I'm running a half elf Blood hunter order of lycan in an Avernus campaign.

Every time I try to use Hunter's Bane to get any kind of information about what we could be facing, even if I roll high, the DM rules that I wouldn't know that information. 

I understand, given it's a lot of devils, but also a lot of undead.

So how is it supposed to work? I know it's not that I'm suddenly the encyclopedia britannica for every monster known to man if I roll high enough, but how is it supposed to work?

So, as always in these threads. Let's look at the offical passage on Hunters Bane, and then extrapolate.  We'll try and use official sources to somewhat corroborate things too since Blood Hunter is a homebrew class, but Matt Mercer obviously takes great steps in trying to make things balanced.

Hunter's Bane is both a way of life and a class ability.  It describes the ascent(descent?) into the order, and then of course the ability itself.  Due to your training with the order or your studies into fey, fiends and undead you gain advantage on intelligence checks to recall information about them.

In the PHB, we're given some sample DCs for abilities checks:

When it comes to published material, by and large the hardest DC you will encounter on the Arcana front is a DC 20.  The only thing I can think of higher is in Tomb of Annihilation, theres a DC25.  This isn't to say that others don't exist, that's just my memory.  Arcana is typically what's used to identify magical monsters, but I could see History or Nature being used too.  Regardless, any of those checks when it comes to recalling information specifically about fey, fiends and undead SHOULD be given advantage.  

Without going into detail about your module, I could understand your DM saying you don't know anything.  At the same time, being denied a major feature of your class blows.  Talk with the DM and ask what their understanding of the ability is.  Let them know you aren't looking to get HP totals, obscure resistances etc, but your class literally studied demons(fiends). Do the marks in the wall look like they came from something unnatural?  Do you get some insight into the habits of how lemure demons work, or maybe what they do to ascend the demon heirarchy? They know some stuff.

Thanks for all this!

I texted him yesterday and am waiting on reply.

Having studied these fiends/fey/undead, would it be mechanically okay if I rolled high enough I would recall their weaknesses and resistances outright?

Like if the DC would be 15, I'd be able to know something/get SOME type of hing if I met it, a little less and less the lower I rolled, then anything exceedingly I'd recall it like a photographic memory?

The only nerf I'd put to this would be for any big names and bosses I'd run into. I'd only get so far as their story, but that would be harder.

Like if the DC would be 15, I'd be able to know something/get SOME type of hing if I met it, a little less and less the lower I rolled, then anything exceedingly I'd recall it like a photographic memory?

Not always.  Meeting the DC just means that, you met it.  To put it in a thieves tools sense, if the DC is 15 to open the lock but you got a 27?  You open the lock.  You could flair it like you had your eyes closed not paying attention, but the end result is you still picked it.  When it comes to knowledge rolls, its either you know or you don't. 

Flair could be used to do other things if the DM is feeling friendly, but going back to the Lemure Demon.  They are immune to Fire damage.  If the DC to know that was 15, and you rolled a 14?  You wouldn't know.  If the DC was 15 and you rolled a 25 you don't magically get to know everything about them either.  

Now, for a Blood Hunter specifically to roll say a 25?   Which for a level 1 character is an extremely high INT check AND their class deals with it?  In my game, you know and maybe some extra stuff.  "Since you got a 25, you learned during your original studies and with how you manifest your Crimson Rite of Flame?  You were instructed this doesn't do well on X Y Z, and it includes that demon in front of you."  If the Barbarian rolls 20 untrained?  I'd parse the information differently, maybe how they notice its extremely hot and this demon doesn't seem to be affected.  This is all assuming that Lemure demon, who is a very common demon.  Once this gets to a named monster?  That shit goes out the window.  DCs go up significantly or are potentially non-existant.  There's no need to self nerf an ability where the reality is the knowledge just doesn't exist without some way to research it.  Checks also change too, maybe we're no longer doing a standard Arcana roll, but rather a history roll to see if the name of this person rings bells? 

As a DM in that instance, how I play it might change.  If we spent levels 1-8 in full on exploration mode, with no chances for true downtime?  What you knew at the start + what you learned is what you know.  Period.  The campaign that sits in a major city hub with access to temples, wizards, experts etc?  That person will naturally pick up other things because they have access to other information.

There's no set answer for any of it because at the end the DM is the arbiter of that world.