So I'm still a relatively new DM and I gave the item linked below to my party. Well I made that item thinking about the cool factor way too much and yeah in the hands of the barbarian getting buffed by the bard to almost guarantee a hit, it is way too powerful.

The barbarian dumped 30 health into the bow and hit for 60 plus standard longbow damage. The party of four level 5s was facing a lone young white dragon, so I could have done better balancing the encounter. I have never dm'd a dragon before so I didn't want to tpk them. 

Is this item still okay in your opinion or should I find a way to make it weaker, should I take it from them, should I have it be feeding an ancient demon the more they use it, so there is an implied cost for using it too much? Any feedback would be greatly appreciated. 

https://www.dndbeyond.com/magic-items/2431679-longbow-of-the-blood-pact

That item should probably be Legendary Rarity and would likely be okay for a party of 12th level or higher. Whenever homebrew is involved, telling the players that those homebrews are “subject to change” will save you some headaches down the road.

Here are three suggestions I have:

1. Have the item require a charge to use it and have the charge refresh at dawn. This way he can only use it 1ce/day.
2. Have the item specify that only a number of HP up to the character’s level can be sacrificed. This way it will keep it lower now, but go up as they reach higher levels.
3. Instead of having the character sacrifice HP, instead have them spend hit dice to increase the damage by the amount rolled. Since a PC can only recover half of their Hit Dice each long rest that will require a bit of additional resource management.

I hope some of that helps.

PS- To clarify, I don’t mean to use all three suggestions at the same time, I meant those are three completely different suggestions of ways to rebalance the item.

Quote from DamienSteinrich >>

So I'm still a relatively new DM and I gave the item linked below to my party. Well I made that item thinking about the cool factor way too much and yeah in the hands of the barbarian getting buffed by the bard to almost guarantee a hit, it is way too powerful.

The barbarian dumped 30 health into the bow and hit for 60 plus standard longbow damage. The party of four level 5s was facing a lone young white dragon, so I could have done better balancing the encounter. I have never dm'd a dragon before so I didn't want to tpk them. 

Is this item still okay in your opinion or should I find a way to make it weaker, should I take it from them, should I have it be feeding an ancient demon the more they use it, so there is an implied cost for using it too much? Any feedback would be greatly appreciated. 

https://www.dndbeyond.com/magic-items/2431679-longbow-of-the-blood-pact

 Yes, it's OP, but it can be balanced. First, you should make it so that HP lost through the bow reduces their hit point maximum until the finish a long rest, too. Second, make the damage equal to half the expended HP. Third, give it three charges, and have it regain 1 charge at dawn so that they have to manage their resources. 

Those are pretty amazing ideas! I think I will definitely impose the charge limit. But because they have used it twice in a day before I may say that there is an awakening occurring the more its used he has felt it growing stronger during the times he has used it twice in a day. So that he understands that maybe he can use it twice but that might not be the best decision if he values his life. I also will not allow a third use. 

Just double checking: A longbow is a ranged weapon that must use dexterity as a modifier. You can only use Reckless Attack on attacks made with Strength.

True and they haven't tried to reckless it. I just mentioned barbarian because that health pool. 

Eh, the easy way to limit the item is to cap X. A limit of around 5 shouldn't be a balance issue as a Rare item.

Quote from Pantagruel666 >>

Eh, the easy way to limit the item is to cap X. A limit of around 5 shouldn't be a balance issue as a Rare item.

 Or Constitution modifier.

Another approach is to have the magic item (bow) use this encounter as an "advertising special" to convince the character to keep it. Then reduce the stats to what you want them to be until the bow or the character levels up by completing a quest line of your invention. Perhaps one of the Candle Keep adventures?

You can use a stepped approach or have it be an all or nothing arrangement. Have them discover lore about it to increase its abilities. Then you control the weapon advancement. Weave the item into you world as a set piece.

Quote from WilsonJ >>

Another approach is to have the magic item (bow) use this encounter as an "advertising special" to convince the character to keep it. Then reduce the stats to what you want them to be until the bow or the character levels up by completing a quest line of your invention. Perhaps one of the Candle Keep adventures?

You can use a stepped approach or have it be an all or nothing arrangement. Have them discover lore about it to increase its abilities. Then you control the weapon advancement. Weave the item into you world as a set piece.

 I like the way you framed that. It would give me control without having to retcon what has already happened. 

I say don't be afraid to just tell your party, "Hey everyone, I didn't realize this weapon wouldn't be balanced when I homebrewed it. To keep it from breaking the game, it's going to work differently going forward." I know it can be unsatisfying to defy the canon established in your game that way,  but it's good to set a precedent that, occasionally, for the sake of the game, some things will change without requiring any kind of complicated in-world explanation. Homebrewing is complicated, and even a lot of official content finds itself horribly unbalanced (did you know Winged Boots are considered Uncommon, but the much-less-useful Boots of Levitation are considered Rare?). 

Of course, you've gotten some pretty clever suggestions for how to address the problem in-story. But even if you do go for one of those explanations, I still think it's best to be honest about why you're making the changes, and don't just spring it on your party and try to pretend you were planning this change all along.

Quote from TransmorpherDDS >>

I say don't be afraid to just tell your party, "Hey everyone, I didn't realize this weapon wouldn't be balanced when I homebrewed it. To keep it from breaking the game, it's going to work differently going forward." I know it can be unsatisfying to defy the canon established in your game that way,  but it's good to set a precedent that, occasionally, for the sake of the game, some things will change without requiring any kind of complicated in-world explanation. Homebrewing is complicated, and even a lot of official content finds itself horribly unbalanced (did you know Winged Boots are considered Uncommon, but the much-less-useful Boots of Levitation are considered Rare?). 

Of course, you've gotten some pretty clever suggestions for how to address the problem in-story. But even if you do go for one of those explanations, I still think it's best to be honest about why you're making the changes, and don't just spring it on your party and try to pretend you were planning this change all along.

 Thank you. I've kept them informed. I am a firm believer in communication. They would probably have just been cool with me changing it but I wanted there to feel like a reason in game as well. Thank you for the input. 

I’d say don’t do anything to it just make it a plot hook. It is feeding some imprisoned fiend and when they’ve used it enough welp he’s free and it doesn’t work anymore. Oh and by the way he’s gonna end the world or something. Guess they have to do something about him. Now they have a whole different story to follow and it’s waY more fun than the DM nerfed my item.

Wrapping it in the plot is a good way to go. They are sacrificing their HP so as long as you don't just throw one bad guy at them each battle, they should reconsider spending 30hp to kill someone.

One option is to have an NPC talk to them and say that they should give the bow over to their holy order which protects the world from things like this. Then, if they don't, have the BBEG hear of this powerful bow and send people after it. Thieves in the night, outright attacks, people spiking their drinks, etc. Then, when he eventually gets the bow, they will be rightly scared of the BBEG!

You can also add a curse - they shouldn't know about it so pick an event that hasn't happened yet (EG using it 3 times, or a total HP to sacrifice) and make something happen - perhaps the demon the bow was feeding awakens. The bow might then split, or the PC might see a hand materialize on the bow, then an arm, until the demon is stood right beside them. Maybe the bow awakens as a leech-like demon creature which attempts to drain the life from the party and escape, leading to a spate of mysterious deaths the party has to investigate.

You could always make the bow cursed and a soul eater and every creature killed with the bow has its soul devoured or sent to the lower plains or something. Once the party figures that out then only an evil creature should want to continue using the bow. Maybe a BBE steals it and the party has to go on a quest to recover and destroy the foul artifact before it can claim anyone else.

Track the HP he’s put into the bow. Add that HP to the Fiend that’s unleashed later. Give him nightmares that that his beast is growing every time he uses it.