Dear dungeon masters,

I had my party follow a whole story arc which took like 6 sessions to complete which ended in a boss fight with their first creature using lair actions and legendary actions.

We did this fight but I gave the creature too many strong allies which resulted in the party loosing and feeling that they didn't have a chance of winning. Eventually this made me feel really shitty about ending an important story arc like that. Because of that i decided to redo the fight starting at the point when initiative was rolled, giving the party once more the option to resolve this encounter through role-play as the creature gave them the option to bow down to him or otherwise be destroyed. 
The party still decided they wanted to fight the creature and this second time around they defeated it as it only had two stronger allies and had a lair action that could summon minions. (creatures with a set AC, 1 +4 1d6+1 damage attack and just a single hitpoints so i don't have to pay attention to how much hp they had)
This fight was still tough and the paladin went unconcious, the bard was paralyzed for 3 rounds because of a hold person spell, most players got under 20 hitpoints left after the fight (7th level players btw) and overall this second fight ended more satisfying and everyone is happier with this version of events.

The problem is, I'm afraid the party might see this as something that could happen more often. Im not planning on doing this again but im afraid that lets say, a character dies, the party would want to redo that fight. Have I opened a wrong door here? 

In the first fight they were on the defensive from the start and eventually dicided to try and retreat, but it wasnt looking too hot for them.

Did i make a mistake by doing this?
- thanks

First of all - if you made a mistake, then all I can say is: we all do, it just makes you human. Mistakes will be made. The only thing you can do wrong with mistakes is refuse to learn from them, and/or refuse to try and fix the damage.

Also - all I can give you is my perspective, and what I think I would have done in your place. That doesn't make my advice right :)

First, dynamically balancing combat situations, in a way that's fair, and invisible to the Players, is a tough skill to learn. It's essential that every good GM learn it, but it's hard to master. It probably would have been better to do that then to give the Players a "redo".

Secondly - provided that you've had a Session Zero, or a "talk" with your Players about the acceptabilty of the Party outright failing and/or Chracters dying - I would have let the Party fail. That doesn't mean they lose the Campaign. That doesn't mean a TPK. It isn't the end of the Adventure. They just failed to defeat the "big bad" this time. They've showed their hand, they've let the BBEG know they're gunning for them, and then the BBEG has kicked their ass. It means that the BBEG will react to that. It means the BBEG will be stronger next time. It means that the Party will need more power, allies, resources to defeat the BBEG next time. Great! Now you have new adventure arcs for them to play through: finding items, finding allies, finding new ways to erode the BBEG power base, etc. Eventually, they've have power and resources needed to go after the BBEG and win next time.

As for Table management, I'd sit down and talk to your Players. Admit you screwed up the Encounter - it was way harder than you thought ( and that's not something to beat yourself up over; that has happened to every DM who has ever stuck with the role, trust me! ), so you gave them a "do over", but that it will not be general policy. In fact you'll probably never allow that again.

 Just communicate with your Players so you all understand how things are going to run, from here on in.

+1 to vedexent on the everyone makes mistakes, and overall.

I would say the re-do was the mistake, more than the tough fight. Besides the fact that they may now expect it, it seems like it would make that second fight’s victory feel pretty hollow. And you missed out on the chance for a recurring villain, which is always fun to have. 
Losing a fight doesn’t mean they lost D&D, in fact it can make the game better. Think about pretty much every movie or story, the hero doesn’t beat the villain the first time. The hero either fights to a draw or someone narrowly escapes. There seems like there’s this expectation that the PCs should win every fight, but I’m personally not a fan of it. Sometimes they need to run into something tougher than they are and have to flee.  Now they have a tough enemy and a reason to really hate it. And the enemy knows them and that they’re coming, but maybe gets cocky and doesn’t take them seriously. 
So the party goes back, licks their wounds and comes up with a new plan to beat the BBEG. Maybe they level up or get new gear. Then they go fight, and now they’re stronger and they win. How good does that feel, after they got their butts kicked, to walk in and win? It would be super-satisfying, and help them realize how much more powerful they’ve gotten. 

I'd say the root of anything that could be a mistake was misjudging your players. I've run/am running campaigns where if the players lost that fight the reaction would be something along the lines of "Holy crap, that was awesome. I can't believe we got our butts handed to us. We're going to have to figure some stuff out and come back" There are other campaigns where the players would be crushed and would end up feeling like they have no direction. So aside from some good advice given above I'd tailor future encounters to give the players more of what they want. If having crazy close calls and potentially having characters die is something that's going to make your group unhappy than work with that. The overall point is for everyone to have fun. 

Don't be too hard on yourself,  we all make mistakes. As Ved said, we just try to learn from them to hopefully reduce them (or make different ones) in the future.

I don't like the idea of "redoing" a fight.    I think letting the BBEG win and become a recurring villain, or a goal for the chars to prepare for and go after would be what I would have tried.   That also reinforces that the party may need to prepare differently, or perhaps gain more information before some encounters. Again, this is hindsight though, and in the spur of the moment we all make mistakes. 

I would say yes, but everyone makes mistakes, so it's fine! The thing is, if the players realize there's no chance of losing, the game no longer has stakes, so it loses its excitement. Yet if you don't let them do "redos" in the future, they could feel cheated, like you pulled the rug out from under them. At this point, probably talk to your players and let them know that while the redo stands you won't be doing it again. If they're mature adults (or teenagers!) they should understand, especially if you mention there's no success without the potential of failure.

I don't think you should re-play battles. If you want to retcon something, there are other ways. For example, say the evil boss-guy kills the party. Instead of being "dead," they could wake up in some outer plane like Tartarus or Mount Olympus having been rez'ed by some god or demon or some powerful entity. This powerful entity perhaps is also an enemy of evil boss-guy but maybe some cosmic rule prevents him from taking direct action. He then offers a deal. I will send you back to the mortal world alive again, but you must defeat this boss-guy for me (which they will no doubt want to do). But I don't tolerate failure, so find a way to defeat him this time. If you lose again, not only will you die for good but this time *I* will punish you in the afterlife... etc. etc.

Quote from BioWizard >>

Instead of being "dead," they could wake up in some outer plane like Tartarus or Mount Olympus having been rez'ed by some god or demon or some powerful entity. 

 I like BioWizard's idea of having the heroes resurrected by the gods.  Only problem with it would be if the party feels it will happen again the next time they have problems in a fight.  A TPK happens, but there are ways around it.  Going with the resurrection idea you could have the defenses of the BBEG shattered and the party can get in much easier this time.  However, the BBEG now knows more about the party and can use that knowledge against them.  Other way of doing it would be make the party come back years later.  Not Samurai Jack time frames, but enough so that they now have a little more work and you can stretch the game a few more levels.

As long as balance problems do not happen as often, no problems.  One thing you could do is try to do little one shot games.  One of my DMs had gladiator combats with both PvP and PvE where he would run just a straight out fight with quickly made characters.  This way you can throw different amounts and types of monsters and see how balance works.  If you Kill Them All, no problems they were throw away characters even though they were their characters.  If they win, you know what threats the players can handle.

As a player, I don’t mind failure. I think it’s good for storytelling, you regroup and come back stronger and take them down later.

Actually in our Session Zero one of the players said that of all the things he's never done in D&D that he would like to do, it's to legit lose to the bad guy... and then have to play out living with the consequences.

The group I’m in did that before I joined them. The subsequent campaign was basically in dark sun, from what they tell me. 

I mean, saying whether the redo is a mistake or not is kinda moot because it already happened.  I'd just use this as an opportunity to tell the players that you did the redo because you're new and still learning how to do this yourself, and that as you become more experienced it will become less of an option going forward because by then you will have better knowledge about what they can and cannot handle, so they shouldn't expect that to be a regular thing.