First, if you’re in my campaign, Curse of the Idir, please stop reading.

Ok, now context. My players are confronting some pirates that have been enslaving and selling the people on the ships they capture. The captain is a lycanthrope worshipper of Malar and I built him with death cleric abilities to be a boss for the whole party to fight. In a really cool story moment, our swashbuckler challenged him to a duel (partly for his ship, but mostly for revenge for a former pirate he cut down).

So the question: do I keep the boss as is and see where the dice take us? It was an amazing story moment and RP choice that I’d like to encourage, but I’m hesitant to dumb down an enemy that I’ve built up as a major threat. What would you do?

Depends. In theory, you could introduce 'pirate code' 'duelling rules', e.g. "no magic", etc.

keep him as is, and see how it plays out. if you want you can grant your player an inspiration die (a free d20 to use to reroll any check or attack he makes) since it was such a cool and inspiring moment in the story. This should give the player a slight edge. Also, let the party have a few minutes to prepare for the duel. hopefully there's a cleric or bard in the party to grant bardic inspiration or guidance or some similar advantage, give each party member 1 action to help prep the swashbuckler.

Don't give the Villian any similar buffs even if he has the ability to use them.

with some smart party work and the inspiration, the PC should have the advantage. but this is a very dangerous situation death should be on the table 100% and nerfing your Pirate Captian would diminish the RP path your PC went down. 

Is the duel to the death? If not, the player should only need to knock him down to half HP or get into another situation where he would yield. Perhaps he can also pop up later in the campaign?

As a malarite, this is a duel to the death and I definitely gave him inspiration for challenging him. These are excellent suggestions! I’ll put the offer on the table, but my player took a few spellcasting levels so he may not accept.

I’m open to other ideas, too!

I let my players face something similar once. Two of them faced a challenge that would have been tough for the entire party at full strength. Fortunately the big boss was arrogant so he had his minions knock them unconscious instead of killing them and they woke up as slaves and when the rest of the party was there the following week they joined in when the fight happened. It worked out because the boss was arrogant and it made for two fun sessions back to back. 

This is the equivalent of jumping down a dark shaft not knowing where it leads in the Tomb of Horrors and landing smack dab in the middle of an Ad&d Sphere of Annihilation.  A classic blunder.  If your world has integrity and isn't some zoned MMORPG like game there is no reason that this Pirate cannot simply accept the challenge of the Swashbuckler and given the relative power disparity demolish them.   They don't have to kill the swashbuckler, the PC is beneath that priveledge.  This gives the BBEG a chance to give a gloating monologue over the swashbuckler as the makes his Death Saves.   I would also suggest leaving the Swashbuckler with a lasting and dramatic scar, a memento of his brush with death.

Well, might I suggest that something interrupts the duel before either side can win? The result of which is that both get an opportunity to size up the other and make for a final climatic showdown later on. I don't know if he's an honourable villain, but he could maybe poison his weapon and give the other players a chance to investigate this and let the duelist know that his opponent isn't honourable and so that he shouldn't go ahead in a contest that's rigged against him. Or maybe a giant sea monster attacks the ship, seperating the duelists.

Because if the boss is designed to go toe to toe with the whole party, then it is rigged against him, so you might as well give him a story reason to get out of there. It will make the moment when they lock eyes later on, when he knows the villain's time has come, that much sweeter if they're had to chase the villain down a bit.

Can the player realistic win, or do the dice have to be heavy in his favor?

I would try to make it so that he is only 1-3 levels higher than your player, that would mean he should be a deadly encounter but still winnable assuming your player plays smart and rolls good. Or take what Mercer did to talisman. That was an amazing story moment. Not one for the players, but it taught the whole party a lesson.

Malar is the god of hunters; not duellists. The captain will accept the duel but will not in any way feel constrained by any "rules of duelling." He wil, for example, have no isses ordering his crew to help.

He or she will also assume the player characters will not be constrained, and may be confused or surprised when the rest of the characters let the swashbuckler fight unaided. 

As a GM, you could use his willingness to get his crew to interfere as a clue that the players chould do the same.

If they don't take the hint then the captain beats the swashbuckler into unconsciousness (the non-fatal blow rules can be used by the GM as well as players) and then proceeds to attempt the same with the rest of the party.

I would say maybe this duel has some self imposed limitations or a code of conduct, as some others have suggested. 

Now, if you don't think that's in character for a pirate werewolf, maybe here's a workaround: 

They don't follow the dueling rules as part of a code of honor, but as a show of strength to their crew. The old "watch me beat this guy with one arm tied behind my back" gambit, where the powerful villain wants to show he doesn't need the full extent of his power to squash this pesky mortal. 

It's of course up to you whether he sticks to this charade if the duel goes south for him. It could be that the second he thinks he's in legit danger, he calls it off and orders his men to attack, the party joins the fray, and you get that full party boss battle you expected. 

If they are foolish enough to fight the boss before they are ready, perhaps give them some hints that it won't go well.

If they proceed,...let them learn from their mistake.

Malar is a CE god so I don't think using a code of conduct would really fit, its more about dominance and survival of the fittest. I'd go down the route of "You challenge me, I won't kill you but WHEN I defeat you, your life and physical body is mine and i'll sell you as a slave to the deepest, darkest most dangerous mines I can find and your soul belongs to Malar"

A few thoughts:

• Good on the player for suggesting an unexpected and out of the box solution to a problem. That alone would make me want to give careful consideration to agreeing to it.
• Think like the pirate. What does he have to gain from the duel? What does he have to lose?
• You are not limited to "yes" and "no" in your response. I heard a rumor that pirates have been known to occasionally dabble in treachery :)
• If your pirate agrees, but plans to be treacherous, then perhaps agree to the player's terms, but be overly accommodating in his tone of voice. If the players want to try an insight check on him and if they see through his ruse, then so much the better.
• I think you should not tone down your pirate captain for the duel. Play him at the strength you imagined for him.
• Remember that players can also be treacherous and if the fight is going badly, they may not play by the rules either.
• Throw the question back to the players. Why should he duel them? What do they have that he might want that he cannot take by force? Why shouldn't he just order his crew to capture or kill them right then and there?
• If the fight begins to turn against the pirate, what will his crew think? Will they jump in to aid him? Or will they see his weakness and turn on him?
• If the stakes are high and the fight begins to go against the player, how far are you willing to go? If the pirate knocks the character out, maybe he pauses and looks at the party. "Any last words before I send this scalawag over the rail?"

Just remember that when the players throw a curve ball at you, they are giving you a gift. Make the most of it.

Thanks again for all the responses!

The captain was in the middle of convincing other NPC pirates to forgo their traditional ways and join his armada so that they may take down not just mercantile, but naval ships as well. Then the PC challenges him to a duel. For the captain, this is very much a good way to prove his strength and get other pirates to his cause. He gained his position through mutiny and defeating any that challenge him further proves he’s the alpha pirate, so to speak.

The captain is a devout of Malar, but does not follow the typical dogma (as the death cleric of Malar in the party already noted) and likes the prestige and attention from being in charge. This captain likes to take hands as prizes to decorate his ship before selling the victim into slavery (to yuan-ti as it turns out). If my player loses the duel, I could have the captain (threaten to) cut of their hand, gloat, and give the rest of the party a chance to intervene?

Just to be clear, I think what my player did was really cool. I guess my overall question (which many of you have answered) is how to reward his story decision without pulling punches.