I am the DM for a party of 3 (Totem Barbarian/Fighter, Aberrant Mind Sorcerer, Twilight Cleric) all 10th level. We've been playing for over 18 months and are reaching the BBEG of arc II. The TL;DR is the players need this about to face off with the main protagonist of this arc, from whom they need information and help — naturally the BBEG is not inclined to give it.

I have been homebrewing the stat block for the BBEG and struggling to properly balance it so that it is challenging for my party but also gives the BBEG some out-of-combat options, as well as some lair and/or legendary actions. In past, they have bought high-level spellcasters and wiped the floor with them. Any suggestions or guidelines for how to balance the BBEG's HP/AC, with spell-casting, with legendary/lair actions to make it a dynamic encounter? 

Give him minions, give the BBEG a reaction called "Human Shield", he can use a reaction to use a minion that is within 5ft as a meat shield for any spell or attack that requires an attack roll. The minion takes the full damage of the attack. 

I'm not an expert on this but I would say giving the boss some sort of counterspell works great against the player's just slogging your boss with high-level spells.  

Take a look at how hard your players can hit and how easily they can hit. I'm not sure how to say this but make sure your boss can survive a few of the really big damage attacks from your players. Remember you don't want the boss to just fall over but you also don't want them to be unstoppable. look at the above-average damage all of your players combined can do in a single round with their big attacks and work around that for health. you want the boss to survive long enough to where it would be difficult but manageable with good teamwork. This is just my insight to how I would go at this situation. 

Anti-magic fields for the lair could work. specific areas during the rounds get put under a special effect where magic cannot be cast nor concentrated in. This could also be used in a way to draw attention away from the boss. Using pylons or crystals to power these anti-magic fields could create a difficult atmosphere. 

I like that idea — the character is an artificer/alchemist of sorts, so it could be interesting to flavor it as some sort of minion automatron.

Thanks for the advice — anti-magic or protection-focused legendary actions could be a great way to extend the fight out longer as needed.

I sometimes struggle with the average damage of my party since one is a barbarian/fighter, can take and give a lot of damage; while our sorcerer is very weak in both AC and damage output, usually focusing on control spells. The anti-magic fields could be an interesting way to separate focus of the melee vs. spellcasters of the party and give them each somethign they can contribute

I suppose you make specific parts of the battle favor the Sorcerer. making the boss vulnerable to certain damage types at specific times during the fight, maybe after destroying a pylon. you could also make the boss have resistance to standard non-magic melee attacks or give them an ability that does has a pushback effect. a rooting effect could also work to hold back the Barbarian. 

do you have any what the BBEG is supposed to Look like? if not, FEAR NOT ADVENTURER, i have some ideas >:)

1. an cogwork machine, like a computer, with a giant rotary dial at its center. every tme it is hit, it become immune to that damage for the next round, but vulnerable to something else, as shown on the dial
2. some kind of brain kept alive by arcane fluids with various spells for battlefeild manipulation like Reverse gravity, Telekenisis, and Teleport
3. a swarm of insects in human(oid) for that is immune to spells or abilities that only target one creature
4. a human wizard obsessed with chess. the battlefeild is a chessboard with the wizard in the king. the wizard can control the various (giant) chess peices to try and crush the players

With an artificer in particular, the minions could be not creatures. Maybe they set up some turrets that shoot at the PCs. Or the turrets have some kind of damage aura (and the artificer is immune to that kind of damage) it doesn’t have to be a lot of damage, just enough to make the characters weigh wether or not to attack the turret or the artificer. 
And as a fun bonus, they’re objects so there’s many spells that can’t target them. 

Really the whole idea is to make it so not every PC’s attacks are going after the boss. Spread out the characters’ damage.

There were these time clockworks, and being so close to them i think 15', the characters could only do one action, no reactions,  so if moved that is all.  if attacked that is all, no bonus actions.  It would effect the time around them almost like a slow spell continously.    They attacked twice lots of hp.  But you can form it to whatever you want.  Sorry I do not remember the name of them.  Seems I saw Matt Colville introduce them.  

they are called Inexroable   they would go well with your BBEG.  

This is awesome, thank you!