**TLDR: Looking for combat and encounter ideas that are Fey themed for a dangerous party with OP items. I allowed this to happen. Mistakes were made**

So I'm a newish dm. I've been playing the game a couple of years and DMing for almost one. In the campaign I'm currently running I'm running into an issue where my party is a bit too well equipped/balanced for the combat/encounters I'm presenting them with. I'd like to throw some stuff at them that's more challenging and interesting, while both remaining thematic and not simply raising the enemy numbers and CR. The party is pretty well balanced. And while a couple players are new, some are pretty experienced and brought optimized characters (which I was/am ok having in the game). While they'd be challenging as is, I recently got a little over excited and offered/allowed some significantly powerful items that I know will only make things more challenging. I know I allowed a lot of this to happen so please keep the criticism constructive.

The campaign is a lightly modified game of Wild Beyond the Witchlight. The tone is, I think, pretty well balanced between fanciful and macabre. The default appearance is sunshine and rainbows, but the party is very well aware that there's about 50/50 chance that each thing of wonder is covering up a dark/disturbing nature.

**MAJOR SPOILER WARNING FOR WBTW!!!!*

The main story beats are the same with the coven trapping Zybilna and 

The only modifications are that I am adding Graz'zt as a factor. He is having his minions open/enter portals to other planes to find where she has been hiding. I haven't implemented these hardly at all, but I intend to use them first just as random encounters, then as things progress I intend for them to gently pressure/redirect the party when they're moving slower than I would like or if they're unsure what direction to go. Eventually Graz'zt will appear in a portal and while not revealing everything, try and parlay with the party in an attempt to gain complete access to the Feywild/Zybilna. The only other change is one of the player's origins. The party members each have their own reason to be in Prismeer, but beyond that one of them is an Archfey Warlock. Long story short, she is a sliver/clone/phylactery of Tasha herself. She was given sentience and sent to the material plane, not knowing her origin and that her true purpose is to serve as a backup if something ever happened to Tasha. When Tasha was attacked the only thing she had time to do was send a message to that player begging for her to assemble the party and as for help. Alright enough of the backstory. Here's the party make up. They're all about to be level 5 and close to half way through the campaign (in Tither if you know the module)

A stoutwise halfling Divination Wizard. He took Lucky and has a Luck Stone. The player has been a DM for years and specifically chose the character to mess with me as the DM. Not maliciously. Just to be able to say "Actually no" any time he wants a roll to go a certain way, and using halfling luck, portent, or his lucky feat. In battle he does a lot of debuffing and/or is offensive with Magic Missile, or Cantrips. Additionally I recently gave him an item that's way out of the party's league. The Rod of Absorption. I definitely was misreading when I looked up the desciption. I'm talking to him about possibly changing the item out, or nerfing it a bit. It was supposed to be an Uncommon Item, definitely not a Very Rare one. 

A human Armorer Artificer. He took Fey Touched and Telekinetic, and he has a Periapt of Wound Closure. The start of my problem is that with his loadout he has a resting AC of 21. (I did the math and it's legit). In battle his go to is to Booming Blade with his Gauntlets against the most melee focused character, and use Telekinetic to shove the opponent 5 feet away from him. That way if the opponent wants to reengage in battle they have to move and take BB damage. 

The Dragonborn (black) Lore Bard. He took Gift of the MetallicDragon (I approved, even though he's not metallic), and he has a Rhythm Makers Drum, and Bag of Tricks (though I edited the Bag so that only one animal could be pulled at a time. In combat they're pretty well balanced for a Bard. Usually they focus on support, but they fairly often throw out some damage or use their acid spray attack. The only "issue" I'm having is that they use they pulled animal as a mount and I'm not very good at handling mounted combat. 

The tabaxi Gloomstalker Ranger. She took Piercer and has a Nature's Mantle and Moon Sickle. The only reason she doesn't *completely* ruin my encounters is because she's newer to the game. But still with the broken GS damage in the beginning of encounters, plus the damage boost from her feat she causes plenty of challenges. In the first round of combat she usually takes out a couple minions or deals some SERIOUS damage to one of the lead combatants. For the rest of battle she usually stays ranged and picks off whoever needs to be finished or distracted. 

The half elf Archfey Warlock. She took Shadow Touched and has a Broom of Flying (Though I limited it to 50ft movements speed/height limit and made it cost a bonus action to start). She started off as a weirdly "utility" warlock. With Pact of the Tome and ST she was casting a LOT more spells than most warlocks. But she recently switched to being more Melee focused and might start taking classes in Rogue which I'd be happy to see. In combat she is NOT afraid to spend her spell slots on big hitters like Shatter. And once they're expended she usually dives right into melee. 

In combat this is how things usually happen. The GS sets up surprise and either takes a chunk out of the toughest/most important looking enemy, or takes out one or two of the most problematic minions. Then falls back and snipes the rest of the battle. The artificer charges in going engaging/distracting the enemy who looks like they dish out the most damage. The bard uses their mount to get in range of who they need to debuff/defend. The Wizard either rides the mount with the bard (is that allowed?) or stays in the back with the Ranger, usually focusing on whoever is in charge and or orchestrating the fight using things like Hold Person. And the Warlock moves a ton. Either taking out flying enemies, stays out of enemy range with her broom, and casts Shatter when she can get in the right position to do a lot of damage. 

My thoughts. I'm actually NOT struggling to come up with ideas that increase the difficulty. That's relatively simple. My challenge is increasing the difficulty in a way that doesn't "invalidate" my player's design decisions. I don't want to completely take away from what my players designed their characters to do. This is long enough as it is, so I'm going to make myself stop. You see the TLDR. Fey themed and interested in both combat as well as non-combat encounters. 

Could you share their character sheets, stats block, inventory etc.? That is something I would definitely need if I am designing an encounter thoroughly. I think that will help a lot of the experienced DMs too in getting a holistic view of what you are handling. Features and such will also have to be taken into consideration if we have to design a challenging encounter. By any chance, is Flanking available?

By the way, if you don't want them to be invalidated yet the encounter to be challenging, easiest way is giving higher HP to enemies, with the usual stats block and a higher dmg per turn. You can find a chart in the index that tells how CR is determined, the corresponding HP and damage per turn. You can adjust the damage as well if you need a higher damage output. If they are overpowered in damage mostly, just give higher HP so your foes can act a turn or two. Another way to handle such high damage, instead of using HP, take Hits, regardless of the damage, certain minions need a certain number of hits to be slain, and as DM, you don't have to disclose such mechanics to players. Instead of high HP, you can also give resistance to whatever they are facing. Enemies can be using magical items. Looting them might not give them those items since the items could be destroyed during the combat.

You should also consider the skills/magic on your monsters/foes and how they will use it. If there is a lack of dmg to players, usually the DM isn't using the skills/magic accordingly to the design of the monsters. Apart from that, also take a look at their saving throws and pick enemies that prey on their weakness. Of course, choose accordingly to the lore, location, climate etc. It will be unreasonable to see a creature of the Underdark roaming in an open field for no reason.

The other option, Legendary resistance & action, lair action, various traps, with those, even a single opponent can be tough by itself. Do fine tune accordingly and playtest before throwing those in an actual session. Picking a fight in a confined space might also do the trick, if the encounter happens in a cellar/basement. Your players can definitely fly in the basement, just not that high. And casting Shatter might cause structural damage...is that invalidating the players if the leader of a syndicate hides in a cellar? That is entirely on you. If the fight takes place in the right location with right reason, creativity from players will be the highlight. You give a chance for them to go creative and tackle a problem in a different manner.

In the end, you can always homebrew a foe that has everything against them. Give them a mutated monster with 2 brains, 6 hands, 2 hearts, tough skin in a laboratory with magical resistance from experiment. It's not nullifying their spells, petrifying them into stone or stunning them till next century, just incredibly resilient and fully capable of dishing out damages like a boss. Experiment gone bad is literally how we end up with owlbears, and you can find tons of examples from the lore.

Do share the character sheets if you can. Probably the best way to get a better and precise suggestion.

As for Fey themed, you can easily setup various obstacles to burn through their resources before the fight. Meeting creatures in Feywild will be mostly disruptive; it could be harmless fey beings playing a prank etc.; or having a wall of vine reacting to their emotion; or creatures seeing them as hostile invading their territory.

You can also setup an urgency to advance within time limit like Tomb of Annihilation. That way they have to burn through resources along the way and heading into the battle faster, making fewer rests etc.. Having fewer resources do make an encounter more challenging. The message for help sounds urgent to me.

Getting cursed/conditions or a level of exhaustion prior to an encounter is also a plausible mean. Given that creatures/plants/environment in Feywild do have a potential to cause mayhem for players in various ways and with varying effects. Of course, do allow your players to roll if they can see it coming. This way, all you need is a bad roll from them, and that will make the encounter a bit challenging.

Those are some of the common ways to make encounters challenging. You can easily find those for references from episodes of Critical Role and videos from any YouTube channels.

I am not entirely sure you are asking help to setup challenging random//fixed encounters or challenging final encounter...but whatever you are planning for, those are all the tools you can use and guidelines we have.