The party that I am DMing for right now has a very strong tank line. The party is level 10 and consists of a barbarian (totem), a barbarian/fighter (cavalier), a fighter (EK), a ranger (gloom stalker), and a bard (glamour). 

I want to find a way to challenge my party with our encounters, and am trying to find solutions that don't just involve me sending bigger/stronger monsters at them all the time. One obvious answer is to use monsters/NPCs that have abilities requiring mental saving throws, which I intend to try, but with the bard as support for the party I worry it may not be enough. Also, I don't want to rely on just one "trick" to challenge the party.

So I am looking for input on how other DMs have been able to come up with encounters that are challenging for parties with alot of HP, AC, and physical might. 

(it is also worth mentioning that the party has a few magic weapons, so overcoming resistances to nonmagical BPS damage isnt an issue) 

Maybe add more environmental hazards and focus more on the environment in general, difficult terrain and that kind of thing? Maybe use more ranged opponents? Or, maybe you could take the focus away from combat and do more social or exploration-based adventures? I haven't DM'ed at that kind of level before, so I'm probably not the best for advice on this kind of thing.

Make the monsters be in such situations where they are a great distance away or can only be affected by magic skills, or cannot be damaged by normal weapons like swords, and axes. That will get them thinking outside of the box.

You could try the weeniehoard approach and hit them with a few waves of ankle biters, Kobolds, Goblins, Skeletons, that sort of thing.

Have the monsters get their own magical support. A number of cultists supported by a couple fanatics, an acolyte or two, and a priest makes for a fun encounter. But there are other magic users to throw at them too.

But the best way I have found is to wear them down over several encounters per Long Rest and then by the last one they really feel up against a wall. None of those fights were necessarily that tough, but four of them in a day and it starts to wear on them, really gives the impression of an exhausting day of grueling fights.

Simple. You are trying to play the Monsters strengths against the Party's strengths. 5e rules favor the Party in a "balanced" encounter. That's a losing proposition.

Instead, pit the World's strength's against the Party's weaknesses. You're already on that path with your idea of mental based abilities for the antagonists.

The Party is strong in Combat. Give them an Encounter which needs to be resolved by means other than combat.

An Encounter is not synonymous with Combat. An Encounter is a problem, or situation where the Party is opposed by antagonists, and the Party may try and resolve with Combat or they may try and resolve it by other means.

Clearly this Party is heavily focused on 'solve it by hitting it with a big stick' - give them an Encounter when stick hitting isn't a viable tactic, or at least isn't a central one.

That doesn't mean you can't have Combat in the Encounter - just that the means of solving the primary dilemma isn't centered on the Combat.  Maybe the tanks need to hold off the Black Guard long enough for the Bard ( since you don't have a Rogue ) to scale the platform, work out how to dispel the enchantment of the damsel in distress's manacles, and rappel down with her in tow. The encounter is then about the Bard's problem solving abilities, and skill rolls. They win, or lose, depending on whether the Bard can get the Damsel free before the gunpowder kegs explode ( apparently she's been kidnapped by Snidely Whiplash :p ).

In that example, it's never in doubt that the tanks will eventually defeat the Black Guard. The Black Guard doesn't need to be an existential challenge for the tanks. But can the tanks win fast enough? Can they juggle the tactical situation so they can take out the Black Archers before they start sniping at the Bard and ruin the rescue attempt? It gives the tanks a valuable support role in the encounter, even if solving it doesn't center on them.

Don't make the encounter about Combat, even if it contains Combat. Build the encounters so whether or not the Party wins hinges on other skills.

Best of luck :)

The obvious option is to just use multiple monsters -- 4 out of 5 of those characters are basically single target attackers, which means they will handle multiple monsters poorly. Though if they have a lot of AC; maybe something like a hell hound pack (5-6 of them is a medium encounter).

Action economy. Matt Colville has a good video on making encounters more interesting. He basically suggests giving villains bonus actions, extra reactions, and villain actions (kind of like layer/legendary). 

The intention is more for boss type encounters, but you could generalize. 

An example encounter in my campaign:

The Wight Brothers: one is an inventor and likes to experiment on live humans to create new zombie variants. The other is a Hannibal lector type and tolerates his brother, but really just wants to see the world burn. They have saved their baby brother (still baby, still human) who the first brother wants to turn in some way once he is older. Given the chance, the second brother would negotiate for the party to kill the first brother and the baby as he finds the idea absurd.

So now you have 2 wights and a bunch of zombies (regular, feral, abominations, and Ichor zombies of my own concoction) in their lair.

feral zombies move faster and get dual claw attacks

Abominations have a crushing grasp that is hard to break from and does auto damage

ichor zombies are like feral zombies, but if they claw you or if you hit them and fail a dex save you get their ichor on you which does auto damage until you burn a bonus action or reaction to wipe it away 

The wights get:

- bonus action to raise one zombie in near proximity

- reaction when a zombie dies to area of effect life drain near where it fell, reallocating those HP as they see fit to themselves or other zombies

- villain actions (each 1/round after any player): animated decay (works kind of like entangle), enrage zombies (all zombies in range get 1 free action), horror (forces save vs a list of nasty horror effects). 

The key to lots of encounters at his sort of level is depletion of resources. Throw some fodder at them so they burn through their spell slots, preferably their higher level stuff. Then throw some more. Then maybe some more. Then bring in the real trouble. It's all about waves. AOE spells are handy against fighter based, because it's reasonably easy to clump them together, particularly if your bad guy spellcaster doesn't mind some friendly fire.

Next up: PC's at this level would have attracted a certain level of fame (or infamy) so bad guys would be aware of some (most) of their abilities. So don't feel bad about tailoring end level type guys to combat their particular skills. I had a sorcerer who LOVED fire spells. It was kind of his trademark. So the bad guys lured him into a warehouse they had prepped with barrels of oil, and sheets hanging from the rafters doused in oil. A fire spell in there would have been a VERY bad time for everyone involved.

So, let's look at a full encounter-day (part of the problem with challenging PCs is that most people don't use up their daily budget). A party of 5 at level 10 has daily budget of 45,000 xp; if we split that into 6 encounters, that's an average of 7,500 xp per encounter.

Let's start with a theme. Hm. We should be looking for an end boss that's a legendary in the CR 13 range, that suggests a Adult White Dragon, Beholder, or Vampire. We'll go with a Vampire. Now to build a day of encounters:

1. 2 x Vampire Spawn, 5,400 adjusted XP. This is an easy encounter that should actually be easy, it's just to introduce the problem.
2. 20 x Swarm of Bats, 4,000 adjusted XP. (9,400 total) This is actually super mean because swarms are just straight resistant to bludgeoning, piercing, and slashing, have 22 hp each, are immune to a huge number of conditions, and is permitted to stack, so they'll all be able to attack. Unless the bard used bardic secrets to get an area damage spell, it's going to be a huge grind.
3. 4 x Ghost, 8,800 adjusted XP (18,200 total). Huh. I don't see much in the way of Charisma saves in that party. Figure two of them get possessed. They aren't super dangerous since they can't use class features, but I don't see any likely ways for the party to deal with possession, and beating them senseless will eat up lots and lots of healing. If you want a bit less annoying, swap two ghosts for a Banshee and a Wraith.
4. 12 x Dire Wolf, 7,200 adjusted XP (25,400 total). Just straight up fighting, they should do alright.
5. 1 x Flameskull, 1xBone Naga (spirit), 1xVampire Spawn. 8,000 adjusted xp (33,400 total). Frankly, the vampire spawn will be nothing more than a meatwall, all the damage will be Fireball and Lightning Bolt
6. 1x Vampire. 10,000 adjusted xp (43,400 total).

Give them a job escorting civilians. The party may be able to casually steamroll through any challenge, but when they've got a bunch of un-trained, squishy allies to keep track of it will at least force them to think more about combat. 

1) Mixed opposition including melee, ranged and casters.

2) Encounters in waves. Not every opponent appears on the field at once. Others enter on later rounds from different directions.

3)  Throw in some shadow monk opponents that target the bard in the back. Engage the front line, these guys teleport next to the bard and stun him. Suddenly no healing, no crowd control, no support (unless the gloom stalker took cure wounds).

4) Synaptic static ... are any of these guys good at int saves?

5) Banish ... only the bard is likely to be good at charisma saves.

6) Hypnotic pattern, dominate person, slow ... there are TONS of spells that will wreak havoc on this party.

Cast spells from beyond 60' in case the bard has counterspell.

------

Someone suggested an encounter with 2 vampire spawns for this group ... I don't think this would be anything more than a pebble in the road for a group of level 10s. I had a group of 6 level 4 characters defeat 6 vampire spawns though they did have the advantage of terrain and the right spells (protection from good and evil) to make the fight more reasonable.

Quote from David42 >>

Someone suggested an encounter with 2 vampire spawns for this group ... I don't think this would be anything more than a pebble in the road for a group of level 10s

 That was me, and it wouldn't be (it's an Easy), but also not meant to be. It's an "introduce the problem" fight.