Everyone is level 4.

Basically, I have a warlock (Hexblade) with 19 AC, Blur, Mirror image, and nothing that matches their CR can really touch him.
Second character is a fighter (Battle master) that seems to have the ability to deal 30+ damage per turn with a 2d6 weapon.

The other two are decently balanced, a ranger (Fey Wanderer, UA) and a barbarian (Beast, UA) and the challenges seem to match them just fine but get overwhelmed when I set up something with the other two in mind. 

Is there any advice for how to properly set up an encounter or something that would challenge the two power builds without making the two casual builds feel like they're falling behind?

Ah yes, I know what you mean.

I think that every DM finds themselves in a situation just like that eventually.

The best answer for this is to learn the strengths and weaknesses of the players, then build encounters around that.

From what you say, your Hexblade has a high AC (does he have magic armor, or just good stats?), and is able to buff himself really well (on a side note, does he buff before fights, or does he waste two turns casting Blur and Mirror Image?). Maybe a monster has truesight, and neither Blur nor Mirror Image works (don't overuse that or he'll start feeling like his actions don't matter). Maybe a monster has resistance to magical attacks.

Really, you just need to experiment. For now, I'd recommend playing to the strengths of your balanced players.

If all else fails, and you think that your overpowered players are ruining the game for others, just talk to them.

Flying creatures. less single target encounters and more encounters with multiple enemies, have more enemies attack the two more powerful pcs. But like mentioned above, depending on the players, they may not like being picked on if you do it too much. But every once and awhile I'd say it's ok to make an encounter specifically to challenge the two more powerful pcs.

Beholders have "anti-magic cone" I don't know if there are any with that ability for level 4's to fight, but you could homebrew that ability on a Beholder Zombie or Gauth  

Anything with Truesight and blindsight like mentioned above.

Dynamic environments that put the players in situations where they have to expend resources on dealing with things other than attacking. Maybe the ground they are standing on starts to float 40 ft up in the air, pits with illusory floors, stuff like that... The environment is an important enemy as well, I have to remind myself of this often, makes combat more fun in general I think....

AoE attacks should work nicely against blur and mirror image, AoE wont destroy the duplicates but it should still hit the main target.... (correct me if I'm wrong here)

Attacks that force the warlock to make a save instead of making an attack roll against them will be more effective against blur.

I'd probably just make up some quick homebrew monsters applying some of these abilities every once and awhile, that way they can't go look up the monster and argue it's abilities, lol.

you can also buff the ranger and barbarian, making the PCs more equal in power then just use stronger enemies.

It's tough, There's always some like this in our games too and sometimes ya just gotta take it, letting them be OP isn't always a bad thing all the time either.

Exploding Mephits are nasty.

Their breath weapon is 'save for half' and at their CR, you can throw many of them at the group. Your damage dealers can take them out rather quickly, but they explode into a 'save for half' death burst to which they themselves are immune. They pair well with other fire immune monsters of higher CR.

Other monsters with 'save for half' effects or death burst abilities are similarly dangerous.

I think the other thing the Mephit suggestion points to is using multiple weak enemies rather than one big strong enemy. If the enemies all have 15 hp, it really doesn't matter if the fighter does 30 hp on a round or 15... they still die. You can throw tons of 15 hp monsters at them, and the ranger and barbarian will be able to one-shot them as easily as the fighter does. Uber abilities really only help you against strong things. That high AC prevents the boss monster's ONE attack from hitting. But when 20 mephits are steam-breathing you, the high AC is not going to help you that much. And even if they claw you, some of them will hit because there are so many of them.

Also, how about some spells to turn the uber-guys against each other? Like Crown of Madness or something.... high AC doesn't help you with that either.

The fighter is probably using a feat the requires you to take a minus 5 to hit and get an extra 10 damage. High AC enemies is very effective against this. 

He has not taken Great Weapon Master.

Just an 18 STR for a total of +6, action surge, and Mastery Die to get a bonus d8 every hit. He and the warlock pretty much nuke people with damage. Every turn is a potential (2d6+4+1d8 <Action Surge>, 2d6+4+d8) for the fighter (First turn) before switching to duel wielding for (Action: 1d6+4+1d8, BA: 1d6+4+1d8) because of Two Weapon Fighting Style plus he's a Mark of Finding human so he also has Hunter's Mark for an extra 1d6 damage per attack. Total max after action surge is 48 dmg a turn. 

Then the warlock has a homebrew item (Scythe) as a weapon, and using the Deathpact Angel as a base the damage is 2d4+cha (Hexblade) plus Booming Blade/Green-flame blade and then hex+Hexblade's Curse so his damage per turn is (2d4+4+1d6+2) normal, then Green-flame blade adds another d8 to a second target, and booming blade adds potential 1d8. So his total is about 20 a turn.

So those damage outputs, high AC, and bonus damage from the ranger and barbarian means that every boss monster or group of monsters goes down after 2-3 turns. 4 if I'm lucky. And it's just like, idk how to make a challenge for the AC/DPS builds that doesn't destroy the casual players. There's some good advice here though, so I think I have an idea on how to go forward.

So I have a few things for you.  

With the Warlock, just remember to design your encounters so they attack your players with a balanced mix of attack rolls and various saving throws.  If your monsters are ONLY making attack rolls, the high AC + blur build is naturally going to be very very strong.  But if you were to say throw a lightning bolt at him, have him make a dex save, and what do you know he drops concentration on blur, all the sudden he isn't looking as powerful anymore.  Just remember your encounters should be attacking your players regularly with saving throws alongside attack rolls.  

As for the fighter, you are going to need to ask this player to explain themself.  It just doesn't add up.  So assuming he rolls the absolute maximum amount of damage: 2d6 (a greatsword, right?)  for 12, and let's say for argument's sake he rolls really well on his strength and has +5 modifier for 17 total damage, now say he expends a superiority die and rolls max on that for damage, for a total of 25.  Let's say he also has great weapon master feat, for another 10 damage.  That totals to 35, which is in the ballpark of what you were describing.  Something obviously does not add up here.  There is no way he is dealing that kind of damage every turn regularly.  He is clearly either getting something wrong, or pulling a fast one on you man.     

What I was explaining was per turn, not per attack. The first turn or 2 is his great weapon, then he switches to two shortswords. He uses the great weapon only for Action Surge to get the most damage possible. And I showed you the numbers. 
Great Weapon, DOESN'T HAVE MASTER, I have to say that again. (2d6+4 (Weapon) + 1d6 (Hunter's Mark) + 1d8 (Superiority Die) then action Surge, to that x 2.) Max damage for that turn is 4d6+8 (32 max) + 2d6 (12) for 44 after the setup. 
Duel wielding: Action: (1d6+4+1d6+1d8) x2 (For Bonus Action) is 2d6 (12) + 8 + 2d6 (Hunter's mark, 12) + 2d8 (16) for a total max of 48, then he's out of superiority, so every turn after goes down to 32 max. It's all fair, it's all RAW damage. IF you do the numbers right and account for all the die he's using and where they're coming from.

There are lots of ways to counter this. For instance, it would probably be hard to do all that damage after being hit with a Hold Person spell....

1st, manage how much short rest do they have. Ranger and Barbarian don't get anything from short rest other than getting some HP back. On the other hand, Warlock and Fighter get their combat resource back from the short rest. So hindering the short rest will have a bigger impact on those two powerful PC.

2nd, What does Mark of Finding human do? Does it only work on humans or humanoids? If so, put some non-humanoid/non-human.

Mark of Finding human gets Hunter's Mark for free and despite the class having spellcasting or not. Also 1d4 added to Survival and Perception checks, and Darkvision.

• Ambushes with ranged attacks so that the warlock and the fighter aren’t in melee for at least 3-4 rounds will make them sweat
• Area of effect spells
• Opponents that regenerate
• Opponents with massive, massive amounts of HP 
• Large numbers of opponents, doing massive damage to one opponent doesn’t help when the party is facing 10-15 opponents 
• Opponents with damage resistance 

And, of course, let them walk through some combat encounters. I deliberately design some combat encounters that my players win easily. They enjoy them for a change of pace. 

IMO, one of the most effective ways to increase the challenge is to increase the monsters action economy relative to the players action economy.

Quote from WoadBear >>

Mark of Finding human gets Hunter's Mark for free and despite the class having spellcasting or not. Also 1d4 added to Survival and Perception checks, and Darkvision.

 So we are simply talking about his single target burst potential per short rest, because he gets only one action surge, and a couple manuvers per short rest.  Only gets the hunter's mark again on a long rest.  Try a follow up attack by the enemy as they go to rest.  That will challenge him. 

Change your focus. Let me explain.

You talk about a high AC Hexblade with spells that make him even harder to hit. Change the focus. AC, Blur, and Mirror Image help you NOT get hit.

Don't roll to hit. Constant environmental damage that deals 1d4 by simply being in the area doesn't care about your AC. 

Use spells that simply require saves. No roll to hit. No disadvantage from blur and again, I don't care about AC because I'm not rolling to hit. Have the Hexblade make a strength save or suffer exhaustion.

Charm a berserker barbarian or use them as enemies. Advantage/disadvantage cancel out but at least your rolling straight up.

Dispel the blur/mirror image.

More saving throws. It ignores AC.

Use monsters with pack tactics.

Use grapple to move the warlock around. Don't try and hit him, just control him.

Use "shove" to knock him prone. Everyone else has advantage now.

Finally, just ignore the warlock. At some point, he'll have to cope with all his allies dying.

 As to the fighter that dishes out insane damage, it's easy. Have enemies that don't have hit points (at least at first). Have enemies that get more punishing as they take damage. Have a ranged thing far away from the fighter that's an invul bubble. It must be destroyed or you can't deal damage to the bad guy.

Make the fighter make saves vs. int. Use illusion, confusion, exhaustion, poison. Things that restrain and render immobile. Sap her strength. Rust monster and heat metal their weapon.

Change your focus to things where high AC and high damage are of little value.

An encounter isn't challenging because of how many hit points a monster has or how high their ac or damage output is. It's challenging simply by making it so. An encounter in fire sucks without fire resist. An encounter on a bridge that collapses in 2 rounds sucks. As does an encounter where you're so cramped you can't swing a big weapon. Fights in the ocean suck. Fights in the air or against flyers when you can't suck. Fights against ghosts or invisible things. Oozes. In a room with a spawning pit that doesn't end. In a place where the baddies prepared for you (behind cover, set traps, use murder holes, things fall from ceiling, etc.)

Hopefully there's enough there and it helps.

My battlemaster can currently do about 138 max damage on a turn with only action surge and superiority dice but, it seems my burst damage is balanced by having to spend many actions dashing to targets without the chance to attack so, there's that effectively cutting your damage in half!

Well i think the fighter is doing the math wrong or you are bending RAW a little

Hunters mark is a bonus action to cast or move so they can't use the bonus action to both two hand attack and cast or move it in the same turn.

Also while a lot of DM work around this and even official advise is to not enforce it too hard and  people like Mearls and Crawford have expressed that its fine to let people combine this - Technically RAW the PC can't stow and draw a weapon without expending an action. they would have to stow the great sword as a free interaction at the end of one turn and draw one of their swords as a free action next turn then using their action to draw the second - in which case they would not be able to use the two weapon fighting as they haven't take the attack attack action.

So what you are describing in the RAW action economy should take them a whole turn to do, not just be able to do.

also max possible damage is not the same as actual damage - pitting your fighter against a cr 3 hobgoblin captain with an ac of 17 - so they need to roll 11+ to hit, so below average. you add up the actually likelihood - well yeah your fighter can do 60 points of damage (1 per short rest, expanding 50% of their super dice) but its 1.47 million to 1 that they will do that much damage!

in truth you are complaining about them having less than 20% chance on that turn of dealing on average 35 damage (once per short rest expending 50% of you get the idea) on the first turn then a less than 20% of dealing an average of 28 on turn two (for all their dice) unless its against a new target and they had to a) move the hunters mark making it 14 for 1 super dice or 22 for 2 dice and then having to spend the rest of the time until the next short rest with no action surge or super dice and no ability to cast hunters mark until their next long rest.

and that is taking into account you are using a RAI stowing/drawing mechanic that lets them switch with no penalty.

There is only one situation where the above is an issue - you are letting them have a short rest between encounters.

Because if that's room one of the dungeon - from room 2 to the end that fighter is a meat shield with a good right hook and nothing more, no hunters mark, no super dice, no action surge.....

It sounds like your biggest problem is that the players have all their abilities for combat.

As mentioned above, this can be very well remedied by periods of time without rests.

Anywhere dangerous will do (dungeon, forests, criminal underground, you name it).

Quote from Paradox52 >>

It sounds like your biggest problem is that the players have all their abilities for combat.

As mentioned above, this can be very well remedied by periods of time without rests.

Anywhere dangerous will do (dungeon, forests, criminal underground, you name it).

My hammer is overpowered at hitting in nails, and no matter how many nails i put in front of it and line up it keeps hammering them in! Meanwhile the pliers and monkey wrench barely get to hit any nails!