OKay, so I am trying to create monsters which are interesting.

How do I define interesting? Well, I want each monster to feel like a substantial threat for more than just base stats.

I've thought about two archtypes: Fast and strong.

Fast represent monsters who are just beings who are agile and strong well beyond human capabilities. To represent this, I give them legendary actions. Especially legendary actions that have movement baked in. One of the chief inspirations for this kind of monster is Dark Souls bosses and the Dark Souls board game where the bosses get to make an attack after every player action. I also characterise this as fighting a Xenomorph (from Alien, not Aliens).

Strong represent giants and other enemies who hit really hard but maybe shouldn't be swinging 3 times a turn. What I figure these monsters should have are attacks that hit a whole area rather than just one target. Through this, positioning becomes even more important.

Both are more suitible to be a single adversary against a group of adventurers than the typical monster, due to action economy deficiencies.

Can anyone suggest any interesting abilities that monsters like this could have to make unique, memorable monster battles? I specifically want to focus on enemies who could, by itself, hold off a party of adventurers, rather than ones that need several copy and pasted versions.

Hmmm, How about control monsters (can't think of a better name) monsters that move others around with their abilities.

The Soul Monger gets an interesting ability: Soul Thirst. Whenever it reduces a creature to 0 hitpoints, it gains half of that creatures max hp as temporary hitpoints and has advantage on all attacks whilst in possession of those temporary hitpoints.

Look at lair actions for some inspiration. Lair actions tend to be a little more complex and range from “mildly annoying” to “rather dangerous. Some of these can change the battlefield itself by messing with terrain or adding additional obstacles; some of them might effect players in ways other than straight damage.  I’ll often take a lair action or two and have them trigger when a monster reaches certain hit point values (like 50% of max, 75% of max, etc.”).

Another option is to add dynamic challenges. A pair of monsters that can’t be defeated unless the party drops them both in the same round (they heal to 25% if dropped to zero if the other isn’t also dropped before getting to top of the order). Monsters that heal themselves, but there is some way to interrupt the healing.  Those kinds of challenges that require the party figure out something other than “how do we best smash this” can lead to some memorable encounters. 

Resistances can help make things interesting, and effectively boost HP.
My DM once had a monster that had two colors on it: one represented an immunity, the other a vulnerability (he didn’t tell us, we figured it out as the fight progressed). Then every round, the colors would change. We never quite figured out which color worked with which energy type, partly because he avoided the obvious (red was not fire, white was not cold). This was quite a few years ago, so it was, obviously, a pretty memorable fight.

Quote from Xalthu >>

Resistances can help make things interesting, and effectively boost HP.
My DM once had a monster that had two colors on it: one represented an immunity, the other a vulnerability (he didn’t tell us, we figured it out as the fight progressed). Then every round, the colors would change. We never quite figured out which color worked with which energy type, partly because he avoided the obvious (red was not fire, white was not cold). This was quite a few years ago, so it was, obviously, a pretty memorable fight.

 I did that with a hydra, but the fight was over before they figured out the color coding (I think it ended the fight with nine heads).

Personally my dea of an interesting monster is when they seem to fit into the world around them, rather than just being dropped into the world and seeming to only exist in a capacity for adventuring parties to fight them!

To that end, I would think about the area around the monsters - and that falls heavily into the Lair Effects section of monsters. Make the monster fit the area, and know the area, manipulate and usethe area, and it's going to be a much more memorable fight than them moving or hitting fast. I also believe the most interesting fights I've been involved in are the ones which had some sort of ticking clock. This can be environmental, such as the rogue trying to pick the lock o na door as a big monster approaches, or it can be enforced by the plot, such as trying to catch up with someone and being blocked by a monster.

I believe that the abilities which will make a fight scene really interesting though are the non-magical control types - IE, being swallowed, picked up, and run off with. Defeating a huge iron bull with smoke billowing from the joints is all well and good, but doing so whilst it is running through the streets trying to wreak havoc on the town, with one party member caught on the horns and the rest trying to work out how to catch up with it or stop it is going to be the ones which prompt the players to think of interesting solutions, which in turn will make it memorable and interesting.

So I guess what I'm saying is, Motives and Environment will make for interesting encounters more than what the monsters are capable of doing. One kobold, with time to make loads of traps and to lure the party into them, could pull a TPK, but I would consider a kobold with superman-powers to make them fast and strong less interesting than one which devises an elaborate series of lures and traps for the party to fall into!

As for single monsters vs party, they have a tendancy to be a bit of an issue - if they do what's reasonable and hit the same character on one turn, they can drop them to 0hp, or they can have too many HP to make the fight anything but a slug-fest (out of neccesity, to not die immediately). far better to consider it as an encounter with multiple "oponents", EG:

• Fire monster, guarding the door
• the door, which needs a lot of damage to be broken through
• a geyser, which goes off at Initiative 20 every other turn

with a goal to the encounter to be that the party needs to be delayed, so the door is working towards that, the geyser is hurting them, and the fire monster is making it impossible for them to open the door. 

You might be interested in monster roles from 4e. They were a great shorthand for monsters strengths, weaknesses, and basic tactics.

• Artillery - agile ranged damage dealers, much less threatening in melee
• Brutes - high damage, high HP, low AC, low mental stats, charge in swinging
• Controllers - high mental stats, low physical stats, ranged enemies specializing in AOE and applying status effects
• Lurkers - glass cannons relying on stealth or other tactics to catch the party off guard
• Skirmishers - highly mobile melee units, flank attacks or go for the back line
• Soldiers - high AC, low damage with "sticky" mechanics to tie up the front line

I'll often start encounter design with a basic combination of 2-3 of these, and the tactics just emerge naturally. A brute holds the front line while artillery fires from above and a lurker circles around to attack from behind. Soldiers guard the gate as skirmishers move to flank. The controller unleashes her brute and retreats to the balcony where she can drop spells on the party.

Then once you have kind of a tactical overview, you can design one or two moves that emphasize each monster's role in the fight.

It's not terribly hard to use 4th edition monsters in 5e. In terms of damage output, a standard monster has an offensive CR of around 1+level/5, an elite 2+level/3, a solo 6+level; just give it CR-appropriate attack bonuses and hit points and you're done. Or tweak to taste -- for hit points I would down-shift by 3 steps, and compensate by adding 3 points of attack and AC, as that will come closer to actual published monsters.

Interesting things are things that normal combat rules do not apply.

Consider a couple of glass golem with something even worse inside it.   Could be a swarm of insects, could be an elemental, could even just be a lit torch if the room is covered in oil. 

Then there is old mirror trick, the creature takes on your own abilities.  You have to fight yourself.  in a similar vein, creature that takes possesion of your allies,whom you have to fight but not kill.

If you want players to find your monsters interesting, then them being creepy always helps.

Van richtens guide to  ravenloft has a lot of horror like monsters and tips on making your monsters scary.

My attempts to give interesting mechanics have been hit or miss, tbh. 

I ran an encounter today with a necromancer who, upon dropping a PC to 0 HP, could send spirits to possess their bodies as a legendary action so that they would rise and fight for him while still rolling death saving throws as normal. Players figured immediately that if they healed the victims it undid the possession.

I had a bladesinger with a magic cloak that was alive and could fly and be used as a mount and had protection as a reaction. The spells from the wizard were poorly selected though and was notuch of a threat.

I had a psichic sorcerer that could share his pain with others and as a reaction upon being hit could make someone else roll concentration as if hit by the attack. They paired up with a crazy surgeon that could use his reaction to cast cure wounds upon either himself or an ally when they were hit.

I had a doppelganger whose mind reading got stronger every turn. First turn they read the mind of everyone within radius, second turn their mind reading became good enough to cause disadvantage on those within range that attacked him and third round gave him advantage on those he attacked within range. Losing concentration meant that they had to restart the count, but they were also the equivalent of a level 7 rogue in regards to class skills.

There was a guard captain that had advantage in initiative rolls and made it so that any of his soldiers that rolled lower than him went right after him in initiative order and could also summon a new soldier per turn with a bonus action (come to think about it, an interesting mechanic could be a monster whose initiative raises by one every round).

I also considered something like giving a green dragon the ability to use plant growth, having the grown grass release poison spores the next round and having said spores being flamable.

personally, i think to make interesting battles, you really need the boss to have 1 or 2 abilities that really stand out, like troll regeneration or elder tempest's screaming gale. probably the best example of this has to be dragon breath weapons.

last session, my party had an encounter with a green hag, and they had very little trouble defeating it, but it would constantly phase through walls and attack through duplicates, making it quite interesting. and what my players don't know is that one, the hag's hut is a living chicken, and the hag was cursed by their competitor coven member, who was an ancient hag to turn into a revenant to slay whoever killed her.

the party's bard strangled her to death after she put out his fire. (don't mind, in his backstory he suffered from multiple demon lord insanities) so, i plan that once the party reaches lvl 5, they'll have an encounter with the revenant.

A thing to consider about how interesting your fights need to be is: a fight that takes two hours to resolve needs more interesting stuff than a fight that takes half an hour. You don't want boring fights, but it doesn't take interesting mechanics to pull that off, if you drop one or more PCs to 0 hp in round 1 that should achieve necessary excitement.

Quote from Doug_Booshaka >>

The Soul Monger gets an interesting ability: Soul Thirst. Whenever it reduces a creature to 0 hitpoints, it gains half of that creatures max hp as temporary hitpoints and has advantage on all attacks whilst in possession of those temporary hitpoints.

I think this monster is actually an example of very bad monster design. The ability means that when it drops a creature to zero, it becomes much, much harder to kill and even more dangerous than it was before. This means that either (a) no character gets to 0 hit points, and the ability never triggers so it might as well not have it, or (b) it strikes hard enough to drop a character, after which the party are in even more danger, are down a character, and if that character is brought back up with a healing spell, it's likely to drop them again and gain even more hit points. It's a death-spiral, and very poorly thought through.

In terms of designing monsters, I suggest coming up with a Gimmick for each fight like this. The Gimmick is the thing your players will remember about the fight, and it involves them employing some specific tactics to win. Here are some examples from encounters I've run (note that to employ gimmicks, you'll need to be prepared for fights to last longer than the MM type monster hit points allow for, or the gimmick barely gets used):

Encounter 1: Physical and Magic

I ran this for six level 8 PCs. There are two guardian statues that become golems when the players activate something they shouldn't (or should). One of them is immune to all physical forms of damage, and is a powerful spellcaster, which prioritises attacks against melee combatants. The other is immune to all forms of magical damage, and is a powerful melee combatant, and prioritises its attacks against spellcasters. These have been set there so that whoever activates them, one golem will always be immune to their best abilities. The PCs need to figure out which target they should each be attacking, and try to protect one another.

Encounter 2: Deep Breath

I ran this for six level 5 PCs. The PCs find themselves up against a raging ice elemental in a library with many book shelves to act as cover. On turn 1 of the combat, the elemental sucks in all the cold around it, and makes regular attacks. On turn 2, it unleashes the sucked in cold, dealing major cold damage to anything not behind cover. The PCs should figure out that they can go in to strike on the turn it will take its Deep Breath, and need to run for cover (or just prepare for the damage) for every evenly numbered turn. To be effective, you'll want this combat to run for at least 5 turns, preferably as many as 7 or 8, and describe how all the cold is pent up inside it after the deep breath is taken on the odd number of turns. The damage from the release of breath should be overwhelming, so the PCs better figure it out quickly, and there should be lots of cover to hide behind.

Encounter 3: Classic Ritual

I ran this for six level 9 characters. The characters chance a necromancer into a crypt, and find he is conducting a ritual with a glowing red altar. On appearing, two of The Lost spawn and attack the PCs. The altar makes the necromancer completely immune to all attacks, but he is able to cast spells from within the safety of the red glow. The altar has 150 hit points and AC19. Each turn, two more of The Lost spawn anew around the room (it takes a Dash to reach the PCs from the alcoves they spawn in). The PCs need to figure out that they need to destroy the altar so they can deal with the Necromancer once his immunity is gone. To make things even more fun, he can cast resilient sphere after the altar is destroyed. Until the necromancer is slain, The Lost continue to rise, so there's a race against time to take him down. Once he dies, any remaining The Lost fade away.

The Deep Breath one reminded me (on a side level) of the ethereal fisherman encountr I made, which featured a giant trying t ocatch a Void Shark.

The Void Sharks exist on the border ethereal, and they had the ability to shift planes ever 3 turns. So the fight could go for 3 turns, then the shark disappears - in the same way it would in the ocean, but it could appear anywhere. 

I must get on and play that one some time!

No-one mentioned Matt Coleville, yet?

They recently funded a Kickstarter for a monster book, a free preview with a bunch of examples is here: LINK