I'm a newer DM when it comes to DnD, so while I feel confident in many aspects of running a game, I don't like throwing encounters at my players where I don't know that they have a decent chance of winning, or at the very least surviving the defeat. This is the reason I kinda need help.

In some sessions, my group of six by-then Level 4 heroes are gonna traverse the ruins of an old city, half submerged in a deathly mire where the sacrificed, the killed and the damned still slumber. I want to throw a small horde of spectral enemies at them, rising souls who were hurt by the BBEG, a mad, half-lichified ghoulish count that once ruled there. I want the feeling of fighting something not-quite-corporeal, but... Just, the Spectres, Shadows, Ghosts, Will'o'Wisps and Banshees are so gosh darn mean! I fear the very real risk of killing of a player or two if I use them, which I really don't want to do in such a decently inconsequential battle.

So that's it, how do I use them properly? Should I try to play them less aggressive, fill their ranks with a few other undead, just remake them to not be able to kill so easily?

For me I think I'd go for a tester encounter near tyhe beginning of the complex they need to go through. Have a single shadow and half a dozen "ghosts". For the "ghosts" just use the zombie stat block and flavour it that they look spectral and ghost like. See how they handle it and then as you progress on through you can add more powerful spectral undead if the party are coping with it or keep the ghost like zombies if they are having a hard time.

Also think about playing on the players fears and knowledge. If they know about Will O'Wisps have dancing lights (as per the cantrip) flicker into life, move around then disappear. If they know about ghouls, have zombies with a few extra hit points and the gnoll rampage ability to make bite attacks. You could have ghostly appirations just floating around and as the party approach the apparitions scatter and inhabit ancient suits of armour, weapons and rugs and animate them, using the animated armour, flying sword and rug of smothering stat blocks but with an udead creature type (just in case anyone has undead detection abilities). Maybe a slightly underhanded approach but you can alays ask them to make saving throws for no apparant reason, a player walks through some stagnant water...ask for a cons saving throw, whatever they roll, they pass (unless a natural 1 in which case you can add some cosmetic effect such as "you feel your right shin tingle as if you just walked through stinging nettles"), ask for preception checks and on nat 1's they see shadows move, see a ripple in the water that moves towards them etc. That way you can hopefully ramp up the fear of them missing something or that you have a creature stalking them, just do it sparingly.

Other options are to reflavour some spells, use a version of entangle that has undead hands rising out the ground to try and grapple people as they pass by, you could tweak spike growith to add in the heads of undead rising out the ground to bite people as they move over the area before sinking back under the ground. Add the "burst" ability to some undead, when they die they explode dealing damage or producing a stinking cloud effect. All these little odd occurances can then become little lair effects for the BBEG.

One thing you could always do is downgrade the creatures if you feel that thematically they fit the game you are running but you also do not want to TPK the party. 

The following link is a good Matt Colville video on suggestions on how to upscale a creature but more importantly for your case possibly downscale them. https://youtu.be/QgTIGo6zJbs

I've done this in a couple of cases where I wanted a creature because it fits my encounter but would have decimated the party, (if you watch the video, ironically enough it was a Medusa).  The changes I made gave the party just enough of a challenge at their levels (5-Third Level) while not be decimated by the abilities of a Medusa. 

So maybe it could be worth your time to see about keeping the creatures but scaling them to fit a challenge for your party.    

Rob's got a decent idea.  Zombies with their Undead Fortitude replaced by Spectral Movement could do a decent job of standing in for a low-class wandering soul.  Give them the usual mundane damage resistances for spectral enemies, refluff Slam as a more ghostly attack (perhaps with a switch to necrotic damage, though consider downgrading the die a step if you do), and your 'Unquiet Soul' would be a good trainer enemy to show your players how more powerful and dangerous spectral undead work.

You guys are great - I especially like Rob's idea of letting the illusion of something horrible coming for them be the main antagonizing force and let the mire itself be the antagonist in the situation. I'll probably make myself an "Unquiet Soul" mob that I'll share with you guys. I may even make a mob of Shambling Bodies that can move up from the mire to grasp at the players and attempt to drag them under.

Also, it's kinda funny that you gave the idea for a Zombie with Rampage; I created my own Ghouls for the adventure that has Pounce and can attack the players after they die, so I kinda already did. But they've been fighting ghouls since they arrived in town so Imma diversify my undead forces a bit.

Maybe a downgraded wraith statblock for a boss? Those don’t have any punishing mechanics really. Or just a default wraith as a boss of the players are doing really good for the challenge.

You can just take a generic humanoid or beast and

• Give it the resistances and immunities of a Specter.
• Change its damage type to necrotic.
• Give flight equal to its speed (hover) and incorporeal movement.

This will typically double the CR of low CR creatures, more like +2 CR for high CR.