I've got a little gothic horror game planned for my players next week (up to now its been all fun and sillyness ;) through a long and complicated journey I want to get them to a point where they enter a room and each find themselves trapped in their own little hell loop (reliving the same thing over and over unable to escape) based of their back stories and what their characters would be affected by.

Question is; is there a magic that would work like that? I was kinda thinking its more like a dream or an halucination then reality. I need it to affect all of them (except one who will be outside looking in) I don't mind players breaking out but I want to avoid difficulties getting them in, or more accuratly them deciding to question how / why and me not having a good answer. Its going to be caused by a sort of ghost/memory trying to stop them.

Any thoughts would be welcome :)

I mean, you could combine a number of illusion spells to achieve an effect like that, but just... don't bother. You don't need to reveal more to them than "It's powerful illusion magic". There are powerful magics available to certain npcs that are not available to players. That's just how it is.

Regular spells tend to have too many mitigating factors for game balance purposes to use for something like this. Just make up your own effect and if you want to give it a saving throw, assign one as appropriate (probably INT).

This is a hand-wavey purview of the DM.  If you want it to happen... make it so... whether it is a vapor or gas or because it is groundhog day.  The plot demands it... so it happens.

Looks like the consensus here is hand-wave it, don't try to spec it out with the nitty-gritty of the rules.

'Welcome to your own personal hell loop...ITS GROUNDHOG DAY!' XD

What's your intent with little "hell loops"? What is causing this to happen to them, and how can they escape them? For those reasons alone, it's important not to entirely hand-wave this. You should have some general sense of the mechanics of the loops. Why? Because the players and characters will interact with them and ask questions about them, and you want to be able to provide certain answers that might help them face this challenge.

It might help ground this magic in the "reality" of the D&D world if you at least base it in part on some existing spells. If you take a look through some of the illusion and enchantment spells, you can get a sense of some of the ways the players might interact with this and how they might break it.

Yes there is magic that can cause hallucination like that, it's called DM fiat. Just tell your players that they feel an unearthly sleep descending upon them and throughout the dream you can impose WIS or CHA checks for them to try and wake up (so they don't feel like they have no agency), and boom, you've done it.

Remember, as the DM, you're under no obligation to hold yourself to just the spell options in the books. The spells in the books don't encompass the totality of all the magic in the world, just the ones that are most commonly used by adventurers. And remember that there are monsters that have access to abilities the players cannot access, so the developers are essentially giving you permission to act outside of the parameters of the Player Rules as well. So I wouldn't ask myself "is there magic that would work like that?" Instead ask yourself "what is the best way narratively for this dream sequence to work that's not too railroady?"

Instead of it being in a room, it could be while they are long resting, because they chose to camp on some mystical spot or something. I’m only saying this because if you do it when they go in a room, there’s issues like, why would character 3 go in after they see the first two get really weird and quiet. And you can bet there will be complaints about not getting a saving throw to avoid it, or arguments about a class ability that should have allowed them to not be affected. You could still see that with the camping way, but they might be less resistant. 
Just be careful of putting them each in a personal loop during the session. You end up with one player doing something while everyone else sits there bored. Even if it only takes 10 minutes each, that leaves people with 30 just sitting there (in a 4-person party, obviously it gets worse with a bigger party). Maybe set up the loop at the end of the session, and run out with each person between sessions. 

     You could hand-wave it, or you could implement something into the setting that would make other creatures/NPCs capable of using powers that your players could never achieve. 

     I don't know anything about your setting, but one idea, if they're in this unknown land, is to maybe make the inhabitants of this place capable of some weird-ass rituals, not known to the rest of the world. Maybe they require sacrifices, or maybe they are the result of a breakthrough in Arcane Science and now that the players know about it, they can't leave.

     Or, you know, they worship some tentacly aberration that can melt your brain if he thinks too hard in you general direction.

There are lots of ways you could do this, but I'm going to be putting my players into a communal hallucination brought on by smoking some magic weed (which they are going to obtain from a gorgon's lair). I'm doing it this way because sending them each into individual dream-scapes is less interesting as they then have 3/4 non-play time whilst the others have their dreams, and also I want them all to have the same knowledge.

This is what is about to happen for them:
Dream 0 - a trippy psycadelic, happy place full of mushrooms and flumphs. Suddenly it goes bad.
Dream 1 - they appear on a ship. The ship is attacked by the BBEG, catches fire, a water genasi with a baby throws herself overboard. The baby is one of the PCs, but they may not realise that.
Dream 2 - as they hit the water they appear on a battlemap, each in the dark and unable to see one another. They are all standing on paper; they are inside a book which one of the PC's owns, which holds the souls of dead swordmasters. They each must duel one of the spirits locked in the book. When they defeat the enemies and find the centre of the book (and each other) they move to the third dream.
Dream 3 - The PCs appear in a desolate place, where a great old one (the warlock's patron) is chained up. They learn about a puzzle that can set it free, then are chased away by a CR13 monster (they are level 4).
Dream 4 - The final dream is over 100 years into the past, at one of the world's inciting events. The players fight creatures that they can't possible beat, will all die, and end up back in their bodies. They will return to this dream at least twice more during the campaign, progressing further each time until they get to see the end of a major confrontation (this is something like a gauntlet, but should let them feel more powerful each time).

So that's what I'm doing. Maybe it will be able to provide you with some inspiration for your own dream-scape madness. The key is that every scene refers to something from each character's backstory, but that they all get to live it.

I'm planning on letting them all be able to see eachothers loops, their all sort of in a dark space with each loop just a pool of light illuminating them, they can look but not break free without outside help as they keep getting dragged back in. I set the campaign up so about 1/2 is built on players backstories so they will get to go home, explore their old lives and follies, this is just the apetiser for that.These are specificaly going to be based on their secrets and weaknesses:

One I've got planned is I have a changling spy masquerading as an actor (this guise has been passed down through generations of changling spies) He's going to be useing a puppet of himself but it turns on him, cuts its strings, jumps onto him and stabbs his face in a frenzy killing him. Then he will wake up in a puppet booth with a box of puppets to use and the urge to perform a play with 'the' pupet, and so it goes on...and on....and on... untill someone pulls him out.

Sounds like a combo for Dream and Imprisonment spells. Imprisonment to lock them in a sleep effect and Dream to keep them reliving their loops. You can then use the imprisonment spell to lock in the way they end their respective loops. You'd be looking at several 9th level spell effects though so you need a suitably high level boss/group to do it.