So my campaign takes place in a high-magic city, and my level 11 party is gearing up for a heist on the estate of a powerful noble family. The PCs have faced magical security before: it's very common in the city, but most of the applications they've run into previously were limited by the time, funds, or power available to the caster. This family will have no such limitations; if there's magic that can be done, they can afford it. The only restriction is no Wish, because the knowledge of how to cast that spell is lost in this setting.

What kind of magical traps would you cook up with unlimited time and resources? Forbiddance is an obvious one, as is Guards and Wards, but I feel like there's some more creative options to layer on top of those. The security measures don't strictly need to abide by the spells available to PCs; as long as it feels like something a Wizard or an Artificer could do, it's perfect.

Gremishka roaming the halls. Like a crazy cat lady, but with gremishka.

Guards with goggles with a Gem of Seeing in them.

A permanent Anti-magic field — either by artifact or someone re-casting it every hour — around the main vault with the item. 

All staff have scrolls of dispel magic they just carry with them. Not casters, necessarily, just they know how to read that one scroll. 

Some kind of contingency that drops a force cage around stuff. 

I’d recommend taking a look at complex traps in XGE. I think it’s a great way to incorporate magic traps and spells to make a nasty shock for a party. It also is something they likely haven’t seen before, and would make them think on their feet

Other than that, idk if you mean like to protect and hide different parts of buildings or treasures or like offensive traps to eliminate intruders. Golems make good guardian style creatures, and can be created, as can helmed horrors or shield guardians.
Some spells made permanent I’d could be animatic field spells, some permanent spike growth spells. Glyph of warding spells everywhere (doors, doorknobs, doormats, trapdoors, keys to doors) and symbol spells as well.

Hope this helps, and let me know what devious traps and spells you decide on! My unsuspecting players won’t know what’s coming…

It really depends on what you want to say about the caster.

For example :

1) Some one may bind spirits or magical creatures for security which creatures they are says allot about the person who did it.  

• They may imprison evil creatures like demons or souls of dangerous criminals and bind them to their service 
• They may  take good creatures hostage  or make deals with evil creatures 
• They could specialize in a group like fiends, undead or fey
• They could have a personal relationship with a creature 

Specific examples include 

• A guardian spirit of the building like a eidolon  willing or unwilling.
• An arcanaloth librarian or servant. This is a pretty dangerous and potentially sadistic fiend 
• A fiend trapped in a portrait pretending to be an innocent begging to be freed. If found out they switch to using some of their magic to torment the party. You can use a Guardian Portrait or another fiend stat block like a chain devil that can use its unnerving mask and animate chain abilities to attack the party. 
• An object with a spirit inside that possesses people. Use either a ghost or the magic jar for rules. Alternative it could be a possessed doll like a Carrionette which has an ability to swap bodies on an attack
• a elder oblex that acts as an entire team of staff through out the facility
• familiars or pets like imp, quasit, blink dog. If you want to be really wild they may have a werewolf trapped in wolf form  as a pet.
• boggle, goblin or other minor fey  house cleaners 
• statues in the garden that are stone cursed

2) Many mages may use curses but this is a more sadistic option because they often create long term suffering or collateral damage. For example were beasts are very destructive.

• A mage may put a curse on an object that can only be removed by them so that thieves are forced to return 
• A curse may be instantly deadly or imprisoning to prevent them from leaving 

Some specific examples 

• Something enchanted with antipathy/sympathy to keep people in a certain place or lure them into a trap 
• a rug of smothering that is the pelt of animal in fuses with the skin of the person it smothers turning them into a werebeast  that will kill friends and family until the mage removes it. 
• A cursed object that when touched petrifies the target as per flesh to stone if its a gold object it may be gold. The gold of the statue is worthless because any attempt to melt it down or break pieces off  causes the removed part to revert to flesh killing the petrified target. 
• an object rigged to explode with glyph of warding  because it is too dangerous or secret to be allowed to leave 
• a cursed object that entices people to take it but brings misfortune like a disease from contagion or radiation like sickening radiance (entire families have been wiped out by thieves stealing radioactive or biological material unknowingly)

3) Constructs are a pretty classic simple deadly device akin to robots. They work best if they act like a robot with clear rules anything in the construct category can work 

• animated armor in the main corridor that attacks people who don't say a password

4) Magical doors. Doors can require specific spells to open  depending on the specialty of the magic used or act as monsters

• arcane lock and knock are a classic example 
• a  plant door  require tree stride
• a thick stone wall requiring meld into stone
• a mimic door that either acts as a fake door or is trained to not attack the owner
• an elemental wall like  wall of fire requiring magical protection from elements like investiture of flame to get through safely

5) Disposing of the dead 

• compost full of swarm of rot grubs and spawn of kyuss those captured are thrown in the pit with them
• a curse on the house that traps the souls of those who die 

This is all great, thanks so much. I especially like the ideas about doors that require different spells to get through. The estate in question belongs to a bloodline of Deep Earth sorcerers (doesn't exist as such in 5e, but I took inspiration from a 3.5 subclass) so it makes sense that their most secure quarters would just... Not have doors. There would have to be some method for the staff to get around, though, and maybe that's something the PCs can find and take advantage of.

I'll definitely be including summons, constructs, and some automatic curses, too. Much to think about. Follow-up thought: do y'all think a Forcecage could be set up around an Antimagic Field?  Or would the field prevent the cage from working? (Acknowledging that the cage is a cube and the field is a sphere, but that could be adjusted)

the walls of the keep are filled with innumerable Spectators. any spells cast in the keep (besides casters approved by the family) will target allies of the caster because of their reflect spell reaction. (yes I know that's not technically how that works, but its way to cool to not do some tweaking

Trap of True Polymorph with a high saving throw.

I used something similar for parties of level 15-18 recently. They encounter a place filled with traps that are extremely deadly and there is one that essentially casts true polymorph upon the creature who activates the self-reloading trap. This meant that they get surprised by the true polymorph but are effectively given a free way to avoid the next lot of damage from a trap (or magic security). They also then have the option to backtrack and reuse the trap as a strategy to help them navigate all the other security. 

I originally envisioned in this thing for it to be a puzzle where only the initiated knew that the way through particular doors was to go around them by being polymorphed into say a spider, and scampering through a tiny crack in a wall. Likewise if you extended out true polymorph you could make the targeted creature become some kind of creature that has immunity to an environmental danger set out ahead.

Initially the party were panicked by being transformed thinking they'd screwed up by setting off a trap. Took them a little while but eventually they realised the idea was that the polymorph gave them a way of surviving a trap without burning through healing items or spell slots.

In addition if you level the save DC correctly it can feel like choosing to let the trap transform your character is a valid choice - i.e. failure is not a punishment but a reward/the solution. 

I had great fun with the idea, but I can see it might not be for everyone though.

Use mimics where you think appropriate.

Check out Grimtooth's traps. Many are as fun(ny) as they are deadly.

Think on the effect that you want the situation to have. Alarm the area and alert the host, use party resources, trap, injure, maim, kill? 

Think about the tolerance of your players to whatever you throw at them. Remember this is supposed to be fun for everyone.