So, I am running a campaign. This is my first campaign that has actually lasted more than a week :). It is about a lost archaeologist, and you have to explore Egypt and the pyramids to find him, during a time of ottoman occupation. One of my players in particular, is playing an autognome wizard. I’ll cut to the chase. He wants to cast mage hand, and then cast enlarge reduce on it, to pick up the top of the king’s palace, and rip it open to steal a golden key fragment. 
1. According to D&D rules, I’m not sure that’s even allowed

2. The palace is critical to the story of the campaign moving forward. 

Once I told him that he couldn’t do it, he tried to cast enlarge and reduce ona staircase, make 10 guards trip, and kill all but one of them with thunderwave. I’ve threatened to ban him from the campaign, but he’s starting to get angry, and is starting to turn another player against me. I think this is unfair.

What should I do?

So the first thing: Yeah. You're right. If he wanted to do that, just tell him to learn Bigby's Hand. Also, if he tries anything else, just say the the throne is enchanted with undestructible magic. For the second, you are also right. Enlarge reduce affects objects and creatures, not structures, so it wouldn't work. Explain to the players that they need to adhere to the rules. If they keep being annoying, tell them that you might have to make them leave. If they aren't going to respect you, make them.

Okay, this sounds like a problem player where 'ban from the campaign' is the right thing to do.

Enlarge/Reduce cannot target mage hand, but even if it could, giving it advantage on Strength checks does nothing (it doesn't have a score and cannot make checks), doubling carry capacity means it can lift... 20 pounds instead of 10.

Casting Enlarge/Reduce on the stairwell is more of a questionable edge case, since a stair is an object, but in general targeting large objects is done in 10' cubes (see Huge and Gargantuan Objects) and it would probably also fail because architectural objects generally do not have room to grow.

Neither. But the DM has the final rule. 

All three of you need to actually read the descriptions of the spells in order to avoid conflicts like these. Enlarge/Reduce allows the caster to target a creature or an object. A Mage Hand is neither a creature or object. It is a spectral force that manipulates objects and it is limited. Even if you allow the Mage Hand to be Enlarged, the Mage Hand still can only carry an object weighing no more than ten pounds. I don't know if the Ottoman Empire had building codes the way we do now, but a king's palace's roof should probably be heavier than ten pounds. Enlarge only grants advantage on Strength checks/saves and an extra 1d4 on weapon attacks. It does not actually increase the target's strength. And a Mage Hand does not have a Strength score with which to make Strength checks or saves. 

In order for an object to be Enlarged there has to be enough space for it to grow. A staircase is built to fit the space it is located at so has no space to grow. But a staircase could shrink to half its size if it is Reduced if the structure is too weak to hold it in place. (Maybe give him a spellcasting check if you want to allow this.) 

Thunderwave affects a 15' cube. Assuming they're medium sized creatures, each guard takes up a 5' square. Unless they are stacked on top of each other, and each should get a Dex save to avoid being tripped, a maximum of nine targets could be in the 15' cube. At best he could reduce the staircase with the guards on it before initiative is being rolled and then cast Thunderwave on his turn, assuming his initiative is higher than the guards. But if all ten guards are on a staircase, they're probably already in initiative and unless he has Action Surge, he can't cast Enlarge/Reduce and Thunderwave on one turn. 

Gotta love those players that try to us "rule of cool" to bully DM's. This dovetails nicely with the thread about players trying to squeeze every bit of juice out of a PC build. You say this is the first game that has lasted more than a session. (You can't be running a campaign if you have never even had two sessions before this one).

That is a red flag for me, but I don't need to know what the details are there. But yeah, if this guy does not immediately settle down, bounce him. Cheating like this should never be allowed.

Yeah, I have a player like that in my group. Normally, he's not that bad if the DM lets him do his thing, but he has some crazy 'but this should work' ideas. One time he played a warforged and got annoyed when we said that he wasn't allowed to make it so his melee did d8 lightning damage because 'he's a robot so his arm is a taser.'

Ha! That player must not have read the spell section of the Players Handbook.

i like the idea of saying "you can certainly try." enlarge/reduce can't target a spell, but let's say that it does this one time. "that would only double it's carrying capacity. i'll let it happen once. do you want to continue?" and then when they still want to do the thing, narrate a flickering, barely-held-together, giant, spectral (definitely not invisible) hand flapping at the roof of the palace. arcane check to even guide it properly (especially since they're all but guaranteed to start it too far back and have to walk it into range to reach the roof). guards on the roof quickly trace it the 30 feet back to it's origin and begin firing arrows.

there's a certain subset of players who are told 'no' every day of their life and for them the fantasy is being allowed to do a thing even if it's not practical. for this type of player they'll be thrilled just to play out the (losing) scenario. and you'll be their favorite dm even though what you just did was narrate a big 'no, find another way.' but, make no bones about it: this is an accommodation. some players are working through social issues like how to collaborate, how to work with a group, how to accept criticism, how to follow rules, etc. you are not obligated to be every player's second therapist. you are entirely within your rights to remove a player who disrupts the rest of the group (especially if that's a shared opinion within the group).

Quote from Jackleoflanterns >>

2. The palace is critical to the story of the campaign moving forward.

 

 That's not for you to decide. If your players figure out an ingenious way to get what they need from the palace, then you need to play along and congratulate them on their superior playing. Your job as DM is to create the world, respond to the players' actions as fairly as possible, and narrate the events. You probably have a cinematic idea of how your campaign will play out. I would encourage you to instead think of your gameworld as a place for your players to explore. The best trick any DM can learn is to follow from the front. Give your players ambiguous clues and then listen to them speculate about what's going on. Make notes and incorporate their ideas into the world.

You have a player who wants to find creative uses for his spells. Read the rules with him and decide on what his spells can and can't do. Then let them make their assault on the palace, and if they fail then play out their failure and have your NPCs react accordingly. This is how gameworlds are built.

Yeah but rules are rules and annoying players should be banned.

Quote from Faneal >>

Quote from Jackleoflanterns >>

2. The palace is critical to the story of the campaign moving forward.

 

 That's not for you to decide. If your players figure out an ingenious way to get what they need from the palace, then you need to play along and congratulate them on their superior playing. Your job as DM is to create the world, respond to the players' actions as fairly as possible, and narrate the events. You probably have a cinematic idea of how your campaign will play out. I would encourage you to instead think of your gameworld as a place for your players to explore. The best trick any DM can learn is to follow from the front. Give your players ambiguous clues and then listen to them speculate about what's going on. Make notes and incorporate their ideas into the world.

You have a player who wants to find creative uses for his spells. Read the rules with him and decide on what his spells can and can't do. Then let them make their assault on the palace, and if they fail then play out their failure and have your NPCs react accordingly. This is how gameworlds are built.

 If you want to make sure he is using spells properly or trying to do so, you can always put a dice roll on it with a high DC. This gives you time to think and iterate about what happens in the world.

Regarding the enlarge/reduce on Mage Hand: He can cast it, but it fails as it is a spectral hand and not a creature or object. One 2nd spell slot is wasted.
Regarding the enlarge/reduce on the staircase: If it is a wooden staircase or something, this could happen as you can regard it as an object. If it's a stone/concrete staircase it is part of a construction and hereby not able to enlarge or reduce. Also it can also biggen enough as the room allows it. This could also pop out the roof of a house, which could cause repercussion in the nearby town/original owner. Also 2nd spell slot is used.

I have a player who wanted to do a clutch attack, by joining her character in the midst of the fray. Friendlies and enemies around her, but still a Thunderwave. She wanted to retract when I started to say that everyone needs to do a Con Save, that everyone is pushed back who has failed and that the thunder is heard around 300ft. If you want to clutch, read the spells and think of the resulting problems. This is the only way every player needs to think if an action could be smart or not.

Quote from GinoDiClarissio >>

...Regarding the enlarge/reduce on Mage Hand: He can cast it, but it fails as it is a spectral hand and not a creature or object. One 2nd spell slot is wasted...

 if you find yourself about to fizzle someone's spell and 'dm gotchas' didn't come up in session 0, then give the guy a break. at least one break. "your character would know what the player is unclear on: that's not a valid target for the spell you want to cast. all previous experience points to you losing your spell slot on an impulsive mis-cast... but you can certainly try"

 ...i know not everyone likes matt mercer's approach to everything, but in this one thing i think there's a ton of wisdom. the dm asking "are you sure?" isn't immediately forcing people into bad outcomes and they're also not bowing to every hold-my-beer whim. if anything, they're hopefully avoiding both excessive pre-action debate and excessive post-action regret.

I would definitely say no on the use of mage hand  when it is clear he would need one of Bigby  Hand spells  The reduction of the  stairs causing the guard to trip should be rewarded as creative spell usage.  Now if they sneak their way into the  to tower and did not want to set of whatever trap might be on the area the key is in  then they could avail themselves  of Mage Hand

The first thing to do is to have a talk with everyone, not their characters. It really sounds like you didn't have much of a session 0 where you lay out how the game will work.

The first thing you should mention is that the game will be played mostly by the rules ... spells do what they say they do, no more and no less. No matter what the player imagines might be cool - that WILL NOT happen unless it is consistent with the way the rules work in the game. There are rare times when you might allow a cool idea to work once for the fun of it but it is best of these at most bend the rules a bit ... not break them.

After covering this and other session 0 explanations, then go on to explain to the player why their idea would not work at all. It really sounds like the player did not read the text of either spell .. just the names.

1) Mage hand is just a hand made of spectral force that can manipulate things a bit and lift at most 10 pounds. Lots of shenanigans possible but ripping the roof off a palace isn't one of them. (Note that ripping a roof off a palace is also something that Bigby's Hand would be incapable of ... it might be able to punch a hole or break a window but it only has a strength of 26 and is only size large - it isn't ripping a roof off anything).

2) Enlarge/reduce only targets creatures or objects - a mage hand is not an object. All enlarge/reduce does is make the object bigger or smaller and weigh more or less. Attacks can do a bit more or less damage (d4) and you get advantage or disadvantage on strength checks. It doesn't change the strength of the creature/object. If the player had read the spell instead of just the title, they would know that.

3) Enlarge specifically says that an object can't get any bigger if there isn't room for it to do so.

" If there isn't enough room for the target to double its size, the creature or object attains the maximum possible size in the space available."

So, in the case of a staircase, because it is usually attached at top and bottom and because there are usually walls and ceilings around, the stairs can NOT double their size - so the spell makes them as big as possible. However, if the stairs are anchored to the floors at top and bottom then there is NO room for it to change size - so it doesn't. Enlarge doesn't just automatically double the size of an object causing it to destroy any surrounding walls etc.

You can also make some rulings if you want how the spell can be used to be more clear ..

- you could require that ALL of the object needs to be within the 30' range/area of the spell to be affected.

- you could define what an object is for your game .. the DMG uses the following loose definition for objects that can be damaged.

"For the purpose of these rules, an object is a discrete, inanimate item like a window, door, sword, book, table, chair, or stone, not a building or a vehicle that is composed of many other objects."

- you could rule that a building of vehicle and the structural parts of each are not considered objects.

Personally, in most cases, I'd allow a caster to target a staircase with Enlarge/Reduce. Enlarge likely wouldn't do anything since the space for the staircase is constrained while Reduce would make it smaller and it would likely fall down. (I'd consider that a cool use of the spell :) ). Reduce might cause creatures on the staircase to fall making a saving throw to see if they land prone and possibly taking some falling damage.

The bottom line in this case is that most of that player's ideas wouldn't actually work because they don't seem to understand what the spells actually do within the context of the game.

Finally, one last comment on your story, reducing the stairs and tripping or causing the 10 guards to fall then killing all but one with Thunderwave ... sounds unlikely if it was run correctly unless all of the guards only have 4 hit points or so (commoners) and were somehow all bunched up on the stairs and then didn't scatter a bit when the stairs collapsed.

Thunderwave only does 9 points of damage on average (2-16) with a saving throw for 1/2 damage. Thuderwave only affects a 15' cube - a cube 15'x15'x15' - each creature occupies a 5'x5' square ... so the targets would have to be packed together and affecting 10 would not be possible unless some were stacked on top of each other. The floor would block the spell from affecting creatures on the upper level since spells don't go through walls for the most part (some will go around if a route is available). The guards would also have had a turn to move and attack between the Reduce and the Thunderwave.

Keep in mind that you should have everyone roll into initiative before the first spell is cast or attack is made. None of these should typically occur out of combat.

-------------

However, the other discussion you should have with folks is what they are looking for in the game. Some folks just want to cause chaos and destruction, kill things, steal things, and generally be anti-social while others actually want to role play their characters. The different play styles often don't mix well though it can work ok at times. You need to figure out whether the player in this case is just trying to be disruptive for the lolz and will continue to do so no matter what (in which case you will need to ask them to leave) or whether they just need to settle in to the character. It is possible to run games with chaotic players as long as they aren't chaotic 100% of the time.