Hey, I’m new to DMing and I’ve just started out my first campaign, my players know this as they are also new to the game and we are all trying to figure everything out together, and overall I think I’ve been doing a decent job at things as a dm, of course there are things to work on but I’ll get better with experience. But I currently have an issue, one of my players firmly believes that it’s their job as a player to mess with the DM and to do the dumbest sort of things for the soul purpose of trying to get me to become more stressed and mess up as a DM, they have kinda been using the same sort of reasoning of “it’s what my character would do” as they are playing a chaotic neutral. For example, they are in their friends house and the door that leads outside is locked, there are glass windows and they also have thieves tools on them, the first thing they say is that they are going to use the thieves tools but then change their mind and say, “wait no im going to try and set fire to the house”. What do I do and how can I handle this?

Let them do what they want and hand out appropriate consequences. Sure, they can set their friend's house on fire, but then they might no longer have a friend.

It's OK for a player to have their character do "dumb" things. Just consider what, if any, repercussions there might be. 

I forgot to mention that generally speaking, actions against another PC should be forbidden in your game. No attacking another PC, no stealing stuff from another PC. 

These actions are bad news in new and experienced groups.

Remember: as DM you are the King of Karma.

“So you light fire to your friend’s door. It starts to burn, and after about 30 minutes it’ll be weakened enough to break down. In the meantime, the fire spreads. Give me a CON save. OK, so you fail. You are poisoned from smoke inhalation. The house around you continues to catch fire. ... You’re jumping out the window? Acrobatics check at disadvantage due to smoke inhalation...  So you land outside and 50 men from the village have formed a fire brigade. They know what the owner of the house looks like, and know you’re not him.  One yells out “Arsonist!” And they all move to attack you. Roll initiative. At disadvantage.”

I handle things a bit differently. I never like to say no to my players, so when they want to go off the rails, I do some passive aggressive things.

You want to set a fire? Sure, go ahead. What's this, you don't experience with fires on a doorway or on large objects? Roll with disadvantage. You are lighting a door at night? -10 because its misty outside. You are using magic fire? Sure, roll with disadvantage and -10 because your friend knows what kind of friend you are and had the house protected with anti fire magic. Ah, you are using oil to make the fire burn better? Hm, I don't recall you buying burning oil. That looks like the vendor sold you olive oil. Looks like you got fooled. I guess you should've made sure it was burning oil in there. You hired a guy to do it for you? Looks like he failed, his slieght of hand is -10.

Sure. Go ahead. Let them TRY. You, as a DM, can also say "Sure. go ahead. But-" 

https://theangrygm.com/all-aboard-the-plot-train/

scroll down and read the portion about “Contrarian A$&holes”

it sounds to me like you have a contrarian a$&hole in your party. 

the whole article is a good read in general and you may want to send it to this player you are having trouble with.

It's time to talk to the player (not the character).

There are certainly campaigns where it's OK to wreak havoc, and it can be fun from time to time, but most of the time those "campaigns" tend to be short lived. It sounds like your player thinks he/she is playing in such a campaign, while you as a DM does not want your campaign to be like that.

Talk to your player(s), try to explain why this doesn't fit in this campaign. Tell them that friends will be friends no m ore if he keeps on burning down their houses. Tell them the local guards will start to harass them, they won't be allowed to stay at the inn etc. Hiding behind an alignment isn't reason to not be thrown in jail (and it sounds like that's where the character is heading).

The reason I would start this as an out of character talk, is because it sounds like it's mostly about (wrong) player expectations. You say both you and your players are quite new, so I think the player deserves a heads up from you as the DM before you throw some in game consequences in his face. Then, next time he wants to do something like burning down a door, remind him that it will have consequences...

And consider making a three session one shot where they play a band of ravaging orcs that set fire to everything and can do all the stupid things they want to get it out of their system. That can be fun too from time to time, but as said - it usually doesn't end up as long campaigns.

I’d agree with the others, let him burn the house down and deal with the consequences of now being wanted for arson and also not getting whatever it was in the house they wanted. 

Id also agree with the others to have an out of character discussion with this person. I keep hearing about new players who seem to want to act out their anti-social crime fantasies and say “but that’s what my character would do.” Point out that they are not passive observers of their character, they are making their character be a jerk, not the other way around, and no one likes playing with a jerk. 

Worst case, if you really want to show him who’s boss say, “a bolt of lighting comes from the sky with pinpoint accuracy and strikes you dead. You are completely evaporated. It doesn’t even scorch the grass under your feet. It’s like a vengeful god was angry with your behavior. Roll up a new character.” We used to call it getting blue bolted, I’m pretty sure it was in the 1e DMG.

"Hey, could you please stop that, it's making the game less fun for me. Thank you for understanding. Maybe I'd be more OK with it if I had a little more experience DMing, but I'm still trying to figure out a lot of things."

If they respond negatively to a reasonably worded, sincere request, then they probably don't respect you and you shouldn't play with them. 

"It's what my character would do" is universal code for trying to excuse a--hole behaviour that messes with someone else's fun. You should tell the player to stop it.

As usual, Matt Colville has an excellent 10 minute video on this, called "The Wangrod Defense." In Colville's terms, your player is being a wangrod. And he needs to stop.

However, quite apart from that, there is this in your OP:

 one of my players firmly believes that it’s their job as a player to mess with the DM

This player may need to be taught the hard way that it is never, ever, a good idea to mess with the DM. In fact, it is a decidedly bad, potentially character-lethal thing to do, messing with the DM.

In the old Champions rules, they had a section about this, in Champions III, I think. They asked the question, why do the player characters almost always win? It's NOT because they are more powerful -- look at the CRs of some of the monsters. It's not because they are smarter -- villains often have a much higher INT. It's not because they are better at planning -- the DM has had months to plan this out for the villains. So why do they win... Almost EVERY. SINGLE. ENCOUNTER? 

The Champions writers' answer is that the PCs have something the villain NPCs can never have: the GM's good will. The villains are trying to kidnap, kill, murder, destroy, maim, pillage, plunder (usually). And the PCs are heroes, who are risking life and limb to save innocents from being kidnapped, killed, murdered, plundered, etc, etc. So the GM is in this odd position of making up encounters and designing villains for the purpose of the players beating the encounter. The GM, usually, plans to lose. (Or rather, plans for the side he controls to lose.)

But this only holds true if the players retain the GM's good will. If they do something to lose that good will, then all bets are off. Now, things aren't going to go their way all the time -- or maybe any of the time. Because as a GM it is trivially easy to make an unbeatable foe, and unwinnable encounter. Just give the monster an AC of 100 and 20,000 hp. Yeah, good luck beating that....

Now as DMs we almost never, ever do this. But, if your player is being a Wangrod... if he is purposely trying to mess with you as the DM, then, if reasoning with him (which you should try first) doesn't make him stop, then it's time for you, as the DM, to mess with him as the player. I normally don't advocate going tit-for-tat like this but if a player is being impossible, it may be your only choice short of booting him from the campaign (which, for social reasons, we often cannot do).

Try to talk to the person first. Try to reason with the player, rather than taking it out on the character. But if he's going to really mess with you like this, then he needs to see what happens when you mess with him back. You're the DM. It'll be SO much worse.

Beat me to the wangrod defense video.

A lot of people here are suggesting consequences, and that's good advice... if you're comfortable potentially crushing their character to serve as a warning. This method involves bringing the hammer down where appropriate and grinding their character into dust so that others know not to do as that character did. While this may convey your message, it also has the potential to lead to an angry player if their character is suddenly executed and that's pretty stressful as the DM and toxic to the group. Considering and building consequences for stupid actions also adds to your workload, and that's something to avoid.

Consequences and consistency are important, but before you go off the deep end the best thing to do is always to have a talk with the player in question and address their behavior. Discuss your grievances and see if they're willing to listen. This should always be your go-to solution if you're having a problem with a player, especially if it's a toxic player like the one you're describing. Tell him that you're willing to forget what he's done so far, but if he continues you're going to start adding consequences and they could potentially be lethal to his character.

Also, "Chaotic" in general refers to a love of personal freedom. It's used for characters who are governed by their emotions and personal beliefs more than laws or traditions and who hate others wielding authority over them. Chaotic Neutral is the alignment for people who do what benefits them first and foremost and often end up in opposition to authority as a result of their methods. It's the completely selfish alignment, but that's not the same as being a psychopath. Chaotic Neutral characters should still act with their own best interest in mind. Your player isn't playing Chaotic Neutral, they're playing Chaotic Stupid.

This method involves bringing the hammer down where appropriate and grinding their character into dust so that others know not to do as that character did. While this may convey your message, it also has the potential to lead to an angry player if their character is suddenly executed and that's pretty stressful as the DM and toxic to the group.

In general I would agree with this. However, I submit the following points:

First, "consequences" does not necessarily mean "execution." A character could be run out of town, lose his ability to gain audience with the local magistrate, forfeit his share of a bounty, etc. 

Second, I submit to you that, as portrayed in the original post, this player in question does not seem to care that he is angering the DM, and as a result, I'm not sure why the DM should care about angering the player. Angering the rest of the table, maybe. But I suspect if the player is ticking off the DM on purpose as described, the other players are probably unhappy with the situation and they may very well welcome and applaud the DM putting his foot down.

Third, regarding bringing the hammer down being "toxic to the group," the player is already toxic. He's already ruining the DM's fun, and purportedly doing this on purpose. There are only two ways this can go. Either the DM finds a way to stop it, or the group is going to fall apart, because unhappy DM will equal unhappy players before very long at all, if not already. So either the player has to be made to STOP, right NOW, or the group is going to be doomed. 

I agree with you that talking OOC to the player is the first thing to do. But if that does not work -- and if the player is doing this on purpose to upset the DM as described, it may well not work -- then either the player needs to be booted or the hammer NEEDS to be brought down. A situation like this cannot be allowed to continue or it will destroy the campaign.

Troublesome players that are just looking to disrupt the game are always a tricky one to deal with. While in principle, letting your player simply be punished in-game for doing stupid things might seem like you're dealing with the problem, you're in danger of destroying the game.

From their perspective, 'D&D is a game where I do what I want. Why are you dictating my fun?'

By letting them do stupid actions, but then ultimately giving them a (usually character ending) punishment in game, you might feel like you're letting them have their choices; but that's just a thin facade over what you're really doing - which is saying 'no you can't do that'.

It's much better to set the expectation of the game session beforehand. While D&D does have immense freedom, there is an underlying order that holds that freedom together. There's a general assumption that the DM will provide a world, full of stories and adventures, and the players will work together to go on them - and you'll all play the game in a way that furthers that objective.

That breaks down when a player decides that their objective is different. Their objective is to make everyone laugh out-of-game, or to 'beat the DM' by doing stupid things in-game. Their character's behaviour is an in-game excuse for out-of-game behaviour - so don't try and solve it in-game.

It's sadly fairly common with new players that feel there's some competition between the players at the table, and the DM. But it can also be fuelled by the social insecurity of roleplaying. It's easier to play the fool, and do stupid things in game to mask the embarassment or awkwardness of roleplaying a serious character.

It's tackled by setting expectations out of game with a sincere explanation of what's expected from your players - and hopefully the majority of players will agree with you. An out of game chat to explain that their behaviour is ruining the game for you, and that you'd hoped you could take it a little more 'seriously', could really help sort things out.

And if they don't care that they're making the game miserable for you or others, then they probably shouldn't be at your table.