I am a new DM, and my group is three sessions into our first campaign.

One thing that has been on my mind, is that the players often will take actions in combat  that are not the most effective, but that have thematic flair.  I generally like this, but it has led to situations where characters are coming close to dying, in what should be pretty simple combat encounters.  

For example, last night we almost had a total party kill against some twig and vine blights.  Our 1st level barbarian kept wanting to head butt and punch stuff, instead of using his club to hit things.  Also, our 1st level Arakocra rogue, did stuff like attacking with a torch.  Our Gnome cleric hid and did nothing, apparently afraid to get hit.

I like the thematic thinking that is involved with their behavior, but in a life or death situation, the party needs to be thinking about what is effective.  That being said, I can see why a barbarian character wouldn't want to just hit things with a club over and over.  That can get boring.

Any advice here?  Thanks

CR is only a guideline, not a hard and fast rule.

If these are new players, perhaps have some tips to go over at the start of the session, or go over with each player what some of their best options are.

Thank you.  But SHOULD a first level barbarian, generally expect to just spam their simple weapon during combat?  Is that how it generally goes?  How can they act like a barbarian, without being ineffective?

Generally Barbarians have a very limited decision tree of Rage, hit target with weapon, repeat.

If they want to mix in unarmed strikes, by RAW, its not very good. I would consider a feat or magic item that increases unarmed damage if that is a fighting style that want to continue with (or take monk levels), or just flat out increase it while they are raging.

And torches are not supposed to be great weapons. Its a thick stick wrapped in cloth that you set on fire. If you allow the torch to have the finesse property, this would actually be an good move by the rogue, as their sneak attack deals the same damage type as their weapon, and the twig blight is vulnerable to fire (double damage).

A DM I've played with used to give each player a free feat at character creation just to expand the range of decisions we can make. Tavern Brawler is a good feat for that sort of character. Also, if you don't have a monk in the party, I'm sure nobody would be upset if your barbarian says "I headbutt the guy", and you allow them to use the the damage of a greatclub. As long as the player doesn't use this to do meta-gamey bull, then you should be fine.

My advice - Let them do them.  They'll figure it out.  If they're having fun with doing things sub optimally, that's ok.  In the mean time, just make your encounters a little easier.  

Oh yeah, last night, I straight up informed them in the middle of combat, that if they did not start using their weapons, that everyone was likely to die.  I made a decision in the beginning, that I was not going to fudge dice rolls, so I was committed to letting the battle play out naturally. 

To be clear, I love the creative thinking the group is involved in.  Truth be told, part of the creativity should include self preservation though.  In real life, if I am being attacked by a bear, I am not going to punch it, if I have a gun available to use.  Role playing falls apart, if basic motivations are being ignored.

Quote from Epipactis >>

To be clear, I love the creative thinking the group is involved in.  Truth be told, part of the creativity should include self preservation though.  In real life, if I am being attacked by a bear, I am not going to punch it, if I have a gun available to use.  Role playing falls apart, if basic motivations are being ignored.

Danger! Danger! Will Robinson, Danger!

This line of thinking is exactly what you DO NOT want to do either as a player or DM.  You want to engage in sub-optimal choices that further the game.  You want your PC to talk to the seductive voice in the shadows, rather than plug your ears with wax, ignore it and stay inside your camp's perimeter.  Why? Because that way fun lies.  You know as a player that the Dragon has in excess of 230hps compared to your 79, but you charge anyway swinging the battleaxe because that is heroic.

I would suggest that your group wishes to run a game of fun that is removed from RAW and efficiency as much as possible.  Allow them to freely utilize the Rule of Cool, and go ahead and make the Barbarian's headbutts the equivalent of a Monk's Strikes because that is what the Player is wanting.  In the RA Salvatore novels, Wulfgar punches almost as often as he crushes a foe with Aegis-fang.  Let the Cleric be a back line PC but encourage them to use Healing Word, Bless or Bane to help their team.  And, the Rogue was doing a damned good job attacking wood with fire!  Reward them!

Hmm.... the thing is, I generally do appreciate outside the box thinking, and going with the ‘rule of cool’. I am finding myself resistant to an anything goes and will be supported, type of game though. At some point, that stops being DND, and instead becomes Calvin Ball. 

I agree that it was good thinking for the rogue to use fire.  I just want them to apply that idea effectively.  I wouldn’t want them to think they could take out a treant with a book of matches (at least, not without some serious planning). To further clarify, the main issue I saw with attacking with the torch, was how the to hit bonus was sacrificed.  In general, this is the problem with the party not using weapons.  The party kept whiffing left and right, while they were being knocked unconscious, and using up a bunch of healing potions (as well as the actions involved in administering the healing potions).  The torch attack did not connect.

To be clear, I love Critical Role, and how Matt indulges all manner of wacky ideas.  I aspire to similar indulgence.  I am in no way stuck on hyper realism.  Tolkien’s essay ‘on faerie stories’ comes to mind.  I want our world to have internal consistency.

I really appreciate the feedback that has been given thus far.  I hope I don’t sound defensive.  I am just trying to work out how to move foreword.  Thus far, I am inclined to want to figure out how to give the players the opportunity to do what they want to, but to have it happen in a way that is consistent with the realities of our game world, and does not erase the world pushing back on them.

There is absolutely no reason why you can't describe the barbarian's attack as a headbutt while still using the club's damage die. Letting him do that damage while unarmed would change the balance of the game, letting him get in an extra punch at the same damage would change the balance, but if he has club in hand and chooses to narrate his attack as a headbutt anyway it changes nothing. Narrative flair is good. 

Fair enough, but then what happens when he later tries to headbutt, when he does not have a weapon in hand, and is wondering why it is less effective than his previous headbutt?  As a DM, it seems kind of wonky to tell him that his head butts are more effective, when he is holding a weapon. 

If the rogue could narrate even a basic flourish( aka like a baton) with the torch, enhancing the description of the scene, I would let them use Dexterity for attack, rather than Strength, as a reward for their creative thinking. If its just "I hit them with my torch", then no DM help.

For the barbarian, the player should understand that fists are not as good as weapons, and they either need to pay a tax (aka a feat) to make it so, or just be less effective.

If your characters are set on utilizing this out-of-the-box thinking, encourage it by providing ingame rewards that further that.

Barbarian likes to headbutt things? Have them find a Spiked Headband that does 1d8 damage - they could use it as a headband, but could also theoretically wrap it around their wrist to do punching AND spiked damage. Rogue throwing torches? Have the enemy catch on fire, dealing 1d4 at the start of each of it's turns until it puts it out. Instead of fighting against the creative thinking, give them some homebrew items that are off the walls that could encourage them to try new wacky things. Give them a Boomerang Torch that returns to the thrower, a Pitchfork of Bleeding that crits on a 19 or 20 or does an extra 1d4 damage to an unarmed target, a Pot of Thundering that when it smacks into an enemy it does 1d6 bludgeoning and an additional 1d4 thunder.

Set up situations where they can utilize their environment to get creative. Send them into an oil storage warehouse to take down a baddy and watch them light the place up, but maybe they need to bring the baddie in alive so they can't just do it carelessly. Maybe when infiltrating a dungeon they find the enemy has captured some local wildlife that could be fed or otherwise befriended to attack the enemies when freed. Maybe they have a show-off with a bad guy in a castle kitchen where there are cauldrons of boiling soup that the barbarian could lift and fling at an enemy, dealing extra burning damage on each turn.

Also: try having the enemies do their own inefficient behaviors. Bandits? Maybe they try to ambush the party by hiding in a tree and spilling oil on them then lighting them aflame. Maybe THEY try to headbutt right back. Enemies are capable of their own thought and thus their own out-the-box thinking.

When the party gets weird, don't be afraid to get weird with them.

I say play it out as is. There are DMs out there that will homebrew every rule to fit the party, but I'd stick to the rules in this instance. The players will adapt, either by using the most efficient weapons or by picking up feats that allow their character concept to happen. The last thing you want is a 12th level barbarian that still wants to just headbutt and you can't throw level appropriate monsters at them because of damage resistances.

I suggest having an "out" set up for their first TPK. Have a small mercenary (or other) band of NPCs pop up and bail them out in as much of a story appropriate way as possible. The main merc (or knight, soldier, etc) offers for them to join up and make some money, but first they need to polish up on their skills. Set up a "combat training" situation at the NPCs base, where they do some one vs one and party vs party combat. (Clerics waiting in the wings for healing.) In the 1v1 battles, you could even have them fight exact copies of themselves controlled by you. (Using every weapon and ability to the fullest.) Then help break down the inevitable result with the players. 

Quote from Epipactis >>

Fair enough, but then what happens when he later tries to headbutt, when he does not have a weapon in hand, and is wondering why it is less effective than his previous headbutt?  As a DM, it seems kind of wonky to tell him that his head butts are more effective, when he is holding a weapon. 

If both you and your players are enjoying a narrative style of play, and especially if you're having a hard time balancing encounters, I find it helpful to spend a little time with your players talking about the difference between the story telling and the mechanics. Basically this is what you are capable of doing when, and we are going to have to find fun story reasons why. A barbarian can (simplified) do:

1d12/2d6+mod damage while both hands are occupied during the attack.

1d8+mod damage and have a +2 to ac with both hands occupied for the round.

1d6+mod damage and another d6 damage as a bonus action with both hands occupied during the round.

1+mod damage with no hands occupied when there is a reason they would be weaponless. 

This is again a simplification, but if you and your players understand the limits of what a character can mechanically do, I find it has made for more fun story telling.

If your  players are happy to almost die to role play, it would not suprise me if they would be happy to make up reasons why this headbutt did 12 points of damage and another only did 1.

Edit:spelling

I'd allow the rogue sneak attack damage with the torches so that the attack can do more damage output - if that's what you're worried about. I like gamsii's idea of a iron headband or spiked headband and wrist guards for the barbarian - that's cool flavor.

It isn't uncommon for some clerics to keep in the back if they view themselves as healers rather than front like fighters.

For me, it's refreshing to see a group that fights with something other than ultra optimal choices - Great Sword blahblah.

I agree with the homebrew idea. If they have a weapon and don't want to use it but instead headbutt, give them some sort of pointy helmet that conveniently does the same amount of damage as that weapon. Keeps the game and the fun going, because if the players aren't having fun then why would they bother to keep playing. Let them be creative and change things around to suit the while still keeping true to dice rolls. Maybe you can even cause dialogue between the PCs to get that cleric to be more open to do stuff by having an NPC that was tailing them call the cleric a coward. 

Quote from Epipactis >>

I am a new DM, and my group is three sessions into our first campaign.

One thing that has been on my mind, is that the players often will take actions in combat  that are not the most effective, but that have thematic flair.  I generally like this, but it has led to situations where characters are coming close to dying, in what should be pretty simple combat encounters.  

For example, last night we almost had a total party kill against some twig and vine blights.  Our 1st level barbarian kept wanting to head butt and punch stuff, instead of using his club to hit things.  Also, our 1st level Arakocra rogue, did stuff like attacking with a torch.  Our Gnome cleric hid and did nothing, apparently afraid to get hit.

I like the thematic thinking that is involved with their behavior, but in a life or death situation, the party needs to be thinking about what is effective.  That being said, I can see why a barbarian character wouldn't want to just hit things with a club over and over.  That can get boring.

Any advice here?  Thanks

 I made a ranger based off of Steve Irwin, and Jigsaw.

i went 7 combats in a row without ATTACKING.

while a murder hobo eldritch Knight in the party magic missled down, an Allosauraus I had jumped on, roped, and ridden off on.

and TWO. River crocodiles, in the same encounter, that I had tied their mouths shut and dragged to shore.

the aarokokra attacking with a torch. Just means 1d4 instead of weapon and no proficiency bonus, so +2 for level 1 shouldn’t be a big deal.

barbarian going unarmed rage attacks vs weapon...little more painful as losing the +2 rage damage, plus just doing straight 1+STR damage. But maybe that’s his flavor and at level 4 he takes Tavern Brawler. Who knows.

i personally am a big fan of let people play how they want to play. If they die. They die.

Edit: saw your later comment about the Treant.

i played a mastermind rogue that at level 3. Used his blanket, an oil flask to cover his blanket in oil and throw it into a treants branches. I got knocked out in its next attack. But the fighter saw what I did and attacked it with a torch instead of his sword, to light the blanket. This lighting the Treant on fire.

should I have instead attacked the Treant with my dagger. And then been knocked unconscious? Would that “more optimal” attack have been better?

slippery slope man. If you as a DM have to many expectations for what your players will think or do, you’re going to end up disappointed or mad.

yeah as a player i love to do things other than "shoot the bad guy with an arrow"...our DM is usually good about these sort of creative attacks...and they really do make memorable fights/encounters. heck those are the only encounters i really do remember besides like Final Boss sort of stuff. I've lassoed a berzerker and hung him over a ledge, I've dropkicked an ogre, backflipped off things, swung down like tarzan into battle. 

the best way is, perhaps, to get the players to explain what they're trying to do and HELP them do that thing if it's possible.

If your player wanted to use fire against a tree blight...cool, what was their goal...did they think it would do extra damage...yes...then encourage it. If the barbarian wanted to get all WRESTLEMANIA on the tree blights...cool, maybe let him throw one into another with a STR check (you figure out the DC)... improvise the damage...1d6 sounds cool, maybe it's knocked prone..maybe not...your call. unarmed strike...let him make 2, 1 as a bonus action. this way the damage evens out as if he was using his club...