Hello, friends. I have been a DM for more than 40 years. The longer I play, the more I learn to see the metagame--the thinking behind the designs and mechanics, stuff like that. In all these years and decades, I have always run games that are pretty linear: there is some kind of an overarching quest (a questline), there are some side-adventures, and there is character progression as the mission (for lack of a better word) is carried out. Most of my game has centered on combat encounters, with smatterings of exploration and role playing inbetween. 

When I play other games (Call of Cthulhu most notably), there is more role-playing and less combat. Part of that is due to the nature of combat in the BRP (Chaosium) system: it is unforgiving and deadly. So I wonder if anyone has some advice about how to make D&D more about story and character development, and less about fighting. Don't misunderstand me: we love the fighting! But, after four decades, I'd like to branch out a bit and see if there can be more of a narrative as the centerpiece of our games. 

One thought: I have been peeking at Candlekeep Mysteries as a possible scenario set that is investigative rather than combative. Has anyone played any of it?

All advice is merrily accepted. Thanks! 

If you look at the mechanical side of things in the core books, combat's really the only thing that's well-developed in the ruleset - exploration and social interaction mechanics are pretty bare-bones. That said, the DMG explicitly recognizes that immersive storytelling is a valid playstyle. My main suggestion would be to impress upon your players the notion that they don't have to consider their characters' actions through the filter of what's written out in the ruleset; that's for the DM to take care of if needed. It's about expectations too: if your players know to expect more and/or more elaborate social encounters, they'll figure out how to deal with that in character, and as DM you'll then translate that into mechanics if necessary. Not everything has to be dependent on dice rolls or character abilities. Bottom line: they'll take what you give them, and give back like in return. GIve them some setting hooks to hang their character backstory on and they'll likely incorporate one or more of them in one way or another. Have them depend on NPC info to figure out what they're getting in the middle of and they'll go out to ask more questions and do more research. Give them challenges that ca't be overcome by applying brute force and they'll look for non-violent solutions.

Without having seen your style, it's hard to say what the "problem" is, and therefore what you should change. Still, some general pointers:

• Have a session 0 in which you explain your intentions and plans. Don't let them think that they're in for a hack 'n' slash and create Ugg the Caveman Barbarian, only to fund out that Charisma is going to be the vital stat for this campaign. Let them know that it will be a more social based campaign, so they can go for Paladins, Sorcerors and Warlocks.
• Let's get the obvious out of the way - set social based quests. Go recruit someone for something. Form alliances. Do a murder mystery. Things that force them to go down the social interaction route.
• Incentivise social solutions. Start off with being obvious, like "if you resolve any given quest nonviolently, you'll gain an additional 10% in xp (or gold, whatever the motivation is). Once they're in the habit of looking for alternatives to combat, you can tone down and then get rid of the incentives, so it becomes a natural option rather than something you're pushing.
• Make it comfortable. One of the big barriers is that people might feel uncomfortable and so seek more mechanical solutions like combat to avoid it. If they feel comfortable though, RPing can be really fun and more rewarding. If they're enjoying RP'ing, they're more likely to seek out solutions that involve it rather than rolling dice to see how their longsword solves their problem.
• Go all out yourself. You don't have to be a Shakespearian actor, but do different voices. Use your body language. Do as much as you feel comfortable doing to make it immersive and they'll feel more inclined to join in. They'll look forward to those scenes.
• Make a big deal of their backstories. Get them immersed in their character because they seem it interacting with the plot and guiding the campaign. They're a lot more likely to participate in the social side rather than just killing if they're invested in their character and their story.
• Make your campaigns more open world. As their characters start to be able to make meaningful choices, they'll be more willing to use social interaction as a viable solution. It tends to help keep their options open better and so they are not so keen to kill a potential source of quests and story.
• I'd also talk with the players. After so long, they'll have the expectation of your game being hack 'n' slash. First, talking to them will meant hat theu are more likely to engage with your direction than just trying to do what they've always gotten. Secondly, they'll be more likely to get on board and enjoy it, which will make it much easier to make it all work. Thirdly, they may not want to go in that direction, and it will just be a source of friction. This is your opportunity to ensure that you're doing what the table wants. If they don't want to do it, then don't. If they do, they're buying into it and will help you out more.

I have been thinking about making a sandbox--I've done that before with other games--and letting them just bump around in it. The question is...what do they do while they are in it? They are always going to be looking for a questline--do this, then this, then this, then this. 

Quote from MichaelCreminIV >>

So I wonder if anyone has some advice about how to make D&D more about story and character development, and less about fighting.

 Well, that's easy. Just include less fighting and more story and character development, same as you do in CoC. And as mentioned, there is no reason why every encounter or challenge has to be solved with combat. You can bribe guards, sneak away from pursuers or trick the monster into attacking someone else or getting lost somewhere.

No need to neccesarily make it sandbox just because you make it less combat-oriented, just make sure that the obstacles your players face aren't just solvable using violence. Which also allows for some sandbox elements. For example, an NPC I plan on introducing is going to be a very rich and powerful and, more importantly, old and bored elf. That elf lives in the middle of the capital, easy to find for anyone which is good, because the player will at some point need something (most likely information) from said elf. BUT! Unless you as a visitor have the right etiquette, style and entertainment value, the elf will never give you the time of the day. So how do the players get an appointment after being snubbed by the doorman who tells them that such pedestrian lowlives aren't welcome? Sure, they could just break in the door and hope to catch the elf before they flee but that will most likely not get them the information wanted. A better idea is probably to spend a bunch of money on completely extravagant and outragous outfits, have the druid shapeshift into an ostridge, dress up the halfling as priest and ride the ostridge/druid will juggling and singing bawdy songs. So where do they get the money for outfits, being poor adventurers? I dunno, perhaps on a sidequest, or borrowing the money from a bank. Or raiding a museum for exotic clothes. Or any number of ways the players can think of. Go crazy. :)

I think Panj put the cards on the table.  When you read the class descriptions, they are loaded with combat options, and a few fluff things.  The best part of the character sheet for out-of-combat player information is Skills and Proficiencies.  Further, the party advances when they accumulate XP, which is usually measured by "What have you killed lately?"

So, to have an RP focused game, you will need to lay it all on the table during session zero.  Then, if the players think that sounds like fun, you will get them to roll up RP characters instead of combat characters.  You may get a lot of Bards and Rogues, and some Paladins and Sorcerers.  Basically, you can expect more reliance on Charisma and less on Strength.

As the DM, it will then fall on you to develop a plethora of NPCs with back-stories.  And then you will have to develop a story that makes these NPCs meaningful.

I think you intent sounds like fun, but I also think I am in the minority of the player base.  But that doesn't matter.  It is about what you and your players want to do.  But be up-front with them when you pull the group together.

Quote from MichaelCreminIV >>

They are always going to be looking for a questline--do this, then this, then this, then this. 

That makes it easy then: if they are looking for a questline, they'll jump on anything you dangle in front of them. Let them overhear an interesting rumour while drinking at an inn, let guards warn them against something when they arrive at the city gates, even just let one of them notice something odd about the townsfolk. Prepare them before the campaign starts that you won't hand them their next goal on a silver platter, that if they stumble across something interesting they should simply go and check it out. You can make it extra easy at first by dropping hints in-game: the rumour can be about a specific location like a local cemetery or a tower, the guards can mention the name of someone who knows more about whatever threat they warned the PCs against, and when you decribe the oddity they notice you can hint it's the sort of thing a physician or hedge witch would know more about. Heck, if they struggle a bit you can flat-out make out-of-character suggestions about things they might try. None of this has to go perfectly from the start. It's more than fine to ease them into things. 

Quote from MichaelCreminIV >>

I have been thinking about making a sandbox--I've done that before with other games--and letting them just bump around in it. The question is...what do they do while they are in it? They are always going to be looking for a questline--do this, then this, then this, then this. 

 As Pangurjan said, that actually makes it easier. If they're not looking, you have to really work to get them to do a quest, and it's a PITA. If they're looking  it gets easier.

You highlight your hooks. For example, you have a quest where a mother's children has gone missing? When you describe a scene, include an upset woman who is clearly distraught off to one side. A person has been kidnapped by goblins? Have an NPC notice their attire and ask finthey can go find this person who went hunting and never came back. An evil Cult gas set up shop nearby? Have a few shifty characters be described, perhaps the characters will notice signs of rituals being performed. Someone needs help? Have them follow the adventurers or stare at them while they're at the tavern.  And so forth. You can also have job boards going for official quests, set up rumour mills for characters to pick up in the local tavern for less official ones. If you want a questline, you can set up a chain in the quests themselves - in the first quest they find clues that lead them to the second, which naturally leads to the third, etc. Yernativrly, you can set up a questmaster.

If you've ever played Elder Scrolls, they have multiple ways to introduce quests, and it's worth paying attention to them. Theynwill need adapting, but then ideas are pretty solid.

The ley to open world/sandbox is that they feel like they're choosing what happens. You have no obligation to make it hard for them to find it. 

Also a 40-year DM here technically only 38), so I feel you. A big thing to keep in mind is it's not just you; your whole table has to buy in to the change. One way to really shift the thinking surrounds xp. It always kind of irked me that the only way to get better at, say your deception skill is to go kill lots of stuff so you eventually raise your proficiency bonus. It never really tracked with me how that makes you a better liar. But if you stop giving xp for killing things, and start giving story xp, or milestone leveling, then it really changes the game. I'm assuming you give xp for overcoming obstacles, not just killing (sneak past the troll and save the hostage gets you as much xp as if you were to kill the troll to save them is pretty standard nowadays.) But make sure you communicate that sort of thing to the players. If the players have the mindset that the only way to advance is to kill things, then that's what they'll do. So make sure they know they'll gain xp, or level up for finding a clever way to overcome a challenge instead of just killing it, and they may start finding other ways to solve problems.

And for the sandbox. Just let them bump around for a while. Within a few sessions, they'll latch on to something and assume that was the plot you had in mind all along.

Let's first be clear about what role-playing is. It's about making choices as your character. A 3-hour tavern scene with no conflict could be a lot of fun but it doesn't make a good story and it's not going to be satisfying for many players.

One thing to consider is that you can use combat as a narrative branch. Drop hard decisions right in the middle of a fight. Hostage situations are an example that's familiar to everyone. Role-playing can and should be an integral part of many combats. Choices made when the stakes are high are much more rewarding than choosing what to drink from the tavern menu.

Introducing a new or alternate goal in the middle of a battle forces characters to choose which to pursue. Save the NPC or ensure the BBEG can't escape? What if the NPC is a criminal? Or a killer? What if the BBEG is well-intentioned and could possibly be groomed into an ally? Try to introduce choices where the morality isn't cut-and-dry so they really have to think about what their character would choose. Choices where the party isn't all likely to fall on the same side. Now they have to think about how their character handles conflict within the party.

The other thing to think about is conflict outside of combat. Introduce political conflict where the goal is to gain the favor of the public or enlist political allies. Instead of raiding the kobolds, build up the defenses of the town so they're no longer a good target. Make sure there's clear options for handling a conflict and at least one of those options isn't "kill stuff."

Most of all, make it clear to the players that they will be rewarded for unconventional choices. If they rid the forest of spiders, the townsfolk will be pleased. But if they stop the killing when the telepathic spider-queen offers a truce and then work on improving relations, the town can eventually have a defensive ally and trading access to fine silks. And now you get to have little flavor scenes like children playing alley-soccer with a young spider as goalie.

The more visible you make the consequences of their past choices, the more they will think about the possible consequences of their future choices. 

Once I started adding these kinds of alternate resolutions in my campaign, my players started actively seeking them out just to see what would happen. Of course sometimes these alternate paths backfire for the party - they shouldn't just automatically be the best way to go or you've just replaced one railroad for another - but as long as they are interesting and force PCs to make choices based on their values and beliefs I think they make for a rewarding story.

I think there is a false dichotomy here and it is a symptom of not understanding what an encounter is.

Combat is not an encounter. Combat is simply one of the tools to resolve an encounter.

An encounter is when there is an obstacle between the PCs and their objective. Maybe its a monster guarding the treasure. Maybe its a guard stopping them from entering the ball. Maybe its a merchant refusing to sell them an item. Maybe it is an official refusing to divulge a piece of information. Maybe it is a locked door.

Roleplaying is simply deciding, "If I were a enven/dwarven/whatever barbarian/warlock/whatever, how would I overcome this obstacle?"

As GMs, we need to stop thinking just about "combats" and instead focus on giving the PCs goals and placing obstacles in their way. We should make the obstacles so that the players need to use all their abilities. Some obstacles should allow for multiple solutions, some should allow only one or two solutions. Many of the obstacles should be set up so that combat is a bad choice. But for every encounter, we need to be continually thinking of what the PCs' goals are. 

A way of keeping this in our memory is to write a dramatic question for every encounter, in a similar way to how old movies used title cards to set scenes.
"Can the PCs gain entry to tbe masked ball?" "
Can the PCs get the location of the cult ritual before it comes to its grisly end?"
"Can our heroes get their audience with the lord of the city?"
"The river is rising fast! Can the rogue open the lock before our heroes meets a watery end?"

Yes. 

Some people forget to roleplay in combat as well. Beside that, simply reviewing your characters stories and weaving character relevant engagement can motivate roleplaying. You can also use milestone so negotiation and survival count toward progress rather than rewarding combat exclusively.

You can target specific players to engage when an NPC is talking, and you can enable roleplaying alternatives to scenarios so the players creativity can guide the way, rather than expecting them to play the scenario you scripted every time.