I've pæayed and DM'ed for a collective ten years by now, but I don't have that much experience playing in or designing dungeons. My old DM used to not make them, and I have only made a few for my current campaign - I feel like they were good and my players enjoyed them, but still, I haven't really found my style yet. So, how do you design dungeons?

I notice, for example, that when I chart the dungeon I have tendency to make a "boss chamber" and then try to create reasons for the players to visit that room last. Do you have a "set course" through the dungeon, from a beginning until the final boss of the dungeon at the end, or do you like more freeform dungeons where you can enter or leave the dungeon in multiple ways? Do you even have a boss?

I like to first imagine what kind of monsters I'd like to have in the dungeon, and base my ideas around that. Sometimes that decides what the dungeon will look like, but other times it's coincidental.

What about dungeon loot? Do you prefer to leave it in one place or do you leave it around the place, so the players have to search every crevasse for it?

Then there's tge practicallity of it - how do you show the players where they are physically without revealing the entire dungeon to them? And what do you do when a random battle emerges that you weren't prepared for?

That's just some of the things I've been thinking about; I'd love to hear how you do it for inspiration! :D

 I start by choosing the dungeon’s purpose.  

A warren for Kobolds needs workspaces, a kitchen, a “throne room” for the worship of their racial god, sleeping areas, mines, a “nursery” for the eggs, etc.  It would be heavily trapped, have narrow tunnels, and multiple entrances and exits.

An ancient temple needs a chamber large enough for the rituals of the faith, sleeping and living areas for the priests, cultists, and visitors, storage areas, workspaces, rooms/enclosures for pets and guard beasts, and perhaps a vault for offerings, tithes, spell components, and so on.  The vault is probably trapped, and there might be a password required at the entrance to the most important chambers.

Then I factor in the age of the dungeon.  The ancient temple has probably suffered the ravages of time; the building might have become buried, passages might have collapsed, sinkholes might have opened up, guard beasts may have bred and/or mutated, or the door to the vault might have been sealed by the pressure from the weight of the earth above.

Inhabitants come last, for me.  The ancient temple might have mummified priests, ghosts of the humanoid sacrifices, guard drakes made immortal by the dragon who was being worshipped, goblins who have moved into the outermost chamber, or maybe even an egg from the dragon “god,” preserved by the dragon’s magic.

I try very hard to maintain a certain amount of realism in my world.  Just because it’s fantasy doesn’t mean it’s illogical!  So I try to have a reason for the primary features in my dungeons.

Like the others said, purpose is the first thing. For me, that’s what is it now and what was it originally. Originally is good enough to start.

 Next, I get a blank piece of graph paper and the dms guide for the random layout section. I don’t use it for the whole time, just long enough to get an idea for what I want the dungeon to be like. Ex: long and windy, straight set of rooms, maze, etc. 

Then I brainstorm who the current main inhabitants of the dungeon are. This can be different for different parts of the dungeon, or the same. Usually I have a shallow monster type, like a group of orcs, who have moved into the first caves, and then some secret or trapped doors that lead to a deeper part with old guardians and monsters left by the original creators. At the end, I plan a final boss, guarding the majority of the treasure or the party’s main goal.

 Finally, it’s time for the descriptions and the random treasure. I usually write room descriptions like published modules, and then dm notes for stats of monsters and terrain difficulties. My descriptions show current state and hints of what it used to be.

I do have a tendency toward final boss rooms- but I try to have more than one way to reach them, so the players don’t feel railroaded. The rooms preboss have order choices, but there is a clear distinction between sections.

My dungeon loot is usually mostly at the end, although I do like to sprinkle treasure throughout rooms as a reward for searching.

 To show the PCs where they are, I use a dry erase board and draw the map as I go, from my DM’s copy behind the screen.

 To avoid random battle problems, I always have my mobile device ready with the encounter builder, to search for stats and check balance. I also have the monster manual and Volos guide with me if the monsters are from that instead of base rules.

The first step is to conceptualize the dungeon. What is this dungeon? Is it a tyrant's castle? An ancient mysterious ruin? A temple to some evil god? Crazy mage's tower? Now give it a bit of history- who built it, why was it built, etc. Finally, answer who or what currently dwells here and why the adventurers are interested in the dungeon.

Now that you have a concept, step 2 is to outline the dungeon. This can be as simple as a rough sketch. Just get a basic idea of the geometry of the dungeon. Sketch the halls, rooms, and any other areas.  Try to use different shapes, such as circular or triangular rooms instead of all just being rectangular.  Have Hallways that shoot off in a diagonal fashion.  

Step 3 is to furnish and document the dungeon.  This is the part where you go hall to hall, room to room, and add in the details of each encounter the players will face and furnish the rooms and halls with objects and treasures, while documenting everything for when you run it.  When thinking about what types of encounters the players will face, I find it helpful to keep a list of different types and add variety to the dungeon.  Here is my list

• Battle- Armed confrontation between the players and villains or monsters.  (roll initiative) 
• Trap-  Hidden and dangerous. 
• Skill Check-  Players need to recognize and apply a skill check of some sort to succeed.  Could be an instrument or tool, too.  
• Hazard- Hazardous environment impeding the players such as a cloud of poison gas. 
• Obstacle- Similar to a hazard but less directly dangerous and more about stopping them in their tracks, such as a great chasm.  
• Puzzle- Could be a riddle or some other tricky puzzle.  
• Secret Door- Should be a vague clue the secret door is in the area, to get them on its trail.   
• Social- High stakes social encounter with consequences or missed opportunity if it goes poorly. 

Try to think about what each room is and what should be in it when furnishing. 

Once the dungeon is furnished and documented, the only thing left really is to balance the encounters and add in tokens for monsters, traps, or objects you intend the players to run into. This can be minis for an in person table top or virtual tokens for online.

I usually go by a process of a few points. 

1. What is the dungeon? Literally, what the dungeon is intended to be. A large abandoned keep in a swamp, a floating facility that's sunken to the bottom of a lake, etc. Pretty much anything can be a dungeon as long as it either keeps the players from easily escaping it or possesses some incentive for them to travel through it.

2. How does this link to the party? The best way to make a dungeon interesting and something fun to look back on, in my experience, is making it so that it is tangentially linked to the party's ambitions, quests, or goals. Yea, you could just make them travel in a mine, but without a good reason for why they're doing it, it will just feel like they're running around in a tunnel murdering kobolds. The link could be anything as simple as a bounty to as complex as directly linking in with a character's detailed backstory. 

3. How is the dungeon structured? Once you know what the dungeon is, and why the party will be going there, now you have to decide on what it's going to look like once they get inside. This should ideally reflect the dungeon well enough; A mansion-style dungeon should have multiple floors and rooms, while a maze-like dungeon should have plenty of winding corridors and tunnels. Also think about how you can flavor the dungeon's innards at this stage. Describing a hall as "Long, with several doors" is succinct and gets the point across, but it doesn't exactly put the image of the surroundings into the player's minds. If the dungeon is a king's castle, add things like decadent, gaudy golden ornaments and banners. If it's a crypt, add some coffins and stuff. Keep the descriptions on-theme to repeatedly convey this imagery.  Hazards should reflect the setting as well. A mage's tower will likely have magical traps, for example. 

4. How are the players rewarded for clearing the dungeon? Finally, once they've clawed through your terrifying beholder den or whatever, the party will expect some reward. I'm big on making the reward thematic with the dungeon, but many times this isn't the best case. Items are always awesome, but its also worth considering less tangible rewards, for lack of a better word. Maybe the party has secured a powerful allegiance with a notable sponsor by completing the dungeon and retrieving an item that would be useless in their hands. Maybe their reward is more story-driven; A family member saved from the clutches of a crazy cult, Justice served to someone who wronged them, etc. A few smaller item-based rewards and one big world-based one has often been received much better than powerful artifact, at least by my players. 

This is just my personal process, I'm still fairly new to DMing but my players seem to enjoy my dungeons, so I guess it works. 

I've been out of the game for a long time and just getting back in, but I'll give you my methods for when I did Bulletin Board (BB) sessions (basically play-by-post in forums) and for designing modules in Neverwinter Nights 1 & 2.

Themes and Use of Case (UoC) are important. Themes are obvious. An abandoned mine, a deep cellar labyrinth, a cave dwelling, etc. UoC is a bit different. Does it make sense for a mine to be where you have placed it? Does it tie into a deeper mystery? Many mystery novels use the trope of discovering an abandoned mine to advance the plot. Maybe you have a cave inhabited by bugbears that have developed an anarchist non-aggression pact society and just want to be left alone, but will viciously defend themselves if needed. These differing kinds of themes can make the same locales seem fresh. Even other more minor inflections, like a normal mine versus one that goes down seemingly for forever, can make a very different thematic difference. Is that wood elf getting anxious about wandering for 3 days deeper into cramped caves? I bet that rock gnome and dwarf feel perfectly at home!

Dungeons for the sake of dungeons is okay if you have a group that wants Diablo-style hack & loot sessions. A lot of people enjoy this kind of stuff, so it should not be discounted and it certainly serves purposes in even more role-play (RP) centric campaigns. When designing this stuff, I always look at the purpose. Is this a story-driven "dungeon" or is it meant solely for tactical combat? Is there a mix of the two? Then plan accordingly. For story-based dungeons, details and careful planning are crucial. You want every turn and room and corridor to have meaning, even if this meaning isn't explicitly stated. For tactics, you want rooms that facilitate what you're doing. Is there a goblin ambush? Then columns, stalagmites, etc. are crucial to cover archers while the melee engages the party. Things like that. Maybe you have a group of bugbears that crawl out of a some holes in a cave, but the room is mostly divided by a large acid pool in the center of the room, which allows spiders to be physically separated from the party and shoot webs.

Storywise, what would fit the motif you are using. Crypts are not going to arranged the same as mines, as dungeons, as cellars, as hidden passages, as caves, etc. What kinds of clues and items can be provided to further the story, provide misdirection, allude to treasure, etc. I find dungeons to be really good pacing devices and a great way to unfurl story content without hoping the party stumbles upon it or force-feeding it to them. Think of it like breadcrumbs. They discover little bits along the way that leads to a grander reveal. For example, a sacred temple in Chult that has been long abandoned may hold numerous secrets to what the party needs to confront the shaman that has gained immense power and is oppressing the locals. This example can be used to provide clues for further locations (heroes go yonder!), provide the actual elements/items needed to confront the Big Baddie (heroes grow fonder!) or a mixture of both.

To answer some specific questions

Do you have a "set course" through the dungeon, from a beginning until the final boss of the dungeon at the end, or do you like more freeform dungeons where you can enter or leave the dungeon in multiple ways? Do you even have a boss?

It depends. Sometimes you don't need a boss. Maybe just finding the skeletal remains of that lost wizard apprentice (whose master claims they don't know what happened to them) is enough. 

What about dungeon loot? Do you prefer to leave it in one place or do you leave it around the place, so the players have to search every crevasse for it?

I like to make loot special. It's rarely just lying around. That's more of a video game trope. If you have a suit of armor worth $150k, would you leave that lying around? Would you leave your $20k sword just sitting in your car with the doors unlocked?

Then there's tge practicallity of it - how do you show the players where they are physically without revealing the entire dungeon to them? And what do you do when a random battle emerges that you weren't prepared for?

I will be using Roll20 for battle maps and screen-sharing on Zoom for my upcoming campaign. Roll20 (the free version) has a manual "fog-of-war" option to reveal terrain to the players. For my campaign, several people are new to D&D and not very tech-savvy, so I will screen-share one monitor of Roll20 from a player perspective and move all tokens for players myself rather than letting them do it. This is easy for me because I have my laptop and a desktop with quad monitors. Which allows me to use FruityStudio for microphone effects (sound like a demon, other realm effects, long echoes, etc.) and to use BananaSplit for sound effects (music and sound FX, like fireballs, death screams, thunder, wind, etc.). It's a fairly involved process, but I don't mind at all. In fact, I love the spending the time to make these theatrics work and I found (from doing a practice session) that the players all got really involved and had a great time.

In person, I would always use a generic blank battlemap and simply draw where trees, streams, doors, walls, etc .were and reveal as players progressed, erased as what they went beyond. If there was something big and amazing, then I would augment with scenery props and even entire dioramas. If it's just some random encounter in the woods, I'm not going to bring out trees and other tiles. If it's a meeting with a wood elf enclave that gets ambushed by Duerger, then I'm going to try and make it look memorable.

For random encounters, I always have a few groups of random bad guys to throw around. Not only for instances of random encounters, but also in case the players rolled really high and decimated a tough foe; I might pepper them with some middle to low difficulty encounters soon after to keep them on their toes. Or maybe a player does something stupid in the city and you need some town guards to reply. Maybe the player is being a jerk and some locals don't take kindly or the bouncer steps in. It's best to always plan a few extra random encounters than just the planned encounters that you know you will need.