I'm going to keep this as brief as possible while not so short that advice is not possible. My 4 players are a mix of lawful evil and chaotic neutral. Their motives are wealth, self glory, and one power within like minded people. The characters work well together and the players have established good virtual relationships between the characters. They have gone on a couple adventures before staring HotDQ. Starting out with incentives to begin the adventure they have the incentive to pilfer and find gold etc in whatever comes as they progress in the adventure. One favors combat with monsters and enemies and finds that motive enough to set out to greenest. The one a dragonborn priest and deity of choice Tiamat seeks power in the ranks of the cult. The last there for a mixed bag of whatever comes along and strikes her fancy. So going forward with the idea that the characters don't care if Tiamat returns or not.

With their incentives established to go to greenest they arrive and save the town folk from the kobolds. But with a dragon attacking after dialog with the town folk they have no interest in going to the keep, the intended direction of the plot. So a quick rewrite to use Save the Mill part of chapter one to move the plot along. They stop the kobolds from burning the mill. Take a short rest and await the return of of town forces to continue to guard the mill. This rewrite moves time forward to when most of the raiders have left added including the dragon they had no interest in getting close to. NPC giving them incentive to go to the keep at this point works and now players are at the keep to move the story forward. This leaves only one part of the chapter left. Half Dragon Champion. I prepare to have to rewrite future story involving this enemy and let the players some freedom with this dual and kill him and take up disguise to infiltrate the cult. Dragonborn or half dragon I'm figuring except cultists close to the half dragon wouldn't know that the player isn't the other. All is good at this point and players feel motivated to infiltrate the raiders camp chapter 2.

Chapter 2 the goal is the players to sneak around camp find Leosin the NPC mentioned to them to see if he is ok or not, before they depart. The incentive is to find out what they are up to and get more riches as a result. The dragonborn has the idea of recruiting kobolds in the camp to sway them that the people whom they are working for don't have their best interests in mind and that they would be better off working for him. Not a problem a little rewrite here and instead of stealthing around the camp this ruse results in camp civil war as the kobolds fight the not kobolds part of the camp and the players move ahead to the expected direction of this chapter. They find Leosin but have no direct interest in freeing him but are impartial to his fate whatever it is. Because written in the adventure Leosin doesn't really want to be rescued because he wants to learn more the players choice here doesn't change that.

Chapter 3 dragon hatchery. Instead of the characters returning to greenest and being sent back to the camp by Leosin who was not rescued and taken back but left at the camp to find out more directs the players to the cave. I use details about how nobody seems to be allowed inside and and Leosins concerns of anything going in or out of the cave. And that the leader of the camp may be inside as incentive to move forward to that part. This chapter progresses with no need to rewrite except for the dragon shrine chamber to make the encounter there to make sense and then direct the players with clues to to guide the characters to find the map that leads to the caravan plan.

At this point I give the players two options written in as a note left to them my Leosin after finally slowly managing to free himself now that the camp was busy fighting itself while the players we doing the hatchery stuff. The options are if you would like to join me I'm going to track this caravan that is leaving the camp. Or I have contacts in waterdeep you can meet up with and report whats going on. This being a plot skipping device to see if the players have interest in a long journey following or infiltrating the caravan or moving ahead to rise of tiamat. Without out of game stating this is a skip content option. They choose the caravan path. And so catching up with Leosin they infiltrate the caravan. So far so good again. After the crazy rewrite of the camp chapter to work with the players idea of how things would proceed.

So with the caravan the players get the crazy idea lets steal everything. Which would derail the entire adventure before it got started. So quick thinking on my part I make the numbers of guards etc in the caravan so great that as players they realize it would be suicide to attempt that. But they still want to pilfer and otherwise cause some mayhem along the way. Their first idea is to trim down the size of the caravan. I let them sabotage a few wagons but the crew of those wagons move the contents to other wagons and stay with the caravan. Players were disappointed it didn't work out as they expected and came up with some new ideas moving forward. I set some monster encounters along the way instead of the ones written in the adventure as they didn't fit with the characters and what not. I should mention at this point the players got a sailing ship that they took down the coast to begin the adventure. So it was sent ahead of them to baldur's gate so that Leosin could possibly set something up for when they get there to try to gain more control or influence within the caravan.

This is the first part I really mess up as the GM. I let the players plan and scheme to take out some caravan leaders to try and redirect the caravan onto their ship. My short sitedness of this I react with another derailment of the plot moving forward so I have things not going that way and the caravan wanting to continue on the road instead of by ship. At this point my players aren't happy enough and feel like they are being forced along a path they don't know where or why to keep going. Ending the session with discussion about how they feel because I felt I let them down by not doing a better job as GM by that point I share with them the reason I gave them a choice to go caravan route or move onto waterdeep to meet with the contacts there. Because I really thought they would not like the caravan path but that I still offered them a choice because I couldn't be certain what they would want to do.

Discussion ends with everyone happy that a rewrite of the tail end of the session take place so that my attempt at avoiding a derailment of the plot instead is a parallel path of it. And because everything to this point is supposed to be a big secret and a complete mystery to the players and characters they feel like they are being dragged along on a long journey etc but no reason to continue and no reason not to just steal everything and by doing so ending the adventure which they appear to not want to do. So I share some character knowledge with them that there is important stuff they are escorting to individuals of higher importance in the cult but that they don't know of what of the stuff is the stuff of importance to those individuals. They are also given leadership of one of the wagons as part of the rewrite of the ruse to take out a leader to drive the caravan to take a ship instead of continue by ground. So cutting part by short but not skipping ahead past the planned road event part I have them depart from the ship at a strategic point on the road they would have continued to keeping the final destination a mystery as the leaders only know a certain point and find the next stop from there.

So now the two characters to be introduced into the adventure are introduced. I have the murder be of Leosin and let them role-play out an ad-lib because they see more into this than just a simple plot device and they don't appear to be on the expected things in the description of that section. The ad-lib appears to go well and I allow things the players would like to see happen in their discussion of what they think they should do etc. The situation strengthens the idea that they are all a part of the group and things move on to the next chapter Carnath Roadhouse.

I describe the roadhouse to them and that the contents of the caravan are being unloaded and into what door they are taking it all. And about the half ork overseeing this transfer. I describe the area based on the details of the essential ingredients section minus the part meant for GM knowledge. I expect that this info alone would guide them to the anticipated direction to investigate where they stored the stuff. So I ask them what they want to do. They appear to be at a loss at what to do. So one watches things for a bit and I have the storage of the goods finish up and a couple caravan leaders go into the dining area followed by the half orc. That appears to have gotten their attention and they go in under the ruse they are looking for something to eat and drink and overhead them talking about the goods are safe in the room below until they are retrieved. Expecting this second clue would put them on the trail they still are lost and voice their frustrations because they feel like they are at a dead end because of all the mystery. They weren't asking things that lead them to the correct path so I described the stuff about the goods below figuring that would put them on track and didn't so the session ended with big disappointment like it did with the rewrite to go by boat incident.

So rather than having the players play things out at this point because they feel like what was the point of everything that took place in the session from meeting the gnome, to the murder, to the this section plays like a spy story from the words of the author that went horribly wrong. They want me to write up a narrative of what happens from the point they got lost in my attempts to clue them to the next direction. And start them off at the next part. My concern is that at some point in the next chapter is going to be some other similar occurance that has turned a session sour and made me feel like I ruined the fun for everyone with one little mistake in story writing to fill in gaps and adjust parts to fit the players expectations and characters.

I'm going to read chapter 6 and plan out the next session starting tomorrow. But I can't seem to plan for everything they do or ask etc and I end up having to ad-lib and try to keep things on plot. But somehow those ad-libs turn things sour at some point and I don't want the past two big issues to keep cropping up. Because I would rather scrap the adventures and the group move onto something else be that new characters with one of them as GM or something totally custom written for their characters because I don't think any adventure module at this point will work without big rewrites. I don't want to give up on them and scrap it all and someone else GM but at the same time I don't want every other session to be some big disappointment and end with discussion on everyone's feelings about what happened and me apologizing for not doing a good enough job of driving the story. And try and explain things from my perspective of how the story is supposed to play out and how I'm trying to rewrite parts to make more sense with their characters. And that I appear to be failing to make the adventure work for them and also give them freedom to drive the story as well. Because if they have something in mind they would do that is not expressly written as a direction to take or a contingency planned by the author I don't want to force them out of it like the first big issue that halted the one session. And again halted the last session because I gave them too much freedom with clues to what the next plot point was and they didn't pick up the clues, and now felt lost because of too much freedom as compensation for not enough freedom last issue.

I hope even though this is as brief as I can be that some of ya'll read this all and have advice for me on what you think I should do. Thanks.

Sadly, it looks like you have run into what many have with some of the released DnD modules.   They are very linear and and require the characters to make really odd decisions.  At the start of HotDQ why would any level 1 character choose to go fight a dragon for example (and not just run away).  They also take a lot of work on the DMs part to really make flow.

You've already been doing this, but I would recommend just using the modules for ideas and some inspiration and crafting your own points around it.  So in this case, you will know what events are happening in the world and some timing. The characters can then explore the sandbox but occasionally intersect with cultist designs etc or political allegiances.   This gives time for them to get pulled in from wherever they are in their own stories and you have the dragon cult always in the background.

Players will very rarely do what you think, and having to ad lib and be creative is part of the joy of DMing.   I think it gets easier over time as well.  Don't get discouraged!  

I think part of the issue that keeps cropping up is that they want sandbox but also want specific direction without either end of the spectrum feeling like that is what's happening. When I guide too much they feel like I could have narrated 5 weeks of sessions into  half a session and go from there because giving them any freedom in that turns into disappointment that things didn't go exactly how they want/expect. If I give them too much freedom they feel like we don't know what to do and we hit the same disappointment wall where the fun is lost and I end up feeling like I ruined the session. I have ran a lot of campaigns in the past and i have never had a group of players where I couldn't write/modify a story and adlib parts to keep things rolling and everyone always having fun. I am failing in how to balance sandbox and plot direction with this group for some reason so that they are getting enough of both and a session doesn't end bad. I have let them know I'm trying to use the written material to carry the story forward and at the same time modify it for what they/their characters wants and goals are. Because of some of the sandbox play I end up at a block and need to read or reread parts of the written adventure to try and course the session and tell more story. Once this was a real problem because the thoughts and questions of the players has no written source on and I had no idea they would consider those thoughts or questions in preparing for the session. Which only complicated things because I feel like they expect me to provide just enough sandbox, just enough direction, and quickly respond to thoughts and questions they have like I prepared for everything they could possibly think up or ask in character. So I try to adlib answers that make the questions a dead end if it needs to be a dead end (like if they are trying to figure something out and the question is offtrack to the solution).

Most of the D&D adventures assume character motivations that are at least nominally heroic, I would expect quite a bit of customization is required to make any of them work well with an evil party. Many of them also have a particular plot flow and thus aren't very sandboxy (not all; Curse of Strahd has stuff going on that you can mess with, but it doesn't give much in the way of clues about an expected flow or what's level-appropriate).

I wrote a narrative as the group requested to move them to the next chapter: Chapter 6: Castle Naerytar. I have the treasure from the caravan transferred into the castle as they safely tail the lizardfolk and an overseer of some kind barks out at the people moving the treasure inside to "get these inside quickly for sorting. We are expected to have everything sorted in a few days so that your masters can depart with what they need." I really hope this guides them to the idea they are meant to explore the castle and figure out the next part of the plot. Because if this next session is a failure I'm afraid it means the end of the campaign; even though I have great ideas how to work everything to the players letting Tiamat enter the world and taking more power within the ranks of the cult or whatever the direction ends up being as we move along.