I'm a new DM, and have only been playing for 3 years now. I'm trying to find a way to make a campaign that has a basis on a plot but has lots of room for improv.

I've run into a couple problems DM-ing for friends. My players including my best friend andh boyfriend, who know how my mind works. I'm hopeful that some of you can help me in figuring out a campaign that will be fun and interesting without them being able to guess where it's going.

I tried a "you've just woken up and forgot everything but your skills and your name" type campaign, which failed immediately. My players were smart and asked questions I couldn't answer without giving everything away, and then almost killed each other. Then I tried "you two are in a bar and a guy begs you to help him." It worked a bit better, but still fell apart after on session. 

Thanks for and help/advice

Why not some kind of prophecy? It's probably pretty generic sounding, but it works, because you have an end goal, the party doesn't need to know all information except the end goal, or maybe not even the end goal if there's a plot twist. That way they can all be normal adventurers that wouldn't have an immediate reason to kill each other. Maybe it's just fate that brings them together, magic forces, or a prophecy keeper? Improvising can work to, like if someone makes a decision, something can still lead to another, resulting in the prophecy happening, and the prophecy could be vague, making it so they can't destroy it so easily, such as "The heroes will go and stop the big bad evil guy".

In my opinion, it seems to just work out and doesn't necessarily have to be a trope or lazy writing. 

You could start with all PCs just being adventurers who got hired for the same job of guarding a caravan and go from there. Have them introduce their characters and talk about their backstories during breaks/rests. Do a couple of bandit and wild animal encounters. Give the characters a chance to bond and become a party.

Then sprinkle plot hooks for along the way. Maybe some rumors about things that happen near their destination. Maybe the caravan is delivering relief aid after a mysterious disaster. Maybe there is a dungeon or ruin near the destination town.

And just kind of evolve organically from there all the while you can have a big bad moving in the background that they most only hear about through rumors or notice indirectly.

You could also go for the unexpected by borrowing elements not commonly found in fantasy from other genres. The big twist in my campaign involves time travel, or you could treat an illithid encounter like an alien invasion, or you could mimic the lead up to WW1 but in a fantasy world. Teach them to expect anything, and therefore nothing. 

Have them write their character's into. If one has the soldier background, and the other is an entertainer, challenge them to figure out how/why they are together at the beginning. What does the soldier want to do now that they are no longer a soldier? Did they hear about a great treasure/evil monster/etc and wanted to do something about it? Were they at a bar, deciding how to track the treasure, and struck up a conversation with the entertainer that was also looking for some extra excitement in their lives? If they have a good story, or even an ok one, chances are you can pull a good plot hook from it to start with. Within that arc, you can introduce larger plot points as you go. 

Have you had a Session 0? That's where you and your players talk about both your expectations for the campaign, create characters together, and come up for a reason for the party to want to work together. You might say something "The campaign will start with a humble request for aid from a local blacksmith, make a character that would be down to help, whether it's because they actually want to help, they are greedy for reward, or they're seeking in influence"

Session zero is super important if you want to have a good story in your campaign because you'll have player buy-in built into session one.

Have you tried to make them think it was going where they think it is but change the story around and make it go in a whole new direction

I use "would your character know/ask/think that?" a lot when dealing with one or two of mine who tend to be a bit too metagamey with their sense of things. Sometimes it's not intentional for them, so I just give them a gentle reminder. It's also OK for an NPC to say they do not know something or become angry at the party's stupid questions. Use that sense of reality, that most people in this world aren't their allies or personal information deliverers, to make the party work to uncover truths. Don't tell them anything that's complete. Give it to them rashomon style, through an NPC's biased coerced perspective.

Also, remember that the game is as much them playing as you DM'ing. If they really are in a "you wake up and cannot remember" scenario, they are as obligated to RP that appropriately as you are to assist them along that narrative. If they approach the game as some kind of competition to force you to reveal more than they could know, then they're kinda being unfair, if not outright disruptive.

So... assuming you get a handle on that, I'd start doing some more premade material with your own strategic edits (just in case). Prewritten modules will help with any predictability issues to do with your style. It's my preferred way to play right now - keeping my games dynamic, modular and adaptive. One session it's the module on smoking out some hobgoblins who seem to be looking for something, the next it's uncovering a dark fallen tower in a dank swamp occupied by a hag where they find a strange object, the next it's a hill giant's forces threatening a nearby town. Then tie them all together with something like - the hobgoblins were working for the hill giant looking for the swamp artifact. Finding out some humans had it, the town got attacked on account of the party. Queue remorse, an offer to defend the town, a large combat encounter, and then a mini-boss battle. 

 Idk if im allowed to post link but I'm running my campaign that is evolving based on what the characters do so its like more satisfying for them like im doing improv as i go and like just have all my lists an encounters ready incase they ask something specific. Like what the town or where everwould be doing if they wernt there and still progressing the timeline. One thing i always say to the players when they as me a question is 'roll play it' or just reply with 'make a check'. And they get negative inspo if they meta game too. Seems to be working well so far. I have a discord that i have some DMs on so you can get a better back and forth and we can help you bounce ideas quickly (dnd beyond please delete if not allowed dont block me from the site its my life lol) https://discord.gg/UDGsaFU

As others have suggested, a Session 0 is very helpful. This might take a bit more work on your end but allow the players to make up their characters and come up with backgrounds at that session. Don't worry about fleshing everything out, just have them make the characters they want to play with the character ideas they want to have. Then give yourself a couple of weeks. Take all those ideas and try to find a common one, thread or theme that ties them together. Write up a little story that would bring them all together. Then start playing sessions introducing them to each other. Have some of them know each other (friends, family, guild member, business associate etc) to make it a little easier on yourself. You might even want to have separate starting sessions for each group so that their introductions are more organic and the group comes together as one when it makes sense. 

Grab a published adventure. You don't have to play it. But if you are afraid they know you too well, steal ideas from it that you might not have come up with yourself. At least enough to get you off and running. 

I have a campaign framework you can have or adapt if you want? Let me know and Ill PM it to you :)

Okay, so one of the things that always helps me is to have a clear goal or direction for the party to be working towards, especially at the beginning. I think you can ease up later, but at the beginning, it might be good to hold a tighter rein on where the party is going. I would do this through circumstance, you control the world, so make it work towards bringing the party together at the beginning. 

A session 0 might help, and you could integrate a starting point into the campaign that the party has to adhere to (you're all soldiers just released from the army on unpaid leave, and you have to find a way to make money). 

I would introduce an element of danger right away, so the party has to respond. As an example:

one of the campaigns I ran had the characters just show up in some village at the annual village fest, and during the feast, they were all appointed tables, and the characters met for the first time at this dinner table, by pure chance or happenstance ( or is it.... destiny??!! muhahaha).  The feast was then attacked by a roving band of goblins, and the party -- being the only competent fighters around -- are forced to work together to fight off the threat. The goblins are defeated, but it turns out that this wasn't the only attack, and another raiding party of goblins kidnapped a bunch of civillians. BAM! first action setpiece and first quest established. 

of course this beginning is quite cliché, but I would argue that it is more valuable to tie the party together strongly through circumstance and need. 

As for outsmarting your friends, I would advise not to set everything in stone yet. you can prepare story arcs and dungeons and encounters, but they are not real until you set them in front of the players. I normally just have a few possible encounters, NPC's, locations and events just floating in flux, until I find the right moment where I think they would fit, at which point I pluck them out of the air, and introduce them to the party. Oh, one disclaimer, if you do choose to take what I like to call the "Prepared improv" route, I would suggest you Take notes of everything that happened in a session after it is done, while it is still fresh in your memory. You don't know what you might have improvved that you might have forgotten the week after. 

Good luck, I hope you succeed