Setting:

A leviathan is a dragon like creature only 3-7 times bigger and about as bright as a horse.  The empire has "domesticated" these and strapped ship hulls to the backs of them.  These ships are armed like tall ships with arrays of cannons giving the empire flying weapon platforms with ranges in the miles.  This will be flying to the area with an Ancient (or older) dragon.

What I want:

Something engaging for the players in combat with that dragon when they come into the area.  I want it to feel high stakes, high risk and worthy of a massive aerial battle but also with the understanding that the party can't fly themselves and it's impractical to dog fight with a tallship.  I also want dice rolling.  Having a set "damage per round from the cannons" doesn't feel like it keeps the spirit of "the dice tell the story" but I'm open to tweeks that keep some level of random outcomes.

What I'm thinking:

The ships cannons will be the primary weapon against the dragon.  The ship has 9 guns a side and enough crew to load and fire one broadside at a time.  Each cannon does 4d10 damage on a hit and cannoners will get a +3 to attack against the dragon's AC of 22.  Most shots will miss but some will creep through for damage.  Doing the math, 4d10 works out to an average of 22 damage.  +3 to hit gives us hits on 19 and 20 or 10% of the time, meaning that on an "average round" the dragon will take 19 damage.  

The dragon has over 550 hp but will leave once it's taking 250 points of damage because while it's interested in a good fight, it doesn't want to risk death over this, yet.  

While this is going on, hordes of Kobolds will land on the ship (beefed up to be a fair fight for the party) and spread out.  A cannon crew that is "engaged" with kobolds won't get their shot to fire.  If a kobold is engaged with a PC or Marine then they won't be engaging the gun crew.  Any gun crew engaged for 2 back to back rounds is incapacitated and out for the fight; they're sailors not marines.  

The dragon will directly be attacking the ship and leviathan and the party is assumed to duck behind the gunwales when the dragon makes a pass with a breath weapon.  After 3-4 rounds the captain will insist they pull back because the leviathan is too wounded and staying longer is a risk to everyone.  This is the loss condition.

Possible actions for the party:

1)  Directly engage the kobolds to keep them out of the fight and manage the waves of them dropping onto the ship.

2)  Find ways to give the gun crews bonuses either through clever actions or other means I haven't considered.  

3)  Possibly throw things directly at the Dragon.  The party has an artificer with a +9 ranged attack.  In theory she could hit and do as much damage as the a cannon all by her self.  The ranger, oddly enough is only a +7 and has a lower damage output.  But given the proximity to the crew, I see the party focus as there until the decks are clear (hopefully 3 rounds).

My goal was 3-4 rounds of combat so that I could play the "we need a win!" shout going into round 4. Looking at the math, though it's unlikely to get to 250 damage in that time.

If we up the bonus for the gun crew to +5, that puts a hit on 1/5, 20%, and ups the damage per round to about 40 from the cannons on average with some real rooms to swing.  After 4 rounds we're still averaging only 160 damage which is 100 short of our goal.    I could also up the damage to "fire ball" of 8d6 which ups the average per shot to 28 and with a +5 to hit puts us at 50 damage per round average and makes the 250 more do-able in 4 rounds.  It also makes the cannons pretty dang bad to get hit with and I'm a little worried about the long term implications of that.  Oddly I'm not worried about CR3 kobolds running around.

So... thoughts?

So your 250 value is already just an arbitrary number. You could just play it by ear and have the dragon retreat when narratively convenient rather than at a certain hp threshold. That's probably how I'd run it. Of course if things went particularly well or poorly, the narrative might change.

It does feel a bit contrived though, because as you say the leviathan can't really dogfight. What's to stop the dragon from just landing right on its back, forcing it to the ground and crushing the party? Or to only attack from below where the guns don't reach? It just seems like a very vulnerable position to fight from.

Perhaps this ship has already sailed so to speak, but i think it would be interesting if the dragon saw the leviathan as a victim/prisoner rather than a rival and was trying to help it rather than kill it. Either by trying to disconnect/wreck the ship on its back or by attacking the party directly.

That's a great mix really!

The only thing I can think of suggesting is to use a Skill Challenge (non 5e concept that's great, more here). This could also connect directly with giving the gun crews bonuses.

It would also work great for "The tallship is taking lots of damage, can we fix some of that to help?" situations and ideas from the characters. Most importantly, the success/failure is still guided by the dice but can also set the narrative for one side to retreat depending on success and failure situations.

So related to the dogfight, I have been presenting the Leviathans as much larger than dragons but not really as smart.  I wanted to capture the vibe of the old tall ships doing their slow turns, big broad sides, kind of thing.  I was aiming for something like Horatio Hornblower and and Novick's "His Majesty's Dragon".  But I don't want to confine the actual Dragons of the world to being that kind of fliers so I was trying to find a way to allow the difference between the two to shine. 

As for the HP numbers, I use a graphical tracker so they can get a sense of "how beat up" something is.  It also tags enemies as "hurt" and "bloodied" so the party can see when a cannon volley that does a lot of damage actually shows a lot of damage on the enemy.  I'm trying to keep a sense of winning as they battle.

I like the idea of skill challenges too but I'm not sure what that looks like while still giving the players agency and actions during the fight itself.  I really want everyone to feel like they are a) useful and b) have options.  The idea of "lead the gun crews" feels like it's a little bit too focused on one style of game play while I also don't want to design a whole game around the combat itself.  Maybe turn "leading" into a "bonus action" and every successful skill check as part of that bonus action becomes an acculumulation of bonsuses to the attack roll for the broadside?

I also know I can do something EVEEL and have the dragon change sides on them.  This means the crew would have to cross the ship to the other battery and that's harder to do when there are, you know kobolds looking to shiv them.  It can take on a whole new kind of dynamicy with the players either taking over a gun, or working to ensure that more crews can get to the other side to set up the next volley.

So it would probably be a skill check of DC15 bonus action to "lead" while fighting.  Let the players call out the style of leading they're using and no repeats per round?

I would consider having the Dragon try to free the Leviathan from the ship. The crew may be trying to repel the Dragon, steer away, and stay aboard.

The dragon may start by looking at the ship. Then it may retaliate if anyone attacks it. Then it starts swooping in and slashing the straps on the leviathan.

The encounter then can go one of two ways - the Leviathan is freed, and the ship is dropped from the sky into the land below, culminating in a cool survival back-to-civilisation sort of adventure, or the Dragon is driven away, and soars onward to the destination as planned.

Expect your players to do more than just try to keep the guns free of kobolds! I don't know what level they are, but you have to consider that either the dragon needs a good reason to not torch the deck of the ship and kill your players, or your players are powerful enough to take it and will immediately step of the wind/fly/teleport and get into full-blown combat with the Dragon! They might try to steer the Leviathan, or pursuade it to fight using Speak with Animals. If they are low level, they will be looking for a way to kill, trap, or otherwise beat the Dragon. They will want to fie the cannons, or they will look to have it swallow a cannon and then fire it - the usual D&D chaos!