There are no wizards in the party, so my players were looking to sell a spellbook they looted off the enemy. How much money are spellbooks worth? Is there a formula based of the number and level of the spells contained in it?

I would use the same base as selling spell scrolls (Described on Chapter 2: Downtime Revisited - Selling Magic Items on Xanathar's).

Considering that spell scrolls of cantrips/lv1 are common, lv 2/3 are uncommon, lv 4/5 are rare, lv 6/7/8 are very rare and lv 9 legendary magic items, I would use the cost table on that section as a reference (Consider that Spell scrolls prices are halved based on the table - even though Spellbooks are not consumables, I would still halve the price, since the spellbook doesn't deliver the same benefit as a spell scroll).

The book also lay down the cost/time required searching for a buyer, the possible resolution and possible complications that can come off trying to sell something that might be perceived as rare.

One of my players is an order of scribes wizard who has run a scam where he sells his spell book, then once he has travelled far enough away from the person he sold it to he then recovers it back. The person who bought the spell book is then left with an empty book. 

He has pulled this off twice with no comeback, he has no idea that at some point I will be making sure the target has ways of tracking him down :). 

Oh that is just a baaaad plan. I would not like to be that player when that wizard who is now 8 levels higher comes back and drops some hurt on them.  LOL or have all of the scammed wizards form a pact to hunt him down...

Second the notion that selling a spell book would be less valuable than a collection of scrolls of the same spells. Half of the price of said scroll collection would maybe be on the high end. Possible variation might be that they find a collector of Arcane tomes or perhaps an apprentice of the recently deceased looking for their Master's work.

Since spell books normally take time and effort to decipher the spells inside of them, I'd say a relatively good starting point would be to take the costs of those spells in the form of scrolls and then take off somewhere between 30-50 percent depending on who they are selling to.

Someone who's going to have to find and pay someone to decipher the spells would probably pay the lesser amount, whereas a skilled wizard who's used to collecting and decoding spell books might give a little more money... unless they want to rip you off. 

For each level of the spell, the process takes 2 hours and costs 50 gp.

An easy formula is to follow the cost of the gold that the wizard spent copying the spells into the spellbook, plus 25gp for the basic book itself (25gp).

So a spellbook containing 1 x 3rd level, 3 x 2nd level and 5 x 1st level spells has a value of: (1x 3 x 50) + (3 x 2 x 50) + (5 x 1 x 50) +25 = 725gp. This might sound high, but it cost that much to produce in the first place so it's fair. It also prevents your wizard PC from wandering into a town and finding a whole bunch of used spellbooks for sale.

Selling it to a merchant, they'll want to turn a tidy profit when they resell it, so they should pay 50% of the value of the book, making this example spellbook sellable for 362gp.

If you can find a wizard who wants all of the spells in the book, they might be prepared to pay more - but they are also individuals and there should be suitable haggling. They won't want to pay for the spells that they already know, for instance.

So another alternative to selling for cash, is trading for magical items or resources.  Xanathar's has pricing for the various levels of magic items, and using the formula that Sanvael points out gives you a rough equivilence.  As a DM you can be as controlling as needed on the trade on what is availible.  So a low level book is work a 4 healing potions, a higher one a sword.  

And it could lead into hooks for "Sure I can make that...but I need one small item...:

I have been pondering the sale price of spellbooks for a while. There is no guide in any Official TSR / WoTAC D&D book.

Apart from the 50gp to by a spellbook, it costs a wizard nothing to write the spells learnt from levelling up into the book. So by L20 there will be 44 spells in the book that cost nothing to write.

So the purchase price of a spellbook would depend on the spells in the book that a would be buyer may want.

Quote from Sanvael >>

For each level of the spell, the process takes 2 hours and costs 50 gp.

 

An easy formula is to follow the cost of the gold that the wizard spent copying the spells into the spellbook, plus 25gp for the basic book itself (25gp).

So a spellbook containing 1 x 3rd level, 3 x 2nd level and 5 x 1st level spells has a value of: (1x 3 x 50) + (3 x 2 x 50) + (5 x 1 x 50) +25 = 725gp. This might sound high, but it cost that much to produce in the first place so it's fair. It also prevents your wizard PC from wandering into a town and finding a whole bunch of used spellbooks for sale.

Selling it to a merchant, they'll want to turn a tidy profit when they resell it, so they should pay 50% of the value of the book, making this example spellbook sellable for 362gp.

If you can find a wizard who wants all of the spells in the book, they might be prepared to pay more - but they are also individuals and there should be suitable haggling. They won't want to pay for the spells that they already know, for instance.

I think this is likely the most reasonable and easy method. It's what I would do for the sake of coming up with a price someone might pay, and not dump too much gold in the players' laps for something that probably wasn't planned to be really valuable loot in the first place when you made the encounter.

VinAlanson is also correct that the book might have cost the original owner almost nothing.  And it's highly dependant on the market. 

But overall that calculation method is pretty fair to get a rough idea. 

Of course, the real answer is - how much money do you want them to make off if it?  If they're level 3 and didn't get much else in the way of treasure, about 300- 500 gold could be a great price.  If they're level 10 and need cash badly for your plot, maybe there is a buyer willing to pay thousands. Realism and fairness are important, but you don't want to ruin your game by giving a 1st level party 100,000 gold haha.

I'd just use the magic item selling rules rather than trying to compute a unique value for the spell book. A price based on the spells individually is likely to be game breakingly high and or a pain to calculate. The magic item rules from xanathar's let you do it based on rarity and you can easily say the rarity is that of the the scroll of the highest level spell in it. Spell books can  basically just be money drops from killing wizards, even if you have a wizard to copy the spells they can sell the book afterwards. 

The only thing you can actually do with a spellbook is copy spells out of it, which means (a) there aren't a lot of people who might even be interested, and (b) even potentially interested parties aren't going to pay for spells they already know, can easily get from another source, or can't cast. As such, spellbooks, while not entirely valueless, are not going to be terribly valuable.

The value of a spellbook, based on normal rules, is very little. The only person it is of any practical use to is a wizard. And the only use they have for it is to copy from it into their own book.

If a spellbook costs the sum of all the spells via the copy rules 50gxlevel, or even worse, the cost of all the same level scrolls for each spell... then a wizard would have to spend outrageous sums of money to buy one of these things just for a single spell or two they wanna copy that afternoon. After which the spellbook they just paid several thousand gold for is now useless to them.

So, we have problems with the demand side of the normal supply/demand equation. They're unlikely to pay the price if they only need a smaller portion of spells from the book. And, they certainly don't need the book after they copied from it. So, they're going to immediately wish to resell it.

The spellbook economy really is a weird one for those reasons. Your best bet will be to try to find a wizard organization to sell it to, so the wizards can each copy the spells, making its value multiplied by each wizard to learn from it.... and yet, a wizard organization is the most likely to already have a copy of all those spells already so won't need it even at all.

There is no easy way to determine the value of a spellbook, because of all that. The value of the book will be as unique as the wizard(s) you're trying to sell it to.

But keep in mind all you're actually selling is access to the spells in it. It has no practical use outside of that, and the wizards will also still have to pay the normal costs for copying out of it.