As wizards, they have access to more spells than their preparation slots allow, so when a player comes up against a wizard who most definitely has their spellbook on them, how do you deal with it? If they're specialists, like the Volo's casters, do you tailor it?

Just give them whatever you think will make the encounter interesting.

Hell, give them stuff after they talk to the PCs. Doesn't really matter. Even if they capture the spellbook on the bad guy, it doesn't HAVE to have all his spells in it.

I'd focus more heavily on what the wizard could prepare, following the same rules on spell preparation as for any wizard and wouldn't include much more than that except more niche/utility spells as opposed to anything that would give players strong advantages if they happened to get the spellbook - particularly if the party also has a wizard who could copy the spells. Saves having to pull DM-bullshit stuff like "oh the spellbook got damaged" or something. Or if the party has a wizard I would include spells they already have.

I would generally only have the wizard's spellbook have more spells of real worth if I treat those spells as rewards. Even if they do not have wizards in their party a wizard spellbook can be sold off - a spellbook with rare or powerful spells would be worth a lot of money to the right buyer. I would consider throwing in more lower level spells in there in case a member of the party wants to multiclass into wizard using this spellbook as the RP mechanism for doing so - I would say their initial spell selections would come only from the spells in the book they found, for instance.

If the wizard is intended to be more than battle-fodder then the spells they have, prepared or not, will be based on what I want this NPC to be able to achieve. If they're high level they might have the Clone spell and this would be in the book but they wouldn't have it prepared and it's not something I'd worry over my party having access to. Similar for things like Glyph of Warding if the wizard set up a lot of traps using this - not something they'd keep prepared everyday but it is something I would want them to be able to do and it's not a spell I would worry about the party having access to given the casting time and cost.

It kind of depends. Naturally you'll have the spells in there that you want to use for the encounter (even things such as mage armor or magic missile and such) and some others. If a wizard player has mentioned wanting a certain spell, I might put it in there for good measure, just like a "treasure" reward. Once I put a scroll in the book which the NPC was studying to put in his spell book as well. I don't want to overwhelm a wizard player with new spells, so I don't give them out too often. Another way to do this is saying that the book has been severely damaged (especially if said spellbook was in the area of say a fireball or something)

I remember that spellbooks were discussed in the Complete Arcane 3.5 book. The average wizard has two types of spellbooks, big fancy grimoires that is kept somewhere safe, and arcanabulae, which are field books with a truncated set of spells. For starters, a wizard encountered randomly probably won't have their grimoire anywhere near them, and their arcanabulae are likely to be scribbled, paraphrased, locked, trapped, magically warded, and/or encoded. For the latter, an encrypted spellbook is going to be nearly impossible to decipher without a key, which could be anything from a mathematical equation to a passage in a random book. It's also possible that a spellbook is written in runes and glyphs, or languages that your player can't read. You could easily modulate the flow of new spells by selectively dropping hints to a player with a foreign spellbook.