I know the spell says the new form is a beast but does that mean it have to be from the beast list?  Wondering because I have an encounter that gets easier the more you polymorph the broodguards into something with higher intelligence.

Yes, it has to be considered a "Beast"-type creature, which has very few creatures with high INT. If your DM is generous, there are Awakened creatures (such as an Awakened Rat or Awakened Elk) that are already statted with human-level INT, but I'd imagine some DMs just won't consider those viable options, since Awakened Creatures are generally consider unnatural in some way and mostly just show up in a few Adventures for story reasons.

If Awakened beast have higher then 8 INT, then I will allow it.  Going up against a Yuan-ti Anathema and some Yuan-ti Broodguard isn't easy unless you are a party of 4 at lvl 14.

Yeah, all of the Awakened Beasts have an INT score of 10

Is there a specific book that the Awakened beast are found in?  Would like to know so I might get the encounter flushed out with different options based on party's actions.

The book with the most Awakened Beasts is Waterdeep: Dungeon of the Mad Mage, which includes a Bear, Elk, and Giant Wasp, which all have basically the same stats as the normal beast just with an INT of 10 and they all speak one language.

You can also look at the Druid spell Awaken if you want to use another beast not listed in a book.