The DMG says this with regards to calculating a homebrew monster's challenge rating: 

Read down the Hit Points column of the Monster Statistics by Challenge Rating table until you find your monster’s hit points. Then look across and note the challenge rating suggested for a monster with those hit points.

Does this mean the average hit points of the monster, or the maximum? If your homebrew monster has 10d8 hit points, do you count its hp as 45 (its average hp using the RAW) or 80?

It would make sense if it was referring to the average. Later in the chapter in "Step 8: Hit Points" it says "you can assign a number of Hit Dice to a monster, then calculate its average hit points. Don’t worry if the hit points aren’t matching up with the expected challenge rating for the monster."

 This suggests that it is balanced based on the average number of hit points. Also, rolling 80 with 10d8 is an outlier. Statistically it will only be rolled less 1% of the time, why balance something on a probability that small?

There are some pretty good online CR calculators. Run a test monster through several of them and go with the consensus result.

Quote from Astromancer >>

Also, rolling 80 with 10d8 is an outlier. Statistically it will only be rolled less 1% of the time, why balance something on a probability that small?

You're assuming I roll my monster hit points instead of just deciding/assigning them.  ;)

I'm still at a loss.....

For example: A CR 13 Devourer has a hit dice of 17d10, even though it's a CR 13 creature. A Jabberwock is a 12d10, even though it's a CR 13 creature. How do you determine the number of hit dice for this?

Thank you

Quote from RonII >>

I'm still at a loss.....

For example: A CR 13 Devourer has a hit dice of 17d10, even though it's a CR 13 creature. A Jabberwock is a 12d10, even though it's a CR 13 creature. How do you determine the number of hit dice for this?

Thank you

 
It isn’t done by Hit Dice, but by Average Hit Points & AC compared to DPR & Attack Bonus/Save DC. The Hit Die size is determined by creature size, the number of them is however many are necessary to get the right number of average hit points for the CR at that AC. The devourer has lower AC than the Jabberwock, so it needs more HP

Defensive Challenge Rating. Read down the Hit Points column of the Monster Statistics by Challenge Rating table until you find your monster’s hit points. Then look across and note the challenge rating suggested for a monster with those hit points.

Now look at the Armor Class suggested for a monster of that challenge rating. If your monster’s AC is at least two points higher or lower than that number, adjust the challenge rating suggested by its hit points up or down by 1 for every 2 points of difference.

Offensive Challenge Rating. Read down the Damage/Round column of the Monster Statistics by Challenge Rating table until you find your monster’s damage output per round. Then look across and note the challenge rating suggested for a monster that deals that much damage.

Now look at the attack bonus suggested for a monster of that challenge rating. If your monster’s attack bonus is at least two points higher or lower than that number, adjust the challenge rating suggested by its damage output up or down by 1 for every 2 points of difference.

If the monster relies more on effects with saving throws than on attacks, use the monster’s save DC instead of its attack bonus.

If your monster uses different attack bonuses or save DCs, use the ones that will come up the most often.

Average Challenge Rating. The monster’s final challenge rating is the average of its defensive and offensive challenge ratings. Round the average up or down to the nearest challenge rating to determine your monster’s final challenge rating. For example, if the creature’s defensive challenge rating is 2 and its offensive rating is 3, its final rating is 3.

With the final challenge rating, you can determine the monster’s proficiency bonus using the Monster Statistics by Challenge Rating table. Use the Experience Points by Challenge Rating table to determine how much XP the monster is worth. A monster of challenge rating 0 is worth 0 XP if it poses no threat. Otherwise, it is worth 10 XP.

Creating a monster isn’t just a number-crunching exercise. The guidelines in this chapter can help you create monsters, but the only way to know whether a monster is fun is to playtest it. After seeing your monster in action, you might want to adjust the challenge rating up or down based on your experiences.

Follow this link to see the chart: (https://www.dndbeyond.com/sources/dmg/dungeon-masters-workshop#MonsterStatisticsbyChallengeRating).

The Devourer has a “Defensive CR” of 8, and an “Offensive CR” around 18ish for an average of CR 13.

The Jabberwock has a Defensive CR around 6ish, an Offensive around 13ish, but that gets ratcheted way up because of the Confusing Burble so it still averages out to around 13ish.

If you go straight across the line at CR 13, your monster would have +8 to its attacks, do between 80-85 damage per round, have an AC of 18, and anywhere from 251-265 HP. If the creature was Tiny (1d4 Hit Die) with 10 Avon it would need about 103ish Hit Dice. If it was Huge (d12 Hit die) with Con 14,  it would only need around 30 of them.