A Homebrewers’ “How-To” FAQ*

Just to save y'all the trouble, you cannot have the homebrew subspecies be nested under the homebrew species the way official stuff are.

You can make a subspecies of an official race and do it but not for homebrew species. That has never been a thing and is unlikely to become a thing due to D&D 5.5e moving away from subspecies.

Your best alternative is using the Options system to set the subspecies as Options instead of being set as subspecies - this is the way new 5.5e does them now, such as 5.5e version of Elf or Tiefling.

but for 17 what if the spells are already in an option like for tieflings you can choose your spellcasting abillity but you get different spells based on your legacy???

Quote from RoNoThwomp >>

but for 17 what if the spells are already in an option like for tieflings you can choose your spellcasting abillity but you get different spells based on your legacy???

 

The way they've done this on the official species that work like this is to have one set of three options that set the spellcasting ability, and then a separate set of options for the legacies where there's actually one for each combination of legacy and spellcasting ability (but only the ones pertinent to the chosen ability are shown).

For example, Tieflings have three legacy options: Abyssal, Chthonic, Infernal. The first set of options has the three spellcasting abilities: Intelligence, Wisdom, Charisma. The second set has nine options: Abyssal/Intelligence, Abyssal/Wisdom, Abyssal/Charisma, Chthonic/Intelligence, Chthonic/Wisdom, etc.

If you want to see the details of how to do this, create a new homebrew species using Tiefling as the template and take a look at how they've implemented it.