Currently, I have 5 spells registered to a single subclass feature on my homebrew subclass. I want all of them to use a single counter for free uses which amounts to the spellcasting modifier. I've used the "USES BY ABILITY MODIFIER" to limit the uses, but unfortunately it creates a use counter for each spell. Any tips on how to do this, new to the homebrew platform of dndbeyond.

First you have to get rid of anything attached to the spells themselves that regulates their number of uses. Then, as part of the same class feature you need to add an Action with Limited Uses set by that Ability modifier.

On the Character sheet each spell will display as “At Will” like a Cantrip, but whenever the Player taps one of those spells, the usual sidebar will slideout, and listed at the top of it will be the name of the class feature that grants those spells. If the player taps that class feature name, the sidebar will change what is displayed to the Description for the class feature. At the bottom of that description, the Action and its limited uses will be accessible to tick of those uses.

That’s how the one or two official features with the same mechanic were entered.

Found out, how to limit the use of an action, but still can't figure out how to attach spells to such action?

Quote from shirofuji >>

Found out, how to limit the use of an action, but still can't figure out how to attach spells to such action?

 You don’t have to. You attach the spells to the class feature, and you attach the action to the same class feature, and the character sheet does the rest.