Ranger: Swarm keeper Prone question

This may have been asked and battled out in the past posts but playing a Swarm keeper Ranger and reading the level 11 perks for mighty swarm reads as:

"If a creature fails its saving throw against being moved by Gathered Swarm, you can also cause the swarm to knock the creature prone."

Question becomes does this get affected by prone sizing rules. Gathered swarm is read as a magical attack since it uses your spell DC to do a prone/move check against an opponent but I'm not sure if there's still a disadvantage trying to knock Huge/gargantuan things down with might swarm or not. I used it in a fight against a Huge dragon once to help the party but I'm not confident if there's a size limitation described in the core rules I'm missing for spells verse physical checks for causing prone status.

Thanks for the help!