A player at our table is really into being an illusionist. He has the spell Silent Image and last outing he created the illusion of a dragon, hoping to scare a bunch of kobolds. Half the kobolds, seeing a dragon, fled. The other half, further away, or maybe feeling some kinship for the winged beast, continued fighting the more immediate threat. I allowed him to achieve this, but don't really want to set a major precedent for this - it's only a first level spell after all.
Game mechanics/guidelines seem pretty thin on the ground regarding illusions - mostly because of the almost infinite ways they can be used during the game.
A couple of thoughts from me : compare the spell to Sleep and have it affect 5D8 HP of creatures. These can have an appropriate condition put on them, such as frightened. Maybe double that 5D8, as unconscious is slightly more inconvenient for the victim than merely frightened or dazed or whatever, but that now seems like a lot. Illusions are partially visually (audible, etc) and partly a mental effect (if you realise they are illusions, either through contact or 'investigation' then they become hazy and see through) so restricting the number of creatures it affects seems reasonable.
Homebrew a morale statistic and use that to adjudicate based on the relevance of the perceived threat. (Note, I am cheap and haven't got the 5th Edition DMG as I have three or four earlier editions, so morale may be in there already.)
Any thoughts or short anecdotes will be appreciated.
Note: the player is not power gaming or attempting to disrupt the game. He is just young with a great imagination, which I am happy to encourage. And a tendency to rage when I don't let the wookie win. :) He is also going to play in another campaign as a sorcerer that is going to be all about the destruction spells; we have to alternate campaigns weekly due to 'stuff.'
Illusions are brilliant but can be a headache for the DM. However, there are a few good solutions.
To use a few examples applicable to your scenario:
Silent Image has a 15x15 ft. size. That is to say, it only works for large creatures. You could make an illusion of a young dragon, but not an adult dragon, and arguably it could not extend its wings without going outside the spell's area.
It's also obviously silent. No intimidating roars, etc.
If a panicked kobold were to shoot one wild attack at it, or even just throw a rock, they would all instantly be able to tell it's an illusion.
And if they figure this out, and some survive, the next kobolds will be suspicious. They may instantly think to use their action to examine it rather than flee, and perhaps have advantage on the check.
Rather than homebrew another stat, I usually just roll vs. Intelligence or Wisdom, depending on the situation. I like the idea of it affecting a certain amount of hit points worth of creatures, like Sleep; I hadn't thought of that before, but seems reasonable. You've got the right idea that frightened is less a penalty than unconscious, so upping the d makes sense. Doubling certainly seems excessive, but another d8 or two feels about right.