Hello, I am having trouble deciding what my bard's college should be.

To give some context my bard's backstory is that he used to be a apart of a mercenary group called the Red Ravens when he was a young adult (20). They taught him that he could use his knack for trickery and magic for something else. Basically they taught him how to use these tricks for helping good people and messing with and hurting bad people. Anyway, I'll sum up the rest by saying there was a mutiny one night and it resulted in a gruesome battle where, miraculously he survived. So now he tries to continue doing what they taught him to do with his talents while also trying to keep a chipper attitude despite the grief (there's other but its a short summary).

My problem is I can't tell if he should be college of lore, due to his knack for magic and trickery.

Or should he be a valor bard due to his time with mercenaries.

All I do know is that I want him to be crowd control and debuff/buff focused (a bit stereotypical for a bard I know but hey why not).

I'd say go lore if you want more magic.  If you want to stab stuff go valor or swords bard.

Lore get cutting words which is a great debuff. At level 6 you can choose any 2 spells in the game to plug in any character need that you don't already have.

If you want to tie this into your backstory you can just do it with your background. You can multiclass into rogue or fighter at level 1. Then go full bard.

Rogue will give some nice skills while fighter gives con saves, armor and weapons.

Try not to optimize too much and make the character work for you. Much more fun that way.

I am playing a mark of shadows sword bard and for flavour I mostly take illusion spells and spells that say "hey look at me". It works for the character but definitely not the best spell choices. 

Thank you very much for the advice, I think I'll go with lore then. It does help me get more spells good for the playstyle I want to go for, so yeah. Thanks again.

well since he spent such little time in the mercinary company and so much time perfecting his trickery, it seems like it makes the most sense to go for the lore bard since his basic cobat training can already be represented by his proficiencies in rapiers, simple weapons and light armor he gets from the bard class, if his knack for skills is what he used to bring to the team his skill usage is most likely what he would continue to focus on

It is more about how you want to play the character whether you go with one college or another.

I was struggling with the same choice weeks ago on my first 5e Bard.

Considering just PHB choices, Lore and Valor, here is the way I see it for me.  If the party is small you may lean toward Valor as you will be in combat more often engaged in melee fighting.  If the party is larger you will be more free to enter and exit melee combat when you wish.  Valor bards get medium armor, shields, more melee attack options and can cover some weaknesses with feats.  Lore bards get Cutting Words, Magical Secrets at level 6, and bonus proficiencies.  This is the strongest spell casting build for a bard allowing the most options to help the party, if you want to make the build for that.  If you want to build your own melee bard, then go with Valor (or swords).

The back-story is secondary to what you will enjoy doing in-game.  Your back-story supports either path.  Good luck and enjoy the game.