Mistakes have been made.

What do you do when you let a player do something that turns out to be game breaking down the line? I'm running a kind of homebrew. I haven't made anything outside of the 5.5E rules but I'm not running a model either. So call it DMs first homebrew. lol 

There are lots of different way a situation can be “game breaking down the line” and each is going to have a different solution with different degrees of subtlety.  It is impossible to give a real solution without knowing all of the intricacies of your particular problem.

That said, 99% of “whoops. We broke something” problems can be fixed with a quick conversation and rebalance.    “Hey, I am new to this. I screwed up and it has created an unbalanced situation that is hard for me to DM for and is not fair to other players” can go a really long way.  Any player worth playing with is going to understands and be willing to work with the DM to fix such a problem. Not a subtle solution or an elegant one, but it gets the job done.

It kind of depends on whether the game-breaking thing is a story element, an ability given to the players, or over-powered items.  Sometimes, you can undo these things purely through storytelling, but it's often easier to just "come clean" to the group and get their input on something else that could replace it, so it feels less like you're "taking something away from them."

It would help to know some more details though.  

What exactly did you do? 

Whatever it is, it might not be as bad as you think it is.

It would be easier to give advice if we knew exactly what you're talking about.

That said, it is easy and totally understandable to tell your players that you messed up and how you're going to fix it. Most of the time they'll let it slide. Everyone makes mistakes.

We've all been there. I have had to rebalance a character on the fly for my campaign because the player made them a were-bear barbarian. I thought it sounded cool, but soon discovered that they were severely overpowered. 

Talk to the player. Explain the problem, ask them what they would be happy to compromise on to get the flavour of their request without the overpowered nature. If they insist that they need the power, make it a feat - that way, they have to spend their level-up gaining said power.