Mistakes have been made.

What do you do when you let a player do something that turns out to be game breaking down the line? I'm running a kind of homebrew. I haven't made anything outside of the 5.5E rules but I'm not running a model either. So call it DMs first homebrew. lol 

There are lots of different way a situation can be “game breaking down the line” and each is going to have a different solution with different degrees of subtlety.  It is impossible to give a real solution without knowing all of the intricacies of your particular problem.

That said, 99% of “whoops. We broke something” problems can be fixed with a quick conversation and rebalance.    “Hey, I am new to this. I screwed up and it has created an unbalanced situation that is hard for me to DM for and is not fair to other players” can go a really long way.  Any player worth playing with is going to understands and be willing to work with the DM to fix such a problem. Not a subtle solution or an elegant one, but it gets the job done.

It kind of depends on whether the game-breaking thing is a story element, an ability given to the players, or over-powered items.  Sometimes, you can undo these things purely through storytelling, but it's often easier to just "come clean" to the group and get their input on something else that could replace it, so it feels less like you're "taking something away from them."

It would help to know some more details though.