So I've been running a game for about seven months now, and it's my first time DMing. I had six players, but a new one just joined this past session. We have been playing for at least twenty sessions and most of my players have enjoyed it. Most. This player, who we will moniker "Sam" for reference sake, came in at the second (or maybe third) session and is a half-elf half-aarakocra druid. His race is custom and his subclass is also homebrewed. He, at first, would help give me ideas and criticism, both of which were helpful and I used them. But now, seven months and many sessions in, "Sam" has gotten way out of hand. He is trying to make his own items without me approving them and then he gets argumentative when I say "no" because the item is too powerful or just he doesn't need it. He already has an Eldunari, and then he has a couple of other misc. magic items. He has a decent subclass and is doing fine for level six. But he constantly asks for more items and more power. And he is even causing problems out of the game. Me and his younger sister, who also is part of the party, have started to develop a relationship of sorts and he is constantly complaining and threatening me. Now I'm an older brother as well, but he is taking it way too far. He complains about everything from combat to social interactions. The other players offer criticism and ideas, he has gotten to the point where he is becoming unbearable. I need help. I'm thinking of saying, "If you can't respect the rules, you're going to have to leave." But the problem is the ripple effect that could have. What do I do?

If a player is giving you threats it stops being about what ripple effects it might have in-game.

I mean... yeah. Pretty much that. If it's hampering the fun of other players at the table then the conversation needs to be had. Heck even if the other players are enjoying the dynamic and it's hampering your fun I would still find a way to address it because DMing a game that makes you miserable in unsustainable. Rampant homebrew in general is tough to manage if you allow it, which is why I generally do not unless its very minor or a carefully thought out exception.

However... some of the other things you've dropped into the story makes this seem like a more complex social dynamic that may impact other areas of your life, so I'm reluctant to offer a serious recommendation with the little context that I have. It sounds like you're been stuck in an appeasement loop due to interest in his sister. That still brings me back to how other people are reacting to the situation - if they are similarly miserable (which I suspect they are if he's getting concessions that make him more powerful) then I would address it quickly or you may not have a group much longer.  Either way, I wouldn't just put up with the threats though - that's just grounds to stop the session where it is because it's not a game anymore.

I should state the threats are out of session and are the "If you hurt my sister..." type threats. And the ripple effect I'm worried about is IRL, not in-game. It's easy to fix the game, it's not as easy with real people.