I am running a 5e campaign in a homebrew setting.  Low magic.  Most magic users are warlocks because magic is hard and warlock is an easy path to power.  Lots of "great old one" type evil powers lurking in the shadows.  Only races are human, wood elf, dwarves, orcs (using half-orc stats) and halflings.  Some other races exist but they are super-rare and players were advised they would not fit the campaign without a GREAT backstory/hook.  It is a fairly dark, serious campaign and was advertised as such.

This is also a family game so there is a fair amount of both love and... poison... built into the player dynamics.  But that is inevitable and not what I am seeking help with.

One of the players is... a grass is greener on the other side of the fence type personality.  He has played 4 different characters so far this campaign and is working on another.   Most of his game time is spent complaining about how his character can't do all the things he wants it to do or asking me questions about what kind of feats I would allow for his next character build while the other players are trying to explore a dungeon or get information from a noble.  He has now revealed his next build is a non-existent (in this campaign world) race with a homebrew class/build.  When I pointed this out, the player pulled the "Magic exists, therefore this character can exist and you are only denying me because you don't want to do it" card on me.  He has also indicated the purpose of this character (which is based on a popular cartoon character) is to break up the structure of the game and make it more silly and fun.  

I consider my purpose as a DM to make the players happy and have a good time.  We are playing via zoom and for many of us this is our weekly break from the world.  The other players are leaving this up to me... they all prefer the darker, more serious tone of the game but also insist I do everything I can to keep this player engaged and a part of the game (as I said, it is a family game).  I feel trapped now.  If I give in to this player, I suspect he will use the new character to break the game - and if unsuccessful will just try again in a few sessions with an even more absurd build.  On the other hand, if I deny the player he has made it clear he will quit and it sounds like at least one of the other players and maybe more will follow him and that will likely be the end of the game.  I must also confess that the kind of game this guy wants to play ("Enter town, pull mayor's pants down, wait for hilarity to ensue") just sounds awful to me and Im not sure I can force myself to put in the kind of work necessary for that game to feel alive and fun for everyone.  (On that note, I had NO idea how much work was involved in running a game.  Its like I have 2 full time jobs now).  And on the other hand (i think that is 3 or 4 hands by now) I am self aware enough to realize I am very attached to the world and setting I've created and at least part of my resistance to this character build is its stated purpose in disrupting and breaking the game.  Perhaps I need to lighten up and - if they want to burn down the world - let the players play.

So what is going to happen next?  Do these situations ever have a happy ending?  Will I look back fondly one day at this as the beginning of my education as a DM?  Or am I doomed?  Because if I'm doomed I would prefer to know in advance so I can plan accordingly.

I think the best advice is to let that Player know that a “fun/silly” character would be inappropriate for that campaign. If they are not liking the darker tone, let them know that you will feel no disrespect if the choose to sit this one out and come back whenever the next campaign starts up.

As to switching characters constantly, I have a rule that a new PC always starts at the bare minimum XP necessary to have the same level as the lowest level member of the party. So, if all the rest of the PCs are 4th level half way to 5th around 4,600 XP, then a new PC would only start with 2,700 XP as that is the bare minimum to be the same level as the rest of them. Once they start realizing how far behind they are falling, they usually stop dead in their tracks.

As to playing stuff you have already said don’t exist in your world, stick to your guns on that. It isn’t fair to the others who followed your rules and haven’t switched Characters if you don’t.

Remember, you’re a player too, your fun matters.

Also, here is a helpful video you might find useful:

I hope that helps.

As a follow up to BigLizard’s advice, my recommendations were ways for you to call their bluff in such a way that they do not “give the authority” back to you, but for you to “take it back.”

In financial negotiations, that tactic is called “standing them up.” It’s a way of indicating that you have hit your hard line and if they don’t like it there’s the door. Letting them know that you are willing to let them walk away from the negotiating table empty handed.

The trick is, when you make that statement, no matter how long it takes, don’t say another word. Make them speak first. They might walk, but probably not. And either way you don’t have the headache anymore. If it isn’t fun to do the ridiculous amount of work a DM does, then why bother.

Most likely they’ll back down. They may bluster a bit first. Don’t respond. Let them holler all they want and you stand firm. When they finally settle down, you have taken your authority back and can then proceed with the good times.

 this character can exist and you are only denying me because you don't want to do it

if I deny the player he has made it clear he will quit and it sounds like at least one of the other players and maybe more will follow him and that will likely be the end of the gamewant to do it

There is so much here that makes me cringe -- in part because I see some milder reflections of it with one of my own players, although it is nothing this extreme. Consequently I have many thoughts about this, not all of which may be helpful. In no particular order:

Point 1: As DM, you are well within your rights to deny something for no "better" reason than that you "don't want to do it." So even if that were your reason, you would have every right to say no.

Point 2: You actually have a good reason. You have a world you have built, and this player wants to just make up whatever he wants. This is a major problem for the verisimilitude of the world. I would thoroughly support denying this player an out-of-theme concept, if I were at your table.

Point 3: As a DM, it is your job to help your players have fun. However, as a DM, it is your right to have fun as well. If allowing this character concept will break the world and ruin the campaign for you, then you will be miserable and I can tell you from personal experience that this will end the game just as if the player and someone else had quit. It will take longer. It will be increasingly awful for you. And then one day you will realize, you don't need this blankety-blank, and you'll just stop DMing. Again this phrase bears repeating: the DM should be having fun, too. 

Point 4: The idea of a player saying "give me what I want or I will quit" is unacceptable to me as a DM. I will not be blackmailed as a DM. I'll stop DMing first. 

I think perhaps the best solution is to say to this player, "look, it's clear you don't like how I am DMing. So why don't you DM and I will play?" In my experience, most of the time, when offered to be made DM, the player will back off. Because really, they don't want to be DM... they know how much work it is. Because DMing is work. Rather a lot of work, in fact. If it's not being appreciated, you don't need to do it. If your table is actually going to expect you to keep pacifying a malcontent or they quit, then I quote Matt Colville: "No D&D is better than bad D&D." I will add to that sentence -- it's infinitely better.

Yeah at this point, if I were in your shoes, I would be soliciting to see if he or someone else would be willing to DM... because I don't need this kind of abuse when I am just trying to make an adventure for everyone to enjoy. And if no one else wanted to do it, I'd say "well, I guess we'll have to find another game to play over zoom... Anyone ever played Talisman?"

You are a player too and deserve to enjoy the game, your friend is being a jerk

Sardonicmonkey beat me to it. Your fun is an important part of the game as well. Don’t be a martyr. You were clear with what was allowed and he came back to you with something completely different. It would be one thing if he had that amazing backstory you mentioned, or if he was at least trying to meet you part way, but it really sounds like he’s just being selfish; you don’t need to play along with that. 

It’s going to sound harsh, but I’d say just kick him. He obviously wants to play a different game from the rest of the group. And based on your limited description, I’d guess he lacks the emotional maturity to find a way to make his character fit with everyone else.  If you allow him to play, he’ll find one way after another to make your (and everyone else’s) experience miserable.

Tell him you don’t think his play style meshes with everyone else’s, and that he’s be better off finding a different kind of game. Then recruit someone else to fill the spot; DMs are a limited resource, players are a dime a dozen. 

It’s going to sound harsh, but I’d say just kick him. 

Although I agree with you, Xalthu... I think the OP is afraid to do this because the entire group might break up. I quote:

if I deny the player he has made it clear he will quit and it sounds like at least one of the other players and maybe more will follow him and that will likely be the end of the game.

I think that the OP is going to need to embrace this possibility. This is why I said, "No D&D is better than bad D&D." 

You cannot have players forcing you to change your rulings or alter your world on threat of quitting the game and taking half the group with them. That completely undermines the DM's authority, and will destroy the game. Better to just end it now.

But the OP is going to have to make peace with that, and it is a hard thing to do, especially if this is family (cousins or something -- OP said it is a family game). If this is people you have to be around and work or play or socialize with at other times, it could be intensely awkward to just kick someone from the group. 

This is why I suggest, instead of kicking, offering the offending player the DM role. Let that person deal with the headaches if they want. If not, then maybe you have some leverage, knowing people want to play but only you are willing to DM. One way or another though, get out of the situation... and if that means not playing D&D, well... then don't play D&D.

No D&D is a thousand times better than bad D&D.

kind wondering. What are the things that his character wants to do? Getting info, casting powerful damage spells, super tanking, deal a lot of damage to one target or multiple targets, charm/convince NPC to have his way, steal stuff...etc. 

Maybe, just maybe there is something in the official stuff that provides what he wants, but he didn't find it.

Also, "Tasha's cauldron of everything" has the rule about custom build race. Maybe tell him to wait and use the rule in the book to build the race. (he has to be the person to buy it not you)

How many players would you have left if some took off? I know that's not the solution you are hoping for, but DMing for small groups is really fun and relaxed. 3 players is actually my personal favorite.

A number of posters here have given good advice on peaceful ways to resolve this. Some involve parting ways, at least for the game. It doesn't have to be bitter on your end, just honestly caring for yourself. 

In the end, you can't control a person who is being difficult. You can only choose how you respond. 

If you end up with 1-2 agreeable players, you could probably invite a friend along. (Clear it with the group first.) You'd be surprised at how many people are interested in trying out this game.

If family time is your first priority here, maybe another type of game for a while for everyone to "reset."

As family maybe try something online. City of Heroes is being run on private servers now and is an easy and fun (and free) game that is not too hard on computers, since it was originally built in 2004. Nearly any computer that is still kicking would have more than enough juice to run COH. And you can play with teams of up to 8, and have giant supergroups... so there would be no limit to who you could invite to join.

And you can let the COH game worry about GMing for ya.