Unarmed Fighting style says, "When you hit with your Unarmed Strike and deal damage, you can deal Bludgeoning damage equal to 1d6 plus your Strength modifier instead of the normal damage of an Unarmed Strike..." 

The Enhanced Unarmed strike part of Tavern Brawler says, "When you hit with your Unarmed Strike and deal damage, you can deal Bludgeoning damage equal to 1d4 plus your Strength modifier instead of the normal damage of an Unarmed Strike."

Both feats say that you have to deal damage first with the unarmed strike to get the enhanced damage version that replace the normal damage. However, you already done damage to get the enhanced version that cancels out the damage you dealt, making so you don't do damage to get the enhance damage.

So, does this break these feats and make it impossible to enhance damage, or do you deal the damage of a normal unarmed strike, cancel out the damage with either or both feats, then do the enhanced damage or damages? And if so, do both feats are meant to be stacked because they are so similar?

Both feats say "instead of the normal damage of an Unarmed Strike" If you use one feat to change the damage calculation of an unarmed strike you are no longer doing the normal damage of an unarmed strike and therefore cannot use the other feat to further change the damage calculation. It's one or the other, not both.

These feats would stack if they made your Unarmed Strike deal 1d4 or 1d6 extra damage but since you can deal 1d4 or 1d6 instead of the normal one, they don't.

They don't stack with Monk Martial Arts Die for the same reason.