So I recently implemented a new house rule for two weapon fighting. By default, the second weapon you're fighting with DOES add the appropriate ability modifier to damage rolls. Those that take the two weapon fighting style can make an attack with their second weapon as part of their first attack on their turn, no longer using up their bonus action, but still limited to only attacking once with their offhand per round. Is this OP? Or is it just P enough?

You can already make attacks with either weapon in either hand during the attack action, and can attack once or more with each if you have extra attack so long as the total number of attacks does not exceed the number you're allowed to make. There isn't really a such thing as "offhand" in 5e. The only restriction on two weapon fighting is that when you make the bonus action attack, you must have used weapon A at least once during your attack action in order to use your twf attack with weapon B, or vice versa.

So for example, a dual wielding character with extra attack can attack in any of the following ways with weapon A and B:

Action: AA, Bonus Action: B
Action: AB or BA, Bonus Action: B
Action: AB or BA, Bonus Action: A
Action: BB, Bonus Action: A

But cannot attack with the same weapon 3 times.

Now, if you're saying that a dual wielding character can make an additional attack, one more than they typically would be able to (with or without extra attack) with just their action, just because they have two weapons, yeah that's OP, leave the rules alone.

If all you're doing is freeing up the bonus action for other non-weapon-attack options (e.g. Second Wind), I don't have a problem with it.

May I suggest something? What about changing the Dual Wielder feat? It would read as follows, ignoring the original text:
"When you are wielding two light weapons in both hands you can, once per round, make an additional attack without spending a bonus action.You can draw or stow two light weapons when you would normally be able to draw or stow only one".

You may ask: "wait, does that mean if my char decides to use its bonus action to attack with the secondary weapon, it would gain two additional attacks"?

The answer is: Yes

So, for example, if a player decides to create a character (fighter) using the variant human, who gains one feat in the first level, he/she could attack 2 times (Action + feat) or 3 times (Action + feat + Bonus action) per round.

Now, reaching the second level, the same fighter gains Action Surge. With it, the number of attacks could be something between this:
2 attacks (Action + feat)
3 attacks (Action + feat + Bonus Action)
4 attacks (Action + feat + Bonus Action + Action Surge)

In the 5th level, however, things start to get serious. The same char now have Extra Attack, making it possible to unleash a furious sequence that varies like this:
3 attacks (Action + Extra Attack + feat)
4 attacks (Action + Extra Attack + feat + Bonus Action)
6 attacks (Action + Extra Attack + feat + Bonus Action + 2 Attacks by Action Surge)

The numbers in the 11th and 20th levels:
11th - 4, 5 or 8 attacks
20th - 5, 6 or 10 attacks.

I would suggest to use the "new" Dual Wielder feat that I posted in this thread. It does almost the same effect that you discribed, minus the ability modifier to dmg, with the price of buying one feat. So, for non-variant humans, it would take at least 4th level to do what you want. Let the Fighting Style feature work the damage modifier as it was originally intended.

Imho Sonyuu while I'd advise against meddling with the action economy anyway, I'd vastly prefer the OP's suggestion to yours from a balancing point of view strictly because the OP is at least limiting the number of attacks per round to what would usually be possible with twf. Particularly because the OP limited it to one "offhand" (a meaningless word in 5e) attack per turn, whereas you seem to be allowing both a bonus action attack and an additional attack from the feat, allowing for 3 attacks per round at lvl 1. That is double dipping and is busted.

From a balancing point of view you've gotta consider extra attacks mean more chances at sneak attack, more smites, more hex/ hunters mark/ crimson rites/spirit shroud. It means giving monks 3 attacks plus 2 flurry of blows (and forcing them to use weapons to achieve this when many just want to use unarmed strikes). It means the (situationally broken) surprise attack the new bugbears get procs an additional time.

To be fair though, with the OP's initial proposal this problem still exists if the player has a good bonus action attack already, such as with martial arts, spiritual weapon, the artificer canon or whatever. 

Nah personally I'd leave it as is, current rules work fine, there are already plenty of benefits to two weapon fighting if you build it correctly with the above modifiers in mind. Whatever you do though I'd recommend not allowing both an additional attack *and* the two weapon fighting bonus action attack.

Rogue is the sticking point here IMO. In that class, the bonus action attack is actually an interesting, competitive option with all the other bonus actions that rogues can use. Even if your main attack doesn't land, it might be better to Cunning Action yourself out of melee range than take an extra attack. Freeing up their bonus action is a significant utility/mobility boost - which is debatably OP depending on how someone feels about rogues (opinions vary widely on how strong they are as is). This is the only class that has real incentive to dual wield in the current ruleset, if only because shields and two-handed weapons are simply not options.

But on ranger - in the past, the definitive dual-wielder - dual wielding just doesn't work. Several subclasses require a bonus action for their built-in damage, Hunter's Mark takes your bonus not just to cast but to move it around, and having your hands full limits your spellcasting options. They also tend to be defensively weaker out of the box because other builds are either using a shield (1h weapon) or take the Defensive fighting style (2h). Here at the other end of the spectrum, ranger needs OP's boost for DW to be on the table at all.

And the classes in between? Meh. I've heard the argument against smiting paladins, but you know what? I've never seen a dual wielding paladin in play. If that bonus attack were that good, you'd think we'd see it even now. Similarly, hexblades are giving up precious, needed defense, and you still need your bonus to cast and move hex.

Particularly because the OP limited it to one "offhand" (a meaningless word in 5e) attack per turn, whereas you seem to be allowing both a bonus action attack and an additional attack from the feat, allowing for 3 attacks per round at lvl 1. That is double dipping and is busted.

This is just being pedantic. What OP clearly meant was your bonus action attack is made as part of the attack and then you don't make a bonus action attack. It just needs correct wording.

For my two cents worth, OP's change is fine. It does give a decent boost to rogue and monk, but it requires them to MC and those are two classes that can take a boost and still not be at the top of the damage-dealing rankings. For any other class that has a two-handed or shield option, it simply helps to make dual wielding not an absolute trap.

Alternative ways I've considered giving out this enhancement is making it a part of the Dual Wielder feat (replacing the non-light-weapons bullet), adding that feature to magic weapon sets so the DM has more control over it, or baking it into individual subclasses. But I do think the method OP suggests is the most simple and elegant solution.