So I'm pretty new to DMing 5e and D&D overall, but I have some experience DMing other systems. I am also a beginner player in 3.5, so I don't know the specifics of most spells by heart yet.

The thing that happened yesterday was:
The PC High-elf Wizard used lightningbolt in the end of a pretty long and difficult encounter. He had planned for it, and had three enemies lined up. He rolled the damage, I rolled their Dex saves. They took damage, two successes one fail.

However... what my weary mind, and probably his too (After 2 hours rping and 4 hours of straight encounter for me, with a large group of 8 PCs and 4 NPCs against16 enemies) missed was that the caster does not chose to evade targets with that spell, and one NPC were in the  5feet wide 100 feet long line. The lightningbolt made 22 damage on a failure. The NPC have a HP maximum of 21 and an average dex so no bonuses and not proficient in the save. So he have to beat a 14 or be fried to death.

Should I enforce the rule, and have the NPC roll a dex first thing next session? Or should I overlook it for now, talking to the player and ask him to read his spells, but also in the coming sessions ask for specifics of every single spell and enforce everything? Or thirdly, should I look up every single spell they use to make sure everything is in order? All my options could result in bad outcomes. Both for the players, and for me for having to ask for specs every time and not being able to trust them.

I'll talk to the player regardless, but I want to know opinions from other DMs first before talking to him.

About the player:
As a player, he is a veteran 3.5 player and DM. He is new to 5e. Usually gets everything right, sometimes mixes up 3.5 and 5e rules but never more than once per rule, and is engaged in the story, and filling the other players in on general lore of the world of Faerun. The fact that he is a veteran player and DM is the main reason I didn't look up specs for the spell immediately when he cast it.

Most of the other players are new to RPing in general. One is a comfortable 5e player. This is the first time this has happened  (They are lvl 5) so I don't belive he wanted to scam me.

Sincerly
Rona

It's not usually good form to retroactively redo combat. I recommend noting to everyone this particular oversight with the intention of letting it slide that time, but remembering it going forward. In general, decent players will know their spells, and you as DM won't have encyclopaedic knowledge of every spell. There are two practical things you can do. 1) Have a spell reference ready. 2) Get familiar with the spells they are casting, and make special note of ones that might have greater rules interactions. E.g. few people will care too much about a cantrip like Acid Splash, but remembering the dimensions and intricacies of a Wall of Force might take some practice.

Like JCAUDM said, point it out, but let it slide since you missed it too.

*I also want to make sure you (and your players) aren't thinking lightning bolt is wider than it is. Line spells width is a diameter not a radius.

Just make the call on the spot as good as you can. Without wasting too much time looking up sources. Then after the session read up the exact ruling. Learn from it and remember it for the future. Do not retcon elements and just let it be the way it was. Mistakes happen and that is fine. As long as the call was reasonable and fair, although flawed, its still acceptable. Unless there is something wrong with you it is pretty much impossible to remember EVERY rule all the time. Mistakes are normal to occur.

As a DM I ask my players to send me their character sheets every level up. This also lets me to look up their specific class, race, feats and whatever. That way, I as DM, can read up on what they're using. Without needing to know the entire book. And this is good form anyhow, because you'll need to do what their spells/items could do for creating interesting encounters/situations.

Don't retcon it. 

We all make mistakes, or can't remember a rule - so we quickly make a call in the moment rather than derail the action flow as everyone dives for their rules, only to discover that we made a call contrary to RAW.

If you watch Critical Role at all, that happens to Matt Mercer all the time - and he is literally a world famous professional Dungeon Master.

I have it baked into my session zero discussion with players that - especially in combat - the DM will make a call, we'll roll with it, we can discuss it again outside of game flow to clarify how it was supposed to work, so we know for next time, but the ruling stands.

Is the Wizard a School of Evocation subclass? Because if so, then they have a second level class feature that grants this exact ability to avoid hitting some chosen creature(s) with an area effect evocation spell like fireball or lightning bolt. Is that possibly what the player was using? 

Those mistakes happen, what I do is if I can work it into the story. Maybe the wizard is resting and rethinking the battle and realised the lightning bolt should have blasted the box, but didn't. Maybe he felt the hand of Mystra , or Azuth takeing a interest in him. At most have him pay a little extra gold to one of their temples.

I'll echo the sentiments of the other DMs here and say that the bar is continuous improvement, not perfection.  Everyone will make honest mistakes and the best thing is to roll with them but correct them moving forward while making the most of an interesting plot twist (it missed you? it's a miracle!).  Make sure everyone understands that expectation (i.e. that mistakes that were allowed to stand in the moment are not precedent setting).  Also, beware the danger in moving back in time (even short amounts) to take back your own actions as this signals to the players that it is okay for them to do so as well.  This devolves into chaos pretty quickly (if I knew that, I *would have done this instead...)