Do you only crit the first weapon attack or if you’re dual wielding do you crit the second attack as well?

If your target is surprised AND you go before them in the initiative order, then all of the hits by the Assassin rogue attacks will be critical hits. In addition, all of the attacks will be made with advantage.

"Starting at 3rd level, you are at your deadliest when you get the drop on your enemies. You have advantage on attack rolls against any creature that hasn’t taken a turn in the combat yet. In addition, any hit you score against a creature that is surprised is a critical hit."

"If you’re surprised, you can’t move or take an action on your first turn of the combat, and you can’t take a reaction until that turn ends."

Surprise is usually assumed to end after the creature's first turn has passed so neither advantage nor critical hits apply if the rogue goes after the target in the initiative order.

An official ruling in Sage Advice Compendium clarify that question;

For triggering the rogue’s Assassinate ability, when does a creature stop being surprised? After their turn in the round, or at the end of the round? A surprised creature stops being surprised at the end of its first turn in combat.