So we started playing a Ultramodern 5 campaign a little while back and my character's class is an Infiltrator. 

My character has two abilities that sort of interact, resulting in a lot of damage. Essentially, an assassination attempt, since it's an infiltrator. The first allows me triple the damage dice, and the second allows me to turn the hit into an automatic crit. 

At first, I interpreted this to mean, for instance, if damage is a D10 +3, and I roll a 10 for a total of 13 damage, I get to triple that damage, giving me 39, and then turn that into a critical hit, resulting in 78 total damage. 

My DM felt that was too OP and we looked over it. It turns out the first ability says I triple the dice, not the damage:

What does the text for Critical Hits say in Ultramodern? The second feature is not adding the dice itself, it is turning the hit into a critical hit, which is not a class feature or archetype.

In base 5e, all damage dice are rolled an extra time (https://www.dndbeyond.com/sources/basic-rules/combat#CriticalHits)

 So with a 1d10+3 regular damage roll, but you had advantage and made it a critical hit it would be:

1d10 (base) + 2d10 (extra damage dice) + 3d10 (critical hit)

So, this is hard to answer canonically, since you're way outside the normal rules.

Edit: I read "triple your damage dice" as "multiply them by three", rather than "triple the number of them you roll". If it's read that way, there's a much stronger argument for you getting to roll 6d10, then adding your bonuses. However, the rules are ambiguously written.

However:

When you score a critical hit, you get to roll extra dice for the attack's damage against the target. Roll all of the attack's damage dice twice and add them together. Then add any relevant modifiers as normal. To speed up play, you can roll all the damage dice at once.

For example, if you score a critical hit with a dagger, roll 2d4 for the damage, rather than 1d4, and then add your relevant ability modifier. If the attack involves other damage dice, such as from the rogue's Sneak Attack feature, you roll those dice twice as well.

If you were to get a critical hit, without the tripling part, you would roll 2d10, and add your bonuses.

With the tripling ability, there are multiple possible interpretations. Also, first of all, the rules here are badly written, as that mechanic could always be interacting with a critical hit, and leaving the question unanswered is very sloppy. (I will not comment on the balance.)

• You roll 1d10x3, 1d10 normal, and add your bonuses. You don't choose which die gets tripled after the roll, but before. This seems to be the DM's interpretation, and it's perfectly reasonable -- it's ruling that the tripling only applies to the base damage die, and not the critical die. Since it's not extra dice, you don't roll them twice.
• You roll 1d10x3, and another 1d10x3, and add your bonuses. This is the best you can expect to get.
• You rule 1d10x3, and that's it. This is ruling that this is a super-critical and replaces the normal critical. Without looking at the actual rules involved, it's hard to justify, but could still be plausibly made.

Just talked to my DM about your answers and we agreed that from now on it will be a roll of 6 D10s plus my bonus. Thanks for the help!

I'm not sure of current 5e rules on multiple multipliers to the same roll, but in previous editions I recall them being additive not multiplicative. This means that applying 3x and 2x multipliers to a single die roll results in actually rolling 4x the original amount of dice, not 6x. The idea is that each multiplier adds an amount to the total damage based on the original amount and not the the result of the original amount that has already been modified by a previous modifier. A 2x (double) multiplier to a single die means you roll that normal one plus one more and a 3x multiplier means you add two extra dice to that original one. In 3.5 this math is used most often for stacking modifiers on critical hits. Regular 5e of D&D simply does not stack such modifiers as all crits are simply "roll twice as many dice" and while abilities and features can add dice to the pool to be doubled nothing changes the number [2] that the dice are being multiplied by.

Quote from Flushmaster >>

I'm not sure of current 5e rules on multiple multipliers to the same roll, but in previous editions I recall them being additive not multiplicative. This means that applying 3x and 2x multipliers to a single die roll results in actually rolling 4x the original amount of dice, not 6x. The idea is that each multiplier adds an amount to the total damage based on the original amount and not the the result of the original amount that has already been modified by a previous modifier. A 2x (double) multiplier to a single die means you roll that normal one plus one more and a 3x multiplier means you add two extra dice to that original one. In 3.5 this math is used most often for stacking modifiers on critical hits. Regular 5e of D&D simply does not stack such modifiers as all crits are simply "roll twice as many dice" and while abilities and features can add dice to the pool to be doubled nothing changes the number [2] that the dice are being multiplied by.

 I'm not sure it is mentioned anywhere explicitly. The only case I'm aware of is that if a Paladin crits, he can us Diviine Smite and that gets doubled too. Which suggests that it is multiplicative, but isn't exactly rock solid.

Quote from Tillable >>

So we started playing a Ultramodern 5 campaign a little while back and my character's class is an Infiltrator. 

My character has two abilities that sort of interact, resulting in a lot of damage. Essentially, an assassination attempt, since it's an infiltrator. The first allows me triple the damage dice, and the second allows me to turn the hit into an automatic crit. 

At first, I interpreted this to mean, for instance, if damage is a D10 +3, and I roll a 10 for a total of 13 damage, I get to triple that damage, giving me 39, and then turn that into a critical hit, resulting in 78 total damage. 

My DM felt that was too OP and we looked over it. It turns out the first ability says I triple the dice, not the damage:

Once per turn, if you hit on an attack you have advantage on, you triple your weapon’s base damage dice (not any additional damage dice from class features or ar- chetypes).

So that means, instead of tripling the damage I rolled, I get to roll 3 D10s, plus 3. So max damage if I land on all three tens, 33 damage. I agreed with that part, as it does say 'damage dice' not damage. But now, I argue, that that 33 should get doubled when I add the other ability that turns it into a critical hit. Here's what that ability says:

Instead of using your Hit Dice to recover hit points, you can expend one during an attack action after hitting to turn the attack into a critical hit.

So I should be able to expend one Hit Dice to turn the 33 damage hit into a critical hit, giving me 66 total damage. But my DM is saying that the critical hit only applies to one of the three hit dice. So if I roll 3 d10s, I can double the first one. Let's say they all land on ten, then it's 20 +10 +10 +3, for a total of 43 damage.

I disagree. I think the tripled hit dice becomes the damage, which then gets doubled because its a critical. Should it be 66 damage or 43?

 I'm not familiar with "Ultramodern". I'll just run with how I'd interpret it in a 5e context.

The first ability talks about the weapon's base dice, and explicitly excludes additional damage dice from other features. You're then trying to use another ability to add damage (which I'd say is a "feature") and therefore is explicitly excluded. The DM is correct.

Of course, there is the nomenclature. If "ability" refers to a separate category of things to "class features or archetypes", ie "abilities" isn't a subset of "class features or achetypes", or at least that ability isn't, then it isn't excluded, then the question becomes "Are the increased damage dice due to it being a critical hit classed as the "weapon's base damage dice"?". Again, I'd lean to it not being the base damage dice because to me the "base damage dice" is what the weapon normally is, not when subjected to the  various mechanics. The base damage of a shortsword in 5e is d6. Criticals add on to that base damage.

But as I said, there is the issue of nomenclature. It really depends on how the game describes it and how it organises itself, which I don't have access to.