Last session out Barbarian fought a Mammoth. I hit him with enlarge. Could i have taken him from large to giant with a metamagic or upcasting?

No, you could not have doubled up on Enlarge/Reduce.  If casting the spell at a higher spell slot gave additional or different effects, the text of the spell would tell you.  It does not, therefore it does not do anything extra at different levels of spell slot.

As for Metamagic, there's a way it might work, but I'm uncertain.  You could potentially Twin the spell, and target the Mammoth to shrink it (Con save to resist).  But I'm not entirely sure Twin interacts that way with a multiple choice spell like Enlarge/Reduce.  On one hand, you might be allowed to effectively cast the spell twice, creating 2 unrelated castings.  On the other hand, you might cast the spell, then copy it after it's already cast, which would require the same effect as the first.

Quote from drag0n_77 >>

No, you could not have doubled up on Enlarge/Reduce.  If casting the spell at a higher spell slot gave additional or different effects, the text of the spell would tell you.  It does not, therefore it does not do anything extra at different levels of spell slot.

As for Metamagic, there's a way it might work, but I'm uncertain.  You could potentially Twin the spell, and target the Mammoth to shrink it (Con save to resist).  But I'm not entirely sure Twin interacts that way with a multiple choice spell like Enlarge/Reduce.  On one hand, you might be allowed to effectively cast the spell twice, creating 2 unrelated castings.  On the other hand, you might cast the spell, then copy it after it's already cast, which would require the same effect as the first.

 RAW, Twinning allows you to pick two different effects, so yes your mammoth example would work with a twinned spell.

Not sure about the RAW. Twin Spell doesn't let you cast it a second time, it just adds a target to what the spell does. So when you target one creature or object with the spell you now target two. It isn't stated at what point you choose the effect - at casting or at targeting. It technically doesn't say you even get to choose which effect just that the target will enlarge or reduce in size.

The likely and easy assumption is the caster chooses which effect when targeting, so yes on twinning you could enlarge one and shrink another. However, the RAW does not specify that you choose which effect happens at all. It's implied you do, of course saying you cause the chosen target to go bigger/smaller, but causing something and choosing something are not the same. You can cause your skin to turn blue when a wild sorc casting a spell, doesn't mean you chose it.

I as DM would certainly let you choose per target, and I imagine most would, although some may interpret choice at casting - which means you decide big/small option for the whole spell, therefore applying the same option to all targets. This is less likely but possible.

However, neither is actually raw, since RAW technically doesn't specifically state you choose at all. Likewise conjure animals states you conjure them, it doesn't state you choose which ones beyond the CR. In these circumstances it's a DM to decide.

I'm 99.99% sure you're meant to (by RAI) choose big/small option in the spell, since it's 50% useless otherwise, but it's technically incorrect to say you do by RAW, since RAW doesn't specify one way or another.

There are a lot of problems with making you twice as large.  For one thing, the Enlarge spell itself cheats - it gives you advantage on strength and strength saving throws rather than actually increasing your strength.  

Personally, I would make a new spell called "Huge Size" rather than try to do something to the Enlarge/Reduce spell.