so we had a lvl3 half-orc barb with 2hp (32Max),and an ogre scored a crit and rolled max damage to boot. So the orgre dealt 36 dmg total while the barb was raging. My question is, does his resistance apply to all the damage, ie he's taking 18 dmg and his relentless endurance can kick in, or does the resistance only last until he hits 0hp  (4 points of damage) and then the remaining 32 damage kills him instantly (no relentless endurance)?

Let's go to the Damage and Healing rules.

Hit Points

(...)

Whenever a creature takes damage, that damage is subtracted from its hit points. The loss of hit points has no effect on a creature's capabilities until the creature drops to 0 hit points.

Damage Resistance and Vulnerability

(...)

Resistance and then vulnerability are applied after all other modifiers to damage. For example, a creature has resistance to bludgeoning damage and is hit by an attack that deals 25 bludgeoning damage. The creature is also within a magical aura that reduces all damage by 5. The 25 damage is first reduced by 5 and then halved, so the creature takes 10 damage.

You take damage and then subtract HP. Resistance and vulnerability change the damage you take.

You had resistance to the damage at the time you got hit. Having resistance reduces the damage from 36 to 18. Then you deduct that damage from your HP, and note that there's still 16 damage left. That's not enough to kill you, so you fall unconscious.