Can the character casting zone of truth or other mind altering spells tell the spell failed? If someone succeeded theirsaving throw, does the spellcaster know? Could the person who cast the spell ask questions thinking the spell succeeded, and thus reveal what they are after to the person they thought they were controlling?

Zone of truth explicitly says that you know whether a creature succeeds or fails its saving throw. For something like charm person or modify memory, you could probably tell based on how they act, but you wouldn't know. It's also quite likely that the target wouldn't know what spell you were using and so wouldn't be able to fake being under the effect. Now I'm thinking about the possibilities of a sorcerer's subtle counterspell...

Gameplay of a failed attempt at Charm or Geas could be fun but tricky. 

I would have to say that the caster would know because the target doesn't act the way it should based on the effect wanted from the spell. However, in a literal sense, no, there is no specific indicator that the caster would know that the spell has failed unless specifically stated in the spell's description.