If Counterspell was subtly cast, or hidden in such a way that the other caster didn't know their spell was countered, would they ALWAYS be aware that their spell was countered?

For example, say a creature has Nystul's Magic Aura on them to disguise themselves as having a different alignment or having no spells on them, and the party's spellcaster has cast Detect Magic and discovered nothing, but something feels off, so they cast Dispel Magic when they think the creature isn't paying attention. Unknown to them, however, the creature, a hidden ally of the creature, or a discrete magical object cast Counterspell on the spellcaster casting Dispel Magic.

Is the spellcaster immediately aware that their spell was countered, or do they assume that it was cast successfully and that they really WERE just being paranoid?

I hate when these don't have a simple answer.

There are no rules on whether a caster knows their spell had an effect. The closest I could find is XGtE invalid spell targets, but that only says that the spell seems to fail and isn't fully applicable to this question.

I would say they can definitely tell their spell did nothing. Whether or not the can tell they were countered might vary from table to table/group to group.

I actually like this answer a lot. Knowing it didn't do anything and knowing it failed are two very different things, but it is a good compromise between "You have successfully dispelled any magic if it existed," and "Your spell was countered, but from where? Find out next session (while I frantically rewrite a major plot point)!" It also takes a lot of inspiration from your reference to XGtE, which is probably as close to RAW as it gets, at least from my research.

I generally assume that a spellcaster can "feel" their magic taking shape and would then know if that shape is disrupted. However, when in doubt, have the player roll an arcana/spellcasting check.

Giving the player a chance to understand what is happening is likely to play better than stone walling them.

I would aim to have the subtle counterspell roll against a passive skill (probably whatever their spellcasting skill is) to hide the counterspell. If they don't roll high enough, then the caster knows somethings up, and if they roll high enough the caster doesn't notice anything. I guess with the "shape of magic" it would be about timing - dispelling after the magic leaves the caster but before it lands to make it unnoticeable.

If I recall correctly, casting a spell of any kind, by default, breaks stealth and reveals yourself. However, I think an argument could be made that Counterspell, with no verbal or material components, could be cast without necessarily revealing oneself to the intended target. Although I suppose that's not really the question being asked... we could just say that the minion doing the casting is a sorcerer who uses Subtle Spell and still have the same primary question.

I would argue, though, that a player would not know that their spell was Counterspelled if, for whatever reason, they don't see the minion casting the spell. Many spells without obvious visual effects will note that a target is aware when they are targeted by the spell (mostly charm spells), but Counterspell doesn't make that distinction. The specific wording in the spell itself is also "The spell fails and has no effect". Not necessarily that the spell is stopped or the energy is syphoned or something like that... just that it fails and has no effect, which is exactly the same wording as you would get if casting Dispel Magic on something that's not enchanted in the first place.