Not all offensive uses of spells are obvious. Some are even intended to be used outside of combat (Charm Person, for example).
Let's consider a sorceror with subtle spell. You're in a crowded bar. You cast subtle phantasmal force on someone. Do his friends realize he's been attacked? It just looks like a psychotic episode. Does the target even realize he's been attacked? (Aside from potentially by whatever his phantom fear is). Should initiative even be rolled? What if he passes his saving throw, instead. He doesn't even know he was targeted by a spell, right?
This is 100% DM territory. All people can tell is how they'd personally handle it. I'll give how I would, but there's no official answer.
Phantasmal Force: In a world where magic is common enough, I'd definitely start initiative. People might start getting accosted and threatened if the culprit doesn't come forward and it would likely be a chaotic mess. That is... If they freak out. Phantasmal Force doesn't actually deal damage unless you make it something hazardous. So you could just make it a creature or object that disturbs them but isn't actively harmful... That's less likely to cause suspicion, unless it happens to multiple people or comes completely out of the blue. So such a situation would be determined by just what the effect is and the circumstances around the use. (Phantasmal Killer would likely be the first result here, as it definitely deals damage consistently and the person would be more distressed.)
Dominate Person: I'd only call for initiative in this situation if this was used to start a fight. If used in a way that was subtle enough to mask the behavior, I'd say nothing happens immediately. And if the players had studied their target and were having it act within the bounds of their normal activity while around people who knew them (and not being overly suspicious around strangers) there would be no checks moving forward. However, if they had the person acting suspiciously (which is a lower bar around people who know them) there would be Insight checks from NPCs to determine what's wrong, and possible investigation.
[...] What if he passes his saving throw, instead. He doesn't even know he was targeted by a spell, right? [...]
I remember similar threads about a spell being noticeable (*). There are some rules in the PHB ("Awareness of Being Targeted") and in XGtE ("Perceiving a Caster at Work"). Also Do you always know when you’re under the effect of a spell?
So based on those rules, I'd say that when using something similar to Sorcerer's Subtle Spell or the Psychic Spells feature, if the creature succeeds on the Saving Throw and there are no secondary effects after succeeding, it won't know it was targeted by spells such as Charm Person, Dominate Person, Phantasmal Force, or Suggestion.
Regarding your main question, I think, as Athanar90 said, it's DM territory and IMO it would depend on how the spells are used, whether they deal damage, etc.
Not all offensive uses of spells are obvious. Some are even intended to be used outside of combat (Charm Person, for example).
Let's consider a sorceror with subtle spell. You're in a crowded bar. You cast subtle phantasmal force on someone. Do his friends realize he's been attacked? It just looks like a psychotic episode. Does the target even realize he's been attacked? (Aside from potentially by whatever his phantom fear is). Should initiative even be rolled? What if he passes his saving throw, instead. He doesn't even know he was targeted by a spell, right?
What about subtle Dominate Person?
When combat starts between two or more creatures, every participant rolls Initiative to determine the order of turns.
If a combatant is surprised by combat starting, that combatant has Disadvantage on their Initiative roll. For example, if an ambusher starts combat while hidden from a foe who is unaware that combat is starting, that foe is surprised.
As DM i'd give Advvantage to ambusher's Initiative, Disadvvantage to surprised creature and use Initiative scores instead of rolling Initiative for following result;
Now in very rare case where the surprised creature in combat still doesn't know who are the other participants or what was attempted when it's turn comes up, as DM i can say it an intuition of imminent danger is felt despite not noticing any result from the spell.
Failed Dominate Person described as the creature about to drink briefly losing control of its arm holding the mug inexplicably, or resisting voice in its head urging to do something to me is a freakin way to start combat 😊
Ask the players how they would like it to be handled at the table, letting them know it goes both ways. Personally as a player and DM id like some things to not start combat as yeah people don't know something is up.
But I can understand if the players would think it is unfair they did not get turns while things were going down while they were unaware of it. In which case when the players cast dominate or whatever subtly the opposition gets a feeling something is off and roll init as they are on guard. Sort of like in video games where the guards spot enough to look for things, but they haven't fully seen you their awareness meter is at half. they start looking around, trying to find out what is up. but they are not charging at you swords drawn.
The Dungeon Master Guide offer more guidance on rolling Initiative;
Rolling Initiative
Combat starts when—and only when—you say it does. Some characters have abilities that trigger on an Initiative roll; you, not the players, decide if and when Initiative is rolled. A high-level Barbarian can't just punch their Paladin friend and roll Initiative to regain expended uses of Rage.
In any situation where a character's actions initiate combat, you can give the acting character Advantage on their Initiative roll. For example, if a conversation with an NPC is cut short because the Sorcerer is convinced that NPC is a doppelganger and targets it with a Chromatic Orb spell, everyone rolls Initiative, and the Sorcerer does so with Advantage. If the doppelganger rolls well, it might still act before the Sorcerer's spell goes off, reflecting the monster's ability to anticipate the spell.
Now in very rare case where the surprised creature in combat still doesn't know who are the other participants or what was attempted when it's turn comes up, as DM i can say it an intuition of imminent danger is felt despite not noticing any result from the spell.
Failed Dominate Person described as the creature about to drink briefly losing control of its arm holding the mug inexplicably, or resisting voice in its head urging to do something to me is a freakin way to start combat 😊
A creature only knows they were the target of the spell if the spell does something obvious. A failed dominate person doesn't do anything obvious, so you wouldn't know you were the target of a spell, RAW. ("Unless a spell has a perceptible effect, a creature might not know it was targeted by a spell at all. An effect like crackling lightning is obvious, but a more subtle effect, such as an attempt to read a creature’s thoughts, typically goes unnoticed, unless a spell says otherwise.")
Indeed, even a successful dominate person doesn't let the target know they were charmed (unlike charm person - the only reason someone knows they were the target of a charm person is because of this sentence: "When the spell ends, the target knows it was Charmed by you."). Maybe using the telepathic link lets it know it was the target of a spell (but which spell? Several spells allow telepathy, as do several racial and subclass features.) So even a successful dominated person would be unaware they were the target of a spell, especially if you didn't bother with the telepathic commands.
Not all offensive uses of spells are obvious. Some are even intended to be used outside of combat (Charm Person, for example).
Let's consider a sorceror with subtle spell. You're in a crowded bar. You cast subtle phantasmal force on someone. Do his friends realize he's been attacked? It just looks like a psychotic episode. Does the target even realize he's been attacked? (Aside from potentially by whatever his phantom fear is). Should initiative even be rolled? What if he passes his saving throw, instead. He doesn't even know he was targeted by a spell, right?
What about subtle Dominate Person?
This is 100% DM territory. All people can tell is how they'd personally handle it. I'll give how I would, but there's no official answer.
Phantasmal Force: In a world where magic is common enough, I'd definitely start initiative. People might start getting accosted and threatened if the culprit doesn't come forward and it would likely be a chaotic mess. That is... If they freak out. Phantasmal Force doesn't actually deal damage unless you make it something hazardous. So you could just make it a creature or object that disturbs them but isn't actively harmful... That's less likely to cause suspicion, unless it happens to multiple people or comes completely out of the blue. So such a situation would be determined by just what the effect is and the circumstances around the use. (Phantasmal Killer would likely be the first result here, as it definitely deals damage consistently and the person would be more distressed.)
Dominate Person: I'd only call for initiative in this situation if this was used to start a fight. If used in a way that was subtle enough to mask the behavior, I'd say nothing happens immediately. And if the players had studied their target and were having it act within the bounds of their normal activity while around people who knew them (and not being overly suspicious around strangers) there would be no checks moving forward. However, if they had the person acting suspiciously (which is a lower bar around people who know them) there would be Insight checks from NPCs to determine what's wrong, and possible investigation.
I remember similar threads about a spell being noticeable (*). There are some rules in the PHB ("Awareness of Being Targeted") and in XGtE ("Perceiving a Caster at Work"). Also Do you always know when you’re under the effect of a spell?
So based on those rules, I'd say that when using something similar to Sorcerer's Subtle Spell or the Psychic Spells feature, if the creature succeeds on the Saving Throw and there are no secondary effects after succeeding, it won't know it was targeted by spells such as Charm Person, Dominate Person, Phantasmal Force, or Suggestion.
Regarding your main question, I think, as Athanar90 said, it's DM territory and IMO it would depend on how the spells are used, whether they deal damage, etc.
(*) this is one of them: 2024 Warlock 'Psychic Spells' Feature
When combat starts between two or more creatures, every participant rolls Initiative to determine the order of turns.
If a combatant is surprised by combat starting, that combatant has Disadvantage on their Initiative roll. For example, if an ambusher starts combat while hidden from a foe who is unaware that combat is starting, that foe is surprised.
As DM i'd give Advvantage to ambusher's Initiative, Disadvvantage to surprised creature and use Initiative scores instead of rolling Initiative for following result;
Ambusher 15 + DEX mod
Surprised 5 + DEX mod
Now in very rare case where the surprised creature in combat still doesn't know who are the other participants or what was attempted when it's turn comes up, as DM i can say it an intuition of imminent danger is felt despite not noticing any result from the spell.
Failed Dominate Person described as the creature about to drink briefly losing control of its arm holding the mug inexplicably, or resisting voice in its head urging to do something to me is a freakin way to start combat 😊
Ask the players how they would like it to be handled at the table, letting them know it goes both ways. Personally as a player and DM id like some things to not start combat as yeah people don't know something is up.
But I can understand if the players would think it is unfair they did not get turns while things were going down while they were unaware of it. In which case when the players cast dominate or whatever subtly the opposition gets a feeling something is off and roll init as they are on guard. Sort of like in video games where the guards spot enough to look for things, but they haven't fully seen you their awareness meter is at half. they start looking around, trying to find out what is up. but they are not charging at you swords drawn.
The Dungeon Master Guide offer more guidance on rolling Initiative;
A creature only knows they were the target of the spell if the spell does something obvious. A failed dominate person doesn't do anything obvious, so you wouldn't know you were the target of a spell, RAW. ("Unless a spell has a perceptible effect, a creature might not know it was targeted by a spell at all. An effect like crackling lightning is obvious, but a more subtle effect, such as an attempt to read a creature’s thoughts, typically goes unnoticed, unless a spell says otherwise.")
Indeed, even a successful dominate person doesn't let the target know they were charmed (unlike charm person - the only reason someone knows they were the target of a charm person is because of this sentence: "When the spell ends, the target knows it was Charmed by you."). Maybe using the telepathic link lets it know it was the target of a spell (but which spell? Several spells allow telepathy, as do several racial and subclass features.) So even a successful dominated person would be unaware they were the target of a spell, especially if you didn't bother with the telepathic commands.
RAW PC don't know monsters resist any damage dealt though i like to describe it still. 😊