I am playing in my first (and second) 5e game(s) and I am trying to learn how to run the game myself.  I have all kinds of ideas, but scaling the encounters is something I am very wary of at this point.  I would like to share the details of my last encounter and ask if y'all think I don't have the right feel for this.

We have a party of 7 PCs, and we were going to a place several days travel away so we elected to take a shortcut through the swamp.  I don't know why, but we did.  Near the end of the first day in the swamp, the second day of travel, we encountered four humanoid figures sitting around a campfire about 60 feet away.  There was a number of places marked as 'water' and an equal number marked as 'land' so we wove our way through the maze of land to approach the humanoids inviting us into their campfire.  We were not meta-gaming or we would have simply all drawn our ranged weapons and started combat.  Before any of us took our first 'move' the DM had us set up on the battle map and roll initiative.  I had the top roll so I went first.  As a bard, I moved forward about half the distance and stopped.  The other players followed suit.  The humanoids stood up as we approached.

In the next 'round' I closed about half the distance so now I am about 20 feet from the humanoids.  Our rogue is hiding in the bushes with his bow ready.  The remainder of the party approach with our dwarven paladin getting within 10 feet of the humanoids along with our human spellcaster.  Our other paladin and war cleric are just a bit back and our ranger is standing next to me.  The DM calls for a perception check and then says, there is something not right about these humanoids.  They have reptilian eyes, kinda like snakes and they have forked tongues.  After another two or three seconds he announces, and you are attacked!

The enemy come from everywhere.  The four humanoids we were facing were Yuan-Ti Purebloods (CR=1, 200xp ea.).  From the west (my side) came four Yuan-Ti "fighters" (CR=3, 700xp ea.).  From our east came three Yuan-Ti "fighters" (CR=3, 700xp ea.) and from our north (the rear) came one Yuan-Ti "boss" (CR=7, 2,900xp).  So facing 12 'critters' with a total XP of 8,600 against our 7-level 5 PCs this is way into deadly encounter territory.

In the 'surprise' event the enemy attacked or rushed forward and I believe we only received one strike for 19pts of damage on our rogue as he shot an arrow and connected for about 12 damage.

The ensuing battle lasted 4 rounds with 2 'chase rounds' where we finished off two of the five fleeing critters.

The Yuan-Ti have three attacks each, or a grapple attack or a spell casting attack or a bite/weapon attack.  Our party had up to 10 melee attacks/ round or less with typically three spell attacks and seven melee attacks.  So since the Yuan-Ti had no obstacles to prevent them from swarming us we were facing over 20 attacks / round and returning something under ten/round.  As soon as I noticed the Yuan-Ti had multiple attacks in addition to having us out numbered 12-7, I thought we would be lucky to survive the encounter.  But nobody even dropped unconscious.  Admittedly our DM has given us some boons that I am a little shaky about, such as being able to retrieve a potion and consume it as 'interact with an object', I am shocked that we came away so good.

Our War Cleric was the hit of the show.  'He' was delivering damage with a spiritual weapon and Spirit Guardians for the turns 2-4, which because of the proximity of the enemies was very significant.  As the bard, I attracted a lot of attention because they wished to undo my debuff spell (Bane) so many of the enemy concentrated on me and somehow missed until late in the second round.  I used Cutting Words to keep our rogue in the fight by negating hits from the 'boss' Yuan-Ti and the rogue used his ability Uncanny Dodge to reduce the damage by half.  One of our paladins held down three Yuan-Ti 'fighters' and a second one held down the four Purebloods.  Our ranger wasn't lucky nor unlucky, but she helped block off some Yuan-Ti fighters from me.

Am I mistaken in thinking this should have been a near TPK?  Do you think our dice rolling must have been crazy good?  Or do you think the DM probably didn't use the best attack ideas in handling the Yuan-Ti?

In the aftermath, we ended the session before looting the bodies or anything.  But three Yuan-Ti escaped and I fear they will bring many reinforcements.  We killed the boss, five of seven 'fighters' and three of four Purebloods.

Am I mistaken in thinking this should have been a near TPK?  Do you think our dice rolling must have been crazy good?  Or do you think the DM probably didn't use the best attack ideas in handling the Yuan-Ti?

I can't say what the DM did, but it does not surprise me that you walked out of it alive and un-face-planted.

The easy/hard/deadly encounter algorithm is based on the assumption that the encounter is one of 5-7 others you had that day in between long rests (for a total of 6-8 per day). A single encounter all by itself can be significantly harder without crushing the party, because you will have all of your abilities. I just had my party of 4 level 4s plus an NPC beastmaster pet face, without resting and after 3 other battles, what should easily have been a deadly encounter (6 skeletons and a legendary creature equivalent to about a CR 7), and they won without anyone going down too. And I sure as heck did not pull any punches (just ask them!).

So, if that was the only encounter your party had that day, and you were able to blow all your abilities in that one fight, I am not surprised you won handily.

I actually feel bad now. I put my players though way harder stuff then that. In some cases encounters that should be hard went bad because of my bad dice rolls. But the players love that. Least mine do. 

I don't use cr anymore. I create monsters based on how much I know about my players characters and what they can do with them. Cr is a basic guess. It doesn't take into consideration of weapons, skills and team playing. 

7 level 5s. Should have no problem like you have seen. I'm sure the DM could have killed you but that's no fun. No DM.. no good DM will purposely tpk a party by just over powering the party. A good one will put up a good fight and wear out the players to the brink of death. 

But even then. It's all about the dice rolls. The dice control the game. Not the cr. 

I am a little confused about what you are asking here, it feels like you want us to rate your DM. You said this was about learning how to encounter build and balance fights, but you haven’t asked about that. Your only question is “should this of TPK’d the party”

which is hard to say because we didn’t run it, you have also made assumptions about what the yuan ti were as fighters and boss aren’t creatures.

if you want to know the mechanics of why the fight went the way it did to learn the only DM you can ask is your own

Well, I'm trying to dance around the issue so the conclusion can't be specifically "your DM didn't do it right."

So to clear up things, I'm not a meta-gamer, so I don't spend a great deal of time memorizing the MM or anything like that.  As I said, everything about the encounter trumpeted "This is going to be a fight, no matter what!"  But we chose to play it as if it could be a wilderness encounter with other travelers or possibly helpful natives.  But ignoring the other 'clues', when the DM says, "Well lets roll initiative", before anyone in the party has even drawn a weapon, what do you Think is going to happen in a five hour session that hasn't seen combat yet?  So I'm not a meta-gamer, but the Yuan-Ti were exactly what is described in the MM because I looked it up after the session was over to help me learn how to build encounters.  So the Yuan-Ti "fighter" I described is the Yuan-Ti Malison, I just couldn't remember the name when I typed out the original post, and likewise the Yuan-Ti "boss" is the Yuan-Ti Abomination.

My question is ... Should this fight have been a TPK for our party?  The basis for the question is two-fold.  First, according to the DM guide and the Encounter Builder tool here at D&D Beyond, this encounter's numeric value places it over 3x the threshold for a deadly encounter for my party.  Second, ignoring the numeric analysis, I understand that difficulty is often mis-calculated when one side has more attacks per round, and the side that has the advantage in attacks per round has a greater advantage than a normal numeric comparison would indicate.  This also supports the conclusion that the party was in grave danger.  But it didn't turn out that way.

Now BioWizard's observation helps me understand how the 'calculation method' could be wildly off in my circumstance.  The party had traveled two days without a combat encounter and upon drawing to the end of the second day we have the encounter.  Figuring we will have a long rest shortly after the encounter, I bet most of the players  felt, "Well, use 'em or lose 'em."  So we probably used more spell slots than we might have if we fought this encounter in the morning or middle of the day with more travel ahead.  My bard used 3 of 4-1st level spell slots and 2 of 3-second level spell slots.  He also used 3 of 4-bardic inspiration on Cutting Words, but I can get those back on a short rest.

Are there other 'tricks' that are not relying on the encounter builder formulas to come up with balanced encounters or do you just have to play thousands of hours and then adjust for the 'character' or 'nature' of the Party you DM for?  If that is the case, then I'll just have to make many many more observations and try to figure it out.  But while experience is the best teacher, there is nothing wrong with learning from someone else's experience.  So it is my hope that some of the seasoned DMs here might have some advice that would help me.  I'm not interested in fixing blame for anything here, and actually, I regret that you would even suggest that I would.

Your DM did nothing wrong. I don't think anyone here will tell you that your DM didn't do it right. You seem to be the one who thinks so, because the "book" says that this encounter should have been way over your party's head.

Except we have already told you how this was not the case, because the "book value" on encounters assumes a # of encounters per day, and does not work when that assumption is wildly violated. Just like when there is a "blue book value" on a car, say a 2010 Camry. The value is based on average condition of a car that is 10 years old. If you beat the thing to death over muddy roads and got into 17 accidents with it in 10 years, it's not going to have the book value. This doesn't make the book wrong -- it means your individual car violates the basic assumptions of the book's writers.

Similarly, you can't use the difficulty rating of an encounter that's supposed to be one of 8 in the day, but you only had one in the day. Of course you guys went nova on that encounter -- you could afford to. The DM probably expected you to, which is why it was full of so many bad guys. Your DM realized that the CR ratings don't apply quite the way they would if this were one of several successive encounters in the dungeon.

Also in terms of reading the MM -- I think that if you intend to read through the MM on what monsters do, even for a learning experience, talk it over with your DM. There may be many more Yuan-Ti malisons and abominations in your near future and you could have just spoiled it for yourself by reading up. I know you want to learn but, wait until that part of the adventure is over first. Let your DM know what you want to do -- learn about building encounters -- and ask when it will be "safe" for you to look up Yuan-Ti. Otherwise, you could ruin the whole adventure for yourself.

Are you sure it was an abomination? It’s a large monstrosity, so would be like 4 5ft squares, was it? You also described it as sneaking up on you.

the answer is as before that if you want to know more speak to your DM, they might not even be using the creatures you think they are.

what issue is it you are trying to dance around?