I was thinking about players shooting projectiles at an enemy while an ally is standing in between. If the attack misses how would I determine if they would hit their ally? I want the group to pay attention to their positions so not to hit each other making it more realistic in my mind.

I was thinking about a critical one doing this yet my experience people do damage to themselves when rolling a natural one.

There is an optional rule in the DMG for hitting cover, yes. If you miss a target but would have hit had they not had cover, the attack is instead delivered against the cover, and if the roll beats the cover’s AC, the cover takes the damage.

For example, I’ve got a bow and my melee allies are positioned such that my fighter friend is providing half cover the enemy. Both my friend and the enemy have ACs of 19, but the enemy gets +2 from cover, so it’s 21. I make my attack and roll a 20. That would be a hit without the cover, and it also beats the covering fighter’s 19AC, so I miss my enemy and hit my friend instead.

This is an optional rule, of course, not a core mechanic.

Definitely not a bad idea to remind people to pay attention to their surroundings. A couple weeks ago, a character cast Call Lightning on two enemies not realizing the were engaged by my character and another player's character. The DM called no backsies so, we all rolled DEX saves. I believe both enemies took half damage while the two PCs took full damage. We made the player use the rest of his spell slots to heal and cast support spells with his character, LOL.

In general I do not rule that misses will hit any living creatures in between the attacker and the target, except in extraordinary circumstances.

In a normal tactical situation, using a 5' square grid (or hex grid) battle map, you have 1 character per square. But a character does not take up 25 square feet of space (or, if you are doing volume, 125 cubic feet). A 155 lb human would take up only about 2.5 cubic feet, and probably around 2 square feet (since 1.4 x 1.4 x 1.4 = about 2.5), which means that 98% of the volume, or 92% of the area in a square occupied by a human person on a 5' grid would be empty air. In addition to this, although you only "go once" on a 6 second round for about a second (or less), when it's not your turn, a character is not standing there like it's the Statue of Liberty play in American football. He or she is ducking, running, parrying blows, looking around a corner, what have you. Basically, moving throughout the square (but not leaving it) and doing defensive type actions, or turning in place to get a better view of the battle or what have you. The combat is resolved by ordered turns but the characters are not just standing in one spot waiting for their turn to go, IC.

Given how little area/volume a character takes up of an actual 5x5x5 foot area on a map, and how much he or she is moving around, the idea that a straight shot (say from a crossbow bolt) that crosses that character's square on its flight to a target would "accidentally" hit that character is just too unlikely for me to ask for a resolution of such a thing on a miss. This doesn't even account for the fact that arrows do not travel directly in a straight line from A to B vertically, even if they do travel that way horizontally. A character launching an arrow from "ear height" at the target's chest would have to aim UP so that as the arrow falls down with gravity, it comes down to the targeted level. (There is a reason why in old school D&D, ranges were measured in feet for weapons like bows indoors, but in yards outdoors -- because you could not aim "up" past a ceiling and so your range was limited by how far the arrow would fall due to gravity as it travels forward, from the 10' max height of the ceiling.) If you are 120 feet from the target and fire at it, and Joe is halfway in between, you're going to be arcing the arrow up anyway, so even if Joe happens to be horizontally directly between you and the target, the shot will probably go over his head in a parabolic arc anyway. 

All of this adds up to -- it is very unlikely that you are going to hit Joe when you fire your arrow at the goblin who is (on the map) directly opposite him. Joe's square on the battle map may be directly between you and the gobbo, but his actual person doesn't take up the whole square, and you're probably firing pretty high anyway (at least over his shoulder), which means most of his body is not in the line anyway, and what little is there, is very unlikely to be in a direct path between you and the gobbo, especially on a miss (since the shot went wide).

Now, in exceptional circumstances, like if someone is standing very close and you are doing something like thrusting a javelin through a narrow doorway that is only a couple feet wide, and your burly barbarian friend is actually standing in the doorway, then yeah, a miss might hit him. But under normal conditions... I do not bother to roll to see if a missed shot hits anyone else (ally, or enemy). It just "goes wide."