So, the players in my campaign have an NPC ally who helps them out on adventures, and has become an important part of the campaign, developing a relationship with each of the characters. however, we have reached a point in the story where it seems it would be instrumental to the developing plotlines (and would hopefully incur not too much grief from the players) to kill off this ally. but even though I think it would ultimately help the story to develop, I'm still not sure if it's acceptable for me to just twist the rules of the game so that the NPC dies no matter what happens (as this is the first campaign I've actually run). this just seems a bit unfair to the players even if they're not that attached to this ally, and perhaps it's just not within the rights of the DM to seal the fate of a side character in this way. if anyone has any thoughts or advice on this, it would be much appreciated.    : )

Is there a reason the NPC has to die? It could be easier for the story to have them leave, and then the players may not feel as maligned. If the story needs the character to die over simply departing, I think the best bet is for that death to be a large event. Don't force them into a battle where they fight foolishly and get themselves killed, instead, their death could be sacrificial, or perhaps their death is from a disease they contract or curse they fall under. There are a lot of ways to kill of an NPC that are intentional and dont feel unfair. All in all you want it to feel like their death is a progression of story, not an incidental thing that happened because you needed or wanted it to as DM.  If their time has come in the book that is your campaign, what did their death mean? 

D&D is storytelling. Ideally collaborative, but ultimately in the hands of the DM. It is perfectly fair for you to kill off an NPC to further the plot, add intrigue, or merely because you don't want to bother running it anymore. Just be conscious of how you do it. If you kill them off screen, then it's just a narrative point, but if you do it in battle with the PCs, they'll expect to be able to influence events.

Manage expectations.

It’s 100% within the DMs rights to do that. Not to a PC, but NPCs are little more than plot devices. And NPCs are, basically, your characters. You can do what you want with them.
I’d say you don’t want to do it in such a way that the PCs are there and aren’t permitted to stop it. Like you wouldn’t say “you go to heal him and cure wounds isn’t working for some reason” that’s no good. But the enemies focusing fire on the NPC , and keep hitting it so there no chance for a death save/revival is fair game; they’re on a mission to kill this person. 
If you’re worried about it being unfair, you can always do it off camera. The PCs go to the inn, NPC goes for a walk and doesn’t come back. That kind of thing. 

By level 5 or so, most parties tend to have numerous ways to prevent or reverse death.  Remember to take these into account when planning to kill an NPC: killing them offscreen is generally easier.

And of course, the other issue is that if the party has really gotten attached to this character, they might get obsessed with chasing down the killers instead of following the plot (assuming that's not the plot).  So be prepared if that happens.

Exactly. You are free to kill of any NPC at any time.  But if the players love them, they will stop at nothing to prevent it.  If they are there when a fight happens, they will all die before giving up.  They will think they have a chance to change the course of events.  And if you deprive them of that chance, only then will they feel truly cheated.  If they come across the NPC already dead, they will make it their mission to find someone who can resurrect them.  So you have to be very careful.  

If death is the only way to move the plot, think hard about how it will happen.  If you can get the same results by letting the players save the NPC from near death, that's better.  If your plot isn't centered around trying to revive the NPC and avenge them, then it's best to avoid it. 

The only thing a player cares about more than their own character, without fail, is a beloved NPC.

You can do that, but it’s only fair to give the players a chance to save them.

I am fairly new to DND beyond but i've run a chunk of campaigns in 3.5 and now a first time 5e DM.  I think that my advice would be to think about 3 things, which hopefully will help you think of a way to solve the issue you are having

1.  Is the plot that you want to move forward with a plot that the players want?  and as a followup is it a campaign that you aren't having fun with if they are having fun with the current way of progression?
2.  Is death the only option?  Depending on the backstory of the NPC in question, they might need to return home to be with a family member who has suffered a medical issue or an injury of some sort.  Or maybe they are at the point in there life where they need to make a pilgrimage in isolation to show their faith to their diety/god.  There are a lot of ways where the NPC can leave for progression, and then come back later on or be featured again in a side story, or perhaps even rescued from a situation depending on what exactly happened.  If they like training people up, maybe they're ready to go help the next group now that they've taught the current group all they can.
3.  What are the death mechanics with regards to characters?  Some campaigns make it possible for resurrection, so if you kill off an NPC would the party simply revive the NPC?

With my 3.5 groups i was more than happy to hold side bars for feedback going both ways out of character so that i knew the players issues and they knew mine.  If it fits with your group dynamic, it might be helpful to consider this option.  I haven't yet gotten there with the 5e group as we just had our first session, but i anticipate that we'll probably also use these sessions on a monthly basis as it's easier to keep the group alive and kicking if everyone feels their issues are at least heard. 

Good luck with solving the NPC issue you are having, it would be cool to hear how you resolved it and how the group handled it all :)

There's nothing inherently wrong killing off npc's.  (spoiler alert)

Okay, you've been warned.  Recently happened in the much watched stream, Critical Role.  The party has lost their patron and they were nowhere around when Esteras was taken down.

Absolutely! Besides, the longer an NPC travels with a party, the more the party will come to rely on the NPC for information or doing chores or whatever. The NPC becomes simply a stand-in for the DM, and the players will use the NPC to try to metagame too much information from the DM, and will eventually stop doing the hard work of exploring, and critical thinking, and problem solving opting instead to simply rely on the NPC.

The DM giveth and the DM taketh away. The fact that the players are emotionally invested in the NPC is all the more reason to kill off the NPC, because that will propel the story forward and motivate the players even more! 

Go for it!

It's fair for a DM to kill an NPC, especially if it advance the campaign's story. The trickiest part is to make it look natural, and not forced in any way. Its even more dramatic if the party had a chance to save it but failed to do so. If it's unavoidable (''rock fall and die'') it's much less remarkable. But killing such a character, especially at higher level, is harder than it seem, especially if it can be dying, as opposed to die at 0 HP. Some DM often have to resort to fudging to make it happen.

Given the presence of spells like Revivify, Resurrection and True Resurrection , killing an NPC is a challenge if they've been with you for a while. 

Personally, I only allow those spells to temporarily bring them back to life. If the players want to make it permanent, then I'll set up a quest to get special materials. That way it costs them something more meaningful than just gold.

Otherwise, unless it's low level...give the NPC a reason to leave. Or have them die of old age, Yoda style. The party is likely to be able to stop the death otherwise.

You could have the NPC get killed, let the party raise them, and have them decide that they would like to retire on the grounds that after surviving death once they don't want to risk it again.

But those spells all require that the spirit be "free and willing" to return to the body. The players may not know that NPC as well as they think. They certainly won't know all that NPC's deep dark secrets. Maybe the NPC sold their soul to a devil some years back, so their spirit is not "free" to return. Or maybe the NPC just got reunited with their parents in the afterlife, so their spirit is no longer "willing" to return.

Sometimes a dead body is just a dead body. 

Technically correct, but players often tend to feel (not without justification) that it's being used to arbitrarily block their attempt to use resources they have access to in order to solve a problem and railroad in the plot on a path they can't actually affect.

Whether player or non-player, a character's violent death can certainly be a motivator for adventurers, maybe even a campaign game changer.  I'm fine with NPCs getting killed off, I'm also fine with PCs dying.  My only caveat would be how the NPC usually operates in game play, you need the death scenario to be consistent with that.  By that I mean is the NPC sorta the party's Gandalf. That is, if it's the sort of NPC who sorta shows up with an occasionally deus ex machina and otherwise serves as Chief Exposition Renderer, that character is fine for off camera execution, or having the characters arrive too late to save them, etc.  On the other hand if this NPC was more a mechanical asset or tool of the party (hireling, retainer, sidekick) and generally fought alongside the party or served some sort of indispensable support role in the character's encounters, there you'd want to actually play out a fighting chance scenario (maybe allow some sort of grievous injury if the deathtrap doesn't work).  

Generally players are fine with "larger events" happening around them, and the death of a patron would be such a larger event.  However, players don't like having their stuff taken from them, especially if it's a thing/person the players have a role playing bond with, so the level of discretion largely depends on the specific nature of the NPC whether it's cool to just "He's Dead Jim" the NPC or play the deathtrap.

The players get their PCs. They show up on game day, play their characters for a few hours, and then go home and don’t have to think about the game anymore for another week. In return for this minimal investment, they get to be the heroes, the story is all about them, they are the stars of the show. The DM works for several hours in between sessions every week, and in return for that investment we get to be all the villains and monsters the PCs beat up, as well as their narrators, tour guides and benefactors. What do we get out of it? We get the rest of the world, including all NPCs. If a DM wants to kill off an NPC, then that’s their prerogative.

—fin—

Its sort of like Gandolf.

Sometimes they die, sometimes they just go away for a long time and sometimes they comeback.

Never let a party get around to relying on an NPC to do all the dirty work.

Quote from WoodsGirl0402 >>

You can do that, but it’s only fair to give the players a chance to save them.

 Agree, 100%.
If he dies let it be for a good narrative reason.

Opposite example: In a campaign in which I’m a player (by dafault I’m the DM), there was the character of a former player that stopped playing the campaign that (in game) betrayed the party.

The master had in mind cool narrative things for the character, but my PC (that hated the former PC-NPC the most) tried to kill him and did so both in a spectacularly cool way (spider climb to run vertically on a very big birch tree, then jump spell and jumping over with a smith hammer from 20m hight on the back of the adversary. Spoiler my PC is still with broken legs in search of a specialized cleric, able to fix disintegrated bones) and with very good dice rolls  (a series of 19 and 20 natural + bonuses), that the master could not deny me that. Despite he had other plans.

So, opposite situation: don’t deny your PC the option to save him. Make it hard and unlikely, but don’t bring away free will from them.