Does anyone have guidance on how much GP (or other rewards) NPCs should realistically offer as a bounty or fee for a quest? I know that this would be scaled for tier and complexity of the mission, but I'm at a loss as to what a fair rate would be. This is my first time DMing and one of the areas where I'm lacking in confidence.

In the old days, they used to say that 1 GP was equal to 1 XP. So if the party got 1,000 GP of treasure, they earned 1,000 XP for that. So, if you wanted to figure out what a reasonable treasure was back then, you created a value to the treasure based on how much XP you wanted to award. 

For example, let us imagine a party of 5 characters, all 2nd level. To get to 3rd level, each will need to gain 600 XP (from 300 to 900). This is a total of 600 x 5 = 3,000 xp they would need to level up all the way from 2 to 3. Now, monsters are worth XP too, but the idea would be, you would give them about 3,000 gp worth of treasure (or maybe, 1,500 worth of monsters and 1,500 worth of treasure).

So now if you want to figure out a good reward for a 2nd level party, you would have a ballpark. "I will pay you 1,500 gold pieces to do X." This would immediately have (back in the day) told them that it's half a level worth of XP, which is a pretty good reward. Then you can just scale it to their level. For instance, it takes 30,000 XP to go from level 16-17. Times 5 players would be 150,000 XP. So a 75,000 gp reward would be to a level 16 party what a 1,500 gp reward would be to a level 2 party.

That's incredibly helpful, thank you!

The 1gp=1xp was what was actually awarded to the characters in 1st edition. If they found a 1000gp of treasure it would be worth 1000xp.

However,

1) XP in first edition scaled completely differently. Second level for a wizard was 2500xp, 3rd level was 5000, 4th was 10,000, 10th level was 250,000 xp. Each class had different XP rates of advancement so if you apply this concept in 5e, it may not work quite as well.

2) How much gold you want to give the characters depends on what they have to spend it on. There actually isn't that much. Trade in magic items is "discouraged" in 5e. Magic items are rare, difficult to obtain and are rarely if ever sold. So from a mundane point of view - gold costs consist of lifestyle expenses (a few gold a day even for the best lifestyle) and a few expensive mundane items like plate armor for 1500gp. If you give the characters about 1gp for every xp you award, they will have enough gold to buy plate armor and whatever else they want by level 3. Personally, I think that is a bit much but it depends on the campaign you are running and whether you want your characters to be able to purchase houses, estates, ships, businesses, hire mercenary companies or armies.

3) Adventurers League imposes relatively low gold limits that leave classes like clerics and wizards at a disadvantage since wizards don't have enough funds to scribe many spells and both classes can have issues with purchasing expensive spell components. AL uses 80gp/level in tier 1 and 240gp/level in tier2. If I was looking for a rule of thumb, I would probably double these amounts to about 200gp/level in tier1 and 500/level in tier2 average depending one what treasure the party finds. If you look at 5e published modules, the treasure rewards are much lower compared to modules that started in earlier editions. As an example, White Plume Mountain in Tales from the Yawning Portal is a 5e conversion from an earlier edition. It is a tier2 adventure but the treasure rewards are about 10,000gp/person and there are a lot of magic items. If you compare this to the more recent adventures created for 5e like Descent into Avernus, Tomb of Annihilation and others, the gp and magic items rewarded are far less.

TL;DR

Anyway, the bottom line is that how much the character's need depends on what the DM puts in the game for them to spend it on. Other versions of D&D rewarded far more treasure than 5e. If I wanted a rule of thumb it would be something like 200gp/character in tier1 and 500gp/character in tier2 on average.

David makes a good point. How much reward you give depends largely on your world. If there is a lot to buy, or spend money on, you should give them enough to do so as appropriate. If there is not enough to buy, then you give less. As a simple example, if you're going to make them keep track of rations every day, they're going to need 5 sp/PC just to eat. If you're a DM who thinks it's not worth tracking that stuff, just assume you have it, then that's 5 sp less per day they need to spend. If you're going to follow the "5e tradition" that magic items should be rare, hard to buy and sell, etc., then the party will have a lot less gold than if you let them sell magic items for large sums of money.

So, my party has killed some pretty scarry dudes already including a hydra and a spectator.  We got big piles of gold.  Are we supposed to receive the XP for killing the monster and XP for the gold too?

Bio, I remember the days of 1gp = 1xp, but my puny level 1 wizard had to get 2500XP to level up to second and have 2d4+Con bonus HP.  That was a long time ago.

Quote from MusicScout >>

So, my party has killed some pretty scarry dudes already including a hydra and a spectator.  We got big piles of gold.  Are we supposed to receive the XP for killing the monster and XP for the gold too?

No no. That was AD&D and early editions. XP is done as per the DMG now, which is based on the CR of what you fought.

My rule of thumb is this. 1cp=$1(US), 1gp=$100. I use 1980’sish pricing so a PC could buy lunch for 3cp, 5cp is lunch and a couple beers. A room at a cheap in (Motel 6 style) runs about 2-3sp. A peasant farmer could subsist off of as little as 10gp/month. Magic Items in my world are pricy. A +1 anything will add as much as 500-800gp above the price of the mundane item depending on accessibility (big cities have higher availability, etc.) That gives me an idea on cost of living for both civilians and adventurers.

After that, there’s really only one question  I have to find an answer for: How much are these people willing to risk their lives for?

Looking at the rewards for the essentials kit, the rewards are 25-50gp for initial quests, then 100gp to 250gp once the characters are 3rd level.

In Lost Mine of Phandelver, the rewards are 50,100 and up to 500.

I'd recommend for the first few levels to reward 50 to 100gp. Once the characters get to 3rd you should have a feel if that's too much or too little for your game. It really depends on how much the characters are into spending. My players are mostly just looking for magic items.

You could alternatively use the treasure hoard tables on p. 137 and following in the DMG. For example, a CR 0-4 treasure hoard has appx 2100 cp, 1050 sp, and 70 gp. This converts to 21+105+70=196 gp... say 200. The hoard can also include gems and magic items... you could use a median roll on the table (50 on d100), which adds 350 gp worth of gems and about 3 magic items from table A. Again using a roll of 50, that'd be 3 potions of healing, another 150 gp. The total gp value of a CR 0-4 hoard would thus be something on the order of... 200+350+150=700 gp. This is what you'd expect to find in the hoard of a CR 3 or so boss monster + minions.

So... instead of putting it into the hoard, you could have a local noble offer them 700 gp for bringing him the head of the monster, and give the monster no treasure. Or, what I'd probably do... is offer 350 gp worth of gems for the reward, and let the monster have 3 weak magic items and some gold and silver on him... just because it wouldn't make sense for a CR 3 creature + minions not to have anything at all on them.

when catapults are outlawed, only outlaws will have catapults? I like :)

To add to what others have said, you also need to keep in mind the cost of certain spells to cast or learn. For example, revifify costs 300 gold worth of crystals. My level 7 Cleric has had access to the revivify spell for a few levels now but there is no point me preparing it because I've yet to recieve any gold whatsoever. 

Quote from bluepumpkinpie >>

It is my first time as DM, and I am making a homebrew campaign. I am not giving XP, just leveling my players up when I feel like they need it, & also they are starting at level 5. I want to reward them with something big at the end of the quest, because I'm throwing a bunch of white dragons and goblins at them, but I don't know what a good reward would be. I also want to maybe add magical items. What would you recommend?

 Be careful starting at 5th level, there is a big jump character ability from level 6 onwards and many DMs finish campaigns by level 8 because balancing becomes harder, also if your players are new there will be a huge learning curve for them. 

Also if you have just 4-6 characters 1 white dragon could potentially be a TPK so sending a bunch at them with goblins as well I would be very careful about how dangerous your making things. If the dragons will be one at a time after the first one it can get a bit samey dealing with the same enemy type over and over. 

Thanks!