I'm setting up a caravan convoy. the lead is the merchant's families carriage followed by three wagons of goods, then a supply wagon caring tents, food, extra clothes and cooking kits... ect.  The merchant's group includes their own wagon driver's and some have families. in total 13 non-combat NPCs. the owner is wealthy but is willing to pay no more then 3gp/day. 

The mercenaries will have their own wagon and mounts. 

how many mercenaries should the owner hire to protect a large group like this? 

I'm imagining around 5-10 guards or thugs, probably not all with individual wagons because their mounts need to be more mobile to do their jobs. Only the wealthiest of merchants can afford more powerful guards in large numbers because of how uncommon they are, and because they are unlikely to settle for the 3gp offer. Simple guards or thugs, however, could easily be considered "skilled hirelings" that charge about 2gp per day (Player's Handbook, page 159).

These guards are plenty to keep honest people honest and even scare away thieves and bandits who prefer to avoid resistance. They might even cause a small force of goblins or kobolds to abandon the mission and search for an easier target. Should such a type of force attack, they might be able to hold them off. In the end, however, there is not an easy, feasible way for anyone other than the absolute wealthiest of merchants, those who can afford guards with the fighting power of knights, to protect themselves from an adventuring party more powerful than level 2 or so. If that's what you're trying to accomplish with these guards, you unfortunately won't be able to.

 the reason why its so large is that the merchant is fleeing a war zone before the fighting starts. the journey will take 5 day travel.

1st ideal: PC are the guards. I'm planning on sending a large force to attack the caravan. 

2nd ideal: PC have to attack the caravan. if this one, i need to know many guards would be the appropriate amount to defend without being OP or too weak. 

For your first scenario, you shouldn't add guards other than the PC's. The merchant already hired them as elite guards, and he doesn't need more. In addition, I know from experience that giving the party NPC helpers in combat, in particular a large number of weak ones, is a DMing blunder. Watching mobs of NPC's kill each other can bore even the most enthusiastic of players.

For your second one, I'll return to my earlier advice:

Quote from Panda_wat >>

there is not an easy, feasible way for anyone other than the absolute wealthiest of merchants, those who can afford guards with the fighting power of knights, to protect themselves from an adventuring party more powerful than level 2 or so. If that's what you're trying to accomplish with these guards, you unfortunately won't be able to.

If the party level is 1 or 2, you can reference the encounter balance rules on page 82 of the DMG.

thank you