One of my potential encounters for my party is them traveling to a town said to be killing any who visited it. In actuality, the town keeps on getting attacked and now treats everyone with hostility. I'm trying to figure out how many magical/fighting people there would be to protect the town, which has around 4,000 people.

If the town has formed a militia, it's probably about 200. You could make a homebrew "Militia" or lower the number for convenience and have 10 different town warriors. If the party is fighting all of them, definitely do not have 200 separate people. Your Militia homebrew could be like a "swarm" of people.

That's entirely up to you and your world.

If they are getting attacked by small groups of bandits, then a defense force of a few dozen guards should be plenty. However, if they are getting attacked by armies, then they may need a few hundred people ready to respond.

If the are a well funded town, then a smaller fighting force can be more effective with better training and equipment. However, if the town is poor, it might be everyone and their grandmothers armed with axes and pitchforks. Maybe the town has one very strong champion that handles all of the threats. Or maybe the town has a shrine that can empower a handful of villagers at a time.

If the town is fortified, or can take advantage of terrain, that also comes into play.

If you want realism, then there are a lot of factors to consider.

If you just want an encounter, then don't overthink it.

I think the biggest question you need to answer first is how you want this group of NPCs to interact with the party. 

Do you just want people the party can beat up once?

Do you want them to capture the party?

Do you want the party to fight along side them in the future? 

If so, do you imagine a large battle?

Or a few brave warriors facing an army like The Seven Samurai?

Think about how you want the story to progress after first contact. That will help inform you of what kind of defenses the town should have and what they look like. In DnD especially, one person can change the course of a battle.  So you could have hundreds of militia or just a few heroes with some class levels.  What really matters is what happens next.  How you want the players to be involved.  And how many individuals you want to run. 

Historically, there were laws and customs in place to ensure that everyone was capable of defence. Heck a few countries in the real world require civil defence service from every citizen.

In England for example the custom was that every young boy was taught from an early age how to use a bow and eventually a longbow. So in a reasonably small town (around 1000 people) you'd have at least 500 people who could at least utilise some form of weaponry. How effective they are is another matter.  Such a town likely would be left alone most of time with guards only send out during tax collection or other official business.

So, if you want to be realistic you're looking at anywhere from 100-500 fighting peasants. Maybe half a dozen guards.

If your world is less realistic it can be whatever you want.

Realistically, the number of professional military that can be maintained is very low (1% is high for a society that isn't actively at war), so your town probably doesn't have more than 50-100 active combatants (most of which will be things like guards). It could have several times that as reserves (people who have been trained but don't practice actively) and probably 1000+ who could be raised as a mob of commoners.

However, D&D is not generally big on realistic society, nor are your players likely to care, so you can certainly increase the numbers.

Yeah, probably somewhere in the neighborhood of 3 dozen guards lead by 3 nobles, up to 400 (10%) militia (commoners with spears), a priest with up to 3 acolytes, a wizard (probably an illusionist wizard, but maybe a conjurer wizard, transmuter wizard or enchanter wizard) and an apprentice wizard.

Well, if the town has been getting attacked so often that they felt the need to close their borders, then they have probably developed a stronger than average defense force. They probably have more citizens capable of defending the town than the average town would. And in D&D terms, a town of 4,000 is definitely not a "small" town! It's a quite large town and very nearly a small city. Normally a population of that size would be spread out over a dozen or so square miles. But with defense being a prime concern, they have probably concentrated much of the population into a smaller area, behind multiple rows of erected defenses. 

For starters, let's lay out some simple numbers. I would say a large town under constant threat of attack would have 5% of its population under arms. These are trained full-time soldiers living in barracks, guarding the walls, and conducting regular patrols. That's 200 people.

Another 5% would be the Ready Reserve. These people have regular day jobs, many of which may be in support of the soldiery - blacksmiths, millers, farriers, carpenters, etc. But they have received basic combat training, and they drill maybe once a month on a Sunday afternoon. They can be called up on short notice to bolster the guard when needed. They're weekend warriors. So that's another 200 people.

Then there's another 10% who are on the rolls as the Inactive Reserve. These are old retired guards, strong teenagers, simpleton farmhands, and maybe even a few indentured servants. These folks don't train or take part in drills, but if the crap hits the fan these folks can potentially pick up a weapon and swing it. 

So this might sound like a lot of people. But remember, it's usually just the 200 full-time soldiers. And those 200 are never all on duty at once. There would be three or maybe four shifts per day, so there's usually only 50 to 70 guards on duty at any one time. And they would be spread out - guards on the main gate, guards on the rear gate, guards in the streets, guards protecting important people, guards on the walls, guards patrolling outside the walls, guards patrolling the nearby fields, guards protecting the wells. 

Of the 200 full-time soldiers, 150 would be CR 1/8 Guards, 20 would be CR 1/2 Scouts, 10 are CR 3 Knights, 5 would be CR 1/4 Acolytes, 4 are CR 2 Priests, 1 is a level 5 Cleric, 4 would be CR 1/4 Apprentice Wizards, 1 is a level 6 Wizard, 4 are level 5 Fighters, and 1 is a level 9 Fighter. 

Again - this is not a roster that I would typically assign to a normal town. This is a roster that would specifically be for a large town that has been under nearly constant attacks or threats of attacks for a prolonged time.