So in my current campaign my party had a suspicion that a person was bad and causing a lot of there problems. (He actually was causing all their problems.) So the cleric casted detect good and evil when the suspect BBEG was nearby. I freaked out mentally, but wasn't going to lie. I revealed his intentions where evil.

So the questions are, is there anyway that the BBEG could sense the spell was casted on him? (he is level 12, level 7 wizard, level 5 blood hunter, not sure if that helps.)

Is there anyway he could've resisted showing he was evil (I can't take it back, just wondering for future reference.) 

DG&E doesn't reveal a creature's alignment, but arcanist's magic aura should work.

yeah, the answer is "know what spells do"

Detect Evil and Good simply does the following:

For the duration, you know if there is an aberration, celestial, elemental, fey, fiend, or undead within 30 feet of you, as well as where the creature is located. Similarly, you know if there is a place or object within 30 feet of you that has been magically consecrated or desecrated.

I had the cleric read me the spell details, and I pointed that out. I said that would mean it would point out a good or evil aberration, celestial, elemental, fey, fiend, or undead. But he is neither. He is just a plain old human. But the party disagreed so I went with it not wanting to cause an argument.

Are you not the DM? Because the PCs DON'T get to decide who things work.

I am. Unfortunately sometimes my party disagrees a lot to my rulings. So I try and find the mid ground. With this one however I was not understanding the spell well enough, and I did not want to start a fight (Not that we have had any. I am just trying to keep it with 0 fights.) So I let them use their interpretation.

Well, you already homebrewed the spell to have it detect alignment instead of just creature types, so just homebrew it further to say that, when it's used that way, the target is aware that they're being scanned.

Quote from Cindrellic >>

So in my current campaign my party had a suspicion that a person was bad and causing a lot of there problems. (He actually was causing all their problems.) So the cleric casted detect good and evil when the suspect BBEG was nearby. I freaked out mentally, but wasn't going to lie. I revealed his intentions where evil.

So the questions are, is there anyway that the BBEG could sense the spell was casted on him? (he is level 12, level 7 wizard, level 5 blood hunter, not sure if that helps.)

Is there anyway he could've resisted showing he was evil (I can't take it back, just wondering for future reference.) 

Actual text of the spell.

How to best get out of this depends on how you actually had the spell work.  There are myriad solutions, but which one is the best writing decision depends a lot on how you handled it.  The more detailed you were, the harder this will be.

Here are some example solutions:

• Define evil as being relative to the cleric's deity - work out a religion for said deity that includes a laundry list of sins such that it's impossible to make rigorous conclusions about what evil he's guilty of.  Maybe he showed up as evil because he wears wool socks every second Wednesday.
• Since you had the spell reveal his "intentions", give him a magic item that reveals fake thoughts when telepathy spells are used on him, like a Ring of Mind Shielding only with extra magic to provide false positives to the spell.  If it somehow wasn't telepathy, same logic, but the base item is an Amulet of Proof Against Detection and Location.  You can also have the item tell him when it's used on him.
• Make such a spell deeply, incredibly, maximum illegal, due to the fact that if such a spell is allowed to exist, it leads directly to genocide, and the powers that be are tired of going down that road.  Have the Inquisition show up and ask pointed questions about casting the spell without a license.

Here is exactly my storyline so that you have full understanding. Buckle up, this is a lot:

This is set in the town of Secomber of Faerun. The party came to Secomber right as they were holding their annual election. Secomber is politics crazy and everyone loves it. The party watched the debate. It was Treskar Selarn, a human ranger who has been the mayor for 12 years (he is the BBEG), and Bardez, a badger-folk, and a politician with good ideas. The party enjoyed seeing it after watching, the winner was announced saying that it was Treskar Selarn, who won by a landslide. Afterwards the party went to sleep. They took a small side quest which has nothing to do with the main plot. After the side quest, they came back to Secomber, and watched yet another political crazy thing in which the newly elected Lord would show off his talents. Treskar is good at sword fighting, darts, and charming. Remember the charming part.

The party had had multiple interactions with the villagers of Secomber in which they seemed "glitchy" and refused to say anything bad about Treskar. After much deliberation they decided to try and get front your seats to the "Lord's Talent Show" and try and cast some spells to gain some knowledge before they decided that Treskar was the bad guy. So the party went and got front row seats. Then the cleric (her patron is The Traveler of Critical Role fame) had the idea of casting detect good and evil. (I freaked cause I never suspecting that they would do that, and I also had limited knowledge of the spell, so I thought that meant it would literally "Detect good and evil.") I then had the cleric read the spell description, and I pointed out that it is determining evil creatures not alignments. But the whole party disagreed. So I decided to not play bad cop. 

If you remember the charming part, Treskar is the bad guy. He made a deal with the ultimate BBEG in which he could charm most of the town of Secomber to follow his will. Using the help of his patron he has held control for the 12 years of his service. Treskar is power hunger and doesn't want to give it up easily.

This is exactly what I said. "As you cast the spell, there are multiple creatures who light up as evil. It does seem as well seem that Treskar has evil intentions."

I think what my plan is, (I am going to discuss it with the party.) I will create a homebrew spell called "Detect Alignment," and I will give it to the cleric. The spell is instantaneous, however the person you cast it on must succeed a WIS 15, if they succeed then they will know that the spell was cast on them.

I think that what you and the party likely were thinking of when they cast Detect Evil and Good was a Paladin's "Divine Sense" ability, which more explicitly says that "The presence of strong evil registers on your senses like a noxious odor, and powerful good rings like heavenly music in your ears." I mean, that also doesn't actually mechanically detect alignment or evil intent, it just detects creature types... but it at least thematically seems like it could detect exceptionally evil people, behaviors, or occurences.. I think that the spell that the party was trying to get the benefit of, was probably Detect Thoughts, which also doesn't just tell you "he's evil," but does at least let you read their surface thoughts and maybe something of their attitude towards you.

But it's not a big deal that you let them sense that he had evil intentions or even was an "evil" person. Just roll with it, use it against the party the next time they try to pretend to not be heroes (or heck, use it against them when they ARE trying to convince people they're heroes, if any of them have non-good alignments!). But beyond that, your original question was basically just, "do creatures know when a spell is cast on them?" and the answer to that is, "depends on the verbal, somatic, and material components." From the PHB, Chapter 10:

Unless a spell has a perceptible effect, a creature might not know it was targeted by a spell at all. An effect like crackling lightning is obvious, but a more subtle effect, such as an attempt to read a creature's thoughts, typically goes unnoticed, unless a spell says otherwise.

Detect Evil and Good has verbal and somatic components: that means, if the target was looking at or within hearing range of your spellcaster, they probably noticed them casting a spell while staring at them! But otherwise, the spell doesn't normally describe that a creature knows its been detected, so I don't imagine that they felt anyone poking around in their brain... unless you want them to, because again, you're already doing things with the spell that it doesn't normally do! 

The party urged you to make a houserule, and you accommodated. If you want to balance that out by imposing other drawbacks on using the spell that way, go wild, you're the boss. 

Quote from Cindrellic >>

I had the cleric read me the spell details, and I pointed that out. I said that would mean it would point out a good or evil aberration, celestial, elemental, fey, fiend, or undead. But he is neither. He is just a plain old human. But the party disagreed so I went with it not wanting to cause an argument.

 In previous editions of the game, the spell worked differently.

So often, players don't actually read the 5E version of spells properly in order to understand how they are different from earlier editions.

Problems of using the spell wrong and how to avoid that in the future aside...

Alignment is very subjective, but good vs evil can loosely be broken down into how you will treat others (it doesn't have to mean literally everybody) compared to yourself and your goal.

• Good will make some personal sacrifices or put aside their goals to help others.
• Neutral will be less inclined to make sacrifices to help others, nor will they harm others for their benefit.
• Evil puts their goals first to the detriment of others, and may even seek to harm others as a means of advancing.

If you told the party the guy was evil, that doesn't mean he was the BBEG. Plenty of perfectly functioning members of society could be evil (especially in positions of power).