My friends recently told me about his campaign he started and offered me a spot in it. I was happy to join it. He explained that he had some small homebrew changes and the main one was about spells. He decided to use a spell point system, but every spell has a different cost including cantrips. He bases how much points it cost on the size or radius of it, which leads to some caltrips costing more points than some actual spells. He also decided that some spells have a limit for their damage, as in if you roll really good then your damage will be reduced. I have been debating whether to play or not due to this and I dont know if this is fair or maybe a different way that in just unfamiliar to. So I am kinda just asking if im just being stupid or something.

You could always play as a martial.

No, it's not normal.

A spell point system may not be a terrible idea, but a "different cost for each spell" based on several criteria sounds overly complicated and is a red flag. Putting a point cost on cantrips is another red flag, and effectively removes any reason to use offensive cantrips. Capping the damage is a third red flag. This DM seems to dislike spellcasters, so I would avoid playing one in that system, and would discourage any fellow player from playing one as well.

I will start by saying there isn't really a "normal" in D&D. Every table has it's quirks and will feel and play differently.

However

No D&D is better than bad D&D. The only way you know whether it is bad D&D or not is based on if you are enjoying yourself. If you are not having fun then move on. If you are having fun then good! If you feel like maybe you could be having more fun under those rules with a different character then let the DM know.

Honestly, the one big thing that makes me want to run for the hills is when the word "homebrew" is mentioned in relation to mechanics. It's rare that people understand what they're doing when messing with mechanics and usually end up making things worse if they've gone beyond "a few tweaks" and entered "homebrew" territory.

This is one of those cases.

Spell points are fine. Spell points based on radius? Cantrips, which are meant to be free so they can act as a back up costing more than levelled spells which are meant to be a carefully managed resource?

That's not something I'd be happy with, by the sounds of it. That it's based on radius screams a fundamental misunderstanding of how the mechanics work and how spells need to be regulated.

Of course, I've only heard one side of the story and I don't know the actual mechanics involved, but it's definitely ringing alarm bells for me.

There is a spell points system that is a variant rule. However, it is based only on spell level and cantrips are still free. 

That homebrew sounds janky to me but that doesn't mean it might not be fun. It does sound overly complicated, in a world where running a full caster is already very complicated. Up to you if complexity like that is something you enjoy.

It sounds like a nerf for casters, which arguably they could use.  Not simple or elegant though. 

I made my character as a cleric since original warlock was my idea but the cantrip costing more than leveled spells scared me, so i went to cleric. He has now went through and changed what every single spell does and made some useless. He changed guiding bolt to it just helps projectiles with a plus two, which makes no sense. I also told him they were wrong and he said to use his instead. Also he lowered all my stats to 15 being my highest when it was before 19, and my strength went down from 11 to 8 since clerics are weaker than avergae people. I really wanna quit, but one friend who im closer with wants me to stay since he was very excited to play with me. Just gotta rant about this.

Time to move on and find a decent table.

Yeah, that's some major red flag behavior.  The GM sounds like they've got serious biases against certain classes and is intentionally making them unplayable.  I don't predict having a lot of fun at that table.

Regardless of the DMs character, which can't be communicated well here, it sounds like your bond of trust is broken and you're getting frustrated.

I would consider leaving very carefully before your resentment boils over into your relationships. It's not necessarily doomed...but it's quite likely to happen, and no D&D is better than that. That you're already ranting (regardless of justification) shows a deep incompatibility.

I agree with the above comments, there's so many red flags there as a DM I'd be running for the hills

Normal is a range, and this DM does exceed that range.  Magic is a big part of the game, and these are not simple changes but huge drastic no longer D&D type of changes.  

One thing D&D always had as a base core stereotype.  At low levels fighters protect mages and at high levels mages can protect everyone.   That simple stereotype, does not appear to be still in effect. 

That is the biggest flag of all.