Hi, I am currently creating a high level (15-20) Viking-themed campaign (I have a years-long obsession with norse mythology and norwegian folklore), but I have some problems creating my monsters and I was wondering if anyone have some good tips. 

First:

- I am not doing skinjobs. And I will not start doing that just because it is faster and easier. 

- I am creating original monsters, and sometimes inventing traits and actions. They are build upon creatures from folklore and norse mythology. For example, I have created a trait for a monster ("nøkken" - a creature famous for luring people into the water to drown them with his fiddle, usually found in marshlands and lakes) called "play the fiddle". Basically he spends his action (recharge 3 rounds after interruption) jumping up on a rock some distance away from the players and plays some hardcore fiddle, binding the players to his will, making them go out into the water drowning themselves (if they fail their save).

So, my issue is to balance the defense challenge rating and offensive challenge rating. If we take Nøkken as an example. He is supposed to be easy to hit AC-wise, and quite weak. but the trick is that he is a slippery one. He has a version of the trait "flyby" called "swimby", and a modified version of "nimble escape" (no advantage on attacks). Coupled with the trait mentioned "play the fiddle" he should make for a real challenge for my team of 4 lvl 15s. I am trying to use the DMG 274 table, and AngryGMs tips on creating cool monsters, but at least the first seems to be badly designed for high CR monsters. Have been trying to check with MM, VGtM and MToF, but the monsters don't generally seem to add up with the table on DMG 274. 

So, here's my real issue: you can move defensive rating or offensive rating up and down according to AC and +attack. And DMG 280-281 give some guidelines on how to calculate "effective CR". But when I try to use all this my monsters have a tendency to end up hitting way harder than I want them to, or being too weak HP-wise than I want them to, or the other way around. For example my Nøkken has a CR-goal of 18. He hits for 102 damage per round. But I don't want him to hit that hard. It doesn't make narratively sense. But for Nøkken to be technically legal he has to hit that hard. Cause his AC is 14, I cannot bump damage too low, or I end up with a weaker monster CR-wise than what I want.  

Anyone with some experience creating high-CR custom monsters willing to share some tips and secrets on how they calculate all this?

You need to make it's AC higher by giving it natural armor. Otherwise, your players will probably never miss with an attack. I would recommend a minimum of 16 AC, and a Maximum of 19.

Here is some chance to hit math with AC along the top and attack bonus on the side.

     1        2      3      4      5       6      7      8       9     10     11   12    13   14     15   16    17    18    19  20
0 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10% 5%
1 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15% 10%
2 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20% 15%
3 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25% 20%
4 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30% 25%
5 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35% 30%
6 130% 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40% 35%
7 135% 130% 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45% 40%
8 140% 135% 130% 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 45%
9 145% 140% 135% 130% 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% 50%
10 150% 145% 140% 135% 130% 125% 120% 115% 110% 105% 100% 95% 90% 85% 80% 75% 70% 65% 60% 55%