So I was thinking about grappling with the Astral Arms and wether or not the 10 feet reach made you safe from melee attacks (only from that one enemy you grappled of course).

Now I know this tweet https://mobile.twitter.com/jeremyecrawford/status/618276037122658304 from Jeremy Crawford states that you can attack the thing used to grapple you even if most of the grappler's body is out of reach, and that makes total sense, but would hitting astral arms with an axe or a sword do anything? The only indication I found is they are mentioned as "spectral", but that seems vague enough.

A couple things. As far as how I read it and understand it, per the arms of the astral self description, you have an extra 5 foot reach only when making unarmed strikes. And a grapple is a special attack, not an unarmed strike. 

you could, however grapple with your physical arms and then attack with your astral arms 

also, the description says you can make unarmed strikes with the arms so even if you were within 5 feet you couldn’t use the arms themselves to grapple as, again, a grapple is not an unarmed strike. 

and I don’t believe the arms can be harmed otherwise the description would say so and list AC and Hit points. 

Edit: you would still be able to use WIS for your grapple attempt as using that only requires that they be active.