I'm a bit confused on how the rabbit hop ability works. 

"Once during each of your turns when you walk at least 5 feet, you can hop, rolling a d12 and moving that many feet in a direction of your choice. This extra distance doesn’t cost movement, but you can hop only if your speed isn’t 0."

If I roll a 3 how would I represent (on the map) how much I moved? Since the maps are drawn with each square being the equivalent of 5 feet. This would also be a problem when determining if I am in a enemies reach or not. 

D&D isn't actually built around maps. The "map grid" is only really for easier reference particularly in battle. Over-reliance on this is a hinderance, because there are various times where numbers are not going to align with such a grid. The ability also might be useful outside of battle too. "There is a gap in the bridge, 7 ft across," - so a roll of 7 or higher lets you hop across, but anything less might be a problem!

If you were 20 feet from a Medium enemy, so there are 4 empty squares between you, and you moved 5 ft then Hopped getting a 3, you are now 8 ft closer. This would put you in a second square towards them so there would now be 2 empty squares between you.

Or just round to nearest 5. 3 ft rounds to 5, 2 ft rounds to 0.

A normal jump is also specific in feet, just like Hop, and the only difference is that Hop is a d12 and doesn't count towards movement and a normal jump is based on Strength score and does use movement. So if you're having an issue with this, you have an issue with all forms of jumping, so your complaint isn't about this feature, but the base D&D 5e rules.