why does this need to last so long

Quote from TheHighElfOfDarkness226ba54599 >>

why does this need to last so long

 So a Warlock doesn't have to use multiple spell slots for it

This is balanced to be cast once.  Early on, once per dungeon unless you are short resting which gives you the spell slot back anyway.  Later on, just once per day.

I do think it needs better scaling though because it is incredibly important at lower levels and all but worthless by midgame.

It definitely isn't "all but worthless" if you're playing the standard issue eldritch blast machine gun warlock. It's an extra 1d6 per beam, or per swing if you're pact of the blade. There are tons of ways to get even more, like sorcerer dip for quickened spell, or picking up weapon mastery for extra hits in melee. Not to mention imposing disadvantage so your party can pull things like grapples off more easily.

I guess it depends on how your table plays.  If you are doing a lot of back to back encounters and need to stretch the benefits from those precious few spell slots, then hex makes itself worth it pretty quick.  If you are doing relatively few encounters between short rests (1 or 2) then by the time you have access  to 4th and 5th level spells, spending one of those much more valuable spell slots on hex makes a lot less sense.  Often you get a lot more bang out of one of those higher level spell slots casting something else.

Though if you can get your hands on a spell storing ring or something and toss a few first level hexes into it during downtime, it can make a huge difference without taking up the whole ring.

I just have a hard time justifying using one of 2 or 3 5th level spell slots, my concentration, AND untold numbers of bonus actions on hex when there are so many other fun toys I could cast that could use my concentration and free up my bonus actions.  Also since it only applies to ability checks, not attacks or saving throws, beyond grappling, I don't remember it EVER coming up.  If I am casting hex on it, we are fighting, the vast majority of ability checks that are not attacks or breaking grapples don't happen during combat.  Though if you have a dedicated grappler in the party, like a monk, then it does vastly increase the utility of this spell.

If it lost concentration and/or the bonus actions to re-apply it at some higher level or even with a feat or invocation, then hex would be indispensable later on as well.  Just losing the ability to use concentration spells or bonus actions really hamper it later on.

In the Unearthed Arcana (Horror), warlocks at 3rd level can cast Hex, without using a spell slot, a number of times up to the CHA modifier, which resets after a long rest.    But how long does this non-spell slot Hex last?  The above Hex spell's duration is 1 hour if using a 1st level slot, 4 hours using a 2nd level slot, 8hours using a 3-4 level slot, and 24 hours using a 5+ level slot.   Any thoughts?

When a feature does not specify the level at which a spell is cast, the spell is cast at the lowest possible level.

The UA Hex would be cast at level 1.  This is the case for all features, feats, items, etc. that grant spells - they're always the lowest level unless the feature says otherwise.

Why don't you just make Hex Spell Scrolls in downtime? They take 1 day each to make since it's a level 1 spell.