Without it being an overnight trip I seem to start to rethink at a little over two hours. I tend to do gravel events that take me 3-4 hours and there was one yesterday that was 45 minutes further away than a great race I did earlier this year and it just sounded like a really long day so I opted to stay and just do the usual group ride.
Getting up horribly early before a race seems fine because my sleep before an event isn’t great anyway, but the crampy drive home late in the day can be a lot.
In my tri days I would do stuff like this for half distance + events where I was training for months on end for this one specific event, similar to how you have been working toward Lutsen and Leadville.
As a roadie it’s a lot harder to justify driving more than 3hrs each way for races considering around here there are 2-3 accessible races each month. And as others pointed out, crits are different than longer stuff.
So I think it very much depends on the type of event we are talking about. 3 hrs for a 45 min crit? Probably not. But for a 2 hr proper road race or TT maybe (because those are more rare!)
Maybe take an A, B, C Race approach?
A unlimited
B 600km
C 60km
Or whatever
We’re lucky to have 1 a month unfortunately.
It’s funny, because one of the reasons I was thinking of driving was because it’s a crit (2 races, cat 3/4 and 2/3). We have 1 true crit and another crit/ciruit race (it’s a 1.4 mile oval with no actual turns so I hesitate to call it a crit) all year. The 4 other races we have are all road races. And the two crits don’t offer any overlapping fields to double up on races (e.g. 3/4 and 2/3). So 6 total races within an hour drive all season. Maybe 2 or 3 others that are about 2-2.5 hours away. The racing scene is terrible here. So it’s not a super important race, but I’d double the number of crits I could race by driving to it.
Yeah back in the times where I did multiple crits in a day I would drive more. These days it seems my eligible races are back to back unless I want to do the pro field late in the day and more often than not I just don’t want to be there all day long. I’ve done races back to back before and normally the second one is a fairly miserable experience. 
TBH I’m not a big time crit guy anyway.
Longest drive on paper was 8 hours each way a couple of times but both those trips were long weekends visiting my brother and sister in law who just happen to live literally right on the course for one for Tour of America’s Dairyland crits. Actual time from “home” to start line was less than 2 minutes.
2-3 hours is about tops for me unless it can be rolled into a weekend away with the wife.
As a general rule, I won’t drive much longer (time) than the race will take me. Although I did once drive almost 2 hours each way for a 1 mile race (about 8 minutes). But that was on Southend Pier which is a fairly unique place to run.
i won’t really drive for crits anymore, and I start to question my life choice when the first digit in travel time is a 6.
That said, driving 12h for a road race then gravel race the next day, and hyped to split it into 2 six hour drives and ride in two very random places along the way!
trying to find more adventure outside of just the races helps me palate it all
I used to go up to 3 hours in New England to road races, but I was in early 20s with no responsibilities. I’ve got kids now so that’s a hard no, although I’m contemplating a 10hr drive this fall for a CX weekend! I’m living in a spot now that mostly all races are 2hrs away. I haven’t done any yet, but it’s the only choice and not an easy one.
If you’re young and free go for it. If your older like me, maybe consider doing some gravel races to at least make the drive work it in ‘time on the bike’.