Ha. I’m all for N+1! I was just surprised you said you have wider tires on your roadie than your gravel bike.
IME riding an actual road bike will “feel” faster, but as others have mentioned it’ll come down to many factors like geometry, gearing, tires, etc.
I have a dedicated road bike (typical road gearing, Conti GP5k S TR 28mm) and a gravel bike that leans more towards “gravel race” geometry (not quite as slack as most gravel bikes, but not quite as aggressive as road bikes - similar to “endurance” geometry). The gravel bike has a 2x drivetrain (46/34t front, 11-34 cassette) and in road-mode wears 700x32mm Conti GP5k S TR. The gravel bike is no slouch in road-mode but the gearing and geometry of the road bike make it feel noticeably faster.
Um, well, I did have wider (taller) tires at one point on the roadie. The LBS said they would fit, but didn’t recommended them. Well, they did fit, but the wheels wouldn’t turn! (The knobby tread) So, I bought a gravel bike to use the tires I bought that wouldn’t fit my roadie. Really. (n+1, right?) 
And LOVE those tires. It’s like riding on rails. It might not be faster, but the control and flexibility I get with it can’t be beat.
Buy the gravel bike n+1!!
(I got up this morning to a barfing dog and a screaming wife. I still feel somewhat sleep deprived. On my earlier (deleted) post, I had such high hopes that they would work, I Freudian forgot the whole tire mess. The LBS was shocked, and so was I. I bought the gravel bike, and Happy Days. I did go as wide as I could on the roadie, but it won’t do (wide) knobby tires no matter how hard I try to talk it into doing them. Apologies for the misstatement)
That sounds like some next level justification logic there… I mean, what else could a reasonable person do when the tires wouldn’t work? Well done.
I should be in politics!! 
Yeah, the story is, I was looking for a bike that would handle taller/wider tires. I walked in to the LBS (local bike shop) and on the wall was a dark purple-ish Cervelo Aspero. I was immediately drawn to the color😍. It looked like a purple velvet dress the wife has. Then I looked further and it was a ‘gravel bike’. I asked if it would handle ‘the tires’, and they said it had slightly larger tires on it already. I took it for a test ride, and it felt right. Wow… And it was a hold-over too. (Where were you when I was looking months ago) I bought it that day. I gave them a couple days to check it out, and tweak it a bit, and picked it up. And, yeah, those tires are awesome. And what model tires? Um… {The chase is better than the catch(https://www.youtube.com/watch?v=PktB0bxo2oQ)?} But great bike. Just put another 30+ clicks on it this weekend.
Pirelli Centuro Gravel M Great tires…
I can’t tell you how badly I wanted that link to lead to Motorhead!! Well done. 
Team Angry Music… It’s amazing how music can help on some rides. Music is a drug. Rock on…
I have a Canyon Grizl gravel bike that I ride on the road with Rene Herse Bon Jons (35 mm spec, more like 38mm IRL). Up until a month or so ago my road bike was a BMC Roadmachine (an endurance bike) with Enve 5.6s with GP5000s, 25mm up front and 28mm in the back.
I have done comparisons on a 4.5 mile TT stretch between the two, and the BMC is noticeably faster than the Grizl. Over a min at similar power. Caveat I have not stuck the Enves on the Grizl. Keep meaning to try that.
But I just built up a Supersix Evo, and now it has the Enves. I absolutely love this bike. The handling is so different than the BMC. I wouldn’t call it twitchy, but it’s way livelier. After a few rides I don’t think about it so much. What still amazes me is how compliant the Supersix is. I never thought the Roadmachine was harsh. Actually, I think it rides great. But the Supersix with the exact same wheels/tires and saddle is way more comfy. The road feels damped. And that’s with last gen wheels and a 25mm tire. The SuperSix also feels somewhat compact. You would not be forced into a super low race fit.
The SuperSix is very comfortable with racy handling. It’ll absolutely be “different” than your gravel bike, but your call whether that’s “better.” We have a sporty SUV and two-seater sports car. I like them both, but they’re very different.
Good luck w/ the decision! I’d get the SuperSix 
Last summer I was traveling and expected to mostly ride gravel, with one or two road group rides mixed in. I brought my gravel bike (Giant Revolt Advanced) and a second pair of road wheels/tires I pulled off my road bike (Tarmac SL6).
On a very fast/ competitive road group ride, I was shocked at how slow the gravel bike felt with my road wheels/tires. Since it was the same wheelset I use normally riding and racing road, it could only be the frame that felt different, and when standing to follow attacks/surge up hills it was tangible how much extra effort it took to push power through the more compliant/ heavier frame. I was also getting momentarily distanced during hard surges from riders I normally have no trouble keeping up with, though I was able to rejoin once the speed slowed slightly and I could bridge back.
The lesson for me was that the extra compliance is definitely welcome on gravel, but does make the bike more sluggish when things are smooth, especially during standing hard efforts. I love my Tarmac and my Revolt, but I don’t see either as the right bike for all conditions. If I was mostly riding road, I’d absolutely choose the Tarmac. But these days I mostly ride gravel so my Revolt sees most of the usage.
Interesting recent Escape Collective podcast that touches on whether more comfortable bikes are or aren’t faster: Podcast: Tires are getting softer, but road bikes are getting … stiffer? - Escape Collective
TL;DR (as I remember it) was that comfort and compliance are incredibly hard to model and quantify, unlike weight and aerodynamics. In general there’s very little evidence that perceived frame comfort varies inversely with speed or efficiency. BUT of course we often perceive a super-comfy frame as slower or less “snappy,” and pro riders often request very stiff frames; if you’re convinced the bike you’re riding is robbing you of watts you’re probably not going to ride your best.
And related to the current conversation, I think the more race-oriented gravel bikes are typically stiffer and less compliant, so would probably not feel like riding a sofa if kitted out with road wheels/tires.
I have the same two bikes and agree 100% with everything you said. The Tarmac doesn’t just feel faster. It IS faster. By quite a bit.
Yeah, stiff and snappy don’t translate to any significant speed unless it’s addressing a handling limiter on a technical course. Aero (and to a lesser degree weight) are the significant difference makers if you are comparing 2 bikes running the same wheel/tire combo. Included in aero is the aerodynamics of the frame/bars, but the geometry can also have a big impact on aero if it allows you to get in a more aero position on one bike vs. another. Body position is usually the biggest aero factor, typically more than frame, wheels, helmet, etc. combined.
I’ve done some back to back testing on my madone aero race bike vs. my checkpoint gravel bike and it’s about a 15 watt difference at 25mph on a flat road running same tires and trying my best to mimic body position. 15 watts is good for about .5mph at those speeds, so nothing to sneeze at, but you obviously don’t get all that if you are riding in a pack or at lower speeds. That comparison also assumes you have the positions set up similar on both bikes. Many people will ride a gravel bike in a more upright position (either by choice or because the frames don’t support the same position). That could make the difference significantly bigger. I don’t ride in an aggressive position on my road bike, so it’s pretty easy to get in a very similar position on my gravel.
In my opinion, sofa bikes are underrated. Smooth/comfortable is fast, especially on long rides. My madone sits in the garage unless a race demands it. But I totally get the desire/fun of riding a racy bike (especially on short rides), I’ve been there.