Foveated streaming! That's a great idea. Foveated rendering is complicated to implement with current rendering APIs in a way that actually improves performance, but foveated streaming seems like a much easier win that applies to all content automatically. And the dedicated 6 GHz dongle should do a much better job at streaming than typical wifi routers.
> Just like any SteamOS device, install your own apps, open a browser, do what you want: It's your PC.
It's an ARM Linux PC that presumably gives you root access, in addition to being a VR headset. And it has an SD card slot for storage expansion. Very cool, should be very hackable. Very unlike every other standalone VR headset.
> 2160 x 2160 LCD (per eye) 72-144Hz refresh rate
Roughly equivalent resolution to Quest 3 and less than Vision Pro. This won't be suitable as a monitor replacement for general desktop use. But the price is hopefully low. I'd love to see a high-end option with higher resolution displays in the future, good enough for monitor replacement.
> Monochrome passthrough
So AR is not a focus here, which makes sense. However:
> User accessible front expansion port w/ Dual high speed camera interface (8 lanes @ 2.5Gbps MIPI) / PCIe Gen 4 interface (1-lane)
Full color AR could be done as an optional expansion pack. And I can imagine people might come up with other fun things to put in there. Mouth tracking?
One thing I don't see here is optional tracking pucks for tracking objects or full body tracking. That's something the SteamVR Lighthouse tracking ecosystem had, and the Pico standalone headset also has it.
More detail from the LTT video: Apparently it can run Android APKs too? Quest compatibility layer maybe? There's an optional accessory kit that adds a top strap (I'm surprised it isn't standard) and palm straps that enable using the controllers in the style of the Valve Index's "knuckles" controllers.
> Foveated streaming! That's a great idea.
Back when I was in Uni, so late 80s or early 90s, my dad was Project Manager on an Air Force project for a new F-111 flight simulator, when Australia upgraded the avionics on their F-111 fighter/bombers.
The sim cockpit had a spherical dome screen and a pair of Silicon Graphics Reality Engines. One of them projected an image across the entire screen at a relatively low resolution. The other projector was on a turret that pan/tilted with the pilot's helmet, and projected a high resolution image but only in a perhaps 1.5m circle directly in from of where the helmet was aimed.
It was super fun being the project manager's kid, and getting to "play with it" on weekends sometimes. You could see what was happening while wearing the helmet and sitting in the seat if you tried - mostly ny intentionally pointing your eyes in a different direction to your head - but when you were "flying around" it was totally believable, and it _looked_ like everything was high resolution. It was also fun watching other people fly it, and being able to see where they were looking, and where they weren't looking and the enemy was speaking up on them.
I'll share a childhood story as well.
Somewhere between '93 and '95 my father took me abroad to Germany and we visited a gaming venue. It was packed with typical arcade machines, games where you sit in a cart holding a pistol and you shoot things on the screen while cart was moving all over the place simulating bumpy ride, etc.
But the highlight was a full 3D experience shooter. You got yourself into a tiny ring, 3D headset and a single puck hold in hand. Rotate the puck and you move. Push the button and you shoot. Look around with your head. Most memorable part - you could duck to avoid shots! Game itself, as I remember it, was full wireframe, akin to Q3DM17 (the longest yard) minus jump pads, but the layout was kind of similar. Player was holding a dart gun - you had a single shot and you had to wait until the projectile decayed or connected with other player.
I'm not entirely sure if the game was multiplayer or not.
I often come back to that memory because shortly after within that time frame my father took me to a computer fair where I had the opportunity to play doom/hexen with VFX1 (or whatever it was called) and it was supposed to revolutionize the world the way AI is suppose to do it now.
Then there was a P5 glove with jaw dropping demo videos of endless possibilities of 3D modelling with your hands, navigating a mech like you were actually inside, etc.
It never came.
That sounds like you're describing dactyl nightmare. [1] I played a version where you were attacking pterodactyls instead of other players, but it was more or less identical. That experience is what led me to believe that VR would eventually take over. I still, more or less, believe it even though it's yet to happen.
I think the big barrier remains price and experiences that are focusing more on visual fidelity over gameplay. An even bigger problem with high end visual fidelity tends to result in motion sickness and other side effects in a substantial chunk of people. But I'm sticking to my guns there - one day VR will win.
I played that game in Berlin in the late 90s. There were four such pods, iirc, and you could see the other players. The frame rate was about 5 frames per second, so it was borderline unplayable, but it was fun nevertheless.
Later, I found out that it was a game called āDactyl Nightmareā that ran on Amiga hardware:
Maybe something like this?
https://en.wikipedia.org/wiki/Virtuality_(product)
I think I played with the 1000CS or similar in a bar or arcade at some point in early 90's
>It never came.
Everything you described and more is available from modern home Vr devices you can purchase right now.
Mecha, planes, skyrim, cinema screens. In VR, with custom controllers or a regular controller if you want that. Go try it! Itās out and itās cheap and itās awesome. Set IPD FIRST.
Thatās reality cool. My first job out of college was implementing an image generator for the simulator for the landing signal officer on the USS Nimitz, also using SGI hardware. I would have loved to have seen the final product in person but sadly never had the chance.
I remember there was a flight simulator project that had something like that, or even it was that.
it was called ESPRIT, which I believe was eye slaved programmed retinal insertion technique.
> 2160 x 2160 LCD (per eye) 72-144Hz refresh rate
I question that we could not create a special purpose video codec that handles this without trickery. The "per eye" part sounds spooky at first, but how much information is typically different between these frames? The mutual information is probably 90%+ in most VR games.
If we were to enhance something like x264 to encode the 2nd display as a residual of the 1st display, this could become much more feasible from a channel capacity standpoint. Video codecs already employ a lot of tricks to make adjacent frames that are nearly identical occupy negligible space.
This seems very similar (identical?) to the problem of efficiently encoding a 3d movie:
I'm entirely unfamiliar with the vr rendering space, so all I have to go on is what (I think) your comment implies.
Is the current state of VR rendering really just rendering and transporting two videostreams independent of eachother? Surely there has to be at least some academic prior-art on the subject, no?
Foveated streaming is cool. FWIW the Vision Pro does that for their Mac virtual display as well, and it works really well to pump a lot more pixels through.
It's the same amount of pixels though, just with reduced bitrate for unfocused regions so you save time in encoding, transmitting, and decoding, essentially reducing latency.
For foveated rendering, the amount of rendered pixels are actually reduced.
At least when we implemented this in the first version of Oculus Link, the way it worked is that it was distorted (AADT [1]) to a deformed texture before compression and then rectilinear regenerated after compression as a cheap and simple way to emulate fixed foveated rendering. So itās not that thereās some kind of adaptive bitrate which applies less bits outside the fovea region but achieves a similar result by giving it fewer pixels in the resulting image being compressed; doing adaptive bitrate would work too (and maybe even better) but encoders (especially HW accelerated ones) donāt support that.
Foveated streaming is presumably the next iteration of this where the eye tracking gives you better information about where to apply this distortion, although Iām genuinely curious how they manage to make this work well - eye tracking is generally high latency but the eye moves very very quickly (maybe HW and SW has improved but they allude to this problem so Iām curious if their argument about using this at a low frequency really improves meaningfully vs more static techniques)
[1] https://developers.meta.com/horizon/blog/how-does-oculus-lin...
That depends on the specifics of the encode/decode pipeline for the streamed frames. Could be the blurry part actually is lower res and lower bitrate until it's decoded, then upscaled and put together with the high res part. I'm not saying they do that, but it's an option.
Itās the same number of pixels rendered but it lets you reduce the amount of data sent , thereby allowing you to send more pixels than you would have been able to otherwise
I think it works really well to pump the same amount of pixels, just focusing them on the more important parts.
Always PIP, Pump Important Pixels
It lets you pump more pixels in a given bandwidth window.
People are conflating rendering (which is not what Iām talking about) with transmission (which is what Iām talking about).
Lowering the quality outside the in focus sections lets them reduce the encoding time and bandwidth required to transmit the frame over.
Foveated streaming is wild to me. Saccades are commonly as low as 20-30ms when reading text, so guaranteeing that latency over 2.4Ghz seems Sisyphean.
I wonder if they have an ML model doing partial upscaling until the eyetracking state is propagated and the full resolution image under the new fovea position is available. It also makes me wonder if there's some way to do neural compression of the peripheral vision optimized for a nice balance between peripheral vision and hints in the embedding to allow for nicer upscaling.
I worked on a foveated video streaming system for 3D video back in 2008, and we used eye tracking and extrapolated a pretty simple motion vector for eyes and ignored saccades entirely. It worked well, you really don't notice the lower detail in the periphery and with a slightly over-sized high resolution focal area you can detect a change in gaze direction before the user's focus exits the high resolution area.
Anyway that was ages ago and we did it with like three people, some duct tape and a GPU, so I expect that it should work really well on modern equipment if they've put the effort into it.
It is amazing how many inventions duck tape found its way into.
Foveated rendering very clearly works well with a dedicated connection, wiht predictable latency. My question was more about the latency spikes inherent in a ISM general use band combined with foveated rendering, which would make the effects of the latency spikes even worse.
They're doing it over 6GHz, if I understand correctly, which with a dedicated router gets you to a reasonable latency with reasonable quality even without foveated rendering (with e.g. a Quest 3).
With foveated rendering I expect this to be a breeze.
Even 5.8Ghz is getting congested. There's a dedicated router in this case (a USB fob), but you still have to share spectrum with the other devices. And at the 160Mhz symbol rate mode on WiFi6, you only have one channel in the 5.8GHz spectrum that needs to be shared.
The real trick is not over complicating things. The goal is to have high fidelity rendering where the eye is currently focusing so to solve for saccades you just build a small buffer area around the idealized minimum high res center and the saccades will safely stay inside that area within the ability of the system to react to the larger overall movements.
Picture demonstrating the large area that foveated rendering actually covers as high or mid res: https://www.reddit.com/r/oculus/comments/66nfap/made_a_pic_t...
It was hard for me to believe as well but streaming games wirelessly on a Quest 2 was totally possible and surprisingly latency-free once I upgraded to wifi 6 (few years ago)
It works a lot better than youād expect at face value.
This is the first standalone headset with an open ecosystem. That's a big deal.
Meta Quests & Apple Visions require developer verification to run your own software, and provide no root access, which slowed down innovation significantly.
Valve giving users root access out of the box is huge. It puts the headset in the same category as a real PC
Praise be to gaben
> first standalone headset with an open ecosystem
What about the Lynx XR1? Running Android sure but officially rooted (details https://lynx.miraheze.org/wiki/Rooting_Process ) and with Linux proper (details https://wiki.postmarketos.org/wiki/Lynx_R1_(lynx-r1) ) even though experimental.
There is but one issue with the Lynx XR1 - no one really got it. A few backers randomly got a few pieces but many others (including myself) are still waiting for their device to arrive (and will most likely wait for ever).
This has a serious impact on the developer ecosystem - there are still a few people who got their devices and are doing interesting work, but with so few users actually having devices the community is too small for much progress to be expected.
It's kinda similar to the old Jolla Tablet - it was a very interesting device (an x86 tablet running an open Linux distro in 2013!) but it ended up in too few hands due to funding issues & the amount of Sailfish OS apps actually supporting the tablet (eg. big screen, native x86 builds, etc.) reflected that.
> many others (including myself) are still waiting for their device
Sucks, sorry to hear that :(
Yes we released our headset with root access and an open bootloader. We are going to announce our next headset in a couple of months :-)
Cool! Do you have a link for a store where I can buy it ? ;-)
Sorry Stan, forgot about the Lynx, huge fan of your work!
Not to mention Meta abandoned the Quest 1 very quickly. I bought a game when it came out and never got around to playing it (had kids). I tried to play it recently and it no longer even works! Ā£30 down the drain, thanks Zuck.
I guess I can't complain too much given that I got it for free.
I bought an Oculus Go last year for ā¬ 30. Its support has been dropped for quite some time, and you can only activate developer mode and sideloading through an old version of the Meta Horizons app [1]. But if you do that, there are 71 GiBs of games to explore on the Internet Archive [2]. Some need patching to remove an online check to a server that no longer exists, but that is easy enough to do with a (regrettably Windows) tool someone published.
The Go is not the best headset of course, but the games are a different style because of the 3DoF tracking without camera's. Somewhat slower paced and sitting down. A style I personally like more.
You can also unlock the device to get root on it [3], which is quite neat, although there doesn't seem to be any homebrew scene at all. Not even the most bare-bones launcher that doesn't require a Meta login.
[1] That doesn't even seem intentional, but it does mean that once the old version of the app can't communicate with Meta servers anymore, any uninitialized Go turns into a brick.
[2] https://archive.org/details/gear-vr-oculus-go
[3] https://developers.meta.com/horizon/blog/unlocking-oculus-go...
That's not quite true - when did you get your free Quest 1? Only January of this year did Meta officially stop allowing devs to support those devices which IMO is not nice, but probably necessary to put resources towards newer devices since it was extremely outdated and very hard to keep supporting. The Quest 1 launched in May 2019, so it got almost 6 years of updates and if you have one, you can still install older versions of existing apps that choose to support it (which admittedly is very rare). I shut off support for my game back in 2024 when they recommended it, since the device is less than half as powerful as the Quest 2, very few users still had one, and the Q1 was a hard target to hit performance-wise vs newer devices. If you spend $50 to get a Quest 2 you'll get a couple years of updates or even better, spend $299 to get a 3S which is an amazing piece of kit and will probably be supported for at least 5 more years since it just came out.
> Ā£30 down the drain, thanks Zuck.
I'm sure he put it to good use. Like 500ms worth of upkeep for one of his yachts.
sorry, maybe i missed it. But how do you know the ecosystem is open?
from the link we don't know if the OS can be changed (might be locked like many Android phones) or if a connected machine is required to run their DRM/Steam. The drivers may also not be open source
It's SteamOS and SteamVR - you can run arbitrary aarch64 Linux binaries that talk to SteamVR and they should just work
Yep, I'm back into VR with this move, specially if the price is closer to $500 than $1000.
Unless the lenses/displays are bad, but I figure we would have heard by now?
from a cursory look . it seems SteamVR is intended to be used with their DRM platform and isn't open source. Maybe its a bit less limiting vs Meta's offering?
i wouldnt characterize this as an "open ecosystem" though
It said its running Steam OS, which is just Linux.
Android is also just Linux. But i cant install Debian on my phone
Even just have direct access to hardware apis is already a big win. On Oculus quest. The closest you can get is running with webxr. But webxr suffer from all those performance problem of web platforms. (And bug of meta softwares. The recent quest browser have bug that prevent you from disabling spatial audio, rendering it not usable for watch video at all)
Frame is obviously the main headline here, but they've also launching a new SteamOS mini-PC and a new controller.
https://store.steampowered.com/sale/steammachine
https://store.steampowered.com/sale/steamcontroller
No prices listed for any of them yet, as far as I can tell.
Oh hell yes. There was a leak of specs (via a benchmarking database) of an upcoming machine from Valve and I had my fingers crossed that it was a mini PC and not some VR thingy, saw this thread, and was sad for a moment before I spotted this post.
6x as powerful as the Steam deck (that I use plugged in anyway 98% of the timeāIād have bought a Steam Deck 2, but Iām glad I get the option to put money toward more performance instead of battery and screen that I donāt use) is great. Not a lot of games I want to play wonāt run well at least at 1080p with specs like that.
What is the draw of the Steam machine though? They say the price is comparable to similarly specced PC. So why not just buy/build any mini PC? There's plenty of options for that
A good while back I abandoned PC gaming because I was sick of driver issues, compatibility, and always having to update hardware to play the next game. Instead, I embraced consoles and haven't considered PC gaming since then. This, however, has me reconsidering that. I want it to "just work". When I want to play games, I don't want to deal with all of that other crap. I'm old, ain't nobody got time for that.
As someone who has been building my PCs for decades, I have to admit seeing some appeal here:
It's apparently small, quiet, capable, and easy.
I'll keep building my own, but most people don't, and the value of saved time and reduced hassle should not be underestimated.
If comparing this device to other pre-built systems, consider that this one is likely to be a first class target for game developers, while others are not.
Some people really don't want to spend time exchanging parts when the memory they buy turns out to be incompatible or that the GPU doesn't fit the sleek mITX case. There's a lot of research to ensure all parts are compatible and optimal when building a PC - for some it's time that could be better spent on using the PC instead of building one.
It's tiny. It runs SteamOS which is built to be used with a controller on a TV. And it will probably be a performance target for many developers.
But I think the biggest feature might be the quick suspend and resume. Every modern console has that, but not PCs. You can try to put a computer to sleep, but many games won't like that.
Snapdragon doesn't really have a good history of supporting proper desktop games. Windows for ARM had kinda bad compatibility. It seems the aim is to have most games just be playable like with the Deck. Fingers crossed but I have some reservations.
Their new mini PC isnāt ARM (the Frame is, though), itās AMD hardware like the Steam Deck. Appears to be x86, should play basically anything in my library at 1080p or higher as long as it works under SteamOS.
Real shame itās only 60Hz at 4k. Thereās a gap for good 120Hz@4k streaming.
Hoping the next Apple TV will do it.
Edit - updated specs claim it can do this, but itās limited to HDMI 2.0
(rewriting this comment because the spec sheet has seemingly been updated)
Looks like it can do 4k 120hz, but since it's limited to HDMI 2.0 it will have to rely on 4:2:0 chroma subsampling to get there. Unfortunately the lack of HDMI 2.1 might be down to politics, the RDNA3 GPU they're using should support it in hardware, but the HDMI Forum has blocked AMD from releasing an open source HDMI 2.1 implementation.
https://arstechnica.com/gadgets/2024/02/hdmi-forum-to-amd-no...
It seems it supports DP 1.4 as well, so perhaps you could get an adapter if your display only supports HDMI 2.1
... but isn't it using a wireless dongle to connect to the headset to the PC so HDMI doesn't get involved?
It seems to me the wireless is pretty important. I have an MQ3 and I have the link cable. For software development I pretty much have to plug the MQ3 into my PC and it is not so bad to wander around the living room looking in a Mars boulder from all sides and such.
For games and apps that involve moving around, particularly things like Beat Saber or Supernatural the standalone headset has a huge advantage of having no cable. If I have a choice between buying a game on Steam or the MQ3 store I'm likely to buy the MQ3 game because of the convenience and freedom of standalone. A really good wireless link changes that.
So, in the specs for the mini-pc, it claims the video out can do 4K @ 120Hz (even faster if displayport). I assume the 4K @ 60Hz you saw is from the "4K gaming at 60 FPS with FSR" line.
I reckon it can probably stream at 4K@120 if it can game at half that.
Interesting. I also saw HDMI 2.0 - I guess itās technically possible but with subsampling?
This is not true, from the specs:
HDMI 2.0
Up to 4K @ 120Hz
Supports HDR, FreeSync, and CEC
I have zero doubts the device can do 4k @ 120Hz streaming Hardware wise. In the end it is just a normal Linux desktop.
Considering how much they talk about Foveated rendering, I think it might not be constrained by the traditional limitations of screens - instead of sending a fixed resolution image at whatever frequency, it'll send a tiny but highly detailed image where your eyes are focusing, with the rest being considerably lower resolution.
Or that's what I think I may be completely wrong.
I am incredibly excited for the new controller. The og steam controller for me was unmatched as a controller, I could never play any first person game on anything else other than mouse and keyboard, not to mention it allows playing rts or point and clicks from the couch.
When they cancelled production I bought 8.
The controller looks pretty cool for sure, my biggest fear is the dpad though. I hope they go for a clicky feel like on the latest xbox controllers, and not the mushy feel you've got on the Dualshock 5 or even the 8BitDo Pro 2, which, for me, really is the only think missing from those. I'm more of a "Dpad in the top left" kind of guy, but I want it to be clicky like on the Xbox controllers :( We'll see!
I'm with you on the dpad. For me I've never found better dpads outside of retro focused controllers from companies like 8bitdo, so when I want to play a retro game with dpad I just grab one of those and use my steam controller for everything else.
A bit of topic, but I was wondering how much bigger is the steam machine compared to the mac mini m4, since that's what I have and is my frame of reference. Obviously comparing apples to oranges and only talking about physical volume, not features, compatibility, price, personal preferences, etc.
Mac Mini m4: 127 x 127 x 50 mm = 0.8 L
Steam Machine: 156 x 162 x 152 = 3.8 L
That's 4.76 times more volume.
> Obviously comparing apples to oranges
Or is it ācomparing apples to steam enginesā?
Given that Valve are the ones who released the Orange Box, methinks the original comparison is valid
It's only a little bigger than Mac Studio.
9.5 x 19.7 x 19.7 cm = 3,687 cmĀ³
and half the size of my SFFPC @ 8.3L
This is going to be an instant buy for me, and my first VR device ever. I've used the previous Steam VR headset over at a friends' place many times, but never bit the bullet to get one myself.
The fact that this can run standalone, doesn't have a bunch of wires dangling from it, and is pretty much a fully working Linux box makes this am almost on-brainer for me.
I do _hope_ the price is reasonable though, if it ends up being like Apple VR I might not buy into it immediately, but I'm hoping for a reasonable $1000 max price.
Not to mention this comes from a company that I respect and that has a proven record of trying to respect its users, unlike literally every other company making VR headsets. The fact that they are trying to making this an open device, and that the controllers have user-replaceable batteries is almost unheard of in any consumer device these days.
Valve kinda shows how a well-managed private business ought to run: respect your customers, find a cash cow and use it for slowly expand into related markets to your niche, develop good products over a long period (SteamOS took many years to become something actually useful) without focusing on the mentality of hyper-growth, keep the stereotypical contemporary MBA thinking away, have a small but competent team.
There are, of course, the issues with lootboxes but even there they've kept their hands much cleaner than any other game developer.
It's a very well oiled machine, I had another VR headset ordered for sim racing, immediately canceled it when saw the Frame announcement because even if specs-wise it's a bit of a downgrade, I want to buy what Valve is selling.
> There are, of course, the issues with lootboxes but even there they've kept their hands much cleaner than any other game developer.
They do seem to get a pretty big pass on that. Wonder what it is about.
Almost every other aspect of the company I find great, and I do wish they would release more games. Maybe Alyx 2 will come out with the headset? Could be what HLX has been this whole time, where people think it is HL3.
On sim racing in VR, absolute game changer. I would never go back to screens, it's the perfect application for VR.
Word is they're aiming for less than the full Index kit (which was $1000), so good news there. I suspect it'll be fairly high up in that range though given the hardware.
See "cheaper than index": https://www.uploadvr.com/valve-steam-frame-official-announce...
Thanks for the article link. Nice quote from the article:
> Unlike the Index controllers, Steam Frame Controllers don't have built-in hand grip straps. But Valve says it will sell them as an optional accessory for people who want them, a similar strategy to Meta.
I was disappointed seeing no hand grip straps. I've never used a Valve Index but they seemed very useful. Very glad that they will still be available.
As an Index controler and Pico user: The back straps are pretty much essential for any serious use; the controller purportedly includes finger tracking (capacitive) but you can't really open your hand without dropping the controller unless you have the strap.
If as I currently intend I end up purchasing this device, I will definitely endeavour to obtain the controller straps as well as the top strap for the headset at the same time, and I recommend others do the same.
You're gonna love where VR is at right now. If you had been holding off until it's good enough, then I think you've timed it well. The Quest 3 from an experience point of view was the watershed headset for me, but the ecosystem being Meta makes it less good from a privacy and ownership point of view.
But this headset solves the ecosystem aspect and brings that visual experience with it.
They've cut some fairly shallow corners, like mono vs color cameras so I imagine getting it within a decent price range has been of high importance. I really doubt it'll be any thing close to $1k.
I think itās possible that thereās a technical reason for monochrome cameras. For example, to let in the maximum amount of IR light for tracking. Bayer filters reduce the amount of light getting in, so it might help the IR LEDs be visible on surrounding walls in the dark.
Still hoping that youāre right, though.
Such a miss not having good full-color AR included. Iām a VR enthusiast with a Meta Quest 3, and itās a shame that this headset is better than the Quest in every way except the most important one.
In my opinion, VR gaming never becomes more than a gimmick. It adds a questionable improvement in graphics and immersion at the incredibly high cost of excluding yourself from the real world. Right now itās not worth it, and I donāt think it ever will be, no matter how good the graphics get. Thatās assuming they even solve the motion sickness problem, which doesnāt seem solvable to me at this point.
The motion controls in VR will also always be severely limited by the fact that you canāt see your surroundings. You canāt meaningfully move around or swing your arms fast in any realistic home environment when youāre in full VR. Youāre constantly at risk of punching something or breaking something, or both. So the controls have to become really stiff and avoid requiring wide movement, at which point you might as well just push buttons on a gamepad.
But AR is a completely different thing. No motion sickness, no risk in any movement, you can move around without silly threadmills, and no exclusion from the world. Itās truly amazing. The AR boxing, pickleball, ping pong and golf are so much closer to real thing then to a videogame adaptation, even the shitty Quest graphics don't ruin the magic. Those AR experiences don't work on videogame rules and really deserve their own name and category - they're as different from gaming as books are from movies. If VR headsets donāt die out, AR is going to be the thing that brings them to the mainstream. I just wish it had more attention, more apps, and more non-Meta mainstream platforms. Not this time, sadly.
Valve is focused on making a device that works well with their existing game catalog. It's a Steam device first, and it needs to be inexpensive to compete with Quest (which is subsidized by Meta), so they need to prioritize which features get included. I wouldn't be surprised to see a first party AR camera attachment a while after launch. The expansion port seems specifically designed for this, with the inclusion of MIPI CSI lanes for two cameras.
I wonder if this will be a VR trojan horse.
The Steam Deck was wildly popular for a non-Nintendo device. It's got Linux up to 3% of total Steam playtime. If this has a similar draw (play every game on Steam without having to buy a TV), maybe the install base of VR will grow to a point where it's more feasible to make games that support it.
It also makes SteamVR relevant again in a world where Oculus has been eating a lot of the mindshare by releasing affordable headsets and buying the most successful game studios.
It will be more expensive than Quest 3s and so is unlikely to grow the VR market significantly beyond what Meta has achieved so far IMO. I'd love to be wrong.
I don't think the greyscale camera is mainly a cost concern. I imagine the greyscale camera has better low light and noise performance, which is quite important for tracking.
The big difference seems to be that this headset doesn't have AR cameras at all, but reuse the mapping camera for some light passthrough duty.
The headsets that have AR cameras don't use them for tracking AFAIK. They all have monochrome cameras for that. The AR cameras are an additional cost that is only used for AR.
I get that there needed to be tradeoffs, I just disagree with this particular one. I could suggest many other ways to save ten bucks in hardware costs. Any other cost saving measure would still allow to play the same games, just with worse performance. But this choice cuts the stock device off from an entire class of apps - in my opinion the best of them all.
I'm sure they did their market research. For me it's the exact opposite. Performance is absolutely key to me, and AR is just a fad in my eyes. All it does for me is give a glimpse into the real world if I'm about to bump into something. AR games are scarce and have never truly impressed me.
AR is a gimmick. VR has real games people spend many hours in. People don't want to see their boring surroundings unless it's to find the couch or a bag of chips.
The real reason the Frame is monochrome AR is because the cameras are also used for IR tracking which is better in monochrome. You can use the Frame in the dark or a dimly lit room - Quest 3 you can't. For real VR users the trade off is worth it.
> You canāt meaningfully move around or swing your arms fast in any realistic home environment when youāre in full VR. Youāre constantly at risk of punching something or breaking something, or both.
You clear the area within the boundaries, leave a little buffer space to the walls, and respect the boundary warnings in game. No problems. You do need a few square meters without any furniture to do this.
Boxing and ping pong feel just as great in VR as they do in AR. It's more a matter of the level of immersion: AR works well for table tennis, but fantasy games are severely limited in what they can do. The most impressive experiences are always in VR - "flying in space" doesn't work while looking at your living room walls.
> It adds a questionable improvement in graphics and immersion at the incredibly high cost of excluding yourself from the real world.
That's a feature for a good number of games, if not most. For example, Resident Evil 4/8 in VR are by far the best horror experiences I've had, and part of it is that you stop seeing your living room while playing.
> The motion controls in VR will also always be severely limited by the fact that you canāt see your surroundings.
There is zero chance that aiming with a controller is more intuitive than point-and-shoot. What I get from your comment is that the movement can be awkward which is absolutely true, but plenty of games have neat ways around that. And then there are games that require no actual movement, like racing games with a sim setup.
The whole "foveated streaming" sounds absolutely fascinating. If they can actually pull off doing it accurately in real time, that would be incredible. I can't even imagine the technical work behind the scenes to make it all work.
I'd really like to know what the experience is like of using it, both for games and something like video.
There's an awesome shader on shadertoy that illustrates just how extreme the fovea focus is: https://www.shadertoy.com/view/4dsXzM
Linus the shrill/yappy poodle and his channel are less than worthless IMO.
When you full screen this, it's crazy how tiny the area that spins is. For me it's like an inch or inch and a half on a 32 inch 4k display at a normal seated position.
(If I move my head closer it gets larger, further and it gets smaller)
That's crazy. I feel dumb for initial thinking it was somehow doing eye tracking to achieve this, despite having no such hardware installed.
I would be curious to see a similar thing that includes flashing. Anecdotally, my peripheral vision seems to be highly sensitive to flashing/strobing even if it is evidently poor at seeing fine details. Make me think compression in the time domain (e.g. reducing frame rate) will be less effective. But I wonder if the flashing would "wake up" the peripheral vision to changes it can't normally detect.
Not sure what the random jab at Linus is about.
Itās normal to be "more sensitive" to brightness differences in the peripheral areas compared to the fovea. The fovea has more color receptors, in the other areas, there are comparatively more monochromatic receptors (brightness). The general density of the fovea is also much larger.
It is doing eye tracking for the foveated rendering - it has 2 cameras inside the visor for it.
That's quite harsh, and definitely not accurate.
Imagine if we could hook this into game rendering as well. Have super high resolution models, textures, shadows, etc near where the player is looking, and use lower LoDs elsewhere.
It could really push the boundaries of detail and efficiency, if we could somehow do it real-time for something that complex. (Streaming video sounds a lot easier)
Foveated rendering is already a thing. But since it needs to be coded for in the game, it's not really being used on PC games. Games designed for Playstation with the PS VR 2 in mind do use foveated rendering since they know their games are being played with hardware that provides eye tracking.
That's foveated rendering. Foveated streaming, which is newly presented here, is a more general approach which can apply to any video signal, be it from a game, movie or desktop environment.
They are complementary things. Foveated rendering means your GPU has to do less work which means higher frame rates for the same resolution/quality settings. Foveated streaming is more about just being able get video data across from the rendering device to the headset. You need both things to get great results as either rendering or video transport could be a bottleneck.
Game rendering is what they're talking about here. John Carmack has talked about this a bunch if you'd like to seed a google search.
As a lover of ray/path tracing I'm obligated to point out: rasterisation gets its efficiency by amortising the cost of per-triangle setup over many pixels. This more or less forces you to do fixed-resolution rendering; it's very efficient at this, which is why even today with hardware RT, rasterisation remains the fastest and most power-efficient way to do visibility processing (under certain conditions). However, this efficiency starts to drop off as soon as you want to do things like stencil reflections, and especially shadow maps, to say nothing of global illumination.
While there are some recent'ish extensions to do variable-rate shading in rasterisation[0], this isn't variable-rate visibility determination (well, you can do stochastic rasterisation[1], but it's not implemented in hardware), and with ray tracing you can do as fine-grained distribution of rays as you like.
TL;DR for foveated rendering, ray tracing is the efficiency king, not rasterisation. But don't worry, ray tracing will eventually replace all rasterisation anyway :)
[0] https://developer.nvidia.com/vrworks/graphics/variableratesh...
[1] https://research.nvidia.com/sites/default/files/pubs/2010-06...
Foveated streaming should be much easier to implement than foveated rendering. Just encode two streams, a low res one and a high res one, and move the high res one around.
There is a LTT video: https://www.youtube.com/watch?v=dU3ru09HTng
Linus says he cannot tell it is actually foveated streaming.
I believe in Linus very little. I'll keep my eyes peeled to see what others say. It's certainly possible though, Valve has the chops to pull it off.
Norm from Tested said the same in his video.
Thats not what he said. What he said was even rapidly moving his eyes around he could not spot the lower resolution part.
If you are going to be pedantic then at least do it right. Because that's also not what he said. He said that no matter how fast he moved his eyes he wasn't able to catch it.
How is that meaningfully different than not being able to tell that it's foveated?
Same with Dave2D https://youtu.be/356rZ8IBCps.
Also mentions 1-2ms latency on a modern GPU
I'm super curious how they will implement it, if it's a general api in steam vr that headsets like the Bigscreen Beyond could use or if it's more tailored towards the Frame. I hope it's the first as to me it sounds like all you need is eye input and the two streams, the rest could be done by steam-vr.
The dedicated communication dongle between the PC and the headset sounds like a real game changer.
Right now getting fast enough and reliable wireless connection means either tweaking to death one's setup or spending car money on the entire setup. In particular normal people usually don't realize how crappy their wi-fi and assume it's all the same, which would end in blaming the poor perf on the headset.
Reminds me of Apple's AWDL, a similar workaround for crappy networks when the devices need a high speed low latency network. I do wonder if the headset here will do similar channel hoping tricks to join both the dongle's network and the normal wifi network.
As I understand it it's two separate radio and stack to have continuous link to both.
This is fantastic!
A while ago I bought the Quest 3 and set it up with WiFi 6 for streaming games. It's a decent setup, but I only bought it cause I was tired of waiting for the "rumored new headset by Valve".
And it seems everything on my wishlist is here:
- foveated rendering based on eye tracking - this is excellent, and was I think only available in the Quest Pro until now
- a dedicated wireless streaming dongle, with multiple radios on the headset - awesome, tuning WiFi 6 got me to a good-enough state, but I'm looking forward to a dedicated out-of-the-box solution
- pancake lenses
- inside-out tracking
In general, having had the Valve Index previously, and then using the Quest 3, it's a night-and-day difference to play something like Alyx wireless. Much better clarity with pancake lenses, too.
Main surprise here is their usage of a Snapdragon chip and not AMD, didn't expect this. I thought it would effectively be a steam deck hardware wise. Curious to see how well that works, esp. for standalone gaming. In practice though you'll likely want to be streaming any "pc-first" titles anyway.
I think they made the right choice with Snapdragon chip... it will drop in and work as a dev kit for all the android toolchains that support quest3, devs will easily port quest3 games etc... so it's basically a non-spyware quest3 which is what everyone wants at this point. Custom drivers on the wifi 6 dongle are going to likely offer the best wireless experience, which again is what everyone wants.
I'm curious how meta responds imo the only way to compete is on price/ease of use but i'm not interested in another quest the 'social features' are just an excuse to collect data.
100%, a non-spyware Quest 3 is what I wanted. The Q3 is a fantastic headset, easily the best amalgamation of features at the right level of performance. Very pragmatic.
But Meta basically having access to my room in 3D, full audio, is not ideal. The very last company I want to invite into my home.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.