Juno 2 Released: Supports Metal Upscaling of 360/180° Videos
Christian Selig has announced version 2 of Juno, his visionOS app for watching YouTube (in the absence of an official YouTube app for visionOS).
Perhaps the most notable feature is its upscaling of immersive videos, which uses the Metal engine.
When watching an immersive video on YouTube, Christian says:
Juno will offers [sic] to open it in an immersive space, where the video will fully surround you as you watch it. (...) The twist is, Juno even takes this a step further, and leverages Apple’s Metal engine to perform advanced AI upscaling and take the existing limitation of 4K video, and upscale it to 6K.
How has Christian accomplished this?
From a technical perspective, this was a TON of work. (...) With just the [YouTube] website, and no direct access to the video, creating an immersive experience is incredibly challenging, but using Metal and impressive GPU on the Vision Pro, Juno essentially takes a series of rapid snapshots of the web browser and stitches them together in a video feed, running each frame through Apple’s MetalFX machine learning upscaling engine, and then projecting that frame onto a sphere that surrounds the user. The result is truly incredible and I’ve seen some really mind-blowing videos.
So, Juno essentially screenshots each frame of the video, runs it through Apple's Metal engine, and re-assembles the resulting upscaled frames into a video that it then plays for you. Bonkers. I can't wait to try it!