Tech Focus: Low cost stereo 3D
Ninja Theory co-founder Mike Ball on Enslaved's 3D implementation
Despite take-up of 3DTVs being rather low, the video games industry's commitment to the new format has been surprisingly robust. While we can assume that there has been extensive evangelism for the new display tech from Sony that will have filtered down to publishers and developers, it's interesting to see that a great many cross-format games have supported stereoscopy on both PlayStation 3 and Xbox 360 - Call of Duty: Black Ops being the most high-profile example.
It's all the more surprising bearing in mind the development challenges that supporting 3D represents. Fully implemented stereoscopic 3D can run smoothly on PC with very few issues even on a mid-range graphics card: there have been several generational leaps in the power of AMD and NVIDIA's GPUs in the years since the Xbox 360 and PlayStation 3 launched and in a market dominated by console conversions, it's an innovative use for the extra processing power beyond higher frame-rates and resolution.
However, neither Microsoft nor Sony's hardware was ever designed to handle 3D, and while the quality of the stereoscopic 3D games seen thus far has often been very high, there have been compromises in the quality of the visuals, and often, by extension, the gameplay too. Black Ops looks cool in 3D, but the precise controls offered by 60Hz feedback just isn't there any more.
There are two key challenges in working with 3D. First of all, true HDMI 1.4 stereoscopic 3D sees pixel throughput double - a clear challenge for PS3's RSX chip. Secondly, geometry needs to be processed twice to produce discrete views for each eye.
Guerrilla Games' Killzone 3 is a good example of how cutting edge software is adapted to support stereo 3D: resolution per eye is cut in half, effectively solving the fill-rate issue, while more aggressive LODs are employed to minimise the geometry issues. However, even with these compromises, frame-rate can still drop significantly against the traditional 2D game in certain areas.
Some developers are experimenting with new techniques based on the process of reprojection. Here, a traditional 2D image is created and information from the depth buffer is used to extrapolate out the second view. It's a technique that Sony demonstrated at GDC last year, and according to developer Zipper Interactive, an iteration of this technique is being used in the upcoming SOCOM 4.
But it's possible to get a good look at reprojection techniques in play on a fully fledged cross-platform AAA title in the here and now. Ninja Theory's Enslaved: Odyssey to the West added TriOviz 3D to the game in its recent Pigsy's Perfect Ten DLC update. While Darkworks' 3D-enabling middleware has already been deployed on Rocksteady's tremendous Batman: Arkham Asylum, this is the first time we've seen the tech operating on HDMI 1.4 3DTVs. Not only that, but TriOviz has also been integrated into Unreal Engine 3, making it an inexpensive upgrade for UE3 titles currently in development.
"We had seen the TriOviz tech long before it appeared in UE3 and decided that we had to have it," says Mike Ball, co-founder and chief technical director of Enslaved developer Ninja Theory.
"As soon as we saw it we were blown away that here was a practical way to introduce 3D into our game. We knew that the beautiful environments that we had built in Enslaved would shine even more when viewed with 3D depth."
TriOviz is noteworthy because of its claims about performance, citing extremely low processing times. This means that - in theory - developers can add 3D without having to reduce resolution or pare back geometry.
As Darkworks' product manager Alexis Arragon told Digital Foundry when Arkham Asylum shipped:
"The native integration of TriOviz for Games into a game eats up approximately 2ms each frame. That's below a 10 per cent hit in a 30FPS game. For more technologically-demanding games, there are memory and CPU optimisations to implement to go below the 2ms barrier. One of our prospects successfully integrated our solution in a 60FPS game. Because our SDK works like a shader, there are multiple ways to optimise performances when using TriOviz for Games."
"We had considered a full stereoscopic 3D solution but we had long dismissed this as being impractical for the renderer setup that we have," says Ninja Theory's Mike Ball.
"One of the benefits we found with TriOviz is that it has a fixed performance cost that we were able to easily accept by removing the motion blur post processing whilst 3D is active."
Darkworks reckon that an existing game can support TriOviz with around two to three weeks' development time, and cite a memory overhead of between 2MB and 10MB to include the tech.
"To generate the appropriate back buffer containing the combined left and right eye view there are some intermediary buffers that are required," confirms Ball.
"These can change size depending on platform and output format."
Getting the best possible PlayStation 3 solution also added to the perennial RAM headache. Put simply, more pixels being rendered requires more memory to store them.
"We had to spend some time scrambling to find the additional memory needed to support HDMI 1.4 version on PS3 which is always a little difficult due to the split memory architecture. It wasn't easy but it was certainly worth it to support the full resolution HDMI 1.4 3D," says Ball, who also corroborates Darkworks' claims about how quickly TriOviz can be integrated.
"The initial integration was very short, in the 2-3 week period but we also had to make a number of modifications to support the new addition of the HDMI 1.4 modes so this caused additional work for us. Now that the technology has settled down and an Unreal integration is available, the addition of TriOviz technology should be pretty simple."
While the notion of full resolution 3D with a small performance overhead might sound too good to be true (and PS3 performance analysis suggests the overhead of HDMI 1.4 is significant), there are disadvantages to the stereo reprojection technique.
First of all, as the source of the image is 2D in nature, camera separation potential is quite limited compared to a full stereoscopic 3D solution - and that's due to the lack of available visual information. Imagine walking up to the edge of a wall that obscures your left eye, while your right eye can see beyond it. With TriOviz there would be no discrete view for the right eye - you'd see nothing instead.
Enslaved's third person view helps somewhat and in terms of speedy integration, Ninja Theory actually saw the limited camera as a plus point.
"One of the strengths of the TriOviz system is the way that it automatically handles the cameras in your game by analysing the depth buffer and working out a good focal point for the stereoscopy," Mike Ball contends.
"In Enslaved the cameras were already placed so we were fortunate that we did not need to make any large scale modifications due to this system. Obviously if you want to get in there and have more fine detail control you can."
There are reprojections artifacts though, and they are all limited to the right eye view, making for a curious effect. They are especially noticeable in the cut-scenes where characters leap out from the screen.
The second issue concerns elements in the rendering process that are not found in the depth buffer.
"Transparencies are unfortunately problematic with the z-buffer analysis that TriOviz performs," Ball agrees.
"Either your alpha components write depth or they don't but they are always combined with the colour of the object behind. This has the unfortunate effect of either projecting the alpha texture over the geometry in the background or pulling the geometry behind the alpha to the same scene depth as the alpha plane. Either way just looks plain odd.
"The solution would be to render the alpha effects separately which would add additional rendering cost. Minimising the amount of alpha used in your scene will obviously reduce the amount of artifacting that you will get. Let's face it, minimising alpha coverage is always good."
Implementation of stereoscopic 3D on Xbox 360 is also problematic due to the hardware not supporting HDMI 1.4. While the Microsoft console's HANA chip has been cajoled into outputting larger resolutions (such as 1080p), the stereo 3D 1280x1470 framebuffer employed by the PlayStation 3 appears to be beyond its capabilities.
In theory, 1080p resolution could be employed to give a 960x1080 output per eye in the side-by-side configuration. This would actually offer more resolution than the HDMI 1.4 spec and it would be compatible with 3DTVs, but on the minus side, after the image is processed by the 3DTV, pixels would be rectangular with a 2:1 aspect ratio. The solution employed right now is mostly to render at 720p, half resolution per eye, and this is one of the formats Enslaved employs.
"On Xbox 360 the left and right image are encoded into the same display buffer according to a use selected patter. This effectively halves the resolution in the process. The 3DTV is then responsible for extracting the left and right images from the single buffer according to the encoding used," explains Mike Ball.
"This is an option that the user needs to manually select through their TV's interface. There are a number of encoding formats available - checkerboarded, vertically or horizontally interlaced as well as side-by-side or top and bottom. 3DTVs generally support at least one of these modes so there is a little trial and error to get the correct setup. There is also the option to encode the display buffer utilising TriOviz's patented colour encoding that allows users without a 3DTV to experience the effect by wearing a pair of TriOviz glasses. Their special colour encoding actually preserves some of the original colour in the scene unlike other colour encoding schemes."
For the right games, and with the right implementation, it's clear that reprojection techniques - be they from TriOviz or generated by the developer itself - are an interesting solution for studios looking to get low-cost 3D. In the case of Enslaved on PS3, the performance deficit is noticeable, but some of this appears to be down to the use of v-sync. In the basic 2D version, the game drops v-sync when frame-rate dips below 30FPS, in common with most UE3 titles.
Sony says that the forthcoming SOCOM 4 runs at 30 frames per second in both 2D and 3D modes, while Crytek has made a number of claims about the reprojection techniques being used for Crysis 2 effectively offering zero-cost 3D. However, analysis of the Xbox 360 demo suggested that the developer was actually using a fixed pixel offset to create a diorama effect of three distinct 2D layers as opposed to any real attempt at stereoscopy. Improvements are being promised for the full game, out at the end of this month.
Ninja Theory's Mike Ball looks back positively on the integration of stereoscopic 3D into Enslaved.
"It was definitely an interesting experience for us. We were able to add an additional dimension to the game. The response from anyone who experienced the game in 3D was so good that it justified some of the teething pains we had with HDMI 1.4," he says.
"Overall though the 3D functionality provided by the TV manufacturers still feels very much like the wild west with many different competing technologies but overall TriOviz provided a solution that allowed us to target as many different TV types as possible with minimal code and art change to the product."