Tech Focus: Second Screen Gaming

Digital Foundry takes a look at SmartGlass, the Wii U GamePad and the PS3/Vita connection.

Eight years after the release of the Nintendo DS, it seems that all three major platform holders now believe that dual-screen gaming is an integral part of the future of console gaming. At E3 this year, we saw the launch line-up for the new twin-screen Wii U, an announcement for smartphone/tablet support from Microsoft in the form of the SmartGlass initiative, while Sony offered up some interesting new integration ideas for PlayStation 3 and the PS Vita handheld.

The move towards dual-screen functionality is most likely prompted by a number of factors. Firstly there's a growing belief that gaming is moving away from the living room TV and towards discrete screens per player. It's an idea fuelled by the amount of time that people are spending using their mobile devices in the home, not just outside of it. Secondly, with the heavily rumoured Apple TV display just around the corner, we can almost certainly expect to see a greater level of convergence between the Cupertino giant's devices and there's a definite sense that the established forces in the games industry don't want to be caught off-guard in embracing the ideas this represents. Finally, for Sony at least, it represents a big opportunity to sell Vita to its mammoth PS3 installed base.

"The Wii U GamePad is the star of the second screen movement, but Microsoft SmartGlass and the PS3/Vita hook-up each have their own specific charms. It's all a question of take-up."

The very different approaches we see from Nintendo, Microsoft and Sony are intriguing. Each implementation appears to have its own relative strengths and weaknesses, but it's clear that in Wii U, Nintendo is very much in the driving seat. The advantages of its tech are numerous - the most strikingly obvious being that every single Wii U owner will possess the requisite technology out of the box, meaning that developers can concentrate their efforts on supporting the tablet knowing that it will reach every owner of the console.

There are also the fundamental strengths of the technology itself. There's a latency free connection between the tablet controller and the console, so game control is as fast and fluid as a conventional joypad (something you can't say for Apple's AirPlay) and the ability to mirror HDTV gameplay onto the tablet screen is an excellent ace-in-the-hole. For the first time, console gaming needn't encroach on regular TV viewing - the action can be switched between the main screen and the tablet simply and quickly.

Nintendo describe the tablet video feed as being latency free - something we were dying to try out. At a recent press event in London, we had the ability to film both tablet and screen simultaneously, and found that with mirrored content, the tablet actually received a fresh image up to seven frames - or 116ms - earlier than the LG HDTVs to which the Wii U was connected. This may suggest that the main displays were hopelessly laggy of course, and we need to factor in that the dev Wii U hardware was tethered to the unit (the suggestion being that AV data may have been beamed across from it) but even in a simple wired HDMI vs. tether contest, the results are still highly revealing about how seriously Nintendo is taking latency. WHDI technology operates with 1ms lag, and if Nintendo has opted for this solution, there should be no problem at all in this regard.

Probably the biggest surprise for us at the recent London preview of the E3 Wii U offerings was the sheer lack of latency on the tablet controller. This is tethered dev hardware so we've no idea if the tablet cable is just for power or also for data transmission, but even as a basic wired AV comparison, the lack of lag on the GamePad is quite remarkable.

While Nintendo Land may not be quite the killer app the firm hoped it would be, it does show off a number of great concepts for dual-screen gameplay - specifically when it comes to multiplayer action. The Zelda mini-game for example sees a three-way split-screen on the main HDTV while the fourth player gets his own discrete view on the tablet. The potential for new concepts here is clear enough and third party developers are also coming up with some nice ideas. Rayman Legends' implementation of touch mechanics and a rhythm action mini-game also highlights the benefits of the low latency display and couldn't really be done on competing platforms. Platinum Games' P-100 allows you to draw shapes on the touchscreen, shepherding your units into aeroplanes to fly across gaps or swords for taking on enemies in direct combat. There's also dross amongst the gems though. Ninja Gaiden 3: Razor's Edge, for example, simply uses the screen to display potential combo branches. As you'll be preoccupied with what's happening on the main screen at any point, the touchscreen functionality is pretty much a waste of time.

"Xbox 360 has no Bluetooth functionality, suggesting that SmartGlass would need to interface with the console over WiFi - this suggests some degree of latency that could limit some of its gaming potential."

It's a bit difficult to properly assess what Microsoft wants to achieve with SmartGlass - its own second-screen solution. There is some genuinely revelatory stuff here, not so much from a technological perspective but certainly in terms of the business decision behind it. The bottom line is that Microsoft is enabling Android and iOS products from competing manufacturers to be tightly integrated with the Xbox 360 both in terms of enriching media playback (which is arguably where its strengths lie) and also by offering SmartGlass app integration with 360 gaming. The demo it came up with at E3 involved Madden NFL plays being plotted on the touchscreen before being beamed to the game: hardly revolutionary, but functional enough. Other demos appeared to offer slick-looking menu-driven functionality.

Microsoft has the advantage of being able to rely on an enormous installed base of existing devices in order to generate some momentum for SmartGlass support, but developers will need to create standalone apps for the touchscreen devices: it's believed that these are coded in HTML5 in order to provide a common ground across all mobile devices. Potentially then, Xbox 360 games can gain access to a lot of input-related data: touchscreen motions, gyroscope data, even GPS. Hopefully there is some degree of control on what personal information may be lifted from your devices: Xbox Live (and indeed PSN) already phones home with an immense amount of information on how you interact with the console, but we would hope that the HTML5 protocol would limit access by default to stop this happening.

Conceivably then, SmartGlass development should be fairly stress-free but it is perhaps somewhat limited in comparison to the capabilities of the Wii U. Touchscreen apps couldn't offer anything like the richness we see with the GamePad, and even the notion of using the device as a controller could perhaps be limited. Xbox 360 has no Bluetooth receiver: therefore it cannot pair directly with the device - interactions will need to be carried out through WiFi, and here what we're not sure is if the connection is direct, or whether it will have to negotiate a router, potentially adding varying levels of lag.

Microsoft's 'highlight package' of the SmartGlass reveal at this year's E3, mostly concentrating on movie and TV support but with some quick flashes of gameplay.

At E3, SmartGlass was touted primarily as a means to explain what the hell is going on in Game of Thrones via the use of an interactive map of Westeros, and other streaming info related to what was happening during the show. It may well be the case that gaming applications will be rather more limited, and there'll almost certainly be no gameplay streaming or the pin-sharp response we see from the Wii U GamePad. Sony's second-screen offering does have some potential though.

"Sony's pursuit of CrossPlay titles and its excellent LittleBigPlanet 2 PS3/Vita upgrade offers an interesting, if somewhat resource-intensive, take on second screen gaming."

PlayStation has its own proprietary device that is eminently fit for purpose, if not quite as ubiquitous as SmartGlass devices, or as tightly integrated as the Wii U GamePad - but exciting new dual-screen gameplay concepts could surely give PlayStation Vita a shot in the arm. One of the most intriguing innovations we saw from Sony at E3 was a patch for LittleBigPlanet 2 that offered a colossal wealth of new dual-screen functionality - much of it a match for what we've seen from the Wii U playbook.

As Vita features full Bluetooth functionality, it has the potential to connect with the PlayStation 3 just as seamlessly - and with just as little latency - as the standard Dual Shock 3 controller. This in itself opens up some intriguing options: not only does Vita have face-buttons and twin analogue sticks, it can also relay gyroscope and touch data across to the main console - meaning new ways to interact with the PS3 version of the game.

The LittleBigPlanet 2 demo seen at E3 also showed us a wide range of potential applications for Vita as a second-screen. Asynchronous gameplay was revealed - we saw the Vita player immersed in a shooter, while the PS3 gamer was working his way through a maze situated inside the Vita gamer's ship: movements in the shooting game impacted the viewpoint as seen in the PS3 level. On top of that we also saw more traditional LBP gameplay with puzzles on the PS3 screen resolved through the loading of mini-levels on Vita. It's not exactly Nintendo Land, but with LBP2 being so adaptable, we wouldn't be surprised to see user-generated levels recreating the Wii U title's mini-games!

The Vita connection comes across rather like a mixture of the second-screen approaches seen in SmartGlass and Wii U: there should be the sharp level of response we see from the Nintendo console, but in common with SmartGlass we strongly suspect that a bespoke app would need to be loaded onto the Vita, necessitating development of standalone code that won't address a majority of users. New content would need to be streamed to the handheld (what we're seeing on Vita didn't look much like streaming video/Remote Play as we know it) and this seems to incur some delay. Regardless, it's an interesting evolution of what we've already seen from CrossPlay titles on Vita, where the same game is developed for both systems (MotorStorm RC amongst others) with some excellent use of multi-system gaming - WipEout HD being a great example.

"A big concern with SmartGlass is its status as something akin to a value-added extra. If it's not available to all Xbox 360 owners out of the box, why spend precious resources developing for it?"

Overall then: three dual-screen systems, three very different levels of implementation, each with their own unique strengths - and varying levels of developmental investment required.

Microsoft's SmartGlass approach has the benefit of actually finding a decent level of take-up owing to the vast amount of devices supported and the use of HTML5, but actual potential here is limited by the need to be compatible with so many devices, plus there are question marks over latency.

Sony's approach with Vita is already paying dividends, with the LBP2 demo showing plenty of potential, but the need to effectively generate two different versions of the game must surely sap developer resources significantly and for that reason, third party take-up must be questionable. On the plus side however, two autonomous pieces of code ensures that both PS3 and Vita can dedicate processing resources to their own players - something not even Wii U can provide.

Which leaves us with new Nintendo console, the star of the second screen movement. The principal difference here is that the tablet isn't just a value-added extra as it is with the Xbox 360 and PS3/Vita offerings - it's the core of the experience, with the most exciting, versatile technology. Hands-on with Wii U, we were drawn to the concepts Nintendo came up with and surprised by how robust the third party exclusives were. PS3/360 conversions with tablet functionality tagged on proved to be somewhat less compelling - and this is probably our principal worry with SmartGlass's gaming potential. If developers aren't going to make money with it (as they might from Vita double-dipping), and the applications are limited to value-added extras only, will SmartGlass attract the resources and ideas it needs to be successful?

Latest comments (20)

Peter Dwyer Games Designer/Developer 5 years ago
Given the 360 has usb ports I'm kind of guessing that you'd simply plug a smart glass dongle (i.e. bluetooth 4.0) into one of those or was that too difficult a conclusion to come to for some people?
0Sign inorRegisterto rate and reply
Joe Winkler trained retail salesman, Expert5 years ago
Lol @Peter Dwyer. That's exactly the comment I wanted to make.. Great article though. It's intresting to see the dual screen gaming evolve from a simple "Play and Watch" in the 80's to a HD streaming monster in your hands in 2012.
0Sign inorRegisterto rate and reply
Matt Walker Production Coordinator, Capcom5 years ago
Has MS made any indications that this functionality would indeed be carried out using Bluetooth? If so, that is certainly a conclusion that could be made. It is reasonable to assume that this article was written from the viewpoint of "based on what has already been announced" rather than, "this is how I would do it".
0Sign inorRegisterto rate and reply
Show all comments (20)
Curt Sampson Sofware Developer 5 years ago
The problem with using a Bluetooth USB dongle is that it's yet one more thing for the consumer to buy, and thus yet one more thing to get in the way of take-up of the product. Not to mention the inevitable compatibility issues and so on.
0Sign inorRegisterto rate and reply
Patrick Frost QA Project Monitor 5 years ago
Again, Nintendo is put in the position where the hard work that developers put in to create unique experiences will get built upon or simply ported over to other platforms.
0Sign inorRegisterto rate and reply
AFAICT, SmartGlass will work over the web. It is unlikely to use Bluetooth or local Wifi. All the data will flow through a online server run by MS, and the user will probably need to be connected online on Xbox live.

This makes it more flexible, and easier to operate across a range of devices. The "applications" are then HTML apps effectively run through a browser.

On iOS there is no other way to do it - unless MS are willing to release a new app for every SmartGlass app (as Apple restrict apps from having "dynamic" content).
0Sign inorRegisterto rate and reply
Peter Warman CEO & Co Founder, Newzoo5 years ago
Newzoo recently published an infographic showing that 33% of all console gamers (US, EU) already actively use a tablet. 41% for Xbox 360, 40% of PS3 gamers and 35% of Wii Gamers. Also, an impressive 49% of console gamers plays games on a smartphone. Absolute numbers and more can be found at the newzoo website (infographics).
0Sign inorRegisterto rate and reply
Kevin Patterson musician 5 years ago
I have always wanted a 2nd screen but I'm wondering how well smartglass will work in a gaming situation. Smartglass seems like a neat idea but holding a tablet then switching to a controller then back to tablet would be a hassle. It seems to me that it could be a interim product until the rumored glasses are announced.
I think that will be interesting, the gui could be on the glasses using the smartglass type of tech, the action on the main screen, and the kinect using voice and tracking hand movement.
0Sign inorRegisterto rate and reply
Kieren Bloomfield Software Engineer, EA Sports5 years ago
I keep asking myself the same question: Do I want this? The answer is no.
0Sign inorRegisterto rate and reply
Ken Varley Owner & Freelance Developer, Writer, Devpac5 years ago
Its really simple. Duel screen will be native to the Wii-U. Developers can take for granted that it's there.

For Xbox/SmartGlass and PS3/Vita, developers cannot. Just like Kinect and Move, it will be an after thought.
0Sign inorRegisterto rate and reply
Mike F Student, Arizona State University5 years ago
I haven't used a second screen yet but it seems like it would be distracting to have to look down at another screen while watching/playing something then looking back. I can only look at one screen at a time...

Some of the extra interaction like drawing strategies on screen would be cool though.
0Sign inorRegisterto rate and reply
Jim Webb Executive Editor/Community Director, E-mpire Ltd. Co.5 years ago
@Peter Warman, I hope you don't mind if I posit my dubious stance regarding those numbers.
0Sign inorRegisterto rate and reply
Scott K Davis CEO, QONQR5 years ago
On a tangential note, Id be interested to see this second screen topic from the scenario of the mobile phone being used as an offline second screen for quest based games or MMO. The core gaming experience stays on the console, but mini-quests or maintenance/farming can happen on the mobile device between console gaming sessions. These small in-between sessions can then help you back in the core game when you return to the console. Again, I know that wasnt the topic here with real-time integration being the focus, but extending the gameplay beyond the sofa in the basement is an interesting topic worth exploring.
0Sign inorRegisterto rate and reply
Daniel Chenoweth Freelance Editor, Reviewer, Writer 5 years ago
The article seems to conclude that the Wii U's second screen implementation has the most promise, and I'm inclined to agree, but it also creates another hindrance that third-party publishers are no doubt considering.

For multi-platform titles (at least as long as this current console generation lasts), third party studios can create their games near functionally identically for 360 and PS3, but if they want to do a Wii U version they're going to have to invest a significant amount of time in making use of the controller screen.

As the article suggests with Ninja Gaiden, if they merely use the second screen to display combos or a map, critics will sour on it for not taking proper advantage of the Wii U's potential.

We already see this happening with the PC versions of multi-platform titles, where PC reviewers roast games that don't offer DX11 features or a mouse-centric UI, and I imagine even greater disapproval would result if for example, Activision bring Black Ops 2 to Wii U and don't do anything meaningful on the touchscreen.

As a result of this (as well as the impending next-gen console jump) I don't think Nintendo are going to enjoy much better third party support this generation than the last, despite soon having a console powerful enough to run the same HD games (for a year or so at least) as its competitors.

That said, they've always been big winners on their in-house stuff, so maybe it's really not a big deal.
0Sign inorRegisterto rate and reply
Bruce Everiss Marketing Consultant 5 years ago
I have been using a second screen for years. Firstly with a netbook whilst watching television, now with a laptop. I have seen research that says that most tablet users do the same.
It is especially good whilst watching F1. The normal coverage on TV and the car tracker and social media on the laptop.

This concept translates perfectly to games where you have two completely different sorts of information, perhaps even two completely different sources of information.

Also the DS has had two screens for a few years now, the clue is in the name.
0Sign inorRegisterto rate and reply
Jim Webb Executive Editor/Community Director, E-mpire Ltd. Co.5 years ago
Daniel, the Wii U will still be able to run many PS4/Next X titles the same way a mid range GPU can run the same game a high end GPU can. Game engines are very scalable now and you won't have to worry about rebuilding the game engine or most of the game assets as you did between PS3/X360 and Wii.
0Sign inorRegisterto rate and reply
Daniel Chenoweth Freelance Editor, Reviewer, Writer 5 years ago
@Jim: the scalability of modern engines is a good point, and will definitely contribute to the Wii U being able to endure longer against next-gen systems. PC is probably the biggest winner with that, with the same games being able to run across an even broader range of systems than ever before.

My main point though was that the uniqueness of the U's second screen setup relative to other platforms may be to its detriment with third party support. As third party pubs consider the negative implications of only half-assing support for the tablet controller into their multi-platform games, thus deciding that it's perhaps not worth the extra overhead involved in making a Wii U version of the game.

With the Wii U's "pro controller" Nintendo have definitely gestured for wider third party support with multi-platform titles this time around, but I posit that the uniqueness of the tablet controller is perhaps a bigger step in the opposite direction.

Maybe we'll see some amazing applications of it and everybody will want to add that stuff to their multi-platform games and market the wii u version as the superior port, but I see it as going the same way as the first Wii, where the novelty wears out and third party pubs just stop bothering with a Nintendo version of their games.

But again, Nintendo have always rocked it with first party titles, so they'll probably keep doing just fine without greater third party support this time too.
0Sign inorRegisterto rate and reply
Jim Webb Executive Editor/Community Director, E-mpire Ltd. Co.5 years ago
Daniel, the necessity of developing a separate control design was a minor inconvenience for 3rd parties compared to the drastic architectural differences between Wii and PS3/X360. I've heard very few complain that they didn't support Wii because of the Wii remote itself and even most of them still harped on the architectural gap.

With regards to Wii U, not only do you have the architecture issue fixed but developing for the Wii U GamePad can be as intricate or minimal as the developer wants. And it's my understanding that it's not as difficult to work with as it appears. Take what Vigil stated regarding pushing the image from the TV to the game GamePad for instance.

"When we first got it up and running, you can have the game download to and run on the pad, and everybody was like, that could take a while, little worried," Fitzloff explained. "It took a programmer two lines of code and five minutes. Working with it is not difficult."

I sincerely hope 2 lines of code and 5 minutes of work are not enough to put off developers from supporting the Wii U.

And then you have, as you noted already, the Pro Controller which removes any need for GamePad programming at all.

You had others that didn't go for Wii because of the minimal online infrastructure. The Wii U fixes that.

The Wii U is not going to be, at any point in the life cycle, in the same position with 3rd parties that the Wii was. Might it have less power than the PS4 and Next X? Most likely but not to the degree or with architectural problems that limit the system from having solid 3rd party support.
0Sign inorRegisterto rate and reply
Rick Lopez Illustrator, Graphic Designer 5 years ago
I for one look foward to what the WiiU gampad can bring to RPG and strategy games. The touch screen will allow for customizable extra button inputs. I can free buttons up for more action oriented functions and leave the touch screen to add more command, squad, formation and tactics commands. I can select different squad members using the touch screen instead of using buttons on the controller, saving those buttons for using abilities and powers assigned to them and I dont have to use a circle menu to pause action select the ability and use it. I see many possibilities with the WiiU set up. Actually i see it as becoming a standard edition to the traditional controller, much like shoulder buttons and analog sticks became standard. And I think the WiiU set up is ideal, because its straight out of the box.

If Nintendo can get the cost of the console at a right price then I think the WiiU offers a staggering amount of possibiliies for games at a reasonable cost... because SONY's PS3+VITA setup is too expensive and having a 300$ controller is hardly any value to me and out of the question. If I buy a VITA its because i will use it as a game console, not a controller.

With the way graphics are now a days, a consoles horsepower and specs is not my top priority. I value smart, attractive and entertaining game design. The Wii you may have less specs, but even with lower Specs, if the specs of the machine are anything over current generation consoles then, its good enough for me. Nintendo has great IP's and Ill be satisfied if the games are well designed and fun.
0Sign inorRegisterto rate and reply
Nicholas Pantazis Senior Editor, VGChartz Ltd5 years ago
@ Daniel The Ubisoft Montreal developer I was talking to at E3 said the Wii U was incredibly easy to work with, emphasizing that it took less effort to get Assassin's Creed 3 going on Wii U than it did to get it working on PS3. I really don't think it faces the same issues the Wii did, which were again largely caused by an architectural gap.
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.