Tech Focus: Data Streaming and the Future of the Optical Drive

This week's release of Battlefield 3 presents a certain proportion of the Xbox 360 audience with an unenviable choice: buy a hard drive or other form of storage media device, or face up to playing a woefully inadequate version of the game. The arrival of BF3, along with other hard drive optimised titles such as id software's Rage suggests that the requirements of game development now outstrip the capabilities of optical drives.

Make no mistake, the difference between Battlefield 3 running with the "HD content" installed and running the game without it is absolutely remarkable. The game is awash with low resolution textures and it's difficult to imagine that DICE would want any one to play through the game with assets as rough as these.

"There's nothing magic about it," BF3 executive producer Patrick Bach told Gamerzines.

"It's the same thing we do for PC and PS3, so there's nothing extra. I think the controversy about this is that we actually let you do it on 360 for once. So what it does is it gives you the same abilities, kind of, as the PC and PS3. You can actually stream information from the hard drive. That's new for Xbox 360, but it's not a new idea for the gaming industry as a whole. No one has really tried to do it properly, so us doing it will create question marks."

Rage and Battlefield 3 get best performance from a 360 hard drive install - is the age of the optical disc nearing its end?

So why is the hard drive install necessary? The problem developers are facing is that the 512MB of RAM in current generation consoles simply isn't enough, so the solution is to stream in gameplay assets - visual and audio - on the fly. Up until recently, on Xbox 360, this has been achieved exclusively by using the optical drive.

The 360's 12x DVD drive can offer some very fast seek and transfer times, and development tools offered by Microsoft allow developers to strategically place content on different areas of the disc. The closer the laser moves to the centre of the disc, the slower data transfer becomes, so developers can prioritise content by moving it to the faster areas of the disc. Up until now, that has proved to be enough: even up against PlayStation 3 titles that partially install to the hard drive, the Xbox 360 has proven to be competitive in terms of streaming and loading times in general without requiring any mandatory installations.


Battlefield 3 and Rage are two recent cross-platform titles where the Xbox 360 version has suffered considerably when the developers have not had access to the hard drive, or other forms of mass storage media.

However, as the demands on streaming have increased, the ability to keep high detail artwork streaming in quickly enough has now seemingly moved beyond the spec of the optical drive, necessitating the install of the most challenging material to a faster medium. With id software's Rage, there is also a very significant difference between the gameplay experience when played with and without a hard disk install: texture pop-in is a bit of an issue even in an optimal configuration, but it's so much worse when the HDD is not available.

Rage on PS3 also demonstrates quite convincingly that even when the hard drive is available, there are still some issues with performance. Responding to the gripes of PS3 Rage players, John Carmack recently tweeted that the level of RAM occupied by the PS3 OS in combination with the lack of unbuffered IO gives the Xbox 360 version of the game a significant advantage, something borne out in the Digital Foundry Rage Face-Off I completed earlier this month.

So, how does DICE's decision to go for optionally installed content work out for the 360 audience in particular? In short, there is going to be a section of gamers that only gets to see the game with visibly diminished visuals. Quite what the percentage is who don't own any kind of mass storage device remains unknown.

"The thing with the 360 is that you need to be able to give consumers a game where you don't have to install it on a hard drive, because there are 360s without a hard drive. So we need to give you the option of installing it, rather than just demanding it. You could call it a 'standard-def' version for the 360 if you don't have a hard-drive," explains Patrick Bach.

In fairness, Microsoft has offered up a lot of different options to ensure that BF3 users can easily benefit from the enhanced "HD" texture pack without having to shell out the cash for one of Microsoft's premium-priced hard drives. The additional content itself weighs in at around 1.6GB, so that means that even users with entry-level Xbox 360S "Slim" consoles should be able to accommodate BF3's lush visuals by installing directly to the 4GB of internal flash storage, with plenty of room left over for player profiles and LIVE downloads. [Updated: For reasons unknown, it appears that DICE has disabled installs to any device other than the hard drive]

Over and above that, Microsoft has also expanded the Xbox 360 platform by allowing for USB flash drives to be utilised as hard drive replacements.

As you can see from the table below, culled from Digital Foundry's extensive flash drive benchmarking on the Microsoft console, the raw throughput of a relatively inexpensive USB stick can match the 360's hard drive. Indeed, as there are no mechanical moving parts, seek times between files could actually be faster than a hard drive.

Only when we went a bit mad and used a cheap no-brand MicroSD card in a USB adaptor did we see a radically reduced level of performance, but in most of our benchmarks, the cheap 16GB flash drive we bought from Amazon was a sterling performer.

Halo 3: Level Loading Tests

Section TestedDVD 360 HDD8GB MicroSD Card16GB ByteStor40GB USB HDD128GB SSD
Sierra 11700:40.000:45.801:34.200:39.400:45.300:38.9
Crow's Nest00:52.701:01.002:02.100:50.700:59.200:50.7
Tsavo Highway00:43.500:49.201:42.600:40.900:47.700:40.8

So what does the Battlefield 3 and Rage experience tell us about the future of the optical drive? After all, when the next-gen consoles arrive, core game assets will be even more data-intensive. Well, we can safely assume that the drive will actually still be there: pressing optical discs - even dual layer 50GB Blu-rays - is going to be a lot cheaper than shipping game-specific SSD-style cartridges. It's also safe to say that broadband infrastructure will not be sufficient to support workable downloads in all current territories, though if all games don't ship digitally day-and-date with their retail counterparts, that would be hugely unfortunate.

The problem I have with a migration across to PC-style game installing is that it impacts the console's defining "plug and play" characteristic. Already we have plenty of complaints about this - especially on PlayStation 3, where playing a new game can be friction-filled experience consisting of system updates, patches and of course mandatory installs. In an ideal world, I'd like to see any kind of forced install banned from next-gen consoles, but bearing in mind the Rage and BF3 experiences, is this possible?

Well, it's safe to say that optical drive performance can radically improve. Let's assume that next-gen consoles will ship with Blu-ray drives: 12x speed BD-ROM units are now in the mainstream, offering an enormous boost to throughput when stacked up against the 2x drive that is standard in the PlayStation 3. There'll still be issues with seek times of course - but we can assume that these will be faster, and we'd hope that all systems would ship with a decent amount of flash memory, a mechanical hard drive - or, as is the case with the higher-end Xbox 360 SKUs in the here and now, both. If Microsoft can ship an entry-level 360 unit with 4GB of flash RAM, who knows how much cheaper storage will be when the next-gen console hits in 2013/2014?

Assuming that all consoles ship with some form of fast storage as standard, perhaps Naughty Dog's Uncharted 3 can offer up a glimpse of the future. The game effectively maxes out a full 50GB dual layer Blu-ray disc, cramming it with 46.3GB of data driven by a mere 18MB executable file.


Uncharted 3 provides a technological tour de force and is a masterpiece in terms of data streaming - there's no mandatory install and the game uses Blu-ray and hard drive cache to provide a superb, seamless gameplay experience.

Diving deeper into the make-up of the new game reveals some interesting facts. 22GB is taken up by movie files - with every cinematic rendered twice: once in 2D, once in 3D. That's a colossal amount of data, but it's still less than half of the space used on the disc, meaning a phenomenal amount of assets related directly to gameplay. In addition to that, we see a large number of files compressed with Sony's proprietary archive format too - so not only did Naughty Dog require a 50GB Blu-ray, they even needed to compress their data still further to make it all fit.

That is rather impressive, but the fact is that Uncharted 3 has no mandatory install whatsoever. The seamless experience it provides is achieved entirely through some extreme hardcore streaming, just like Rage and Battlefield 3, the different being that Naughty Dog had a single, fixed platform to develop on (though even that caused some issues, as this hair-raising and very revealing post-mortem feature by 1UP explains).

While every game design will have its own hardware challenges, Uncharted 3 demonstrates quite effectively that extreme streaming of game assets needn't necessarily mean mandatory installs. On-the-fly streaming can achieve a hell of a lot, and the benefits of a high-speed storage device in addition to the optical drive can preserve the "plug and play" console experience. As we move towards the release of the next-gen consoles, the make-up of the optical drive and the flash medium will come into focus. Hopefully a standard platform along these lines at launch will mean that sub-optimal experiences with certain hardware configurations a la Rage/BF3 will no longer occur...

Latest comments (12)

Lewis Brown Snr Sourcer/Recruiter, Electronic Arts6 years ago
I really dont get what all the fuss is about, I have a 360 Arcade and bought a hard drive for about 15 pound a long time ago, I cant imagine thier are many out thier who havent added a HDD. It seems like a quite simple and obvious way to go, to optimise an ageing system. I'm speaking purely as a consumer here not an EA person. I would encourage others to do the same If it drives up quality.
0Sign inorRegisterto rate and reply
This is exactly why we need consoles to go back to cartridge/solid-state media. Running USB3 speed (or higher) would not only be more than fast enough to replace a hard disk, it would be getting to the speed needed for direct texture access (i.e. main memory) - effectively increasing your overall system memory. Just look at the PSP vrs PSV...
0Sign inorRegisterto rate and reply
Sam Maxted Journalist / Community / Support 6 years ago
I don't think Halo 3 was the best example to use here, as it's something of an abnormality. As far as I'm aware, Halo 3 results in slower loading times reading from the 360's hard disc because of how it uses the HD's cache when loading a level (see I'd be interested in seeing results for other titles on the console, though.
0Sign inorRegisterto rate and reply
Show all comments (12)
Sam Maxted Journalist / Community / Support 6 years ago
For some reason the site won't let me edit my post to fix the broken link. It should be as follows:
0Sign inorRegisterto rate and reply
Nicholas Muise Assistant Producer, HB Studios6 years ago
"This is exactly why we need consoles to go back to cartridge/solid-state media. Running USB3 speed (or higher) would not only be more than fast enough to replace a hard disk, it would be getting to the speed needed for direct texture access (i.e. main memory) - effectively increasing your overall system memory. Just look at the PSP vrs PSV... "

I don't know pricing figures for what cartridge/solid-state media currently is, but I certainly would not want to return to the 80-90$ games of N64 days.
0Sign inorRegisterto rate and reply
Dave Herod Senior Programmer, Codemasters6 years ago
I think what gamers will expect is this: If you don't install extra textures you should get the *same* level of quality as every other game on the market, and if you do install, you get better visuals. It seems like that's not the case though, in that you actually get an experience that's worse than comparable titles if you don't install, and I imagine that's not so forgivable to many gamers.
0Sign inorRegisterto rate and reply
David Spender Lead Programmer 6 years ago
I completely agree Dave. They are trying to spin it the opposite direction but gamers are saying: you sold us the 360 arcade with the promise it would be able to play all games equally. Now that's not the case. They have every reason to be upset.
0Sign inorRegisterto rate and reply
@Nicholas: Tech has come a long way since then - again, look at the PSV for an example. I didn't think I would ever see the day where Sony *dumped* optical media, but they did it. The issue with the N64 was more the tiny cartridge space, than the cost - cartridges/sold-state media mass production cost is comparible to optical discs. Space is probably still an issue though, not sure how easy it is to put 20GB on a cart.
0Sign inorRegisterto rate and reply
Adam Campbell Game Production Manager, Azoomee6 years ago
Interesting article and good overview of the issues.

@Michael Shamgar

On a technical level I can see your point, but on price I think it would be very impractical and not as good as the trade-offs you have with optical discs (very high capacity, dirt cheap but slow).

I can see it working for some hand-helds like the Vita, but we're still talking about a major gulf in the capacity of these things compared to optical discs considering the price. 4GB for a Vita cart (maybe more if pushed) isn't that great.. I'd prefer bigger, cheaper in-built storage be it for a hand-held or home console over cartridges :-)

0Sign inorRegisterto rate and reply
Adam Campbell Game Production Manager, Azoomee6 years ago
"Space is probably still an issue though, not sure how easy it is to put 20GB on a cart."

Very easy indeed. Look at SD cards or even the tiny MicroSD variants happily at 32GB with anything between 2MB and 8MB per second speeds (there may be class 10 and 12 now but I'm not sure). It's just a matter of cost, even at the lower end.

0Sign inorRegisterto rate and reply
Andrew Goodchild Studying development, Train2Game6 years ago
@Adam, as you said, it is a matter of cost, last time I checked on SD cards or USB sticks you pay roughly 1 per Gb, where a blank DVD costs about 12p when bought in bulk. These are obviously consumer prices, but I'd guess the pattern is similar. Most people given the choice would not pay 20-30 extra for that game to come on a 32Gb flash stick rather than a Blue-ray disc.
0Sign inorRegisterto rate and reply
James Berg Games User Researcher, EA Canada6 years ago
Another thing to keep in mind is that 3rd party developers have the knowledge and hardware for designing for and testing disc-based games. Proprietary formats make that a lot more difficult (*cough*UMD*cough*).

How many small-scale developers would think twice about investing in a pile of expensive new hardware to develop on your platform if someone else is doing things differently?
0Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.