In this extensive interview, Digital Foundry talks with Guerrilla Games' technical director Michiel van der Leeuw about the development of Sony's latest first person shooter, Killzone 3.
Covering topics as diverse as improvements to the core rendering tech, upgrades to AI and physics systems, advanced SPU utilisation, stereoscopic 3D, PlayStation Move and the choice of codec for video playback, this piece is a fascinating insight into the creation of one of the most technologically advanced games on console.
The interview kicks off with insights into Guerrilla's approach to the post mortem process, its impact on Killzone 3, and the internal systems established at Sony that allows its many studios to share tech and general development know-how.
Q:Let's go back in time a bit. You've just shipped Killzone 2, the most technologically advanced first person shooter ever made, you've read the reviews and you're getting your first feedback from players. This must have been a pretty epic post mortem - what were your major takeaways from your own analysis of the game?
Michiel van der Leeuw:We certainly did get a lot of good feedback from the audience, and it was epic in a lot - but not all - regards. We take a very good look at the feedback we get from reviews, forums, fans and other studios. We always do a pretty thorough analysis and then see what people comment on a lot, to see if there are recurring patterns. When that's done, we filter the feedback and bring it back in as input points for the next game. Some of the issues that we wanted to tackle were improving controls, getting more variety in the game (gameplay, environment characters etc), fixing any hiccups/glitches, and improving storytelling.
"Some of the issues that we wanted to tackle were improving controls, getting more variety in the game... and improving storytelling."
Q:Aside from your own post mortem, what kind of feedback do you get from elsewhere within Sony? Do other Sony studios contribute their thoughts on each other's games?
Michiel van der Leeuw:Yes, definitely. There's a pretty good positive competition and sense of learning from each other between the Sony studios. We were doing a studio tour right after we finished Killzone 2 to get inspiration from other studios and we often found people in other studios tearing apart a preview copy of Killzone 2 to see how we did. Those were great opportunities for other studios to feed off us, but also for them to give feedback.
We also get to see what other studios are working on and we have meetings to share experiences. We've been on-site with Naughty Dog and Sony Santa Monica before their games were released to exchange experiences, and we regularly have conference calls on specific subjects like animation, scale, storytelling, rendering and so on.
We also have a Sony yearly conference where all the studios attend and there are regular meetings between various disciplines, the Technical Directors have their annual meeting and so on. So there's no single "Studio A judges Studio B's game" but there's a lot of communication back and forth.
Q:Were they any major elements in Killzone 2 you couldn't include but decided to "save" for inclusion in the sequel?
Michiel van der Leeuw:Maybe not so much save, but we wanted to bite off what we could chew. I think the exclusion of a few things like a complete co-op campaign, big vehicles, on-rails sections and a few others were things we felt would stretch us too thin. They'd been on our wish-list to master for a long time and the time just wasn't there yet for Killzone 2. But no, there wasn't a single moment where we went "oh, no don't stick this in, we'll save this for Killzone 3!"
Q:Post-Killzone 2, other Sony studios have pushed technical barriers in terms of rendering (Uncharted 2, God of War III), and networking (MAG). You have direct access to their work. In what ways do your colleagues worldwide help you to make better games?
Michiel van der Leeuw:We have a pretty open dialogue with all of our sister studios worldwide. Apart from regular, organised meetings we're also pretty easy in picking up the phone if we quickly want to run something by somebody. We also have something called SHIP ("SHared Information Portal") within Sony, which is a platform for sharing ideas, technology, etc. It's a combination between a Wiki, bugtracker, file manager, and so on.
It allows any studio or person within the organisation to start a project and contribute things for sharing across all studios. Not only does it spawn a lot of useful projects, it also makes it easier for people to approach one another on chat programmes or by phone - it's easier to know who your colleagues are.
Q:Let's move on to a closer examination of your rendering tech. Killzone 2 was one of the first games to shift to a fully deferred rendering pipeline. Were any changes made to the basic technology in the move to Killzone 3? Were there any changes to the fat g-buffer set-up?
Michiel van der Leeuw:I think that the basic technology is still the same, but we re-organised the g-buffers slightly so we had more dynamic range for colour and lighting. We ended up getting more bang for every bit in the buffer, but it stayed as fat as ever.
Q:In a past presentation from Guerrilla, Logluv was mentioned as a possibility for extending the colour range. Did you experiment with this? What's your final approach in Killzone 3?
Michiel van der Leeuw:We did experiment with it, but found it unsuitable. Both blending and interpolation don't come naturally to Logluv and fixing those issues cost more cycles than it was worth. We deal with colour differently slightly (by pre-multiplying light intensity with exposure before it is colour corrected), but we're still in RGB colour space.
We may experiment with LUV space again, but probably not because we'd like to compress dynamic range into fewer bits, but because chroma-based colour spaces have properties which resemble the way the human eye works. This makes them interesting for things like colour correction, texture sampling and compression as well.
Q:Previously you've stated that you had a fair amount of SPU processing time left over in Killzone 2. What has your approach been to getting more out of these processing elements in the new game?
Michiel van der Leeuw:We've been doing a lot more stuff on SPUs this time around. One of the things we're very proud of is one that nobody sees: We're doing a full depth rasterisation of tens of thousands of triangles in a software rasteriser on SPUs so we can do occlusion culling against it. This allows us to do much more aggressive culling, which in the end allows us to do more complex scenes and further draw distance. The work flow is also much nicer than what we had and it reduces the frame rate spikes which were associated with the portal/occluder system we previously had.
Besides this sort of heavy work, more AI code moved to SPUs, MLAA got added, we do more characters, more physics objects, rendering and streaming book-keeping and a few more game code and physics systems. By the end of Killzone 2 we had a lot of people on the team who were pretty comfortable with writing SPU code, so it was just a natural progression for us to continue using it. We got to the point where they were all full though, so we had to optimise our SPU code quite a bit to fit it all in at the end.
Q:The use of ATG's MLAA tech is a headline addition to the Killzone 3 renderer, yet with the quincunx technique used in Killzone 2, we had a look that seemed to be a great fit with the aesthetic of the game. Why did you decide to make the switch?
Michiel van der Leeuw:A major reason to switch was the fact that we really liked the look of MLAA. Quincunx always had a slight blurring effect on the screen and although it had its charms, it also made everything a bit murky. With MLAA everything was much crisper and textures look sharper, and if we want everything a bit softer we have depth of field, bloom and motion blur which we can tweak.
Q:Aside from the aesthetics, what are the technical advantages in switching to MLAA from a multisampling approach? Assuming you're freeing up extra RAM and GPU resources, how does this benefit the game directly?
Michiel van der Leeuw:With MLAA we reduced our load on the RSX quite a bit, we halved the size of the G-buffers and some depth/stencil re-primes between passes were no longer necessary, which gave us additional speed boosts. In the end it gave us quite a bit of RSX cycles and memory back, which we invested back into texture streaming, draw distance, texture sampling quality and - more indirectly - the variety in characters, environments and so on.
Q:How has the switch to MLAA changed the post-process effects chain?
Michiel van der Leeuw:We were already doing most of our post process effects on the SPUs, but with MLAA moving to SPUs as well we had the entire post chain running there. We also need to store a full resolution colour buffer in XDR RAM (the main RAM) all of sudden, so we had to take a hit there.
Q:MLAA clearly has issues with sub-pixel artefacts, something that clearly wasn't an issue in Killzone 2. Did you experiment with combining MLAA with a multisampling anti-aliasing approach - the best of both worlds?
Michiel van der Leeuw:Yes, we've entertained the thought, but the cost of doing so didn't seem too appealing to us. Although it'd look incredible, we'd have to trade off other factors such as draw distance or variety which were much higher on our wish list.
"We could keep some assets from Killzone 2, but in general we had to up the texture resolution, model complexity, shader complexity and colour use between the two titles."
Q:With Killzone 3 we see a visual look very much removed from the last game. Is this entirely due to the artistic direction, or are there any new effects added to the renderer?
Michiel van der Leeuw:This is largely an artistic direction, but there is some technology that played parts in it as well. There is a delicate balance in our colour pipeline, from what the artwork writes into the albedo buffer, to how it's lit and then how colour-correction deals with it. We were quite happy with the general look, but we also thought it was a bit too monochrome and it was very difficult to colour correct properly.
Together with the Art Director we worked on improving the algorithms that we use and improving our colour pipeline. We have specific hue controls at the end of the pipeline and preserve precision and colour better throughout.
Q:How do art assets compare between Killzone 2 and the new game? Are there just tweaks or are there dramatic changes to elements such as texture resolution and model complexity?
Michiel van der Leeuw:We could keep some assets from Killzone 2, but in general we had to up the texture resolution, model complexity, shader complexity and colour use between the two titles.
Q:Where do animation and physics fit into the rendering and SPU usage?
Michiel van der Leeuw:The physics in Killzone 2 is done with Havok middleware and a lot of our custom work as well. We've been working closely with Havok to move more code to SPUs over time and also reduce the memory use and SPU time spent in physics. We run Havok in parallel with a lot of graphics work on the same SPUs. It's definitely not cheap, but we do a lot of physics in quite a short time.
Animation is all done with PlayStation Edge for the lower level parts and our own code for the higher level. The Edge part is really fast, but we noticed that the high level parts (which was still running on PPU) was becoming quite a bottleneck. At some point we'll have to move that to SPUs as well, but it'll be quite a re-factor!
Q:Are there any significant changes to systems like AI or networking in Killzone 3?
Michiel van der Leeuw:We did improve a lot on the AI, but the major systems stayed the same. We optimised more code to run on SPUs so we could do more AI and bigger battles, but the focus was really on the behaviours this time. We've spent a lot of time making them more diverse and recognisable for different enemy types, as well as improving their overall quality so they're more responsive, their animations look better, they're good in close combat and brutal melee and they're just more fun to play with.
A lot of the network code was scratched and re-written from the ground up. For us community is one of the most important aspects of multiplayer and although we were quite happy with Killzone 2's feature set, we felt our networking systems wouldn't scale with what we wanted to do in future. It's too bad that you don't get street credit for re-inventing yourselves, but we've got a much more flexible system now which is much more stable, responsive, gives us richer stats and provides a solid foundation for things to come.
Q:Based on Naughty Dog's presentations, the SPUs can play a key role in mesh pre-processing (culling, visibility checks etc). You're handling some incredibly detailed scenes in Killzone 3. Do you use the same style of technique? What are the savings you gain here between the make-up of the core assets and what actually ends up on-screen?
Michiel van der Leeuw:We're doing occlusion culling of (sub) meshes on screen and it's difficult to quantify what it gives you exactly because you need to compare it against something. Compared to Killzone 2 I think we have 10 to 30 per cent less overdraw and primitives being wasted (so more of the stuff that we send to the RSX ends up on-screen and being drawn), but it depends on the circumstances.
Q:Frame-rate in Killzone 3 is more consistent than it was in its predecessor. Can you tell us about the ways and means by which you review and optimise game performance?
Michiel van der Leeuw:For Killzone 2 we already had some pretty good stats to track frame-rate. With the flip of a dip-switch on the devkit an artist or programmer can see what is being drawn, what it costs (and who's fault if may be!). Although this gave us insight, it didn't always end up in a smooth frame-rate, some specific sections were bad or we'd have occasional hiccups.
For Killzone 3 we did a few things. First of all we wrote a tool called Autobot, which is a robot (well... server) that starts up each section of the game a few times a day and makes snapshots at artist-specified positions. It records frame-rate, polygons drawn and other vital stats, as well as make a screenshot. We have a technical artist that leads the optimisation who puts cameras at the worst points in a level and Autobot presents all of this on a web page. We can then see graphs over time of how close we are to hitting our performance target and our artists use these numbers to crunch down on geometry and shaders until everything fits.
That helps with the static analysis (of just the art), but what really matters is what happens when it gets in the hands of the players, on the final disc. So we have a special cheat code in the (pre-release) game that turns on some vital statistics on screen like poly count, frame rate and so on, and we capture this as movies. A tester plays the entire game from start to finish each build and we save the movies to a network share. The numbers on screen flash up in red when they go over our thresholds. This allows our optimising artists and coders to quickly scrub through the game and look for performance problems.
It's quite a simple mechanism, but it helped us find dozens of performance hotspots, frame-rate drops when certain actions happened and so on. This used to be difficult to spot and slipped through the nets, but the movies were very effective.
Q:Controller lag was an issue you tackled directly during the production of Killzone 3. From a technical perspective, what were the key changes you made? There was a sense that lag was amplified by frame-rate drops in Killzone 2, but even in heavy scenes response seems much improved in the new game.
Michiel van der Leeuw:There were a lot of changes to all systems related to input, movement, gunplay and player response. We fixed a few bugs where we inadvertently added latency, but we also tightened up deadzones and the responsiveness of particle effects to show your bullet hit something. Anything that wouldn't change the weight, but improve the feel and control of the player we polished up.
"3D is all about immersion... there is such a difference between a vista looking out over an icy landscape on a flat screen, or in 3D where 3km viewing range feels like 3km."
We did a lot of testing with high-speed cameras to verify our changes and see if we were still on track. We also added a system to the engine that would track an event through the engine from the initial button press as it would travel through code, all the way down to when the RSX would send it to the TV. So you can see when the button was pressed, when the game code responded to it, when the bullet was fired, when the particle was spawned and so on. If the high-speed camera then indicated we had long latency or jitter, we could pin-point the exact problem.
Q:What was your first reaction to the suggestion that Killzone 3 should support full stereoscopic 3D?
Michiel van der Leeuw:Madness! Well, actually no. I don't recall it was suggested that we should do it as such. We got light of some 3D research going on and got to play with some prototype hardware which really impressed us. We got offered the opportunity to do it, played with the numbers a bit and then decided to go for it. We're always up for a challenge and this all seemed very promising.
Q:The 3D team have talked about getting the best performance by building a rendering engine with 3D in mind, yet obviously you're re-using a lot of tech from Killzone 2. What was your basic technical approach for integrating 3D?
Michiel van der Leeuw:Our basic technical approach for 3D was building on the work we did for split-screen. We knew we were going to have to run the renderer twice for two players looking in completely different directions with no coherency between them. We must be able to use that technology and optimise it to pieces if we have two eyes looking in the same direction, right?
Q:Is it fair to say that fill-rate is your biggest challenge here, as opposed to generating the geometry twice - once per eye? Is there still the fixed resolution per eye, as per the E3 demo or is it dynamic according to load like Evolution's Motorstorm work?
Michiel van der Leeuw:We have a fixed resolution per eye. We've played with the idea of doing dynamic resolution, but I don't think the technique translates that well to shooters. We focused our efforts on optimisations so we wouldn't have to!
Q:Technicalities aside, what did you want the player to get from the 3D experience and what did you do to achieve this?
Michiel van der Leeuw:3D is all about immersion, feeling that you're really there, forgetting you're sitting on the couch, being on Helghan and wanting to get off the planet. There is such a difference between a vista looking out over an icy landscape on a flat screen, or in 3D where 3km viewing range feels like 3km.
We went for a full stereoscoping solution, not re-projection. This allows us to pull depth separation apart much further and prevent the layered feeling other techniques have. We also get proper stereoscopic reflections to boot and we can be much more free with things like depth of field.
The real trick of 3D is to not make it fall apart. Most people like 3D, but when something is off - your crosshair 'feels' unnaturally placed, you gun feels like it sticks through the wall, something you try to focus on stays blurry - it breaks for people. And when it breaks, it breaks the immersion.
We did a lot of work to avoid it breaking, we dynamically scale depth perception based on circumstances - some of them artist-driven, some of them automatic, depending on what the player is doing. The trick is in getting people to forget they're watching 3D and assume it as real.
Q:Campaign co-op is offline only - what are the challenges in enabling co-op over PSN and why couldn't you include it in Killzone 3?
Michiel van der Leeuw:A co-op campaign and split-screen rendering are pretty tough challenges to face. As a studio we hadn't done this in a long time and that's what we set as a challenge for ourselves to do, for entire campaign, no cheating, no short-cuts.
Doing it on PSN as well would be tricky because - like doing a 'normal' multiplayer game - game state needs to be synced over the network and in case of packet loss or other issues, things need to be arbitrated and made robust so nobody can get stuck, or cheat. Because we do the entire campaign in co-op, that means that all of our game code, even the one-offs, the end-bosses, they all need to be coded with network conditions in mind. We worried that taking on too much might distract us from delivering a solid offline co-op experience, so we decided to focus on that for Killzone 3.
Q:Killzone 2 was around 13GB in all while the sequel almost touches 42GB. A large amount of this new data takes the form of high bitrate Bink-format video. What are the reasons behind the switch to non-realtime video sequences?
Michiel van der Leeuw:The reason to do pre-rendered cut-scenes was based on a desire to remove any loading screens, any interruption of the experience from the user. We personally found the loading screens in Killzone 2 very jarring and disruptive of the experience.
We didn't actually change anything to the rendering process, the cut-scenes are rendered in native 720p with the Killzone rendering and exhibit the exact same quality as the game. We could have cheated and oversampled them or done something else to make them look fancy, but it would defeat the purpose - people would notice and it'd drag them out of the game.
Q:Bink's a fairly ancient video codec by today's standards, and Sony's own libraries support higher quality, more space-efficient h.264 playback. Why go for Bink specifically?
Michiel van der Leeuw:Because we have the entire game streaming in data while we're showing the movies; we have to be very light on CPU usage and we have only very little memory to spare. Often games play movies when there are no game assets loaded, we play movies when all of the assets are loaded. The more popular H.264 profiles require large buffer sizes, to deal with VBR or for reference frames. To use H.264 we'd have to drop down to a simpler profile and probably write a player which is optimised for our particular use, instead of quality and speed.
In the end, Bink fares pretty well compared to other codecs when they're starved on memory and it was easy for us to integrate. That said, it's not something we feel precious about; anything that looks good, runs with little memory and is fast will do for us. I'm a bit tempted to write a special h.264 derivative, or maybe invent something brand new...
Q:Did you need to convince Sony to go for a more expensive dual layer Blu-ray for the game, or did they present it to you as an option?
Michiel van der Leeuw:There was little negotiation involved. When you need it, you need it. I think we were asked what our format was at some point and that was it.
I don't think we'll be creating a brand new engine for this gen, we're pretty close to what you can achieve in this particular direction, although there's always room for improvement.
Q:Let's talk about PlayStation Move support. On the face of it, the basic set-up is generally similar to what we've seen on Wii FPS titles, but your actual implementation is rather more ingenious. Can you talk us through the development process and how you've exploited Move's many advantages?
Michiel van der Leeuw:When the Move was first presented to us it was still in early prototyping stages. There was a desire, but not a big push, for us to support it. We're always interested in new developments, so we were quite keen to see if it would work for us.
To our surprise, our early tests were unexpectedly positive - our experienced Killzone players could actually play the game and almost enjoy it with a basic prototype. The precision of the Move helped a lot in that respect. When we made the decision to research further, one of our programmers took the project to heart and tried to make the best motion controls we possibly could. We found out that - like with 3D - there are a lot of false assumptions and expectations and you really have to try a lot for yourself.
After each experiment we playtested the game, both on our own testers and team members, as well on different people off the streets. We experimented with gestures, aiming and moving mechanics, aim assist and direct controls until we got to the point where a lot of people preferred playing with the Move!
Q:You've got a stereoscopic 3D game and a powerful 3D controller. At the same time we have Evolution Studios looking at holographs - marrying head-tracking with 3D gameplay. The applications in Killzone must surely be mouth-watering...
Michiel van der Leeuw:Yes. They are. We're always on the lookout for things which will make the experience more immersive than the previous.
Q:It's fair to say that Guerrilla raised the technological bar for first person shooters on console, but Killzone 3 is perhaps more evolution rather than revolution. With two PS3 games under your belt now, do you think there's time left this gen to create a brand new game/engine that is as much of a leap as KZ2 was? Or are we looking at a case of diminishing technological returns that makes refining your current engine the better bet?
Michiel van der Leeuw:I don't think we'll be creating a brand new engine for this gen, we're pretty close to what you can achieve in this particular direction, although there's always room for improvement. That doesn't say we can't do anything really fresh, we're going to be doing more than just Killzone in future and new ideas lead to new techniques and fresh experiences.