Ghost in the Machine

That's an interesting one. You're right, on the platform side, and Dance Central is a great example, there is no lag. Let me talk about Kinect Adventures for a minute, because what I'm definitely not saying is that there's a half-second of lag on Kinect Adventures.

Kinect Adventures, as a title, is designed not only to enable you to have fun, and to have fun instantaneously, but also to teach you about the platform. It's one of the titles that's supposed to be simple, fun and approachable to everyone.

From that perspective it's meant to be an easy-in, easy-out type of experience where a lot of people can be having fun together, cheering each other on. This is the game that probably went through the most playtesting, and the most usability testing to make sure we tuned the experience so that, being the packed-in title, it's something that everyone can enjoy. Something that everyone can get learning from, and understanding of how to use the system.

I'm not sure what you've seen, which makes it hard for me to answer, but I will say that in terms of game mechanics, those mechanics were optimised to make sure that they were instantly fun and accessible to everyone. Does that answer your question?

Okay, well now I think I'm narrowing down on your problem and I can give you a more precise answer. On the jumping side, you have to think about the amount of fun people are having in the living room. This is the part where I say, we playtested the crap out of this game. If you have kids moving up and down, the last thing you want to do is have a false positive.

The last thing you want to do is have that raft or the obstacle course, or the avatar somewhere jumping when you are not jumping. If you take little kids, running around the room, this is where the human lag comes into play. You look at it and you say, 'how do I ensure, positively, that really was a jump?' How to make sure it wasn't just a crouch?

Think about the range of human motions. If I just crouch, and lift myself really quickly, how's that different from a jump? The answer is that you can detect those things, but you have to be very careful. This is where the game is tuned to make sure that you are building trust in the robustness of the platform. The last thing we wanted was to be getting a lot of false positives on River Rush because people in our playtest lab were having tons of fun, playing it.

So I'm not saying there's a half a second of lag, in that game, but I will say that, for the thing that you talked about, it's more about being intentional about design and making sure we're only actuating mechanics and actions when we're one hundred per cent sure that they've occurred in the living room.

Let's bring it to a world we understand - let's talk philosophy for a little bit. This is no different to thinking about a racing genre. Think about the difference between a Forza and a PGR. One has an arcadey feel to it, the other one has a simulation feel to it. As a game designer I choose where I put the needle. If I'm a Forza designer then I'm going against the demographic of customers that really enjoys the simulation level of the experience. I'm going to be extremely precise, to the point that, if you're not good, you're not going to be able to drive the car.

Project Gotham Racing is made to be much more approachable. More arcadey in its driving, less simulation. It allows for a different type of experience. The same is true here, and I'll use the two examples we just talked about. Dance Central is made to be a simulation type game. You're simulating dancing - you want it to be precise, you want it to be real-time and guess what? If you're not good at it, you're just going to suck at dancing. But they also have the 'Break it Down' feature that really teaches you how to be a great dancer.

Take Kinect Adventures, the needle moves to a different side. It is made to be a fun, simple, approachable game that gets people acquainted with the platform. Those remain toolsets - paint colours and paintbrushes that game designers, the storytellers, get to choose. I think you'll see that the platform has range, that it has the range to go down a more realtime simulation, all the way out to a less simulation sort of game. That's what I think you're seeing here.

That is true.

Sure. One of the major key ingredients of the experience is machine learning. Machine learning in our world is defining a world of probabilities. Machine learning, particularly our kind, which is probabilistic, is not really about what you know, it's about what you don't know. It's about being able to look at the world and not see duality, zeroes and ones, but to see infinite shades of grey. To see what's probable. You should imagine that, in our machine learning piece of the brain, which is just one component of the brain, pixels go in and what you get out of it is a probability distribution of likelihood.

So a pixel may go in and what comes out of it may be - hey, this pixel? Eighty per cent chance that this pixel belongs to a foot. Sixty per cent chance it belongs to a head, twenty per cent chance that it belongs to the chest. Now this is where we chop the human body into the 48 joints which we expose to our game designers. What you see is infinite levels of probability for every pixel and if it belongs to a different body part.

That operation is, as you can imagine, a highly, highly parallelisable operation. It's the equivalent of saying, pixel in, work through this fancy maths equation and imagine you get a positive number, a positive answer you branch right, you get a negative answer you branch left. Imagine doing this over a forest of probabilities. This is stuff where you'll get a thousand times performance improvement if you put it on the GPU rather than the CPU.

GPUs are machines designed for these types of operations. The core of our machine learning algorithm, the thing that really understands meaning, and translates a world of noise to the world of probabilities of human parts, runs on the GPU.

In a way, yes. A little bit differently though. The Kinect brain works in the same way as your brain, or my brain. Our brains are machines designed to essentially be this massive blob of signal to noise. We push away the noise and very quickly focus in on the signal.

I like to give the example of how you and I came into being, and then you'll understand how Kinect does the same. Imagine we have a fictitious baby. She or he is zero years old, you show this baby a human and a lion, and you say - 'here baby, tell the difference between them'. Turns out that a brand new baby cannot. Time goes by and now this baby has enough reference data, historical data, to be able to predict, next time you show it, the difference between the two.

Now that baby a male and a female. It's not going to have any idea how to differentiate those things. Some time passes and it has enough historical data, enough training to be able to tell the difference. Now show the baby two females, a random one and one the baby knows, how does the baby tell them apart? The baby can't.

Now fast forward to you and me. The second you meet someone that you've never seen before, you instantaneously have a gigantic amount of information. You know roughly that person's age, you roughly know their size, you roughly know where they're from, their age, whether they're male or female. You roughly know the position that they're in.

Our world works in the same way. Your brain doesn't just know everything that it sees. As you walk through the world, it's using previous historical data to essentially predict, based on probabilities, what you're seeing now. Kinect works in the same way, that's the fundamental principle. What we've done is shown it a sample of statistically significant data which allows us to comprehend the world in a way similar to the way your brain operates.

Let me tell you something slightly different. If I think of the search base - the ranges of human motion, the shapes of bodies, the conditions of living rooms worldwide, I get a search base that will equate to somewhere around ten to the power of 23. I can't even say that number. If I'm doing cause/effect based programming, that's a lot of conditions.

I could have an army of people coding, their entire lives - I still don't have a product that can recognise it. Which is why you have to break it, to move away from traditional programming, you have to create breakthroughs that really allow you to work in a different way. It's not that you code, in the system, what it's going to see, instead you teach it to recognise the real world. If you can teach it, much like your brain, when you get in front of it, it doesn't know, it has to look at previous history to predict the future. In that world you're talking about a much more fuzzy world of signal to noise - you're talking in terms of probabilities instead of actualities.