Machine learning's potential for players and developers

By now, Haven Studios should be comfortable working on the technological edge.

After all, much of the studio's founding team came from Google, where they were working to create first-party Stadia games that would showcase the power of cloud streaming technology.

Haven is with Sony these days – the PlayStation maker provided investment for the start-up and outright acquired it earlier this year – but the devs have still been relying on cloud tech to help with hybrid remote/local development, and are looking to push the envelope to see how other emerging technologies can change the way games are made.

"We really have this vision to bring a new type of player-centric, community-driven game to life. And to do that, these investments are very important"

Jade Raymond

In formally launching the studio's Montreal office space last week, Haven co-founder Jade Raymond said more than 30% of the studio is currently working on machine learning, artificial intelligence, and cloud-based dev tools.

"We really have this vision to bring a new type of player-centric, community-driven game to life," Raymond said. "And to do that, these investments are very important."

While it's not quite clear yet exactly what kind of payoff those investments will have, Haven shared an early look at technology that it believes could potentially be used in game development down the road.

Haven director of creative machine learning Logan Olson started by showing a brief clip from the HBO sci-fi show Westworld, where a person working with a futuristic interface describes a story and a computer generates a rough visual that adapts to the changing story details in real-time. While it's nowhere close to practical at the moment, Olson said it was a "pretty rad north star" for people to work towards.

It also may not be as far away an aspiration as it seems, he says, given the advances made by AI text-to-image efforts like Midjourney and DALL-E 2 in recent years. Such programs are now beginning to tackle video and 3D models as well, and Olson says the rate of progress is so fast it's difficult to keep up with.

As for what Haven is working on, Olson demonstrated a differentiable rendering tool that takes a 3D model of a human head in a photorealistic style and alter the image based on a user prompt, whether it be "evil clown," "aquatic alien," or "werewolf." He cautioned there were still a number of hurdles to be cleared before any such output would be realistically usable by the development team, but he was clearly enthusiastic about the future potential.

After Olson's demonstration, Raymond sat on a panel with Haven CTO Leon O'Reilly and PS5 lead system architect Mark Cerny, discussing the potential of machine learning to change the way games are made. Cerny began by reflecting on all that's changed over his decades-long career in the industry.

"When I look back, of course CPUs and GPUs are exponentially more powerful, and we talk about that quite a bit," Cerny said. "What we tend not to talk about is the corresponding increase in the power of the tools we use to create games."

He recalled working on his arcade hit Marble Madness in 1984, when the process of changing a single pixel of the graphics would take several hours.

"It was a multi-step process that involved things like burning the art data into ROMs and then physically inserting the ROMs into a prototype motherboard," Cerny said. "And that's terrible, of course, because fast iteration is key. Fast iteration has been key for the entire history of video games. We never get it right the first time."

If the tools are too cumbersome, creators have a harder time translating their ideas into reality, Cerny said, adding he was excited about the technology Olson showed off and how it might help more developers realize their visions.

O'Reilly has a lengthy history in development himself, having started with the Spectrum ZX. He said iteration times in games steadily improved after the days Cerny was lamenting, but only to a point, and they've actually been getting worse in recent years.

"Iteration times probably reached their peak around the early 2000s, when... the data sets were small enough that we could iterate quickly"

Leon O'Reilly

"Iteration times probably reached their peak around the early 2000s, when the codebases we had were lean enough and small enough, the data sets were small enough that we could iterate quickly," O'Reilly said. "But as we moved away from that and started to build these big open-world games with lots of assets and bigger teams, iteration times really took a hit. There's a lot more data to crunch than there was previously, and the code bases themselves really ballooned in size."

While many of the code bases used today date back to that early 2000s era, O'Reilly said they have had so many additional layers piled on top of them that they have lost their original advantages.

Raymond echoed the importance of iteration speed, saying, "It doesn't take you too many games to understand that the faster you can get to a good view and knowledge of what your game is – and the more you can then iterate on that – that's how good your game is going to be."

She acknowledged Haven's roots at Stadia, and while she said the team was excited about what the cloud could do for the end user experience, she was more keen on what it could do for game engines and dev tools. She said she sees experiments with machine learning in game development in much the same way.

Cerny was particularly interested in how tools like Olson demonstrated could be used to create lots of content quickly that may not be "directly publishable AAA content," but is good enough to get an idea across.

"...before the game even begins, the whole process of conceptualization is going to get so much freer and faster"

Mark Cerny

"The first thing I see is before the game even begins, the whole process of conceptualization is going to get so much freer and faster," Cerny said.

"Today, without these strategies, we go to a conference room, we chat, we scribble on a whiteboard, and when we kind of know where we're heading, we send out for storyboards, and then a week or two later we chat again. Well based on this, you can see we'll soon be communicating at the speed of AI. We will be able to converge so much more rapidly on what our world feels like, and what a level feels like."

He adds that the technology could empower teams "of a moderate size" to do much more.

"That's really important because what we're doing is a creative endeavor," Cerny said. "Yes, you can brute force it. Yes, you can bring in 800 or a thousand people to make your game, but something like maintaining thematic coherency across the project when you have a headcount that high? The left hand doesn't know what the right hand does at one point."

He also echoed Olson's point about the potential for such tools to be used for unique customized content, saying, "What's important is that you're expressing your individuality as a player and it's a little bit less important whether a serious artist would look at that and say, 'You know we could do a little bit better over here.'"

Raymond concluded: "Our north star at Haven is creating a new IP that can be owned by the fans, where the fans are really driving that. Once you launch a service game, the fans do take over. And what does that mean, and where can we bring that, if we can get these tools to the players?"