Why you should consider off-the-shelf crash reporting

Debugging is arguably one of the most frustrating aspects of game development. Even when you think you're done with it, your game will never be 100% free from bugs, errors and other crashes. However, you always want to come as close as possible to this 100%.

While some studios prefer to build their own crash reporting solution, homegrown options are not everyone's cup of tea.

That's Chuck Tolman's case -- he's the CTO of Skydance Interactive (formerly The Workshop Entertainment), the California-based studio behind VR titles Archangel, Archangel Hellfire and work-in-progress The Walking Dead: Saints & Sinners.

"Homegrown, build-it-yourself solutions can be the right choice in some circumstances," he says. "But in a lot of circumstances, it ends up being just as expensive, if not more expensive, to try to do it your own way."

That's why Skydance Interactive uses crash reporting and error management solution Backtrace. To put it simply, if your game crashes then Backtrace is the product you would use to capture and understand the information to be able to debug that issue.

"A service like Backtrace provides so much functionality," Tolman says. "It's already there for you and can be integrated so quickly that you're solving what's one of the most important things to solve, which is getting off the ground quickly, having working solutions as rapidly as possible. And that's important in game development, because there's a lot of schedule pressure. You have to commit to dates and amounts of money to spend, and trying to build your own solutions can torpedo you."

"Backtrace provides so much functionality: it can be integrated so quickly that you're solving what's one of the most important things to solve, which is getting off the ground quickly."

Chuck Tolman, Skydance Interactive

Tolman advises developers to go for Backtrace before shipping their game.

"However, I could also imagine picking it up after shipping on a product that is experiencing a lot of issues or just volume of data and struggling with reports coming in, or not having a good system to handle it," he adds.

Backtrace works with all platforms including PlayStation, Xbox, PC, and mobile and is integrated with all the major engines. Its cost structure means that if you are a small studio and you don't have as much activity and as much actual usage, you don't spend as much money. So you can ramp up gradually, quickly and save yourself a lot of headaches.

Backtrace's pricing options even adapt to the smallest developers, with a free version available if you have fewer than 250 crashes a month.

"We have a number of very small, independent shops using Backtrace," CEO and co-founder Abel Mathew confirms. "What we've done is created a full spectrum of product packages for every type of game development shop out there. For studios that want to get something off the ground very quickly with minimal investment, we have an option to start at $99 a month, and then fractions of a penny for every monthly active user that you have. Then, when they see much higher usage levels, they pay a progressive amount according to their monthly active user base.

"We also have everything up to AAA-grade packages where you are able to upload tens, if not hundreds, of millions of crashes, give full access to your teams, and get 24/7 support."

Having said that, Mathew understands why some developers could be tempted to build their own solution, too.

"The default mindset of a studio is to build instead of buy, even when the parameters of cost, enablement, and complexity clearly show that buy is the right choice. The idea that building all of your own technology actually leads to a tangible competitive advantage is still very much the mindset in the gaming world. There are very specific aspects of game development where this logic might make sense, for instance graphics or audio effects, but that is becoming less and less the case.

"Off-the-shelf solutions today have an incredible amount of flexibility and customisability. People are still exploring the nooks and crannies of what you can do with these solutions and building very compelling games from it."

Tolman agrees that "there are definitely some things in terms of technology" when it could be advantageous to build yourself. However, he reckons that as crash reporting solutions are not forward facing to the customer, "it's not going to be something that can distinguish your product from other products in any meaningful way."

He adds: "If you have a good [crash and error reporting] solution, then you can create a more stable product. I think that Backtrace in particular is one of those tools that there's not much competitive advantage to be had in trying to cook your own."

Easy to get started

Backtrace is particularly easy to set up, Tolman reports, with Skydance Interactive hooked up within a couple of days.

"After that, we did a fair amount of iterating on it, fine tuning how we were using it, making sure we were uploading all of the data that we wanted to have available in each crash report, adding the Slack integration," he says. "So there was an incremental process of getting these additional details right. But the initial getting it up and running was very fast."

Mathew adds that if you're using a market-available engine or standard frameworks in game development, the Backtrace team has tried "to make it as simple and easy as possible to get going."

He adds: "We strive for the initial integration to take minutes, if not an hour or a day. Once you are up and running, we focus on the idea of extensibility and workflow integration. Tweaking the system to be able to capture more data, to be able to have the workflows that you want -- Backtrace provides you the ability to extend the platform so that you can get exactly what you need to be able to resolve crashes day-in and day-out."

Integrate with your collaboration and issue tracking tools

Extensibility and customisation options are two of Backtrace's strengths, with Tolman expanding on how useful the Slack integration has been for Skydance.

"We use Slack internally for messages and communications across the projects and within the projects," he says. "So we have this Slack integration that spits out Backtrace crash report summaries right into a channel in Slack. That's super helpful because we get instant visibility. Anybody on the team can see the crash reports. You click on the link, it takes you to the web page to show you the details about the crash; the call stacks are right there.

"It speeds up the process of uncovering problems and communicating them and understanding what's going on. Visibility and communication improve productivity in multiple ways. You can instantly download data related to the crash. You can download the crash dump and you can bring it up in your debugger and look at more details. It stores the log file. You have it all available at your fingertips so you can look at the log from the execution."

Slack is not the only ace up Backtrace's sleeve, with the platform being easily adaptable to developers' workflow in general.

"Our thought process was: just capturing the data is not enough. We also want to embed that data where people work, where they collaborate."

Abel Mathew, Backtrace

Mathew says: "Our thought process was: just capturing the data is not enough. We also want to embed that data where people work, where they collaborate. Putting that information front and center so that they can immediately act on that data is one way that we improve the workflow of the teams that use Backtrace.

"Another example of a workflow integration would be integrating it into ticket management systems like Jira. Oftentimes a studio's game development team will be using some sort of ticket management system to be able to manage their work and in Backtrace not only can you integrate into these systems, data in the two systems will sync.

"For example, when a team member marks a crash resolved in Jira, that information will sync back into Backtrace. When you are looking at outstanding crashes in Jira, Jira tickets will have links to directly jump into the full context of the crash with information such as the callstack, metadata, log files, screenshots, and so on. In just a matter of a couple of clicks, you will be able to gather the information you need to drive resolution.

Accelerate development with a turnkey solution

Prior to Backtrace, Tolman explored other options, though for most of them, you have to host yourself. Ultimately he was "looking for something that was a little less work on [Skydance's] part to maintain and that would be cloud accessible."

The concepts of accessibility and ease are really at the core of Backtrace's mission, with Mathew adding that he really wanted to "provide a product that is oriented around the goal of solving your game crashes and exceptions."

He concludes: "We centered our user experience and our product to be able to streamline and operationalise that workflow while the underlying technology we've built can handle the largest of games. By using Backtrace, Skydance doesn't have to worry about scale, maintenance, console and cross-platform support, nor team adoption due to our user-friendly UX.

"Ultimately, this frees up your resources to focus on what you do best: creating an exceptional game."

To find out more about Backtrace, click here.