Sections

Riot uses LoL chatlogs to weed out toxic employees

Being a troll in-game translates IRL, finds developer

Riot has used the behaviour of its employees when playing League of Legends to identify potential problems in the workplace, finding a high correlation between those who are toxic in-game and unacceptable workplace behaviour, Re:Work reports.

Because all Riot employees play League, and have the same persistent player records as everyone else, it's a relatively easy matter to take stock of which employees have been attracting bad feedback from team mates and opponents. That done, Riot looked back at the dismissals it had made over the last year and found a correlation between poor in-game reputation and problem behaviour at work - a quarter of all fired employees had been unpleasant players. Toxic players tended to be toxic employees, even if the reverse wasn't always true.

Following that realisation, Riot took a proactive stance, looking at the most problematic LoL records of current employees and examining their workplace behaviour. Looking at the traits exhibited, snarky passive aggression and over authoritative behaviour were found to be the most common negative associations, with some employees 'pulling rank' on other players to intimidate them. This behaviour tended to fluctuate with mood, unsurprisingly, but the situation was bad enough that the company singled out the 30 worst offenders and separated them into two groups: those who needed a warning, and those who needed to leave.

These 30 employees were brought into resolution meetings and confronted with their chat logs. Most, the company reports, were appalled when asked to justify the behaviour, showing contrition almost across the board.

"Pretty much everyone we spoke with was appalled at their own behaviour. We actually received some essays from employees vowing to change their ways and become not just more considerate gamers but better people," said Riot's Talent head Jay Moldenhauer-Salazar.

Now, the logs form an active part of recruitment. All prospective employees are asked for their in-game account names so their behaviour can be tracked. All are then rated on a traffic light system to highlight any potential troublesome proclivities. In addition, Riot is now using the study to try and find out which players work best together, identifying the best team players to create more efficient employee groups.

Related stories

Riot closing Brighton offices

Creative Services branch to be relocated to Dublin headquarters

By Dan Pearson

Riot Games hires Warhammer writer

Graham McNeill joins League Of Legends studio

By Rachel Weber

Latest comments (15)

Renaud Charpentier Game Director, The Creative Assembly10 months ago
Whaaaa, this is so wrong on so many levels! From public contrition meetings and self improvement essays from the worst Stalinian trials to scanning of your private life (game logs) to get a job... welcome to Gattaka! What the worse totalitarian regimes clumsily did with pens and papers seems all fine as soon as it is modern and digital. This is dreadful and frightening to the core individual liberties.
14Sign inorRegisterto rate and reply
Steven Hodgson Programmer, Code in Progress Ltd10 months ago
Sounds like the new graphology, at least ex-convicts aren't allowed to be discriminated unless the job requires a criminal record check. How far back in time do they check for toxic behaviour, as they have already pointed out that they were mainly new to the working world so probably quite young and ignorant.
However, I have avoided League of Legends myself because of what I've heard about the way players treat each other. Hopefully they'll learn something from those they did confront and try to make changes to the game to try to address the issue and influence better player behaviour.

Edited 1 times. Last edit by Steven Hodgson on 10th June 2016 12:02pm

5Sign inorRegisterto rate and reply
Daniel Trezub QA Analyst, GameLoft10 months ago
John Wooden — 'The true test of a man's character is what he does when no one is watching.'

"I lost my job because I am an asshole. This is so unfair" -- former Riot employee

Edited 1 times. Last edit by Daniel Trezub on 10th June 2016 6:52pm

10Sign inorRegisterto rate and reply
Show all comments (15)
Guillaume Provost Studio Head, Compulsion10 months ago
I agree with Renaud. This is just plain wrong on so many levels. I can certainly see how there would be a statistical correlation between player toxicity and workplace behaviors, but I just don't see how that makes it okay to breach people's privacy in this way. I don't really see any real leap from this to diving into people's medical and financial history to weed out people with bad credit history or illnesses.
1Sign inorRegisterto rate and reply
Andrew Watson Tools Programmer 10 months ago
This is exactly why I keep my real life and internet presence as separated as possible.
2Sign inorRegisterto rate and reply
Ethan Larson Senior Programmer, Behaviour Interactive Inc10 months ago
Chat in LoL games is visible to 9 other people, and anyone watching potential Twitch streams. Your behavior in LoL games is hardly what I would call "private". Riot first dug around in their archives to verify there was a correlation, and there was. Then, those found to be toxic ingame were given a chance to reform. I don't get where Renaud got that the contrition meetings were public. From the article, they could be, or it could be just a manager and an employee with nobody else around.

I think the commenters here are making this out to be more evil than it is. Personally, I was shocked when a workmate killed me at the start of a round of Counter Strike because I accidentally shot one of them the previous round during my first game ever. Had I known which workmate it was, it would have given me a sour impression of them from that day forward.

We already give companies we work at permission to read all our work email and other correspondance. You work at Riot, it's hardly surprising they would look at your behavior in Riot's own game. Already they don't hire people who don't play the game.
6Sign inorRegisterto rate and reply
Benjamin Hoyt Founder & CEO, 47 Games, Inc.10 months ago
Big brother much?
0Sign inorRegisterto rate and reply
Klaus Preisinger Freelance Writing 10 months ago
So the game is called LoL, yet it is taken so serious that people riot in chat enough to get kicked out by a company called Riot. The same company whose business model depends on people taking the game way more serious than video games should be taken to begin with. Talk about messaging being all over the place, this is more confusing than all of California's cults thrown together.

The bright future of 2016, be ethnic, gay and transgender all you want, but please strictly adhere to the ideological code of uniformity. At which point is the solution the bigger problem here? Remember when people believed in being able to change other people by use of language? Screw that, big data can sort them, HR can deport them.

Bottom line, do not play with random people, do not use company resources (such as in game chat) to talk to anybody outside work and you shall not be coerced. Hurray for the bright future of multiplayer.
3Sign inorRegisterto rate and reply
Felix Nenz Fix It Felix, Riot Games10 months ago
Working for Riot Games I want to share my perspective.

We welcome players from all over the world to play our game. We ask you that if you do you please treat others like you want to be treated. As part of this we have our "summoners code", that I think was developed with player feedback. Early in our journey we even had a player run tribunal where players be the judges of others reported to be in breach of the code.

At riot we aspire to become the most player-focused company in the world, providing genre-defining experiences for our players. A big part of that is feedback, self-reflection and improvement in everything we do. As a company we are very active on trying to create experiences that are fun and enjoyable, finding ways to reduce toxicity as we believe it directly counters the fun for others.

As rioters we are players. We by the vast majority only hire gamers. We want our team to be able to relate to and know what it feels like to have 5% packet loss in a really close match when you are the last one on the team against two CT's about to defuse your c4.

When it comes to toxicity in our game we hold ourselves to the highest standard. We try to be a role model for behaviour. We want to engage with our players, meet them, have fun and create an open dialogue. Being toxic does not allow for that. Worse, it actively hinders it.

Yes, as sad as it sounds, you might be the most prolific engineer in the world. But if you are an asshole that when losing threatens other players and try to use some kind of authority to get your way - then you are better off working in finance or general business. Not because you are not a brain washed assimilated drone worker, but because you lack the humanity and compassion that we need to help each other making us deliver better player value.
13Sign inorRegisterto rate and reply
Jamie Firth Video Games Production 10 months ago
Anonymity needs to be expunged form the Internet I think.

Too many people choose to use it very poorly.
5Sign inorRegisterto rate and reply
Klaus Preisinger Freelance Writing 10 months ago
I gave this article a long thought and wrote too many words about where it itched me.

Event 1:
In the past, Riot sacked a bunch of people for reasons we do not know. The article leaves that part entirely open to our imagination. Nothing out of the ordinary and the total anonymity of the process means we so far have a non-event.

Event 2:
Riot ran a big data analysis on former employees. Imho, this is the entry point of the creepy territory. Let's write that off as curiosity, though. The result is a correlation of data. Riot presents this data about former employees internally (much later even to the public) and basically says "oh look, wasn't just us". But wait, even if parts of Riot now feel validated on some suspected opinion they had in the past, Riot now made a statement about ALL former employees. People could have left on the best of terms, but Riot now positioned itself as the company that kicks you for being a prick anywhere they can track you. Remind me, why would anybody sign a project based contract with such a company again?

Event 3:
Riot now gets proactive. Having correlated the data in one direction "Prick at Riot correlates with prick Ingame", Riot now removes the "correlates with" and replaces it with "equates to". Even just inverting it would be wrong! By logic of Riot, the big data analysis works in both directions. As a result, 30 people were "caught".

Intermission:
At this point, take a step back and think about the narrative we were given. 30 people worked at Riot seemingly unchallenged about their behavior, until their chat logs 'revealed' them. Is there no other corporate mechanism in case your co-worker or boss is a prick all week? Does that mean as long as you are nice inside the game, you can wreck havoc at Riot, since they lack other mechanisms of reigning people in? Hold those thoughts.

Event 4:
This is the part where 30 people are forced into contrition and were appalled by themselves. Please take special note of the fact that the original article makes a point that this were young people new to the working world! Imho, this makes the things about to happen considerably worse. What you probably really have now is 30 young people suddenly being very scared and very strategic about the truth. Because the source of all this was a video game chat. It was not punching the lady at the reception in the face after working all night on crystal meth. Which I am not implying happens at Riot at all, just as Riot does not imply all people sacked are big data confirmed sociopaths; see?

Event 5:
Some people are now sacked and some are reprimanded. The narrative and justification is exactly the same as in stories where fast food managers are telling teenage cooks not to spit in burgers because that harms company profits when stories get out. It is a stock justification, very corporate, few humans argue like that. Felix Nenz argues like a human in this thread when he says, "treat others like you want to be treated". The article takes the inhuman route by saying "320% less likely to come back again". Also notice how the article still makes no mention of other corporate mechanisms! At this point, events are presented being entirely chat log based. But it gets worse, some people wrote essays, as if still in high school. If you combine that with the fact that the 30 persons were allegedly new to the working world, this makes for one dark story called "My first Job: Riot". Too many things add up in a bad way here.

Event 6:
At this point, you could still laugh it off, you could still tell yourself, this was all one sided reporting on HR's latest pet project, surely there are far more and better mechanisms in place. Maybe Riot forgot to tell re:Work about them. Such as, I don't know, managers listening to complaints and acting like adults? But Riot had to be there for re:Work to receive the information in the article and yet here I am, the reader, stuck between making excuses for Riot, or thinking they are the devil. Then arrives the final paragraph making another point of hammering home, that big data is all Riot needs for a hire, or fire, these days.

Final Event:
the organization wants to make sure they don’t make any bad hires
and yet the real lesson learned is, never reveal your off the clock persona to Riot. Killing anonymity may kill a part of yourself you thought you could keep to yourself. You may think being a recreational prick in a video game for 20 minutes is fun (you know, enact power fantasies in a virtual environment and all that). Riot do not think that at all and they have no mechanism to deal with your juvenile behaviors other than to coerce or fire you (in the context of this article). They do not judge you based on your psychological makeup requiring relief after a stressful day, they will only judge your footprint on their profits. I get it, educating people takes time, company time to be exact, which in turn is company money, the company wants to make more money, so no time to educate. The morals of this story are as brutal and inhumane as they come. Set company_messaging_traffic_light red

Cherry on top:
study to see what types of personalities compose a great team. They’re hoping to come up with an understanding of the personalities that make the best team composition, and how they can proactively create high performing, positive teams.
To at least end this on a joke:
If your co-workers suddenly seem that more attractive to you, it is probably your company starting a breeding program.

---

Dear Felix,
somewhere in your company is a person on his first job, who was scared enough by the company he works for to turn in an essay in an effort to fight for his job. That is the human behind this story, that is what I read about Riot today. I hope the re:Work article is as far from the real truth as could be and still I am sorry for him. (<- emotional exploitation mechanism detected.)
4Sign inorRegisterto rate and reply
Bonnie Patterson Narrative Designer, Writer 10 months ago
So Riot don't use Belbin Team Roles for team structuring and recruitment? I wonder it is about the system that fails there - how to blend types in a team for optimal results is something Belbin largely left up to individual instinct; it always seems to me to be more focused on how to make best use of the personality types that you have rather than choosing types for the best team, so maybe it's that. Still, I'd find it fascinating to know the failings of current systems that are driving Riot to seek new ones. I guess I'll have to wait until all the research gets published.

(Both Belbin and Myers-Briggs give you some great ways of actually coding player-satisfaction into a narrative, which is why they interest me, as well as practically providing a tick-list for developing NPC personalities that will be diverse and plausible rather than cloney and hollow.)

As for the rest, well, note that it was bad in-office behaviour that resulted in warnings and dismissals - it was correlated with toxic behaviour in-game, but in-game behaviour alone was not the reason for dismissals.

Certainly, when I've worked for a company in ANY capacity, I have expected my in-game behaviour and chatlogs to be available for company scrutiny. They have to know that I'm not trading on my position for favours or to intimidate and bully, and they need to be able to prove that I'm not using any associated powers or authority to give friends an in-game advantage, or enemies a disadvantage. As a watchman, I expect to be watched.

But then again, that was always made very clear to me by the companies I was involved with.

Plus acting in a way known to drive away players when they're your employers paying customers seems a lot like biting the hand.

As for "toxic players turn into toxic employees", that's something that psychologists have been pointing out for the last several years. There is no such thing as "only trolling" - they don't hit an off-switch and suddenly become not racist, not sexist, not homophobic, they just think they're not because they believe certain racist/sexist/homophobic myths to be true and thus fair and reasonable to hold.

A bully in-game will turn into a bully in the office. Someone who is passive-aggressive and manipulative in-game will continue to be so out of it. And neither is a good thing to have in your office where there are already enough external sources of stress.

It's all rather sad, though. It's easy to punish, but I'd much rather we'd never allowed this to become a massive problem in the first place. Contrary to popular belief, the base of human nature is not self-centred and arrogant - the limbic system is that of a social, co-operative animal, and it's this that allows a person to feel empathy for his or her fellows, to experience a squeamishness over harming others. Suppress that, as some parts of society have pushed increasingly to do with each generation, and you create a mess that takes quite some time and effort to fix.
4Sign inorRegisterto rate and reply
Renaud Charpentier Game Director, The Creative Assembly10 months ago
And now the article has been "redacted" removing the most horrifying part of it about public shaming and "improvement" essays.
Sorry dear Felix, but only that speaks volume on how wrong all that is: someone in you perfect world realised that was dreadful PR. One day you will (bleakly) realise that absolute transparency is absolute fascism.
3Sign inorRegisterto rate and reply
Alfonso Sexto Lead Tester, Ubisoft Germany10 months ago
@Andrew: Because you are toxic ;) :P (Joke)
1Sign inorRegisterto rate and reply
Felix Nenz Fix It Felix, Riot Games10 months ago
Second comment from me, adding to my original above.
I gave this article a long thought and wrote too many words about where it itched me.
Yeah, reading it "on it's own" there definitely are a few sections that can make it feel oppressive or harsh. That is why I signed up and added my comment, to share my perspective. I have nothing to do with the actual study or the reporting of it. I merely work for Riot Games since 2011 and have been a part growing this company from a few hundred to over 1500 people world wide.
Then arrives the final paragraph making another point of hammering home, that big data is all Riot needs for a hire, or fire, these days.: "the organization wants to make sure they don’t make any bad hires"
Yeah that's totally not how it actually works. We do try to avoid making "bad" hires due to the impact they can have on existing teams. Especially if it ends in a way that the only resolution is termination ("Likely an employee extremely toxic to his teams"). We do this as part of our interview process. As a candidate, you will meet many rioters and talk about many topics. We have conversations and we try to see if we are a good fit for each other. For us, culture is paramount. Part of this culture is being a gamer, self-awareness and empathy with an eagerness to grow. Yes, as part of this study we looked at in-game behavior relating to workplace behavior (some more later). We are looking for long term relationships, not hookups. We sell you as much on the opportunities that we have as you sell us on how awesome you are. Our careers page reads:
We’re looking for humble but ambitious, razor-sharp professionals who can teach us a thing or two. We promise to return the favor. These individuals will become part of entrepreneurial teams who prefer action (kicking ass) over process and bureaucracy (taking names). We embrace those who see things differently, who aren't afraid to experiment, and who have a healthy disregard for the rules.

Applicants should first learn about Riot and play League of Legends because we're looking for people who, like us, take play seriously and for whom it's never "just a game." We play lots of games, especially our own, because hands-on gaming experience is our ultimate guiding compass.

We're hiring for multiple levels, from associate to lead, so don't get hung up on the title; it's all about the team. For the roles below that are listed by team name, we're hiring for varying levels of XP.
If we both decide to go forward we will spend a significant amount of time onboarding you, giving you hopefully all the knowledge you need to make something amazing. As part of onboarding we do discuss what our expectations are from you in public. Not 100% certain as I have not gone trough this recently, but I believe in game behavior is part of this. And yes, there is a pretty significant ask of you as a person when you want to become a rioter. We acknowledge that this might not be for everybody, so we established our queue dodge program. If you join at any time within your first 6 months we encourage you to quit if you feel like it's not the right thing for you. We will pay you 10% of your annual salary to make that decission not bound by money. For more read this.
somewhere in your company is a person on his first job, who was scared enough by the company he works for to turn in an essay in an effort to fight for his job. That is the human behind this story, that is what I read about Riot today. I hope the re:Work article is as far from the real truth as could be and still I am sorry for him. (<- emotional exploitation mechanism detected.)
Absolutely, we are all humans. Being on the receiving side of that conversation is mortifying. You just scored an awesome job at a company that you love, you might even be delivering amazing work and now suddenly you are faced with the real possibility of loosing that. Especially as behavior like that is not as easy to fix as let's say performance. How to make it better? Well many people will fall back to things they know. An essay might be the chosen path of someone when they are less secure talking about the issue. From the conversations we had with all employees (We do monthly AMA with leadership where we can ask ANY question and talk just about anything), people that were identified as toxic where not just amongst the most toxic of riot, but notably so across the entire player base. We knew what riots expectation of our in-game behaviour is. The individuals that where identified had private conversations with their managers and even senior leadership. Which brings the next concern:
And now the article has been "redacted" removing the most horrifying part of it about public shaming and "improvement" essays.
Sorry dear Felix, but only that speaks volume on how wrong all that is: someone in you perfect world realised that was dreadful PR. One day you will (bleakly) realise that absolute transparency is absolute fascism.
Likely as this was wrong. There is no public shaming of employees or essays we ask of people to write. It is dealt with similarly to performance between you and your manager. Yes, just like the Government and any of its 20+ partner organizations world-wide things I do at work are visible to others. The way I engage with one of our players is deeply visible to them. If I am an asshole I do impact their opinion of riot. If I break the summoners code, a code the community is trying to uphold, am I saying I am above this code? I simply can not be. Yes you can have bad games. Believe me I have many many many of them, but hopefully I have the tools to deal with my frustration in a better way. But it's not like the company is just trying to find ways to fire me. As with most of American companies they do not need a reason to do so. Employee at will. Coming from Europe this is definitely different here.

On the issue of privacy, it's a mixed bag for me. But our chat logs are not private. Your chat is seen by at least your team mates and the other team. We use chat logs amongst a ton of other signals to try to make people better players, better team mates. That's why we now have systems like team builder and rewards for positive behaviour. We believe the carrot works better than the stick.
As for "toxic players turn into toxic employees", that's something that psychologists have been pointing out for the last several years. There is no such thing as "only trolling" - they don't hit an off-switch and suddenly become not racist, not sexist, not homophobic, they just think they're not because they believe certain racist/sexist/homophobic myths to be true and thus fair and reasonable to hold.
I think this is what the study attempted to proof / disproove. Having a whole team dedicated to psychological research into player behaviour and toxicity opened this up as an opportunity.
So Riot don't use Belbin Team Roles for team structuring and recruitment?
Not sure. Internally we are trying a lot of methods like Myers-Briggs, Gallup Strength Finder, Crucial Conversations, Coaching for performance to make our teams more efficient. We use a lot of agile. But teams largely find the way they want to work with help of our dev mgmt teams. For recruitment for a short while we used personality tests and things alike, but at least I never got much use out of them. But we keep iterating, trying to make our process faster and better.

If you want to know more about some ancient history in how and why we are focusing so much on culture, Brandon Beck gave a closing Keynote at the IGDA Leaders conference in 2011. You can find it here.
4Sign inorRegisterto rate and reply

Sign in to contribute

Need an account? Register now.