Study: Loot box labels inconsistently applied

Loot box researcher Leon Xiao has found that rating board warning labels about randomized loot boxes are being inconsistently applied.

Xiao's latest study looked at the whether the North American ESRB and Europe's PEGI rating systems are applying the "In-game purchases (includes random items)" labels consistently. The labels were rolled out by both groups with essentially identical criteria after government scrutiny on loot boxes.

Separately, Xiao also took a random sampling of 100 games on Google Play that were known to have loot boxes in them and check to see if the comparable IARC label regarding loot boxes was included alongside each game's rating.

After eliminating ports, multiple releases of the same game and titles released in one market but not the other, the study found 66 different games that presumably should have carried the randomized in-game purchases label in both the ESRB and PEGI databases. However, only 26 of those games (39.4%) sported the label in both systems, while 40 games (60.6%) had the label in one system but not the other.

PEGI appeared to be more active with the label, as it warned consumers about 35 games with loot boxes that the ESRB did not. On the other hand, there were only 5 games to carry the warning in the ESRB database but not PEGI.

Xiao singled out Genshin Impact as one example, calling it "highly unsatisfactory" that such a high profile game that relies on loot boxes does not carry a randomized purchase label from the ESRB.

He noted that PEGI appeared to have retroactively applied the label to some games after introducing it whereas the ESRB did not, which may account for some of the inconsistency. But even when applying a different methodology to give the ESRB the benefit of the doubt, it still only managed to be 74.3% consistent with PEGI, with the ESRB failing to attach a label to 4 games that PEGI did.

As for Google Play, Xiao found only 29% of the 100 random games known to have loot boxes actually displayed the label. To ensure the developers had not removed the loot boxes (as has been done recently with titles like Brawl Stars and Mario Kart Tour), Xiao replayed the games to verify they still contained loot boxes; all of them did.

Xiao suggested ESRB and PEGI along with German rating board USK collaborate going forward, applying loot box labels to all games the others have labelled (in particular older games PEGI retroactively applied a label to).

"A centralised resource should also be developed: if any one rating system decides to label a historical game or indeed a new game as containing loot boxes, that decision should be communicated with others to ensure consistency across the various rating systems," he said.

"Such cooperation can act as a safeguard to check each other’s mistakes and omissions and enhance the accuracy of the labelling process."

[Correction]: This article originally stated that when giving the ESRB the benefit of the doubt on retroactive labelling, it failed to attach a label to 9 games that PEGI did. That number has been corrected to 4 games.