Valve targeting ‘off-topic review bombs’ in Steam user reviews

Valve says it’s refining its approach to how customer reviews are handled on Steam, announcing today that it plans to identify and remove “off-topic review bombs” that might affect user review scores.

Review bombing has been a longstanding problem for digital storefronts with user reviews, and Valve has tried to combat their impact on Steam with other methods, including reviews of user reviews and graphs that display “temporary distortions” in user feedback. But to date, Valve has still factored those reviews into the game’s overall user score.

And while some review bombing situations may be tied to legitimate issues with games, many are unrelated or tangentially related to the product itself. Games like Firewatch, PlayerUnknown’s Battlegrounds, and Dota 2 have been targeted with waves of negative reviews for incidents well beyond the scope of the games themselves, either over political and ideological disagreements or decisions made in other games by the developers. Horror game Devotion is the most recent high-profile example of this behavior; the game’s Steam page was flooded with negative reviews after users discovered an in-game poster mocking China’s president.

All of which leads to an obvious question: How does Valve deem something “off-topic?” Valve explained in a post on Steam how it determines that, and the mix of automated tools and human interaction that will help confirm it.

We define an off-topic review bomb as one where the focus of those reviews is on a topic that we consider unrelated to the likelihood that future purchasers will be happy if they buy the game, and hence not something that should be added to the Review Score.

Obviously, there’s a grey area here, because there’s a wide range of things that players care about. So how will we identify these off-topic review bombs? The first step is a tool we’ve built that identifies any anomalous review activity on all games on Steam in as close to real-time as possible. It doesn’t know why a given game is receiving anomalous review activity, and it doesn’t even try to figure that out. Instead, it notifies a team of people at Valve, who’ll then go and investigate. We’ve already run our tool across the entire history of reviews on Steam, identifying many reasons why games have seen periods of anomalous review activity, and off-topic review bombs appear to only be a small number of them.

Valve said that some topics closely tied to games, specifically digital rights management (DRM) software and end-user license agreements (EULA), will fall under the off-topic category.

We had long debates about these two, and others like them. They’re technically not a part of the game, but they are an issue for some players. In the end, we’ve decided to define them as off-topic review bombs. Our reasoning is that the “general” Steam player doesn’t care as much about them, so the Review Score is more accurate if it doesn’t contain them. In addition, we believe that players who do care about topics like DRM are often willing to dig a little deeper into games before purchasing - which is why we still keep all the reviews within the review bombs. It only takes a minute to dig into those reviews to see if the issue is something you care about.

And here’s where it gets trickier: Valve’s system for identifying off-topic user reviews will also remove neutral and positive reviews from the overall score calculation if they were made during the review bombing window. Valve says “our data shows us that review bombs tend to be temporary distortions, so we believe the Review Score will still be accurate, and other players will still be able to find and read your review within the period.”

More detail on the change is available on Steam.