Monday, July 21, 2014

Fighting Fair in APB Reloaded with FairFight

...and the sad naming and shaming of 200+ players.

I bet many of you know “FairFight” as the anti-cheat system currently in use in several top tier games, including Battlefield 4 and Titanfall.

But I bet you don’t know that the Reloaded team has been working closely with the FairFight team for the past several months to integrate and optimize FairFight in APB.

When were able to combine our existing tools and systems with the FairFight system, we were suddenly able to review a vast range of information and get much better visibility into what players were doing in the game.

Some of these new insights made us very happy; for example it turns out that the numbers of actual cheaters in APB is much lower than the community would believe. But some insights made us terribly sad; players with as many as 8,000 hours of playtime were caught red-handed cheating.

What kind of cheating did we catch? The current combination of tools can now catch a wide range of activities using almost every conceivable method of detection. The FairFight system has been great, since it has acted as a “key” to let us pull together a lot of disparate the data into one single effort. As a result we found lots of players who actively had a cheat running, but, sort-of didn’t use it much, and often appeared to be running mostly legit, while relying on their purchased cheat tool “to cheat just a little bit.”

Of course any cheat is a gross violation of our terms of service, and just as when pregnant; there is no “a-little-bit-of-pregnant” category in the human gestational cycle, just as there is no “a-little-bit-of-cheating” category when playing a fair game.

The sad naming and shaming of 200+ players

Today as we formally move out of “stealth mode” to ensure a fair APB game going forward we are also publicly listing a sampling of 212 people whom were detected cheating sometime in the last few weeks or months. The list of names can be found here. It’s not comprehensive (the total number of bans are several times this list), and it includes players caught using several different detection methods. A few were even tagged by the “trusty” old Punkbuster as well, though FairFight enabled us to truly be efficient in tracking down all these miscreants.

In the “name and shame” section we also opted to not only list the character names of those who cheated, but decided to take it a big step further and list the time spent playing the game, player rank, server, clan and the range of real cash each player had spent in the game before getting terminated.

APB Cheat Stats

There is some good news that came out of our intense cheat review; on a daily basis, the number of cheaters who log in to the game stayed pretty constant around 1.2% of all players (some of which were re-rolls). Inversely 98.8% of players in APB played on a daily basis without any trace of any cheat (which we are very thankful to see)! So as usual, it’s the 1% who is messing up the world for the rest of humanity.

For some even more startling facts; 60% of the players banned in this sample were paid players. That’s surprising on several levels. For comparison, about 10% of the monthly unique players in APB make a new purchase in the game in any given month. On a cumulative basis about 30% of all players playing the game have paid money for something at some point during their account history.  So in simple terms, paid players turned out to be 2X over-represented in the initial ban sample, compared to the APB population as a whole.

Another startling fact; paid players who were banned had on average 260 hours of play time in the game, had played for over 1 year, and had on average spent $180 on virtual goods.

Now clearly they got their monies worth already (in comparison they could have chosen to pay $59.99 for 20-40 hours of play time in a traditional single-player game, or pay for 12 months of MMO subscription time for about that amount of money) and it's crazy that they chose to have themselves eliminated from the "sport of APB" by cheating.

A Few Bad Apples Spoiling All the Fun

So why has there been such a giant perception that cheaters are rampant in APB, if the ratio of cheaters, even before FairFight went live, was only around 1% (when we were just observing what was going on during stealth-mode)?

Well, there are really two explanations.

First, yes the few cheaters that exist have a disproportionately negative impact on the matches (snapping, appearance of cheating etc.) partly BECAUSE they also tend to be active players. So they do affect a much larger number of matches than 1%.

Second, it also turns out that APB is a REALLY HARD GAME to master and play. In spite of what everyone says, APB takes some serious skill (much more so than money). Locational awareness, amazing twitch-skill, strategic planning, and a flawless group to play with are critical to success in the game. And in fact we found ourselves analyzing some matches where EVERYONE accused everyone else of cheating, but no one was actually cheating!

A few of the players in those matches were in fact just THAT much better at the game than everyone else. It’s a really tough feeling when you realize you got crushed in some inexplicable way, and a very common immediate reaction is to accuse everyone of cheating. Statistically there is a very good chance they actually were not cheating the last time you played them. They probably (though not always of course) were just better players, a better group etc.

Cheating “just a little bit”?

Even more fascinating, as we go through the list of cheaters, we realize that often they did NOT use a cheat to crank all their stats (well of course, some idiots did), and our only conclusion must be that some of them have been using cheats as a ‘crutch,’ probably because they thought the other people around them were beating the crap out of them (even if statistically they were likely NOT to be cheating).

Notice that the overall K/D ratio of the cheaters in the sample name and shame list is around 1.7 – meaning for every 17 kills the cheaters died 10 times. That’s around 70% ahead of break-even, but not massively better than players with equivalent time spent in game.

And finally, we also found accounts where it appears the players have been playing legit in the account for a long time, and then for some reason or another decide “to just try a little cheating.” That’s sort of like “just trying a little meth.” Your mom and your local DEA officials always tell you “don’t do it” to the meth. And we say the same about cheats. Don’t do it. One day you will be caught.

Should we enable Purgatory?

So while we are very glad to see some of our community members go due to these flagrant violations of our terms of service, one idea that we keep kicking around is to potentially enable a “Purgatory” District.

The idea is that cheaters don’t get banned right away (which of course doesn’t happen since the system runs on various delays), and instead freshly detected cheaters get kicked into special cheaters-only districts, where they can continue playing against other cheaters without initially realizing they have been tagged (or whole groups get thrown in there as long as they have a cheater on their team). Titanfall has implemented a similar system, and it seems to work reasonably well for them.  The main reason for us to consider doing this is purely for research. We benefit from continuing to capture data from those who cheat.

But we also realize this is a sensitive item for all players. We’d be curious to hear your feedback on this type of district (maybe it becomes a ‘mass-spectate’ thing where you can view them all cheating each other)? No final decision has been made so far about this type of new district type.

In fact – this is NOT free!

In jest I had added a post-script to last week’s ‘teaser’ post about making an announcement that said “hint: it’s free.”

Make no mistake. Spending THIS many resources to combat cheating, even with an excellent partner like FairFight (who have a great team, but of course needs to charge for their services), combined with all the other related internal and external tools involved, it all takes a massive amount of resources away from the Reloaded team. And again puts us in a position where we have to prioritize anti-cheat efforts which directly takes away critical resources that could be better spent on new shiny features and new systems in the game.

But in this case we feel the investment will be worth it in the long-term. And we are hopeful we will continue getting great support from our devoted players who spend countless hours in the game every month.

Final words

The other big project that’s ongoing is of course the engine upgrade, which continues moving forward at a steady pace. However, what started as one project has now morphed into two related projects that have much larger teams attached. The details of which I cannot release “next week,” but I promise that it will be “sooner than SOON™.” And I am personally super-thrilled about what our teams are already working on, and again, I hope to issue a really detailed update as soon as I am able to disclose all the details.

In the meantime, we continue being incredibly thankful for all your ongoing support, and hope to share more exacting news as it becomes available.

And just to cap it off – as an APB player, did you know that APB Reloaded is in fact the 19th most played game on Steam (out of 2750 total games distributed) of ALL TIME, just ahead of Borderlands2 and behind Call of Duty: Modern Warfare 2? At least according to this Ars Technica article (and which, from our internal numbers, we can certainly corroborate, if not ponder that Ars may have under-counted some stuff). That makes APB more played than 99.3% of all other PC games (!). There is some additional data in this Ars Technica update as well.

In fact turns out Reloaded Games have 2 of the games in the Top-100 out of 2750 games: APB Reloaded at No. 19 and Fallen Earth at No. 94.

In short, we are very much looking forward to when we can share our next set of exciting updates.

In the meantime – play fair! And if you want to see the conversations that this has sparked – then you can go here to check out threads like these in our social district forums.


Friday, July 18, 2014

Next week

Since we've been very quiet recently, we wanted to make a quick post ahead of the weekend to tease something we will be sharing with everyone next week. 

As you know our teams have been working very hard and very long on several different projects, and at least in one of those cases we are ready to make an announcement next week that we think will make most people very happy...

What we are going to announce is not the biggest project we have going at the moment, but it's a good one, and worthy of some detailed discussion.

See you next week!


(hint: it's free)