End of Week 10 Update : Busting some Punks?

Alright - the time has come to start having the rest of the team start chiming in on the development to help shed more light on what's actually happening under the surface. First up is Johann, who originally had worked on the PB integration for the original release, and is now responsible for the overall anti-hack/anti-cheat detection for the game as a whole. My plan is to let key members share their ideas on this blog regularly, and hopefully that will spark some interesting discussions.

Johann - take it away:

----------------------------------------------------------------------------------

Busting some Punks

by Aphadon

 

Hi, I’m Johann aka ‘Aphadon’ - previously a Living City / AI Engineer for Realtime Worlds, and now a ‘Pretty-Much-Anything’ Engineer on APB Reloaded. When TechMech recently asked the team if we’d like to share anything on the blog, I thought I’d take the opportunity to discuss in greater detail our plans for addressing the cheating problem. This is a subject we are quite passionate about, and while it may be virtually impossible to ever win the war against cheaters completely, we certainly intend to give it our best shot.

But before I get into the details of where we stand with that now, let me start off by explaining a bit of the development history. 

We originally integrated PunkBuster into APB in January 2009. After this was completed, it was disabled and essentially left untested until near the end of Closed Beta (~ April 2010), since it was not deemed an essential feature to enable before launch. However when we finally did turn it on, we were caught off guard by a whole bunch of unexpected issues. Many people were getting kicked due to PunkBuster configuration problems, the servers were hitting lag spikes upwards of 10 seconds every few minutes (we used to joke internally and call them “lag mountains” since they were too big to be called spikes), and some people even reported BSOD’s starting on the day we turned it on. So in a nutshell,  it was quite a mess.

We were forced to disable PunkBuster again straight away until we could fix it, and ended up launching the game with it turned off. I offered to help out at this point since the developers in charge of PunkBuster at the time were tied up with other work, and I spent the next two months working closely with Even Balance to track down and fix the majority of issues. We eventually managed to turn it back on without issues for a few weeks before the servers were shut down, and -  contrary to what the aimbot sites would have you believe - during that period PunkBuster was able to successfully detect every single commercially available aimbot for APB. Over 1000 cheating players were silently identified and logged by the system, but unfortunately RTW was already having financial difficulties by then and we never had the opportunity to get our hands on the list of cheaters to ban them (since once you stop paying the people with the info, they are unlikely to give it to you).

Fast forward to today, and we’ve just completed a full upgrade of PunkBuster to the very latest version. I’ve been going over the issues we’ve seen before, and Even Balance is confident that all of them will be fixed in this version, along with new and improved detection methods and many other behind-the-scenes improvements in addition to Punk Buster alone. We also plan to have anti-cheating enabled on all servers from the very start of Closed Beta, allowing us to get as much testing done as possible. That way if any more surprises turn up, we should have more than enough time to fix them.

Ok, the non-technical people may wake up again now.  The next bit of information I’d like to share is our proposed banning policy. This isn’t set in stone yet, and some parts of it may not even be finished during beta, but this is what we would like to work towards: 

If a player is positively identified as running a known aimbot, he/she will be immediately kicked from the district and a notification message will be sent to all other players in that district. The cheater’s account and PC will be immediately banned for a period of time, during which the person will not be able to play using that account on any computer. If an aimbot is detected a second time, the account and computer will immediately be permanently banned, all monies paid will be forfeited and any upgrades or customizations will be revoked. 

This might seem quite harsh, but there really is no excuse for aimbotting and this time around it will be treated with zero tolerance. We believe this policy will cut down on cheating significantly, and ensure a fairer and more fun experience for everyone.

Next week I believe Qwentle - our Designer - should have a somewhat more light-hearted post to share with you, containing some juicy tidbits about what we’ve been up to recently (aside from plotting the doom of cheaters, that is).

Cheers

Johann / Aphadon.

----------------------------------------------------------------------------------

There you have it. Many thanks to Aphadon for sharing, and as he is alluding to, there are actually in this build going to be things other than Punk Buster to help keep the playing field level. What are those? Who knows :) What we all do know - we take game cheating extremely seriously. At some point we want to introduce bullet-proof vests, and for that to work, we clearly need headshots also be something you can actually achieve.

And everyone should realize that downloading hacks (even commercial ones) almost always entails installing a keylogger and malware on your own machine. Why anyone would ever do that is beyond us.

Til next week,

End of Week 4 Update : UK Studio Strategy and "Hackers and Cheaters"

First things first; the Scottish snow really sucked this past week (link). Being stuck in transit between London and Scotland made me really appreciate some of the lovely California weather we have come to expect. But once I made it up there, I do have to say it was worth the very painful trip getting there.

The talent we have been able to find in the UK and have been contracting with has been really great, so our next step is now to set up a new game studio in the UK in February 2011. Right now we are just debating WHERE to put such a studio as an extension to our US operation. The clear front-runners at the moment appear to be Edinburgh (near Weaverly Station which would put us right next to Rockstar North of all companies) or Cambridge (near JagEx and others). Other options include Dundee, Liverpool, Newcastle, Birmingham, Manchester or London/Guildford. Each of these offer various advantages (and in some cases various incentives), and we are spending the next four weeks determining the right location both for us and our staff. I will keep this blog updated with our progress as we evaluate all our options (and I might even invite people to make suggestions) before we pull the trigger.

Hackers and Cheaters

Now on to the main issue this week. Hackers and Cheaters. This is a tricky topic to talk about in anything except "hushed tones" and using code-words and only partial information. But I will give it a shot, so that during the rest of the relaunch process we have a basic framework for the discussion when the topic comes back up again.

First off, let me define two separate terms; we usually refer to people that do bad things in-game as cheaters, spammers or griefers. We really try hard not calling them "hackers" - only because hackers tend to connote much more sinister stuff like DDoS and Malware attacks. But sometimes we slip, and we call the cheaters "hackers." But I digress.

So what can we do about Cheaters in APB? Well there are a ton of "cheat-busting"solutions out there such as NProtect, HackShield, PunkBuster, XTrap etc. All those have various advantages and disadvantages, though the most annoying disadvantage is of course that none of them are perfect in catching all cheats, some of them are easily circumvented, and some tend to interfere with normal gameplay to lesser or greater degrees. But everyone pretty much agrees that using some form of execution protection is a must, and the newest types of protection are now much more custom and game-specific, rather than generic. That's generally the type of protection we are now moving toward.

First the good news; as a mostly server-driven game APB is naturally much more resilient to cheaters than most other F2P MMO's. There is a good reason F2P games usually don't use server driven architectures; they cost a lot more per CCU to operate than P2P communication solutions, so in APB we are going to run a giant experiment to basically determine if hardware costs/specs have progressed far enough to make F2P server-driven games financially viable. Our current calculations seem to support that it will in fact work. By avoiding using P2P to transmit action data, the ability to attack or hack the core APB functions drop significantly. This is true for all types of hack attacks except one key category of cheats; aim-bots. Because aiming relies on visual data that by necessity HAS to exist on the client (after all - if you didn't have this data, you wouldn't be able to shoot anyone, since you wouldn't be able to see them) it makes them the trickiest category to protect against.

But even here there are some good news; before APB was shut down most aim-bots were actually actively being detected. Unfortunately (or fortunately) the data was not being acted on for various reasons (analysis was on of the reasons).

There is another issue - most people who lose any match in any shooter game tend to presume that they lost to a cheater, even when they lost fair and square. Part of human nature I suppose, which means cheating is actually over-reported. When we have run live in-person tournaments we basically end up concluding that there ARE many people out there who are basically MACHINES, and play at a level many many steps above where everyone else is.

On to hackers. Well we have been particularly aggressive trying to deal with hackers, since in some cases they have made our lives a living hell. The most interesting case being the one in 2007 when an employee of an unnamed telco-company decided to DDoS us, and basically the DDoS lasted during the hours he was on the job every day. Eventually the particular perpetrator was tracked down.

More recently we were hit by a string of DDoS attacks starting on December 25, 2009 (thanks for that Christmas present). That was actually the start of more than 50 attacks over the last 12 months alone.


To combat DDoS, which we now seem to be pretty good at doing, let's just say that "we do things," and in some cases "we use some outside providers" and internally "we have some good stuff," and we finally collaborate with some pretty mean people to help us out bringing people to justice. Just two days ago the newspapers in Turkey released this story: http://www.sabah.com.tr/Gundem/2010/12/10/ikileaks (here is the somewhat cryptic Turkish to English Google Translation). These guys were part of the group that were specifically targeting Knight Online.

DDoS attacks are clearly annoying (and very destructive to the business), but they are not as dangerous as the targeted malware attempts by Chinese goldfarmers that have become prevalent toward game companies (think of Stuxnet, but aiming for game companies instead). Sometimes it boggles the mind that there are people out there with an actual business model that requires some form of hacking to improve the margins of their operation.

That's a quick overview of the situation. Apologies for not being more detailed given that we have to keep things quiet in order to be effective. But once the game gets back up and running I am happy to share more specific details about specific items as they arise when we are live.

Cheers, Bjorn / TechMech