Friday, January 24, 2014

APB Storm Watch; dealing with DDoS attacks

Here’s what’s going on right now:

For the last few weeks miscreants have been messing with APB by attempting both small and large DDoS attacks daily. The attacks range from small scale attempts, all the way to Derp Trolling taking down APB during their big take-down spree against Eve Online, LoL, Steam and (on an aside, if anyone has any information about these events, we, and the FBI, are all ears and we will share more info on how the community can help out too).

In the meantime, choosing a mitigation response that is both effective and doesn’t destroy performance is hard, but that part is complete and a first solution has now been deployed. Unfortunately there is no free lunch; in this case, mitigating attacks comes at the cost of higher latency in the short term. 

We are now working with our provider to optimize the mitigation to bring pings back down with DDOS mitigation turned on. As if our job wasn’t hard enough without this extra layer of work, thankfully our team has added this challenge to the long list things they just ‘have to do.’

Being DDOSed sucks, but at least dealing with it is our job. Unfortunately for you, this whole mess is harming how you’re trying to spend time when you should be just relaxing and having fun in APB.

To that end we’re sending out 10 days of Premium to everyone who logged in to the game anytime between January 1 and January 23. It doesn't make up for the fact that you had to go through this with us, but hopefully it helps you get back on your feet and back to enjoying the game.

Along those same lines, let’s talk about the game engine's performance. This is an ongoing issue that we’ve made great strides toward solving. Additional deep network logging in the last four weeks helped us identify several deeply buried culprits in the game code itself, and we are rolling out code fixes to those issues in the next couple of patches. In December we rolled out more Overkill servers to Joker, and we’re rolling out two more this month (though the DDOS attacks have disrupted that schedule a bit).

Yesterday we also found and corrected an issue with the mail system. This change comes with certain limits on mail volumes users can send, so we’ll be looking at the effects of that going forward. This may cut down on scamming; it may also potentially impact players who use the mail system as a pseudo-inventory. Have any thoughts on this? Please share them in the comments so we can make an informed decision on how these tradeoffs might affect your quality of life. Edit: This change will not impact your stored mails or your ability to continue to store mails or attachments.

Finally, our system-wide Unreal engine upgrade is coming along on schedule. Currently progress is being made on rendering, the lighting system and the district/client interactions. I hope to share something more tangible in the coming weeks so you can see just how far along it’s come.

And while we’re smack in the middle of a Winter Warmup event, we’re also looking ahead towards Valentine’s Day events, which you'll see in a couple weeks.

Lastly, we've also noticed recently that the number of new players logging in for the first time has steadily increased. If this means you, welcome to the game! Soon we'll be rolling out early game content to help new players such as yourself find your bearings in the sometimes harsh and unforgiving streets of San Paro.

Thanks for reading, and thanks for playing APB!

Tl;dr DDOS solution in place but causing high pings (which we're working on), 10-days Premium free for everyone logged in January so far, engine still being upgraded, fixed a mail abuse bug, performance being worked on, more overkill servers, and more in-game events around the corner.