No EU Overkill?
Since we first announced Overkill back in early 2013 (http://apbreloaded.gamersfirst.com/2013/02/optimization-magic.html) - we purchased, built and deployed these ultra-optimized machines in the US in response to your demand for better server-side performance. The Overkill machines are deployed in Colby and Joker provide increased server tick-rates (30 fps compared to 25) and higher maximum district capacities (50v50 compared to 40v40).The graphs below illustrate how Overkill performs at higher district capacities compared to the standard spec:
(a) Standard (original) server running 4 district instances at 80 capacity, with 40ms frame time/25 FPS.
(b) Overkill server running 5 district instances at 100 capacity, around 33ms frame time/30 FPS. In both cases, frame time starts to increase exponentially as district population exceeds certain amounts (70 in standard servers and 90 in Overkill servers). Since we try to maintain a minimum frame time 25 FPS (40ms frame time) for ideal performance, there is always a careful balancing act between minimizing latency and maximizing district capacities.
Why haven’t we simply launched them in the EU?
It basically boiled down to the logistics of setting up these highly-customized machines in Frankfurt (where our EU servers are located). This isn't like building a PC at home – birthing an Overkill server overseas takes lots of work putting together the individual components, escorting it to the proper location, racking it in the data center and configuring it for use. Unfortunately once it's in place, it's a hard piece of equipment to maintain, especially relying exclusively on remote assistance.Therefore we have been on a continual hunt for a more "enterprise-class" solution that was better suited for remote datacenters, especially when we need to deploy them in Germany, where we don't have our own team on the ground.
We never lost sight of upgrading the servers that our EU players call home. The good news; we are launching the final "Gen 2" servers for testing in the EU later this week!
APB in the Cloud
Over the last several months we have been working with several potential cloud-hosting or hybrid hosting providers – like AWS and SoftLayer – to experiment with alternative server solutions. "The Cloud" of course offers benefits like a selection of ultra-modern and high-performance servers already located near existing APB infrastructure. Unfortunately, "The Cloud" can also be unpredictable and unreliable when compared with "bare-metal" hosting.AWS offers a "pure" cloud solution, whereas SoftLayer provides a "hybrid" hosting solution where they actually spin up new physical servers and give us more physical control over actual "hardware."
Still, finding the right servers able to handle the massive demands of running APB district servers has been no easy task. So far we have put several different configurations through real-world tests.
How did they stack up?
(c) Amazon AWS C3 2680v2 (10 cores, 2.8Ghz) running 6 district instances at 90 capacity.
(d) SoftLayer 2690 (8 cores, 3.0 GHz) running 6 district instances and 100 capacity.
(e) SoftLayer 1270 V3 (4 cores, 3.5 GHz/ 3.9Ghz TURBO, thanks to hybrid hosting) running 3 district instances at 100 capacity.
The clear winner here is the SoftLayer 1270v3 at 3.9Ghz (TURBO). That's a bit surprising, since technically it's the "smallest" of the three processors types (AWS C3 is a E5-2680v2 with 10 cores at 2.8Ghz, and the SoftLayer E5-2690v2 is an 8 core 3.0Ghz processor) and by far the cheapest.
However, Unreal eats processor clock for breakfast, and does not properly use multiple cores for its processing tasks. So the smaller processor actually wins out.
For the 1270v3 even with each district at full 100 capacity, frame rate remained close to 30 FPS, and dropped only very occasionally to 25 FPS. This is higher than the current servers’ frame rate of 25 FPS under the best conditions (down to 18 FPS at 80 capacity), and actually better than the US based Overkill servers which run closer to 22 FPS under the same 100-player load!
Sadly, our experiments with Amazon (where we had a lot of hopes in Australia) did not turn out to work very well. While it's a fine piece of hardware, at these extreme performance levels the limited hardware tuning options made the AWS solution less suitable for this very unique hosting requirement.
Over all - the hybrid hosting options have shown to be incredibly promising and if you’re still reading along at this point, you’re probably wondering when you’ll be able to take these for a spin yourself. The good news is that we are configuring both EU1 and EU2 to incorporate the new SoftLayer servers, and will add 12 new districts running at 100 CCU max capacity in the live environment later this week. Once they are live, they will run through the rest of February and depending on our test results and your feedback, we hope this solution will be the final version we deploy for all EU districts.
New Server Structure in Preparation for Console Launches
So to end on a bit of a bombshell, and reward you for patiently reading about low level technical items, in the next blog post we will share the new game structure that we will implement over the next few months in preparation for the PS4 and Xbox One launches.The new upcoming World structure will feature three Central US Worlds (one each for PC, PS4 and XBox One) and three Central EU worlds (PC, PS4 and XBox One). Each new World instance will have greater capacity than the old ones to handle additional players, and we will no longer need to split players on a single platform (PC/PS4 etc.) onto multiple World servers. Han will remain as a PC world in Asia, and may transition to the new server technology if the EU tests go well.
Til next time,
Jobs
