Optimization Magic - Part 2

(link to Part 1 of Optimization Magic)
(a slightly circular reference to forum discussion)

First; barring any unforeseen issues we are launching 1.10.2 globally in a few hours (of course with the caveat that anything could be rolled back). We expect 1.10.2 to result in a 15% server-side performance improvement across the board on the existing hardware. So while that does NOT completely address the current performance spikes, it should generally improve the experience and allow the game to run more stable at its current 25 FPS server-side tick rate. Also 1.10.2 is the first of a series of optimization focused updates.

SPCT and trashing the new hardware.

Second; a big thanks to the SPCT (San Paro City Testers) who jumped in and tried trashing the new software and new server hardware on Saturday as well as some invited guests (and for those who don't know, you can join the SPCT program to participate in the public tests of new builds by applying through our forums. Keep in mind that part of your entrance exam is figuring out HOW to apply...)

It was a pretty fun event since we basically put up the new hardware, upped the CCU to 100 per server core, then unlocked the FPS-lock on the new software, and asked everyone to show up in the same spot and "shoot and holler" and basically perform as many close encounter actions as they could (which in turn trashed client FPS for most players, since rendering 100 other players within 90m actively shooting at you and setting off explosions turns out to have its own client FPS issues, though that's separate from the server effects).

In short we tried creating the worst possible scenario for the game (server wise) and we think we pulled that off pretty well.

The good news is that on the new hardware running 1.10.2, we saw a stable server side FPS around 40 FPS (basically frame ticks stayed around 25ms), with the biggest spikes at 28ms (which is a server side FPS of 35FPS).

This means we should be able to comfortable return to 100 CCU and set the server tick to 35FPS using the new hardware combined with the upcoming software version of 1.10.2.

You can see the video of the carnage here:

Here is the resulting performance graph from the test above:

In fact, as an added bonus, during the 1.10.2 maintenance we are going to put SOME of the new hardware into production for SOME (not all) of the Financial Districts on US-West.

Due to some quirks in the configuration system, we will not be able to change the 25FPS limit until ALL the districts have new hardware, but at minimum US-West should start seeing more and more stable performance in Financial as a result. After all the US-West Financial have been upgraded, then we will move on to Waterfront.

Next steps

Next step is to evaluate 1.10.2 after the software patch, find other obvious performance issues, and then review what other optimizations we will focus on next. After that we will monitor and measure the effects of the OverKill hardware being put in US-West in production, and hope it doesn't physically explode or blow up, or catch on fire in our datacenter given it's 'pushed' pretty hard. 

Presuming all goes well, we will expand the roll-out of the new hardware and progressively replace Financial and Waterfront in each of the worlds, while at the same time continuing software optimization.

One of the challenges on the client side remains the giant particle-effect slow-downs. Unreal has a known issue where unlit particles cause large FPS hits. Epic addressed some that in the 2009 build of the engine with the introduction of the Light Mass system, but unfortunately APB Reloaded pre-dates that particular engine build, and does not have access to that lighting system.

We have various ideas of how to upgrade or replace parts of the engine over the next few months, and some are more radical than others (maybe just creating a whole new APB game using the new Unreal engine instead?.... :) ), but the basic direction is to continue with optimizations in any way shape or form we can think of, and to continue keeping you informed via the blog and forums as we roll out these changes.

See you in San Paro!

/TechMech