Pages

Monday, April 17, 2017

Console Latency Issues and Network Outages

At the start of the weekend we began seeing an issue on our PS4 environment. We initially thought it was due to the recent patch or seasonal event but as time went on the issue spread to our XB1 environment as well. We have now identified the issue to be related to a new type of DDOS attack against our servers which spams our network and is causing latency and connectivity issues within the game.

Today we also received approval for the Xbox One to be patched with the seasonal event however we’re holding off on patching while the DDOS issues are present.  

We apologize for the disruptions and are working with our mitigation provider to combat the attacks moving forward.

Thursday, April 6, 2017

PlayStation 4 Update



With the soft launch finally done, we now have a new build being tested to resolve the key issues reported. The main issue so far has been the frame rate drop as new characters load in, which some players may have reported as lag. The cause is actually not “lag” (which would imply server or network issues), but the frame drops have to do with the custom character baking when new characters show up in the district, which is especially noticeable in the social district.

Frame Render

As you know APB Reloaded gives you the ability to customize your characters to an unbelievably high level (one of our favorite features), but the downside is that the system creates the custom character textures and shapes in real-time based on each player’s character data, and then pushes the content to everyone else who happens to be in the same district. The code breaks this complex process down into smaller components and spreads the work over multiple frames, but there appears to be an issue with performance on PS4 which causes some of the computations to take considerably longer than expected when combined with real world customized characters and the number of characters that load in and out of a district in a session (especially in Social). We are right now working hard on a new build that will dramatically reduce this effect, and have been working on fixes we can quickly get in to the first patch. Then we are planning a more extensive overhaul in subsequent patches to eliminate the issue.

Crash

Two thirds of the crashes from the first weekend were due to a single memory issue (again related to the complexity of real world custom characters), and we have identified the cause and are testing a fix in the live environment right now. Once it passes Sony QA we’ll push it out to live which should remove the main stability issue in that first patch.