Open Beta Release On Hold until next weekRelease 1.5.1.85 maintenance today 4pm PST

Here is Thursday's update - and the aim here is clearly to be crazy-transparent with what we are doing, so for those not that interested in the technical stuff, the short version is; Open Beta is on hold until next week. For Closed Beta users we are going to roll out patch 1.5.1.85 at 4pm PST / 7pm EST / 24:00 GMT / 01:00 CEST today

For the longer version continue reading below;

The Closed Beta issue we have run in to causes the disconnection of all players on a server. Unfortunately it happens to be of those vexing problems that only shows up at full production load, and an issue that leaves almost no traces of its origin in our logs. During QA and testing loads the issue doesn't materialize (throw 50 people on a server and everything seems just fine), even with large static loads and testing the game from multiple locations.

Specifically the issue manifests itself where districts with 60-100 players suddenly lose IPC (inter-process communication) functionality which then leads to the district dropping all players and the district crashing. This happens anywhere from 1-6 hours from the district starting up. Normally this would be easily grabbed by our crash processor and it's usually a pretty straight forward to identify the location of any GPF, however with this particular issue, the logs don't show anything that helps narrow down the scope other than it's part of the changes for Open Beta.

Rolling back several versions is certainly not the preferred option as it puts us firmly back into Closed Beta, but could be done (it would take us back to somewhere in the 1.5.0.7x range), so instead we are making one last push to get through the current issue with the production load, and we continue to ask our Closed Beta testers to help hammer the servers over the weekend so we can reevaluate where we are on Monday.

Today we are also pushing ahead with release of 1.5.1.85 which further adds information to what is happening, fixes an unrelated client-side GPF that happened when a song changed tracks (a general Audio GPF), and potentially MIGHT solve the district disconnect issue (but again, won't be visible until it's on full production load for a few hours).

Once we enter Open Beta, the patching schedule becomes very different (and slower). We will also at that point recruit our first "Test Monkey Crew" for APB - basically volunteers who will get early access to builds in return for running live tests on a public test world, specifically designed to catch these types of hard-to-QA issues. The TMC gets various in-game benefits for being early testers. But again - that system goes in to place after Open Beta goes live and Closed Beta has ended.

I will continue updating this blog daily until the the game is in formal Open Beta. Also while this is being worked on, be mindful that very few of our tech staff will be active on the boards or blogs, since this is pretty much an all-consuming issue for everyone.