On Thursday, May 5th at 1:17 PM CST version 4.7.1 of Fleetio Go on Android was promoted to Production and began to be available for update on Google Play. We received our first customer report of the app crashing on launch on Friday, May 6th at 7:43 AM CST. Upon investigation, we determined that a development build had inadvertently replaced the 4.7.1 release that was originally staged. We then rolled back to the previous working release version 4.7.0 and deployed the hot fix as version 4.7.0.1 which was made available for update on Google Play at 11:40 AM CST.

With every release of Fleetio Go we perform rigorous regression testing of the app to ensure problems like these don’t arise. This testing occurred with the 4.7.1 release that was originally prepared for deployment, but a development build inadvertently overwrote the staged build from a developer’s local machine after QA had already occurred. We are taking the following steps to safeguard against these type of scenarios in the future:

• Restrict the access in which a build can be uploaded to Google Play for release.
• Change our promotion process to take advantage of alpha and beta lanes to ensure the build being tested is what is deployed to production.
• Utilize phased rollouts and beta testing groups to ensure app quality with a smaller user base before a release makes it out to all customers.

We realize that Fleetio Go is mission critical for our users. We would like to apologize for the adverse effects this outage had on our customers. As always, we'll continue to analyze the events involving this incident and will use these learnings to improve Fleetio.