Insights

Case Summary

Company B was left high and dry following the departure of their technical cofounder. This lead them to search for a team that could keep the application running, fix any existing issues, and implement new changes to the active app visited by their users. We jumped into existing infrastructure on limited documentation, and broken access to the production code, to ensure a fully functioning pipeline for new application development and a stably running code base.

The Problem

Company B had lost their technical cofounder and the code was left dormant for over a year. When Axon came on the scene the application code was in a repository separate from the original deployment repo, in a different state than the application in use by customers. In short Company B had an application that could not be maintained and was luckily still running. Made evident from a server crash several weeks into our engagement.

The Options

Company B could have totally abandoned the code base, but that would have lost years of time & investment, and they had active customers using the application. They could have just hired an application developer, hoping that the individual could tweak the existing deployment pipeline and restart the servers without any problems. This would have been a high risk, and would have ultimately resulted in down-time for the customers.

The Solution

Instead, Company B decided to work with our experts who made a plan to extract the existing value and maintain uptime. We took a very measured strategy of discovering the different resources being used, attempting to access them, then extracting whatever code or configuration was accessible. All of this was done unobtrusively, and left the existing resources running for the active users to interact with. Little did we know that our skills would be so necessary: several weeks into the initial engagement, one of their production servers went down. This of course killed the app, requiring one of our experts to jump in, with no notice, and restart the VM, touching the services to ensure they were back running.

A Bit More Detail

We started with access to GitHub and AWS. Technically we also had access to their original BitBucket, but following Mercurial’s deprecation this was useless. We pulled the code and compared it to the existing application. It was clear that what was actively used was quite different from what was stored in our migrated repository. We began the journey of trying to extract the application code from the EC2 resources. All SSH keys were lost, direct AWS connection was not enabled, and we could not restart the servers to fix any of this. We instead opted to create snapshots of the VMs/volumes and create our own duplicates. From there we were able to extract the code and docker images which became the foundation of our recreation of the system. Of course, it wasn’t just the EC2 instances we had to work with, In creating new operating environments we created new DNS routes in Route53, new S3 duplicates and the necessary CloudFront façade. We had to update the SES verified emails for new functionality, update CloudWatch to properly log and notify on health checks, and implement new Mongo replica-sets and Elasticsearch with proper synchronization. As mentioned, we also had to deal with an outage caused by a rogue process, requiring us to jump in and manually touch the servers since we couldn’t rely on the OpsWorks deployment. At this point we injected new SSH keys and also were able to get the services back up. We have continued working with the client to implement new application changes now that their infrastructure is stable. All feedback has been positive and we anticipate a long fruitful partnership with Company B.