Cloud To Cloud Migration

Summary

Our customer, a large investment company, was acquired by a competitor. This required their AWS Cloud platform to be migrated and integrated into the new owner's platform. Because the new owner is based in the United States, we had to adhere to both European and US regulations. This was quite a challenge. Meanwhile, we still had to maintain the existing platform, and keep the current flow of data and services, including payment services, running.

The Challenge

To implement this migration, we went through several phases. Our first approach was to use an internal tool that the new owner had developed. This tool was designed to help developers quickly onboard new AWS accounts and applications. All of this was backed by their internal Cloud platform and security teams. As first order of business, we had to investigate how to migrate the most critical services, the databases. We had to find a way to migrate the data from the old platform to the new one, while keeping the data secure and compliant with European and US regulations. Technically, this was not a big challenge, because AWS provides ample tooling to migrate databases. Our implementation on a technical level was to create a snapshot of the database, copy it to the new account, and then restore it. It was then that we found out that this internal tooling did not support the restoration method. After some investigation, and with help of the Security and Compliance teams, we decided that this was a fruitless effort. Because of the myriad of regulations in the financial sector, both in the EU and the US, we had to find a different approach. Now because we could not use the internal tool, we also had to find a different way to create a new platform and organize our accounts and cloud-based applications. This was a challenge, because we were originally provided with a sort of desired-state for our new platform, but now we had to redesign our platform from scratch. In the meanwhile, payment services had to keep running, and we had to keep the data flowing. This started to become an issue, because the migration was delayed because of aforementioned issues.

The Solution

After quite some deliberation, we managed to find a satisfying solution to all of these issues. Luckily, another company was acquired a while before our customer, and they managed to migrate their platform to the new owner's platform.
We contacted them, and they were willing to share their experiences with us. They used Terraform to create their new platform, and managed to onboard all their accounts and applications in a way that was also useful for us. Though this process of migrating is still ongoing, we're able to use this process as well.

For the payment flows, we had to implement a custom solution. As shown in the diagram, we had to extract the data from the old plaform, sometimes directly from the database, and sometimes through a Kubernetes cluster, where the data was generated by an application. This data was then sent to an S3 bucket, from where it was picked up by a Lambda function. This function processed and/or enriched the data, and put it in another S3 bucket. From there it is picked up by a secure data transfer service and sent to the platform of the new owner. This was a complex process, because we had to work closely together with the teams on the new owner's side, who built their solution in Java, and are still restricted to using on-premise servers and databases.
Once the data was picked up on their side, it was processed and the payments were reconciled.

For designing and building the the data transfer application, we used the Cloud Development Kit (CDK). This allowed us to create the infrastructure as code, and deploy it in a secure and repeatable way. We used a combination of Python, Node.js and TypeScript to create the Lambda functions and the data transfer application. This allowed us to quickly iterate on the application, and to create a robust and secure application, and to properly communicate with other teams and responds to their requests. The data transfer application was designed as an Event Driven Architecture, where the data was processed in small chunks, and where the data was enriched and validated in a secure way.