AUTOMATED COMPUTE GRIDS

Summary

A research institute was triggered for Cloud migration because the old compute grid was end of support / service life. This international team of researchers used this compute grid for various kind of scientific use cases. Some of these workloads ran for long periods of time. By running an MVP with Azure Compute and Azure Batch showed the immediate value of using the public cloud platform for these workloads. This made their decision to move these activities to Azure.

The Challenge

Create the tools and services for researchers to run their workloads in a consistent way on Azure. Help sizing their workloads and make use of cloud services, like Azure Batch, Azure Functions and more. Cost and security controls should be in place. able to autonomously deploy changes to various environments on this platform.

The Solution

At first we have created the solution design that defines all services, components, networking requirements, naming and tagging conventions. We created their complete landingzone with Terraform, Ansible (+AWX) through automated CI/CD pipelines in GitlabCI. For different research groups we defined blueprints, which could automatically be deployed based upon input parameters. The blueprints contains automatically created dashboards and billing alerts so research project leaders automatically got control on their budget and resource allocation. Secure access to the platform was provided through on-premise Site-to-Site VPN connectivity as well as a Point-to-Site VPN solution for remote workers. By making use of these cloud services we achieved a significant reduction in the durations of these workloads while also saving costs. For example, the runtime of a specific calculation was reduced from 2 months to 46 hours.