Utilising Azure Functions for Python Projections

This blog post is aimed at a technical audience and assumes prior knowledge of C# and Azure Functions.

At Moneybox, we’re on a mission to help everyone save and invest for their future, by providing the tools and information to achieve their financial goals. In our app you’ll find tools like our Time machine, which forecasts what a customer’s savings and investments could look like in the future by changing their deposit amounts.

We categorise these tools as projections and as we looked for the ability to deliver quick updates and introduce new projections, it became apparent that we needed to evolve our current solution into something more robust, manageable and scalable.

Lifetime ISA Time machine

The problem

Python is the most widely used programming language for data science due to its simplicity and speed. It’s what our Data Insight team predominantly uses to build any complex calculations and projections for the Moneybox app.

The challenge we faced is integrating the Python code into our backend C# .NET codebase. We wanted to ensure that we could update the calculations over time and that these ran efficiently for the best user experience.

Existing solution — Python to C# conversion

Our existing system is simple; take the Python code and manually translate it into C#. This was conceived at a time when delivering the feature at speed was a priority and Azure Functions were in their infancy.

Although there is tooling available, this mundane conversion process is time-consuming and inefficient — with lots of room for error. We needed to maintain two versions, written in different languages, both requiring extensive testing to ensure they return the same results. Any subsequent changes made to the original Python code required a rewrite of the C# counterpart.

This worked but we needed to find a more mature solution.

Python to C# mapping solution

As a next step, we explored the idea of a Python to C# mapping solution. We built a new .NET solution with an outward facing API, which would host and interact with the Python projection code at runtime via a series of mapping classes.

Initially we found that this solution was inefficient. Any calls made to the new endpoints were delayed by up to 10 seconds. Keen to resolve this, we made a switch to import ‘Python for use in build scenarios’ via nuget and zipping up the Python projection code. These optimisations were successful and reduced the response times to less than 200 milliseconds.

This approach worked and was certainly a step in the right direction. It gave us the ability to run scripts through our usual infrastructure and test processes, and reduced the replication of code across two different languages. However this still didn’t feel like the right solution; it required a lot of setup and manual mapping between the Python and C# components.

Azure Function App

Introducing Azure Functions, a Microsoft cloud-based solution that offers a whole range of benefits. There was a bit of a eureka moment when this was first suggested. It offers all of the advantages from the previous Python to C# mapping solution, with none of the dependency mapping and setup, all within an environment still hosted inside our existing infrastructure.

The plan is to build a function app that will host each of our insight projections as individual functions. These functions will execute on simple event triggers, such as HTTP Requests, and can be called directly from the client or via our API solution.

One of the key selling points of Azure Functions was that they are “serverless”. Despite its terminology, this does not actually mean that the application is hosted without a server. Rather, it allows you to run code on-demand without having to set up App Services or Virtual Machines. As a Function as a Service (FaaS), this reduced management allows us to focus more on code than infrastructure.

Azure Functions currently support the following languages: C#, Javascript, F#, Java, Powershell, Python, and Typescript. The ability to create Azure Functions in Python was only made possible in 2019, provided that this is deployed in a Linux environment. This recent support meant that we could encapsulate the Python projections code into its own functions without the need for any mapping or manual conversion to C#.

For the first time, our Data Insight Team can manage and update their projection code via source control without the need for involvement from a Cloud Apps Engineer. Thanks to our automated Azure DevOps pipelines, they will be able to see the immediate effects of making a change in our test environments. This will go through our usual QA process and can then be released into our production environment for our live Moneybox users.

Current state of play

So where are we now? The projections Function App is now set up and deployed in Azure alongside all of our other apps via Terraform. To utilise the new app, we’ll employ a strangler pattern, slowly transitioning the python code for existing projects to the new system and creating any new projections directly there.

We still have quite a journey to go, but we are looking forward to seeing the improved performance and ease of maintainability of our new solution.