This page explains how the PlanX backend is set up to handle variability in traffic to the PlanX webite. In particular, this inclues the API and Hasura GraphQL layer (which sits in between the website and the database proper).
As a planning officer, I want to be able to view and edit my flows in the PlanX editor without any significant slowdown.
As a user stepping through a flow in order to make a submission, I want to interact with a snappy, responsive interface.
The PlanX editor (https://editor.planx.uk/) relies on background services in order to run.
For example, it makes a lot of calls to both our API (https://api.editor.planx.uk) and our Hasura GraphQL layer (https://hasura.editor.planx.uk/).
These servers need to be able to handle varying amounts of traffic. In general, at night, we can expect very few people to be using PlanX. However, one can also imagine a case where there is a deadline approaching which applies nationwide to a certain kind of submission, in which case we may see a significant and sustained spike in traffic, as users rush to make their submission in time.
To minimise resource use, we provision our services such that they are responsive to changes in traffic. That is, we tell AWS to adjust the amount of compute capacity available to a given service in reaction to observed usage patterns, in a matter of minutes. This is called ‘auto-scaling’.
We’ll use Hasura as an exemplar case.
When lots of people are using PlanX at once (including via a planningservices.council.gov.uk URL), many of them will be pulling down published flows (see How editing and publishing flows work). This is one of the most data-intensive requests the application ever needs to make.
Such requests are made to a Hasura ‘service’ running on AWS ECS as 1 or more ‘tasks’ (which we can think of like Docker containers). Each task has a certain amount of CPU and memory resources allocated to it. If any task is subject to too many simultaneous requests, it may not have the resources to fulfil them, resulting in a failed request, or may do so very slowly. Either way, the resulting user experience will be sub-optimal.
Therefore, the service distributes incoming requests equally between the tasks to avoid overwhelming any single one. We prescribe in code that if the CPU or memory usage of the service as a whole should exceed certain thresholds, new Hasura tasks should be created, thereby easing the load on any single task. This is called ‘scaling out’. When traffic, and therefore resource use, dips again for a sustained period, the service will start to shut tasks down. This is called ‘scaling in’.
The appropriate thresholds at which this auto-scaling should take place are ascertained by running manual load testing exercises on the PlanX staging platform, and observing the results (see Uptime & resilience for more on how we load test).
As described above, auto-scaling currently applies to the PlanX API and Hasura. Both of these services ultimately rely on and connect to an AWS RDS hosted PostgreSQL database. Our database itself is not currently configured to automatically scale. Based on our usage patterns and load testing exercises, this is not yet a concern but may be prioritised in a future phase of work.