Data Resiliency at Scalingo
For a couple of weeks now, some of our users and clients have been increasingly curious on the support chat about data resiliency at Scalingo. They are asking a broad range of questions. How do you store my data? Where is it stored? How do you handle backups? How do you handle data redundancy?
While all general information about data handling is publicly available in our Data Processing Agreement, it's hard to figure out actual data flows.
That's why we decided to sum it all up in a diagram, with the help of our hosting provider 3DS Outscale.
Our goal is to be as transparent as possible about the infrastructure we are using. This article will only cover data flows and how we ensure data redundancy.
For those who just want a quick look, you can discover the diagram in the first part of the article.
The second part will give you more detail and explanation about the infrastructure.
TL;DR - The diagram
What is the data managed by Scalingo?
Applications
When you push your code, we build a Docker image that is stored in our private registry which is backed by Outscale Object Storage (OOS). We'll refer to this Docker image later simply as code.
We run those images as container on instances managed by us. We attach disk to these instances through Outscale software-defined storage.
When your applications is running, it generates logs and metrics which are consumed by their respective Scalingo internal services.
Databases
Databases at Scalingo are Docker containers run out of pre-built Docker images. Unlike applications, those containers are using persistence disk storage.
From the first paid plan, we do daily backups of your database. On Scalingo PostgreSQL as a Service we do PITR backups (Point-in-Time Recovery). Backups are stored in Outscale Object Storage.
Databases logs and metrics are consumed by their respective Scalingo internal services.
Scheduler
The scheduler is the brain of the platform. It knows every containers running on the platform. This data, written as configuration in the diagram, is stored in regular Scalingo Database as a Service offering.
It's worth to notice that the scheduler, the authentication service, the billing service, the metrics service, the logs service and all internal services are regular Scalingo applications benefiting from the same data resiliency mechanisms. We're eating our own dog food!
The crispy details
We host your data and applications on the infrastructure of our hosting provider 3DS Outscale (Cloud subsidiary of Dassault Systemes).
We are currently offering two regions: osc-fr1 and osc-secnum-fr1. Those regions are hosted on completely different datacenters.
In the diagram you can discover how our region osc-fr1 works. Our other region osc-secnum-fr1 works similarly.
All datacenters from the region osc-fr1 are located in France (Magny-les-Hameaux, Pantin and a third site in Paris' suburbs).
Outscale Region eu-west-2
As you can see the region eu-west-2 is divided into 2 Availability Zone (AZ): eu-west-2a and eu-west-2b. Availability Zones are distinct datacenters linked by dark fibre (dedicated optic fibre, only used by Outscale).
Each zone in a region has redundant and separate power sources, networking and connectivity to reduce the likelihood of two zones failing simultaneously.
Let's dive into what is stored in each Availability Zone and give you some insights on how Scalingo uses them.
Outscale Availability Zone eu-west-2a
This is the primary availability zone where we host all applications and databases. Its datacenter is located in Magny-les-Hameaux.
Your applications, databases and scheduler are all hosted using Outscale virtual instances on Outscale software-defined storage which are all persisted on Outscale physical storage. (even if we in the "cloud" there is always a physical storage as you might have guessed.)
All your applications files (code, logs, metrics), databases (daily backups, PITR backups, logs, metrics) and scheduler (configuration) are stored in Outscale Object Storage (OOS). See below for more details about OOS.
Outscale Availability Zone eu-west-2b
This availability zone is located in Pantin which is also in Paris's suburbs.
Outscale Object Storage files are available in this Availability Zone as well.
Outscale Object Storage (OOS)
Outscale Object Storage is an object storage service (hence the name) with an S3 compatible API.
Data is distributed between the 2 Availability Zone (AZ) of the region.
The redundancy of OOS is setup to sustain the complete loss of one of the two AZ.
Outscale Backup Site
Finally 3DS Outscale also has a backup site located at about 30 kilometers away from Magny-les-Hameaux and about 5 kilometers away from Pantin.
This backup site is also called DRP which means Disaster Recovery Plan.
In this backup site, full backups of Outscale physical storage from both AZ are made every 24 hours. One for eu-west-2a and one for eu-west-2b.
Backup 3-2-1 Rule
Using Outscale Object Storage to store backups helps us achieve the Golden 3-2-1 rule for backups:
- Keep at least 3 copies of your data
- Store 2 backup copies on different devices or storage media
- Keep at least 1 backup copy offsite
Conclusion
At Scalingo our vision is to ease the developers' job and to build the best European cloud platform for them.
We take a lot of responsibilities for you so you don't have to worry about them and can focus on your added value: the application you're writing and the features you're bringing to your customers.
Likewise, we are standing on the shoulder of giants: Outscale IaaS is ISO 27001 certified and their datacenters are themselves ISO 27001 certified.
If you need to host your data feel free to contact me or anyone in the team.
Photo by Alicia Mary Smith on Unsplash
