Deploying a Future Proof Traffic Router Based on OpenResty

Thousands of applications are hosted on Scalingo, they can be deployed, restarted or scaled at any time. Each of these operations impacts the internal topology of our infrastructure. Traffic routers handle the incoming requests and route them dynamically into the infrastructure. Since the beginning, we relied on Nginx to power these servers. With our growth, this solution became unsatisfying in terms of performance, reliability and flexibility, restraining the implementation of new features. We needed to find a way to improve or replace this piece of software to keep growing. In this article, we offer you a travel in the depths of Scalingo infrastructure, telling you how we ended up using OpenResty.

In this article, we will cover the main cons of the current solution leading to the deployment of a brand new shining reverse proxy. Then, we will show how this critical new version has been tested and released to minimize the risk for our customer's applications.

The purpose of a reverse proxy

Our infrastructure is composed of a multitude of servers running our customer's applications. These servers are not reachable from the wild outside world, they are hidden behind some frontend servers called reverse proxies. A reverse proxy is a HTTP server which receives requests and forwards them to the application containers. When a request is received, the proxy looks at the targeted domain name thanks to the Host HTTP header for a simple HTTP request or to the SNI extension in the case of a secured HTTPS connection. Once the domain name is defined, the connection is forwarded to the container(s) of the application matching this domain name. In a second time, the application container processes the request and sends the answer to the client through the reverse proxy.

Configuring a reverse proxy commonly takes place in one or multiple configuration files in order to setup the destination of the requests depending on the targeted domain name, sometime called virtual hosts. When the reverse proxy boots, it reads the configuration files and creates data structures, it builds a routing table which is stored in memory. This table is not modified after the initialization to speed up the processing of the requests. In our case, a modification of the routing table is mandatory in different cases: the deployment of an application, the restart of an application, the scaling of an application, etc. When such event occurred, we had to reload the configuration by reading all the files again and re-build the routing table in memory. This method to handle the configuration is limiting when you have thousands of websites to serve.

The cons of the current solution

Since the beginning of Scalingo, these reverse proxies where powered by good old Nginx instances with thousands of configuration files, one for each hosted application. The cons of such a solution became too important to be ignored:

• The routes are not instantly updated when needed. For example, when a new application is deployed, because of the large amount of domains to handle, the delay to reload the configuration ranged from 3 to 8 seconds with an average of 4 seconds. This problem caused noticeable delay for our customers after an update of an application configuration.
• Long lasting connections like WebSockets could be brutally cut off if the configuration was reloaded too many times during their life: when Nginx reloads its configuration, the server creates a copy of itself (fork/exec), and the child Nginx read the new configuration files and starts responding to requests. The problem is what is happening to the parent process: it is kept alive until the last connection ends. The obvious problem is that with long running connections, it never happens. Finally we had to garbage collect old Nginx instance to free memory on our front servers, breaking the oldest connections.

All these problems became more frequent with the number of configuration changes increasing over time.

For these reasons we have been investigating solutions to make the configuration of a reverse proxy more dynamic.

Here comes the Holy Grail of reverse proxy

In our quest to find a good dynamic reverse proxy, we found OpenResty, a dynamic web platform based on Nginx and LuaJIT. Being used by big companies like Cloudflare, we are confident about the serious and durability of this project. OpenResty is not an out-of-the-box solution. It still requires some Lua development to perfectly fit our infrastructure. Three main components have to be developed for our OpenResty setup to be able to fully replace the legacy Nginx:

1. TLS Certificate / Key: every application hosted on Scalingo has a certificate associated. With our OpenResty integration, certificates are now stored in a Redis database and are fetched dynamically when a request reaches a reverse proxy. Additionally, to prevent hitting Redis at each incoming HTTPS request and reducing latency, a in-memory LRU cache has been developed. When the configuration changes the cache is invalidated using Redis Pub/Sub.

2. Locate application inside Scalingo's infrastructure: every application is available on one or multiple servers on Scalingo's infrastructure. The reverse proxy must know the destination of all incoming requests. These information are also stored in Redis and cached to get the best performance possible.

3. Logs management: when using our legacy Nginx setup, it was configured to store the logs in one file for each application. No aggregation was possible, and it was a really static manner log the connection information. Given the dynamic nature of OpenResty, much more possibilities are opened to us. A micro-service written in Go has been created, which aims at receiving logs stream and handle these logs exactly as we wish (file system, indexing, message queue, etc.)

Testing OpenResty to ensure it is not a false Grail

The biggest part in the development of this new reverse proxy was the testing part. We wanted to ensure that OpenResty really outperforms the legacy Nginx setup in regard with the outlined cons. Both the new and the old setup have been extensively tested in order to compare them. In this section, we will explain the experiments we conducted and provide the results to extensively compare OpenResty with Nginx.

In the remaining of this document, we will call Legacy the old setup with Nginx only and OpenResty the new one.

Load Tests

Load tests were the first achieved experiments on both reverse proxies. The tool Vegeta v6.1.1 has been used in order to send a high number of HTTP requests to each reverse proxy. Four parameters are varying for this experiment:

• Web protocol: HTTP and HTTPS. In production, we currently have three times more HTTPS connection than HTTP. HTTPS requests require more computation from the reverse proxy.
• Duration of an experiment: 1, 2, 5 and 10 minutes.
• Requests rate: 100, 250, 500 and 750 requests per second.
• With and without reloading the configuration. As explained above, reloading the Nginx configuration is problematic when a high amount of applications are configured. Such a reload happens on average 11 times an hour, with some spike rate of multiple reloads every minute. (As you could expect, there are more reloading during the European office hours than Sundays at midnight CET). The worst case scenario will be tested by reloading the configuration every 20 seconds.

Metrics

In order to compare both solutions, the following metrics have been defined:

• Mean response time: the response time is the time between the beginning of the request sending and the end of receiving the associated answer.
• 99th percentile for the response time: 99% of the requests response time are below the 99th percentile.

Results

On the following images, the mean response time and the 99th percentile are displayed. It concerns the experiments without reloading the configuration. Results are displayed for the 1 minute long experiment as there wasn't any noticeable difference between the different durations. The x-axis is the number of requests per second and the y-axis is the mean response time. The pink bars show the results for the experiments with Legacy and the blue bars show the results for OpenResty. Each bar displays a line on top to show the 99th percentile. Its purpose is to help us understand how are distributed the values behind the average. Long story short: the longer the line, more distributed the values are (it's bad!).

The HTTP results are really close between OpenResty and Legacy. At 500 requests per second, the average response time fluctuate from 22 ms to 27 ms. At 750 requests per second, the average response time is slightly higher. But above all, some requests take much more time to receive an answer with a 99th percentile of 45 ms. At 1000 requests per second, the variation of response time becomes unreasonable with a 99th percentile of 246 ms.

When using HTTPS, the results are slightly unfavourable for Legacy with unreasonable response time starting from 500 requests per second (187 ms on average) whereas response time with OpenResty is 50 ms on average.

Let's now have a look at the following experiments in which the configuration is reloaded frequently. The results are still in favor of OpenResty. However, in both cases (OpenResty and Legacy), the performances degraded due to the frequent reload of the configuration.

Regarding this first batch of experiments we can conclude that OpenResty and Legacy both provide similar results with a slight advantage for OpenResty when handling HTTPS requests which is a good point as said previously, three quarters of incoming requests are using HTTPS. However, beyond 500 requests per second, the reverse proxy does not respond in reasonable delay in the worst case scenario. As of today, in production, our reverse proxies receive from 80 to 170 requests per second. This results is a good gauge for us to detect when additional proxies need to be deployed.

WebSockets Tests

After validating that the OpenResty setup can handle the actual load of Scalingo, we want to ensure that some of the problems raised when using Legacy fade out with OpenResty. One of these really annoying problem affect WebSocket connections. WebSockets are full-duplex channels often used to ease real time communication between a client and a server.

When using Legacy, updating the routing table requires to reload the configuration files. Nginx spawns new workers which read the configuration files then kill the previous worker when the newcomers are ready to accept connections. This method ensures no downtime in the process as Nginx does not break existing connections. If one of the old workers still has an open WebSocket connection, it goes to the state shutting down: it does not close the existing connections but do not accept new one. When all the existing connections are closed, the worker can gracefully shutdown. In the meantime, these workers are using resources: 400 MB of memory on average. If a high number of configuration reload occurs in a short interval of time, resources used by these workers are not negligible. Hence, we developed a script which aims at closing the WebSocket connections to free the resources. In production, this script must be executed every 15 minutes at peak time. This sudden connection cut led to problem for some applications and many questions about that to our support team.

Thanks to its dynamic nature, OpenResty configuration is not stored in configuration files but in a Redis database. Hence, the configuration files do not need to be reloaded: Nginx does not need to spawn new workers when the routing table are updated. Nginx workers should not be in the shutting down state and ongoing WebSocket connections do not need to be brutally killed to free the resources.

The following experiments aim at confirming that long running WebSocket connections are not affected by an update of the Nginx routing table. To make these experiments, A WebSocket connection is opened to each reverse proxy (Legacy and OpenResty) using the command line tool wscat. The connection can be kept opened for a long time if it sends a keepalive message at regular intervals. (Idle connections are closed after 30 seconds). The command is:

> wscat --connect ws://{openresty,legacy}.scalingo.com/sockjs/6l78edj/websocket --keepalive 30

Then, the routing table is updated by changing the number of containers associated to the targeted application (sample-ruby-sinatra):

> SCALINGO_API_URL=https://api-staging.scalingo.com scalingo --app sample-ruby-sinatra scale web:+1

As expected, the Nginx worker does not shutdown immediately with Legacy as it waits for the WebSocket connection to close. The following commands highlights this behavior:

> ps aux | grep nginx
www-data 13377  3.9  6.4 [...] 15:04 0:06 nginx: worker process is shutting down
www-data 13740  8.2  6.4 [...] 15:05 0:08 nginx: worker process

The process 13377 waits for the WebSocket connection to shutdown. The process 13740 is the new process spawned after the configuration reload. Killing the process 13377 will shut the connection down.

On OpenResty side, no worker is in the shutting down state:

> ps aux | grep nginx
root      4044  0.0  0.0 [...] Feb06 0:00   nginx: master process openresty
www-data  4046 10.8  3.0 [...] Feb06 414:38 nginx: worker process

WebSocket connections are not stopped by an update of the routing table.

Logs Service Test Load

The last case to test is about the logs. When using Legacy, Logs were configured to be stored in one file per application. Given the dynamic nature of OpenResty, the same simple setup is not usable anymore. A micro-service written in Go has been built, which aims at receiving logs stream and handle them as we wish (writing on disk, send to indexation service etc.). OpenResty has been configured to send the logs per packet of 64 KB. We need to ensure that this service will handle the load.

The first log storage backend developed has been to the file system: when receiving logs, the logger opens one file for the logs of each application. It keeps internally a structure of all the open file descriptors. At some point, log rotation becomes mandatory to prevent a too big log file. Our service must close the old file descriptor after a log rotation to acquire the file descriptor of the new log file. As this procedure use resources, we need to ensure that the performance do not degrade too quickly with a frequent reload of the file descriptors.

For this experiment, a small program has been written to send logs per packet of 64 KB to the logger service. Two parameters vary for these tests:

• The number of logs packets sent every second: 100, 250, 312, 375 and 500.
• With or without reloading the file descriptors every 20 seconds.

The duration of the experiment is 1 minute.

Metrics

We want to ensure that the service handles the incoming logs faster than they arrive. At the end of the experiment, the amount of remaining lines of logs are displayed. If it is greater than 0, this service do not handle the lines of logs quickly enough.

Results

The results displayed here are those of the worst case scenario (i.e. with a reload of the file descriptors) in the following table.

This service handles up to 250 packets of 64 KB logs per second in the worst case scenario. Hence, the logger handles a maximum of \(250\times 64 = 16000~KB/s\). We noted in a previous section that one reverse proxy cannot handle more than 500 requests per second. This service would be overloaded if every log line weights \(\frac{16\,000}{500} = 32~KB\). As it seems unreasonably high, this service is ready for production use!

Conclusion

New reverse proxies have been deployed a few weeks ago without any of you noticing (nearly ;-) and are very happy with the results. With this new traffic router based on OpenResty in place, we are ready to sustain Scalingo's growth!

In the process, the tests we conducted on the reverse proxies grew our knowledge about the resilience of the infrastructure.

Thanks to the acquired dynamist, deployment of new features had been greatly eased. It already let us deploy the wonderful Let's Encrypt integration. Stay tuned, new features are on the way!