Surpass Pokemon Go hype: minimizing impact on the infrastructure
Except if you’ve lived in a cave, disconnected from the outside world since the beginning of the summer, you should have heard of Niantic Labs releasing the augmented reality game Pokemon Go. Continent after continent, country after country, the game keeps getting more and more audience and players. Nowadays, it also means more and more applications around the game are created and here starts this story.
The emergence of Pokemon Go apps
The 6 days following the initial release of PokemonGo, some players/developers from all around the world have been taking time to reverse-engineer the game in order to understand how the mobile application communicates with Niantic Labs servers. They finally released a first Python third-party API client. With a few additional days, clients in any programming language were available, resulting in a very large amount of applications being created by the community. The main problems people are trying to solve are: “What are the pokemons around me?” and “Where are the rare pokemons I’m tracking for days?”
The 16th of July (2 days after the release of the python library), the project PokemonGo-Map got started, to propose a simple python application to visualize pokemons around a specific location given as configuration. The server pretends being a specific user at a specific location walking around and discovering pokemons, displaying them on a GoogleMap map.
The app was several weeks on Github trending page, making its popularity rocketing to the hype world of the game Pokemon Go. A lot of people have been willing:
- To use the application
- To deploy the application
- To contribute to the application
From that point, we assumed Scalingo could help to solve the problem number 2.
One-Click Deployment
In December last year, we’ve released our 1-click deployment feature. It let you put a button on your webpages, or Github project page to allow people deploying an open-source project in a single click.
We made a pull request to the map project which has been accepted. From that point people coming on the project page were able to deploy the application in less than two minutes on the platform.
SQLite - Houston there might be a problem
Almost 1,000 instances of the application got deployed in a few weeks. We knew in advance that all these people would be doing it, because it’s free, enjoying our free trial, and in exchange they would discover the advantages of a platform like Scalingo.
After the first tens of deployments, the tech team got alarmed: disk usage has raised abnormally, after a short analysis, the PokemonGo-Map applications were using, in average, 10MBps of disk bandwidth. Let’s make a simple computation: 1000 * 10 = 10GBps, even spread on multiple servers, it is a problem.
Why is the application using so much IO? One answer: SQLite.
Usually, we discourage the use of SQLite not because of the disk usage but because it does not fit the PaaS model. SQLite writes its data in a single file, usually suffixed ’.db’ alongside an app. In the PaaS model, disk around your app is temporary and any deployment or restart operation would result in data loss as the ’.db’ file would disappear.
As the data gathered from Niantic API are only temporary, representing pokemon states at a given time, it was not a problem to use SQLite, even if restarted or redeployed, data would be fetched again.
Here is an example from one of our server monitoring, do you guess when was the IO pike?
Solving the IO problem
From that point, we had two solutions:
- Stopping the one-click deploy feature for this app
- Finding a way to handle such IO load
Stopping deployments would be radical but we were under the spotlights, it was not interesting from a business point of view to stop that now, this solution should only be our last resort.
Our tech team took a glance at the project to get how it is working, to analyse if there was something do-able to reduce the disk usage. However it is not our policy to modify the source code of what is deployed on the platform, we should accept anything and the platform should adapt to the app needs.
One of us made the following assumption: “what if the SQLite database was in memory instead of being on the disk”. That’s an idea which was worth thinking about it. The data from the SQLite database was never really large (average of 10MB), putting it in RAM would increases the app memory consumption reasonably and decreases the IO to… 0.
From certain point of view, it is a dirty hack, from another it is an elegant solution to a very precise problem. A few hours later, we were deploying in our production cluster a hook detecting if the application is a PokemonGo-Map, and if it’s the case, creating dynamically a TMPFS directory where the SQLite database is created by the app.
“TMPFS is a temporary filesystem that resides in memory and/or your swap partition(s), depending on how much you fill it up. Mounting directories as TMPFS can be an effective way of speeding up accesses to their files, or to ensure that their contents are automatically cleared upon reboot.” - Archlinux Wiki
Conclusion
After restarting all the concerned applications, the disk usage dropped back to normal and we were able to handle more and more deployments without fearing to decrease performance of our customers production applications.
It’s not in our habits to develop custom things for specific apps, but it is sometimes required to keep our infrastructure stable and its cost normal (to make our business model actually works).
This article illustrates one aspect of the Pokemon Go effervescence, other projects related to the game are also hosted on the platform, with their set of problems, but that’s another story.
