Improved build time in Elixir without a DB

My engineering team at Moz is undertaking a comprehensive overhaul of our backend architecture to improve the performance and speed of our application and to unlock significant new features in high demand by our customers. We are abandoning MySQL database storage—our current infrastructure’s decisive bottleneck—in favor of a database-free architecture and an Elixir-driven data indexing model.

In this post on the Moz Blog, I discuss our new architecture, which uses Elixir to process data from cold storage in S3, and load it into memory. We use Erlang’s binary marshaling format to facilitate efficient data indexing, and we load customer datasets into memory on a few API hosts.

With the Elixir solution, we achieved some impressive results:

1. 1.

   Compressed data files consume 63 times less disk space

2. 2.

   API serves data 20 times faster, with <50 ms average response time

3. 3.

   Customer data sets build 30 times faster, usually in under a minute

Read the full post on the Moz Developer Blog.