MagLab / Case studies / High-load platform
Ten microservices, a terabyte a day and one source of truth
High-load platform: consulting, assessment and implementation
The platform grew from a monolith into ten services, each running on its own schedule. Data was lost, duplicated and reached reports a day or two late. We rebuilt it into one pipeline: events flow into one database around the clock.
Problem
The platform was losing and duplicating data, and reports ran two days late.
- Each of the ten services ran on its own schedule.
- Data was lost and duplicated between services, and copies of the same data took up space.
- Reports waited up to two days for data.
What we did
We took the whole product apart, designed a new architecture, took it through the architecture committee and rebuilt the platform together with the client's team.
- We went into a running system of ten services and studied how it all worked.
- We found the cause: data consistency and processing speed.
- We proposed an architecture with one source of truth: one event stream and one database.
- We rolled it out together with the company's teams.
What changed
- One pipeline: events flow around the clock, with no copies and no losses.
- One database, a single source of truth for all data.
- Data reaches reports in 1-2 hours instead of two days, at a terabyte of data a day and a thousand requests per second.
Built with
- Apache Kafka
- Apache Spark
- Kubernetes
Also: Elasticsearch, Grafana and Docker.
Other cases
01Restaurant chain
The whole business on one screen
The money lived in five systems. Now the whole business is on one screen every morning.
03Knowledge graph
AI read a whole library and built a knowledge graph from it
The answers were buried in thousands of papers. Now you ask a question and get the answer with its source.
04Mosena
Mosena: we did it all ourselves, start to finish
There was an idea for an app. Now there is a working AI product, built start to finish.
Let's grow together
Tell us what eats your time and money. We reply within a working day and suggest a time for a call.
The first conversation is 30 minutes, free, and you talk to an engineer, not a salesperson. Email: hello@maglab.app
Who you will talk to
Gleb Maksimov, founder of MagLab. Eight years in data, ML and high-load systems: startups, research labs, corporations. LinkedIn