45 billion records and counting – Salpo builds giant data lake

This is big. Literally. We’ve built one of the largest elastic search clusters in the world, having processed more than 45 billion (you heard right) records at time of writing. And this figure is growing all the time.

The project is being completed for our client, Acuris Risk Intelligence (ARI). They transform unstructured dark web data into meaningful customer insights to support the fight against cyber crime. Their old legacy system was struggling to cope with demand, so we stepped in to build a robust, future-proof solution.

ARI’s previous software solution, while functional, had scalability and performance issues, preventing growth and providing a sub-optimal user experience. There was also limited support for this aging legacy product.

We built a high-performance data lake and multi-clustered search interface, using advanced AWS components and streamlined code. Having ingested and processed more than 60 billion records to date, this database is among the largest elastic search clusters in AWS and likely the world.

“Salpo built a high-performance big data solution, capable of fulfilling our growing data requirements and meeting global customer expectations.”

Nick Parfitt, Head of Market Planning, ARI

Click to read the full Acuris Risk Intelligence case study. If you’ve worked with Salpo in the past and would like to be featured, please get in touch.