Writing MapReduce Jobs Using Rust and Efflux

During my work life, I spend a lot of time working with MapReduce-style workflows, particularly with Hadoop infrastructure. A lot of this work is spent with larger amounts of data in order to implement the batch layer of the Lambda architecture. Due to this, the largest concern is that the…

Simulating Distributed Clusters for Elixir Unit Tests

During the recent addition of distribution to Cachex, I realised that testing distributed behaviour is quite annoying. Although there's a tonne of useful inside the OTP standard library, it's not particularly obvious if you don't know what you're looking for. Most projects (such as Phoenix) roll their distributed tests around…

Quickly Concatenating Files in Amazon S3

Amazon S3 is a storage solution used by pretty much everyone these days. Due to this there are naturally a bunch of tools for doing almost everything you can think of with S3. This post is about a tool I wrote to retrieve concatenate files efficiently in S3 buckets. If…

Easily Analyzing Your S3 Buckets

Amazon S3 is a storage solution used by pretty much everyone these days. Due to this there are naturally a bunch of tools for doing almost everything you can think of with S3. This post is about a tool I wrote to retrieve metadata about S3 buckets. If you don't…

Filtering Duplicate Log Entries via Rust and Runiq

If you work on any kind of backend system, odds are you're familiar with the pains of trawling through files looking for something specific only to discover huge amounts of entries which are basically all the same thing. Perhaps you have a really cool API that everyone calls all the…