Writing MapReduce Jobs Using Rust and Efflux

During my work life, I spend a lot of time working with MapReduce-style workflows, particularly with Hadoop infrastructure. A lot of this work is spent with larger amounts of data in order to implement the batch layer of the Lambda architecture. Due to this, the largest concern is that the…

Simulating Distributed Clusters for Elixir Unit Tests

During the recent addition of distribution to Cachex, I realised that testing distributed behaviour is quite annoying. Although there's a tonne of useful inside the OTP standard library, it's not particularly obvious if you don't know what you're looking for. Most projects (such as Phoenix) roll their distributed tests around…

Quickly Concatenating Files in Amazon S3

Amazon S3 is a storage solution used by pretty much everyone these days. Due to this there are naturally a bunch of tools for doing almost everything you can think of with S3. This post is about a tool I wrote to retrieve concatenate files efficiently in S3 buckets. If…

Cachex v3.1 and the Return of Distributed Caches

If you work with Elixir on web services, you may have come across a library I work on named Cachex. Although the name makes it somewhat obvious, this is a library based on caching expensive data in memory. It works inside your application layers using the Erlang Term Storage (ETS)…

Optimizing Docker Images for Maven Projects

I recently wrote a post on optimizing Docker images for Rust projects, which turned out to be quite popular, so I figured I'd follow it up with another for those Java developers out there who have to painfully sit through slow Maven builds inside Docker. Just like the other post,…

Adding Custom Handlebars Template Helpers in Ghost

Ghost has come a long way since I wrote my first post on safely creating custom Handlebars helpers, but unfortunately not far enough that it's an easy feat. Whilst there's some movement in enabling developers to add Handlebars helpers via Ghost Apps, it's still a pain to get working (I…

Easily Analyzing Your S3 Buckets

Amazon S3 is a storage solution used by pretty much everyone these days. Due to this there are naturally a bunch of tools for doing almost everything you can think of with S3. This post is about a tool I wrote to retrieve metadata about S3 buckets. If you don't…

Filtering Duplicate Log Entries via Rust and Runiq

If you work on any kind of backend system, odds are you're familiar with the pains of trawling through files looking for something specific only to discover huge amounts of entries which are basically all the same thing. Perhaps you have a really cool API that everyone calls all the…

Optimizing Docker Images for Rust Projects

This post will be the first of several addressing Docker image optimizations for different project types. It stems from my recent experiences with badly written Dockerfiles, which result in sitting around for 10 minutes every time you build, only to then need to upload images of over 1GB in size.…

Automatic Bang! Function Generation in Elixir

Elixir developers typically conform to the standard of returning Tuple structures from functions which can either succeed or fail in order to effectively signal their status to the caller. Whilst this encourages the caller to directly handle the various cases which can arise, sometimes you just know that everything was…