Easily Analyzing Your S3 Buckets

Amazon S3 is a storage solution used by pretty much everyone these days. Due to this there are naturally a bunch of tools for doing almost everything you can think of with S3. This post is about a tool I wrote to retrieve metadata about S3 buckets. If you don't…

Filtering Duplicate Log Entries via Rust and Runiq

If you work on any kind of backend system, odds are you're familiar with the pains of trawling through files looking for something specific only to discover huge amounts of entries which are basically all the same thing. Perhaps you have a really cool API that everyone calls all the…

Optimizing Docker Images for Rust Projects

This post will be the first of several addressing Docker image optimizations for different project types. It stems from my recent experiences with badly written Dockerfiles, which result in sitting around for 10 minutes every time you build, only to then need to upload images of over 1GB in size.…

Automatic Bang! Function Generation in Elixir

Elixir developers typically conform to the standard of returning Tuple structures from functions which can either succeed or fail in order to effectively signal their status to the caller. Whilst this encourages the caller to directly handle the various cases which can arise, sometimes you just know that everything was…

Achieving Multiple Routing Keys in RabbitMQ Exchanges

One of the key features inside RabbitMQ is the ability to easily route messages between exchanges and queues, using any number of provided routing patterns. Out of the box, RabbitMQ provides a variety of algorithms to use when routing (based on the AMQP). You can route using the following algorithms:…

Bulk Indexing with the Elasticsearch Java REST Client

With the introduction of Elasticsearch v5.0.0, Elastic began to recommend moving to their new REST client for server interactions rather than the transport client that was shipped previously. The intent was to make a HTTP client which would act with little to no interaction with the internal Elasticsearch…

A Year with Cachex in Production

A little over a year ago I began working on what would become my first real project in Elixir. Without going into too much detail, it was a re-imagination of a legacy system designed to handle many incoming data streams quickly and reliably. It seemed like a good fit for…

Introducing Vessel: A Bridge Between Elixir and Hadoop

Disclaimer: I should point out that I wrote this project for those who are already working with Elixir and have need of integrating their existing codebases with Hadoop. I'm going to try assume limited knowledge of Hadoop/Elixir within this post - but I apologise if it goes a little…

Stash: a user-friendly wrapper for ETS/DTS caching in Elixir

Erlang (and by extension, Elixir) comes with a sweet little memory storage library called ETS (Erlang Term Storage). This provides an awesome way to store a whole bunch of values in memory, without having to write your own implementation, or pull in large third-party libraries. The issue with ETS is…

Writing SipHash: my introduction to NIFs in Elixir

After recently discovering SipHash when searching for a solid algorithm to use in another project, I decided to take a shot at writing an implementation in Elixir as an exercise (having only started picking up Elixir in the last few weeks or so). The project is available on GitHub and…