For those who are unaware, the Titanium SDK has long been a good option for cross-platform mobile development. With the recent move to it being a fully open source project (via TiDev) I decided to revisit for some minor experimentation. This post is going to cover working with JSON data (in particular, the storage of JSON data inside a mobile application).

Data Persistence

Storing data on disk to persist across sessions is a pretty common occurrence, whether it be to cache remote data for quicker access, or keep track of application state. When it comes to storing data on a mobile device, the Titanium SDK offers support for various methods of persistence:

  • Application Properties
    • Ideal for simple key/value pairs.
    • Typically stores data on the state of the application.
  • SQLite Databases
    • Great for structured data (i.e. using a schema).
    • Provides flexibility in querying and retrieving data.
    • Offers transactional APIs to ensure consistency.
  • Filesystem Storage
    • Generally used for arbitrary files storage.
    • Doesn't require a schema or structure.
    • Supports binary types and images.

The strengths and weaknesses of each of these options are well documented here, but the above should give a pretty broad view of the types of situations you'd reach for each option. When it comes to working with the Titanium SDK, most of what you write will be in JavaScript. As you will no doubt be aware, JavaScript tends to come along with at least some amount of JSON usage. When we look at each of these options, none of them are ideal for storing data of this format:

  • Application Properties don't scale to the amount of data we'd be storing, and it's only intended for simple key/value pairs.
  • SQLite could serve this use case well, but we'd need a layer between our data and the database to convert JSON <-> SQL.
  • Filesystem storage would store our data well, but there's no support for querying as it's just raw file storage.

In a time where I wasn't playing around for fun, I might default to SQLite. As this was already intended as a learning experience I began searching for embedded JSON databases, with the hopes that I could adapt one for Titanium.

LokiJS - A JavaScript Document Store

My search led me through an endless stream of embedded databases, but unfortunately none of them quite fit the bill - they generally had too many restrictions or some requirements my data could not fulfill. In the end I eventually landed on LokiJS, which seemed like a good fit for my use case. The documentation was pretty solid and the repository seemed fairly active. At the time of writing, the GitHub repository for LokiJS states:

LokiJS is a document oriented database written in javascript, published under MIT License. Its purpose is to store javascript objects as documents in a nosql fashion and retrieve them with a similar mechanism.

My interest was now piqued; many of the things I was looking for seemed to be readily available in LokiJS. Even just this snippet above hits on many of the major points relevant to my use case:

  • Written in JavaScript
  • Storing JSON data (i.e. JavaScript objects)
  • Unstructured storage (i.e. NoSQL)
  • Querying stored data

So how does it work? Well, Loki essentially stores your data in memory and provides indexing and query capability around it. The API is relatively similar to a lot of other tools in the JavaScript space, as well as other database syntaxes such as MongoDB. Below is a small sample of storing and querying a couple of user records, loosely lifted from the LokiJS documentation:

// import our loki module
let Loki = require('lokijs');

// create a new example database
let db = new Loki("example.db");

// create a collection of user documents
let users = db.addCollection("users");

// insert a couple of example users
users.insert({ "name": "John", "age": 32 });
users.insert({ "name": "Jane", "age": 26 });

// read back all users over 30 years old
let results = users.find({ "age": { "$gte": 30 } });

// check the results match
assert(results.length === 1);
assert(results[0].name === "John");

As you can see, it's a pretty simple process. This example was already enough to convince me that I should at least try to get Loki running with Titanium - it seemed like it would become a great option for storing JSON data.

Integrating with the Titanium SDK

I initially thought compatibility with the Titanium SDK might be complicated, but LokiJS is written completely in JavaScript and so worked out of the box with Titanium. Of course, an in-memory database isn't going to get you far with a mobile application - every time you close the app, you'd lose all your data! Fortunately, LokiJS already has the concept of persistence "adapters". These adapters allow a developer to write a serialized version of their database to disk, so that Loki can pick it up later and re-initialize itself from a snapshot.

There are a couple of options with Titanium that I explored here. I thought about writing the JSON <-> SQL translation layer I mentioned earlier, which would map the JSON objects of Loki through to SQLite tables. I decided that it didn't really make sense, given that SQLite was acting only as storage - which meant it was just a layer of indirection over a filesystem. So the solution left was to simply map the data to the filesystem. It was at this point that I wrote a simple library for Titanium: loki-titanium-adapter.

This library provides a very simple adapter to bridge the gap between the Titanium SDK and LokiJS. All persistence is taken care of via the Titanium Filesystem APIs, by serializing data to the device storage. Data is written to disk in the JSONL format to allow for streamed (de)serialization, and collections may be written to the disk concurrently. The structure on disk used for a database is as follows:

/{name}     <- parent directory for a database
/{name}/_   <- metadata file storing database metadata
/{name}/0   <- JSONL file storing data for collection 0
/{name}/1   <- JSONL file storing data for collection 1
/{name}/2   <- etc.

This design offered quick read/write from the disk, and makes it easy to clean up a given database (by just removing the parent directory). I did some rough benchmarking and throughput seemed pretty great; even serializing a database with 1,000,000 records takes only a few seconds. Eventually I'll probably clean these benchmarks up and get them into the repository.

Using LokiJS in Titanium

This section will be pretty short, as it's really best to visit the LokiJS documentation for usage. I included an example Titanium project (v12) in the repository, for those of you who want to look it over. For anyone who wants to give it a try in their own projects, both LokiJS and the Titanium adapter are available as packages from npm:

npm install lokijs loki-titanium-adapter

You can then use both of these modules inside your application code to initialize a database which will then sync across application sessions. To do this, simply attach an instance of TitaniumAdapter against your database during creation:

const Loki = require('lokijs');
const TitaniumAdapter = require('loki-titanium-adapter');

const db = new Loki('my_database', {
	adapter: new TitaniumAdapter(),
	autoload: true,
	autosave: true,
	autosaveInterval: 3000,
	autoloadCallback: function () {
		// initialize collections and UI
	},
});

This example will load your database from the device storage on startup, and will sync it back to storage every 3 seconds. Of course these things can be tuned as you need, or you can opt for manual syncing by calling db.saveDatabase(callback) and db.loadDatabase(opts, callback). The autoloadCallback can be used to initialize base data, or instantiate any UI which depends on your data being loaded.

It's worth mentioning that the TitaniumAdapter itself also has some configuration options, so make sure to check the repository README for the latest features and options. Most of these options will be related to reading and writing from disk, so you can assume the defaults will be "good enough" for most cases. Of course if there's anything we could improve in future, please file an issue or let me know!

One final thing to note here is that the data lives in memory when your database is loaded. This means that a very large database might consume a large amount of memory. Due to this it's best to save and close your database if you're not actively using it, to preserve resources on device. Given that it's generally fast to load from disk, there's no harm in opening it again when you need it. If you're already using SQLite and are thinking of switching over, this shouldn't be a new concept - but just make sure to double check!