A quick post about moving my Node app tests to Mocha and Chai. These are frameworks used where I work, and I figured I may as well learn something new and have more confidence in my Node apps when I deploy them, so how hard can it be?

Turns out, quite easy. These testing frameworks are deigned to be flexible and resemble plain-english test descriptions. In fact, the new testing code looks a lot like my home-grown test framework. Have a look below for a comparison (taken from the led-server project):

(You can see the implementation of testBed module in the node-common project)

Home-grown

async function testSetAll(expected) {
  const response = await testBed.sendConduitPacket({
    to: 'LedServer',
    topic: 'setAll',
    message: { all: [ 25, 25, 52 ] }
  });

  testBed.assert(response.status === 200 && response.message.content === 'OK',
    'setAll: response contains status:200 and content:OK');
}

async function testSetPixel(expected) {
  const response = await testBed.sendConduitPacket({
    to: 'LedServer',
    topic: 'setPixel',
    message: {
      '0': [ 25, 25, 52 ],
      '1': [ 100, 100, 100 ]
    }
  });

  testBed.assert(response.status === 200 && response.message.content === 'OK',
    'setPixel: response contains status:200 and content:OK');
}

Mocha/Chai

describe('Conduit topic: setPixel', () => {
  it('should return 200 / OK', async () => {
    const response = await testBed.sendConduitPacket({
      to: 'LedServer',
      topic: 'setPixel',
      message: {
        '0': [ 10, 20, 30],
        '1': [30, 50, 60]
      }
    });

    expect(response.status).to.equal(200);
    expect(response.message.content).to.equal('OK');
  });
});

describe('Conduit topic: setAll', () => {
  it('should return 200 / OK', async () => {
    const response = await testBed.sendConduitPacket({
      to: 'LedServer',
      topic: 'setAll',
      message: { all: [64,64,64] }
    });

    expect(response.status).to.equal(200);
    expect(response.message.content).to.equal('OK');
  });
});

As a result, my script to run all the test suites of each Node app (after booting them all together locally) looks like a lot of this – all green, and no red!

Hopefully this new skill will enable me to write better code both personally and professionally in the future – I may even try out TDD for my next project!

A problem I had found when setting up my Node.js services on a new Raspberry Pi (or resetting one that had gotten into a bad state) was keeping track of the individual port numbers of each one. This might typically look like this:

• New Headlines Backend – 5000
• Tube Status Backend – 5050
• LED Server – 5001
• Backlight Server – 5005
• Attic – 5500
• Spotify Auth – 5009

…and so on. This wasn’t only a problem with setup, but also with maintaining all the numerous config.json files for each app that needed to talk to any of the other ones.

So to do something about it, I decided to have a go implementing a central message broker service (nominally called Message Bus, or MBus) from scratch (one of the key features of my hobbyist development, as you tend to learn a lot more this way). This new service had to be generic to allow all kinds of messages to flow between the services that they define themselves. It had to be fault tolerant and so should use JSON Schema to make sure the messages are all of the correct format. And lastly, it shouldn’t care what the connection details are for each app at startup.

Client Registration and Message Exchange

To solve this last problem, each app uses a common Node.js modules that knows the port of a local instance of MBus and requests a port assignment. MBus responds with a randomly rolled port number from a range (making sure it isn’t already allocated to another app), and the client app then creates an Express server that listens on the allocated port. If MBus receives a message with a known client app as the destination, it simply sends it on to that port within the local machine, where the client app will be listening as expected. These two processes are summarised below:

Client Implementation

To implement a new client to talk to MBus, it includes the mbus.js common module, and registers itself at runtime. It also specifies the message schema it will expect from MBus using conventional JSON Schemas:

const mbus = require('../node-common').mbus();

const GET_MESSAGE_SCHEMA = {
  type: 'object',
  required: [ 'app', 'key' ],
  properties: {
    app: { type: 'string' },
    key: { type: 'string' }
  }
};

const SET_MESSAGE_SCHEMA = {
  type: 'object',
  required: [ 'app', 'key', 'value' ],
  properties: {
    app: { type: 'string' },
    key: { type: 'string' },
    value: {}
  }
};

async function setup() {
  await mbus.register();

  mbus.addTopic('get', require('../api/get'), GET_MESSAGE_SCHEMA);
  mbus.addTopic('set', require('../api/set'), SET_MESSAGE_SCHEMA);
}

Once this is done, the config.json is also updated to specify where it can find the central MBus instance and the name it is to be identified as when messages are destined for it:

{
  "MBUS": {
    "HOST": "localhost",
    "PORT": 5959,
    "APP": "LedServer"
  }
}

The mbus.js module also takes care of the message metadata and the server checks the overall packet schema:

const MESSAGE_SCHEMA = {
  type: 'object',
  required: [ 'to', 'from', 'topic', 'message' ],
  properties: {
    status: { type: 'integer' },
    error: { type: 'string' },
    to: { type: 'string' },
    from: { type: 'string' },
    topic: { type: 'string' },
    message: { type: 'object' },
    broadcast: { type: 'boolean' }
  }
};

Example Implementations

You can find the code for MBus in the GitHub repository, and also check some example clients including Attic, LED Server, and Monitor.

Barring a few client app updates (luckily no very serious user-facing apps depend on these services for core functionality right now), all the main services now use MBus to talk to each other. The image below shows these setups for the main machines they are deployed on:

Finally, over the next few months I’ll be updating what clients there are to talk to their remote counterparts in this manner, and also take advantage of the fact it is now each to add and address future services in the same manner without needing to configure ports and addresses for every individual service.

Adding Raspberry Pi based backlighting to my desktop PC with backlight-server, and moving to a new flat gave me an interesting idea – add an API to the backlight server to set the lights to the dominant colour of whatever album is playing in my Spotify account. How hard could it be?

The first step was to read up on the Spotify API. I quickly found the ‘Get the User’s Currently Playing Track’ API, which fit the bill. Since it deals with user data, I had to authenticate with their Authorization Code Flow, which requires multiple steps as well as a static address for a callback containing the authorization code granted to my application. I experimented with giving the Spotify Developer site the IP address of my Server Pi, but that could change, which would mean editing the application listing each time that happened, which was unacceptable for a seamless ‘setup and forget’ experience I was aiming for.

The solution was to resurrect my DigitalOcean account to host a small Node.js app with a simple task – receive the callback from Spotify with the authorization code with which access and refresh codes would be granted, and fetch and determine the dominant colour of the album art currently playing. This service would in turn be used by backlight-server to light up my living room with the appropriate colour.

This authorization flow took a long time to get right, both from a code perspective (I used the spotify-web-api-node npm package to make things programmatically easier), as well as a behavioural perspective (when should the token be refreshed? How to propagate errors through different services? How can the app know it is authorized at any given time?), but once it worked, it was very cool to see the room change colour as my playlist shuffled by.

I had a half-hearted attempt at figuring out the dominant colour myself using buckets and histograms, but in the end decided to preserve my sanity and use the node-vibrant package instead, which worked like magic!

So this is basically how the whole thing works, and you can see the code for the spotify-auth microservice on GitHub. The diagram below may also help explain:

So what next? Well, those smart RGB light bulbs are looking a lot more interesting now…