To render a scene on a Game Boy or Game Boy Color, one must create a tileset of tiles of 8x8 pixels, and assign these tiles to the background. The Game Boy background is 32x32 tiles and the screen renders a 20x18 portion of it.

It therefore makes a lot of sense to render text using 1 tile per character, resulting in a fixed-width 8x8 font. Most popular games use this.

My goal is to find a way to render variable-width fonts, with the intention to render more than 20 characters on a single line. I do intend to stick to the 8 pixels vertically, but horizontally I want to “ignore” the 8-pixel grid and create a software text renderer for the Game Boy.

Finding a font

The first step is finding a font. I’m really lazy, so I’ll grab a free pixelated font from FontSpace.com.

I like Pixel Millennium, because it has a variety of character widths, making it suitable for this demo. This font is free for personal use, and I’ve purchased a commercial license for my project.

Encoding variable-width characters in C

The next step is to convert the TTF to a map of characters I can use as a basis to render the text width. My instinct tells me to render the characters as a 1-bit bitmap, where each byte represents 8 pixels of either “background color” (0, for white) and “text color” (1, for black). My bitmap should contain the width of each character.

I haven’t actually checked, but I assume the smallest character is 1 pixel wide and the largest character is 5 pixels wide, so I need at least 3 bits to encode the width of a character. 3 bits nicely encodes 0 up to 7, which I expect to cover all characters. I’ll also assume a fixed height of 5 pixels for all characters.

Finally, I’m going to encode all characters in a single buffer of bytes named font_data, and because they will all have different sizes, I’m going to need some kind of index. This index will mirror an ASCII table for common characters, starting at 33 (!) and ending at 126 (~), for 94 characters total. The index will contain pointers to the start of the font_data for each character.

To make the pointers work, I’m going to have to align each character in the font_data with the start of a byte, so I expect there to be some padding bits between them.

To summarize, I expect to encode the font as C code as follows:

const uint8_t font_letter_spacing = 1;
const uint8_t font_space_width = 3;
const uint8_t font_height = 5;
const uint8_t font_data[] = { 0xFF, /* encoded pixels for all characters */ };
const uint16_t font_indices[] = {
  0, // !
  2, // "
  5, // #
  // etc
};

// example usage...
const uint8_t *character_bitmap = &font_data[font_indices['#'-33]];

Whether this is an optimal solution, I don’t know. But I’ll find out, by doing. Let’s generate some font data by rendering the TTF to font data!

Encoding the font

The easiest way to encode the characters to font data I can think of is by using Javascript. I can render each character onto a canvas, extract the pixels and convert them to my desired font_data and indices.

First, I render the characters to a canvas. For some reason the font keeps rendering slightly blurry, so I’ll just render the font pretty big and query the center of “assumed pixels”. I rendered the “query pixels” green, the borders red and the font blue:

By rendering these characters, I immediately notice the characters not only have a variable height, they also have a variable vertical position. My plan to encode the characters with a fixed height of 5 will fail anyway since some characters are clearly taller than that, with the $ being 7 pixels tall while the . (period) is 1 pixel tall. The underscore, _ starts at the very bottom and the ' starts at the very top. The @ is not supported by the font, so I’m going to keep that character blank for now.

To support a variable width, height and vertical offset, I’m going to encode the width, height and vertical offset of each character. Analyzing all characters, I found that the width, height and vertical offsets all vary by at least 5, I’m going to need at least 3 bits for each value, 9 bits total. I might still pack them as 1 byte per value, depending on the size vs performance tradeoff, since extracting 3 bits from a byte is likely extra work.

Packing the pixel data tightly with the 9 bits of metadata, the majority of characters will fit in 3 or 2 bytes. Since most characters are 3x5 pixels, they need 15 bits. Add 9 bits of metadata, and you get a nice round 24 bits.

I suspect I’ll be needing to calculate the width of a string without actually rendering it, so I want to pack the width at the beginning for easy access.

Skipping the implementation details, the result looks like this:

font_data.h

#pragma once

#include <stdint.h>

#define font_data_ascii_offset 33
#define font_data_ascii_max 126

extern const uint8_t font_data[278];
extern const uint16_t font_data_indices[94];

font_data.c

#include "font_data.h"

const uint8_t font_data[278] = {
  /* <blank> */ 0x00,
  /* ! */ 105, 46,
  /* " */ 83, 90,
  /* # */ 109, 212, 87, 95, 1,
  /* ... */
  /* | */ 57, 254,
  /* } */ 59, 34, 89, 10,
  /* ~ */ 212, 180, 0,
};

const uint16_t font_data_indices[94] = {
  /* ! */ 1,
  /* " */ 3,
  /* # */ 5,
  /* ... */
  /* | */ 269,
  /* } */ 271,
  /* ~ */ 275,
};

It compiles without error! Now let’s see if I can render the characters on a Game Boy.

Rendering characters

Let’s attempt to write the characters as tiles on a Game Boy, rendering one character per tile. I’m going to create 2 new files - font.h and font.c; These will contain functions for rendering the characters. Before I deal with the Game Boy’s native 2-bits-per-pixel encoding, I’m going to use GBDK-2020’s (Game Boy Development Kit) 1bpp to simplify the implementation, where one byte simply represents 1 row of 8 pixels, and each tile of 8x8 pixels is represented by 8 bytes.

font.h

#pragma once

#include <gb/gb.h>
#include <stdint.h>

uint8_t font_render_character_1bpp(uint8_t *tile, int8_t dx, int8_t dy, char c);

With 1bpp, one can use the following GBDK functions to render the character:

// Set the color for the high pixels (fgcolor) and low pixels (bgcolor)
void set_1bpp_colors(uint8_t fgcolor, uint8_t bgcolor);

// Upload `ntiles` tiles to VRAM at `start` with 8 bytes per tiles from `src`
void set_bkg_1bpp_data(uint16_t start, uint16_t ntiles, const void *src);

// Render a tile from VRAM, identified by `t`, at `x`,`y` on the background
uint8_t * set_bkg_tile_xy(uint8_t x, uint8_t y, uint8_t t);

Let’s define font_render_character_1bpp, returning 0, which is the minimum definition that compiles. I am going to implement this using Test-Driven-Development:

font.c

#include "font.h"
#include "font_data.h"

uint8_t font_render_character_1bpp(uint8_t *tile, uint8_t dx, uint8_t dy, char c) {
  (void)tile;
  (void)dx;
  (void)c;

  return 0;
}

font_test.h

#pragma once

char *font_test(void);

font_test.c

#include "font_test.h"
#include "minunit.h"
#include "font.h"

static uint8_t tile_data[32];

static char *test_font_render_character_1bpp_missing_character(void) {
  uint8_t size = font_render_character_1bpp(tile_data, 0, 0, ' ');
  mu_assert_eq(size, 0, "%d");
  return 0;
}

char *font_test(void) {
  mu_run_test(test_font_render_character_1bpp_missing_character);
  return 0;
}

This passes, as expected. Let’s add a test!

static char *test_font_render_character_1bpp_known_character(void) {
  uint8_t width = font_render_character_1bpp(tile_data, 0, 0, '!');
  mu_assert_eq(width, 1, "%d");
  return 0;
}

As expected, this fails. In TDD spirit, let’s only return the width, as that is the only thing being tested. I’m going to extend my test suite to cover all these cases, and clean up the suite a bit:

font_test.c

#include "font_test.h"
#include "minunit.h"
#include "font.h"
#include "font_data.h"

static uint8_t tile_data[32];

static uint8_t render(char c) {
  return font_render_character_1bpp(tile_data, 0, 0, c);
}

static char *test_font_render_character_1bpp_missing_character(void) {
  mu_assert_eq(render(' '), 0, "%d"); // Test a space
  mu_assert_eq(render('@'), 0, "%d"); // Test a known missing character
  mu_assert_eq(render(200), 0, "%d"); // Test beyond 7-bit ascii range
  mu_assert_eq(render(18), 0, "%d"); // Test before visible characters
  return 0;
}

static char *test_font_render_character_1bpp_known_character(void) {
  mu_assert_eq(render('!'), 1, "%d"); // Known 1-pixel wide
  mu_assert_eq(render('#'), 5, "%d"); // Known 5-pixel wide
  return 0;
}

char *font_test(void) {
  mu_run_test(test_font_render_character_1bpp_missing_character);
  mu_run_test(test_font_render_character_1bpp_known_character);
  return 0;
}

font.c

#include "font.h"
#include "font_data.h"

uint8_t font_render_character_1bpp(uint8_t *tile, uint8_t dx, uint8_t dy, char c) {
  if (c < font_data_ascii_offset
  || c > font_data_ascii_max)
    return 0;

  uint16_t i = font_data_indices[(uint8_t)c - font_data_ascii_offset];
  if (i == 0)
    return 0;

  uint8_t *char_data = &font_data[i];
  uint8_t width = char_data[0] & 0x07;

  (void)tile;
  (void)dx;
  (void)dy;

  return width;
}

Now to render the actual tiles. But first, a test! I don’t know a pretty way to test rendered graphics, but adding this after render('#') this should suffice for now:

static char *test_font_render_character_1bpp_known_character(void) {
  mu_assert_eq(render('!'), 1, "%d"); // Known 1-pixel wide

  memset(tile_data, 0, sizeof(tile_data));
  mu_assert_eq(render('#'), 5, "%d"); // Known 5-pixel wide
  mu_assert_eq(tile_data[0], 0b00000000, "%d");
  mu_assert_eq(tile_data[1], 0b01010000, "%d");
  mu_assert_eq(tile_data[2], 0b11111000, "%d");
  mu_assert_eq(tile_data[3], 0b01010000, "%d");
  mu_assert_eq(tile_data[4], 0b11111000, "%d");
  mu_assert_eq(tile_data[5], 0b01010000, "%d");
  mu_assert_eq(tile_data[6], 0b00000000, "%d");
  mu_assert_eq(tile_data[7], 0b00000000, "%d");

  return 0;
}

As expected, the test fails. Now to implement the actual rendering:

uint8_t font_render_character_1bpp(uint8_t *tile, int8_t dx, int8_t dy, char c) {
  if (c < font_data_ascii_offset
  || c > font_data_ascii_max)
    return 0;

  uint16_t i = font_data_indices[(uint8_t)c - font_data_ascii_offset];
  if (i == 0)
    return 0;

  uint8_t *char_data = &font_data[i];
  uint8_t width = char_data[0] & 0x07;
  uint8_t height = (char_data[0] >> 3) & 0x07;
  dy += ((char_data[0] >> 6) & 0x03) | ((char_data[1] & 0x01) << 2);

  uint8_t bit_index = 9;

  for (uint8_t y = 0; y < height; y++) {
    int8_t py = dy + (int8_t)y;

    if (py < 0 || py >= 8)
      continue;

    for (uint8_t x = 0; x < width; x++, bit_index++) {
      int8_t px = dx + (int8_t)x;

      if (px < 0
      || px >= 8
      || !((char_data[bit_index >> 3] >> (bit_index & 7)) & 1u))
        continue;

      tile[py] |= (uint8_t)(1u << (7 - px));
    }
  }

  return width;
}

This implementation is obviously not efficient, but the test does pass. Now to actually put them on the screen! Exciting stuff!

main.c (partial)

#include "font.h"
#include "font_data.h"
#include <string.h>

#define _tile_size 8
#define _char_count 94

static uint8_t tile_data[_tile_size * _char_count];

static void render_font_characters(void) {
  memset(tile_data, 0, sizeof(tile_data));

  for (uint8_t i = 0; i < _char_count; i++)
    font_render_character_1bpp(&tile_data[i * _tile_size], 0, 0,
      font_data_ascii_offset + i);
  set_bkg_1bpp_data(1, _char_count, tile_data);

  for (uint8_t y = 0; y < 8; y++)
    for (uint8_t x = 0; x < 12; x++)
      if (y * 12 + x < _char_count)
        set_bkg_tile_xy(x + 1, y + 1, 1 + y * 12 + x);
}

It works! With the dx & dy arguments, I can render the characters anywhere in a tile. This should make the next step easy: Variable-Width Text Rendering.

Variable-Width Rendering

For my next function, rendering a line of text, I’m going to use a function like this:

font.h

#define font_letter_spacing 1
#define font_space_width 2

typedef struct {
  const char *remainder;
  uint8_t pixel_count;
  uint8_t tile_count;
} font_render_line_result_t;

// Render a string into a 1bpp tile buffer.
//
// Characters from the string are rendered using font_render_character_1bpp,
// starting at dx & dy. Characters are rendered until the end of the string, the
// end of the line or the end of the available tiles.
//
// Once the end is reached, `remainder` is returned, representing a pointer to
// the next character in the string not rendered, or the end of the string.
// Additionally, the amount of tiles written to is returned as `tile_count`, and
// the amount of pixels rendered is returned as `pixel_count`.
font_render_line_result_t font_render_line_1bpp(
  uint8_t *tiles,
  uint8_t tiles_length,
  int8_t dx,
  int8_t dy,
  const char *string
);

Let’s write some tests for this function:

font_test.c

#define tile_data_length 4
static uint8_t tile_data[tile_data_length * 8];
static char *test_font_render_line_1bpp_simple(void) {
  memset(tile_data, 0, sizeof(tile_data));
  font_render_line_result_t result =
    font_render_line_1bpp(tile_data, tile_data_length, 0, 0, "Hi");

  // Note: I've created this font_get_line_width function as well,
  // but that's outside the scope of this blog to keep it brief.
  uint8_t expected_pixels = font_get_line_width("Hi");
  uint8_t expected_tiles = (expected_pixels + 7) / 8;

  mu_assert_eq(*result.remainder, '\0', "%c");
  mu_assert_eq(result.pixel_count, expected_pixels, "%d");
  mu_assert_eq(result.tile_count, expected_tiles, "%d");

  return 0;
}

static char *test_font_render_line_1bpp_too_long(void) {
  const char *text = "This is a long string!";
  memset(tile_data, 0, sizeof(tile_data));
  font_render_line_result_t result =
    font_render_line_1bpp(tile_data, tile_data_length, 0, 0, text);

  // because the tile_data is too short to fit the string, the remainder should
  // point to a non-NULL character
  mu_assert(*result.remainder != '\0');

  // The remainder should also be greater than the start of the text
  mu_assert(result.remainder > text);

  mu_assert_eq(result.tile_count, tile_data_length, "%d");
  mu_assert(result.pixel_count <= tile_data_length * 8);

  return 0;
}

Now to implement them.

font.c

font_render_line_result_t font_render_line_1bpp(
  uint8_t *tiles,
  uint8_t tiles_length,
  int8_t dx,
  int8_t dy,
  const char *string
) {
  uint8_t x = dx;
  const char *current_char = string;

  while (*current_char) {
    char c = *current_char;

    if (c == '\n')
      break;

    if (c == ' ') {
      x += font_space_width;
      current_char++;
      continue;
    }

    uint8_t tile_x = x / 8;
    int8_t offset_x = x % 8;

    if (tile_x >= tiles_length)
      break;

    uint8_t w = font_render_character_1bpp(&tiles[tile_x * 8], offset_x, dy, c);

    if (offset_x + w > 8 && tile_x + 1 < tiles_length) {
      offset_x -= 8;
      tile_x++;
      font_render_character_1bpp(&tiles[tile_x * 8], offset_x, dy, c);
    }

    x += w + font_letter_spacing;
    current_char++;
  }

  uint8_t rendered_width = (uint8_t)(x - dx);

  font_render_line_result_t result;
  result.remainder = current_char;
  result.pixel_count = rendered_width > 0
    ? rendered_width - font_letter_spacing : 0;
  result.tile_count = (uint8_t)((x + 7) / 8);

  return result;
}

main.c

static void render_line(const char *text) {
  // create and clear a buffer for 20 1bpp tiles
  uint8_t text_tile_data[_tile_size * 20];
  memset(text_tile_data, 0, sizeof(text_tile_data));

  // render the characters into 20 tiles, starting at x=4, y=0
  font_render_line_1bpp(text_tile_data, 20, 4, 0, text);

  // upload the 20 tiles to VRAM at id=100
  set_bkg_1bpp_data(100, 20, text_tile_data);

  // render the VRAM tiles id=100-120 at x=0-20, y=10 on the background
  for (uint8_t i = 0; i < 20; i++)
    set_bkg_tile_xy(i, 10, 100 + i);
}

void main(void) {
  render_line("Hello, world!! This is a variable-width string!");
}

It works!

And here’s the result running on real hardware:

There’s lots of room for optimization and rendering of new-lines, but that’s outside the scope of this already lengthy blog. Thanks for reading!

Modern frontend development is dominated by React and other functional-like patterns. There’s likely 1000s of frontend libraries that all attempt the same thing: Creating reusable components, embracing a functional pattern around immutable data structures. However, there is a huge problem with these libraries:

Javascript data structures are inherently not immutable.

This causes tons of issues that need careful crafting. Manipulating an object inside a useEffect can cause subtle bugs. There’s tons of potential solutions that “enforce” immutability: Immer, Redux, ImmutableJS - but they all require you to model your data around these tools. In some cases, these tools appear to fix the problems, but you will still end up with hidden gotchas.

I feel like all these functional-like frameworks try to make Javascript something it is not: A functional language with immutable datastructures.

So what is Javascript?

Javascript is a prototype-based, loosely-typed language with robust Object Oriented Programming support

Can we design a frontend development style that embraces Javascript for what it is, instead of trying to make it something it is not? I think we can. We can start looking in patterns that work for Object-Oriented-Programming languages: Model-View-Controller (MVC) and Model–View–ViewModel (MVVM).

Reading MVC, you might immediately jump to your experience with MVC platforms like Ruby on Rails, Django, ASP.NET or Laravel, but those are not the kind of MVC I want to explore today.

I am thinking of a fully client-side MVC experience. One where the views somehow convert data to HTML, controllers that handle events triggered by the views, and models that are rendered by the view and updated by the controller, all within the browser.

Testable, independent models, views and controllers

I think testing is very important. I want to systematically test every line of code written either by me or AI, including my views. Tests must run fast and have no external dependencies. Ideally, I want to test the models, views or controllers independently from each other. Let’s break them down.

The Model

The model is a very important part of the application and requires thorough testing. I want my models to be completely independent from any framework-specific nonsense. Ideally, they are just plain Javascript objects or classes with simple exposed members. They should be easily reconstructible in any state for easy testing.

class Counter {
  constructor(count = 0) {
    this.count = count;
  }

  increment() {
    this.count++;
  }
}

test(".increment increments counter", () => {
  const counter = new Counter(0);
  counter.increment();
  assert.strictEqual(counter.count, 1);
});

As is common in Object-Oriented-Programming, this model couples functions and data, which does not guarantee the model to be easily reconstructible or serializable. It would also expose all functions to any view rendering the model.

A procedural alternative, separating functions and data, could be:

const Counter = {
  create: (count = 0) => {
    return { count };
  },

  increment: (counter) => {
    counter.count++;
  },
};

test(".increment increments counter", () => {
  const counter = Counter.create();
  Counter.increment(counter);
  assert.strictEqual(counter.count, 1);
});

This separation of functions and data allows you to render the view without exposing the class methods to the view. It also allows you to work on serialized and deserialized data (e.g. data received via an API, JSON-deserialized data) without the need to reconstruct the object prototypes.

Either approach has its pros and cons, which is beyond the scope of this blog for now.

The View

I want my (client-side) views to be rendered in any state based on a model that can easily be initialized in any state. It will likely involve some kind of HTML template or existing HTML fragment which can be manipulated in a render() call, rendering the model by updating the DOM. Any user interaction should trigger an event, the view should avoid local state.

These views should be able to exist self-contained without external styles or other components. This should make it possible to preview and test the view with only the model (or a mock copy of the model), completely independent from any other views in any webpage.

One could design the views without relying on the DOM, but this will likely involve some kind of vdom or other means of patching the DOM based on rendered template strings. From a programmer’s perspective, this is nice and easy, but from a optimization perspective - I’m not a fan.

Instead, I think views should embrace the dom, either injecting a template once or relying on an existing HTML document, and manipulating it based on the model’s properties on render.

<template id="counter-view-template">
  <style>
    .root {
      color: blue;
      border: 1px solid blue;
      padding: 3px;
    }

    .count {
      font-weight: bold;
    }
  </style>
  <span class="root">
    Count: <span class="count" data-count></span>
    <button data-increment>+</button>
  </span>
</template>

class CounterView extends HTMLElement {
  constructor() {
    super();
    const template = document.getElementById("counter-view-template");
    const shadowRoot = this.attachShadow({ mode: "open" });
    shadowRoot.appendChild(template.content.cloneNode(true));

    this.countElement = shadowRoot.querySelector("[data-count]");

    shadowRoot.querySelector("[data-increment]")
      .addEventListener("click", this.handleIncrementClickEvent.bind(this));
  }

  render(counter) {
    this.countElement.textContent = String(counter.count ?? 0);
  }

  handleIncrementClickEvent(event) {
    event.preventDefault();
    this.dispatchEvent(new CustomEvent("counter-increment", {
      bubbles: true,
      composed: true
    }));
  }
}

window.customElements.define("counter-view", CounterView);

The view can be used like this:

<counter-view id="my-counter-view"></counter-view>

const counter = new Counter(69);
const myCounterView = document.getElementById("my-counter-view");

myCounterView.render(counter);

myCounterView.addEventListener("counter-increment", () => {
  counter.increment();
  myCounterView.render(counter);
});

Example fiddle: https://jsfiddle.net/cv23r7kd/7/

Testing the view

In my opinion, views need to be visually inspected in any state. You can achieve this by rendering your view in various states on a demo page. This will demonstrate how your view can be used and at the same time it can be used to verify they’re rendered correctly in the browser you’re using.

If you want automated tests, you can either run tests on NodeJS with JSDOM, or you can run tests using a client-side test runner like Mocha, in a real browser. Ideally, you want to design these tests in a way you can run them in the browser and in NodeJS. The NodeJS test suite can then run the first pass to catch any issues with the code, and another more sophisticated tool can run the test suite in a wide array of browsers and report back browser-specific issues.

Your tests could look something like this:

let counter;
let view;

beforeEach(() => {
  counter = new Counter(69);
  view = document.createElement("counter-view");
  view.render(counter);
});

test("render renders the counter's count", () => {
  expect(view.shadowRoot.querySelector("[data-count]").textContent)
    .toEqual("69")
});

test("clicking increment button emits counter-increment event", () => {
  const events = [];
  view.addEventListener("counter-increment", (e) => { events.push(e); });
  view.shadowRoot.querySelector("[data-increment]").click();
  expect(events).toMatchObject([{ type: "counter-increment" }]);
});

You can use Mocha and Jest’s expect to run your tests in the browser. You can use JSDom to run the same tests in NodeJS.

The Controller

The controller depends on the model and view. The implementation is already obvious from the previous example on how the view and model are used together:

class CounterController {
  constructor(counter, counterView) {
    this.counter = counter;

    this.handleCounterIncrement = () => {
      this.counter.increment();
      this.counterView.render(this.counter);
    };

    this.setView(counterView);
  }

  setView(newCounterView) {
    if (this.counterView)
      this.removeView();

    this.counterView = newCounterView;

    this.counterView.render(this.counter);
    this.counterView.addEventListener(
        "counter-increment", this.handleCounterIncrement);
  }

  removeView() {
    if (!this.counterView)
      return;

    this.counterView.removeEventListener(
        "counter-increment", this.handleCounterIncrement);
    this.counterView = null;
  }
}

The usage example can then be changed to:

const counter = new Counter(69);
const counterView = document.getElementById("my-counter-view");
new CounterController(counter, counterView);

You can test it here: https://jsfiddle.net/cv23r7kd/11/

You might look at this controller and think nothing much of it, but for “real” use-cases, the controller can grow fast. It can coordinate the rendering of multiple views of the same model, it can handle async actions of the view or model, and it could dynamically swap out models and views.

Testing the controller

Since the controller also relies on the views, it must be tested in the same environment as the views: The browser and/or JSDom. For simplicity, I choose to use a custom view for testing, since there’s no easy way to assert the rendered result on the CounterView. My TestCounterView extends EventTarget for addEventListener and removeEventListener.

class TestCounterView extends EventTarget {
  state = null;

  render(counter) {
    this.state = { count: counter.count };
  }
}

let counter;
let counterView;
let controller;

beforeEach(() => {
  counter = new Counter(0);
  counterView = new TestCounterView();
  controller = new CounterController(counter, counterView);
});

test("controller renders the view", () => {
  expect(counterView.state).toEqual({ count: 0 });
});

test("controller increments and renders on counter-increment event", () => {
  counterView.dispatchEvent(new CustomEvent("counter-increment"));
  expect(counter.count).toEqual(1);
  expect(counterView.state).toEqual({ count: 1 });
});

test("removeView removes the counter-increment listener", () => {
  controller.removeView();

  counterView.dispatchEvent(new CustomEvent("counter-increment"));
  expect(counter.count).toEqual(0);
  expect(counterView.state).toEqual({ count: 0 });
});

No observing of the model state

A lot of MVC and MVVM patterns currently present on the web (Backbone, Knockout, Angular 1) relied heavily on observables or automatic bindings. This makes it easy for multiple views to automatically update when a model changes, but event-driven programming is hard to debug.

I therefore choose not to make my models observable. I want my models to remain completely agnostic of any knowledge of the frontend.

For applications that rely on a large web of models, views and controllers, the controller layer needs to coordinate the updates between all models and views.

The verbosity of the DOM

The current implementation is pretty verbose and includes more boilerplate than I would like. I expect most of the boilerplate can be DRYed up using small helpers. For example:

<template id="counter-view-template">
  <style>
    .root {
      color: blue;
      border: 1px solid blue;
      padding: 3px;
    }

    .count {
      font-weight: bold;
    }
  </style>
  <span class="root">
    Count: <span class="count" data-ref="count"></span>
    <button data-ref="increment">+</button>
  </span>
</template>

class BoundTemplateHTMLElement extends HTMLElement {
  refs = {};

  constructor(templateId) {
    super();

    const template = document.getElementById(templateId);
    this.attachShadow({ mode: "open" });
    this.shadowRoot.appendChild(template.content.cloneNode(true));

    for (const elem of this.shadowRoot.querySelectorAll("[data-ref]"))
      this.refs[elem.getAttribute("data-ref")] = elem;
  }

  emit(type) {
    this.dispatchEvent(new CustomEvent(type, {
      bubbles: true,
      composed: true
    }));
  }
}

class CounterView extends BoundTemplateHTMLElement {
  constructor() {
    super("counter-view-template");

    this.refs.increment.addEventListener(
      "click", this.handleIncrementClickEvent.bind(this));
  }

  render(counter) {
    this.refs.count.textContent = String(counter.count ?? 0);
  }

  handleIncrementClickEvent(event) {
    event.preventDefault();
    this.emit("counter-increment");
  }
}

window.customElements.define("counter-view", CounterView);

An obvious alternative is LitElement, which fits in nicely with this pattern. Lit does have reactive properties, but you can just use these to read and write the view components (which also makes testing easier).

In a future blog post, I’d love to provide an example of a simple TODO app with server-side communications.

Asking ChatGPT to write a Vertex2D library will get you a complete, (likely) functional library in 30 seconds. AI is really good at writing libraries around existing problems that fit in context.

Easily accessible models (Examples: Claude 3.7, Gemini 2.5) these days have a context size of around a million tokens. As of writing this post, all blog posts we’ve written will fit in about 250K tokens, so that leaves us with a lot of tokens remaining to ask questions, discuss the blog posts, and much more. Modern AI models are big enough to be able to load complete applications into context, if the application is small enough.

I believe that a future in AI-written applications lies in creating applications consisting of many small modules, where each module is small enough to fit in the context of an AI agent, together with all the tests, documentation and requirements for that module.

How do you make applications small enough? Easy: Split them up. Delegate implementation details to dependencies. These dependencies? AI can write them for you.

AI Agents: A new employee for every task

Querying a large language model is basically like asking an experienced developer a question on his first day. Sure, he knows a lot of things, but he knows nothing about your application, business, conventions or expectations. You’ll have to tell them, every time.

Imagine you have a new employee and the only way to communicate with him is via email. To have him perform a task, you send him all relevant documentation and instruct the employee what to do.

Without thinking about it, the employee reads all the documentation and attempts to solve the problem right away.

To have this employee succeed, you’ll have to send pretty detailed instructions, and ideally you also give this employee access to anything it needs.

After the task is completed, the employee leaves. If you have follow-up questions or tasks, a new employee is assigned. This new employee knows nothing and needs to read the previous conversation to attempt to solve your problem.

These employees are your AI agents. The instructions you provide to these employees is called a prompt. You need a pretty detailed prompt to give these employees a proper chance to produce something you need, but you also need to provide no more than required or they’ll hallucinate, make stuff up, or fix unrelated things.

When these AI agents are tasked with anything non-trivial, you want the AI agents to be able to debug their code, and only their code. You want the AI to be able to write tests and run the test suite with only their code.

Dependencies need to be incredibly simple or need to be mocked predictably. If a test fails, it must be because the AI agent screwed up his own code, not because a dependency is failing.

Testable modules

We want to split our application into smaller modules, and ideally you can develop and test these modules independently from each other. Designing composable modules has always been a good idea, but it has been too easy to just create a “big ball of mud” where everything relies on everything else.

I think we will need to define very strong boundaries, almost like you’re designing micro-services, but without the actual micro-services. Instead of literally hosting many slow, inter-communicating HTTP servers, I imagine we can just create a framework that allows you to define boundaries. These boundary definitions can then be used to validate inputs and outputs of modules and isolate errors thrown by a module.

That way, calls between modules can easily be mocked in tests, and it is easier to identify which module is misbehaving when something goes wrong. By validating the boundaries and catching errors around boundaries, we can quickly identify a lot of problems early on, directly at the source.

To aid with validating and enforcing boundaries, I think it’s helpful to strongly prefer functional-style functions, only accepting and returning primitive values, or objects composed of primitive values. These value objects should be clonable and replacable by literals: ideally they’re JSON-serializable.

Let’s start with an example, imagine the structure of a blog-like website.

The Blog Structure

A blog website can have modules like this:

• SqliteDatabase: A database connection, taking SQL queries and returning the results
• PostsRepo: A module responsible for loading and saving posts on persistent storage, like a database. Can be implemented using a SqlitePostsRepo (using SqliteDatabase), FilePostsRepo saving the posts in JSON files or MemoryPostsRepo keeping them in-memory. (useful for testing?)
• PublicPosts: A module responsible for accessing posts accessible by the public, only returning posts that are published and only returning properties that should be accessible by the public. Uses a PostsRepo to load posts.
• AdminPosts: A module for managing posts, like creating, editing, publishing and deleting posts. Uses a PostsRepo to load and save posts.
• PublicPostsRss: A module wrapping PublicPosts, returning published posts as an RSS feed.
• PublicWeb: A module wrapping PublicPosts, returning published posts as HTML pages, like a homepage and a detailed post page, and maybe some additional navigational elements like “posts per month”.
• PostsWebComponents: A module rendering PublicPosts as HTML elements. Can be a react-like thing or just render to plain HTML. Is used by PublicWeb to render the page. Because styling and designing a webpage can be complicated, I would like this to be a separate module, so we can render the pages with just a bit of dummy data.
• ErrorWebComponents: A module for rendering error pages and components.

Rendering posts

To render the homepage, we’ll render a list of published posts. This will involve a variety of modules, including:

• PostsRepo: A module responsible for loading and saving posts on persistent storage, like a database.
• PublicPosts: A module using a repo to load published posts.

The boundary can look like this:

{
  "id": "PublicPosts",
  "schemas": {
    "PublicPost": {
      "id": "PublicPost",
      "type": "object",
      "properties": {
        "title": { "type": "string" },
        "publishedAt": { "$ref": "ZonedDateTime" },
        "author": { "$ref": "PublicAuthor" },
        "contentHtml": { "type": "string" },
        "href": { "type": "string" }
      }
    },
    "getLatestPosts": {
      "id": "getLatestPosts",
      "type": "function",
      "parameters": [
        { "id": "page", "type": "integer", "min": 1 }
      ],
      "output": {
        "type": "object",
        "properties": {
          "posts": { "type": "array", "items": { "$ref": "PublicPost" } }
        }
      }
    },
    "getPost": {
      "id": "getPost",
      "type": "function",
      "parameters": [
        { "id": "id", "type": "integer" }
      ],
      "output": {
        "type": "object",
        "properties": {
          "post": { "oneOf": [{ "$ref": "PublicPost" }, { "type": "null" }] },
          "error": { "oneOf": [{ "type": "string" }, { "type": "null" }] }
        }
      }
    }
  },
  "runtimeDependencies": ["PostsRepo", "PublicRoutes"]
}

We can define this boundary using zod. Notice how all inputs and outputs are JSON serializable. This PublicPosts module might be implemented like this:

export async function getLatestPosts(limit, offset) {
  const posts = await PostsRepo.findPublishedPostsDescending(offset, limit);
  return {
    posts: posts.map((post) => renderPublicPost(post))
  };
}

export async function getPost(id) {
  const post = await PostsRepo.findPublishedPostById(id);
  if (!post)
    return { error: "not-found" };
  return { post: renderPublicPost(post) };
}

function renderPublicPost(post) {
  return {
    title: post.title,
    publishedAt: post.publishedAt,
    author: renderPublicAuthor(post.author),
    contentHtml: markdown2html(post.contentMd),
    href: PublicRoutes.postPath(post),
  };
}

function renderPublicAuthor(user) {
  return {
    name: user.name,
    href: PublicRoutes.authorPath(user),
  };
}

Now imagine the PublicWeb module, which depends on PublicPosts:

{
  "id": "PublicWeb",
  "schemas": {
    "getHome": {
      "id": "getHome",
      "type": "function",
      "parameters": [
        { "id": "req", "$ref": "HttpRequest" }
      ],
      "output": { "$ref": "HttpResponse" }
    },
    "getPostDetail": {
      "id": "getPostDetail",
      "type": "function",
      "parameters": [
        { "id": "id", "type": "integer" }
      ],
      "output": { "$ref": "HttpResponse" }
    }
  },
  "runtimeDependencies": [
    "PublicPosts",
    "PostsWebComponents",
    "ErrorWebComponents"
  ]
}

export async function getHome(req) {
  const page = parseInt(req.params.page || "1", 10);
  const limit = 30;
  const response = await PublicPosts.getLatestPosts(limit, (page - 1) * limit);
  return res(200, PostsWebComponents.renderIndex(response));
}

export async function getPostDetail(req) {
  const id = parseInt(req.params.id || "", 10);
  const response = await PublicPosts.getPost(id);
  if (response.error == "not-found")
    return res(404, ErrorWebComponents.renderError("Post not found"));
  return res(200, PostsWebComponents.renderDetail(response));
}

function res(status, body) {
  return { status, headers: { contentType: "text/html; charset=utf-8" }, body };
}

In these modules, there’s a clear separation of concerns. Note that a view (PostsWebComponents.renderIndex) can’t access any data that isn’t previously loaded - no magic auto-loading of relations or unintentional access of properties here.

The PublicWeb controller can only access PublicPosts, not AdminPosts, because only PublicPosts is listed as a dependency in the schema. Even if AdminPosts were to be added as a dependency, both the call to AdminPosts.* and the addition of the AdminPosts dependency would be clear indicators that a PublicWeb module suddenly uses a module only meant in modules related to admin access.

In many applications, you can access any module or class from anywhere, even in views. Loading Admin-only sensitive data on the homepage would be just as easy as loading any other data, and you rely on code reviews to catch mistakes or carelessness.

I imagine we will have an agent designing these schemas, and other agents implementing the modules. Because the agent writing the module can’t alter any dependencies, it can automatically not access any data it isn’t supposed to access. If a module agent needs additional dependencies, it should request the schema agent.

Module agents

I don’t intend to give the agents access to a couple of source files directly. Instead, I want to give the module agent access to a virtual file containing an array of functions, tests and references (dependencies, imports) using the initial prompt and a set of tools:

• getFunctionNames(), getFunction(name), setFunction(name, params, body), deleteFunction(name): Create, update or delete a function, uniquely identified by name.
• getTestNames(), getTest(description), setTest(description, body), deleteTest(description): Create, update or delete a test, uniquely identified by description.
• getReferences(), setReference(alias, source): Request a dependency to be added, uniquely identified by alias. If the dependency is not present in the schema, the agent (or human) managing the schema will be requested to allow the dependency.
• testAll(): Runs all tests. Includes code coverage.
• setFunctionAndTest(fun, test): A test-driven-development cycle, combining setFunction and setTest. Runs a sequence of steps, aborting if any of the steps fail. Steps: 1. Run all tests, assert they succeed, 2. Add the test, assert it fails, 3. Add the function, assert the new test succeeds, 4. Run all tests, assert they succeed.

The agent will be initialized with:

• a list of all modules in the application
• a list of all possible dependencies
• full source code and tests of its module
• its current schema
• general documentation about the whole application
• a prompt for code-style

You can imagine this initial prompt will be pretty big.

I want to put a lot of emphasis on test-driven-development and getting 100% code coverage.

Mocking dependencies

In my example, PublicWeb depends on PublicPosts, which depends on PostsRepo which would depend on SqliteDatabase. We likely want to mock these dependencies at one point.

I think it works best if runtime dependencies are mocked directly. PublicWeb relies on PublicPosts, so it should mock PublicPosts. This makes sure that each module is only concerned about its declared interface, even in tests.

This leaves a strong demand for integration or end-to-end tests. I can imagine to have a separate agent writing integration tests, which doesn’t mock any module except modules that rely on infrastructure (the database, http calls, APIs, etc) and another one for end-to-end tests, which just tests against a running application.

The integration tests will likely mock the database, or run a clean in-memory database, and use real modules to setup a scenario to test with.

For example, an integration test for PostPosts and AdminPosts can look like this:

const publicRoutes = inject(PublicRoutes, { host: "http://localhost:9999/" })
const db = inject(SqliteDatabase, { databasePath: ":memory:" });
const postsRepo = inject(PostsRepo, { SqliteDatabase: db });
const adminPosts = inject(AdminPosts, { PostsRepo: postsRepo });
const publicPosts = inject(PublicPosts,
  { PostsRepo: postsRepo, PublicRoutes: publicRoutes })

const createdPost = await adminPosts.createPost({
  title: "Hello",
  contentMd: "Hello, **world**!"
});
await adminPosts.publishPost(createdPost.id);

const publishedPost = await publicPosts.getPost(post.id);
expect(publishedPost).toEqual({
  href: "http://localhost:9999/posts/1337-hello",
  title: "Hello",
  publishedAt: "2025-12-25T12:00:00+01:00[Europe/Amsterdam]",
  contentHtml: `<p>Hello, <strong>world</strong>!</p>`,
  author: {
    user: "Toby",
    href: "http://localhost:9999/users/422-toby"
  },
});

Schema changes & refactoring across boundaries

For graceful refactoring & schema changes, I’m thinking to add the possibility to have boundary migrations where you can define a set of functions to transform between old and new versions of the boundary. For example, let’s imagine we’re changing the name field of the PublicAuthor to be givenName and familyName. A transform might be defined like this:

{
  id: "PublicAuthorNameTransform",
  target: "PublicAuthor",
  from: {
    name: { type: "string" },
  },
  to: {
    givenName: { type: "string" },
    familyName: { type: "string" },
  },
  transform(author) {
    return {
      ...author,
      givenName: author.name,
      familyName: "",
    };
  }
}

This way, we can have focus on our new module with a new version of the schema without breaking any other modules.

Once the old version of the schema has been removed, the transform can be automatically deleted. The framework could automatically detect any schema mismatches and automatically detect an appropriate schema transform, and also automatically delete any transform that is no longer required.

Notice how the transform is only concerned with relevant properties. That way, multiple versions of the same schema can exist at the same time, allowing for gracefully migrations between them.

Implementation

There are a lot of tools today that take on autonomous bugfixing and software development, but these are mostly general purpose tools you can drop into an existing codebase.

I’m currently working on creating the tools to implement this (not open source, at this moment)

I imagine by designing the infrastructure, prompts and the framework together, we could get better, faster and/or cheaper results. Or it will be a huge, impractical mess that serves no purpose for real-world applications. We’ll see!

This blog is a part 3 of my bluebird blog. See part 1 and part 2

I don’t think there’s an easy way to send email without dependencies. I suppose if I really wanted email, I can use some email-as-a-service API using NodeJS’ HTTP client. However, the email-as-a-service could be considered a dependency, so I won’t.

Traditionally, if the user forgets their password, they can get a password reset link via email. Without email, this obviously isn’t possible. For bluebird, my twitter-clone, forgetting your password results in losing access to your account.

Let’s create a signup and login process. Before I start writing HTML, I want to implement the authentication (signup + login) process in a module for easy testing.

The authentication process will be handled by a class named Auth and will feature at least 2 methods: login and signup.

Signup

The signup method needs to create a user if it doesn’t already exists. The created user should be returned. If the user does exist, some kind of error is returned.

For a method to return either an error or a user, I’m going to make the function method return an object with either an error property for failure, or a user property for success.

Following the Red-Green-Refactor rule, I’m writing a failing test first, then write a minimal implementation to make it pass.

// test/auth.test.mjs
import { describe, it, beforeEach } from "node:test";
import { Auth } from "../src/auth.mjs";
import assert from "node:assert";

describe("Auth", () => {
  describe("signup", () => {
    it("returns created user", async () => {
      const auth = new Auth();
      const { user } = await auth.signup("foo", "bar");
      assert(user);
    });
  });
});

// src/auth.mjs
export class Auth {
  async signup(name, pass) {
    return { user: { name } };
  }
}

Now to check for existing users. Note how I use JSDoc for typing. JSDoc is completely ignored by NodeJS, but it does help in some editors when the type of a variable isn’t obvious.

// test/auth.test.mjs (partial)
describe("signup", () => {
  /** @type {Auth} */
  let auth;

  beforeEach(() => {
    auth = new Auth();
  });

  it("returns created user", async () => {
    const { user } = await auth.signup("foo", "bar");
    assert(user);
  });

  it("returns user-exists error if user already exists", async () => {
    const { user } = await auth.signup("foo", "bar");
    assert(user);

    const { error } = await auth.signup("foo", "bar");
    assert.equal(error, Auth.USER_EXISTS);
  });
});

// src/auth.mjs
/**
 * @typedef {Object} User
 * @property {string} name
 */

export class Auth {
  static USER_EXISTS = Symbol("USER_EXISTS");

  /** @type {Map<string, User>} */
  #users = new Map();

  /**
   *
   * @param {string} name
   * @param {string} pass
   * @returns {Promise<{ user: User }|{ error: symbol }>}
   */
  async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name };
    this.#users.set(name, user);
    return { user };
  }
}

Using the current implementation, I’m storing the created users in a variable. Later, I’ll find a way to persist them in some kind of database.

Also note how I haven’t implemented any password hashing. There is no way to test whether a password is stored or hashed properly without writing an additional method to verify the password, so that’s what I’m going to do now using the login-method.

Login

I continue writing tests.

// test/auth.test.mjs (partial)
describe("login", () => {
  it("for unknown user, return USER_NOT_FOUND", async () => {
    const { error } = await auth.login("foo", "bar");
    assert.equal(error, Auth.USER_NOT_FOUND);
  });
});

// src/auth.mjs (partial)
class Auth {
  /**
   *
   * @param {string} name
   * @param {string} pass
   * @returns {Promise<{ user: User }|{ error: symbol }>}
   */
  async login(name, pass) {
    if (!this.#users.has(name))
      return { error: Auth.USER_NOT_FOUND };
  }
}

OWASP considers it a security risk to let the user know whether a username exists or not. They suggest to respond with a generic error message regardless of wether the password was incorrect, the account doesn’t exist or the account is disabled.

However, in many implementations, there’s still other ways for any user to find whether a username exists or not. Most notably, the signup screen generally shows you a user with that username already exists, and password recovery often also returns some kind of error indicating whether the email address is known or not.

OWASP suggests the signup page and account recovery to not report whether a user exists or not and instead send an email to the user. For this project, I don’t have email available, so that’s not going to work.

Additionally, I don’t think the security issue is worth the hit on usability. I’m perfectly comfortable with rendering useful errors to users at the cost of this potential security risk. Let’s continue!

// test/auth.test.mjs (partial)
it("for known user but invalid password, return INVALID_PASSWORD", async () => {
  await auth.signup("foo", "bar");
  const { error } = await auth.login("foo", "invalid");
  assert.equal(error, Auth.INVALID_PASSWORD);
});

// src/auth.mjs (partial)
class User {
  async login(name, pass) {
    if (!this.#users.has(name))
      return { error: Auth.USER_NOT_FOUND };
    return { error: Auth.INVALID_PASSWORD };
  }
}

// test/auth.test.mjs (partial)
it("for known user and valid password, return user", async () => {
  await auth.signup("foo", "bar");
  const { user } = await auth.login("foo", "bar");
  assert(user);
});

What’s the minimal implementation to check a password? Yep: plain text!

class User {
    async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name, pass }; // <-- Don't do this!
    this.#users.set(name, user);
    return { user };
  }

  async login(name, pass) {
    const user = this.#users.get(name);
    if (!user)
      return { error: Auth.USER_NOT_FOUND };
    if (user.pass != pass) // <-- Don't do this!
      return { error: Auth.INVALID_PASSWORD };
    return { user };
  }
}

All tests pass. Now to find a way to make it not pass. First of all, I never want the password exposed. There’s no perfect way to test whether the password is exposed by the returned user object, but a very minimal, naive approach would be to check none of the returned properties has the password as a value.

I’m going to reuse the last test I wrote for that purpose, iterating all properties of the returned object and checking for the password.

it("for known user and valid password, return user", async () => {
  await auth.signup("foo", "secret");
  const { user } = await auth.login("foo", "secret");
  assert(user);
  for (const k in user)
    assert.notEqual(user[k], "secret");
});

To hide the password, I’m going to hash the password in some way. The last time I hashed passwords myself, I used a single round of md5. This was a long time ago. After that, I only used bcrypt.

But that’s a dependency. NodeJS seems to offer scrypt which is basically the same as bcrypt and only differs in one letter.

Unlike the bcrypt implementations I’m used to, scrypt requires you to bring your own salt. Scrypt seems to accept all kinds of parameters and options, so I’m going to wrap scrypt in my own hashing module which basically has a method for hashing a password and comparing a hash with a password.

I’m going to create a verify and hash method in the Pass class and their behaviour is pretty simple.

test/pass.test.mjs

import { describe, it, beforeEach } from "node:test";
import { Pass } from "../src/pass.mjs";
import assert from "node:assert";

describe("Pass", () => {
  /** @type {Pass} */
  let pass;

  beforeEach(() => {
    pass = new Pass({ cost: 2 });
  });

  it("hash creates a password hash", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(typeof hash, "string");
  });

  it("Pass.verify returns true for the correct password", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(await Pass.verify(hash, "my-password"), true);
  });

  it("Pass.verify returns false for any other password", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(await Pass.verify(hash, "wrong-password"), false);
  });
});

src/pass.mjs

import crypto from "node:crypto";
import { promisify } from "node:util";

const scrypt = promisify(crypto.scrypt);
const randomBytes = promisify(crypto.randomBytes);

const pattern = /^\$s0\$(?<cost>\d+)\$(?<keylen>\d+)\$(?<salt>[^$]+)\$(?<key>[^$]+)$/

export class Pass {
  #keylen = 60;
  #cost = 16384;

  constructor({ keylen, cost } = {}) {
    this.#keylen = keylen ?? 60;
    this.#cost = cost ?? 16384;
  }

  /**
   *
   * @param {string} password
   * @returns {Promise<string>}
   */
  async hash(password) {
    return Pass.hash(password, await randomBytes(24), this.#keylen, this.#cost);
  }

  /**
   *
   * @param {string} hash
   * @param {string} password
   * @returns {Promise<boolean>}
   *
   */
  static async verify(hash, password) {
    const result = hash.match(pattern);
    if (!result) throw new Error(`Pass.verify was given an invalid string as a hash`);
    const cost = parseInt(result.groups.cost, 10);
    const keylen = parseInt(result.groups.keylen, 10);
    const salt = Buffer.from(result.groups.salt, "base64");
    const expectedHash = await Pass.hash(password, salt, keylen, cost);
    return expectedHash === hash;
  }

  /**
   *
   * @param {string} password
   * @param {Buffer} salt
   * @param {number} keylen
   * @param {number} cost
   */
  static async hash(password, salt, keylen, cost) {
    const key = await scrypt(password, salt, keylen, { cost });
    return `$s0$${cost}$${keylen}$${salt.toString("base64")}$${key.toString("base64")}`;
  }
}

I intentionally made Pass.verify a static method to make sure it doesn’t rely on the config of a Pass-instance. That way, the tests automatically prove that a generated hash can be verified regardless the settings used at creation, since there is no way to access instance variables from a static method.

I put all required parameters inside the generated password hash string, like bcrypt does. That way, I can safely change the parameters like cost, salt size or keylen in future releases without breaking existing password hashes.

Why $s0$? I don’t know, I liked it. Bcrypt uses a similar prefix for versioning, so I thought I use it as well: s for scrypt and 0 for version 0.

For testing, I’m going to set the cost very low, but for real applications I would put the cost on a value to ensure password hashing takes about a second.

Back to the Auth-class. It’s going to need an instance of Pass for hashing passwords.

// test/auth.test.mjs (partial)
beforeEach(() => {
  auth = new Auth({
    pass: new Pass({ cost: 2 })
  });
});

// src/auth.mjs
import { Pass } from "./pass.mjs";

/**
 * @typedef {Object} User
 * @property {string} name
 * @property {string} passHash
 */

export class Auth {
  static INVALID_PASSWORD = Symbol("INVALID_PASSWORD");
  static USER_EXISTS = Symbol("USER_EXISTS");
  static USER_NOT_FOUND = Symbol("USER_NOT_FOUND");

  /** @type {Map<string, User>} */
  #users = new Map();

  /** @type {Pass} */
  #pass;

  /** @param  opts */
  constructor({ pass }) {
    this.#pass = pass;
  }

  async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name, passHash: await this.#pass.hash(pass) };
    this.#users.set(name, user);
    return { user };
  }

  async login(name, pass) {
    const user = this.#users.get(name);
    if (!user)
      return { error: Auth.USER_NOT_FOUND };
    if (!await Pass.verify(user.passHash, pass))
      return { error: Auth.INVALID_PASSWORD };
    return { user };
  }
}

All tests now pass and the modules are ready for signup and login. Let’s create the login and signup routes in App.

App: Controllers & Views

Our app already includes routes for login, specifically:

// src/app.mjs (partial)
export class App {
  async "GET /login"() {
    return render("login.ejs", { errorCode: null, params: { username: "" } });
  }

  async "POST /login"({ body }) {
    const { username, password } = body;
    const { user, error } = await this.#auth.login(username, password);

    if (user)
      return redirect("/profile", { cookies: { username: user.name } });

    return render("login.ejs", {
      errorCode: error.description,
      params: { username }
    });
  }
}

These routes were part of my previous blog.

The signup routes are just as simple. The GET /signup will just render a template with default params, and the POST /signup will delegate the params to the model (the Auth-module)

But before I write those, I’m going to write some failing tests first!

// app.test.mjs (partial)
describe("App", () => {
  describe("GET /signup", () => {
    it("renders signup template", async () => {
      await GET("/signup");
      assertRender("signup.ejs",
        { params: { username: "" }, errorCode: null });
    });
  });

  describe("POST /signup", () => {
    it("for existing user, return a USER_EXISTS error", async () => {
      await POST("/signup", { username: "foo", password: "bar" });

      assertRender("signup.ejs", {
        params: { username: "foo" },
        errorCode: "USER_EXISTS",
      });
    });

    it("with valid params, sets cookie and redirects to profile", async () => {
      await POST("/signup", { username: "foo", password: "bar" });
      assertRedirect("/profile", { cookies: { username: "foo" } });
    });
  });
});

These tests are very similar to the tests for login. The implementation is also very similar. So similar, I might as well copy/paste them.

// src/app.mjs (partial)
export class App {
  async "GET /signup"() {
    return render("signup.ejs", { errorCode: null, params: { username: "" } });
  }

  async "POST /signup"({ body }) {
    const { username, password } = body;
    const { user, error } = await this.#auth.signup(username, password);

    if (user)
      return redirect("/profile", { cookies: { username: user.name } });

    return render("signup.ejs", {
      errorCode: error.description,
      params: { username }
    });
  }
}

The template is just as interesting:

<!-- signup.ejs -->
<h1>Signup!</h1>

<form method="POST" action="/signup">
  <% if (errorCode) { %>
    <p><%= {
      "USER_EXISTS": "User exists"
    }[errorCode] ?? errorCode %></p>
  <% } %>

  <label>
    <strong>Username</strong><br>
    <input type="text" name="username" value="<%= params.username %>">
  </label><br>

  <label>
    <strong>Password</strong><br>
    <input type="password" name="password">
  </label><br>

  <button type="submit">Signup</button>
</form>

This completes a very basic signup feature. However, the /profile page does not yet exist. To complete this blog, I’m going to implement that page!

The Profile Page

I intent to make cookies readable like this:

// src/app.mjs (partial)
export class App {
  "GET /profile"({ cookies }) {
    return render("profile.ejs", { name: cookies.username });
  }
}

<!-- templates/profile.ejs -->
<h1>Hello, <%= name %>!</h1>

This route and template serve as a placeholder while I implement the handling of cookies. Let’s first implement that any cookie gets passed to the request handler.

Insecure cookies

Let’s first update the AppRequest type definition.

// src/app.mjs (partial)
/**
 * @typedef {Object} AppRequest
 * @property {AppRequestMethod} method
 * @property {string} path
 * @property {AppRequestParams} body
 * @property {Record<string, string|null>} cookies
 */

This has no use in NodeJS whatsoever, but it helps documenting the code and adds type hints to the editor. Now we just have to parse cookies from the cookie header:

// src/main.mjs (partial)
import { parseCookies } from "./cookies.mjs";

// inside http.createServer's callback:
let { status, json, headers, template, variables, cookies, location } =
  await app.handleRequest({
    method: req.method,
    path: url.pathname,
    cookies: parseCookies(req.headers.cookie),
    body,
  });

I was too lazy to implement parseCookies myself so I asked ChatGPT:

// cookies.mjs
export function parseCookies(cookieHeader) {
  if (!cookieHeader) return {};

  return cookieHeader.split(';').reduce((cookies, cookie) => {
    const [key, value] = cookie.split('=').map(part => part.trim());
    if (key) cookies[key] = decodeURIComponent(value || '');
    return cookies;
  }, {});
}

ChatGPT’s implementation seems totally reasonable so I’ll just go with it.

I have no intention to create a test for src/main.mjs at the moment, so I’ll just manually test it (by starting the server and signing up via the browser). After a quick check, it seems to work perfectly fine.

I do intent to test /profile, so let’s just break TDD’s rules and write a test after writing the implementation.

First, I need to modify my GET and POST methods to handle cookies. I add an overrides param which can be used to include cookies. I use this new param in my new tests:

// app.test.mjs (partial)
describe("GET /profile", () => {
  it("with cookie, returns user", async () => {
    await GET("/profile", { cookies: { username: "foo" } });
    assertRender("profile.ejs", { name: "foo" });
  });

  it("without cookie, redirects to login", async () => {
    await GET("/profile");
    assertRedirect("/login");
  });
});

async function POST(path, body, overrides = {}) {
  res = await app.handleRequest({ method: "POST", path, body, cookies: {}, ...overrides });
}

async function GET(path, overrides = {}) {
  res = await app.handleRequest({ method: "GET", path, cookies: {}, ...overrides });
}

I need to handle the unauthenticated case in my GET /profile method:

// app.mjs
"GET /profile"({ cookies, path }) {
  if (!cookies.username)
    return redirect("/login")
  return render("profile.ejs", { name: cookies.username });
}

There is a huge flaw with this approach: Cookies are basically user input. Any visitor can alter the contents of the cookie, so anyone can now login as any user by just submitting a cookie with the user’s username.

To reject tainted cookies, or cookies that are otherwise manipulated, I’ll complete this blog by signing the cookies with a secret.

First, I’ll generate a secret and save it in my .envrc:

echo "export SECRET='$(openssl rand -base64 48)'" >> .envrc
direnv allow

Now to sign the cookies with this secret using HMAC. I’m going to create two helper functions for this. I’ll also have all cookies signed by default, and verify all cookies in every request.

// src/cookies.mjs (partial)
import crypto from "node:crypto";

/**.
 * @param {string} value
 * @param {string} [secret=process.env.SECRET]
 * @returns {string}
 */
function signCookie(value, secret = process.env.SECRET) {
  const hmac = crypto.createHmac("sha256", secret);
  hmac.update(value);
  const signature = hmac.digest("base64url");
  return `${value}.${signature}`;
}

/**
 * @param {string} signedValue
 * @param {string} [secret=process.env.SECRET]
 * @returns {string|null}
 */
function verifyCookie(signedValue, secret = process.env.SECRET) {
  const [value, signature] = signedValue.split(".");
  if (!value || !signature) return null;

  const expectedSignature = crypto
    .createHmac("sha256", secret)
    .update(value)
    .digest("base64url");

  if (crypto.timingSafeEqual(
    Buffer.from(signature), Buffer.from(expectedSignature))) {
    return value;
  }

  return null;
}

export function parseCookies(cookieHeader, secret = process.env.SECRET) {
  if (!cookieHeader) return {};

  return cookieHeader.split(';').reduce((cookies, cookie) => {
    const [key, value] = cookie.split('=').map(part => part.trim());
    if (key)
      cookies[key] = verifyCookie(decodeURIComponent(value || ''), secret);
    return cookies;
  }, {});
}

// src/main.mjs (partial)
if (cookies)
  for (const key in cookies)
    if (cookies.hasOwnProperty(key)) {
      const value = [
        `${key}=${signCookie(cookies[key]) ?? ""}`,
        `Path=/`,
        `HttpOnly`,
        `SameSite=Strict`,
        cookies[key] ? `Max-Age=${YEAR_SECONDS}` : `Max-Age=-1`
      ].join("; ");
      res.setHeader("set-cookie", value);
    }

Since the signing and verifying of cookies is handled in src/main.mjs, none of the request handlers need to change their implementation. The handlers receive verified cookies, and they can set cookies which are always signed automatically. I don’t have to change any of my handlers or their tests.

Again, I tested the change manually by checking if the browser gets assigned a signed cookie, and altering the cookie breaks the login. It did, so it works! In a real production-ready application, I would very likely add some tests specifically testing the HTTP request & response handling.

This blog is a part 2 of my bluebird blog. See part 1

For my twitter clone without dependencies, I’ve decided to design my route handlers in a way that they’ll return variables and a template name. This allows for easy testing, as my tests can just do assertions on the template name and variables rather than inspecting a HTML document.

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  template: "public-profile-show",
  variables: {
    user: {
      id: 54,
      name: "John Doe",
    },
    posts: [
      { id: 55412, message: "Have you seen the new iThing?",
        createdAt: 1699788972 }
    ]
  }
}

In this blog, I’m going to implement this template language.

Designing the template language

The template language I need, needs to output HTML documents with only a set of variables as input. I want a template to be compiled to a JS function. For example, I want Hello <%= name %> to compile to something like this:

({ name }) => `Hello ${escapeHtml(name)}`;

I’ll go with a classic <%= %> syntax, because it’s incredibly common and well known. Most developers that see this syntax will intuitively know that you can just write regular code in there and the output of the code will be added to the output.

It must support variables and auto-escape HTML entities. Loops, if/else statements and including other templates must also be supported. It would be nice if we could invoke arbitrary functions and do some basic math;

So basically, I want it to be able to execute arbitrary javascript.

The implementation

I guess I just get started writing code and see where I end up. First, a test.

it("simple template", () => {
  const fn = Template.parse("Hello, <%= name %>");
  assert.equal(fn({ name: "world" }), "Hello, world");
});

The simplest implementation I can think of is to use a regular expression. All content outside <%= %> will just be added to the output, and the content between will be executed as JS.

The regular expression used is /(.*?)<%(.*?)%>/sg. This regular expression captures any text until the first <% it finds using (.*?)<%. Then it captures any text until %> using (.*?)%>. The s modifier allows the . (dot) to match newlines. The g modifier allows multiple matches.

Javascript’s replace function on string allows executing code for every match while also returning a replacement value, "" in my code. Because every match is replaced by an empty string, only the text after the last %> is returned by the replace-function, which I call the tail.

I use JSON.stringify to create a string literal.

const Template = {
  parse(template) {
    let body = [
      "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
      `out = [];`
    ];
    const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
      body.push(`out.push(${JSON.stringify(content)});`);

      if (code.startsWith("="))
        body.push(`out.push(${code.substr(1)});`);

      return "";
    });

    body.push(`out.push(${JSON.stringify(tail)});`);
    body.push(`return out.join("");`);

    return new Function('vars', body.join("\n"))
  }
};

For the template in the test, this function returns a function like this:

function(vars) {
  eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);
  out = [];
  out.push("Hello, ");
  out.push(name);
  out.push("");
  return out.join("");
}

Another notable part of this code is the eval statement. To allow the template to refer to any variable in vars (name in this example), I need to make the properties in vars available as local variables in the function.

There is no easy way to determine the possible variables while compiling, so I generate them at runtime. The only way I know to assign arbitrary local variables at runtime, is to use eval.

var { foo } = { foo: 1 };
// foo = 1
eval('var { bar } = { bar: 2 }');
// bar = 2

Another method is to use the with-statement, which is discouraged. Let’s try it anyway.

function(vars) {
  with (vars) {
    out = [];
    out.push("Hello, ");
    out.push(name);
    out.push("");
    return out.join("");
  }
}

The generated function works perfectly. Too bad the feature is discouraged, legacy or deprecated, depending on who you ask. So far my options are evil eval or deprecated with. Ideally, I want to determine the variables used at compile-time, but this requires compiling the Javascript code to determine the variables used.

There is no easy way to get an abstract syntax tree of some piece of Javascript using plain NodeJS.

Now to escape HTML entities, support if/else statements and add some minor fixes.

// Template.parse
let body = [
  "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
  `out = [];`
];
const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
  if (content)
    body.push(`out.push(${JSON.stringify(content)});`);

  if (code.startsWith("="))
    body.push(`out.push(escapeHtml(${code.substr(1)}));`);
  else if (code.startsWith("-"))
    body.push(`out.push(${code.substr(1)});`);
  else
    body.push(code);

  return "";
});

if (tail.length > 0) body.push(`out.push(${JSON.stringify(tail)});`);

body.push(`return out.join("");`);
body = body.join("\n");

const fn = new Function('vars', body);
return (vars) => fn({ ...vars, ...Template.locals });

// Template.locals
locals: {
  escapeHtml: (str) => `${str}`.replace(/[<>&"']/g, s =>
    ({ "<": "&lt;", ">": "&gt;", "&": "&amp;", '"': "&quot;", "'": "&#39;" })[s])
}

I also added some more tests.

describe("Template.parse", () => {
  it("simple template", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "world" }), "Hello, world");
  });

  it("math template", () => {
    const fn = Template.parse("1 + 1 = <%= 1 + 1 %>");
    assert.equal(fn({}), "1 + 1 = 2");
  });

  it("function template", () => {
    const fn = Template.parse("Hello <%= foo() %>");
    assert.equal(fn({ foo: () => "world" }), "Hello world");
  });

  it("if-else template", () => {
    const fn = Template.parse(`Answer: <% if (answer) { %>Yes<% } else { %>No<% } %>`);
    assert.deepEqual(
      [fn({ answer: true }), fn({ answer: false })],
      ["Answer: Yes", "Answer: No"]);
  });

  it("multiline template", () => {
    const fn = Template.parse(`
    Answer:
    <% if (answer) { %>
      Yes
    <% } else { %>
      No
    <% } %>
  `);
    assert.deepEqual(
      [delws(fn({ answer: true })), delws(fn({ answer: false }))],
      ["Answer: Yes", "Answer: No"]);
  });

  it("escape html", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "<script> & \" '" }), "Hello, &lt;script&gt; &amp; &quot; &#39;");
  });
});

function delws(str) {
  return str.replace(/^\s+|\s+$/g, "").replace(/\s+/g, " ");
}