tech

Good and bad APIs: IndexedDB

The web has its own database: IndexedDB. But, it's API is so bad everyone stays away from it. But, other solutions are not ready yet. So, I am working on making IndexedDB better with Effect.


Sandro Maglione

Sandro Maglione

Software

I ventured on a quest (aka fight) with IndexedDB: bending its weird API to conform to the power of the effect model 👊

I learned a couple of things (and more) about API design, let me show you 👇


IndexedDB is madness

Let me give you a TLDR of why IndexedDB is so bad:

  • No Promise, all callback (errors included)
  • Versioning and migrations restricted inside a callback (onupgradeneeded)
  • Severely untyped
const database = yield* Effect.async(() => {
  const request = window.indexedDB.open(identifier, version);

  request.onblocked = (event) => {
    // ...
  };

  request.onerror = (event) => {
    // ...
  };

  // If `onupgradeneeded` exits successfully, `onsuccess` will then be triggered
  request.onupgradeneeded = (event) => {
    const idbRequest = event.target as IDBRequest<IDBDatabase>;
    const database = idbRequest.result;
    const transaction = idbRequest.transaction;
    const oldVersion = event.oldVersion;

    if (transaction !== null) {
      transaction.onabort = (event) => {
        // ...
      };

      transaction.onerror = (event) => {
        // ...
      };
    }
  };

  request.onsuccess = (event) => {
    const idbRequest = event.target as IDBRequest<IDBDatabase>;
    const database = idbRequest.result;
    // ...
  };
});

Forget about modern TypeScript. You mostly cannot even rely on try/catch. The database on the web of today is all callbacks-hell and jumping between scopes 🤯

DexieJs to the rescue (partially)

The madness of IndexedDB made solutions like Dexie a must-have.

You just create a new Dexie instance, add your keys and schema, and then interact with a sane API (get, add, delete, all Promise-based):

import Dexie, { type EntityTable } from 'dexie';

interface Friend {
  id: number;
  name: string;
  age: number;
}

const db = new Dexie('FriendsDatabase') as Dexie & {
  friends: EntityTable<
    Friend,
    'id'
  >;
};

db.version(1).stores({
  friends: '++id, name, age'
});

export type { Friend };
export { db };

I used Dexie a lot in many projects (DexieJs: Reactive local state in React) 👈

But, I am still not satisfied. The model is still too weak and not strongly-typed enough.

I want a clean IndexedDB effect module: composition, layers, full type-safety, better migrations, and error handling.

That's what I started to implement 🫡

API design for full type-safety

My goals:

  • Full type safety (typed errors, schema, migrations, queries)
  • Schema and data migrations
  • Composition with services and layers
  • Completely abstract away the underlying IndexedDB API

As of right now (still work in progress), this is what I have:

  • IndexedDbTable: Model for a table with Schema
  • IndexedDbVersion: Single database version (collection of IndexedDbTable)
  • IndexedDbMigration: Schema/data migrations (sequence of IndexedDbVersion)
  • IndexedDb: Service to open database and apply migrations
  • IndexedDbQuery: Service to perform type-safe queries

This requires some tricky magic with TypeScript types 🪄

export const make = <
  Tables extends readonly IndexedDbTable.IndexedDbTable.Any[]
>(
  // Not empty list of generic tables, each unique based on a shared model, all inferred 🙌
  ...tables: Tables & { 0: IndexedDbTable.IndexedDbTable.Any }
): IndexedDbVersion<Tables[number]> => {
  return makeProto({
    tables: HashMap.fromIterable(
      tables.map((table) => [table.tableName, table])
    ),
  });
};

The end result for the user is a minimal API that just works, that also accommodates all use cases without manual typing:

// 1️⃣ Define tables with schema and options
const Table1 = IndexedDbTable.make(
  "table1",
  Schema.Struct({
    id: Schema.UUID,
    value: Schema.Number,
  }),
  { keyPath: "id" }
);

const Table2_1 = IndexedDbTable.make(
  "table2",
  Schema.Struct({
    name: Schema.String,
    age: Schema.Number,
  }),
  { keyPath: "name" }
);

const Table2_2 = IndexedDbTable.make(
  "table2",
  Schema.Struct({
    nameAndAge: Schema.String,
  }),
  { keyPath: "nameAndAge" }
);

// 2️⃣ Define database version (collection of tables)
const CurrentDb = IndexedDbVersion.make(Table1, Table2_2);

// 3️⃣ Define migration (initial database creation and first migration)
const Migration1 = IndexedDbMigration.make({
  fromVersion: IndexedDbVersion.makeEmpty,
  toVersion: CurrentDb,
  execute: (_, toQuery) =>
    Effect.gen(function* () {
      yield* toQuery.createObjectStore("table1");
      yield* toQuery.createObjectStore("table2");
    }),
});

const Migration2 = IndexedDbMigration.make({
  fromVersion: CurrentDb,
  toVersion: CurrentDb,
  execute: (_, toQuery) =>
    Effect.gen(function* () {
      yield* toQuery.deleteObjectStore("table2");
    }),
});

// 4️⃣ Compose layers with database name and migration sequence
const layer = IndexedDbQuery.layer.pipe(
  Layer.provide(IndexedDb.layer("db", Migration1, Migration2))
);

// 5️⃣ Use type-safe API to query database
export const main = Effect.gen(function* () {
  const { makeApi } = yield* IndexedDbQuery.IndexedDbApi;
  const api = makeApi(CurrentDb);
  const key = yield* api.insert("table1", {
    id: "2",
    value: 2,
  });
  yield* Effect.log(key);
}).pipe(Effect.provide(layer));

This is the result of multiple iterations, and questions on "But why?" and "Is this even possible?" 🫠

I aim to make this functional enough to land inside @effect/platform-browser. I plan to open a PR once I am confident this works as it should 🔜


I am pushing for the client to become more and more prominent. One of the key features of any app is a database, and the client deserves a great API for it 🫡

Expect more details coming this and next week, and more behind the scenes 🔜

See you next 👋

Start here.

Every week I dive headfirst into a topic, uncovering every hidden nook and shadow, to deliver you the most interesting insights

Not convinced? Well, let me tell you more about it