I ventured on a quest (aka fight) with IndexedDB: bending its weird API to conform to the power of the effect
model 👊
Okay, something works here with IndexedDb and @EffectTS_ The IndexedDb layer creates the database, IndexedDbQuery provides a type-safe API based on IndexedDbVersion Keep working on it 🔬
I learned a couple of things (and more) about API design, let me show you 👇
IndexedDB is madness
Let me give you a TLDR of why IndexedDB is so bad:
- No
Promise
, all callback (errors included) - Versioning and migrations restricted inside a callback (
onupgradeneeded
) - Severely untyped
const database = yield* Effect.async(() => {
const request = window.indexedDB.open(identifier, version);
request.onblocked = (event) => {
// ...
};
request.onerror = (event) => {
// ...
};
// If `onupgradeneeded` exits successfully, `onsuccess` will then be triggered
request.onupgradeneeded = (event) => {
const idbRequest = event.target as IDBRequest<IDBDatabase>;
const database = idbRequest.result;
const transaction = idbRequest.transaction;
const oldVersion = event.oldVersion;
if (transaction !== null) {
transaction.onabort = (event) => {
// ...
};
transaction.onerror = (event) => {
// ...
};
}
};
request.onsuccess = (event) => {
const idbRequest = event.target as IDBRequest<IDBDatabase>;
const database = idbRequest.result;
// ...
};
});
Forget about modern TypeScript. You mostly cannot even rely on try
/catch
. The database on the web of today is all callbacks-hell and jumping between scopes 🤯
DexieJs to the rescue (partially)
The madness of IndexedDB made solutions like Dexie a must-have.
You just create a new Dexie
instance, add your keys and schema, and then interact with a sane API (get
, add
, delete
, all Promise
-based):
import Dexie, { type EntityTable } from 'dexie';
interface Friend {
id: number;
name: string;
age: number;
}
const db = new Dexie('FriendsDatabase') as Dexie & {
friends: EntityTable<
Friend,
'id'
>;
};
db.version(1).stores({
friends: '++id, name, age'
});
export type { Friend };
export { db };
I used Dexie a lot in many projects (DexieJs: Reactive local state in React) 👈
But, I am still not satisfied. The model is still too weak and not strongly-typed enough.
I want a clean IndexedDB
effect
module: composition, layers, full type-safety, better migrations, and error handling.That's what I started to implement 🫡
I did some experiments with IndexedDB in @EffectTS_ IndexedDB: Weird API, mostly untyped, full of callbacks, errors behind every corner Then you get into migrations, queries, and reactivity and it all breaks down I am (nearly) convinced that is better to bet on SQLite+File
API design for full type-safety
My goals:
- Full type safety (typed errors, schema, migrations, queries)
- Schema and data migrations
- Composition with services and layers
- Completely abstract away the underlying IndexedDB API
As of right now (still work in progress), this is what I have:
IndexedDbTable
: Model for a table withSchema
IndexedDbVersion
: Single database version (collection ofIndexedDbTable
)IndexedDbMigration
: Schema/data migrations (sequence ofIndexedDbVersion
)IndexedDb
: Service to open database and apply migrationsIndexedDbQuery
: Service to perform type-safe queries
This requires some tricky magic with TypeScript types 🪄
export const make = <
Tables extends readonly IndexedDbTable.IndexedDbTable.Any[]
>(
// Not empty list of generic tables, each unique based on a shared model, all inferred 🙌
...tables: Tables & { 0: IndexedDbTable.IndexedDbTable.Any }
): IndexedDbVersion<Tables[number]> => {
return makeProto({
tables: HashMap.fromIterable(
tables.map((table) => [table.tableName, table])
),
});
};
The end result for the user is a minimal API that just works, that also accommodates all use cases without manual typing:
// 1️⃣ Define tables with schema and options
const Table1 = IndexedDbTable.make(
"table1",
Schema.Struct({
id: Schema.UUID,
value: Schema.Number,
}),
{ keyPath: "id" }
);
const Table2_1 = IndexedDbTable.make(
"table2",
Schema.Struct({
name: Schema.String,
age: Schema.Number,
}),
{ keyPath: "name" }
);
const Table2_2 = IndexedDbTable.make(
"table2",
Schema.Struct({
nameAndAge: Schema.String,
}),
{ keyPath: "nameAndAge" }
);
// 2️⃣ Define database version (collection of tables)
const CurrentDb = IndexedDbVersion.make(Table1, Table2_2);
// 3️⃣ Define migration (initial database creation and first migration)
const Migration1 = IndexedDbMigration.make({
fromVersion: IndexedDbVersion.makeEmpty,
toVersion: CurrentDb,
execute: (_, toQuery) =>
Effect.gen(function* () {
yield* toQuery.createObjectStore("table1");
yield* toQuery.createObjectStore("table2");
}),
});
const Migration2 = IndexedDbMigration.make({
fromVersion: CurrentDb,
toVersion: CurrentDb,
execute: (_, toQuery) =>
Effect.gen(function* () {
yield* toQuery.deleteObjectStore("table2");
}),
});
// 4️⃣ Compose layers with database name and migration sequence
const layer = IndexedDbQuery.layer.pipe(
Layer.provide(IndexedDb.layer("db", Migration1, Migration2))
);
// 5️⃣ Use type-safe API to query database
export const main = Effect.gen(function* () {
const { makeApi } = yield* IndexedDbQuery.IndexedDbApi;
const api = makeApi(CurrentDb);
const key = yield* api.insert("table1", {
id: "2",
value: 2,
});
yield* Effect.log(key);
}).pipe(Effect.provide(layer));
This is the result of multiple iterations, and questions on "But why?" and "Is this even possible?" 🫠
I aim to make this functional enough to land inside
@effect/platform-browser
. I plan to open a PR once I am confident this works as it should 🔜
Okay, something works here with IndexedDb and @EffectTS_ The IndexedDb layer creates the database, IndexedDbQuery provides a type-safe API based on IndexedDbVersion Keep working on it 🔬
I am pushing for the client to become more and more prominent. One of the key features of any app is a database, and the client deserves a great API for it 🫡
Expect more details coming this and next week, and more behind the scenes 🔜
See you next 👋