Schema Evolution

shorn has no schema evolution. A payload is decodable only by the schema that wrote it, and the fingerprint reports a mismatch rather than resolving one.

This is a deliberate format decision. Field tags would enable evolution but make every record larger. The Person fixture is 8 bytes in shorn and Avro, 9 in SchemaPack, and 10 in msgpackr records.

The policy: dispatch, don’t resolve

const v1 = fingerprinted(compile(PersonV1));
const v2 = fingerprinted(compile(PersonV2));

const byVersion = new Map([
  [v1.fingerprintHex, v1],
  [v2.fingerprintHex, v2],
]);

function read(payload: Uint8Array) {
  const key = [...payload.subarray(0, 3)]
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");
  const codec = byVersion.get(key);
  if (!codec) throw new Error(`No codec for schema ${key}`);
  return codec.decode(payload);
}

Use fingerprintHex as the map key. fingerprint returns a new array on each access, so it cannot be used directly as a stable Map key. Write with the newest codec and read with the codec that matches each payload. Remove an old codec only after no payloads use its fingerprint.

Migrate old payloads in a batch instead of converting them on every read. shorn does not include a schema registry; the dispatch table is an ordinary Map. A generic decodeAny would return an impractically broad union type.

Why no field tags

On-wire tags make tiny records larger. Protobuf uses them to support evolution, but that tradeoff conflicts with shorn’s focus on compact payloads.

Position-based IDs do not solve optional fields. The presence bitmap uses ceil(n / 8) bytes, and each field’s bit depends on its rank among optional fields. Adding a ninth optional field grows the bitmap and causes old payloads to be misread. Prefixing the bitmap with its length would add overhead and still would not support a new required field. shorn therefore makes no partial compatibility promise.

Avro already provides mature schema evolution and ties shorn on size by keeping the writer’s schema out of band. Use Avro when you need automatic resolution. See ADR 0002.

In practice

• Persistent and queued payloads MUST be fingerprinted. A bare stored payload has no recoverable identity once the schema moves.
• Applications retain every historical codec used to write data. This requires explicit version bookkeeping.
• Adding a new wire type does not break existing fingerprints, because it does not alter the signature of any existing shape. Safe in a minor release.

The format itself is frozen

The wire format is frozen for existing shapes. A new wire type may be added, but it cannot change payloads that do not use it. Changing the encoding of an existing shape would invalidate stored payloads, so shorn will not do that in a compatible release.

If this guarantee ever changes, it will happen in a major release with migration instructions: pin the old package, re-encode stored data, and then upgrade.

The payload does not contain a separate format-version byte. Such a byte could identify the required decoder but could not supply it; only the old package can read the old format. The fingerprint instead turns a silent misread into an explicit error.

Migrating a schema

1. Keep the old schema object, renamed. Do not edit it.
2. Add the new one.
3. Register both fingerprints.
4. Write with the new codec; read with whichever matches.
5. Optionally batch-re-encode and drop the old entry.

function migrate(payload: Uint8Array) {
  const old = v1.decode(payload); // throws if it was not v1
  const [firstName, ...rest] = old.name.split(" ");
  return v2.encode({ ...old, firstName, lastName: rest.join(" ") });
}

Without the fingerprint, line one would have returned a plausible object with the values in the wrong fields.

What would reopen this

• A wire-format change making the presence bitmap length-prefixed for other reasons.
• Evidence of applications hand-maintaining large dispatch tables, which is the signal a registry earns its keep.
• Cross-language consumers, which would make out-of-band schema distribution unavoidable. Currently a non-goal.