Schema evolution & compatibility

Skir is designed for long-term data persistence and distributed systems. It ensures that your application can evolve its data structures while maintaining compatibility with older data (backward compatibility) and older clients (forward compatibility).

Core concepts

• Backward compatibility: New code can read old data. This is essential for reading records stored in a database created with an older schema.
• Forward compatibility: Old code can read new data. This is critical in distributed systems where different services or clients may be running different versions of your application.

Safe schema changes

The following changes are safe and preserve both backward and forward compatibility:

Adding fields to a struct

New code reading old data will use default values for missing fields:

• Numbers: 0
• Booleans: false
• Strings/Bytes: Empty string/bytes
• Arrays: Empty array []
• Structs: A struct with all fields at their default values
• Enums: The implicit UNKNOWN variant
• Optional types: null

Adding variants to an enum

Old code encountering a new variant will treat it as the implicit UNKNOWN variant.

Renaming types, fields, and variants

Skir uses numeric identifiers (field numbers) in its binary and compact JSON formats, not names. Therefore, renaming any element is safe.

Removing fields or variants

You must mark the field or variant number as removed to prevent accidental reuse.

Compatible type changes

You can change a type if the new type is backward-compatible with the old one:

• bool → int32, int64, hash64
• int32 → int64
• float32 → float64
• float64 → float32 (precision loss possible)
• [A] → [B] (if A → B is valid)
• A? → B? (if A → B is valid)

Unsafe changes

The following changes will break compatibility:

• Changing a field/variant number, or reordering fields/variants if using implicit numbering.
• Changing the type of a field, wrapper variant, method request or method response to an incompatible type.
• Changing a method's stable identifier.
• Reusing a removed field or variant number.
• Deleting a field or variant without marking it as removed.
• Changing a constant variant to a wrapper variant or vice-versa.

Automated compatibility checks

The Skir compiler includes a snapshot tool to prevent accidental breaking changes.

How it works

The npx skir snapshot command helps you manage schema evolution by maintaining a history of your schema state. When you run this command, two things happen:

1. Verification: Skir checks for a skir-snapshot.json file. If it exists, it compares your current .skir files against it. If breaking changes are detected, the command reports them and exits.
2. Update: If no breaking changes are found (or if no snapshot exists), Skir creates or updates the skir-snapshot.json file to reflect the current schema.

Tracked types and stable identifiers

To track compatibility across renames, Skir needs a stable identifier for your types. You can assign a random integer ID to any struct or enum:

// "User" is now tracked by ID 500996846
struct User(500996846) {
  name: string;
}

If you rename User to Account but keep the ID 500996846, Skir knows it's the same type and will validate the change safely.

Handling intentional breaking changes

If you must make a breaking change (e.g., during early development), simply delete the skir-snapshot.json file and run npx skir snapshot again to establish a new baseline.

Recommended workflow

1. During development

While drafting a new schema version, use the --dry-run flag to check for backward compatibility without updating the snapshot:

npx skir snapshot --dry-run

This confirms that your changes are safe relative to the last release (snapshot).

If you are using the official VSCode extension, breaking changes will be highlighted directly in your editor as you type.

2. Before release

Run npx skir snapshot without flags to verify compatibility and commit the new schema state to the snapshot file.

3. Continuous integration

Add the command to your CI pipeline or pre-commit hook to prevent accidental breaking changes. The --ci flag ensures the snapshot is up-to-date and compatible:

- name: Ensure Skir snapshot up-to-date
  run: npx skir snapshot --ci

Round-tripping unrecognized data

Consider a service in a distributed system that reads a Skir value, modifies it, and writes it back. If the schema has evolved (e.g., new fields were added) but the service is running older code, it may encounter data it doesn't recognize.

When deserializing, you can choose to either drop or preserve this unrecognized data.

• Drop (default): Unrecognized fields and variants are discarded. This is safer but results in data loss if the object is saved back to storage.
• Preserve: Unrecognized data is kept internally and written back during serialization. This enables "round-tripping".

Example

Consider a schema evolution where a field and an enum variant are added:

struct UserBefore(999) {
  id: int64;
  subscription_status: enum {
    FREE;
    PREMIUM;
  };
}

struct UserAfter(999) {
  id: int64;
  subscription_status: enum {
    FREE;
    PREMIUM;
    TRIAL;  // Added
  };
  name: string;  // Added
}

Default behavior: drop

By default, unrecognized data is lost during the round-trip.

// Old code reads and writes the data
const oldUser = UserBefore.serializer.fromJson(originalJson);
const roundTrippedJson = UserBefore.serializer.toJson(oldUser);

// New code reads the result
const result = UserAfter.serializer.fromJson(roundTrippedJson);

assert(result.id === 123);
assert(result.name === "");  // Lost: reset to default
assert(result.subscriptionStatus.union.kind === "UNKNOWN");  // Lost: became UNKNOWN

Preserve behavior

You can configure the deserializer to keep unrecognized values.

// Old code reads with "keep-unrecognized-values"
const oldUser = UserBefore.serializer.fromJson(
  originalJson,
  "keep-unrecognized-values"
);
const roundTrippedJson = UserBefore.serializer.toJson(oldUser);

// New code reads the result
const result = UserAfter.serializer.fromJson(roundTrippedJson);

assert(result.id === 123);
assert(result.name === "Jane");  // Preserved!
assert(result.subscriptionStatus.union.kind === "TRIAL");  // Preserved!