Core Concepts

This page explains the fundamental concepts of the Unbroken Protocol. Understanding these concepts is essential for implementing clients and servers.

Streams

A stream is a URL-addressable, append-only sequence of bytes. Streams are the fundamental unit of the protocol.

https://api.example.com/streams/conversations/conv-123
https://api.example.com/sync/users/changes
https://myapp.com/events/room/abc

Key Properties

Property	Description
URL-Addressable	Every stream has a unique URL
Append-Only	Data can only be added to the end, never modified or deleted in place
Durable	Once acknowledged, data persists until stream deletion or expiry
Immutable by Position	Bytes at a given offset never change
Ordered	All data has a strict total order defined by offset

Stream Lifecycle

┌─────────┐     ┌──────────┐     ┌─────────┐
│ Create  │────▶│  Active  │────▶│ Deleted │
│  (PUT)  │     │(GET/POST)│     │ (DELETE)│
└─────────┘     └──────────┘     └─────────┘
                     │
                     ▼
               ┌──────────┐
               │ Expired  │
               │  (TTL)   │
               └──────────┘

Streams transition through these states:

1. Created: Stream is established via PUT request
2. Active: Stream accepts reads and writes
3. Deleted: Stream is removed via DELETE request
4. Expired: Stream is automatically removed after TTL/expiry time

Offsets

An offset is an opaque token that identifies a position within a stream. Clients use offsets to track their reading position and resume after disconnection.

Offset Properties

Offsets have three critical properties:

1. Opaque: Clients MUST NOT interpret or parse offset values
2. Lexicographically Sortable: Comparing offsets as strings determines their order
3. Persistent: Offsets remain valid for the stream’s lifetime

Using Offsets

Initial Read:    GET /stream?offset=-1
                 Response: data, Stream-Next-Offset: abc123

Subsequent Read: GET /stream?offset=abc123
                 Response: more data, Stream-Next-Offset: def456

After Restart:   GET /stream?offset=abc123  ← Resume from saved offset
                 Response: data from abc123 onwards

Clients SHOULD persist offsets locally (localStorage, database, file) to enable resumability across restarts.

Special Offset Values

Value	Meaning
-1	Start of stream (read from beginning)
Omitted	Same as -1
<token>	Resume from after this position

Content Types

Every stream has a content type set at creation that describes the format of its data.

Supported Content Types

The protocol supports any MIME content type:

Content Type	Use Case
application/octet-stream	Binary data (default)
application/json	JSON with message boundary preservation
application/ndjson	Newline-delimited JSON
text/plain	Plain text
application/x-protobuf	Protocol Buffers

JSON Mode

Streams with Content-Type: application/json have special semantics:

1. Message Boundaries: Each append is stored as a distinct message
2. Array Flattening: Top-level arrays are flattened (batch append)
3. Response Format: Reads return data wrapped in a JSON array

// Append: {"event": "click"}
// Stored: {"event": "click"}

// Append: [{"a": 1}, {"b": 2}]
// Stored: {"a": 1}, {"b": 2}  (two separate messages)

// Read Response:
[{ "event": "click" }, { "a": 1 }, { "b": 2 }]

Read Modes

The protocol supports three modes for reading stream data:

Catch-Up Read

Returns immediately with available data. Used for replaying historical data.

GET /stream?offset=abc123

Response includes:

• Data from the requested offset
• Stream-Next-Offset header for the next read
• Stream-Up-To-Date: true when at the end

Long-Poll Read

Waits for new data if none is available. Efficient for live tailing.

GET /stream?offset=abc123&live=long-poll

Server behavior:

• Returns immediately if data exists
• Waits up to timeout (typically 30s) for new data
• Returns 204 No Content if timeout expires with no new data

SSE (Server-Sent Events)

Maintains an open connection for real-time streaming. Best for browser clients.

GET /stream?offset=abc123&live=sse

Response format:

event: data
data: {"message": "hello"}

event: control
data: {"streamNextOffset": "def456"}

Headers

The protocol uses custom HTTP headers for stream metadata:

Request Headers

Header	Operation	Description
Stream-TTL	PUT	Relative time-to-live in seconds
Stream-Expires-At	PUT	Absolute expiry (RFC 3339)
Stream-Seq	POST	Writer sequence for coordination

Response Headers

Header	Description
Stream-Next-Offset	Next offset for subsequent reads
Stream-Up-To-Date	Present when response includes all available data
Stream-Cursor	Cursor for CDN collapsing (optional)

Writer Coordination

Multiple writers can safely append to a stream using the Stream-Seq header:

POST /stream
Stream-Seq: 001
Content-Type: application/json

{"event": "first"}

The server rejects appends with sequence values less than or equal to the last accepted sequence, preventing out-of-order writes.

Next Steps

• Protocol Overview - Complete method reference
• Reading Operations - Detailed read documentation
• Writing Operations - Create and append details
• Quick Start Guide - Hands-on examples