blog.heidiks.com
Go back

Designing ROZ: a file service that never touches your files

Roz, the deadpan records clerk from Monsters, Inc.

ROZ Storage — short for Routing Object Zoning — is a small Go service with one job: hand out files. The name is also a nod to Roz from Monsters, Inc., the deadpan clerk who guards the paperwork and never lets a misfiled form slide. Fitting, for a service whose whole job is brokering access to stored files.

TL;DR

What follows is each of those decisions and the tradeoff behind it, in the order they mattered — because that order is the architecture.

Where it started: bytes through the front door

Here’s the system I was replacing. To upload a file, the client streamed it to the frontend, which streamed it to the backend, which streamed it to Azure — every byte of every file passing through every layer, in 1MB multipart chunks, all the way down.

graph LR
    U([User]) --> F[frontend]
    F -->|metadata + multipart upload, 1MB chunks| B[backend]
    B -->|metadata + multipart upload, 1MB chunks| AZ
    subgraph AZ["Azure (single cloud)"]
        OS[(Object Storage)]
        TS[(Table Storage)]
    end

Two things are wrong with this picture, and they’re the reasons ROZ exists:

  1. Every layer is on the data path. The frontend and backend aren’t deciding anything about the file — they’re just plumbing, copying bytes from one socket to the next. That’s CPU, memory, and bandwidth spent on work that adds no value, multiplied by every layer and every megabyte.
  2. One cloud. The whole thing is welded to Azure. There’s no way to send one tenant’s files to a cheaper or closer store without rewriting the path.

Where it ended up

graph LR
    U([User]) --> F[frontend]
    F <-->|"1. metadata / 3. complete"| B[backend]
    B <-->|"1. metadata / 3. complete"| R[ROZ-storage]
    R -->|"location, pre-signed URL"| B
    R -->|metadata| DB[(Database)]
    F -.->|"2. multipart upload, 1MB chunks (direct)"| CLOUD

    subgraph CLOUD["Object storage (provider-agnostic)"]
        GCP[(GCP)]
        AZURE[(Azure Blob)]
        R2[(Cloudflare R2)]
    end

Same three-tier app on the control path — frontend asks backend asks ROZ for a place to put the file — but the bytes no longer flow through any of them. ROZ returns a pre-signed URL, the frontend uploads its 1MB chunks directly to whichever cloud that tenant is routed to, and only a “complete” call comes back through the stack. The app layer went from carrying the file to merely brokering it.

Everything else in this post is the set of decisions that turned the first diagram into the second.

Decision 1: The valet key pattern

The first and most consequential decision: ROZ does not proxy file content. It never receives an upload and never streams a download. Instead it issues short-lived, pre-signed URLs and steps out of the way. The client uploads its bytes directly to the object store, and downloads by following a redirect to a signed URL.

If you’ve never met the pattern, here’s the everyday version of it.

Think about how a parcel locker works. The courier doesn’t carry your package into your living room, and the store doesn’t keep it behind its own counter indefinitely. It drops the box in a locker and texts you a code. That code opens exactly one door, for a limited time, and then it’s useless. The store doesn’t have to be in the room when you show up, and you never get a key to the whole locker wall — just the one door, just for a while.

A pre-signed URL is that locker code. ROZ doesn’t hold your file and it doesn’t hand out the master key to the bucket. It signs a URL that grants one narrow permission — “write this object” or “read this object” — that expires in minutes. This is the valet key pattern: you don’t give the valet your car, you give them a key that only does one limited thing.

sequenceDiagram
    participant C as Client
    participant ROZ as ROZ API
    participant M as Metadata store
    participant OS as Object store

    C->>ROZ: POST /uploads (file_name, content_type, ...)
    ROZ->>M: persist metadata
    ROZ->>OS: sign a write URL
    ROZ-->>C: { upload_url, required_headers }
    C->>OS: PUT bytes directly (no ROZ in the path)

    Note over C,OS: later...
    C->>ROZ: GET /files/:id
    ROZ->>M: read metadata
    ROZ->>OS: sign a download URL
    ROZ-->>C: 302 Found -> signed URL
    C->>OS: GET bytes directly

Why I committed to this:

The tradeoff I accepted: I give up inspecting or transforming content in flight. No virus scanning at the gateway, no on-the-fly transcoding, no counting bytes as they pass. If I ever need those, they happen out-of-band — a worker triggered after upload — not inline. For a file manager, that was an easy trade. For a file pipeline, it wouldn’t be.

There’s also a leak in the abstraction worth naming: a signed PUT URL isn’t enough on its own. The signature is computed over certain headers (Content-Type, Content-Encoding, Cache-Control…), so the client must send those exact headers back. That’s why the API response carries a required_headers map the client has to echo verbatim. The locker code comes with a sticky note of instructions.

Decision 2: Provider-agnostic, behind a three-method interface

We run across more than one object store: Cloudflare R2 and Oracle OCI (both S3-compatible), plus Azure Blob (its own SDK and its own SAS-token model). I did not want the business logic to know or care which one it was talking to.

So the entire surface a provider must satisfy is three methods:

type Provider interface {
    GeneratePresignedURL(ctx, params) (url string, requiredHeaders map[string]string, err error)
    GenerateDownloadURL(ctx, objectPath string, ttlInMinutes int) (string, error)
    DeleteFile(ctx, objectPath string) error
}

Two implementations live behind it: an s3 one (R2 + OCI, same SDK) and an azure one. The catalog models each connection as a Provider and an SDK — because the provider is branding (CLOUDFLARE_R2, OCI_OBJECT_STORAGE, AZURE_BLOB) but the SDK is what actually decides the code path (S3_COMPATIBLE vs AZURE_SDK).

Why three methods and not thirty: the interface is the contract every future store has to honor, so I wanted it to be the smallest contract that still does the job. Sign a write, sign a read, delete. Everything tenant-, path-, and policy-related lives above the interface, in the service layer, written once and shared by every provider.

And this is where pluggability stops being a buzzword. Adding a brand-new cloud — Google Cloud Storage, say — isn’t a refactor. It’s implementing those three methods and adding one entry to the datasource catalog. That’s the entire price of admission. The diagram above shows GCP next to Azure and R2 for exactly that reason: not because it’s wired up today, but to make the point that a new provider is a leaf, not a root.

The tradeoff: the abstraction is lowest-common-denominator on purpose. Provider-specific superpowers (R2 bindings, Azure lifecycle tiers, OCI retention locks) don’t get a home in the interface. When I need them, they leak into the concrete implementation or into operational config — which is exactly what happened with lifecycle rules, the quirk that reshaped the whole path scheme below.

Decision 3: Routing config is embedded in the binary, not in a database

Which tenant’s files go to which store? That’s a routing question, and the obvious instinct is “put it in a table.” I went the other way: two JSON files — datasources.json (the catalog of connections) and router-rules.json (the tenant/category → datasource mapping) — compiled into the binary with go:embed.

//go:embed configs/datasources.json
var embeddedDatasources []byte

//go:embed configs/router-rules.json
var embeddedRouterRules []byte

Secrets aren’t baked in, though — the embedded JSON contains ${ENV_VAR} placeholders expanded at startup with os.ExpandEnv. The shape is static; the credentials are injected at runtime.

Why I took the database off the request path:

The router resolves in a strict priority order, which is itself a small design statement:

graph TD
    A[tenant_id + category] --> B{tenant rule?}
    B -- yes --> T[use that datasource]
    B -- no --> C{category rule?}
    C -- yes --> CT[use that datasource]
    C -- no --> D[default datasource]

Tenant beats category beats default. A specific tenant override always wins; a category rule (“all documents go to one store”) is the broad stroke; and there’s always a default, so an unknown tenant is never a 500 — it just lands in the general-purpose bucket.

The sharp version of this — and the part I like most: issuing a pre-signed URL touches no database at all. Routing is an in-memory map lookup over the embedded config, and signing is a pure cryptographic operation — an HMAC over the object path, expiry, and headers using the datasource’s key. There is no row to read in order to hand out a URL. ROZ does record file metadata separately, but the act of authorizing an upload or a download is stateless and in-memory. Only memory is enough. (You’ll see in the next decision how downloads can skip the database entirely, too.)

The tradeoff — and it’s a real one: changing routing means a redeploy. There’s no runtime knob, no admin endpoint. For a config that changes a handful of times a year, I’ll take “boring redeploy” over “live mutable state in a database” every time. If routing churned daily, this would be the wrong call and I’d move it to a store with a cache. Know your change frequency before you copy this.

Decision 4: The file ID is the timestamp (UUIDv7)

Every file gets a UUIDv7 as its ID. Why UUIDv7 specifically, and not v4 or an auto-increment? Because v7 is time-ordered: the creation timestamp is encoded in its leading bits. The ID isn’t just a random label — it’s a clock you can read.

The object path is partitioned by month:

{tenant_id}/{category}/{YYYY-MM}/{file_id}

That YYYY-MM comes straight out of the UUIDv7. Because the ID carries its own creation time, three things fall out for free:

That last point is what lets downloads skip the database. There’s a bulk-download path in ROZ that takes a list of UUIDs and signs every URL in memory, deriving each object path directly from the UUIDv7’s timestamp — no metadata read at all. The valet key (Decision 1), the in-memory routing (Decision 3), and the self-describing ID compound into the same property: you can authorize access to a file with only the ID and some math.

The tradeoff: I’ve coupled my path logic to the ID format. UUIDv7’s time bits are now load-bearing, not just a primary key. If I ever changed ID schemes, the path logic would change with them. I think that’s a fine coupling — the ID and the path are both “identity of a file” — but it’s a coupling, and it should be a conscious one. The payoff shows up immediately, in the next decision.

Decision 5: A path migration with zero file migration

Here’s the quirk that reshaped everything, and the decision I’m proudest of because it cost almost nothing.

The quirk: Cloudflare R2’s lifecycle rules don’t support wildcards. You can expire objects under a fixed prefix, but you can’t write */documents/* to expire one category across all tenants. With my original path — {tenant_id}/{category}/... — the tenant is the root prefix, so “expire this category everywhere” is impossible without enumerating every tenant by hand and updating the rules each time a new one is added. That doesn’t scale.

The fix is to invert the path so category is the root:

V1 (old): {tenant_id}/{category}/{YYYY-MM}/{file_id}
V2 (new): {category}/{tenant_id}/{YYYY-MM}/{file_id}

Now documents/ is a clean fixed prefix and a single lifecycle rule covers every tenant.

The constraint: the V1 files are already in production. Migrating objects in R2 — copying and re-keying millions of them — is expensive and risky. Non-starter. So the service has to read old files where they are and write new files where they belong, at the same time, forever, without anyone having to think about it.

This is where the UUIDv7 decision paid off. The version of the path is a pure function of the file’s timestamp versus a configurable cutover instant:

graph TD
    Op["upload / download / delete"] --> G["getObjectPath(timestamp)"]
    G --> Q{timestamp < cutover?}
    Q -- yes --> V1["V1: tenant / category / YYYY-MM / id"]
    Q -- no --> V2["V2: category / tenant / YYYY-MM / id"]

The same function gives the right answer for both, so reads and deletes of old files just work — no flag, no caller awareness, no schema change.

And the safety detail I care about most: the cutover (PATH_V2_CUTOVER_AT) defaults to a date in the far future (9999-12-31). With no configuration, every timestamp is before the cutover, so everything stays V1. The new behavior cannot switch on by accident in an environment where someone forgot to set the variable. Safe by default, opt-in by configuration, reversible by changing one env var before the next deploy. That’s the posture I want every risky change to have.

The tradeoff: I now carry two path schemes indefinitely, and the dual-path logic is a small permanent tax on anyone reading the code. But “permanent small tax” beats “one heroic, risky mass migration” — especially when the heroic option can corrupt production.

What I’d put on the highlight reel

And the quirks

The takeaway

None of these decisions are clever in isolation. The valet key is a documented pattern. A three-method interface is just good taste. go:embed is a stdlib feature. UUIDv7 is an RFC. What made ROZ feel designed rather than assembled was choosing each one for an explicit reason, naming the tradeoff out loud, and letting earlier decisions pay dividends for later ones — UUIDv7 making the migration cheap is the clearest example.

If there’s one habit I’d pass on, it’s that last one: when you make a structural choice, write down what it costs you. The cost is the part you’ll forget, and it’s the part that tells the next person — often future you — whether the decision still holds.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Next Post
The Return of the 3-Letter Acronyms