Three years of protobuf in production

My project has a data service that serves everything over gRPC. It has five clients: a Next.js frontend, a sitemap generator, a Laravel backend, a Go service, and an api-gateway that nobody has touched since September 2024. That is three languages: TypeScript, PHP, and Go.

The contract between them lives in its own schema repository. After three years it holds:

274 commits and 141 tags
36 .proto files, about 2,640 lines
16 services, 87 RPCs, about 270 messages

Writing the schema was the easy part. The hard parts sat around it: packaging in 2024 and the network in 2026.

Why gRPC

I wanted a typed contract for TypeScript, PHP, and Go. I wanted a binary format over HTTP/2, and streams. And I wanted to try it.

How a change ships

I edit a .proto file in src/. Every file uses one package, package services;, so each language gets one namespace and imports stay short. Every file must also set php_namespace and go_package.
I push a git tag 0.0.N+1. Nothing builds locally.
On the tag, GitHub Actions runs three jobs in parallel:
- ts: protoc with ts-proto (outputServices=nice-grpc), an ESM and a CJS build, then a publish to GitHub Packages with the tag as the version.
- php: protoc --php_out --grpc_out, then a copy into a separate PHP repo, with a commit and the same tag.
- go: protoc-gen-go and protoc-gen-go-grpc into a separate Go repo, then go mod tidy, a commit, and a tag.
Each client bumps the version on its own.

The server has bumped its schema version 138 times. The frontend has done it 46 times.

Dates are strings

On day two I replaced google.protobuf.Timestamp with string. It stayed that way: every date in the schema is a string.

The reason was compatibility with the API I already had. The database and the HTTP API of the data service already returned dates as ISO strings. gRPC went in next to the HTTP API, and clients had to get the same values from both.

The same reason drove a change in October 2025. In RatingsService, int64 timestamp became string date, so the daily ratings histogram now returns dates in the same format as the rest of the API.

2024: packaging was harder than the schema

The first commit is from December 28, 2023. Nine more followed that day: “fix ga”, “fix protoc”, “add nice-grpc”, “debug”. The schema barely moved. I was fixing the build.

On January 18, 2024, I moved the package to ESM. That took about 15 commits in one day: “support esm”, then “fix imports” v2 through v8. At one point ts-proto came from a local folder on my laptop. long.js was swapped for a patched fork and reverted the same day.

One line from that day still runs in CI:

sed -i -E -e "s/as \w+ //" src/index.ts

ts-proto generates export * as services from "./index.services.js". The sed turns it into export * from …, so clients import ApplicationsServiceDefinition directly, with no namespace. One of the versions between v2 and v7 ran sed without -i, and it silently changed nothing.

After that, services arrived with features: events, applications, reviews, search, ratings, keywords, charts, achievements. The PHP SDK came in September 2024, and I had to add php_namespace to every file. Go came in December, after five commits in a row before the build worked.

2025: FieldMask and `optional`

In January 2025, GetApplications, GetReviewsData, GetSimilars, and GetEvents got a google.protobuf.FieldMask. The client asks only for the fields it needs, and the server skips the JOINs nobody asked for.

This raised a question at once: how do you tell “not requested” from “empty”? In proto3 without optional, you can’t. An empty string and an unset string look the same on the wire. A day later, the fields of ApplicationShort became optional.

2026: the network

Most of the lessons came from here. Each fix started with a concrete symptom.

No retries for six months. In March 2026 I added a retryPolicy to the frontend channel options. I put it at the top level of grpc.service_config. grpc-js reads it only inside methodConfig and ignores it anywhere else, with no error and no warning. I found it in September. In the same fix I removed DEADLINE_EXCEEDED from the retryable codes: when the upstream is slow, a retry only burns what is left of the deadline.

No default deadline. Neither nice-grpc nor PHP gRPC sets one. In PHP, wait() hung until Octane killed the FrankenPHP worker after 30 seconds. The fix was a trait that puts an 8-second deadline on every unary call. In the frontend, one stuck call held an SSR page open forever. The fix there was a middleware with a 5-second deadline.

The last pod got no traffic. Load balancing is round_robin over a headless service. grpc-js does not re-resolve DNS while its connections are healthy. After a rolling restart, the last pod sat idle until the clients restarted. The fix was max_connection_age = 5 min on the server.

That fix broke streams. The grace period for max_connection_age was 30 seconds. Sitemap streams run for 3 to 5 minutes. A stream that started near the end of a connection’s life got CANCELLED. I raised the grace period to 10 minutes.

Postgres cut streams too. StreamReviews failed because statement_timeout = 30s also counted the time the client spent reading. The fix was SET LOCAL statement_timeout = '10min' inside the stream’s transaction.

Over the same months I added a composite RPC, GetItemPage. One call returns everything the item page needs. The subpages followed: one subpage went from two calls to one, the review body from four to three.

Rules I broke

In three years I used the protobuf evolution tools exactly four times: two reserved and two [deprecated = true]. I broke the rules more often than that.

Changing a type under the same field number. January 2025:

// before
bool deleted = 28;
// after
optional string deletedAt = 28;

On the wire, a bool is a varint and a string is length-delimited. An old client cannot read the new field. The same thing happened with timestamp to date in RatingsService.

A breaking removal shipped. In May 2026 charts learned to return bundles. I replaced repeated ChartPosition data = 1 with entries = 2, a oneof of Application and Bundle. Tag 0.0.192 went out without the data field. The same day, 0.0.193 brought back data = 1 [deprecated = true]. No npm client ever installed 0.0.192.

Renumbering fields. November 2026:

// 0.0.229, 0.0.230
repeated string deviceModels = 50;
repeated PrivacyType privacyTypes = 51;
repeated ArtworkItem headerArtwork = 52;

// 0.0.231, 18 minutes later
repeated PrivacyType privacyTypes = 50;
repeated ArtworkItem headerArtwork = 51;

No reserved. If the frontend had installed 0.0.230, it would read privacyTypes from field 51, where the new server sends headerArtwork. It would get garbage or a parse error. Nobody installed 0.0.229 or 0.0.230, so nothing happened.

I got away with all of this because one person deploys every client, and the server and clients update together. That is luck, and I shouldn’t plan around it.

Versions, halfway

There are 141 tags, but only the npm clients use them. The PHP backend pins dev-main. The Go service uses a pseudo-version from a commit. CI also creates tags in the PHP and Go repos, and nothing reads them.

CI has its own weak spot. Tag 0.0.222 failed in the Go job on arduino/setup-protoc with API rate limit exceeded. The action fetches protoc releases from the GitHub API without a token.

The schema and AI agents

In 2026, 56 of the 60 commits to the schema repo carry Co-Authored-By: Claude. Here the schema helped a lot. The agent changes a .proto file, and the generated types in each client show what to fix.

The agent also did the renumbering from the section above. Types don’t catch that. After regeneration everything compiles, and only wire compatibility breaks.

What I would do differently

Run buf breaking in CI. It would catch the type change, the renumbering, and the removed field.
Add reserved every time a field goes away.
Set a default deadline in every client from day one.
Make PHP and Go depend on tags, not on the latest commit.