monorepo.
monorepo55 min read

Buf Gen Protobuf Rust Python Typescript

The first time I tried to share a single protobuf schema across a Rust service, a Python data job, and a TypeScript frontend, I ended up with three subtly different message definitions and a Slack thread full of "wait, which field is optional again?" The schemas had drifted in a week. I had copy-pasted .proto files between three repos like it was 2014, hand-rolled three different codegen invocations, and convinced myself the CI would catch any mismatch. CI caught nothing — a staging deploy did, at 11pm, when a TypeScript client serialized a field the Rust server had quietly renamed.

This article walks through the fix I wish I had reached for sooner: a single monorepo where buf owns the .proto schemas, buf.gen.yaml drives codegen for prost/tonic in Rust, grpcio-tools in Python, and @bufbuild/protobuf in TypeScript, and buf lint plus buf breaking run as CI gates so drift becomes a failing check instead of a 2am pager. By the end you will have a working monorepo that round-trips a message across all three languages and a CI workflow that refuses to merge a breaking schema change. The rule worth screenshotting: if the same schema lives in more than one file, it is already two schemas — and one of them is wrong.

This is written for backend and platform engineers who already ship in at least one of Rust, Python, or TypeScript and want the other two to stop being someone else's problem. You will not need deep gRPC theory; you will need patience for one cohesive toolchain instead of three improvised ones.

Step 1: Laying the Polyglot Monorepo Foundation with a Pinned buf CLI

A polyglot Protobuf monorepo lives or dies by its first few decisions: where the .proto files live, how each language consumes them, and which buf release is in charge of stamping out code. If those answers drift between contributors, generated Rust, Python, and TypeScript stop agreeing on field numbers and JSON casing — the worst kind of bug because the compiler is happy.

This step builds nothing fancy yet. It carves out the directory layout for proto/, rust/, python/, and typescript/, writes the smallest valid buf.work.yaml, ships a pinned installer for the buf CLI, and locks the skeleton in place with pytest assertions. Every later step in this article series leans on these invariants.

Setup

The repository is a single workspace; no language toolchains are required to run this step's tests beyond Python 3.9+ and pytest. The files we create are:

  • buf.work.yaml — declares the workspace and points it at the proto/ module root.
  • proto/.gitkeep, rust/.gitkeep, python/.gitkeep, typescript/.gitkeep — placeholder files so git tracks the empty subprojects.
  • scripts/install-buf.sh — a portable, version-pinned installer for the buf CLI.
  • Makefile — three thin targets (install-buf, check-buf, test) so the contract is the same on a laptop and in CI.
  • pyproject.toml + tests/test_bootstrap.py — pytest harness that fails loudly if any of the above drifts.

There are no third-party Python dependencies in this step. The test file uses only the standard library (os, stat, pathlib) so a fresh clone can run pytest -q immediately without an environment dance.

Implementation

We start by telling buf where Protobuf modules live. The workspace file is intentionally minimal — one module, rooted at proto/, with version: v1 so we can grow into multiple modules later without rewriting the schema.

version: v1
directories:
  - proto

That single declaration is what lets buf build, buf lint, and buf generate find sources regardless of which language subtree initiated the command. Pinning it now means none of the upcoming Rust, Python, or TypeScript codegen steps need to argue about source paths — they all defer to this file.

Next, the installer. We deliberately do not rely on whatever buf happens to be on a contributor's PATH. Generated code is downstream of the compiler, so the compiler must be reproducible.

#!/usr/bin/env bash
set -euo pipefail

BUF_VERSION="${BUF_VERSION:-1.50.0}"
INSTALL_DIR="${INSTALL_DIR:-${HOME}/.local/bin}"

detect_os() {
  case "$(uname -s)" in
    Linux*)  echo "Linux" ;;
    Darwin*) echo "Darwin" ;;
    *) echo "unsupported" ;;
  esac
}

detect_arch() {
  case "$(uname -m)" in
    x86_64|amd64) echo "x86_64" ;;
    arm64|aarch64) echo "aarch64" ;;
    *) echo "unsupported" ;;
  esac
}

The script uses set -euo pipefail so a single failure aborts the whole installation; a half-installed buf is worse than no buf because it would silently mask version mismatches. BUF_VERSION is an env-overridable default (1.50.0) — CI can pin newer or older releases without editing the script. The OS/arch detection covers the four combinations we actually ship to (macOS arm64, macOS x86_64, Linux x86_64, Linux aarch64), and anything else fails fast with a clear message.

The main routine fetches the matching release asset from the official bufbuild/buf GitHub releases, drops it into $HOME/.local/bin, and marks it executable.

main() {
  local os arch url
  os="$(detect_os)"
  arch="$(detect_arch)"
  if [ "${os}" = "unsupported" ] || [ "${arch}" = "unsupported" ]; then
    echo "Unsupported platform: $(uname -s) $(uname -m)" >&2
    exit 1
  fi
  url="https://github.com/bufbuild/buf/releases/download/v${BUF_VERSION}/buf-${os}-${arch}"
  mkdir -p "${INSTALL_DIR}"
  echo "Downloading buf v${BUF_VERSION} (${os}/${arch}) -> ${INSTALL_DIR}/buf"
  curl -fsSL "${url}" -o "${INSTALL_DIR}/buf"
  chmod +x "${INSTALL_DIR}/buf"
  echo "Installed: $(${INSTALL_DIR}/buf --version 2>&1)"
}

main "$@"

curl -fsSL is the safe default: fail on HTTP errors (-f), stay silent on progress (-s), follow redirects (-L), and surface real errors (-S). Echoing the version of the freshly-installed binary at the end gives contributors a one-line confirmation that the installer worked and matches what the article claims.

The Makefile wraps three contract surfaces — install, sanity-check, and run tests — so we never have to memorize the underlying commands.

.PHONY: help install-buf check-buf test

help:
	@echo "Targets:"
	@echo "  install-buf  - Download a pinned buf CLI release into \$$HOME/.local/bin"
	@echo "  check-buf    - Print the buf version (fails if buf is not on PATH)"
	@echo "  test         - Run pytest against the bootstrap checks"

install-buf:
	./scripts/install-buf.sh

check-buf:
	@buf --version

test:
	pytest -q

check-buf deliberately calls buf from PATH. If a contributor forgot to add $HOME/.local/bin to their shell, this target fails fast instead of letting later codegen steps emit a cryptic "command not found".

Finally, the pytest harness encodes everything we just promised. The fixture-free style keeps the tests readable; each function asserts exactly one invariant.

from __future__ import annotations

import os
import stat
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parent.parent


def _read(path: Path) -> str:
    return path.read_text(encoding="utf-8")


def test_language_subprojects_exist():
    for sub in ("proto", "rust", "python", "typescript"):
        target = REPO_ROOT / sub
        assert target.is_dir(), f"missing monorepo subdir: {sub}"

The first assertion locks the four-subtree shape. If anyone deletes rust/ or renames typescript/ in a future refactor without updating the test, the suite turns red and the article's promises are honored by CI rather than by memory.

def test_buf_workspace_file_lists_proto_directory():
    workspace = REPO_ROOT / "buf.work.yaml"
    assert workspace.is_file(), "buf.work.yaml is required at the repo root"
    text = _read(workspace)
    assert "version: v1" in text
    assert "- proto" in text


def test_install_script_is_executable_and_pins_buf_version():
    script = REPO_ROOT / "scripts" / "install-buf.sh"
    assert script.is_file(), "scripts/install-buf.sh is required"
    mode = script.stat().st_mode
    assert mode & stat.S_IXUSR, "install-buf.sh must be executable"
    text = _read(script)
    assert "BUF_VERSION" in text, "installer must declare a buf version variable"
    assert "bufbuild/buf/releases/download" in text

The workspace test verifies both the schema version and the proto directory listing, so a typo in either drops the harness. The installer test refuses to accept a non-executable file (stat.S_IXUSR) — a common breakage when scripts are added through a Windows editor or copied without the execute bit, which silently turns make install-buf into a no-op on first run.

There is also a privacy guard that scans shipped artifacts for operator-private absolute paths or container roots, so no stray developer-machine path ever makes it into the public companion repo. The full file lives in the codebase repo; the commit linked at the end of this step is the authoritative copy.

Verification

Run the bootstrap test suite from the repository root:

pytest -q

The expected output is seven dots followed by a passing summary:

.......                                                                  [100%]
7 passed in 0.02s

That's seven invariants locked in: the four language subtrees exist, the buf.work.yaml shape is correct, the installer is executable + version-pinned, the installer's shebang is safe, the Makefile exposes the three contract targets, no shipped artifact leaks an operator-private path, and the Python runtime meets the 3.9+ floor we rely on.

What we built

We turned an empty directory into a four-language monorepo skeleton with a single source of truth for Protobuf inputs. buf.work.yaml declares the workspace, the proto/, rust/, python/, and typescript/ subtrees mark the consumer slots, and .gitkeep files make sure git ships the empty layout intact.

We also boxed the buf CLI into a reproducible installer. The script pins buf 1.50.0 by default but allows CI or contributors to override the version through BUF_VERSION, and the Makefile provides one-shot entry points so nobody has to remember the install path or the version flag.

Most importantly, every promise in this step is enforced by pytest. The harness is fast (sub-second), fixture-free, and uses only the standard library, so a brand-new contributor can clone the repo and confirm the skeleton with a single command — no buf, no Rust toolchain, no Node install required.

That trio — layout, reproducible compiler, executable contract — is what unlocks the next step. We can now drop a real .proto file into proto/ and have a defensible answer to the question "but which buf ran it?"

Repository

The state of the code after this step: d6c2e65

Step 2: Sealing the Schema Contract with buf.yaml Lint and Breaking-Change Rules

Step 1 gave us an empty four-language skeleton with a pinned buf CLI and a pytest harness. The proto/ directory existed, but it held no actual schemas, so there was nothing for the Rust, Python, or TypeScript targets to generate against. Without a frozen contract, every codegen step downstream would be writing checks against a moving target.

This step authors the first real schemas — monorepo/v1/money.proto and monorepo/v1/ledger.proto — and bolts a buf.yaml module file in front of them. The buf.yaml switches on DEFAULT lint coverage so naming, packaging, and style drift get caught at the source, and it enables FILE-scope breaking-change rules so a renamed field or shifted field number cannot ship past CI. A new pytest module then encodes every schema invariant we care about, even when buf itself is not on the PATH.

Setup

This step adds three categories of file to the codebase/ repo. None of them require new third-party dependencies; the test layer still runs on the standard library and the pytest harness installed in step 1.

  • proto/buf.yaml — the module manifest that names the workspace buf.build/vytharion/monorepo, enables lint: DEFAULT, and enables breaking: FILE.
  • proto/monorepo/v1/money.proto — the shared Money value type, used by every monetary field in the system.
  • proto/monorepo/v1/ledger.proto — the Account aggregate plus the LedgerService RPC surface (Transfer, GetAccount).
  • Makefile — extended with two new targets: lint (runs buf lint) and breaking (compares the working tree against main).
  • tests/test_proto_schemas.py — eleven pytest checks that lock the schema layout, the buf configuration, and the privacy guardrails.

The proto/ directory previously held only a .gitkeep placeholder. The new monorepo/v1/ subdirectory is the canonical package path — buf uses it both as the proto package identifier and as the on-disk layout, so any drift between the two is mechanical to detect.

Implementation

The module manifest is intentionally tiny. It declares the module name, picks the DEFAULT lint category (covers field naming, package suffix, RPC casing, and import organization), and picks the FILE breaking-change category so each file is compared independently rather than at the wire-format level.

version: v1
name: buf.build/vytharion/monorepo
lint:
  use:
    - DEFAULT
breaking:
  use:
    - FILE

DEFAULT is the right floor: it bundles the curated set of rules buf recommends for clean schemas without enforcing the stricter style categories like COMMENTS. FILE-scope breaking catches the realistic mistakes — a deleted field, a changed type, a renamed RPC — without flagging every wire-compatible refactor (PACKAGE and WIRE scopes have different trade-offs we don't need yet).

Next, the shared Money value type. Floats are a non-starter in cross-language Protobuf code because Rust, Python, and TypeScript each round and serialize floating-point edges differently. We carry an integer amount in the currency's minor units instead.

syntax = "proto3";

package monorepo.v1;

option go_package = "github.com/vytharion/buf-gen-protobuf-rust-python-typescript/gen/go/monorepo/v1;monorepov1";
option java_multiple_files = true;
option java_outer_classname = "MoneyProto";
option java_package = "com.vytharion.monorepo.v1";
option csharp_namespace = "Vytharion.Monorepo.V1";

// Money carries a fixed-precision amount so Rust, Python, and TypeScript
// clients never disagree on rounding. The unit is the smallest indivisible
// piece of the named currency (cents for USD, satoshi for BTC, etc).
message Money {
  string currency_code = 1;
  int64 amount_minor_units = 2;
}

The option go_package line is what makes this file portable across all of buf's generators — even targets we are not shipping yet inherit a sensible default. Pinning it under the public vytharion repository namespace also means the privacy gate has no opportunity to flag operator-private literals later in the pipeline.

The Ledger schema layers on top of Money and the well-known google.protobuf.Timestamp. Importing the well-known type instead of inventing a custom one keeps the wire format compatible with every Protobuf runtime out of the box.

syntax = "proto3";

package monorepo.v1;

import "google/protobuf/timestamp.proto";
import "monorepo/v1/money.proto";

message Account {
  string id = 1;
  string display_name = 2;
  Money balance = 3;
  google.protobuf.Timestamp created_at = 4;
}

message TransferRequest {
  string from_account_id = 1;
  string to_account_id = 2;
  Money amount = 3;
  string idempotency_key = 4;
}

message TransferResponse {
  string transfer_id = 1;
  google.protobuf.Timestamp posted_at = 2;
}

service LedgerService {
  rpc Transfer(TransferRequest) returns (TransferResponse);
  rpc GetAccount(GetAccountRequest) returns (GetAccountResponse);
}

TransferRequest carries an idempotency_key so retries are safe regardless of which language calls the service; that field is part of the contract, not a convenience the Rust client invents on its own. Account.balance reuses the imported Money message rather than inlining currency_code and amount_minor_units fields — exactly the kind of structural reuse that buf's DEFAULT lint helps enforce.

The Makefile grows two thin targets so contributors and CI use the same commands:

lint:
	buf lint

breaking:
	buf breaking --against '.git#branch=main'

buf breaking --against '.git#branch=main' reads the protos as they exist on the main branch and diffs the current working tree against them. That gives us a meaningful baseline for pull requests without needing a remote registry or a tagged release.

Finally, the pytest module. It runs without buf on the PATH — every assertion fires against file content directly — which keeps the suite green on fresh machines and inside minimal CI containers. The module-level constants pin the expected layout and the public repository prefix.

REPO_ROOT = Path(__file__).resolve().parent.parent
PROTO_DIR = REPO_ROOT / "proto"
PROTO_PACKAGE_DIR = PROTO_DIR / "monorepo" / "v1"

GO_PACKAGE_PREFIX = (
    "github.com/vytharion/buf-gen-protobuf-rust-python-typescript"
)

The first cluster of tests pins the buf configuration, the proto3 syntax declaration, and the package monorepo.v1 identifier on every file. They catch the most common drift modes long before buf lint would even run.

def test_buf_yaml_enables_default_lint_and_file_breaking():
    cfg = PROTO_DIR / "buf.yaml"
    text = cfg.read_text(encoding="utf-8")
    assert "version: v1" in text
    assert "lint:" in text and "DEFAULT" in text
    assert "breaking:" in text and "FILE" in text


def test_every_proto_declares_proto3_and_versioned_package():
    for proto_file in sorted(PROTO_PACKAGE_DIR.glob("*.proto")):
        text = proto_file.read_text(encoding="utf-8")
        assert 'syntax = "proto3";' in text
        assert "package monorepo.v1;" in text

A regex-driven check then walks every message { ... } body and flags duplicate field numbers. That is the single most expensive Protobuf bug — two fields sharing a tag silently corrupt the wire format — and the test makes sure no merge can introduce one.

def test_no_duplicate_field_numbers_within_a_message():
    field_pattern = re.compile(
        r"^\s*(?:repeated\s+|optional\s+)?[\w.]+\s+\w+\s*=\s*(\d+)\s*;",
        re.MULTILINE,
    )
    message_pattern = re.compile(r"message\s+\w+\s*\{([^}]*)\}", re.DOTALL)
    for proto_file in sorted(PROTO_PACKAGE_DIR.glob("*.proto")):
        text = proto_file.read_text(encoding="utf-8")
        for body in message_pattern.findall(text):
            numbers = [int(n) for n in field_pattern.findall(body)]
            assert len(numbers) == len(set(numbers))

Two targeted assertions confirm that Money stays integer-only and that the LedgerService RPC surface is present with both Transfer and GetAccount. The final test is a privacy guard that scans every shipped artifact for operator-private path fragments so no developer-machine literal ever lands in the public companion repo.

Verification

Run the full suite from the repository root after applying the step 2 changes:

pytest -q

The bootstrap tests from step 1 still pass, and the eleven new schema tests join them for an eighteen-dot summary:

..................                                                       [100%]
18 passed in 0.04s

That covers seven step-1 invariants plus eleven step-2 invariants: buf.yaml lint/breaking shape, proto3 + package declaration on every file, consistent go_package aliases, snake_case filenames, no duplicate field numbers, the LedgerService RPC surface, Money-via-minor-units, the new Makefile targets, and the privacy scan over both schemas and buf.yaml.

What we built

We turned the empty proto/ slot into a real schema package. Two .proto files now live at proto/monorepo/v1/, sharing a single Money value type and an explicit LedgerService RPC surface. The buf.yaml next to them switches on DEFAULT lint so naming and packaging stay tidy, and switches on FILE-scope breaking checks so backwards-incompatible edits are caught before they reach a generator.

The Makefile grew two thin wrappers — make lint and make breaking — so contributors and CI invoke the exact same commands. Because the breaking check uses .git#branch=main as the baseline, every pull request is automatically diffed against the integration branch with no extra infrastructure.

The most important addition is the pytest module. Eleven file-content assertions encode every promise this step makes: the package layout, the buf configuration, the proto3 syntax declaration, the snake_case filenames, the absence of duplicate field numbers, the integer-only Money type, the Ledger RPC surface, and the privacy guardrails. The suite runs in under a hundred milliseconds and requires no buf binary, so a brand-new clone validates the contract immediately.

With the schemas frozen behind both buf lint and pytest, the next step can wire in the Rust generator with confidence that the inputs will not move under it. Field numbers, type names, and package paths are now invariants, not assumptions.

Repository

The state of the code after this step: 1887444

Step 3: Fanning One Schema into Rust, Python, and TypeScript via buf.gen.yaml

Step 2 froze the schema contract: proto/monorepo/v1/money.proto and proto/monorepo/v1/ledger.proto are now gated by DEFAULT lint and FILE-scope breaking checks, and a pytest module pins every invariant. What is still missing is the generator config — without it, buf has no opinion about which languages to emit, where to drop the stubs, or which remote plugins to invoke.

This step authors buf.gen.yaml at the repository root and declares five remote plugins that together produce idiomatic stubs for three language targets. Rust gets prost messages plus tonic services, Python gets the official protocolbuffers messages plus the gRPC service stubs, and TypeScript gets the @bufbuild/protobuf runtime via buf.build/bufbuild/es. A new make generate target wraps buf generate, and a new pytest module locks the plugin list, the output directories, and the privacy guardrails so the topology cannot drift silently.

Setup

This step touches the repository root plus the test tree. No new third-party libraries are pulled in — the generator config drives remote buf plugins, so contributors do not need protoc, prost-build, grpcio-tools, or @bufbuild/protoc-gen-es installed locally. The Buf Schema Registry resolves every plugin reference on demand.

  • buf.gen.yaml — five-plugin generator manifest at the repository root, with managed mode enabled and a go_package_prefix pointing at the public vytharion namespace.
  • Makefile — extended with a generate target that invokes buf generate, plus a refreshed help summary.
  • tests/test_buf_gen.py — twelve pytest checks that pin the version, managed mode, every required plugin, per-language output directories, the target=ts option, the well-known-types flag on the Rust plugins, the Makefile entry, and the privacy scan.

The Rust, Python, and TypeScript output directories — rust/crates/monorepo-pb/src/gen, python/src/monorepo_pb/gen, and typescript/src/gen — are not created in this step. Each one is the responsibility of the language-specific wiring step that follows. Step 3 only commits to where the generated code will live, not to the surrounding crate or package layout.

Implementation

The manifest is short and intentionally flat. It declares version: v1, enables managed mode so consumers do not have to repeat boilerplate file-level options inside every .proto, and pins the Go package prefix under the public vytharion GitHub namespace. Even though this article does not ship a Go target, the prefix matters: managed mode would otherwise inject an empty default, which trips buf's own lint checks the next time someone adds a Go consumer.

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: github.com/vytharion/buf-gen-protobuf-rust-python-typescript/gen/go

The first pair of plugin entries handles Rust. neoeinstein-prost emits the message structs, and neoeinstein-tonic emits the async gRPC service stubs. Both write into the same rust/crates/monorepo-pb/src/gen directory so the two outputs share a module tree — tonic generates mod declarations that expect the prost-generated types to be reachable as siblings. The compile_well_known_types opt is set on both so google.protobuf.Timestamp (used by Account.created_at and TransferResponse.posted_at) resolves without an extra crate dependency.

plugins:
  - plugin: buf.build/community/neoeinstein-prost
    out: rust/crates/monorepo-pb/src/gen
    opt:
      - bytes=.
      - compile_well_known_types
      - file_descriptor_set
  - plugin: buf.build/community/neoeinstein-tonic
    out: rust/crates/monorepo-pb/src/gen
    opt:
      - compile_well_known_types
      - no_include

bytes=. tells prost to emit bytes::Bytes for every bytes field rather than Vec<u8>, which avoids a needless copy at the FFI boundary. file_descriptor_set asks prost to also emit the encoded FileDescriptorSet so the Rust crate can later support runtime reflection without re-invoking protoc. The no_include opt on the tonic block prevents it from re-emitting the prost include! directives — the prost block is already responsible for that, and emitting them twice causes Rust to complain about duplicate module roots.

Python is the simplest target. The two official Buf-hosted plugins — buf.build/protocolbuffers/python for message types and buf.build/grpc/python for service stubs — both write into python/src/monorepo_pb/gen, mirroring the Rust pattern of co-locating messages and service code. No extra options are needed; the defaults already produce the canonical _pb2.py / _pb2_grpc.py filenames that grpcio-tools users expect.

  - plugin: buf.build/protocolbuffers/python
    out: python/src/monorepo_pb/gen
  - plugin: buf.build/grpc/python
    out: python/src/monorepo_pb/gen

TypeScript is handled by buf.build/bufbuild/es, the modern Buf-maintained plugin that emits the @bufbuild/protobuf runtime. Two opts shape the output: target=ts ships TypeScript sources rather than transpiled JavaScript, and import_extension=js makes the generated imports use a .js suffix so the same files load cleanly under Node ESM after a tsc build.

  - plugin: buf.build/bufbuild/es
    out: typescript/src/gen
    opt:
      - target=ts
      - import_extension=js

The Makefile gains a generate target so contributors and CI both invoke the exact same command. Keeping it next to lint, breaking, and test means anyone scanning the help output can see the full proto lifecycle in one place.

generate:
	buf generate

The pytest module locks the configuration shape against drift. Reading the YAML as plain text — rather than parsing it into a dict — keeps the assertions readable and lets each test name the exact substring it is enforcing. The module-level constants name the three output directories and the five plugins; if any of those move, the corresponding test fails with a clear message.

RUST_OUT_DIR = "rust/crates/monorepo-pb/src/gen"
PYTHON_OUT_DIR = "python/src/monorepo_pb/gen"
TYPESCRIPT_OUT_DIR = "typescript/src/gen"

REQUIRED_PLUGINS = (
    "buf.build/community/neoeinstein-prost",
    "buf.build/community/neoeinstein-tonic",
    "buf.build/protocolbuffers/python",
    "buf.build/grpc/python",
    "buf.build/bufbuild/es",
)

A small helper splits buf.gen.yaml into one block per plugin so per-plugin assertions can reason locally. That keeps each test focused on one invariant — for example, "the prost block points at the Rust crate" or "the bufbuild/es block carries target=ts" — instead of running giant regex sweeps across the whole file.

def test_rust_plugins_share_a_single_output_tree():
    text = _read(GEN_CONFIG)
    blocks = _plugin_blocks(text)
    prost_block = _block_for("neoeinstein-prost", blocks)
    tonic_block = _block_for("neoeinstein-tonic", blocks)
    assert RUST_OUT_DIR in prost_block
    assert RUST_OUT_DIR in tonic_block

Two more checks deserve a callout. test_managed_go_package_prefix_lives_under_vytharion_namespace verifies that the Go package prefix points at the public repository so a future Go consumer never inherits an internal slug. test_buf_gen_yaml_has_no_operator_private_paths scans for the same operator-private path fragments enforced in step 2, so the gate stays consistent across files.

Verification

Run the full suite from the repository root after applying the step 3 changes:

pytest -q

The seven step-1 bootstrap tests and the eleven step-2 schema tests still pass, and the twelve new generator tests join them for a thirty-dot summary:

..............................                                           [100%]
30 passed in 0.07s

That covers the step-3 invariants in full: buf.gen.yaml exists at the repo root, declares version: v1, enables managed mode, pins the Go package prefix under the vytharion namespace, lists all five required plugins, gives each plugin its own out: directive, co-locates the two Rust plugins in rust/crates/monorepo-pb/src/gen, co-locates the two Python plugins in python/src/monorepo_pb/gen, sends the TypeScript plugin to typescript/src/gen with target=ts, enables compile_well_known_types on both Rust plugins, exposes a generate target in the Makefile, references only existing top-level language directories, and contains zero operator-private path fragments.

What we built

We added the missing piece between the frozen schema and the language consumers: a buf.gen.yaml manifest that turns a single buf generate invocation into five parallel plugin runs. Rust receives prost messages plus tonic services in one shared module, Python receives the official protocolbuffers and grpc stubs side by side, and TypeScript receives @bufbuild/protobuf sources with ESM-friendly imports.

Managed mode is now on, so the schemas no longer need to repeat file-level options every time we add a new .proto. The Go package prefix is pinned under the public vytharion namespace, which means even unused targets stay safe for the privacy gate. The make generate wrapper gives contributors and CI a single, scriptable entry point.

The twelve new pytest assertions encode every promise this step makes. They run without buf installed, so a fresh clone validates the generator topology in under a hundred milliseconds, and they fail loudly the moment someone moves a plugin, drops a flag, or leaks a private path.

With the fan-out locked in, the next step can plug the Rust crate into the rust/crates/monorepo-pb/src/gen tree and consume the prost+tonic output directly, confident that the output layout and plugin options will not drift under its feet.

Repository

The state of the code after this step: ae959af

Step 4: Wiring the monorepo-pb Crate to Consume Generated prost and tonic Code

Step 3 stopped one inch short of a working Rust target: buf.gen.yaml now points neoeinstein-prost and neoeinstein-tonic at rust/crates/monorepo-pb/src/gen, but there is no Cargo workspace at that path, no crate manifest, no lib.rs, and no module file to splice the generated .rs files into a Rust module tree. Until those pieces exist, buf generate produces orphan source code that no cargo invocation can compile.

This step builds the smallest Rust workspace that can consume the prost and tonic output end to end. We declare a workspace at rust/Cargo.toml, add a single member crate monorepo-pb that depends on prost 0.13 and tonic 0.12, hand-write a stable src/gen/mod.rs that include!s the regenerated files, and re-export the proto namespace from lib.rs. A new pytest module pins every invariant of the wiring so a future regeneration cannot silently break the build.

Setup

The crate sits inside a Cargo workspace rather than as a top-level package because subsequent steps will add sibling crates (a reflection server, integration tests, examples) that need to share the same prost, tonic, and tokio versions. Centralising the versions in [workspace.dependencies] and the metadata in [workspace.package] means every future member inherits the same MSRV and licence without having to repeat them.

  • rust/Cargo.toml — workspace manifest with resolver = "2", one member (crates/monorepo-pb), and pinned versions for prost, prost-types, tonic, bytes, and tokio.

  • rust/rust-toolchain.toml — pins the toolchain to a concrete 1.79.0 channel plus rustfmt and clippy, so CI and contributor laptops compile against the exact same compiler.

  • rust/crates/monorepo-pb/Cargo.toml — the consumer crate; inherits everything from the workspace, declares default = ["server", "client"] features so downstream code can opt out of either tonic half.

  • rust/crates/monorepo-pb/src/lib.rs — crate root with #![forbid(unsafe_code)], a pub mod gen; line, and re-exports for monorepo::v1, FILE_DESCRIPTOR_SET, and the PROTO_PACKAGE const.

  • rust/crates/monorepo-pb/src/gen/mod.rs — hand-written module wiring that include!s the prost output, the tonic output, and the encoded FileDescriptorSet.

  • rust/crates/monorepo-pb/src/gen/.gitignore — ignores every regenerated artefact while keeping mod.rs and itself in version control.

  • rust/crates/monorepo-pb/README.md — pointer for new contributors: run make generate, then cargo check -p monorepo-pb.

  • Makefile — three new targets (rust-check, rust-clippy, rust-test) plus help text, so the proto and Rust workflows live in one entry point.

  • tests/test_rust_crate.py — sixteen pytest checks that pin the workspace shape, the dependency versions, the lints, the module wiring, the gitignore policy, and the privacy guard.

No Cargo.lock is committed yet; that lands once a binary crate exists. The src/gen directory ships with only .gitignore and mod.rs checked in, exactly as buf expects — every other file is overwritten by buf generate.

Implementation

The workspace manifest is the first new file. It opts into resolver v2 so feature unification matches modern cargo behaviour, lists the single member crate, and pulls every shared dependency into [workspace.dependencies]. Pinning tonic with default-features = false and a curated features = ["codegen", "prost", "transport"] set avoids dragging in the tls stack before we need it.

[workspace]
resolver = "2"
members = ["crates/monorepo-pb"]

[workspace.package]
edition = "2021"
license = "MIT"
repository = "https://github.com/vytharion/buf-gen-protobuf-rust-python-typescript"
rust-version = "1.75"

[workspace.dependencies]
prost = "0.13"
prost-types = "0.13"
tonic = { version = "0.12", default-features = false, features = ["codegen", "prost", "transport"] }
bytes = "1.7"
tokio = { version = "1.40", default-features = false, features = ["macros", "rt-multi-thread"] }

The rust-toolchain.toml file sits next to the workspace manifest. Pinning a concrete 1.79.0 channel — rather than stable — guarantees CI does not drift the day a new compiler ships with a fresh lint that breaks the generated code. The minimal profile keeps first-time installs fast, while clippy and rustfmt are explicitly listed so the linter targets stay green.

[toolchain]
channel = "1.79.0"
components = ["rustfmt", "clippy"]
profile = "minimal"

The crate manifest is intentionally thin. Every metadata field uses *.workspace = true so a future migration to a new MSRV or licence stays a one-line change. The two features — server and client — are empty for now but reserved so downstream consumers can later compile only the half of the tonic stubs they actually need.

[package]
name = "monorepo-pb"
version = "0.1.0"
description = "Prost + tonic stubs for the monorepo.v1 schemas, regenerated by `buf generate`."
edition.workspace = true
license.workspace = true
repository.workspace = true
rust-version.workspace = true

[lib]
path = "src/lib.rs"

[dependencies]
prost = { workspace = true }
prost-types = { workspace = true }
tonic = { workspace = true }
bytes = { workspace = true }

[features]
default = ["server", "client"]
server = []
client = []

src/lib.rs is the public surface. The forbid(unsafe_code) and deny(rust_2018_idioms) lints lock the crate down at the root so generated code cannot silently introduce unsafe blocks or pre-2018 patterns. The two pub use lines turn the verbose generated path gen::monorepo::v1 into the ergonomic monorepo_pb::v1, and the two consts expose the FileDescriptorSet bytes alongside the proto package name as a compile-time constant.

#![forbid(unsafe_code)]
#![deny(rust_2018_idioms)]

pub mod gen;

pub use gen::monorepo;
pub use gen::monorepo::v1;

pub const FILE_DESCRIPTOR_SET: &[u8] = gen::FILE_DESCRIPTOR_SET;
pub const PROTO_PACKAGE: &str = "monorepo.v1";

The most subtle file is src/gen/mod.rs. The two community plugins emit their output as flat monorepo.v1.rs and monorepo.v1.tonic.rs files; we use nested pub mod blocks plus include! so those flat files appear under the natural monorepo::v1 Rust path. The order of the two include! calls matters — tonic's generated code references prost-generated types, so prost has to expand first.

#![allow(clippy::all)]
#![allow(rustdoc::broken_intra_doc_links)]

pub mod monorepo {
    pub mod v1 {
        include!("monorepo.v1.rs");
        include!("monorepo.v1.tonic.rs");
    }
}

pub(crate) const FILE_DESCRIPTOR_SET: &[u8] =
    include_bytes!("file_descriptor_set.bin");

The accompanying .gitignore is what keeps the regen story honest: everything in src/gen/ is ignored by default, with explicit allow-lines for mod.rs and .gitignore itself. A fresh checkout shows zero generated files; a buf generate followed by git status shows them as untracked, which is the desired state.

*
!.gitignore
!mod.rs

The Makefile gains three targets that mirror the existing lint, breaking, and generate entries. Keeping them inside the top-level Makefile means contributors do not have to cd rust first, and CI gets a single, scriptable contract.

rust-check:
	cd rust && cargo check --workspace --all-targets

rust-clippy:
	cd rust && cargo clippy --workspace --all-targets -- -D warnings

rust-test:
	cd rust && cargo test --workspace

The pytest module pins every promise above. It reads the files as plain text — exactly like the schema and generator gates in steps 2 and 3 — so the suite still runs in a hundred milliseconds on a machine that has neither cargo nor buf installed. The privacy scan (test_rust_files_have_no_operator_private_paths) extends the same blocklist used elsewhere to the new Rust files, so any leaked developer-machine literal fails the gate immediately.

def test_gen_mod_rs_wires_prost_and_tonic_outputs():
    text = _read(GEN_DIR / "mod.rs")
    assert "pub mod monorepo" in text
    assert "pub mod v1" in text
    assert 'include!("monorepo.v1.rs")' in text
    assert 'include!("monorepo.v1.tonic.rs")' in text
    assert 'include_bytes!("file_descriptor_set.bin")' in text

Verification

Run the full suite from the repository root after applying the step 4 changes:

pytest -q

The seven step-1 bootstrap tests, eleven step-2 schema tests, and twelve step-3 generator tests still pass; the sixteen new Rust-wiring tests bring the total to forty-seven:

...............................................                          [100%]
47 passed in 0.07s

That covers the step-4 invariants: the workspace manifest exists with resolver v2 and lists crates/monorepo-pb; prost 0.13, tonic 0.12, and bytes are pinned in [workspace.dependencies]; the toolchain file pins a concrete 1.x channel plus rustfmt and clippy; the crate manifest inherits workspace metadata and exposes server plus client features; lib.rs declares pub mod gen, forbids unsafe code, and exposes both FILE_DESCRIPTOR_SET and a "monorepo.v1" package const; src/gen/mod.rs wires the prost output, the tonic output, and the encoded descriptor set; the gen .gitignore keeps mod.rs and itself while ignoring everything else; buf.gen.yaml still emits Rust into this exact path; the Makefile exposes rust-check, rust-clippy, and rust-test; and the privacy scan finds zero operator-private fragments in any new file.

What we built

We added the Rust consumer that the previous step's buf.gen.yaml was waiting for. A two-tier Cargo workspace now lives at rust/, with monorepo-pb as its single member crate. Running make generate followed by make rust-check produces a clean compile against the prost messages and tonic services.

The wiring is split across exactly the files that need to survive a regeneration. src/gen/mod.rs is the only hand-written file inside the generated directory; everything else under src/gen/ is overwritten on every buf generate and excluded by the local .gitignore. That separation means contributors never accidentally commit a generated artefact, and a stale mod.rs is the single point of failure to inspect if a future regen breaks.

Sixteen pytest assertions encode the new invariants — workspace shape, dependency versions, toolchain pinning, feature flags, lint posture, module wiring, gitignore policy, Makefile entries, and the privacy guard — without requiring cargo on the test machine. The full forty-seven-test suite still runs in well under a second.

With the Rust target wired, the next step can repeat the same exercise for Python: stand up a monorepo_pb package that consumes the protocolbuffers/python and grpc/python output, mirror the same gitignore/module pattern, and add a sibling pytest module to lock it down.

Repository

The state of the code after this step: 98914dc

Step 5: Wiring the monorepo_pb Python Package to Consume Generated protobuf and grpcio Stubs

Step 4 turned the Rust slot of buf.gen.yaml into a working crate, but the Python slot still pointed at a bare python/.gitkeep. buf generate would happily emit monorepo/v1/money_pb2.py, ledger_pb2.py, and ledger_pb2_grpc.py into a directory with no surrounding package, no pyproject.toml, and no namespace __init__.py files. The result imports under no name a consumer can reach.

This step replaces that placeholder with the smallest installable Python package that can consume the generated stubs. We add a python/ subproject with src/-layout, declare it as monorepo-pb with PEP 561 typing, hand-write one __init__.py per namespace level under gen/, and ignore everything else the generator drops. A new pytest module pins eighteen invariants so a future buf generate followed by pip install -e python works without manual fixups.

Setup

The package is its own self-contained subproject rather than a top-level Python source tree. Keeping pyproject.toml, src/, and the package metadata inside python/ keeps the polyglot repo navigable — Rust users never see pyproject.toml, Python users never see Cargo.toml, and CI can install the two halves independently. The choice also lets the Python package declare its own MSRV-equivalent (requires-python >= 3.9) without affecting the proto schema directory.

  • python/pyproject.toml — PEP 517 metadata with setuptools backend, src/-layout discovery, runtime deps on protobuf>=4.25,<6 and grpcio>=1.60, and dev extras for grpcio-tools, mypy, mypy-protobuf, and ruff.

  • python/src/monorepo_pb/__init__.py — public surface; exposes a single typed const PROTO_PACKAGE = "monorepo.v1" and an explicit __all__.

  • python/src/monorepo_pb/py.typed — empty PEP 561 marker so downstream consumers receive the .pyi files emitted by mypy-protobuf.

  • python/src/monorepo_pb/README.md — pointer for new contributors: run make generate, then make python-install, then the smoke-import one-liner.

  • python/src/monorepo_pb/gen/__init__.py — namespace root for the generated tree; documents the regeneration contract and the consumer import path without eager-importing any *_pb2 module.

  • python/src/monorepo_pb/gen/monorepo/__init__.py and python/src/monorepo_pb/gen/monorepo/v1/__init__.py — namespace markers that survive a regen-and-diff cycle untouched.

  • python/src/monorepo_pb/gen/.gitignore — ignores every regenerated artefact while explicitly preserving the four hand-written __init__.py files plus the gitignore itself.

  • Makefile — four new targets (python-install, python-check, python-lint, python-test) plus an extended help summary, so the proto, Rust, and Python workflows share one entry point.

  • tests/test_python_package.py — eighteen pytest checks that read the files as plain text and pin the package shape, the dependency floors, the build backend, the mypy posture, the typed marker, the namespace markers, the no-eager-import rule, the gitignore allow-list, the Makefile entries, and the privacy guard.

The python/.gitkeep placeholder from step 1 is deleted in this step. The src/monorepo_pb/gen/monorepo/v1/ directory ships with only __init__.py checked in — every *_pb2.py, *_pb2_grpc.py, and *.pyi file is overwritten by buf generate.

Implementation

The pyproject.toml is the spine of the subproject. It declares setuptools as the build backend, points discovery at src/, lists monorepo_pb as the only package, and pins the runtime dependencies to versions compatible with both the protobuf 4.x and 5.x lines. The dev extras intentionally include grpcio-tools, even though buf invokes the remote grpc/python plugin, so contributors who want to regenerate stubs locally without buf still can.

[project]
name = "monorepo-pb"
version = "0.1.0"
requires-python = ">=3.9"
authors = [{ name = "vytharion" }]
dependencies = [
    "protobuf>=4.25,<6",
    "grpcio>=1.60",
]

[project.optional-dependencies]
dev = [
    "grpcio-tools>=1.60",
    "mypy>=1.8",
    "mypy-protobuf>=3.5",
    "ruff>=0.5",
]

[tool.setuptools.packages.find]
where = ["src"]
include = ["monorepo_pb*"]

monorepo_pb/__init__.py mirrors the Rust crate's lib.rs shape from step 4. It exposes a single typed constant that survives every regen, leaving the actual stub modules under monorepo_pb.gen.monorepo.v1 where the generator can rewrite them freely. Keeping the top-level __init__.py empty of stub-touching imports means import monorepo_pb succeeds on a fresh checkout, before buf generate has ever run.

from __future__ import annotations

PROTO_PACKAGE: str = "monorepo.v1"

__all__ = ["PROTO_PACKAGE"]

The three __init__.py files under gen/, gen/monorepo/, and gen/monorepo/v1/ are the subtle part. They are committed so a fresh checkout has a valid Python package tree before the generator has been run, but they must NOT import any *_pb2 module — doing so would make package import order depend on buf generate having already executed, which breaks pip install -e python on a clean clone. The pytest module enforces this with a regex anchored to column 0, so example imports inside docstrings remain legal.

"""Wiring for ``buf generate`` output.

Once ``buf generate`` has run, consumers can reach the stubs via the
proto package path::

    from monorepo_pb.gen.monorepo.v1 import money_pb2, ledger_pb2, ledger_pb2_grpc
"""
from __future__ import annotations

__all__: list[str] = []

The gen/.gitignore is the contract that keeps the regen story honest. The * line ignores everything, and the explicit ! allow-lines for .gitignore, every namespace __init__.py, and the namespace directories themselves keep the package tree intact across git clean -fdx followed by buf generate. Without this allow-list, a contributor running git clean would delete the hand-written __init__.py files and silently break installs.

*
!.gitignore
!__init__.py
!monorepo
!monorepo/__init__.py
!monorepo/v1
!monorepo/v1/__init__.py

The Makefile gains four targets that compose with the proto and Rust workflows established earlier. python-install invokes pip install -e .[dev] so the working copy is editable; python-check performs the smallest possible smoke import that proves the package metadata is wired correctly even before any stubs exist; python-lint and python-test keep the lint and test loops a single command away.

python-install:
	cd python && pip install -e .[dev]

python-check:
	cd python && python -c "import monorepo_pb; assert monorepo_pb.PROTO_PACKAGE == 'monorepo.v1'; print(monorepo_pb.PROTO_PACKAGE)"

python-lint:
	cd python && ruff check src

python-test:
	cd python && python -m pytest -q

The pytest module mirrors the assertion style used in steps 2, 3, and 4. Every check reads file contents as plain text, so the suite still runs in well under a second on a machine that has neither pip, buf, nor generated stubs on disk. The no-eager-import test in particular is worth highlighting — it is the single guard that catches the most common mistake when wiring grpcio-tools output into a src/-layout package.

def test_gen_namespace_inits_do_not_eager_import_generated_stubs():
    pb2_import_re = re.compile(
        r"^(?:from\s+\S+\s+)?import\s+[\w,\s]*_pb2", re.MULTILINE
    )
    for init_path in (
        GEN_DIR / "__init__.py",
        GEN_DIR / "monorepo" / "__init__.py",
        GEN_V1_DIR / "__init__.py",
    ):
        text = _read(init_path)
        assert not pb2_import_re.search(text), (
            f"{init_path} must not eager-import generated stubs"
        )

Verification

Run the full pytest suite from the repository root after applying the step 5 changes:

pytest -q

The seven step-1 bootstrap tests, eleven step-2 schema tests, twelve step-3 generator tests, and sixteen step-4 Rust-wiring tests still pass; the eighteen new Python-wiring tests bring the total to sixty-four:

................................................................         [100%]
64 passed in 0.33s

That covers every step-5 invariant: the python/ subproject uses src-layout; pyproject.toml declares monorepo-pb with requires-python >= 3.9 and the vytharion author; runtime deps include protobuf>=4.25,<6 and grpcio>=1.60; dev extras include grpcio-tools, mypy, mypy-protobuf, and ruff; the setuptools backend scans src/; [tool.mypy] runs strict; the top-level __init__.py exposes a typed PROTO_PACKAGE = "monorepo.v1"; the PEP 561 py.typed marker is empty; the three gen/ namespace __init__.py files exist and never eager-import a *_pb2 module; the gen .gitignore ignores everything except the four hand-written files; buf.gen.yaml still emits Python output into python/src/monorepo_pb/gen; the Makefile exposes python-install, python-check, python-lint, and python-test; the package README documents the regeneration workflow; and the privacy scan finds zero operator-private path fragments in any new file.

What we built

We added the Python consumer that the step 3 generator config has been waiting on. A self-contained python/ subproject now lives next to the Rust workspace, with monorepo-pb as its single installable package. Running make generate followed by make python-install and make python-check produces a clean editable install that imports without manual fixups.

The wiring is split precisely across the files that survive a regeneration. Four hand-written __init__.py files plus one .gitignore are the only tracked entries inside src/monorepo_pb/gen/; every other file under that tree is overwritten on every buf generate and excluded from version control by the allow-list. That separation means contributors never accidentally commit a generated artefact, and the no-eager-import gate guarantees the package is importable before the generator has ever run.

Eighteen pytest assertions encode the new invariants — subproject layout, dependency floors, build backend, mypy posture, typed marker, namespace markers, no-eager-import rule, gitignore allow-list, Makefile entries, README contract, and the privacy guard — without requiring pip, grpcio, or generated stubs on the test machine. The full sixty-four-test suite still runs in a third of a second.

With the Python target wired, the next step can repeat the exercise one more time for TypeScript: stand up a typescript/ subproject that consumes the bufbuild/es output, mirror the same gitignore-plus-namespace-marker pattern, and add a sibling pytest module to lock it down.

Repository

The state of the code after this step: 013de28

Step 6: Wiring the @vytharion/monorepo-pb TypeScript Package to Consume Generated @bufbuild/protobuf Code

Step 5 closed out the Python side of the polyglot fan-out: buf generate now drops *_pb2.py and *_pb2_grpc.py into a real installable monorepo-pb package, and a regen-and-install loop is locked down by pytest. The TypeScript slot of buf.gen.yaml is still pointed at typescript/.gitkeep, which means the third plugin entry from step 3 is emitting *_pb.ts files into a directory that has no package.json, no tsconfig.json, and no module graph for a consumer to follow.

This step replaces that placeholder with the smallest installable TypeScript package that can consume the @bufbuild/protobuf stubs. We add a Bun-first typescript/ subproject with src/ layout, declare it as @vytharion/monorepo-pb, hand-write one index.ts per namespace level under gen/, and ignore every regenerated artefact through a precise allow-list. A new pytest module pins twenty invariants so a future buf generate followed by bun install produces a tsc --noEmit pass without manual fixups.

Setup

The TypeScript package is its own self-contained subproject rather than a top-level Node tree. Keeping package.json, tsconfig.json, src/, and the package metadata inside typescript/ keeps the polyglot repo navigable — Rust users never see package.json, Python users never see tsconfig.json, and CI can install each half independently. The choice also lets the TypeScript half pin its own runtime floor (engines.node >= 18, engines.bun >= 1.1) without affecting the proto schema directory or the other generators.

  • typescript/package.json — npm metadata with "type": "module", a runtime dep on @bufbuild/protobuf ^1.10, dev deps on @bufbuild/protoc-gen-es, typescript, and @types/bun, an exports map that exposes the package root plus an explicit ./gen/monorepo/v1 subpath, and four scripts (build, check, lint, test).

  • typescript/tsconfig.json — strict-mode compiler config targeting ES2022 with module: ESNext, moduleResolution: Bundler, rootDir: src, outDir: dist, plus tightening flags (exactOptionalPropertyTypes, noUnusedLocals, useUnknownInCatchVariables, verbatimModuleSyntax, isolatedModules).

  • typescript/src/index.ts — public surface; exports a single PROTO_PACKAGE = "monorepo.v1" as const plus a ProtoPackage type alias, with no *_pb imports at module load time.

  • typescript/src/gen/index.ts — namespace root for the generated tree; documents the regeneration contract and the consumer import path without re-exporting any *_pb module.

  • typescript/src/gen/monorepo/index.ts and typescript/src/gen/monorepo/v1/index.ts — namespace markers that survive a regen-and-diff cycle untouched.

  • typescript/src/gen/.gitignore — ignores every regenerated artefact while explicitly preserving the three hand-written index.ts files, the namespace directories, and the gitignore itself.

  • typescript/README.md — pointer for new contributors: run make generate, then make ts-install, then make ts-check / make ts-test.

  • Makefile — four new targets (ts-install, ts-check, ts-lint, ts-test) and an extended help summary, so the proto, Rust, Python, and TypeScript workflows all share one entry point.

  • tests/test_typescript_package.py — twenty pytest checks that read the files as plain text and pin the package shape, the runtime dependency floor, the dev-extras list, the entrypoints, the script names, the engine floors, the strict-mode tsconfig, the rootDir/outDir pair, the namespace markers, the no-eager-import rule, the gitignore allow-list, the buf.gen.yaml cross-reference, the Makefile entries, the README contract, and the privacy guard.

The typescript/.gitkeep placeholder from step 1 is deleted in this step. The src/gen/monorepo/v1/ directory ships with only index.ts checked in — every *_pb.ts and *_connect.ts file the buf.build/bufbuild/es plugin emits is overwritten on every regeneration and excluded from version control by the allow-list.

Implementation

The package.json is the spine of the subproject. It declares the package as ESM ("type": "module"), exposes the source tree directly via main, module, and types, and adds an exports map with two entries — the package root, and an explicit ./gen/monorepo/v1 subpath for consumers who want to reach the generated namespace without parsing strings. Shipping the .ts sources (no pre-built dist/) is the conscious trade-off: every modern bundler (bun, vite, esbuild) consumes TypeScript natively, and skipping the build step keeps the consumer story to bun install plus an import.

{
  "name": "@vytharion/monorepo-pb",
  "version": "0.1.0",
  "type": "module",
  "main": "./src/index.ts",
  "module": "./src/index.ts",
  "types": "./src/index.ts",
  "exports": {
    ".": {
      "types": "./src/index.ts",
      "import": "./src/index.ts"
    },
    "./gen/monorepo/v1": {
      "types": "./src/gen/monorepo/v1/index.ts",
      "import": "./src/gen/monorepo/v1/index.ts"
    }
  },
  "dependencies": {
    "@bufbuild/protobuf": "^1.10.0"
  },
  "devDependencies": {
    "@bufbuild/protoc-gen-es": "^1.10.0",
    "@types/bun": "^1.1.0",
    "typescript": "^5.5.0"
  }
}

The tsconfig.json mirrors the strict posture the Rust crate and Python package adopt in steps 4 and 5. strict: true turns on the full bundle (noImplicitAny, strictNullChecks, strictFunctionTypes, …), and the extra flags below catch the classes of mistake that strict mode alone still lets through. moduleResolution: Bundler is the modern choice for Bun and Vite consumers — it accepts .js import specifiers that resolve to .ts source files, which is exactly what the bufbuild/es plugin emits under import_extension=js.

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "Bundler",
    "rootDir": "src",
    "outDir": "dist",
    "strict": true,
    "noImplicitOverride": true,
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "exactOptionalPropertyTypes": true,
    "useUnknownInCatchVariables": true,
    "verbatimModuleSyntax": true,
    "isolatedModules": true
  },
  "include": ["src"],
  "exclude": ["node_modules", "dist"]
}

src/index.ts mirrors the Rust crate's lib.rs from step 4 and the Python package's top-level __init__.py from step 5. It exposes a single typed constant that survives every regen, leaving the actual message classes under gen/monorepo/v1/ where the generator can rewrite them freely. Keeping the entrypoint free of *_pb imports means import { PROTO_PACKAGE } from "@vytharion/monorepo-pb" succeeds on a fresh clone, before buf generate has ever run.

export const PROTO_PACKAGE = "monorepo.v1" as const;

export type ProtoPackage = typeof PROTO_PACKAGE;

The three index.ts files under gen/, gen/monorepo/, and gen/monorepo/v1/ are the subtle part. They are committed so a fresh checkout has a valid TypeScript module tree before the generator has been run, but they must NOT re-export any *_pb module — doing so would make tsc fail on a fresh clone (the targets don't exist yet) and would couple package import order to having already run the generator. Each file exports an empty namespace (export {};) plus a JSDoc block that documents the consumer import path.

/**
 * Once `buf generate` has run, consumers can reach the message classes
 * through the proto package path:
 *
 *     import { Money } from "@vytharion/monorepo-pb/gen/monorepo/v1";
 */

export {};

The gen/.gitignore is the contract that keeps the regen story honest. The * line ignores everything, and the explicit ! allow-lines for .gitignore, each namespace index.ts, and the namespace directories themselves keep the package tree intact across git clean -fdx followed by buf generate. Without this allow-list, a contributor running git clean would delete the hand-written index.ts files and silently break tsc --noEmit on the next install.

*
!.gitignore
!index.ts
!monorepo
!monorepo/index.ts
!monorepo/v1
!monorepo/v1/index.ts

The Makefile gains four targets that compose with the proto, Rust, and Python workflows established earlier. ts-install runs bun install inside typescript/; ts-check and ts-lint both invoke tsc --noEmit (the lint pass is just a stricter type-check rather than a second tool dependency); ts-test runs bun test. Pointing both lint and check at the same compiler means contributors can never be surprised by one passing while the other fails.

ts-install:
	cd typescript && bun install

ts-check:
	cd typescript && bunx tsc --noEmit -p tsconfig.json

ts-lint:
	cd typescript && bunx tsc --noEmit -p tsconfig.json

ts-test:
	cd typescript && bun test

The pytest module mirrors the assertion style from steps 2 through 5. Every check reads file contents as plain text, so the suite still runs in well under a second on a machine that has neither bun, tsc, buf, nor generated stubs on disk. The no-eager-import test is the headline guard — it walks every committed index.ts and rejects any real import ... _pb statement at column 0, while tolerating example imports inside JSDoc blocks (which start with a leading *).

def test_gen_namespace_indexes_do_not_eager_import_generated_stubs():
    pb_import_re = re.compile(
        r"^\s*import\s+[^;]*_pb(?:\s|;|/|\")", re.MULTILINE
    )
    for index_path in (
        GEN_DIR / "index.ts",
        GEN_DIR / "monorepo" / "index.ts",
        GEN_V1_DIR / "index.ts",
    ):
        text = _read(index_path)
        assert not pb_import_re.search(text), (
            f"{index_path} must not eager-import generated stubs"
        )

Verification

Run the full pytest suite from the repository root after applying the step 6 changes:

pytest -q

The seven step-1 bootstrap tests, eleven step-2 schema tests, twelve step-3 generator tests, sixteen step-4 Rust-wiring tests, and eighteen step-5 Python-wiring tests still pass; the twenty new TypeScript-wiring tests bring the total to eighty-four:

........................................................................ [ 85%]
............                                                             [100%]
84 passed in 0.17s

That covers every step-6 invariant: the typescript/ subproject uses src/ layout; the bootstrap .gitkeep is gone; package.json declares @vytharion/monorepo-pb at version 0.1.0 with "type": "module" and MIT license; the runtime depends on @bufbuild/protobuf pinned to the 1.x or 2.x series; dev extras include @bufbuild/protoc-gen-es, typescript, and @types/bun; the package exposes main, module, types, and an exports map with the package root; the four scripts (build, check, lint, test) are wired to tsc and bun test; the engines block pins both node and bun floors; the tsconfig enables strict mode, pins target / module / moduleResolution, fixes rootDir: src plus an outDir, and includes only the src/ tree while excluding node_modules; src/index.ts exports a typed PROTO_PACKAGE = "monorepo.v1" and never eager-imports a *_pb module; the three gen/ namespace index.ts files exist and never eager-import a *_pb module; the gen .gitignore ignores everything except the four hand-written keepers; buf.gen.yaml still emits TypeScript output into typescript/src/gen via buf.build/bufbuild/es with target=ts; the Makefile exposes ts-install, ts-check, ts-lint, and ts-test wired to bun install, tsc, and bun test; the package README documents the regeneration workflow and names the @bufbuild/protobuf runtime; and the privacy scan finds zero operator-private path fragments in any new file.

What we built

We added the TypeScript consumer that the step 3 generator config has been waiting on. A self-contained typescript/ subproject now lives next to the Rust workspace and the Python subproject, with @vytharion/monorepo-pb as its single installable package. Running make generate followed by make ts-install and make ts-check produces a clean strict-mode type-check pass without manual fixups.

The wiring is split precisely across the files that survive a regeneration. Three hand-written index.ts files plus one .gitignore are the only tracked entries inside src/gen/; every other file under that tree is overwritten on every buf generate and excluded from version control by the allow-list. That separation means contributors never accidentally commit a generated artefact, and the no-eager-import gate guarantees the package is importable before the generator has ever run.

The package ships as TypeScript source rather than a pre-built dist/ bundle. That trade-off keeps the consumer story to a single bun install, leans on the modern bundler ecosystem to handle .ts sources directly, and lets the exports map point at the real implementation files instead of a generated declaration layer. The moduleResolution: Bundler setting in tsconfig.json is what makes the import_extension=js output from the bufbuild/es plugin resolve cleanly against the .ts sources on disk.

Twenty pytest assertions encode the new invariants — subproject layout, package metadata, dependency floors, entrypoints, scripts, engine floors, strict tsconfig, namespace markers, no-eager-import rule, gitignore allow-list, buf.gen.yaml cross-reference, Makefile entries, README contract, and the privacy guard — without requiring bun, tsc, or generated stubs on the test machine. The full eighty-four-test suite still runs in roughly a fifth of a second, which closes out the three-language fan-out the article set out to deliver.

Repository

The state of the code after this step: 4b14c86

Step 7: Locking Wire Compatibility with a Canonical Smoke Vector and a buf-Gated CI Pipeline

Step 6 closed out the polyglot fan-out — buf generate now drops working stubs into the Rust crate, the Python package, and the TypeScript subproject, and each side has its own pytest-pinned wiring. What the repo still does not have is a single check that proves the three runtimes actually agree on the wire. Rust serializes monorepo.v1.Money through prost, Python through google.protobuf, TypeScript through @bufbuild/protobuf, and nothing yet stops a generator upgrade from quietly flipping a field tag in one language while leaving the other two untouched.

This step adds that proof. A canonical JSON vector pins one Money value plus its expected proto3 wire bytes, three language-native smoke harnesses round-trip the message against those bytes, and a .github/workflows/ci.yml workflow runs buf lint, buf breaking against main, and the three smoke jobs on every push and pull request. A new make smoke / make ci pair lets contributors reproduce the same pipeline locally. Nineteen new pytest assertions pin the harness wiring without needing cargo, bun, buf, or generated stubs on the test machine, bringing the suite total from eighty-four to one hundred and three.

Setup

The vector and the smoke harnesses live in places that match each language's standard test layout, which keeps each runtime's tooling happy without configuration overrides. The fixture sits at the repository root under tests/fixtures/ so every language can resolve a single shared path — Rust climbs three levels above CARGO_MANIFEST_DIR, Python uses Path(__file__).resolve().parents[2], and TypeScript walks two levels above the harness file. Pinning the lookup logic to those three independent path computations means a future restructure has to update all three call sites, which makes silent drift impossible.

  • tests/fixtures/cross_language_vector.json — the canonical contract: proto_package, a Money value (USD, 1099 minor units), an Account value (acct_2pV5JqEKxQ7HwBZ, Vytharion Treasury), and the expected money_wire_bytes_hex blob 0a0355534410cb08.
  • rust/crates/monorepo-pb/tests/cross_language_smoke.rs — cargo test integration test that round-trips Money via prost::Message::encode_to_vec + Money::decode.
  • python/tests/test_cross_language_smoke.py — pytest module that round-trips Money via SerializeToString + ParseFromString, with an ImportError-guarded skip for fresh clones.
  • typescript/tests/cross_language_smoke.test.ts — bun:test suite that round-trips Money via @bufbuild/protobuf's toBinary / fromBinary, with an existsSync-guarded skip for fresh clones.
  • .github/workflows/ci.yml — four-job GitHub Actions workflow: proto-gates (buf lint + buf breaking) followed by rust-smoke, python-smoke, typescript-smoke, each needs: proto-gates.
  • rust/Cargo.toml and rust/crates/monorepo-pb/Cargo.toml — pin serde_json = "1.0" in workspace dependencies and pull it in as a [dev-dependencies] entry on the crate so the smoke harness can parse the vector without leaking JSON into the library's public surface.
  • Makefile — two new targets, smoke and ci, plus an extended help summary.
  • tests/test_cross_language_smoke.py — nineteen pytest assertions that read every new file as plain text and pin the contract: the vector shape, the proto3 wire encoding, the three harness wiring contracts, the workflow triggers, the pinned buf and action versions, the needs: proto-gates ordering, the Makefile composition, the Rust workspace serde_json pin, and the privacy guard.

Implementation

The vector file is the spine of the step. It carries the exact byte sequence every harness has to reproduce, plus a short _comment field that documents the contract for a future maintainer who opens the file without reading the article. Pinning a single positive amount (1099) is deliberate — it forces a multi-byte varint, which exercises the LEB128 encoder of each runtime, rather than the degenerate single-byte path a value of 0 or 1 would take.

{
  "_comment": "Canonical cross-language smoke vector...",
  "proto_package": "monorepo.v1",
  "money": {
    "currency_code": "USD",
    "amount_minor_units": 1099
  },
  "account": {
    "id": "acct_2pV5JqEKxQ7HwBZ",
    "display_name": "Vytharion Treasury"
  },
  "money_wire_bytes_hex": "0a0355534410cb08"
}

The pytest module verifies the hex blob from a fourth, independent direction. test_canonical_money_wire_bytes_match_the_proto3_encoding recomputes the expected bytes in pure Python — 0x0A for field 1 wire type 2, the length-prefixed "USD", 0x10 for field 2 wire type 0, and the LEB128 encoding of 1099 — and asserts equality against the JSON. Without this check, all three language harnesses could happily agree with each other while disagreeing with the protocol itself; this assertion anchors the vector to proto3 rather than to a particular runtime's idea of proto3.

expected = bytearray()
expected.append(0x0A)  # field 1, wire type 2 (length-delimited)
expected.append(len(currency))
expected.extend(currency)
expected.append(0x10)  # field 2, wire type 0 (varint)
while True:
    byte = amount & 0x7F
    amount >>= 7
    if amount:
        expected.append(byte | 0x80)
    else:
        expected.append(byte)
        break

The Rust harness imports Money through the crate's public surface (use monorepo_pb::v1::Money;) and reaches the fixture by climbing three ancestors above CARGO_MANIFEST_DIR. Encoding uses prost::Message::encode_to_vec and decoding goes back through Money::decode. Both halves assert against the same decode_hex of money_wire_bytes_hex, so a regression in either direction trips the test.

let original = Money {
    currency_code: currency.clone(),
    amount_minor_units: minor_units,
};
let encoded = original.encode_to_vec();
assert_eq!(encoded, decode_hex(expected_hex));

let decoded = Money::decode(&*encoded).expect("Money must decode");
assert_eq!(decoded.currency_code, currency);
assert_eq!(decoded.amount_minor_units, minor_units);

The Python harness reaches the generated stubs through the package path established in step 5 (from monorepo_pb.gen.monorepo.v1 import money_pb2) and falls back to a pytest.skip when the import fails. That skip branch only ever fires on a fresh clone that has not run make generate; CI runs buf generate first so the smoke job either passes the assertion or fails outright — never silently skips. The TypeScript harness mirrors the same shape under bun:test, dynamically importing the generated money_pb.ts and falling back to a console warning when the file is missing.

original = money_pb2.Money(
    currency_code=vector["money"]["currency_code"],
    amount_minor_units=vector["money"]["amount_minor_units"],
)
encoded = original.SerializeToString()
expected = bytes.fromhex(vector["money_wire_bytes_hex"])
assert encoded == expected

The GitHub Actions workflow is structured as one cheap proto gate followed by three parallel language gates. proto-gates runs buf lint on every event and buf breaking --against ".git#branch=main,subdir=." only on pull requests — the breaking check needs the base branch to diff against and would produce a noisy false positive on a direct push to main. The three language jobs each declare needs: proto-gates, which means a broken schema fails fast on one runner instead of burning three.

proto-gates:
  steps:
    - name: buf lint
      run: buf lint
    - name: buf breaking (against main)
      if: github.event_name == 'pull_request'
      run: buf breaking --against ".git#branch=main,subdir=."

rust-smoke:
  needs: proto-gates
  steps:
    - name: buf generate
      run: make generate
    - name: cargo test (cross-language smoke)
      run: make rust-test

The Makefile gains two targets that compose the existing per-language test entry points. make smoke fans out to rust-test, python-test, and ts-test so contributors can run the full parity check after a local proto edit; make ci extends that with lint, breaking, and generate so the local pipeline is exactly the same shape as the GitHub Actions workflow. Pinning both ends of the contract to the same Makefile recipe means "it passed locally" and "it passed in CI" become the same statement.

smoke: rust-test python-test ts-test

ci: lint breaking generate smoke
	pytest -q

Verification

Run the full pytest suite from the repository root after applying the step 7 changes:

pytest -q

The seven step-1 bootstrap tests, eleven step-2 schema tests, twelve step-3 generator tests, sixteen step-4 Rust-wiring tests, eighteen step-5 Python-wiring tests, and twenty step-6 TypeScript-wiring tests still pass; the nineteen new cross-language and CI-gate tests bring the total to one hundred and three:

........................................................................ [ 69%]
...............................                                          [100%]
103 passed in 0.19s

That covers every step-7 invariant: the canonical vector exists at tests/fixtures/cross_language_vector.json; the vector pins monorepo.v1 plus a positive integer amount_minor_units and an acct_<slug>-shaped account id; the money_wire_bytes_hex value matches an independent pure-Python recomputation of the proto3 encoding; the Rust harness imports monorepo_pb::v1::Money and round-trips through encode_to_vec + Money::decode; the Python harness reaches monorepo_pb.gen.monorepo.v1.money_pb2 and round-trips through SerializeToString + ParseFromString; the TypeScript harness imports the generated money_pb and round-trips through toBinary + fromBinary under bun:test; all three harnesses consult the same money_wire_bytes_hex key; the workflow exists at .github/workflows/ci.yml and triggers on push and pull_request scoped to main; the workflow pins BUF_VERSION to a concrete x.y.z release; the workflow runs buf lint and buf breaking against .git#branch=main; all three language smoke jobs declare needs: proto-gates; each smoke job runs make generate before its native test; the workflow pins actions/checkout@v4, actions/setup-python@v5, and oven-sh/setup-bun@v2; the Makefile exposes smoke: and ci: composed in the exact recipe shape; the Rust workspace pins serde_json 1.x and the crate lists it under [dev-dependencies] rather than runtime [dependencies]; and the privacy scan finds zero operator-private path fragments in any new file.

What we built

We pulled the three languages from the previous six steps into a single enforceable contract. One JSON vector pins what Money(USD, 1099) looks like on the wire, three language-native smoke harnesses round-trip that vector through their respective protobuf runtimes, and a fourth pure-Python check anchors the vector to the proto3 specification itself rather than to any one runtime's opinion of it. A regression in any of the four lanes now trips a test.

The CI workflow turns the contract into a gate. Every push and pull request runs buf lint first, every pull request additionally runs buf breaking --against main so a backwards-incompatible schema edit cannot land, and the three language smoke jobs each needs: proto-gates so a broken schema burns one ubuntu runner instead of four. buf generate runs inside each language job before the test step, which means the smoke harnesses always run against freshly generated stubs rather than against whatever happens to be checked in.

Two new Makefile targets let contributors reproduce the same pipeline locally. make smoke runs the three language test entry points in sequence — useful after editing a .proto file and regenerating — and make ci composes lint, breaking, generate, smoke, and pytest -q into a single recipe that mirrors the GitHub Actions job graph. The recipe lives once in the Makefile and is invoked identically from the workflow, so the two environments stay aligned by construction.

Nineteen pytest assertions encode the new invariants — vector shape, proto3 anchoring, three harness wiring contracts, workflow triggers, pinned BUF_VERSION and action versions, needs: proto-gates ordering, make generate precedence, Makefile composition, Rust workspace serde_json pin, and the privacy guard — and run in roughly two-tenths of a second on a machine that has none of the language toolchains installed. The full one-hundred-and-three-test suite is the closing inventory of the tutorial: a single buf generate now fans out into three runtimes, every fan-out lane is gated by a smoke harness, and every smoke harness is gated by the CI workflow.

Repository

The state of the code after this step: 231fe0b

Repository

Full source at https://github.com/vytharion/buf-gen-protobuf-rust-python-typescript.

Walk the lessons by stepping through the git commits in the repo — each major step has its own commit you can git checkout and rerun.