I Taught a Bucket to Speak Git: A Git Server Backed by Object Storage
Here's a question worth sitting with: what happens if you just point a git server at an object storage bucket? Xe Iaso actually built it — a project called objgit, a git server with no local disk, no /usr/bin/git, and no database. The writeup is one of those engineering posts where every "obvious" assumption a filesystem gives you for free falls apart the moment a network round trip sits at the core. Worth walking through, because the lessons generalize well past the specific stack.
Git Was Always an Object Store
Strip away the porcelain and a git repository is four things:
- Objects — compressed blobs of data. Most objects in any repo are files.
- Trees — objects that map to other objects. Trees are folders.
- Commits — objects pointing at one tree and their parent commit, pinning down which files belong to one logical change set.
- Refs — branches and tags; tiny mutable pointers into the pile of objects.
⚠️ One correction to a common mental model, because it's load-bearing for everything that follows: git does not store patches against an empty folder and reconstruct history from diffs. It keeps the entire file content for every version. The diff model works fine for daily use, but it's wrong at the storage layer — which is exactly why big binary blobs wreck git tooling, and exactly the layer this project lives in.
Make a fresh repo and commit a README.md, and the .git folder looks like this:
$ tree .git
.git
├── COMMIT_EDITMSG
├── config
├── HEAD
├── index
├── objects
│ ├── 5e
│ │ └── b8151eb669aa4467b6dea2c4bce19183cd0b41
│ ├── 6a
│ │ └── 6a8ecfcae2632152486aca3d9150ef83dedd66
│ ├── f4
│ │ └── d2487a1c6d742c8037c0296ddf80625190bd80
│ ├── info
│ └── pack
└── refs
├── heads
│ └── main
└── tags
Three objects: a commit, a tree, and the README. The branch is just a file pointing at the commit:
$ cat .git/refs/heads/main
5eb8151eb669aa4467b6dea2c4bce19183cd0b41
The useful insight: half of this is content-addressed and never changes once committed. That maps beautifully onto append-only object storage. The things that do change — the refs — are tiny files an object store handles trivially.
Why Bother? Git Hosting Is a Stateful-Service Problem
Self-hosted git is one of the most stateful services in an otherwise stateless, cloud-native world. The whole model relies on git objects being 1:1 with filesystem objects, because everyone — GitHub included — shells out to the git binary, which demands a big mounted filesystem. That's a single point of failure on a single machine that can and will break. Git is theoretically decentralized, but in practice most of us put our repos in one big store with questionable uptime and call it GitHub.
If that bothers you enough to build an alternative, you have to "speak git" somehow, and the conventional options are bad:
- Shell out to the git binary — now your "library" is the argv of a subprocess and your error handling is screen-scraping stdout. Git's internals are held together by load-bearing calls to
die()that kill the process. - Link libgit — you inherit the same
die()-on-error behavior, so your app crashes at random. Bad for uptime. - Use libgit2 — the library-shaped rewrite, but it's GPL with a linking exception (explain that to your lawyers), you pay a jump to C on every git operation, development has stalled, the Go bindings are archived, and it still assumes a local filesystem.
The escape hatch is go-git: a pure-Go implementation of git's protocol and internals, no cgo, no /usr/bin/git, and — crucially — no assumption that repos live on a local filesystem. Its storage interface is written against billy, a filesystem abstraction. Teach a bucket to satisfy the billy interface and go-git's native language ("filesystem") suddenly speaks to object storage.
"Oh No, It Works"
The first cut needed exactly one filesystem call added to boot — MkdirAll. Wire the transport to a socket, point it at a bucket, push a repo, and it worked: push, pull, log, blame, tag, the whole thing. No reimplementing git — just shoving a square peg into a round hole until it went in.
In hindsight it makes sense. A bare repo is those four object types on a filesystem. Swap the filesystem for object storage and everything else Just Works, because git's on-disk format is its database schema. Fake open/stat/rename convincingly enough and the façade holds.
The shipped feature set ended up being push/pull over three transports (HTTP, git://, SSH), repos created on first push, Prometheus metrics, and a single Go binary with no local state — even the generated SSH keys live in the bucket. The rest of the story is everything between "oh no, it works" and "actually usable," and that's where the engineering lessons live.
⚠️ Standard disclaimer the author makes and I'll echo: this is an experiment, not vetted for correctness. Do not move your monorepo onto this. The value here is the lessons, not a production recommendation.
Disaster 1: The One POSIX Idiom That Doesn't Translate
Git is paranoid about durability, and its whole strategy is one Unix idiom: write new data to a temp file, then rename(2) it into place once it's correct. POSIX guarantees rename is atomic, so a reader sees either the old file or the new one, never a half-written intermediate. Packfiles land as temp files then get renamed home. Refs are written as locked temp files then renamed over. It's rename all the way down.
Object storage traditionally has no atomic rename. S3's "answer" is to manufacture exactly the intermediate state git tries to avoid: CopyObject to the new key, then DeleteObject on the old one. That's a window where the file exists in both places, or neither — and git's most load-bearing durability idiom falls apart.
⚠️ This is the first thing to internalize if you ever build on object storage: atomic rename is a filesystem guarantee, not an object-storage one. Any design that leans on rename-for-atomicity needs a different strategy on S3-style backends. Some object stores offer a server-side rename extension (the one used here moves metadata server-side in a single round trip, mapping the Unix idiom 1:1), but on vanilla S3 you're stuck with copy-then-delete and have to design around the non-atomicity — usually with conditional writes or a separate consistency mechanism.
A second, sneakier violation hides in the same path: when go-git writes a temp file, it immediately opens it for reading to build the pack index. You can't read and write the same live object in any object store — you're one or the other, never both. The workaround was buffering newly written packfiles in memory. ⚠️ That worked until pushing a large repo (gcc.git) ran the process out of RAM — a preview of the kind of resource cliff you hit when an in-memory shim stands in for a filesystem affordance.
Disaster 2: Death by a Thousand stat() Calls
Pushing the golang/go repo worked but took forever. The metrics showed biblical quantities of HeadObject calls. The git library uses stat() to check whether an object exists, and the flow is: client has object X → check if X exists → check if any pack has X → repeat, endlessly.
On a local filesystem this is fine — stat() resolves in microseconds. Across a network to a remote object-storage region, each one is tens of milliseconds.
It compounded: the SSH transport (and git://, sharing the same code path) was exploding every packfile into loose objects on push. Each loose write cost two round trips — stat() to check existence, then open()/write() to store. A 100,000-object packfile became 200,000 object-storage calls. At ~10ms each, that's over half an hour of waiting, mostly for responses saying "404, nothing here." Caching can't save you — these are writes to 100,000 paths that have never been read and never will be.
The root cause was a genuinely clever deadlock story. The fast path stores an incoming pack whole, copying from the connection until io.EOF. Over HTTP that's fine — the request body ends, EOF arrives. Over git:// and SSH the connection is a persistent socket the client holds open, waiting for the server's status report. EOF never comes. The copy waits forever, the client waits forever — a two-party distributed deadlock. The original workaround disabled the whole-pack path on those transports, falling back to the loose-object writer. Hence the stat storm.
⚠️ The fix is a nice general lesson: stop depending on EOF. Packfiles are self-delimiting — the header declares the object count and a trailing checksum marks the end. So a scanner walks the stream and stops at the trailer while a TeeReader mirrors exactly those bytes into the pack writer. Two clean uploads (packfile + index) instead of a torrent of meaningless round trips. When a protocol gives you explicit length or terminator framing, prefer it over connection-close signaling — the latter breaks the moment the transport is a kept-alive socket.
Disaster 3: Cloning Re-Un-Cached the One Cacheable Workload
With pushing fixed, the read path had its own latency curse. Emulating ReadAt via ranged GetObject requests let the library read individual objects from packfiles — but cloning a trivial 318-object repo with a 200KiB packfile fired over 8,500 GetObject calls before it got killed.
A cloning client reads packfiles thousands of times with random access, walking objects and candidate delta bases repeatedly. On local disk the page cache eats that for breakfast. When every read is an HTTP request, a 200KiB repo becomes dozens of megabytes of round trips, and a 20MiB repo is effectively unservable.
⚠️ The fix leans on a property worth remembering: packfiles are immutable and content-addressed, so pack-<sha>.pack never changes as long as it exists — which makes it trivially cacheable to a faster medium with zero invalidation logic. objgit downloads packs to a local temp folder and serves reads from there, with LRU eviction so the temp folder doesn't blow up. First request is slower; everything after is at filesystem speed. Yes, that reintroduces local disk — but only as a disposable cache, which is a worthwhile trade against clones that never terminate. Content-addressed immutability is the gift that makes aggressive caching safe; lean on it wherever your data model offers it.
Disaster 4: A Flood of ListObjectsV2 That Never Stopped
One disaster remained: every clone triggered a flood of ListObjectsV2 calls that kept going after the clone finished.
Two things compounded. First, when git looks up an unpacked object it probes for a loose object at objects/<xx>/<rest-of-hash>. objgit keeps packs whole, so there are no loose objects, so every probe misses — and each miss across a distinct two-hex prefix triggered a directory listing. There are 256 possible prefixes, so one clone could fire up to 256 listings whose collective answer was "there is nothing here."
Second, and more embarrassing: the listing cache already had an optimization for exactly this — collapsing subtree lookups into recursive scans so one listing answers every probe beneath it. It was completely dead in production. The cache matched recursive prefixes against the repo root (refs/), but every repo is chrooted to its own directory, so real keys look like myrepo.git/refs/heads/main. The prefix check didn't know about chroots, so it never matched. Nobody noticed, because a cache that silently degrades to "no caching" still returns correct answers — just slowly. To twist the knife, a cache warmer was re-listing every useless prefix every 30 seconds for 10 minutes after each clone.
⚠️ The fix was tiny — register each repo's chroot as a recursive subtree root in the cache — but the lesson is large: a broken cache that still returns correct results is invisible until you graph the miss rate. Correctness masks the performance bug. If you have a cache, instrument its hit/miss ratio, or you'll never know it's doing nothing.
The Meta-Lesson
None of these disasters were exotic. They're the things filesystems and kernels hand you for free — atomic rename, microsecond stat(), a page cache that absorbs random reads — and every one of them shattered the moment a network round trip sat at the core. Serving git repositories turns out to be an accidental filesystem-latency benchmark: if your storage abstraction has a weak point, git will find it, and the metrics will show you exactly where.
That's the takeaway for anyone building on object storage, git server or not. Object storage is not a filesystem with a different API. It's a fundamentally different consistency, latency, and atomicity model wearing a filesystem-shaped mask. Every place your code assumed a POSIX guarantee, you now own that guarantee yourself.
Bonus: Sandboxed Post-Receive Hooks
One genuinely nice touch: post-receive hooks run in a locked-down sandbox. Push with hooks enabled and objgit looks for a hook in the tree of the commit you just pushed, then spins up a sandbox with a checkout mounted at /src and writable temp at /tmp. It gets coreutils and nothing else — no host filesystem, no network, no arbitrary binaries. Output streams back as remote: lines like a normal push.
⚠️ That's the right default for running pushed code server-side. Self-hosted git hooks have historically been a soft spot — they run with whatever privileges the git user has, on the host, with full network. A coreutils-only, no-network, no-host-fs sandbox is how server-side hook execution should look. If you run your own git server with post-receive hooks today, that's the bar to hold your own setup to.
Where This Could Go
objgit is an experiment, explicitly tested on a workstation rather than production, with latency to the object store as the big open problem. The interesting direction is CI: wiring hook commands to "apply a Kubernetes object" or "kick off a pipeline run" so CI runs through your own cluster — a self-hosted GitHub Actions equivalent with no disk and no git binary underneath. A web UI is harder, mostly because of the current reality of aggressive scrapers hammering git servers for every scrap of RAM.
A git server with no disk, no git binary, and no database is a genuinely fun proof that git's "database schema is its on-disk format" design is more portable than anyone intended. Whether it ever becomes production-grade is beside the point — the disasters along the way are the education.
