ENGINEERING NOTES ·IMAGE GENERATION · SELF-HOSTED
Rivalling
Gemini's Images
從零打造 · 對標 Gemini 的地端生圖
Notes from building an image-generation system that reaches for what a frontier model like Gemini does best — feed it a face, a logo and a product, and it composites them into one coherent scene while keeping every identity intact; ask in plain language and it edits rather than repaints; it renders readable text inside the image. All of it self-hosted on consumer GPUs, OpenAI-compatible, and owned end to end. This is how I got there — including the wrong turn I took first.
What “matching Gemini” actually means
“Rival Gemini” is easy to say and slippery to pin down. Plenty of open models make a pretty picture from a prompt — that part is basically solved. The thing that makes a frontier image model feel like a different category is not raw text-to-image. It's four capabilities, and they're the actual target:
Multi-reference compositing
Hand it several images — a person, a brand mark, a product — and get back one scene that fuses them, relit and re-posed, instead of a collage.
Identity preservation
The person still looks like that person; the logo is still the logo. Compositing without identity drift is the hard half.
Instruction editing
“Make it dusk,” “add a sign that says OPEN” — it edits the given image rather than hallucinating a new one from scratch.
Text that reads
Legible words rendered inside the image — a neon sign, a poster headline — not the melted glyph-soup older models produce.
Add the constraints I actually cared about — it must run on hardware I own (consumer 24 GB cards, not a rented cluster), speak an OpenAI-compatible API so anything can call it, and never send a customer's face to someone else's server — and the target is clear. The rest of these notes is how a system reaches those four bars from a bare GPU.
The wrong turn: two models
The obvious first architecture is to use the right tool for each job. So I ran two specialists: a fast turbo text-to-image model — Z-Image-Turbo — for “make a picture from a prompt,” and a separate editing model for “change this picture.” Each was excellent in isolation. Together they were a mistake, for two reasons that only show up once real users touch it.
VRAM thrashing
→ On one 24 GB card only one model fits at a time. Every switch between “generate” and “edit” meant a 20–60s reload. Interactive work died in the swap.
Mis-routing
→ Free-form requests don't announce which model they want. An edit intent would land on the txt2img path and cheerfully throw the user's image away.
Two great models glued together made a worse product than one.
Z-Image earned its retirement honestly — it's a genuinely good turbo generator. But the split itself was the flaw. The lesson pointed straight at how the frontier does it: not a router in front of specialists, but one model that does everything.
The insight: one model, conditioned on 0…N images
A frontier image model isn't a text-to-image model with an editor bolted on. It's a single model that takes a prompt and some number of reference images — zero, one, or several — and produces an image conditioned on all of them. Text-to-image is just the zero-reference case of the same machine. That reframing is the whole method:
The non-obvious part: an editing model can generate from nothing at all. Give it an empty canvas of the target size plus a few-step distilled sampler, and it produces text-to-image output that's as good as the specialist I retired — with noticeably stronger in-image text. So one open-weight instruction-edit model, quantized to fit the card, now covers all three modes. No router. No reload. One thing to load, keep hot, and reason about.
Collapse the modes, don't route between them.
Every switch you can remove is a class of bugs and a chunk of latency you never have to handle. The generate / edit / composite distinction is a detail of how many images you attach — not three different models. Verify the unified path on real prompts first, then delete the specialists.
How identity-preserving compositing works
This is the capability that separates a real system from a demo, and it's worth understanding model-agnostically. The mode that does it — call it multi-edit — is really a routing decision: how each input image is fed into the model. Every reference is read by two encoders at once, and both readings are folded into the conditioning that steers generation:
A vision-language encoder
Reads what is in each image — this is a face, that is a logo, this is a product — as semantics the model can reason about alongside your prompt. The “what.”
A latent (VAE) encoder
Encodes each image's pixels into the latent space the diffusion model works in — the concrete appearance to preserve, not just the description. The “look.”
Because every reference goes through both paths, the model knows both “there is a person” and exactly what that person looks like. That dual reading is why it can relight and recompose while identity survives. Order carries meaning: the first image is the primary subject slot (and, absent an explicit size, sets the output frame); the rest are pure references.
The subtle part is what the starting canvas contributes. Image 1 is scaled and VAE-encoded into a
start latent — but the sampler runs at full denoise (1.0), which discards that latent's
pixels and keeps only its shape. So the composite is not a paint-over of image 1;
every identity — image 1's included — arrives through the conditioning, carried by the two encoders.
That is precisely the routing that makes multi-edit fuse rather than overwrite. Drop denoise
below 1.0 and you get the opposite behaviour: classic img2img that keeps the source pixels.
The VAE is not a free choice. It has to be the one paired with the diffusion backbone — trained jointly, so its latent space is exactly what the sampler expects; bolt on a VAE from a different model family and it decodes to mush or shifted colour. Reach for a modern high-compression image VAE — the current generation packs more latent channels than the old four-channel, SD-era autoencoders — because fine detail and crisp in-image text live or die on how faithfully the VAE round-trips pixels ↔ latent. And it works both ends: the same VAE encodes every reference into the latent that feeds conditioning, and decodes the final latent back to pixels. One loader, used at input and output.
With reference images driving the content through conditioning, the starting canvas only has to decide the output dimensions. So an explicit size can swap in a blank latent of exactly the requested shape without disturbing any reference — the composite still preserves identity, you just control the frame. Omit the size and it inherits the first image's aspect ratio. Same graph, one optional node, fully backward-compatible.
Fitting a frontier idea on a 24 GB card
The capability is one thing; making it run — and stay running — on hardware you can actually buy is the engineering.
Quantize to fit
- The full stack — the diffusion weights, the vision-language text encoder, the VAE — is squeezed with a quantized build so one model fits on a single 24 GB card.
- One model resident, kept hot. No swapping.
A few-step sampler
- A distillation LoRA cuts sampling to a handful of steps — the difference between ~9 s and “too slow to use.”
- It's also what makes editing edit rather than rebuild the image from noise.
Wrap it in an adapter
- A node-graph diffusion runtime (the ComfyUI family) does the actual work; a thin adapter turns its graph into OpenAI-style
/v1/images/*endpoints. - Callers never see the graph — they see an image API.
Convert before it leaves
- The runtime emits PNG; the adapter re-encodes to WebP on the GPU box before anything ships — ~23× smaller for web use.
- Small bytes all the way down the wire.
With two GPU workers behind a load balancer, roughly half of first requests failed with an out-of-memory error — and the retry always succeeded. The tell: it only happened on a cold card. Loading the full model set from scratch hit a peak-VRAM-plus-allocator-fragmentation spike right as the text encoder loaded; the model itself fits fine and a hot worker sat happily at ~19 GB. Switching the allocator to the framework's native caching allocator flattened the spike — 0 OOM across a sustained burst on both cards. A one-line launch flag, host-independent, checked straight into the compose file.
Making the model actually listen
Here's a failure that cost real debugging time. The model kept taking prompts far too literally — asked for a clean product shot, it would scribble menu text, fake UI, and stray watermarks into the frame — and no negative prompt made any difference. It was ignoring “don't.”
The root cause was two coupled defaults baked into the pipeline. The guidance scale was pinned at
1.0, and the negative-prompt input was wired to a dead node that produced nothing. At
a guidance of 1.0 the classifier-free-guidance math collapses to “positive only” — the negative
term cancels out entirely — so even a real negative prompt would have been a no-op. Both had to be
fixed together, and neither was even exposed for a caller to set.
The fix ran end to end: replace the dead negative node with a real encoder, expose both guidance and negative prompt through every layer — workflow, adapter, gateway, client — and set sensible defaults centrally so one change propagates everywhere. Crucially it stays backward-compatible: at the default guidance of 1.0 the new nodes are mathematically inert, so nothing that worked before changes. Turn guidance up and the negatives finally do their job — cleaner plates, no scribbled text.
Guidance is only one dial on the sampler. A node-graph runtime exposes the whole diffusion process as editable inputs, and most “it won't do what I asked” problems are one of these set wrong — so it's worth knowing what each actually controls:
| Parameter | What it controls |
|---|---|
| steps | Denoising iterations. A few-step distillation LoRA tunes this to ~4 — more is not better here; the LoRA expects the low count it was trained for. |
| cfg | Guidance scale — how hard the sampler pulls toward the positive prompt and away from the negative. 1.0 collapses to positive-only (negatives inert); 2–4 makes the prompt and negatives bite. |
| denoise | How much of the start latent is replaced with noise before sampling. 1.0 = start from pure noise, so references arrive through conditioning, not start pixels (edit / composite); <1.0 = keep some of the source (img2img strength). |
| seed | The RNG seed. Fix it to reproduce a result exactly; vary it to explore alternatives from one prompt. |
| sampler / scheduler | The numerical solver and its noise schedule — together they set the sampling trajectory and the speed-vs-character trade-off. |
| latent source | The starting canvas: an empty latent sized to the request (dimensions only), or one VAE-encoded from a source image (dimensions + pixels). |
| positive / negative | The two encoded conditionings — the prompt and references to move toward, and the terms to move away from (only active once cfg > 1). |
A knob you can't set is a knob that's stuck.
The negative prompt “didn't work” not because the idea was wrong but because it was never plumbed out to where anyone could turn it — and a second baked default silently neutralised it even if they had. Expose the controls end to end, wire real defaults in one place, and keep the old path inert so exposing them changes nothing until someone opts in.
Turning a model into a service
A model on a GPU isn't a product. What turns it into one is the boring layer in front: a gateway that sits ahead of several GPU workers and makes them behave like a single, fair, observable API. The shape:
| Concern | What it does |
|---|---|
| Load balance | One image in flight per GPU; healthy workers share the load round-robin; a failing worker is skipped automatically. |
| Fair queue | Not one FIFO but one queue per user, round-robined between them — a heavy user dumping a batch can't bury someone sending three. |
| Sync and async | Block-and-return by default (OpenAI-compatible), or opt into a job id you poll — so long jobs don't hold a connection and you can see your place in line. |
| Honest error codes | The worker's real HTTP status passes straight through instead of being flattened to a generic 502 — the code you see is the layer that actually failed. |
| Small by default | WebP responses; the server keeps no images — results live in memory briefly, then expire. |
| Observability | A live dashboard: per-worker state, queue depth per user, GPU health across every box. |
Inside the private network it's all trusted over VPN — zero tokens, zero setup. For the outside world, the port is never exposed directly; a tunnel dials out and an access layer with service tokens guards the front. The endpoints and request bodies are identical either way, so a cloud function calls it exactly like a laptop on the VPN does.
One deployment lesson worth stating plainly: don't co-locate the image service with the language model. It's tempting — one GPU box, one thing to manage — but the two have opposite appetites. A language model wants a large context resident in memory; image workers want to churn short, bursty GPU jobs. Share a host and they fight for VRAM, and restarting one drags down the other — you've made two unrelated services into a single point of failure. Give the image system its own host (or its own small cluster behind the queue) and each one scales, restarts, and fails on its own.
A standalone image API is an agent other agents can call.
Once it's decoupled and OpenAI-compatible, it stops being “a feature of one app” and becomes a peer on the network — the clean seam for agent-to-agent (A2A) composition. The highest-leverage peer to put in front of it is a prompt-refinement agent: a small language model that takes a user's rough request, rewrites it into the phrasing the image model actually responds to, looks at the result (vision), and iterates — “the sign text is clipped, regenerate wider.” The image model never has to understand loose human intent; a language agent translates and closes the loop. Keeping the hosts separate is exactly what turns that — one agent calling another — from tangled into clean.
Three request shapes, one endpoint family: zero reference images generates, one edits, two-or-three composite. The client sends a normal OpenAI-style image request and gets back base64 — it never needs to know there's a queue, two GPUs, and a graph runtime behind the curtain.
The point
Matching Gemini wasn't one magic model. It was one model, well served.
The frontier capability — multi-reference compositing that preserves identity, editing that edits, text that reads — came not from out-teching a trillion-dollar lab but from three unglamorous moves: collapse three specialists into one instruction-edit model, understand the dual-encoder conditioning well enough to trust it, and wrap it in a serving layer that's fair, observable, and small on the wire. The wrong turns — the two-model thrash, the negative prompt that couldn't fire — taught more than the wins. What you get at the end is something a hosted API can't be: private, owned, and running all day on hardware in your own rack.
Companion to my Local Coding Agent and Harness Engineering notes — same philosophy, pointed at image generation instead of code.