One vocabulary behind many vendors

I counted, at one point, four different names for the same number. How many tokens went into a request. One provider called it input_tokens, another prompt_tokens, a third promptTokenCount, a fourth prompt_eval_count. Same idea, four spellings, and that was before you got to the output side.

Multiply that by every field, across every vendor, and you can feel a codebase starting to rot.

The temptation is to spread the vendor everywhere#

The lazy path is to let each provider’s shape travel through the whole system. Code in the middle checks which provider it is talking to, reaches for whichever field that one happens to use, and moves on. It works. It also means every provider is now wired into every layer, and adding the next one means editing all of them.

I did the opposite. Each provider gets a thin adapter, and the adapter’s job is translation. Whatever the vendor calls its token counts, the adapter maps them to one internal shape: tokensIn and tokensOut, every time. The rest of the system has never heard of prompt_eval_count and never will.

It is the same move for storage. S3, Drive, Dropbox, OneDrive, four SDKs with four personalities, all hidden behind one stream-first port. A factory hands you something that satisfies the port, and the calling code streams bytes without knowing or caring which cloud is on the other end.

A capability interface, not a lowest common denominator#

The hard part is that vendors do not all do the same things. If you flatten to only the features every provider shares, you lose the good ones. If you expose every provider’s quirks, you are back to leaking vendors.

The answer that held up across roughly twenty-five adapters was a capability interface with optional methods. Every adapter satisfies the core contract. Beyond that, a provider only implements what it actually supports. Signature verification is the clean example: some providers can verify a webhook, most cannot, so only the ones that can implement verifyWebhookSignature. The orchestration code checks whether the capability is present, rather than assuming it.

Behind that sits a normalisation boundary. Every inbound payload, whatever its native shape, gets mapped to one internal shape at the edge. So the twenty-five providers speak twenty-five dialects to the adapter, and the adapter speaks one language to everyone else.

Orchestration code should reference the interface and never a concrete provider. Adding the twenty-sixth then means writing one file that satisfies the contract, not editing twenty-five that already do.

The streaming gotcha#

One detail cost me more time than it should have, so here it is for you.

Token usage on a streamed response is not always where you expect it. Some providers report usage only at the very end of the stream, in the final chunk, so if you tear the connection down the moment the text stops arriving, the numbers are simply gone. Others will not send usage at all unless you ask, via an explicit flag on the request that says “include usage”.

If your adapter reads usage the same way for a streamed call as a plain one, it quietly records zeros and you do not notice until the accounting looks wrong. If you bill customers or budget by those token counts, recording zeros means undercharging, blown cost estimates, or both, discovered only when the invoice does not add up. The normaliser has to know that “usage arrives at the end” and “usage is opt-in” are real provider behaviours, and handle them at the boundary, not in the code that thinks it is done.

The whole approach is one idea. Normalise at the edge so the middle stays clean. Let each vendor be as strange as it likes inside its own adapter, map it to one vocabulary on the way in, and keep the orchestration provider-agnostic. Then the next integration is boring, which, for an integration, is exactly what you want.