Log these fields per request
- Feature/endpoint tag — which part of the app made the call.
- Model identifier — costs differ per model; don't aggregate blindly.
- Input tokens, output tokens — from the API response's usage object, not estimated.
- Latency and retries — processed retries can add usage; correlate them with cost.
- Timestamp and user/tenant — for anomaly detection and chargeback.
Aggregate into decisions
- Cost per feature per day — a useful budget view.
- P50/P95/P99 tokens per request — averages hide the pathological tail.
- Cost per active user — ties spend to growth so unit economics stay visible.
- Retry rate and its cost share — investigate unexpected repeat calls.
Watch the tail, not the average
Some workloads have an expensive tail of large documents, long outputs, or repeated tool calls. Set a per-request usage threshold and investigate requests above it.
Assumptions and limits
- Implementation details vary by provider SDK; log the usage object the API returns.
- Respect privacy: avoid logging prompt contents where sensitive data may appear — log counts and metadata.
Common questions
Should I log prompt contents or just token counts?
Log counts and metadata by default; avoid storing prompt contents unless you have a reason and a retention policy. Counts answer cost questions; contents create privacy and compliance obligations.
Can I trust the token counts the API returns?
Use the API response's usage object for observed token counts, then reconcile those counts with provider billing. Cached rates, service tiers, tools, and other charges can make the final bill differ from a simple token estimate.
What token metric matters most?
Cost per feature per day and a high-percentile per-request token count are useful diagnostics. Averages alone can hide expensive requests.
Last reviewed: . Developer guides for estimating AI API costs. Provider pricing changes often — always verify current rates with the provider.