修「token 统计恒 0」+ token 细分(缓存命中 / 推理)
老板 2026-09-28:「ai 网关 token 数统计有问题,一直都是 0;token 是不是可以细分,
命中缓存之类的;能细分就细分」。
【真因】/v1/chat/completions 的 **managed 流式路径**(改写分支,QwenPaw 实际走的就是它)
只把累计的 usage **转发给客户端**,从没写进指标。上游每轮都发 usage(实测 chunk 里带
prompt_tokens 与 prompt_tokens_details.cached_tokens)。
现场取证:网关记了 **663 次调用、tokens 恒为 {prompt: 0, completion: 0}**。
对照:直传路径(messages/responses)与非流式路径本来就有记账 —— 只漏了「流式 + 改写」。
【修复】该路径补 _record_usage_metrics(stream_usage)。
【顺带做 token 细分】(上游 usage 本来就有,此前只记总数):
- cached = prompt_tokens_details.cached_tokens(命中缓存的输入量,已含在 prompt 内)
- reasoning = completion_tokens_details.reasoning_tokens(输出里的思考量,已含在 completion 内)
- Metrics.record_tokens(prompt, completion, cached=0, reasoning=0):全局与按账号两条桶都细分
- 四处记录点全接上;直传流式的扫描器补采 reasoning_tokens
- 新增辅助 _usage_detail()/_record_usage_metrics(),一处判据四路复用
- Prometheus 新增 api_gateway_tokens_cached_total / api_gateway_tokens_reasoning_total(含 HELP/TYPE)
- 前端「指标」页新增两张卡:缓存命中 Token(占输入 x%)、推理 Token(占输出 x%);中英文案对称
【上线实测】
- 瘦请求 → tokens {prompt: 62, completion: 2, cached: 0, reasoning: 1}
- 4k token 同前缀请求两次:第 1 次 cached=0;第 2 次 cached=1920 / prompt=4076
→ 网关累计 cached=1920(输入命中率 22.9%)
- Prometheus 导出同值
【测试】新增两条回归(managed 流式必须记账含细分、细分累加互不干扰),202 → 204 全过;
ci_check.sh 全绿;i18n 中英 759/759 对称。
This commit is contained in:
1 parent
9e2fc8e40a
commit
3b35b14a8a
7 files changed
+201
-9
No files matched your search
+34
-5
@@ -65,6 +65,11 @@ class Metrics:
|
||||
self.providers: dict[str, dict[str, int]] = {} # provider -> {ok, fail}
|
||||
self.tokens_prompt: int = 0
|
||||
self.tokens_completion: int = 0
|
||||
# 第三十四轮: token 细分 —— 上游 usage 里本来就有这两项,之前只记了总数:
|
||||
# cached = prompt_tokens_details.cached_tokens(命中前缀缓存的输入量)
|
||||
# reasoning = completion_tokens_details.reasoning_tokens(输出里的思考量)
|
||||
self.tokens_cached: int = 0
|
||||
self.tokens_reasoning: int = 0
|
||||
self.latency_samples: dict[str, deque[float]] = {} # model -> 最近 N 次耗时
|
||||
# 功能 11: 在途请求计数
|
||||
self.in_flight: int = 0
|
||||
@@ -81,7 +86,7 @@ class Metrics:
|
||||
if acc is None:
|
||||
acc = {
|
||||
"calls": {}, "providers": {},
|
||||
"tokens": {"prompt": 0, "completion": 0},
|
||||
"tokens": {"prompt": 0, "completion": 0, "cached": 0, "reasoning": 0},
|
||||
"latency": {},
|
||||
}
|
||||
self.by_account[account] = acc
|
||||
@@ -128,17 +133,29 @@ class Metrics:
|
||||
self._bucket(acc["providers"], provider)["ok" if ok else "fail"] += 1
|
||||
|
||||
|
||||
def record_tokens(self, prompt: int, completion: int, account: str | None = None) -> None:
|
||||
def record_tokens(self, prompt: int, completion: int, account: str | None = None,
|
||||
cached: int = 0, reasoning: int = 0) -> None:
|
||||
"""记一次调用的 token。
|
||||
|
||||
第三十四轮新增 cached / reasoning 细分(默认 0,老调用点不受影响):
|
||||
`cached` 是输入里命中缓存的部分(**已含在 prompt 里**,不是额外的量),
|
||||
`reasoning` 是输出里的思考部分(**已含在 completion 里**)——两者都只作细分展示,
|
||||
求和时不要与 prompt/completion 相加。
|
||||
"""
|
||||
if not self.enabled:
|
||||
return
|
||||
a = account or ACCOUNT.get()
|
||||
with self._lock:
|
||||
self.tokens_prompt += prompt
|
||||
self.tokens_completion += completion
|
||||
self.tokens_cached += cached
|
||||
self.tokens_reasoning += reasoning
|
||||
if a and a != "anonymous":
|
||||
acc = self._acc(a)
|
||||
acc["tokens"]["prompt"] += prompt
|
||||
acc["tokens"]["completion"] += completion
|
||||
acc["tokens"]["cached"] += cached
|
||||
acc["tokens"]["reasoning"] += reasoning
|
||||
|
||||
def record_latency(self, model: str, seconds: float, account: str | None = None) -> None:
|
||||
if not self.enabled:
|
||||
@@ -177,7 +194,8 @@ class Metrics:
|
||||
return dict(uptime_seconds=round(time.time() - self.started_at, 1),
|
||||
in_flight=self.in_flight, in_flight_peak=self.in_flight_peak,
|
||||
calls={}, providers={},
|
||||
tokens={"prompt": 0, "completion": 0}, latency={})
|
||||
tokens={"prompt": 0, "completion": 0, "cached": 0, "reasoning": 0},
|
||||
latency={})
|
||||
lat = {}
|
||||
for m, dq in acc["latency"].items():
|
||||
if dq:
|
||||
@@ -188,7 +206,7 @@ class Metrics:
|
||||
"in_flight_peak": self.in_flight_peak,
|
||||
"calls": {k: dict(v) for k, v in acc["calls"].items()},
|
||||
"providers": {k: dict(v) for k, v in acc["providers"].items()},
|
||||
"tokens": dict(acc["tokens"]),
|
||||
"tokens": _tokens_out(acc["tokens"]),
|
||||
"latency": lat,
|
||||
}
|
||||
return self._snapshot_global()
|
||||
@@ -207,13 +225,24 @@ class Metrics:
|
||||
"in_flight_peak": self.in_flight_peak,
|
||||
"calls": {k: dict(v) for k, v in self.calls.items()},
|
||||
"providers": {k: dict(v) for k, v in self.providers.items()},
|
||||
"tokens": {"prompt": self.tokens_prompt, "completion": self.tokens_completion},
|
||||
"tokens": {"prompt": self.tokens_prompt, "completion": self.tokens_completion,
|
||||
"cached": self.tokens_cached, "reasoning": self.tokens_reasoning},
|
||||
"latency": latency_out,
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
def _tokens_out(t: dict) -> dict[str, int]:
|
||||
"""token 细分输出的统一形状(老桶可能缺 cached/reasoning,这里补 0)。"""
|
||||
return {
|
||||
"prompt": int(t.get("prompt", 0) or 0),
|
||||
"completion": int(t.get("completion", 0) or 0),
|
||||
"cached": int(t.get("cached", 0) or 0),
|
||||
"reasoning": int(t.get("reasoning", 0) or 0),
|
||||
}
|
||||
|
||||
|
||||
def _latency_summary(dq: deque) -> dict[str, float]:
|
||||
"""把某模型/账号的耗时样本汇总成 avg/p50/p95(nearest-rank)。"""
|
||||
s = sorted(dq)
|
||||
|
||||
Reference in new issue
Block a user