diff --git a/docs/LIVE-EXECUTION-REPAIR-PLAN.md b/docs/LIVE-EXECUTION-REPAIR-PLAN.md index 13d86e79..1c0a7955 100644 --- a/docs/LIVE-EXECUTION-REPAIR-PLAN.md +++ b/docs/LIVE-EXECUTION-REPAIR-PLAN.md @@ -1,6 +1,6 @@ # 真实执行可靠性修复:下一切片 -起点为 [真实模型/Purge原样本结果](LIVE-MODEL-PURGE-RESULTS.md)。六例未交付,当前优先修复阻止工程闭环的基础问题;[1.25要求与交付复核](PLAN-1.25.md)和后续产品路线继续保留。A已完成源码修复和两协议16K工程链路验收,见[QA-LIVE-CONTEXT-A](QA-LIVE-CONTEXT-A.md);B已完成所属连接/未知POST/原片段修复,见[QA-LIVE-NETWORK-B](QA-LIVE-NETWORK-B.md)。尚未部署正式,C–E按下表继续,不代表真实模型或Purge收益已经通过。 +起点为 [真实模型/Purge原样本结果](LIVE-MODEL-PURGE-RESULTS.md)。六例未交付,当前优先修复阻止工程闭环的基础问题;[1.25要求与交付复核](PLAN-1.25.md)和后续产品路线继续保留。A已完成源码修复和两协议16K工程链路验收,见[QA-LIVE-CONTEXT-A](QA-LIVE-CONTEXT-A.md);B已完成所属连接/未知POST/原片段修复,见[QA-LIVE-NETWORK-B](QA-LIVE-NETWORK-B.md);C已完成实际Task模型身份、已知用量校准与每次发送前核对,149/149、两协议各五组和两视口通过,见[QA-LIVE-BUDGET-C](QA-LIVE-BUDGET-C.md)。尚未部署正式,D–E按下表继续,不代表真实模型或Purge收益已经通过。 ## 执行顺序 diff --git a/docs/QA-LIVE-BUDGET-C.md b/docs/QA-LIVE-BUDGET-C.md new file mode 100644 index 00000000..3db7d62e --- /dev/null +++ b/docs/QA-LIVE-BUDGET-C.md @@ -0,0 +1,27 @@ +# 真实执行修复 C:模型身份与后续预算 + +已修正验收报告冒用启动时模型名、已知用量未校准后续预留,以及错误响应超额后的停止状态。此次为源码切片,未部署正式 1.24、未改原 Provider 或 Key;本轮真实模型请求为 0。真实自主交付和原版 Purge 收益仍需后续独立实测。 + +## 实际行为 + +- 验收报告从所属 Task 的 Run 消费记录读取实际请求模型、协议、连接和每次尝试。混合模型、零请求、消费缺失、最近 Run 投影覆盖不足、最终刷新失败分别标明;启动配置只保留为计划值。 +- 每次真实 HTTP 记录原线体的本地估算及 SHA。仅同 Task、模型、协议、连接和 endpoint 的已报告 usage 参与校准;后续预留取输入比例、已知输出和总量下限的保守最大值,不降低本地估算、不回写旧消费、不把未知当零。 +- 规划与每次重试、切池、原生实际发送前都核对剩余预算。首次 45,100 已报告、70,000 预算时,剩余 24,900 已低于校准预留,legacy/池均只发一次并暂停;保留原 HTTP 503。 +- 单个在途响应仍可能超出估算。55,017 已报告、35,000 运行预算的场景保留完整用量和 20,017 超出,停止后续请求;单连接、池、legacy 一致。原错误、请求 ID 和原响应仍可核对,未知请求继续单列本地预留。 +- 原生外层工具回传属于协调步骤,不重复预留第二次模型 HTTP;内层实际请求独立校准。完成的读取结果可以先正常回传,下一 HTTP 预算不足时停止。新增原生请求使用版本化预留核验,旧工件沿原契约核验。 + +## 验收 + +最终 425 份源码输入冻结后,15 文件相关回归 **149/149**、TypeScript/桌面/手机前端/服务端构建通过。Chat 与 Responses 的真实 Harness 0.2.1-rc.2、固定 16K、自有 HTTP 各 **5 组**工程链路通过:保留原九项失败、读取分页日志、修复三模块并形成完整审查批次、父项目九项测试和构建、实际 Git 及原暂存/未暂存保护。该结果证明代理链路,不证明模型自主推理。 + +实际编译宿主的 Edge **1440/390px 两组**通过并已目视回看原图:用量/预算/超出/未知分别呈现、原 HTTP 503 保留、续接入口可达、无页面横向溢出。续接预览、关闭和刷新不增加 HTTP、不替换原 Task、不丢原消费。全部自有请求与正式配置分离。 + +A、B 和原真实六例的只读离线核验均通过,原 manifest SHA 分别保持 `f0bfd70ba570ed48391511a9279e8857b3f1ab9c3319d738d355d1b03548b9f5`、`df97b53c80204f7f0e6b352f707ff2b345d7360ecdd0e08ac4e00c86f2462c7b`、`100fb4f89ac7d78a61f8b9d6324c0cae488e510eca0dd7198dde6cfd9d079b04`。原 20 请求/419,597 已报告 tokens/一次未知、六例未交付保持,不改写为成功。 + +## 原失败与回顾 + +开发轮 147 项中 146 通过/1 失败、原生 6/7 和诊断失败、三次 UI 失败均保留。UI 最后一次是验证器 label 不匹配实际控件;前两次暴露单连接超额状态。原生失败暴露外层工具协调被重复预留,已修正。首冻结 147/147 通过后,审查又补上“尚未超额但剩余不足”重试边界,重新冻结并执行上述最终 149 项,未借用旧轮通过。 + +Responses 首轮四个响应后发生一次未知/reset,工作台只记一次未知且未自动重放;原本机夹具在 catch 直接断开,没有保存内部原因。补充只收集错误的夹具诊断后独立五组复验通过,最终源码的五组也通过;首次断开原因仍未知,不据未复现推断已找到根因。原结果、消费和工件保留为失败样本。 + +证据按原字节封存于 [manifest](live-budget-c-evidence/manifest.json),可用 `node scripts/verify-model-budget-evidence.mjs` 只读核对。下一 D 固定原版插件加载闭包、隔离配置与权限边界,先无真实 Key 验收,再进入 E 的新真实批次;本地 Git 收尾,不远端推送。 diff --git a/findings.md b/findings.md index ab5912ac..4c6cfe45 100644 --- a/findings.md +++ b/findings.md @@ -914,3 +914,9 @@ I轮实际Edge/Electron各7组、手机真实配对TLS网关观察2组,共16 原UND_ERR_CONNECT_TIMEOUT属于建连阶段。Provider现在拥有固定Undici7.30.1 dispatcher,双栈候选在请求内选择,不更改全局DNS或关闭IPv6;代理连接自身也使用同一候选/超时配置,环境代理与NO_PROXY只输出存在位。POST的ECONNRESET或已发送后等待超时不能证明上游未处理,应保存unknown、禁止自动重试/切池。只有完整、有界、全部明确预发送的错误链可以安全重试,截断/环形/含未知叶节点均不能据已知部分推断。 诊断旧测试9/12原失败保持在.qa/live-network-diagnostics.tap;修正的是“reset可安全重试”的过时假设。新增显式续接不重放已批准命令断言,明确ECONNREFUSED继续覆盖重试耗尽/退避停止,12/12通过。HTTP正文中途断裂现保存64KiB以内原片段/请求ID,完整性标为interrupted,不把缺失usage填零。 + +C新增发现:实际usage必须绑定实际线体本地估算和同Task/模型/协议/连接/endpoint的消费记录,不能用bootstrap模型。已报告用量只增校准后续预留,未知不参与校准也不当零。单连接最后一次HTTP错误没有下一次路由迭代,因此还需在原响应记账后的异常分支核对预算,暂停同时保留原诊断。原生外层是工具回传协调,不是第二次模型HTTP;对其重复套入内层计费比例会阻止原结果回传,校准门控应只作用于实际模型请求。 + +C实际渲染回看:1440/390px原截图均显示55,017上游报告、35,000运行预算、20,017记账超出及未知0/0,原HTTP503仍在预算面板;窄屏完整换行、续接入口可达,无页面横向溢出。真实编译宿主的两视口续接预览/关闭/刷新均不增加HTTP、不替换原Task。验证器第三次原失败仅为label“新Run预算tokens”不匹配实际“新Run累计tokens”;修正选择器后独立UUID两组通过,产品字节未变。固定16K的Chat原生三模块/原九项失败与后续修复/父审查/Git五组已通过;Responses和最终相关回归仍在运行。本轮恢复期间一次批量读取及通配输出过大,后续精确限定文件和短段;一次误写修复计划文件名不存在,已按实际LIVE-EXECUTION-REPAIR-PLAN.md读取,不计作验证。 + +C后续审查:首冻结147/147通过后,Responses独立首轮4响应+1unknown/reset,未自动重发,原失败保留;原夹具catch直接destroy且未保存原因。增加只收集error的诊断后新UUID五组通过,不能把未复现断开归因于已知断言。另修正每次重试/切池前的预留核对:45,100已报告/70,000预算时虽尚未超额,剩余24,900已低于校准预留,legacy和池都必须只发一次并保留503。最终425输入重新冻结,首轮147不算新来源149通过。D只读确认原插件默认autoRetry=true/retryMax3/hooksDenyBypass=true;完整原patch另含身份与redteam组件,必须隔离配置/加载闭包,不能直接改正式宿主。 diff --git a/progress.md b/progress.md index a184efeb..b9ac027e 100644 --- a/progress.md +++ b/progress.md @@ -1396,3 +1396,7 @@ A独立源码12批合计716/716、0失败/取消/跳过/待办;实际Electron # 真实执行 B 接续:网络请求归属与重放边界 继续沿用planning-with-files和原修复方案;未启动子代理,未操作正式宿主或Key,未发起真实模型请求。A原实现与证据保持。B相关94/94、构建、原engine-worker14/14和最终编译宿主2/2通过;原诊断9/12、编译冷却误等待14/16及0/2均保留。实现e73bca9;18份/228436字节证据SHA df97b53c80204f7f0e6b352f707ff2b345d7360ecdd0e08ac4e00c86f2462c7b离线核对通过,A原封存保持。C身份纯函数4/4后已接入实际验收器,最终停止/刷新后以Task消费为准;预算校准仍待实施。工具恢复时一次多文件读取输出截断,随后限定短段,截断内容不作完整审查结论。 + +C最终接续:预算专项7/7、最终编译通过;新UI独立目录model-budget-ui-7c0de90d-1afd-4165-911e-e949b60b7d30两视口通过并已目视回看原图。原label失败保留,不改产品。Chat16K独立目录controlled-context-fe746aa6-377a-40be-945d-1a849ad675e2五组通过;Responses和最终15文件相关回归运行中。B/原六例离线核验已通过,A离线核验运行中。未使用真实Key、未改正式宿主、未加载原版Purge。 + +C最终结论:重试前剩余预算不足的两项回归补齐后,新425输入冻结保持,15文件149/149及编译通过;Chat controlled-context-417c46cd-bd9a-4600-a2cb-33a028b5568d、Responses controlled-context-6471c96c-cab2-4d8c-85fe-ec53876cdccc各5组;UI model-budget-ui-01f13fac-4234-4ea9-89f3-02225857cf31两组并已目视两张最终原图通过。此前147/147及原Responses未知失败、诊断复验均独立保留,不混作最终来源。A/B/原六例离线通过;真实请求0、正式宿主未动。D只读预研中两次猜测桥文件名不存在,随后由rg实际清单定位controlled-plugin/controlled-installation,未执行原插件。 diff --git a/scripts/live-model-observation.mjs b/scripts/live-model-observation.mjs new file mode 100644 index 00000000..156d9623 --- /dev/null +++ b/scripts/live-model-observation.mjs @@ -0,0 +1,42 @@ +// This reports observed Task consumption, never the model selected in bootstrap. +// Public execution views may contain only the most recent Runs; gaps stay explicit. +export function observedTaskModels(state,taskId=state?.execution?.task?.id){ + const execution=state?.execution??{},runs=new Map(),attempts=new Map(),issues=[]; + for(const run of [...(execution.runs??[]),execution.run].filter(Boolean)){ + if(taskId&&run.taskId===taskId&&typeof run.id==='string')runs.set(run.id,run); + } + const task=execution.task?.id===taskId?execution.task:null; + const validCount=value=>Number.isSafeInteger(value)&&value>=0; + let declared=task&&validCount(task.usage?.requests)?task.usage.requests:null; + if(declared===null&&runs.size&&[...runs.values()].every(run=>validCount(run.usage?.requests))) + declared=[...runs.values()].reduce((total,run)=>total+run.usage.requests,0); + for(const run of runs.values())for(const receipt of run.consumption??[]){ + if(typeof receipt.attemptId!=='string'||!receipt.attemptId){issues.push({kind:'missing_attempt_id',runId:run.id});continue;} + const attempt={runId:run.id,attemptId:receipt.attemptId,requestId:receipt.requestId??null, + model:typeof receipt.model==='string'&&receipt.model.trim()?receipt.model:null, + protocol:typeof receipt.protocol==='string'&&receipt.protocol.trim()?receipt.protocol:null, + connectionId:receipt.connectionId??null,outcome:receipt.outcome??null, + usageStatus:receipt.usageStatus??'unknown',usage:receipt.usage??null}; + const previous=attempts.get(attempt.attemptId); + if(previous){ + if(JSON.stringify(previous)!==JSON.stringify(attempt))issues.push({kind:'conflicting_attempt',attemptId:attempt.attemptId,runIds:[previous.runId,run.id]}); + continue; + } + attempts.set(attempt.attemptId,attempt); + if(!attempt.model||!attempt.protocol)issues.push({kind:'missing_route_identity',attemptId:attempt.attemptId}); + if(attempt.usageStatus==='reported'&&!attempt.usage)issues.push({kind:'missing_reported_usage',attemptId:attempt.attemptId}); + } + const observed=[...attempts.values()],models=[...new Set(observed.map(value=>value.model).filter(Boolean))],routes=new Map(); + for(const attempt of observed){ + const identity={connectionId:attempt.connectionId,model:attempt.model,protocol:attempt.protocol},key=JSON.stringify(identity); + if(!routes.has(key))routes.set(key,{...identity,requests:0,reportedUsageRequests:0,unknownUsageRequests:0}); + const route=routes.get(key);route.requests++; + if(attempt.usageStatus==='reported'&&attempt.usage)route.reportedUsageRequests++;else route.unknownUsageRequests++; + } + if(!taskId)issues.push({kind:'missing_task_identity'}); + if(declared!==null&&declared!==observed.length)issues.push({kind:'request_coverage_gap',recordedRequests:declared,observedAttempts:observed.length}); + const status=issues.length?'incomplete':!observed.length?declared===0?'not_observed':'unavailable':models.length>1?'mixed':'observed'; + return {schemaVersion:1,source:'task_run_consumption',taskId:taskId??null,status, + model:['observed','mixed'].includes(status)&&models.length===1?models[0]:null,models, + observedAttempts:observed.length,recordedRequests:declared,routes:[...routes.values()],attempts:observed,issues}; +} diff --git a/scripts/verify-controlled-projects.mjs b/scripts/verify-controlled-projects.mjs index 12b0aec8..b81f1d9d 100644 --- a/scripts/verify-controlled-projects.mjs +++ b/scripts/verify-controlled-projects.mjs @@ -52,7 +52,7 @@ const frozen = inputs(), mark = message => { cases.push(message); console.log('P async function runSample(kind, { sampleName = kind, strategy = null, comparison = null } = {}) { const startedAt = Date.now(), label = sampleName; const disposers = [], t = { after: value => disposers.push(value) }, f = fixture(t), sample = path.join(directory, sampleName); - fs.mkdirSync(sample); let runtime, group, result, desktopQa, cycledRequest = false, cycledApproval = false; const reviews = [], modelRequests = []; + fs.mkdirSync(sample); let runtime, group, result, desktopQa, model, cycledRequest = false, cycledApproval = false; const reviews = [], modelRequests = []; try { const baseline = contextMode ? 'docs/live-model-purge-1.24-evidence/source-baselines/r1-off' : kind === 'react' ? 'benchmarks/react-delivery-v1/baseline' : 'benchmarks/workers-delivery-v1/' + kind; fs.cpSync(path.resolve(baseline), f.projectRoot, { recursive: true, filter: file => !file.split(path.sep).includes('node_modules') }); @@ -74,7 +74,7 @@ async function runSample(kind, { sampleName = kind, strategy = null, comparison }; if (kind === 'react') install(f.projectRoot, 'parent'); let phase = 'new'; - const model = await (responsesMode ? responsesEngineeringHttp : compatibleFixture)(t, async (route, input) => { + model = await (responsesMode ? responsesEngineeringHttp : compatibleFixture)(t, async (route, input) => { assert.equal(route, '/v1/chat/completions'); assert.equal(input.model, 'owned-controlled-engineering'); assertToolPairs(input.messages); const worker = [...f.store.core.worker.values()][0], session = f.store.session(worker.sessionId); @@ -237,7 +237,7 @@ async function runSample(kind, { sampleName = kind, strategy = null, comparison child: f.store.core.worker.get(child.id), engineRuns, checks: [...f.store.core.check_job.values()], parentCheck: check, browserJobs, commit: delivery.commit, writable, reviews, modelRequests, ...(desktopQa ? { desktop: { events: desktopQa.events, screenshots: desktopQa.screenshots, scope: desktopQa.scope, errors: desktopQa.errors } } : {}), protectedFiles: Object.fromEntries(Object.entries(protectedFiles).map(([file, bytes]) => [file, hash(bytes)])), realModelRequests: 0 }; return result; - } catch (error) { await desktopQa?.failure(error).catch(() => {}); result = { success: false, version, kind, sampleName, comparison, error: error.stack, group, reviews, modelRequests, engineRuns: [...f.store.core.engine_run.values()], checks: [...f.store.core.check_job.values()], realModelRequests: 0 }; throw error; } + } catch (error) { await desktopQa?.failure(error).catch(() => {}); result = { success: false, version, kind, sampleName, comparison, error: error.stack, group, reviews, modelRequests, ownedFixtureErrors:model?.errors??[], engineRuns: [...f.store.core.engine_run.values()], checks: [...f.store.core.check_job.values()], realModelRequests: 0 }; throw error; } finally { await desktopQa?.close(); await runtime?.stopAll(); f.store.storage.database.exec('PRAGMA wal_checkpoint(TRUNCATE)'); diff --git a/scripts/verify-live-engineering.mjs b/scripts/verify-live-engineering.mjs index 2524fd98..ce9764be 100644 --- a/scripts/verify-live-engineering.mjs +++ b/scripts/verify-live-engineering.mjs @@ -10,6 +10,7 @@ import { createPythonDeliveryProject, pythonProtected } from '../tests/fixtures/ import { deliveryGit, writeDeliveryFile } from '../tests/fixtures/delivery-project.mjs'; import { captureWorkspaceSource, assertWorkspaceSource } from '../tests/fixtures/workspace-project.mjs'; import { verifyLiveReactClient } from './live-react-client.mjs'; +import { observedTaskModels } from './live-model-observation.mjs'; if (process.argv.includes('--help')) { console.log('node scripts/verify-live-engineering.mjs --case react|python [--url http://127.0.0.1:4318] — fresh owned Git source/workspace, configured model and bounded review/check/delivery acceptance'); process.exit(0); @@ -149,7 +150,7 @@ try { fs.writeFileSync(path.join(root, 'delivery-report.json'), reportText); fs.writeFileSync(path.join(root, 'delivery-report.md'), markdown); const browser = kind === 'react' ? await verifyLiveReactClient(childProject.root, path.join(root, 'screenshots')) : null; fs.writeFileSync(path.join(root, 'session.json'), JSON.stringify(await api(route + '/export'), null, 2) + '\n'); - result = { success: true, version: bootstrap.version, testedAt: new Date().toISOString(), kind, model: bootstrap.config.model, root, + result = { success: true, version: bootstrap.version, testedAt: new Date().toISOString(), kind, root, taskId, sessionId: childSession.id, projectId: childProject.id, sourceProjectId: parentProject.id, workspaceId: workspace.id, workspaceRoot: childProject.root, sourceCommit: workspace.source.head, commit: completedDelivery.commit, tests, checks: passing.results, usage: state.execution.task.usage, elapsedSeconds: Math.round((Date.now() - startedAt) / 1000), applications, decisions, continuations, browser, @@ -158,11 +159,20 @@ try { credentialsHandling: 'The original Provider executes requests in its own process; no model key is read, copied or written.' }; succeeded = true; } catch (error) { - result = { success: false, version: bootstrap.version, kind, testedAt: new Date().toISOString(), root, model: bootstrap.config.model, + result = { success: false, version: bootstrap.version, kind, testedAt: new Date().toISOString(), root, taskId, sessionId: childSession?.id, projectId: childProject?.id, workspaceId: workspace?.id, error: error.stack, elapsedSeconds: Math.round((Date.now() - startedAt) / 1000), usage: state?.execution?.task?.usage, state: state?.status, stopReason: state?.execution?.run?.stopReason, applications, decisions, continuations, checks: state?.execution?.checks }; process.exitCode = 1; -} finally { if (!succeeded && route) await api(route + '/stop', {}).catch(() => {}); } +} finally { + if (!succeeded && route) await api(route + '/stop', {}).catch(() => {}); + let refreshError=null; + if(route){try{state=await api(route);taskId??=state.execution?.task?.id;}catch(error){refreshError=error.message;}} + const observation=observedTaskModels(state,taskId); + if(refreshError){observation.status='incomplete';observation.model=null;observation.issues.push({kind:'final_state_refresh_failed',message:refreshError});} + result.model=observation.model;result.modelObservation=observation;result.plannedModel=bootstrap.config.model; + if(state?.execution?.task?.id===taskId)result.usage=state.execution.task.usage; +} fs.writeFileSync(path.join(root, 'result.json'), JSON.stringify(result, null, 2) + '\n'); fs.writeFileSync(path.resolve('docs', 'live-' + kind + '-engineering-validation.json'), JSON.stringify(result, null, 2) + '\n'); -console.log(JSON.stringify({ success: result.success, kind, root, taskId, commit: result.commit, tests: result.tests, usage: result.usage, error: result.error })); +console.log(JSON.stringify({ success: result.success, kind, root, taskId, model:result.model,modelObservation:result.modelObservation.status, + commit: result.commit, tests: result.tests, usage: result.usage, error: result.error })); diff --git a/scripts/verify-model-budget-evidence.mjs b/scripts/verify-model-budget-evidence.mjs new file mode 100644 index 00000000..8c3e23a0 --- /dev/null +++ b/scripts/verify-model-budget-evidence.mjs @@ -0,0 +1,61 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import {createHash} from 'node:crypto'; +import {execFileSync} from 'node:child_process'; +import {originalEngineRun,verifyEngineModelInput} from '../src/shared/contracts/engine-execution.mjs'; +import {observedTaskModels} from './live-model-observation.mjs'; + +// Offline only: no Provider, Runtime, credentials, model request or formal data. +const root=path.resolve('docs/live-budget-c-evidence'),sha=value=>createHash('sha256').update(value).digest('hex'); +const inside=relative=>{const file=path.resolve(root,relative);assert.ok(file.startsWith(root+path.sep));return file;}; +const raw=fs.readFileSync(inside('manifest.json')),manifest=JSON.parse(raw),read=relative=>JSON.parse(fs.readFileSync(inside(relative),'utf8')); +assert.equal(manifest.schemaVersion,1);assert.equal(manifest.realModelRequests,0);assert.equal(manifest.formalServiceChanged,false); +const entries=new Map();let bytes=0; +for(const entry of manifest.entries){assert.ok(!entries.has(entry.path));entries.set(entry.path,entry);const data=fs.readFileSync(inside(entry.path)); + assert.equal(data.length,entry.bytes,entry.path);assert.equal(sha(data),entry.sha256,entry.path);bytes+=data.length;} +assert.equal(entries.size,manifest.count);assert.equal(bytes,manifest.bytes); +for(const input of manifest.sourceInputs){ + assert.match(input.path,/^(?:src|scripts|tests)\/[A-Za-z0-9_./-]+$|^package(?:-lock)?\.json$/);assert.ok(!input.path.split('/').includes('..')); + const blob=execFileSync('git',['show',manifest.sourceCommit+':'+input.path],{windowsHide:true,maxBuffer:16*1024*1024}); + const data=input.snapshot?fs.readFileSync(inside(input.snapshot)):blob; + assert.equal(sha(blob),input.gitBlobSha256);assert.equal(sha(data),input.sha256);assert.equal(data.length,input.bytes); + assert.equal(data.toString('utf8').replaceAll('\r\n','\n'),blob.toString('utf8').replaceAll('\r\n','\n')); +} +const receipt=name=>fs.readFileSync(inside(manifest.receipts.find(value=>value.name===name).file),'utf8'); +assert.match(receipt('final-related'),/tests 149\b/);assert.match(receipt('final-related'),/pass 149\b/);assert.match(receipt('final-related'),/fail 0\b/); +assert.match(receipt('final-build'),/Server bundle built\./); +const ui=read(manifest.ui.result),session=read(manifest.ui.session); +assert.equal(ui.passed,true);assert.equal(ui.cases.length,2);assert.equal(ui.ownedHttpRequests,1);assert.deepEqual(ui.errors,[]); +assert.equal(session.status,'paused');assert.equal(session.execution.run.stopReason,'token_limit'); +assert.equal(session.execution.run.budgetPause.tokenAccounting.run.reportedTokens,55017); +assert.equal(session.execution.run.budgetPause.tokenAccounting.run.exceededTokens,20017); +assert.equal(session.execution.run.budgetPause.tokenAccounting.run.unknownRequests,0); +assert.equal(session.execution.run.budgetPause.modelFailure.httpStatus,503); +assert.equal(observedTaskModels(session).model,'owned-budget-ui'); +let originalRequests=0,artifactRecords=0; +for(const sample of manifest.samples){ + const result=read(sample.result),records=read(sample.records),sessions=new Map(read(sample.sessions).map(value=>[value.id,value])),core={}; + for(const row of records){const value=JSON.parse(row.payload);(core[row.kind]??=new Map()).set(value.id,value);} + artifactRecords+=core.artifact.size;const artifacts=new Map(sample.artifacts.map(value=>[value.id,value])); + const store={core,owned(kind,id,sessionId){const value=core[kind]?.get(id);assert.equal(value?.sessionId,sessionId);return value;}, + artifactStore:{read(artifact){const data=fs.readFileSync(inside(artifacts.get(artifact.id).file));assert.equal(sha(data),artifact.sha256);assert.equal(data.length,artifact.bytes);return data.toString('utf8');}}, + savedToolResult(sessionId,callId){const tool=[...core.tool.values()].find(value=>value.sessionId===sessionId&&value.callId===callId);return JSON.parse(this.artifactStore.read(core.artifact.get(tool.resultArtifactId)));}}; + assert.equal(result.success,sample.passed);assert.equal(result.realModelRequests,0); + if(!sample.passed){assert.ok(result.error);continue;} + assert.equal(result.metrics.contextLimit,16000);assert.equal(result.parentCheck.results[0].tests.passed,9); + assert.ok(result.checks.some(job=>job.status==='failed'&&job.results[0].tests.failed===9));assert.equal(result.writable.length,3); + for(const job of core.engine_run.values()){ + const owner=sessions.get(job.sessionId);originalEngineRun(store,owner,job); + assert.equal(job.status,'completed');assert.equal(job.native.cleanup.treeConfirmed,true);assert.equal(job.native.cleanup.rootExited,true); + for(const request of job.requests){ + verifyEngineModelInput(store,owner,job,request);assert.equal(request.reservationVersion,1);originalRequests++; + assert.ok(request.tokenReservation.routes.every(route=>route.inputRatio>=1)); + } + } +} +assert.deepEqual(manifest.samples.filter(value=>value.passed).map(value=>value.name).sort(),['chat','responses']); +for(const name of ['a-offline','b-offline','pilot-offline'])assert.equal(JSON.parse(receipt(name)).passed,true); +console.log(JSON.stringify({passed:true,sourceCommit:manifest.sourceCommit,manifestSha256:sha(raw),files:entries.size,bytes, + sourceInputs:manifest.sourceInputs.length,artifactRecords,originalRequests,realModelRequests:0, + scope:'Offline raw bytes/source, known usage reservation, actual Task model identity and owned native engineering evidence; original failures retained, real model and original Purge not measured.'})); diff --git a/scripts/verify-model-budget-ui.mjs b/scripts/verify-model-budget-ui.mjs new file mode 100644 index 00000000..8c5afe68 --- /dev/null +++ b/scripts/verify-model-budget-ui.mjs @@ -0,0 +1,56 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import {randomUUID} from 'node:crypto'; +import {chromium,expect} from 'playwright/test'; +import {fixture,compatibleFixture,launchWorkbench,until} from '../tests/helpers.mjs'; +import {observedTaskModels} from './live-model-observation.mjs'; + +const directory=path.resolve('.qa','model-budget-ui-'+randomUUID());fs.mkdirSync(directory,{recursive:true}); +const disposers=[],t={after:fn=>disposers.push(fn)},f=fixture(t),screenshots=[],cases=[],errors=[]; +let host,browser,report,state; +try{ + const model=await compatibleFixture(t,()=>({status:503,body:{error:{message:'Owned UI budget failure'}, + usage:{prompt_tokens:55000,completion_tokens:17,total_tokens:55017}}})); + f.store.close();host=await launchWorkbench(t,f.data);assert.notEqual(new URL(host.url).port,'4318'); + await host.api('/config',{...host.bootstrap.config,contextLimit:16000,maxOutputTokens:512,maxRunTokens:35000,maxTaskTokens:1000000}); + const connection=await host.api('/model-center/connections',{name:'自有预算验收模型',model:'owned-budget-ui',baseUrl:model.baseUrl,requiresKey:false}); + await host.api('/model-center/select',{type:'connection',id:connection.id}); + const route='/sessions/'+f.session.id; + await host.api(route+'/messages',{content:'读取README,不修改文件,不运行检查。'}); + await until(async()=>{state=await host.api(route);return !['running','awaiting_approval','stopping'].includes(state.status);},15000); + assert.equal(state.status,'paused',JSON.stringify({status:state.status,failure:state.failure?.message,stopReason:state.execution.run?.stopReason,usage:state.execution.task?.usage,budget:state.execution.run?.budget})); + assert.equal(state.execution.run.stopReason,'token_limit');assert.equal(model.requests.length,1); + assert.equal(observedTaskModels(state).model,'owned-budget-ui'); + browser=await chromium.launch({channel:'msedge',headless:true}); + for(const width of [1440,390]){ + const page=await browser.newPage({viewport:{width,height:1000}});page.on('pageerror',error=>errors.push(error.message)); + await page.goto(host.url);await page.waitForLoadState('networkidle'); + const project=page.getByLabel('当前项目',{exact:true}); + if(!await project.isVisible())await page.getByRole('button',{name:'展开项目栏',exact:true}).click(); + await project.selectOption(f.project.id); + const panel=page.getByLabel('任务预算诊断',{exact:true});await expect(panel).toBeVisible(); + for(const text of ['本次运行上游报告用量','55,017','35,000','20,017','用量未知请求 / 本地预留','0 / 0','单个在途响应可能超过预估'])await expect(panel).toContainText(text); + await panel.getByRole('button',{name:'核对预算并续接原任务',exact:true}).click(); + const dialog=page.getByRole('dialog',{name:'续接原任务',exact:true}); + await dialog.getByLabel('新Run累计tokens',{exact:true}).fill('100000'); + await dialog.getByRole('button',{name:'准备续接预览',exact:true}).click();await expect(dialog.getByRole('status')).toContainText('将创建原Task的新Run'); + assert.equal(model.requests.length,1,'A budget preview cannot dispatch another model request.'); + const box=await dialog.boundingBox();assert.ok(box.x>=0&&box.x+box.width<=width+1); + await dialog.getByRole('button',{name:'关闭窗口',exact:true}).click();await page.reload();await expect(panel).toContainText('20,017'); + const current=await host.api(route);assert.equal(current.execution.task.id,state.execution.task.id);assert.equal(current.execution.task.usage.requests,1); + assert.equal(current.execution.run.consumption.length,1);assert.equal(model.requests.length,1); + assert.ok(await page.evaluate(()=>document.documentElement.scrollWidth<=innerWidth+1),'No viewport overflow'); + await panel.scrollIntoViewIfNeeded();const screenshot=path.join(directory,'reported-overrun-'+width+'.png'); + await page.screenshot({path:screenshot,fullPage:true});screenshots.push(screenshot); + fs.writeFileSync(path.join(directory,'reported-overrun-'+width+'.html'),await page.content()); + cases.push(width+': actual reported overrun persists, zero unknown use stays distinct, and preview does not dispatch or replace the original Task');await page.close(); + } + assert.deepEqual(errors,[]);fs.writeFileSync(path.join(directory,'session.json'),JSON.stringify(await host.api(route),null,2)); + report={passed:true,cases,screenshots,errors,realModelRequests:0,ownedHttpRequests:model.requests.length, + modelObservation:observedTaskModels(state),scope:'Actual compiled host and owned HTTP reported overrun, persistence and preview UI; real-model autonomy/Purge benefit remain separate.'}; +}catch(error){if(state)fs.writeFileSync(path.join(directory,'failed-state.json'),JSON.stringify(state,null,2));report={passed:false,cases,screenshots,errors,error:error.stack,realModelRequests:0};process.exitCode=1;} +finally{ + await browser?.close();await host?.close();for(const dispose of disposers.reverse())await dispose(); + fs.writeFileSync(path.join(directory,'result.json'),JSON.stringify(report,null,2));console.log(JSON.stringify({...report,directory})); +} diff --git a/src/client/BudgetPausePanel.tsx b/src/client/BudgetPausePanel.tsx index 5ba4e93c..1f88f253 100644 --- a/src/client/BudgetPausePanel.tsx +++ b/src/client/BudgetPausePanel.tsx @@ -17,6 +17,7 @@ export function BudgetPausePanel({session,config,onSession,onRecovery}:{session: const [selection,setSelection]=useState(''),[options,setOptions]=useState<{value:string;name:string}[]>([]); const run=session.execution?.run,pause=run?.budgetPause,reason=pause?.reason??run?.stopReason??''; const controlled=session.execution?.task?.origin==='isolated-worker'||run?.engine==='deepseek-workbench'; + const calibrationSources=new Set(pause?.tokenReservation?.routes.flatMap(route=>route.sources.map(source=>source.attemptId))??[]).size; if(!['paused','failed'].includes(session.status)||!reasons[reason])return null; async function start(){setOpen(true);setError('');setPlan(null);try{const center=await api<{selected:{type:string;id:string};connections:{id:string;name:string;contextLimit:number}[];pools:{id:string;name:string}[]}>('/model-center');setSelection(center.selected.type+':'+center.selected.id);setOptions([...center.connections.map(row=>({value:(row.id==='legacy-default'?'legacy':'connection')+':'+row.id,name:row.name+' · '+row.contextLimit.toLocaleString()+' 上下文'})),...center.pools.map(row=>({value:'pool:'+row.id,name:row.name+' · 按成员最小容量'}))]);}catch(cause){setError((cause as Error).message);}} async function preview(){setBusy(true);setError('');setPlan(null);try{const [type,id]=selection.split(':');const value=await api('/sessions/'+session.id+'/resume-preview',{...fields,...(selection?{selection:{type,id}}:{})});setPlan(value);onSession(await api('/sessions/'+session.id));}catch(cause){setError((cause as Error).message);}finally{setBusy(false);}} @@ -25,6 +26,9 @@ export function BudgetPausePanel({session,config,onSession,onRecovery}:{session: {pause?.originalInputTokens!==undefined&&

原生完整历史估算 {pause.originalInputTokens.toLocaleString()} tokens。原始输入、历史索引与拒绝正文已保存为任务工件,本次拒绝没有发送模型请求。

} {!!pause?.transportReservationTokens&&

准备原生任务时另留 {pause.transportReservationTokens.toLocaleString()} tokens 给引擎提示和消息封装;实际原生请求仍单独核对容量。

} {!!pause?.imageReservationTokens&&

图像使用单独的本地预留,属于容量估算;实际计费与服务报告用量分列。

} + {pause?.modelFailure&&

本次模型原错误:{pause.modelFailure.message}{pause.modelFailure.upstreamRequestId?<> · 上游请求 ID:{pause.modelFailure.upstreamRequestId}:null}。原响应和请求诊断仍可核对。

} + {pause?.tokenReservation&&

{calibrationSources?<>依据同一模型线路的 {calibrationSources} 次已知响应校准,:'按本地估算,'}下一次请求需预留 {pause.tokenReservation.reservedTokens.toLocaleString()} tokens{pause.remainingTokens!==undefined?<>,剩余 {pause.remainingTokens.toLocaleString()}:null}。已在发送前暂停;这项预留用于后续预算,原始用量记录保持。

} + {pause?.tokenAccounting&&<>
本次运行上游报告用量
{pause.tokenAccounting.run.reportedTokens.toLocaleString()}
用量未知请求 / 本地预留
{pause.tokenAccounting.run.unknownRequests.toLocaleString()} / {pause.tokenAccounting.run.estimatedUnknownTokens.toLocaleString()}
本次运行预算 / 记账超出
{pause.tokenAccounting.run.maximumTokens>0?pause.tokenAccounting.run.maximumTokens.toLocaleString():'不限'} / {pause.tokenAccounting.run.exceededTokens.toLocaleString()}
任务累计预算 / 记账超出
{pause.tokenAccounting.task.maximumTokens>0?pause.tokenAccounting.task.maximumTokens.toLocaleString():'不限'} / {pause.tokenAccounting.task.exceededTokens.toLocaleString()}

单个在途响应可能超过预估。超额后停止后续请求;用量未知的请求仍按本地预留控制预算,不补造实际计费。

} {controlled?<>

受限子任务在隔离协作中处理。当前协作沿用已封存容量与累计预算;若配置不足,先保留现场,再为新目标选择合适配置。已完成工具和原失败不会自动重放。

{onRecovery&&}:} {open&&{if(!busy)setOpen(false);}} wide>

Task {session.execution?.task?.id}
原 Run {run?.id}

{([['contextLimit','上下文窗口',1000,200000],['maxOutputTokens','响应预留',256,16000],['maxRunTokens','新Run累计tokens',1000,2000000],['maxTaskTokens','原任务累计tokens',1000,10000000],['maxSteps','新Run工具轮次',1,30],['maxRunDurationMs','新Run时间(分钟)',1,60]] as const).map(([key,label,min,max])=>)}

容量须符合服务实际限制;池按成员最小容量核对。新增预算只用于这次明确续接,不修改全局设置。

{error&&

{error}

}
{plan&&
将创建原Task的新Run

可用模型 {plan.eligibleConnections.length?plan.eligibleConnections.map(row=>row.name+' / '+row.model).join('、'):plan.config.model} · 上下文 {plan.budget.contextLimit.toLocaleString()} · 可用输入 {plan.budget.inputBudget.toLocaleString()} tokens

保留 {plan.completedToolIds.length} 项完成工具、{plan.attachmentIds.length} 张附件;{plan.unknownToolIds.length} 项未知结果仍须核对。

{plan.note}

预览 SHA-256 {plan.sha256}

}
} ; diff --git a/src/client/types.ts b/src/client/types.ts index 118a81bd..c61a2a26 100644 --- a/src/client/types.ts +++ b/src/client/types.ts @@ -47,6 +47,9 @@ export interface InputAttachment { id:string;name:string;status:string;mediaType export interface BudgetPause { reason:string;requiredInputTokens?:number;contextLimit?:number;inputBudget?:number;outputReservation?:number;safetyTokens?:number;runRemainingTokens?:number|null;taskRemainingTokens?:number|null;imageReservationTokens?:number;breakdown?:Record } export interface Run { budgetPause?:BudgetPause } export interface BudgetPause { originalInputTokens?:number;transportReservationTokens?:number } +export interface BudgetPause { modelFailure?:RuntimeFailure } +export interface TokenBudgetCounter { maximumTokens:number;reportedTokens:number;unknownRequests:number;estimatedUnknownTokens:number;accountedTokens:number;remainingTokens:number|null;exceededTokens:number } +export interface BudgetPause { remainingTokens?:number;tokenReservation?:{schemaVersion:number;reservedTokens:number;routes:{sources:{runId:string;attemptId:string}[]}[]};tokenAccounting?:{schemaVersion:number;singleInFlightResponseMayExceed:boolean;run:TokenBudgetCounter;task:TokenBudgetCounter} } export interface ToolEvent { id: string; kind: string; label: string; phase: string; createdAt: string; tool?: string; input?: Record; output?: string; durationMs?: number; startedAt?: string; finishedAt?: string; diff --git a/src/core/context/request-plan.mjs b/src/core/context/request-plan.mjs index 8972aa00..463f9923 100644 --- a/src/core/context/request-plan.mjs +++ b/src/core/context/request-plan.mjs @@ -6,6 +6,7 @@ import { latestToolDocument, latestToolMessage, verifyLatestToolDocument } from import { projectMemoryMessage, MEMORY_CONTEXT_LIMITS } from '../memory/project-memory.mjs'; import { skillCatalogMessage, SKILL_LIMITS } from '../skills/skill-registry.mjs'; import { mcpCatalogMessage, MCP_LIMITS } from '../tools/mcp-contract.mjs'; +import { modelReservation,taskConsumption,verifyModelReservation } from '../../shared/contracts/model-reservation.mjs'; const hash = value => createHash('sha256').update(typeof value === 'string' ? value : JSON.stringify(value), 'utf8').digest('hex'); const PLACEHOLDER_ID = '00000000-0000-0000-0000-000000000000'; @@ -158,7 +159,7 @@ export class RequestPlanner { return { artifact_id: artifact.id, sha256: artifact.sha256, group_sha256: group.sha256, bytes: Buffer.byteLength(text, 'utf8'), chunks: chunks.length }; } - prepare(session, run, config, { system, tools, stepId, projectMemory = null, skillCatalog = null, mcpCatalog = null, transportReservationTokens = 0 }) { + prepare(session, run, config, { system, tools, stepId, projectMemory = null, skillCatalog = null, mcpCatalog = null, transportReservationTokens = 0,engineStepOnly=false }) { const task = this.store.owned('task', run.taskId, session.id); this.store.owned('run', run.id, session.id); const groups = historyGroups(session.providerMessages), budget = budgetFor(config, run, task, transportReservationTokens); @@ -306,12 +307,17 @@ export class RequestPlanner { } const body = completionBody(config, messages, tools), metrics = requestMetrics(body); if (metrics.estimatedInputTokens > budget.inputBudget) throw pause(budget, metrics, breakdown); + const tokenReservation=engineStepOnly?null:modelReservation(config,body,taskConsumption(this.store,task.id)); + const runCredit=budget.runRemainingTokens??Infinity,taskCredit=budget.taskRemainingTokens??Infinity,credit=Math.min(runCredit,taskCredit); + if(tokenReservation&&tokenReservation.reservedTokens>credit)throw pause({...budget,limitingReason:runCredit<=taskCredit?'token_limit':'task_token_limit', + tokenReservation,remainingTokens:credit},metrics,breakdown); const requestArtifact = this.artifact(session, run, body, 'model-request'); const record = { schemaVersion: 1, id: randomUUID(), sessionId: session.id, projectId: session.projectId, taskId: run.taskId, runId: run.id, stepId, requestIndex: [...this.store.core.context_plan.values()].filter(plan => plan.runId === run.id).length + 1, status: 'prepared', createdAt: new Date().toISOString(), finishedAt: null, requestSha256: metrics.sha256, requestBytes: metrics.bytes, estimatedInputTokens: metrics.estimatedInputTokens, requestArtifactId: requestArtifact.id, + ...(tokenReservation?{tokenReservation,maxBudgetAttempts:Math.min(8,Math.floor(credit/tokenReservation.reservedTokens))}:{engineStepOnly:true}), estimateMethod: metrics.estimateMethod, beforeInputTokens: before.estimatedInputTokens, beforeBytes: before.bytes, budget, ...(metrics.imageCount?{imageCount:metrics.imageCount,imageReservationTokens:metrics.imageReservationTokens,imageEstimate:metrics.imageEstimate, inputImages:config.inputImages.map(({dataUrl,...identity})=>identity)}:{}), @@ -360,11 +366,14 @@ export function verifyPlanEvidence(store, session, plan, cache = new Map()) { }; const request = get(plan.requestArtifactId, 'model-request', plan.requestSha256); const body = JSON.parse(store.artifactStore.read(request)), metrics = requestMetrics(body); + if(plan.tokenReservation)verifyModelReservation(store,session,plan.taskId,{...store.owned('run',plan.runId,session.id).model, + modelRouting:store.owned('run',plan.runId,session.id).model.routing},body,plan.tokenReservation); if (metrics.bytes !== plan.requestBytes || metrics.estimatedInputTokens !== plan.estimatedInputTokens || requestVariants(body).some(value => outputReservation(value) !== plan.budget.outputReservation) || metrics.estimatedInputTokens > plan.budget.inputBudget) throw new Error('请求预算与原正文不匹配。'); historyGroups(requestHistory(body)); for (const variant of requestVariants(body)) historyGroups(requestHistory(variant)); const frozen = JSON.parse(store.artifactStore.read(get(plan.artifactId, 'context-plan'))); + for(const key of ['tokenReservation','maxBudgetAttempts'])if(JSON.stringify(frozen[key]??null)!==JSON.stringify(plan[key]??null))throw new Error('模型用量预留与发送前封存不一致。'); for(const key of ['imageCount','imageReservationTokens','imageEstimate','inputImages']) { if(JSON.stringify(frozen[key]??null)!==JSON.stringify(plan[key]??null))throw new Error('图片身份或预算与发送前封存不匹配。'); } diff --git a/src/core/engines/engine-runner.mjs b/src/core/engines/engine-runner.mjs index 487a7aac..fca5b625 100644 --- a/src/core/engines/engine-runner.mjs +++ b/src/core/engines/engine-runner.mjs @@ -7,6 +7,7 @@ import { resolveControlledInstallation, assertControlledInstallation, controlled import { engineRunInput, engineRunReport, verifyEngineInput, nativeMessagesToChat } from '../../shared/contracts/engine-execution.mjs'; import { engineError } from '../../shared/contracts/engines.mjs'; import { modelSnapshot, accountedTokens, checkBudget, BudgetExceeded } from '../../shared/contracts/execution.mjs'; +import { modelReservation,taskConsumption,checkModelDispatchBudget } from '../../shared/contracts/model-reservation.mjs'; import { completionBody, requestMetrics } from '../../shared/contracts/model-request.mjs'; import { workerForSession } from '../../shared/contracts/workers.mjs'; import { selectPurgeStrategy, assertPurgeModelInput } from './purge-strategy.mjs'; @@ -208,7 +209,8 @@ class EngineRunDriver { inputBudget:projected.inputBudget,safetyTokens:projected.safetyTokens,retainedGroups:projected.retainedGroups,archivedGroups:projected.archivedGroups}; const maximumAttempts = config.mode === 'demo' ? 1 : config.modelRouting?.maxAttempts ?? 1; const credit = Math.min(this.run.record.budget.maxTokens - accountedTokens(this.run.record.usage), task.budget.maxTokens - accountedTokens(task.usage)); - const estimate = metrics.estimatedInputTokens + 128 + config.maxOutputTokens; + const tokenReservation=modelReservation(config,body,taskConsumption(this.store,task.id),128); + const estimate = tokenReservation.reservedTokens; const attempts = Math.min(maximumAttempts, Math.floor(credit / estimate), this.run.worker.budget.maxSteps - task.usage.requests); if (metrics.estimatedInputTokens>projected.inputBudget || attempts < 1) { const overhead=requestMetrics(completionBody(config,[],[])).estimatedInputTokens; @@ -216,7 +218,7 @@ class EngineRunDriver { const fixedInstructionTokens=requestMetrics(completionBody(config,originalMessages.filter(x=>['system','developer','user'].includes(x.role)),[])).estimatedInputTokens-overhead; const contextBudget={contextLimit:config.contextLimit,outputReservation:config.maxOutputTokens,safetyTokens:projected.safetyTokens, inputBudget:projected.inputBudget,requiredInputTokens:metrics.estimatedInputTokens,originalInputTokens:projected.before.estimatedInputTokens, - remainingTokens:credit,runRemainingTokens:Math.max(0,this.run.record.budget.maxTokens-accountedTokens(this.run.record.usage)), + remainingTokens:credit,tokenReservation,runRemainingTokens:Math.max(0,this.run.record.budget.maxTokens-accountedTokens(this.run.record.usage)), taskRemainingTokens:Math.max(0,task.budget.maxTokens-accountedTokens(task.usage)),maxAttempts:Math.max(0,attempts),sourceArtifactId:source.id,sourceSha256:source.sha256, breakdown:{requestOverheadTokens:overhead,toolDefinitionTokens,fixedInstructionTokens, retainedHistoryAndViewTokens:metrics.estimatedInputTokens-overhead-toolDefinitionTokens-fixedInstructionTokens}, @@ -231,6 +233,7 @@ class EngineRunDriver { request = { id: randomUUID(), bridgeTraceId: trace.id, sourceSha256: controlledDigest(input), requestSha256: metrics.sha256, contextViewVersion: 1, requestBytes: metrics.bytes, estimatedInputTokens: metrics.estimatedInputTokens, reservedPerAttempt: estimate, + reservationVersion:1,tokenReservation, maxAttempts: attempts, accountedAttempts: 0, observedAttempts: 0, status: 'prepared', startedAt: at() }; const plan = { ...request, ...owner }; const artifact = this.store.artifactStore.write(this.session, this.run.record, @@ -259,7 +262,7 @@ class EngineRunDriver { modelEvent.label = progress.switching ? '引擎切换模型池连接' : progress.phase === 'failed' ? '引擎模型请求失败' : progress.phase === 'completed' ? '引擎模型响应已收到' : '引擎请求模型服务'; if (['completed', 'failed'].includes(progress.phase)) modelEvent.finishedAt = at(); this.save('model_progress'); - }, plan, tools, { attempts: 1 }); + }, plan, tools, { attempts: 1,beforeDispatch:({previousFailure}={})=>checkModelDispatchBudget(this.store,this.session,this.run.record,config,body,128,previousFailure) }); const count = Math.max(1, request.observedAttempts); request.actualAttemptsFinal = count; request.responseReceivedAt = at(); request.status = 'responded'; while (request.accountedAttempts < count - 1) { diff --git a/src/core/models/model-hub.mjs b/src/core/models/model-hub.mjs index f0e4367f..1a98c93d 100644 --- a/src/core/models/model-hub.mjs +++ b/src/core/models/model-hub.mjs @@ -282,12 +282,12 @@ export class ModelHub { } } finally { this.waiting--; } } - async complete(config, body, signal, onProgress, plan, execute) { + async complete(config, body, signal, onProgress, plan, execute,beforeDispatch) { const operation = this.provider.requestOperation?.(signal); - try { return await this.completeOwned(config, body, operation?.signal ?? signal, onProgress, plan, execute); } + try { return await this.completeOwned(config, body, operation?.signal ?? signal, onProgress, plan, execute,beforeDispatch); } finally { operation?.finish(); } } - async completeOwned(config, body, signal, onProgress, plan, execute) { + async completeOwned(config, body, signal, onProgress, plan, execute,beforeDispatch) { const routing = config.modelRouting, excluded = new Set(), deadline = Date.now() + Math.min(routing.queueTimeoutMs, 60000); const requestId = plan?.id ?? randomUUID(), runId = plan?.runId ?? requestId; const requirements = config.capabilityProbe ? [] : ['text', ...(requestVariants(body).some(value => value.tools?.length) ? ['tools'] : []), @@ -303,6 +303,7 @@ export class ModelHub { let lastError; for (let count = 1; count <= routing.maxAttempts; count++) { signal?.throwIfAborted(); + beforeDispatch?.({previousFailure:lastError?.diagnostic}); let lease; try { lease = await this.acquire(eligibleRouting, runId, excluded, signal, deadline, requirements); } catch (cause) { throw lastError ?? cause; } diff --git a/src/server/provider.mjs b/src/server/provider.mjs index a7358666..306ee904 100644 --- a/src/server/provider.mjs +++ b/src/server/provider.mjs @@ -8,6 +8,7 @@ import { normalizeProtocolResponse, protocolRoute, validateModelResponse, report import { readModelStream } from '../core/models/model-stream.mjs'; import { modelWaitSettings, requestDeadline } from '../core/models/request-deadline.mjs'; import { ModelTransport } from '../core/models/transport.mjs'; +import { localModelEstimate } from '../shared/contracts/model-reservation.mjs'; import { ERROR_CAPTURE_LIMIT, readErrorResponse, upstreamCapture } from '../core/models/upstream-error.mjs'; export const engineToolDefinitions = [ @@ -156,19 +157,22 @@ export class Provider { const waits = modelWaitSettings(config), timeoutMs = body ? waits.modelResponseTimeoutMs : 20000; const policy = { attempts: 3, delays: [500, 1500], timeoutMs, totalTimeoutMs: timeoutMs, idleTimeoutMs: waits.modelStreamIdleTimeoutMs, ...this.retryPolicy, ...options }; + policy.attempts=Math.min(policy.attempts,options.maxBudgetAttempts??policy.attempts); const operation = this.requestOperation(signal); signal = operation.signal; const deadline = Date.now() + policy.totalTimeoutMs; const startedAt = Date.now(); - let attempts = 0; + let attempts = 0, previousFailure = null; this.activeRequests = (this.activeRequests ?? 0) + 1; try { for (let attempt = 1; attempt <= policy.attempts; attempt++) { signal?.throwIfAborted(); + options.beforeDispatch?.({previousFailure}); attempts = attempt; const timeout = requestDeadline({ signal, timeoutMs: Math.max(1, Math.min(policy.timeoutMs, deadline - Date.now())), idleTimeoutMs: policy.idleTimeoutMs }); const requestSignal = timeout.signal; let readingStream = false, streamTerminalSeen = false, dispatched = false, usage = null, outcome = 'unknown'; - const receipt = { attemptId: randomUUID(), endpoint: base, model: config.model, protocol: config.protocol ?? 'chat-completions', startedAt: new Date().toISOString() }; + const receipt = { attemptId: randomUUID(), endpoint: base, model: config.model, protocol: config.protocol ?? 'chat-completions', startedAt: new Date().toISOString(), + ...(body?{localEstimate:localModelEstimate(body)}:{}) }; const context = { endpoint: base, model: config.model, route, attempt, attemptId: receipt.attemptId }; onProgress?.({ phase: 'running', attempt, maxAttempts: policy.attempts, endpoint: base, route, wireAttemptId: receipt.attemptId }); let responseDetails = null; @@ -255,6 +259,7 @@ export class Provider { const retry = diagnostic.retryable && attempt < policy.attempts && remaining > waitMs; diagnostic.retryDecision = retry ? 'retry_after_wait' : readingStream ? 'stream_not_replayed' : !diagnostic.retryable ? 'not_retryable' : attempt >= policy.attempts ? 'attempt_limit' : 'wait_budget'; diagnostic.remainingWaitBudgetMs = Math.max(0, remaining); + previousFailure = diagnostic; this.lastRequest = { ...diagnostic, status: retry ? 'retrying' : 'failed', durationMs: Date.now() - startedAt }; onProgress?.({ phase: retry ? 'retrying' : 'failed', attempt, maxAttempts: policy.attempts, diagnostic, waitMs: retry ? waitMs : 0, durationMs: Date.now() - startedAt }); @@ -289,7 +294,7 @@ export class Provider { return this.request(selected, protocolRoute(selected), input, signal, key, progress, { ...(config.modelRouting ? { attempts: 1 } : {}), ...requestPolicy, transformResponse: raw => validateModelResponse(normalizeProtocolResponse(selected, raw), tools) }); }; - const result = config.modelRouting && this.hub ? await this.hub.complete(config, body, signal, onProgress, requestPlan, execute) + const result = config.modelRouting && this.hub ? await this.hub.complete(config, body, signal, onProgress, requestPlan, execute,requestPolicy.beforeDispatch) : await execute(config, this.apiKey, body, onProgress); const received = result.choices?.[0]?.message; if (!received) throw new Error('兼容服务响应中缺少 choices[0].message。'); diff --git a/src/server/runtime.mjs b/src/server/runtime.mjs index 449806cb..2dcf974d 100644 --- a/src/server/runtime.mjs +++ b/src/server/runtime.mjs @@ -16,6 +16,7 @@ import { projectContext, toolDefinitions, researchToolDefinitions, engineToolDef import { now, repairToolPairs } from './store.mjs'; import { executionEnvironment, sessionContext, sessionSnapshot } from './diagnostics.mjs'; import { BudgetExceeded, checkBudget } from '../shared/contracts/execution.mjs'; +import { tokenBudgetState,checkModelDispatchBudget } from '../shared/contracts/model-reservation.mjs'; import { taskToolAllowed, taskContract } from '../shared/contracts/task-control.mjs'; import { CheckRunner } from '../core/check-runner.mjs'; import { inspectProject } from '../core/project-inspector.mjs'; @@ -362,7 +363,7 @@ export class Runtime { const projectMemory = await this.memoryManager().context(session, { redact: value => this.provider.redact(value) }); signal.throwIfAborted(); const planned = this.requestPlanner().prepare(session, run.record, config, { system: context, tools, stepId: stepRecord.id, projectMemory, skillCatalog, mcpCatalog, - transportReservationTokens:run.worker?.engine?ENGINE_SEED_RESERVATION:0 }); + transportReservationTokens:run.worker?.engine?ENGINE_SEED_RESERVATION:0,engineStepOnly:!!run.worker?.engine }); if (planned.record.archivedGroups && !projectionLogged) { projectionLogged = true; session.compactions++; this.store.event(session, { kind: 'context', label: '整理请求上下文', phase: 'completed', contextPlanId: planned.record.id, @@ -379,7 +380,7 @@ export class Runtime { requestStarted = true; const progressHandler = progress => { if (progress.consumption) { - this.store.recordConsumption(session, run.record, { ...progress.consumption, requestId: planned.record.id }, planned.record.estimatedInputTokens + config.maxOutputTokens); + this.store.recordConsumption(session, run.record, { ...progress.consumption, requestId: planned.record.id }, planned.record.tokenReservation?.reservedTokens??planned.record.estimatedInputTokens + config.maxOutputTokens); responseAccounted = true; if (run.worker) { planned.record.workerUsageAccounted = true; this.store.save(session); } return; @@ -418,9 +419,11 @@ export class Runtime { if (['completed', 'failed'].includes(progress.phase)) requestEvent.finishedAt = now(); this.store.save(session); }; + const requestConfig=config.modelRouting&&!externalEngine?{...config,modelRouting:{...config.modelRouting,maxAttempts:Math.min(config.modelRouting.maxAttempts,planned.record.maxBudgetAttempts)}}:config; response = externalEngine ? await this.engineRunner().complete(session, run, planned.messages, progressHandler, planned.record, tools) - : await this.provider.complete(config, planned.messages, run.project, signal, progressHandler, planned.record, tools); + : await this.provider.complete(requestConfig, planned.messages, run.project, signal, progressHandler, planned.record, tools, + {maxBudgetAttempts:planned.record.maxBudgetAttempts,beforeDispatch:({previousFailure}={})=>checkModelDispatchBudget(this.store,session,run.record,requestConfig,planned.body,0,previousFailure)}); if (response.compatibility?.length && requestEvent) { requestEvent.compatibility = response.compatibility; this.store.save(session); } this.requestPlanner().finish(session, planned.record, { response, attempts: planned.record.requestAttempts }); if (run.worker && !externalEngine && !responseAccounted) { @@ -431,10 +434,15 @@ export class Runtime { if (requestEvent?.stream && requestEvent.stream.status !== 'completed') { requestEvent.stream.status = signal.aborted ? 'interrupted' : 'incomplete'; requestEvent.phase = signal.aborted ? 'interrupted' : 'failed'; requestEvent.finishedAt = now(); } this.requestPlanner().finish(session, planned.record, { error: Object.assign(new Error(this.provider.redact(error.message ?? String(error))), { name: signal.aborted ? 'AbortError' : error.name }), attempts: planned.record.requestAttempts }); if (run.worker && requestStarted && !responseAccounted) { - this.store.recordResponse(session, run.record, null, planned.record.estimatedInputTokens + config.maxOutputTokens); + this.store.recordResponse(session, run.record, null, planned.record.tokenReservation?.reservedTokens??planned.record.estimatedInputTokens + config.maxOutputTokens); planned.record.workerUsageAccounted = true; this.store.save(session, { type: 'worker.request_usage_unknown' }); } + try { checkBudget(run.record,task); } + catch(budget){ + if(budget instanceof BudgetExceeded&&error.diagnostic)budget.contextBudget={modelFailure:structuredClone(error.diagnostic)}; + throw budget; + } throw error; } signal.throwIfAborted(); @@ -543,8 +551,9 @@ export class Runtime { run.stopReason = budget?.reason ?? 'cancelled'; if(budget)run.record.budgetPause={schema:1,reason:budget.reason,createdAt:now(),taskId:run.taskId,runId:run.id, ...(budget.contextBudget??{}),runUsage:structuredClone(run.record.usage),taskUsage:structuredClone(this.store.core.task.get(run.taskId).usage), + tokenAccounting:tokenBudgetState(run.record,this.store.core.task.get(run.taskId)), runBudget:structuredClone(run.record.budget),taskBudget:structuredClone(this.store.core.task.get(run.taskId).budget), - note:'本地预留估算与服务实际用量分开记录;图片预留不是厂商计费或容量保证。'}; + note:'本地预留与上游报告用量分开记录。单个在途响应可能超过预估;超额后停止后续请求,未知用量保持未知。'}; for (const event of session.events.filter(item => item.kind === 'model' && item.runId === run.id && ['running', 'retrying'].includes(item.phase))) { event.phase = 'interrupted'; event.finishedAt = now(); event.output = '模型请求已停止,已完成的工具结果保留。'; } diff --git a/src/server/store.mjs b/src/server/store.mjs index 7f0aa0a1..199ad8b8 100644 --- a/src/server/store.mjs +++ b/src/server/store.mjs @@ -209,7 +209,7 @@ export class Store extends EventEmitter { if (plan.workerDispatchedAt && !plan.workerUsageAccounted && !plan.engineStepOnly) { const run = this.core.run.get(plan.runId); if (run && this.core.task.get(run.taskId)?.workerId) { - this.recordResponse(session, run, null, plan.estimatedInputTokens + (run.model.maxOutputTokens ?? 2048)); + this.recordResponse(session, run, null, plan.tokenReservation?.reservedTokens??plan.estimatedInputTokens + (run.model.maxOutputTokens ?? 2048)); plan.workerUsageAccounted = true; plan.workerUsageRecovered = true; changed = true; } } @@ -553,6 +553,7 @@ export class Store extends EventEmitter { model: receipt.model, endpoint: receipt.endpoint, protocol: receipt.protocol, startedAt: receipt.startedAt, finishedAt: receipt.finishedAt, outcome: receipt.outcome, usageStatus: receipt.usage ? 'reported' : 'unknown', usage: receipt.usage ?? null, + ...(receipt.localEstimate?{localEstimate:structuredClone(receipt.localEstimate)}:{}), estimatedUnknownTokens: receipt.usage ? 0 : Math.max(0, Math.ceil(estimate)) }); this.recordResponse(session, run, receipt.usage, estimate); this.save(session, { type: 'model.consumption', detail: { attemptId: receipt.attemptId, usageStatus: receipt.usage ? 'reported' : 'unknown' } }); diff --git a/src/shared/contracts/engine-execution.mjs b/src/shared/contracts/engine-execution.mjs index 67d31d6e..1252674f 100644 --- a/src/shared/contracts/engine-execution.mjs +++ b/src/shared/contracts/engine-execution.mjs @@ -1,6 +1,7 @@ import { engineError, engineCanonicalDigest as controlledDigest } from './engines.mjs'; import { completionBody, requestMetrics, requestVariants, outputReservation } from './model-request.mjs'; import { engineContextIndex,engineContextEvidence,engineContextView,engineSeedPrompt } from './engine-context.mjs'; +import { verifyModelReservation } from './model-reservation.mjs'; export const ENGINE_RUN_ACTIVE = new Set(['queued', 'starting', 'initializing', 'running', 'waiting_tool', 'closing', 'stopping']); export function selectedEngineInput(input) { @@ -64,12 +65,16 @@ export function verifyEngineModelInput(store, session, job, request) { if (input.plan[key] !== request[key]) throw engineError('原引擎模型请求或尝试预算与封存输入不一致。'); } if(request.contextViewVersion!==undefined&&(request.contextViewVersion!==1||input.plan.contextViewVersion!==request.contextViewVersion||!input.view))throw engineError('原生模型上下文视图版本缺失或不匹配。'); + if(request.reservationVersion!==undefined){ + if(request.reservationVersion!==1||input.plan.reservationVersion!==1||controlledDigest(request.tokenReservation)!==controlledDigest(input.plan.tokenReservation))throw engineError('原生用量预留版本或输入不一致。'); + verifyModelReservation(store,session,job.taskId,{...job.model,modelRouting:job.model.routing},input.body,request.tokenReservation); + }else if(input.plan.reservationVersion!==undefined||input.plan.tokenReservation!==undefined)throw engineError('原生用量预留版本缺失。'); if (input.owner.sessionId !== job.sessionId || input.owner.taskId !== job.taskId || input.owner.runId !== job.runId || input.owner.projectId !== job.projectId || controlledDigest(input.native) !== request.sourceSha256 || metrics.sha256 !== request.requestSha256 || metrics.bytes !== request.requestBytes || !Number.isSafeInteger(request.maxAttempts) || request.maxAttempts < 1 || request.maxAttempts > 8 || !Number.isSafeInteger(input.native.maxTokens) || input.native.maxTokens < 1 || !requestVariants(input.body).every(body => outputReservation(body) === input.native.maxTokens) - || request.reservedPerAttempt !== metrics.estimatedInputTokens + 128 + input.native.maxTokens) throw engineError('原引擎模型输入、归属或预算核对失败。'); + || request.reservedPerAttempt !== (request.reservationVersion===1?request.tokenReservation.reservedTokens:metrics.estimatedInputTokens + 128 + input.native.maxTokens)) throw engineError('原引擎模型输入、归属或预算核对失败。'); if(input.view){ const view=input.view,source=store.owned('artifact',view.sourceArtifactId,session.id),sourceValue=JSON.parse(store.artifactStore.read(source)); if(view.schemaVersion!==1||sourceValue.schemaVersion!==1||source.kind!=='engine-native-input'||source.taskId!==job.taskId||source.runId!==job.runId||source.sha256!==view.sourceSha256 diff --git a/src/shared/contracts/model-reservation.mjs b/src/shared/contracts/model-reservation.mjs new file mode 100644 index 00000000..1d9bd5ae --- /dev/null +++ b/src/shared/contracts/model-reservation.mjs @@ -0,0 +1,93 @@ +import {actualRequest,outputReservation,requestMetrics} from './model-request.mjs'; +import {accountedTokens,checkBudget,BudgetExceeded} from './execution.mjs'; + +const count=value=>Number.isSafeInteger(value)&&value>=0; +const bounded=value=>Math.min(Number.MAX_SAFE_INTEGER,Math.max(0,Math.ceil(value))); +const endpoint=value=>String(value??'').replace(/\/+$/,''); +export function localModelEstimate(body){ + const metrics=requestMetrics(body); + return {schemaVersion:1,inputTokens:metrics.estimatedInputTokens,outputTokens:outputReservation(body), + requestSha256:metrics.sha256,requestBytes:metrics.bytes,estimateMethod:metrics.estimateMethod}; +} +export function taskConsumption(store,taskId){ + return [...store.core.run.values()].filter(run=>run.taskId===taskId) + .flatMap(run=>(run.consumption??[]).map(receipt=>({...receipt,runId:run.id}))); +} + +export function checkModelDispatchBudget(store,session,run,config,body,overheadTokens=0,previousFailure=null){ + const task=store.owned('task',run.taskId,session.id); + try{ + checkBudget(run,task); + const tokenReservation=modelReservation(config,body,taskConsumption(store,task.id),overheadTokens); + const remaining=value=>value.budget.maxTokens>0?Math.max(0,value.budget.maxTokens-accountedTokens(value.usage)):Infinity; + const runRemainingTokens=remaining(run),taskRemainingTokens=remaining(task),remainingTokens=Math.min(runRemainingTokens,taskRemainingTokens); + if(tokenReservation.reservedTokens>remainingTokens){ + const error=new BudgetExceeded(runRemainingTokens<=taskRemainingTokens?'token_limit':'task_token_limit','下一次模型请求的用量预留超过剩余预算,已在发送前暂停。原用量和已完成工具结果保留。'); + error.contextBudget={tokenReservation,remainingTokens,runRemainingTokens:Number.isFinite(runRemainingTokens)?runRemainingTokens:null, + taskRemainingTokens:Number.isFinite(taskRemainingTokens)?taskRemainingTokens:null}; + throw error; + } + return tokenReservation; + }catch(error){ + if(error instanceof BudgetExceeded&&previousFailure)error.contextBudget={...error.contextBudget,modelFailure:structuredClone(previousFailure)}; + throw error; + } +} + +// A conservative reservation for the next dispatch, not corrected historical +// billing or a promise about an unseen upstream response. Never lowers estimates. +export function modelReservation(config,body,receipts=[],overheadTokens=0){ + if(!count(overheadTokens))throw new Error('模型请求封装预留无效。'); + const members=config.modelRouting?.members??[{id:null,model:config.model,protocol:config.protocol??'chat-completions',baseUrl:config.baseUrl}]; + const routes=members.map(member=>{ + const wire=actualRequest(body,member),local=localModelEstimate(wire),seen=new Set(),sources=[]; + let inputRatio=1,observedOutputTokens=local.outputTokens,observedTotalFloor=0; + if(!count(local.inputTokens)||!count(local.outputTokens))throw new Error('模型请求输入或输出预留无效。'); + for(const receipt of receipts){ + const estimate=receipt.localEstimate,usage=receipt.usage; + if(receipt.model!==member.model||(receipt.protocol??'chat-completions')!==(member.protocol??'chat-completions') + ||(receipt.connectionId??null)!==(member.id??null)||endpoint(receipt.endpoint)!==endpoint(member.baseUrl) + ||receipt.usageStatus!=='reported'||!usage||estimate?.schemaVersion!==1||!count(estimate.inputTokens)||estimate.inputTokens<1 + ||!receipt.attemptId||seen.has(receipt.attemptId))continue; + if(!count(usage.total_tokens)&&!(count(usage.prompt_tokens)&&count(usage.completion_tokens)))continue; + seen.add(receipt.attemptId); + if(count(usage.prompt_tokens))inputRatio=Math.max(inputRatio,usage.prompt_tokens/estimate.inputTokens); + if(count(usage.completion_tokens))observedOutputTokens=Math.max(observedOutputTokens,usage.completion_tokens); + const total=count(usage.total_tokens)?usage.total_tokens:usage.prompt_tokens+usage.completion_tokens; + observedTotalFloor=Math.max(observedTotalFloor,total); + sources.push({runId:receipt.runId,attemptId:receipt.attemptId,localInputTokens:estimate.inputTokens, + reportedInputTokens:count(usage.prompt_tokens)?usage.prompt_tokens:null,reportedOutputTokens:count(usage.completion_tokens)?usage.completion_tokens:null,totalTokens:total}); + } + const reservedTokens=bounded(Math.max(local.inputTokens*inputRatio+observedOutputTokens,observedTotalFloor)+overheadTokens); + return {connectionId:member.id??null,model:member.model,protocol:member.protocol??'chat-completions',endpoint:member.baseUrl, + localInputTokens:local.inputTokens,outputReservation:local.outputTokens,inputRatio,observedOutputTokens,observedTotalFloor,reservedTokens,sources}; + }); + if(!routes.length)throw new Error('模型预留没有实际候选线路。'); + return {schemaVersion:1,method:'observed-route-max-v1',overheadTokens,reservedTokens:Math.max(...routes.map(route=>route.reservedTokens)),routes}; +} + +export function verifyModelReservation(store,session,taskId,config,body,reservation){ + if(reservation?.schemaVersion!==1||reservation.method!=='observed-route-max-v1'||!Array.isArray(reservation.routes))throw new Error('模型用量预留版本或线路缺失。'); + const receipts=new Map(); + for(const route of reservation.routes)for(const source of route.sources??[]){ + const run=store.owned('run',source.runId,session.id); + if(run.taskId!==taskId)throw new Error('模型预算校准来源不属于当前Task。'); + const receipt=run.consumption?.find(value=>value.attemptId===source.attemptId); + if(!receipt)throw new Error('模型预算校准缺少原消费记录。'); + receipts.set(source.attemptId,{...receipt,runId:run.id}); + } + const expected=modelReservation(config,body,[...receipts.values()],reservation.overheadTokens); + if(JSON.stringify(expected)!==JSON.stringify(reservation))throw new Error('模型预算预留与原估算/用量/实际线路不一致。'); + return reservation; +} + +export function tokenBudgetState(run,task){ + const state=value=>{ + const reportedTokens=value.usage?.totalTokens??0,estimatedUnknownTokens=value.usage?.estimatedUnknownTokens??0, + accountedTokens=reportedTokens+estimatedUnknownTokens,maximumTokens=value.budget?.maxTokens??0; + return {maximumTokens,reportedTokens,unknownRequests:value.usage?.unknownRequests??0,estimatedUnknownTokens,accountedTokens, + remainingTokens:maximumTokens>0?Math.max(0,maximumTokens-accountedTokens):null, + exceededTokens:maximumTokens>0?Math.max(0,accountedTokens-maximumTokens):0}; + }; + return {schemaVersion:1,singleInFlightResponseMayExceed:true,run:state(run),task:state(task)}; +} diff --git a/task_plan.md b/task_plan.md index c7e3c081..76d84c33 100644 --- a/task_plan.md +++ b/task_plan.md @@ -7,10 +7,15 @@ A已提交0a58c0d/6257bba并封存。继续先完成网络与报告中的确定 1. [complete] 固定Node24.22.1/Undici7.30.1;原失败明确连接阶段,当前HTTP(S)_PROXY/NO_PROXY仅存在位核对、均无设置。连接/响应等待/正文中断分开,不导出代理或密钥。 2. [complete] Provider所属双栈与代理/NO_PROXY连接策略;未知POST不自动重发/切池,明确完整预发送错误链才可重试。HTTP正文中断保留有界原片段、上游请求ID与未知消费;取消后关闭所属dispatcher,可显式重建。 3. [complete] 自有双栈/代理/绕过、未知POST/正文、取消复用与工具不重放包含在94/94相关回归;构建通过。engine-worker原14/14和最终编译宿主200/503正文中断、冷却后显式续接2/2分别通过。A原621份/646工件/20夹具请求离线核对保持。 -4. [in_progress] 修正真实验收器实际模型身份来源,区分零请求、混合路由与未知用量;按范围本地Git和回顾,不把诊断连通当真实模型鉴权通过。C身份纯函数四项通过,尚未正式接入验收器或预算校准。 +4. [complete] C实际Task模型身份、同线路已知usage校准后续预留、每次重试/切池前核对、单次超额后保留原错误并停止均完成。最终425输入冻结后149/149、构建、Chat/Responses固定16K各5组工程链路和编译UI两视口通过并目视回看;A/B/原真实六例离线核验保持。见docs/QA-LIVE-BUDGET-C.md,源码与证据分次本地提交,不改正式宿主。 +5. [in_progress] D原版Purge隔离只读预研:确认默认重试/权限及原patch额外组件,下一固定完整加载输入和独立home/配置/工作区,保留宿主外部审批和受限桥。未加载原版插件、未发真实模型请求,E新真实批次仍待D验收。 B原94/94、构建和A封存通过。编译/引擎首轮14/16与状态复核0/2保留:两项是验收在原连接冷却结束前立刻续接,最终按原cooldownUntil等待后2/2,未重复已通过14项。B见docs/QA-LIVE-NETWORK-B.md。一次猜测budget.mjs不存在已改按rg实际定位execution.mjs;文档新增尾空行已清理。 +C开发回顾:首26/26后广泛开发轮147项146通过/1失败,日志保持;开发期间UI实测发现单连接最后一次503超额没有预算暂停,已修正并保留原错误诊断。原生专项进一步证明外层工具回传属于协调步骤,不能把内层真实用量校准再套成外层新请求预留,否则会挡住已完成工具回传;已明确engineStepOnly,在内层实际HTTP前校准。原UI两次失败和原生状态复核保持,最终源码冻结后另行验收,不将开发混合来源记为最终整轮。一次多文件补丁因无效import锚点原子拒绝,已按真实锚点修正,无部分变更。 + +C收口审查新增:首冻结147/147及编译UI两组已过;Responses首次unknown/reset停止、四成功一次未知,原失败保持。给自有夹具增加错误收集后独立复验五组通过,未复现原断开,原因保持未知。静态审查另发现“未超额但剩余已不足”仍可能重试,已在Provider/池/原生每次发送前重算同Task线路预留,保留前次原HTTP错误;新增两项专项与身份合计13/13、构建通过。当前最终425输入重新冻结,149项/两协议/编译UI重新按最终来源验证;不套用首冻结结果、不重跑旧真实六例。 + ## 当前修复:真实执行上下文与可诊断容量(1.24.1切片) 接续b67b4f2,分支fix/live-execution-context-1.24.1。用户继续授权执行已记录修复方案;先A上下文,再B连接/C预算身份,不开启子代理。正式1.24/PID36552/hasKey=true、无活动,原池保持;开发与验证使用隔离数据/端口,先离线复现,不续跑原六例。 diff --git a/tests/fixtures/responses-engineering-http.mjs b/tests/fixtures/responses-engineering-http.mjs index 0ffd1d2a..8c48eceb 100644 --- a/tests/fixtures/responses-engineering-http.mjs +++ b/tests/fixtures/responses-engineering-http.mjs @@ -5,7 +5,7 @@ import { requestHistory } from '../../src/shared/contracts/model-request.mjs'; // An owned wire fixture only: the original engineering algorithm receives the // same canonical history. Runtime/Harness/checks/Git still execute for real. export async function responsesEngineeringHttp(t, algorithm) { - const requests = [], sockets = new Set(); + const requests = [], errors = [], sockets = new Set(); const server = http.createServer(async (request, response) => { try { const parts = []; for await (const part of request) parts.push(part); @@ -24,10 +24,10 @@ export async function responsesEngineeringHttp(t, algorithm) { else { send('response.output_item.added', { item: { ...item, arguments: '', status: 'in_progress' } }); send('response.function_call_arguments.delta', { item_id: item.id, delta: item.arguments }); send('response.function_call_arguments.done', { item_id: item.id, arguments: item.arguments }); } } send('response.completed', { response: final }); response.end(); - } catch { response.destroy(); } + } catch (error) { errors.push({message:error.message,stack:error.stack}); response.destroy(); } }); server.on('connection', socket => { sockets.add(socket); socket.on('close', () => sockets.delete(socket)); }); await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); t.after(() => new Promise(resolve => { for (const socket of sockets) socket.destroy(); server.close(resolve); })); - return { baseUrl: 'http://127.0.0.1:' + server.address().port + '/v1', requests }; + return { baseUrl: 'http://127.0.0.1:' + server.address().port + '/v1', requests, errors }; } diff --git a/tests/live-model-observation.test.mjs b/tests/live-model-observation.test.mjs new file mode 100644 index 00000000..a7715b0d --- /dev/null +++ b/tests/live-model-observation.test.mjs @@ -0,0 +1,39 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { observedTaskModels } from '../scripts/live-model-observation.mjs'; + +const receipt=(attemptId,model='actual-claude',extra={})=>({attemptId,model,protocol:'chat-completions',connectionId:'owned-connection', + outcome:'completed',usageStatus:'reported',usage:{prompt_tokens:23,completion_tokens:7,total_tokens:30},...extra}); +const state=(requests,runs)=>({config:{model:'planned-gpt'},execution:{task:{id:'owned-task',usage:{requests}},runs,run:runs.at(-1)}}); +const run=(id,receipts,extra={})=>({id,taskId:'owned-task',consumption:receipts,usage:{requests:receipts.length},...extra}); + +test('actual model comes from owned consumption, excludes other Tasks and deduplicates the current Run',()=>{ + const value=state(2,[run('first',[receipt('one')]),run('second',[receipt('two')]),run('foreign',[receipt('foreign','other')],{taskId:'other-task'})]); + value.execution.run=value.execution.runs[1];const original=structuredClone(value); + const report=observedTaskModels(value,'owned-task'); + assert.equal(report.status,'observed');assert.equal(report.model,'actual-claude');assert.equal(report.observedAttempts,2); + assert.deepEqual(report.models,['actual-claude']);assert.equal(report.routes[0].requests,2);assert.deepEqual(value,original); +}); + +test('mixed routes retain each original model/protocol/outcome and unknown use is never reported as zero',()=>{ + const value=state(2,[run('mixed',[receipt('one','actual-gpt',{outcome:'unknown',usageStatus:'unknown',usage:null}), + receipt('two','actual-claude',{protocol:'responses',connectionId:'second'})])]); + const report=observedTaskModels(value); + assert.equal(report.status,'mixed');assert.equal(report.model,null);assert.equal(report.routes.length,2); + assert.equal(report.attempts[0].usage,null);assert.equal(report.routes[0].unknownUsageRequests,1); + assert.equal(report.attempts[1].protocol,'responses');assert.deepEqual(report.attempts[1].usage,value.execution.run.consumption[1].usage); +}); + +test('zero dispatch, missing ledger, missing model and truncated Run history never borrow configured model identity',()=>{ + assert.equal(observedTaskModels(state(0,[])).status,'not_observed'); + assert.equal(observedTaskModels({}).status,'incomplete'); + for(const value of [state(3,[run('old',[])]),state(1,[run('missing',[receipt('one',null)])]),state(2,[run('recent',[receipt('last')])])]){ + const report=observedTaskModels(value);assert.equal(report.status,'incomplete');assert.equal(report.model,null);assert.ok(report.issues.length); + } +}); + +test('repeated attempt identities across Runs and contradictory usage stay explicit instead of inflating successes',()=>{ + const report=observedTaskModels(state(2,[run('one',[receipt('same')]),run('two',[receipt('same','changed',{usage:null})])])); + assert.equal(report.status,'incomplete');assert.equal(report.observedAttempts,1);assert.equal(report.model,null); + assert.ok(report.issues.some(issue=>issue.kind==='conflicting_attempt'));assert.ok(report.issues.some(issue=>issue.kind==='request_coverage_gap')); +}); diff --git a/tests/model-reservation.test.mjs b/tests/model-reservation.test.mjs new file mode 100644 index 00000000..a20fe54b --- /dev/null +++ b/tests/model-reservation.test.mjs @@ -0,0 +1,114 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import {modelReservation,localModelEstimate,verifyModelReservation} from '../src/shared/contracts/model-reservation.mjs'; +import {completionBody} from '../src/shared/contracts/model-request.mjs'; +import {Provider} from '../src/server/provider.mjs'; +import {Runtime} from '../src/server/runtime.mjs'; +import {fixture,compatibleFixture,completion,call} from './helpers.mjs'; +import {controlledWorker} from './fixtures/controlled-worker.mjs'; +import {verifyEngineModelInput} from '../src/shared/contracts/engine-execution.mjs'; + +const config={mode:'compatible',model:'owned-A',protocol:'chat-completions',baseUrl:'http://127.0.0.1:1/v1',maxOutputTokens:512,contextLimit:16000}; +const body=completionBody(config,[{role:'user',content:'Owned bounded input.'}],[]); +const observed=(extra={})=>({attemptId:'owned-attempt',runId:'owned-run',model:config.model,protocol:config.protocol,endpoint:config.baseUrl, + connectionId:null,outcome:'completed',usageStatus:'reported',localEstimate:localModelEstimate(body), + usage:{prompt_tokens:4000,completion_tokens:800,total_tokens:4800},...extra}); + +test('reservation uses actual known usage from the same route, never lowers estimates or rewrites original receipts',()=>{ + const receipts=[observed(),observed({attemptId:'unknown',usageStatus:'unknown',usage:{total_tokens:9999999}}), + observed({attemptId:'other',model:'other-model',usage:{total_tokens:9999999}}),observed({attemptId:'legacy',localEstimate:undefined,usage:{total_tokens:9999999}})]; + const original=structuredClone(receipts),reservation=modelReservation(config,body,receipts,128); + assert.equal(reservation.reservedTokens,4928);assert.equal(reservation.routes[0].sources.length,1); + assert.ok(reservation.routes[0].inputRatio>1);assert.deepEqual(receipts,original); + const small=modelReservation(config,body,[observed({usage:{prompt_tokens:1,completion_tokens:1,total_tokens:2}})]); + assert.equal(small.reservedTokens,localModelEstimate(body).inputTokens+512); + const pool={...config,modelRouting:{members:[{id:'A',model:config.model,protocol:config.protocol,baseUrl:config.baseUrl}, + {id:'B',model:'owned-B',protocol:config.protocol,baseUrl:'http://127.0.0.1:2/v1'}]}}; + const mixed=modelReservation(pool,completionBody(pool,[{role:'user',content:'Owned bounded input.'}],[]),[observed({connectionId:'A'})]); + assert.equal(mixed.routes[0].sources.length,1);assert.equal(mixed.routes[1].sources.length,0);assert.equal(mixed.routes[1].inputRatio,1); +}); + +test('reservation verification binds original Task/Run consumption and rejects changed usage or foreign provenance',()=>{ + const receipt=observed(),run={id:'owned-run',taskId:'owned-task',sessionId:'owned-session',consumption:[receipt]},session={id:'owned-session'}; + const store={owned(kind,id,sessionId){assert.equal(kind,'run');assert.equal(id,run.id);assert.equal(sessionId,run.sessionId);return run;}}; + const reservation=modelReservation(config,body,[receipt],128); + assert.deepEqual(verifyModelReservation(store,session,'owned-task',config,body,reservation),reservation); + const altered=structuredClone(reservation);altered.routes[0].inputRatio=1; + assert.throws(()=>verifyModelReservation(store,session,'owned-task',config,body,altered),/不一致/); + assert.throws(()=>verifyModelReservation(store,session,'other-task',config,body,reservation),/当前Task/); + run.consumption[0]={...receipt,usage:{...receipt.usage,total_tokens:4900}}; + assert.throws(()=>verifyModelReservation(store,session,'owned-task',config,body,reservation),/不一致/); +}); + +test('known high input calibrates the next request and stops before dispatch while preserving the completed read',async t=>{ + const f=fixture(t),model=await compatibleFixture(t,()=>({...completion([call('owned-read','read_file',{path:'README.md'})]), + usage:{prompt_tokens:45000,completion_tokens:100,total_tokens:45100}})); + f.store.saveConfig({...config,baseUrl:model.baseUrl,maxRunTokens:70000,maxTaskTokens:1000000}); + const provider=new Provider(),runtime=new Runtime(f.store,provider);t.after(()=>runtime.stopAll()); + runtime.start(f.session.id,'Read README. Do not modify files or run checks.');await runtime.active.get(f.session.id).promise; + const run=f.store.core.run.get(f.session.runId),first=structuredClone(run.consumption[0]); + assert.equal(model.requests.length,1);assert.equal(f.session.status,'paused');assert.equal(run.stopReason,'token_limit'); + assert.equal(run.usage.totalTokens,45100);assert.equal(run.usage.unknownRequests,0); + assert.equal(f.session.events.filter(event=>event.tool==='read_file'&&event.phase==='completed').length,1); + assert.ok(run.budgetPause.tokenReservation.reservedTokens>run.budgetPause.remainingTokens); + assert.equal(run.budgetPause.tokenReservation.routes[0].sources[0].attemptId,first.attemptId); + assert.deepEqual(run.consumption[0],first);assert.ok(first.localEstimate.requestSha256); +}); + +for(const mode of ['legacy','pool','connection'])test('one reported overrun stops '+mode+' requests and keeps actual use and excess',async t=>{ + const f=fixture(t),model=await compatibleFixture(t,()=>({status:503,body:{error:{message:'owned failed request'},usage:{prompt_tokens:55000,completion_tokens:17,total_tokens:55017}}})); + f.store.saveConfig({...config,baseUrl:model.baseUrl,maxRunTokens:35000,maxTaskTokens:1000000}); + const provider=new Provider({retryPolicy:{delays:[1,1]}}); + if(mode!=='legacy'){const hub=provider.modelHub(f.store),one=hub.upsert({name:'owned first',model:config.model,baseUrl:model.baseUrl,requiresKey:false}); + if(mode==='connection')hub.select({type:'connection',id:one.id}); + else{ + const two=hub.upsert({name:'owned unused second',model:'owned-B',baseUrl:model.baseUrl,requiresKey:false}); + const pool=hub.pool({name:'owned overrun',members:[{connectionId:one.id},{connectionId:two.id}]});hub.select({type:'pool',id:pool.id});}} + const runtime=new Runtime(f.store,provider);t.after(()=>runtime.stopAll()); + runtime.start(f.session.id,'Read README without changing files.');await runtime.active.get(f.session.id).promise; + const run=f.store.core.run.get(f.session.runId); + assert.equal(model.requests.length,1);assert.equal(f.session.status,'paused');assert.equal(run.stopReason,'token_limit'); + assert.equal(run.consumption.length,1);assert.equal(run.usage.totalTokens,55017);assert.equal(run.usage.unknownRequests,0); + assert.equal(run.budgetPause.tokenAccounting.run.exceededTokens,20017);assert.equal(f.store.core.tool.size,0); + if(mode!=='legacy')assert.equal(provider.modelHub(f.store).view().attempts.length,1); + if(mode==='connection')assert.equal(run.budgetPause.modelFailure.httpStatus,503); +}); + +for(const mode of ['legacy','pool'])test('reported usage below the limit prevents an unaffordable '+mode+' retry before another dispatch',async t=>{ + const f=fixture(t),model=await compatibleFixture(t,()=>({status:503,body:{error:{message:'owned response below maximum'}, + usage:{prompt_tokens:45000,completion_tokens:100,total_tokens:45100}}})); + f.store.saveConfig({...config,baseUrl:model.baseUrl,maxRunTokens:70000,maxTaskTokens:1000000}); + const provider=new Provider({retryPolicy:{delays:[1,1]}}); + if(mode==='pool'){ + const hub=provider.modelHub(f.store),one=hub.upsert({name:'owned charged route',model:config.model,baseUrl:model.baseUrl,requiresKey:false}), + two=hub.upsert({name:'owned unused route',model:'owned-B',baseUrl:model.baseUrl,requiresKey:false}); + const pool=hub.pool({name:'owned remaining budget',members:[{connectionId:one.id},{connectionId:two.id}]});hub.select({type:'pool',id:pool.id}); + } + const runtime=new Runtime(f.store,provider);t.after(()=>runtime.stopAll()); + runtime.start(f.session.id,'Read README without changing files.');await runtime.active.get(f.session.id).promise; + const run=f.store.core.run.get(f.session.runId); + assert.equal(model.requests.length,1);assert.equal(f.session.status,'paused');assert.equal(run.stopReason,'token_limit'); + assert.equal(run.usage.totalTokens,45100);assert.equal(run.usage.unknownRequests,0);assert.equal(run.consumption.length,1); + assert.equal(run.budgetPause.remainingTokens,24900);assert.ok(run.budgetPause.tokenReservation.reservedTokens>24900); + assert.equal(run.budgetPause.modelFailure.httpStatus,503);assert.equal(run.budgetPause.tokenAccounting.run.exceededTokens,0); + if(mode==='pool')assert.equal(provider.modelHub(f.store).view().attempts.length,1); +}); + +test('native bridge seals calibrated refusal with original consumption and preserves versioned request verification',async t=>{ + const f=fixture(t),model=await compatibleFixture(t,()=>({...completion([call('native-owned-read','read_file',{path:'README.md'})]), + usage:{prompt_tokens:45000,completion_tokens:100,total_tokens:45100}})),context=controlledWorker(f); + context.provider.complete=Provider.prototype.complete.bind(context.provider); + f.store.saveConfig({...config,baseUrl:model.baseUrl,maxRunTokens:70000,maxTaskTokens:1000000}); + context.definition.budget.maxTokens=70000; + const group=await context.manager.plan(f.session,context.definition); + await (await context.manager.prepare(f.session,group.id,group.fingerprint)).promise;assert.equal(group.status,'ready',group.summary); + await (await context.manager.start(f.session,group.id,group.fingerprint)).promise; + const job=[...f.store.core.engine_run.values()].find(value=>value.collaborationId===group.id),session=f.store.session(job.sessionId),run=f.store.core.run.get(job.runId); + assert.equal(model.requests.length,1);assert.equal(group.status,'paused');assert.equal(run.stopReason,'engine_input_budget',JSON.stringify({budget:run.budget,usage:run.usage, + task:f.store.core.task.get(run.taskId).usage,requests:job.requests.map(value=>({accounted:value.accountedAttempts,observed:value.observedAttempts,max:value.maxAttempts,estimate:value.reservedPerAttempt,status:value.status})),rejections:job.inputRejections?.map(value=>({remaining:value.remainingTokens,reserved:value.tokenReservation?.reservedTokens})),error:job.error})); + assert.equal(job.calls.length,1);assert.equal(job.calls[0].status,'returned');assert.equal(job.requests[0].reservationVersion,1); + assert.doesNotThrow(()=>verifyEngineModelInput(f.store,session,job,job.requests[0])); + const refusal=job.inputRejections[0];assert.ok(refusal.tokenReservation.reservedTokens>refusal.remainingTokens); + assert.equal(refusal.tokenReservation.routes[0].sources[0].attemptId,run.consumption[0].attemptId); + assert.equal(run.usage.totalTokens,45100);assert.equal(run.usage.requests,1);assert.equal(run.usage.unknownRequests,0); +});