test: freeze AA live compatibility and delivery validation
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# AA:1.25.3 真实供应商、工程交付与原版 Purge 验证
|
||||
|
||||
用户于2026-11-05(Asia/Shanghai)明确要求验证真实供应商兼容、自主工程交付及Purge收益,并完整保留历史异常。新批次从干净cef636fd开始;正式安装来源1bd94856、1.25.3、PID36552及原Provider保持。旧Z3/Z2/Z/Y/W/V/T/R/P/O/H/E等均不续跑,不覆写其证据,不把新结果倒写历史。
|
||||
|
||||
## 冻结与次序
|
||||
|
||||
1. 只读冻结实际安装清单/源码身份、当前公开连接、原预算/选线、全表行SHA/全部导出/工件SHA/完整原尝试。用原Provider内部凭据发送;不读取、复制、解密、提取或重存任何Key,不重启宿主、不启用已停用连接。
|
||||
2. 对当前启用的DeepSeek、apikey-Claude、apikey GPT分别获取一次真实模型目录(产品原60秒/20页/1000项/4MB边界);记录实际型号和能力缺失,取消临时目录,不批量导入或保存连接。每条连接另做一次所属DNS/TLS诊断,仅观察,不以诊断成功替代模型API成功。
|
||||
3. 每条连接两个独立、无工具、单请求/单尝试预检:关闭推理、开启低强度推理。Chat OpenAI使用reasoning_effort=none/low;DeepSeek使用thinking.disabled或thinking.enabled+reasoning_effort=low;Messages使用thinking.disabled或thinking.enabled+budget_tokens1024、effort low。每次16K输入总预算、最多2048输出、320秒,严格要求上游型号匹配、完整答复包含随机nonce。开关参数只进入本次Run快照,不保存原连接。验证字段支持与实际响应分别报告;仅上游身份字段符合不认证权重。
|
||||
4. 单个新的Node自主工程样本:按固定优先级DeepSeek→apikey-Claude→apikey GPT,取关闭推理预检通过的第一条线路。目录缺失或开启推理失败分别保留,不阻断已通过关闭模式的工程测试;工程固定使用该关闭参数,不能失败后换线重跑。同原三模块/九测试/build/owner staged+unstaged+untracked保护,只给业务要求与通用流程,模型自己实现、封存变更、规划检查/Git;监督仅审查模型原提案、批准原检查及确认本地Git。
|
||||
5. Node真实闭环后新React/Python工程,再独立原版受限Purge AB/BA(off,on,on,off)四新工程。Purge1.1.78/Harness0.2.1-rc.2、purge-original-restricted-v1,固定源、模型、关闭推理参数、输入/预算/检查,逐请求证明原版实际加载和模型输入,确认所属进程树结束。不能用工作台独立提示实验代替原版。
|
||||
|
||||
## 预算、成功与停止
|
||||
|
||||
整批最多100次真实模型请求、700000报告tokens;每次派发前保留原宿主用量/未知预留检查。六次兼容预检每例最多1次/16000报告tokens。Node新样本固定24次/140000报告tokens/600000ms,16K上下文、2048输出与原1600保留预算;这个新预算不同旧8次样本,不作单因素性能比较。React/Python每例24次/140000,Purge每例24次/105000,均受总上限与剩余额度约束;不为后续阶段扩大总额。在途报告超额如实记录,不追加派发。
|
||||
|
||||
身份不符、网络/协议失败、未知消费、原预算/无进展停止、来源或用户配置变化均停止本例并封存;不改测试、加实现提示、人工代写、放宽期限或自动重放。配置写入如工程所需,通过原API临时修改身份/原生参数/预算/选择,finally按原语义恢复,仅允许已声明的单调修订/时间戳/观察健康变化;原凭据修订不变。外部编辑不覆盖,明确核对差异。
|
||||
|
||||
工程成功需原测试+build(React另类型/真实页面)、模型来源的完整变更集、实际应用、当前输入证据和本地已确认Git交付,原owner修改及暂存不变。审批监督次数、请求/用量/缓存/unknown/墙钟/错误/停止原因独立列出。Purge更早失败不算节省;只有四例与同成功标准成立才给描述性对比,不宣称统计显著或费用。
|
||||
|
||||
如Node未闭环,React/Python/Purge前置未满足,记录明确未执行与原因;不制造对照收益。发现确定产品缺陷先封存失败,修复/回归/发行和另冻实验独立,不在原样本中改变代码或预算。
|
||||
|
||||
## 封存与提交
|
||||
|
||||
冻结前驱动自检:本地隔离六组合已通过(`.qa/aa-driver-check-973a49f4-f5e8-432b-bc51-f6eaaefcfc33`,6次合成请求、真实请求0)。首账本接口404、次Messages夹具数组误作text的两份失败保持;只修验收驱动,没有改变产品响应完整性守卫。驱动语法与diff检查后本地提交,再开始实际冻结。
|
||||
|
||||
新驱动scripts/live-aa、证据.qa/live-validation-aa;write-once,不递归搜索全部.qa,不改旧封存。每阶段对账原状态、核对实际请求/响应SHA、回顾并本地提交,全部完成后离线验证新manifest及Git原字节。不委派、不推送。原网络异常、型号不符、空响应、length、重复读取、预留边界、真实交付及Purge未知各自保留,新观察只作追加结论。
|
||||
@@ -1,5 +1,9 @@
|
||||
# 汇总发现
|
||||
|
||||
AA驱动复核:模型账本实际由`GET /model-center`的attempts提供(最近100),没有`/model-center/attempts`接口;完整原账本仍只读metadata中的modelAttempts。首隔离失败未发送真实请求,修正另轮不覆盖。GPT原身份来源不为空,Node入口必须与冻结原配置比较而不能硬断言空来源;恢复保留原来源。
|
||||
|
||||
AA起步:干净cef636fd,正式1.25.3/PID36552/instance4c910dd5、空闲/窗口0;启用DeepSeek revision3/凭据1、apikey-Claude revision5/凭据3、apikey GPT93d revision7/凭据1。均16K/2048/SSE、record,原默认高性能池保持。新验证按LIVE-VALIDATION-AA-PLAN,不续跑旧样本。既有驱动Z固定1.25.2/旧修订和256输出,需适配新Run参数与2048推理;不能直接复用运行。首次宽文件搜索包含大量旧证据而截断,及若干猜测扁平server/model-reasoning/retest-h路径读取失败均未触真实请求;随后限定rg --files定位src/core/models及已有H驱动目录。
|
||||
|
||||
1.25.3最终交付:source1bd94856/候选20261104181218275,完整926/926来自6741510b,两个验收驱动差异的严格SHA证明保持。独立657输入重建22/22、源码78组/便携126组/真实安装12生命周期及103UI通过。正式283191ms保留PID36552/instance/Provider、75项目91会话6723记录14320事件及outbox和全原行/导出/工件/配置;网页两视口及原生回看后final exact、窗口0/9229关闭。主1635份138998818字节已封存;旧fidelity缺health、Messages字段testedVersion的原失败保持,封存后QA元数据永久修复+同便携6组10图/6个实际版本门检查,补验16份771613字节另封存。真实目录/模型请求均0,上游覆盖/稳定、自主交付、原网络/重复读取/真实预留边界/Purge未知不变。以下“待发行”等按历史时点理解。
|
||||
|
||||
上游模型目录:原Provider.models仅字符串/100条,模型中心手填;新目录入口复用原Hub/池,Chat/Responses/DeepSeek GET及Messages根/v1分页/鉴权。新型号通过源连接凭据引用,原对象与DPAPI块不重存;不同账号即使同地址型号也不去重。目录元数据为声明、未知不伪造、推理档位限制可选项,未接入执行功能仍显明。自动适配是能力/健康/并发加池策略;原Run冻结目录/凭据源代次。68相关+4升级、网页/原生各8组通过,真实推理0。细节docs/MODEL-CATALOG.md;正式和独立发行仍待完成。
|
||||
|
||||
+11
@@ -1,5 +1,16 @@
|
||||
# 统一执行进度
|
||||
|
||||
## AA 真实验证开始
|
||||
|
||||
- 新UUID隔离夹具`973a49f4-f5e8-432b-bc51-f6eaaefcfc33`六组合全部通过(6次本地模拟请求,真实目录/模型0);新驱动全部语法通过。增加每次预检独立离线核对原会话/导出/工件/元数据/2048原生参数与零工具的验证器;Node缺失outputTokenField仅允许规范化为文档默认max_tokens,原身份来源与参数恢复。未改产品或发行,无需重复产品完整安装。
|
||||
|
||||
- 隔离驱动第二轮DeepSeek关闭/开启两组通过,Claude关闭在夹具响应中把Messages文本数组误作text字符串,被产品协议守卫正确拒绝;原`.qa/aa-driver-check-4e646483-343f-4cf6-a906-c32a32fbd80f`保留。仅修正本地模拟服务的合法响应,不改产品协议判定,不触正式请求。
|
||||
|
||||
- 隔离驱动首轮完成DeepSeek-off的单请求/2048输出/参数/nonce/零工具检查后,因驱动读取不存在的`/model-center/attempts`接口失败。原失败保留在`.qa/aa-driver-check-5e95f790-38fe-4afc-a834-41e994c48173`;按实际`/model-center`的attempts字段修正,另开UUID验证六组合。正式上游请求仍0。审查同时修正Node对GPT原identityAliasSource的保留断言及沿用旧标题/错字,未更改原业务题与测试。
|
||||
|
||||
- 用户明确授权真实供应商兼容、自主工程交付与Purge收益验证,完整保留历史异常。planning-with-files接续,session-catchup无待同步输出;工作树cef636fd干净、正式1.25.3/PID36552空闲。读取旧计划/驱动和当前公开连接,无Key读取、上游目录或模型请求。
|
||||
- 新AA计划固定三路线目录/两种推理单请求、优先DeepSeek的单线路Node,Node闭环后React/Python/原版ABBA;整批100请求700000tokens,Node另冻24请求140000(不能与旧8次作单因素收益比较)。历史断言/旧失败/未知不变。
|
||||
|
||||
## 1.25.3 最终发行与保留状态升级
|
||||
|
||||
- 完整源码6741510b为926/926、零失败/取消/跳过;最终1bd94856仅两个QA驱动差异,所有产品/测试输入及22编译相同。最终候选20261104181218275已完成657输入独立离线重建22/22、三归档、真实便携、源码78组/便携126组、真实隔离安装12生命周期及安装副本14套103UI。
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { root, experiment, connectionId, readRoot, save, client, meta } from './common.mjs';
|
||||
assert.ok(experiment); assert.equal(fs.existsSync(path.join(root, 'catalog')), false);
|
||||
const original = readRoot(), { api, boot, close } = await client();
|
||||
let diagnosticId, previewId;
|
||||
try {
|
||||
const center = await api('/model-center'), connection = center.connections.find(c => c.id === connectionId);
|
||||
assert.equal(boot.version, '1.25.3'); assert.equal(connection.revision, original.connectionRevision);
|
||||
const attemptsBefore = meta('modelAttempts') ?? []; save('catalog/attempts-before.json', attemptsBefore);
|
||||
save('catalog/contract.json', { connection, requestLimit: 'one catalog operation, product pagination only, no retries', noConfigurationSave: true, modelRequests: 0 });
|
||||
try {
|
||||
const started = await api('/model-center/network-diagnostics', { id: connectionId, revision: connection.revision });
|
||||
diagnosticId = started.job.id; save('catalog/network-start.json', started);
|
||||
const deadline = Date.now() + 25000; let status;
|
||||
do { status = await api('/model-center/network-diagnostics'); if (!status.active) break; assert.ok(Date.now() < deadline); await delay(250); } while (true);
|
||||
const job = status.jobs.find(row => row.id === diagnosticId); save('catalog/network.json', job);
|
||||
assert.equal(job.httpRequests, 0); assert.equal(job.modelRequests, 0); assert.equal(job.credentialAccess, false);
|
||||
} catch (error) { save('catalog/network-error.json', { error: error.stack, response: error.response, noReplay: true }); }
|
||||
const requestId = randomUUID(), startedAt = new Date().toISOString(); previewId = requestId;
|
||||
try {
|
||||
const preview = await api('/model-center/catalog', { id: connectionId, revision: connection.revision, requestId }, 70000);
|
||||
save('catalog/preview.json', preview);
|
||||
save('catalog/result.json', { success: true, connectionId, startedAt, finishedAt: new Date().toISOString(), models: preview.entries.length,
|
||||
complete: preview.complete, selectedModel: connection.model, selectedModelListed: preview.entries.some(row => row.id === connection.model),
|
||||
declaredCapabilities: preview.entries.filter(row => Object.values(row.capabilityDeclarations).some(value => value !== 'unknown')).length,
|
||||
declaredReasoning: preview.entries.filter(row => row.efforts.length || row.modes.length || row.features.thinking !== 'unknown').length,
|
||||
normalizedPreviewOnly: true, rawHttpResponseArtifactAvailable: false, realModelRequests: 0, configurationSaved: false });
|
||||
} catch (error) { save('catalog/result.json', { success: false, connectionId, startedAt, finishedAt: new Date().toISOString(), error: error.message, status: error.status, response: error.response, automaticReplay: false, realModelRequests: 0, configurationSaved: false }); }
|
||||
if (previewId) save('catalog/cancel.json', await api('/model-center/catalog/cancel', { id: previewId }));
|
||||
const after = await api('/model-center');
|
||||
assert.deepEqual(after.selected, center.selected); assert.deepEqual(after.connections, center.connections); assert.deepEqual(after.pools, center.pools);
|
||||
assert.deepEqual(meta('modelAttempts') ?? [], attemptsBefore);
|
||||
save('catalog/restoration.json', { passed: true, connectionsSelectionPoolsUnchanged: true, modelAttemptsUnchanged: true, credentialsReadOrResaved: false });
|
||||
console.log(fs.readFileSync(path.join(root, 'catalog/result.json'), 'utf8'));
|
||||
} finally {
|
||||
if (diagnosticId) { const status = await api('/model-center/network-diagnostics'); if (status.active && ['running', 'pending'].includes(status.jobs.find(row => row.id === diagnosticId)?.status)) save('catalog/network-stop.json', await api('/model-center/network-diagnostics/' + diagnosticId + '/stop', {})); }
|
||||
await close();
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { DatabaseSync } from 'node:sqlite';
|
||||
import { Agent, fetch } from 'undici';
|
||||
export const origin = 'http://127.0.0.1:4318';
|
||||
export const batch = path.resolve('.qa/live-validation-aa');
|
||||
export const routes = [
|
||||
{ key: 'deepseek', id: '1a7a55a9-29f2-4b9f-8ea2-5a55afd7eacd', off: { thinking: { type: 'disabled' } }, on: { thinking: { type: 'enabled' }, reasoning_effort: 'low' } },
|
||||
{ key: 'claude', id: '9ae348a5-d0a6-42ed-8664-0da5f9effe34', off: { thinking: { type: 'disabled' } }, on: { thinking: { type: 'enabled', budget_tokens: 1024 }, output_config: { effort: 'low' } } },
|
||||
{ key: 'gpt', id: '93d0cf6d-4f32-4918-9da8-6f125aa8a6e5', off: { reasoning_effort: 'none' }, on: { reasoning_effort: 'low' } },
|
||||
];
|
||||
export const target = process.argv[2];
|
||||
export const experiment = routes.find(row => row.key === target);
|
||||
export const root = experiment ? path.join(batch, target) : batch;
|
||||
export const connectionId = experiment?.id;
|
||||
if (experiment && fs.existsSync(path.join(root, 'current.json'))) {
|
||||
const frozen = JSON.parse(fs.readFileSync(path.join(root, 'current.json'))).experiment;
|
||||
experiment.model = frozen.model; experiment.protocol = frozen.protocol;
|
||||
const connection = JSON.parse(fs.readFileSync(path.join(root, 'current.json'))).center.connections.find(c => c.id === connectionId);
|
||||
experiment.credentialRevision = connection.credentialRevision;
|
||||
}
|
||||
export const sha = value => createHash('sha256').update(value).digest('hex');
|
||||
export const saveAt = (directory, relative, value) => {
|
||||
const file = path.join(directory, relative); fs.mkdirSync(path.dirname(file), { recursive: true });
|
||||
fs.writeFileSync(file, typeof value === 'string' || Buffer.isBuffer(value) ? value : JSON.stringify(value, null, 2) + '\n', { flag: 'wx' });
|
||||
};
|
||||
export const save = (relative, value) => saveAt(root, relative, value);
|
||||
export const readRoot = () => JSON.parse(fs.readFileSync(path.join(root, 'current.json')));
|
||||
export const dataRoot = () => JSON.parse(fs.readFileSync(path.join(batch, 'current.json'))).host.identity.dataRoot;
|
||||
export function meta(name) {
|
||||
assert.ok(['config', 'modelRegistry', 'modelAttempts', 'modelNetworkDiagnostics'].includes(name));
|
||||
const db = new DatabaseSync(path.join(dataRoot(), 'workbench.sqlite'), { readOnly: true });
|
||||
try { const row = db.prepare('SELECT value FROM metadata WHERE key=?').get(name); return row ? JSON.parse(row.value) : null; }
|
||||
finally { db.close(); }
|
||||
}
|
||||
export async function client() {
|
||||
const dispatcher = new Agent({ connect: { autoSelectFamily: true, timeout: 5000 } }); let token;
|
||||
const api = async (route, input, timeout = 30000) => {
|
||||
const response = await fetch(origin + '/api' + route, { dispatcher,
|
||||
method: input === undefined ? 'GET' : 'POST', redirect: 'error', signal: AbortSignal.timeout(timeout),
|
||||
headers: { ...(token ? { 'X-Workbench-Token': token } : {}), 'Content-Type': 'application/json', Connection: 'close' },
|
||||
...(input === undefined ? {} : { body: JSON.stringify(input) }) });
|
||||
const value = await response.json();
|
||||
if (!response.ok) throw Object.assign(new Error(route + ': ' + (value.error ?? response.status)), { status: response.status, response: value });
|
||||
return value;
|
||||
};
|
||||
const boot = await api('/bootstrap'); token = boot.token;
|
||||
return { api, boot, close: () => dispatcher.close() };
|
||||
}
|
||||
export async function captureAttempts(api, attempts, relative) {
|
||||
const captures = [];
|
||||
for (const attempt of attempts) for (const kind of ['request', 'response']) {
|
||||
const id = attempt[kind + 'ArtifactId']; if (!id) continue;
|
||||
let content = '', offset = 0, page;
|
||||
do { page = await api('/artifacts/' + id + '?session=' + attempt.sessionId + '&offset=' + offset + '&limit=16000');
|
||||
assert.ok(page.content.length || !page.truncated); content += page.content; offset += page.content.length;
|
||||
} while (page.truncated);
|
||||
assert.equal(sha(content), page.sha256); assert.equal(page.sha256, attempt[kind + 'Sha256']);
|
||||
const name = `${relative}/${attempt.id}-${kind}.${kind === 'request' ? 'json' : attempt.streaming ? 'sse' : 'txt'}`;
|
||||
save(name, content); captures.push({ attemptId: attempt.id, kind, artifactId: id, path: name, sha256: sha(content), bytes: Buffer.byteLength(content) });
|
||||
}
|
||||
return captures;
|
||||
}
|
||||
export function batchUsage() {
|
||||
const before = new Set(JSON.parse(fs.readFileSync(path.join(batch, 'original-attempts.json'))).map(row => row.id));
|
||||
const attempts = (meta('modelAttempts') ?? []).filter(row => !before.has(row.id));
|
||||
return { requests: attempts.length, tokens: attempts.reduce((sum, row) => sum + (row.usage?.total_tokens ?? 0), 0), unknown: attempts.filter(row => row.usageStatus !== 'reported').length };
|
||||
}
|
||||
export function assertBatchBudget() { const usage = batchUsage(); assert.ok(usage.requests < 100 && usage.tokens < 700000, 'AA batch budget exhausted'); return usage; }
|
||||
@@ -0,0 +1,37 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { publicSnapshot } from '../retained-host-state.mjs';
|
||||
import { batch, routes, sha, client, saveAt } from './common.mjs';
|
||||
assert.equal(fs.existsSync(batch), false, 'New AA batch only, never replay.');
|
||||
assert.equal(execFileSync('git', ['status', '--porcelain'], { encoding: 'utf8', windowsHide: true }).trim(), '');
|
||||
const { api, boot, close } = await client();
|
||||
try {
|
||||
const host = await api('/host/status'), center = await api('/model-center');
|
||||
assert.equal(boot.version, '1.25.3'); assert.equal(host.identity.pid, 36552); assert.equal(host.activity.active, false); assert.equal(host.desktopWindows.count, 0);
|
||||
for (const row of routes) {
|
||||
const c = center.connections.find(c => c.id === row.id);
|
||||
assert.ok(c?.enabled && c.hasKey); assert.equal(c.streaming, true); assert.equal(c.contextLimit, 16000); assert.equal(c.maxOutputTokens, 2048);
|
||||
assert.equal(c.modelIdentityPolicy, 'record'); assert.deepEqual(c.modelIdentityAliases, []);
|
||||
}
|
||||
const candidate = JSON.parse(fs.readFileSync('releases/latest.json'));
|
||||
const manifest = fs.readFileSync(path.join(boot.runtimeRoot, 'MANIFEST.json'));
|
||||
assert.equal(sha(manifest), sha(fs.readFileSync(path.join(candidate.portableRoot, 'MANIFEST.json'))));
|
||||
const identity = JSON.parse(fs.readFileSync('docs/model-catalog-1.25.3-release-identity.json'));
|
||||
assert.equal(sha(manifest), identity.portableManifestSha256);
|
||||
for (const row of JSON.parse(fs.readFileSync('docs/model-catalog-1.25.3-evidence/release-verified/freeze.json')).compiled) {
|
||||
assert.equal(sha(fs.readFileSync(path.join(boot.runtimeRoot, row.path))), row.sha256);
|
||||
}
|
||||
const baseline = await publicSnapshot('http://127.0.0.1:4318', host.identity.dataRoot);
|
||||
const current = { host, config: boot.config, projects: boot.projects, center, baseline, sourceCommit: candidate.documentation.sourceCommit,
|
||||
reviewedHead: execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8', windowsHide: true }).trim(), runtimeManifestSha256: sha(manifest), observedAt: new Date().toISOString() };
|
||||
saveAt(batch, 'current.json', current);
|
||||
const { meta } = await import('./common.mjs'); saveAt(batch, 'original-attempts.json', meta('modelAttempts') ?? []);
|
||||
saveAt(batch, 'original-diagnostics.json', meta('modelNetworkDiagnostics') ?? []);
|
||||
for (const row of routes) {
|
||||
const connection = center.connections.find(c => c.id === row.id);
|
||||
saveAt(path.join(batch, row.key), 'current.json', { ...current, experiment: { ...row, model: connection.model, protocol: connection.protocol }, connectionRevision: connection.revision });
|
||||
}
|
||||
console.log(JSON.stringify({ success: true, version: boot.version, pid: host.identity.pid, projects: baseline.projects, sessions: baseline.exports.length, routes: routes.map(row => row.key), realModelRequests: 0 }));
|
||||
} finally { await close(); }
|
||||
@@ -0,0 +1,58 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import net from 'node:net';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { boundedProviderPreflight } from './payload.mjs';
|
||||
export async function inspectorClosed() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const socket = net.createConnection({ host: '127.0.0.1', port: 9229 }); socket.setTimeout(1500);
|
||||
socket.once('connect', () => { socket.destroy(); reject(Error('Inspector already open.')); });
|
||||
socket.once('error', e => { socket.destroy(); e.code === 'ECONNREFUSED' ? resolve(true) : reject(e); });
|
||||
socket.once('timeout', () => { socket.destroy(); reject(Error('Inspector state unknown.')); });
|
||||
});
|
||||
}
|
||||
class Protocol {
|
||||
constructor(socket) {
|
||||
this.socket = socket; this.nextId = 1; this.pending = new Map();
|
||||
socket.addEventListener('message', event => {
|
||||
const m = JSON.parse(String(event.data)); if (!m.id) return;
|
||||
const p = this.pending.get(m.id); if (!p) return; this.pending.delete(m.id); clearTimeout(p.timer);
|
||||
m.error ? p.reject(Error(m.error.message)) : p.resolve(m.result);
|
||||
});
|
||||
socket.addEventListener('close', () => { for (const p of this.pending.values()) { clearTimeout(p.timer); p.reject(Error('Inspector closed; outcome unknown.')); } this.pending.clear(); });
|
||||
}
|
||||
send(method, params, timeout = 10000) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const id = this.nextId++, timer = setTimeout(() => { this.pending.delete(id); reject(Error('Inspector timeout; do not replay.')); }, timeout);
|
||||
this.pending.set(id, { resolve, reject, timer }); this.socket.send(JSON.stringify({ id, method, params }));
|
||||
});
|
||||
}
|
||||
}
|
||||
export async function invokeProbe(input) {
|
||||
await inspectorClosed(); let socket, protocol, enabled = false;
|
||||
try {
|
||||
process._debugProcess(input.pid); enabled = true;
|
||||
let targets;
|
||||
for (let i = 0; i < 40 && !targets; i++) {
|
||||
await delay(100);
|
||||
try { targets = await (await fetch('http://127.0.0.1:9229/json/list', { signal: AbortSignal.timeout(700) })).json(); } catch {}
|
||||
}
|
||||
assert.equal(targets?.length, 1);
|
||||
assert.ok(targets[0].webSocketDebuggerUrl.startsWith('ws://127.0.0.1:9229/'));
|
||||
socket = new WebSocket(targets[0].webSocketDebuggerUrl);
|
||||
await new Promise((resolve, reject) => { socket.addEventListener('open', resolve, { once: true }); socket.addEventListener('error', reject, { once: true }); });
|
||||
protocol = new Protocol(socket);
|
||||
const identity = await protocol.send('Runtime.evaluate', { expression: 'process.pid', returnByValue: true });
|
||||
assert.equal(identity.result.value, input.pid);
|
||||
const expression = '(' + boundedProviderPreflight.toString() + ')(' + JSON.stringify(input) + ')';
|
||||
const result = await protocol.send('Runtime.evaluate', { expression, awaitPromise: true, returnByValue: true }, 345000);
|
||||
if (result.exceptionDetails) throw Error(result.exceptionDetails.exception?.description ?? result.exceptionDetails.text);
|
||||
return { result: result.result.value, expression };
|
||||
} finally {
|
||||
if (protocol && enabled) {
|
||||
await protocol.send('Runtime.evaluate', { expression: "setTimeout(() => process.getBuiltinModule('inspector').close(), 25); 'closing'", returnByValue: true });
|
||||
await delay(200);
|
||||
}
|
||||
socket?.close();
|
||||
await inspectorClosed();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { gitEnvironment } from '../../src/core/git-process.mjs';
|
||||
import { publicSnapshot, canonical } from './state.mjs';
|
||||
import { root, origin, dataRoot, connectionId, save, meta, client, sha, captureAttempts, readRoot, experiment } from './common.mjs';
|
||||
// Bounded real Node engineering with supervisor-in-the-loop review. Model
|
||||
// implements the three allowed modules autonomously; supervisor only approves
|
||||
// the original checks, reviews/applies the allowed change set and confirms Git.
|
||||
assert.equal(JSON.parse(fs.readFileSync(path.join(root, 'preflight-off/result.json'))).passed, true);
|
||||
assert.equal(fs.existsSync(path.join(root, 'node')), false, 'Never replay or overwrite a live sample.');
|
||||
const original = readRoot();
|
||||
experiment.model = original.experiment.model; experiment.protocol = original.experiment.protocol;
|
||||
const { api, boot, close } = await client();
|
||||
const allowed = ['src/amount.mjs', 'src/summary.mjs', 'src/report.mjs'];
|
||||
const budgetFields = ['contextLimit', 'maxSteps', 'maxOutputTokens', 'maxRunTokens', 'maxTaskTokens', 'maxRunDurationMs'];
|
||||
const connectionFields = ['id', 'name', 'brand', 'model', 'baseUrl', 'enabled', 'requiresKey', 'streaming', 'protocol',
|
||||
'capabilityDeclarations', 'contextLimit', 'maxOutputTokens', 'concurrency', 'nativeParameters', 'outputTokenField',
|
||||
'modelIdentityPolicy', 'modelIdentityAliases', 'modelIdentityAliasSource', 'parameterProfile'];
|
||||
const connectionInput = value => Object.fromEntries(connectionFields.filter(key => Object.hasOwn(value, key)).map(key => [key, value[key]]));
|
||||
const originalIdentity = value => ({ modelIdentityPolicy: value.modelIdentityPolicy ?? 'record',
|
||||
modelIdentityAliases: value.modelIdentityAliases ?? [], modelIdentityAliasSource: value.modelIdentityAliasSource ?? '' });
|
||||
let before, centerBefore, publicMetadata, originalAttempts, connection, configured, project, session, state, taskId, route, budgetWritten = false, selectionWritten = false;
|
||||
let result, delivery, succeeded = false, failure;
|
||||
const approvals = [], applications = [], continuations = [], observations = [], start = Date.now();
|
||||
const sourceRoot = path.join(root, 'node/expense-project');
|
||||
const git = args => execFileSync('git', ['--no-pager', '--no-optional-locks', '-C', sourceRoot, ...args], { encoding: 'utf8', windowsHide: true, env: gitEnvironment() });
|
||||
let protectedBytes, stagedBefore;
|
||||
const protect = () => {
|
||||
for (const [name, bytes] of Object.entries(protectedBytes)) assert.deepEqual(fs.readFileSync(path.join(sourceRoot, name)), bytes, name);
|
||||
assert.equal(git(['show', ':owner-staged.txt']), stagedBefore, 'Preserve original index and working-copy states.');
|
||||
};
|
||||
const waitFor = async (predicate, timeout = 30000) => {
|
||||
const deadline = Date.now() + timeout;
|
||||
while (!await predicate()) { assert.ok(Date.now() < deadline, 'Local operation deadline exceeded; verify outcome, never replay.'); await delay(350); }
|
||||
};
|
||||
const review = async (kind, value) => {
|
||||
const relative = `node/reviews/${kind}-${applications.length + 1}`;
|
||||
const fingerprint = value.fingerprint ?? sha(JSON.stringify(value));
|
||||
save(relative + '/input.json', { ...value, fingerprint });
|
||||
console.log(JSON.stringify({ reviewRequired: true, kind, path: relative + '/input.json', fingerprint }));
|
||||
const decisionPath = path.join(root, relative, 'decision.json');
|
||||
await waitFor(async () => fs.existsSync(decisionPath), Math.max(1, 600000 - (Date.now() - start)));
|
||||
const decision = JSON.parse(fs.readFileSync(decisionPath));
|
||||
assert.equal(decision.approve, true, 'Supervisor declined the original model proposal.');
|
||||
assert.ok(typeof decision.reason === 'string' && decision.reason.length > 20, 'Record substantive review.');
|
||||
assert.equal(decision.fingerprint, fingerprint);
|
||||
return decision;
|
||||
};
|
||||
const remainingRun = async () => {
|
||||
const { assertBatchBudget } = await import('./common.mjs'); assertBatchBudget();
|
||||
const requests = state?.execution.task.usage.requests ?? 0;
|
||||
assert.ok(requests < 24, 'The sealed 24-request budget is exhausted.');
|
||||
const budgets = { contextLimit: 16000, maxOutputTokens: 2048, maxSteps: 24 - requests,
|
||||
maxRunTokens: 140000, maxTaskTokens: 140000, maxRunDurationMs: 600000 };
|
||||
await api('/model-center/runtime-budgets', budgets); budgetWritten = true;
|
||||
continuations.push({ beforeRequests: requests, remainingSteps: budgets.maxSteps, at: new Date().toISOString() });
|
||||
};
|
||||
try {
|
||||
assert.equal(boot.version, '1.25.3');
|
||||
const host = await api('/host/status'); assert.equal(host.identity.pid, 36552); assert.equal(host.activity.active, false);
|
||||
before = await publicSnapshot(origin, dataRoot()); save('node/before.json', before);
|
||||
centerBefore = await api('/model-center'); connection = centerBefore.connections.find(value => value.id === connectionId);
|
||||
assert.equal(connection.enabled, true); assert.equal(connection.hasKey, true); assert.equal(connection.revision, original.connectionRevision);
|
||||
assert.equal(connection.model, experiment.model); assert.equal(connection.protocol, experiment.protocol); assert.equal(connection.contextLimit, 16000);
|
||||
assert.deepEqual(originalIdentity(connection), originalIdentity(original.center.connections.find(value => value.id === connectionId)));
|
||||
publicMetadata = { config: meta('config'), registry: meta('modelRegistry') };
|
||||
save('node/diagnostics-before.json', meta('modelNetworkDiagnostics') ?? []);
|
||||
save('node/original-public-metadata.json', publicMetadata);
|
||||
originalAttempts = meta('modelAttempts') ?? []; save('node/original-attempts.json', originalAttempts);
|
||||
save('node/contract.json', { kind: 'new-real-Node-engineering', version: boot.version, connectionId, connection,
|
||||
maxRequests: 24, maxTokens: 140000, contextLimit: 16000, maxOutputTokens: 2048, maxDurationMs: 600000,
|
||||
allowed, checks: ['test', 'build'], expectedTests: 9, commands: 'Only model-planned npm test/build, no arbitrary run_command',
|
||||
review: 'Model seals all three files, supervisor reviews and applies; model plans Git, supervisor confirms.',
|
||||
continuation: 'Only review-action continuation of the same task with remaining steps, no extra implementation prompts.',
|
||||
originalImplementationSupplied: false, automaticRetry: false, poolSwitching: false,
|
||||
originalSelected: centerBefore.selected, originalBudgets: Object.fromEntries(budgetFields.map(key => [key, boot.config[key]])),
|
||||
restore: 'Original budget values, native parameters, identity source and selection restored. Original absent fields may become explicit API defaults; raw pre-state retained and only declared default normalization accepted. Revision counters remain monotonic.' });
|
||||
fs.cpSync('benchmarks/live-delivery-v1/node', sourceRoot, { recursive: true, errorOnExist: true });
|
||||
fs.writeFileSync(path.join(sourceRoot, 'README.md'), '# Expense report acceptance\nRepair the three source modules using the immutable original nine tests; do not edit checks or owner files.\n', { flag: 'wx' });
|
||||
fs.writeFileSync(path.join(sourceRoot, '.gitignore'), 'dist/\n.awb-*\nnode_modules/\n', { flag: 'wx' });
|
||||
for (const name of ['owner-staged.txt', 'owner-unstaged.txt']) fs.writeFileSync(path.join(sourceRoot, name), 'Owner original baseline\n', { flag: 'wx' });
|
||||
git(['init', '--initial-branch=main']); git(['config', 'core.autocrlf', 'false']);
|
||||
git(['config', 'user.name', 'Workbench QA']); git(['config', 'user.email', 'qa@workbench.invalid']);
|
||||
git(['add', '--', '.']); git(['commit', '--no-gpg-sign', '-m', 'Seal expense engineering baseline and original checks']);
|
||||
fs.writeFileSync(path.join(sourceRoot, 'owner-staged.txt'), 'Owner staged change\n'); git(['add', '--', 'owner-staged.txt']);
|
||||
stagedBefore = git(['show', ':owner-staged.txt']);
|
||||
fs.writeFileSync(path.join(sourceRoot, 'owner-staged.txt'), 'Owner working copy differs from staging\n');
|
||||
fs.writeFileSync(path.join(sourceRoot, 'owner-unstaged.txt'), 'Owner unstaged change\n');
|
||||
fs.writeFileSync(path.join(sourceRoot, 'owner-untracked.txt'), 'Owner untracked change\n', { flag: 'wx' });
|
||||
const protectedNames = ['README.md', '.gitignore', 'package.json', 'test/expense.test.mjs', 'scripts/build.mjs', 'owner-staged.txt', 'owner-unstaged.txt', 'owner-untracked.txt'];
|
||||
protectedBytes = Object.fromEntries(protectedNames.map(name => [name, fs.readFileSync(path.join(sourceRoot, name))]));
|
||||
const sourceFiles = [...protectedNames, ...allowed];
|
||||
for (const name of sourceFiles) save('node/source-before/' + name, fs.readFileSync(path.join(sourceRoot, name)));
|
||||
save('node/source-identity.json', { head: git(['rev-parse', 'HEAD']).trim(), status: git(['status', '--porcelain=v1']), stagedBefore,
|
||||
files: Object.fromEntries(sourceFiles.map(name => [name, sha(fs.readFileSync(path.join(sourceRoot, name)))])) });
|
||||
project = await api('/projects', { root: sourceRoot }); session = await api('/sessions', { projectId: project.id }); route = '/sessions/' + session.id;
|
||||
save('node/owned.json', { project, session });
|
||||
configured = await api('/model-center/connections', { ...connectionInput(connection), revision: connection.revision,
|
||||
nativeParameters: experiment.off, modelIdentityPolicy: 'require-match', modelIdentityAliases: [], modelIdentityAliasSource: '' });
|
||||
assert.equal(configured.credentialRevision, connection.credentialRevision); assert.equal(configured.hasKey, true);
|
||||
save('node/connection-frozen.json', configured);
|
||||
await remainingRun(); await api('/model-center/select', { type: 'connection', id: connectionId }); selectionWritten = true;
|
||||
const goal = '修复这个Node费用报表工程的amount、summary、report三个模块。金额用精确分单位处理,接受测试约定的十进制字符串、有限数字、退款、及空白;拒绝格式错误、超精度和不安全金额。按独立币种规范化、排序汇总并保留原数组,计数准确、安全求和,报表两位小数含负分金额。以原九项测试为完整验收约定。'
|
||||
+ '先inspect_project,再plan_checks选择["test","build"]并run_checks保留真实基线失败,读取原测试和相关源码。只允许修改src/amount.mjs、src/summary.mjs、src/report.mjs;不改测试、配置、README、构建脚本、owner文件或原暂存,不使用run_command,不直接提交Git。'
|
||||
+ 'propose_file/propose_patch准备完整三个模块后,用prepare_change_set一次封存全部三文件,等待审查应用。应用后继续同一个Task,重新plan_checks/run_checks,通过原9项测试和build后,用plan_delivery为三个文件准备本地Git预览并等待确认。模型必须自己实现,禁止用口头完成替代检查。预算最多24次模型请求、140000tokens,输出2048,上下文16K,请合并独立工具调用并避免重复观察。';
|
||||
save('node/goal.txt', goal); protect();
|
||||
await api(route + '/messages', { content: goal });
|
||||
while (Date.now() - start < 600000) {
|
||||
state = await api(route); taskId ??= state.execution.task.id; assert.equal(state.execution.task.id, taskId); protect();
|
||||
const usage = state.execution.task.usage; assert.ok(usage.requests <= 24); assert.ok(usage.totalTokens <= 140000);
|
||||
const progress = { status: state.status, requests: usage.requests, tokens: usage.totalTokens, lastTool: state.events.filter(event => event.kind === 'tool').at(-1)?.tool,
|
||||
checks: state.execution.checks.map(job => job.status), elapsedSeconds: Math.round((Date.now() - start) / 1000) };
|
||||
if (JSON.stringify({ ...progress, elapsedSeconds: 0 }) !== JSON.stringify({ ...observations.at(-1), elapsedSeconds: 0 })) { observations.push(progress); console.log(JSON.stringify(progress)); }
|
||||
for (const change of state.changes.filter(change => ['pending', 'conflict'].includes(change.status))) assert.ok(allowed.includes(change.path), 'Out-of-scope proposal: ' + change.path);
|
||||
for (const approval of state.approvals.filter(item => item.status === 'pending')) {
|
||||
const plan = state.execution.plans.find(value => value.id === approval.checkPlanId);
|
||||
const allow = !!plan && plan.taskId === taskId && plan.checks.length === 2 && plan.checks.every(check => ['test', 'build'].includes(check.name) && !check.lifecycle?.length)
|
||||
&& approval.cwd === sourceRoot && approval.command === plan.checks.map(check => check.command).join('\n');
|
||||
approvals.push({ approvalId: approval.id, planId: plan?.id, command: approval.command, allow });
|
||||
await api(route + '/approvals/' + approval.id, { allow }); assert.equal(allow, true, 'Only the sealed original checks are authorized.');
|
||||
}
|
||||
if (['running', 'awaiting_approval', 'stopping'].includes(state.status)) { await delay(650); continue; }
|
||||
if (state.status === 'failed') throw new Error(state.failure?.message ?? 'Actual-model task failed');
|
||||
const plan = state.execution.deliveryPlans?.findLast(value => value.status === 'planned' && value.taskId === taskId);
|
||||
if (plan) {
|
||||
assert.deepEqual(plan.paths.slice().sort(), allowed.slice().sort()); const requests = usage.requests;
|
||||
assert.ok(state.events.some(event => event.tool === 'plan_delivery' && event.phase === 'completed'));
|
||||
const passing = state.execution.checks.filter(job => job.status === 'passed').at(-1);
|
||||
assert.equal(passing?.results.find(value => value.name === 'test')?.tests?.passed, 9);
|
||||
await review('git', { plan, fingerprint: plan.fingerprint, checks: state.execution.checks, source: state.changes,
|
||||
status: git(['status', '--porcelain=v1']), stagedBefore, stagedAfter: git(['show', ':owner-staged.txt']) });
|
||||
await api(route + '/delivery-jobs', { planId: plan.id });
|
||||
await waitFor(async () => { state = await api(route); delivery = state.execution.deliveries.at(-1); return !!delivery?.reportArtifactId; });
|
||||
assert.equal(delivery.status, 'completed'); assert.equal(state.execution.task.usage.requests, requests); break;
|
||||
}
|
||||
const pending = state.changes.filter(change => change.status === 'pending');
|
||||
const batch = state.execution.changeSets?.findLast(value => value.taskId === taskId && value.status === 'prepared');
|
||||
if (pending.length) {
|
||||
assert.ok(state.execution.checks.some(job => job.status === 'failed' && job.results.some(value => value.tests?.failed > 0)), 'Model must preserve the original failing tests.');
|
||||
assert.equal(batch?.source, 'agent', 'Model must seal its own complete change set.');
|
||||
assert.deepEqual(batch.files.map(file => file.changeId).sort(), pending.map(change => change.id).sort());
|
||||
if (!applications.length) assert.deepEqual(batch.files.map(file => file.path).sort(), allowed.slice().sort());
|
||||
assert.ok(applications.length < 3);
|
||||
await review('changes', { fingerprint: batch.fingerprint, batch, changes: pending, checks: state.execution.checks });
|
||||
await api(route + '/change-sets/' + batch.id + '/apply', { fingerprint: batch.fingerprint });
|
||||
await waitFor(async () => (await api(route + '/change-sets/' + batch.id)).status === 'completed');
|
||||
for (const change of pending) assert.equal(sha(fs.readFileSync(path.join(sourceRoot, change.path))), change.sha256);
|
||||
applications.push({ id: batch.id, fingerprint: batch.fingerprint, paths: batch.files.map(file => file.path), source: batch.source }); protect();
|
||||
await remainingRun(); await api(route + '/continue-reviewed', {}); continue;
|
||||
}
|
||||
throw new Error('Model stopped before verified Git delivery: ' + state.status + '/' + state.execution.run.stopReason);
|
||||
}
|
||||
assert.ok(delivery, 'The sealed engineering deadline ended without complete delivery.');
|
||||
const passing = state.execution.checks.filter(job => job.status === 'passed').at(-1); assert.ok(passing);
|
||||
const tests = passing.results.find(value => value.name === 'test')?.tests; assert.equal(tests?.total, 9); assert.equal(tests.passed, 9); assert.equal(tests.failed, 0);
|
||||
assert.deepEqual(git(['diff-tree', '--no-commit-id', '--name-only', '-r', 'HEAD']).trim().split(/\r?\n/).sort(), allowed.slice().sort()); protect();
|
||||
succeeded = true;
|
||||
result = { success: true, commit: delivery.commit, tests, delivery, checkJobId: passing.id };
|
||||
} catch (error) {
|
||||
failure = { name: error.name, message: error.message, stack: error.stack, at: new Date().toISOString(), automaticReplay: false };
|
||||
result = { success: false, failure }; process.exitCode = 1;
|
||||
} finally {
|
||||
if (route) {
|
||||
state = await api(route);
|
||||
if (['running', 'stopping', 'awaiting_approval'].includes(state.status)) { await api(route + '/stop', {}); state = await api(route); }
|
||||
taskId ??= state.execution.task?.id;
|
||||
save('node/session.json', state); save('node/export.json', await api(route + '/export'));
|
||||
const attempts = (meta('modelAttempts') ?? []).filter(value => value.sessionId === session.id); save('node/attempts.json', attempts);
|
||||
const captures = await captureAttempts(api, attempts, 'node/captures');
|
||||
const identitiesValid = attempts.every(value => value.connectionId === connectionId && value.poolId === null && value.ordinal === 1
|
||||
&& value.status === 'completed' && value.modelIdentity?.status === 'matched' && value.modelIdentity.complete === true && value.modelIdentity.policy === 'require-match');
|
||||
result = { ...result, version: boot.version, sessionId: session.id, projectId: project.id, taskId, sourceRoot,
|
||||
connectionId, requestedModel: experiment.model, usage: state.execution.task?.usage, state: state.status, stopReason: state.execution.run?.stopReason,
|
||||
attempts: attempts.length, identitiesValid, captures, approvals, applications, continuations, observations,
|
||||
modelImplementationSuppliedBySupervisor: false, modelWeightsAuthenticated: false, originalPurgeBenefitTested: false,
|
||||
elapsedSeconds: Math.round((Date.now() - start) / 1000) };
|
||||
if (succeeded) assert.equal(identitiesValid, true);
|
||||
save('node/model-outcome.json', result);
|
||||
for (const name of allowed) save('node/source-after/' + name, fs.readFileSync(path.join(sourceRoot, name)));
|
||||
save('node/git-final.json', { head: git(['rev-parse', 'HEAD']).trim(), status: git(['status', '--porcelain=v1']), stagedBefore, stagedAfter: git(['show', ':owner-staged.txt']), diff: git(['diff', 'HEAD', '--', ...allowed]) });
|
||||
}
|
||||
const host = await api('/host/status'); assert.equal(host.activity.active, false, 'Do not overwrite another active task.');
|
||||
if (configured) {
|
||||
const center = await api('/model-center'), current = center.connections.find(value => value.id === connectionId);
|
||||
assert.equal(current.revision, configured.revision, 'Do not overwrite an external connection edit.');
|
||||
const restored = await api('/model-center/connections', { ...connectionInput(connection), nativeParameters: connection.nativeParameters ?? {}, ...originalIdentity(connection), revision: current.revision });
|
||||
assert.equal(restored.credentialRevision, connection.credentialRevision); save('node/connection-restored.json', restored);
|
||||
assert.deepEqual(originalIdentity(restored), originalIdentity(connection));
|
||||
}
|
||||
if (budgetWritten) await api('/model-center/runtime-budgets', Object.fromEntries(budgetFields.map(key => [key, boot.config[key]])));
|
||||
if (selectionWritten) await api('/model-center/select', centerBefore.selected);
|
||||
const after = await publicSnapshot(origin, dataRoot()); save('node/after.json', after);
|
||||
if (before) {
|
||||
for (const item of before.exports) assert.deepEqual(after.exports.find(value => value.id === item.id), item, 'Original export: ' + item.id);
|
||||
for (const table of ['sessions', 'records', 'events', 'outbox']) {
|
||||
const rows = new Map(after.database.tables[table].rows.map(row => [row.id, row]));
|
||||
for (const row of before.database.tables[table].rows) assert.deepEqual(rows.get(row.id), row, 'Original ' + table + '/' + row.id);
|
||||
}
|
||||
const artifactMap = new Map(after.persistent.artifacts.map(value => [value.path, value]));
|
||||
for (const item of before.persistent.artifacts) assert.deepEqual(artifactMap.get(item.path), item, 'Original artifact: ' + item.path);
|
||||
const finalBoot = await api('/bootstrap'), normalized = { ...finalBoot.config, configRevision: boot.config.configRevision };
|
||||
assert.deepEqual(canonical(normalized), canonical(boot.config));
|
||||
assert.deepEqual(finalBoot.projects.filter(value => value.id !== project?.id), boot.projects, 'Every original public project, excluding only the new owned sample.');
|
||||
assert.deepEqual(after.modelCenter.selected, centerBefore.selected); assert.deepEqual(after.modelCenter.pools, centerBefore.pools);
|
||||
for (const item of centerBefore.connections.filter(value => ![connectionId, 'legacy-default'].includes(value.id))) assert.deepEqual(after.modelCenter.connections.find(value => value.id === item.id), item);
|
||||
const finalMetadata = { config: meta('config'), registry: meta('modelRegistry') };
|
||||
save('node/final-public-metadata.json', finalMetadata);
|
||||
const configNormalized = { ...finalMetadata.config, configRevision: publicMetadata.config.configRevision };
|
||||
assert.deepEqual(canonical(configNormalized), canonical(publicMetadata.config), 'Raw public disk config except its monotonic revision.');
|
||||
const registryNormalized = structuredClone(finalMetadata.registry);
|
||||
assert.ok(registryNormalized.revision > publicMetadata.registry.revision);
|
||||
registryNormalized.revision = publicMetadata.registry.revision;
|
||||
const originalConnection = publicMetadata.registry.connections.find(value => value.id === connectionId);
|
||||
const finalConnection = registryNormalized.connections.find(value => value.id === connectionId);
|
||||
assert.equal(finalConnection.revision, originalConnection.revision + 2);
|
||||
finalConnection.revision = originalConnection.revision; finalConnection.updatedAt = originalConnection.updatedAt;
|
||||
const registryBefore = structuredClone(publicMetadata.registry);
|
||||
const normalizedBeforeConnection = registryBefore.connections.find(value => value.id === connectionId);
|
||||
if (!Object.hasOwn(normalizedBeforeConnection, 'outputTokenField') && Object.hasOwn(finalConnection, 'outputTokenField')) assert.equal(finalConnection.outputTokenField, 'max_tokens', 'Only the documented Chat/Messages default can be normalized.');
|
||||
const identityNormalization = Object.fromEntries(Object.entries(originalIdentity(normalizedBeforeConnection)).filter(([key]) => !Object.hasOwn(normalizedBeforeConnection, key)));
|
||||
const normalization = { ...identityNormalization, ...(!Object.hasOwn(normalizedBeforeConnection, 'nativeParameters') ? { nativeParameters: {} } : {}), ...(!Object.hasOwn(normalizedBeforeConnection, 'outputTokenField') && Object.hasOwn(finalConnection, 'outputTokenField') ? { outputTokenField: finalConnection.outputTokenField } : {}) };
|
||||
Object.assign(normalizedBeforeConnection, normalization);
|
||||
save('node/declared-identity-default-normalization.json', { connectionId, added: normalization,
|
||||
originalRawStatePreserved: true, originalSemanticsRestored: true, reason: 'The public partial-update API retains missing fields; restore explicit original default semantics rather than silently retaining require-match.' });
|
||||
for (const value of [registryBefore, registryNormalized]) value.connections.sort((a, b) => a.id.localeCompare(b.id));
|
||||
assert.deepEqual(canonical(registryNormalized), canonical(registryBefore), 'Registry except declared revision/timestamp/order and explicit original identity defaults.');
|
||||
const finalAttempts = meta('modelAttempts') ?? [];
|
||||
for (const item of originalAttempts) assert.deepEqual(finalAttempts.find(value => value.id === item.id), item, 'Original complete model attempt: ' + item.id);
|
||||
const metadataRows = new Map(after.database.tables.metadata.rows.map(row => [row.id, row]));
|
||||
const diagnosticsBefore = JSON.parse(fs.readFileSync(path.join(root, 'node/diagnostics-before.json')));
|
||||
const diagnosticsAfter = meta('modelNetworkDiagnostics') ?? [];
|
||||
assert.deepEqual(diagnosticsAfter.slice(0, diagnosticsBefore.length), diagnosticsBefore, 'Every original diagnostic field and order.');
|
||||
save('node/diagnostics-after.json', { originalExact: true, jobs: diagnosticsAfter, externalAdditions: diagnosticsAfter.slice(diagnosticsBefore.length) });
|
||||
for (const row of before.database.tables.metadata.rows.filter(row => !['config', 'modelRegistry', 'modelAttempts', 'modelNetworkDiagnostics'].includes(row.id))) assert.deepEqual(metadataRows.get(row.id), row);
|
||||
save('node/restoration.json', { passed: true, originalExports: before.exports.length, originalArtifactFiles: before.persistent.artifacts.length,
|
||||
originalRowsExact: ['sessions', 'records', 'events', 'outbox'], budgetsAndSelectionRestored: true, poolsUnchanged: true,
|
||||
originalPublicProjectsExact: true, originalAttemptsExact: originalAttempts.length, rawConfigExceptRevisionExact: true,
|
||||
registryExceptDeclaredRevisionTimestampOrderAndIdentityDefaultsExact: true, identityDefaultNormalization: normalization, otherMetadataRowsExact: true,
|
||||
originalCredentialRevision: connection?.credentialRevision, finalCredentialRevision: after.modelCenter.connections.find(value => value.id === connectionId)?.credentialRevision,
|
||||
changedOnlyMonotonicRevisionsObservedHealthAndDeclaredIdentityDefaults: true, pid: after.pid, version: after.version, noKeyExtractedOrResaved: true });
|
||||
}
|
||||
save('node/result.json', result ?? { success: false, failure, noModelDispatch: true }); await close();
|
||||
}
|
||||
console.log(JSON.stringify({ success: result.success, attempts: result.attempts, usage: result.usage, state: result.state,
|
||||
stopReason: result.stopReason, commit: result.commit, tests: result.tests, error: result.failure?.message }));
|
||||
@@ -0,0 +1,87 @@
|
||||
// One-shot QA adapter. Runs inside the existing formal host via the inspector;
|
||||
// never reads a credential value. Mirrors the sealed R preflight payload.
|
||||
export async function boundedProviderPreflight(input) {
|
||||
const ensure = (value, message) => { if (!value) throw new Error(message); };
|
||||
ensure(process.pid === input.pid, 'Unexpected host PID.');
|
||||
const servers = process._getActiveHandles().filter(h => h?.listening && h.address?.()?.address === '127.0.0.1' && h.address?.()?.port === input.port);
|
||||
ensure(servers.length === 1, 'Expected one owned loopback server.');
|
||||
const context = servers[0].workbenchHostAuthority?.context;
|
||||
ensure(context?.provider && context?.store && context?.runtime, 'Original context missing.');
|
||||
const { store, provider, runtime } = context;
|
||||
ensure(runtime.appDirectory === input.runtimeRoot, 'Loaded runtime changed.');
|
||||
ensure(![...store.sessions.values()].some(s => ['running', 'stopping', 'awaiting_approval'].includes(s.status)), 'Host is busy.');
|
||||
const session = store.session(input.sessionId), project = store.project(session.projectId);
|
||||
ensure(session.projectId === input.projectId && session.messages.length === 0 && !session.runId, 'Probe session must be new and owned.');
|
||||
const hub = provider.modelHub(store);
|
||||
const connection = hub.view().connections.find(c => c.id === input.connectionId);
|
||||
ensure(connection?.hasKey && connection.credentialRevision === input.credentialRevision, 'Original saved credential revision changed.');
|
||||
ensure(connection.enabled && connection.protocol === input.protocol && (connection.modelIdentityPolicy ?? 'record') === 'record', 'Probe protocol/auth/identity mismatch.');
|
||||
ensure(connection.baseUrl === input.baseUrl && connection.model === input.model && connection.revision === input.connectionRevision, 'Connection changed.');
|
||||
const config = hub.prepare({ ...store.config, mode: 'compatible', maxSteps: 1, maxOutputTokens: 2048,
|
||||
contextLimit: 16000, maxRunTokens: 16000, maxTaskTokens: 16000, maxRunDurationMs: 320000,
|
||||
modelResponseTimeoutMs: 300000, modelStreamIdleTimeoutMs: 120000 }, { type: 'connection', id: input.connectionId });
|
||||
config.nativeParameters = input.nativeParameters;
|
||||
config.modelRouting.members = config.modelRouting.members.map(member => ({ ...member, nativeParameters: input.nativeParameters, modelIdentityPolicy: 'require-match', modelIdentityAliases: [], modelIdentityAliasSource: '' }));
|
||||
ensure(config.modelRouting.members.length === 1 && config.modelRouting.maxAttempts === 1 && config.maxOutputTokens === 2048, 'One-request hard limits missing.');
|
||||
globalThis.__providerAAProbeOnce ??= new Set();
|
||||
ensure(!globalThis.__providerAAProbeOnce.has(input.sessionId), 'This sample cannot be replayed.');
|
||||
globalThis.__providerAAProbeOnce.add(input.sessionId);
|
||||
session.title = input.title;
|
||||
const run = store.beginRun(session, input.prompt, false, config, { forceNewTask: true });
|
||||
store.message(session, 'user', input.prompt);
|
||||
session.providerMessages.push({ role: 'user', content: input.prompt });
|
||||
const step = store.beginStep(session, run), planner = runtime.requestPlanner();
|
||||
let plan, response, diagnostic = null, requestEvent, receipts = 0, wireStarts = 0;
|
||||
const wireIds = new Set();
|
||||
const signal = AbortSignal.timeout(320000);
|
||||
let stopReason = 'identity_probe_complete';
|
||||
try {
|
||||
plan = planner.prepare(session, run, config, { system: 'This is a connection identity preflight. Reply to the user exactly. No tools are available. Do not perform any project work.', tools: [], stepId: step.id });
|
||||
const variants = plan.body.kind === 'model-route-plan' ? plan.body.variants.map(v => v.body) : [plan.body];
|
||||
ensure(variants.length === 1 && variants.every(b => !(b.tools?.length) && b.max_tokens === 2048 && b.model === input.model), 'Wire request violates no-tool contract.');
|
||||
ensure(plan.record.tokenReservation.reservedTokens <= 16000, 'Reservation exceeds sealed budget.');
|
||||
const progress = event => {
|
||||
if (event.consumption) {
|
||||
receipts += Number(store.recordConsumption(session, run, { ...event.consumption, requestId: plan.record.id }, plan.record.tokenReservation.reservedTokens));
|
||||
return;
|
||||
}
|
||||
if (event.phase === 'running' && event.wireAttemptId) { wireIds.add(event.wireAttemptId); wireStarts = wireIds.size; ensure(wireStarts === 1, 'No retries permitted.'); }
|
||||
requestEvent ??= store.event(session, { kind: 'model', source: 'model_connection', label: input.title, phase: 'running', runId: run.id, contextPlanId: plan.record.id, startedAt: new Date().toISOString() });
|
||||
if (event.modelAttemptId) requestEvent.modelAttemptId = event.modelAttemptId;
|
||||
if (event.connectionId) requestEvent.connectionId = event.connectionId;
|
||||
if (event.model) requestEvent.actualModel = event.model;
|
||||
if (event.modelIdentity) requestEvent.modelIdentity = event.modelIdentity;
|
||||
if (event.requestUrl) requestEvent.requestUrl = event.requestUrl;
|
||||
if (event.diagnostic) requestEvent.diagnostic = event.diagnostic;
|
||||
if (event.stream) requestEvent.stream = event.stream;
|
||||
if (event.phase) requestEvent.phase = event.phase;
|
||||
plan.record.requestAttempts = wireStarts; step.modelAttempts = wireStarts;
|
||||
store.save(session);
|
||||
};
|
||||
response = await provider.complete(config, plan.messages, project, signal, progress, plan.record, [], { attempts: 1, maxBudgetAttempts: 1 });
|
||||
ensure(!response.message.tool_calls?.length, 'Unsolicited tools are never executed.');
|
||||
ensure(receipts === 1, 'Expected one original consumption receipt.');
|
||||
planner.finish(session, plan.record, { response, attempts: wireStarts });
|
||||
session.providerMessages.push(response.message); store.message(session, 'assistant', response.message.content);
|
||||
const total = (run.usage?.reportedTokens ?? run.usage?.totalTokens ?? 0);
|
||||
session.status = total > 16000 ? 'paused' : 'completed';
|
||||
if (total > 16000) stopReason = 'task_token_limit';
|
||||
} catch (cause) {
|
||||
diagnostic = cause.diagnostic ?? { source: 'workbench', origin: 'local', stage: 'identity_preflight', kind: signal.aborted ? 'cancelled' : 'runtime', title: '身份预检需要核对', message: provider.redact(cause.message), occurredAt: new Date().toISOString() };
|
||||
session.status = diagnostic.kind === 'model_identity' || signal.aborted ? 'paused' : 'failed';
|
||||
stopReason = diagnostic.kind;
|
||||
session.lastFailure = { ...diagnostic, runId: run.id };
|
||||
const message = store.message(session, 'system', diagnostic.message);
|
||||
session.failure = { ...session.lastFailure, messageId: message.id };
|
||||
if (plan) planner.finish(session, plan.record, { error: new Error(diagnostic.message), attempts: wireStarts });
|
||||
} finally {
|
||||
if (requestEvent) { requestEvent.phase = diagnostic ? 'failed' : 'completed'; requestEvent.finishedAt = new Date().toISOString(); if (diagnostic) requestEvent.diagnostic = diagnostic; }
|
||||
store.finishStep(session, step, diagnostic ? 'failed' : 'completed');
|
||||
store.finishRun(session, run, stopReason);
|
||||
}
|
||||
const attempts = hub.attempts.filter(a => a.sessionId === session.id);
|
||||
ensure(attempts.length <= 1 && wireStarts <= 1, 'Single-request limit exceeded.');
|
||||
return { sessionId: session.id, projectId: project.id, taskId: run.taskId, runId: run.id, status: session.status,
|
||||
stopReason, attempts: attempts.length, receipts, wireStarts, actualTools: run.toolCalls, usage: run.usage,
|
||||
contextPlanId: plan?.record.id ?? null, diagnostic, message: response?.message ?? null };
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { publicSnapshot, hash } from './state.mjs';
|
||||
import { invokeProbe, inspectorClosed } from './inspector.mjs';
|
||||
import { root, origin, dataRoot, connectionId, save, meta, client, captureAttempts, readRoot, experiment } from './common.mjs';
|
||||
// Single no-tool provider identity preflight. One request max, strict identity, stop on failure.
|
||||
assert.equal(fs.existsSync(path.join(root, 'preflight-' + process.argv[3])), false, 'Never replay or overwrite a real sample.');
|
||||
const original = readRoot();
|
||||
assert.ok(['off', 'on'].includes(process.argv[3]));
|
||||
const nativeParameters = experiment[process.argv[3]];
|
||||
const prefix = 'preflight-' + process.argv[3];
|
||||
const { assertBatchBudget } = await import('./common.mjs');
|
||||
assertBatchBudget();
|
||||
const { api, boot, close } = await client();
|
||||
let fatal;
|
||||
try {
|
||||
await inspectorClosed();
|
||||
assert.equal(boot.version, '1.25.3');
|
||||
const host = await api('/host/status'); assert.equal(host.identity.pid, 36552); assert.equal(host.activity.active, false);
|
||||
const before = await publicSnapshot(origin, dataRoot()); save(prefix + '/before.json', before);
|
||||
assert.deepEqual(before.active, []);
|
||||
const connection = before.modelCenter.connections.find(value => value.id === connectionId);
|
||||
assert.equal(connection.enabled, true); assert.equal(connection.hasKey, true); assert.equal(connection.credentialRevision, experiment.credentialRevision);
|
||||
assert.equal(connection.protocol, experiment.protocol); assert.equal(connection.model, experiment.model);
|
||||
assert.equal(connection.revision, original.connectionRevision);
|
||||
assert.equal(connection.contextLimit, 16000); assert.equal(connection.modelIdentityPolicy ?? 'record', 'record');
|
||||
const originalAttempts = meta('modelAttempts') ?? []; save(prefix + '/original-attempts.json', originalAttempts);
|
||||
const targetName = connection.name + " · " + process.argv[3];
|
||||
const nonce = randomUUID(), prompt = `这是独立模型连接身份预检,不是工程任务。只回复 OK ${nonce},不要调用工具、读写文件或执行命令。`;
|
||||
save(prefix + '/contract.json', { kind: 'single-real-provider-identity-preflight', connectionId, connection,
|
||||
version: boot.version, runtimeRoot: boot.runtimeRoot, maxRequests: 1, maxAttempts: 1, maxOutputTokens: 2048, nativeParameters,
|
||||
maxTokens: 16000, contextLimit: 16000, toolDefinitions: 0, toolExecution: 0,
|
||||
poolSwitching: false, automaticReplay: false, modelIdentityPolicy: 'require-match in this experiment snapshot only',
|
||||
noGlobalConfigurationWrite: true, noCredentialExtractionOrResave: true, prompt, nonce,
|
||||
originalBaselineSha256: hash(fs.readFileSync(path.join(root, prefix + '/before.json'))) });
|
||||
const projectRoot = path.join(root, prefix + '/identity-only'); fs.mkdirSync(projectRoot);
|
||||
fs.writeFileSync(path.join(projectRoot, 'README.md'), '# AA provider identity preflight\nNo engineering or command execution.\n', { flag: 'wx' });
|
||||
const project = await api('/projects', { root: projectRoot });
|
||||
const session = await api('/sessions', { projectId: project.id }); save(prefix + '/owned.json', { project, session });
|
||||
const input = { pid: host.identity.pid, port: 4318, runtimeRoot: boot.runtimeRoot, sessionId: session.id, projectId: project.id,
|
||||
nativeParameters, connectionId, connectionRevision: connection.revision, credentialRevision: connection.credentialRevision,
|
||||
baseUrl: connection.baseUrl, model: connection.model, protocol: connection.protocol, title: `AA · ${targetName}单次预检`, prompt };
|
||||
save(prefix + '/input.json', input);
|
||||
console.log(JSON.stringify({ stage: 'single-provider-preflight', connectionId, model: connection.model, protocol: connection.protocol, baseUrl: connection.baseUrl, maxRequests: 1, toolDefinitions: 0 }));
|
||||
const invoked = await invokeProbe(input); save(prefix + '/expression.js', invoked.expression); save(prefix + '/host-result.json', invoked.result);
|
||||
const previous = new Set(originalAttempts.map(value => value.id));
|
||||
const attempts = (meta('modelAttempts') ?? []).filter(value => !previous.has(value.id)); save(prefix + '/attempts.json', attempts);
|
||||
assert.ok(attempts.length <= 1); for (const attempt of attempts) { assert.equal(attempt.sessionId, session.id); assert.equal(attempt.connectionId, connectionId); assert.equal(attempt.poolId, null); }
|
||||
const state = await api('/sessions/' + session.id); save(prefix + '/session.json', state);
|
||||
save(prefix + '/export.json', await api('/sessions/' + session.id + '/export'));
|
||||
const captures = await captureAttempts(api, attempts, prefix + '/captures');
|
||||
const attempt = attempts[0];
|
||||
const passed = attempts.length === 1 && attempt.status === 'completed' && attempt.modelIdentity?.status === 'matched'
|
||||
&& attempt.modelIdentity.complete === true && attempt.modelIdentity.policy === 'require-match' && attempt.responseRedacted === false
|
||||
&& attempt.usageStatus === 'reported' && attempt.usage.total_tokens <= 16000 && captures.length === 2
|
||||
&& invoked.result.actualTools === 0 && state.status === 'completed' && invoked.result.message?.content?.includes(nonce);
|
||||
const result = { passed, engineeringTestAllowed: passed, testedAt: new Date().toISOString(), version: boot.version, nativeParameters, reasoningMode: process.argv[3],
|
||||
connectionId, sessionId: session.id, projectId: project.id, taskId: invoked.result.taskId,
|
||||
requestedModel: connection.model, reportedModels: attempt?.modelIdentity?.reportedModels ?? [], identity: attempt?.modelIdentity,
|
||||
requestUrl: attempt?.requestUrl, requests: attempts.length, usage: attempt?.usage ?? null, usageStatus: attempt?.usageStatus ?? 'not_dispatched',
|
||||
actualTools: invoked.result.actualTools, stopReason: invoked.result.stopReason, captures,
|
||||
noRetryOrPoolSwitch: attempts.every(value => value.ordinal === 1 && value.poolId === null),
|
||||
modelWeightsAuthenticated: false, engineeringDeliveryTested: false, purgeBenefitTested: false,
|
||||
defaultSelectionChanged: false, connectionConfigChanged: false, credentialValuesReadOrSaved: false };
|
||||
save(prefix + '/result.json', result); console.log(JSON.stringify({ passed, requests: result.requests, usage: result.usage, reportedModels: result.reportedModels, stopReason: result.stopReason }));
|
||||
} catch (error) {
|
||||
fatal = { name: error.name, message: error.message, stack: error.stack, recordedAt: new Date().toISOString(), automaticReplay: false };
|
||||
save(prefix + '/driver-error.json', fatal); process.exitCode = 1;
|
||||
} finally {
|
||||
save(prefix + '/after.json', await publicSnapshot(origin, dataRoot()));
|
||||
save(prefix + '/inspector-final.json', { closed: await inspectorClosed() });
|
||||
await close();
|
||||
}
|
||||
if (fatal) console.error(fatal.message);
|
||||
@@ -0,0 +1 @@
|
||||
export { hash, canonical, digest, databaseSnapshot, publicSnapshot, compareOriginalState } from '../retained-host-state.mjs';
|
||||
@@ -0,0 +1,54 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { fixture, launchWorkbench } from '../../tests/helpers.mjs';
|
||||
import { invokeProbe, inspectorClosed } from './inspector.mjs';
|
||||
import { routes } from './common.mjs';
|
||||
const output = path.resolve('.qa/aa-driver-check-' + randomUUID()); fs.mkdirSync(output);
|
||||
const cleanup = [], t = { after: action => cleanup.push(action) }, f = fixture(t), wire = [], checks = [];
|
||||
const server = http.createServer(async (request, response) => {
|
||||
const chunks = []; for await (const chunk of request) chunks.push(chunk); const body = JSON.parse(Buffer.concat(chunks)); wire.push(body);
|
||||
const content = body.messages.at(-1).content;
|
||||
const text = Array.isArray(content) ? content.map(block => block.text ?? '').join('') : content;
|
||||
response.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify(request.url.includes('messages')
|
||||
? { id: 'msg-owned', type: 'message', role: 'assistant', model: body.model, content: [{ type: 'text', text }], stop_reason: 'end_turn', usage: { input_tokens: 10, output_tokens: 20 } }
|
||||
: { id: 'owned', model: body.model, choices: [{ message: { role: 'assistant', content: text }, finish_reason: 'stop' }], usage: { prompt_tokens: 10, completion_tokens: 20, total_tokens: 30 } }));
|
||||
});
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
let result;
|
||||
try {
|
||||
await inspectorClosed();
|
||||
const host = await launchWorkbench(t, f.data);
|
||||
for (const route of routes) {
|
||||
const protocol = route.key === 'claude' ? 'anthropic-messages' : 'chat-completions';
|
||||
const connection = await host.api('/model-center/connections', { name: 'AA owned ' + route.key, brand: route.key === 'deepseek' ? 'DeepSeek' : '自定义',
|
||||
model: 'owned-aa-' + route.key, baseUrl: 'http://127.0.0.1:' + server.address().port + '/v1', protocol,
|
||||
requiresKey: true, apiKey: 'owned-aa-synthetic', contextLimit: 16000, maxOutputTokens: 2048 });
|
||||
for (const mode of ['off', 'on']) {
|
||||
const session = await host.api('/sessions', { projectId: f.project.id });
|
||||
const prompt = 'OK owned-aa-' + randomUUID(), before = wire.length;
|
||||
const value = await invokeProbe({ pid: host.child.pid, port: new URL(host.url).port * 1, runtimeRoot: host.bootstrap.runtimeRoot,
|
||||
sessionId: session.id, projectId: f.project.id, connectionId: connection.id, connectionRevision: connection.revision,
|
||||
credentialRevision: connection.credentialRevision, baseUrl: connection.baseUrl, model: connection.model, protocol,
|
||||
title: 'AA owned ' + mode, prompt, nativeParameters: route[mode] });
|
||||
fs.writeFileSync(path.join(output, route.key + '-' + mode + '.json'), JSON.stringify(value.result, null, 2));
|
||||
assert.equal(value.result.status, 'completed'); assert.equal(value.result.wireStarts, 1); assert.equal(value.result.actualTools, 0);
|
||||
assert.equal(value.result.message.content, prompt); assert.equal(wire.length, before + 1);
|
||||
const sent = wire.at(-1); assert.equal(sent.max_tokens, 2048); assert.ok(!sent.tools?.length);
|
||||
for (const [key, expected] of Object.entries(route[mode])) assert.deepEqual(sent[key], expected);
|
||||
const { attempts } = await host.api('/model-center');
|
||||
assert.equal(attempts.find(row => row.sessionId === session.id).modelIdentity.status, 'matched');
|
||||
checks.push(route.key + '-' + mode);
|
||||
}
|
||||
}
|
||||
assert.equal(checks.length, 6); await inspectorClosed();
|
||||
result = { passed: true, checks, syntheticRequests: wire.length, realModelRequests: 0, realCatalogRequests: 0, originalFormalHostTouched: false };
|
||||
} catch (error) { result = { passed: false, checks, error: error.stack, realModelRequests: 0 }; process.exitCode = 1; }
|
||||
finally {
|
||||
server.closeAllConnections(); await new Promise(resolve => server.close(resolve));
|
||||
for (const action of cleanup.reverse()) await action();
|
||||
fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2)); console.log(JSON.stringify({ ...result, output }));
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
// Offline preservation and wire checks for one already-completed AA preflight.
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import assert from 'node:assert/strict';
|
||||
import { root, save, sha, experiment } from './common.mjs';
|
||||
const mode = process.argv[3]; assert.ok(['off', 'on'].includes(mode));
|
||||
const prefix = 'preflight-' + mode;
|
||||
const read = file => JSON.parse(fs.readFileSync(path.join(root, prefix, file)));
|
||||
const before = read('before.json'), after = read('after.json'), result = read('result.json');
|
||||
const owned = read('owned.json'), attempts = read('attempts.json'), session = read('session.json');
|
||||
for (const field of ['pid', 'version', 'runtimeRoot', 'hasKey', 'configSha256']) assert.equal(after[field], before[field], field);
|
||||
assert.deepEqual(before.active, []); assert.deepEqual(after.active, []);
|
||||
assert.equal(after.projects, before.projects + 1); assert.equal(after.exports.length, before.exports.length + 1);
|
||||
for (const item of before.exports) assert.deepEqual(after.exports.find(value => value.id === item.id), item);
|
||||
for (const table of ['sessions', 'records', 'events', 'outbox']) {
|
||||
assert.equal(after.database.tables[table].schemaSha256, before.database.tables[table].schemaSha256);
|
||||
const rows = new Map(after.database.tables[table].rows.map(value => [value.id, value]));
|
||||
for (const row of before.database.tables[table].rows) assert.deepEqual(rows.get(row.id), row, table + '/' + row.id);
|
||||
}
|
||||
const metadata = new Map(after.database.tables.metadata.rows.map(value => [value.id, value]));
|
||||
for (const row of before.database.tables.metadata.rows.filter(value => value.id !== 'modelAttempts')) assert.deepEqual(metadata.get(row.id), row);
|
||||
const artifacts = new Map(after.persistent.artifacts.map(value => [value.path, value]));
|
||||
for (const row of before.persistent.artifacts) assert.deepEqual(artifacts.get(row.path), row);
|
||||
assert.deepEqual(after.modelCenter.selected, before.modelCenter.selected); assert.deepEqual(after.modelCenter.pools, before.modelCenter.pools);
|
||||
assert.equal(attempts.length, result.requests); assert.ok(attempts.length <= 1);
|
||||
for (const attempt of attempts) {
|
||||
assert.equal(attempt.sessionId, owned.session.id); assert.equal(attempt.poolId, null); assert.equal(attempt.ordinal, 1);
|
||||
assert.equal(attempt.model, experiment.model); assert.equal(attempt.protocol, experiment.protocol);
|
||||
}
|
||||
assert.equal(result.actualTools, 0); assert.equal(session.execution.run.toolCalls, 0); assert.equal(read('inspector-final.json').closed, true);
|
||||
for (const capture of result.captures) {
|
||||
const bytes = fs.readFileSync(path.join(root, capture.path)); assert.equal(bytes.length, capture.bytes); assert.equal(sha(bytes), capture.sha256);
|
||||
if (capture.kind === 'request') {
|
||||
const request = JSON.parse(bytes); assert.equal(request.max_tokens, 2048); assert.ok(!request.tools?.length);
|
||||
for (const [key, expected] of Object.entries(experiment[mode])) assert.deepEqual(request[key], expected);
|
||||
}
|
||||
}
|
||||
const verification = { passed: true, route: experiment.key, reasoningMode: mode, modelPassed: result.passed,
|
||||
originalExportsExact: before.exports.length, originalArtifactFilesExact: before.persistent.artifacts.length,
|
||||
originalRowsExact: ['sessions', 'records', 'events', 'outbox'], metadataExactExceptNewAttempt: true,
|
||||
budgetsSelectionPoolsAndCredentialsUnchanged: true, originalPidPreserved: after.pid, inspectorClosed: true,
|
||||
preflightRequests: result.requests, reportedTokens: result.usage?.total_tokens ?? null, toolsExecuted: 0, verificationModelRequests: 0 };
|
||||
save(prefix + '/verification.json', verification); console.log(JSON.stringify(verification));
|
||||
@@ -1,5 +1,15 @@
|
||||
# Agent Workbench:核心代理能力研究与产品计划
|
||||
|
||||
## 当前:AA 真实兼容、自主交付与 Purge 验证
|
||||
|
||||
用户明确要求进入真实验证,保留历史异常。计划docs/LIVE-VALIDATION-AA-PLAN.md;原1.25.3/PID36552/Provider/Key保持,不续跑旧样本、不委派/推送。
|
||||
|
||||
1. [in_progress] 核对干净cef636fd、当前三条启用连接与旧驱动,冻结新计划/驱动/正式原状态。
|
||||
2. [pending] 三连接真实目录和所属DNS/TLS,关闭/开启推理各一个新无工具单请求,原输入/响应/身份/用量封存。
|
||||
3. [pending] 固定优先级选已通过关闭预检的单线路,新Node24请求/140000tokens/16K2048,原九测试/build/模型批次/审查Git。
|
||||
4. [pending] Node闭环后React/Python及原版Purge AB/BA,整批100请求/700000tokens,不具前置按事实保留未测。
|
||||
5. [pending] 原数据/凭据/配置恢复对账,证据离线/Git核验、回顾及各阶段本地提交。
|
||||
|
||||
## 当前:1.25.3 模型目录与推理配套(已本地交付)
|
||||
|
||||
用户在独立复核后授权全量修复、开发、验证与本地提交。基线532b9d60干净;采用planning-with-files,先解决已复现问题,再新冻真实样本。正式PID/Provider/Key保持,不委派、不推送、不续跑旧样本。原失败和未知不改成通过。
|
||||
|
||||
Reference in New Issue
Block a user