diff --git a/docs/LIVE-PURGE-AF-PLAN.md b/docs/LIVE-PURGE-AF-PLAN.md new file mode 100644 index 00000000..974a8fb4 --- /dev/null +++ b/docs/LIVE-PURGE-AF-PLAN.md @@ -0,0 +1,21 @@ +# AF:开放模式下原版 Purge AB/BA + +前置AD Node和AE React/Python各自已封存并独立交付通过。原来的有限额度、网络、输出截断、未交付、重复读取与未知缓存/真实预留记录全部保持;不续跑旧样本或用户项目。正式1.25.8/a6e31dec/PID71628保持,本阶段不改产品或重打包。 + +冻结后按ab-off→ab-on→ba-on→ba-off串行,各新父项目、新子工作区和新会话。基线复制原Node费用三模块/原九测试,固定Git元数据令初始提交内容完全相同,不复用已有成功源码或对话。模型仅收一次相同业务目标和允许路径,自行读、改、plan_checks/run_checks;不提供实现、人工审批、继续或失败后重试。子工程保持原HEAD/index,父工程只按真实成果自动归集、合并、检查、源码限定Git并finalize,这些自动动作逐项保留,不能冒认模型自己运行了父流程。 + +模型固定AD/AE已核对的DeepSeek/deepseek-flash、Chat/streaming、nativeParameters={}及原凭据引用;整批只公开暂选单线和require-match一次,全部组同连接修订,结束恢复原高性能池与record策略,单调修订和时间变化声明。每次HTTP完整matched、原请求/响应SHA与实际用量/unknown单列,不认证底层权重,不自动换池,不读取、复制、解密或重存真实Key。 + +关闭组仍运行同一个Harness0.2.1-rc.2、工具与宿主桥;开启组加载dsh-purge1.1.78原入口和apply,使用purge-original-restricted-v1固定可信来源profile。Harness controlled SHA为5f4cb8dfd15484724771494634dbb0a221a30c2585a2eadbbe63356ab6fc863a,Purge源SHA为b71679347cac069667d6cc63cb3b7736e69508eda58d50ee72746639bfc40fb0,393运行输入SHA为16e74cf9fc46cb78d128b60710f02fe327514e34faaf52021a2bf685c76ff45e;运行时再次核对,不下载替换。60原文件/两个原生模块、最终完整模型输入、入口/apply及所属进程树结束必须有证据。 + +原版profile仍关闭自动更新、宿主改写、原版操作工具、自动重试/继续。使用新冻结的开放工程提示,明确宿主自动处理审批,不沿用旧默认提示里的“等待审批”。这个提示只属于on的原版postPrompt因素;off没有独立提示实验段。固定源码profile与当前权限区别如实记录,不宣称测试了上游完整默认插件或任意插件操作能力。 + +两组均保持unrestricted/full/passthrough/preserve:context、输出、累计token、请求/工具次数和产品时长不设本地额度,不向wire附加输出上限,不裁剪原模型历史。每例外部观察10分钟,连续3分钟无新读取/源码/检查事实才停止自己的任务;这是两组共同的实验观察终点,不改变产品默认。未知或在途消费原样保留,不改称零成本。 + +成功要求原九测试和build、仅三允许源码提交且当前文件=提交内容、owner原字节及原暂存/未暂存/未跟踪状态、子工程原HEAD/index、原生账本/全部进程结束及每请求身份和用量均核对。父检查与独立重跑分别保存。失败后不追加提示;若是模型/网络停止则保留该例并可执行下一全新组,若驱动/封存/来源或配置漂移则停止批次,先修复后另冻实验,不用后见之明放宽本批门槛。 + +报告四例全部分母、输入/输出/上游缓存tokens、unknown、请求数、延迟与总墙钟、读取/重复/工具和自动父动作。四例均达相同成功标准才能描述token/请求/延迟差异;提前失败不等于省tokens,无账单不编造费用,仅两对顺序平衡的小样本不作因果或普遍收益结论。 + +执行前本机原生夹具open-original-engine-13b6c268两组通过:各四次自有HTTP、12000068合成报告tokens仍继续,完整历史/无输出cap、原版入口/apply/两个原生模块、自动父工程检查/Git/owner及进程树均过,真实请求0。首夹具66b4dc31错用manager.merge,次夹具在成功提交后误要求整个Git HEAD/index不变;两原失败保留,修正为实际prepareMerge和保护非目标index/owner,不能借夹具宣称真实推理或Purge收益。 + +完整真实驱动af-driver-c6ca1f0f四组自有HTTP验收和原状态审计、477份证据封存均通过,真实请求0;三次旧驱动/审计失败见QA,原件保留。先本地提交再冻结正式current.json。原数据审计覆盖全部原表行、导出、工件、模型中心/凭据存在状态与原池;新增父子项目/会话必须全部核对归属。各阶段回顾、本地提交,不委派、不推送。两处已知产品缺口(内置Python环境选择、终端Git交付账本整合)不在对照途中改动,保留另切修复。 diff --git a/docs/QA-LIVE-PURGE-AF.md b/docs/QA-LIVE-PURGE-AF.md new file mode 100644 index 00000000..b7f37598 --- /dev/null +++ b/docs/QA-LIVE-PURGE-AF.md @@ -0,0 +1,13 @@ +# AF 原版Purge对照:准备与实际结果 + +契约见[LIVE-PURGE-AF-PLAN.md](LIVE-PURGE-AF-PLAN.md)。AD Node和AE React/Python已各自独立交付封存,本阶段不改产品,正式仍1.25.8/a6e31dec/PID71628。真实对照尚未开始;未观察前收益unknown。 + +原生开放契约本机验证open-original-engine-13b6c268通过,两组各四次自有HTTP和12000068合成报告tokens、完整历史与无输出cap、原版入口/apply/两个原生模块、父工程检查/Git与owner/index/进程树退出均过,真实请求0。前两个夹具的驱动函数名错误和提交后误要求整个HEAD/index保持的断言失败保留。 + +完整四组驱动首af-driver-5e056c46在ab-off实际父交付通过后,取证把model-route-plan误当直接HTTP正文,找不到顶层tools而失败;修正为核对唯一实际variant,原失败保持。次af-driver-6ca7a530四组16自有HTTP和原九测试/原生/父Git全部通过,但最终审计器假定已有modelAttempts/modelNetworkDiagnostics数组,而全新夹具基线是null,原审计失败保持。该轮旧driver-result错误写了success=true,不能用它盖过失败audit/退出1;已增加必须读取audit.success的断言,并在新夹具冻结前建立与生产一致的两个空数组行。不改通用严格生产审计,也不重写原夹具结果。新的完整验收仍待结算。 + +第三af-driver-dc0cb1df四工程仍通过,最终审计抓到恢复错误:新连接原modelIdentityPolicy省略,恢复POST省略该字段会保留临时require-match。原审计拒绝通过且driver-result=false/退出1保持;修正为显式恢复原值或默认record/空别名,不改凭据。这轮问题是驱动恢复,不将四组工程通过当作完整验收通过。修正后再另建新夹具结算。 + +最终af-driver-c6ca1f0f完整四组通过:16自有HTTP、每例原九测试/build与父Git/owner/index、两组真实原版入口/apply/两个原生模块及进程树退出通过。原11行/1导出/原工件保持,新增8父子项目/8会话和16尝试全部自有,原连接1→3/config2→4仅声明修订/显式默认字段,原池/凭据状态恢复,9229关闭。原状态审计809e90c1通过;封存477份19505631字节,逐实际HTTP/模型计划/schema和全部文件SHA核对通过。该合成夹具的输出表中realRequests指16次自有HTTP,顶层driver-result明确realModelRequests=0,不能纳入真实模型分母。 + +全部固定实现仅发给回环自有HTTP,不发真实供应商。上述运行不是自主推理或Purge收益证据。正式current.json即将单次冻结,真实AB/BA结果另附;未执行前收益unknown。 diff --git a/findings.md b/findings.md index 852ff3c6..9e8bc1e6 100644 --- a/findings.md +++ b/findings.md @@ -1302,3 +1302,7 @@ V调用前检查:新增11文件SHA断言首语法检查发现多一个右括 # AD真实结果 Node新样本已真实交付:11请求180554tokens,7次读取分别属于不同文件,30工具schema原HTTP与prefixAudit v2逐次保持,上游报告122624缓存输入tokens。不能据此归因C收益或普遍不重复读。终端Git成功且owner/index保持,但内置delivery_report仍in_progress,未登记delivery_job;实际Git独立证据与产品报告状态分别记录。Claude原生Messages要求max_tokens,完全省略字段会原样400;不加任意额度或改写历史。两网络目录失败保留。完整阶段见QA-LIVE-VALIDATION-AD。 + +AE两工程均真实自主交付通过,但Python内置检查选中了Hermes的无pytest环境,模型自主使用目标中现成解释器的终端验证才通过。目标文字没有成为检查环境的结构化选择,不能把终端绿测补写成原check_job通过。React/Python均未登记内部delivery_job;报告必须区分真实Git与内部登记的证据。这两处需独立产品修复,不能在Purge对照中更换实现。 + +原版Harness开放模式由collaborationInput将四项budget规范化为0;worker configFor跳过有限额度计算,engine credit以Infinity处理,capacityMode=passthrough不投影裁剪也不发输出cap。本机原生off/on各四请求和12000068合成tokens后继续完成父检查/Git,入口/apply、两个原生模块与进程树退出通过。夹具证明执行契约,不能代替真实自主交付/Purge收益。 diff --git a/progress.md b/progress.md index 99845da7..5e5b81ed 100644 --- a/progress.md +++ b/progress.md @@ -1942,3 +1942,5 @@ AD驱动初次本机夹具失败在连接health由unverified转responded的错 AD现已完成并封存216份60197251字节。真实兼容8/10,jojocode/Claude目录网络异常及Claude缺max_tokens的两次400保持。DeepSeek新Node11请求180554tokens、9/9/build/三模块Git88bb25cb、全部owner/index通过,0人工审批/继续;原高性能池恢复。首Node审计末尾换行假设错误保留,按真实atomicJson精确逆算原SHA后审计node-577d9308通过,原46111行/105导出/工件保持。下步另冻AE React/Python,Purge仍待前置。 AE已封存225份52216677字节:React/Python均10请求、合计367365tokens/0unknown,自主原检查/实际页面及限定Git/owner/index独立通过。Python原内置检查误选无pytest的Hermes环境,失败保留,模型自主用目标提供的现成解释器验证成功;两例内部交付job缺失保持。最终审计原48100行/116导出/工件精确保留,新增2项目2会话均归属AE,原池与配置恢复、正式空闲/9229关闭。下一先本机核对开放原版引擎,再另冻PurgeAB/BA。 + +AF准备完成:原生开放off/on四请求12000068合成tokens仍继续完成父交付;完整af-driver-c6ca1f0f四组16本机HTTP、原九测试/build/父Git/owner/index和原版入口/apply/模块/进程树全部通过,原状态审计及477份证据封存通过。旧函数名、Git断言、route-plan取证、null元数据和省略身份策略恢复失败全部保留。下一本地提交并唯一冻结AF真实ABBA;本机固定响应不算真实自主交付/Purge收益,不改产品和默认额度。 diff --git a/scripts/live-ad/common.mjs b/scripts/live-ad/common.mjs index f424aa63..e8e8c5fc 100644 --- a/scripts/live-ad/common.mjs +++ b/scripts/live-ad/common.mjs @@ -7,6 +7,14 @@ import { Agent, fetch } from 'undici'; export let root = path.resolve('.qa/live-validation-ad'), origin = 'http://127.0.0.1:4318'; export function engineeringEvidence() { root = path.resolve('.qa/live-engineering-ae'); } +let purgeFixture; +export function purgeEvidence() { root = purgeFixture?.root ?? path.resolve('.qa/live-purge-af'); if (purgeFixture) origin = purgeFixture.origin; } +export function configurePurgeFixture(options) { + const target = path.resolve(options.root), url = new URL(options.origin); + assert.equal(path.dirname(target), path.resolve('.qa')); assert.match(path.basename(target), /^af-driver-[a-f0-9-]{36}$/); + assert.equal(url.hostname, '127.0.0.1'); assert.notEqual(url.port, '4318'); assert.ok(url.port); + purgeFixture = { root: target, origin: url.origin }; purgeEvidence(); +} // Used only by the isolated local driver verifier, never by a real batch. export function configureFixture(options) { const target = path.resolve(options.root), url = new URL(options.origin); diff --git a/scripts/live-af/contract.mjs b/scripts/live-af/contract.mjs new file mode 100644 index 00000000..7ed93c01 --- /dev/null +++ b/scripts/live-af/contract.mjs @@ -0,0 +1,8 @@ +export const allowed = ['src/amount.mjs', 'src/summary.mjs', 'src/report.mjs']; +export const order = [ + { id: 'ab-off', enabled: false }, { id: 'ab-on', enabled: true }, + { id: 'ba-on', enabled: true }, { id: 'ba-off', enabled: false }, +]; +export const purgePrompt = '核对原目标和实际工具结果,自行完成允许范围的源码修复和真实检查。当前完全访问权限由宿主自动处理审批,不等待人工审查。保留原测试、配置和用户已有修改,失败如实报告,不自动重试或继续。完成后返回实际结果,由父工程统一提交。'; +export const goal = '独立修复这个Node费用报表工程的三个源码模块,使原九项测试和build通过。只修改src/amount.mjs、src/summary.mjs、src/report.mjs,按原测试理解金额、币种汇总和报表要求;自行读取、修改,并使用plan_checks/run_checks取得当前源码的test/build实际结果。不得修改原测试、package配置、构建脚本、README或owner文件;保留已有暂存/未暂存/未跟踪状态。当前为完全访问权限,写入和检查由宿主自动执行,无需人工审查。当前运行在所属子工作区,不在子工程执行git add/commit或修改Git状态;完成后返回实际结果,由父工程自动归集、复查并完成本地Git交付,不推送远端。不要只解释方案或口头完成。'; +export const observations = { maximumMs: 600000, noNewFactMs: 180000 }; diff --git a/scripts/live-af/finish.mjs b/scripts/live-af/finish.mjs new file mode 100644 index 00000000..ae3f0c2d --- /dev/null +++ b/scripts/live-af/finish.mjs @@ -0,0 +1,16 @@ +import { randomUUID } from 'node:crypto'; +import { purgeEvidence, read, save, client, origin } from '../live-ad/common.mjs'; +import { inspectorClosed } from '../live-ad/inspector.mjs'; +import { auditExperiment } from '../retained-experiment-state.mjs'; +purgeEvidence(); +const frozen = read('current.json'), owned = read('owned.json'), restoration = read('restoration.json'); +const prefix = 'audits/final-' + randomUUID(), { api, boot, close } = await client(); let result; +try { + await inspectorClosed(); const host = await api('/host/status'); + const changed = !!restoration.connection; + const audited = await auditExperiment({ origin, host, boot, frozen, owned, + edits: changed ? [{ id: frozen.connection.id, revisionDelta: 2 }] : [], registryRevisionDelta: changed ? 4 : 0, configRevisionDelta: changed ? 2 : 0 }); + save(prefix + '/after.json', audited.after); result = { ...audited.result, inspectorClosed: true }; +} catch (error) { result = { success: false, error: error.stack }; process.exitCode = 1; } +finally { save(prefix + '/result.json', result); await close(); } +console.log(JSON.stringify({ prefix, ...result })); diff --git a/scripts/live-af/initial.mjs b/scripts/live-af/initial.mjs new file mode 100644 index 00000000..aa55be94 --- /dev/null +++ b/scripts/live-af/initial.mjs @@ -0,0 +1,36 @@ +import fs from 'node:fs'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { purgeEvidence, root, save, client, publicMetadata } from '../live-ad/common.mjs'; +import { publicSnapshot } from '../retained-host-state.mjs'; +import { inspectorClosed } from '../live-ad/inspector.mjs'; +import { resolveControlledInstallation } from '../../src/core/engines/controlled-installation.mjs'; +import { resolveOriginalPurgeSource } from '../../src/core/engines/purge-strategy.mjs'; +import { order, allowed, goal, purgePrompt, observations } from './contract.mjs'; + +const ae = JSON.parse(fs.readFileSync('.qa/live-engineering-ae/summary.json')); +assert.ok(ae.samples.length === 2 && ae.samples.every(row => row.success)); +const fixturePath = process.argv[2]; assert.ok(fixturePath, 'The completed open native fixture result is required.'); +const fixture = JSON.parse(fs.readFileSync(fixturePath)); assert.equal(fixture.success, true); assert.equal(fixture.realModelRequests, 0); +assert.equal(fixture.completeSamples, 4); assert.equal(fixture.originalStateAuditPassed, true); assert.equal(fixture.actualBatchDriverExecuted, true); +purgeEvidence(); assert.equal(fs.existsSync(root), false, 'Never refreeze AF after observing any real result.'); +assert.equal(execFileSync('git', ['status', '--porcelain'], { encoding: 'utf8', windowsHide: true }).trim(), ''); +const { api, boot, close } = await client(); +try { + await inspectorClosed(); const host = await api('/host/status'), center = await api('/model-center'); + assert.equal(boot.version, '1.25.8'); assert.equal(host.activity.active, false); assert.equal(host.desktopWindows.count, 0); + for (const [key, value] of Object.entries({ executionMode: 'unrestricted', approvalMode: 'full', capacityMode: 'passthrough', historyPolicy: 'preserve', + maxSteps: 0, maxRunTokens: 0, maxTaskTokens: 0, maxRunDurationMs: 0 })) assert.equal(boot.config[key], value); + const connection = center.connections.find(row => row.id === '1a7a55a9-29f2-4b9f-8ea2-5a55afd7eacd'); + assert.equal(connection.enabled, true); assert.equal(connection.hasKey, true); assert.equal(connection.model, 'deepseek-flash'); + assert.equal(connection.credentialRevision, 1); assert.deepEqual(connection.nativeParameters, {}); + const installation = resolveControlledInstallation(process.cwd()), source = resolveOriginalPurgeSource(process.cwd(), undefined, installation); + assert.equal(installation.controlledStatus, 'available'); assert.equal(source.status, 'available'); + const baseline = await publicSnapshot('http://127.0.0.1:4318', host.identity.dataRoot); + save('current.json', { host, config: boot.config, center, connection, originalProjects: boot.projects, originalPublicMetadata: publicMetadata(host.identity.dataRoot), + baseline, installation, source, order, allowed, goal, purgePrompt, observations, fixturePath, fixture, + driverCommit: execFileSync('git', ['rev-parse', 'HEAD'], { encoding: 'utf8', windowsHide: true }).trim() }); + console.log(JSON.stringify({ frozen: true, requests: 0, driverCommit: JSON.parse(fs.readFileSync(root + '/current.json')).driverCommit, + version: boot.version, pid: host.identity.pid, projects: baseline.projects, sessions: baseline.database.sessionIds.length, + connectionRevision: connection.revision, harnessSha256: installation.controlledSha256, purgeSha256: source.sourceSha256 })); +} finally { await close(); } diff --git a/scripts/live-af/project.mjs b/scripts/live-af/project.mjs new file mode 100644 index 00000000..952b8bf5 --- /dev/null +++ b/scripts/live-af/project.mjs @@ -0,0 +1,47 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import { execFileSync, spawnSync } from 'node:child_process'; +import { gitEnvironment } from '../../src/core/git-process.mjs'; +import { sha } from '../live-ad/common.mjs'; +import { allowed } from './contract.mjs'; + +export function prepareNode(output, save) { + const sourceRoot = path.join(output, 'project'); + assert.equal(fs.existsSync(sourceRoot), false); + fs.cpSync('benchmarks/live-delivery-v1/node', sourceRoot, { recursive: true, errorOnExist: true }); + fs.writeFileSync(path.join(sourceRoot, 'README.md'), '# Independent expense engineering comparison\nRepair only the three src modules according to the original nine tests. Preserve owner files and their original staging.\n', { flag: 'wx' }); + fs.writeFileSync(path.join(sourceRoot, '.gitignore'), 'dist/\n.awb-*\nnode_modules/\n', { flag: 'wx' }); + for (const name of ['owner-staged.txt', 'owner-unstaged.txt']) fs.writeFileSync(path.join(sourceRoot, name), 'Owner baseline\n', { flag: 'wx' }); + const git = (args, baseline = false) => execFileSync('git', ['--no-pager', '--no-optional-locks', '-C', sourceRoot, ...args], + { encoding: 'utf8', windowsHide: true, env: { ...gitEnvironment(), ...(baseline ? { GIT_AUTHOR_DATE: '2026-11-06T05:00:00Z', GIT_COMMITTER_DATE: '2026-11-06T05:00:00Z' } : {}) } }); + git(['init', '--initial-branch=main']); git(['config', 'core.autocrlf', 'false']); + git(['config', 'user.name', 'Workbench QA']); git(['config', 'user.email', 'qa@workbench.invalid']); + git(['add', '--', '.']); git(['-c', 'maintenance.auto=false', 'commit', '--no-gpg-sign', '-m', 'Seal AF original expense baseline'], true); + const initialHead = git(['rev-parse', 'HEAD']).trim(); + fs.writeFileSync(path.join(sourceRoot, 'owner-staged.txt'), 'Owner staged change\n'); git(['add', '--', 'owner-staged.txt']); + fs.writeFileSync(path.join(sourceRoot, 'owner-staged.txt'), 'Owner unstaged over staging\n'); + fs.writeFileSync(path.join(sourceRoot, 'owner-unstaged.txt'), 'Owner unstaged change\n'); + fs.writeFileSync(path.join(sourceRoot, 'owner-untracked.txt'), 'Owner untracked change\n', { flag: 'wx' }); + const all = git(['ls-files', '-z']).split('\0').filter(Boolean).concat('owner-untracked.txt'); + const files = Object.fromEntries(all.map(name => [name, sha(fs.readFileSync(path.join(sourceRoot, name)))])); + const protectedNames = all.filter(name => !allowed.includes(name)); + const originalIndex = git(['ls-files', '--stage', '--', ...protectedNames]); + const protect = () => { + for (const name of protectedNames) assert.equal(sha(fs.readFileSync(path.join(sourceRoot, name))), files[name], 'Protected original file ' + name); + assert.equal(git(['ls-files', '--stage', '--', ...protectedNames]), originalIndex, 'Protected original index'); + }; + const checks = prefix => { + const result = {}; + for (const [name, args] of [['test', ['--test', '--test-reporter=tap', 'test/expense.test.mjs']], ['build', ['scripts/build.mjs']]]) { + const run = spawnSync(process.execPath, args, { cwd: sourceRoot, encoding: 'utf8', windowsHide: true, env: gitEnvironment(), maxBuffer: 32 * 1024 * 1024 }); + save(prefix + '/' + name + '.txt', (run.stdout ?? '') + (run.stderr ?? '')); + const stats = name === 'test' ? Object.fromEntries(['tests', 'pass', 'fail', 'cancelled', 'skipped', 'todo'].map(key => [key, Number(run.stdout?.match(new RegExp('^# ' + key + ' (\\d+)$', 'm'))?.[1] ?? NaN)])) : null; + result[name] = { status: run.status, signal: run.signal, error: run.error?.message ?? null, stats, + stdoutSha256: sha(run.stdout ?? ''), stderrSha256: sha(run.stderr ?? '') }; + } + protect(); save(prefix + '/result.json', result); return result; + }; + save('source-before.json', { initialHead, files, protectedIndex: originalIndex, status: git(['status', '--porcelain=v1']) }); + return { sourceRoot, git, initialHead, protect, checks, files }; +} diff --git a/scripts/live-af/run.mjs b/scripts/live-af/run.mjs new file mode 100644 index 00000000..60b663f9 --- /dev/null +++ b/scripts/live-af/run.mjs @@ -0,0 +1,213 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import { DatabaseSync } from 'node:sqlite'; +import { setTimeout as delay } from 'node:timers/promises'; +import { purgeEvidence, root, read, save, client, sha, captureSession, connectionDefinition, connectionInput, compareConfig } from '../live-ad/common.mjs'; +import { prepareNode } from './project.mjs'; +import { order, allowed, goal, purgePrompt, observations as limits } from './contract.mjs'; +import { ORIGINAL_PURGE_KIND, assertPurgeModelInput } from '../../src/core/engines/purge-strategy.mjs'; +import { resolveControlledInstallation } from '../../src/core/engines/controlled-installation.mjs'; +import { resolveOriginalPurgeSource } from '../../src/core/engines/purge-strategy.mjs'; +import { requestMetrics, requestVariants } from '../../src/shared/contracts/model-request.mjs'; + +purgeEvidence(); const frozen = read('current.json'); +assert.equal(fs.existsSync(path.join(root, 'batch-start.json')), false, 'No replay or partial batch continuation.'); +const { api, boot, close } = await client(); let original, configured, selected = false, currentRoute, results = [], finalFailure; +const owned = []; +const settle = async () => { if (currentRoute) await api(currentRoute + '/stop', {}, 120000); }; +async function waitParent(route, active, deadline = Date.now() + 120000) { + for (;;) { + const state = await api(route), group = state.execution.collaborations.at(-1); + if (!active.includes(group.status)) return { state, group }; + assert.ok(Date.now() < deadline, 'Owned parent operation observation ended.'); await delay(750); + } +} +async function artifact(sessionId, id) { + let content = '', page; + do { page = await api('/artifacts/' + id + '?session=' + sessionId + '&offset=' + content.length + '&limit=16000'); content += page.content; } while (page.truncated); + assert.equal(sha(content), page.sha256); return content; +} +function engineJobs(sessionId) { + const db = new DatabaseSync(path.join(frozen.host.identity.dataRoot, 'workbench.sqlite'), { readOnly: true }); + try { return db.prepare("SELECT payload FROM records WHERE kind='engine_run' AND session_id=?").all(sessionId).map(row => JSON.parse(row.payload)); } + finally { db.close(); } +} +async function captureEngine(sessionId, prefix) { + const jobs = engineJobs(sessionId); save(prefix + '/engine-jobs.json', jobs); + const proof = []; + for (const job of jobs) { + const base = prefix + '/engines/' + job.id; + const input = await artifact(sessionId, job.inputArtifactId); assert.equal(sha(input), job.inputSha256); save(base + '/input.json', input); + if (job.reportArtifactId) { + const report = await api('/sessions/' + sessionId + '/engine-runs/' + job.id + '/download'); + assert.equal(sha(JSON.stringify(report)), job.reportSha256); save(base + '/report.json', JSON.stringify(report)); + } + if (job.seedContext) save(base + '/seed.json', await artifact(sessionId, job.seedContext.artifactId)); + for (const request of job.requests) { + const text = await artifact(sessionId, request.inputArtifactId); assert.equal(sha(text), request.inputSha256); + const input = JSON.parse(text); assertPurgeModelInput(input.native, job.installation.strategy); + const metrics = requestMetrics(input.body); assert.equal(metrics.sha256, request.requestSha256); assert.equal(metrics.bytes, request.requestBytes); + assert.equal(input.view.mode, 'full'); assert.equal(input.view.inputBudget, null); + const variants = requestVariants(input.body); + assert.equal(variants.length, 1); + assert.ok(variants.every(body => ['max_tokens', 'max_output_tokens', 'max_completion_tokens'].every(key => !Object.hasOwn(body, key)))); + save(base + '/model-' + request.id + '.json', text); + if (input.view.sourceArtifactId) save(base + '/source-' + request.id + '.json', await artifact(sessionId, input.view.sourceArtifactId)); + proof.push({ jobId: job.id, requestId: request.id, requestSha256: metrics.sha256, + variants: variants.map(body => ({ bodySha256: sha(JSON.stringify(body)), toolSha256: sha(JSON.stringify(body.tools)), toolCount: body.tools.length })), + nativeSha256: request.sourceSha256, fullHistory: true, outputTokenCap: false }); + } + for (const call of job.calls) for (const type of ['input', 'output']) { + if (call[type + 'ArtifactId']) save(base + '/tool-' + sha(call.sourceCallId) + '-' + type + '.json', await artifact(sessionId, call[type + 'ArtifactId'])); + } + } + save(prefix + '/engine-proof.json', proof); return { jobs, proof }; +} +try { + const host = await api('/host/status'), center = await api('/model-center'); + assert.equal(host.activity.active, false); assert.equal(host.desktopWindows.count, 0); assert.equal(boot.version, '1.25.8'); + compareConfig(frozen.config, boot.config); assert.deepEqual(center.selected, frozen.center.selected); assert.deepEqual(center.pools, frozen.center.pools); + original = center.connections.find(row => row.id === frozen.connection.id); assert.deepEqual(connectionDefinition(original), connectionDefinition(frozen.connection)); + save('batch-start.json', { order, goal, purgePrompt, limits, config: boot.config, connection: original, localCapsChanged: false, startedAt: new Date().toISOString() }); + configured = await api('/model-center/connections', { ...connectionInput(original), modelIdentityPolicy: 'require-match' }); + assert.equal(configured.hasKey, original.hasKey); assert.equal(configured.credentialRevision, original.credentialRevision); + await api('/model-center/select', { type: 'connection', id: original.id }); selected = true; + save('active-model.json', { connection: configured, config: (await api('/bootstrap')).config }); + let baseHead; + for (const sample of order) { + const prefix = sample.id; assert.equal(fs.existsSync(path.join(root, prefix)), false); + const store = (file, value) => save(prefix + '/' + file, value), startedAt = Date.now(); + let own, session, worker, group, workerState, failure, result, route; + try { + const installation = resolveControlledInstallation(process.cwd(), frozen.installation.root); + const source = resolveOriginalPurgeSource(process.cwd(), frozen.source.root, installation); + assert.equal(installation.controlledSha256, frozen.installation.controlledSha256); + assert.equal(source.sourceSha256, frozen.source.sourceSha256); assert.equal(source.runtimeSha256, frozen.source.runtimeSha256); + const current = await api('/model-center'); + assert.deepEqual(connectionDefinition(current.connections.find(row => row.id === original.id)), connectionDefinition(configured)); + assert.deepEqual(current.selected, { type: 'connection', id: original.id }); + own = prepareNode(path.join(root, prefix), store); + baseHead ??= own.initialHead; assert.equal(own.initialHead, baseHead, 'Every comparison starts with identical original Git content.'); + const before = own.checks('baseline'); assert.equal(before.test.stats.tests, 9); assert.equal(before.test.stats.fail, 9); assert.equal(before.build.status, 0); + const project = await api('/projects', { root: own.sourceRoot }); session = await api('/sessions', { projectId: project.id }); + route = '/sessions/' + session.id; currentRoute = route; owned.push({ projectId: project.id, sessionId: session.id }); + store('owned-parent.json', { project, session }); + const planned = await api(route + '/collaborations', { goal, engine: { id: 'deepseek-harness', root: installation.root, + controlledSha256: installation.controlledSha256, ...(sample.enabled ? { strategy: { kind: ORIGINAL_PURGE_KIND, + root: source.root, sourceSha256: source.sourceSha256, runtimeSha256: source.runtimeSha256, prompt: purgePrompt } } : {}) }, + children: [{ label: 'expense-candidate', goal, paths: allowed }] }); + group = planned.group; store('group-planned.json', group); + assert.deepEqual(group.budget, { maxTokens: 0, maxSteps: 0, maxToolCalls: 0, maxDurationMs: 0 }); + const groupRoute = route + '/collaborations/' + group.id, fingerprint = group.fingerprint; + await api(groupRoute + '/prepare', { fingerprint }, 120000); + group = (await waitParent(route, ['preparing'])).group; assert.equal(group.status, 'ready', group.summary); + worker = group.children[0]; owned.push({ projectId: worker.projectId, sessionId: worker.sessionId }); store('owned-worker.json', worker); + workerState = await api('/sessions/' + worker.sessionId); store('worker-prepared.json', workerState); + const workerProject = (await api('/bootstrap')).projects.find(row => row.id === worker.projectId); assert.ok(workerProject); + store('worker-project.json', workerProject); + await api(groupRoute + '/start', { fingerprint }, 120000); + const runningAt = Date.now(); let novelAt = runningAt, signature = '', previous = '', latestMessageAt = 0; + const observed = []; + for (;;) { + const parent = await api(route); group = parent.execution.collaborations.at(-1); + workerState = await api('/sessions/' + worker.sessionId); own.protect(); + const changes = allowed.map(name => sha(fs.readFileSync(path.join(workerProject.root, name)))); + const completedTools = workerState.events.filter(row => row.kind === 'tool' && row.phase === 'completed'); + const facts = completedTools.filter(row => ['read_file', 'read_artifact', 'list_files', 'search_files', 'query_code'].includes(row.tool)) + .map(row => row.tool + JSON.stringify(row.input) + sha(row.output ?? '')); + const next = JSON.stringify({ changes, facts: [...new Set(facts)].sort(), checks: workerState.execution.checks.map(row => ({ status: row.status, source: row.sourceFingerprint })) }); + if (signature !== next) { signature = next; novelAt = Date.now(); } + const observation = { sample: prefix, group: group.status, worker: workerState.status, + requests: workerState.execution.task?.usage.requests, tokens: workerState.execution.task?.usage.totalTokens, + changes: workerState.changes.length, checks: workerState.execution.checks.map(row => row.status), + engine: workerState.execution.engineRuns?.at(-1)?.status, lastTool: completedTools.at(-1)?.tool, elapsedMs: Date.now() - runningAt }; + const key = JSON.stringify({ ...observation, elapsedMs: 0 }); + if (key !== previous || Date.now() - latestMessageAt > 45000) { + observed.push(observation); console.log(JSON.stringify(observation)); previous = key; latestMessageAt = Date.now(); + } + if (!['starting', 'running', 'stopping'].includes(group.status)) break; + if (Date.now() - runningAt > limits.maximumMs || Date.now() - novelAt > limits.noNewFactMs) { + failure = { reason: 'experiment_observation_ended', elapsedMs: Date.now() - runningAt, noNewFactMs: Date.now() - novelAt, productCapsChanged: false }; + await settle(); break; + } + await delay(1000); + } + store('observations.json', observed); group = (await api(route)).execution.collaborations.at(-1); + if (!failure && group.status === 'ready_to_collect') { + await api(groupRoute + '/collect', { fingerprint }, 120000); group = (await waitParent(route, ['collecting'])).group; + assert.equal(group.status, 'collected', group.summary); + await api(groupRoute + '/merge', { fingerprint }, 120000); group = (await waitParent(route, ['merge_preparing'])).group; + assert.equal(group.status, 'merge_prepared', group.summary); + const checkState = await api(route + '/check-plans', { names: ['test', 'build'] }); const plan = checkState.execution.plans.at(-1); + await api(route + '/check-jobs', { planId: plan.id }); + let state; do { state = await api(route); if (['queued', 'running'].includes(state.execution.checks.at(-1).status)) await delay(500); else break; } while (Date.now() - startedAt < 900000); + assert.equal(state.execution.checks.at(-1).status, 'passed'); + const deliveryState = await api(route + '/delivery-plans', { message: 'Complete frozen AF expense engineering' }); + await api(route + '/delivery-jobs', { planId: deliveryState.execution.deliveryPlans.at(-1).id }); + do { state = await api(route); if (['queued', 'preparing', 'prepared', 'publishing', 'committed', 'reporting'].includes(state.execution.deliveries.at(-1).status)) await delay(500); else break; } while (Date.now() - startedAt < 900000); + assert.equal(state.execution.deliveries.at(-1).status, 'completed'); + await api(groupRoute + '/finalize', { fingerprint }, 120000); group = (await api(route)).execution.collaborations.at(-1); + assert.equal(group.status, 'completed'); + } else failure ??= { reason: 'model_or_engine_stopped', groupStatus: group.status, summary: group.summary }; + const final = own.checks('independent-final'), commit = own.git(['rev-parse', 'HEAD']).trim(); + const committedPaths = own.git(['diff', '--name-only', own.initialHead, commit]).trim().split(/\r?\n/).filter(Boolean).sort(); + const committedCurrent = allowed.every(name => sha(Buffer.from(own.git(['show', commit + ':' + name]))) === sha(fs.readFileSync(path.join(own.sourceRoot, name)))); + result = { sample: prefix, enabled: sample.enabled, success: !failure && group.status === 'completed' && final.test.stats.pass === 9 + && final.test.stats.fail === 0 && final.build.status === 0 && commit !== own.initialHead && committedCurrent + && JSON.stringify(committedPaths) === JSON.stringify(allowed.slice().sort()), + failure: failure ?? null, final, commit, initialHead: own.initialHead, committedPaths, committedCurrent, + parentSessionId: session.id, workerSessionId: worker.sessionId, ownerBytesAndIndexPreserved: true, + supervisorApprovals: 0, supervisorContinuations: 0, automaticParentCollectionAndChecks: true }; + } catch (error) { + result = { sample: prefix, enabled: sample.enabled, success: false, failure: { error: error.stack, retry: false }, parentSessionId: session?.id, workerSessionId: worker?.sessionId }; + } finally { + const cleanupErrors = []; + try { await settle(); } catch (error) { cleanupErrors.push({ phase: 'settle', error: error.stack }); } + if (worker) { + try { + const captured = await captureSession(api, worker.sessionId, prefix + '/worker'), engine = await captureEngine(worker.sessionId, prefix + '/worker'); + const identities = captured.rows.length > 0 && captured.rows.every(row => row.status === 'completed' && row.connectionId === original.id + && row.modelIdentity?.complete && row.modelIdentity.status === 'matched'); + const nativePassed = engine.jobs.length === 1 && engine.jobs.every(job => job.status === 'completed' && !job.requiresVerification && job.verified + && job.native?.cleanup?.rootExited && job.native?.cleanup?.treeConfirmed && (!sample.enabled + ? !job.native.originalPurge : job.native.originalPurge?.readiness.originalEntryLoaded && job.native.originalPurge?.readiness.originalApplyCalled)); + result = { ...result, success: result.success && identities && nativePassed && captured.state.execution.task.usage.unknownRequests === 0, + identitiesPassed: identities, nativePassed, requests: captured.rows.length, usage: captured.state.execution.task.usage, + reportedCachedInputTokens: captured.rows.reduce((sum, row) => sum + (row.usage?.prompt_tokens_details?.cached_tokens ?? 0), 0), + elapsedMs: Date.now() - startedAt, engineElapsedMs: engine.jobs.reduce((sum, job) => sum + Date.parse(job.finishedAt) - Date.parse(job.startedAt), 0), + modelIdentityPolicy: 'require-match', captures: captured.captures, engineRequestProof: engine.proof }; + } catch (error) { cleanupErrors.push({ phase: 'capture-worker', error: error.stack }); } + } + if (session) try { await captureSession(api, session.id, prefix + '/parent'); } catch (error) { cleanupErrors.push({ phase: 'capture-parent', error: error.stack }); } + if (own) { try { own.protect(); store('git-status.txt', own.git(['status', '--porcelain=v1'])); store('git-log.txt', own.git(['log', '--stat', own.initialHead + '..HEAD'])); } catch (error) { cleanupErrors.push({ phase: 'owner', error: error.stack }); } } + if (cleanupErrors.length) result = { ...result, success: false, cleanupErrors }; + store('result.json', result); save('owned-' + prefix + '.json', owned); currentRoute = null; results.push(result); + console.log(JSON.stringify({ sample: prefix, success: result.success, requests: result.requests, tokens: result.usage?.totalTokens, failure: result.failure, cleanupErrors: result.cleanupErrors })); + if (cleanupErrors.length || result.failure?.error) { finalFailure = 'Driver or capture failed; later samples not dispatched.'; break; } + } + } +} catch (error) { finalFailure = error.stack; process.exitCode = 1; } +finally { + const errors = [], restoration = {}; + try { await settle(); } catch (error) { errors.push({ phase: 'settle', error: error.stack }); } + try { + if (configured) { + const center = await api('/model-center'), current = center.connections.find(row => row.id === original.id); + if (current.revision === configured.revision) restoration.connection = await api('/model-center/connections', { ...connectionInput(original), revision: current.revision, + modelIdentityPolicy: original.modelIdentityPolicy ?? 'record', modelIdentityAliases: original.modelIdentityAliases ?? [], modelIdentityAliasSource: original.modelIdentityAliasSource ?? '' }); + else { restoration.externalConnectionChangePreserved = true; errors.push({ phase: 'connection', error: 'External connection revision changed; preserved.' }); } + } + if (selected) { + const center = await api('/model-center'); + if (center.selected.type === 'connection' && center.selected.id === original.id) { await api('/model-center/select', frozen.center.selected); restoration.selection = frozen.center.selected; } + else { restoration.externalSelectionChangePreserved = true; errors.push({ phase: 'selection', error: 'External selection changed; preserved.' }); } + } + restoration.config = (await api('/bootstrap')).config; restoration.comparison = compareConfig(frozen.config, restoration.config); + } catch (error) { errors.push({ phase: 'restore', error: error.stack }); } + restoration.keysReadOrResaved = false; restoration.errors = errors; save('restoration.json', restoration); + save('owned.json', owned); save('result.json', { success: results.length === 4 && results.every(row => row.success) && !errors.length && !finalFailure, + samples: results, finalFailure: finalFailure ?? null, restorationErrors: errors, purgeBenefit: 'not yet assessed', historicalSamplesReplayed: false }); + await close(); +} +if (finalFailure) console.log(JSON.stringify({ finalFailure })); diff --git a/scripts/live-af/seal.mjs b/scripts/live-af/seal.mjs new file mode 100644 index 00000000..bdca8634 --- /dev/null +++ b/scripts/live-af/seal.mjs @@ -0,0 +1,76 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import { purgeEvidence, root, read, save, sha } from '../live-ad/common.mjs'; + +purgeEvidence(); +const auditPath = process.argv[2]; assert.ok(auditPath, 'Explicit completed audit path is required.'); +const audit = read(auditPath); assert.equal(audit.success, true); +const frozen = read('current.json'), result = read('result.json'), samples = [], allAttempts = [], tools = new Set(); +for (const sample of result.samples) { + const prefix = sample.sample, state = read(prefix + '/worker/session.json'), attempts = read(prefix + '/worker/attempts.json'); + const jobs = read(prefix + '/worker/engine-jobs.json'), proof = read(prefix + '/worker/engine-proof.json'); + assert.equal(attempts.length, state.execution.task.usage.requests); + for (const attempt of attempts) { + allAttempts.push(attempt); + for (const type of ['request', 'response']) { + if (!attempt[type + 'ArtifactId']) continue; + const names = fs.readdirSync(path.join(root, prefix, 'worker/captures')).filter(name => name.startsWith(attempt.id + '-' + type)); + assert.equal(names.length, 1); + const bytes = fs.readFileSync(path.join(root, prefix, 'worker/captures', names[0])); assert.equal(sha(bytes), attempt[type + 'Sha256']); + if (type === 'request') { + const body = JSON.parse(bytes); + assert.ok(['max_tokens', 'max_output_tokens', 'max_completion_tokens'].every(key => !Object.hasOwn(body, key))); + const request = proof.find(row => row.requestId === attempt.requestId); assert.ok(request); + assert.ok(request.variants.some(row => row.bodySha256 === sha(bytes))); + const toolSha256 = sha(JSON.stringify(body.tools)); tools.add(toolSha256); + assert.ok(request.variants.some(row => row.toolSha256 === toolSha256)); + } + } + } + const reads = state.events.filter(row => row.tool === 'read_file' && row.phase === 'completed').map(row => row.input); + const turns = state.execution.run.progressShadow?.turns ?? []; + samples.push({ ...sample, captures: undefined, engineRequestProof: undefined, reads, + requestTimings: attempts.map(row => ({ id: row.id, startedAt: row.startedAt, finishedAt: row.finishedAt, + elapsedMs: row.finishedAt ? Date.parse(row.finishedAt) - Date.parse(row.startedAt) : null, + firstTokenLatencyMs: null, firstTokenLatencyNote: 'The attempt ledger does not record this metric; no estimate substituted.' })), + toolCalls: state.events.filter(row => row.kind === 'tool' && row.phase === 'completed').map(row => ({ tool: row.tool, + startedAt: row.startedAt ?? null, finishedAt: row.finishedAt ?? null, durationMs: row.durationMs ?? null })), + shadowRepeatedReads: turns.filter(row => row.phase === 'settled').reduce((sum, row) => sum + (row.delta?.repeated_reads ?? 0), 0), + originalPurge: jobs.map(job => ({ id: job.id, kind: job.installation.strategy?.kind ?? null, + sourceSha256: job.installation.strategy?.source.sourceSha256 ?? null, + entryLoaded: job.native?.originalPurge?.readiness.originalEntryLoaded ?? false, + applyCalled: job.native?.originalPurge?.readiness.originalApplyCalled ?? false, + loadedModules: job.native?.originalPurge?.loadedModules?.length ?? 0, + nativeLoads: job.native?.originalPurge?.nativeLoads?.length ?? 0, + rootExited: job.native?.cleanup?.rootExited ?? false, treeConfirmed: job.native?.cleanup?.treeConfirmed ?? false })) }); +} +const fourSuccessful = samples.length === 4 && samples.every(row => row.success), sameTools = tools.size === 1; +const metric = (enabled, key) => samples.filter(row => row.enabled === enabled).reduce((sum, row) => sum + (key === 'tokens' ? row.usage?.totalTokens ?? 0 : row[key] ?? 0), 0); +const comparison = fourSuccessful && sameTools ? { eligible: true, off: { tokens: metric(false, 'tokens'), requests: metric(false, 'requests'), engineElapsedMs: metric(false, 'engineElapsedMs') }, + on: { tokens: metric(true, 'tokens'), requests: metric(true, 'requests'), engineElapsedMs: metric(true, 'engineElapsedMs') }, + sampleSize: 2, interpretation: 'Descriptive totals for two order-balanced pairs only; no universal or causal benefit and no billing claim.' } + : { eligible: false, reason: 'Four equal-success samples with stable tools are required; early failure is not token savings.' }; +const summary = { schema: 1, version: '1.25.8', driverCommit: frozen.driverCommit, samples, + requests: allAttempts.length, reportedRequests: allAttempts.filter(row => row.usageStatus === 'reported').length, + reportedTokens: allAttempts.reduce((sum, row) => sum + (row.usage?.total_tokens ?? 0), 0), + unknownRequests: allAttempts.filter(row => row.usageStatus === 'unknown').length, + unknownLocalEstimate: samples.reduce((sum, row) => sum + (row.usage?.estimatedUnknownTokens ?? 0), 0), + actualToolSha256: [...tools], sameTools, comparison, originalStateAudit: auditPath, originalStateAuditResult: audit, + keysReadOrResaved: false, historicalSamplesReplayed: false, priorFailedExperimentsRemainUnchanged: true }; +save('summary.json', summary); +const files = []; +function walk(directory) { + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { + const file = path.join(directory, entry.name); assert.equal(entry.isSymbolicLink(), false); + if (entry.isDirectory()) walk(file); + else { assert.equal(entry.isFile(), true); const value = fs.readFileSync(file); files.push({ path: path.relative(root, file).replaceAll('\\', '/'), bytes: value.length, sha256: sha(value) }); } + } +} +walk(root); files.sort((a, b) => a.path.localeCompare(b.path)); const bytes = files.reduce((sum, row) => sum + row.bytes, 0); +save('FINAL-MANIFEST.json', { schema: 1, files, bytes, scope: 'All AF raw requests/responses, original engine inputs/reports, source/Git, checks and audits. Historical evidence is not repackaged.' }); +for (const row of files) { const value = fs.readFileSync(path.join(root, row.path)); assert.equal(value.length, row.bytes); assert.equal(sha(value), row.sha256); } +if (!frozen.fixture) fs.writeFileSync('docs/live-purge-af-evidence.json', JSON.stringify({ summary, evidence: { root, files: files.length, bytes, + manifestSha256: sha(fs.readFileSync(path.join(root, 'FINAL-MANIFEST.json'))) } }, null, 2) + '\n', { flag: 'wx' }); +console.log(JSON.stringify({ success: true, realRequests: summary.requests, reportedTokens: summary.reportedTokens, unknown: summary.unknownRequests, + allFourSuccessful: fourSuccessful, sameTools, comparison, files: files.length, bytes })); diff --git a/scripts/live-af/verify-driver.mjs b/scripts/live-af/verify-driver.mjs new file mode 100644 index 00000000..aeb87bf5 --- /dev/null +++ b/scripts/live-af/verify-driver.mjs @@ -0,0 +1,62 @@ +// Synthetic fixed model output exercises the real driver and native processes. +// These implementation bytes are never sent to any real provider. +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import { randomUUID } from 'node:crypto'; +import { DatabaseSync } from 'node:sqlite'; +import { fixture, compatibleFixture, launchWorkbench, completion, call } from '../../tests/helpers.mjs'; +import { configurePurgeFixture, save, publicMetadata } from '../live-ad/common.mjs'; +import { publicSnapshot } from '../retained-host-state.mjs'; +import { resolveControlledInstallation } from '../../src/core/engines/controlled-installation.mjs'; +import { resolveOriginalPurgeSource } from '../../src/core/engines/purge-strategy.mjs'; +import { allowed } from './contract.mjs'; + +const output = path.resolve('.qa/af-driver-' + randomUUID()), disposers = [], t = { after: fn => disposers.push(fn) }; +fs.mkdirSync(output); const f = fixture(t, { modernDefaults: true }); +f.store.saveConfig({ approvalMode: 'full', mode: 'compatible' }); +// Model the existing production metadata shape before freezing this fresh fixture. +for (const key of ['modelAttempts', 'modelNetworkDiagnostics']) f.store.storage.database.prepare('INSERT OR IGNORE INTO metadata(key,value) VALUES (?,?)').run(key, '[]'); +f.store.close(); let result; +try { + const solution = path.resolve('.qa/live-validation-ad/node/expense-project'); + assert.equal(JSON.parse(fs.readFileSync('.qa/live-validation-ad/node/result.json')).success, true); + const files = Object.fromEntries(allowed.map(name => [name, fs.readFileSync(path.join(solution, name), 'utf8')])); + let requests = 0; + const model = await compatibleFixture(t, (_url, body) => { + requests++; const step = (requests - 1) % 4 + 1; + assert.ok(['max_tokens', 'max_output_tokens', 'max_completion_tokens'].every(key => !Object.hasOwn(body, key))); + let calls; + if (step === 1) calls = Object.entries(files).map(([name, content], index) => call('owned-file-' + index, 'propose_file', { path: name, content, reason: 'Local fixed HTTP fixture only.' })); + else if (step === 2) calls = [call('owned-plan', 'plan_checks', { names: ['test', 'build'] })]; + else if (step === 3) { + const db = new DatabaseSync(path.join(f.data, 'workbench.sqlite'), { readOnly: true }); + let plans; try { plans = db.prepare("SELECT payload FROM records WHERE kind='check_plan' ORDER BY rowid").all().map(row => JSON.parse(row.payload)); } finally { db.close(); } + calls = [call('owned-check', 'run_checks', { plan_id: plans.at(-1).id })]; + } else calls = []; + assert.ok(requests <= 16); console.log(JSON.stringify({ fixtureRequest: requests, step })); + return { ...completion(calls, calls.length ? '' : 'Owned fixture done.'), model: body.model }; + }); + const host = await launchWorkbench(t, f.data), originalCenter = await host.api('/model-center'); + const connection = await host.api('/model-center/connections', { name: 'Owned AF local fixture', brand: 'Owned fixture', model: 'owned-af', protocol: 'chat-completions', + baseUrl: model.baseUrl, apiKey: 'owned-local-fixture-only', streaming: false, nativeParameters: {} }); + const boot = await host.api('/bootstrap'), center = await host.api('/model-center'), health = await host.api('/host/status'); + configurePurgeFixture({ root: output, origin: host.url }); + const installation = resolveControlledInstallation(process.cwd()), source = resolveOriginalPurgeSource(process.cwd(), undefined, installation); + save('current.json', { host: health, config: boot.config, center, connection, originalProjects: boot.projects, + originalPublicMetadata: publicMetadata(f.data), baseline: await publicSnapshot(host.url, f.data), installation, source, fixture: true }); + await import('./run.mjs'); const batch = JSON.parse(fs.readFileSync(path.join(output, 'result.json'))); + assert.equal(batch.success, true, JSON.stringify(batch.samples.map(row => ({ sample: row.sample, failure: row.failure, cleanupErrors: row.cleanupErrors })))); + assert.equal(requests, 16); assert.equal(batch.samples.length, 4); assert.ok(batch.samples.every(row => row.final.test.stats.pass === 9)); + await import('./finish.mjs'); + const audits = fs.readdirSync(path.join(output, 'audits')); assert.equal(audits.length, 1); + const auditPath = 'audits/' + audits[0] + '/result.json'; + assert.equal(JSON.parse(fs.readFileSync(path.join(output, auditPath))).success, true); + const oldArgument = process.argv[2]; process.argv[2] = auditPath; + try { await import('./seal.mjs'); } finally { process.argv[2] = oldArgument; } + const after = await host.api('/model-center'); assert.deepEqual(after.selected, originalCenter.selected); + result = { success: true, realModelRequests: 0, fixedHttpRequests: requests, actualBatchDriverExecuted: true, + originalTestsPerSample: 9, completeSamples: 4, originalStateAuditPassed: true, implementationSuppliedToLocalFixtureOnly: true }; +} catch (error) { result = { success: false, error: error.stack, realModelRequests: 0 }; process.exitCode = 1; } +finally { for (const dispose of disposers.reverse()) await dispose(); fs.writeFileSync(path.join(output, 'driver-result.json'), JSON.stringify(result, null, 2)); } +console.log(JSON.stringify({ output, ...result })); diff --git a/scripts/verify-open-original-engine.mjs b/scripts/verify-open-original-engine.mjs new file mode 100644 index 00000000..18202f2a --- /dev/null +++ b/scripts/verify-open-original-engine.mjs @@ -0,0 +1,120 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import assert from 'node:assert/strict'; +import { randomUUID } from 'node:crypto'; +import { fixture, compatibleFixture, call } from '../tests/helpers.mjs'; +import { createDeliveryProject, deliveryGit } from '../tests/fixtures/delivery-project.mjs'; +import { ownerBytes } from '../tests/fixtures/workers-project.mjs'; +import { Provider } from '../src/server/provider.mjs'; +import { Runtime } from '../src/server/runtime.mjs'; +import { EngineRunner } from '../src/core/engines/engine-runner.mjs'; +import { resolveControlledInstallation } from '../src/core/engines/controlled-installation.mjs'; +import { resolveOriginalPurgeSource, ORIGINAL_PURGE_KIND, assertPurgeModelInput } from '../src/core/engines/purge-strategy.mjs'; +import { originalEngineRun, verifyEngineModelInput } from '../src/shared/contracts/engine-execution.mjs'; +import { purgePrompt as prompt } from './live-af/contract.mjs'; + +const directory = path.resolve('.qa/open-original-engine-' + randomUUID()); +fs.mkdirSync(directory); +const installation = resolveControlledInstallation(process.cwd()); +assert.equal(installation.controlledStatus, 'available', installation.message); +const source = resolveOriginalPurgeSource(process.cwd(), undefined, installation); +assert.equal(source.status, 'available', source.message); +const results = []; +for (const enabled of [false, true]) { + const name = enabled ? 'on' : 'off', output = path.join(directory, name), disposers = []; + fs.mkdirSync(output); + const t = { after: fn => disposers.push(fn) }, f = fixture(t, { modernDefaults: true }); + let runtime, group, timeout, result; + try { + createDeliveryProject(f); + const owner = ownerBytes(f); + const ownerIndex = deliveryGit(f, ['ls-files', '--stage', '--', 'owner-staged.txt', 'owner-unstaged.txt']); + let requests = 0; + const model = await compatibleFixture(t, async (_route, body) => { + requests++; + console.log(JSON.stringify({ name, ownedHttpRequest: requests })); + assert.ok(['max_tokens', 'max_output_tokens', 'max_completion_tokens'].every(key => !Object.hasOwn(body, key))); + const worker = group.children[0], plan = [...f.store.core.check_plan.values()].find(row => row.taskId === worker.taskId); + const tools = requests === 1 ? [call('owned-read', 'read_file', { path: 'src/add.mjs' }), + call('owned-fix', 'propose_file', { path: 'src/add.mjs', content: 'export const add = (a, b) => a + b;\n', reason: 'Owned HTTP fixture addition repair.' })] + : requests === 2 ? [call('owned-plan', 'plan_checks', { names: ['test', 'build'] })] + : requests === 3 ? [call('owned-check', 'run_checks', { plan_id: plan.id })] : []; + assert.ok(requests <= 4, 'No hidden fixture retry or continuation.'); + return { model: 'owned-open-engine', choices: [{ message: { role: 'assistant', content: tools.length ? '' : 'Owned fixture completed.', + ...(tools.length ? { tool_calls: tools } : {}) }, finish_reason: tools.length ? 'tool_calls' : 'stop' }], + usage: { prompt_tokens: 3000000, completion_tokens: 17, total_tokens: 3000017 } }; + }); + f.store.saveConfig({ mode: 'compatible', model: 'owned-open-engine', baseUrl: model.baseUrl, approvalMode: 'full', + executionMode: 'unrestricted', capacityMode: 'passthrough', historyPolicy: 'preserve', + maxRunTokens: 0, maxTaskTokens: 0, maxSteps: 0, maxRunDurationMs: 0 }); + const provider = new Provider(); provider.setKey('owned-local-http-fixture-only'); + runtime = new Runtime(f.store, provider); + runtime.controlledEngines = new EngineRunner(f.store, provider, { directoryRoot: path.join(f.root, 'owned-engine') }); + const manager = runtime.workerManager(); + group = await manager.plan(f.session, { goal: 'Repair the original addition fixture with full automatic access; preserve owner state.', + engine: { id: 'deepseek-harness', root: installation.root, controlledSha256: installation.controlledSha256, + ...(enabled ? { strategy: { kind: ORIGINAL_PURGE_KIND, root: source.root, sourceSha256: source.sourceSha256, + runtimeSha256: source.runtimeSha256, prompt } } : {}) }, + children: [{ label: 'candidate', goal: 'Repair only src/add.mjs, run the original test/build using plan_checks/run_checks; do not stage or commit the child workspace. Return evidence for parent delivery.', paths: ['src/add.mjs'] }] }); + assert.deepEqual(group.budget, { maxTokens: 0, maxSteps: 0, maxToolCalls: 0, maxDurationMs: 0 }); + await (await manager.prepare(f.session, group.id, group.fingerprint)).promise; + assert.equal(group.status, 'ready', group.summary); + const worker = group.children[0], session = f.store.session(worker.sessionId); + timeout = setTimeout(() => { runtime.stopAll().catch(() => {}); }, 120000); + await manager.start(f.session, group.id, group.fingerprint).promise; + assert.equal(group.status, 'ready_to_collect', group.summary); + const job = [...f.store.core.engine_run.values()].find(row => row.taskId === worker.taskId); + originalEngineRun(f.store, session, job); + assert.equal(job.status, 'completed', job.error); + assert.equal(job.requests.length, 4); assert.equal(requests, 4); + assert.equal(job.native.cleanup.rootExited, true); assert.equal(job.native.cleanup.treeConfirmed, true); + assert.equal(f.store.core.task.get(worker.taskId).usage.totalTokens, 12000068); + for (const request of job.requests) { + const input = verifyEngineModelInput(f.store, session, job, request); + assertPurgeModelInput(input.native, job.installation.strategy); + assert.equal(input.view.mode, 'full'); assert.equal(input.view.inputBudget, null); + } + if (enabled) { + assert.equal(job.native.originalPurge.readiness.originalEntryLoaded, true); + assert.equal(job.native.originalPurge.readiness.originalApplyCalled, true); + assert.equal(job.native.originalPurge.nativeLoads.length, 2); + } else assert.equal(job.native.originalPurge, undefined); + await manager.collect(f.session, group.id, group.fingerprint).promise; + await manager.prepareMerge(f.session, group.id, group.fingerprint).promise; + assert.equal(group.status, 'merge_prepared', group.summary); + assert.equal(f.store.core.change_set.get(group.merge.changeSetId).status, 'completed'); + const checks = runtime.checkRunner(), plan = await checks.plan(f.session, ['test', 'build']); + const checked = await (await checks.start(f.session, plan.id)).promise; + assert.equal(checked.status, 'passed'); + const git = runtime.gitDelivery(), deliveryPlan = await git.plan(f.session, { message: 'Owned open engine fixture delivery' }); + const delivery = await (await git.start(f.session, deliveryPlan.id)).promise; + assert.equal(delivery.status, 'completed'); + await manager.finalize(f.session, group.id, group.fingerprint); + assert.equal(group.status, 'completed'); assert.deepEqual(ownerBytes(f).files, owner.files); + assert.equal(deliveryGit(f, ['ls-files', '--stage', '--', 'owner-staged.txt', 'owner-unstaged.txt']), ownerIndex); + assert.notEqual(delivery.commit, owner.head); + assert.ok(session.approvals.every(row => row.status === 'approved' && row.decisionSource === 'full-access-policy')); + result = { success: true, enabled, realModelRequests: 0, ownedHttpRequests: requests, + fixtureReportedTokens: 12000068, zeroBudgetsMeanUnlimited: true, fullHistory: true, outputCapAbsent: true, + entryAndApplyVerified: enabled, allProcessesExited: true, ownerBytesAndIndexPreserved: true, + humanApprovals: 0, group, job, check: checked, delivery, modelRequests: model.requests.map(({ headers, ...value }) => value) }; + console.log(JSON.stringify({ name, success: true, ownedHttpRequests: requests, fixtureReportedTokens: 12000068, commit: delivery.commit })); + } catch (error) { + result = { success: false, enabled, error: error.stack, group, realModelRequests: 0 }; + process.exitCode = 1; + console.log(JSON.stringify({ name, success: false, error: error.message.split('\n')[0] })); + } finally { + clearTimeout(timeout); + await runtime?.stopAll(); + f.store.storage.database.exec('PRAGMA wal_checkpoint(TRUNCATE)'); + fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2)); + fs.cpSync(f.root, path.join(output, 'original-fixture'), { recursive: true, filter: file => !file.split(path.sep).includes('node_modules') }); + for (const dispose of disposers.reverse()) await dispose(); + } + results.push({ name, success: result.success, error: result.error ?? null }); + if (!result.success) break; +} +const result = { success: results.length === 2 && results.every(row => row.success), directory, results, + realModelRequests: 0, scope: 'Actual pinned native engines with local synthetic HTTP only; no autonomous reasoning or Purge benefit claim.' }; +fs.writeFileSync(path.join(directory, 'result.json'), JSON.stringify(result, null, 2)); +console.log(JSON.stringify(result));