Files
Agent-Workbench/docs/reproducible-env-release-evidence/source/tests-05.tap
T

505 lines
23 KiB
Plaintext

TAP version 13
# Subtest: PNG bytes are fully decoded, metadata-free and bounded; JPEG EXIF orientation is applied to the sent PNG
ok 1 - PNG bytes are fully decoded, metadata-free and bounded; JPEG EXIF orientation is applied to the sent PNG
---
duration_ms: 104.8995
type: 'test'
...
# Subtest: forged type, short headers, CRC damage, incomplete chunks, trailing payloads, animation and dimension bombs reject
ok 2 - forged type, short headers, CRC damage, incomplete chunks, trailing payloads, animation and dimension bombs reject
---
duration_ms: 24.2905
type: 'test'
...
# Subtest: draft image ownership, withdrawal, reload and atomic Task binding retain original SHA and original bytes
ok 3 - draft image ownership, withdrawal, reload and atomic Task binding retain original SHA and original bytes
---
duration_ms: 1666.5949
type: 'test'
...
# Subtest: a failed session transaction restores draft status and cannot leave a Run consuming the attachment
ok 4 - a failed session transaction restores draft status and cannot leave a Run consuming the attachment
---
duration_ms: 1028.3996
type: 'test'
...
# Subtest: draft limits, duplicate selection, source changes and cross-Task association reject before dispatch
ok 5 - draft limits, duplicate selection, source changes and cross-Task association reject before dispatch
---
duration_ms: 2297.9942
type: 'test'
...
# Subtest: demo and legacy paths refuse image dispatch without creating a Task or consuming the draft
ok 6 - demo and legacy paths refuse image dispatch without creating a Task or consuming the draft
---
duration_ms: 1489.713
type: 'test'
...
# Subtest: chat-completions sends one owned image per actual request, pairs original tools and verifies exact sealed bytes
ok 7 - chat-completions sends one owned image per actual request, pairs original tools and verifies exact sealed bytes
---
duration_ms: 3292.337
type: 'test'
...
# Subtest: responses sends one owned image per actual request, pairs original tools and verifies exact sealed bytes
ok 8 - responses sends one owned image per actual request, pairs original tools and verifies exact sealed bytes
---
duration_ms: 3386.5595
type: 'test'
...
# Subtest: mixed pool excludes unknown vision before lease and sends zero HTTP to the incompatible member
ok 9 - mixed pool excludes unknown vision before lease and sends zero HTTP to the incompatible member
---
duration_ms: 2261.4956
type: 'test'
...
# Subtest: vision probe requires correct randomized pixel order and evidence expires when the connection revision changes
ok 10 - vision probe requires correct randomized pixel order and evidence expires when the connection revision changes
---
duration_ms: 1437.7343
type: 'test'
...
# Subtest: HTTP upload/read/withdraw uses session authorization, returns byte SHA and refuses unknown ownership
ok 11 - HTTP upload/read/withdraw uses session authorization, returns byte SHA and refuses unknown ownership
---
duration_ms: 1076.4396
type: 'test'
...
# Subtest: image accounting excludes Base64 as text but seals every original request byte and reserves image input separately
ok 12 - image accounting excludes Base64 as text but seals every original request byte and reserves image input separately
---
duration_ms: 3.0214
type: 'test'
...
# Subtest: MCP registration stays disabled until an explicit real stdio probe and schema-based tool selection, with actual process cleanup
ok 13 - MCP registration stays disabled until an explicit real stdio probe and schema-based tool selection, with actual process cleanup
---
duration_ms: 2710.7659
type: 'test'
...
# Subtest: MCP probe holds the project operation lock until its owned process and report are settled
ok 14 - MCP probe holds the project operation lock until its owned process and report are settled
---
duration_ms: 2595.0158
type: 'test'
...
# Subtest: MCP validates the declared JSON schema subset and rejects unsupported schemas before external dispatch
ok 15 - MCP validates the declared JSON schema subset and rejects unsupported schemas before external dispatch
---
duration_ms: 0.6264
type: 'test'
...
# Subtest: actual model MCP call approves long schema-valid arguments, validates the sent catalog and stores one original result without leaking environment secrets
ok 16 - actual model MCP call approves long schema-valid arguments, validates the sent catalog and stores one original result without leaking environment secrets
---
duration_ms: 6267.2185
type: 'test'
...
# Subtest: denying an MCP call never starts another process or executes the external tool
ok 17 - denying an MCP call never starts another process or executes the external tool
---
duration_ms: 4959.1561
type: 'test'
...
# Subtest: actual handshake drift rejects tools/call even if program entry hash is unchanged
ok 18 - actual handshake drift rejects tools/call even if program entry hash is unchanged
---
duration_ms: 5057.9694
type: 'test'
...
# Subtest: cancelled dispatched external side effect stays unknown, closes its owned process and cannot replay on reopen
ok 19 - cancelled dispatched external side effect stays unknown, closes its owned process and cannot replay on reopen
---
duration_ms: 5595.7184
type: 'test'
...
# Subtest: changed executable input, foreign project IDs and corrupted original contracts block enable and dispatch
ok 20 - changed executable input, foreign project IDs and corrupted original contracts block enable and dispatch
---
duration_ms: 1620.425
type: 'test'
...
# Subtest: registered MCP HTTP API uses fixed execution contracts and project-owned probe records with no model requests
ok 21 - registered MCP HTTP API uses fixed execution contracts and project-owned probe records with no model requests
---
duration_ms: 4303.7707
type: 'test'
...
# Subtest: unknown MCP pauses later tool calls and all project writers; fingerprinted acknowledgement clears only the review gates and explicit resume does not replay
ok 22 - unknown MCP pauses later tool calls and all project writers; fingerprinted acknowledgement clears only the review gates and explicit resume does not replay
---
duration_ms: 7564.115
type: 'test'
...
# Subtest: MCP acknowledgement persistence failure restores both job and tool review gates; corrupt evidence cannot be acknowledged
ok 23 - MCP acknowledgement persistence failure restores both job and tool review gates; corrupt evidence cannot be acknowledged
---
duration_ms: 5637.0898
type: 'test'
...
# Subtest: actual host exit after an external effect restores unknown jobs without a fabricated report, allows source-bound review and never replays the call
ok 24 - actual host exit after an external effect restores unknown jobs without a fabricated report, allows source-bound review and never replays the call
---
duration_ms: 4653.6584
type: 'test'
...
# Subtest: changed enabled MCP programs can be explicitly disabled without launching or trusting their changed source
ok 25 - changed enabled MCP programs can be explicitly disabled without launching or trusting their changed source
---
duration_ms: 2521.8206
type: 'test'
...
# Subtest: stdio preserves split UTF-8 frames and final replies drained after process exit
ok 26 - stdio preserves split UTF-8 frames and final replies drained after process exit
---
duration_ms: 2019.7841
type: 'test'
...
# Subtest: server-initiated model requests are refused instead of inheriting provider privileges
ok 27 - server-initiated model requests are refused instead of inheriting provider privileges
---
duration_ms: 1030.8448
type: 'test'
...
# Subtest: invalid or over-budget protocol invalid closes only its owned process and is not retried
ok 28 - invalid or over-budget protocol invalid closes only its owned process and is not retried
---
duration_ms: 1519.6333
type: 'test'
...
# Subtest: invalid or over-budget protocol oversized closes only its owned process and is not retried
ok 29 - invalid or over-budget protocol oversized closes only its owned process and is not retried
---
duration_ms: 1049.56
type: 'test'
...
# Subtest: cancel and timeout do not resend a dispatched request and pre-cancel has no request trace
ok 30 - cancel and timeout do not resend a dispatched request and pre-cancel has no request trace
---
duration_ms: 1315.1274
type: 'test'
...
# Subtest: startup failure remains undispatched and has no process PID to terminate
ok 31 - startup failure remains undispatched and has no process PID to terminate
---
duration_ms: 795.6057
type: 'test'
...
# Subtest: REA launch filters a fixture credential canary and rejects changed entry identity before a new connection
ok 32 - REA launch filters a fixture credential canary and rejects changed entry identity before a new connection
---
duration_ms: 1006.0149
type: 'test'
...
# Subtest: schema1 connections remain readable without inventing observed capabilities or migrating original state
ok 33 - schema1 connections remain readable without inventing observed capabilities or migrating original state
---
duration_ms: 769.0945
type: 'test'
...
# Subtest: protocol, unsupported capabilities and stream choice persist; client claims cannot forge verified evidence
ok 34 - protocol, unsupported capabilities and stream choice persist; client claims cannot forge verified evidence
---
duration_ms: 755.0772
type: 'test'
...
# Subtest: ability evidence belongs to exact connection and credential revision; unknown and unsupported reject required routing
ok 35 - ability evidence belongs to exact connection and credential revision; unknown and unsupported reject required routing
---
duration_ms: 0.2828
type: 'test'
...
# Subtest: invalid declarations and inconsistent stream settings reject before registry, revision and credentials mutate
ok 36 - invalid declarations and inconsistent stream settings reject before registry, revision and credentials mutate
---
duration_ms: 759.1922
type: 'test'
...
# Subtest: model registry keeps original credentials/endpoint intact, rejects stale edits, and forgets only volatile keys on restart
ok 37 - model registry keeps original credentials/endpoint intact, rejects stale edits, and forgets only volatile keys on restart
---
duration_ms: 1283.9964
type: 'test'
...
# Subtest: config import previews exact revisions, keeps secrets volatile, syncs only when enabled and preserves manual edits on conflicts
ok 38 - config import previews exact revisions, keeps secrets volatile, syncs only when enabled and preserves manual edits on conflicts
---
duration_ms: 1523.0038
type: 'test'
...
# Subtest: real HTTP pool fails over 429 to another model and never repeats completed tools; attempts seal actual request bytes and Run identity
ok 39 - real HTTP pool fails over 429 to another model and never repeats completed tools; attempts seal actual request bytes and Run identity
---
duration_ms: 3396.969
type: 'test'
...
# Subtest: pool rotation and weights select members on new Runs; disabled and missing credentials are excluded without requests
ok 40 - pool rotation and weights select members on new Runs; disabled and missing credentials are excluded without requests
---
duration_ms: 5111.6252
type: 'test'
...
# Subtest: 401 blocks the credential revision; updating the key restores availability without leaking error echoes
ok 41 - 401 blocks the credential revision; updating the key restores availability without leaking error echoes
---
duration_ms: 1916.8038
type: 'test'
...
# Subtest: queued request cancellation and in-flight disable release only owned leases and do not dispatch another HTTP request
ok 42 - queued request cancellation and in-flight disable release only owned leases and do not dispatch another HTTP request
---
duration_ms: 943.7949
type: 'test'
...
# Subtest: compiled model-center HTTP API persists public profiles, blocks forged origins, and local operations never invoke a model
ok 43 - compiled model-center HTTP API persists public profiles, blocks forged origins, and local operations never invoke a model
---
duration_ms: 2530.8623
type: 'test'
...
# Subtest: model registry disk failures restore routing, pool profiles and credential references without partial updates
ok 44 - model registry disk failures restore routing, pool profiles and credential references without partial updates
---
duration_ms: 826.5944
type: 'test'
...
# Subtest: model selection and saved configuration commit together and failures preserve the original mode and endpoint
ok 45 - model selection and saved configuration commit together and failures preserve the original mode and endpoint
---
duration_ms: 1017.4359
type: 'test'
...
# Subtest: configuration imports roll back the entire batch and retain one usable preview after validation or commit failure
ok 46 - configuration imports roll back the entire batch and retain one usable preview after validation or commit failure
---
duration_ms: 1090.7032
type: 'test'
...
# Subtest: HTTP quota retains native code/type/param/request id, bounded original and allowlisted headers, never repeats that connection
ok 47 - HTTP quota retains native code/type/param/request id, bounded original and allowlisted headers, never repeats that connection
---
duration_ms: 2710.6532
type: 'test'
...
# Subtest: Retry-After above 90 seconds remains exact and a short local budget cannot trigger an early retry
ok 48 - Retry-After above 90 seconds remains exact and a short local budget cannot trigger an early retry
---
duration_ms: 1.302
type: 'test'
...
# Subtest: Retry-After date, expired date, huge value and invalid header retain explicit semantics
ok 49 - Retry-After date, expired date, huge value and invalid header retain explicit semantics
---
duration_ms: 0.8638
type: 'test'
...
# Subtest: unclassified 429 remains unspecified and every bounded actual retry retains its own original error
ok 50 - unclassified 429 remains unspecified and every bounded actual retry retains its own original error
---
duration_ms: 2695.2364
type: 'test'
...
# Subtest: oversized HTTP error saves exactly the bounded prefix and labels its incomplete range
ok 51 - oversized HTTP error saves exactly the bounded prefix and labels its incomplete range
---
duration_ms: 3069.7286
type: 'test'
...
# Subtest: Chat SSE error preserves upstream fields, original failed event, partial tool and reported usage without executing or retrying
ok 52 - Chat SSE error preserves upstream fields, original failed event, partial tool and reported usage without executing or retrying
---
duration_ms: 2234.9671
type: 'test'
...
# Subtest: Responses failed SSE retains original event/response identity and usage
ok 53 - Responses failed SSE retains original event/response identity and usage
---
duration_ms: 2209.5466
type: 'test'
...
# Subtest: Responses incomplete SSE retains original event/response identity and usage
ok 54 - Responses incomplete SSE retains original event/response identity and usage
---
duration_ms: 2712.3843
type: 'test'
...
# Subtest: unsupported successful Responses output retains original JSON and usage with a local protocol origin
ok 55 - unsupported successful Responses output retains original JSON and usage with a local protocol origin
---
duration_ms: 2639.5313
type: 'test'
...
# Subtest: HTTP 200 native error is still an upstream failure rather than a fabricated successful answer
ok 56 - HTTP 200 native error is still an upstream failure rather than a fabricated successful answer
---
duration_ms: 0.5987
type: 'test'
...
# Subtest: JSON-escaped credential echoes are redacted in the original preview and capture as well as parsed fields
ok 57 - JSON-escaped credential echoes are redacted in the original preview and capture as well as parsed fields
---
duration_ms: 0.4172
type: 'test'
...
# Subtest: quota failover uses another independent pool member and never replays a failed stream
ok 58 - quota failover uses another independent pool member and never replays a failed stream
---
duration_ms: 3232.4205
type: 'test'
...
# Subtest: native parameters are isolated by pool member, preserve Responses state and bind the original output reservation
ok 59 - native parameters are isolated by pool member, preserve Responses state and bind the original output reservation
---
duration_ms: 0.3175
type: 'test'
...
# Subtest: unsupported native fields and invalid values reject before dispatch instead of silently dropping or overriding constraints
ok 60 - unsupported native fields and invalid values reject before dispatch instead of silently dropping or overriding constraints
---
duration_ms: 0.2946
type: 'test'
...
# Subtest: native settings for legacy and registered profiles persist without changing credentials or old records and reject stale edits
ok 61 - native settings for legacy and registered profiles persist without changing credentials or old records and reject stale edits
---
duration_ms: 2141.6635
type: 'test'
...
# Subtest: compiled native settings API saves/restores exact parameters and rejects reserved fields without mutating the profile
ok 62 - compiled native settings API saves/restores exact parameters and rejects reserved fields without mutating the profile
---
duration_ms: 2786.9461
type: 'test'
...
# Subtest: legacy connection test sends its saved native fields and selected output limit field without changing the configuration
ok 63 - legacy connection test sends its saved native fields and selected output limit field without changing the configuration
---
duration_ms: 3310.5525
type: 'test'
...
# Subtest: saved engine budgets verify all Chat/Responses route variants and reject a mismatched output allowance
ok 64 - saved engine budgets verify all Chat/Responses route variants and reject a mismatched output allowance
---
duration_ms: 1479.7463
type: 'test'
...
# Subtest: owned CC Switch SQLite imports the selected TOML Responses wire_api and leaves unknown clients untouched
ok 65 - owned CC Switch SQLite imports the selected TOML Responses wire_api and leaves unknown clients untouched
---
duration_ms: 1718.2007
type: 'test'
...
# Subtest: Responses executes one original tool through Runtime, carries encrypted Items, reports raw usage and seals exact wire bytes
ok 66 - Responses executes one original tool through Runtime, carries encrypted Items, reports raw usage and seals exact wire bytes
---
duration_ms: 4195.6387
type: 'test'
...
# Subtest: mixed-protocol pool keeps both actual variants and changes route on 429 without re-executing completed tools
ok 67 - mixed-protocol pool keeps both actual variants and changes route on 429 without re-executing completed tools
---
duration_ms: 4139.6957
type: 'test'
...
# Subtest: Responses incomplete, failed, missing call identity and unsupported hosted outputs never dispatch a tool
# Subtest: Reject {"body":{"id":"resp_owned","status":"incomplete","output":[{"type":"function_call","id":"fc_partial"
ok 1 - Reject {"body":{"id":"resp_owned","status":"incomplete","output":[{"type":"function_call","id":"fc_partial"
---
duration_ms: 2584.3229
type: 'test'
...
# Subtest: Reject {"body":{"id":"resp_owned","status":"failed","output":[{"type":"function_call","id":"fc_failed","cal
ok 2 - Reject {"body":{"id":"resp_owned","status":"failed","output":[{"type":"function_call","id":"fc_failed","cal
---
duration_ms: 2726.2969
type: 'test'
...
# Subtest: Reject {"body":{"id":"resp_owned","status":"completed","output":[{"type":"function_call","id":"fc_bad","cal
ok 3 - Reject {"body":{"id":"resp_owned","status":"completed","output":[{"type":"function_call","id":"fc_bad","cal
---
duration_ms: 2841.0854
type: 'test'
...
# Subtest: Reject {"body":{"id":"resp_owned","status":"completed","output":[{"type":"computer_call","id":"not-supporte
ok 4 - Reject {"body":{"id":"resp_owned","status":"completed","output":[{"type":"computer_call","id":"not-supporte
---
duration_ms: 2458.1422
type: 'test'
...
1..4
ok 68 - Responses incomplete, failed, missing call identity and unsupported hosted outputs never dispatch a tool
---
duration_ms: 10610.6913
type: 'test'
...
# Subtest: invalid JSON in a nonstream Chat or Responses tool group prevents every tool and pool replay
# Subtest: chat-completions
ok 1 - chat-completions
---
duration_ms: 1606.2024
type: 'test'
...
# Subtest: responses
ok 2 - responses
---
duration_ms: 2467.0039
type: 'test'
...
1..2
ok 69 - invalid JSON in a nonstream Chat or Responses tool group prevents every tool and pool replay
---
duration_ms: 4073.5583
type: 'test'
...
# Subtest: wire adapter preserves tool IDs but never passes Responses-private Items to Chat or another endpoint
ok 70 - wire adapter preserves tool IDs but never passes Responses-private Items to Chat or another endpoint
---
duration_ms: 0.1986
type: 'test'
...
# Subtest: connection JSON import accepts explicit Responses protocol and continues to reject unknown protocols without reading client sessions
ok 71 - connection JSON import accepts explicit Responses protocol and continues to reject unknown protocols without reading client sessions
---
duration_ms: 1186.4728
type: 'test'
...
# Subtest: capability routing skips text-only members before acquiring a lease and records exact refusal in the original Run
ok 72 - capability routing skips text-only members before acquiring a lease and records exact refusal in the original Run
---
duration_ms: 1624.0536
type: 'test'
...
# Subtest: all incompatible members reject with no network request, no lease, preserved Task and explicit non-dispatched ledger
ok 73 - all incompatible members reject with no network request, no lease, preserved Task and explicit non-dispatched ledger
---
duration_ms: 1180.596
type: 'test'
...
# Subtest: explicit synthetic tool probe verifies only actual matching call, never executes it and loses verification on revision changes
ok 74 - explicit synthetic tool probe verifies only actual matching call, never executes it and loses verification on revision changes
---
duration_ms: 821.9758
type: 'test'
...
# Subtest: plain text probe cannot verify tools; wrong tool nonce and client supplied evidence cannot claim verified support
ok 75 - plain text probe cannot verify tools; wrong tool nonce and client supplied evidence cannot claim verified support
---
duration_ms: 1151.0725
type: 'test'
...
# Subtest: a disabled connection invalidates an in-flight probe even when toggle mutates its original registry object
ok 76 - a disabled connection invalidates an in-flight probe even when toggle mutates its original registry object
---
duration_ms: 1016.8477
type: 'test'
...
1..76
# tests 82
# suites 0
# pass 82
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 86935.6444