diff --git a/docs/M2-NATIVE-CACHE-STATE-PLAN.md b/docs/M2-NATIVE-CACHE-STATE-PLAN.md new file mode 100644 index 00000000..ff6e18e8 --- /dev/null +++ b/docs/M2-NATIVE-CACHE-STATE-PLAN.md @@ -0,0 +1,15 @@ +# M2 第五批:原生缓存块与跨 Run 状态 + +2026-11-07,基线 `bdfeb29c85b3231bfd7bc24b8b5f49d40a3beebf`,干净工作树。承接 I14/I12 的冻结契约/当次 fetch 数据区分,完成 I09 原生块与工具缓存注解往返、I11 可配置会话/Task亲和与供应商状态绑定;I27/I28 CI 与 M3 单独续接。 + +1. [complete] 固定当前官方缓存/状态来源、复用 I15 原件并核SHA;8份来源清单见 tests/fixtures/native-cache-state/SOURCE.json 与 docs/NATIVE-CACHE-STATE-SOURCES.md。保留原丢注解、跨 Run 选线和状态身份缺项观察。Mnemo新结果只读登记,不作为本地验收。 +2. [in_progress] 保留 Messages 系统/用户/工具块及工具元数据;新增原生状态身份绑定和会话/Task/Run亲和,原用户设置、原快照、完整历史和未知原件不变。默认不偷偷添加缓存参数。 +3. [pending] 自有 HTTP JSON/SSE、混池、跨 Run、手选换模、修订/停用、停止/恢复/重启与所有工具结果不重放的确定性核对;上游原错误保留。 +4. [pending] 复用 Orven 模型中心/原生参数/上下文,完成真实保存恢复与宽窄网页/实际Electron验收、独立构建及逐输入/编译核对。 +5. [pending] 每阶段回顾、限定本地提交、最终证据清单与普通GitAtlas增量推送;旧封存根只读,原失败保留。 + +本批不委派、不读真实Key、不请求真实模型、不操作正式数据/配置/安装/重启,不续旧实验或用户工程。默认 unrestricted/passthrough/preserve 与持久 ask/auto/full 保持,不增加预算、上下文、输出、头或历史额度。工具权限以当前模式为准,不能把状态核对做成新增审批。缓存使用与供应商命中由原 usage 表示,保留 unknown;稳定前缀/亲和或相同指纹不是缓存收益证明。 + +W/AD/AE/AG 网络/unknown、重复读取普遍收益未证明、R/W v2真实排除边界与 Purge收益unknown保持。M3须真实 schema2迁移增量及新处理器产生结果后的回退,不能删新账本或直接移除逐表全等守卫。新证据只写 `.qa/m2-native-cache-state-2026-11-07/`。 + +接续保留原 source-1/source-2/source-3:首夹具错误、亲和来源断言失败和跨线状态整正文搜索失败仍待准确核对,不能把接续摘要当全绿。首次接续 mkdir 遇既有目录即停止;改为读取已有原件并使用新文件名,未覆盖证据。规划辅助只读路径/输出错误见 progress.md。 diff --git a/docs/NATIVE-CACHE-STATE-SOURCES.md b/docs/NATIVE-CACHE-STATE-SOURCES.md new file mode 100644 index 00000000..8992f780 --- /dev/null +++ b/docs/NATIVE-CACHE-STATE-SOURCES.md @@ -0,0 +1,18 @@ +# M2 原生缓存与状态来源 + +2026-11-07,沿 `bdfeb29c` 接续 I09/I11。来源清单与正文 SHA 在 [SOURCE.json](../tests/fixtures/native-cache-state/SOURCE.json)。OpenAI 两篇完整页面复用本批已取得的官方 Docs 原件,Claude/DeepSeek 六篇复用 I15 的原正文,并逐 SHA 只读核对。接续时再次检索并打开 OpenAI 当前页面;新查询不覆盖先前原件或失败记录。 + +| 官方来源 | 本批采用范围 | +| --- | --- | +| [OpenAI conversation state](https://developers.openai.com/api/docs/guides/conversation-state) | 手动历史保留完整 output 数组、encrypted reasoning 与 assistant phase;不把仅有 response ID 当完整本地状态 | +| [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching) | 显式 content breakpoint、缓存 key/TTL/retention 独立透传;会话持续存在或前缀相同不保证命中 | +| [Claude prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) | 系统、消息、工具和工具结果的 cache_control、各块 TTL 及原生元数据保持原位置和顺序 | +| [DeepSeek thinking](https://api-docs.deepseek.com/guides/thinking_mode) / [chat](https://api-docs.deepseek.com/api/create-chat-completion) | 同供应商范围内历史 assistant reasoning_content 完整保留;工具身份与原结果不重放 | +| [DeepSeek cache](https://api-docs.deepseek.com/guides/kv_cache) | 缓存是上游行为,用原 usage 区分读写、未报告和 unknown | +| [DeepSeek Responses](https://api-docs.deepseek.com/api/create-response.md) / [Messages](https://api-docs.deepseek.com/api/create-message.md) | 兼容接口原字段与原生状态分开,不按协议名称推断账户或供应商状态可互换 | + +本地状态绑定记录实际使用的协议、请求 URL、型号、连接、凭据修订和有效请求头模板。它用于防止不透明历史在不同供应商范围误用,不认证缓存命中、模型身份或供应商兼容。普通生成参数和连接修订不等于凭据变更。旧无完整身份的状态保留原件和适用原因;常规文本与原工具调用可继续进入新请求。 + +新模型池默认会话保持;已有池沿原 Run 范围或关闭设置。用户可选择关闭、Run、Task、会话;手动选模和当前可用成员优先。跨 Run 亲和仅从已完成响应保存,诊断独立、未知请求不补成成功,不新增执行审批或 token 额度。 + +该来源核对及合成验证不能证明全球实际兼容、真实缓存收益、重复读取普遍改善或 Purge 收益。W/AD/AE/AG 的网络与 unknown、R/W v2 真实排除边界和原工程结果保持。 diff --git a/findings.md b/findings.md index 06fa760a..ccd8a556 100644 --- a/findings.md +++ b/findings.md @@ -1,5 +1,9 @@ # 汇总发现 +I09/I11接续决策:新池可默认会话保持,但旧池必须保留原Run范围;亲和只来自完成的真实响应且诊断路径独立。状态身份要核对实际请求路径/认证模板/版本头/型号/连接与凭据修订,改变原生普通生成参数不应误认凭据变化。状态无法适用时保留原件和原因、以常规文本/原调用构造新历史,不丢用户数据或自动重放工具。沿既有Orven池编辑与上下文/请求账本补真实控制和反馈,不另立页面。 + +I09/I11具体缺口:messagesBody重建文本和工具时丢非结构元数据,messagesHistory反向只取text并改变tool_result顺序;Messages/Responses原生助手状态只有endpoint/model绑定,缺实际请求版本/头/凭据范围;ModelHub sticky仅runId且重启不持久。需要按来源区分声明原生输入与不透明输出状态,当前状态不能跨供应商或凭据误用,原件不能因此丢弃。 + 原生状态核对新增源方参考:冻结预设/用户设置不等于actual standard会话采用,创建失败也不代表发过模型请求;输入估算/额外预留、压缩请求值与当次fetch数据必须分别核对。手机客户端期限/确认、服务端原任务成功、容量样本和被清理的分段证据不能互相补数或推定延迟段。新同步仅按源方报告登记,自己的透传/预算/身份和原未知结论不改变。 I14收尾复核:实际请求头观察必须绑定prepared fetch参数,原Run只能表示冻结设置;认证模板/独立Key引用、HTTP传输层补头和真实缓存命中证据分别说明。最终两端各8组验证通知正常滚动、原配置/密文/快照保持;871输入与29输出一致,176后端/209测试未变可复用原143/143。首身份验证误用TAP计数不是产品失败,原件保留后仅修验证器。源码交付不升级为真实供应商/自主工程/Purge收益通过;迁移增量与激活新处理器后的回退仍必须在M3证明。 diff --git a/progress.md b/progress.md index 8f3ecac2..e12120d6 100644 --- a/progress.md +++ b/progress.md @@ -1,5 +1,13 @@ # 统一执行进度 +I09/I11来源提交前git diff --check发现官方原文6处原有尾空白,来源字节/SHA保持不修剪;限定排除这两份官方正文后核对其它新增文件,后续另核Git blob原字节。不是产品检查失败,不改原来源来制造通过。 + +I09/I11阶段1来源复核完成:8份官方来源固定,SOURCE.json SHA ced3703c4afb174c5b939ffb3dd8572f22ff8e5f3fb8489f627f45880315b799;OpenAI完整Docs原件复用、Claude/DeepSeek六份逐SHA一致。接续再次打开当前两篇页面,原source-1/2/3失败保留,未重跑旧轮次;首次mkdir既有目录拒绝后改为新文件名写入。下一完成核心与用户连接保持范围及确定性边界,限定来源提交先行。 + +I09/I11接续复核(2026-11-07):沿bdfeb29c与已有未提交实现继续,planning catchup退出0且无补遗;已核AGENTS最新范围、采用计划、1.25计划与M2第五批计划。工作树含缓存块/身份/亲和核心及新测试,界面/来源封存/验证/提交尚未完成,不把简短接续摘要当验收。初次工作区根AGENTS不存在、过宽清单输出截断和假定CI目录不存在仅属只读调查错误;以后限定Agent-Workbench真实路径和输出范围。当前不委派/真实模型/Key/正式写入或重启,先核未提交输入再完成本批。 + +I09/I11阶段1开始:确认Messages转换丢系统/内容/工具缓存注解、亲和仅在Run;先保存当前OpenAI官方缓存/会话原件并复用I15 Claude/DeepSeek来源。planning catchup退出0/无补遗。初读误猜model-settings/model-native-settings文件名及PowerShell rg glob、过量来源结果输出仅属调查错误,后续改为真实文件列表/摘要输出。Mnemo新增准确报告/固定计划SHA与SSE partial只读资料另保留,不替代工作台验收;主任务继续。 + Mnemo新调用方预算/状态同步已仅作只读登记:源方新版手机短3/3、900秒49/60发起/48客户端确认/p95约48.4/125新鲜采样仍失败;后查49服务端原任务成功不补成客户端窗口通过。47无网络响应与8离线/过期、分段事件清理后的延迟unknown分别保留。Harness2048→23→1/额外4096预留、修后2048与默认压缩65536/standard身份缺项分开;最新启动不能创建会话、0模型请求,14公开检查与独立边界失败分别记录,完整读改测/恢复未通过。固定旧计划SHA一致,明确reviews路径未找到。见[只读同步](docs/integration/MNEMO-CALLER-BUDGET-STATE-READONLY-2026-11-07.md)。工作台M2下一I09/I11/CI及M3顺序、开放默认/模型偏好/历史unknown保持,无真实模型/正式写入/委派或向源线程回复。 I14/I12本批阶段5完成:3207份1498866416字节已封存,manifest 3963ad9e02d3f62c1ae4d5fefcb59fdc0bcdc63682be1c72109542a60a2b408c;封存后全3207原件/871工作输入只读复核,35改变输入Git原字节与17当前文档链接通过,2个文档仅规范化行尾、产品输入未变。来源337d1bc9、核心f95507ab、界面26472d28及验收04c0556c已普通增量推送至既有GitAtlas feat/task-agreement-1.25,远端逐SHA一致。回执和完成接续最后另提交/推送再核对,不递归改回执。下一I09/I11缓存块/跨Run状态与I27/I28 CI,再M3真实schema2迁移增量/新处理器结果后回退;未发行/正式操作/真实模型,W/AD/AE/AG与重复读/R/W边界/Purge收益unknown保持。 diff --git a/task_plan.md b/task_plan.md index 8f3bf6d9..6955f076 100644 --- a/task_plan.md +++ b/task_plan.md @@ -1,5 +1,7 @@ # Agent Workbench:核心代理能力研究与产品计划 +M2第五批I09/I11已从bdfeb29c干净基线开始,见[原生缓存与状态计划](docs/M2-NATIVE-CACHE-STATE-PLAN.md):保留Messages块/工具注解及原生历史,配置会话/Task/Run亲和,状态按实际协议/地址/型号/凭据/版本身份核对;旧历史不改写,亲和/同指纹不当作缓存命中。随后CI及M3真实迁移增量/结果后回退。无真实模型/Key/正式写入/委派,历史unknown保持。 + I14/I12阶段1–4完成:来源337d1bc9/核心f95507ab/界面26472d28,最后相关38/38和首候选143/143后端输入未变;最终871输入/29编译/74runtime、网页与实际Electron各8组/8图/三配色/1280及390,原metadata/cipher/存储快照exact、原I13/I15原件只读一致、自有进程/已知监听0。原失败与验收器错误保持。阶段5已完成封存/回顾/授权普通GitAtlas增量推送,源码与验收04c0556c远端逐SHA一致,回执及完成接续最后提交再核对,见[请求契约计划](docs/M2-REQUEST-CONTRACT-PLAN.md)和[验收](docs/QA-M2-REQUEST-CONTRACT.md)。下一I09/I11/CI、M3真实迁移增量及新处理器结果后回退,真实收益和Purge unknown保持。 M2第四批I14/I12从c7d1ad5e干净/远端一致基线开始,见[请求契约计划](docs/M2-REQUEST-CONTRACT-PLAN.md)。已实际取得官方OpenAI overview与Anthropic版本/头说明,先固定来源和旧/新默认分离,再请求版本/头/路径/快照与去凭据指纹。I15原根只读,0真实Key/模型/正式写入,不委派或重启;I09/I11/CI/M3与历史unknown保持。 diff --git a/tests/fixtures/native-cache-state/SOURCE.json b/tests/fixtures/native-cache-state/SOURCE.json new file mode 100644 index 00000000..d4394799 --- /dev/null +++ b/tests/fixtures/native-cache-state/SOURCE.json @@ -0,0 +1,113 @@ +{ + "schemaVersion": 1, + "baseline": "bdfeb29c85b3231bfd7bc24b8b5f49d40a3beebf", + "sources": [ + { + "id": "openai-conversation-state", + "provider": "openai", + "url": "https://developers.openai.com/api/docs/guides/conversation-state", + "checkedAt": "2026-11-07", + "file": "sources/openai-conversation-state.md", + "sha256": "8a62162c2ef6ff7e94dfdad5ba06f417f5a6c00ea6521e6466d462ff2fcd9b96", + "bytes": 24788, + "captureSha256": "df7776f8ac54a84b55cbc0d42bc2b80192428921d96b2e272d21754e2f3d5f8b", + "representation": "official-docs-mcp-markdown-utf8", + "scope": "complete-captured-page", + "reusedReadOnly": true + }, + { + "id": "openai-prompt-caching", + "provider": "openai", + "url": "https://developers.openai.com/api/docs/guides/prompt-caching", + "checkedAt": "2026-11-07", + "file": "sources/openai-prompt-caching.md", + "sha256": "01e57dacc2e5c0fb881b2d1909649983e9e327a5d6c17a98bb1183ee00e9d334", + "bytes": 56425, + "captureSha256": "ed312ad7e34d4de7f5861317481ae82b1bc3b9a1ae78d525c916a0d94231d522", + "representation": "official-docs-mcp-markdown-utf8", + "scope": "complete-captured-page", + "reusedReadOnly": true + }, + { + "id": "claude-build-with-claude__prompt-caching", + "provider": "anthropic", + "url": "https://platform.claude.com/docs/en/build-with-claude/prompt-caching", + "checkedAt": "2026-11-06", + "sha256": "ff35232788f3f7fbd0f9aae002351567a9ea8e68e5b749069d96dab52882747d", + "file": "../official-model-registry/sources/claude-build-with-claude-prompt-caching.md", + "originalSha256": "ff35232788f3f7fbd0f9aae002351567a9ea8e68e5b749069d96dab52882747d", + "representation": "extracted-official-markdown", + "scope": "complete-captured-source", + "bytes": 164465, + "reusedReadOnly": true + }, + { + "id": "deepseek-create", + "provider": "deepseek", + "url": "https://api-docs.deepseek.com/api/create-chat-completion", + "checkedAt": "2026-11-06", + "sha256": "b43ca950bd04ae7f6f537b42e6cb729f543d942dcd848ec43ad4e9bfe059fe1f", + "file": "../official-model-registry/sources/deepseek-create.body", + "originalSha256": "b43ca950bd04ae7f6f537b42e6cb729f543d942dcd848ec43ad4e9bfe059fe1f", + "representation": "received-html-bytes", + "scope": "complete-captured-source", + "bytes": 129498, + "reusedReadOnly": true + }, + { + "id": "deepseek-thinking", + "provider": "deepseek", + "url": "https://api-docs.deepseek.com/guides/thinking_mode", + "checkedAt": "2026-11-06", + "sha256": "42151b5f86827bff663a61693080a61cdbf99074dee94af756c9fce855baa7cd", + "file": "../official-model-registry/sources/deepseek-thinking.body", + "originalSha256": "42151b5f86827bff663a61693080a61cdbf99074dee94af756c9fce855baa7cd", + "representation": "received-html-bytes", + "scope": "complete-captured-source", + "bytes": 117778, + "reusedReadOnly": true + }, + { + "id": "deepseek-cache", + "provider": "deepseek", + "url": "https://api-docs.deepseek.com/guides/kv_cache", + "checkedAt": "2026-11-06", + "sha256": "061c0c7055ceb96d0a522ae1829ff5f7126ff08f26af217643e371abb740a081", + "file": "../official-model-registry/sources/deepseek-cache.body", + "originalSha256": "061c0c7055ceb96d0a522ae1829ff5f7126ff08f26af217643e371abb740a081", + "representation": "received-html-bytes", + "scope": "complete-captured-source", + "bytes": 35850, + "reusedReadOnly": true + }, + { + "id": "deepseek-response", + "provider": "deepseek", + "url": "https://api-docs.deepseek.com/api/create-response.md", + "checkedAt": "2026-11-07", + "sha256": "5e56e45d25db47f2b6966bbb0b2471d324035c33f0c8f81eaaf8fc444e6869da", + "file": "../official-model-registry/sources/deepseek-response.md", + "originalSha256": "5e56e45d25db47f2b6966bbb0b2471d324035c33f0c8f81eaaf8fc444e6869da", + "representation": "received-document-bytes", + "scope": "complete-captured-source", + "bytes": 33870, + "reusedReadOnly": true + }, + { + "id": "deepseek-message", + "provider": "deepseek", + "url": "https://api-docs.deepseek.com/api/create-message.md", + "checkedAt": "2026-11-07", + "sha256": "46b31d10f429910bf93ea2166537fb803cd2f49f7c5b4ad2ee4f08257b2e17e8", + "file": "../official-model-registry/sources/deepseek-message.md", + "originalSha256": "46b31d10f429910bf93ea2166537fb803cd2f49f7c5b4ad2ee4f08257b2e17e8", + "representation": "received-document-bytes", + "scope": "complete-captured-source", + "bytes": 51134, + "reusedReadOnly": true + } + ], + "newRealModelRequests": 0, + "previousEvidenceRootsWritten": 0, + "scope": "request-block-fidelity-provider-state-and-affinity-contracts-not-live-cache-benefit" +} diff --git a/tests/fixtures/native-cache-state/sources/openai-conversation-state.md b/tests/fixtures/native-cache-state/sources/openai-conversation-state.md new file mode 100644 index 00000000..9b95c8f9 --- /dev/null +++ b/tests/fixtures/native-cache-state/sources/openai-conversation-state.md @@ -0,0 +1,795 @@ +# Conversation state + +OpenAI provides a few ways to manage conversation state, which is important for preserving information across multiple messages or turns in a conversation. + + + When troubleshooting cases where GPT-5.5 treats an intermediate update as + the final answer, verify your integration preserves the assistant message + `phase` field correctly. See [Phase + parameter](https://developers.openai.com/api/docs/guides/reasoning#phase-parameter) for details. + + +## Manually manage conversation state + +While each text generation request is independent and stateless, you can still implement **multi-turn conversations** by providing additional messages as parameters to your text generation request. Consider a knock-knock joke: + + + + Manually construct a past conversation + +```javascript +import OpenAI from "openai"; + +const openai = new OpenAI(); + +const response = await openai.responses.create({ + model: "gpt-6-astra", + input: [ + { role: "user", content: "knock knock." }, + { role: "assistant", content: "Who's there?" }, + { role: "user", content: "Orange." }, + ], +}); + +console.log(response.output_text); +``` + +```python +from openai import OpenAI + +client = OpenAI() + +response = client.responses.create( + model="gpt-6-astra", + input=[ + {"role": "user", "content": "knock knock."}, + {"role": "assistant", "content": "Who's there?"}, + {"role": "user", "content": "Orange."}, + ], +) + +print(response.output_text) +``` + +```go +package main + +import ( + "context" + "fmt" + + "github.com/openai/openai-go/v3" + "github.com/openai/openai-go/v3/responses" +) + +func main() { + client := openai.NewClient() + + response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + Input: responses.ResponseNewParamsInputUnion{ + OfInputItemList: responses.ResponseInputParam{ + responses.ResponseInputItemParamOfMessage("Knock knock.", responses.EasyInputMessageRoleUser), + responses.ResponseInputItemParamOfMessage("Who's there?", responses.EasyInputMessageRoleAssistant), + responses.ResponseInputItemParamOfMessage("Orange.", responses.EasyInputMessageRoleUser), + }, + }, + }) + if err != nil { + panic(err) + } + + fmt.Println(response.OutputText()) +} +``` + +```java +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.models.responses.EasyInputMessage; +import com.openai.models.responses.ResponseCreateParams; +import com.openai.models.responses.ResponseInputItem; +import java.util.List; + +ResponseCreateParams params = + ResponseCreateParams.builder() + .model("gpt-6-astra") + .inputOfResponse( + List.of( + ResponseInputItem.ofEasyInputMessage( + EasyInputMessage.builder() + .role(EasyInputMessage.Role.USER) + .content("Knock knock.") + .build()), + ResponseInputItem.ofEasyInputMessage( + EasyInputMessage.builder() + .role(EasyInputMessage.Role.ASSISTANT) + .content("Who's there?") + .build()), + ResponseInputItem.ofEasyInputMessage( + EasyInputMessage.builder() + .role(EasyInputMessage.Role.USER) + .content("Orange.") + .build()))) + .build(); + +client.responses().create(params).output().stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); +``` + +```csharp +using OpenAI.Responses; +#pragma warning disable OPENAI001 + +string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!; +ResponsesClient client = new(key); + +ResponseResult response = await client.CreateResponseAsync( + "gpt-6-astra", + [ + ResponseItem.CreateUserMessageItem("Knock knock."), + ResponseItem.CreateAssistantMessageItem("Who's there?"), + ResponseItem.CreateUserMessageItem("Orange."), + ] +); + +Console.WriteLine(response.GetOutputText()); +``` + +```ruby +require "openai" + +client = OpenAI::Client.new + +response = client.responses.create( + model: "gpt-6-astra", + input: [ + { + role: :user, + content: "Knock knock." + }, + { + role: :assistant, + content: "Who's there?" + }, + { + role: :user, + content: "Orange." + } + ] +) + +puts(response.output_text) +``` + + + +By using alternating `user` and `assistant` messages, you capture the previous state of a conversation in one request to the model. + +To manually share context across generated responses, include the model's previous response output as input, and append that input to your next request. + +For stateless reasoning-model requests, preserve every item in the response's `output` array. The Responses API returns encrypted reasoning items by default. Replaying the complete output keeps reasoning items and assistant `phase` values intact. Models that support persisted reasoning can use `reasoning.context: "all_turns"` to render the available reasoning from earlier turns into the next sample. See [preserve reasoning across calls](https://developers.openai.com/api/docs/guides/reasoning#preserve-reasoning-across-calls). + +In the following example, we ask the model to tell a joke, followed by a request for another joke. Appending previous responses to new requests in this way helps ensure conversations feel natural and retain the context of previous interactions. + + + + + Manually manage conversation state with the Responses API. + +```javascript +import OpenAI from "openai"; +import { toResponseInputItems } from "openai/lib/responses/ResponseInputItems"; + +const openai = new OpenAI(); + +let history = [ + { + role: "user", + content: "tell me a joke", + }, +]; + +const response = await openai.responses.create({ + model: "gpt-6-astra", + input: history, + store: false, +}); + +console.log(response.output_text); + +// Add replayable output items, including reasoning items, to the history +history.push(...toResponseInputItems(response.output)); + +history.push({ + role: "user", + content: "tell me another", +}); + +const secondResponse = await openai.responses.create({ + model: "gpt-6-astra", + input: history, + store: false, +}); + +console.log(secondResponse.output_text); +``` + +```python +from openai import OpenAI + +client = OpenAI() + +history = [{"role": "user", "content": "tell me a joke"}] + +response = client.responses.create( + model="gpt-6-astra", + input=history, + store=False, +) + +print(response.output_text) + +# Add all response output items, including encrypted reasoning items, to the conversation +history += response.output + +history.append({"role": "user", "content": "tell me another"}) + +second_response = client.responses.create( + model="gpt-6-astra", + input=history, + store=False, +) + +print(second_response.output_text) +``` + +```go +package main + +import ( + "context" + "encoding/json" + "fmt" + + "github.com/openai/openai-go/v3" + "github.com/openai/openai-go/v3/responses" +) + +func main() { + client := openai.NewClient() + history := responses.ResponseInputParam{ + responses.ResponseInputItemParamOfMessage("tell me a joke", responses.EasyInputMessageRoleUser), + } + first, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + Input: responses.ResponseNewParamsInputUnion{OfInputItemList: history}, + Store: openai.Bool(false), + }) + if err != nil { + panic(err) + } + fmt.Println(first.OutputText()) + + history = append(history, outputAsInput(first.Output)...) + history = append(history, responses.ResponseInputItemParamOfMessage("tell me another", responses.EasyInputMessageRoleUser)) + second, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + Input: responses.ResponseNewParamsInputUnion{OfInputItemList: history}, + Store: openai.Bool(false), + }) + if err != nil { + panic(err) + } + fmt.Println(second.OutputText()) +} + +func outputAsInput(output []responses.ResponseOutputItemUnion) []responses.ResponseInputItemUnionParam { + input := make([]responses.ResponseInputItemUnionParam, 0, len(output)) + for _, item := range output { + var converted responses.ResponseInputItemUnion + if err := json.Unmarshal([]byte(item.RawJSON()), &converted); err != nil { + panic(err) + } + input = append(input, converted.ToParam()) + } + return input +} +``` + +```java +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.core.JsonValue; +import com.openai.models.responses.EasyInputMessage; +import com.openai.models.responses.ResponseCreateParams; +import com.openai.models.responses.ResponseInputItem; +import java.util.ArrayList; + +var history = new ArrayList(); +history.add( + ResponseInputItem.ofEasyInputMessage( + EasyInputMessage.builder() + .role(EasyInputMessage.Role.USER) + .content("Tell me a joke.") + .build())); + +var first = + client + .responses() + .create( + ResponseCreateParams.builder() + .model("gpt-6-astra") + .inputOfResponse(history) + .store(false) + .build()); +first.output().stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); +first.output().stream() + .map(item -> JsonValue.from(item).convert(ResponseInputItem.class)) + .forEach(history::add); +history.add( + ResponseInputItem.ofEasyInputMessage( + EasyInputMessage.builder() + .role(EasyInputMessage.Role.USER) + .content("Tell me another.") + .build())); + +client + .responses() + .create( + ResponseCreateParams.builder() + .model("gpt-6-astra") + .inputOfResponse(history) + .store(false) + .build()) + .output() + .stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); +``` + +```csharp +using OpenAI.Responses; +#pragma warning disable OPENAI001 + +string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!; +ResponsesClient client = new(key); + +List history = +[ + ResponseItem.CreateUserMessageItem("Tell me a joke."), +]; + +CreateResponseOptions options = new("gpt-6-astra", history) +{ + StoredOutputEnabled = false, + IncludedProperties = + { + IncludedResponseProperty.ReasoningEncryptedContent, + }, +}; +ResponseResult first = await client.CreateResponseAsync(options); +Console.WriteLine(first.GetOutputText()); + +history.AddRange(first.OutputItems); +history.Add(ResponseItem.CreateUserMessageItem("Tell me another.")); + +options = new("gpt-6-astra", history) +{ + StoredOutputEnabled = false, + IncludedProperties = + { + IncludedResponseProperty.ReasoningEncryptedContent, + }, +}; +ResponseResult second = await client.CreateResponseAsync(options); +Console.WriteLine(second.GetOutputText()); +``` + +```ruby +require "openai" + +client = OpenAI::Client.new +history = [ + { + role: :user, + content: "Tell me a joke." + } +] + +first = client.responses.create( + model: "gpt-6-astra", + input: history, + store: false +) +puts(first.output_text) + +history.concat(first.output) +history << { + role: :user, + content: "Tell me another." +} + +second = client.responses.create( + model: "gpt-6-astra", + input: history, + store: false +) +puts(second.output_text) +``` + + + +## OpenAI APIs for conversation state + +Our APIs make it easier to manage conversation state automatically, so you don't have to pass inputs manually with each turn of a conversation. + + + + + +### Using the Conversations API + +The [Conversations API](https://developers.openai.com/api/reference/resources/conversations/methods/create) works with the [Responses API](https://developers.openai.com/api/reference/resources/responses/methods/create) to persist conversation state as a long-running object with its own durable identifier. After creating a conversation object, you can keep using it across sessions, devices, or jobs. + +Conversations store items, which can be messages, tool calls, tool outputs, and other data. + + Create a conversation + +```javascript +const conversation = await client.conversations.create(); +``` + +```python +conversation = openai.conversations.create() +``` + +```go +conversation, err := client.Conversations.New(context.Background(), conversations.ConversationNewParams{}) +if err != nil { + panic(err) +} +``` + +```java +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; + +var conversation = client.conversations().create(); + +System.out.println(conversation.id()); +``` + +```ruby +conversation = client.conversations.create +``` + + +In a multi-turn interaction, you can pass the `conversation` into subsequent responses to persist state and share context across subsequent responses, rather than having to chain multiple response items together. + + Manage conversation state with Conversations and Responses APIs + +```javascript +const response = await client.responses.create({ + model: "gpt-6-astra", + input: [{ role: "user", content: "What are the five Ds of dodgeball?" }], + conversation: conversation.id, +}); + +console.log(response.output_text); +``` + +```python +response = openai.responses.create( + model="gpt-6-astra", + input=[{"role": "user", "content": "What are the 5 Ds of dodgeball?"}], + conversation=conversation.id, +) +``` + +```go +response, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + Conversation: responses.ResponseNewParamsConversationUnion{ + OfString: openai.String(conversation.ID), + }, + Input: responses.ResponseNewParamsInputUnion{ + OfString: openai.String("What are the five Ds of dodgeball?"), + }, +}) +if err != nil { + panic(err) +} +fmt.Println(response.OutputText()) +``` + +```java +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.models.responses.ResponseCreateParams; + +var conversation = client.conversations().create(); + +var response = + client + .responses() + .create( + ResponseCreateParams.builder() + .model("gpt-6-astra") + .conversation(conversation.id()) + .input("What are the five Ds of dodgeball?") + .build()); + +response.output().stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); +``` + +```ruby +response = client.responses.create( + model: "gpt-6-astra", + conversation: conversation.id, + input: "What are the five Ds of dodgeball?" +) + +puts(response.output_text) +``` + + +### Passing context from the previous response + +Another way to manage conversation state is to share context across generated responses with the `previous_response_id` parameter. This parameter lets you chain responses and create a threaded conversation. + + Chain responses across turns by passing the previous response ID + +```javascript +import OpenAI from "openai"; + +const openai = new OpenAI(); + +const response = await openai.responses.create({ + model: "gpt-6-astra", + input: "tell me a joke", + store: true, +}); + +console.log(response.output_text); + +const secondResponse = await openai.responses.create({ + model: "gpt-6-astra", + previous_response_id: response.id, + input: [{ role: "user", content: "explain why this is funny." }], + store: true, +}); + +console.log(secondResponse.output_text); +``` + +```python +from openai import OpenAI + +client = OpenAI() + +response = client.responses.create( + model="gpt-6-astra", + input="tell me a joke", +) +print(response.output_text) + +second_response = client.responses.create( + model="gpt-6-astra", + previous_response_id=response.id, + input=[{"role": "user", "content": "explain why this is funny."}], +) +print(second_response.output_text) +``` + +```go +package main + +import ( + "context" + "fmt" + + "github.com/openai/openai-go/v3" + "github.com/openai/openai-go/v3/responses" +) + +func main() { + client := openai.NewClient() + + first, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + Input: responses.ResponseNewParamsInputUnion{ + OfString: openai.String("Tell me a joke."), + }, + }) + if err != nil { + panic(err) + } + fmt.Println(first.OutputText()) + + second, err := client.Responses.New(context.Background(), responses.ResponseNewParams{ + Model: "gpt-6-astra", + PreviousResponseID: openai.String(first.ID), + Input: responses.ResponseNewParamsInputUnion{ + OfString: openai.String("Explain why this is funny."), + }, + }) + if err != nil { + panic(err) + } + fmt.Println(second.OutputText()) +} +``` + +```java +import com.openai.client.OpenAIClient; +import com.openai.client.okhttp.OpenAIOkHttpClient; +import com.openai.models.responses.ResponseCreateParams; + +var first = + client + .responses() + .create( + ResponseCreateParams.builder().model("gpt-6-astra").input("Tell me a joke.").build()); + +first.output().stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); + +var second = + client + .responses() + .create( + ResponseCreateParams.builder() + .model("gpt-6-astra") + .input("Explain why this is funny.") + .previousResponseId(first.id()) + .build()); +second.output().stream() + .flatMap(item -> item.message().stream()) + .flatMap(message -> message.content().stream()) + .flatMap(content -> content.outputText().stream()) + .forEach(text -> System.out.println(text.text())); +``` + +```csharp +using OpenAI.Responses; +#pragma warning disable OPENAI001 + +string key = Environment.GetEnvironmentVariable("OPENAI_API_KEY")!; +ResponsesClient client = new(key); + +ResponseResult first = await client.CreateResponseAsync( + "gpt-6-astra", + "Tell me a joke." +); +Console.WriteLine(first.GetOutputText()); + +ResponseResult second = await client.CreateResponseAsync( + "gpt-6-astra", + "Explain why this is funny.", + previousResponseId: first.Id +); +Console.WriteLine(second.GetOutputText()); +``` + +```ruby +require "openai" + +client = OpenAI::Client.new + +first = client.responses.create( + model: "gpt-6-astra", + input: "Tell me a joke." +) +puts(first.output_text) + +second = client.responses.create( + model: "gpt-6-astra", + previous_response_id: first.id, + input: "Explain why this is funny." +) +puts(second.output_text) +``` + + +#### `previous_response_id` in WebSocket mode + +If you are using [the Responses API WebSocket mode](https://developers.openai.com/api/docs/guides/websocket-mode), continuation uses the same `previous_response_id` semantics as HTTP mode, but over a persistent socket with repeated `response.create` events. + +The connection-local cache keeps recent previous responses in memory for low-latency continuation. When you use `stream_id`, each lane can retain its latest response; `previous_response_id` still controls lineage, so a new lane can fork from a response on another lane while that response remains available. If an uncached ID cannot be resolved, send a new turn with `previous_response_id` set to `null` and pass full input context. + + + + + +##### Data retention for model responses + + + Response objects are saved for 30 days by default. They can be viewed in the dashboard + [logs](https://platform.openai.com/logs?api=responses) page or + [retrieved](https://developers.openai.com/api/reference/resources/responses/methods/retrieve) via the API. + You can disable this behavior by setting `store` to `false` + when creating a Response. + + Conversation objects and items in them are not subject to the 30 day TTL. Any response attached to a conversation will have its items persisted with no 30 day TTL. + + OpenAI does not use data sent via API to train our models without your explicit consent—[learn more](https://developers.openai.com/api/docs/guides/your-data). + + + + + + +Even when using `previous_response_id`, all previous input tokens for responses in the chain are billed as input tokens in the API. + + + +## Managing the context window + +Understanding context windows will help you successfully create threaded conversations and manage state across model interactions. + +The **context window** is the maximum number of tokens that can be used in a single request. This max tokens number includes input, output, and reasoning tokens. To learn your model's context window, see [model details](https://developers.openai.com/api/docs/models). + +### Managing context for text generation + +As your inputs become more complex, or you include more turns in a conversation, you'll need to consider both **output token** and **context window** limits. Model inputs and outputs are metered in [**tokens**](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them), which are parsed from inputs to analyze their content and intent and assembled to render logical outputs. Models have limits on token usage during the lifecycle of a text generation request. + +- **Output tokens** are the tokens generated by a model in response to a prompt. Each model has different [limits for output tokens](https://developers.openai.com/api/docs/models). For example, `gpt-4o-2024-08-06` can generate a maximum of 16,384 output tokens. +- A **context window** describes the total tokens that can be used for both input and output tokens (and for some models, [reasoning tokens](https://developers.openai.com/api/docs/guides/reasoning)). Compare the [context window limits](https://developers.openai.com/api/docs/models) of our models. For example, `gpt-4o-2024-08-06` has a total context window of 128k tokens. + +If you create a large prompt—often by including extra context, data, or examples for the model—you run the risk of exceeding the allocated context window for a model, which might result in truncated outputs. + +Use the [tokenizer tool](https://platform.openai.com/tokenizer), built with the [tiktoken library](https://github.com/openai/tiktoken), to see how many tokens are in a particular string of text. + + + +For example, when making an API request to the [Responses API](https://developers.openai.com/api/reference/resources/responses) with a reasoning enabled model, like the [o1 model](https://developers.openai.com/api/docs/guides/reasoning), the following token counts will apply toward the context window total: + +- Input tokens (inputs you include in the `input` array for the [Responses API](https://developers.openai.com/api/reference/resources/responses)) +- Output tokens (tokens generated in response to your prompt) +- Reasoning tokens (used by the model to plan a response) + + +Tokens generated in excess of the context window limit may be truncated in API responses. + +![context window visualization](https://cdn.openai.com/API/docs/images/context-window.png) + +You can estimate the number of tokens your messages will use with the [tokenizer tool](https://platform.openai.com/tokenizer). + + + +### Compaction + +Detailed compaction guidance now lives in +[Compaction](https://developers.openai.com/api/docs/guides/compaction). + +- For `/responses` with `context_management` and `compact_threshold`, see + [Server-side compaction](https://developers.openai.com/api/docs/guides/compaction#server-side-compaction). +- For explicit compaction control, see + [Standalone compact endpoint](https://developers.openai.com/api/docs/guides/compaction#standalone-compact-endpoint) + and the [`/responses/compact` API reference](https://developers.openai.com/api/reference/resources/responses/methods/compact). + +## Next steps + +For more specific examples and use cases, visit the [OpenAI Cookbook](https://developers.openai.com/cookbook), or learn more about using the APIs to extend model capabilities: + +- [Receive JSON responses with Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs) +- [Extend the models with function calling](https://developers.openai.com/api/docs/guides/function-calling) +- [Enable streaming for real-time responses](https://developers.openai.com/api/docs/guides/streaming-responses) +- [Build a computer-using agent](https://developers.openai.com/api/docs/guides/tools-computer-use) \ No newline at end of file diff --git a/tests/fixtures/native-cache-state/sources/openai-prompt-caching.md b/tests/fixtures/native-cache-state/sources/openai-prompt-caching.md new file mode 100644 index 00000000..4fe7ab00 --- /dev/null +++ b/tests/fixtures/native-cache-state/sources/openai-prompt-caching.md @@ -0,0 +1,1127 @@ +# Prompt caching + +## Why prompt caching matters + +Prompt caching reuses work when requests share the same prompt prefix. This provides three main benefits: + +- **Compute-efficient:** Avoid recalculating a prompt prefix that the model has already processed. +- **Cheaper input tokens:** Pay the model's reduced cached-input rate for reused tokens, discounted up to 95%. +- **Faster:** Reduce the time spent processing input before the response starts. + +Prompt caching is enabled by default for supported OpenAI models. Use the [Prompt Caching Dashboard](https://platform.openai.com/usage?usage_section=prompt-caching) to monitor cache read hit rates and use the [Prompt Cache Diagnostics tool](https://developers.openai.com/api/docs/guides/prompt-caching/diagnostics) to diagnose cache misses and improve cache reuse. + +Agents API model calls use the same prompt-caching behavior as the Responses API. Reusing context within a session can preserve a shared prompt prefix, but maintaining a session doesn't guarantee a cache hit. See [Observability and usage](https://developers.openai.com/api/docs/guides/agents-api/observability) for session usage fields and subagent accounting. + +Prompt caching pricing varies by model. See [API pricing](https://developers.openai.com/api/docs/pricing) for current cached-input and cache-write rates. Cache-write pricing is not an additive fee: input tokens use the uncached-input, cached-input, or cache-write rate. + +## What is the prompt cache? + +When the model processes input tokens, it must calculate intermediate states, known as key-value (KV) states. These states let the model refer back to earlier tokens while processing new input and generating output tokens. + +Prompt caching preserves that state for a reusable **prefix**: the unchanged tokens at the beginning of a prompt. When a later request has the same prefix and finds a matching cache entry, the model can reuse the saved state instead of processing those tokens again. It still needs to process any new input to generate a new response. + +The prompt cache stores key-value (KV) tensors, not the tokens themselves. + + + +Ask ChatGPT for a deeper explanation + + + +OpenAI caches the model's full rendered context including OpenAI-provided instructions, [developer messages](https://developers.openai.com/api/docs/guides/prompt-engineering#message-roles-and-instruction-following), [tool definitions](https://developers.openai.com/api/docs/guides/function-calling), and [conversation history](https://developers.openai.com/api/docs/guides/conversation-state) containing [text](https://developers.openai.com/api/docs/guides/text), [images](https://developers.openai.com/api/docs/guides/images-vision), [documents](https://developers.openai.com/api/docs/guides/file-inputs), and supported [audio](https://developers.openai.com/api/docs/guides/audio). + +Cache reuse requires the entire rendered prefix to match. If content or a relevant setting changes before a breakpoint, the prefix after that change cannot match the existing cache entry. + + + + + + + + + +### Which settings affect the cached prefix? + + + +Changing a request does not necessarily discard an existing cache entry. What matters is whether a subsequent request has the same prefix and can find an eligible matching breakpoint. The main settings to check are: + +| Setting | Impact | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [`model`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20model%20%3E%20%28schema%29) | A different model can use different weights and caching behavior. | +| [`tools`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20tools%20%3E%20%28schema%29) | Changes tool names, descriptions, schemas, ordering, or tool-specific instructions. | +| [`parallel_tool_calls`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20parallel_tool_calls%20%3E%20%28schema%29) | Can change instructions about calling multiple tools in one turn. | +| [`text.format`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20text%20%3E%20%28schema%29) ([Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs)) | On supported GPT-6 and later models, [append a format update](https://developers.openai.com/api/docs/guides/structured-outputs?api-mode=responses#change-the-response-format-mid-conversation) to keep the original schema in the prefix. | +| [`reasoning.effort`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20reasoning%20%3E%20%28schema%29) | Can change model-side reasoning instructions. On supported models, use a [configuration update](#change-reasoning-effort-without-rewriting-the-prefix) to change effort while preserving the earlier prefix. | +| [`text.verbosity`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20text%20%3E%20%28schema%29) | Can change instructions about response detail. | +| [`context_management`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20context_management%20%3E%20%28schema%29) ([Compaction](https://developers.openai.com/api/docs/guides/compaction)) | Replaces earlier conversation content with a compacted context that can prevent reuse from the first changed token onward. | + + + + + +## How caching works + +A **cache breakpoint** marks the end of a prompt prefix that OpenAI can save to the cache and reuse in later requests. The first request writes an eligible prefix to the cache and subsequent requests look for the longest matching cached prefix available, working backward through eligible breakpoints until they find a match. + +A prompt prefix must meet the model's **minimum cacheable token length** before it can be cached. Tokens in the OpenAI-provided hidden system content do not count toward this minimum. The minimum cacheable prompt length is 1,024 tokens for GPT-5.6 and later and varies by request settings for earlier models. See the [model comparison](#summary-of-model-differences) for details. + +After the minimum cacheable token length, you can choose where to place cache breakpoints explicitly, or let OpenAI choose their locations implicitly. The available options depend on the model. + + + + + + + +### GPT-5.6 and later + + + +For GPT-5.6 and later, cache writes cost 1.25× the standard, uncached input-token rate for the default 30-minute lifetime. In the Responses API, you can choose a [12-hour minimum lifetime](#cache-lifetime) at a write rate of 2.5× ordinary input. Subsequent reads cost 0.1× the ordinary input rate on most of these models and 0.05× on [GPT-6.1 Sol](https://developers.openai.com/api/docs/models/gpt-6.1-sol) and [GPT-6.1 Terra](https://developers.openai.com/api/docs/models/gpt-6.1-terra), regardless of the selected lifetime. + +At the 0.1× read rate with 30-minute writes, writing a prefix once and fully reusing it once costs 1.35× its ordinary input cost, compared with 2× for processing it twice without caching. Across ten requests, one write and nine full reads cost 2.15× at that rate, compared with 10× without caching. + +Both implicit and explicit caching are supported, where explicit caching gives you more control over which context is written to cache. + +**Explicit mode:** You choose where to place cache breakpoints based on your context management. + +- Set `prompt_cache_options.mode` to `explicit` to use only developer-selected breakpoints and mark each desired breakpoint by adding `prompt_cache_breakpoint: { "mode": "explicit" }` to a supported content block inside an input message. +- When no explicit breakpoints are placed, the request does not use prompt caching or create cache writes. +- Explicit-only mode lets you choose where cache writes end. Content after the last selected breakpoint is processed at the uncached input-token rate without a cache-write charge, so you can avoid writing changing content that is unlikely to be reused. +- Multiple explicit breakpoints can preserve prefixes that change at different rates. Each request can create up to four cache writes. +- `additional_tools` input items do not currently accept `prompt_cache_breakpoint`. + +Top-level `instructions` cannot contain an explicit breakpoint. To mark reusable developer instructions, place them in an `input_text` block inside a developer message. + +**Implicit mode:** OpenAI chooses breakpoint locations out of the box that work well for most use cases. + +- When `prompt_cache_options.mode` is `implicit`, OpenAI places a breakpoint at the end of the latest eligible message. Eligible messages are: + - user messages + - the last tool response in a consecutive group of tool responses + - the last developer message in the initial consecutive group of developer messages. +- You can add explicit breakpoints without turning off the implicit breakpoint; an implicit breakpoint uses one of the four cache write slots to leave three usable explicit cache write slots. + + + + + + + + + + + +### Earlier models + + + +Only implicit caching is supported. OpenAI places implicit breakpoints at [model-dependent intervals](#summary-of-model-differences), counted from the beginning of the hidden OpenAI system message. Only breakpoints at or beyond the minimum cacheable length (counted from the end of the hidden context) are eligible. + +Reported `cached_tokens` is calculated by subtracting the hidden system tokens from the last matched breakpoint, then rounding down to the nearest multiple of 128. + + + + + +### How prefix matching works + +OpenAI walks through only the **cache lookup boundaries** (explained below) in the incoming request, from longest prefix to shortest, looking for an available matching prefix already cached on the machine. + +For GPT-5.6 and later, the cache lookup boundaries in the incoming request are: + +- **Explicit-only mode:** The first 2 and latest 50 explicit breakpoints. +- **Implicit mode:** The first 2 and latest 50 explicit breakpoints, the implicit breakpoint, up to 20 earlier eligible message endings, and the endpoint of the initial consecutive block of developer messages. This lets implicit mode reuse a prefix ending at an earlier message without explicit breakpoints there. + +## Cache lifetime + +Cache entries are not stored indefinitely. The minimum cache lifetime, retention settings, and behavior when a request reuses a prefix [depend on the model](#summary-of-model-differences). + + + + + + + + + +### GPT-5.6 and later + + + +In the Responses API, set `prompt_cache_options.ttl` to choose the minimum cache lifetime: + +| Value | Minimum lifetime | Cache-write rate | When to use it | +| ----------------- | ---------------- | ----------------------------------- | -------------------------------------------------------------------- | +| `"30m"` (default) | 30 minutes | 1.25× the uncached input-token rate | Conversations and agent steps that reuse context frequently. | +| `"12h"` | 12 hours | 2.5× the uncached input-token rate | Stable instructions or reference material reused across longer gaps. | + +These are total write rates, not surcharges on top of ordinary input. Cache-read rates are the same for both lifetimes. OpenAI may retain entries longer than the requested minimum; `"12h"` does not set a deletion deadline or change the [maximum data-retention policy](https://developers.openai.com/api/docs/guides/your-data#storage-requirements-and-retention-controls-per-endpoint). + +The request-level `ttl` applies to the implicit breakpoint and to explicit breakpoints without their own `ttl`. Set `prompt_cache_breakpoint.ttl` to override it for an individual explicit breakpoint. Omitting both settings preserves the default `"30m"` behavior. Unsupported values return `400 invalid_request_error`. These settings do not change the older-model `prompt_cache_retention` parameter. + +**Mix lifetimes in one request** + +Keep material reused throughout the day at the start of the prompt and give its breakpoint a `"12h"` lifetime. Use `"30m"` for session context that changes more frequently. This example sets a 30-minute request default and overrides it for the shared prefix. Explicit-only mode leaves the current question after the last breakpoint, so that content does not incur a cache-write charge. + +Responses API request with 12-hour and 30-minute breakpoints + +```json +{ + "model": "gpt-6.1-sol", + "prompt_cache_options": { "mode": "explicit", "ttl": "30m" }, + "input": [ + { + "role": "developer", + "content": [ + { + "type": "input_text", + "text": "Shared instructions and reference material reused throughout the day...", + "prompt_cache_breakpoint": { "mode": "explicit", "ttl": "12h" } + } + ] + }, + { + "role": "user", + "content": [ + { + "type": "input_text", + "text": "Session context reused over the next few minutes...", + "prompt_cache_breakpoint": { "mode": "explicit" } + }, + { + "type": "input_text", + "text": "The current question, which does not need a cache write..." + } + ] + } + ] +} +``` + + +Replace the placeholder text with your content. Each prefix you want to cache must meet the model's [minimum cacheable length](#summary-of-model-differences). + +**Overlapping writes and lifetime changes** + +A breakpoint covers the entire prefix before it. Within one request, each token is billed once at the longest lifetime of the selected writes covering it. A later 12-hour write also gives earlier selected breakpoints within that prefix 12-hour coverage. + +These examples assume any existing cached prefix matches an [eligible lookup boundary](#how-prefix-matching-works) in the new request. + +| Cache state before the request | Selected breakpoints | Billed reads | Billed 30-minute writes | Billed 12-hour writes | +| --------------------------------------------- | ---------------------------------------------- | ------------ | ----------------------- | --------------------- | +| No cached prefix | `"12h"` at 2,000 tokens, then `"30m"` at 5,000 | 0 | 3,000 | 2,000 | +| No cached prefix | `"30m"` at 2,000 tokens, then `"12h"` at 5,000 | 0 | 0 | 5,000 | +| Valid 30-minute coverage through 2,000 tokens | `"12h"` at 5,000 | 0 | 0 | 5,000 | +| Valid 12-hour coverage through 2,000 tokens | `"30m"` at 5,000 | 2,000 | 3,000 | 0 | + +Upgrading valid 30-minute coverage to 12 hours charges a new 12-hour write for those tokens, without credit for the earlier write. Any portion already covered by a valid 12-hour entry remains a cache read. + +A 30-minute read refreshes coverage to at least 30 minutes after that read. If it uses an entry with valid 12-hour coverage, it preserves the later of the existing expiry and 30 minutes after the read. It does not restart a 12-hour clock or incur another write charge. + +The minimum lifetime governs eligibility for cache-read pricing. It does not guarantee that every request reuses stored KV tensors or has lower latency. + + + + + + + + + + + + + +### Earlier models + + + +Use `prompt_cache_retention`, with supported values that depend on the model: + +- `in_memory`: Entries typically remain active for around 5 to 10 minutes of inactivity, up to one hour. +- `24h`: Extended retention typically keeps entries available for around 30 minutes and can retain them for up to 24 hours. + +**Retention defaults and Zero Data Retention** + +Prompt caching may store encrypted key/value tensors in GPU-local storage as application state. For models that support both `in_memory` and `24h`, the default depends on your organization's data retention policy: + +- Organizations _without_ Zero Data Retention enabled default to `24h`. +- Organizations _with_ Zero Data Retention enabled default to `in_memory`. + +Verify the available retention policies for your model and organization before selecting a value. + + + + + + + + + + + +## Cache location + +Cached states live on individual machines, where traffic above 15 requests per minute can lead to overflow routing. A request can reuse a cached prefix only if it reaches a machine holding a matching entry that has not expired. Routing requests to the right machine is therefore important for cache reuse. + +Caches are not shared across organizations and cannot be reused across [regional processing boundaries](https://developers.openai.com/api/docs/guides/your-data#data-residency-controls). + +OpenAI handles routing automatically. Within an organization and processing region, routing for a given model depends on: + +- Current machine load and available capacity. +- A hash of the initial tokens after the hidden OpenAI content, including tool definitions when present. The number of tokens hashed varies by model. +- A supplied [`prompt_cache_key`](#prompt-cache-keys), which separates cache reuse between groups of requests and helps optimize cache routing on models before GPT-5.6. + + + + + + + +### Prompt cache keys + + + +On models before GPT-5.6, use a stable [`prompt_cache_key`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20prompt_cache_key%20%3E%20%28schema%29) for requests that share a reusable prefix to help route related requests to the same cache. For busy groups, aim for about 15 requests per minute in total across all prefixes using each key. Partition higher-volume traffic across multiple keys using a stable, deterministic mapping. Keep related requests on the same `prompt_cache_key` so they can reuse its cache. Keys influence routing; they do not pin requests to a machine or guarantee a cache hit. + +On GPT-5.6 and later, OpenAI handles cache routing automatically; the key is not needed to optimize caching. You can use separate keys to maintain separate cache accounting for customers or users within your application. + +Using separate keys can make cached token usage and billing easier to explain for each customer or user. For example, separate keys help prevent cache-hit probing across users: submitting candidate prompts and observing cache hits to learn whether matching content was previously cached. See [Separate cache accounting with keys](#separate-prompts-with-cache-keys). + + + + + + + +## Summary of model differences + +| Behavior | GPT-5.6 and later | GPT-5.5 and GPT-5.5 Pro | Other earlier models | +| -------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------- | +| Implicit breakpoints | At the end of the latest eligible message. | Spaced at regular 2,048-token intervals. | Spaced at regular, model-dependent intervals. | +| Explicit breakpoints | Supported | Not supported | Not supported | +| `prompt_cache_key` | Optional for separate cache accounting | Use a stable key to optimize cache routing | Use a stable key to optimize cache routing | +| Minimum cacheable prefix | 1,024 visible input tokens | Varies by request settings | Varies by request settings | +| Cached-token reporting | Exact eligible boundary, excluding hidden tokens | Excludes hidden tokens and rounds down to a multiple of 128 | Excludes hidden tokens and rounds down to a multiple of 128 | +| Cache read charge | 0.1× uncached input (0.05× for GPT-6.1 Sol and GPT-6.1 Terra) | Model-dependent cached-input rate | Model-dependent cached-input rate | +| Cache write charge | 1.25× for `"30m"`; 2.5× for `"12h"` in the Responses API | No additional cache-write charge | No additional cache-write charge | +| Cache lifetime control | `prompt_cache_options.ttl` | `prompt_cache_retention` | `prompt_cache_retention` | +| Supported retention values | `"30m"` (default), or `"12h"` in the Responses API | `"24h"` only | `"in_memory"` or `"24h"`[\*](#extended-retention-models) | +| Cache lifetime | Minimum lifetime of 30 minutes or 12 hours; see [refresh behavior](#cache-lifetime) | Typically around 30 minutes, up to 24 hours | Typically 5 to 10 minutes inactive for `in_memory`, or up to 24 hours for `24h` | + + + + + + +\* Extended retention is supported by `gpt-5.5`, `gpt-5.5-pro`, `gpt-5.4`, `gpt-5.2`, `gpt-5.1-codex-max`, `gpt-5.1`, `gpt-5.1-codex`, `gpt-5.1-codex-mini`, `gpt-5.1-chat-latest`, `gpt-5`, `gpt-5-codex`, and `gpt-4.1`. + + + + +For models before GPT-5.6, the minimum cacheable input length varies with request settings, including tools, images, output schemas, reasoning effort, and verbosity. + + + +Ask ChatGPT to find the cache minimum for my request + + + + + +## How to optimize prompt caching + +Focus on [preserving conversation history](#preserve-conversation-history), [keeping tool definitions stable](#manage-tools-with-append-only-updates), and choosing where caching occurs. On GPT-5.6 and later, use [`prompt_cache_options.mode` and `prompt_cache_breakpoint`](#choose-a-caching-mode) to control cache breakpoints. You can also use an optional [`prompt_cache_key`](#separate-prompts-with-cache-keys) if your application needs separate cache accounting for customers. On models before GPT-5.6, use a stable `prompt_cache_key` to optimize cache routing for requests that share a reusable prefix. + + + +Ask ChatGPT to optimize my prompt caching + + + + + + + + + +### Preserve conversation history + + + +In multi-turn applications, reusing the growing conversation history can save more input tokens than caching only the initial instructions. Preserve earlier messages and tool results so later turns can reuse the full shared prefix. + +- **Keep the prefix stable.** Put stable developer instructions and shared reference material first. If developer instructions or shared material contain timestamps, user-specific content, or other dynamic content, place those at the end rather than the beginning, or move them into later conversation messages. +- **Preserve conversation history.** Append new messages rather than rewriting earlier turns. Summarization, [compaction](#compaction-can-reduce-cache-reuse), or context truncation can change the prefix and reset cache reuse. +- **Change reasoning effort without rewriting the prefix.** On GPT-6 models, append a `configuration_update` input item to change reasoning effort between responses while keeping request-level `reasoning.effort` unchanged. This preserves the original prefix for cache reuse. See [Change reasoning mid-conversation](https://developers.openai.com/api/docs/guides/reasoning#change-reasoning-mid-conversation) for examples and compatibility limits. + +Keep changing content after the breakpoint + +```json +{ + "model": "gpt-6.1-sol", + "reasoning": { "effort": "low", "context": "all_turns" }, + "text": { "verbosity": "medium" }, + "prompt_cache_options": { "mode": "explicit" }, + "input": [ + { + "role": "developer", + "content": [ + { + "type": "input_text", + "text": "Stable instructions and shared reference material...", + "prompt_cache_breakpoint": { "mode": "explicit" } + } + ] + }, + { + "role": "developer", + "content": "Dynamic developer instructions, such as user-specific content and timestamps..." + }, + { + "role": "user", + "content": "The user's current question..." + } + ] +} +``` + + + + + + + + + + + + +### Change reasoning effort without rewriting the prefix + + + +On supported GPT-6 and later models, append a `configuration_update` input item to [change reasoning effort during a conversation](https://developers.openai.com/api/docs/guides/reasoning?api-mode=responses#change-reasoning-mid-conversation) while preserving the earlier cached prefix. Keep the top-level `reasoning.effort` at its original value as changing that setting can rewrite instructions in the hidden system instructions. + +The latest update to `reasoning.effort` controls the effort for later responses. Append this item to `input` to switch to `high` effort. + +Item to append to the input array + +```json +{ + "type": "configuration_update", + "reasoning": { "effort": "high" } +} +``` + + + + + + + + + + + + + + +### Manage tools with append-only updates + + + +When the tools your application needs vary between requests, change which tools are callable while keeping their definitions stable to preserve reusable prefixes. + +- **Keep tools consistent.** Preserve tool definitions, ordering, and schemas. +- **Disable tool use for a request.** Set [`tool_choice`](https://developers.openai.com/api/docs/guides/function-calling#tool-choice) to `"none"` instead of removing the tool definitions. +- **Enable only selected tools.** Use [`allowed_tools`](https://developers.openai.com/api/docs/guides/function-calling#tool-choice) to restrict which tools are callable while keeping the supplied `tools` list stable. +- **Load tools when needed.** Use [tool search](https://developers.openai.com/api/docs/guides/tools-tool-search) with `defer_loading: true` to reduce input tokens spent on tool definitions in early requests of multi-turn threads. Discovered tools are appended at the end of context, preserving earlier reusable content. +- **Preserve tool-loading history.** Use a developer-role [`additional_tools` input item](https://developers.openai.com/api/docs/guides/tools-tool-search#add-tools-at-a-specific-point-in-the-input) to add tools during a thread according to your application's logic. + + + + + + + + + + + +### Choose a caching mode + + + +On GPT-5.6 and later, two controls determine where cache breakpoints are placed: `prompt_cache_options.mode` selects implicit or explicit-only caching, and `prompt_cache_breakpoint` marks a boundary you choose. + +- **Place breakpoints automatically.** Use implicit caching to place a breakpoint at the end of the latest eligible message. This is convenient for multi-turn threads that append to existing context. +- **Choose breakpoints deliberately.** Place explicit markers at the end of stable content. Use explicit-only mode to avoid unnecessary cache writes for changing suffixes. + + + +> Illustration: In explicit-only mode, tools and schemas precede a stable developer-message prefix and breakpoint 1. One branch adds a variable developer suffix and more conversation turns before breakpoint 2, then splits into new user inputs. Another branch has an unselected variable suffix. Content after each branch's last selected breakpoint is charged at the uncached input rate without a cache-write charge. + + + + + + + + + + + + + +### Prewarm the cache + + + +For GPT-5.6 and later, prepare known context ahead of time to reduce time to first token on a subsequent request. For example, an interactive application can prewarm shared instructions, tool definitions, or reference material during startup, before the user asks their first question. + +Set [`prompt_cache_options.prewarm`](https://developers.openai.com/api/reference/resources/responses/methods/create#%28resource%29%20responses%20%3E%20%28method%29%20create%20%3E%20%28params%29%200.non_streaming%20%3E%20%28param%29%20prompt_cache_options%20%3E%20%28schema%29%20%3E%20%28property%29%20prewarm) to `true` in a Responses API request to prepare the prompt cache without generating output. Once it completes, send your actual request with the same prompt prefix and `prewarm` omitted or set to `false`. + +Prewarm the cache + +```json +{ + "model": "gpt-6.1-sol", + "input": [ + { + "role": "developer", + "content": "Your app's shared instructions and reference material..." + } + ], + "prompt_cache_options": { + "prewarm": true + } +} +``` + + +Send a follow-up request + +```json +{ + "model": "gpt-6.1-sol", + "input": [ + { + "role": "developer", + "content": "Your app's shared instructions and reference material..." + }, + { + "role": "user", + "content": "The user's question..." + } + ] +} +``` + + +Tokens written to the cache during a prewarm request are billed at the [cache-write rate for the selected lifetime](#cache-lifetime). + + + + + + + + + + + + + + + + + +### Separate cache accounting with keys + + + +On GPT-5.6 and later, use `prompt_cache_key` when you want to maintain separate cache accounting for customers, users, or workspaces within your application. This can make cached token usage and billing easier to explain within each group. The key is optional and is not needed to optimize caching on these models. + +- **Choose how to separate cache accounting.** Assign a distinct key to each customer or user whose cache accounting should remain separate. For example, `support:customer_123` and `support:customer_456` maintain separate cache accounting for two customers, even when their requests contain the same prefix. +- **Keep keys stable within each group.** Reuse the same key for a customer's related requests. Generate a separate key for a session or thread only when it needs its own cache accounting. +- **Apply keys consistently.** Use the customer's key across their requests to maintain separate cache accounting. This also helps prevent cache-hit probing across customers. + +On models before GPT-5.6, `prompt_cache_key` is important for optimizing cache hit rates. Use a stable key for requests that share a reusable prefix to help route them to the same cache. For busy groups, follow the [guidance for distributing traffic across more keys](#prompt-cache-keys). + + + + + + + + + + + + + +### Configure cache retention + + + +For Responses API models that support `prompt_cache_options`, use `"30m"` for frequently reused context and consider `"12h"` for stable prefixes reused across longer gaps. Compare the higher 12-hour write rate with the cost of rewriting the prefix after 30-minute coverage expires. Use breakpoint overrides to give shared material a longer lifetime than session-specific context. See [Cache lifetime](#cache-lifetime) for examples and billing rules. + +For earlier models, prefer setting `prompt_cache_retention` to `"24h"` for extended retention when the model and your data-retention requirements allow it. See [Cache lifetime](#cache-lifetime) for supported settings and defaults. + + + + + + + + + + + + + +### Escape the minimum cacheable length cost trap + + + +If many requests reuse the same developer instructions and tool definitions, but that shared prefix falls below the model's [minimum cacheable length](#summary-of-model-differences), consider shortening it or expanding it with useful, stable instructions, examples, or reference material. Measure whether cache reuse offsets the additional input tokens and any cache-write charges, and ensure evaluations and behaviour remain stable. + +The chart highlights the minimum cacheable length cost trap where short prefix lengths can cost more uncached than expanding to the minimum cacheable token length. It uses the 30-minute cache-write rate for GPT-5.6 and later. + + + + + +#### Mathematical details + + + +For a cost-only comparison, let $$M$$ be the minimum cacheable length, $$L < M$$ the original prefix length, $$r$$ the cache-read multiplier, $$w$$ the cache-write multiplier, and $$N$$ the total number of requests. Assume the expanded prefix is exactly $$M$$ tokens, is written once, and is fully reused on every later request. In uncached-input-token equivalents, keeping the original prefix costs $$N \times L$$, while expanding it costs $$M \left[w + (N - 1)r\right]$$. The break-even original length is: + +$$ +L_{\mathrm{break\text{-}even}} = M\left(r + \frac{w-r}{N}\right) +$$ + +Expand when $$L > L_{\mathrm{break\text{-}even}}$$; keeping the shorter prefix costs less when $$L < L_{\mathrm{break\text{-}even}}$$. At equality, the costs are the same. The smallest whole-token length for which expansion is cheaper is $$\left\lfloor L_{\mathrm{break\text{-}even}} \right\rfloor + 1$$. Conversely, shrinking a cacheable prefix below $$M$$ loses caching: under the same assumptions, the shorter uncached prefix must fall below $$L_{\mathrm{break\text{-}even}}$$ to cost less than caching $$M$$ tokens. There is no universal maximum-cost prompt length; the crossover depends on reuse and pricing. + +For example, using the usual cache-read rate and 30-minute writes with $$M = 1{,}024$$, $$r = 0.1$$, and $$w = 1.25$$, the crossover is $$102.4 + \frac{1{,}177.6}{N}$$ tokens. Across 10 requests, expanding an original prefix of at least 221 tokens to 1,024 tokens is cheaper. As reuse grows, the crossover approaches 102.4 tokens. A 103-token prefix needs at least 1,963 total requests to benefit; a prefix of 102 tokens or fewer never does under these assumptions. This comparison excludes performance, output tokens, and unchanged request costs. Additional misses, writes, or different model rates change the result. + + + + + + + + + + + + + + + +### Monitor cache performance + + + +- **Measure actual cache performance.** Track `usage.input_tokens_details.cached_tokens`, `usage.input_tokens_details.cache_write_tokens`, the write counts by lifetime, input-token counts, latency, and realized cost. Track the token cache-hit rate by dividing total cached tokens by total input tokens, aggregating both counts by user, workspace, day, or another useful grouping. +- **Calculate input cost.** Use the token counts in `response.usage` and the model's [prices per million tokens](https://developers.openai.com/api/docs/pricing). +- **Use the prompt caching dashboard.** Monitor cache hit rates in the [Prompt Caching Dashboard](https://platform.openai.com/usage?usage_section=prompt-caching). + +For Responses API models that support `prompt_cache_options`, `input_tokens_details.cache_write_tokens_details` breaks writes into `ttl_30m_tokens` and `ttl_12h_tokens`. These counts do not overlap and sum to `cache_write_tokens`. Both fields are returned, including when one is zero. `cached_tokens` counts billed reads and excludes tokens charged as writes, including upgrades from 30-minute to 12-hour coverage. + +Input usage fields with both write lifetimes + +```json +{ + "usage": { + "input_tokens": 12000, + "input_tokens_details": { + "cached_tokens": 4000, + "cache_write_tokens": 7000, + "cache_write_tokens_details": { + "ttl_30m_tokens": 3000, + "ttl_12h_tokens": 4000 + } + } + } +} +``` + + +In this usage example, 1,000 input tokens are ordinary input: `12000 - 4000 - 7000`. To calculate input cost, multiply ordinary input tokens by the model's input price, `cached_tokens` by its cached-input price, `ttl_30m_tokens` by 1.25× its input price, and `ttl_12h_tokens` by 2.5× its input price. Add these amounts and divide by 1,000,000 when using prices per million tokens. Do not add `cache_write_tokens` again after counting its two components. + +For example, at an input price of $1 per million tokens and a cached-input price of $0.10 per million, this usage costs `(1000 + 4000 × 0.1 + 3000 × 1.25 + 4000 × 2.5) / 1000000 = $0.01515` for input. Use your model's actual rates, including its cached-input rate. + +The following helper uses one cache-write multiplier for all writes. Use `1.25` when all writes have a 30-minute lifetime, or `2.5` when all have a 12-hour lifetime. For requests with both lifetimes, calculate each write component separately as described here. + +Calculate input cost with one cache-write rate + +```javascript +function calculateInputCost( + usage, + inputPricePerMillion, + cacheInputMultiplier = 0.1, + cacheWriteMultiplier = 1.25 +) { + const inputTokens = usage.input_tokens; + const cachedTokens = usage.input_tokens_details.cached_tokens; + const cacheWriteTokens = usage.input_tokens_details.cache_write_tokens; + const ordinaryInputTokens = inputTokens - cachedTokens - cacheWriteTokens; + + const weightedInputTokens = + ordinaryInputTokens + + cachedTokens * cacheInputMultiplier + + cacheWriteTokens * cacheWriteMultiplier; + const inputCost = (weightedInputTokens * inputPricePerMillion) / 1_000_000; + return inputCost; +} +``` + +```python +from openai.types.responses import ResponseUsage + + +def calculate_input_cost( + usage: ResponseUsage, + input_price_per_million: float, + cache_input_multiplier: float = 0.1, + cache_write_multiplier: float = 1.25, +) -> float: + input_tokens = usage.input_tokens + cached_tokens = usage.input_tokens_details.cached_tokens + cache_write_tokens = usage.input_tokens_details.cache_write_tokens + ordinary_input_tokens = input_tokens - cached_tokens - cache_write_tokens + + weighted_input_tokens = ( + ordinary_input_tokens + + cached_tokens * cache_input_multiplier + + cache_write_tokens * cache_write_multiplier + ) + input_cost = weighted_input_tokens * input_price_per_million / 1_000_000 + return input_cost +``` + +```go +func calculateInputCost(usage responses.ResponseUsage, pricePerMillion, cacheInputMultiplier, cacheWriteMultiplier float64) float64 { + ordinary := usage.InputTokens - usage.InputTokensDetails.CachedTokens - usage.InputTokensDetails.CacheWriteTokens + weighted := float64(ordinary) + float64(usage.InputTokensDetails.CachedTokens)*cacheInputMultiplier + float64(usage.InputTokensDetails.CacheWriteTokens)*cacheWriteMultiplier + return weighted * pricePerMillion / 1_000_000 +} +``` + +```java +import com.openai.models.responses.ResponseUsage; + +static double calculateInputCost( + ResponseUsage usage, + double pricePerMillion, + double cacheInputMultiplier, + double cacheWriteMultiplier) { + long cached = usage.inputTokensDetails().cachedTokens(); + long written = usage.inputTokensDetails().cacheWriteTokens(); + long ordinary = usage.inputTokens() - cached - written; + return (ordinary + cached * cacheInputMultiplier + written * cacheWriteMultiplier) + * pricePerMillion + / 1_000_000; +} +``` + +```ruby +def calculate_input_cost( + usage, + input_price_per_million, + cache_input_multiplier = 0.1, + cache_write_multiplier = 1.25 +) + input_tokens = usage.input_tokens + details = usage.input_tokens_details + cached_tokens = details.cached_tokens + cache_write_tokens = details.cache_write_tokens + ordinary_input_tokens = input_tokens - cached_tokens - cache_write_tokens + + weighted_input_tokens = ordinary_input_tokens + + (cached_tokens * cache_input_multiplier) + + (cache_write_tokens * cache_write_multiplier) + (weighted_input_tokens * input_price_per_million) / 1_000_000 +end +``` + + + + + + + + + + + + +### Migrate prompt caching from an earlier model to GPT-5.6 and later + + + +- Keep existing stable prefixes. +- If you use `prompt_cache_key`, keep existing values to preserve separate cache accounting for customers or users. +- Replace `prompt_cache_retention` with `prompt_cache_options.ttl`. +- Confirm that reusable prefixes meet the model's [minimum cacheable length](#summary-of-model-differences). +- If the default breakpoint includes content that changes between requests, add an explicit breakpoint after the stable prefix. +- Use `prompt_cache_options.mode: "explicit"` when later content is not worth writing. +- [Compare `cached_tokens`, `cache_write_tokens`, latency, and total cost](#monitor-cache-performance) before and after migration. + + + + + +## Examples + +The following examples apply to GPT-5.6 and later models. + + + + + + + +### Single-turn LLM-as-a-Judge + + + +Consider a single-turn LLM judge that determines whether a completed interaction shows evidence that the user is satisfied after an interaction with a chatbot. Each request uses the same grading rubric and labeled few-shot examples to evaluate a different interaction. + +- **Preserving the prefix:** The fixed rubric and examples come first. Their combined length is deliberately kept just above the model's [minimum cacheable length](#summary-of-model-differences), using material that helps calibrate the judge. The interaction being evaluated comes last. +- **Caching mode and breakpoint:** Explicit-only caching is enabled, with a breakpoint after the fixed rubric and examples. The user–chatbot conversation being evaluated comes after that breakpoint and is not written to the cache, avoiding a cache-write charge for content that is unlikely to be reused. + +An example deployment using these principles reported a **token cache-hit rate of ~70%**. This figure illustrates a possible outcome. Actual cache-hit rate ceilings will depend upon your context and application usage. + +Responses API request for a single-turn judge + +```json +{ + "model": "gpt-6.1-sol", + "reasoning": { "effort": "medium", "context": "all_turns" }, + "text": { "verbosity": "low" }, + "prompt_cache_options": { "mode": "explicit" }, + "input": [ + { + "role": "developer", + "content": [ + { + "type": "input_text", + "text": "Judge whether the completed interaction provides evidence that the user is satisfied. Return true or false. Full grading rubric and labeled few-shot examples...", + "prompt_cache_breakpoint": { "mode": "explicit" } + } + ] + }, + { + "role": "user", + "content": "Completed interaction to evaluate..." + } + ] +} +``` + + + + + + + + + + + + +### Multi-turn agent + + + +Consider a multi-turn agent with long, shared developer instructions and frequent tool calls. Typical usage sees users running multiple sessions with the agent at once, and often forking the threads. + +- **Preserving the prefix**: Each turn appends new messages, tool calls, and results without rewriting earlier context, so the reusable prefix grows over time. +- **Optional prompt cache key:** This example uses `agent_123_v1:user_456` to maintain separate cache accounting for user 456, making their cached token usage and billing easier to explain. This also helps prevent cache-hit probing across users. The key stays the same across that user's sessions and forks with the agent. Omit it if your application does not need this separation. +- **Implicit caching mode:** Implicit caching is enabled so the latest eligible user or tool message provides a breakpoint. +- **Explicit breakpoints:** A breakpoint is added after each tool result to preserve earlier reusable prefixes and improve cache efficiency of forking. + +An example deployment using these principles reported a **token cache-hit rate >90%**. This figure illustrates a possible outcome. Actual cache-hit rate ceilings will depend upon your context and application usage. + +Responses API request for a multi-turn agent + +```json +{ + "model": "gpt-6.1-sol", + "reasoning": { "effort": "medium", "context": "all_turns" }, + "text": { "verbosity": "medium" }, + "prompt_cache_key": "agent_123_v1:user_456", + "prompt_cache_options": { "mode": "implicit" }, + "tools": [ + { + "type": "function", + "name": "function_name", + "description": "Function description", + "parameters": { "...": "..." } + } + ], + "input": [ + { + "role": "developer", + "content": "Stable developer instructions and reference material..." + }, + { "role": "user", "content": "Can you do...?" }, + { + "type": "function_call", + "call_id": "call_123", + "name": "function_name", + "arguments": "..." + }, + { + "type": "function_call_output", + "call_id": "call_123", + "output": [ + { + "type": "input_text", + "text": "Tool result...", + "prompt_cache_breakpoint": { "mode": "explicit" } + } + ] + }, + { "role": "assistant", "content": "Assistant response..." }, + { "role": "user", "content": "Can you also do...?" } + ] +} +``` + + + + + + + + +## Gotchas + + + + + + + +### A shared prefix is not always a cached prefix + + + +This is particularly prevalent when [migrating from earlier models to GPT-5.6 or later](#migrate-prompt-caching-from-an-earlier-model-to-gpt-5-6-and-later) due to the change in implicit caching behaviour. If requests share a long prefix but have different suffixes, caching the first complete request implicitly-only does not make the shorter shared prefix reusable. + +Consider a static developer message followed by a dynamic user message in each request. This request writes through the dynamic content. Changing that content in the next request does not match the longer cached prefix, and there is no separate breakpoint after the static content. + +Without a breakpoint after the static content + +```json +{ + "model": "gpt-6.1-sol", + "reasoning": { "effort": "medium", "context": "all_turns" }, + "text": { "verbosity": "low" }, + "prompt_cache_options": { "mode": "implicit" }, + "input": [ + { "role": "developer", "content": "Static content..." }, + { "role": "user", "content": "Dynamic content..." } + ] +} +``` + + +To remediate, place an explicit breakpoint after the static content in both requests. The first request writes the reusable prefix; the next can reuse it even when the dynamic content changes. This example uses explicit-only mode to avoid writing the dynamic content to cache. + +With a breakpoint after the static content + +```json +{ + "model": "gpt-6.1-sol", + "reasoning": { "effort": "medium", "context": "all_turns" }, + "text": { "verbosity": "low" }, + "prompt_cache_options": { "mode": "explicit" }, + "input": [ + { + "role": "developer", + "content": [{ + "type": "input_text", + "text": "Static content...", + "prompt_cache_breakpoint": { "mode": "explicit" } + }] + }, + { "role": "user", "content": "Dynamic content..." } + ] +} +``` + + + + + + + + + + + + +### Switching to explicit-only mode can miss an implicit cache write + + + +Suppose request 1 uses implicit mode and caches a prefix through the end of a user message, then follow-up request 2 preserves that prefix but switches to `prompt_cache_options.mode: "explicit"`. As explained in [How prefix matching works](#how-prefix-matching-works), request 2 checks only the explicit breakpoints in its own input, so it will not reuse that saved implicit prefix from request 1 (unless one of the explicit breakpoints in request 2 matches the cached endpoint from request 1). + +```text +▼ = breakpoint + +- Request 1: implicit mode + [Developer message][User message] ▼ + +- Request 2: explicit-only mode. Does not hit cache. + [Developer message][User message][Follow-up] ▼ +``` + +To reuse the implicit prefix from request 1, place an explicit breakpoint at the matching content-block boundary in request 2, or keep implicit mode enabled so the earlier eligible message ending remains a lookup candidate. + + + + + + + + + + + +### Extending a message can prevent reuse of its cached prefix + + + +Even when both requests use implicit mode, preserving the same initial tokens is not always enough. Suppose request 1 ends with a user message containing `Content A`, then follow-up request 2 extends that same message to `Content A + Content B`. The old endpoint after `Content A` is now inside a message, rather than at its end. As explained in [How prefix matching works](#how-prefix-matching-works), without an explicit breakpoint at that boundary, request 2 does not reuse the prefix saved there. + +```text +▼ = breakpoint + +- Request 1: implicit mode + [Developer message][User message: Content A] ▼ + +- Request 2: implicit mode. Cannot reuse the prefix through Content A. + [Developer message][User message: Content A + Content B] ▼ +``` + +When the conversation structure permits, preserve the original message and append a new message instead. Otherwise, keep the reusable text in a separate content block and place an explicit breakpoint after it in both requests. + + + + + + + + + + + +### Not all developer messages are automatic implicit mode cache lookup boundaries + + + +In implicit mode, developer messages after the initial consecutive block of developer messages are not automatic cache lookup boundaries. Add an explicit breakpoint at the end of the reusable developer message to preserve that breakpoint in subsequent requests so OpenAI can check for a matching cached prefix. + + + + + + + + + + + +### Minimum cacheable length varies by model + + + +A prefix that qualifies for caching on one model may be too short on another. Check the [model comparison](#summary-of-model-differences) and measure the reusable prefix with the model and settings you actually use. When changing models, repeat that check rather than assuming the previous model's threshold still applies. + + + + + + + + + + + +### Compaction can reduce cache reuse + + + +[Compaction](https://developers.openai.com/api/docs/guides/compaction) replaces earlier conversation context with a shorter representation. That can change the prefix, so the first request after compaction may reuse less of the previous cache even when the conversation is logically the same. + +Keep reusable instructions and reference material stable where possible, then let subsequent turns build on the compacted context. Compare total input cost before and after compaction: fewer input tokens can still save money even when the cache-hit rate falls. + + + + + +## Frequently asked questions + + + + + + + +### Does prompt caching affect output generation? + + + +No. Prompt caching does not change how the model generates output tokens. The model generates a new response using the cached prefix, so identical requests are not guaranteed to produce identical outputs. + + + + + + + + + + + +### Can I manually clear the cache? + + + +No. Manual cache clearing is not currently available. Cache entries expire according to the model's [cache lifetime](#cache-lifetime) and retention settings. + + + + + + + + + + + +### Do cached prompts count toward rate limits? + + + +Yes. Cached input tokens still count toward tokens-per-minute limits. Prompt caching does not change how [rate limits](https://developers.openai.com/api/docs/guides/rate-limits) are calculated. + + + + + +## Data retention + +All prompt caching features are compatible with [Zero Data Retention (ZDR)](https://developers.openai.com/api/docs/guides/your-data#zero-data-retention). + +The prompt cache stores key-value (KV) tensors rather than the original prompt text. These tensors may be stored as encrypted application state in GPU-local storage, with a maximum retention period of 24 hours. See [Cache lifetime](#cache-lifetime) for model-specific cache lifetimes and refresh behavior. + +Caches are not shared across organizations and cannot be reused across [regional processing boundaries](https://developers.openai.com/api/docs/guides/your-data#data-residency-controls). For details about data handling and ZDR eligibility across API features, see [Your data](https://developers.openai.com/api/docs/guides/your-data). \ No newline at end of file