发布日期: 2026-06-01
版本号: v1.17.9

Dapr 1.17.9 修复了一个在使用 Azure Cosmos DB 作为工作流 Actor 状态存储时，因 customStatus 未持久化而导致已完成的工作流无法被保留清除策略正常清除的 bug。当工作流完成但没有对应的 customStatus 数据时，清除任务会因删除不存在的数据而失败，并陷入每秒重试的死循环。根本原因是清除逻辑无条件地尝试删除 customStatus 键，而 Azure Cosmos DB 的批处理操作要求所有操作成功才能提交。修复方案是仅在确认 customStatus 存在时才执行删除。升级到 1.17.9 后，之前卡在此状态的工作流会自动恢复正常。

更新内容 （中文）

Dapr 1.17.9

此更新包含以下错误修复：

当 customStatus 未持久化时，在 Azure Cosmos DB 上工作流保留清除失败

问题

当状态存储为 Azure Cosmos DB 时，一个已完成且其 customStatus 行在 Actor 状态存储中不存在的工作流，无法被保留提醒器清除。 保留提醒器每秒无限期触发，且该工作流在超过其配置的保留 TTL 后仍保持在 Completed 状态。

影响

每当工作流到达终态且存储中未持久化 customStatus 行时，所有使用 Azure Cosmos DB 状态存储 (state.azure.cosmosdb) 作为工作流 Actor 状态存储的部署都会受到影响。

这包括：

  • 首次由 customStatus 之前的 daprd 版本保存，且后来已升级的工作流。
  • customStatus 行被带外移除（手动清理、从备份部分还原等）的工作流。
  • 已调度但从未超过初始状态推进的工作流，因此从未有历史记录增量触发 customStatus 的更新或插入。

可见症状包括：

  • 一个已完成的工作流在超过配置的 stateRetentionPolicy.anyTerminal TTL 后仍保持在 Completed 状态。
  • 调度器保留保留提醒器并无限期地每秒重新触发一次。
  • dapr_runtime_workflow_operation_count{operation=purge_workflow,status=failed} 指标每个受影响的工作流每秒递增一次。
  • daprd 以每个受影响的工作流每秒一次的频率记录日志 failed to invoke scheduled actor reminder named: retention due to: transaction failed。
  • 仅在操作员手动从调度器中删除保留器作业后，工作流才会恢复。

根本原因

pkg/runtime/wfengine/state/state.go 中的 GetPurgeRequest 无条件地为 customStatus 键发出删除操作，与元数据、收件箱和历史记录的删除一起，无论该行是否实际持久化。

Azure Cosmos DB 状态存储 (components-contrib/state/azure/cosmosdb) 将 state.TransactionalStore.Multi 转换为单个 Cosmos 事务批处理。 Cosmos 批处理是原子的：如果批处理中的任何操作失败，整个批处理将回滚，每个操作都返回 FailedDependency。 对不存在的行执行删除会返回 NotFound 并中止批处理。 状态组件的“容忍无 etag 删除时的 NotFound”路径仅适用于单操作调用，不适用于批处理调用，因此清除事务被回滚，工作流状态得以保留。

保留器提醒器的创建策略为 Constant{Interval: 1s, MaxRetries: nil}（每秒重试，无限次），因此调度器无限期地每秒重试相同的注定失败的批处理。

解决方案

State 现在将 customStatusPersisted 作为显式观察结果进行跟踪，在加载时从批量获取的 ETag 设置，并在 ResetChangeTracking 中维护，以反映最近一次保存提交的任何更新或插入。 GetPurgeRequest 查询此标志，仅在知道该行存在于存储中时才发出 customStatus 删除。

已经在此状态下卡住的工作流，在现有 1.17 部署上，一旦 Sidecar 升级到 1.17.9 将自动恢复。 重启后，在下一次保留提醒器触发时，daprd 将从 Cosmos 重新加载工作流状态，通过其缺失的 ETag 观察到缺失的 customStatus 行，省略删除操作，Cosmos 接受该批处理。 提醒器被耗尽，工作流以正常方式被清除。 升级后无需操作员干预或手动调度器删除。

更新内容 （原始）

Dapr 1.17.9

This update contains the following bug fix:

Workflow retention purge fails on Azure Cosmos DB when customStatus is not persisted

Problem

A completed workflow whose customStatus row does not exist in the actor state store cannot be purged by the retention reminder when the state store is Azure Cosmos DB. The retention reminder fires every second indefinitely and the workflow stays in the Completed state past its configured retention TTL.

Impact

Any deployment using the Azure Cosmos DB state store (state.azure.cosmosdb) for the workflow actor state store is affected whenever a workflow reaches a terminal state without a customStatus row persisted in the store.

This includes:

  • Workflows that were first saved by a pre-customStatus daprd version and have since been upgraded.
  • Workflows whose customStatus row was removed out of band (manual cleanup, partial restore from backup, etc.).
  • Workflows scheduled but never advanced past the initial state, so no history delta ever triggered the customStatus upsert.

Visible symptoms include:

  • A completed workflow stays in the Completed state past the configured stateRetentionPolicy.anyTerminal TTL.
  • The scheduler retains the retentioner reminder and re-fires it once per second indefinitely.
  • The dapr_runtime_workflow_operation_count{operation=purge_workflow,status=failed} metric increments once per second per affected workflow.
  • daprd logs failed to invoke scheduled actor reminder named: retention due to: transaction failed at one tick per second per affected workflow.
  • The workflow recovers only after an operator manually deletes the retentioner job out of the scheduler.

Root Cause

GetPurgeRequest in pkg/runtime/wfengine/state/state.go unconditionally emitted a delete for the customStatus key alongside the metadata, inbox, and history deletes, regardless of whether the row was actually persisted.

The Azure Cosmos DB state store (components-contrib/state/azure/cosmosdb) translates state.TransactionalStore.Multi into a single Cosmos transactional batch. Cosmos batches are atomic: if any operation in the batch fails, the whole batch is rolled back and every operation returns FailedDependency. A delete for a row that does not exist returns NotFound and aborts the batch. The state component’s “tolerate NotFound on etag-less delete” path only applies to single-operation calls, not to batched ones, so the purge transaction was rolled back and the workflow state stayed in place.

The retentioner reminder is created with a failure policy of Constant{Interval: 1s, MaxRetries: nil} (retry every second, forever), so the scheduler retried the same doomed batch once per second indefinitely.

Solution

State now tracks customStatusPersisted as an explicit observation, set from the bulk-get ETag at load time and maintained in ResetChangeTracking to reflect whichever upserts the most recent save committed. GetPurgeRequest consults this flag and only emits the customStatus delete when the row is known to exist in the store.

Workflows already stuck in this state on existing 1.17 deployments recover automatically once the sidecar is upgraded to 1.17.9. On the next retention reminder fire after restart, daprd reloads the workflow state from Cosmos, observes the missing customStatus row from its absent ETag, omits the delete, and Cosmos accepts the batch. The reminder drains and the workflow is purged in the normal way. No operator intervention or manual scheduler delete is required after the upgrade.

下载链接