dapr v1.18.4 版本更新介绍
发布日期: 2026-09-09
版本号: v1.18.4
Dapr 1.18.4 是一个以工作流(Workflow)缺陷修复为主的补丁版本,主要修复内容如下:一是工作流启动提醒(start reminder)相关的多个问题,包括调度器重启、放置重平衡或调度器不可用期间创建的工作流会永久停留在 PENDING 状态,通过改为先提交状态再创建提醒、使用确定性提醒名称、将失败注册交由运行时生命周期的分离重试以及状态轮询时自动重新注册提醒来解决。二是 ContinueAsNew 相关问题:父工作流创建与上一代子工作流实例 ID 冲突的子工作流时会永久死锁,现在会返回 already exists 错误;SDK 派生的确定性值(如 NewGuid)在 ContinueAsNew 后重复,原因是工作流工作项未携带执行 ID,现已补上;新代首次执行丢失或上一轮结果被错误复用会导致工作流卡在 RUNNING,已通过正确持久化新代、丢弃旧事件以及修复会合 actor 的响应匹配来解决。三是历史签名(WorkflowHistorySigning)相关问题:强制清除在启用签名时总是失败、过期的完成事件被误判为篡改而永久终止工作流、子工作流创建被拒或子工作流 ContinueAsNew 后无法向父工作流报告完成,均已修复。四是终止与状态查询问题:终止事件与其他事件同批投递时被静默丢弃,现改为直接将实例置为 TERMINATED;工作流处于停滞状态时调度和状态等待会报错,现改为直接读取状态并退避重试。五是其他修复:子工作流完成先于自身状态提交导致状态不一致,现改为先提交再通知;强制清除未驱逐内存中的工作流 actor,现会先驱逐;热重载的访问策略在元数据中列出但尚未生效,现改为先编译生效再更新存储;通过 gRPC 调用输出绑定时会把二进制追踪元数据(grpc-trace-bin)转发给组件,导致 Azure Service Bus 和 Blob 存储绑定失败,现已过滤以 -bin 结尾的元数据;可插拔 pub/sub 组件消息逐条串行投递导致吞吐骤降、批量订阅失效,现恢复为并发分发并保持优雅停机语义。
更新内容 (中文)
Dapr 1.18.4
本次更新包含以下问题修复:
- 工作流的启动提醒在调度器重启期间失败,导致工作流永久停留在 PENDING
- 工作流的启动提醒在创建提交之前触发,导致工作流永久滞留在 PENDING
- 调度器故障期间,放置再平衡中断了工作流的启动提醒注册,导致工作流永久停留在 PENDING
- ContinueAsNew 之后创建的子工作流与仍在运行的子实例 ID 冲突,导致父工作流永久死锁
- 通过 gRPC 的输出绑定调用将二进制链路追踪元数据转发给了组件
- 启用历史签名时,所有工作流的强制清除均失败
- 在 ContinueAsNew 之后或状态存储瞬时故障后收到过期的完成事件时,工作流被错误地判定为被篡改
- 工作流终止指令与其他事件在同一批次中投递时被静默丢弃
- 工作流处于停滞状态时,工作流调度和状态等待以 actor 停滞错误失败
- ContinueAsNew 之后,工作流 NewGuid 及其他 SDK 派生的确定性值出现重复
- 热重载的工作流访问策略在生效之前就已出现在 metadata 中
- 启用历史签名时,子工作流创建被拒绝导致工作流被错误地判定为被篡改
- 子工作流的完成事件在子工作流提交自身状态之前就已投递给父工作流
- 强制清除后,工作流 actor 仍在用其过期的内存状态对外服务并写入
- 启用历史签名时,执行了 ContinueAsNew 的子工作流无法上报其完成状态
- 新一代首次执行丢失后,工作流在 ContinueAsNew 之后卡在 RUNNING
- 启用 WorkflowsClusteredDeployment 时,某一轮次以上一轮的结果作答,导致工作流卡在 RUNNING
- 可插拔 pub/sub 组件一次只能向应用投递一条消息
工作流的启动提醒在调度器重启期间失败,导致工作流永久停留在 PENDING
问题
在调度器副本正在关闭时创建工作流,可能使新实例永久卡在 PENDING 状态。
工作流 actor 先提交其状态(状态为 PENDING),随后作为一次独立的调度器调用注册启动执行的 start- 提醒。
当该调用以 rpc error: code = Unknown desc = cron is closed 失败时,该错误被当作永久性错误:提醒的创建只尝试一次便被丢弃,而已提交的状态则被保留。
于是该实例无限期地报告 PENDING,没有注册任何提醒,从未执行,也永远不会转入 FAILED,因此监控不会收到任何错误。
在调度器重启期间,同样的窗口通常会产生 Unavailable 错误,这些错误会被正确重试;只有更狭窄的关闭中窗口会产生不被重试的 Unknown 错误。
影响
如果在 dapr-scheduler-server 副本重启期间创建工作流(包括例行的滚动升级),你就会受到影响。
单个副本重启就足够了。
受影响的实例被静默丢失:轮询它们的调用方永远看到一个看似健康的 PENDING 状态,这些实例只能通过清除并重建来恢复。
根本原因
调度器将内部的 cron 关闭错误作为普通错误返回,它以 gRPC 码 Unknown 跨越传输。
actor 提醒创建周围的有界重试只将 Unavailable 和 DeadlineExceeded 分类为瞬时错误,因此 Unknown 立即被抛出而没有任何重试,丢弃了唯一能够启动工作流的信号。
解决方案
工作流提醒的创建现在对所有错误都采用有界指数退避进行重试,明确的永久性请求错误(InvalidArgument、PermissionDenied、Unauthenticated、FailedPrecondition、Unimplemented、NotFound)以及需要运维人员介入并会立即反馈给调用方的调度器存储耗尽(ResourceExhausted)除外。
调度器现在以 gRPC 码 ResourceExhausted(而非 Unknown)返回 etcd 空间配额耗尽,这样客户端就能将其与瞬时故障区分开。
此前只有 Unavailable 和 DeadlineExceeded 会被重试,因此任何其他瞬时错误(包括以 Unknown 形式跨网络传输的调度器关闭错误和 etcd 错误)都会在第一次尝试时被丢弃。
这适用于所有与工作流相关的提醒创建(start、timer、activity 和 termination 提醒),而真正的永久性错误仍然会在有界重试预算内浮出。
提醒调度错误日志现在还包含 actor 类型和 ID,因此任何失败都可以归因到受影响的工作流实例。
工作流的启动提醒在创建提交之前触发,导致工作流永久滞留在 PENDING
问题
在放置再平衡或调度器领导权更替期间创建工作流,可能使新实例永久卡在 PENDING,历史为空且没有为它注册任何提醒。
工作流 actor 在提交实例状态之前创建了一次性 start 提醒。
在再平衡期间,该提醒可能在提交落地之前在另一台主机上触发:触发的主机找不到可处理的收件箱,于是确认成功,调度器删除了该一次性提醒。
随后提交落地到一个唯一执行触发器已不复存在的实例上。
影响
如果工作流的创建与放置传播(主机加入或离开)或调度器 quorum 丢失并发发生,你就会受到影响,现场在 1.18.4-rc.2 上曾有此类报告。
受影响的实例永远报告 PENDING,从未执行,也永远不会转入 FAILED。
由于 start 提醒的名称包含随机后缀,重试创建无法重新注册它:重试只会返回 already exists 错误,因此实例只能通过清除并重建来恢复。
根本原因
start 提醒在 ExecutionStarted 收件箱事件被持久化之前创建,因此在该窗口内触发的提醒会发现空收件箱并被确认为完成,从而永久消耗掉这个一次性提醒。
随机的提醒名称使该丢失不可恢复:之后没有任何手段能确定性地重新声明同一个提醒。
解决方案
工作流 actor 现在先持久化 ExecutionStarted 收件箱事件,并在提交之后以派生自已保存事件的确定性名称创建 start 提醒。
提醒不再可能针对未提交的状态触发,提醒创建失败现在会作为可重试错误反馈给调用方,而不是让实例陷入滞留。
对已保存但缺少 start 提醒的实例重试创建,会以确定性名称重新声明该提醒并报告成功,从而恢复该实例,而不是以 already exists 失败。
调度器故障期间,放置再平衡中断了工作流的启动提醒注册,导致工作流永久停留在 PENDING
问题
在调度器不可达时创建工作流,如果恰好同时发生针对该工作流 actor 类型的放置传播(例如应用在调度器滚动更新期间扩缩容),新实例可能永久卡在 PENDING。
工作流 actor 先提交状态,然后注册启动执行的 start- 提醒,并在调度器不可用期间重试该注册。
该重试运行在调用的放置 claim 上下文中,而传播轮次会在排空超时后取消该上下文,无论调度器是否健康,于是注册被放弃,而状态已经提交。
创建返回了错误,但实例确实存在:每次状态读取都报告 PENDING,没有注册提醒,从未执行,也永远不会转入 FAILED。
重试创建同样无法恢复它。 重试在重新声明之前会检查 start 提醒是否缺失,而该检查是一次调度器往返,恰好在丢失提醒的这场故障中失败,因此重试中止而没有重新声明。 如果客户端读取工作流状态后,只要实例非终态就跳过创建(文档示例中使用的模式),就根本不会再发起任何创建,也没有其他任何代码触及该实例。
影响
如果在调度器中断期间创建工作流(包括在复用的实例 ID 下重建),且应用的 pod 数量在创建前后几秒内发生变化,你就会受到影响。
单纯的调度器故障不会导致此问题;必须与放置轮次巧合重叠才行。
受影响的实例被静默丢失:轮询它们的调用方永远看到一个看似健康的 PENDING 状态,这些实例只能通过清除并重建来恢复。
根本原因
状态提交之后执行的 start 提醒注册受调用的放置 claim 上下文限制,而不是受运行时生命周期限制,因此针对该工作流 actor 类型的一次传播轮次可以放弃一个已提交实例的唯一驱动。
创建重试中的恢复路径依赖调度器 Get 成功,而客户端使用状态读取完全绕过了工作流 actor,因此没有任何代码路径会重新声明缺失的提醒。
解决方案
一旦状态提交,任何因永久性请求错误之外原因失败的 start 或唤醒提醒注册,都会交给一个基于运行时生命周期的脱离式重试,这样被取消的 claim 上下文或耗尽的重试预算都不可能在 sidecar 运行期间让一个已提交的实例失去驱动。
创建仍然会把错误返回给调用方,而注册会在调度器重新可达时立即完成。
创建重试现在只要无法连上调度器进行提醒检查,就会重新声明 start 提醒,而不是直接失败。
最后,对于 start 已逾期超过 5 秒的 PENDING 实例,状态读取(包括状态 API 和状态等待)会重新声明其 start 提醒,这样对于在脱离式重试完成之前 sidecar 就已死亡的实例,任何轮询它的客户端都能将其恢复,而无需新的创建。
ContinueAsNew 之后创建的子工作流与仍在运行的子实例 ID 冲突,导致父工作流永久死锁
问题
一个父工作流调用了 ContinueAsNew,随后创建的子工作流实例 ID 仍被同一父工作流上一次执行产生的、仍在运行的子实例占用,导致该父工作流永久挂起。
子工作流从未启动,父工作流的子工作流任务既未完成也未出错,工作流状态无限期停留在 RUNNING。
唯一的痕迹是一条 debug 级日志:ignoring duplicate child workflow creation from parent '<id>'。
自动生成的子实例 ID 派生自父实例 ID 和一个按执行计数的动作计数器,而该计数器在每次 ContinueAsNew 时都会归零,因此每一代的第一个自动生成子 ID 都是相同的。
同样,诸如 .NET SDK 的 WorkflowContext.NewGuid() 之类的确定性 SDK API 在 ContinueAsNew 之后会再次返回相同的值,因此将子实例 ID 固定为这类值的代码会重新发出一个已在使用的 ID。
影响
如果工作流使用了 ContinueAsNew,并创建的子工作流的上一代对应实例可能仍在运行(例如在 ContinueAsNew 之前启动的即发即忘子工作流),你就会受到影响。
根本原因
创建实例 ID 已存在的子工作流是有意进行去重的,其场景是:父工作流在成功创建子工作流之后、持久化自身状态之前崩溃,其重新执行会发出相同的创建请求,这必须被忽略以保证重放安全。 该去重检查只比较了父实例 ID 和创建方任务的 ID。 ContinueAsNew 保留父实例 ID 并重置其历史,因此任务 ID 从零重新开始,后续执行中真正的新子工作流创建与崩溃重放的重复无法区分。 创建被丢弃,丢弃被报告为成功,而父工作流挂起的子工作流任务永远不会完成或出错。
解决方案
每次工作流执行本就携带一个唯一的执行 ID,它在每次 ContinueAsNew 和实例重建时都会新铸,而且已经包含在子创建请求中。
去重检查现在也对它进行比较:当已存储子实例的父执行 ID 与传入创建的父执行 ID 都存在且不同时,该创建就是真正的新创建,而非崩溃重放,现在它会以 already exists 错误失败,而不是被静默丢弃。
该错误会使父工作流的子工作流任务出错,于是父工作流要么处理它,要么带着清晰的消息转入 FAILED,这与在被无关运行中工作流占用的实例 ID 下创建子工作流时的既有行为一致。
复用其先前持有者已完成的子实例 ID,其行为与之前完全一致。
通过 gRPC 的输出绑定调用将二进制链路追踪元数据转发给了组件
问题
通过 gRPC API 调用输出绑定时,会把传入 gRPC 元数据的每一个条目都复制到交给绑定组件的元数据中,包括二进制元数据(键以 -bin 结尾,最典型的是 grpc-trace-bin)。
二进制 gRPC 元数据会在运行时看到之前被 gRPC 服务器进行 base64 解码,因此原始的追踪字节被写入组件的字符串元数据,键为 dapr-grpc-trace-bin。
要求文本元数据的组件在这些字节上会失败。
Azure Service Bus 绑定直接拒绝该消息:
Error invoking output binding <binding-name>: (connlost): not a valid UTF-8 string
使用 Shared Key 认证的 Azure Blob Storage 绑定会间歇性失败,只要追踪 ID 中恰好包含一个水平制表符字节(0x09)——该字节通过了组件的头部清洗器,却被 Azure 的规范化处理移除:
403 AuthenticationFailed
The MAC signature found in the HTTP request is not the same as any computed signature
影响
如果你的应用使用发送 grpc-trace-bin 头的客户端,通过 gRPC Dapr API 调用输出绑定,且目标组件将元数据视为文本,你就会受到影响。
.NET SDK 自 1.18.5 起为每次 gRPC 调用附加 grpc-trace-bin,Java SDK 更是早已如此,因此升级 .NET SDK 会让这一故障出现在此前正常的部署中。
由于问题字节来自随机的 trace 和 span ID,故障是间歇性的,取决于具体的追踪。
对于 Blob Storage,不基于请求签名的认证方式(如托管标识)不会使请求失败;被破坏的值取而代之地被静默写入所存储 blob 的元数据。
根本原因
gRPC InvokeBinding 处理器把所有传入的 gRPC 元数据合并进绑定请求,以便文本追踪头(traceparent、tracestate、baggage)和自定义元数据传播到组件。
该合并没有区分二进制元数据:grpc-trace-bin 被重命名为 dapr-grpc-trace-bin,其原始解码字节被存入字符串元数据映射,随后原样传给组件。
HTTP 绑定 API 完全从请求体中获取元数据,因此未受影响。
解决方案
二进制 gRPC 元数据(任何以 -bin 结尾的键)不再从传入的 gRPC 调用复制到输出绑定组件元数据中。
在 InvokeBinding 请求本身上显式设置的元数据不受过滤,文本追踪元数据(traceparent、tracestate、baggage)继续原样传播到组件。
分布式追踪不受影响:运行时直接从 gRPC 调用中消费 grpc-trace-bin 用于 span 传播,而从不从绑定元数据中获取,也没有任何组件读取它。
启用历史签名时,所有工作流的强制清除均失败
问题
启用 WorkflowHistorySigning 预览特性后,强制清除一个工作流实例总会以配置错误失败,提示该工作流拥有已签名历史但未配置签名器。
这使得本意用于清除无法正常清除的实例的强制清除逃生通道,在任何启用签名的安装上都不可用。
影响
如果你启用了 WorkflowHistorySigning 并尝试强制清除(无论是否递归),你就会受到影响:两种强制模式都会走到同一条加载路径。
任何已签名实例都无法被强制清除;调用在删除任何内容之前就失败了。
根本原因
强制清除路径加载工作流状态时,没有像其他所有加载点那样把运行时的签名器和命名空间传给状态加载器。 加载器的防篡改保护在没有可用签名器进行校验时会拒绝加载已签名历史,因此加载(进而清除)总是失败。
解决方案
强制清除路径现在会把签名器和命名空间传给状态加载器,与其他调用点保持一致,这样已签名历史能正常校验,清除得以继续。
在 ContinueAsNew 之后或状态存储瞬时故障后收到过期的完成事件时,工作流被错误地判定为被篡改
问题
启用 WorkflowHistorySigning 后,工作流可能以错误类型 DAPR_WORKFLOW_HISTORY_TAMPERED 终态失败,而实际上并未发生任何篡改。
有两种良性情况会触发它:子工作流的完成事件在父工作流调用 ContinueAsNew 之后到达,以及 activity 的重试完成事件在一次瞬时状态存储故障回滚了记录该 activity 调度的保存之后到达。
一旦被标记为墓碑(tombstone),实例便不可恢复:所有提醒被删除,其后的每一个事件都被拒绝。
影响
如果你启用了 WorkflowHistorySigning 并使用 ContinueAsNew,而子工作流可能在状态转换前后完成,或者你的 actor 状态存储在高负载下出现瞬时写入失败,你就会受到影响。
受影响的实例会因完整性并未被破坏而永久以篡改错误失败,需要清除并重建。
根本原因
对入站完成事件的证明(attestation)校验,会在已签名历史中查找该完成事件的任务调度 ID。 ContinueAsNew 会合法地重置历史(因此上一次执行的调度事件已消失),被回滚的保存会合法地撤回一个已派发的调度事件。 校验把由此产生的“未知任务调度 ID”与签名不匹配同等对待,并把工作流标记为墓碑,而非签名路径则只是丢弃这类过期完成事件。
解决方案
现在,引用了已签名历史中不存在的任务调度 ID 的完成事件会被丢弃且不持久化任何内容,这与非签名行为一致;发送方会被告知该实例不再接受它并停止重投。 当该 ID 存在但记录的是一次不同的调用时同理:任务 ID 在 ContinueAsNew 之后会重新开始,因此一个迟到的完成事件可能落在属于新一代自身子工作流或 activity 的复用 ID 上;一个签名有效但其输入摘要与所记录调用不匹配的完成事件,会被当作过期事件丢弃,而不是把工作流标记为墓碑。 这是安全的,因为该事件在写入之前就被丢弃:一个使用未知 ID 的伪造完成事件对攻击者毫无价值。 此外,在因任何其他校验失败而标记墓碑之前,运行时现在会针对新加载的持久状态重新校验该事件,这样过期的内存缓存就无法再诬陷一个健康的工作流;真正的签名不匹配仍会像以前一样被标记墓碑。
工作流终止指令与其他事件在同一批次中投递时被静默丢弃
问题
终止工作流有时不起作用:工作流状态保持 RUNNING,并继续执行 activity 和 timer,仿佛从未发出过终止指令。
阻塞式的终止调用(例如 .NET SDK 中的 TerminateWorkflowAsync)永不返回,因为它们在等待实例进入一个永不到来的终态。
启用 debug 日志后,一个被丢弃的终止会显示 ExecutionTerminated 事件被投递并消费,但毫无效果:
received work item with 2 new event(s): [ExecutionTerminated, TaskCompleted#1]
workflow execution returned with status 'ORCHESTRATION_STATUS_RUNNING'
处于挂起状态时被终止的工作流也会受到同样的影响,且是确定性的:状态保持 SUSPENDED,发给它的每一次终止都丢失了。
影响
如果你终止的是正在积极推进的工作流,无论使用哪种 SDK 语言,你都会受到影响。
时间窗口的大小取决于收件箱压力:当终止与其他事件在同一个工作项批次中到达、且不是该批次的最后一个事件时,它会被丢弃,因此在短 activity 和 timer 上循环的长时工作流(无尽轮询器、监控器、continue_as_new 循环)最易受影响,而停留在单个慢 activity 或 timer 上的工作流几乎总是单独收到其终止,不受影响。
被丢弃的终止会随其批次一起被消费且永不重投,实例因此继续运行;重试的终止会再次撞上同一窗口。
被丢弃的递归终止也永远不会级联下去,留下作为孤儿继续运行的子工作流。
对于挂起的工作流则没有窗口:在工作流被恢复之前,每一次终止都丢失了。
根本原因
工作流引擎把一个实例的待处理事件作为一个批次交给工作流执行器,并完全依赖 SDK 执行器把 ExecutionTerminated 事件转化为终止结果。
SDK 执行器在处理该事件时登记了终止,但仍把该批次中剩余的事件继续喂给工作流代码,于是工作流越过自身终止而恢复运行,并产生一个竞争结果:调度更多工作、正常完成,或通过 continue_as_new 重启。
continue_as_new 结果总会丢弃终止;其他结果则以不确定的方式与之竞争。
在工作流处于挂起状态时投递的终止根本不产生任何结果,因为挂起抑制了执行器本应返回的每一个动作。
由于引擎信任执行器的结果,而事件又随批次被消费,终止便永久丢失。
解决方案
当投递的批次包含 ExecutionTerminated 事件且执行器并未完成工作流时,daprd 会丢弃注定失败的执行的待处理工作,并把该实例标记为 TERMINATED 完成。
与终止一同返回的 continue_as_new 不再启动新的迭代,终止一个挂起的工作流现在无需先恢复它即可完成终止。
工作流处于停滞状态时,工作流调度和状态等待以 actor 停滞错误失败
问题
调度一个首批轮次很快陷入停滞的工作流,或等待一个已停滞工作流的状态,可能以 actor 处于停滞状态的错误失败。
实例本身是健康的:它被创建、运行,并进入了预期的 STALLED 状态,但观察它的调用返回了错误而非状态。
影响
如果工作流发生停滞(例如因 PAYLOAD_SIZE_EXCEEDED 或版本不匹配),而客户端在停滞期间调度实例或等待其状态,你就会受到影响。
ScheduleNewWorkflow 可能对一个已成功创建的实例返回错误,状态等待也可能失败而非等待恢复。
根本原因
陷入停滞的轮次会持有工作流 actor 的轮次锁挂起,以避免热重试该停滞。 在该持有期间注册的状态观察流会以 actor 停滞错误被拒绝,该错误被传播给等待中的客户端,而不是被当作一个需要等待通过的状态。
解决方案
当状态观察发现 actor 停滞时,会直接从状态存储读取已保存的工作流状态:如果实例已达到等待条件(例如调度在等待工作流启动),该等待会立即完成,其他任何等待都会以退避方式重新注册,直到停滞解除。
ContinueAsNew 之后,工作流 NewGuid 及其他 SDK 派生的确定性值出现重复
问题
工作流 SDK 按执行派生的确定性值,在工作流调用 ContinueAsNew 之后出现重复。
最明显的是,.NET SDK 的 WorkflowContext.NewGuid() 在新一代中再次返回相同的 GUID 序列,因此将子工作流实例 ID 固定为这些 GUID 的代码会重新发出已在使用的 ID,并且从 1.18.4 起会收到 already exists 失败,而不是得到新的子工作流。
影响
如果工作流把 ContinueAsNew 与派生确定性值的 SDK API(例如 .NET SDK 中的 NewGuid() 或自动生成的子工作流实例 ID)一起使用,你就会受到影响。
每一代都会复现上一代的值。
根本原因
SDK 从每个工作流工作项上携带的执行 ID 为这些派生播种,当该字段为空时回退到实例 ID。
实例 ID 在 ContinueAsNew 之间保持不变,而 SDK 的派生计数器会随新历史一起重新开始,因此执行 ID 为空时每一代都会复现相同的值。
sidecar 从未填充该字段:工作流引擎发送的每个工作项执行 ID 都为空,尽管每一代的 ExecutionStarted 历史事件都携带一个新铸的执行 ID。
解决方案
工作项现在携带其所属世代的执行 ID,取自 ExecutionStarted 事件,并且先扫描新事件再扫描过往事件,这样执行了 continue-as-new 或重建的运行会报告其自身的执行,而不是过期的执行。
每一代的执行 ID 在 ContinueAsNew 和实例重建时新铸,但会被持久化到历史中,因此 SDK 派生的值在每一代内唯一,同时在同一代内的重放中保持稳定。
在升级期间仍在飞行中的工作流会在下一轮次用其执行 ID 重新播种;其历史中已记录的值仍像以前一样从历史中重放。
热重载的工作流访问策略在生效之前就已出现在 metadata 中
问题
当 WorkflowAccessPolicy 资源在运行时被创建、更新或删除时,sidecar 的 metadata 端点会在新的策略集合真正生效之前一小段时间列出该新集合。
等待 metadata 报告收紧后的策略再调用工作流操作的调用方,仍可能基于此前更宽松的策略集合被授权;对于刚被删除的策略,情况正好相反。
影响
将 metadata 端点作为策略变更已生效信号的运维人员或自动化,可能观察到一个短暂窗口,在此期间上报的策略与生效的策略不一致。 稳态执行不受影响:该窗口在新的集合编译完成后即关闭。
根本原因
热重载协调器先把策略加入资源存储,之后才编译完整集合并将其交换进工作流 API。 编译策略表达式需要一小段时间,而 metadata 端点列出的是资源存储而非已编译的集合。
解决方案
协调器现在会先编译候选策略集合并将其交换进执行,然后再更新资源存储,更新和删除皆是如此,因此 metadata 端点列出的策略始终是已经生效的策略。
启用历史签名时,子工作流创建被拒绝导致工作流被错误地判定为被篡改
问题
启用 WorkflowHistorySigning 后,子工作流创建被运行时拒绝的父工作流会以错误类型 DAPR_WORKFLOW_HISTORY_TAMPERED 和消息 child failure missing required attestation for task <n> 终态失败。
有两种拒绝会触发它:在仍被同一父工作流上一次执行的运行中子实例占用的实例 ID 下创建子工作流(本版本引入的 already exists 失败),以及被 WorkflowAccessPolicy 拒绝的子工作流创建。
一旦被标记为墓碑,实例便不可恢复:其提醒被删除,旧子实例之后的完成事件以 no such instance exists 被丢弃。
影响
如果你启用了 WorkflowHistorySigning,且工作流使用了 ContinueAsNew 并重建了上一代对应实例可能仍在运行的子工作流,或创建了被访问策略拒绝的子工作流,你就会受到影响。
没有签名时,同一个父工作流只会观察到一个它可以处理的普通子工作流失败;有签名时,它每次出现都会确定性地永久失败。
被访问策略拒绝的 activity 存在同样的潜在缺陷。
根本原因
当运行时拒绝子工作流创建时,它会为父工作流合成一个 ChildWorkflowInstanceFailed 事件,并通过提醒投递。
这类事件按设计不携带证明,因此当事件的 router 把父工作流自身的应用指定为来源时,证明校验会豁免它。
合成该事件的代码从未设置 router,因此豁免从未匹配,该事件被当作来自不可信发送方的无证明完成事件处理。
解决方案
合成的子工作流失败和合成的 activity 失败现在会把父工作流的应用记录为它们的来源,这样证明校验就会将它们识别为本地生成,父工作流也能如预期地观察到该失败。
子工作流的完成事件在子工作流提交自身状态之前就已投递给父工作流
问题
当子工作流完成时,其完成事件在子工作流自身的终态写入状态存储之前就投递给了父工作流。
观察到父工作流为 COMPLETED 并立即查询子工作流的客户端,可能仍看到子工作流为 RUNNING 或 PENDING,而在此窗口内 sidecar 崩溃或存储报错会重跑子工作流的最后一轮并再次投递完成事件。
父工作流仅按任务 ID 去重这类完成事件,因此在父工作流 ContinueAsNew(任务 ID 会重新开始)之后,一个迟到的重复事件可能被归到错误的子任务上。
影响
任何使用子工作流的工作流:父子状态短暂不一致、依赖父端去重的重复完成投递,以及 ContinueAsNew 之后罕见的完成事件错归。
根本原因
在这一轮次中,完成事件的派发排在子工作流状态保存之前,因此父工作流的完成事件副本先于子工作流的副本变得持久。
解决方案
子工作流现在先提交其终态,并同时记录一个持久化标记,表明其父工作流尚未确认该完成事件,随后投递完成事件,并在父工作流收到后清除该标记。
如果其间投递失败或 sidecar 重启,该通知会根据子工作流持久化的历史重建并重发,直到父工作流确认为止,因此父工作流得知的完成事件一定是已经持久化的;在标记存在期间,清除子工作流或复用其实例 ID 的操作会因未完成而被拒绝。
父工作流还会拒绝其发送方不是它当前世代为该任务创建的子工作流、或是在父工作流上一次执行下创建的完成事件,这样过期的完成事件在 ContinueAsNew 之后就无法再被归到新的子工作流上;已完成的父工作流会确认迟到的子完成事件而不运行任何轮次。
启用历史签名时,执行了 ContinueAsNew 的子工作流无法上报其完成状态
问题
启用 WorkflowHistorySigning 后,另一个应用中调用了 ContinueAsNew 的子工作流永远无法解决其父工作流的任务:父工作流把子工作流发来的每一个完成事件都当作过期事件丢弃,父工作流一直运行到被终止。
影响
任何已签名的、跨应用的、执行了 continue-as-new 的子工作流。父工作流无法完成。
根本原因
父工作流用它创建子工作流时的输入来校验子工作流的完成证明。ContinueAsNew 之后,子工作流当前的启动事件携带的是延续后的输入,而子工作流用该输入做了证明,因此摘要永不匹配。
解决方案
子工作流在创建时记录其被创建时所用的输入(仅限已签名的子工作流,因为只有证明会读取它),并在任意多次 ContinueAsNew 调用中始终用该输入证明其完成。 在本修复之前创建、且已经执行了 continue-as-new 的已签名子工作流,其任何位置都不再保留创建时的输入,因此其完成事件仍无法校验;这类实例正是跨越升级运行的那些。
强制清除后,工作流 actor 仍在用其过期的内存状态对外服务并写入
问题
强制清除删除了工作流的状态存储行和提醒,但没有把仍驻留在内存中的工作流 actor 驱逐出去。 该 actor 继续用其缓存状态响应状态流和跨应用 metadata 读取,因此一个已清除的实例仍可能报告为运行中,之后针对它的事件或轮次可能把其缓存状态写回,部分复活已清除的实例。
影响
在 actor 仍处于活跃状态时使用强制清除来移除卡住或损坏实例的运维人员,以及此后观察该实例的任何客户端。
根本原因
强制清除是绕过 actor 的存储级操作,而 actor 保存后的版本检查无法区分被清除的行和不返回版本令牌的存储,于是它落到了无条件写入。
解决方案
强制清除现在会在删除行之前,先在运行该实例的主机上驱逐其工作流 actor,这样任何后续访问都会从存储重新加载并发现什么都没有。 元数据行在其下方消失的 actor 现在会丢弃其缓存并停用,而不是把它写回;一个已知行版本丢失的保存会被拒绝,而不是无条件执行。
新一代首次执行丢失后,工作流在 ContinueAsNew 之后卡在 RUNNING
问题
启用了 WorkflowsClusteredDeployment 后,调用了 ContinueAsNew 的工作流可能永远卡在 RUNNING。
运行时为 ContinueAsNew 轮次记录了 failed before receiving workflow result 和 aborting workflow execution,随后新一代的首次执行收到了上一代的事件,例如触发 ContinueAsNew 的 timer,以及上一代启动的子工作流的完成事件。
当新一代以相同任务 ID 创建子工作流时,过期的完成事件被记录在创建之前,新子实例的真实完成事件被当作重复事件丢弃(dropping duplicate completion event already present in history/inbox),父工作流从此不再推进。
影响
如果你启用了 WorkflowsClusteredDeployment 并使用了 ContinueAsNew,你就会受到影响;那些在每一代中还创建子工作流或 timer 的工作流最容易滞留,在负载下或 pod 重启期间每次 ContinueAsNew 有百分之几的发生率。
当工作流超出引擎的紧循环 ContinueAsNew 限制时,任何部署都可能触达同一缺陷。
根本原因
当工作流引擎在工作流已经 continue-as-new 之后放弃一个工作项时,运行时会持久化新一代以免进度丢失。 它把新一代的启动事件写入历史,但只有在上一代留有需要结转的未处理外部事件时才会替换已消费的收件箱。 没有结转时,上一代已消费的事件会留在收件箱中,并作为新一代的第一个工作项被再次投递,此时任务 ID 序列已重新开始,因此上一代的完成事件可能接管新一代的某个任务 ID。
解决方案
被放弃的 continue-as-new 世代现在会与一个全新创建的工作流完全一样地持久化:空历史、其启动事件引领收件箱(后接任何结转的事件),以及在状态保存之后创建的、驱动它的启动提醒。 上一代已消费的事件会被丢弃,新一代始终独立运行。
启用 WorkflowsClusteredDeployment 时,某一轮次以上一轮的结果作答,导致工作流卡在 RUNNING
问题
启用 WorkflowsClusteredDeployment 特性后,一个使用了 ContinueAsNew 且每一代都运行 activity 的工作流可能停止推进并永远报告 RUNNING。
一个卡住实例的最后已提交历史以一个重复的 TaskScheduled 结束(该 activity 早已完成),其收件箱为空,也没有任何提醒来驱动它。
影响
如果你在负载均衡器后运行启用了 WorkflowsClusteredDeployment 的 daprd,且你的工作流执行了 continue-as-new(尤其是在每一代都运行 activity 或子工作流的循环中),你就会受到影响。
受影响的实例看起来健康(RUNNING)但永不完成,不抛出任何错误,只能通过终止或清除并重建来恢复。
根本原因
在 WorkflowsClusteredDeployment 下,应用的执行结果经由一个按实例的会合(rendezvous)actor 传递,因为完成 RPC 可能落在集群中的任意 daprd 上。
该 actor 仅按实例 ID 将响应与等待中的执行匹配:在没有执行在等待时到达的响应会被暂存,同一实例的下一个执行会取走任何被暂存的响应。
ContinueAsNew 紧循环会让同一实例的两个执行背靠背运行;会合 actor 在第一个执行之后被异步停用,第二个执行的等待与该停用竞争并立即失败。
工作流 actor 正确地把新一代作为待启动恢复,但丢失执行的响应仍从应用侧到达并被暂存。
从此该实例的每一轮次都以上一轮的响应作答:投递 activity 结果的轮次收到了过期的 ScheduleTask 动作,actor 跳过了派发(认为已解析)并持久化了一个重复的 TaskScheduled,真正的完成事件被暂存而没有消费者,实例再也不推进。
解决方案
执行现在在其工作项被派发给应用之前就向会合 actor 注册,且注册会丢弃上一个执行遗留的任何响应,这样过期的响应就不可能再交给下一个轮次。 当存在注册或等待中的执行时,actor 不会再被停用,落在被并发停用关闭的 actor 上的调用会在一个新 actor 上重试,因此 ContinueAsNew 循环背靠背的执行不再丢失其结果。 作为纵深防御,工作流 actor 会拒绝其响应重建了已提交历史中已存在操作的轮次(这是过期响应的特征),并重试该轮次而不是持久化重复项。 该变更与滚动升级兼容:等待修复前 daprd 的执行,在面对已修复的 daprd 时保持此前行为;而注册被修复前 daprd 拒绝的已修复 daprd,会针对该工作项回退到此前行为。
可插拔 pub/sub 组件一次只能向应用投递一条消息
问题
从可插拔 pub/sub 组件收到的消息被串行派发给应用。
接收循环阻塞在处理器上,该处理器调用应用、等待其响应,然后才确认,因此在上一条消息被完全处理之前无法接收下一条消息。
每个订阅被限制为同时只有一条在途消息,其上限变为单条消息延迟的倒数。
发布不受影响,因此订阅者相对于此前能跟上的发布者会稳步落后,队列不断增长却没有任何错误日志。
批量订阅也随之劣化。
运行时从并发的处理器调用中组装批量批次,因此每次只有一个调用方时,配置的 maxMessagesCount 永远无法达到,每条消息都变成了它自己的应用请求。
影响
如果你在 v1.16.0 或更高版本上通过可插拔 pub/sub 组件消费消息,你就会受到影响。 以一个可插拔 RabbitMQ 组件按 5,000 msg/s 发布 300,000 条消息进行测量,消费速率从 5,004 msg/s 降至 3,026 msg/s,p95 端到端延迟从 15 ms 升至 37,945 ms,批量批次从平均 10.1 条消息坍缩到恰好 1 条。 吞吐量上限跟随处理器延迟,因此应用处理器越慢,影响越严重:一个耗时 60 ms 的处理器无论 broker 或组件容量如何,上限都在 16 msg/s 附近。 内置 pub/sub 组件不受影响。
根本原因
这种串行化是一项变更的附带损害,该变更本意是阻止在途输入绑定事件在优雅关闭期间被取消。
那次变更移除了 pullMessages 用来等待在途处理器的 WaitGroup(当时已冗余,因为运行时在 pkg/runtime/subscription 中自行跟踪在途消息),并连带压平了派发的 goroutine。
输入绑定在同一次变更中保留了其并发派发,而本用于让并发派发安全的 safeSend 互斥锁在这里被留下了。
解决方案
消息恢复并发派发,沿用输入绑定已有的形态,因此每个订阅可以有多条消息在途。
流上的发送仍由现有互斥锁串行化。
现在会在关闭流之前等待在途处理器,这样在订阅被拆除时仍在运行的处理器仍能确认它已处理的消息,而不是留给消息重投。
优雅关闭语义未变:排空、暂停和截止时间都在 pkg/runtime/subscription 中,统计的是处理器而非流 goroutine。
更新内容 (原始)
Dapr 1.18.4
This update contains the following bug fixes:
- Workflow left permanently PENDING when its start reminder fails during a scheduler restart
- Workflow stranded permanently PENDING when its start reminder fired before the creation commit
- Workflow left permanently PENDING when a placement rebalance interrupted its start reminder registration during a Scheduler outage
- Parent workflow deadlocked forever when a child workflow created after ContinueAsNew collided with a still-running child’s instance ID
- Output binding invocations over gRPC forwarding binary trace metadata to components
- Force purge failed for every workflow when history signing was enabled
- Workflow wrongly failed as tampered when a stale completion arrived after ContinueAsNew or a transient state store fault
- Workflow terminate silently dropped when delivered in the same batch as other events
- Workflow schedule and status waits failed with a stalled-actor error while a workflow was stalled
- Workflow NewGuid and other SDK-derived deterministic values repeated after ContinueAsNew
- Hot-reloaded workflow access policy listed in metadata before it was enforced
- Workflow wrongly failed as tampered when a child workflow creation was rejected under history signing
- Child workflow completion was delivered to the parent before the child committed its own state
- Force purge left the workflow actor serving and writing its stale in-memory state
- A child workflow that continued as new could not report its completion under history signing
- Workflow stuck RUNNING after ContinueAsNew when the new generation’s first execution was lost
- Workflow stuck RUNNING under WorkflowsClusteredDeployment when a turn was answered with the previous turn’s result
- Pluggable pub/sub components delivered messages to the application one at a time
Workflow left permanently PENDING when its start reminder fails during a scheduler restart
Problem
Creating a workflow while a Scheduler replica was shutting down could leave the new instance permanently stuck in PENDING.
The workflow actor commits its state (status PENDING) and then registers the start- reminder that launches execution as a separate Scheduler call.
When that call failed with rpc error: code = Unknown desc = cron is closed, the error was treated as permanent: the reminder create was attempted once and dropped, while the committed state remained.
The instance then reported PENDING indefinitely, had no reminder registered, never executed, and never transitioned to FAILED, so no error surfaced to monitoring.
The same window during a Scheduler restart usually produces Unavailable errors, which were correctly retried; only the narrower mid-shutdown window produced the non-retried Unknown error.
Impact
You were affected if workflows were created while a dapr-scheduler-server replica restarted, including routine rolling upgrades.
A single replica restart was sufficient.
Affected instances were silently lost: callers polling them saw a healthy-looking PENDING status forever, and the instances could only be recovered by purging and recreating them.
Root Cause
The Scheduler returns its internal cron shutdown error as a plain error, which crosses gRPC with code Unknown.
The bounded retry around actor reminder creation classified only Unavailable and DeadlineExceeded as transient, so Unknown was surfaced immediately without any retry, dropping the only signal that could ever start the workflow.
Solution
Workflow reminder creation now retries every error with bounded exponential backoff, except clearly-permanent request errors (InvalidArgument, PermissionDenied, Unauthenticated, FailedPrecondition, Unimplemented, NotFound) and Scheduler storage exhaustion (ResourceExhausted), which requires operator intervention and is surfaced to the caller immediately.
The Scheduler now returns etcd space-quota exhaustion with gRPC code ResourceExhausted instead of Unknown, so clients can distinguish it from transient failures.
Previously only Unavailable and DeadlineExceeded were retried, so any other transient error, including the Scheduler’s shutdown and etcd errors that cross the wire as Unknown, was dropped on the first attempt.
This applies to all workflow-related reminder creates (start, timer, activity, and termination reminders), and a genuinely permanent error still surfaces within the bounded retry budget.
The reminder scheduling error log now also includes the actor type and ID, so any failure can be attributed to the affected workflow instance.
Workflow stranded permanently PENDING when its start reminder fired before the creation commit
Problem
Creating a workflow during placement rebalancing or Scheduler leadership churn could leave the new instance permanently stuck in PENDING, with an empty history and no reminder registered for it.
The workflow actor created the one-shot start reminder before committing the instance’s state.
Under rebalancing, the reminder could fire on another host before that commit landed: the firing host found no inbox to process, acknowledged success, and the Scheduler deleted the one-shot reminder.
The commit then landed on an instance whose only execution trigger was already gone.
Impact
You were affected if workflows were created concurrently with placement dissemination (hosts joining or leaving) or Scheduler quorum loss, as reported in the field on 1.18.4-rc.2.
Affected instances reported PENDING forever, never executed, and never transitioned to FAILED.
Because the start reminder’s name contained a random suffix, no retry of the create could re-register it: retries only ever returned an already exists error, so the instance could only be recovered by purging and recreating it.
Root Cause
The start reminder was created before the ExecutionStarted inbox event was persisted, so a reminder firing in that window found an empty inbox and was acknowledged as complete, permanently consuming the one-shot.
The random reminder name made the loss unrecoverable: nothing could deterministically re-assert the same reminder afterwards.
Solution
The workflow actor now persists the ExecutionStarted inbox event first and creates the start reminder after the commit, under a deterministic name derived from the saved event.
A reminder can no longer fire against uncommitted state, and a reminder creation failure now surfaces to the caller as a retryable error instead of stranding the instance.
Retrying the create for an instance that was saved but whose start reminder is missing re-asserts the reminder under its deterministic name and reports success, reviving the instance instead of failing with already exists.
Workflow left permanently PENDING when a placement rebalance interrupted its start reminder registration during a Scheduler outage
Problem
Creating a workflow while the Scheduler was unreachable could leave the new instance permanently stuck in PENDING if a placement dissemination for the workflow actor type happened at the same time, for example the application scaling up or down during a Scheduler rollout.
The workflow actor commits its state first and then registers the start- reminder that launches execution, retrying the registration while the Scheduler is unavailable.
That retry ran under the placement claim context of the invocation, which a dissemination round cancels after the drain timeout regardless of the Scheduler’s health, so the registration was abandoned with the state already committed.
The create returned an error, but the instance existed: it reported PENDING on every status read, had no reminder registered, never executed, and never transitioned to FAILED.
Retrying the create did not recover it either. The retry checks whether the start reminder is missing before re-asserting it, and that check is a Scheduler round-trip that failed during the very outage that lost the reminder, so the retry aborted instead of re-asserting. A client that reads the workflow status and skips the create while the instance is not terminal, the pattern used in the documentation examples, never issued another create at all, and nothing else ever touched the instance.
Impact
You were affected if workflows were created, including re-created under a reused instance ID, while the Scheduler was disrupted and the application’s pod count changed within a few seconds of the create.
A Scheduler outage alone did not cause it; the coincidence with a placement round did.
Affected instances were silently lost: callers polling them saw a healthy-looking PENDING status forever, and the instances could only be recovered by purging and recreating them.
Root Cause
The start reminder registration performed after the state commit was bounded by the placement claim context of the invocation rather than by the lifetime of the runtime, so a dissemination round for the workflow actor type could abandon the only driver of an already committed instance.
The recovery path in the create retry depended on a Scheduler Get succeeding, and the status read used by clients bypassed the workflow actor entirely, so no code path re-asserted the missing reminder.
Solution
Once the state is committed, a start or wake-up reminder registration that fails for any reason other than a permanent request error is handed to a detached retry on the runtime lifetime, so a cancelled claim context or an exhausted retry budget can no longer leave a committed instance without a driver while the sidecar runs.
The create still returns the error to the caller, and the registration completes as soon as the Scheduler is reachable again.
The create retry now re-asserts the start reminder whenever the reminder check cannot reach the Scheduler, instead of failing.
Finally, a status read of a PENDING instance whose start is overdue by more than 5 seconds re-asserts its start reminder, through both the status API and status waits, so an instance whose sidecar died before the detached retry completed is recovered by any client that polls it, without a new create.
Parent workflow deadlocked forever when a child workflow created after ContinueAsNew collided with a still-running child’s instance ID
Problem
A parent workflow that called ContinueAsNew and then created a child workflow whose instance ID was still held by a running child from a previous execution of the same parent hung forever.
The child was never started, the parent’s child workflow task never completed or faulted, and the workflow status stayed RUNNING indefinitely.
The only trace was a debug-level log line: ignoring duplicate child workflow creation from parent '<id>'.
Auto-generated child instance IDs are derived from the parent instance ID and a per-execution action counter, and that counter restarts at zero on every ContinueAsNew, so the first auto-generated child ID of each generation is identical.
Similarly, deterministic SDK APIs such as the .NET SDK’s WorkflowContext.NewGuid() return the same value again after ContinueAsNew, so code that pins child instance IDs from such values re-issues an ID that is already in use.
Impact
You were affected if a workflow used ContinueAsNew and created child workflows whose previous-generation counterparts could still be running, for example a fire-and-forget child started before the ContinueAsNew.
Root Cause
Creating a child workflow whose instance ID already exists is deliberately deduplicated in the case when the parent crashed after successfully creating the child but before persisting its own state, its re-execution re-issues the same creation, which must be ignored for replay safety. That dedup check compared only the parent’s instance ID and the creating task’s ID. ContinueAsNew keeps the parent’s instance ID and resets its history, so task IDs restart from zero, and a genuinely new child creation from a later execution was indistinguishable from a crash-replay duplicate. The creation was dropped, the drop was reported as success, and nothing ever completed or faulted the parent’s pending child workflow task.
Solution
Every workflow execution already carries a unique execution ID, minted fresh on each ContinueAsNew and on instance recreation, and it is already included in child creation requests.
The dedup check now compares it as well: when the stored child’s parent execution ID and the incoming creation’s parent execution ID are both present and differ, the creation is a genuinely new one, not a crash-replay, and it now fails with an already exists error instead of being silently dropped.
That error faults the parent’s child workflow task, so the parent either handles it or transitions to FAILED with a clear message, matching the existing behavior for creating a child under an instance ID occupied by an unrelated running workflow.
Reusing a child instance ID whose previous holder has completed also works exactly as before.
Output binding invocations over gRPC forwarding binary trace metadata to components
Problem
Invoking an output binding through the gRPC API copied every incoming gRPC metadata entry into the metadata handed to the binding component, including binary metadata (keys ending in -bin, most notably grpc-trace-bin).
Binary gRPC metadata is base64-decoded by the gRPC server before the runtime sees it, so raw trace bytes were written into the component’s string metadata under the key dapr-grpc-trace-bin.
Components that require text metadata failed on these bytes. The Azure Service Bus binding rejected the message outright:
Error invoking output binding <binding-name>: (connlost): not a valid UTF-8 string
The Azure Blob Storage binding with Shared Key authentication failed intermittently, whenever the trace ID happened to contain a horizontal TAB byte (0x09) that survived the component’s header sanitizer but was removed by Azure’s canonicalization:
403 AuthenticationFailed
The MAC signature found in the HTTP request is not the same as any computed signature
Impact
You were affected if your application invoked output bindings through the gRPC Dapr API using a client that sends the grpc-trace-bin header, and the target component treats metadata as text.
The .NET SDK attaches grpc-trace-bin to every gRPC call since 1.18.5, and the Java SDK has long done the same, so upgrading the .NET SDK made this failure appear in previously working deployments.
Because the offending bytes come from the random trace and span IDs, the failures were intermittent and depended on the individual trace.
With Blob Storage, authentication methods not based on request signing (such as Managed Identity) did not fail the request; the corrupted value was instead silently written into the stored blob metadata.
Root Cause
The gRPC InvokeBinding handler merged all incoming gRPC metadata into the binding request so that text tracing headers (traceparent, tracestate, baggage) and custom metadata propagate to components.
The merge did not distinguish binary metadata: grpc-trace-bin was renamed to dapr-grpc-trace-bin and its raw decoded bytes were stored in the string metadata map, then passed verbatim to the component.
The HTTP binding API takes metadata exclusively from the request body and was not affected.
Solution
Binary gRPC metadata (any key ending in -bin) is no longer copied from the incoming gRPC call into output binding component metadata.
Metadata set explicitly on the InvokeBinding request itself is not filtered, and text tracing metadata (traceparent, tracestate, baggage) continues to propagate to components unchanged.
Distributed tracing is unaffected: the runtime consumes grpc-trace-bin directly from the gRPC call for span propagation, never from binding metadata, which no component reads.
Force purge failed for every workflow when history signing was enabled
Problem
With the WorkflowHistorySigning preview feature enabled, force-purging a workflow instance always failed with a configuration error stating the workflow has signed history but no signer is configured.
This made the force purge escape hatch, intended for removing instances that cannot be purged normally, unusable on any signing-enabled installation.
Impact
You were affected if you ran with WorkflowHistorySigning enabled and attempted a force purge, whether recursive or not: both forced modes reach the same load path.
No signed instance could ever be force-purged; the call failed before deleting anything.
Root Cause
The force purge path loaded workflow state without passing the runtime’s signer and namespace to the state loader, unlike every other load site. The loader’s tamper protection rejects loading signed history when no signer is available to verify it, so the load, and therefore the purge, always failed.
Solution
The force purge path now passes the signer and namespace to the state loader, matching the other call sites, so signed history verifies normally and the purge proceeds.
Workflow wrongly failed as tampered when a stale completion arrived after ContinueAsNew or a transient state store fault
Problem
With WorkflowHistorySigning enabled, a workflow could be terminally failed with error type DAPR_WORKFLOW_HISTORY_TAMPERED without any tampering having occurred.
Two benign situations triggered it: a child workflow’s completion arriving after the parent had called ContinueAsNew, and an activity’s retried completion arriving after a transient state store fault rolled back the save that recorded the activity’s scheduling.
Once tombstoned, the instance was unrecoverable: all reminders were deleted and every subsequent event was rejected.
Impact
You were affected if you ran with WorkflowHistorySigning enabled and used ContinueAsNew with child workflows that could complete around the transition, or if your actor state store experienced transient write failures under load.
Affected instances failed permanently with a tampering error despite no integrity violation, requiring a purge and re-creation.
Root Cause
Attestation verification of an inbound completion looks up the completion’s task scheduled ID in signed history. ContinueAsNew legitimately resets history (so a previous execution’s scheduling events are gone), and a rolled-back save legitimately retracts a scheduling event that was already dispatched. Verification treated the resulting “unknown task scheduled id” identically to a signature mismatch and tombstoned the workflow, whereas the non-signing path simply drops such stale completions.
Solution
A completion referencing a task scheduled ID absent from signed history is now dropped without persisting anything, matching the non-signing behavior; the sender is told the instance no longer accepts it and stops redelivering. The same applies when the ID is present but records a different invocation: task IDs restart after ContinueAsNew, so a straggler completion can land on a reused ID belonging to the new generation’s own child or activity; a validly signed completion whose input digest does not match the recorded invocation is dropped as stale instead of tombstoning the workflow. This is safe because the event is discarded before it is written: a forged completion with an unknown ID gains an attacker nothing. Additionally, before tombstoning on any other verification failure, the runtime now re-verifies the event against freshly loaded durable state, so a stale in-memory cache can no longer condemn a healthy workflow; genuine signature mismatches still tombstone as before.
Workflow terminate silently dropped when delivered in the same batch as other events
Problem
Terminating a workflow intermittently had no effect: the workflow’s status stayed RUNNING and it kept executing activities and timers as if the terminate had never been issued.
Blocking terminate calls (such as TerminateWorkflowAsync in the .NET SDK) never returned, because they wait for the instance to reach a terminal status that never came.
With debug logging enabled, a dropped terminate showed the ExecutionTerminated event being delivered and consumed with no effect:
received work item with 2 new event(s): [ExecutionTerminated, TaskCompleted#1]
workflow execution returned with status 'ORCHESTRATION_STATUS_RUNNING'
A workflow terminated while it was suspended was affected the same way, deterministically: the status stayed SUSPENDED and every terminate sent to it was lost.
Impact
You were affected if you terminated workflows that were actively making progress, regardless of SDK language.
The window depends on inbox pressure: the terminate is dropped when it arrives in the same work item batch as another event and is not the last event in that batch, so long-running workflows that loop over short activities and timers (endless pollers, monitors, continue_as_new loops) were the most exposed, while a workflow idling on a single slow activity or timer almost always received its terminate alone and was unaffected.
A dropped terminate is consumed with its batch and never redelivered, so the instance kept running; a retried terminate raced the same window again. A recursive terminate that was dropped also never cascaded, leaving child workflows running as orphans. For suspended workflows there was no window: every terminate was lost until the workflow was resumed.
Root Cause
The workflow engine hands an instance’s pending events to the workflow executor as one batch, and relied entirely on the SDK executor to turn an ExecutionTerminated event into a termination outcome.
SDK executors registered the termination when they processed the event but kept feeding the remaining events of the batch into the workflow code, which resumed past its own termination and produced a competing outcome: scheduling more work, completing normally, or restarting via continue_as_new.
A continue_as_new outcome always discarded the termination; other outcomes raced it nondeterministically.
A terminate delivered while the workflow was suspended produced no outcome at all, because suspension suppressed every action the executor would have returned.
Since the engine trusted the executor’s result and the event was consumed with the batch, the terminate was lost permanently.
Solution
When a delivered batch contains an ExecutionTerminated event and the executor does not complete the workflow, daprd discards the doomed execution’s pending work and completes the instance as TERMINATED.
A continue_as_new returned alongside a terminate no longer starts a new iteration, and terminating a suspended workflow now terminates it without requiring a resume.
Workflow schedule and status waits failed with a stalled-actor error while a workflow was stalled
Problem
Scheduling a workflow whose first turns stalled quickly, or waiting on the status of an already-stalled workflow, could fail with an error stating the actor is stalled.
The instance itself was healthy: it was created, ran, and entered the intended STALLED state, but the call observing it returned an error instead of the status.
Impact
You were affected if workflows stalled, for example on PAYLOAD_SIZE_EXCEEDED or a version mismatch, and clients scheduled instances or waited on their status while the stall was in place.
ScheduleNewWorkflow could return an error for an instance that was created successfully, and status waits could fail instead of waiting for recovery.
Root Cause
A stalling turn parks holding the workflow actor’s turn lock so the stall is not retried hot. A status watch stream registering during that hold was rejected with the stalled-actor error, and the error propagated to the waiting client instead of being treated as a state to wait through.
Solution
When a status watch finds the actor stalled, the stored workflow status is read directly from the state store: a wait whose condition the instance already reached, such as a schedule waiting for the workflow to start, resolves immediately, and any other wait re-registers with backoff until the stall clears.
Workflow NewGuid and other SDK-derived deterministic values repeated after ContinueAsNew
Problem
Deterministic values that workflow SDKs derive per execution were repeated after a workflow called ContinueAsNew.
Most visibly, the .NET SDK’s WorkflowContext.NewGuid() returned the same sequence of GUIDs again in the new generation, so code that pinned child workflow instance IDs from those GUIDs re-issued IDs already in use and, from 1.18.4, received already exists failures instead of fresh children.
Impact
You were affected if a workflow used ContinueAsNew together with SDK APIs that derive deterministic values, such as NewGuid() or auto-generated child workflow instance IDs in the .NET SDK.
Every generation reproduced the previous generation’s values.
Root Cause
SDKs seed these derivations from the execution ID carried on each workflow work item, falling back to the instance ID when the field is empty.
The instance ID is unchanged across ContinueAsNew while the SDK’s derivation counters restart with the new history, so with an empty execution ID every generation reproduced the same values.
The sidecar never populated the field: the workflow engine sent every work item with an empty execution ID, even though each generation’s ExecutionStarted history event carries a freshly minted one.
Solution
Workflow work items now carry the execution ID of the generation they belong to, taken from the ExecutionStarted event, scanning new events before past events so a continued-as-new or recreated run reports its own execution rather than a stale one.
Each generation’s execution ID is minted fresh on ContinueAsNew and instance recreation but is persisted in history, so SDK-derived values become unique per generation while remaining stable across replays within a generation.
Workflows in flight across the upgrade re-seed from the execution ID on their next turn; values already recorded in their history replay from history as before.
Hot-reloaded workflow access policy listed in metadata before it was enforced
Problem
When a WorkflowAccessPolicy resource was created, updated or deleted at runtime, the sidecar’s metadata endpoint listed the new set of policies a small amount of time before the new set was actually enforced.
A caller that waited for metadata to report a tightened policy and then invoked a workflow operation could still be authorized against the previous, more permissive policy set; the reverse held for a policy that had just been deleted.
Impact
Operators or automation using the metadata endpoint as the signal that a policy change is live could observe a brief window in which the reported and the enforced policies disagreed. Steady-state enforcement was unaffected: the window closed as soon as the new set finished compiling.
Root Cause
The hot-reload reconciler added the policy to the resource store first and only then compiled the full set and swapped it into the workflow API. Compiling the policy expressions takes a small amount of time, and the metadata endpoint lists the resource store rather than the compiled set.
Solution
The reconciler now compiles the prospective policy set and swaps it into enforcement before it updates the resource store, for both updates and deletes, so a policy listed by the metadata endpoint is always one that is already enforced.
Workflow wrongly failed as tampered when a child workflow creation was rejected under history signing
Problem
With WorkflowHistorySigning enabled, a parent workflow whose child workflow creation was rejected by the runtime was terminally failed with error type DAPR_WORKFLOW_HISTORY_TAMPERED and the message child failure missing required attestation for task <n>.
Two rejections trigger it: creating a child under an instance ID still held by a running child of a previous execution of the same parent (the already exists failure introduced in this release), and a child creation denied by a WorkflowAccessPolicy.
Once tombstoned, the instance was unrecoverable: its reminders were deleted and the old child’s later completion was dropped with no such instance exists.
Impact
You were affected if you ran with WorkflowHistorySigning enabled and a workflow used ContinueAsNew and re-created a child whose previous-generation counterpart could still be running, or created children denied by an access policy.
Without signing the same parent observes an ordinary child workflow failure it can handle; with signing it failed permanently and deterministically on every occurrence.
An activity denied by an access policy had the same latent defect.
Root Cause
When the runtime rejects a child workflow creation it synthesizes a ChildWorkflowInstanceFailed event for the parent and delivers it through a reminder.
Such an event carries no attestation by design, so attestation verification exempts it when the event’s router names the parent’s own application as its source.
The code that synthesizes the event never set the router, so the exemption never matched and the event was treated as an unattested completion from an untrusted sender.
Solution
The synthesized child workflow failure and the synthesized activity failure now record the parent’s application as their source, so attestation verification recognises them as locally authored and the parent observes the failure as intended.
Child workflow completion was delivered to the parent before the child committed its own state
Problem
When a child workflow completed, its completion was delivered to the parent workflow before the child’s own terminal state was written to the state store.
A client that observed the parent as COMPLETED and immediately queried the child could still see the child as RUNNING or PENDING, and a sidecar crash or a store error in that window re-ran the child’s final turn and delivered the completion a second time.
The parent deduplicates such completions by task ID only, so after a parent ContinueAsNew, where task IDs restart, a late duplicate could be attributed to the wrong child task.
Impact
Any workflow using child workflows: transiently inconsistent parent and child statuses, duplicated completion deliveries relying on parent-side deduplication, and a rare misattributed completion after ContinueAsNew.
Root Cause
The completion dispatch was ordered before the child’s state save in the turn, so the parent’s copy of the completion became durable before the child’s.
Solution
The child now commits its terminal state first, together with a durable marker recording that its parent has not yet acknowledged the completion, then delivers the completion and clears the marker once the parent has it.
If the delivery fails or the sidecar restarts in between, the notification is rebuilt from the child’s persisted history and re-sent until the parent acknowledges it, so the parent only ever learns of a completion that is already durable, and purging the child or reusing its instance ID is refused as not completed while the marker is present.
The parent also rejects a completion whose sender is not the child its current generation created for that task, or which was created under a previous execution of the parent, so a stale completion can no longer be attributed to a new child after ContinueAsNew, and a parent that has already completed acknowledges a late child completion without running a turn.
A child workflow that continued as new could not report its completion under history signing
Problem
With WorkflowHistorySigning enabled, a child workflow in another application that called ContinueAsNew never resolved its parent’s task: the parent dropped every completion the child sent as a stale one, and the parent stayed running until it was terminated.
Impact
Any signed, cross-application child workflow that continues as new. The parent could not complete.
Root Cause
The parent verifies the child’s completion attestation against the input it created the child with. After ContinueAsNew the child’s current start event carries the continued input, and the child attested with that, so the digests never matched.
Solution
The child records the input it was created with at creation (signed children only, since only the attestation reads it) and attests its completion with that input across any number of ContinueAsNew calls.
A signed child created before this fix that has already continued as new no longer holds its creation input anywhere, so its completion still cannot be verified; such instances are the ones running across the upgrade.
Force purge left the workflow actor serving and writing its stale in-memory state
Problem
A force purge deleted a workflow’s state store rows and reminders but did not evict the workflow actor still resident in memory. That actor kept answering status streams and cross-app metadata reads from its cached state, so a purged instance could still report as running, and a later event or turn against it could write its cached state back, partially resurrecting the purged instance.
Impact
Operators using force purge to remove a stuck or corrupted instance while its actor was still active, and any client observing that instance afterwards.
Root Cause
Force purge is a store-level operation that bypassed the actor, and the actor’s post-save version check could not distinguish a purged row from a store that returns no version tokens, so it fell through to an unconditional write.
Solution
Force purge now evicts the instance’s workflow actor on the host that runs it before deleting the rows, so any later access reloads from the store and finds nothing. An actor whose metadata row disappears underneath it now drops its cache and deactivates instead of writing it back, and a save whose known row version was lost is refused rather than performed unconditionally.
Workflow stuck RUNNING after ContinueAsNew when the new generation’s first execution was lost
Problem
A workflow that called ContinueAsNew could get stuck RUNNING forever with WorkflowsClusteredDeployment enabled.
The runtime logged failed before receiving workflow result and aborting workflow execution for the ContinueAsNew turn, and the new generation’s first execution then received the previous generation’s events, for example the timer that triggered the ContinueAsNew and the completion of a child workflow started by the previous generation.
When the new generation created a child workflow under the same task ID, the stale completion was recorded ahead of the creation, the new child’s genuine completion was dropped as a duplicate (dropping duplicate completion event already present in history/inbox), and the parent never advanced.
Impact
You were affected if you ran with WorkflowsClusteredDeployment enabled and used ContinueAsNew; workflows that also created child workflows or timers in each generation were the most likely to strand, at a rate of several percent per ContinueAsNew under load or during pod restarts.
The same defect could be reached on any deployment when a workflow exceeded the engine’s tight-loop ContinueAsNew limit.
Root Cause
When the workflow engine abandons a work item after the workflow has already continued-as-new, the runtime persists the new generation so that progress is not lost. It wrote the new generation’s start events to history but replaced the consumed inbox only when the previous generation left unprocessed external events to carry over. Without carryover the previous generation’s consumed events stayed in the inbox and were delivered again as the new generation’s first work item, where the task ID sequence has restarted, so a previous generation’s completion could take over a task ID of the new generation.
Solution
An abandoned continued-as-new generation is now persisted exactly like a freshly created workflow: an empty history, its start event leading the inbox followed by any carried-over events, and a start reminder that drives it, created after the state is saved. The previous generation’s consumed events are discarded, and the new generation always runs on its own.
Workflow stuck RUNNING under WorkflowsClusteredDeployment when a turn was answered with the previous turn’s result
Problem
With the WorkflowsClusteredDeployment feature enabled, a workflow that used ContinueAsNew and ran an activity in each generation could stop making progress and report RUNNING forever.
The last committed history of a stuck instance ended with a duplicate TaskScheduled for an activity that had already completed, its inbox was empty, and no reminder was left to drive it.
Impact
You were affected if you ran daprd behind a load balancer with WorkflowsClusteredDeployment enabled and your workflows continued as new, in particular in a loop that ran activities or child workflows per generation.
Affected instances looked healthy (RUNNING) but never completed, raised no error, and could only be recovered by terminating or purging and recreating them.
Root Cause
Under WorkflowsClusteredDeployment the application’s execution results travel through a per-instance rendezvous actor, because the completion RPC can land on any daprd in the cluster.
That actor matched a response to a waiting execution by instance ID only: a response arriving while no execution was waiting was parked, and the next execution of the same instance took whatever was parked.
The ContinueAsNew tight loop runs two executions of one instance back to back; the rendezvous actor was deactivated asynchronously after the first one, so the second execution’s wait raced the deactivation and failed instantly.
The workflow actor correctly recovered the new generation as a pending start, but the response of the lost execution still arrived from the application and was parked.
From then on every turn of the instance was answered with the previous turn’s response: the turn that delivered the activity result received a stale ScheduleTask action, the actor skipped the dispatch as already resolved and persisted a duplicate TaskScheduled, the real completion was parked with no consumer, and the instance never advanced.
Solution
The execution now registers with the rendezvous before its work item is dispatched to the application, and the registration discards any response left behind by a previous execution, so a stale response can no longer be handed to the next turn. The actor is no longer deactivated while a registration or a waiting execution is present, and a call that lands on an actor closed by a concurrent deactivation is retried on a fresh one, so back-to-back executions of a ContinueAsNew loop no longer lose their result. As defence in depth, the workflow actor rejects a turn whose response re-creates an operation that already exists in committed history (the signature of a stale response) and retries the turn instead of persisting the duplicate. The change is compatible with a rolling upgrade: executions waiting on a pre-fix daprd keep the previous behaviour against a fixed daprd, and a fixed daprd whose registration is rejected by a pre-fix daprd falls back to the previous behaviour for that work item.
Pluggable pub/sub components delivered messages to the application one at a time
Problem
Messages received from a pluggable pub/sub component were dispatched to the application serially. The receive loop blocked on the handler, which invokes the application, waits for its response and only then acknowledges, so the next message could not be received until the previous one was fully handled. Each subscription was limited to a single message in flight and its ceiling became the inverse of the per-message latency. Publishing was unaffected, so a subscriber fell steadily behind a publisher it had previously kept up with, and the queue grew without any error being logged.
Bulk subscribe degraded along with it.
The runtime assembles bulk batches out of concurrent handler calls, so with one caller at a time the configured maxMessagesCount could never be reached and every message became its own application request.
Impact
You were affected if you consumed messages through a pluggable pub/sub component on v1.16.0 or later. Measured against a RabbitMQ pluggable component publishing 300,000 messages at 5,000 msg/s, consumption fell from 5,004 to 3,026 msg/s, p95 end-to-end latency rose from 15 ms to 37,945 ms, and bulk batches collapsed from a mean of 10.1 messages to exactly 1. The throughput ceiling tracks handler latency, so the slower the application handler, the more severe the effect: a handler taking 60 ms tops out near 16 msg/s regardless of broker or component capacity. Built-in pub/sub components were not affected.
Root Cause
The serialization was collateral damage of a change that set out to stop in-flight input binding events from being cancelled during graceful shutdown.
That change removed the WaitGroup pullMessages used to wait on in-flight handlers, by then redundant because the runtime tracks in-flight messages itself in pkg/runtime/subscription, and collapsed the dispatching goroutine along with it.
Input bindings kept their concurrent dispatch in that same change, and the safeSend mutex that exists to make concurrent dispatch safe was left in place here.
Solution
Messages are dispatched concurrently again, following the shape input bindings already use, so more than one message can be in flight per subscription.
Sends on the stream remain serialized by the existing mutex.
In-flight handlers are now waited on before the stream is closed, so a handler that is still running when the subscription is torn down can still acknowledge the message it has already processed instead of leaving it to be redelivered.
Graceful shutdown semantics are unchanged: draining, pausing and the deadline live in pkg/runtime/subscription and count handlers rather than stream goroutines.
下载链接
- daprd_darwin_amd64.tar.gz
- daprd_darwin_amd64.tar.gz.sha256
- daprd_darwin_arm64.tar.gz
- daprd_darwin_arm64.tar.gz.sha256
- daprd_linux_amd64-stablecomponents.tar.gz
- daprd_linux_amd64-stablecomponents.tar.gz.sha256
- daprd_linux_amd64.tar.gz
- daprd_linux_amd64.tar.gz.sha256
- daprd_linux_arm-stablecomponents.tar.gz
- daprd_linux_arm-stablecomponents.tar.gz.sha256
- daprd_linux_arm.tar.gz
- daprd_linux_arm.tar.gz.sha256
- daprd_linux_arm64-stablecomponents.tar.gz
- daprd_linux_arm64-stablecomponents.tar.gz.sha256
- daprd_linux_arm64.tar.gz
- daprd_linux_arm64.tar.gz.sha256
- daprd_windows_amd64.zip
- daprd_windows_amd64.zip.sha256
- grafana-actor-dashboard.json
- grafana-actor-dashboard.json.sha256
- grafana-sidecar-dashboard.json
- grafana-sidecar-dashboard.json.sha256
- grafana-system-services-dashboard.json
- grafana-system-services-dashboard.json.sha256
- injector_darwin_amd64.tar.gz
- injector_darwin_amd64.tar.gz.sha256
- injector_darwin_arm64.tar.gz
- injector_darwin_arm64.tar.gz.sha256
- injector_linux_amd64.tar.gz
- injector_linux_amd64.tar.gz.sha256
- injector_linux_arm.tar.gz
- injector_linux_arm.tar.gz.sha256
- injector_linux_arm64.tar.gz
- injector_linux_arm64.tar.gz.sha256
- injector_windows_amd64.zip
- injector_windows_amd64.zip.sha256
- operator_darwin_amd64.tar.gz
- operator_darwin_amd64.tar.gz.sha256
- operator_darwin_arm64.tar.gz
- operator_darwin_arm64.tar.gz.sha256
- operator_linux_amd64.tar.gz
- operator_linux_amd64.tar.gz.sha256
- operator_linux_arm.tar.gz
- operator_linux_arm.tar.gz.sha256
- operator_linux_arm64.tar.gz
- operator_linux_arm64.tar.gz.sha256
- operator_windows_amd64.zip
- operator_windows_amd64.zip.sha256
- placement_darwin_amd64.tar.gz
- placement_darwin_amd64.tar.gz.sha256
- placement_darwin_arm64.tar.gz
- placement_darwin_arm64.tar.gz.sha256
- placement_linux_amd64.tar.gz
- placement_linux_amd64.tar.gz.sha256
- placement_linux_arm.tar.gz
- placement_linux_arm.tar.gz.sha256
- placement_linux_arm64.tar.gz
- placement_linux_arm64.tar.gz.sha256
- placement_windows_amd64.zip
- placement_windows_amd64.zip.sha256
- scheduler_darwin_amd64.tar.gz
- scheduler_darwin_amd64.tar.gz.sha256
- scheduler_darwin_arm64.tar.gz
- scheduler_darwin_arm64.tar.gz.sha256
- scheduler_linux_amd64.tar.gz
- scheduler_linux_amd64.tar.gz.sha256
- scheduler_linux_arm.tar.gz
- scheduler_linux_arm.tar.gz.sha256
- scheduler_linux_arm64.tar.gz
- scheduler_linux_arm64.tar.gz.sha256
- scheduler_windows_amd64.zip
- scheduler_windows_amd64.zip.sha256
- sentry_darwin_amd64.tar.gz
- sentry_darwin_amd64.tar.gz.sha256
- sentry_darwin_arm64.tar.gz
- sentry_darwin_arm64.tar.gz.sha256
- sentry_linux_amd64.tar.gz
- sentry_linux_amd64.tar.gz.sha256
- sentry_linux_arm.tar.gz
- sentry_linux_arm.tar.gz.sha256
- sentry_linux_arm64.tar.gz
- sentry_linux_arm64.tar.gz.sha256
- sentry_windows_amd64.zip
- sentry_windows_amd64.zip.sha256