发布日期: 2026-03-26
版本号: v1.16.11

Dapr 1.16.11版本更新主要包括将Go版本升级至1.25.8，并修复了几个关键错误。具体修复内容包括：解决了在领导权变更期间Scheduler因状态机错误崩溃的问题；修复了集群扩容后Scheduler实例可能静默停止参与集群工作的问题；以及解决了由于镜像清单中缺少OSVersion字段导致Windows边车容器在AKS上无法启动的问题。此次更新提升了系统的稳定性和安全性。

更新内容 （中文）

Dapr 1.16.11

本次更新包含 Go 版本升级和关键缺陷修复：

Go 版本升级至 1.25.8

问题描述

Dapr 1.16.10 使用 Go 1.25.7 构建。 Go 1.25.8 包含对 html/template、net/url 和 os 包的安全修复，以及对 go 命令、编译器和 os 包的缺陷修复。

影响范围

使用 Go 1.25.7 构建的 Dapr 用户可能暴露在 Go 1.25.8 中已修补的已知漏洞下。

解决方案

将 Go 版本从 1.25.7 升级至 1.25.8。

Scheduler 在领导权变更期间因"致命状态机错误"崩溃

问题描述

在高作业吞吐量期间，Dapr Scheduler 进程因致命错误在领导权仲裁变更时崩溃：

level=fatal msg="Fatal error running scheduler: catastrophic state machine error: lost inner loop reference"

一旦 Scheduler 实例崩溃，剩余实例将经历进一步的仲裁不稳定，这可能导致这些实例也发生相同崩溃，从而在整个 Scheduler 集群中引发级联故障。所有工作流执行、计划作业和 Actor 提醒均停止触发，直至 Scheduler 集群恢复。

影响范围

所有以多实例（高可用）配置运行 Scheduler 的 Dapr 部署均受影响。 当 Scheduler 实例重启时（无论由于 Kubernetes Pod 驱逐、OOM 终止、节点维护、滚动更新或瞬态 etcd 连接问题），且集群处于高作业吞吐量状态下，崩溃即被触发。 这对使用 Dapr 工作流的部署尤为严重，因为 Scheduler 负责调度所有工作流活动和编排定时器。

受影响版本：v1.16.0 至 v1.16.10。

根因分析

Scheduler 的内部定时任务引擎使用路由器管理每个作业的计数器循环。 当作业完成或被删除时，计数器循环向路由器发出 CloseJob 事件，路由器从其内部映射中移除该计数器。

在领导权仲裁变更期间（例如，当 Scheduler 实例宕机时分区数从 3 变为 2），定时任务引擎将重启。 在旧引擎关闭与新引擎启动之间的短暂窗口期，来自旧引擎计数器循环的陈旧 CloseJob 事件可能到达新引擎的路由器。 由于新路由器以空计数器映射启动，这些陈旧事件引用的计数器不存在。

此前，此条件返回致命错误（"catastrophic state machine error: lost inner loop reference"），该错误通过路由器的事件循环传播，取消路由器的父上下文，并终止整个 Scheduler 进程。

解决方案

路由器现将针对不存在计数器的 CloseJob 事件视为良性空操作：陈旧事件被丢弃并记录调试日志，路由器继续正常处理。 此修复安全，因为：

  1. 计数器已被清理——无需关闭。
  2. 该事件来自先前的引擎生命周期，不再相关。
  3. 所有其他 CloseJob 代码路径（计数器存在、计数器使用新索引重用）不受影响。

此修复位于 go-etcd-cron 依赖中（diagridio/go-etcd-cron#127）。

集群扩容后 Scheduler 实例静默停止参与调度

问题描述

在 Scheduler Pod 重启、滚动更新或集群扩容事件后，一个或多个 Scheduler 实例可能静默停止参与集群。 受影响实例的 Pod 保持运行并通过健康检查，但其定时任务引擎退出且永不重启。 该实例停止向 WatchHosts API 发布其地址，因此 daprd sidecar 永远无法发现它。 分配到受影响实例分区的作业、Actor 提醒或工作流永不被触发。

从用户角度看，工作流、计划作业和 Actor 提醒在 Scheduler Pod 重启后随机停止触发。 此问题具有间歇性，取决于重启相对于其他集群活动的精确时序。

影响范围

所有以多实例（高可用）配置运行 Scheduler 的 Dapr 部署均受影响。 当 Scheduler 实例加入或重新加入集群导致领导权仲裁变更时（例如，分区数从 2 变为 3），此问题即被触发。 可能性随已连接 daprd sidecar 数量增加而增大，因为更多 WatchHosts 订阅者意味着主机广播耗时更长，竞争窗口更宽。

当此缺陷触发时：

  • 受影响的 Scheduler 实例拥有分区但无法在其上交付作业。分配到这些分区的所有工作流活动定时器、计划作业和 Actor 提醒停止触发，并被记录为 UNDELIVERABLE。
  • 调用 WatchHosts 的 daprd sidecar 可能接收不完整的主机列表（缺失受影响实例），或可能永远收不到响应——导致 sidecar 无限期停留在 scheduler-watch-hosts 就绪门控，阻止应用进入就绪状态。
  • 其余健康实例无法接管受影响实例的分区，因为其领导权密钥仍存在于 etcd 中（租约续期独立继续）。集群陷入所有实例均运行但仲裁永远无法收敛到正确分区数的状态。

唯一恢复方法是同时重启所有 Scheduler Pod，强制进行新的领导权选举。

根因分析

当 Scheduler 的内部定时模块在分区变更后达到领导权仲裁时，它会调用 runEngine 启动定时任务引擎。 在启动引擎前，runEngine 将更新后的集群主机地址发送到内部无缓冲 Go 通道（WatchLeadership），以便 WatchHosts API 将其广播给已连接的 sidecar。

如果通道消费者正忙（例如，向多个 WatchHosts 订阅者广播之前的主机更新），发送会阻塞。 同时，如果另一个 Scheduler 实例加入并导致第二次仲裁变更，当选定的上下文在发送仍阻塞时被取消。

由于定时模块已退出，它永远不会调用 Reelect 来使用新的分区总数更新其领导权密钥。 其他 Scheduler 实例看到此陈旧密钥，无法达成仲裁协议，阻止整个集群收敛。

解决方案

Scheduler 定时包装器中的内部通道消费者已被非阻塞事件循环（events/loop）替换。 该循环的 Enqueue 方法永不阻塞。 如果当前分段已满，它会分配新分段。 这意味着来自定时库的通道发送总是立即完成，无论消费者广播多忙。

由于发送不再阻塞，导致静默退出的上下文取消竞争不再可能发生。 定时循环继续在每次仲裁变更后调用 Reelect，领导权密钥使用正确的分区总数更新，所有实例正常收敛。

因镜像清单缺失 OSVersion 导致 Windows sidecar 容器在 AKS 上启动失败

问题描述

从 Dapr v1.16.9 开始，Dapr sidecar 容器（daprd）在 AKS Windows 节点上启动失败，错误为：

hcs::CreateComputeSystem daprd: The container operating system does not match the host operating system.

影响范围

v1.16.9 及之后所有基于 Windows 的 AKS Dapr sidecar 部署均受影响。daprd 容器进入 CrashLoopBackOff 状态且永不启动。Linux 部署不受影响。

根因分析

在 v1.16.9 中，docker/docker.mk 中的 docker-manifest-create 目标从使用 docker manifest create / docker manifest push 变为使用 docker buildx imagetools create。

docker manifest 命令自动从每个源镜像的配置中读取 os.version 并将其包含在清单列表条目中。docker buildx imagetools create 命令不会将 os.version 字段传递到清单列表中。

Windows 清单条目中缺少 os.version，导致 Windows 容器运行时无法区分两个 windows/amd64 镜像（Server 2019 和 Server 2022），并为主机 OS 构建拉取错误的变体。

解决方案

将 docker-manifest-create 目标恢复使用 docker manifest create 和 docker manifest push，恢复 Windows 镜像清单列表条目中的 os.version 字段。

更新内容 （原始）

Dapr 1.16.11

This update includes a Go version bump, and critical bug fixes:

Go version updated to 1.25.8

Problem

Dapr 1.16.10 was built with Go 1.25.7. Go 1.25.8 includes security fixes to the html/template, net/url, and os packages, as well as bug fixes to the go command, the compiler, and the os package.

Impact

Users running Dapr built with Go 1.25.7 may be exposed to known vulnerabilities that have been patched in Go 1.25.8.

Solution

Updated the Go version from 1.25.7 to 1.25.8.

Scheduler crashes with “catastrophic state machine error” during leadership changes

Problem

The Dapr Scheduler process crashes with a fatal error during leadership quorum changes under high job throughput:

level=fatal msg="Fatal error running scheduler: catastrophic state machine error: lost inner loop reference"

Once a Scheduler instance crashes, the remaining instances experience further quorum instability, which can trigger the same crash on those instances as well, leading to a cascading failure across the entire Scheduler cluster. All workflow executions, scheduled jobs, and actor reminders stop firing until the Scheduler cluster recovers.

Impact

Any Dapr deployment running Scheduler in a multi-instance (HA) configuration is affected. The crash is triggered when a Scheduler instance restarts, whether due to a Kubernetes pod eviction, OOM kill, node maintenance, rolling update, or transient etcd connectivity issue, while the cluster is under high job throughput. This is especially severe for deployments using Dapr Workflows, as the Scheduler is responsible for dispatching all workflow activity and orchestration timers.

Affected versions: v1.16.0 through v1.16.10.

Root Cause

The Scheduler’s internal cron engine uses a router that manages per-job counter loops. When a job completes or is deleted, the counter loop emits a CloseJob event to the router, which removes the counter from its internal map.

During a leadership quorum change (e.g., partition count changes from 3 to 2 when a Scheduler instance goes down), the cron engine restarts. In the narrow window between the old engine shutting down and the new engine starting up, stale CloseJob events from the old engine’s counter loops can arrive at the new engine’s router. Because the new router starts with an empty counter map, these stale events reference counters that do not exist.

Previously, this condition returned a fatal error ("catastrophic state machine error: lost inner loop reference"), which propagated through the router’s event loop, cancelled the router’s parent context, and terminated the entire Scheduler process.

Solution

The router now treats a CloseJob event for a missing counter as a benign no-op: the stale event is dropped with a debug-level log message and the router continues processing normally. This is safe because:

  1. The counter was already cleaned up — there is nothing to close.
  2. The event is from a previous engine lifecycle and is no longer relevant.
  3. All other CloseJob code paths (counter exists, counter reused with new index) are unaffected.

This fix is in the go-etcd-cron dependency (diagridio/go-etcd-cron#127).

Scheduler instance silently stops participating after cluster scale-up

Problem

After a Scheduler pod restart, rolling update, or cluster scale-up event, one or more Scheduler instances can silently stop participating in the cluster. The affected instance’s pod remains running and passes health checks, but its cron engine exits and never restarts. The instance stops publishing its address to the WatchHosts API, so daprd sidecars never discover it. Jobs, Actor Reminders or Workflows assigned to the affected instance’s partitions are never triggered.

From a user’s perspective, workflows, scheduled jobs, and actor reminders randomly stop firing after a Scheduler pod restart. The issue is intermittent and depends on the exact timing of the restart relative to other cluster activity.

Impact

Any Dapr deployment running the Scheduler in a multi-instance (HA) configuration is affected. The issue is triggered when a Scheduler instance joins or rejoins the cluster, causing a leadership quorum change (e.g., partition count changes from 2 to 3). The likelihood increases with the number of connected daprd sidecars, as more WatchHosts subscribers means the host broadcast takes longer, widening the race window.

When the bug is triggered:

  • The affected Scheduler instance owns partitions but cannot deliver jobs on them. All workflow activity timers, scheduled jobs, and actor reminders assigned to those partitions stop firing and are logged as UNDELIVERABLE.
  • Daprd sidecars that call WatchHosts may receive an incomplete host list (missing the affected instance), or may never receive a response at all — leaving the sidecar stuck on the scheduler-watch-hosts readiness gate indefinitely, preventing the application from becoming ready.
  • The remaining healthy instances cannot take over the affected instance’s partitions because its leadership key remains in etcd (the lease keep-alive continues independently). The cluster is stuck in a state where all instances are running but quorum can never converge on the correct partition count.

The only recovery is to restart all Scheduler pods simultaneously, which forces a fresh leadership election.

Root Cause

When the Scheduler’s internal cron module reaches leadership quorum after a partition change, it calls runEngine to start the cron engine. Before starting the engine, runEngine sends the updated cluster host addresses to an internal unbuffered Go channel (WatchLeadership) so that the WatchHosts API can broadcast them to connected sidecars.

If the channel consumer is busy (for example, broadcasting a previous host update to many WatchHosts subscribers), the send blocks. Meanwhile, if another Scheduler instance joins and causes a second quorum change, the elected context is cancelled while the send is still blocked.

Because the cron module has exited, it never calls Reelect to update its leadership key with the new partition total. The other Scheduler instances see this stale key and cannot reach quorum agreement, preventing the entire cluster from converging.

Solution

The internal channel consumer in the Scheduler’s cron wrapper has been replaced with a non-blocking event loop (events/loop). The loop’s Enqueue method never blocks. If the current segment is full, it allocates a new one. This means the channel send from the cron library always completes immediately, regardless of how busy the consumer is with broadcasting.

Since the send no longer blocks, the context cancellation race that caused the silent exit can no longer occur. The cron loop continues to call Reelect after each quorum change, leadership keys are updated with the correct partition total, and all instances converge normally.

Windows sidecar container fails to start on AKS due to missing OSVersion in image manifest

Problem

Starting with Dapr v1.16.9, the Dapr sidecar container (daprd) fails to start on AKS Windows nodes with the error:

hcs::CreateComputeSystem daprd: The container operating system does not match the host operating system.

Impact

All Windows-based Dapr sidecar deployments on AKS are broken from v1.16.9 onward. The daprd container enters CrashLoopBackOff and never starts. Linux deployments are unaffected.

Root Cause

In v1.16.9, the docker-manifest-create target in docker/docker.mk was changed from using docker manifest create / docker manifest push to docker buildx imagetools create.

The docker manifest commands automatically read os.version from each source image’s config and include it in the manifest list entries. The docker buildx imagetools create command does not carry the os.version field through to the manifest list.

Without os.version on the Windows manifest entries, the Windows container runtime cannot distinguish between the two windows/amd64 images (Server 2019 and Server 2022) and pulls the wrong variant for the host OS build.

Solution

Reverted the docker-manifest-create target to use docker manifest create and docker manifest push, restoring the os.version field in the manifest list entries for Windows images.

下载链接