发布日期: 2025-09-25
版本号: v1.10.0

Longhorn v1.10.0 是一个重要的版本，主要专注于提升稳定性、性能和用户体验。该版本对核心功能进行了显著增强，特别是 V2 数据引擎，并简化了配置以便于管理。关键亮点包括 V2 数据引擎的改进、增强的系统弹性、简化的配置以及更好的可观测性。需要特别注意的是，v1.10.0 版本的 longhorn-manager:v1.10.0 镜像存在一个回归问题，可能导致管理器崩溃和升级后无法部署新的 share-manager Pod，官方提供了热修复镜像 longhorn-manager:v1.10.0-hotfix-1 以供替换。从 v1.9 升级到 v1.10 前，强烈建议手动执行自定义资源迁移，将所有 v1beta1 版本的资源迁移到 v1beta2 版本，并验证 CRD 存储版本中仅包含 v1beta2，以避免兼容性问题。本次发布引入了多项新特性，包括：V2 数据引擎新增中断模式、卷和快照克隆、副本重建 QoS 控制、卷扩容，以及无需大页内存的支持；V1 数据引擎新增 IPv6 支持；统一了 V1 和 V2 引擎的全局设置格式；支持 CSIStorageCapacity 实现感知存储容量的 Pod 调度；可配置备份块大小；以及在界面中展示卷的挂载摘要。此外，版本还包含大量的功能改进、错误修复和文档更新。安装或升级前需确保 Kubernetes 集群版本不低于 v1.25。

更新内容 （中文）

Longhorn v1.10.0 发布说明

Longhorn v1.10.0 是一个主要版本发布，专注于提高稳定性、性能和整体用户体验。此版本对其核心功能进行了重大增强，包括 V2 数据引擎，并简化了配置以便于管理。

主要亮点包括 V2 数据引擎的改进、增强的弹性、简化的配置以及更好的可观测性。

我们欢迎反馈和贡献，以帮助持续改进 Longhorn。

有关 Longhorn 版本的术语和上下文，请参阅 发布版本。

[!WARNING]

热修复

longhorn-manager:v1.10.0 镜像受新的 share-manager Pod 退避逻辑引入的 回归问题 影响。此 bug 可能导致 longhorn-manager 中出现空指针解引用 panic，导致重复崩溃并在升级后无法部署新的 share-manager Pod。为缓解此问题，请将 longhorn-manager:v1.10.0 替换为热修复镜像 longhorn-manager:v1.10.0-hotfix-1。

您可以按照以下步骤应用更新：

  1. 禁用升级版本检查

    • Helm 用户：在 values.yaml 文件中将 upgradeVersionCheck 设置为 false。
    • Manifest 用户：从部署清单中移除 --upgrade-version-check 标志。

  2. 更新 longhorn-manager 镜像

    • 在相应文件中将镜像标签从 v1.10.0 更改为 v1.10.0-hotfix-1：
      • Helm 用户：更新 values.yaml
      • Manifest 用户：直接更新部署清单。

  3. 继续升级

  • 使用您的标准 Helm 升级命令应用更改，或重新应用更新后的清单。

升级

如果您的 Longhorn 集群最初是使用早于 v1.3.0 的版本部署的，则自定义资源 (CR) 是使用 v1beta1 API 创建的。虽然从 Longhorn v1.8 到 v1.9 的升级会自动将所有 CR 迁移到新的 v1beta2 版本，但在从 Longhorn v1.9 升级到 v1.10 之前，强烈建议手动迁移 CR。

某些操作（如 etcd 或 CRD 恢复）可能会遗留 v1beta1 数据。手动迁移您的 CR 可确保所有 Longhorn 数据都正确更新到 v1beta2 API，防止与新 Longhorn 版本发生潜在的兼容性问题和意外行为。

手动迁移后，请验证 v1beta1 已从 CRD 存储版本中移除，以确保完成并成功升级。

更多详情，请参阅 Kubernetes 官方文档关于 CRD 存储版本 和 Issue #11886。

Longhorn v1.10 升级前的迁移要求

在从 Longhorn v1.9 升级到 v1.10 之前，请执行以下手动 CRD 存储版本迁移。

注意：如果您的 Longhorn 安装使用除 longhorn-system 之外的命名空间，请在整个命令中将 longhorn-system 替换为您自定义的命名空间。

# 临时禁用 CR 验证 webhook，以允许更新只读设置 CR。
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= map(select(. != "UPDATE")) else . end)')"

# 迁移曾存储过 v1beta1 资源的 CRD
migration_time="$(date +%Y-%m-%dT%H:%M:%S)"
crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name'))
for crd in "${crds[@]}"; do
  echo "Migrating ${crd} ..."
  for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do
    # 附加额外注释以触发在最新存储版本中更新 v1beta1 资源。
    kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}'
 done
 # 清理 CRD 状态中的存储版本
 kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status
done

# 重新启用 CR 验证 webhook。
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= (. + ["UPDATE"] | unique) else . end)')"

迁移验证

运行脚本后，请使用此命令验证 CRD 存储版本：

kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"

“}{end}'


关键是在继续 v1.10 升级之前，所有 Longhorn CRD 的 `storedVersions` 中必须仅列出 `"v1beta2"`（即必须完全不存在 `"v1beta1"`）。

成功输出的示例：

backingimagedatasources.longhorn.io: [“v1beta2”] backingimagemanagers.longhorn.io: [“v1beta2”] backingimages.longhorn.io: [“v1beta2”] backupbackingimages.longhorn.io: [“v1beta2”] backups.longhorn.io: [“v1beta2”] backuptargets.longhorn.io: [“v1beta2”] backupvolumes.longhorn.io: [“v1beta2”] engineimages.longhorn.io: [“v1beta2”] engines.longhorn.io: [“v1beta2”] instancemanagers.longhorn.io: [“v1beta2”] nodes.longhorn.io: [“v1beta2”] orphans.longhorn.io: [“v1beta2”] recurringjobs.longhorn.io: [“v1beta2”] replicas.longhorn.io: [“v1beta2”] settings.longhorn.io: [“v1beta2”] sharemanagers.longhorn.io: [“v1beta2”] snapshots.longhorn.io: [“v1beta2”] supportbundles.longhorn.io: [“v1beta2”] systembackups.longhorn.io: [“v1beta2”] systemrestores.longhorn.io: [“v1beta2”] volumeattachments.longhorn.io: [“v1beta2”] volumes.longhorn.io: [“v1beta2”]


完成这些步骤后，Longhorn 升级到 v1.10 现在应该可以顺利进行。

#### 升级到 Longhorn v1.10 期间 CRD 升级故障的故障排除

如果您未应用所需的预升级迁移步骤，且 CR 未完全迁移到 `v1beta2`，则 `longhorn-manager` Pod 可能无法正常工作。此问题的常见错误消息为：

Upgrade failed: cannot patch “backingimagedatasources.longhorn.io” with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io “backingimagedatasources.longhorn.io” is invalid: status.storedVersions[0]: Invalid value: “v1beta1”: missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions


要修复此问题，您必须执行**强制降级**，回退到失败升级尝试之前运行的**确切 Longhorn v1.9.x 版本**。

##### 降级步骤（kubectl 安装）

如果 Longhorn 是使用 `kubectl` 安装的，则在降级之前必须修补 `current-longhorn-version` 设置。在以下命令中，将 `v1.9.x` 替换为升级前的原始版本。

```bash
# 附加注释以允许修补 current-longhorn-version。
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}'
# 临时覆盖当前版本以允许旧版本安装
# 将值 `"v1.9.x"` 替换为升级前的原始版本。
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'

修改 current-longhorn-version 后，您可以继续降级到原始的 Longhorn v1.9.x 部署。

降级步骤（Helm 安装）

如果 Longhorn 是使用 Helm 安装的，则通过禁用 preUpgradeChecker.upgradeVersionCheck 标志可以允许降级。

降级后

降级完成且 Longhorn 系统在 v1.9.x 版本上稳定后，您必须立即按照 Longhorn v1.10 升级前的迁移要求 中概述的步骤操作。此步骤对于在再次尝试 Longhorn v1.10 升级之前，将所有剩余的 v1beta1 CR 迁移到 v1beta2 至关重要。

已移除

longhorn.io/v1beta1 API

v1beta1 Longhorn API 版本已被移除。

详情请参阅 GitHub Issue #10249。

replica.status.evictionRequested 字段

已弃用的 replica.status.evictionRequested 字段已被移除。

详情请参阅 GitHub Issue #7022。

主要亮点

新的 V2 数据引擎功能

中断模式支持

V2 数据引擎新增了中断模式，以帮助降低 CPU 使用率。此功能对于具有空闲或低 I/O 工作负载的集群特别有益，因为节省 CPU 资源比最小化延迟更重要。

虽然中断模式降低了 CPU 消耗，但与轮询模式相比，它可能会引入略高的 I/O 延迟。此外，当前实现采用混合方法，即使在启用中断时，仍会产生最小、恒定的 CPU 负载。

[!NOTE] 限制： 仅支持 AIO 磁盘。

详情请参阅 中断模式 和 GitHub Issue#9834。

卷和快照克隆

V2 卷现在支持两种类型的克隆：

  • 全拷贝克隆：创建一个新的 PVC，其中包含源数据的完整、独立副本，提供完全隔离。
  • 链接克隆（快速/智能克隆）：创建一个与源卷共享数据块的 PVC，实现近乎即时的创建。非常适合临时工作负载、备份或测试。链接克隆轻量、快速，并减少存储开销。

详情请参阅 卷克隆支持 和 GitHub Issue#7794。

副本重建 QoS

为 V2 卷副本重建提供服务质量 (QoS) 控制。您可以全局或按卷配置带宽限制，以防止源节点和目标节点的存储吞吐量过载。

详情请参阅 副本重建 QoS 和 GitHub Issue#10770。

卷扩展

Longhorn 现在支持 V2 数据引擎卷的扩展。您可以通过 UI 或修改 PVC 清单来扩展卷。

详情请参阅 V2 卷扩展 和 GitHub Issue#8022。

支持在没有大页的情况下运行

这减轻了低规格节点上的内存压力，并提高了部署灵活性。与在有大页的情况下运行相比，性能可能会降低。

详情请参阅 GitHub Issue#7066。

新的 V1 数据引擎功能

IPv6 支持

V1 卷现在支持单栈 IPv6 Kubernetes 集群。

警告： 此版本不支持双栈 Kubernetes 集群和 V2 卷。

详情请参阅 GitHub Issue #2259。

统一的全局设置

为了简化管理，Longhorn 设置现在在 V1 和 V2 数据引擎之间统一，使用新的、更灵活的 JSON 格式。

  • 单一值（适用于所有数据引擎）：非 JSON 字符串（例如，1024）。
  • 特定于数据引擎：JSON 对象（例如，{"v1": "value1", "v2": "value2"}）。
  • 仅限 V1：带有 v1 键的 JSON 对象（例如，{"v1":"value1"}）。
  • 仅限 V2：带有 v2 键的 JSON 对象（例如，{"v2":"value1"}）。

详情请参阅 Longhorn 设置 和 GitHub Issue#10926。

使用 CSIStorageCapacity 进行 Pod 调度

Longhorn 现在支持 CSIStorageCapacity，允许 Kubernetes 在使用带有 WaitForFirstConsumer 的 StorageClass 调度 Pod 之前验证节点存储。这减少了调度错误并提高了可靠性。

详情请参阅 GitHub Issue #10685。

可配置的备份块大小

创建卷时现在可以配置备份块大小，以优化性能和效率。

详情请参阅 创建 Longhorn 卷 和 GitHub Issue#5215。

卷附加摘要

UI 现在在每个卷页面上显示附加票证的摘要，以提高可见性。

详情请参阅 GitHub Issue #11400。

安装

[!IMPORTANT] 在安装 Longhorn v1.10.0 之前，请确保您的集群运行的是 Kubernetes v1.25 或更高版本。

您可以使用多种工具安装 Longhorn，包括 Rancher、Kubectl 和 Helm。有关安装方法和要求的更多信息，请参阅 Longhorn 文档中的 快速安装。

升级

[!IMPORTANT] 在从 Longhorn v1.9.x 升级到 v1.10.0 之前，请确保您的集群运行的是 Kubernetes v1.25 或更高版本。

Longhorn 仅允许从受支持的版本进行升级。有关升级路径和过程的更多信息，请参阅 Longhorn 文档中的 升级。

发布后已知问题

有关此版本后发现的问题的信息，请参阅 Release-Known-Issues。

亮点

  • [FEATURE] V2 卷支持克隆 7794 - @yangchiu @PhanLe1010
  • [FEATURE] V2 支持卷扩展 8022 - @davidcheng0922 @chriscchien
  • [UI][FEATURE] V2 卷支持克隆 11736 - @yangchiu @houhoucoop
  • [FEATURE] V2 卷支持中断模式 9834 - @yangchiu @c3y1huang
  • [FEATURE] 支持无大页的 V2 卷 7066 - @derekbit @chriscchien
  • [FEATURE] 可配置的备份块大小 5215 - @COLDTURNIP @yangchiu
  • [UI][FEATURE] 可配置的备份块大小 11586 -
  • [FEATURE] 添加 QoS 支持以限制副本重建负载 10770 - @hookak @roger-ryao
  • [FEATURE] V2 的卷粒度设置以匹配 V1 数据引擎 10926 - @derekbit @chriscchien
  • [IMPROVEMENT] 支持 Longhorn CSI 驱动程序中的 CSIStorageCapacity 以实现容量感知的 Pod 调度 10685 - @bachmanity1 @roger-ryao
  • [FEATURE] V1 数据引擎的 IPV6 2259 - @yangchiu @c3y1huang
  • [FEATURE] 使用增量快照的增量副本重建：控制平面和数据平面 10037 - @shuo-wu @roger-ryao
  • [FEATURE] 在 Longhorn v1.10 中移除 v1beta1 API CRD 10249 - @derekbit @roger-ryao

功能

  • [FEATURE] 在大页更新后通过 longhornctl 添加选项重启 kubelet 11241 - @chriscchien @bachmanity1
  • [UI][FEATURE] 可配置的备份块大小 11351 - @yangchiu @houhoucoop
  • [UI][FEATURE] 在单个卷的概览页面中显示附加票证的摘要 11401 - @yangchiu @houhoucoop
  • [UI][FEATURE] 添加 QoS 支持以限制副本重建负载 11306 - @davidcheng0922 @houhoucoop @roger-ryao
  • [UI][FEATURE] V2 的卷粒度设置以匹配 V1 数据引擎 11354 - @chriscchien @houhoucoop
  • [FEATURE] 在单个卷的概览页面中显示附加票证的摘要 11400 - @yangchiu @davidcheng0922
  • [FEATURE] 允许 longhorn 在启用 当卷意外分离时自动删除工作负载 Pod 功能时，使用自定义控制器重启 Pod 8353 - @derekbit @roger-ryao
  • [FEATURE] 用于覆盖容器镜像注册表的标准化方式 11064 - @marcosbc @yangchiu @roger-ryao
  • [FEATURE] 用于指定镜像拉取密钥的标准化方式 11062 - @marcosbc @chriscchien

改进

  • [IMPROVEMENT] 添加 Longhorn 安装变体的使用指标 11792 - @derekbit
  • [IMPROVEMENT] 允许为 V1 和 V2 数据引擎应用不同的快照校验和相关设置值 11537 - @chriscchien @nzhan126
  • [IMPROVEMENT] 使 longhornctl 可在气隙环境中使用 11291 - @chriscchien @bachmanity1
  • [IMPROVEMENT] SAST longhorn-manager 中 controller/volume_controller.go 可能的空指针解引用 11780 - @c3y1huang
  • [IMPROVEMENT] 从由设置 log-path 定义的主机目录收集日志 11522 - @c3y1huang @roger-ryao
  • [IMPROVEMENT] 增强离线重建，支持资源感知和重试退避 11270 - @mantissahz @chriscchien
  • [IMPROVEMENT] 在支持捆绑包中收集挂载表、进程状态和进程表 8397 - @mantissahz @chriscchien
  • [IMPROVEMENT] 卷附加应自动排除设置了 disable-v2-data-engine="true" 的节点 11695 - @derekbit @chriscchien
  • [IMPROVEMENT] 为设置引入 系统信息 类别 11656 - @derekbit @roger-ryao
  • [IMPROVEMENT] RBAC 权限 11345 - @davidcheng0922 @chriscchien
  • [IMPROVEMENT] 将 Longhorn Pod 日志精度提高到纳秒 11596 - @derekbit @roger-ryao
  • [IMPROVEMENT] 更新 V2 数据引擎的验证逻辑 11600 - @derekbit @chriscchien
  • [IMPROVEMENT] 改进 longhorn-engine、tgt 和 liblonghorn 的日志消息以用于故障排除 11545 - @yangchiu @derekbit
  • [IMPROVEMENT] 重命名后台镜像管理器以减少 CR 名称冲突的概率 11455 - @COLDTURNIP @chriscchien
  • [IMPROVEMENT] 移除 longhorn/longhorn 中过时的先决条件安装脚本 11430 - @yangchiu @roger-ryao @sushant-suse
  • [UI][IMPROVEMENT] 为强制分离操作添加 UI 警告，以防止 Kubernetes 和 Longhorn VolumeAttachments 不同步 9944 - @yangchiu @houhoucoop
  • [IMPROVEMENT] 为 longhornctl 添加 node-selector 选项，以选择运行 DaemonSet 的节点 11213 - @yangchiu @bachmanity1
  • [IMPROVEMENT] 改进卷 Scheduled 条件消息 11460 - @yangchiu @derekbit @chriscchien
  • [IMPROVEMENT] 启动新的机制以收集实例管理器日志 5948 - @yangchiu @derekbit
  • [IMPROVEMENT] 调整后台镜像下载的硬编码超时限制 11309 - @COLDTURNIP @roger-ryao
  • [IMPROVEMENT] 使实例管理器 Pod 的存活性探测参数可配置 10788 - @yangchiu @derekbit
  • [IMPROVEMENT] 增强 Longhorn CLI 的菜单描述 8998 - @roger-ryao @sushant-suse
  • [IMPROVEMENT] 改进 longhorn-engine 控制器日志消息 11507 - @derekbit @chriscchien
  • [IMPROVEMENT] 添加注释以解释实例管理器控制器中 isSettingDataEngineSynced 的作用 11321 - @mantissahz
  • [IMPROVEMENT] 冗长且误导性的日志消息 Deleting orphans on evicted node ... 11500 - @yangchiu @derekbit
  • [IMPROVEMENT] 拒绝 V1 数据引擎的 volume.spec.replicaRebuildingBandwidthLimit 更新 11497 - @derekbit @roger-ryao
  • [IMPROVEMENT] 如果重建无法启动，则分离离线重建卷 11274 - @mantissahz
  • [IMPROVEMENT] 后台镜像处理节点磁盘删除事件 10983 - @COLDTURNIP @chriscchien
  • [IMPROVEMENT] 将 RebuildingMbytesPerSecond 重命名为 ReplicaRebuildBandwidthLimit 11403 - @derekbit @roger-ryao
  • [IMPROVEMENT] 使同步代理可进行性能分析 11386 - @COLDTURNIP @yangchiu
  • [IMPROVEMENT] 添加 Longhorn 磁盘 I/O 的性能指标 11223 - @hookak @DamiaSan
  • [IMPROVEMENT] 使 CLI 预检检查对后续检查是非阻塞的 9877 - @davidcheng0922 @DamiaSan
  • [IMPROVEMENT] 为 cli 预检检查添加命名空间参数/参数 9749 - @davidcheng0922 @DamiaSan
  • [IMPROVEMENT] 孤立数据 不应位于设置下 10383 - @houhoucoop @DamiaSan @sushant-suse
  • [IMPROVEMENT] 在 longhorn-ui 中升级 Node v20 11315 - @chriscchien @houhoucoop
  • [IMPROVEMENT] 来自 /v1/backuptargets 的有用错误消息未在 UI 中显示 10428 - @houhoucoop @DamiaSan
  • [IMPROVEMENT] 在创建备份、备份后台镜像和系统备份之前检查备份目标是否可用 10085 - @yangchiu @nzhan126
  • [IMPROVEMENT] 资源受限场景中实例管理器 Pod 重新创建的退避重试间隔 10263 - @yangchiu @bachmanity1
  • [IMPROVEMENT] 记录 webhook 拒绝迁移附加票证时的详细信息 11150 - @COLDTURNIP @roger-ryao
  • [IMPROVEMENT] 处理包含混合无效条件的凭证密钥 8537 - @yangchiu @nzhan126
  • [IMPROVEMENT] 添加为 guaranteed-instance-manager-cpu 和 node.spec.instanceManagerCPURequest 设置浮点值的可能性 11179 - @yangchiu @gigabyte132
  • [IMPROVEMENT] 移除 CRD 的 preserveUnknownFields: false 补丁 11263 - @derekbit @chriscchien
  • [IMPROVEMENT] 当本地性为 尽力而为 时，至少在本地调度一个副本 11007 - @chriscchien @bachmanity1
  • [IMPROVEMENT] 改进磁盘空间不可调度条件消息 10436 - @yangchiu @davidcheng0922
  • [IMPROVEMENT] 改进引擎镜像检查的条件消息 9845 - @derekbit @chriscchien
  • [IMPROVEMENT] 改进在同一备份目标上检测到同一卷的多个备份卷时的日志记录 11152 - @PhanLe1010 @chriscchien
  • [IMPROVEMENT] 为 longhorn/cli 实现文档验证 11229 - @derekbit
  • [IMPROVEMENT] 将验证从每个资源删除移至验证 webhook 5156 - @derekbit @roger-ryao
  • [IMPROVEMENT] 验证 node.longhorn.io 资源规范字段 11079 - @Felipalds @chriscchien
  • [IMPROVEMENT] 在 Longhorn Helm Chart 的 UI 服务中添加对自定义注释的支持 11031 - @josimar-silva @roger-ryao
  • [IMPROVEMENT] 为 longhorn-csi-plugin 在尝试联系 longhorn-manager Pod 时添加重试逻辑 9482 - @PhanLe1010 @roger-ryao

缺陷

  • [BUG] 无法从配置映射加载 DataEngineSpecific 布尔设置 11810 - @COLDTURNIP @roger-ryao
  • [BUG] V2 停止工作 - connectNVMfBdev() -> “code”: -95,“message”: “Operation not supported” (1.10.0-rc2) 11761 - @yangchiu @c3y1huang
  • [BUG] [UI] 克隆卷中数据引擎的默认值不一致 11802 - @houhoucoop @roger-ryao
  • [BUG] 如果某些节点标记为 disable-v2-data-engine=true，系统备份可能会卡在 CreatingVolumeBackups 状态 11774 - @mantissahz @roger-ryao
  • [BUG] 从快照创建时，卷调整大小期间可能存在数据损坏 11484 - @yangchiu @PhanLe1010
  • [BUG] 块磁盘重新添加后可能永远不会变为 可调度 11760 - @derekbit @chriscchien
  • [BUG] 节点重启后，V2 卷可能会卡在 正在分离/故障 状态 10112 - @yangchiu @shuo-wu
  • [BUG] 如果在 PVC 创建前后台镜像不存在，则无法使用后台镜像动态配置 V2 卷 11762 - @COLDTURNIP @yangchiu
  • [BUG] 源卷扩展并备份后，V2 DR 卷故障 11767 - @davidcheng0922 @roger-ryao
  • [BUG] longhorn manager 在安装期间崩溃 11743 - @derekbit @chriscchien
  • [BUG] 无法从远程备份存储同步现有备份 11758 - @yangchiu @mantissahz
  • [BUG] Longhorn PVC 处于挂起状态 11654 - @yangchiu @derekbit
  • [BUG] 写操作期间副本节点磁盘空间耗尽时，卷会发生故障 10718 - @yangchiu @mantissahz
  • [BUG] [v1.10.0-rc1] longhornctl trim volume 命令挂起 11704 - @davidcheng0922 @chriscchien
  • [BUG] longhornctl preflight install 应加载并检查 iscsi_tcp 内核模块 11706 - @mantissahz @chriscchien
  • [BUG] 由于 bdev_channel_destroy_resource() 断言失败，副本重建后 spdk_tgt 崩溃 11109 - @hookak @chriscchien
  • [BUG] 创建 V2 卷时无法设置副本亲和性 (test_soft_anti_affinity_scheduling_volume_enable) 11642 - @yangchiu @derekbit
  • [BUG] BackupBackingImage 可能从未就绪的 BackingImageManager 创建 11675 - @WebberHuang1118 @roger-ryao
  • [BUG] 使用 200 Mi 后台镜像创建 2 Gi 卷被拒绝，并提示“卷大小应大于后台镜像大小” 11362 - @COLDTURNIP @yangchiu
  • [BUG] V2 卷的副本自动平衡磁盘压力失败 10551 - @yangchiu @hookak
  • [BUG] 当 ownerID 分配给标记为 node.longhorn.io/disable-v2-data-engine: “true” 的节点时，备份卡住 11619 - @davidcheng0922 @roger-ryao
  • [BUG] 快速卷分离后引擎进程继续运行 11605 - @COLDTURNIP @yangchiu
  • [BUG] 节点 CR 中剩余的未知操作系统条件 11612 - @COLDTURNIP @roger-ryao
  • [BUG] Longhorn Manager 无限期地继续向实例管理器发送 V2 卷的副本删除请求 11553 - @yangchiu @shuo-wu
  • [BUG] 即使没有 V2 卷、后台镜像或孤立数据，也无法禁用 v2-data-engine 11330 - @shuo-wu @roger-ryao
  • [BUG] longhorn-manager 重复发出 No instance manager for node xxx for update instance state of orphan instance orphan-xxx.. 11597 - @COLDTURNIP @chriscchien
  • [BUG] 卷在变为只读时无法重新挂载 8572 - @derekbit @chriscchien
  • [BUG] 实时迁移意外终止时的悬挂卷状态 11479 - @PhanLe1010 @chriscchien
  • [BUG] S3 备份目标随机恢复为之前的值 9581 - @yangchiu @mantissahz
  • [BUG] Longhornctl / CLI - 未提供配置，尝试设置 KUBERNETES_MASTER 环境变量 10094 - @davidcheng0922 @chriscchien
  • [BUG] longhorn-images.txt 指定的 CSI 组件仓库标签未找到 11575 - @yangchiu @derekbit
  • [BUG] DR 卷的备份块大小应从最新备份设置 11580 - @COLDTURNIP @yangchiu
  • [BUG] longhornctl –enable-spdk 不支持 arm64 11551 - @yangchiu @davidcheng0922
  • [BUG] 集群脑裂期间可能会创建额外的无效 BackupVolumeCR 11154 - @mantissahz @roger-ryao
  • [BUG] 系统备份错误 11232 - @c3y1huang @roger-ryao
  • [BUG] 无法使用 UI 中的 Create Backup 图标创建备份 11451 - @yangchiu @mantissahz @houhoucoop
  • [BUG] 无法为 V2 卷上的旧快照创建备份：未能找到快照 lvol 范围 11461 - @c3y1huang @chriscchien
  • [BUG] 卸载失败，因为发现 backuptargets 残留 11486 - @COLDTURNIP @yangchiu
  • [BUG] 设置 V2 数据引擎时可能未满足大页要求，导致错误的 V2 实例管理器 CR 悬挂系统 11519 - @yangchiu @derekbit
  • [BUG] 无法在存储网络环境中设置备份目标：找不到节点的运行实例管理器 11478 - @yangchiu @derekbit
  • [BUG] V2 卷上的卷迁移负面测试用例失败 10800 - @shuo-wu @chriscchien
  • [BUG] V2 卷无法清理错误副本并重建新副本 - test_data_locality_basic 10335 - @shuo-wu @chriscchien
  • [BUG] 在 Talos 集群上自动检测 NVMe 驱动器的问题（vfio-pci 驱动程序而非预期的 vfio_pci） 11127 - @Hugome @roger-ryao
  • [BUG] 测试用例 test_running_volume_with_scheduling_failure 由于意外创建新副本而失败 11512 - @derekbit @chriscchien
  • [BUG] 回归测试用例由于无法清理虚拟备份而失败 11487 - @COLDTURNIP @yangchiu
  • [BUG][v1.9.0-rc1] 删除 V2 实例管理器后意外创建了孤立数据 10829 - @yangchiu @derekbit
  • [BUG] 重建后 V2 引擎在分离和附加状态之间循环 10396 - @shuo-wu @roger-ryao
  • [BUG] test_basic.py::test_backup_status_for_unavailable_replicas 失败 11416 - @derekbit @roger-ryao
  • [BUG] 由于过时的 SLES 仓库导致构建失败 11481 - @PhanLe1010
  • [BUG] 如果备份已存在，则无法设置 S3 备份目标 11337 - @mantissahz @chriscchien
  • [BUG][UI] 快照和备份 图表卡在加载中，控制台显示错误消息 10529 - @yangchiu @davidcheng0922
  • [BUG] Longhorn 文档中的备份 YAML 示例不工作 11216 - @roger-ryao @nzhan126
  • [BUG] 网络断开后，V2 卷工作负载 IO 可能收到 Bad message 错误 10113 - @yangchiu @shuo-wu
  • [BUG] 测试用例 Test Replica Auto Balance Node Least Effort 在 V2 卷上失败 10977 - @yangchiu @c3y1huang
  • [BUG] ublk 客户端包装错误时 IsJSONRPCRespErrorNoSuchDevice 失败 11361 - @yangchiu @davidcheng0922
  • [BUG] longhorn-manager 因 SIGSEGV: segmentation violation 而崩溃 11420 - @derekbit @chriscchien
  • [BUG] 测试用例 test_replica_auto_balance_node_least_effort 有时失败 11388 - @derekbit @chriscchien
  • [BUG] SPDK V2 引擎的卷间歇性卡在分离/故障状态 10724 - @DamiaSan
  • [BUG] 配置参数拼写错误：“offlineRelicaRebuilding” 应为 “offlineReplicaRebuilding” 11380 - @yangchiu @in-jun
  • [BUG][DOC] OpenShift 文档 11174 - @yangchiu @mlacko64
  • [BUG] 在单节点 Harvester 中，longhorn-manager 中记录了无尽的“无法调度副本” 3708 - @derekbit @chriscchien
  • [BUG] 由于不允许删除正在运行的 Longhorn 节点，卸载失败 11131 - @COLDTURNIP @roger-ryao
  • [BUG] 实例管理器 Pod 删除后，V2 引擎 I/O 阻塞超过 1-2 分钟 10167 - @c3y1huang @chriscchien
  • [BUG] 回归测试用例失败：预期卷已分离但实际已附加 11273 - @yangchiu @mantissahz
  • [BUG] 卷扩展失败并出现“不支持的磁盘加密格式 ext4” 11120 - @COLDTURNIP @mantissahz @roger-ryao
  • [BUG] longhorn-spdk-engine 重建单元测试可能卡住 2 分钟 11099 - @shuo-wu @roger-ryao
  • [BUG] 当 迁移节点宕机后确认迁移 时，V2 卷可能会卡在 分离/已分离 循环中 10157 - @yangchiu @PhanLe1010
  • [BUG] 在实例管理器 Pod 重启后触发竞争条件时，Longhorn 将不会重用失败的 V2 副本 11188 - @shuo-wu @roger-ryao
  • [BUG] 自动生成的 CLI 文档被 make 覆盖 11219 - @bachmanity1
  • [BUG] longhorn chart/values.yaml 中 remove-snapshots-during-filesystem-trim 的值不正确 11264 - @derekbit @chriscchien
  • [BUG][v1.9.0-rc1] V2 卷在节点宕机后未按预期重用失败副本 10828 - @yangchiu @shuo-wu
  • [BUG] CSI 插件重启触发了可迁移 RWX 卷工作负载的意外重启 11158 - @c3y1huang @roger-ryao
  • [BUG] 在浏览器 UI 中：卷 -> 克隆卷导致浏览器页面损坏 11165 - @houhoucoop @roger-ryao
  • [BUG] StorageClass 中的 “mkfsParams” 未传递给 share-manager 以进行文件系统格式化 11107 - @Florianisme @roger-ryao
  • [BUG] 测试用例 test_engine_image_not_fully_deployed_perform_volume_operations 失败：无法分离卷 10874 - @mantissahz @chriscchien
  • [BUG] 创建 support-bundle 导致 NPE panic 11169 - @c3y1huang @roger-ryao
  • [BUG] 由于 CMAKE 兼容性问题无法构建 Longhorn-Share-Manager 镜像 11159 - @derekbit @roger-ryao
  • [BUG] 由于不允许删除默认引擎镜像，卸载失败 11130 - @COLDTURNIP @chriscchien
  • [BUG] kubelet 重启后，V2 卷卡在降级状态并持续重建/删除副本 10107 - @shuo-wu
  • [BUG] SPDK API bdev_lvol_detach_parent 未按预期工作 11046 - @DamiaSan @roger-ryao
  • [BUG] 分配给默认组时循环作业失败 11016 - @c3y1huang @chriscchien
  • [BUG] V2 卷副本重建后数据校验和不匹配 10118 - @yangchiu @shuo-wu
  • [BUG] 由于无法更新设置，大部分回归测试用例失败 11042 - @yangchiu @mantissahz
  • [BUG] 节点驱逐后无法清理后台镜像卷副本 11053 - @COLDTURNIP @roger-ryao
  • [BUG] 驱逐节点期间后台镜像卷副本 NPE 崩溃 11034 - @COLDTURNIP @chriscchien
  • [BUG] 降级的 DR 卷在激活后保持待机和附加状态 2107 - @roger-ryao
  • [BUG] 如果只有一个正在重建的副本在运行，DR 卷会卡住 2753 - @c3y1huang @roger-ryao

稳定性

  • [DOC] 记录由 HDD 上 I/O 延迟引起的卷稳定性风险 11240 - @chriscchien @sushant-suse

其他

  • [DOC] 添加有关副本容错性的信息 11526 - @roger-ryao @sushant-suse
  • [TASK] 备份存储锁冲突错误消息的知识库 11293 - @yangchiu @pratikjagrut
  • [DOC] 记录副本重建机制及其限制 11119 - @mantissahz @chriscchien
  • [DOC] V2 引擎用法矛盾 11409 - @shuo-wu @roger-ryao
  • [DOC] Talos 新卷策略 11015 - @yangchiu @DrummyFloyd
  • [TASK] [pytest] 如果启用 RUN_V2_TEST，则自动添加 -m v2_volume_test 11376 - @chriscchien
  • 修订关于节点空间的文档 3021 - @derekbit @chriscchien
  • [DOC] XFS 文件系统挂载失败的故障排除知识库 11214 - @derekbit @roger-ryao
  • [DOC] 解释 Longhorn VolumeAttachment 操作和行为 11142 - @derekbit @roger-ryao
  • [DOC] 更新网站上的损坏链接 11288 - @yangchiu @sushant-suse
  • [DOC] 在 chart README 中阐明 privateRegistry.createSecret 和 registrySecret 的用法 11251 - @chriscchien
  • [DOC] 知识库：VM 升级期间完成卷迁移失败 11149 - @COLDTURNIP @chriscchien
  • [DOC] 详细说明如何使用 kubectl 启用 DR 卷 10958 - @derekbit @chriscchien
  • [DOC] 从气隙安装指南中移除 defaultSettings.registrySecret 引用 11237 - @chriscchien
  • [TASK] 移除已弃用的 replica.status.evictionRequested 字段 7022 - @yangchiu @derekbit @roger-ryao
  • [TASK] 创建 longhorn/spdk longhorn-v25.05 分支 11048 - @derekbit @chriscchien
  • [TASK] 创建 libqcow 的专用存储库以提高可维护性和构建管理 10988 - @derekbit @chriscchien
  • [DOC] 在 1.9.0 上将使用已弃用的 crd 版本 v1beta1 的示例更新为 v1beta2 11019 - @falmar @roger-ryao
  • [TASK] UI 扩展迁移的概念验证 10516 - @houhoucoop
  • [TASK] 确保 support-bundle-kit 构建使用已供应商化的依赖项 11106 - @yangchiu @c3y1huang
  • [DOC] 修复文档中存在的损坏链接 11028 - @chriscchien @sushant-suse
  • [DOC] 更新网站主页以包含社区会议链接 10890 - @yangchiu @divya-mohan0209 @sushant-suse
  • [REFACTOR] 使用 go pkg 进行系统操作，而不是依赖于通过 shell 命令进行的外部系统调用 5193 - @c3y1huang

新贡献者

  • @Felipalds
  • @Florianisme
  • @Hugome
  • @divya-mohan0209
  • @falmar
  • @gigabyte132
  • @in-jun
  • @josimar-silva
  • @mlacko64
  • @pratikjagrut

贡献者

  • @COLDTURNIP
  • @DamiaSan
  • @DrummyFloyd
  • @PhanLe1010
  • @WebberHuang1118
  • @bachmanity1
  • @c3y1huang
  • @chriscchien
  • @davidcheng0922
  • @derekbit
  • @hookak
  • @houhoucoop
  • @innobead
  • @mantissahz
  • @marcosbc
  • @nzhan126
  • @roger-ryao
  • @shuo-wu
  • @sushant-suse
  • @yangchiu

更新内容 （原始）

Longhorn v1.10.0 Release Notes

Longhorn v1.10.0 is a major release focused on improving stability, performance, and the overall user experience. This version introduces significant enhancements to our core features, including the V2 Data Engine, and streamlines configuration for easier management.

The key highlights include improvements to the V2 Data Engine, enhanced resilience, simplified configuration, and better observability.

We welcome feedback and contributions to help continuously improve Longhorn.

For terminology and context on Longhorn releases, see Releases.

[!WARNING]

HotFix

The longhorn-manager:v1.10.0 image is affected by a regression issue introduced by the new share-manager pod backoff logic. This bug may cause a nil pointer dereference panic in the longhorn-manager, leading to repeated crashes and failure to deploy new share-manager pods after an upgrade. To mitigate this issue, replace longhorn-manager:v1.10.0 with the hotfixed image longhorn-manager:v1.10.0-hotfix-1.

You can apply the update by following these steps:

  1. Disable the upgrade version check

    • Helm users: Set upgradeVersionCheck to false in the values.yaml file.
    • Manifest users: Remove the --upgrade-version-check flag from the deployment manifest.

  2. Update the longhorn-manager image

    • Change the image tag from v1.10.0 to v1.10.0-hotfix-1 in the appropriate file:
      • For Helm: Update values.yaml
      • For manifests: Update the deployment manifest directly.

  3. Proceed with the upgrade

  • Apply the changes using your standard Helm upgrade command or reapply the updated manifest.

Upgrade

If your Longhorn cluster was initially deployed with a version earlier than v1.3.0, the Custom Resources (CRs) were created using the v1beta1 APIs. While the upgrade from Longhorn v1.8 to v1.9 automatically migrates all CRs to the new v1beta2 version, a manual CR migration is strongly advised before upgrading from Longhorn v1.9 to v1.10.

Certain operations, such as an etcd or CRD restore, may leave behind v1beta1 data. Manually migrating your CRs ensures that all Longhorn data is properly updated to the v1beta2 API, preventing potential compatibility issues and unexpected behavior with the new Longhorn version.

Following the manual migration, verify that v1beta1 has been removed from the CRD stored versions to ensure completion and a successful upgrade.

For more details, see Kubernetes official document for CRD storage version, and Issue #11886.

Migration Requirement Before Longhorn v1.10 Upgrade

Before upgrading from Longhorn v1.9 to v1.10, perform the following manual CRD storage version migration.

Note: If your Longhorn installation uses a namespace other than longhorn-system, replace longhorn-system with your custom namespace throughout the commands.

# Temporarily disable the CR validation webhook to allow updating read-only settings CRs.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= map(select(. != "UPDATE")) else . end)')"

# Migrate CRDs that ever stored v1beta1 resources
migration_time="$(date +%Y-%m-%dT%H:%M:%S)"
crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name'))
for crd in "${crds[@]}"; do
  echo "Migrating ${crd} ..."
  for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do
    # Attach additional annotations to trigger v1beta1 resource updating in the latest storage version.
    kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}'
 done
 # Clean up the stored version in CRD status
 kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status
done

# Re-enable the CR validation webhook.
kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \
  --type=merge \
  -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \
  jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then
    .operations |= (. + ["UPDATE"] | unique) else . end)')"

Migration Verification

After running the script, verify the CRD stored versions using this command:

kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"\n"}{end}'

Crucially, all Longhorn CRDs MUST have only "v1beta2" listed in storedVersions (i.e., "v1beta1" must be completely absent) before proceeding to the v1.10 upgrade.

Example of successful output:

backingimagedatasources.longhorn.io: ["v1beta2"]
backingimagemanagers.longhorn.io: ["v1beta2"]
backingimages.longhorn.io: ["v1beta2"]
backupbackingimages.longhorn.io: ["v1beta2"]
backups.longhorn.io: ["v1beta2"]
backuptargets.longhorn.io: ["v1beta2"]
backupvolumes.longhorn.io: ["v1beta2"]
engineimages.longhorn.io: ["v1beta2"]
engines.longhorn.io: ["v1beta2"]
instancemanagers.longhorn.io: ["v1beta2"]
nodes.longhorn.io: ["v1beta2"]
orphans.longhorn.io: ["v1beta2"]
recurringjobs.longhorn.io: ["v1beta2"]
replicas.longhorn.io: ["v1beta2"]
settings.longhorn.io: ["v1beta2"]
sharemanagers.longhorn.io: ["v1beta2"]
snapshots.longhorn.io: ["v1beta2"]
supportbundles.longhorn.io: ["v1beta2"]
systembackups.longhorn.io: ["v1beta2"]
systemrestores.longhorn.io: ["v1beta2"]
volumeattachments.longhorn.io: ["v1beta2"]
volumes.longhorn.io: ["v1beta2"]

With these steps completed, the Longhorn upgrade to v1.10 should now proceed without issues.

Troubleshooting CRD Upgrade Failures During Upgrade to Longhorn v1.10

If you did not apply the required pre-upgrade migration steps and the CRs are not fully migrated to v1beta2, the longhorn-manager Pods may fail to operate correctly. A common error message for this issue is:

Upgrade failed: cannot patch "backingimagedatasources.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "backingimagedatasources.longhorn.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions

To fix this issue, you must perform a forced downgrade back to the exact Longhorn v1.9.x version that was running before the failed upgrade attempt.

Downgrade Procedure (kubectl Installation)

If Longhorn was installed using kubectl, you must patch the current-longhorn-version setting before downgrading. Replace v1.9.x with the original version before upgrade in the following commands.

# Attaching annotation to allow patching current-longhorn-version.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}'
# Temporarily override current version to allow old version installation
# Replace the value `"v1.9.x" to the original version before upgrade.
kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'

After modifying current-longhorn-version, you can proceed to downgrade to the original Longhorn v1.9.x deployment.

Downgrade Procedure (Helm Installation)

If Longhorn was installed using Helm, the downgrade is allowed by disabling the preUpgradeChecker.upgradeVersionCheck flag.

Post-Downgrade

Once the downgrade is complete and the Longhorn system is stable on the v1.9.x version, you must immediately follow the steps outlined in the Migration Requirement Before Longhorn v1.10 Upgrade. This step is crucial to migrate all remaining v1beta1 CRs to v1beta2 before attempting the Longhorn v1.10 upgrade again.

Removal

longhorn.io/v1beta1 API

The v1beta1 Longhorn API version has been removed.

See GitHub Issue #10249 for details.

replica.status.evictionRequested Field

The deprecated replica.status.evictionRequested field has been removed.

See GitHub Issue #7022 for details.

Primary Highlights

New V2 Data Engine Features

Interrupt Mode Support

Interrupt mode has been added to the V2 Data Engine to help reduce CPU usage. This feature is especially beneficial for clusters with idle or low I/O workloads, where conserving CPU resources is more important than minimizing latency.

While interrupt mode lowers CPU consumption, it may introduce slightly higher I/O latency compared to polling mode. In addition, the current implementation uses a hybrid approach, which still incurs a minimal, constant CPU load even when interrupts are enabled.

[!NOTE] Limitation: Supports AIO disks only.

See Interrupt Mode and GitHub Issue#9834 for details.

Volume and Snapshot Cloning

V2 volumes now support two types of cloning:

  • Full-Copy Clone: Creates a new PVC with a complete, independent copy of the source data, providing full isolation.
  • Linked-Clone (Fast/Smart Clone): Creates a PVC that shares data blocks with the source volume for near-instant creation. Ideal for temporary workloads, backups, or testing. Linked-clones are lightweight, fast, and reduce storage overhead.

See Volume Clone Support and GitHub Issue#7794 for details.

Replica Rebuild QoS

Provides Quality of Service (QoS) control for V2 volume replica rebuilds. You can configure bandwidth limits globally or per volume to prevent storage throughput overload on source and destination nodes.

See Replica Rebuild QoS and GitHub Issue#10770 for details.

Volume Expansion

Longhorn now supports volume expansion for V2 Data Engine volumes. You can expand the volume through the UI or by modifying the PVC manifest.

See V2 Volume Expansion and GitHub Issue#8022 for details.

Support for Running Without Hugepages

This reduces memory pressure on low-spec nodes and increases deployment flexibility. Performance may be lower compared to running with Hugepages.

See GitHub Issue#7066 for details.

New V1 Data Engine Features

IPv6 Support

V1 volumes now support single-stack IPv6 Kubernetes clusters.

Warning: Dual-stack Kubernetes clusters and V2 volumes are not supported in this release.

See GitHub Issue #2259 for details.

Consolidated Global Settings

To simplify management, Longhorn settings are now unified across V1 and V2 Data Engines, using a new, more flexible JSON format.

  • Single value (applies to all Data Engines): Non-JSON string (e.g., 1024).
  • Data-engine-specific: JSON object (e.g., {"v1": "value1", "v2": "value2"})
  • V1-only: JSON object with v1 key (e.g., {"v1":"value1"}).
  • V2-only: JSON object with v2 key (e.g., {"v2":"value1"}).

See Longhorn Settings and GitHub Issue#10926 for details.

Pod Scheduling with CSIStorageCapacity

Longhorn now supports CSIStorageCapacity, allowing Kubernetes to verify node storage before scheduling pods using StorageClasses with WaitForFirstConsumer. This reduces scheduling errors and improves reliability.

See GitHub Issue #10685 for details.

Configurable Backup Block Size

Backup block size can now be configured when creating a volume to optimize performance and efficiency.

See Create Longhorn Volumes and GitHub Issue#5215 for details.

Volume Attachment Summary

The UI now shows a summary of attachment tickets on each volume page for improved visibility.

See GitHub Issue #11400 for details.

Installation

[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.10.0.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.9.x to v1.10.0.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Highlight

  • [FEATURE] V2 Volume Supports Cloning 7794 - @yangchiu @PhanLe1010
  • [FEATURE] v2 supports volume expansion 8022 - @davidcheng0922 @chriscchien
  • [UI][FEATURE] V2 Volume Supports Cloning 11736 - @yangchiu @houhoucoop
  • [FEATURE] V2 volumes support interrupt mode 9834 - @yangchiu @c3y1huang
  • [FEATURE] Support v2 volume without hugepage 7066 - @derekbit @chriscchien
  • [FEATURE] Configurable Backup Block Size 5215 - @COLDTURNIP @yangchiu
  • [UI][FEATURE] Configurable Backup Block Size 11586 -
  • [FEATURE] Add QoS support to limit replica rebuilding load 10770 - @hookak @roger-ryao
  • [FEATURE] Volume granular setting parity for V2 to match V1 data engine 10926 - @derekbit @chriscchien
  • [IMPROVEMENT] Support CSIStorageCapacity in Longhorn CSI driver to enable capacity-aware pod scheduling 10685 - @bachmanity1 @roger-ryao
  • [FEATURE] IPV6 for V1 Data Engine 2259 - @yangchiu @c3y1huang
  • [FEATURE] Delta Replica Rebuilding using Delta Snapshot: Control and Data Planes 10037 - @shuo-wu @roger-ryao
  • [FEATURE] Remove v1beta1 API CRD in Longhorn v1.10 10249 - @derekbit @roger-ryao

Feature

  • [FEATURE] Add option to restart kubelet through longhornctl after huge page update 11241 - @chriscchien @bachmanity1
  • [UI][FEATURE] Configurable Backup Block Size 11351 - @yangchiu @houhoucoop
  • [UI][FEATURE] Display a summary of the attachment tickets in an individual volume’s overview page 11401 - @yangchiu @houhoucoop
  • [UI][FEATURE] Add QoS support to limit replica rebuilding load 11306 - @davidcheng0922 @houhoucoop @roger-ryao
  • [UI][FEATURE] Volume granular setting parity for V2 to match V1 data engine 11354 - @chriscchien @houhoucoop
  • [FEATURE] Display a summary of the attachment tickets in an individual volume’s overview page 11400 - @yangchiu @davidcheng0922
  • [FEATURE] Allow longhorn to restart pods with custom controllers, while the Automatically Delete Workload Pod when The Volume Is Detached Unexpectedly feature is enabled 8353 - @derekbit @roger-ryao
  • [FEATURE] Standardized way to override container image registry 11064 - @marcosbc @yangchiu @roger-ryao
  • [FEATURE] Standardized way to specify image pull secrets 11062 - @marcosbc @chriscchien

Improvement

  • [IMPROVEMENT] Add usage metrics for Longhorn installation variant 11792 - @derekbit
  • [IMPROVEMENT] Allow applying different values of snapshot checksum related settings for v1 and v2 data engine 11537 - @chriscchien @nzhan126
  • [IMPROVEMENT] Make longhornctl usable in air-gapped environments 11291 - @chriscchien @bachmanity1
  • [IMPROVEMENT] SAST Potential dereference of the null pointer in controller/volume_controller.go in longhorn-manager 11780 - @c3y1huang
  • [IMPROVEMENT] Collect Logs from the Host Directory Defined by the Setting log-path 11522 - @c3y1huang @roger-ryao
  • [IMPROVEMENT] Enhance Offline Rebuilding with Resource Awareness and Retry Backoff 11270 - @mantissahz @chriscchien
  • [IMPROVEMENT] Collect mount table, process status and process table in support bundle 8397 - @mantissahz @chriscchien
  • [IMPROVEMENT] Volume attachment should automatically exclude nodes with disable-v2-data-engine="true" 11695 - @derekbit @chriscchien
  • [IMPROVEMENT] Introduce System Info Category for Settings 11656 - @derekbit @roger-ryao
  • [IMPROVEMENT] RBAC permissions 11345 - @davidcheng0922 @chriscchien
  • [IMPROVEMENT] Improve Longhorn Pods Logging Precision to Nanoseconds 11596 - @derekbit @roger-ryao
  • [IMPROVEMENT] Update validation logics for v2 data engine 11600 - @derekbit @chriscchien
  • [IMPROVEMENT] Improve log messages of longhorn-engine, tgt and liblonghorn for troubleshooting 11545 - @yangchiu @derekbit
  • [IMPROVEMENT] rename the backing image manager to reduce the probability of CR name collision 11455 - @COLDTURNIP @chriscchien
  • [IMPROVEMENT] Remove outdated prerequisite installation scripts in longhorn/longhorn 11430 - @yangchiu @roger-ryao @sushant-suse
  • [UI][IMPROVEMENT] Add UI Warning for Force-Detach Actions to Prevent Out-of-Sync Kubernetes and Longhorn VolumeAttachments 9944 - @yangchiu @houhoucoop
  • [IMPROVEMENT] Add node-selector option to longhornctl to select nodes on which to run DaemonSet 11213 - @yangchiu @bachmanity1
  • [IMPROVEMENT] Improve volume Scheduled condition message 11460 - @yangchiu @derekbit @chriscchien
  • [IMPROVEMENT] Launching a new mechanism to collect instance manager logs 5948 - @yangchiu @derekbit
  • [IMPROVEMENT] adjust the hardcoded timeout limitation for backing image downloading 11309 - @COLDTURNIP @roger-ryao
  • [IMPROVEMENT] Make liveness probe parameters of instance-manager pod configurable 10788 - @yangchiu @derekbit
  • [IMPROVEMENT] Enhance menu descriptions for Longhorn CLI 8998 - @roger-ryao @sushant-suse
  • [IMPROVEMENT] Improve longhorn-engine controller log messages 11507 - @derekbit @chriscchien
  • [IMPROVEMENT] Add a comment to explain what isSettingDataEngineSynced does in the instance manager controller. 11321 - @mantissahz
  • [IMPROVEMENT] Flooding and misleading log message Deleting orphans on evicted node ... 11500 - @yangchiu @derekbit
  • [IMPROVEMENT] Reject volume.spec.replicaRebuildingBandwidthLimit update for V1 Data Engine 11497 - @derekbit @roger-ryao
  • [IMPROVEMENT] Detach an offline rebuilding volume if rebuilding can not start 11274 - @mantissahz
  • [IMPROVEMENT] backing image handle node disk deleting events 10983 - @COLDTURNIP @chriscchien
  • [IMPROVEMENT] Rename RebuildingMbytesPerSecond to ReplicaRebuildBandwidthLimit 11403 - @derekbit @roger-ryao
  • [IMPROVEMENT] Make the sync agent profilable 11386 - @COLDTURNIP @yangchiu
  • [IMPROVEMENT] Add performance metrics for Longhorn disk I/O 11223 - @hookak @DamiaSan
  • [IMPROVEMENT] Make CLI preflight check non-blocking for subsequent checkups 9877 - @davidcheng0922 @DamiaSan
  • [IMPROVEMENT] Add namespace argument/parameter to cli pre-flight check 9749 - @davidcheng0922 @DamiaSan
  • [IMPROVEMENT] Orphaned Data should not be placed under Settings 10383 - @houhoucoop @DamiaSan @sushant-suse
  • [IMPROVEMENT] Upgrade Node v20 in longhorn-ui 11315 - @chriscchien @houhoucoop
  • [IMPROVEMENT] useful error message from /v1/backuptargets is not displayed in UI 10428 - @houhoucoop @DamiaSan
  • [IMPROVEMENT] Check if the backup target is available before creating a backup, backup backing image, and system backup 10085 - @yangchiu @nzhan126
  • [IMPROVEMENT] Backoff Retry Interval for Instance Manager Pod Re-creation in Resource Constraint Scenarios 10263 - @yangchiu @bachmanity1
  • [IMPROVEMENT] record the detail while webhook rejecting migration attachment tickets 11150 - @COLDTURNIP @roger-ryao
  • [IMPROVEMENT] Handle credential secret containing mixed invalid conditions 8537 - @yangchiu @nzhan126
  • [IMPROVEMENT] Add the possibility of setting floating point values for guaranteed-instance-manager-cpu and node.spec.instanceManagerCPURequest 11179 - @yangchiu @gigabyte132
  • [IMPROVEMENT] Remove the Patch preserveUnknownFields: false for CRDs 11263 - @derekbit @chriscchien
  • [IMPROVEMENT] Schedule at least one replica locally when locality is best-effort 11007 - @chriscchien @bachmanity1
  • [IMPROVEMENT] Improve the disk space un-schedulable condition message 10436 - @yangchiu @davidcheng0922
  • [IMPROVEMENT] Improve the condition message of engine image check 9845 - @derekbit @chriscchien
  • [IMPROVEMENT] Improve the logging when detecting multiple backup volumes of the same volume on the same backup target 11152 - @PhanLe1010 @chriscchien
  • [IMPROVEMENT] Implement Documentation Validation for longhorn/cli 11229 - @derekbit
  • [IMPROVEMENT] Move validation from each resource deletion to validation webhook 5156 - @derekbit @roger-ryao
  • [IMPROVEMENT] Validate node.longhorn.io resource spec fields 11079 - @Felipalds @chriscchien
  • [IMPROVEMENT] add support for custom annotations in the UI service on Longhorn Helm Chart 11031 - @josimar-silva @roger-ryao
  • [IMPROVEMENT] Adding retry logic for longhorn-csi-plugin when it trying to contact the longhorn-manager pods 9482 - @PhanLe1010 @roger-ryao

Bug

  • [BUG] failed to load DataEngineSpecific boolean setting from configmap 11810 - @COLDTURNIP @roger-ryao
  • [BUG] V2 stop working - connectNVMfBdev() -> “code”: -95,“message”: “Operation not supported” (1.10.0-rc2) 11761 - @yangchiu @c3y1huang
  • [BUG] [UI] Inconsistent Default Value for Data Engine in Clone Volume 11802 - @houhoucoop @roger-ryao
  • [BUG] System backup could get stuck in CreatingVolumeBackups if some nodes are labeled with disable-v2-data-engine=true 11774 - @mantissahz @roger-ryao
  • [BUG] Potential Data Corruption During Volume Resizing When Created from Snapshot 11484 - @yangchiu @PhanLe1010
  • [BUG] Block disk may never become Schedulable after re-adding 11760 - @derekbit @chriscchien
  • [BUG] v2 volume could get stuck in Detaching/Faulted state after nodes reboot 10112 - @yangchiu @shuo-wu
  • [BUG] Fail to dynamically provision a v2 volume with a backing image if the backing image doesn’t exist before PVC creation 11762 - @COLDTURNIP @yangchiu
  • [BUG] v2 DR volume faulted after origin volume expand and backuped 11767 - @davidcheng0922 @roger-ryao
  • [BUG] longhorn manager crash in installation 11743 - @derekbit @chriscchien
  • [BUG] Unable to sync existing backups from a remote backup store 11758 - @yangchiu @mantissahz
  • [BUG] Longhorn pvcs are in pending state. 11654 - @yangchiu @derekbit
  • [BUG] Volume becomes faulted when its replica node disks run out of space during a write operation 10718 - @yangchiu @mantissahz
  • [BUG] [v1.10.0-rc1] longhornctl trim volume command hangs 11704 - @davidcheng0922 @chriscchien
  • [BUG] longhornctl preflight install should load and check iscsi_tcp kernel module. 11706 - @mantissahz @chriscchien
  • [BUG] spdk_tgt crash after replica rebuilding due to bdev_channel_destroy_resource() assert failure 11109 - @hookak @chriscchien
  • [BUG] Unable to set replica affinity when creating a v2 volume (test_soft_anti_affinity_scheduling_volume_enable) 11642 - @yangchiu @derekbit
  • [BUG] BackupBackingImage may be created from an unready BackingImageManager 11675 - @WebberHuang1118 @roger-ryao
  • [BUG] Creating a 2 Gi volume with a 200 Mi backing image is rejected with “volume size should be larger than the backing image size” 11362 - @COLDTURNIP @yangchiu
  • [BUG] Replica auto balance disk in pressure fails on v2 volumes 10551 - @yangchiu @hookak
  • [BUG] Backup stuck when ownerID is assigned to a node with node.longhorn.io/disable-v2-data-engine: “true” 11619 - @davidcheng0922 @roger-ryao
  • [BUG] Engine process continues running after rapid volume detachment 11605 - @COLDTURNIP @yangchiu
  • [BUG] remaining unknown OS condition in node CR 11612 - @COLDTURNIP @roger-ryao
  • [BUG] Longhorn Manager continues to send replica deletion requests to the Instance Manager for the v2 volume indefinitely 11553 - @yangchiu @shuo-wu
  • [BUG] Unable to disable v2-data-engine even though there is no v2 volumes, backing images or orphaned data 11330 - @shuo-wu @roger-ryao
  • [BUG] longhorn-manager repeatedly emits No instance manager for node xxx for update instance state of orphan instance orphan-xxx.. 11597 - @COLDTURNIP @chriscchien
  • [BUG] Volumes fails to remount when they go read-only 8572 - @derekbit @chriscchien
  • [BUG] Dangling Volume State When Live Migration Terminates Unexpectedly 11479 - @PhanLe1010 @chriscchien
  • [BUG] S3 Backup target reverts randomly to previous value 9581 - @yangchiu @mantissahz
  • [BUG] Longhornctl / CLI - no configuration has been provided, try setting KUBERNETES_MASTER environment variable 10094 - @davidcheng0922 @chriscchien
  • [BUG] longhorn-images.txt specifies CSI component repo tags not found 11575 - @yangchiu @derekbit
  • [BUG] DR volume’s backup block size should be set from the latest backup 11580 - @COLDTURNIP @yangchiu
  • [BUG] longhornctl –enable-spdk doesn’t support arm64 11551 - @yangchiu @davidcheng0922
  • [BUG] extra invalid BackupVolumeCR may be created during cluster split-brain 11154 - @mantissahz @roger-ryao
  • [BUG] system backup error 11232 - @c3y1huang @roger-ryao
  • [BUG] Can not create backup using Create Backup icon in UI 11451 - @yangchiu @mantissahz @houhoucoop
  • [BUG] Unable to create backup for old snapshots on v2 volumes: failed to find snapshot lvol range 11461 - @c3y1huang @chriscchien
  • [BUG] Uninstall fail because find backuptargets remaining 11486 - @COLDTURNIP @yangchiu
  • [BUG] Setting v2 data engine can be enabled without fulfilling the hugepage requirement, causing error v2 instance manager CR dangling in the system 11519 - @yangchiu @derekbit
  • [BUG] Unable to setup backup target in storage network environment: cannot find a running instance manager for node 11478 - @yangchiu @derekbit
  • [BUG] Volume migration negative test cases fail on v2 volumes 10800 - @shuo-wu @chriscchien
  • [BUG] V2 volume fails to cleanup error replica and rebuild new one - test_data_locality_basic 10335 - @shuo-wu @chriscchien
  • [BUG] Issue auto detecting nvme drive on talos cluster (vfio-pci driver instead of expected vfio_pci) 11127 - @Hugome @roger-ryao
  • [BUG] Test case test_running_volume_with_scheduling_failure failed due to unexpected new replica created 11512 - @derekbit @chriscchien
  • [BUG] Regression test cases failed due to unable to clean up dummy backups 11487 - @COLDTURNIP @yangchiu
  • [BUG][v1.9.0-rc1] Unexpected orphaned data are created after v2 instance managers deleted 10829 - @yangchiu @derekbit
  • [BUG] v2 Engine loops in detaching and attaching state after rebuilding 10396 - @shuo-wu @roger-ryao
  • [BUG] test_basic.py::test_backup_status_for_unavailable_replicas is failed 11416 - @derekbit @roger-ryao
  • [BUG] Build fails due to outdated SLES repo 11481 - @PhanLe1010
  • [BUG] Unable to set up S3 backup target if backups already exist 11337 - @mantissahz @chriscchien
  • [BUG][UI] Snapshots and Backups graph is stuck loading and console shows error messages 10529 - @yangchiu @davidcheng0922
  • [BUG] The Backup YAML example in the Longhorn doc does not work 11216 - @roger-ryao @nzhan126
  • [BUG] v2 volume workload IO could get Bad message error after network disconnect 10113 - @yangchiu @shuo-wu
  • [BUG] Test case Test Replica Auto Balance Node Least Effort failed on v2 volume 10977 - @yangchiu @c3y1huang
  • [BUG] IsJSONRPCRespErrorNoSuchDevice fails on wrapped errors for ublk client 11361 - @yangchiu @davidcheng0922
  • [BUG] longhorn-manager is crashed due to SIGSEGV: segmentation violation 11420 - @derekbit @chriscchien
  • [BUG] Test Case test_replica_auto_balance_node_least_effort Is Sometimes Failed 11388 - @derekbit @chriscchien
  • [BUG] volume gets stuck at detaching/faulted state for spdk v2 engine intermittently 10724 - @DamiaSan
  • [BUG] Typo in configuration parameter: “offlineRelicaRebuilding” should be “offlineReplicaRebuilding” 11380 - @yangchiu @in-jun
  • [BUG][DOC] OpenShift documentation 11174 - @yangchiu @mlacko64
  • [BUG] In single node Harvester, endless “unable to schedule replica” is logged in longhorn-manager 3708 - @derekbit @chriscchien
  • [BUG] Uninstallation fail due to deleting the running Longhorn node is not allowed 11131 - @COLDTURNIP @roger-ryao
  • [BUG] Engine v2 I/O Blocked Over 1-2 Minutes After Instance Manager Pod Deletion 10167 - @c3y1huang @chriscchien
  • [BUG] Regression test cases failed: expecting volume to be detached but it’s attached 11273 - @yangchiu @mantissahz
  • [BUG] Volume expansion fails with “unsupported disk encryption format ext4” 11120 - @COLDTURNIP @mantissahz @roger-ryao
  • [BUG] longhorn-spdk-engine rebuilding unit tests may get stuck for 2 minutes 11099 - @shuo-wu @roger-ryao
  • [BUG] v2 volume could get stuck in detaching/detached loop when Migration Confirmation After Migration Node Down 10157 - @yangchiu @PhanLe1010
  • [BUG] Longhorn will not reuse the failed v2 replicas when a race condition is triggered after the instance manager pod restart 11188 - @shuo-wu @roger-ryao
  • [BUG] Auto-generated CLI document overwrite by make 11219 - @bachmanity1
  • [BUG] Incorrect value of remove-snapshots-during-filesystem-trim in longhorn chart/values.yaml 11264 - @derekbit @chriscchien
  • [BUG][v1.9.0-rc1] v2 volumes don’t reuse failed replicas as expected after a node goes down 10828 - @yangchiu @shuo-wu
  • [BUG] CSI Plugin restart triggers unintended restart of migratable RWX volume workloads 11158 - @c3y1huang @roger-ryao
  • [BUG] in the browser UI: Volume -> Clone Volume results in the broken browser page 11165 - @houhoucoop @roger-ryao
  • [BUG] “mkfsParams” in StorageClass are not passed to share-manager for filesystem formatting 11107 - @Florianisme @roger-ryao
  • [BUG] Test case test_engine_image_not_fully_deployed_perform_volume_operations failed: unable to detach a volume 10874 - @mantissahz @chriscchien
  • [BUG] Creating support-bundle panic NPE 11169 - @c3y1huang @roger-ryao
  • [BUG] Unable to Build Longhorn-Share-Manager Image Due to CMAKE Compatibility 11159 - @derekbit @roger-ryao
  • [BUG] Uninstallation fail due to deleting the default engine image is not allowed 11130 - @COLDTURNIP @chriscchien
  • [BUG] v2 volume gets stuck in degraded state and continuously rebuilds/deletes replicas after a kubelet restart 10107 - @shuo-wu
  • [BUG] SPDK API bdev_lvol_detach_parent does not work as expected 11046 - @DamiaSan @roger-ryao
  • [BUG] Recurring jobs fail when assigned to default group 11016 - @c3y1huang @chriscchien
  • [BUG] v2 volume data checksum mismatch after replica rebuilding 10118 - @yangchiu @shuo-wu
  • [BUG] Most of regression test cases are failing due to unable to update settings 11042 - @yangchiu @mantissahz
  • [BUG] unable to clean up the backing image volume replica after node eviction 11053 - @COLDTURNIP @roger-ryao
  • [BUG] backing image volume replica NPE crash during evicting node 11034 - @COLDTURNIP @chriscchien
  • [BUG] A degraded DR volume remains in standby and attached after activation. 2107 - @roger-ryao
  • [BUG] DR volume gets stuck if there is only a rebuilding replica running 2753 - @c3y1huang @roger-ryao

Stability

  • [DOC] Document Volume Stability Risks Caused by I/O Latency on HDDs 11240 - @chriscchien @sushant-suse

Misc

  • [DOC] Add Information on Replica Failure Tolerance 11526 - @roger-ryao @sushant-suse
  • [TASK] KB for backup store lock conflict error message 11293 - @yangchiu @pratikjagrut
  • [DOC] Document Replica Rebuilding Mechanisms and Their Limitations 11119 - @mantissahz @chriscchien
  • [DOC] V2 Engine usage contradiction 11409 - @shuo-wu @roger-ryao
  • [DOC] Talos new volumes strategy 11015 - @yangchiu @DrummyFloyd
  • [TASK] [pytest] automatically add -m v2_volume_test if RUN_V2_TEST enabled 11376 - @chriscchien
  • Revise the document about node space 3021 - @derekbit @chriscchien
  • [DOC] Troubleshooting KB for Mount Failure with XFS Filesystem 11214 - @derekbit @roger-ryao
  • [DOC] Explain Longhorn VolumeAttachment operation and behavior 11142 - @derekbit @roger-ryao
  • [DOC] Update Broken Links on Website 11288 - @yangchiu @sushant-suse
  • [DOC] Clarify privateRegistry.createSecret and registrySecret usage in chart README 11251 - @chriscchien
  • [DOC] KB: failed to complete volume migration during VM upgrade 11149 - @COLDTURNIP @chriscchien
  • [DOC] Elaborate how to enable DR volume using kubectl 10958 - @derekbit @chriscchien
  • [DOC] Remove defaultSettings.registrySecret reference from air gap installation guide 11237 - @chriscchien
  • [TASK] Remove deprecated replica.status.evictionRequested field 7022 - @yangchiu @derekbit @roger-ryao
  • [TASK] Create longhorn/spdk longhorn-v25.05 branch 11048 - @derekbit @chriscchien
  • [TASK] Create a Dedicated Repository for libqcow to Improve Maintainability and Build Management 10988 - @derekbit @chriscchien
  • [DOC] Update examples using deprecated crd version v1beta1 to v1beta2 on 1.9.0 11019 - @falmar @roger-ryao
  • [TASK] POC for ui-extension migration 10516 - @houhoucoop
  • [TASK] Ensure support-bundle-kit builds use vendored dependencies 11106 - @yangchiu @c3y1huang
  • [DOC] Fix the broken links present in documentation 11028 - @chriscchien @sushant-suse
  • [DOC] Update website front page to include community meeting links 10890 - @yangchiu @divya-mohan0209 @sushant-suse
  • [REFACTOR] Use go pkg for system operation instead of relying on external system call via shell command 5193 - @c3y1huang

New Contributors

  • @Felipalds
  • @Florianisme
  • @Hugome
  • @divya-mohan0209
  • @falmar
  • @gigabyte132
  • @in-jun
  • @josimar-silva
  • @mlacko64
  • @pratikjagrut

Contributors

  • @COLDTURNIP
  • @DamiaSan
  • @DrummyFloyd
  • @PhanLe1010
  • @WebberHuang1118
  • @bachmanity1
  • @c3y1huang
  • @chriscchien
  • @davidcheng0922
  • @derekbit
  • @hookak
  • @houhoucoop
  • @innobead
  • @mantissahz
  • @marcosbc
  • @nzhan126
  • @roger-ryao
  • @shuo-wu
  • @sushant-suse
  • @yangchiu

下载链接