longhorn v1.10.1 版本更新介绍
发布日期: 2025-11-12
版本号: v1.10.1
Longhorn v1.10.1 版本发布,主要包含稳定性、安全性和功能方面的改进与修复。该版本提供了一个热修复镜像(v1.10.1-hotfix-2)以解决 v1.10.1 版本中存在的几个关键问题,包括可能导致崩溃的空指针引用、块模式卷迁移卡住,以及V2卷克隆缓慢或无法挂载工作负载等问题。应用热修复需要禁用版本检查并更新镜像。
重要升级说明:从 v1.9 升级至 v1.10 前,强烈建议手动进行自定义资源(CR)存储版本的迁移,将遗留的 v1beta1 资源迁移至 v1beta2。升级前必须验证所有Longhorn CRD的 storedVersions 中已完全移除 v1beta1,否则可能导致升级失败。若升级失败,需先降级回 v1.9.x 版本并完成迁移后再重试。
本次版本还修复了多个重要问题:修复了V2数据引擎下实例管理器的Goroutine泄漏;修复了V2卷在中断模式下使用NVMe磁盘可能无法完成挂载的问题(v1.10.1起支持NVMe中断模式);修复了在仅启用IPv4的节点上无法部署Longhorn UI的问题;以及解决了读写多(RWX)卷的共享管理器内存使用过高的问题。此外,包含了多项错误修复和改进,涉及备份、克隆、容量展示、错误日志记录等多个方面。
安装和升级此版本要求集群运行 Kubernetes v1.25 或更高版本。
更新内容 (中文)
Longhorn v1.10.1 发布说明
Longhorn 1.10.1 引入了若干改进和错误修复,旨在提升系统质量、韧性、稳定性和安全性。
我们欢迎反馈和贡献,以帮助持续改进 Longhorn。
有关 Longhorn 版本的术语和背景,请参阅 版本。
[!WARNING]
热修复
longhorn-manager:v1.10.1镜像受以下问题影响:
- 回归问题:
- \[BUG\] 无效内存地址或空指针解引用,在某些条件下可能触发空指针解引用,导致意外崩溃。
- \[BUG\] 块模式卷迁移卡住,可能导致块模式卷迁移无限期卡住。
- 首日问题:
- V2 卷克隆:
- \[BUG\] 由 VolumeAttachment Webhook 导致的 V2 卷 CSI 克隆缓慢,可能导致 V2 卷克隆操作出现显著延迟。
- \[BUG\] [Bug] 克隆的卷无法附加到工作负载,阻止克隆的卷附加到工作负载。
- 副本自动平衡:
- \[BUG\] 已停止卷的副本自动平衡磁盘压力阈值停滞,可能导致在磁盘压力条件下停止卷时,副本自动平衡功能停滞。
为缓解这些问题,请将
longhorn-manager:v1.10.1替换为热修复镜像longhorn-manager:v1.10.1-hotfix-2。按照以下步骤应用更新:
禁用升级版本检查
- Helm 用户:在
values.yaml文件中将upgradeVersionCheck设置为false。- Manifest 用户:从部署清单中移除
--upgrade-version-check标志。更新
longhorn-manager镜像
- 在相应文件中将镜像标签从
v1.10.1更改为v1.10.1-hotfix-2:
- Helm 用户:更新
values.yaml- Manifest 用户:直接更新部署清单。
继续升级
- 使用标准 Helm 升级命令应用更改或重新应用更新后的清单。
升级
如果您的 Longhorn 集群最初部署的版本早于 v1.3.0,则自定义资源是使用
v1beta1API 创建的。虽然从 Longhorn v1.8 升级到 v1.9 会自动将所有 CR 迁移到新的v1beta2版本,但强烈建议在从 Longhornv1.9升级到v1.10之前手动迁移 CR。某些操作(如
etcd或 CRD 恢复)可能会留下v1beta1数据。手动迁移 CR 可确保所有 Longhorn 数据正确更新到v1beta2API,防止与新 Longhorn 版本出现潜在的兼容性问题和意外行为。手动迁移后,请验证 CRD 存储版本中已移除
v1beta1以确保完成并成功升级。有关更多详细信息,请参阅 Kubernetes 官方 CRD 存储版本文档 和 Issue #11886。
Longhorn v1.10 升级前的迁移要求
在从 Longhorn v1.9 升级到 v1.10 之前,请执行以下手动 CRD 存储版本迁移。
注意:如果您的 Longhorn 安装使用的命名空间不是
longhorn-system,请在整个命令中将longhorn-system替换为您自定义的命名空间。# 临时禁用 CR 验证 webhook,以允许更新只读设置 CR。 kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \ --type=merge \ -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \ jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then .operations |= map(select(. != "UPDATE")) else . end)')" # 迁移曾存储 v1beta1 资源的 CRD migration_time="$(date +%Y-%m-%dT%H:%M:%S)" crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name')) for crd in "${crds[@]}"; do echo "正在迁移 ${crd} ..." for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do # 附加额外注释以在最新存储版本中触发 v1beta1 资源更新。 kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}' done # 清理 CRD 状态中的存储版本 kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status done # 重新启用 CR 验证 webhook。 kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \ --type=merge \ -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \ jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then .operations |= (. + ["UPDATE"] | unique) else . end)')"迁移验证
运行脚本后,使用此命令验证 CRD 存储版本:
kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"
“}{end}'
关键点:在继续升级到 v1.10 之前,所有 Longhorn CRD 的 `storedVersions` 中必须仅列出 `"v1beta2"`(即完全不存在 `"v1beta1"`)。 成功输出的示例:backingimagedatasources.longhorn.io: [“v1beta2”] backingimagemanagers.longhorn.io: [“v1beta2”] backingimages.longhorn.io: [“v1beta2”] backupbackingimages.longhorn.io: [“v1beta2”] backups.longhorn.io: [“v1beta2”] backuptargets.longhorn.io: [“v1beta2”] backupvolumes.longhorn.io: [“v1beta2”] engineimages.longhorn.io: [“v1beta2”] engines.longhorn.io: [“v1beta2”] instancemanagers.longhorn.io: [“v1beta2”] nodes.longhorn.io: [“v1beta2”] orphans.longhorn.io: [“v1beta2”] recurringjobs.longhorn.io: [“v1beta2”] replicas.longhorn.io: [“v1beta2”] settings.longhorn.io: [“v1beta2”] sharemanagers.longhorn.io: [“v1beta2”] snapshots.longhorn.io: [“v1beta2”] supportbundles.longhorn.io: [“v1beta2”] systembackups.longhorn.io: [“v1beta2”] systemrestores.longhorn.io: [“v1beta2”] volumeattachments.longhorn.io: [“v1beta2”] volumes.longhorn.io: [“v1beta2”]
完成这些步骤后,Longhorn 升级到 v1.10 应该可以顺利进行。 #### 升级到 Longhorn v1.10 时 CRD 升级失败的故障排除 如果您未应用所需的预升级迁移步骤,且 CR 未完全迁移到 `v1beta2`,则 `longhorn-manager` Pod 可能无法正常运行。此问题的常见错误消息是:Upgrade failed: cannot patch “backingimagedatasources.longhorn.io” with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io “backingimagedatasources.longhorn.io” is invalid: status.storedVersions[0]: Invalid value: “v1beta1”: missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersions
要修复此问题,您必须执行**强制降级**回升级失败前运行的**确切 Longhorn v1.9.x 版本**。 ##### 降级步骤(kubectl 安装) 如果 Longhorn 是使用 `kubectl` 安装的,则必须在降级前修补 `current-longhorn-version` 设置。在以下命令中将 `v1.9.x` 替换为升级前的原始版本。 ```bash # 附加注释以允许修补 current-longhorn-version。 kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}' # 临时覆盖当前版本以允许旧版本安装 # 将值 `"v1.9.x"` 替换为升级前的原始版本。 kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'修改
current-longhorn-version后,您可以继续降级到原始的 Longhorn v1.9.x 部署。降级步骤(Helm 安装)
如果 Longhorn 是使用 Helm 安装的,通过禁用
preUpgradeChecker.upgradeVersionCheck标志允许降级。降级后
降级完成且 Longhorn 系统在 v1.9.x 版本上稳定后,您必须立即按照 Longhorn v1.10 升级前的迁移要求 中概述的步骤操作。此步骤对于在再次尝试升级 Longhorn v1.10 之前,将所有剩余的
v1beta1CR 迁移到v1beta2至关重要。
重要修复
此版本包含多项关键稳定性和性能改进:
实例管理器中的协程泄漏(V2 数据引擎)
修复了使用 V2 数据引擎时实例管理器中的协程泄漏问题。此问题可能导致内存使用增加,长期来看可能引发稳定性问题。
有关更多详细信息,请参阅 Issue #11962。
中断模式下 V2 卷附加失败
修复了使用 NVMe 磁盘的中断模式的 V2 卷可能无法完成附加过程,导致卷无限期保持附加状态的问题。
在 Longhorn v1.10.0 中,中断模式仅支持 AIO 磁盘。从 v1.10.1 开始支持 NVMe 磁盘的中断模式。
有关更多详细信息,请参阅 Issue #11816。
仅 IPv4 节点上的 UI 部署失败
修复了 v1.10.0 中引入的一个错误,即 Longhorn UI 在仅启用 IPv4 的节点上部署失败。UI 现在可以正确支持仅 IPv4 配置,无需 IPv6。
有关更多详细信息,请参阅 Issue #11875。
共享管理器内存使用过高
修复了 RWX(ReadWriteMany)卷共享管理器中内存消耗过高的问题。该组件在正常运行下现在保持稳定的内存使用。
有关更多详细信息,请参阅 Issue #12043。
安装
[!IMPORTANT] 在安装 Longhorn v1.10.1 之前,请确保您的集群运行的是 Kubernetes v1.25 或更高版本。
您可以使用多种工具安装 Longhorn,包括 Rancher、Kubectl 和 Helm。有关安装方法和要求的更多信息,请参阅 Longhorn 文档中的 快速安装。
升级
[!IMPORTANT] 在从 Longhorn v1.9.x 升级到 v1.10.1 之前,请确保您的集群运行的是 Kubernetes v1.25 或更高版本。
Longhorn 仅允许从受支持的版本升级。有关升级路径和程序的更多信息,请参阅 Longhorn 文档中的 升级。
发布后已知问题
有关此版本发布后发现的查询信息,请参阅 发布已知问题。
已解决问题
改进
- [BACKPORT][v1.10.1][IMPROVEMENT]
auto-delete-pod-when-volume-detached-unexpectedly应仅关注 Kubernetes 内置工作负载。12125 - @derekbit @chriscchien - [BACKPORT][v1.10.1][IMPROVEMENT]
CSIStorageCapacity对象必须显示可调度(可分配)容量 12036 - @chriscchien @bachmanity1 - [BACKPORT][v1.10.1][IMPROVEMENT] 改进节点发布卷期间挂载失败的错误日志记录 12033 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.10.1][IMPROVEMENT] 改进 Helm Chart defaultSettings 处理,支持自动引用和多类型 12020 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][IMPROVEMENT] 避免在迁移期间副本不可用时重复引擎重启 11945 - @yangchiu @shuo-wu
- [BACKPORT][v1.10.1][IMPROVEMENT] 将 GuaranteedInstanceManagerCPU 的最大值调整为较大值 11968 - @mantissahz
- [BACKPORT][v1.10.1][IMPROVEMENT] 添加 Longhorn 安装变体的使用指标 11795 - @derekbit
错误
- [BACKPORT][v1.10.1][BUG] 备份目标指标损坏 12089 - @mantissahz @roger-ryao
- [BACKPORT][v1.10.1][BUG] 网络断开后备份镜像下载卡住 12094 - @COLDTURNIP @chriscchien
- [BACKPORT][v1.10.1][BUG] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 at longhorn-engine/pkg/controller/control.go:218 +0x2de 12088 - @roger-ryao
- [BACKPORT][v1.10.1][BUG] 由于剩余 backuptarget 无法完成卸载 11964 - @mantissahz @roger-ryao
- [BACKPORT][v1.10.1][BUG] share-manager 内存使用过高 12043 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][BUG] v2 数据引擎中未找到 NVME 磁盘(无法找到 BDF 设备) 12029 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] 循环作业执行期间出现 NPE 错误 11926 - @yangchiu @shuo-wu
- [BACKPORT][v1.10.1][BUG] 在 talos 节点上 v2 卷创建失败 12026 - @c3y1huang @chriscchien
- [BACKPORT][v1.10.1][BUG] CSI 节点发布卷期间挂载错误未正确处理 12008 - @COLDTURNIP
- [BACKPORT][v1.10.1][BUG] 并发向同一节点添加多个磁盘时偶尔失败 12018 - @davidcheng0922 @roger-ryao
- [BUG] 从 1.9.1 升级到 1.10.0 失败,因为旧资源仍处于 v1beta1 11886 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.10.1][BUG] 如果引擎镜像从附加节点删除,DR 卷会卡在
unknown状态 11998 - @yangchiu @shuo-wu - [BACKPORT][v1.10.1][BUG] 如果引擎镜像未部署在某个节点上,卷会卡在
attaching状态 11996 - @yangchiu @shuo-wu - [BACKPORT][v1.10.1][BUG] 重新启用 v2 数据引擎后无法通过 BDF 重新添加块类型磁盘 12000 - @yangchiu @davidcheng0922
- [BACKPORT][v1.10.1][BUG]
test_system_backup_and_restore测试用例在 master-head 上失败 12005 - @derekbit @chriscchien - [BACKPORT][v1.10.1][BUG] 修复 SPDK v25.05 CVE 问题 11970 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] V2 卷卡在卷附加中(V2 中断模式) 11976 - @c3y1huang @chriscchien
- [BACKPORT][v1.10.1][BUG] RWX 卷导致进程不可中断睡眠 11958 - @COLDTURNIP @chriscchien
- [BACKPORT][v1.10.1][BUG] 从 1.9.2 升级到 1.10.0 后 longhorn-manager 启动失败 11865 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] 块磁盘删除失败但没有错误消息 11954 - @davidcheng0922 @roger-ryao
- [BACKPORT][v1.10.1][BUG] 使用 v2 数据引擎时 instance-manager 中的协程泄漏 11962 - @PhanLe1010 @chriscchien
- [BACKPORT][v1.10.1][BUG] 无效内存地址或空指针解引用 11942 - @bachmanity1 @roger-ryao
- [BACKPORT][v1.10.1][BUG] 如果缺少 dataEngine 参数,csi-provisioner 静默创建 CSIStorageCapacity 失败 11918 - @yangchiu @bachmanity1
- [BACKPORT][v1.10.1][BUG] longhorn-engine 的 UI 崩溃 11901 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][BUG] 如果活动副本数量大于
volumme.spec.numberOfReplicas,卷无法升级 11895 - @yangchiu @derekbit - [BACKPORT][v1.10.1][BUG] 在 v1.10.0 版本中仅启用 IPv4 的节点上 UI 部署失败 11875 - @yangchiu @c3y1huang
- [BACKPORT][v1.10.1][BUG] 标记
disable-v2-data-engine=true后无法分离 v2 卷 11801 - @mantissahz
杂项
- [BACKPORT][v1.10.1][REFACTOR] UI 组件的 SAST 检查 11992 - @chriscchien
- [HOTFIX] 为 longhorn-manager:v1.10.0 创建热修复镜像 11951 - @c3y1huang @roger-ryao
贡献者
- @COLDTURNIP
- @PhanLe1010
- @bachmanity1
- @c3y1huang
- @chriscchien
- @davidcheng0922
- @derekbit
- @forbesguthrie
- @innobead
- @mantissahz
- @rebeccazzzz
- @roger-ryao
- @sushant-suse
- @shuo-wu
- @yangchiu
更新内容 (原始)
Longhorn v1.10.1 Release Notes
Longhorn 1.10.1 introduces several improvements and bug fixes that are intended to improve system quality, resilience, stability and security.
We welcome feedback and contributions to help continuously improve Longhorn.
For terminology and context on Longhorn releases, see Releases.
[!WARNING]
HotFix
The
longhorn-manager:v1.10.1image is affected by
- Regression:
- [BUG] invalid memory address or nil pointer dereference that can trigger a nil-pointer dereference under certain conditions, potentially causing unexpected crashes.
- [BUG] Block Mode Volume Migration Stuck that can cause block mode volume migration to get stuck indefinitely.
- Day-one issues:
- V2 volume clone:
- [BUG] V2 Volume CSI Clone Slowness Caused by VolumeAttachment Webhook that can lead to significant delays during V2 volume cloning operations.
- [BUG] [Bug] A cloned volume cannot be attached to a workload that prevents cloned volumes from being attached to workloads.
- Replica auto-balance:
- [BUG] Replica auto balance disk pressure threshold stalled with stopped volumes that can cause the replica auto-balance feature to stall when volumes are stopped under disk pressure conditions.
To mitigate the issues, replace
longhorn-manager:v1.10.1with the hotfixed imagelonghorn-manager:v1.10.1-hotfix-2.Follow these steps to apply the update:
Disable the upgrade version check
- Helm users: Set
upgradeVersionChecktofalsein thevalues.yamlfile.- Manifest users: Remove the
--upgrade-version-checkflag from the deployment manifest.Update the
longhorn-managerimage
- Change the image tag from
v1.10.1tov1.10.1-hotfix-2in the appropriate file:
- For Helm: Update
values.yaml- For manifests: Update the deployment manifest directly.
Proceed with the upgrade
- Apply the changes using your standard Helm upgrade command or reapply the updated manifest.
Upgrade
If your Longhorn cluster was initially deployed with a version earlier than v1.3.0, the Custom Resources (CRs) were created using the
v1beta1APIs. While the upgrade from Longhorn v1.8 to v1.9 automatically migrates all CRs to the newv1beta2version, a manual CR migration is strongly advised before upgrading from Longhornv1.9tov1.10.Certain operations, such as an
etcdor CRD restore, may leave behindv1beta1data. Manually migrating your CRs ensures that all Longhorn data is properly updated to thev1beta2API, preventing potential compatibility issues and unexpected behavior with the new Longhorn version.Following the manual migration, verify that
v1beta1has been removed from the CRD stored versions to ensure completion and a successful upgrade.For more details, see Kubernetes official document for CRD storage version, and Issue #11886.
Migration Requirement Before Longhorn v1.10 Upgrade
Before upgrading from Longhorn v1.9 to v1.10, perform the following manual CRD storage version migration.
Note: If your Longhorn installation uses a namespace other than
longhorn-system, replacelonghorn-systemwith your custom namespace throughout the commands.# Temporarily disable the CR validation webhook to allow updating read-only settings CRs. kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \ --type=merge \ -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \ jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then .operations |= map(select(. != "UPDATE")) else . end)')" # Migrate CRDs that ever stored v1beta1 resources migration_time="$(date +%Y-%m-%dT%H:%M:%S)" crds=($(kubectl get crd -l app.kubernetes.io/name=longhorn -o json | jq -r '.items[] | select(.status.storedVersions | index("v1beta1")) | .metadata.name')) for crd in "${crds[@]}"; do echo "Migrating ${crd} ..." for name in $(kubectl -n longhorn-system get "$crd" -o jsonpath='{.items[*].metadata.name}'); do # Attach additional annotations to trigger v1beta1 resource updating in the latest storage version. kubectl patch "${crd}" "${name}" -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"migration-time":"'"${migration_time}"'"}}}' done # Clean up the stored version in CRD status kubectl patch crd "${crd}" --type=merge -p '{"status":{"storedVersions":["v1beta2"]}}' --subresource=status done # Re-enable the CR validation webhook. kubectl patch validatingwebhookconfiguration longhorn-webhook-validator \ --type=merge \ -p "$(kubectl get validatingwebhookconfiguration longhorn-webhook-validator -o json | \ jq '.webhooks[0].rules |= map(if .apiGroups == ["longhorn.io"] and .resources == ["settings"] then .operations |= (. + ["UPDATE"] | unique) else . end)')"Migration Verification
After running the script, verify the CRD stored versions using this command:
kubectl get crd -l app.kubernetes.io/name=longhorn -o=jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.storedVersions}{"\n"}{end}'Crucially, all Longhorn CRDs MUST have only
"v1beta2"listed instoredVersions(i.e.,"v1beta1"must be completely absent) before proceeding to the v1.10 upgrade.Example of successful output:
backingimagedatasources.longhorn.io: ["v1beta2"] backingimagemanagers.longhorn.io: ["v1beta2"] backingimages.longhorn.io: ["v1beta2"] backupbackingimages.longhorn.io: ["v1beta2"] backups.longhorn.io: ["v1beta2"] backuptargets.longhorn.io: ["v1beta2"] backupvolumes.longhorn.io: ["v1beta2"] engineimages.longhorn.io: ["v1beta2"] engines.longhorn.io: ["v1beta2"] instancemanagers.longhorn.io: ["v1beta2"] nodes.longhorn.io: ["v1beta2"] orphans.longhorn.io: ["v1beta2"] recurringjobs.longhorn.io: ["v1beta2"] replicas.longhorn.io: ["v1beta2"] settings.longhorn.io: ["v1beta2"] sharemanagers.longhorn.io: ["v1beta2"] snapshots.longhorn.io: ["v1beta2"] supportbundles.longhorn.io: ["v1beta2"] systembackups.longhorn.io: ["v1beta2"] systemrestores.longhorn.io: ["v1beta2"] volumeattachments.longhorn.io: ["v1beta2"] volumes.longhorn.io: ["v1beta2"]With these steps completed, the Longhorn upgrade to v1.10 should now proceed without issues.
Troubleshooting CRD Upgrade Failures During Upgrade to Longhorn v1.10
If you did not apply the required pre-upgrade migration steps and the CRs are not fully migrated to
v1beta2, thelonghorn-managerPods may fail to operate correctly. A common error message for this issue is:Upgrade failed: cannot patch "backingimagedatasources.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "backingimagedatasources.longhorn.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": missing from spec.versions; v1beta1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1beta1 and removes v1beta1 from status.storedVersionsTo fix this issue, you must perform a forced downgrade back to the exact Longhorn v1.9.x version that was running before the failed upgrade attempt.
Downgrade Procedure (kubectl Installation)
If Longhorn was installed using
kubectl, you must patch thecurrent-longhorn-versionsetting before downgrading. Replacev1.9.xwith the original version before upgrade in the following commands.# Attaching annotation to allow patching current-longhorn-version. kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"metadata":{"annotations":{"longhorn.io/update-setting-from-longhorn":""}}}' # Temporarily override current version to allow old version installation # Replace the value `"v1.9.x" to the original version before upgrade. kubectl patch settings.longhorn.io current-longhorn-version -n longhorn-system --type=merge -p='{"value":"v1.9.x"}'After modifying
current-longhorn-version, you can proceed to downgrade to the original Longhorn v1.9.x deployment.Downgrade Procedure (Helm Installation)
If Longhorn was installed using Helm, the downgrade is allowed by disabling the
preUpgradeChecker.upgradeVersionCheckflag.Post-Downgrade
Once the downgrade is complete and the Longhorn system is stable on the v1.9.x version, you must immediately follow the steps outlined in the Migration Requirement Before Longhorn v1.10 Upgrade. This step is crucial to migrate all remaining
v1beta1CRs tov1beta2before attempting the Longhorn v1.10 upgrade again.
Important Fixes
This release includes several critical stability and performance improvements:
Goroutine Leak in Instance Manager (V2 Data Engine)
Fixed a goroutine leak in the instance manager when using the V2 data engine. This issue could lead to increased memory usage and potential stability problems over time.
For more details, see Issue #11962.
V2 Volume Attachment Failure in Interrupt Mode
Fixed an issue where V2 volumes using interrupt mode with NVMe disks could fail to complete the attachment process, causing volumes to remain stuck in the attaching state indefinitely.
In Longhorn v1.10.0, interrupt mode supports only AIO disks. Interrupt mode for NVMe disks is supported starting in v1.10.1.
For more details, see Issue #11816.
UI Deployment Failure on IPv4-Only Nodes
Fixed a bug introduced in v1.10.0 where the Longhorn UI failed to deploy on nodes with only IPv4 enabled. The UI now correctly supports IPv4-only configurations without requiring IPv6.
For more details, see Issue #11875.
Share Manager Excessive Memory Usage
Fixed excessive memory consumption in the share manager for RWX (ReadWriteMany) volumes. The component now maintains stable memory usage under normal operation.
For more details, see Issue #12043.
Installation
[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.10.1.
You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.
Upgrade
[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.9.x to v1.10.1.
Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.
Post-Release Known Issues
For information about issues identified after this release, see Release-Known-Issues.
Resolved Issues
Improvement
- [BACKPORT][v1.10.1][IMPROVEMENT] The
auto-delete-pod-when-volume-detached-unexpectedlyshould only focus on the kubernetes builtin workload. 12125 - @derekbit @chriscchien - [BACKPORT][v1.10.1][IMPROVEMENT]
CSIStorageCapacityobjects must show schedulable (allocatable) capacity 12036 - @chriscchien @bachmanity1 - [BACKPORT][v1.10.1][IMPROVEMENT] improve error logging for failed mounting during node publish volume 12033 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.10.1][IMPROVEMENT] Improve Helm Chart defaultSettings handling with automatic quoting and multi-type support 12020 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][IMPROVEMENT] Avoid repeat engine restart when there are replica unavailable during migration 11945 - @yangchiu @shuo-wu
- [BACKPORT][v1.10.1][IMPROVEMENT] Adjust maximum of GuaranteedInstanceManagerCPU to a big value 11968 - @mantissahz
- [BACKPORT][v1.10.1][IMPROVEMENT] Add usage metrics for Longhorn installation variant 11795 - @derekbit
Bug
- [BACKPORT][v1.10.1][BUG] Backup target metric is broken 12089 - @mantissahz @roger-ryao
- [BACKPORT][v1.10.1][BUG] Backing image download gets stuck after network disconnection 12094 - @COLDTURNIP @chriscchien
- [BACKPORT][v1.10.1][BUG] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 at longhorn-engine/pkg/controller/control.go:218 +0x2de 12088 - @roger-ryao
- [BACKPORT][v1.10.1][BUG] Unable to complete uninstallation due to the remaining backuptarget 11964 - @mantissahz @roger-ryao
- [BACKPORT][v1.10.1][BUG] share-manager excessive memory usage 12043 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][BUG] NVME disk not found in v2 data engine (failed to find device for BDF) 12029 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] NPE error during recurring job execution 11926 - @yangchiu @shuo-wu
- [BACKPORT][v1.10.1][BUG] v2 volume creation failed on talos nodes 12026 - @c3y1huang @chriscchien
- [BACKPORT][v1.10.1][BUG] mounting error is not properly hanedled during CSI node publish volume 12008 - @COLDTURNIP
- [BACKPORT][v1.10.1][BUG] Adding multiple disks to the same node concurrently may occasionally fail 12018 - @davidcheng0922 @roger-ryao
- [BUG] upgrading from 1.9.1 to 1.10.0 fails due to old resources still being in v1beta1 11886 - @COLDTURNIP @roger-ryao
- [BACKPORT][v1.10.1][BUG] DR volume gets stuck in
unknownstate if engine image is deleted from the attached node 11998 - @yangchiu @shuo-wu - [BACKPORT][v1.10.1][BUG] Volume gets stuck in
attachingstate if engine image image is not deployed on one of nodes 11996 - @yangchiu @shuo-wu - [BACKPORT][v1.10.1][BUG] Unable to re-add block-type disks by BDF after re-enable v2 data engine 12000 - @yangchiu @davidcheng0922
- [BACKPORT][v1.10.1][BUG]
test_system_backup_and_restoretest case failed on master-head 12005 - @derekbit @chriscchien - [BACKPORT][v1.10.1][BUG] Fix SPDK v25.05 CVE issue 11970 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] V2 volume stuck in volume attachment (V2 interrupt mode) 11976 - @c3y1huang @chriscchien
- [BACKPORT][v1.10.1][BUG] RWX volume causes process uninterruptible sleep 11958 - @COLDTURNIP @chriscchien
- [BACKPORT][v1.10.1][BUG] longhorn-manager fails to start after upgrading from 1.9.2 to 1.10.0 11865 - @derekbit @roger-ryao
- [BACKPORT][v1.10.1][BUG] Block disk deletion fails without error message 11954 - @davidcheng0922 @roger-ryao
- [BACKPORT][v1.10.1][BUG] Goroutine leak in instance-manager when using v2 data engine 11962 - @PhanLe1010 @chriscchien
- [BACKPORT][v1.10.1][BUG] invalid memory address or nil pointer dereference 11942 - @bachmanity1 @roger-ryao
- [BACKPORT][v1.10.1][BUG] csi-provisioner silently fails to create CSIStorageCapacity if dataEngine parameter is missing 11918 - @yangchiu @bachmanity1
- [BACKPORT][v1.10.1][BUG] longhorn-engine’s UI panics 11901 - @derekbit @chriscchien
- [BACKPORT][v1.10.1][BUG] Volume is unable to upgrade if the number of active replicas is larger than
volumme.spec.numberOfReplicas11895 - @yangchiu @derekbit - [BACKPORT][v1.10.1][BUG] UI fails to deploy when only IPv4 is enabled on nodes with v1.10.0 version 11875 - @yangchiu @c3y1huang
- [BACKPORT][v1.10.1][BUG] Unable to detach a v2 volume after labeling
disable-v2-data-engine=true11801 - @mantissahz
Misc
- [BACKPORT][v1.10.1][REFACTOR] SAST checks for UI component 11992 - @chriscchien
- [HOTFIX] Create hotfixed image for longhorn-manager:v1.10.0 11951 - @c3y1huang @roger-ryao
Contributors
- @COLDTURNIP
- @PhanLe1010
- @bachmanity1
- @c3y1huang
- @chriscchien
- @davidcheng0922
- @derekbit
- @forbesguthrie
- @innobead
- @mantissahz
- @rebeccazzzz
- @roger-ryao
- @sushant-suse
- @shuo-wu
- @yangchiu