longhorn v1.9.0 版本更新介绍
发布日期: 2025-05-27
版本号: v1.9.0
Longhorn v1.9.0 版本引入了多项新功能、增强和问题修复,旨在提升系统稳定性和用户体验。主要亮点包括V2数据引擎改进、孤儿实例删除、离线副本重建、周期性系统备份以及Longhorn资源可观测性的增强。该版本存在一个已知问题:longhorn-manager v1.9.0受回归问题影响,可能导致周期性作业失败,用户需将镜像替换为热修复版本
v1.9.0-hotfix-1。此外,移除了已弃用的environment_check.sh脚本,并更新了orphan-resource-auto-deletion设置。longhorn.io/v1beta1API已被标记为弃用,并将在v1.10.0中移除。安装或升级前需确保集群运行Kubernetes v1.25或更高版本。
更新内容 (中文)
Longhorn v1.9.0 发布说明
Longhorn v1.9.0 引入了新功能、增强和错误修复,旨在提升系统稳定性和用户体验。主要亮点包括 V2 数据引擎改进、孤立实例删除、离线副本重建、周期性系统备份以及 Longhorn 资源可观测性的增强。
Longhorn 团队感谢您的贡献,并期待收到关于此版本的反馈。
关于 Longhorn 发布的术语和背景,请参阅 发布版本。
警告
longhorn-manager v1.9.0 受到一个 回滚问题 的影响,该问题会导致周期性作业失败。要解决此问题,请将 longhorn-manager:v1.9.0 替换为热修复镜像 longhorn-manager:v1.9.0-hotfix-1。
您可以按照以下步骤应用更新:
-
禁用升级版本检查
- Helm 用户:在
values.yaml文件中将upgradeVersionCheck设置为false。 - Manifest 用户:从部署清单中移除
--upgrade-version-check标志。
- Helm 用户:在
-
更新
longhorn-manager镜像- 在相应文件中将镜像标签从
v1.9.0更改为v1.9.0-hotfix-1:- 对于 Helm:更新
values.yaml - 对于 Manifest:直接更新部署清单。
- 对于 Helm:更新
- 在相应文件中将镜像标签从
-
继续升级
- 使用标准的 Helm 升级命令或重新应用更新后的清单来应用更改。
移除
环境检查脚本
在 v1.7.0 中弃用的 environment_check.sh 脚本已在 v1.9.0 中移除。请使用 Longhorn 命令行工具 (longhornctl) 检查您的环境是否存在潜在问题。
孤立资源自动删除
orphan-auto-deletion 设置在 v1.9.0 中已被 orphan-resource-auto-deletion 取代。要复制之前的行为,请在 orphan-resource-auto-deletion 的值中包含 replica-data。在升级期间,原始的 orphan-auto-deletion 设置会自动迁移。
更多信息,请参阅 孤立数据清理。
longhorn.io/v1beta2 CRD 中的已弃用字段
CRD 中的已弃用字段已被移除。详情请参阅问题 #6684。
弃用与不兼容性
longhorn.io/v1beta1 API
Longhorn API 的 v1beta1 版本在 v1.9.0 中被标记为不提供服务且不受支持,并将在 v1.10.0 中移除。
更多详情,请参阅 问题 #10250。
V2 支持镜像的破坏性更改
从 Longhorn v1.9.0 开始,V2 支持镜像与早期版本不兼容,原因是 SPDK 支持镜像逻辑卷使用的扩展属性 (xattrs) 存在命名冲突。因此,在升级过程中必须删除并重新创建 V2 支持镜像。由于支持镜像在使用它们的卷仍然存在时无法删除,您必须先备份、删除这些卷,然后按以下步骤恢复它们:
- 在升级到 v1.9.0 之前:
- 验证备份目标是否正常运行。
- 为所有使用 V2 支持镜像的卷创建完整备份。
- 备份完成后,分离并删除这些卷。
- 在 支持镜像 页面,保存所有 V2 支持镜像的规格,包括名称和镜像源。
- 删除所有 V2 支持镜像。
- 升级后:
- 使用相同的名称和镜像源重新创建 V2 支持镜像。
- 从备份中恢复卷。
更多详情,请参阅 问题 #10805。
主要亮点
新的 V2 数据引擎功能
虽然 V2 数据引擎在本版本中仍处于实验状态,但几项核心功能已得到显著改进:
- 支持 UBLK 前端:在 V2 数据引擎中支持 UBLK 前端,以实现更好的性能和资源利用率。
- 存储网络:在 V2 数据引擎中引入存储网络支持,允许网络隔离。
- 离线副本重建:支持离线副本重建,允许降级卷即使在卷分离时也能自动恢复副本。此功能确保了高数据可用性,无需手动干预。
周期性系统备份
从 Longhorn v1.9.0 开始,您可以创建用于系统备份创建的周期性作业。
离线副本重建
Longhorn 引入了离线副本重建,此功能允许降级卷即使在卷分离时也能自动恢复副本。此功能最大限度地减少了手动恢复步骤的需要,加速了恢复过程,并确保了高数据可用性。默认情况下,离线副本重建处于禁用状态。要启用它,请在 Longhorn UI 或 CLI 中将 offline-replica-rebuilding 设置为 true。
孤立实例删除
Longhorn 现在可以跟踪和移除孤立实例,这些实例是不再与活动卷关联的剩余资源(如副本或引擎)。这些实例可能由于意外故障或不完全清理而累积。
为减少资源使用并维护系统性能,Longhorn 支持自动和手动清理。默认情况下,此功能处于禁用状态。要启用它,请在 Longhorn UI 或 CLI 中将 orphan-resource-auto-deletion 设置为 instance。
改进的副本、引擎和重建状态指标
Longhorn 通过新的 Prometheus 指标改进了可观测性,这些指标公开了副本和引擎 CR 的状态和身份,以及重建活动。这些指标使监控跨集群的重建更加容易。
安装
[!IMPORTANT] 在安装 Longhorn v1.9.0 之前,请确保您的集群运行的是 Kubernetes v1.25 或更高版本。
您可以使用多种工具安装 Longhorn,包括 Rancher、Kubectl 和 Helm。有关安装方法和要求的更多信息,请参阅 Longhorn 文档中的 快速安装。
升级
[!IMPORTANT] 在从 Longhorn v1.8.x 升级到 v1.9.0 之前,请确保您的集群运行的是 Kubernetes v1.25 或更高版本。
Longhorn 仅允许从受支持的版本升级。有关升级路径和过程的更多信息,请参阅 Longhorn 文档中的 升级。
发布后已知问题
有关此发布后发现的问题信息,请参阅 发布已知问题。
亮点
- [功能] 如果 v1 资源已删除,则清理孤立的卷运行时资源 6764 - @COLDTURNIP @chriscchien
- [功能] v2 卷支持 UBLK 前端 9456 - @PhanLe1010 @chriscchien
- [功能] V1 和 V2 卷离线副本重建 8443 - @mantissahz @roger-ryao
- [任务] 将 v1beta1 CR 迁移到 v1beta2 10250 - @COLDTURNIP @roger-ryao
- [功能] V2 数据引擎的存储网络 6450 - @c3y1huang @roger-ryao
- [功能] 周期性系统备份 6534 - @yangchiu @c3y1huang
功能
- [功能] 使用增量快照进行增量副本重建:SPDK API 开发 10799 - @yangchiu @DamiaSan
- [功能] 卷表中的运行副本字段 10817 - @xelab04 @roger-ryao
- [功能] Longhorn UI 支持孤立实例 CR 管理 10760 - @yangchiu @houhoucoop
- [功能] 允许在从快照创建备份时自动删除该快照。 9213 - @yangchiu @mantissahz
- [功能] 添加按节点/集群的卷/副本数量的缺失指标 7599 - @c3y1huang @roger-ryao
改进
- [改进] 移除不必要的 lasso 依赖 10856 - @derekbit @chriscchien
- [改进] 孤立实例 CR 创建的可配置等待间隔 10904 - @derekbit @chriscchien
- [改进] 防止在删除 v1 实例时创建孤立实例 10888 - @COLDTURNIP @chriscchien
- [改进] 防止在禁用离线重建时删除卷产生错误日志。 10889 - @mantissahz @chriscchien
- [改进] 为副本和引擎 CR 添加 Prometheus 指标 10722 - @hookak @chriscchien
- [改进] 导出 longhorn 引擎重建状态作为 Prometheus 指标 10550 - @hookak @chriscchien
- [改进] 为单副本 V1 卷禁用快照校验和计算 10518 - @derekbit @chriscchien
- [改进] 在图表中添加 extraObject 10835 - @DrummyFloyd @chriscchien
- [改进] 在 v2 快照被删除时禁用其哈希计算 10563 - @shuo-wu @roger-ryao
- [改进] v2 校验和计算和更新应遵循 v1 流程 10480 - @shuo-wu @roger-ryao
- [改进] 在升级路径中为更新选项添加严格的字段验证 10644 - @ChanYiLin @chriscchien
- [改进] 在备份进行期间显示快照大小 9783 - @yangchiu @houhoucoop
- [改进] 在对 v2 卷进行快照之前不要同步所有文件系统 9023 - @yangchiu @DamiaSan
- [改进] spdk_tgt 可以在存在高优先级任务时取消 lvol 校验和计算 10421 - @yangchiu @DamiaSan
- [改进] 如果 Blob 正忙,则不强制移除 Lvol 10474 - @yangchiu @DamiaSan
- [改进] 从 CRD 中移除已弃用的字段 6684 - @derekbit @roger-ryao
- [改进] Longhorn CLI 无法识别 Raspbian OS 10676 - @bachmanity1 @roger-ryao
- [改进] 减少分离卷的自动均衡日志噪音 10691 - @dihmandrake @roger-ryao
- [改进] 移除 v2-data-engine-guaranteed-instance-manager-cpu 的上限 10662 - @derekbit @roger-ryao
- [改进] 清理 BackupTarget 条件消息处理 8224 - @chriscchien @houhoucoop
- [改进] Longhorn CLI 支持 SLES micro 9256 - @yangchiu @DamiaSan
- [改进] 允许从 helm 值设置
volumeBindingMode10592 - @ruant @roger-ryao - [文档] 准备支持镜像在升级期间故障排除的知识库 10590 - @ChanYiLin @chriscchien
- [改进] 缺失的引擎 v2 卷的 Prometheus 指标 10472 - @hookak @roger-ryao
- [改进] 创建卷 UI 改进,根据
v1或v2选择自动过滤支持镜像 10086 - @houhoucoop @roger-ryao - [UI][改进] 改进移除
Block类型磁盘失败时的警告消息 10580 - @houhoucoop @roger-ryao - [改进] 将完整的备份模式选项传递给 CSI 卷快照类型备份 9785 - @ChanYiLin @roger-ryao
- [UI][改进] 清理 BackupTarget 条件消息处理 10579 - @houhoucoop
- [改进] 改进移除
Block类型磁盘失败时的警告消息 10522 - @yangchiu @ChanYiLin - [改进] 将
SettingNameV2DataEngineHugepageLimit移动到危险区域设置 7746 - @derekbit @chriscchien - [改进] 在支持包中包含 /proc/mounts 文件和 multipath.config 6754 - @c3y1huang @roger-ryao
- [改进] 使用 code-generator/kube_codegen.sh 生成 K8s 桩代码和 CRD 7944 - @derekbit @chriscchien
- [改进] CRD 和 API 代码生成器与 Go 常规源路径解耦 10556 - @COLDTURNIP
- [改进] 支持可配置的 upgrade-responder URL 10437 - @derekbit @roger-ryao
- [改进] 设置更改验证应恢复使用卷状态来确定“是否所有卷都已分离” 10233 - @yangchiu @james-munson
- [改进] 改进更新危险区域设置的用户体验 8070 - @yangchiu @mantissahz
- [UI][功能] V1 和 V2 卷离线副本重建 10581 - @houhoucoop @roger-ryao
- [UI][功能] 周期性系统备份 10262 - @yangchiu @houhoucoop
缺陷
- [缺陷] 容器中 git checkout 出错 10621 - @derekbit @chriscchien
- [缺陷] 测试用例
test_snapshot_prune_and_coalesce_simultaneously_with_backing_image失败 10808 - @yangchiu @c3y1huang - [缺陷] 如果存在支持包
ReadyForDownload,则无法终止命名空间longhorn-system10731 - @yangchiu @c3y1huang - [缺陷] Helm persistence.backupTargetName 未在 storageclass 模板中引用 10961 - @yangchiu @mantissahz
- [缺陷] SPDK API lvol_get_snapshot_range_checksums 无法获得正确结果 10950 - @shuo-wu @roger-ryao
- [缺陷] 从 v1.8.1 升级到 v1.9.x 后,V2 支持镜像未就绪 10805 - @COLDTURNIP @chriscchien
- [缺陷] v2 副本重建将遗漏支持镜像 10909 - @shuo-wu @chriscchien
- [缺陷] yamllint 在 helm 图表中发现的次要间距问题 10681 - @codekow
- [缺陷][UI] 带支持镜像的 v2 卷快照未在
Snapshots and Backups图上显示 10526 - @derekbit @chriscchien - [缺陷] 已删除的孤立数据在页面刷新前仍在页面上渲染 10803 - @COLDTURNIP @chriscchien @houhoucoop
- [缺陷] DR 卷激活时未与最新备份同步 10824 - @c3y1huang @chriscchien
- [缺陷][v1.9.0-rc1] 从
v1.8.1升级到v1.9.0-rc1后,Block 磁盘暂时不可用 10821 - @mantissahz - [缺陷] 创建新支持镜像管理器名称时的命名冲突 10616 - @yangchiu @ChanYiLin
- [缺陷] 在启用
Immediate Data Integrity Check的情况下删除快照后,v2 卷副本状态为error10798 - @shuo-wu - [缺陷] 启用
V2 Data Engine设置后,经过某些负面因子操作后 v2 实例管理器无法启动 10791 - @COLDTURNIP @yangchiu - [缺陷] spdk 在注册 lvol 校验和计算时发出
Device or resource busy10140 - @shuo-wu @roger-ryao - [缺陷] v2 实例管理器在 master-head arm64 环境中持续崩溃 10768 - @yangchiu @PhanLe1010
- [缺陷]
longhorn-images.txt中的镜像名称错误 10774 - @c3y1huang - [缺陷] 节点宕机并强制删除终止的部署 pod 后,卷无法成功附加 10689 - @c3y1huang @chriscchien
- [缺陷] 删除一个 v2 卷的副本也会降级另一个 v2 卷 10527 - @yangchiu @ChanYiLin
- [缺陷] 将不存在的磁盘添加到节点将导致 longhorn-manager 崩溃 10749 - @ChanYiLin @roger-ryao
- [缺陷] 从 v1.8.1 升级到 master-head 导致 longhorn-manager 崩溃 10762 - @yangchiu @mantissahz
- [缺陷] 节点重启和工作负载 pod 重启后,pod 数据大小变为 0,且挂载的卷变为只读 9248 - @yangchiu @c3y1huang
- [缺陷] 测试用例
test_engine_crash_during_live_upgrade因数据丢失而失败 10751 - @c3y1huang @roger-ryao - [缺陷] 系统备份可能无限期地卡在
CreatingBackingImageBackups状态 10740 - @yangchiu @ChanYiLin - [缺陷] 带支持镜像的 v2 卷卡在
Attaching状态 10743 - @yangchiu @ChanYiLin - [缺陷] 无法删除超大的 Not Ready 卷 10741 - @WebberHuang1118 @chriscchien
- [缺陷][UI] 对已分离卷进行批量备份创建返回错误 405,并在浏览器控制台显示错误消息 10460 - @yangchiu @a110605
- [缺陷] 在 CI 测试期间,
spdk_tgt在longhorn-spdk-helper中遇到Lvol store removed with error: -1610622 - @derekbit @roger-ryao - [缺陷] 卸载作业创建后可能创建 2 个卸载 pod,其中一个因
deleting-confirmation-flag is set to false错误而失败,而另一个成功完成 10483 - @yangchiu @derekbit - [缺陷] SPDK 持续发出 “Bad length of checksum xattr” 10399 - @ChanYiLin @chriscchien
- [缺陷] 无法在 Talos 上通过 BDF 创建 v2 block-type 磁盘 10313 - @derekbit @roger-ryao
- [缺陷] 升级 Harvester v1.4.1 期间,VM 迁移时 Longhorn v1.7.2 卷出现 I/O 错误 10495 - @derekbit @roger-ryao
- [缺陷] MultiUnmapper 用关于大小不匹配的警告淹没日志。 6406 - @shuo-wu @roger-ryao
- [缺陷]
spdk_tgt在 CI 测试运行期间在longhorn-spdk-helper中发生段错误。 10598 - @derekbit @roger-ryao - [缺陷] Helm 图表中的备份执行超时设置问题 10323 - @yangchiu @james-munson
- [缺陷] Longhorn 卷加密在 Talos 1.9.x 中不工作 10584 - @c3y1huang @roger-ryao
- [缺陷][UI] 卷设置
Allow snapshots removal during trim中的大小写不一致 10470 - @yangchiu @houhoucoop - [缺陷] “扩展卷”选项在“卷”选项卡下灰显,但在卷详细信息部分可用。 7529 - @yangchiu @houhoucoop
- [缺陷] 实例管理器镜像构建失败 10653 - @shuo-wu
- [缺陷] 节点重启后,周期性作业 pod 卡在待定状态,无法创建新快照 7956 - @c3y1huang @chriscchien
- [缺陷] 使用 v2 支持镜像升级后,实例管理器 pod 卡在终止状态 10520 - @ChanYiLin @chriscchien
- [缺陷] 在引擎镜像未完全部署的环境中创建卷时,会创建额外的副本 8263 - @c3y1huang @chriscchien
- [缺陷] [v1.8.0-rc1] 如果存在支持镜像,卸载将失败,instance-manager pod 卡在终止状态 10044 - @ChanYiLin @chriscchien
- [缺陷] 使用 NFS 备份存储的集成测试运行中,test_statefulset_restore 失败 3451 - @roger-ryao
- [缺陷] 如果备份目标不可用,csi 会持续创建备份 10501 - @mantissahz @roger-ryao
- [缺陷] 当支持镜像副本从规范中删除但同时被驱逐时,出现空指针 10464 - @yangchiu @ChanYiLin
- [缺陷] 在 getLatestBackup 中复制了 Mutex 6965 - @james-munson @roger-ryao
- [缺陷] 副本调度器中整数除以零 10502 - @c3y1huang @chriscchien
- [缺陷] Longhorn UI 中的前导或尾随空格会破坏搜索 10491 - @houhoucoop @roger-ryao
- [缺陷] 副本重建完成时,进度可能是 99 而不是 100 8589 - @shuo-wu @chriscchien
- [缺陷] 当 Kubelet 重启时,使用 RWX 卷的工作负载无法恢复 2933 - @james-munson @chriscchien
- [缺陷][UI] 备份存储设置不适用于克隆卷 10463 - @yangchiu @mantissahz
- [缺陷] Longhorn CSI 插件在罕见的竞态条件下执行错误的文件系统格式化操作导致数据丢失 10416 - @yangchiu @PhanLe1010
- [缺陷] WebUI 卷消失并重现 10314 - @yangchiu @PhanLe1010 @houhoucoop
- [缺陷] Longhorn-manager 在每次备份时记录 “Failed to sync backup status” 10301 - @derekbit @chriscchien
- [缺陷] 如果在恢复期间节点断电,DR 卷的重建会卡住 2747 - @COLDTURNIP @roger-ryao
- [缺陷] 当存在 Longhorn SPDK 卷时,卸载 K3s 而不先卸载 Longhorn 会导致 arm64 挂起 8132 - @roger-ryao
- [缺陷] 如果卷是使用支持镜像创建的,副本重建后 V2 卷校验和会改变 10340 - @shuo-wu @chriscchien
- [缺陷] RWX 卷在节点重新连接后会出错 5658 - @james-munson @chriscchien
- [缺陷] V2 BackingImage 在节点重启后失败 10342 - @ChanYiLin @chriscchien
- [缺陷] 降级的 v2 卷即使有可用磁盘也不会创建新副本 9197 - @c3y1huang
- [缺陷] 快照计数强制执行中的缺陷导致卷出错并卡在分离/附加循环中 10308 - @PhanLe1010 @roger-ryao
- [缺陷] 测试用例
test_csi_mount_volume_online_expansion因无法扩展 PVC 而失败 10411 - @yangchiu @c3y1huang - [缺陷] Longhorn CSI 插件 1.8.0 在尝试创建快照时持续崩溃 10303 - @yangchiu @PhanLe1010
- [缺陷] 当备份操作耗时较长时,工作负载 pod 将无法移动到新节点 10171 - @yangchiu @PhanLe1010
- [缺陷] 如果之前的副本未正确清理,v2 引擎会卡在分离-附加循环中 10293 - @yangchiu @shuo-wu
- [缺陷] [UI] 重新加载页面后,系统备份页面上的“创建”按钮被禁用 10351 - @yangchiu @houhoucoop
- [缺陷] 每次指标抓取时 “metrics_collector.(*BackupCollector).Collect” 出现 “Error get size” 10358 - @derekbit @chriscchien
- [缺陷] 代理 gRPC API ReplicaList 对 v1 和 v2 卷返回不同的输出格式 10347 - @shuo-wu @roger-ryao
- [缺陷] 引擎卡在“停止”状态,阻止卷附加 9938 - @ChanYiLin @roger-ryao
- [缺陷] 升级到 v1.8.0 后,web-ui 上的版本号丢失 10336 - @derekbit
- [缺陷] 升级到 1.8 后持续附加/重新附加卷 10304 - @PhanLe1010
- [缺陷] 有时附加的 DR 卷校验和会波动 9305 - @c3y1huang
- [缺陷] 支持镜像管理器 pod 无法在 RHEL 8.4 上启动 2767 - @roger-ryao
- [缺陷] 无法将 Longhorn 从
v1.8.x-head升级到master-head10143 - @roger-ryao - [缺陷] [v1.5.4-rc2] 当 concurrent-automatic-engine-upgrade-per-node-limit > 0 时,V2 卷执行引擎升级 7930 - @derekbit
- [缺陷] 负面测试用例卡在等待 longhorn-ui pod 8248 - @c3y1huang
杂项
- [文档] 副本计数行为不清楚。 10861 - @hoo29 @chriscchien
- [文档] 更新“升级路径强制执行和降级预防” 10945 - @derekbit @roger-ryao
- [任务] 更新 longhornio/nfs-ganesha 镜像 10878 - @derekbit @c3y1huang @chriscchien
- [文档] “设置/常规”中没有
BackupTargetSecret10858 - @vnwnv @roger-ryao - [文档] 在升级系统之前首先创建系统备份 10633 - @ChanYiLin @chriscchien
- [任务] [UI] [功能] v2 卷支持 UBLK 前端 10735 - @chriscchien @houhoucoop
- [文档] 在数据恢复步骤中添加对 RKE2/k3s 的支持 10714 - @mattmattox @roger-ryao
- [文档] 架构图 6761 - @derekbit @chriscchien
- [任务] 修复 longhorn/website 以支持最新的 Hugo 服务器版本 10632 - @chriscchien @sushant-suse
- [任务] 修复 longhorn-manager 中的 lint 问题 10639 - @COLDTURNIP @chriscchien
- [文档] 知识库中的代码块不换行 8143 - @roger-ryao @sushant-suse
- [任务] 用于 ui-extension 迁移的 Longhorn UI 评估 10487 - @houhoucoop
- [文档] 更新设置 Azure 备份目标的步骤 9688 - @mantissahz
- [文档] 解释从现有卷创建支持镜像的过程 10093 - @ChanYiLin @chriscchien
- [文档] 更新 ArgoCD 安装文档 10588 - @mantissahz
- [任务] 自 v1.9.0 起移除环境检查脚本 9239 - @yangchiu @derekbit
- [任务] 在指标中添加平台架构师和卷加密信息 7047 - @c3y1huang @roger-ryao
- [文档] 更新文档并创建知识库以解决 BackingImage 应为 512B 的倍数的限制 10536 - @ChanYiLin
- [文档] 关于 defaultSettings.defaultDataLocality 和 persistence.defaultDataLocality 用法的说明 10253 - @james-munson @roger-ryao
新贡献者
- @bachmanity1
- @codekow
- @DrummyFloyd
- @hoo29
- @hookak
- @dihmandrake
- @mattmattox
- @ruant
- @vnwnv
- @xelab04
贡献者
- @COLDTURNIP
- @ChanYiLin
- @DamiaSan
- @PhanLe1010
- @WebberHuang1118
- @a110605
- @c3y1huang
- @chriscchien
- @derekbit
- @houhoucoop
- @innobead
- @james-munson
- @mantissahz
- @roger-ryao
- @shuo-wu
- @yangchiu
- @sushant-suse
- @jillian-maroket
- @rebeccazzzz
- @forbesguthrie
- @asettle
更新内容 (原始)
Longhorn v1.9.0 Release Notes
Longhorn v1.9.0 introduces new features, enhancements, and bug fixes aimed at improving system stability and user experience. Key highlights include V2 Data Engine improvements, orphaned instance deletion, offline replica rebuilding, recurring system backups, and enhanced observability of Longhorn resources.
The Longhorn team appreciates your contributions and anticipates receiving feedback regarding this release.
For terminology and background on Longhorn releases, see Releases.
Warning
The longhorn-manager v1.9.0 is impacted by a regression issue, which causes failures of recurring jobs. To resolve this issue, replace longhorn-manager:v1.9.0 with the hotfixed image longhorn-manager:v1.9.0-hotfix-1.
You can apply the update by following these steps:
-
Disable the upgrade version check
- Helm users: Set
upgradeVersionChecktofalsein thevalues.yamlfile. - Manifest users: Remove the
--upgrade-version-checkflag from the deployment manifest.
- Helm users: Set
-
Update the
longhorn-managerimage- Change the image tag from
v1.9.0tov1.9.0-hotfix-1in the appropriate file:- For Helm: Update
values.yaml - For manifests: Update the deployment manifest directly.
- For Helm: Update
- Change the image tag from
-
Proceed with the upgrade
- Apply the changes using your standard Helm upgrade command or reapply the updated manifest.
Removal
Environment Check Script
The environment_check.sh script, deprecated in v1.7.0, has been removed in v1.9.0. Use the Longhorn Command Line Tool (longhornctl) to check your environment for potential issues.
Orphan Resource Auto-Deletion
The orphan-auto-deletion setting has been replaced by orphan-resource-auto-deletion in v1.9.0. To replicate the previous behavior, include replica-data in the orphan-resource-auto-deletion value. During the upgrade, the original orphan-auto-deletion setting is automatically migrated.
For more information, see Orphan Data Cleanup.
Deprecated Fields in longhorn.io/v1beta2 CRDs
Deprecated fields have been removed from the CRDs. For details, see issue #6684.
Deprecation & Incompatibilities
longhorn.io/v1beta1 API
The v1beta1 version of the Longhorn API is marked unserved and unsupported in v1.9.0 and will be removed in v1.10.0.
For more details, see Issue #10250.
Breaking Change in V2 Backing Image
Starting with Longhorn v1.9.0, V2 backing images are incompatible with earlier versions due to naming conflicts in the extended attributes (xattrs) used by SPDK backing image logical volumes. As a result, V2 backing images must be deleted and recreated during the upgrade process. Since backing images cannot be deleted while volumes using them still exist, you must first back up, delete, and later restore those volumes as the following steps:
- Before upgrading to v1.9.0:
- Verify that backup targets are functioning properly.
- Create full backups of all volumes that use a V2 backing image.
- Detach and delete these volumes after the backups complete.
- In the Backing Image page, save the specifications of all V2 backing images, including the name and the image source.
- Delete all V2 backing images.
- After upgrading:
- Recreate the V2 backing images using the same names and image sources.
- Restore the volumes from your backups.
For more details, see Issue #10805.
Primary Highlights
New V2 Data Engine Features
While the V2 Data Engine remains experimental in this release, several core functions have been significantly improved:
- Support UBLK Frontend: Support for UBLK frontend in the V2 Data Engine, which allows for better performance and resource utilization.
- Storage Network: Introduces support for storage networks in the V2 Data Engine to allow network segregation.
- Offline Replica Rebuilding: Support for offline replica rebuilding, which allows degraded volumes to automatically recover replicas even while the volume is detached. This capability ensures high data availability without manual intervention.
Recurring System Backup
Starting with Longhorn v1.9.0, you can create a recurring job for system backup creation.
Offline Replica Rebuilding
Longhorn introduces offline replica rebuilding, a feature that allows degraded volumes to automatically recover replicas even while the volume is detached. This capability minimizes the need for manual recovery steps, accelerates restoration, and ensures high data availability. By default, offline replica rebuilding is disabled. To enable it, set the offline-replica-rebuilding setting to true in the Longhorn UI or CLI.
Orphaned Instance Deletion
Longhorn can now track and remove orphaned instances, which are leftover resources like replicas or engines that are no longer associated with an active volume. These instances may accumulate due to unexpected failures or incomplete cleanup.
To reduce resource usage and maintain system performance, Longhorn supports both automatic and manual cleanup. By default, this feature is disabled. To enable it, set the orphan-resource-auto-deletion setting to instance in the Longhorn UI or CLI.
Improved Metrics for Replica, Engine, and Rebuild Status
Longhorn improves observability with new Prometheus metrics that expose the status and identity of Replica and Engine CRs, along with rebuild activity. These metrics make it easier to monitor rebuilds across the cluster.
For more information, see #10550 and #10722.
Installation
[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before installing Longhorn v1.9.0.
You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.
Upgrade
[!IMPORTANT] Ensure that your cluster is running Kubernetes v1.25 or later before upgrading from Longhorn v1.8.x to v1.9.0.
Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.
Post-Release Known Issues
For information about issues identified after this release, see Release-Known-Issues.
Highlight
- [FEATURE] Cleanup orphaned volume runtime resources if the v1 resources already deleted 6764 - @COLDTURNIP @chriscchien
- [FEATURE] v2 volume supports UBLK frontend 9456 - @PhanLe1010 @chriscchien
- [FEATURE] V1 and V2 volume offline replica rebuilding 8443 - @mantissahz @roger-ryao
- [TASK] Migrate v1beta1 CR to v1beta2 10250 - @COLDTURNIP @roger-ryao
- [FEATURE] Storage network with V2 data engine 6450 - @c3y1huang @roger-ryao
- [FEATURE] Recurring system backup 6534 - @yangchiu @c3y1huang
Feature
- [FEATURE] Delta Replica Rebuilding using Delta Snapshot: SPDK API Development 10799 - @yangchiu @DamiaSan
- [FEATURE] Running replicas field in volume table 10817 - @xelab04 @roger-ryao
- [FEATURE] Longhorn UI supports orphaned instance CRs management 10760 - @yangchiu @houhoucoop
- [FEATURE] Allow auto deleting snapshot when a backup is created from that snapshot. 9213 - @yangchiu @mantissahz
- [FEATURE] Add missing metrics of number of volumes/replicas by node/cluster 7599 - @c3y1huang @roger-ryao
Improvement
- [IMPROVEMENT] Remove unnecessary lasso dependency 10856 - @derekbit @chriscchien
- [IMPROVEMENT] Configurable wait interval for orphaned instance CR creation 10904 - @derekbit @chriscchien
- [IMPROVEMENT] Prevent creating orphans while deleting the v1 instances 10888 - @COLDTURNIP @chriscchien
- [IMPROVEMENT] prevent false logs from deleting volume with offline rebuilding is disabled. 10889 - @mantissahz @chriscchien
- [IMPROVEMENT] Add Prometheus metrics for Replica and Engine CRs 10722 - @hookak @chriscchien
- [IMPROVEMENT] Export longhorn engine rebuild status as prometheus metrics 10550 - @hookak @chriscchien
- [IMPROVEMENT] Disable Snapshot Checksum Calculation for Single-Replica V1 Volume 10518 - @derekbit @chriscchien
- [IMPROVEMENT] add extraObject in charts 10835 - @DrummyFloyd @chriscchien
- [IMPROVEMENT] Disable the v2 snapshot hashing while it is being deleted 10563 - @shuo-wu @roger-ryao
- [IMPROVEMENT] v2 checksum calculation and update should follow the v1 flow 10480 - @shuo-wu @roger-ryao
- [IMPROVEMENT] add strict field validation to the update option in upgrade path 10644 - @ChanYiLin @chriscchien
- [IMPROVEMENT] Show snapshot size during in-progress backup 9783 - @yangchiu @houhoucoop
- [IMPROVEMENT] Don’t synchronize all filesystem before snapshotting a v2 volume 9023 - @yangchiu @DamiaSan
- [IMPROVEMENT] spdk_tgt can cancel lvol checksum calculation while there is high priority task 10421 - @yangchiu @DamiaSan
- [IMPROVEMENT] Lvol is not force-removed if Blob is busy 10474 - @yangchiu @DamiaSan
- [IMPROVEMENT] Remove deprecated fields from CRDs 6684 - @derekbit @roger-ryao
- [IMPROVEMENT] Longhorn CLI fails to recognize Raspbian OS 10676 - @bachmanity1 @roger-ryao
- [IMPROVEMENT] Reduce auto balancing logging noise for detached volumes 10691 - @dihmandrake @roger-ryao
- [IMPROVEMENT] Remove the upper bound of v2-data-engine-guaranteed-instance-manager-cpu 10662 - @derekbit @roger-ryao
- [IMPROVEMENT] Clean up BackupTarget condition message handling 8224 - @chriscchien @houhoucoop
- [IMPROVEMENT] Longhorn CLI supports SLES micro 9256 - @yangchiu @DamiaSan
- [IMPROVEMENT] Allow
volumeBindingModeto be set from helm values 10592 - @ruant @roger-ryao - [DOC] Prepare a knowledge base for backing image trouble shooting during upgrade 10590 - @ChanYiLin @chriscchien
- [IMPROVEMENT] Missing Prometheus Metrics for Engine v2 Volumes 10472 - @hookak @roger-ryao
- [IMPROVEMENT] Create Volume UI improvement, Automatically Filter Backing Image Based on
v1orv2Selection 10086 - @houhoucoop @roger-ryao - [UI][IMPROVEMENT] Improve the Warning Message When Failed to Remove
Block-Type Disks 10580 - @houhoucoop @roger-ryao - [IMPROVEMENT] Pass full backup mode option to CSI volume snapshot type backup 9785 - @ChanYiLin @roger-ryao
- [UI][IMPROVEMENT] Clean up BackupTarget condition message handling 10579 - @houhoucoop
- [IMPROVEMENT] Improve the Warning Message When Failed to Remove
Block-Type Disks 10522 - @yangchiu @ChanYiLin - [IMPROVEMENT] Move
SettingNameV2DataEngineHugepageLimitto danger zone settings 7746 - @derekbit @chriscchien - [IMPROVEMENT] Include the /proc/mounts file and multipath.config in the support-bundle 6754 - @c3y1huang @roger-ryao
- [IMPROVEMENT] Use code-generator/kube_codegen.sh to generate K8s stubs and CRDs 7944 - @derekbit @chriscchien
- [IMPROVEMENT] CRD & API code generator decouple from Go conventional source path 10556 - @COLDTURNIP
- [IMPROVEMENT] Support configurable upgrade-responder URL 10437 - @derekbit @roger-ryao
- [IMPROVEMENT] Settings change validation should go back to using Volume state to determine “are all volumes detached” 10233 - @yangchiu @james-munson
- [IMPROVEMENT] Improve the UX of updating danger zone settings 8070 - @yangchiu @mantissahz
- [UI][FEATURE] V1 and V2 volume offline replica rebuilding 10581 - @houhoucoop @roger-ryao
- [UI][FEATURE] Recurring system backup 10262 - @yangchiu @houhoucoop
Bug
- [BUG] Error on git checkout in a container 10621 - @derekbit @chriscchien
- [BUG] Test case
test_snapshot_prune_and_coalesce_simultaneously_with_backing_imagefails 10808 - @yangchiu @c3y1huang - [BUG] Failed to terminate namespace
longhorn-systemif there is a support bundleReadyForDownload10731 - @yangchiu @c3y1huang - [BUG] Helm persistence.backupTargetName not referenced in storageclass template 10961 - @yangchiu @mantissahz
- [BUG] SPDK API lvol_get_snapshot_range_checksums cannot get the correct result 10950 - @shuo-wu @roger-ryao
- [BUG] V2 Backing image not ready after upgrade from v1.8.1 to v1.9.x 10805 - @COLDTURNIP @chriscchien
- [BUG] v2 replica rebuilding will miss the backing image 10909 - @shuo-wu @chriscchien
- [BUG] minor spacing issues found with yamllint in the helm chart 10681 - @codekow
- [BUG][UI] Snapshots of v2 volume with backing image aren’t shown on the
Snapshots and Backupsgraph 10526 - @derekbit @chriscchien - [BUG] Deleted orphan data still renders on the page until page refresh 10803 - @COLDTURNIP @chriscchien @houhoucoop
- [BUG] DR volume does not sync with latest backup when activation 10824 - @c3y1huang @chriscchien
- [BUG][v1.9.0-rc1] Block disks become temporarily unavailable after the upgrading from
v1.8.1tov1.9.0-rc110821 - @mantissahz - [BUG] Naming collision when creating the name of the new backing image manager 10616 - @yangchiu @ChanYiLin
- [BUG] v2 volume replica status
errorafter snapshot deletion withImmediate Data Integrity CheckEnabled 10798 - @shuo-wu - [BUG] Enabling
V2 Data Enginesetting, v2 instance manager doesn’t start after certain negative factor operations 10791 - @COLDTURNIP @yangchiu - [BUG] spdk emits
Device or resource busywhile registering lvol checksum calculation 10140 - @shuo-wu @roger-ryao - [BUG] v2 instance managers keep crashing on master-head arm64 environment 10768 - @yangchiu @PhanLe1010
- [BUG] Wrong image name in
longhorn-images.txt10774 - @c3y1huang - [BUG] After node down and force delete the terminating deployment pod, volume can not attach success 10689 - @c3y1huang @chriscchien
- [BUG] Deleting a replica of one v2 volume will also degrade the other v2 volume 10527 - @yangchiu @ChanYiLin
- [BUG] Adding a non-existing disk to a node will cause the longhorn-manager to crash 10749 - @ChanYiLin @roger-ryao
- [BUG] Upgrading Longhorn from v1.8.1 to master-head causes longhorn-manager to crash 10762 - @yangchiu @mantissahz
- [BUG] After node rebooted and workload pod restarted, pod data size became 0, and the mounted volume turned read-only 9248 - @yangchiu @c3y1huang
- [BUG] Test case
test_engine_crash_during_live_upgradefailed due to data loss 10751 - @c3y1huang @roger-ryao - [BUG] System backup could get stuck in
CreatingBackingImageBackupsindefinitely 10740 - @yangchiu @ChanYiLin - [BUG] v2 volume with backing image gets stuck in
Attachingstate 10743 - @yangchiu @ChanYiLin - [BUG] Can NOT delete an oversized Not Ready volume 10741 - @WebberHuang1118 @chriscchien
- [BUG][UI] Bulk backup creation with a detached volume returns error 405 and error messages show in browser console 10460 - @yangchiu @a110605
- [BUG]
spdk_tgtencounteredLvol store removed with error: -16inlonghorn-spdk-helperduring a CI test 10622 - @derekbit @roger-ryao - [BUG] 2 uninstall pods could be created after uninstall job was created, one failed with
deleting-confirmation-flag is set to falseerror, while the other completed successfully 10483 - @yangchiu @derekbit - [BUG] SPDK constantly emits “Bad length of checksum xattr” 10399 - @ChanYiLin @chriscchien
- [BUG] Can’t create v2 block-type disk via BDF on Talos 10313 - @derekbit @roger-ryao
- [BUG] I/O errors on Longhorn v1.7.2 volume during VM migration while upgrading Harvester v1.4.1 10495 - @derekbit @roger-ryao
- [BUG] MultiUnmapper floods logs with warnings about size mismatch. 6406 - @shuo-wu @roger-ryao
- [BUG]
spdk_tgtsegfaulted inlonghorn-spdk-helperduring a CI test run. 10598 - @derekbit @roger-ryao - [BUG] Backup Execution Timeout setting issue in Helm chart 10323 - @yangchiu @james-munson
- [BUG] Longhorn Volume Encryption Not Working in Talos 1.9.x 10584 - @c3y1huang @roger-ryao
- [BUG][UI] Inconsistent capitalization in the
Allow snapshots removal during trimvolume setting 10470 - @yangchiu @houhoucoop - [BUG] Expand Volume option is greyed under Volume tab but working in the volume detail section. 7529 - @yangchiu @houhoucoop
- [BUG] Instance manager image build fail 10653 - @shuo-wu
- [BUG] Recurring job pod stuck in pending state and unable to create new snapshots after node reboot 7956 - @c3y1huang @chriscchien
- [BUG] Instance manager pod stuck in Terminating state after upgrade with v2 backing image 10520 - @ChanYiLin @chriscchien
- [BUG] Extra replica created when create volume in a engine image not fully deployed environment 8263 - @c3y1huang @chriscchien
- [BUG] [v1.8.0-rc1] Uninstallation fail if having backing images, the instance-manager pod stuck at terminating 10044 - @ChanYiLin @chriscchien
- [BUG] test_statefulset_restore fails on integration test run with NFS backup store 3451 - @roger-ryao
- [BUG] csi keeps creating backup if the backup target is unavailable 10501 - @mantissahz @roger-ryao
- [BUG] nil pointer when the backing image copy is delete from the spec but also gets evicted at the same time 10464 - @yangchiu @ChanYiLin
- [BUG] Mutex is copied in getLatestBackup 6965 - @james-munson @roger-ryao
- [BUG] integer divide by zero in replica scheduler 10502 - @c3y1huang @chriscchien
- [BUG] Leading or trailing spaces in Longhorn UI break search 10491 - @houhoucoop @roger-ryao
- [BUG] When replica rebuilding completed, the progress could be 99 instead of 100 8589 - @shuo-wu @chriscchien
- [BUG] Workload with RWX volume cannot recover when Kubelet restarts 2933 - @james-munson @chriscchien
- [BUG][UI] Backup store setting doesn’t apply to the cloned volume 10463 - @yangchiu @mantissahz
- [BUG] Data lost caused by Longhorn CSI plugin doing a wrong filesystem format action in a rare race condition 10416 - @yangchiu @PhanLe1010
- [BUG] WebUI Volumes Disappear and Reappear 10314 - @yangchiu @PhanLe1010 @houhoucoop
- [BUG] Longhorn-manager logs “Failed to sync backup status” on every backup 10301 - @derekbit @chriscchien
- [BUG] Rebuilding stuck for DR volume if the node was power down while restoring 2747 - @COLDTURNIP @roger-ryao
- [BUG] Uninstalling K3s without uninstalling Longhorn first when Longhorn SPDK volumes exist hangs on arm64 8132 - @roger-ryao
- [BUG] A V2 volume checksum will change after replica rebuilding if the volume created with backing image 10340 - @shuo-wu @chriscchien
- [BUG] RWX volume becomes faulted after the node reconnects 5658 - @james-munson @chriscchien
- [BUG] V2 BackingImage failed after node reboot 10342 - @ChanYiLin @chriscchien
- [BUG] degraded v2 volume doesn’t create new replica even though there is an available disk 9197 - @c3y1huang
- [BUG] Bug in snapshot count enforcement cause volume faulted and stuck in detaching/attaching loop 10308 - @PhanLe1010 @roger-ryao
- [BUG] Test case
test_csi_mount_volume_online_expansionis failing due to unable to expand PVC 10411 - @yangchiu @c3y1huang - [BUG] Longhorn CSI plugin 1.8.0 crashes consistently when trying to create a snapshot 10303 - @yangchiu @PhanLe1010
- [BUG] Workload pod will not be able to move to new node when backup operation is taking a long time 10171 - @yangchiu @PhanLe1010
- [BUG] v2 engine stuck in detaching-attaching loop if the previous replica is not cleaned up correct 10293 - @yangchiu @shuo-wu
- [BUG] [UI] ‘Create’ button on the System Backup page is disabled after reloading page 10351 - @yangchiu @houhoucoop
- [BUG] “Error get size” from “metrics_collector.(*BackupCollector).Collect” on every metric scrape 10358 - @derekbit @chriscchien
- [BUG] Proxy gRPC API ReplicaList returns different output formats for v1 and v2 volumes 10347 - @shuo-wu @roger-ryao
- [BUG] Engine stuck in “stopped” state, prevent volume attach 9938 - @ChanYiLin @roger-ryao
- [BUG] After upgrading to v1.8.0 the version number lost on the web-ui 10336 - @derekbit
- [BUG] constant attaching/reattaching of volumes after upgrading to 1.8 10304 - @PhanLe1010
- [BUG] Sometimes attached DR volume checksum fluctuates 9305 - @c3y1huang
- [BUG] Backing image manager pods unable to come up on RHEL 8.4 2767 - @roger-ryao
- [BUG] Failed to upgrade Longhorn from
v1.8.x-headtomaster-head10143 - @roger-ryao - [BUG] [v1.5.4-rc2] V2 volume perform engine upgrade when concurrent-automatic-engine-upgrade-per-node-limit > 0 7930 - @derekbit
- [BUG] Negative test case got stuck in waiting for longhorn-ui pods 8248 - @c3y1huang
Misc
- [DOC] Replica count behaviour is unclear. 10861 - @hoo29 @chriscchien
- [DOC] Update “Upgrade Path Enforcement and Downgrade Prevention” 10945 - @derekbit @roger-ryao
- [TASK] Update the longhornio/nfs-ganesha image 10878 - @derekbit @c3y1huang @chriscchien
- [DOC] No
BackupTargetSecretinSettings/General10858 - @vnwnv @roger-ryao - [DOC] Create system backup first before upgrade system 10633 - @ChanYiLin @chriscchien
- [TASK] [UI] [FEATURE] v2 volume supports UBLK frontend 10735 - @chriscchien @houhoucoop
- [DOC] Adding support for RKE2/k3s in data-recovery steps 10714 - @mattmattox @roger-ryao
- [DOC] Architecture Diagram 6761 - @derekbit @chriscchien
- [TASK] Fix longhorn/website to support latest Hugo server version 10632 - @chriscchien @sushant-suse
- [TASK] fix lint problems in longhorn-manager 10639 - @COLDTURNIP @chriscchien
- [DOC] Codeblocks in KB don’t line wrap 8143 - @roger-ryao @sushant-suse
- [TASK] Longhorn UI assessment for ui-extension migration 10487 - @houhoucoop
- [DOC] Update the steps to set up the Azure backup target 9688 - @mantissahz
- [DOC] Explain the process of creating backing images from existing volumes 10093 - @ChanYiLin @chriscchien
- [DOC] Update ArgoCD installation document 10588 - @mantissahz
- [TASK] Remove environment check script since v1.9.0 9239 - @yangchiu @derekbit
- [TASK] Add platform architect and volume encryption info to metrics 7047 - @c3y1huang @roger-ryao
- [DOC] Update the document and create a KB to address the limitation that BackingImage should be multiple of 512B 10536 - @ChanYiLin
- [Doc] Clarifications on defaultSettings.defaultDataLocality and persistence.defaultDataLocality usage 10253 - @james-munson @roger-ryao
New Contributors
- @bachmanity1
- @codekow
- @DrummyFloyd
- @hoo29
- @hookak
- @dihmandrake
- @mattmattox
- @ruant
- @vnwnv
- @xelab04
Contributors
- @COLDTURNIP
- @ChanYiLin
- @DamiaSan
- @PhanLe1010
- @WebberHuang1118
- @a110605
- @c3y1huang
- @chriscchien
- @derekbit
- @houhoucoop
- @innobead
- @james-munson
- @mantissahz
- @roger-ryao
- @shuo-wu
- @yangchiu
- @sushant-suse
- @jillian-maroket
- @rebeccazzzz
- @forbesguthrie
- @asettle