[lustre-discuss] upgrading zfs back end file system lustre doesn't mount anymore

[Bob Ball]

I had the same issue.  I could not find/get an answer, posted to this 
same list about 1mo ago.  I ended up down-grading to the same zfs I had 
previously used, ie, 0.6.3.1 with Lustre 2.7.0

Maybe an answer has come round in the intervening month?  But, it would 
appear that if you use zfs, you are very heavily tied to the version of 
Lustre built with that zfs.

bob

On 6/8/2015 3:41 PM, Kurt Strosahl wrote:
> Hello,
>
>     I recently encountered an issue with zfs on linux that required me to upgrade to a newer version.  Now I'm trying to patch my oss systems.  I've got a new oss (has never had its osts mounted) that I'm using to test the upgrade process, but now that I've installed the new zfs modules it won't mount the osts...
>
> I'm running CentOS 6.5
> rpm -qa | grep lustre
> kernel-2.6.32-431.23.3.el6_lustre.x86_64
> lustre-osd-zfs-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64
> kernel-firmware-2.6.32-431.23.3.el6_lustre.x86_64
> lustre-modules-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64
> lustre-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64
> kernel-headers-2.6.32-431.23.3.el6_lustre.x86_64
> lustre-osd-ldiskfs-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64
>
> rpm -qa | grep zfs
> zfs-0.6.3-1.3.el6.x86_64
> lustre-osd-zfs-2.5.3-2.6.32_431.23.3.el6_lustre.x86_64.x86_64
> kmod-zfs-2.6.32-431.el6.x86_64-0.6.3-1.3.el6.x86_64
> libzfs2-0.6.3-1.3.el6.x86_64
>
> rpm -qa | grep spl
> kmod-spl-devel-2.6.32-431.el6.x86_64-0.6.3-1.3.el6.x86_64
> kmod-spl-2.6.32-431.el6.x86_64-0.6.3-1.3.el6.x86_64
> kmod-spl-devel-0.6.3-1.3.el6.x86_64
> spl-0.6.3-1.3.el6.x86_64
>
>
> When I do a /etc/init.d/lustre start it says that the lustre modules are not loaded... and when I examine the /var/log/messages I see the following for each ost:
>
> Lustre: Lustre: Build Version: 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
> osd_zfs: disagrees about version of symbol dsl_prop_register
> osd_zfs: Unknown symbol dsl_prop_register
> osd_zfs: disagrees about version of symbol zap_cursor_serialize
> osd_zfs: Unknown symbol zap_cursor_serialize
> osd_zfs: disagrees about version of symbol dmu_tx_hold_write
> osd_zfs: Unknown symbol dmu_tx_hold_write
> osd_zfs: disagrees about version of symbol dsl_prop_unregister
> osd_zfs: Unknown symbol dsl_prop_unregister
> osd_zfs: disagrees about version of symbol sa_spill_rele
> osd_zfs: Unknown symbol sa_spill_rele
> osd_zfs: disagrees about version of symbol zap_cursor_advance
> osd_zfs: Unknown symbol zap_cursor_advance
> osd_zfs: disagrees about version of symbol dmu_read
> osd_zfs: Unknown symbol dmu_read
> osd_zfs: disagrees about version of symbol zap_cursor_init_serialized
> osd_zfs: Unknown symbol zap_cursor_init_serialized
> osd_zfs: disagrees about version of symbol dmu_tx_assign
> osd_zfs: Unknown symbol dmu_tx_assign
> osd_zfs: disagrees about version of symbol sa_handle_get
> osd_zfs: Unknown symbol sa_handle_get
> osd_zfs: disagrees about version of symbol zap_cursor_fini
> osd_zfs: Unknown symbol zap_cursor_fini
> osd_zfs: disagrees about version of symbol zrl_add
> osd_zfs: Unknown symbol zrl_add
> osd_zfs: disagrees about version of symbol dmu_tx_create
> osd_zfs: Unknown symbol dmu_tx_create
> osd_zfs: disagrees about version of symbol sa_lookup
> osd_zfs: Unknown symbol sa_lookup
> osd_zfs: disagrees about version of symbol dmu_object_alloc
> osd_zfs: Unknown symbol dmu_object_alloc
> osd_zfs: disagrees about version of symbol dmu_object_free
> osd_zfs: Unknown symbol dmu_object_free
> osd_zfs: disagrees about version of symbol dmu_objset_own
> osd_zfs: Unknown symbol dmu_objset_own
> osd_zfs: disagrees about version of symbol __cv_broadcast
> osd_zfs: Unknown symbol __cv_broadcast
> osd_zfs: disagrees about version of symbol zap_cursor_init
> osd_zfs: Unknown symbol zap_cursor_init
> osd_zfs: disagrees about version of symbol dmu_write
> osd_zfs: Unknown symbol dmu_write
> osd_zfs: disagrees about version of symbol sa_bulk_lookup
> osd_zfs: Unknown symbol sa_bulk_lookup
> osd_zfs: disagrees about version of symbol dmu_return_arcbuf
> osd_zfs: Unknown symbol dmu_return_arcbuf
> osd_zfs: disagrees about version of symbol dmu_assign_arcbuf
> osd_zfs: Unknown symbol dmu_assign_arcbuf
> osd_zfs: disagrees about version of symbol sa_object_size
> osd_zfs: Unknown symbol sa_object_size
> osd_zfs: disagrees about version of symbol dmu_tx_callback_register
> osd_zfs: Unknown symbol dmu_tx_callback_register
> osd_zfs: disagrees about version of symbol sa_setup
> osd_zfs: Unknown symbol sa_setup
> osd_zfs: disagrees about version of symbol sa_update
> osd_zfs: Unknown symbol sa_update
> osd_zfs: disagrees about version of symbol dmu_objset_disown
> osd_zfs: Unknown symbol dmu_objset_disown
> osd_zfs: disagrees about version of symbol dmu_tx_hold_sa_create
> osd_zfs: Unknown symbol dmu_tx_hold_sa_create
> osd_zfs: disagrees about version of symbol dmu_objset_ds
> osd_zfs: Unknown symbol dmu_objset_ds
> osd_zfs: disagrees about version of symbol dmu_objset_spa
> osd_zfs: Unknown symbol dmu_objset_spa
> osd_zfs: disagrees about version of symbol spa_get_dsl
> osd_zfs: Unknown symbol spa_get_dsl
> osd_zfs: disagrees about version of symbol dmu_objset_userused_enabled
> osd_zfs: Unknown symbol dmu_objset_userused_enabled
> osd_zfs: disagrees about version of symbol dmu_tx_hold_zap
> osd_zfs: Unknown symbol dmu_tx_hold_zap
> osd_zfs: disagrees about version of symbol dmu_tx_get_txg
> osd_zfs: Unknown symbol dmu_tx_get_txg
> osd_zfs: disagrees about version of symbol dmu_request_arcbuf
> osd_zfs: Unknown symbol dmu_request_arcbuf
> osd_zfs: disagrees about version of symbol dmu_tx_commit
> osd_zfs: Unknown symbol dmu_tx_commit
> osd_zfs: disagrees about version of symbol txg_wait_callbacks
> osd_zfs: Unknown symbol txg_wait_callbacks
> osd_zfs: disagrees about version of symbol dmu_tx_hold_sa
> osd_zfs: Unknown symbol dmu_tx_hold_sa
> osd_zfs: disagrees about version of symbol dsl_pool_config_enter
> osd_zfs: Unknown symbol dsl_pool_config_enter
> osd_zfs: disagrees about version of symbol dbuf_hold_impl
> osd_zfs: Unknown symbol dbuf_hold_impl
> osd_zfs: disagrees about version of symbol dmu_objset_pool
> osd_zfs: Unknown symbol dmu_objset_pool
> osd_zfs: disagrees about version of symbol dmu_free_range
> osd_zfs: Unknown symbol dmu_free_range
> osd_zfs: disagrees about version of symbol sa_size
> osd_zfs: Unknown symbol sa_size
> osd_zfs: disagrees about version of symbol sa_buf_hold
> osd_zfs: Unknown symbol sa_buf_hold
> osd_zfs: disagrees about version of symbol sa_replace_all_by_template
> osd_zfs: Unknown symbol sa_replace_all_by_template
> osd_zfs: disagrees about version of symbol dmu_tx_hold_bonus
> osd_zfs: Unknown symbol dmu_tx_hold_bonus
> osd_zfs: disagrees about version of symbol spa_freeze
> osd_zfs: Unknown symbol spa_freeze
> osd_zfs: disagrees about version of symbol zap_cursor_retrieve
> osd_zfs: Unknown symbol zap_cursor_retrieve
> osd_zfs: disagrees about version of symbol zrl_remove
> osd_zfs: Unknown symbol zrl_remove
> osd_zfs: disagrees about version of symbol txg_wait_synced
> osd_zfs: Unknown symbol txg_wait_synced
> osd_zfs: disagrees about version of symbol dsl_pool_config_exit
> osd_zfs: Unknown symbol dsl_pool_config_exit
> osd_zfs: disagrees about version of symbol dmu_tx_abort
> osd_zfs: Unknown symbol dmu_tx_abort
> osd_zfs: disagrees about version of symbol sa_handle_destroy
> osd_zfs: Unknown symbol sa_handle_destroy
> osd_zfs: disagrees about version of symbol dmu_objset_space
> osd_zfs: Unknown symbol dmu_objset_space
> osd_zfs: disagrees about version of symbol dbuf_rele
> osd_zfs: Unknown symbol dbuf_rele
> osd_zfs: disagrees about version of symbol dmu_object_set_blocksize
> osd_zfs: Unknown symbol dmu_object_set_blocksize
> osd_zfs: disagrees about version of symbol sa_bulk_update
> osd_zfs: Unknown symbol sa_bulk_update
> osd_zfs: disagrees about version of symbol dmu_tx_hold_free
> osd_zfs: Unknown symbol dmu_tx_hold_free
> LustreError: 158-c: Can't load module 'osd-zfs'
> LustreError: 5581:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs
> LustreError: 5581:0:(obd_config.c:374:class_attach()) Cannot create device lustre2-OST0009-osd of type osd-zfs : -19
> LustreError: 5581:0:(obd_mount.c:195:lustre_start_simple()) lustre2-OST0009-osd attach error -19
> LustreError: 5581:0:(obd_mount_server.c:1689:server_fill_super()) Unable to start osd on lustre-ost9/ost9: -19
> LustreError: 5581:0:(obd_mount.c:1325:lustre_fill_super()) Unable to mount  (-19)
>
> So I tried reinstalling all the lustre rpms (including the kernel) but it still gives the above message.
>
> w/r,
> Kurt J. Strosahl
> System Administrator
> Scientific Computing Group, Thomas Jefferson National Accelerator Facility
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>