[Lustre-discuss] Can't use any lfs commands on clients...

Wed Jun 16 07:04:01 PDT 2010

Dear All,

I have just successfully setup a Lustre enviroment. Every thing seems 
just A-OK so far. But...
After completing some  performance test with iozone and now wanting to 
see the performance difference with striping enabled I noticed that I 
cant use any lfs commands on the client side.
Digging further, I noticed lctl is not showing any nids, on the client
I went through the logs on the OSS'es, nothing noteworthy in there, so 
the only loggin I have is coming from the client, and is not very 
helpful to me so I'm hoping some one here can point at some flaw - or at 
least provide pointers on what to check...I guessing that the fact im 
not seeing any nids is already a bad start, but I don't know how I can 
add one (the nid is listed on the mgs/oss though..)

Any help would be appreciated!

Leen

Outputs (if more info is needed I'll be glad to provide it!)
Client:~# mount
..
192.168.21.32 at tcp0:192.168.21.33 at tcp:/datafs on /data type lustre 
(rw,_netdev,noatime)

Client:~# lctl device_list
   0 UP mgc MGC192.168.21.32 at tcp 2367fdc6-ef75-f1fc-b1cb-611c46a0ced3 5
   1 UP lov datafs-clilov-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 4
   2 UP mdc datafs-MDT0000-mdc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   3 UP osc datafs-OST0000-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   4 UP osc datafs-OST0001-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   5 UP osc datafs-OST0002-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   6 UP osc datafs-OST0003-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   7 UP osc datafs-OST0004-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   8 UP osc datafs-OST0005-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
   9 UP osc datafs-OST0006-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
  10 UP osc datafs-OST0007-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
  11 UP osc datafs-OST0008-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5
  12 UP osc datafs-OST0009-osc-ffff810037802400 
951384b1-8456-c2f3-575e-4926f70919c9 5

Client:~# lctl list_nids
IOC_LIBCFS_GET_NI error 22: Invalid argument

Client:~# lctl list_nids
IOC_LIBCFS_GET_NI error 22: Invalid argument
prd-as-022:~# lfs
lfs > osts
error: can't get lov name: /data: Invalid argument (22)
lfs > df
UUID                 1K-blocks      Used Available  Use% Mounted on
error: llapi_obd_statfs(): Invalid argument (-22)
lfs > getstripe /data
error: can't get lov name: /data: Invalid argument (22)
lfs >

syslog excerpt:
Jun 16 15:26:49 prd-as-022 kernel: Lustre: MGC192.168.21.32 at tcp: 
Reactivating import
Jun 16 15:26:49 prd-as-022 kernel: Lustre: Client datafs-client has started
Jun 16 15:26:57 prd-as-022 kernel: ioctl32(lfs:4645): Unknown cmd fd(4) 
cmd(c00466a4){t:'f';sz:4} arg(ffb0a34c) on /data
Jun 16 15:27:54 prd-as-022 kernel: ioctl32(lfs:4646): Unknown cmd fd(4) 
cmd(c00466a4){t:'f';sz:4} arg(ff90294c) on /data
Jun 16 15:28:44 prd-as-022 kernel: ioctl32(lfs:4648): Unknown cmd fd(4) 
cmd(c00466a4){t:'f';sz:4} arg(ffd925dc) on /data
Jun 16 15:29:00 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(4) 
cmd(c00466a4){t:'f';sz:4} arg(fff32f2c) on /data
Jun 16 15:29:38 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(3) 
cmd(c0046683){t:'f';sz:4} arg(fff37190) on /data
Jun 16 15:29:46 prd-as-022 kernel: ioctl32(lfs:4649): Unknown cmd fd(3) 
cmd(c0046683){t:'f';sz:4} arg(fff37190) on /data
Jun 16 15:31:07 prd-as-022 kernel: ioctl32(lfs:4650): Unknown cmd fd(4) 
cmd(c0046683){t:'f';sz:4} arg(ffcb6750) on /data
Jun 16 15:48:17 prd-as-022 kernel: ioctl32(lctl:4672): Unknown cmd fd(3) 
cmd(c0046532){t:'e';sz:4} arg(fffd23ec) on /dev/lnet
Jun 16 15:51:48 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(4) 
cmd(c0046683){t:'f';sz:4} arg(ffc556f0) on /data
Jun 16 15:51:52 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(4) 
cmd(c00466a4){t:'f';sz:4} arg(ffc5144c) on /data
Jun 16 15:52:04 prd-as-022 kernel: ioctl32(lfs:4674): Unknown cmd fd(3) 
cmd(c0046683){t:'f';sz:4} arg(ffc556b0) on /data

OST's created with
fs-ost-002:~#  mkfs.lustre --ost --mgsnode=192.168.21.32 at tcp 
--mgsnode=192.168.21.33 at tcp --failnode=192.168.21.34 at tcp  
--fsname=datafs /dev/VG1/ost1

    Permanent disk data:
Target:     datafs-OSTffff
Index:      unassigned
Lustre FS:  datafs
Mount type: ldiskfs
Flags:      0x72
               (OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc
Parameters: mgsnode=192.168.21.32 at tcp mgsnode=192.168.21.33 at tcp 
failover.node=192.168.21.34 at tcp

checking for existing Lustre data: not found
device size = 10240MB
2 6 22
WARNING: The e2fsprogs package currently installed on your system does 
not support "mmp" feature.
Please install the latest version of e2fsprogs from 
http://www.clusterfs.com/downloads/public/Lustre/Tools/e2fsprogs/
to enable this feature.
Feature will not be enabled until e2fsprogs is updated and 'tune2fs -O 
mmp %{device}' is run.

formatting backing filesystem ldiskfs on /dev/VG1/ost1
         target name  datafs-OSTffff
         4k blocks     0
         options        -J size=400 -i 16384 -I 256 -q -O 
dir_index,uninit_groups -F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L datafs-OSTffff  -J size=400 -i 16384 
-I 256 -q -O dir_index,uninit_groups -F /dev/VG1/ost1
Writing CONFIGS/mountdata

On OSS:
fs-ost-001:~# lctl list_nids
192.168.21.34 at tcp
fs-mgs-001:~# lctl list_nids
192.168.21.32 at tcp