[Lustre-discuss] LustreError

Thu Jun 5 08:48:00 PDT 2014

Dear Experts,
We are running lustre 2.4.1 with a combined MDT/MGS disk-server mounted
with 4 device-mappers as 4 OSTs. Recently the setup suffered from high
system load and long hang when trying to lfs df -h from client.
could someone shed light on the situation?
Any help would be greatly appreciated

$ sudo tail -n 100 /var/log/messages
Jun  5 14:56:02 disk-server kernel: Lustre: lustre-OST0001: recovery is
timed out, evict stale exports
Jun  5 14:56:45 disk-server kernel: LustreError:
58949:0:(ldlm_resource.c:1165:ldlm_resource_get()) lustre-OST0001:
lvbo_init failed for resource 0xc4:0x0: rc = -2
Jun  5 14:57:27 disk-server kernel: Lustre: lustre-OST0001: Client
7e1e6422-c17b-841e-1703-36dda2141083 (at 192.168.1.61 at o2ib) reconnecting,
waiting for 4 clients in recovery for 0:42
Jun  5 14:59:54 disk-server kernel: Lustre: lustre-OST0000: Not available
for connect from 192.168.1.61 at o2ib (stopping)
Jun  5 15:02:00 disk-server kernel: LustreError: 11-0:
lustre-OST0000-osc-MDT0000: Communicating with 0 at lo, operation ost_connect
failed with -19.
Jun  5 15:02:17 disk-server kernel: LustreError:
4943:0:(ofd_obd.c:1338:ofd_create()) lustre-OST0002: unable to precreate:
rc = -30
Jun  5 15:02:17 disk-server kernel: LustreError:
39979:0:(osp_precreate.c:484:osp_precreate_send())
lustre-OST0002-osc-MDT0000: can't precreate: rc = -30
Jun  5 15:02:17 disk-server kernel: LustreError:
39979:0:(osp_precreate.c:484:osp_precreate_send())
lustre-OST0002-osc-MDT0000: can't precreate: rc = -30
Jun  5 15:02:17 disk-server kernel: LustreError:
39979:0:(osp_precreate.c:989:osp_precreate_thread())
lustre-OST0002-osc-MDT0000: cannot precreate objects: rc = -30
Jun  5 15:04:59 disk-server kernel: LustreError:
4950:0:(tgt_lastrcvd.c:577:tgt_client_new()) lustre-OST0002: Failed to
write client lcd at idx 2, rc -30
Jun  5 15:06:35 disk-server kernel: Lustre: lustre-OST0001: recovery is
timed out, evict stale exports
Jun  5 15:07:18 disk-server kernel: LustreError:
58949:0:(ldlm_resource.c:1165:ldlm_resource_get()) lustre-OST0001:
lvbo_init failed for resource 0xc4:0x0: rc = -2
Jun  5 15:08:00 disk-server kernel: Lustre: lustre-OST0001: Client
7e1e6422-c17b-841e-1703-36dda2141083 (at 192.168.1.61 at o2ib) reconnecting,
waiting for 4 clients in recovery for 0:42
Jun  5 15:09:54 disk-server kernel: Lustre: lustre-OST0000: Not available
for connect from 192.168.1.61 at o2ib (stopping)

some infromation
[user at disk-server]$ uname -r
2.6.32-358.18.1.el6_lustre.x86_64
[user at disk-server]$ rpm -qa|grep lustre
kernel-2.6.32-358.18.1.el6_lustre.x86_64
lustre-iokit-1.4.0-1.noarch
lustre-ldiskfs-4.1.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-debuginfo-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-tests-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-modules-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-ldiskfs-debuginfo-4.1.0-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.4.1-2.6.32_358.18.1.el6_lustre.x86_64.x86_64
kernel-firmware-2.6.32-358.18.1.el6_lustre.x86_64

[user at disk-server]$ lctl dl
  0 UP osd-ldiskfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 12
  1 UP mgc MGC192.168.1.254 at o2ib 48090e07-ab8e-54f2-0d40-1bf99a31c774 5
  2 UP ost OSS OSS_uuid 3
  3 UP mgs MGS MGS 13
  4 UP mgc MGC117.103.97.241 at tcp 9442eac9-407e-56fd-3b6b-91e3e39beb97 5
  5 UP mds MDS MDS_uuid 3
  6 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4
  7 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 17
  8 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4
  9 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4
 10 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5
 11 UP osd-ldiskfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5
 12 ST obdfilter lustre-OST0000 lustre-OST0000_UUID 5
 13 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5
 14 UP osp lustre-OST0002-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5
 15 UP osp lustre-OST0003-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5
 16 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5
 17 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5
 18 UP osd-ldiskfs lustre-OST0002-osd lustre-OST0002-osd_UUID 5
 19 UP obdfilter lustre-OST0002 lustre-OST0002_UUID 9
 20 UP lwp lustre-MDT0000-lwp-OST0002 lustre-MDT0000-lwp-OST0002_UUID 5
 21 UP osd-ldiskfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5
 22 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 11
 23 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5
 24 UP osd-ldiskfs lustre-OST0003-osd lustre-OST0003-osd_UUID 5
 25 UP obdfilter lustre-OST0003 lustre-OST0003_UUID 11
 26 UP lwp lustre-MDT0000-lwp-OST0003 lustre-MDT0000-lwp-OST0003_UUID 5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20140605/caec628f/attachment.htm>