[Lustre-discuss] How to re-activate "inactive device" on client

peter.chiu at stfc.ac.uk peter.chiu at stfc.ac.uk
Fri Dec 16 05:05:59 PST 2011


Dear all,

We have an odd problem on a lustre client in that a couple of OST elements have become "inactive".

However, lctl dl shows all devices are in "UP" state.

The "UP" status is also echoed on the mds as well as on another client - please see below.

I have tried rebooting this client, but with no joy.


I have also searched around the discussion list, and can find a few others had experienced similar problems before.

But unfortunatley none of them has reported back with a good recovery procedure, other than a complete reformatting of the lustre storage.

So can I trouble you experts to give me some clue as to how to recover from this?

Many thanks.

Regards,
Peter

PS. Client: SLES 11 with 2.6.32.29 kernel, Lustre 1.8.5

===================================================================================

Client Capuchin:

capuchin:~ # lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
ceda3-MDT0000_UUID        350.0G      689.2M      329.3G   0% /disks/ceda3[MDT:0]
ceda3-OST0000_UUID          7.2T        6.8T        6.5G  94% /disks/ceda3[OST:0]
ceda3-OST0001_UUID          7.2T        6.8T        6.5G  94% /disks/ceda3[OST:1]
ceda3-OST0002_UUID  : inactive device
ceda3-OST0003_UUID  : inactive device
ceda3-OST0004_UUID          7.2T        6.8T        6.6G  94% /disks/ceda3[OST:4]
ceda3-OST0005_UUID          7.2T        6.8T        6.7G  94% /disks/ceda3[OST:5]
ceda3-OST0006_UUID          7.2T        6.8T        6.2G  94% /disks/ceda3[OST:6]

filesystem summary:        35.8T       34.0T       32.5G  94% /disks/ceda3

capuchin:~ # lctl dl  
  0 UP mgc MGC130.246.191.64 at tcp be6c0da2-275f-c59d-fa3a-c792134a0d9d 5
  1 UP lov ceda3-clilov-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 4
  2 UP mdc ceda3-MDT0000-mdc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  3 UP osc ceda3-OST0000-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  4 UP osc ceda3-OST0001-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  5 UP osc ceda3-OST0002-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  6 UP osc ceda3-OST0003-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  7 UP osc ceda3-OST0004-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  8 UP osc ceda3-OST0005-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
  9 UP osc ceda3-OST0006-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5
capuchin:~ #

Client2 cmip-dn1:
[root at aspre script]# ssh cmip-dn1 lfs df -h
UUID                       bytes        Used   Available Use% Mounted on
ceda3-MDT0000_UUID        350.0G      689.2M      329.3G   0% /disks/ceda3[MDT:0]
ceda3-OST0000_UUID          7.2T        6.8T        6.5G  94% /disks/ceda3[OST:0]
ceda3-OST0001_UUID          7.2T        6.8T        6.5G  94% /disks/ceda3[OST:1]
ceda3-OST0002_UUID          7.2T      451.6M        6.8T   0% /disks/ceda3[OST:2]
ceda3-OST0003_UUID          7.2T      451.8M        6.8T   0% /disks/ceda3[OST:3]
ceda3-OST0004_UUID          7.2T        6.8T        6.6G  94% /disks/ceda3[OST:4]
ceda3-OST0005_UUID          7.2T        6.8T        6.7G  94% /disks/ceda3[OST:5]
ceda3-OST0006_UUID          7.2T        6.8T        6.2G  94% /disks/ceda3[OST:6]

filesystem summary:        50.1T       34.0T       13.6T  67% /disks/ceda3


Mds:
[root at mds02 ~]# lctl dl
  0 UP mgc MGC130.246.191.64 at tcp 8aa29420-11f1-8ca3-a361-ce5135a09be2 5
  1 UP mdt MDS MDS_uuid 3
  2 UP lov ceda3-mdtlov ceda3-mdtlov_UUID 4
  3 UP mds ceda3-MDT0000 ceda3-MDT0000_UUID 9
  4 UP osc ceda3-OST0000-osc ceda3-mdtlov_UUID 5
  5 UP osc ceda3-OST0001-osc ceda3-mdtlov_UUID 5
  6 UP osc ceda3-OST0002-osc ceda3-mdtlov_UUID 5
  7 UP osc ceda3-OST0003-osc ceda3-mdtlov_UUID 5
  8 UP osc ceda3-OST0004-osc ceda3-mdtlov_UUID 5
  9 UP osc ceda3-OST0005-osc ceda3-mdtlov_UUID 5
 10 UP osc ceda3-OST0006-osc ceda3-mdtlov_UUID 5
 11 UP lov ceda4-mdtlov ceda4-mdtlov_UUID 4
 12 UP mds ceda4-MDT0000 ceda4-MDT0000_UUID 9
 13 UP osc ceda4-OST0000-osc ceda4-mdtlov_UUID 5
 14 UP osc ceda4-OST0001-osc ceda4-mdtlov_UUID 5
 15 UP osc ceda4-OST0002-osc ceda4-mdtlov_UUID 5
 16 UP osc ceda4-OST0003-osc ceda4-mdtlov_UUID 5
 17 UP osc ceda4-OST0004-osc ceda4-mdtlov_UUID 5
 18 UP osc ceda4-OST0005-osc ceda4-mdtlov_UUID 5
[root at mds02 ~]#

-- 
Scanned by iCritical.



More information about the lustre-discuss mailing list