[Lustre-discuss] Replace OST

Thu Mar 20 19:52:06 PDT 2014

On 2014/03/20, 3:56 AM, "Jon Tegner" <tegner at foi.se> wrote:

>Thanks a lot!
>
>Much appreciated!
>
>I'm using 2.4.2 on CentOS-6.5. Indeed it seems related to the issue
>Andreas links to. I did some further tests, and it seems that if I run
>the "find" command (or something like "ls -R /lusterMountPoint") BEFORE
>I unmount the OST the "find" command works after the OST has been
>unmounted. Related to some kind of caching?
>
>Tried to use "getstripe" instead (thanks!), with the command:
>
>"lfs getstripe --ost lustre-OST0001 -v -r /home | grep home | xargs -n 1
>unlink"
>
>(where /home is my luster mount point).

The "grep" is probably not very useful.  You could just get rid of "-v"
since every line will have "home" in it...  Hmm, I'm wrong.  Seems that
it still prints the striping info even if "-q" is used.  Need to fix that,
or fix "lfs find" to work with missing OSTs.

>By then following the procedure as indicated in the manual (thanks for
>pointing me to the latest!) I managed to introduce the "new" (in my case
>"reformated") OST in the file system.
>
>At the moment I'm just testing, and the amount of data on the system is
>very small, and I have no clue whether the way I combine "getstripe"
>with "xargs" an "unlink" is an efficient way of doing it?

Seems OK.

>Another thing I didn't understand was that after "reformating" the OST,
>and mounting it, "df" indicated that the file system was growing. Before
>mounting the "failed" OST it seemed as if about half of the space was
>gone (reasonable, since I'm playing around with two OSTs). However,
>after a while it seemed as if the file system was automagically
>repopulated. But not with ALL of the files (when compared with my
>"backup").  But maybe this could be attributed to confusion on my part...

This is a bug in 2.4 and earlier.  If the reformatted OST is returned
to the same index without restoring the "/O/0/LAST_ID" file, it will
frantically try to create the missing objects (without any data, Lustre
does not yet have data redundancy).  That was fixed in 2.5 and later:
http://review.whamcloud.com/7443

Cheers, Andreas

>Anyway, thanks again for your help (realizing that I can get feedback
>from this list will make it less scary to move over to lustre)!
>
>Regards,
>
>/jon
>
>On 03/19/2014 08:51 PM, Dilger, Andreas wrote:
>> On 2014/03/19, 9:35 AM, "Lee, Brett" <brett.lee at intel.com> wrote:
>>
>>> Hi Jon,
>>>
>>> Looks like you are jumping in with both feet (new to Lustre and trying
>>>to
>>> replace an OST).  Pretty ambitious... ;)
>>>
>> >From what I can tell you are following section 14.8.3 in the latest
>>> Lustre manual:
>>>
>>> 14.8.3.  Removing an OST from the File System
>>> 
>>>http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifac
>>>t/
>>> lustre_manual.xhtml
>>>
>>> I'm not familiar with the process (having not needed to do that), but I
>>> believe that step describes how to remove an *active* OST from the file
>>> system.
>>>
>>> Here's what I'd suggest:
>>>
>>> 1.  Verify that each step in the *latest* manual has been executed
>>> successfully.
>>>
>>> 2.  If the "lfs find" process has not completed yet, run "strace -p
>>> <pid>" on the parent process to provide the list more detail.
>> Looks like https://jira.hpdd.intel.com/browse/LU-1738 which hasn't been
>> fixed yet.  In the meantime, you can use "lfs getstripe" instead.
>>
>>> 3.  Provide the list which version of Lustre are you running.
>>>
>>> 4.  Provide the list relevant syslog messages from the client, MDS,
>>>MGS,
>>> and the OSS with the deactivated OST.
>>>
>>> Note that for replacing a *fully* failed OST, I believe you would
>>> reformat the OST and then follow 14.8.5 (in the latest manual).
>>>
>>> Dr. Brett Lee, Solutions Architect
>>> High Performance Data Division, Intel
>>> +1.303.625.3595
>>>
>>>
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: lustre-discuss-bounces at lists.lustre.org [mailto:lustre-discuss-
>>>> bounces at lists.lustre.org] On Behalf Of Jon Tegner
>>>> Sent: Monday, March 17, 2014 10:28 AM
>>>> To: lustre-discuss at lists.lustre.org
>>>> Subject: [Lustre-discuss] Replace OST
>>>>
>>>> Hi,
>>>>
>>>> I'm new to lustre, so please excuse me for probably some stupid
>>>> questions.
>>>>
>>>> I have set up a small test system, consisting of
>>>>
>>>> * 1 MGS/MDT
>>>> * 2 OSS/OSTs
>>>> * 6 clients on infiniband and one on gigabit.
>>>>
>>>> I have verified the scaling effect (increased performance with two
>>>>OSTs
>>>> compared to one). I further wanted to gain some experience when
>>>> components fail, and in a first test I wanted to replace one of my
>>>> OSS/OSTs.
>>>> Did the following (trying to follow chapter 14.7 in the manual):
>>>>
>>>> 1. Unmounted the OST on one of my two OSS/OSTs (simulating a crash).
>>>>
>>>> 2. Deactivating this OST on the MGS/MDT.
>>>>
>>>> 3. Trying to remove the files located on this OST with the command:
>>>>
>>>> lfs find --obd lustre-OST0002 -print0 /home |  tee
>>>>/tmp/files_to_restore
>>>> | xargs -0 -n 1 unlink
>>>>
>>>> The last command was issued from one of the clients, but it just
>>>>hangs.
>>>>
>>>> Are there something wrong with the way I'm trying to do this? Any help
>>>> would be greatly appreciated!
>>>>
>>>> Thanks!
>>>>
>>>> /jon
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>> Cheers, Andreas
>
>

Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division