<div dir="ltr">Did you unmount the OST ? remember to lfs_migrate the data otherwise old data would give errors</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 5, 2021 at 11:59 AM Etienne Aujames via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org">lustre-discuss@lists.lustre.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
There is some process/work in progress on the LU-7668 to remove the OST<br>
directly on the MGS configuration.<br>
<br>
In the comment section Andreas describes a way to remove an OST with<br>
llog_print and llog_cancel (see <a href="https://review.whamcloud.com/41449" rel="noreferrer" target="_blank">https://review.whamcloud.com/41449</a>).<br>
<br>
Stephane Thiell have submitted a patch (<br>
<a href="https://review.whamcloud.com/41449/" rel="noreferrer" target="_blank">https://review.whamcloud.com/41449/</a>) to implement this process directly<br>
inside a lctl command "del_ost".<br>
<br>
This process could be applied live, the changes will take effect only<br>
after whole system remount (when MGS configuration is read by<br>
clients/MDT).<br>
<br>
This process does not replace the migrate/locking parts.<br>
<br>
We tested this process in production, but maybe for now this is bit<br>
risky. So I recommend to backup the MGS configuration.<br>
<br>
Best regards.<br>
<br>
Etienne AUJAMES<br>
<br>
On Fri, 2021-03-05 at 12:56 +0800, Tung-Han Hsieh via lustre-discuss<br>
wrote:<br>
> Dear Angelos,<br>
> <br>
> On Fri, Mar 05, 2021 at 12:15:19PM +0800, Angelos Ching via lustre-<br>
> discuss wrote:<br>
> > Hi TH,<br>
> > <br>
> > I think you'll have to set max_create_count=20000 after step 7<br>
> > unless you<br>
> > unmount and remount your MDT.<br>
> <br>
> Yes. You are right. We have to set max_create_count=20000 for the<br>
> replaced<br>
> OST, otherwise it will not accept newly created files.<br>
> <br>
> > And for step 4, I used conf_param instead of set_param during my<br>
> > drill and I<br>
> > noticed this might be more resilient if you are using a HA pair for<br>
> > the MDT<br>
> > because the MDS might try to activate the inactive OST during<br>
> > failover as<br>
> > set_param is only changing run time option?<br>
> > <br>
> > Regards,<br>
> > Angelos<br>
> <br>
> I am concerning that, sometimes, the replacement of the OST many take<br>
> a<br>
> long time. In between we may encounter some other events that need to<br>
> reboot the MDT servers. I am only sure that we can deactivate /<br>
> reactivate<br>
> the OST by conf_param when MDT server is not rebooted. Once MDT<br>
> server<br>
> is rebooted after setting conf_param=0 on the OST, I am not sure<br>
> whether<br>
> it can be recovered back or not.<br>
> <br>
> So probably I missed another step. Between step 6 and 7, we need to<br>
> reactivate the old OST before mounting the new OST ?<br>
> <br>
> 6. Prepare the new OST for replacement by mkfs.lustre with --replace<br>
> option, and set the index to the old OST index (e.g., 0x8):<br>
> ....<br>
> <br>
> 6.5. Reactivate the old OST index:<br>
> <br>
> lctl set_param osc.chome-OST0008-osc-MDT0000.active=1<br>
> <br>
> 7. Mount the new OST (run in the new OST server).<br>
> <br>
> 8. Release the new OST for accepting new objects:<br>
> <br>
> lctl set_param osc.chome-OST0008-osc-<br>
> MDT0000.max_create_count=20000<br>
> <br>
> <br>
> Cheers,<br>
> <br>
> T.H.Hsieh<br>
> <br>
> <br>
> > On 05/03/2021 11:48, Tung-Han Hsieh via lustre-discuss wrote:<br>
> > > Dear Hans,<br>
> > > <br>
> > > Thank you very much. Replacing the OST is new to me and very very<br>
> > > useful. We will try it next time.<br>
> > > <br>
> > > So, according to the description of the manual, to replace the<br>
> > > OST<br>
> > > we probably need to:<br>
> > > <br>
> > > 1. Lock the old OST (e.g., chome-OST0008) such that it will not<br>
> > > create new files (run in the MDT server):<br>
> > > <br>
> > > lctl set_param osc.chome-OST0008-osc-<br>
> > > MDT0000.max_create_count=0<br>
> > > <br>
> > > 2. Locate the list of files in the old OST: (e.g., chome-<br>
> > > OST0008):<br>
> > > (run in the client):<br>
> > > <br>
> > > lfs find --obd chome-OST0008_UUID /home > /tmp/OST0008.txt<br>
> > > <br>
> > > 3. Migrate the listed files in /tmp/OST0008.txt out of the old<br>
> > > OST.<br>
> > > (run in the client).<br>
> > > <br>
> > > 4. Remove the old OST temporarily (run in the MDT server):<br>
> > > <br>
> > > lctl set_param osc.chome-OST0008-osc-MDT0000.active=0<br>
> > > <br>
> > > (Note: should use "set_param" instead of "conf_param")<br>
> > > <br>
> > > 5. Unmount the old OST partition (run in the old OST server)<br>
> > > <br>
> > > 6. Prepare the new OST for replacement by mkfs.lustre with --<br>
> > > replace<br>
> > > option, and set the index to the old OST index (e.g., 0x8):<br>
> > > (run in the new OST server)<br>
> > > <br>
> > > mkfs.lustre --ost --mgsnode=XXXXXX --index=0x8 --replace<br>
> > > <device_name><br>
> > > <br>
> > > 7. Mount the new OST (run in the new OST server).<br>
> > > <br>
> > > <br>
> > > Best Regards,<br>
> > > <br>
> > > T.H.Hsieh<br>
> > > <br>
> > > <br>
> > > On Thu, Mar 04, 2021 at 04:59:54PM +0100, Hans Henrik Happe via<br>
> > > lustre-discuss wrote:<br>
> > > > Hi,<br>
> > > > <br>
> > > > The manual describe this:<br>
> > > > <br>
> > > > <a href="https://doc.lustre.org/lustre_manual.xhtml#lustremaint.remove_ost" rel="noreferrer" target="_blank">https://doc.lustre.org/lustre_manual.xhtml#lustremaint.remove_ost</a><br>
> > > > <br>
> > > > There is a note telling you that it will still be there, but<br>
> > > > can be<br>
> > > > replaced.<br>
> > > > <br>
> > > > Hope you migrated your data away from the OST also. Otherwise<br>
> > > > you would<br>
> > > > have lost it.<br>
> > > > <br>
> > > > Cheers,<br>
> > > > Hans Henrik<br>
> > > > <br>
> > > > On 03.03.2021 11.22, Tung-Han Hsieh via lustre-discuss wrote:<br>
> > > > > Dear All,<br>
> > > > > <br>
> > > > > Here is a question about how to remove an OST completely<br>
> > > > > without<br>
> > > > > restarting the Lustre file system. Our Lustre version is<br>
> > > > > 2.12.6.<br>
> > > > > <br>
> > > > > We did the following steps to remove the OST:<br>
> > > > > <br>
> > > > > 1. Lock the OST (e.g., chome-OST0008) such that it will not<br>
> > > > > create<br>
> > > > > new files (run in the MDT server):<br>
> > > > > <br>
> > > > > lctl set_param osc.chome-OST0008-osc-<br>
> > > > > MDT0000.max_create_count=0<br>
> > > > > <br>
> > > > > 2. Locate the list of files in the target OST: (e.g., chome-<br>
> > > > > OST0008):<br>
> > > > > (run in the client):<br>
> > > > > <br>
> > > > > lfs find --obd chome-OST0008_UUID /home<br>
> > > > > <br>
> > > > > 3. Remove OST (run in the MDT server):<br>
> > > > > lctl conf_param osc.chome-OST0008-osc-MDT0000.active=0<br>
> > > > > <br>
> > > > > 4. Unmount the OST partition (run in the OST server)<br>
> > > > > <br>
> > > > > After that, the total size of the Lustre file system<br>
> > > > > decreased, and<br>
> > > > > everything looks fine. However, without restarting (i.e.,<br>
> > > > > rebooting<br>
> > > > > Lustre MDT / OST servers), we still feel that the removed OST<br>
> > > > > is<br>
> > > > > still exists. For example, in MDT:<br>
> > > > > <br>
> > > > > # lctl get_param osc.*.active<br>
> > > > > osc.chome-OST0000-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0001-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0002-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0003-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0008-osc-MDT0000.active=0<br>
> > > > > osc.chome-OST0010-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0011-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0012-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0013-osc-MDT0000.active=1<br>
> > > > > osc.chome-OST0014-osc-MDT0000.active=1<br>
> > > > > <br>
> > > > > We still see chome-OST0008. And in dmesg of MDT, we see a lot<br>
> > > > > of:<br>
> > > > > <br>
> > > > > LustreError: 4313:0:(osp_object.c:594:osp_attr_get()) chome-<br>
> > > > > OST0008-osc-MDT0000:osp_attr_get update error<br>
> > > > > [0x100080000:0x10a54c:0x0]: rc = -108<br>
> > > > > <br>
> > > > > In addition, when running LFSCK in the MDT server:<br>
> > > > > <br>
> > > > > lctl lfsck_start -A<br>
> > > > > <br>
> > > > > even after all the works of MDT and OST are completed, we<br>
> > > > > still see that<br>
> > > > > (run in MDT server):<br>
> > > > > <br>
> > > > > lctl get_param mdd.*.lfsck_layout<br>
> > > > > <br>
> > > > > the status is not completed:<br>
> > > > > <br>
> > > > > mdd.chome-MDT0000.lfsck_layout=<br>
> > > > > name: lfsck_layout<br>
> > > > > magic: 0xb1732fed<br>
> > > > > version: 2<br>
> > > > > status: partial<br>
> > > > > flags: incomplete<br>
> > > > > param: all_targets<br>
> > > > > last_completed_time: 1614762495<br>
> > > > > time_since_last_completed: 4325 seconds<br>
> > > > > ....<br>
> > > > > <br>
> > > > > We suspect that the "incomplete" part might due to the<br>
> > > > > already removed<br>
> > > > > chome-OST0008.<br>
> > > > > <br>
> > > > > Is there any way to completely remove the chome-OST0008 from<br>
> > > > > the Lustre<br>
> > > > > file system ? since that OST device has already been<br>
> > > > > reformatted for<br>
> > > > > other usage.<br>
> > > > > <br>
> > > > > Thanks very much.<br>
> > > > > <br>
> > > > > <br>
> > > > > T.H.Hsieh<br>
> > > > > _______________________________________________<br>
> > > > > lustre-discuss mailing list<br>
> > > > > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > > > > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> > > > _______________________________________________<br>
> > > > lustre-discuss mailing list<br>
> > > > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > > > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> > > _______________________________________________<br>
> > > lustre-discuss mailing list<br>
> > > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> > _______________________________________________<br>
> > lustre-discuss mailing list<br>
> > <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> > <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
> _______________________________________________<br>
> lustre-discuss mailing list<br>
> <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
> <a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
_______________________________________________<br>
lustre-discuss mailing list<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org" rel="noreferrer" target="_blank">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br>
</blockquote></div>