I tried using tunefs.lustre to re-set failover parameter for my OST ( although, from dryrun tunefs.lustre output, i saw those parameter ) but it couldn't help. Anyone else has any idea? <br><br>Thank you in advance !!!! <br>
<br><div class="gmail_quote">On Thu, Nov 19, 2009 at 5:33 AM, Dam Thanh Tung <span dir="ltr"><<a href="mailto:tungdt@isds.vn">tungdt@isds.vn</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="gmail_quote"><div><div></div><div class="h5">On Thu, Nov 19, 2009 at 2:00 AM, <span dir="ltr"><<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Send Lustre-discuss mailing list submissions to<br>
<a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:lustre-discuss-request@lists.lustre.org" target="_blank">lustre-discuss-request@lists.lustre.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:lustre-discuss-owner@lists.lustre.org" target="_blank">lustre-discuss-owner@lists.lustre.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Lustre-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. MDS doesn't switch to failover OST node (Dam Thanh Tung)<br>
2. Re: MDS doesn't switch to failover OST node (Brian J. Murrell)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 18 Nov 2009 22:54:28 +0700<br>
From: Dam Thanh Tung <<a href="mailto:tungdt@isds.vn" target="_blank">tungdt@isds.vn</a>><br>
Subject: [Lustre-discuss] MDS doesn't switch to failover OST node<br>
To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Message-ID:<br>
<<a href="mailto:a119d1570911180754i3ee81f30wad5a0dd1cdb47e05@mail.gmail.com" target="_blank">a119d1570911180754i3ee81f30wad5a0dd1cdb47e05@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
Hi list<br>
<br>
I am encountering a problem with OST-MDS connecting. Because of RAID card<br>
hanging, our OST went down this morning and when i tried to mount the faill<br>
over node of that OST, problem occurred :<br>
<br>
MDS only sent request to the OST which was down and didn't connect to our<br>
backup (failover) OST, so our backup solution was useless, we lost all data<br>
from that OST. It's really a disaster for me because we even lost all of our<br>
data before with the same kind of problem: OST can't connect to MDS !!!!<br>
<br>
We use drbd between OSTs to synchronize data. The backup (failover node) was<br>
mounted successfully without any error but didn't have any client to recover<br>
like this:<br>
<br>
cat /proc/fs/lustre/obdfilter/lustre-OST0006/recovery_status<br>
status: RECOVERING<br>
recovery_start: 0<br>
time_remaining: 0<br>
connected_clients: 0/1<br>
delayed_clients: 0/1<br>
completed_clients: 0/1<br>
replayed_requests: 0*/??*<br>
queued_requests: 0<br>
next_transno: 30064771073<br>
<br>
In MDS's message log, we only saw the connection to our dead OST:<br>
<br>
Nov 18 22:44:03 MDS1 kernel: Lustre: Request x1314965674069373 sent from<br>
lustre-OST0006-osc to NID 192.168.1.66@tcp 56s ago has timed out (limit<br>
56s).<br>
......<br>
<br>
The output of* **lctl dl *command from MDS<br>
<br>
lctl dl<br>
0 UP mgs MGS MGS 25<br>
1 UP mgc MGC192.168.1.78@tcp 0681a267-849f-350c-5b2c-6869c794550f 5<br>
2 UP mdt MDS MDS_uuid 3<br>
3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4<br>
4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 15<br>
5 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5<br>
6 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5<br>
7 IN osc lustre-OST0006-osc lustre-mdtlov_UUID 5<br>
8 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5<br>
9 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5<br>
<br>
I did activated OST6 ( lctl --device 7 activate ) but it couldn't help<br>
<br>
<br>
<br>
Could anyone tell me how to route MDS to connect to our backup OST ( with ip<br>
address 192.168.1.67 , for example ) ? , to bring our OST up ?<br>
<br>
Any help would be really appreciated !<br>
<br>
Hope that i can receive your answers or suggestions as soon as possible<br>
<br>
Best Regards<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091118/5b0a96ce/attachment-0001.html" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091118/5b0a96ce/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 18 Nov 2009 11:10:51 -0500<br>
From: "Brian J. Murrell" <Brian.Murrell@Sun.COM><br>
Subject: Re: [Lustre-discuss] MDS doesn't switch to failover OST node<br>
To: <a href="mailto:lustre-discuss@lists.lustre.org" target="_blank">lustre-discuss@lists.lustre.org</a><br>
Message-ID: <<a href="mailto:1258560651.30445.59.camel@pc.interlinx.bc.ca" target="_blank">1258560651.30445.59.camel@pc.interlinx.bc.ca</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
On Wed, 2009-11-18 at 22:54 +0700, Dam Thanh Tung wrote:<br>
> Hi list<br>
<br>
Hi,<br>
<br>
> MDS only sent request to the OST which was down and didn't connect to<br>
> our backup (failover) OST, so our backup solution was useless, we lost<br>
> all data from that OST.<br></blockquote><div> </div></div></div><div>Hi Brian<br>
<br>
Thank you for you fast reply <br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
I don't think you have actually lost any data. It's there. Your<br>
clients (which the MDS is) just don't know to use the failover OSS that<br>
you have set up (but not told Lustre about).<br>
<br>
> It's really a disaster for me because we even lost all of our data<br>
> before with the same kind of problem: OST can't connect to MDS !!!!<br>
<br>
Failures to connect between nodes does not result in data loss. The<br>
data is still there. You just need to have your clients access it.<br>
<br></blockquote></div><div><br><br>
I know that data is still there but i refer to "lost" when i no longer can access it anymore. <br>
<br>
In our client, we mounted with parameter like this: <br>
<br>
mount -t lustre -o flock 192.168.1.78@tcp:192.168.1.80@tcp:/lustre /mnt/lustre/<br><br>We didn't umount our client, just deactivate the dead OST and after mouting the backup one, we activated it, but because MDS coudn't connect and receive any information from the backup ( failover ) OST, clients are the same. <br>
<br> <br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
> Could anyone tell me how to route MDS to connect to our backup OST<br>
> ( with ip address 192.168.1.67 , for example ) ? , to bring our OST<br>
> up ?<br>
<br>
It sounds like you need to review the failover section of the manual.<br>
<br>
In summary, you need to tell the clients about failover nodes<br>
(--failnode) when you create the filesystem. You can add this feature<br>
after-the-fact with tunefs.lustre.<br></blockquote></div><div><br>
In our OST, before it goes down because of RAID card hanging, we made it by: <br>
<br>
mkfs.lustre --ost --mgsnode=192.168.1.78@tcp
--mgsnode=192.168.1.80@tcp --failover=192.168.1.66@tcp --index=6
--verbose --writeconf /dev/drbd6 <br><br>Could you please give some suggestions ? Do i need to provide some information ? <br><br>Many thanks<br></div><div class="im"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
b.<br>
<br>
-------------- next part --------------<br>
A non-text attachment was scrubbed...<br>
Name: not available<br>
Type: application/pgp-signature<br>
Size: 197 bytes<br>
Desc: This is a digitally signed message part<br>
Url : <a href="http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091118/f1c497e1/attachment-0001.bin" target="_blank">http://lists.lustre.org/pipermail/lustre-discuss/attachments/20091118/f1c497e1/attachment-0001.bin</a><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
Lustre-discuss mailing list<br>
<a href="mailto:Lustre-discuss@lists.lustre.org" target="_blank">Lustre-discuss@lists.lustre.org</a><br>
<a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss" target="_blank">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><br>
<br>
<br>
End of Lustre-discuss Digest, Vol 46, Issue 33<br>
**********************************************<br>
</blockquote></div></div><br>
</blockquote></div><br>