<HTML>
<HEAD>
<TITLE>Re: [Lustre-discuss] new OST not mounting up</TITLE>
</HEAD>
<BODY>
<FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>Ok, try this.. I know you’ve done it already, but it may help us to understand:<BR>
<BR>
1. reboot both the ost and the MGS machine<BR>
2. run tunefs.lustre —writeconf /dev/<device><BR>
3. mount the mgs, then the ost<BR>
4. run dmesg on both servers so we can see what that first mount failure says.<BR>
<BR>
Since this also looks like they are on the same machine we may only get one dmesg output here.. Also it looks like you are having trouble communicating with the server at times. Is it possible there are communication errors.<BR>
<BR>
What version of lustre is this?<BR>
<BR>
evan<BR>
<BR>
On 3/1/09 5:19 PM, "Mag Gam" <<a href="magawake@gmail.com">magawake@gmail.com</a>> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Calibri, Verdana, Helvetica, Arial"><SPAN STYLE='font-size:11pt'>OK, I did a tunefs.lustre --writeconf /dev/<BR>
<BR>
and tried to mount it up, still the same error. "Operation already in<BR>
progress". The target service is already running.<BR>
<BR>
I am not sure whatelse I can try...<BR>
<BR>
Any suggestions?<BR>
<BR>
TIA<BR>
<BR>
On Sat, Feb 28, 2009 at 8:39 AM, Mag Gam <<a href="magawake@gmail.com">magawake@gmail.com</a>> wrote:<BR>
> (sorry adding the entire list for Evan's reponse)<BR>
><BR>
> Thankyou got getting back to me on this.<BR>
> So, when I try to mount the **new** ost I keep getting these messages.<BR>
><BR>
> For some reason the new OST is active on the MGS side which I am not<BR>
> sure why. I think I made a mistake by trying to mount up a new OST<BR>
> while clients were still active.<BR>
><BR>
><BR>
> When I try to activaste the bad OST.I get this message.<BR>
><BR>
> Lustre: 11647:0:(ldlm_lib.c:736:target_handle_connect())<BR>
> lfs001-OST0005: cookie lfs001-mdtlov_UUID seen on new NID<BR>
> mds_ip_addr@tcp when existing NID 0@lo is already connected<BR>
> Feb 27 11:59:01 oss_server kernel: Lustre:<BR>
> 11647:0:(ldlm_lib.c:736:target_handle_connect()) Skipped 4 previous<BR>
> similar messages<BR>
> Feb 27 11:59:01 mds_server kernel: Lustre:<BR>
> 3426:0:(import.c:411:import_select_connection()) lfs001-OST0005-osc:<BR>
> tried all connections, increasing latency to 51s<BR>
> Feb 27 11:59:01 oss_server kernel: LustreError:<BR>
> 11647:0:(ldlm_lib.c:1614:target_send_reply_msg()) @@@ processing error<BR>
> (-114) req@ffff8104251a4400 x388745/t0 o8-><?>@<?>:0/0 lens 240/144 e<BR>
> 0 to 0 dl 1235754041 ref 1 fl Interpret:/0/0 rc -114/0<BR>
> Feb 27 11:59:01 mds_server kernel: Lustre:<BR>
> 3426:0:(import.c:411:import_select_connection()) Skipped 6 previous<BR>
> similar messages<BR>
> Feb 27 11:59:01 mds_server kernel: LustreError: 11-0: an error<BR>
> occurred while communicating with oss_ip@tcp. The ost_connect<BR>
> operation failed with -114<BR>
> Feb 27 11:59:01 mds_server kernel: LustreError: Skipped 12 previous<BR>
> similar messages<BR>
><BR>
><BR>
> oss_server kernel: LustreError:<BR>
> 11556:0:(ldlm_lib.c:1614:target_send_reply_msg()) @@@ processing error<BR>
> (-114) req@ffff81042150a000 x388953/t0 o8-><?>@<?>:0/0 lens 240/144 e<BR>
> 0 to 0 dl 1235754240 ref 1 fl Interpret:/0/0 rc -114/0<BR>
><BR>
><BR>
><BR>
> Also, I was wondering if there was a way to reset the state of my OST.<BR>
> Its keep thinking its already mounted. Even after a reboot. Any way to<BR>
> say "hey, I am not mounted" ? :-)<BR>
><BR>
> TIA<BR>
><BR>
> On Sat, Feb 28, 2009 at 8:38 AM, Mag Gam <<a href="magawake@gmail.com">magawake@gmail.com</a>> wrote:<BR>
>> Thankyou got getting back to me on this.<BR>
>> So, when I try to mount the **new** ost I keep getting these messages.<BR>
>><BR>
>> For some reason the new OST is active on the MGS side which I am not<BR>
>> sure why. I think I made a mistake by trying to mount up a new OST<BR>
>> while clients were still active.<BR>
>><BR>
>><BR>
>><BR>
>><BR>
>> When I try to activaste the bad OST.I get this message.<BR>
>><BR>
>> Lustre: 11647:0:(ldlm_lib.c:736:target_handle_connect())<BR>
>> lfs001-OST0005: cookie lfs001-mdtlov_UUID seen on new NID<BR>
>> mds_ip_addr@tcp when existing NID 0@lo is already connected<BR>
>> Feb 27 11:59:01 oss_server kernel: Lustre:<BR>
>> 11647:0:(ldlm_lib.c:736:target_handle_connect()) Skipped 4 previous<BR>
>> similar messages<BR>
>> Feb 27 11:59:01 mds_server kernel: Lustre:<BR>
>> 3426:0:(import.c:411:import_select_connection()) lfs001-OST0005-osc:<BR>
>> tried all connections, increasing latency to 51s<BR>
>> Feb 27 11:59:01 oss_server kernel: LustreError:<BR>
>> 11647:0:(ldlm_lib.c:1614:target_send_reply_msg()) @@@ processing error<BR>
>> (-114) req@ffff8104251a4400 x388745/t0 o8-><?>@<?>:0/0 lens 240/144 e<BR>
>> 0 to 0 dl 1235754041 ref 1 fl Interpret:/0/0 rc -114/0<BR>
>> Feb 27 11:59:01 mds_server kernel: Lustre:<BR>
>> 3426:0:(import.c:411:import_select_connection()) Skipped 6 previous<BR>
>> similar messages<BR>
>> Feb 27 11:59:01 mds_server kernel: LustreError: 11-0: an error<BR>
>> occurred while communicating with oss_ip@tcp. The ost_connect<BR>
>> operation failed with -114<BR>
>> Feb 27 11:59:01 mds_server kernel: LustreError: Skipped 12 previous<BR>
>> similar messages<BR>
>><BR>
>><BR>
>> oss_server kernel: LustreError:<BR>
>> 11556:0:(ldlm_lib.c:1614:target_send_reply_msg()) @@@ processing error<BR>
>> (-114) req@ffff81042150a000 x388953/t0 o8-><?>@<?>:0/0 lens 240/144 e<BR>
>> 0 to 0 dl 1235754240 ref 1 fl Interpret:/0/0 rc -114/0<BR>
>><BR>
>><BR>
>><BR>
>> Also, I was wondering if there was a way to reset the state of my OST.<BR>
>> Its keep thinking its already mounted. Even after a reboot. Any way to<BR>
>> say "hey, I am not mounted" ? :-)<BR>
>><BR>
>> Would a writeconf help on the OST? I am hesitant to run one on it.<BR>
>><BR>
>> TIA<BR>
>><BR>
>> On Fri, Feb 27, 2009 at 11:41 AM, Evan Felix <<a href="evan.felix@pnl.gov">evan.felix@pnl.gov</a>> wrote:<BR>
>>> Mag,<BR>
>>><BR>
>>> Can you send us the output from your kernel log after you try the mount<BR>
>>> command that is failing?<BR>
>>><BR>
>>> Just run 'dmesg' and send us the last 20 lines or so..<BR>
>>><BR>
>>> evan<BR>
>>><BR>
>>><BR>
>>> On 2/26/09 7:12 PM, "Mag Gam" <<a href="magawake@gmail.com">magawake@gmail.com</a>> wrote:<BR>
>>><BR>
>>>> Any ideas?<BR>
>>>><BR>
>>>> I am still unable to mount this new OST. I stopped the client hang<BR>
>>>> problem by disabling the OST via lctl but crazy problem indeed.<BR>
>>>><BR>
>>>><BR>
>>>> I would love to know how to activate the OST.<BR>
>>>><BR>
>>>><BR>
>>>><BR>
>>>> On Wed, Feb 25, 2009 at 4:43 PM, Mag Gam <<a href="magawake@gmail.com">magawake@gmail.com</a>> wrote:<BR>
>>>>> Hello.<BR>
>>>>><BR>
>>>>><BR>
>>>>> We created an OST on an OSS. But when I try to mount up the OST, it<BR>
>>>>> keeps saying.<BR>
>>>>><BR>
>>>>><BR>
>>>>> mount.lustre /dev/vg/ost002 /vol/srv1/ost002<BR>
>>>>> mount.lustre: mount /dev/vg/ost002 at /vol/srv1/ost002 failed:<BR>
>>>>> Operation already in progress<BR>
>>>>> The target service is already running. (/dev/vg/ost002)<BR>
>>>>><BR>
>>>>><BR>
>>>>> However,<BR>
>>>>> mount | grep -i ost002<BR>
>>>>> Nothing is mounted up....<BR>
>>>>><BR>
>>>>> lctl is even showing this OST and also the client is able to see it.<BR>
>>>>> lfs df -h<BR>
>>>>> ...<BR>
>>>>> lfs001-OST0005_UUID 492.2G 445.2G 22.0G 90%<BR>
>>>>> /lfs/srv5/lfs001[OST:5]<BR>
>>>>> ...<BR>
>>>>><BR>
>>>>><BR>
>>>>> The MDS/OSS Version:<BR>
>>>>> lustre: 1.6.5.52<BR>
>>>>> kernel: patchless<BR>
>>>>> build:<BR>
>>>>> 1.6.5.52-19691231190000-PRISTINE-.var.tmp.linux-2.6.18.x86_64-2.6.18-prep<BR>
>>>>> 2.6.18 = Kernel Version<BR>
>>>>><BR>
>>>>> I don't think its bugzilla 11564, because my lustre fs name is only 6<BR>
>>>>> characters long.<BR>
>>>>><BR>
>>>>><BR>
>>>>> Also, when the client tries to access the new OST's space, it simple<BR>
>>>>> hangs. It placed it in "bloc<BR>
>>>>><BR>
>>>>> Any thoughts about this?<BR>
>>>>><BR>
>>>>> TIA<BR>
>>>>><BR>
>>>> _______________________________________________<BR>
>>>> Lustre-discuss mailing list<BR>
>>>> <a href="Lustre-discuss@lists.lustre.org">Lustre-discuss@lists.lustre.org</a><BR>
>>>> <a href="http://lists.lustre.org/mailman/listinfo/lustre-discuss">http://lists.lustre.org/mailman/listinfo/lustre-discuss</a><BR>
>>><BR>
>>><BR>
>><BR>
><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>