From s.wendy.cheng at gmail.com  Sat Oct  5 00:18:52 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Fri, 4 Oct 2013 17:18:52 -0700
Subject: [Lustre-devel] Lustre on Intel Xeon Phi
In-Reply-To: <9FC73D3DBECE0941BD2ED069D268634216EBEEAB@IRSMSX105.ger.corp.intel.com>
References: <CABgxfbFoC1bh4zmJmgZJuufN1zViO1e7AogvVMkeOzVRuPw0hw@mail.gmail.com>
	<9FC73D3DBECE0941BD2ED069D268634216EBEEAB@IRSMSX105.ger.corp.intel.com>
Message-ID: <CABgxfbGSgeNHx0AT+8WeYigM==TpEL5Ecrn-Qk=YDGFCxwiQyQ@mail.gmail.com>

On Fri, Oct 4, 2013 at 4:53 AM, Eremin, Dmitry <dmitry.eremin at intel.com> wrote:
> The support of Lustre for Intel Xeon Phi with MPSS 2.1 was introduced in Lustre 2.4. Then the support was extended for MPSS 3.x but in Lustre 2.5 only.
>

Great thanks !   .. So I took mpss_gold_update_3-2.1.6720-19 source,
plus the master branch by doing a

"git clone git://git.whamcloud.com/fs/lustre-release.git"

The "configure" seems to run fine but "make rpms" wants two non-exist
files as the following:

RPM build errors:
    File not found:
/root/rpmbuild/BUILDROOT/lustre-client-mic-2.4.93-2.6.38.8_g2593b11_g26c55b2.x86_64/opt/lustre/2.4.93/x86_64-k1om-linux/usr/bin/mcreate
    File not found:
/root/rpmbuild/BUILDROOT/lustre-client-mic-2.4.93-2.6.38.8_g2593b11_g26c55b2.x86_64/opt/lustre/2.4.93/x86_64-k1om-linux/usr/bin/munlink

Do I have any workaround ? I would prefer the master branch, instead
of going back to 2.4, if all possible.

Thanks,
Wendy

> How to compile Lustre with MPSS 2.1
>
> Got to http://software.intel.com/en-us/articles/intel-manycore-platform-software-stack-mpss and download the SOURCE package. Unpack it and find the package-full_src-k1om.tar.bz2 file. Unpack this archive in any location, for example in "/opt/intel/mic/src", then go to this location and execute the following commands:
>
> make defconfig-miclinux
> make -C card/kernel ARCH=k1om modules_prepare
>
> Now the Intel Xeon Phi kernel is ready for Lustre compilation. Go to location with Lustre sources and execute the following commands:
>
> # NOTE: You should have "intel-mic-gpl-<version>.x86_64" package installed.
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> sh autogen.sh
> ./configure --with-linux=/opt/intel/mic/src/card/kernel \
> --disable-server --disable-tests --disable-doc --without-o2ib \
> --host=x86_64-k1om-linux --build=x86_64-pc-linux
> make rpms
>
> The following RPMs should be installed on Intel Xeon host with Phi card and Lustre will be deployed to Phi card:
>
> lustre-client-mic-<version>.x86_64.rpm
> lustre-client-mic-modules-<version>.x86_64.rpm
>
> After MPSS service restart it's ready to configure and use.
>
> Dmitry.
>> -----Original Message-----
>> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-
>> bounces at lists.lustre.org] On Behalf Of Wendy Cheng
>> Sent: Tuesday, October 01, 2013 3:18 AM
>> To: Lustre-devel at lists.lustre.org
>> Subject: [Lustre-devel] Lustre on Intel Xeon Phi
>>
>> Do folks have instructions of how to install Lustre client on Intel Xeon Phi ?
>> I'm on a relatively old MPSS  (mpss_gold_update_3,
>> mic-2.1.6720 linux 2.6.18 kernel). Other cluster nodes run Lustre-1.8.8.
>>
>> Thanks, Wendy
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
> --------------------------------------------------------------------
> Closed Joint Stock Company Intel A/O
> Registered legal address: Krylatsky Hills Business Park,
> 17 Krylatskaya Str., Bldg 4, Moscow 121614,
> Russian Federation
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

From dmitry.eremin at intel.com  Fri Oct  4 11:53:15 2013
From: dmitry.eremin at intel.com (Eremin, Dmitry)
Date: Fri, 4 Oct 2013 11:53:15 +0000
Subject: [Lustre-devel] Lustre on Intel Xeon Phi
In-Reply-To: <CABgxfbFoC1bh4zmJmgZJuufN1zViO1e7AogvVMkeOzVRuPw0hw@mail.gmail.com>
References: <CABgxfbFoC1bh4zmJmgZJuufN1zViO1e7AogvVMkeOzVRuPw0hw@mail.gmail.com>
Message-ID: <9FC73D3DBECE0941BD2ED069D268634216EBEEAB@IRSMSX105.ger.corp.intel.com>

The support of Lustre for Intel Xeon Phi with MPSS 2.1 was introduced in Lustre 2.4. Then the support was extended for MPSS 3.x but in Lustre 2.5 only.

How to compile Lustre with MPSS 2.1

Got to http://software.intel.com/en-us/articles/intel-manycore-platform-software-stack-mpss and download the SOURCE package. Unpack it and find the package-full_src-k1om.tar.bz2 file. Unpack this archive in any location, for example in "/opt/intel/mic/src", then go to this location and execute the following commands:

make defconfig-miclinux
make -C card/kernel ARCH=k1om modules_prepare
 
Now the Intel Xeon Phi kernel is ready for Lustre compilation. Go to location with Lustre sources and execute the following commands:

# NOTE: You should have "intel-mic-gpl-<version>.x86_64" package installed.
export PATH=/usr/linux-k1om-4.7/bin:$PATH 
sh autogen.sh
./configure --with-linux=/opt/intel/mic/src/card/kernel \
--disable-server --disable-tests --disable-doc --without-o2ib \
--host=x86_64-k1om-linux --build=x86_64-pc-linux
make rpms

The following RPMs should be installed on Intel Xeon host with Phi card and Lustre will be deployed to Phi card:

lustre-client-mic-<version>.x86_64.rpm
lustre-client-mic-modules-<version>.x86_64.rpm

After MPSS service restart it's ready to configure and use.

Dmitry.
> -----Original Message-----
> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-
> bounces at lists.lustre.org] On Behalf Of Wendy Cheng
> Sent: Tuesday, October 01, 2013 3:18 AM
> To: Lustre-devel at lists.lustre.org
> Subject: [Lustre-devel] Lustre on Intel Xeon Phi
> 
> Do folks have instructions of how to install Lustre client on Intel Xeon Phi ?
> I'm on a relatively old MPSS  (mpss_gold_update_3,
> mic-2.1.6720 linux 2.6.18 kernel). Other cluster nodes run Lustre-1.8.8.
> 
> Thanks, Wendy
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park, 
17 Krylatskaya Str., Bldg 4, Moscow 121614, 
Russian Federation

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

From groschup at stud.uni-heidelberg.de  Mon Oct  7 14:42:48 2013
From: groschup at stud.uni-heidelberg.de (Tobias Groschup)
Date: Mon, 07 Oct 2013 16:42:48 +0200
Subject: [Lustre-devel] Starting and using  a self written LND
Message-ID: <5252C868.2050104@stud.uni-heidelberg.de>

Hi all,

I am still at the task of writing a new LND. Finally got Lustre to build 
with our own software stack.

Now I want to test the new LND. Of course, it does not implement much 
functionality yet. In fact, it only has a stump for the send and receive 
functions, and a startup and shutdown function which sets up the 
software context for the network card and closes it.
I can load the module, and LNet does not crash after adding the new 
entries to the LND types enum in libcfs/include/libcfs/libcfs private.h 
and to libcfs_netstrfns[] in libcfs/libcfs/nidstrings.c.

So the question is: how does one invoke the startup function after 
loading an LND module? I tried lclt and its network up command, which 
returned without error, but I did not see the debug output which is 
printed via printks.

And: are the modifications to the libcfs mentioned above all that are 
needed to declare an LND to libcfs?

Thanks and Greetings
Tobias

From s.wendy.cheng at gmail.com  Thu Oct 10 17:05:56 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Thu, 10 Oct 2013 10:05:56 -0700
Subject: [Lustre-devel] Lustre on Intel Xeon Phi
In-Reply-To: <9FC73D3DBECE0941BD2ED069D268634216EBF312@IRSMSX105.ger.corp.intel.com>
References: <CABgxfbFoC1bh4zmJmgZJuufN1zViO1e7AogvVMkeOzVRuPw0hw@mail.gmail.com>
	<9FC73D3DBECE0941BD2ED069D268634216EBEEAB@IRSMSX105.ger.corp.intel.com>
	<CABgxfbGSgeNHx0AT+8WeYigM==TpEL5Ecrn-Qk=YDGFCxwiQyQ@mail.gmail.com>
	<9FC73D3DBECE0941BD2ED069D268634216EBF312@IRSMSX105.ger.corp.intel.com>
Message-ID: <CABgxfbFqiqg-w=kCise+x0468UE9YqUrnek7UVTt74nfsLaCvw@mail.gmail.com>

On Sat, Oct 5, 2013 at 3:01 AM, Eremin, Dmitry <dmitry.eremin at intel.com> wrote:
> Sorry, after recent reorganization it was broken. I will fix this. But now you can just remove "--disable-tests" option from configure. The build will be a bit longer but error should gone. Those binaries produced for testing needs.
>

Thanks for the help ! The Xeon Phi Lustre client is now up and running
on top of Infiniband RDMA using

* Intel Xeon Phi MPSS 2.1 (mpss_gold_update_3-2.1.6720-16)
* Lustre-release from git master branch
* A private version of OFED

There was a kernel panic at lmv_get_info() during mount time but the
problem has been fixed. As the issue seems to be generic (not MPSS
specific), will start another thread to discuss it.

Again, your responses and help are very appreciated.

-- Wendy

From s.wendy.cheng at gmail.com  Fri Oct 11 17:11:33 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Fri, 11 Oct 2013 10:11:33 -0700
Subject: [Lustre-devel] Kernel crash from "mkfs.lustre --index" setting
Message-ID: <CABgxfbG2DWP-Md7SjQnhEkSUcqaBFqkyGMc0f=t=cVsPHu80sg@mail.gmail.com>

This panic seems to be generic regardless the platform, though I'm
actually on Intel Xeon Phi Lustre (client) nodes.

New to Lustre, I mistakenly thought the "index" option of mkfs.lustre
was for software raid so I formatted one of the server disks as the
following:

server> mkfs.lustre --reformat --fsname=lus1 --mgs --mdt --index=1 /dev/sdd1
server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=1 /dev/sde1

The client mount immediately crashed at lmv_get_info(). The attached
patch fixed that particular panic ... but unfortunately crashed at an
assertion further down the path. I'll be travelling next week so might
give up pursuing this issue. The disks are now subsequently
re-formatted with index=0 - things seem to work fine and performance
numbers collected. Three questions here:

1. What is this "index" option all about ?
2. Does the problem worth being fixed ? Or is it a user error ?
3. The performance numbers (again, NOT Xeon Phi specific) surprise me.
Would this list be a good place to ask questions ?

-- Wendy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: index.patch
Type: application/octet-stream
Size: 1885 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20131011/978dfb1b/attachment.bin>

From cliff.white at intel.com  Fri Oct 11 17:41:33 2013
From: cliff.white at intel.com (White, Cliff)
Date: Fri, 11 Oct 2013 17:41:33 +0000
Subject: [Lustre-devel] Kernel crash from "mkfs.lustre --index" setting
In-Reply-To: <CABgxfbG2DWP-Md7SjQnhEkSUcqaBFqkyGMc0f=t=cVsPHu80sg@mail.gmail.com>
Message-ID: <CE7D8557.721A%cliff.white@intel.com>

On 10/11/13 10:11 AM, "Wendy Cheng" <s.wendy.cheng at gmail.com> wrote:

>This panic seems to be generic regardless the platform, though I'm
>actually on Intel Xeon Phi Lustre (client) nodes.
>
>New to Lustre, I mistakenly thought the "index" option of mkfs.lustre
>was for software raid so I formatted one of the server disks as the
>following:
>
>server> mkfs.lustre --reformat --fsname=lus1 --mgs --mdt --index=1
>/dev/sdd1
>server> mkfs.lustre --reformat --ost --fsname=lus1
>--mgsnode=192.168.20.46 at o2ib0 --index=1 /dev/sde1
>
>The client mount immediately crashed at lmv_get_info(). The attached
>patch fixed that particular panic ... but unfortunately crashed at an
>assertion further down the path. I'll be travelling next week so might
>give up pursuing this issue. The disks are now subsequently
>re-formatted with index=0 - things seem to work fine and performance
>numbers collected. Three questions here:
>
>1. What is this "index" option all about ?
>2. Does the problem worth being fixed ? Or is it a user error ?
>3. The performance numbers (again, NOT Xeon Phi specific) surprise me.
>Would this list be a good place to ask questions ?
>
>-- Wendy
>
1.
--index is used to enumerate OSTs and MDT, when using DNE.
The index MUST be unique, and indexes must not have gaps.
So, you should do this:
server> mkfs.lustre --reformat --fsname=lus1 --mgs --mdt --index=0
/dev/sdd1 /* First MDT */
server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=0 /dev/sde1 /* first OST */
If you add a second OST partition:

server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=1 /dev/sdfoo /* second OST */

And a third:
server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=2 /dev/sdbar /* third OST */


2.- You must fix this, or things won't work. I would suggest starting
again, and doing a reformat
Etc,etc

3. Surprise you how?

HPDD-discuss is likely a better list for these sorts of questions,
lustre-devel is for code development.
Cliffw

From s.wendy.cheng at gmail.com  Fri Oct 11 17:59:41 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Fri, 11 Oct 2013 10:59:41 -0700
Subject: [Lustre-devel] Kernel crash from "mkfs.lustre --index" setting
In-Reply-To: <CE7D8557.721A%cliff.white@intel.com>
References: <CABgxfbG2DWP-Md7SjQnhEkSUcqaBFqkyGMc0f=t=cVsPHu80sg@mail.gmail.com>
	<CE7D8557.721A%cliff.white@intel.com>
Message-ID: <CABgxfbE1LruxE8Uc-Vb_JnEyRnJcC8FUa0Q-kPjNNgT6zR7oCg@mail.gmail.com>

On Fri, Oct 11, 2013 at 10:41 AM, White, Cliff <cliff.white at intel.com> wrote:

> 1.
> --index is used to enumerate OSTs and MDT, when using DNE.
> The index MUST be unique, and indexes must not have gaps.

I see ... index must not have gaps. However, a user error could crash
the kernelr . Does that sound right ? .

>
> 3. Surprise you how?
>
> HPDD-discuss is likely a better list for these sorts of questions,
> lustre-devel is for code development.

Thanks .. I'll move the discuss there sometime next week. It looks to
me Lustre is doing sync to the disks all the time vs. other network
filesystem (e.g. NFS) that does caching quite aggressively.

-- Wendy

From cliff.white at intel.com  Fri Oct 11 18:30:56 2013
From: cliff.white at intel.com (White, Cliff)
Date: Fri, 11 Oct 2013 18:30:56 +0000
Subject: [Lustre-devel] Kernel crash from "mkfs.lustre --index" setting
In-Reply-To: <CABgxfbE1LruxE8Uc-Vb_JnEyRnJcC8FUa0Q-kPjNNgT6zR7oCg@mail.gmail.com>
Message-ID: <CE7D90F6.722B%cliff.white@intel.com>

On 10/11/13 10:59 AM, "Wendy Cheng" <s.wendy.cheng at gmail.com> wrote:

>On Fri, Oct 11, 2013 at 10:41 AM, White, Cliff <cliff.white at intel.com>
>wrote:
>
>> 1.
>> --index is used to enumerate OSTs and MDT, when using DNE.
>> The index MUST be unique, and indexes must not have gaps.
>
>I see ... index must not have gaps. However, a user error could crash
>the kernelr . Does that sound right ? .

Well, creating the filesystem is normally done by admins, not users, but
yes, it shouldn't crash.
Lustre-devel is the place for your patch, sorry I wasn't clear. -discuss
is more for the 'why are their indexes' type of questions. :)
>
>>
>> 3. Surprise you how?
>>
>> HPDD-discuss is likely a better list for these sorts of questions,
>> lustre-devel is for code development.
>
>Thanks .. I'll move the discuss there sometime next week. It looks to
>me Lustre is doing sync to the disks all the time vs. other network
>filesystem (e.g. NFS) that does caching quite aggressively.

Yes, Lustre by design does direct IO to disk, and does not cache data on
the servers. 
Some caching can be enabled, but in general no, you should not see the
servers caching.

However, the clients should be using the normal Linux block cache, if the
clients are not caching
There may be an issue with your setup.
Cliffw

>
>-- Wendy
>


From s.wendy.cheng at gmail.com  Sat Oct 12 00:47:49 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Fri, 11 Oct 2013 17:47:49 -0700
Subject: [Lustre-devel] Kernel crash from "mkfs.lustre --index" setting
In-Reply-To: <CE7D90F6.722B%cliff.white@intel.com>
References: <CABgxfbE1LruxE8Uc-Vb_JnEyRnJcC8FUa0Q-kPjNNgT6zR7oCg@mail.gmail.com>
	<CE7D90F6.722B%cliff.white@intel.com>
Message-ID: <CABgxfbF_E6XZp_aFEewsf7hpg7N9hryeHNgpKMHmL+ObE2E_Bw@mail.gmail.com>

On Fri, Oct 11, 2013 at 11:30 AM, White, Cliff <cliff.white at intel.com> wrote:
> On 10/11/13 10:59 AM, "Wendy Cheng" <s.wendy.cheng at gmail.com> wrote:
>
>>On Fri, Oct 11, 2013 at 10:41 AM, White, Cliff <cliff.white at intel.com>
>>wrote:
>>

>
> Yes, Lustre by design does direct IO to disk, and does not cache data on
> the servers.

I see .. Direct IO . the data makes sense now :)

Thanks !

-- Wendy

From s.wendy.cheng at gmail.com  Sat Oct 12 03:28:45 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Fri, 11 Oct 2013 20:28:45 -0700
Subject: [Lustre-devel] [Patch] Fix Client Kernel Crash with Mis-configured
	Index Numbering
Message-ID: <CABgxfbH-js0JZzSAienCMJgXQYzVsKUYHbpfJWbvPfakyFGT6w@mail.gmail.com>

Ref: http://lists.lustre.org/pipermail/lustre-devel/2013-October/004270.html

I'm not really convinced the "index" setting of mkfs.lustre needs to
be started with "0". However, in the minimum, the client kernel should
not crash. The attached patch does this minimum fix; compiled and
tested with GIT master branch.

Recreated by:
server> mkfs.lustre --reformat --fsname=lus1 --mgs --mdt --index=1 /dev/sdd1
server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=1 /dev/sde1

client> mount.lustre -o flock 192.168.20.46 at o2ib0:/lus1 /mnt/lustre

The client mount crashes at lmv_get_info() without changes

<1>[  215.946538] BUG: unable to handle kernel NULL pointer
dereference at 0000000000000028
<1>[  215.946572] IP: [<ffffffffa07445cb>] lmv_get_info+0x32b/0x560 [lmv]
<0>[  215.947090] Call Trace:^M
<4>[  215.947143]  [<ffffffffa0655b70>] ll_fill_super+0x1f40/0x4330 [lustre]^M
<4>[  215.947214]  [<ffffffffa02cf527>] ?
lustre_start_mgc+0x227/0x2a90 [obdclass]^M
<4>[  215.947275]  [<ffffffffa02d3d60>] lustre_fill_super+0xa20/0x22f0
[obdclass]^M
<4>[  215.947304]  [<ffffffff810de91f>] ? set_anon_super+0x0/0xe0^M
<4>[  215.947361]  [<ffffffffa02d3340>] ? lustre_fill_super+0x0/0x22f0
[obdclass]^M
<4>[  215.947380]  [<ffffffff810df601>] mount_nodev+0x50/0x84^M
<4>[  215.947437]  [<ffffffffa02cc5d9>] lustre_mount+0x29/0x30 [obdclass]^M
<4>[  215.947454]  [<ffffffff810df009>] vfs_kern_mount+0xa8/0x1f3^M
<4>[  215.947471]  [<ffffffff810df1bc>] do_kern_mount+0x4d/0xe1^M
<4>[  215.947489]  [<ffffffff810f54d7>] do_mount+0x67d/0x6d5^M
<4>[  215.947507]  [<ffffffff810f57cc>] sys_mount+0x84/0xbd^M
<4>[  215.947527]  [<ffffffff81002aab>] system_call_fastpath+0x16/0x1b^M

Signed-off-by: Wendy Cheng <wendy.cheng at intel.com>

diff --git a/lustre/lmv/lmv_obd.c b/lustre/lmv/lmv_obd.c
index 3091bfb..5f4a18b 100644
--- a/lustre/lmv/lmv_obd.c
+++ b/lustre/lmv/lmv_obd.c
@@ -2443,6 +2443,16 @@ static int lmv_get_info(const struct lu_env
*env, struct obd_export *exp,
                         RETURN(rc);

                /*
+                * In the case of mis-configured OSS, instead of crashing
+                * the kernel during client mount, give them a warning and
+                * gracefully back out mount process w/ -ENXIO error.
+                */
+               if (lmv->tgts[0] == NULL) {
+                       CDEBUG(D_IOCTL, "NULL index\n");
+                       RETURN(-ENXIO);
+               }
+
+               /*
                 * Forwarding this request to first MDS, it should know LOV
                 * desc.
                 */

From andreas.dilger at intel.com  Sat Oct 12 06:59:03 2013
From: andreas.dilger at intel.com (Dilger, Andreas)
Date: Sat, 12 Oct 2013 06:59:03 +0000
Subject: [Lustre-devel] [Patch] Fix Client Kernel Crash with
 Mis-configured	Index Numbering
In-Reply-To: <CABgxfbH-js0JZzSAienCMJgXQYzVsKUYHbpfJWbvPfakyFGT6w@mail.gmail.com>
References: <CABgxfbH-js0JZzSAienCMJgXQYzVsKUYHbpfJWbvPfakyFGT6w@mail.gmail.com>
Message-ID: <E7E46B0B-79E1-4238-954E-6D6177A9421A@intel.com>

Hi Wendy,
Thanks for the patch. Could you please file a ticket at https://jira.hpdd.intel.com/ and submit the patch to our Gerrit repo (with minor tweaks as suggested below) so it is included in the next Lustre release. For more details please see:

https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes

You are totally correct that no user input should crash the kernel. The support for multiple MDTs in the same filesystem is relatively new (previously only MDT index 0 was allowed), and I guess nobody has ever tested what you did.

Cheers, Andreas

On 2013-10-11, at 21:29, "Wendy Cheng" <s.wendy.cheng at gmail.com<mailto:s.wendy.cheng at gmail.com>> wrote:

Ref: http://lists.lustre.org/pipermail/lustre-devel/2013-October/004270.html

I'm not really convinced the "index" setting of mkfs.lustre needs to
be started with "0". However, in the minimum, the client kernel should
not crash. The attached patch does this minimum fix; compiled and
tested with GIT master branch.

Recreated by:
server> mkfs.lustre --reformat --fsname=lus1 --mgs --mdt --index=1 /dev/sdd1
server> mkfs.lustre --reformat --ost --fsname=lus1
--mgsnode=192.168.20.46 at o2ib0 --index=1 /dev/sde1

client> mount.lustre -o flock 192.168.20.46 at o2ib0:/lus1 /mnt/lustre

The client mount crashes at lmv_get_info() without changes

<1>[  215.946538] BUG: unable to handle kernel NULL pointer
dereference at 0000000000000028
<1>[  215.946572] IP: [<ffffffffa07445cb>] lmv_get_info+0x32b/0x560 [lmv]
<0>[  215.947090] Call Trace:^M
<4>[  215.947143]  [<ffffffffa0655b70>] ll_fill_super+0x1f40/0x4330 [lustre]^M
<4>[  215.947214]  [<ffffffffa02cf527>] ?
lustre_start_mgc+0x227/0x2a90 [obdclass]^M
<4>[  215.947275]  [<ffffffffa02d3d60>] lustre_fill_super+0xa20/0x22f0
[obdclass]^M
<4>[  215.947304]  [<ffffffff810de91f>] ? set_anon_super+0x0/0xe0^M
<4>[  215.947361]  [<ffffffffa02d3340>] ? lustre_fill_super+0x0/0x22f0
[obdclass]^M
<4>[  215.947380]  [<ffffffff810df601>] mount_nodev+0x50/0x84^M
<4>[  215.947437]  [<ffffffffa02cc5d9>] lustre_mount+0x29/0x30 [obdclass]^M
<4>[  215.947454]  [<ffffffff810df009>] vfs_kern_mount+0xa8/0x1f3^M
<4>[  215.947471]  [<ffffffff810df1bc>] do_kern_mount+0x4d/0xe1^M
<4>[  215.947489]  [<ffffffff810f54d7>] do_mount+0x67d/0x6d5^M
<4>[  215.947507]  [<ffffffff810f57cc>] sys_mount+0x84/0xbd^M
<4>[  215.947527]  [<ffffffff81002aab>] system_call_fastpath+0x16/0x1b^M

Signed-off-by: Wendy Cheng <wendy.cheng at intel.com<mailto:wendy.cheng at intel.com>>

diff --git a/lustre/lmv/lmv_obd.c b/lustre/lmv/lmv_obd.c
index 3091bfb..5f4a18b 100644
--- a/lustre/lmv/lmv_obd.c
+++ b/lustre/lmv/lmv_obd.c
@@ -2443,6 +2443,16 @@ static int lmv_get_info(const struct lu_env
*env, struct obd_export *exp,
                        RETURN(rc);

               /*
+                * In the case of mis-configured OSS, instead of crashing

This comment should read "misconfigured MDT" ...

+                * the kernel during client mount, give them a warning and
+                * gracefully back out mount process w/ -ENXIO error.
+                */
+               if (lmv->tgts[0] == NULL) {
+                       CDEBUG(D_IOCTL, "NULL index\n");

"NULL target for MDT0\n"

+                       RETURN(-ENXIO);
+               }
+
+               /*
                * Forwarding this request to first MDS, it should know LOV
                * desc.
                */
_______________________________________________
Lustre-devel mailing list
Lustre-devel at lists.lustre.org<mailto:Lustre-devel at lists.lustre.org>
http://lists.lustre.org/mailman/listinfo/lustre-devel

From s.wendy.cheng at gmail.com  Sun Oct 13 13:42:28 2013
From: s.wendy.cheng at gmail.com (Wendy Cheng)
Date: Sun, 13 Oct 2013 06:42:28 -0700
Subject: [Lustre-devel] [Patch] Fix Client Kernel Crash with
 Mis-configured Index Numbering
In-Reply-To: <E7E46B0B-79E1-4238-954E-6D6177A9421A@intel.com>
References: <CABgxfbH-js0JZzSAienCMJgXQYzVsKUYHbpfJWbvPfakyFGT6w@mail.gmail.com>
	<E7E46B0B-79E1-4238-954E-6D6177A9421A@intel.com>
Message-ID: <CABgxfbE08pFaqmju2ovQQ0aMr2NNjJe02Aw_+JAQF+D-TURtaQ@mail.gmail.com>

On Fri, Oct 11, 2013 at 11:59 PM, Dilger, Andreas
<andreas.dilger at intel.com> wrote:
> Hi Wendy,
> Thanks for the patch. Could you please file a ticket at
> https://jira.hpdd.intel.com/ and submit the patch to our Gerrit repo (with
> minor tweaks as suggested below) so it is included in the next Lustre
> release. For more details please see:
>
> https://wiki.hpdd.intel.com/display/PUB/Submitting+Changes
>
>

Thanks ... Will do the check in sometime next week when I'm back to office .

-- Wendy

From indivar.nair at techterra.in  Fri Oct 18 03:46:26 2013
From: indivar.nair at techterra.in (Indivar Nair)
Date: Fri, 18 Oct 2013 09:16:26 +0530
Subject: [Lustre-devel] Lustre+ZFS Documentation
Message-ID: <CALuPYL1rmx6kBq=ru6GiqsRVTt-T5Awf_O7XNPiza6GqQpTeYg@mail.gmail.com>

Hi ...,

Is there a documentation on how to deploy Lustre over ZFS (MDS and OSS)?

I am not able to find anything on it in the Lustre Documentation page -
http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml

Regards,


Indivar Nair
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20131018/1c115477/attachment.html>

From indivar.nair at techterra.in  Mon Oct 21 13:10:18 2013
From: indivar.nair at techterra.in (Indivar Nair)
Date: Mon, 21 Oct 2013 18:40:18 +0530
Subject: [Lustre-devel] Lustre+ZFS Documentation
In-Reply-To: <CAB+_9Gqw=bXGKYk5PDCS2O6CNRZEgDnqxuRD3AvfWvK-VBxXUQ@mail.gmail.com>
References: <CALuPYL1rmx6kBq=ru6GiqsRVTt-T5Awf_O7XNPiza6GqQpTeYg@mail.gmail.com>
	<CAB+_9Gqw=bXGKYk5PDCS2O6CNRZEgDnqxuRD3AvfWvK-VBxXUQ@mail.gmail.com>
Message-ID: <CALuPYL1feMjW+0L3BueiY6dSqju==eDm9cou8830mhrwA3gmHg@mail.gmail.com>

Thanks Parinay,

The second link is what I was looking for.

However, it states that -
"The Lustre Orion code still requires some polish and you may encounter a
few bugs. It is not currently considered ready for real production usage."

I guess this is an old documentation and the line is no more valid with
Lustre 2.4 and later, right?

Regards,


Indivar Nair



On Mon, Oct 21, 2013 at 3:08 PM, Parinay Kondekar <
parinay_kondekar at xyratex.com> wrote:

> If not seen already ...
>
> http://zfsonlinux.org/lustre.html
>
> http://zfsonlinux.org/lustre-configure-single.html
>
> HTH
>
>
>
> On 18 October 2013 09:16, Indivar Nair <indivar.nair at techterra.in> wrote:
>
>> Hi ...,
>>
>> Is there a documentation on how to deploy Lustre over ZFS (MDS and OSS)?
>>
>> I am not able to find anything on it in the Lustre Documentation page -
>> http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml
>>
>> Regards,
>>
>>
>> Indivar Nair
>>
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20131021/39f72f2f/attachment.html>

From indivar.nair at techterra.in  Tue Oct 22 13:14:43 2013
From: indivar.nair at techterra.in (Indivar Nair)
Date: Tue, 22 Oct 2013 18:44:43 +0530
Subject: [Lustre-devel] Lustre+ZFS Documentation
In-Reply-To: <CAB+_9GoPa7h80Fu3VZNRAR9Mw=E5DHO4XnCdE26u5W3ozWtoDA@mail.gmail.com>
References: <CALuPYL1rmx6kBq=ru6GiqsRVTt-T5Awf_O7XNPiza6GqQpTeYg@mail.gmail.com>
	<CAB+_9Gqw=bXGKYk5PDCS2O6CNRZEgDnqxuRD3AvfWvK-VBxXUQ@mail.gmail.com>
	<CALuPYL1feMjW+0L3BueiY6dSqju==eDm9cou8830mhrwA3gmHg@mail.gmail.com>
	<CAB+_9GoPa7h80Fu3VZNRAR9Mw=E5DHO4XnCdE26u5W3ozWtoDA@mail.gmail.com>
Message-ID: <CALuPYL2r78AtXyyWQDhvy7at9kXVmUrfHrhYBmzOb=bmNTts3g@mail.gmail.com>

Got it.

Thanks Parinay.


On Tue, Oct 22, 2013 at 4:19 PM, Parinay Kondekar <
parinay_kondekar at xyratex.com> wrote:

> 2.4 changelog - https://wiki.hpdd.intel.com/display/PUB/Lustre+2.4 = OSD
> Server stack ; so Yes.
>
> HTH
>
>
> On 21 October 2013 18:40, Indivar Nair <indivar.nair at techterra.in> wrote:
>
>> Thanks Parinay,
>>
>> The second link is what I was looking for.
>>
>> However, it states that -
>> "The Lustre Orion code still requires some polish and you may encounter a
>> few bugs. It is not currently considered ready for real production usage."
>>
>> I guess this is an old documentation and the line is no more valid with
>> Lustre 2.4 and later, right?
>>
>> Regards,
>>
>>
>> Indivar Nair
>>
>>
>>
>> On Mon, Oct 21, 2013 at 3:08 PM, Parinay Kondekar <
>> parinay_kondekar at xyratex.com> wrote:
>>
>>> If not seen already ...
>>>
>>> http://zfsonlinux.org/lustre.html
>>>
>>> http://zfsonlinux.org/lustre-configure-single.html
>>>
>>> HTH
>>>
>>>
>>>
>>> On 18 October 2013 09:16, Indivar Nair <indivar.nair at techterra.in>wrote:
>>>
>>>> Hi ...,
>>>>
>>>> Is there a documentation on how to deploy Lustre over ZFS (MDS and OSS)?
>>>>
>>>> I am not able to find anything on it in the Lustre Documentation page -
>>>> http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml
>>>>
>>>> Regards,
>>>>
>>>>
>>>> Indivar Nair
>>>>
>>>> _______________________________________________
>>>> Lustre-devel mailing list
>>>> Lustre-devel at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20131022/1e58643c/attachment.html>

From hornc at cray.com  Tue Oct 15 20:03:33 2013
From: hornc at cray.com (Chris Horn)
Date: Tue, 15 Oct 2013 20:03:33 -0000
Subject: [Lustre-devel] Starting and using  a self written LND
In-Reply-To: <5252C868.2050104@stud.uni-heidelberg.de>
References: <5252C868.2050104@stud.uni-heidelberg.de>
Message-ID: <CD2D33E7-4FB2-4D37-942F-06C18ACDE3CB@cray.com>

LNET only calls LND startup functions for networks defined in the LNET module setting (modprobe.conf.local or equivalent). See lnet_startup_lndnis() and lnet_parse_networks().

Chris Horn

On Oct 7, 2013, at 9:42 AM, Tobias Groschup <groschup at stud.uni-heidelberg.de> wrote:

> Hi all,
> 
> I am still at the task of writing a new LND. Finally got Lustre to build 
> with our own software stack.
> 
> Now I want to test the new LND. Of course, it does not implement much 
> functionality yet. In fact, it only has a stump for the send and receive 
> functions, and a startup and shutdown function which sets up the 
> software context for the network card and closes it.
> I can load the module, and LNet does not crash after adding the new 
> entries to the LND types enum in libcfs/include/libcfs/libcfs private.h 
> and to libcfs_netstrfns[] in libcfs/libcfs/nidstrings.c.
> 
> So the question is: how does one invoke the startup function after 
> loading an LND module? I tried lclt and its network up command, which 
> returned without error, but I did not see the debug output which is 
> printed via printks.
> 
> And: are the modifications to the libcfs mentioned above all that are 
> needed to declare an LND to libcfs?
> 
> Thanks and Greetings
> Tobias
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

From parinay_kondekar at xyratex.com  Tue Oct 22 10:32:05 2013
From: parinay_kondekar at xyratex.com (Parinay Kondekar)
Date: Tue, 22 Oct 2013 10:32:05 -0000
Subject: [Lustre-devel] Lustre+ZFS Documentation
In-Reply-To: <CALuPYL1feMjW+0L3BueiY6dSqju==eDm9cou8830mhrwA3gmHg@mail.gmail.com>
References: <CALuPYL1rmx6kBq=ru6GiqsRVTt-T5Awf_O7XNPiza6GqQpTeYg@mail.gmail.com>
	<CAB+_9Gqw=bXGKYk5PDCS2O6CNRZEgDnqxuRD3AvfWvK-VBxXUQ@mail.gmail.com>
	<CALuPYL1feMjW+0L3BueiY6dSqju==eDm9cou8830mhrwA3gmHg@mail.gmail.com>
Message-ID: <CAB+_9GoPa7h80Fu3VZNRAR9Mw=E5DHO4XnCdE26u5W3ozWtoDA@mail.gmail.com>

2.4 changelog - https://wiki.hpdd.intel.com/display/PUB/Lustre+2.4 = OSD
Server stack ; so Yes.

HTH


On 21 October 2013 18:40, Indivar Nair <indivar.nair at techterra.in> wrote:

> Thanks Parinay,
>
> The second link is what I was looking for.
>
> However, it states that -
> "The Lustre Orion code still requires some polish and you may encounter a
> few bugs. It is not currently considered ready for real production usage."
>
> I guess this is an old documentation and the line is no more valid with
> Lustre 2.4 and later, right?
>
> Regards,
>
>
> Indivar Nair
>
>
>
> On Mon, Oct 21, 2013 at 3:08 PM, Parinay Kondekar <
> parinay_kondekar at xyratex.com> wrote:
>
>> If not seen already ...
>>
>> http://zfsonlinux.org/lustre.html
>>
>> http://zfsonlinux.org/lustre-configure-single.html
>>
>> HTH
>>
>>
>>
>> On 18 October 2013 09:16, Indivar Nair <indivar.nair at techterra.in> wrote:
>>
>>> Hi ...,
>>>
>>> Is there a documentation on how to deploy Lustre over ZFS (MDS and OSS)?
>>>
>>> I am not able to find anything on it in the Lustre Documentation page -
>>> http://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml
>>>
>>> Regards,
>>>
>>>
>>> Indivar Nair
>>>
>>> _______________________________________________
>>> Lustre-devel mailing list
>>> Lustre-devel at lists.lustre.org
>>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20131022/07ec9791/attachment.html>