<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
On Jan 5, 2023, at 04:12, Nick dan via lustre-discuss <<a href="mailto:lustre-discuss@lists.lustre.org" class="">lustre-discuss@lists.lustre.org</a>> wrote:
<div>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">We have configured the Lustre filesystem and mounted the same on the client. This is being done using IP as highlighted below</div>
<div class=""><br class="">
</div>
</div>
</div>
</blockquote>
<blockquote type="cite" class="">[root@cl01] user]# df -hT</blockquote>
<blockquote type="cite" class="">11.11.1.211@tcp:/lustre lustre 9.2G 42M 8.7G 1% /mnt/lustre</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">Part 2:</div>
<div class="">Now for the next part, we don't want to use this command: (mount -t lustre 11.11.1.211@tcp:/lustre /mnt/lustre) as the connection is through IP.</div>
<div class="">We want to mount a particular block device as a lustre file system. For example, in the image below we have created 2 partitions on the server of 10G each and we are sharing ost partition to the client as a block device.</div>
<div class="">when we are sharing the block device. This is our client and we have got 10G of /mnt/ost. </div>
<div class=""><br class="">
</div>
</div>
</div>
</blockquote>
<blockquote type="cite" class="">[root@cll01 user]# lsblk</blockquote>
<blockquote type="cite" class="">nvme0n1 259:0 0 10G 0 disk</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">After mounting using <b class="">mount /dev/nvme0n1 /mnt/lustre/ </b></div>
</div>
</div>
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">The filesystem type is ext4 and not lustre as mentioned below.</div>
</div>
</div>
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">[root@cl01] user]# df -hT</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">/dev/nvme0n1 ext4 9.3G 42M 8.7G 1% /mnt/lustre</div>
</div>
</div>
</blockquote>
<blockquote type="cite" class=""><br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">We want to mount using the lustre filesystem and not ext4.</div>
<div class="">Is there a need to change the lnet configuration? What else is need to be done?</div>
</div>
</div>
</blockquote>
<br class="">
</div>
<div>What you are doing will not work. Mounting the shared block device directly as type ext4 will (and probably already has) corrupt the filesystem on /dev/nvme0n1 because the two kernels do not know the device is in use on two nodes.</div>
<div><br class="">
</div>
<div>As a starting point, if you have shared NVMe devices (presumably via NVMeoF) visible on multiple nodes you should enable the "mmp" feature (Multi-Mount Protection) on the unmounted ext4 filesystem like "tune2fs -O mmp /dev/nvme0n1" and it will at least
prevent the filesystem from being mounted directly on two nodes at one time.</div>
<div><br class="">
</div>
<div>However, you still cannot mount this device directly on the client. Lustre is a network filesystem and not a shared-block filesystem. If you have IB or RoCE you can use OFED/MOFED with the o2ib LND and it will use RDMA to transfer data to/from the OST
devices. With IB RDMA Lustre can have network bandwidth comparable to locally attached NVMe devices, and can also scale far larger than directly-attached storage would allow.</div>
<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>