[lustre-discuss] Lustre poor performance

Jones, Peter A peter.a.jones at intel.com
Thu Aug 17 20:56:51 PDT 2017


Riccardo

I expect that it will be useful to know which version of ZFS you are using

Peter




On 8/17/17, 8:21 PM, "lustre-discuss on behalf of Riccardo Veraldi" <lustre-discuss-bounces at lists.lustre.org on behalf of Riccardo.Veraldi at cnaf.infn.it> wrote:

>Hello,
>
>I am running Lustre 2.10.0 on Centos 7.3
>I have one MDS and two OSSes, each with one OST
>each OST is a ZFS raidz1 with 6 nvme disks each.
>The configuration of ZFS is done in a way to allow maximum write
>performances:
>
>zfs set sync=disabled drpffb-ost02
>zfs set atime=off drpffb-ost02
>zfs set redundant_metadata=most drpffb-ost02
>zfs set xattr=sa drpffb-ost02
>zfs set recordsize=1M drpffb-ost02
>
>every NVMe disk has 4K byte sector, zfs -o  ashift=12
>
>In a LOCAL raidz1 configuration I get 3.6GB/sec writings and 5GB/sec
>readings.
>
>The same configuration thru Lustre has very poor performances, 1.3GB/sec
>writes and 2GB/sec reads
>
>There must be something else to look for having better performances but
>a local ZFS raidz1 is working pretty good.
>
>this is the Lustre partition client side:
>
>172.21.42.159 at tcp5:/drpffb                  10T  279G  9.8T   3% /drpffb
>
>UUID                       bytes        Used   Available Use% Mounted on
>drpffb-MDT0000_UUID        19.1G        2.1M       19.1G   0% /drpffb[MDT:0]
>drpffb-OST0001_UUID         5.0T      142.2G        4.9T   3% /drpffb[OST:1]
>drpffb-OST0002_UUID         5.0T      136.4G        4.9T   3% /drpffb[OST:2]
>
>filesystem_summary:        10.0T      278.6G        9.7T   3% /drpffb
>
>Tests both on Lustre/ZFS and local ZFS are based on 50 threads writing
>4GB of data each and 50 threads reading using iozone:
>
>iozone  -i 0 -t 50 -i 1 -t 50 -s4g
>
>I do not know what else I can do to improve performances
>
>here some details on the OSSes
>
>OSS01:
>
>NAME                 USED  AVAIL  REFER  MOUNTPOINT
>drpffb-ost01        39.4G  4.99T   153K  none
>drpffb-ost01/ost01  39.4G  4.99T  39.4G  none
>
>  pool: drpffb-ost01
> state: ONLINE
>  scan: none requested
>config:
>
>    NAME         STATE     READ WRITE CKSUM
>    drpffb-ost01  ONLINE       0     0     0
>      raidz1-0   ONLINE       0     0     0
>        nvme0n1  ONLINE       0     0     0
>        nvme1n1  ONLINE       0     0     0
>        nvme2n1  ONLINE       0     0     0
>        nvme3n1  ONLINE       0     0     0
>        nvme4n1  ONLINE       0     0     0
>        nvme5n1  ONLINE       0     0     0
>
>OSS02:
>
>NAME                 USED  AVAIL  REFER  MOUNTPOINT
>drpffb-ost02        62.2G  4.97T   153K  none
>drpffb-ost02/ost02  62.2G  4.97T  62.2G  none
>
>  pool: drpffb-ost02
> state: ONLINE
>  scan: none requested
>config:
>
>    NAME         STATE     READ WRITE CKSUM
>    drpffb-ost02  ONLINE       0     0     0
>      raidz1-0   ONLINE       0     0     0
>        nvme0n1  ONLINE       0     0     0
>        nvme1n1  ONLINE       0     0     0
>        nvme2n1  ONLINE       0     0     0
>        nvme3n1  ONLINE       0     0     0
>        nvme4n1  ONLINE       0     0     0
>        nvme5n1  ONLINE       0     0     0
>
>thanks to anyone who may help giving hints.
>
>Rick
>
>
>
>_______________________________________________
>lustre-discuss mailing list
>lustre-discuss at lists.lustre.org
>http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org


More information about the lustre-discuss mailing list