[lustre-discuss] Lustre poor performance

[Riccardo Veraldi]

Hello,

I am running Lustre 2.10.0 on Centos 7.3
I have one MDS and two OSSes, each with one OST
each OST is a ZFS raidz1 with 6 nvme disks each.
The configuration of ZFS is done in a way to allow maximum write
performances:

zfs set sync=disabled drpffb-ost02
zfs set atime=off drpffb-ost02
zfs set redundant_metadata=most drpffb-ost02
zfs set xattr=sa drpffb-ost02
zfs set recordsize=1M drpffb-ost02

every NVMe disk has 4K byte sector, zfs -o  ashift=12

In a LOCAL raidz1 configuration I get 3.6GB/sec writings and 5GB/sec
readings.

The same configuration thru Lustre has very poor performances, 1.3GB/sec
writes and 2GB/sec reads

There must be something else to look for having better performances but
a local ZFS raidz1 is working pretty good.

this is the Lustre partition client side:

172.21.42.159 at tcp5:/drpffb                  10T  279G  9.8T   3% /drpffb

UUID                       bytes        Used   Available Use% Mounted on
drpffb-MDT0000_UUID        19.1G        2.1M       19.1G   0% /drpffb[MDT:0]
drpffb-OST0001_UUID         5.0T      142.2G        4.9T   3% /drpffb[OST:1]
drpffb-OST0002_UUID         5.0T      136.4G        4.9T   3% /drpffb[OST:2]

filesystem_summary:        10.0T      278.6G        9.7T   3% /drpffb

Tests both on Lustre/ZFS and local ZFS are based on 50 threads writing
4GB of data each and 50 threads reading using iozone:

iozone  -i 0 -t 50 -i 1 -t 50 -s4g

I do not know what else I can do to improve performances

here some details on the OSSes

OSS01:

NAME                 USED  AVAIL  REFER  MOUNTPOINT
drpffb-ost01        39.4G  4.99T   153K  none
drpffb-ost01/ost01  39.4G  4.99T  39.4G  none

  pool: drpffb-ost01
 state: ONLINE
  scan: none requested
config:

    NAME         STATE     READ WRITE CKSUM
    drpffb-ost01  ONLINE       0     0     0
      raidz1-0   ONLINE       0     0     0
        nvme0n1  ONLINE       0     0     0
        nvme1n1  ONLINE       0     0     0
        nvme2n1  ONLINE       0     0     0
        nvme3n1  ONLINE       0     0     0
        nvme4n1  ONLINE       0     0     0
        nvme5n1  ONLINE       0     0     0

OSS02:

NAME                 USED  AVAIL  REFER  MOUNTPOINT
drpffb-ost02        62.2G  4.97T   153K  none
drpffb-ost02/ost02  62.2G  4.97T  62.2G  none

  pool: drpffb-ost02
 state: ONLINE
  scan: none requested
config:

    NAME         STATE     READ WRITE CKSUM
    drpffb-ost02  ONLINE       0     0     0
      raidz1-0   ONLINE       0     0     0
        nvme0n1  ONLINE       0     0     0
        nvme1n1  ONLINE       0     0     0
        nvme2n1  ONLINE       0     0     0
        nvme3n1  ONLINE       0     0     0
        nvme4n1  ONLINE       0     0     0
        nvme5n1  ONLINE       0     0     0

thanks to anyone who may help giving hints.

Rick