[lustre-discuss] POC Performance

[Riccardo Veraldi]

Are you testing this in a virtual machine environment ?

if you are aiming for Lustre performacne you should not run virtual
machines especially on the OSS side.
Also how many clients are you using for reading/writing ? to get
performance out of lustre or any other parallel filesystem, you need to
stripe and read/write in paralle from different clients. Also if you run
Lustre/ZFS you need a good amount of RAM and 8GB on the OSS side sounds
like too few for me.

In my environment I have 8x OSS each one with 128GB RAM and a raidz with
4x Micron NVMe 9200  disks.
Each OSS has a Infiniband EDR connection. In my test I use 32 threads
per client writing from 8 clients at the same time on the Lustre
filesystem and
I can saturate the NVMe SSD disks performance. I get almost 80GB/s write
performance and 90GB/s read performance over Infiniband.



On 4/8/19 12:46 PM, Bill Carlson wrote:
> Hello,
>
> I've been chasing a proof of concept for Lustre, so far performance
> tests are not promising.
> Basic setup:
>
> MGS/MDT: VM, 4 cpu, 8GB ram
> OSS #1: VM, 16 cpu, 8GB ram
> OSS #1: hardware, 20 cpu, 1TB ram
>
> I've been using sybench fileio for tests, 16k on 50GB over 5 minutes.
>
> Basic test results, performed on OSS with mounted FS:
> Base ext4 SSD, OSS #2:
> Sequential write: 1 GB/s
> Random r/w: 551 MB/s read, 367 MB/s write
>
> ZFS dataset SSD, OSS #2:
> Sequential write:  397 MB/s
> Randow r/w: 109 MB/s read, 73 MB/s write
>
> About 5 times slower. Expected?
>
>
> ZFS OST SSD, OSS #2:
> Sequential write: 9 MB/s
> Randow r/w: 18MB/s read, 12.5MB/s write
>
> Over 30-110 times slower than basic disk, that just doesn't seem right.
> I also tried ldiskfs, not much difference.
>
> I tried various changes, ZFS compression on, atime off, xattr sa.
>
> Watching the system via atop during a 5 minute OST test, disks are not
> 100% busy and CPU is mostly idle. Network is all lo.
>
> What am I missing? I assumed random r/w would be pretty slow, but not
> sequential.
>
> Thanks,
>