[lustre-discuss] DF bug with lustre 2.12.4

[Konzem, Kevin P]

Hey all,
Ive been struggling with a problem with our newly updated lustre 2.12 cluster, and I don't really know if its a bug, or configuration problem, or what.
So here's the setup: I've recently set up a small 2-OST single MDT 2.10 cluster to emulate our production cluster, and test the process of upgrading to 2.12.4. The upgrade went fine, however there is a problem with how df reports space on the lustre filesystem that is causing problems with our processing software. The software includes a df check to make sure the filesystem isn't too full before beginning a job. The problem is, that when multiple df commands are run against the lustre filesystem from the same client, occasionally the command will return a 0 in the available field, which in turn makes the software think the filesystem is full, then drop jobs. I can test this by running 'while [ true ];do /bin/df -TP /performance;done' on two sessions on the same client. As soon as I start the second while loop, the outputs go from:
Filesystem                 Type   1024-blocks   Used Available Capacity Mounted on
192.168.0.181 at tcp:/perform lustre    71467728 100416  67664944       1% /performance

to:
Filesystem                 Type   1024-blocks  Used Available Capacity Mounted on
192.168.0.181 at tcp:/perform lustre           0    -0        -0      50% /performance
I am using lustre 2.12.4 on the client as well, so Ive ruled out version mismatch issues at least.

I've checked all the mount settings between the prod 2.10 cluster and the dev 2.12 cluster, and everything I can find looks the same. The 2.10 prod cluster does not have this problem, and the dev cluster did not have the problem before upgrading from 2.10. Has anyone else experienced this? Can anyone at least point me in the right direction on how to fix it?

Thanks,

Kevin Konzem
Linux Systems Administrator
USS Contractor to USGS/EROS
Technical Services Support Contract
47914 252nd street
Sioux Falls, South Dakota 57198
Phone: (605) 594-2509
Fax: (605) 594-2828
Email: kkonzem at contractor.usgs.gov<mailto:kkonzem at contractor.usgs.gov>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20200220/4ac0456f/attachment-0001.html>