[lustre-discuss] ost-survey hangs on lustre-2.10.0 client when using different size values

[Dilger, Andreas]

On Nov 23, 2017, at 18:02, Jae-Hyuck Kwak <jhkwak at kisti.re.kr> wrote:
> 
> Hi, I'm newbie on lustre.
> 
> I am using Lustre-2.10.0. When I use ost-survey with default -s value, 
> it works well. But when I changes -s value, it hangs at read step.
> (see below)
> 
> ost-survey seems to change max_cached_mb to 256 * system page size 
> in MB which is 16 in our lustre environment.
> 
> I changed this value to a larger value and it works well.
> 
> I think minimum max_cached_mb value for ost-survey has something wrong.
> 
> Do you have any comments or something?

It would be useful to get stack traces and/or console messages from the
client and server after it hangs.  Best would be to file a new ticket in
Jira.

Cheers, Andreas

> 
> [root at cn11 ~]# ost-survey /lustre
> /usr/bin/ost-survey: 11/24/17 OST speed survey on /lustre from 10.0.0.111 at o2ib1
> Number of Active OST devices : 8
> Page Size is 4096
> write index 0 done.
> write index 1 done.
> write index 2 done.
> write index 3 done.
> write index 4 done.
> write index 5 done.
> write index 6 done.
> write index 7 done.
> read index 0 done.
> read index 1 done.
> read index 2 done.
> read index 3 done.
> read index 4 done.
> read index 5 done.
> read index 6 done.
> read index 7 done.
> Worst  Read OST indx: 0 speed: 544.158868
> Best   Read OST indx: 7 speed: 745.733589
> Read Average: 642.827346 +/- 63.038560 MB/s
> Worst  Write OST indx: 2 speed: 165.359455
> Best   Write OST indx: 0 speed: 547.385382
> Write Average: 284.413980 +/- 118.452906 MB/s
> Ost#  Read(MB/s)  Write(MB/s)  Read-time  Write-time
> ----------------------------------------------------
> 0     544.159       547.385        0.055      0.055
> 1     597.003       245.347        0.050      0.122
> 2     622.987       165.359        0.048      0.181
> 3     648.340       172.648        0.046      0.174
> 4     730.477       384.788        0.041      0.078
> 5     607.521       218.656        0.049      0.137
> 6     646.398       262.812        0.046      0.114
> 7     745.734       278.317        0.040      0.108
> [root at cn11 ~]# ost-survey -s 10 /lustre
> /usr/bin/ost-survey: 11/24/17 OST speed survey on /lustre from 10.0.0.111 at o2ib1
> Number of Active OST devices : 8
> Page Size is 4096
> write index 0 done.
> write index 1 done.
> write index 2 done.
> write index 3 done.
> write index 4 done.
> write index 5 done.
> write index 6 done.
> write index 7 done.
> read index 0 done.
> read index 1 done.
> read index 2 done.
> read index 3 done.
> read index 4 done.
> read index 5 done.
> read index 6 done.
> read index 7 done.
> Worst  Read OST indx: 4 speed: 323.487301
> Best   Read OST indx: 3 speed: 425.770117
> Read Average: 378.171698 +/- 32.609314 MB/s
> Worst  Write OST indx: 5 speed: 142.140286
> Best   Write OST indx: 0 speed: 361.154509
> Write Average: 248.073472 +/- 75.279234 MB/s
> Ost#  Read(MB/s)  Write(MB/s)  Read-time  Write-time
> ----------------------------------------------------
> 0     335.843       361.155        0.030      0.028
> 1     386.369       244.261        0.026      0.041
> 2     396.778       214.615        0.025      0.047
> 3     425.770       158.509        0.023      0.063
> 4     323.487       330.927        0.031      0.030
> 5     364.589       142.140        0.027      0.070
> 6     386.113       314.592        0.026      0.032
> 7     406.425       218.388        0.025      0.046
> [root at cn11 ~]# ost-survey -s 100 /lustre
> /usr/bin/ost-survey: 11/24/17 OST speed survey on /lustre from 10.0.0.111 at o2ib1
> Number of Active OST devices : 8
> Page Size is 4096
> write index 0 done.
> write index 1 done.
> write index 2 done.
> write index 3 done.
> write index 4 done.
> write index 5 done.
> write index 6 done.
> write index 7 done.
> (hang)
> 
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation