[Lustre-discuss] Loop device performance

[Dilger, Andreas]

On 2013/08/25 6:39 AM, "Nikolay Kvetsinski" <nkvecinski at gmail.com> wrote:

>Hello, I have a production script that do read operations to a lot of
>small files. I read that one can gain performance boost with small files
>by using a loop device on top of Lustre. So a created 500 GB file striped
>across all of my OSTs(which
> are 8). I formatted the file with ext2 fs, and mounted it on a client.
>Just for the sake of testing a simple bash script finds all files with a
>given file type and cat the first 10 lines in /dev/null.
>
>
>When I run the script on the Lustre cluster I get :
>
>
>time sh test.sh
>
>
>real    1m16.804s
>user    0m2.539s
>sys     0m5.363s
>
>
>
>If I immediately re-run the script the time is :
>
>
>real    0m12.158s
>user    0m2.218s
>sys     0m5.430s
>
> 
>
>
>There are 5406 files that meet the filetype criteria.
>
>
>When I run the script on the mounted loop device I get :
>
>
>real    2m30.177s
>user    0m2.290s
>sys     0m4.880s
>
>And immediate re-run gives me :
>
>real    0m7.810s
>user    0m2.187s
>sys     0m5.360s
>
>
>I`m usig lustre-2.4.0-2.6.32_358.6.2.el6_lustre.g230b174.x86_64_gd3f91c4.
>Also
>set all of "small files" optimizations like, no striping for the dirs
>containing the small files, max_dirty_mb=256, max_rpcs_in_flight=32,
>staahead=8192 and lnet.debug=0.
>Is it normal to get two times slower access times with the mounted loop
>device ??

It depends on how the loop device is doing IO on the underlying objects.
It may
be that ext2 isn't the best filesystem for this.  You could try formatting
it with:

     mke2fs -t ext4 -O ^journal {device}

which will enable the flex_bg,extents and other ext4 features but disables
the
journal (which I assume you don't need because you are formatting as ext2
originally).  You should also mount with "-t ext4".

The flex_bg and mballoc features of ext4 may help improve the IO going to
the
back-end storage and improve the performance when running over loop
devices.

Cheers, Andreas
-- 
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division