[lustre-discuss] Question about Single OST Performance Bottleneck

[Zuoru Yang]

Hi all,



I have a question regarding the performance bottleneck of a single OST. We have high-performance SAN storage, which can provide high raw write bandwidth with a single LUN (e.g., about 10GiB/s).



We evaluate the write performance of a single OST (based on a single LUN) via obdfilter-survey in OSS:

$> nobjlo=32 nobjhi=32 thrlo=32 thrhi=32 rszlo=2048 rszhi=2048 size=102400 targets="l_lfs-OST0000" case=disk sh obdfilter-survey



The output write bandwidth of this single OST is around 1.2GiB/s. However, if we increase the write pressure of obdfilter-survey in OSS:

$> nobjlo=64 nobjhi=64 thrlo=64 thrhi=64 rszlo=2048 rszhi=2048 size=102400 targets="l_lfs-OST0000" case=disk sh obdfilter-survey



The output write bandwidth of this single OST cannot increase anymore (i.e., still around 1.2GiB/s), which cannot further exploit the write bandwidth of a single LUN. Also, if we use iostat to check this LUN, its write pressure does not increase (e.g., aqu-sz remains at around 20).



We consider that obdfilter-survey evaluation does not incur the impact on the client and LNet, it would only include oss layer, ofd layer, osd-ldiskfs layer, and ldiskfs layer. For the ldiskfs layer, we think it would not be the performance bottleneck, since we have measured its performance by mounting the single LUN via ldiskfs and issuing "dd" via multi-threading (the write performance can reach 7-8GiB/s with 64 writer threads).



Can anyone provide some insights of performance bottleneck for a single OST? Is there any mechanism to control the write concurrency of a single OST? I also disable writecache and readcache during the evaluation, so it should exclude the impact of pagecache. Thanks.



Regards,

Zuoru
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20240222/3758b748/attachment.htm>