[lustre-discuss] Robinhood exhausting RPC resources against 2.5.5 lustre file systems

[Colin Faber]

Hi Jessica,

What STAGE_GET_FID_threads_max and STAGE_GET_INFO_FS_threads_max values are
you using?

Also what nb_threads values are you using for your robinhood configuration?

Also, when you see the disconnects between the clients and the MDS, what
does the MDS load actually look like? Is the system overloaded?

Did you happen to ever benchmark your MDS to find out what kind of
performance you can expect out of it? (mdtest?)

-cf


On Thu, May 18, 2017 at 12:36 PM, Jessica Otey <jotey at nrao.edu> wrote:

> Hi Megan,
>
> Thanks for your input. We use percona, a drop-in replacement for mysql...
> The robinhood activity floods the MDT, but it does not seem to produce any
> excessive load on the robinhood box...
>
> Anyway, FWIW...
>
> ~]# mysql --version
> mysql  Ver 14.14 Distrib 5.5.54-38.6, for Linux (x86_64) using readline 5.1
>
> Product:         robinhood
> Version:         3.0-1
> Build:           2017-03-13 10:29:26
>
> Compilation switches:
>     Lustre filesystems
>     Lustre Version: 2.5
>     Address entries by FID
>     MDT Changelogs supported
>
> Database binding: MySQL
>
> RPM: robinhood-lustre-3.0-1.lustre2.5.el6.x86_64
> Lustre rpms:
>
> lustre-client-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
> lustre-client-modules-2.5.5-2.6.32_642.15.1.el6.x86_64_g22a210f.x86_64
>
> On 5/18/17 11:55 AM, Ms. Megan Larko wrote:
>
> With regards to (WRT) Subject "Robinhood exhausting RPC resources against
> 2.5.5   lustre file systems", what version of robinhood and what version of
> MySQL database?   I mention this because I have been working with
> robinhood-3.0-0.rc1 and initially MySQL-5.5.32 and Lustre 2.5.42.1 on
> kernel-2.6.32-573 and had issues in which the robinhood server consumed
> more than the total amount of 32 CPU cores on the robinhood server (with
> 128 G RAM) and would functionally hang the robinhood server.   The issue
> was solved for me by changing to MySQL-5.6.35.   It was the "sort" command
> in robinhood that was not working well with the MySQL-5.5.32.
>
> Cheers,
> megan
>
>
>
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
> lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&
> d=DwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=x9pM59OqndbWw-
> lPPdr8w1Vud29EZigcxcNkz0uw5oQ&m=4AakV9Mc0JlUxzY-
> sNPH0a7okSbxM9bxL6_RSKwd4L4&s=k6F_tUj9P27MaMwmajor1q-
> VdHqBPo-xKG2f3quLLgc&e=
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20170518/09d6469e/attachment.htm>