[Lustre-discuss] lnet router tuning

[Michael Kluge]

Has anyone else a 10GE<->IB Lustre router? What are the typical 
performance numbers? How close do you get to 1GB/s?

Michael


Am 10.09.2010 17:55, schrieb Michael Kluge:
> And here are my params:
>
> root at doss05:/home/tests/lnet# for F in /sys/module/lnet/parameters/* ;
> do echo -n "$F: "; cat $F ; done
> /sys/module/lnet/parameters/accept: secure
> /sys/module/lnet/parameters/accept_backlog: 127
> /sys/module/lnet/parameters/accept_port: 988
> /sys/module/lnet/parameters/accept_timeout: 5
> /sys/module/lnet/parameters/auto_down: 1
> /sys/module/lnet/parameters/avoid_asym_router_failure: 0
> /sys/module/lnet/parameters/check_routers_before_use: 0
> /sys/module/lnet/parameters/config_on_load: 0
> /sys/module/lnet/parameters/dead_router_check_interval: 0
> /sys/module/lnet/parameters/forwarding: enabled
> /sys/module/lnet/parameters/ip2nets:
> /sys/module/lnet/parameters/large_router_buffers: 512
> /sys/module/lnet/parameters/live_router_check_interval: 0
> /sys/module/lnet/parameters/local_nid_dist_zero: 1
> /sys/module/lnet/parameters/networks: tcp0(eth2),o2ib(ib1)
> /sys/module/lnet/parameters/peer_buffer_credits: 0
> /sys/module/lnet/parameters/portals_compatibility: none
> /sys/module/lnet/parameters/router_ping_timeout: 50
> /sys/module/lnet/parameters/routes:
> /sys/module/lnet/parameters/small_router_buffers: 8192
> /sys/module/lnet/parameters/tiny_router_buffers: 1024
>
> I have not used ip2nets but configure routing but put explict routing
> statements into the modprobe.d/ files. Is that OK?
>
>
> Michael
>
>
> Am 10.09.2010 um 17:48 schrieb Michael Kluge:
>
>> OK, IB back to back is at 1,2 GB/s, 10GE back to back at 950 MB/s,
>> with additional lnet router I see 550 MB/s. Time for lnet tuning?
>>
>> Michael
>>
>>> Hi Andreas,
>>>
>>> Am 10.09.2010 um 16:35 schrieb Andreas Dilger:
>>>
>>>> On 2010-09-10, at 08:23, Michael Kluge wrote:
>>>>> I have a Lustre 1.8.3 setup where I'd like to some lnet router
>>>>> performance tests with routing between DDR IB<->10GE networks.
>>>>> Currently I have three nodes, one with DDR IB, one with 10GE and
>>>>> one with both that does the routing. A first short lnet test shows
>>>>> 520-550 MB/s performance.
>>>>>
>>>>> Has anyone an idea which of the variables of the lnet module are
>>>>> worth playing with to get this number a bit closer to 1GB/s?
>>>>
>>>> I would start by testing the performance on just the 10GigE side,
>>>> and then separately on the IB side, to verify you are getting the
>>>> expected performance from the components before trying them both
>>>> together. Often it is necessary to tune the ethernet send/receive
>>>> buffers.
>>>
>>> Ethernet back to back is at 950 MB/s. I have not looked at IB back to
>>> back yet.
>>>
>>>
>>> Michael
>>>
>>> --
>>>
>>> Michael Kluge, M.Sc.
>>>
>>> Technische Universität Dresden
>>> Center for Information Services and
>>> High Performance Computing (ZIH)
>>> D-01062 Dresden
>>> Germany
>>>
>>> Contact:
>>> Willersbau, Room WIL A 208
>>> Phone: (+49) 351 463-34217
>>> Fax: (+49) 351 463-37773
>>> e-mail: michael.kluge at tu-dresden.de <mailto:michael.kluge at tu-dresden.de>
>>> WWW: http://www.tu-dresden.de/zih
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org>
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
>> --
>>
>> Michael Kluge, M.Sc.
>>
>> Technische Universität Dresden
>> Center for Information Services and
>> High Performance Computing (ZIH)
>> D-01062 Dresden
>> Germany
>>
>> Contact:
>> Willersbau, Room WIL A 208
>> Phone: (+49) 351 463-34217
>> Fax: (+49) 351 463-37773
>> e-mail: michael.kluge at tu-dresden.de <mailto:michael.kluge at tu-dresden.de>
>> WWW: http://www.tu-dresden.de/zih
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org <mailto:Lustre-discuss at lists.lustre.org>
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> --
>
> Michael Kluge, M.Sc.
>
> Technische Universität Dresden
> Center for Information Services and
> High Performance Computing (ZIH)
> D-01062 Dresden
> Germany
>
> Contact:
> Willersbau, Room WIL A 208
> Phone: (+49) 351 463-34217
> Fax: (+49) 351 463-37773
> e-mail: michael.kluge at tu-dresden.de <mailto:michael.kluge at tu-dresden.de>
> WWW: http://www.tu-dresden.de/zih
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss