[Lustre-discuss] RDMA limitation?

[Jiahua]

Thanks for your answers! More questions:

* Do you only lock for writes? What if I only read? Do you still lock
even for simultaneous reads?
* Is the limitation system wide or just in one server? That is, can I
improve the performance by adding more OSS or OST?
* By RPC bouncing, are you talking about the Linux storage stack? It
is not inherent to Lustre, right?

Thanks,
Jiahua


On Tue, Apr 13, 2010 at 8:43 PM, Liang Zhen <Zhen.Liang at sun.com> wrote:
> It's a kind of story like: "if you have to take dozens of global locks on
> lifetime of a RPC, then the code can't scale well on large SMP system, not
> matter what kind of network you are using”, so the problem is scattered
> everywhere.
> Also, we are trying to reduce RPC bounce between CPUs, in current code, a
> request can be received by CPU A, then queued on CPU B, processed by CPU C,
> and replied by CPU D, it's very bad on large SMP system because of data
> traffic between CPUs.
>
> Regards
> Liang
>
> Jiahua wrote:
>>
>> You mean it is inherent in the code? Can you point me to the actual
>> code if possible? I am just curious why. Any pointers or hints will be
>> appreciated.
>>
>> Thanks,
>> Jiahua
>>
>>
>> On Tue, Apr 13, 2010 at 6:46 PM, Kevin Van Maren <Kevin.Vanmaren at sun.com>
>> wrote:
>>
>>>
>>> Yes, the RPC rate is limited by Lustre code locking to that rate, even
>>> with
>>> rdma.
>>>
>>> Kevin
>>>
>>>
>>> On Apr 13, 2010, at 5:08 PM, Jiahua <jiahua at gmail.com> wrote:
>>>
>>>
>>>>
>>>> Hi all,
>>>>
>>>> This is kind of a followup question of the thread "One or two OSS, no
>>>> difference?" last month. In that thread, Andreas stated:
>>>>
>>>> "There is work currently underway to improve the SMP scaling
>>>> performance for the RPC handling layer in Lustre.  Currently that
>>>> limits the delivered RPC rate to 10-15k/sec or so."
>>>>
>>>> My question is: is the limitation also applied to RDMA on IB? By SMP,
>>>> I guess Andreas was talking about CPU, right? Since RDMA can bypass
>>>> the host CPU, does it mean it can also bypass the limitation?
>>>>
>>>> Thanks,
>>>> Jiahua
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>
>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>