[Lustre-devel] Query to understand the Lustre request/reply message

[Vilobh Meshram]

Hi Alexey,

That surely helps.Thanks for all the help till now.

Thanks,
Vilobh
*Graduate Research Associate
Department of Computer Science
The Ohio State University Columbus Ohio*


On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov <
alexey.lyashkov at clusterstor.com> wrote:

> Hi Vilobh,
>
> interop == interoperability between nodes with different version of
> software.
>
> in general we have two ways to solve that - for requests with mdc_body -
> you can set flag in body and analyze that flag in server/client side.
> if you want add new operation - better way add new flag into  connect_data
>  (look to OBD_CONNECT_* macroses handling)
> that flag can checked via export->connect_flags on client or server side
> for remote side features.
> as example 1.x and 2.0 have a different format for setattr requests :
> int mdc_setattr
> ...
>        if (mdc_exp_is_2_0_server(exp)) {
>
>                 size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr);
>
>                 size[REQ_REC_OFF + 1] = 0; /* capa */
>
>                 size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch);
>
>                 size[REQ_REC_OFF + 3] = ealen;
>
>                 size[REQ_REC_OFF + 4] = ea2len;
>
>                 size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request);
>
>                 offset = REQ_REC_OFF + 5;
>
>                 bufcount = 6;
>
>                 replybufcount = 6;
>
>         } else {
>
>                 bufcount = 4;
>
>         }
>
>
> example of client features are checking version based recovery support for
> client
> mds_version_get_check
> ...
>         if (inode == NULL || !exp_connect_vbr(req->rq_export))
>
>
>
> I hope that help you.
>
>
> On Oct 14, 2010, at 18:29, Vilobh Meshram wrote:
>
> Hi Alexey,
>
> Thanks again for the reply.
>
> Can you briefly give me some pointers about this interop issue and in which
> kind of RPC should this issue arise ? How should we resolve this what kind
> of flag needs to be set in ?
>
> I went through the bugzilla entry mentioned by you it seems like for RPCs
> dealing with LDLM may cause this issue.Please correct me if I am wrong.
>
> Thanks,
> Vilobh
> *Graduate Research Associate
> Department of Computer Science
> The Ohio State University Columbus Ohio*
>
>
> On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov <
> alexey.lyashkov at clusterstor.com> wrote:
>
>> Hi Vilobh,
>>
>> as i see, you touched code related to locking. struct ldm_request used to
>> lock enqueue process - that why i say about interop issue in ELC code, which
>> solved with export flag.
>> for common mdc requests you can resolve interop issue with flags in
>> mdc_body (mdt_body), but that not possible for ldlm requests.
>>
>>
>> On Oct 14, 2010, at 18:04, Vilobh Meshram wrote:
>>
>> Hi Alexey,
>>
>> Thanks again for your reply.
>>
>> I am trying to embed a buffer in the RPC which will get filled in with
>> some values which MDS is aware of which the client calling the RPC is not
>> aware of.It has nothing to do with locking.I just want to fill in the
>> buffer which I embedd in the RPC with some suitable data from the MDS end
>> and then do operations on that data at the client side.So I think the
>> approach suggested by you and Nicholas of just including the sizeof(str)
>> [the size of the expected information from the MDS] in the size[] array
>> should be fine as done below :-
>>
>>
>>
>> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),
>>                                     [DLM_LOCKREQ_OFF]     = sizeof(struct
>> ldlm_request) };
>>
>> ---->>
>>      __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),
>>                                   [DLM_LOCKREQ_OFF]     = sizeof(struct
>> ldlm_request) ,
>>                                   //how to add "char *str=Hello" ofcourse
>> we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF
>> bcz for a specific kind of RPC there are limited number of such MACROS
>>
>>
>> *Please correct me if I am wrong or please guide me if I need to consider
>> few corner cases to handle this use case.
>>
>> *Thanks again.
>>
>> Thanks,
>> Vilobh
>> *Graduate Research Associate
>> Department of Computer Science
>> The Ohio State University Columbus Ohio*
>>
>>
>> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov <
>> alexey.lyashkov at clusterstor.com> wrote:
>>
>>> Andreas,
>>>
>>> On Oct 14, 2010, at 17:31, Andreas Dilger wrote:
>>>
>>> > On 2010-10-13, at 23:18, Nicolas Williams wrote:
>>> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote:
>>> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote:
>>> >>>> Yes, it's possible to add buffers to requests.  It's not possible to
>>> add
>>> >>>> buffers to _replies_ to existing RPCs unless you know the client
>>> expects
>>> >>>> those additional buffers -- existing clients expect a given maxsize
>>> for
>>> >>>> each reply, and if your reply is bigger then it will get dropped.
>>> >>> It is wrong for last ~1year.
>>> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for
>>> reply, and resend a request.
>>> >>
>>> >> Ah, I didn't know that was in 1.8.  Are there interop issues (with
>>> older
>>> >> clients) though with sending larger replies than expected?
>>> >
>>> > Nico, it has always been possible in the past to increase the size of
>>> any buffer in a request, or in a reply (if the total reply size will fit
>>> into the pre-allocated reply buffer).  An older peer would just ignore the
>>> bytes beyond the known part of the buffer.
>>> >
>>> I think that question don't about rebalance buffers size in message,
>>> i think that sending large reply in smaller reply buffer.
>>> LNet don't able to put large reply to small buffer (without truncate
>>> flag, which is not exist in older ptlrpc version).
>>> without that flag you will see messages
>>> >>
>>>                CERROR("Matching packet from %s, match "LPU64
>>>                       " length %d too big: %d left, %d allowed\n",
>>>                       libcfs_id2str(src), match_bits, rlength,
>>>                       md->md_length - offset, mlength);
>>> >>
>>> and LNet will drop message without notify PtlRPC.
>>>
>>>
>>> > Is that not true with the 2.x RPC handling?
>>> >
>>> 2.x able to rebalance space between buffers (but looks by hand), and able
>>> adjust reply buffer after truncated reply.
>>>
>>>
>>>
>>> --------------------------------------
>>> Alexey Lyashkov
>>> alexey.lyashkov at clusterstor.com
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20101014/83686fd2/attachment.htm>