<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">can you please attach diff file ?<div><br><div><div>On Oct 15, 2010, at 03:58, Vilobh Meshram wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><font color="#3333ff"><font size="2"><font face="times new roman,serif">Hi Alexey/Nicholas, <br><br>I modified the code in following way in the way Nicholas suggested yesterday in-order to get some information filled in a fixed sized buffer sent from client side.Here I am sending a buffer called "str" (whose size is 16) which will be updated at the MDS side by the string "hello"(whose size is 7 much less than original size of buffer "str" i.e 16).But I am not able to perform the operation successfully and I am getting an error <br>
"LustreError: 4209:0:(file.c:3143:ll_inode_revalidate_fini()) failure -14 inode 31257"<br><br>which seems to be related to DLM_REPLY_REC_OFF since I have modified this offset in my code.Can you please review my code and suggest me if I am making any mistake.I will be done with my task if I can resolve this problem.<br>
<br>Following are the modifications .The text in BOLD and Italics (blue color) are my modification at Client and MDS side for <b>Lustre 1.8.1.1</b>:-<br><br><i style="color: rgb(102, 0, 0);">At Client side :- lustre/ldlm/ldlm_lockd.c</i><b><i style="color: rgb(102, 0, 0);"><br>
<br></i></b> 655 int ldlm_cli_enqueue(.........)<br> 665 __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),<br> 666 [DLM_LOCKREQ_OFF] = sizeof(*body),<br> 667 [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len :<br>
668 sizeof(struct ost_lvb),<br><i><b> 669 16};</b></i><br><br> 717 if (reqp == NULL || *reqp == NULL) {<br> <i>718 req = ldlm_prep_enqueue_req(exp, <b>4,</b> size, NULL, 0);<br>
|<br> |<br> v<br><br> 575 struct ptlrpc_request *ldlm_prep_elc_req(.......)<br>
584 void *str=NULL;<br> 585 char *bufs[4] = {NULL,NULL,NULL,str};<br> 616 req = ptlrpc_prep_req(class_exp2cliimp(exp), version,<br> 617 opc, bufcount, size, <b>bufs</b></i><i>);<br>
<br><br><span style="color: rgb(102, 0, 0);">At MDS side :- lustre/ldlm/ldlm_lockd.c</span><br><br> 992 int ldlm_handle_enqueue(.........)<br> 996 {<br>1000 void *str;<br><b> __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),<br>
[DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep)<br></b>1009 <b>char *org = "hello";<br><br><br></b></i>1119 existing_lock:<br>1120 <br>1121 if (flags & LDLM_FL_HAS_INTENT) {<br>
1122 /* In this case, the reply buffer is allocated deep in<br>1123 * local_lock_enqueue by the policy function. */<br>1124 cookie = req;<br>1125 } else {<br><i><b>1126 int buffers = 4;</b></i><br>
1127 <br>1128 lock_res_and_lock(lock);<br>1129 if (lock->l_resource->lr_lvb_len) {<br><i><b> size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len;<br> buffers = 4;</b></i><br>
1132 }<br>1133 unlock_res_and_lock(lock);<br>1134 <br>1135 if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR))<br>1136 GOTO(out, rc = -ENOMEM);<br>
<i><b> str = lustre_msg_buf(req->rq_reqmsg, DLM_REPLY_REC_OFF+1, 1);<br> memcpy ( str , org , 7);<br> size[DLM_REPLY_REC_OFF + 1] = 16;</b><b><br><br></b><br></i><br><br clear="all">
</font></font></font><font style="color: rgb(51, 51, 255);" size="2" color="#3333ff"><font face="'times new roman', serif">Thanks,<br>Vilobh<br>
</font></font><i><font style="color: rgb(51, 51, 255);" size="2">Graduate Research Associate<br>Department of Computer Science<br>The Ohio State University Columbus Ohio</font></i><br>
<br><br><div class="gmail_quote">On Thu, Oct 14, 2010 at 12:25 PM, Vilobh Meshram <span dir="ltr"><<a href="mailto:vilobh.meshram@gmail.com">vilobh.meshram@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<font color="#3333ff"><font size="2"><font face="times new roman,serif">Hi Alexey,<br><br>That surely helps.Thanks for all the help till now.<br><br clear="all"></font></font></font><div class="im"><font style="color: rgb(51, 51, 255);" size="2" color="#3333ff"><font face="'times new roman', serif">Thanks,<br>
Vilobh<br>
</font></font><i><font style="color: rgb(51, 51, 255);" size="2">Graduate Research Associate<br>Department of Computer Science<br>The Ohio State University Columbus Ohio</font></i><br>
<br><br></div><div><div></div><div class="h5"><div class="gmail_quote">On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov <span dir="ltr"><<a href="mailto:alexey.lyashkov@clusterstor.com" target="_blank">alexey.lyashkov@clusterstor.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="word-wrap: break-word;">Hi Vilobh,<div><br></div><div>interop == interoperability between nodes with different version of software.</div><div><br></div><div>in general we have two ways to solve that - for requests with mdc_body - you can set flag in body and analyze that flag in server/client side.</div>
<div>if you want add new operation - better way add new flag into connect_data (look to OBD_CONNECT_* macroses handling)</div><div>that flag can checked via export->connect_flags on client or server side for remote side features.</div>
<div>as example 1.x and 2.0 have a different format for setattr requests :</div><div>int mdc_setattr</div><div>...</div><div><div> if (mdc_exp_is_2_0_server(exp)) { </div>
<div> size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); </div><div> size[REQ_REC_OFF + 1] = 0; /* capa */ </div><div>
size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); </div><div> size[REQ_REC_OFF + 3] = ealen; </div><div> size[REQ_REC_OFF + 4] = ea2len; </div>
<div> size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); </div><div> offset = REQ_REC_OFF + 5; </div><div>
bufcount = 6; </div><div> replybufcount = 6; </div><div> } else { </div>
<div> bufcount = 4; </div><div> } </div><div>
</div></div><div>example of client features are checking version based recovery support for client </div><div>mds_version_get_check</div><div>...</div><div><div> if (inode == NULL || !exp_connect_vbr(req->rq_export)) </div>
</div><div><br></div><div><br></div><div>I hope that help you.</div><div><div></div><div><div><br></div><div><br><div><div>On Oct 14, 2010, at 18:29, Vilobh Meshram wrote:</div><br><blockquote type="cite"><font color="#3333ff"><font size="2"><font face="times new roman,serif">Hi Alexey,<br>
<br>Thanks again for the reply.<br><br>Can you briefly give me some pointers about this interop issue and in which kind of RPC should this issue arise ? How should we resolve this what kind of flag needs to be set in ?<br>
<br>I went through the bugzilla entry mentioned by you it seems like for RPCs dealing with LDLM may cause this issue.Please correct me if I am wrong.<br><br clear="all"></font></font></font><font style="color: rgb(51, 51, 255);" size="2" color="#3333ff"><font face="'times new roman', serif">Thanks,<br>
Vilobh<br>
</font></font><i><font style="color: rgb(51, 51, 255);" size="2">Graduate Research Associate<br>Department of Computer Science<br>The Ohio State University Columbus Ohio</font></i><br>
<br><br><div class="gmail_quote">On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov <span dir="ltr"><<a href="mailto:alexey.lyashkov@clusterstor.com" target="_blank">alexey.lyashkov@clusterstor.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="word-wrap: break-word;">Hi Vilobh,<div><br></div><div>as i see, you touched code related to locking. struct ldm_request used to lock enqueue process - that why i say about interop issue in ELC code, which solved with export flag.</div>
<div>for common mdc requests you can resolve interop issue with flags in mdc_body (mdt_body), but that not possible for ldlm requests.</div><div><div></div><div><div> </div><div><br><div><div>On Oct 14, 2010, at 18:04, Vilobh Meshram wrote:</div>
<br><blockquote type="cite"><font color="#3333ff"><font size="2"><font face="times new roman,serif">Hi Alexey,<br>
<br>
Thanks again for your reply.<br>
<br>
I am trying to embed a buffer in the RPC which will get filled in with some values which MDS is aware of which the client calling the RPC is not aware <a href="http://of.It/" target="_blank">of.It</a> has nothing to do with locking.I just want to fill in the buffer which I embedd in the RPC with some suitable data from the MDS end and then do operations on that data at the client side.So I think the approach suggested by you and Nicholas of just including the sizeof(str) [the size of the expected information from the MDS] in the size[] array should be fine as done below :-<br>
<br><br><br></font></font></font><font face="'times new roman', serif"><font color="#3333ff">__u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),</font></font>
<div><font face="'times new roman', serif"><font color="#3333ff"> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) };</font></font></div>
<div><font face="'times new roman', serif"><font color="#3333ff"><br></font></font></div><div><font face="'times new roman', serif"><font color="#3333ff">---->> </font></font></div>
<div><font face="'times new roman', serif"><font color="#3333ff"> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body),</font></font></div><div>
<font face="'times new roman', serif"><font color="#3333ff"> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) ,</font></font></div>
<div><font face="'times new roman', serif"><font color="#3333ff">
//how to add "char *str=Hello" ofcourse we
will have sizeof(str) but how to choose the MACRO like </font></font><span style="font-family: 'times new roman',serif; color: rgb(51, 51, 255);">DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS</span></div>
<font color="#3333ff"><font size="2"><font face="times new roman,serif"> <br><br><b>Please correct me if I am wrong or please guide me if I need to consider few corner cases to handle this use case.<br><br></b>Thanks again.<br>
<br clear="all"></font></font></font><font style="color: rgb(51, 51, 255);" size="2" color="#3333ff"><font face="'times new roman', serif">Thanks,<br>Vilobh<br>
</font></font><i><font style="color: rgb(51, 51, 255);" size="2">Graduate Research Associate<br>Department of Computer Science<br>The Ohio State University Columbus Ohio</font></i><br>
<br clear="all"><font style="color: rgb(51, 51, 255);" size="2" color="#3333ff"><font face="'times new roman', serif"></font></font><br><div class="gmail_quote">On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov <span dir="ltr"><<a href="mailto:alexey.lyashkov@clusterstor.com" target="_blank">alexey.lyashkov@clusterstor.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Andreas,<br>
<div><br>
On Oct 14, 2010, at 17:31, Andreas Dilger wrote:<br>
<br>
> On 2010-10-13, at 23:18, Nicolas Williams wrote:<br>
>> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote:<br>
>>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote:<br>
>>>> Yes, it's possible to add buffers to requests. It's not possible to add<br>
>>>> buffers to _replies_ to existing RPCs unless you know the client expects<br>
>>>> those additional buffers -- existing clients expect a given maxsize for<br>
>>>> each reply, and if your reply is bigger then it will get dropped.<br>
>>> It is wrong for last ~1year.<br>
>>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request.<br>
>><br>
>> Ah, I didn't know that was in 1.8. Are there interop issues (with older<br>
>> clients) though with sending larger replies than expected?<br>
><br>
> Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer.<br>
><br>
</div>I think that question don't about rebalance buffers size in message,<br>
i think that sending large reply in smaller reply buffer.<br>
LNet don't able to put large reply to small buffer (without truncate flag, which is not exist in older ptlrpc version).<br>
without that flag you will see messages<br>
>><br>
CERROR("Matching packet from %s, match "LPU64<br>
" length %d too big: %d left, %d allowed\n",<br>
libcfs_id2str(src), match_bits, rlength,<br>
md->md_length - offset, mlength);<br>
>><br>
and LNet will drop message without notify PtlRPC.<br>
<div><br>
<br>
> Is that not true with the 2.x RPC handling?<br>
><br>
</div>2.x able to rebalance space between buffers (but looks by hand), and able adjust reply buffer after truncated reply.<br>
<div><div></div><div><br>
<br>
<br>
--------------------------------------<br>
Alexey Lyashkov<br>
<a href="mailto:alexey.lyashkov@clusterstor.com" target="_blank">alexey.lyashkov@clusterstor.com</a><br>
<br>
<br>
<br>
<br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</blockquote></div><br></div></body></html>