<font size="2"><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF">Amazing...Thanks Nicholas and Alexey for your time and detailed reply.</font></font></font><div>
<font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><br></font></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF">I will try out the new API to create new RPC as per the steps mentioned by you for Lustre 2.0 (since I am using 1.8.1.1 right now) .</font></font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><br></font></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF">Thanks again.</font></font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><br></font></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF">Thanks,<br>
Vilobh<br></font></font><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><i>Graduate Research Associate</i></font></font></div><div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><i>Department of Computer Science</i></font></font></div>
<div><font class="Apple-style-span" face="'trebuchet ms', sans-serif"><font class="Apple-style-span" color="#3333FF"><i>The Ohio State University Columbus Ohio</i></font></font><i><br></i>
<br><br><div class="gmail_quote">On Wed, Oct 13, 2010 at 1:42 AM, Nicolas Williams <span dir="ltr"><<a href="mailto:Nicolas.Williams@oracle.com">Nicolas.Williams@oracle.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote:<br>
> Thanks a lot Alexey for the reply.The information will be really useful.<br>
><br>
> Since I am using 1.8.1.1 for my research project I will have to rely on the<br>
> old API.Since in the source tree prior to 2.0 we do not have a<br>
> mdt/mdt_handler.c and layout.c files will have to work with the low level<br>
> buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a<br>
> place or a function which make use of the old API which I can use as a<br>
> reference to write the RPC for my task.<br>
<br>
</div>The new API is _much_ easier to use than the old API.<br>
<br>
To add an RPC you must:<br>
<br>
- decide what it looks like<br>
<br>
Every PTLRPC has an opcode and one or more "buffers", with each<br>
buffer containing a C struct, a string, whatever. If a buffer<br>
contains a C struct, then it has to be fixed sized. The first buffer<br>
is struct ptlrpc_body.<br>
<br>
A single RPC opcode can denote multiple different layouts, depending<br>
on contents of various buffers. A single layout is called a<br>
"layout". See below.<br>
<br>
- add any struct, enum, and other C types you need to lustre_idl.h<br>
<br>
You must make sure to use the base types we use in lustre_idl.h, such<br>
as __u64.<br>
<br>
- create swabber functions for your data, if necessary<br>
<br>
- add handlers for the new RPC to mdt_handler.c (for the MDS) or<br>
ost_handler.c (for the OST), and so on<br>
<br>
The handlers are responsible for knowing which buffers contain what,<br>
and for swabbing them. You have to make sure that you don't swab a<br>
buffer more than once.<br>
<br>
The new API allows you define formats quite nicely, and it takes care of<br>
calling swabbers and ensuring that no buffer is swabbed more than once.<br>
The formats are defined in lustre/ptlrpc/layout.c and look like this:<br>
<br>
struct req_format RQF_MDS_SYNC =<br>
DEFINE_REQ_FMT0("MDS_SYNC", mdt_body_capa, mdt_body_only);<br>
...<br>
static const struct req_msg_field *mdt_body_capa[] = {<br>
&RMF_PTLRPC_BODY,<br>
&RMF_MDT_BODY,<br>
&RMF_CAPA1<br>
};<br>
static const struct req_msg_field *mdt_body_only[] = {<br>
&RMF_PTLRPC_BODY,<br>
&RMF_MDT_BODY<br>
};<br>
...<br>
<br>
An RPC consists of a request and reply, with their formats given in the<br>
DEFINE_REQ_FMT0() macro (there's other macros). Each message format<br>
defines a layout of buffers or, as we call them now, "fields", and each<br>
field has a format definition as well, such as:<br>
<br>
struct req_msg_field RMF_PTLRPC_BODY =<br>
DEFINE_MSGF("ptlrpc_body", 0,<br>
sizeof(struct ptlrpc_body), lustre_swab_ptlrpc_body, NULL);<br>
<br>
for a struct buffer. Other types of RMFs are possible (e.g., strings);<br>
see layout.c.<br>
<br>
So an MDS_SYNC RPC consists of a three-field (buffer) request and<br>
two-field reply. The request's fields are: PTLRPC_BODY, MDT_BODY, and<br>
CAPA1. The reply's fields are: PTLRPC_BODY and MDT_BODY. PTLRPC_BODY<br>
is a fixed-sized field containing a C structure, and that the swabber<br>
for this field is lustre_swab_ptlrpc_body(). And so on.<br>
<br>
If you look at Lustre 2.0's mdt_handler.c and ost_handler.c you'll find<br>
that one of the first things done is to initialize a "capsule", and that<br>
the expected message format of a request is decided based on its opcode.<br>
That is, the mapping of opcode to RQF is not given by some array, but<br>
decided as we go. Indeed, the RQF of a capsule can be changed<br>
mid-stream, with some constraints.<br>
<br>
So, with the new API you:<br>
<br>
- add C types to lustre_idl.h for on-the-wire data<br>
- add any swabbers to lustre/ptlrpc/pack_generic.c (declare them in<br>
lustre_idl.h)<br>
- add RQFs and, possibly, RMFs to layout.c<br>
- declare the RQFs/RMFs in lustre/include/lustre_req_layout.h<br>
<br>
- on the server-side:<br>
- Modify the relevant handler to add an arm to the existing switch<br>
on the request's opcode, call req_capsule_set() to set the<br>
capsule's format, then call a function that will use<br>
req_capsule_*get*() to get at the fields (buffers) (both, request<br>
and reply buffers) to read from (request) or write to (reply).<br>
<br>
- on the client-side:<br>
- You'll do something very similar, except that there's no handler<br>
function -- the pattern is less consistent, so you'll have to read<br>
mdc*.c and so on to get a flavor for this... Typically you'll<br>
allocate a request using ptlrpc_request_alloc_pack(), fill in its<br>
fields (again, using req_capsule_client_get() and friends), then<br>
you'll send it using, for example, ptlrpc_queue_wait().<br>
<br>
Take a good look at mdc_request.c in 2.0 to get a better idea of<br>
how to build client stubs for your new RPCs.<br>
<br>
I haven't described the wirecheck part -- I can do that later, once<br>
you've made enough progress. (We have a wirecheck/wiretest program pair<br>
to check that only backwards interoperable changes are made to<br>
lustre_idl.h.)<br>
<br>
I hope that helps. Yes, it'd be nice to have something closer to an<br>
actual IDL. The RQF/RMF/wirecheck/wiretest stuff could be extended to:<br>
<br>
- auto-generate swabbers from lustre_idl.h structs<br>
- provide a default opcode->RQF mapping<br>
- provide more static type safety (by having req_capsule_*get() be<br>
macros that cast the buffer address to the right type)<br>
- auto-generate simple request constructors (that take pointers to<br>
values of an RQF's correct request field C types)<br>
<br>
Compared to the old thing, the new API is much closer to an IDL. It's a<br>
good thing. I strongly recommend that you use it,<br>
<br>
Nico<br>
<font color="#888888">--<br>
</font></blockquote></div><br></div>