From crquan at gmail.com Thu Oct 7 05:56:32 2010 From: crquan at gmail.com (Cheng Renquan) Date: Thu, 7 Oct 2010 13:56:32 +0800 Subject: [Lustre-devel] Could you setup also a lustre git over http protocol (and a gitweb interface)? Message-ID: My company has a firewall that blocks git protocol (port 9418), that my git clone over git protocol could never succeed, http://wiki.lustre.org/index.php/Accessing_Lustre_Code git clone git://git.lustre.org/prime/lustre.git I think there may be many other ones also suffering this problem, so could you also setup git over http protocol? Thanks a lot! and if also provide a gitweb interface that would be better, like: http://git.kernel.org -- Cheng Renquan (程任全), from Singapore From jay.zen.lan at gmail.com Sat Oct 9 01:16:05 2010 From: jay.zen.lan at gmail.com (Jay Lan) Date: Fri, 08 Oct 2010 18:16:05 -0700 Subject: [Lustre-devel] How was lustre-1.8.4.tar.gz created? Message-ID: <4CAFC255.8090906@gmail.com> There is a lustre-1.8.4.tar.gz at the download site. At the top dir, comparing to the tree you check out from git, it does not have augogen.sh and misses the libcfs directory completely, but it contains tree_status that should be generated later in building. So, how was lustre-1.8.4.tar.gz generated? I used 'git archive v1_8_4' to populate a workarea, and built lustre software with that. The resulting rpm caused 2.6.18-194.3.1 kernel to crash when modprobe lustre.ko at ib_core:rdma_node_get_transport+0x14/0x2. However, if I built using the tar ball from the download area, it worked great. So, what was the trick? How would I be able to build a 1.8.4 for rhel5.5 from the git repository? Thanks, Jay From vilobh.meshram at gmail.com Tue Oct 12 17:55:12 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Tue, 12 Oct 2010 13:55:12 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message Message-ID: I want to understand the message encoding and decoding logic in lustre.I am planning to send a request to the MDS and based on the reply from MDs want to populate the struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg */ struct lustre_msg *rq_repmsg; I am trying this for a simple "Hello" message but not seeing the expected output.Sometime I even see Kernel Crash. If you can please give me some insight on the way the Lustre File system encodes decodes the messages sent accross nodes it would be helpful. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Tue Oct 12 18:21:05 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Tue, 12 Oct 2010 21:21:05 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: Message-ID: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> Hi Vilobh, ldlm_cli_cancel_req is good example to use old PtlRPC API. for first you need allocate request buffer via ptlrpc_prep_req next is - allocate reply buffer via ptlrpc_req_set_repsize next - call ptlrpc_queue_wait to send message and wait reply. osc_getattr_async good example for new PtlRPC API and async RPC processing. if that isn't help you - please show a yours code to find a error. On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: > I want to understand the message encoding and decoding logic in lustre.I am planning to send a request to the MDS and based on the reply from MDs want to populate the > > struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg */ > struct lustre_msg *rq_repmsg; > > I am trying this for a simple "Hello" message but not seeing the expected output.Sometime I even see Kernel Crash. > If you can please give me some insight on the way the Lustre File system encodes decodes the messages sent accross nodes it would be helpful. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Tue Oct 12 22:17:06 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Tue, 12 Oct 2010 18:17:06 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> Message-ID: Thanks Alexey.It was helpful. I have one more question :- If we want to add a new RPC with a new opcode are there any guidlines to be followed in the Lustre File System. Also , 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract the buffer information from the ptlrpc_message. 2)For every new RPC is the message length which is to be sent on wire ( including the fixed header size + the buffer size) dependent on the number of buffers in the lustre request message i.e the count field in the ptlrpc_prep_req() or the size of the size[] array. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > Hi Vilobh, > > ldlm_cli_cancel_req is good example to use old PtlRPC API. > for first you need allocate request buffer via ptlrpc_prep_req > next is - allocate reply buffer via ptlrpc_req_set_repsize > next - call ptlrpc_queue_wait to send message and wait reply. > > osc_getattr_async good example for new PtlRPC API and async RPC processing. > > if that isn't help you - please show a yours code to find a error. > > On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: > > I want to understand the message encoding and decoding logic in lustre.I am > planning to send a request to the MDS and based on the reply from MDs want > to populate the > > struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg > */ > struct lustre_msg *rq_repmsg; > > I am trying this for a simple "Hello" message but not seeing the expected > output.Sometime I even see Kernel Crash. > If you can please give me some insight on the way the Lustre File system > encodes decodes the messages sent accross nodes it would be helpful. > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Tue Oct 12 22:39:16 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Tue, 12 Oct 2010 18:39:16 -0400 Subject: [Lustre-devel] Adding a new RPC to Lustre Message-ID: If we want to add a new RPC with a new opcode are there any guidlines to be followed in the Lustre File System. Also , 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract the buffer information from the ptlrpc_message. 2)For every new RPC is the message length which is to be sent on wire ( including the fixed header size + the buffer size) dependent on the number of buffers in the lustre request message i.e the count field in the ptlrpc_prep_req() or the size of the size[] array. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Wed Oct 13 03:46:26 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 06:46:26 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> Message-ID: <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> That is depend of rpc type - is that RPC want to be return lock to caller or not, and is that rpc want to have special code to reconstruct in replay phase. in general you need look to mdt/mdt_handler.c. mdt_get_info is good example of simple rpc processing - but it use new PtlRPC api. that is API hide of low level request structures and provide api to access to message buffer by identifier. to use that API you need define structure of own message in ptlrpc/layout.c, and own command in enum mds_cmd_t, adjust array with commands and write own handler. On Oct 13, 2010, at 01:17, Vilobh Meshram wrote: > Thanks Alexey.It was helpful. > > I have one more question :- > > If we want to add a new RPC with a new opcode are there any guidlines to be followed in the Lustre File System. > > Also , > 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract the buffer information from the ptlrpc_message. > 2)For every new RPC is the message length which is to be sent on wire ( including the fixed header size + the buffer size) dependent on the number of buffers in the lustre request message i.e the count field in the ptlrpc_prep_req() or the size of the size[] array. > > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov wrote: > Hi Vilobh, > > ldlm_cli_cancel_req is good example to use old PtlRPC API. > for first you need allocate request buffer via ptlrpc_prep_req > next is - allocate reply buffer via ptlrpc_req_set_repsize > next - call ptlrpc_queue_wait to send message and wait reply. > > osc_getattr_async good example for new PtlRPC API and async RPC processing. > > if that isn't help you - please show a yours code to find a error. > > On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: > >> I want to understand the message encoding and decoding logic in lustre.I am planning to send a request to the MDS and based on the reply from MDs want to populate the >> >> struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg */ >> struct lustre_msg *rq_repmsg; >> >> I am trying this for a simple "Hello" message but not seeing the expected output.Sometime I even see Kernel Crash. >> If you can please give me some insight on the way the Lustre File system encodes decodes the messages sent accross nodes it would be helpful. >> >> Thanks, >> Vilobh >> Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Wed Oct 13 04:06:09 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Wed, 13 Oct 2010 00:06:09 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> Message-ID: Thanks Alexey for the reply.Thanks a lot. I will try out the steps mentioned by you and see if I can add a new RPC for the task which I am thinking of to implement in Lustre. The RPC of which I am thinking of will not return the lock to the caller.Yes that rpc will have special code to reconstruct in replay phase. Just a last question from which release of Lustre can we make use of the new API.Is their any documentation which lists the use of the new API.If yes can you please point me to that ? Thanks again. Thanks, Vilobh *Graduate Research Associate* ***Department of Computer Science* ***The Ohio State University Columbus Ohio* On Tue, Oct 12, 2010 at 11:46 PM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > That is depend of rpc type - is that RPC want to be return lock to caller > or not, and is that rpc want to have special code to reconstruct in replay > phase. > in general you need look to mdt/mdt_handler.c. mdt_get_info is good example > of simple rpc processing - but it use new PtlRPC api. > that is API hide of low level request structures and provide api to access > to message buffer by identifier. > to use that API you need define structure of own message in > ptlrpc/layout.c, and own command in enum mds_cmd_t, adjust array with > commands and write own handler. > > > On Oct 13, 2010, at 01:17, Vilobh Meshram wrote: > > Thanks Alexey.It was helpful. > > I have one more question :- > > If we want to add a new RPC with a new opcode are there any guidlines to be > followed in the Lustre File System. > > Also , > 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract > the buffer information from the ptlrpc_message. > 2)For every new RPC is the message length which is to be sent on wire ( > including the fixed header size + the buffer size) dependent on the number > of buffers in the lustre request message i.e the count field in the > ptlrpc_prep_req() or the size of the size[] array. > > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > > > On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov < > alexey.lyashkov at clusterstor.com> wrote: > >> Hi Vilobh, >> >> ldlm_cli_cancel_req is good example to use old PtlRPC API. >> for first you need allocate request buffer via ptlrpc_prep_req >> next is - allocate reply buffer via ptlrpc_req_set_repsize >> next - call ptlrpc_queue_wait to send message and wait reply. >> >> osc_getattr_async good example for new PtlRPC API and async RPC >> processing. >> >> if that isn't help you - please show a yours code to find a error. >> >> On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: >> >> I want to understand the message encoding and decoding logic in lustre.I >> am planning to send a request to the MDS and based on the reply from MDs >> want to populate the >> >> struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than >> msg */ >> struct lustre_msg *rq_repmsg; >> >> I am trying this for a simple "Hello" message but not seeing the expected >> output.Sometime I even see Kernel Crash. >> If you can please give me some insight on the way the Lustre File system >> encodes decodes the messages sent accross nodes it would be helpful. >> >> Thanks, >> Vilobh >> *Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio* >> _______________________________________________ >> Lustre-devel mailing list >> Lustre-devel at lists.lustre.org >> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Wed Oct 13 04:20:34 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 07:20:34 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> Message-ID: <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> On Oct 13, 2010, at 07:06, Vilobh Meshram wrote: > Thanks Alexey for the reply.Thanks a lot. > > I will try out the steps mentioned by you and see if I can add a new RPC for the task which I am thinking of to implement in Lustre. > > The RPC of which I am thinking of will not return the lock to the caller.Yes that rpc will have special code to reconstruct in replay phase. In that case possible you need look to 'setattr' functions - it have own mdt_reconstruct_setattr reconstructor, but that is need support on client side. typically that say - you need have special field in message to copy data from server reply. > > Just a last question from which release of Lustre can we make use of the new API. > Is their any documentation which lists the use of the new API.If yes can you please point me to that ? lustre 1.x have a old PtlRPC API, lustre 2.0 have a mix of new and old, but have a migrate to use new API. > > Thanks again. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > On Tue, Oct 12, 2010 at 11:46 PM, Alexey Lyashkov wrote: > That is depend of rpc type - is that RPC want to be return lock to caller or not, and is that rpc want to have special code to reconstruct in replay phase. > in general you need look to mdt/mdt_handler.c. mdt_get_info is good example of simple rpc processing - but it use new PtlRPC api. > that is API hide of low level request structures and provide api to access to message buffer by identifier. > to use that API you need define structure of own message in ptlrpc/layout.c, and own command in enum mds_cmd_t, adjust array with commands and write own handler. > > > On Oct 13, 2010, at 01:17, Vilobh Meshram wrote: >> Thanks Alexey.It was helpful. >> >> I have one more question :- >> >> If we want to add a new RPC with a new opcode are there any guidlines to be followed in the Lustre File System. >> >> Also , >> 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract the buffer information from the ptlrpc_message. >> 2)For every new RPC is the message length which is to be sent on wire ( including the fixed header size + the buffer size) dependent on the number of buffers in the lustre request message i.e the count field in the ptlrpc_prep_req() or the size of the size[] array. >> >> >> Thanks, >> Vilobh >> Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio >> >> >> On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov wrote: >> Hi Vilobh, >> >> ldlm_cli_cancel_req is good example to use old PtlRPC API. >> for first you need allocate request buffer via ptlrpc_prep_req >> next is - allocate reply buffer via ptlrpc_req_set_repsize >> next - call ptlrpc_queue_wait to send message and wait reply. >> >> osc_getattr_async good example for new PtlRPC API and async RPC processing. >> >> if that isn't help you - please show a yours code to find a error. >> >> On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: >> >>> I want to understand the message encoding and decoding logic in lustre.I am planning to send a request to the MDS and based on the reply from MDs want to populate the >>> >>> struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg */ >>> struct lustre_msg *rq_repmsg; >>> >>> I am trying this for a simple "Hello" message but not seeing the expected output.Sometime I even see Kernel Crash. >>> If you can please give me some insight on the way the Lustre File system encodes decodes the messages sent accross nodes it would be helpful. >>> >>> Thanks, >>> Vilobh >>> Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Wed Oct 13 04:35:13 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Wed, 13 Oct 2010 00:35:13 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> Message-ID: Thanks a lot Alexey for the reply.The information will be really useful. Since I am using 1.8.1.1 for my research project I will have to rely on the old API.Since in the source tree prior to 2.0 we do not have a mdt/mdt_handler.c and layout.c files will have to work with the low level buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a place or a function which make use of the old API which I can use as a reference to write the RPC for my task. Thanks, Vilobh *Graduate Research Associate* *Department of Computer Science* *The Ohio State University Columbus Ohio* On Wed, Oct 13, 2010 at 12:20 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > > On Oct 13, 2010, at 07:06, Vilobh Meshram wrote: > > Thanks Alexey for the reply.Thanks a lot. > > I will try out the steps mentioned by you and see if I can add a new RPC > for the task which I am thinking of to implement in Lustre. > > The RPC of which I am thinking of will not return the lock to the > caller.Yes that rpc will have special code to reconstruct in replay phase. > > In that case possible you need look to 'setattr' functions - it have > own mdt_reconstruct_setattr reconstructor, but that is need support on > client side. > typically that say - you need have special field in message to copy data > from server reply. > > > Just a last question from which release of Lustre can we make use of the > new API. > > Is their any documentation which lists the use of the new API.If yes can > you please point me to that ? > > lustre 1.x have a old PtlRPC API, lustre 2.0 have a mix of new and old, but > have a migrate to use new API. > > > Thanks again. > > Thanks, > Vilobh > *Graduate Research Associate* > ***Department of Computer Science* > ***The Ohio State University Columbus Ohio* > > On Tue, Oct 12, 2010 at 11:46 PM, Alexey Lyashkov < > alexey.lyashkov at clusterstor.com> wrote: > >> That is depend of rpc type - is that RPC want to be return lock to caller >> or not, and is that rpc want to have special code to reconstruct in replay >> phase. >> in general you need look to mdt/mdt_handler.c. mdt_get_info is good >> example of simple rpc processing - but it use new PtlRPC api. >> that is API hide of low level request structures and provide api to access >> to message buffer by identifier. >> to use that API you need define structure of own message in >> ptlrpc/layout.c, and own command in enum mds_cmd_t, adjust array with >> commands and write own handler. >> >> >> On Oct 13, 2010, at 01:17, Vilobh Meshram wrote: >> >> Thanks Alexey.It was helpful. >> >> I have one more question :- >> >> If we want to add a new RPC with a new opcode are there any guidlines to >> be followed in the Lustre File System. >> >> Also , >> 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract >> the buffer information from the ptlrpc_message. >> 2)For every new RPC is the message length which is to be sent on wire ( >> including the fixed header size + the buffer size) dependent on the number >> of buffers in the lustre request message i.e the count field in the >> ptlrpc_prep_req() or the size of the size[] array. >> >> >> Thanks, >> Vilobh >> *Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio* >> >> >> On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov < >> alexey.lyashkov at clusterstor.com> wrote: >> >>> Hi Vilobh, >>> >>> ldlm_cli_cancel_req is good example to use old PtlRPC API. >>> for first you need allocate request buffer via ptlrpc_prep_req >>> next is - allocate reply buffer via ptlrpc_req_set_repsize >>> next - call ptlrpc_queue_wait to send message and wait reply. >>> >>> osc_getattr_async good example for new PtlRPC API and async RPC >>> processing. >>> >>> if that isn't help you - please show a yours code to find a error. >>> >>> On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: >>> >>> I want to understand the message encoding and decoding logic in lustre.I >>> am planning to send a request to the MDS and based on the reply from MDs >>> want to populate the >>> >>> struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than >>> msg */ >>> struct lustre_msg *rq_repmsg; >>> >>> I am trying this for a simple "Hello" message but not seeing the expected >>> output.Sometime I even see Kernel Crash. >>> If you can please give me some insight on the way the Lustre File system >>> encodes decodes the messages sent accross nodes it would be helpful. >>> >>> Thanks, >>> Vilobh >>> *Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio* >>> _______________________________________________ >>> Lustre-devel mailing list >>> Lustre-devel at lists.lustre.org >>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Wed Oct 13 04:41:49 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 07:41:49 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> Message-ID: mds_handle (start processing), MDS_CHECK_RESENT() macro to handle reconstruction. mds_set_info_rpc - to simple rpc processing, possible mds_setxattr (mds_setxattr_internal) with generic reconstruction function. On Oct 13, 2010, at 07:35, Vilobh Meshram wrote: > Thanks a lot Alexey for the reply.The information will be really useful. > > Since I am using 1.8.1.1 for my research project I will have to rely on the old API.Since in the source tree prior to 2.0 we do not have a mdt/mdt_handler.c and layout.c files will have to work with the low level buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a place or a function which make use of the old API which I can use as a reference to write the RPC for my task. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Wed, Oct 13, 2010 at 12:20 AM, Alexey Lyashkov wrote: > > On Oct 13, 2010, at 07:06, Vilobh Meshram wrote: > >> Thanks Alexey for the reply.Thanks a lot. >> >> I will try out the steps mentioned by you and see if I can add a new RPC for the task which I am thinking of to implement in Lustre. >> >> The RPC of which I am thinking of will not return the lock to the caller.Yes that rpc will have special code to reconstruct in replay phase. > > In that case possible you need look to 'setattr' functions - it have own mdt_reconstruct_setattr reconstructor, but that is need support on client side. > typically that say - you need have special field in message to copy data from server reply. > >> >> Just a last question from which release of Lustre can we make use of the new API. >> Is their any documentation which lists the use of the new API.If yes can you please point me to that ? > > lustre 1.x have a old PtlRPC API, lustre 2.0 have a mix of new and old, but have a migrate to use new API. > >> >> Thanks again. >> >> Thanks, >> Vilobh >> Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio >> >> On Tue, Oct 12, 2010 at 11:46 PM, Alexey Lyashkov wrote: >> That is depend of rpc type - is that RPC want to be return lock to caller or not, and is that rpc want to have special code to reconstruct in replay phase. >> in general you need look to mdt/mdt_handler.c. mdt_get_info is good example of simple rpc processing - but it use new PtlRPC api. >> that is API hide of low level request structures and provide api to access to message buffer by identifier. >> to use that API you need define structure of own message in ptlrpc/layout.c, and own command in enum mds_cmd_t, adjust array with commands and write own handler. >> >> >> On Oct 13, 2010, at 01:17, Vilobh Meshram wrote: >>> Thanks Alexey.It was helpful. >>> >>> I have one more question :- >>> >>> If we want to add a new RPC with a new opcode are there any guidlines to be followed in the Lustre File System. >>> >>> Also , >>> 1)How does the MDS process the ptlrpc_request i.e how does the MDS extract the buffer information from the ptlrpc_message. >>> 2)For every new RPC is the message length which is to be sent on wire ( including the fixed header size + the buffer size) dependent on the number of buffers in the lustre request message i.e the count field in the ptlrpc_prep_req() or the size of the size[] array. >>> >>> >>> Thanks, >>> Vilobh >>> Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio >>> >>> >>> On Tue, Oct 12, 2010 at 2:21 PM, Alexey Lyashkov wrote: >>> Hi Vilobh, >>> >>> ldlm_cli_cancel_req is good example to use old PtlRPC API. >>> for first you need allocate request buffer via ptlrpc_prep_req >>> next is - allocate reply buffer via ptlrpc_req_set_repsize >>> next - call ptlrpc_queue_wait to send message and wait reply. >>> >>> osc_getattr_async good example for new PtlRPC API and async RPC processing. >>> >>> if that isn't help you - please show a yours code to find a error. >>> >>> On Oct 12, 2010, at 20:55, Vilobh Meshram wrote: >>> >>>> I want to understand the message encoding and decoding logic in lustre.I am planning to send a request to the MDS and based on the reply from MDs want to populate the >>>> >>>> struct lustre_msg *rq_repbuf; /* client only, buf may be bigger than msg */ >>>> struct lustre_msg *rq_repmsg; >>>> >>>> I am trying this for a simple "Hello" message but not seeing the expected output.Sometime I even see Kernel Crash. >>>> If you can please give me some insight on the way the Lustre File system encodes decodes the messages sent accross nodes it would be helpful. >>>> >>>> Thanks, >>>> Vilobh >>>> Graduate Research Associate >>>> Department of Computer Science >>>> The Ohio State University Columbus Ohio >>>> _______________________________________________ >>>> Lustre-devel mailing list >>>> Lustre-devel at lists.lustre.org >>>> http://lists.lustre.org/mailman/listinfo/lustre-devel >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Williams at oracle.com Wed Oct 13 05:42:34 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 00:42:34 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> Message-ID: <20101013054233.GC1635@oracle.com> On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote: > Thanks a lot Alexey for the reply.The information will be really useful. > > Since I am using 1.8.1.1 for my research project I will have to rely on the > old API.Since in the source tree prior to 2.0 we do not have a > mdt/mdt_handler.c and layout.c files will have to work with the low level > buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a > place or a function which make use of the old API which I can use as a > reference to write the RPC for my task. The new API is _much_ easier to use than the old API. To add an RPC you must: - decide what it looks like Every PTLRPC has an opcode and one or more "buffers", with each buffer containing a C struct, a string, whatever. If a buffer contains a C struct, then it has to be fixed sized. The first buffer is struct ptlrpc_body. A single RPC opcode can denote multiple different layouts, depending on contents of various buffers. A single layout is called a "layout". See below. - add any struct, enum, and other C types you need to lustre_idl.h You must make sure to use the base types we use in lustre_idl.h, such as __u64. - create swabber functions for your data, if necessary - add handlers for the new RPC to mdt_handler.c (for the MDS) or ost_handler.c (for the OST), and so on The handlers are responsible for knowing which buffers contain what, and for swabbing them. You have to make sure that you don't swab a buffer more than once. The new API allows you define formats quite nicely, and it takes care of calling swabbers and ensuring that no buffer is swabbed more than once. The formats are defined in lustre/ptlrpc/layout.c and look like this: struct req_format RQF_MDS_SYNC = DEFINE_REQ_FMT0("MDS_SYNC", mdt_body_capa, mdt_body_only); ... static const struct req_msg_field *mdt_body_capa[] = { &RMF_PTLRPC_BODY, &RMF_MDT_BODY, &RMF_CAPA1 }; static const struct req_msg_field *mdt_body_only[] = { &RMF_PTLRPC_BODY, &RMF_MDT_BODY }; ... An RPC consists of a request and reply, with their formats given in the DEFINE_REQ_FMT0() macro (there's other macros). Each message format defines a layout of buffers or, as we call them now, "fields", and each field has a format definition as well, such as: struct req_msg_field RMF_PTLRPC_BODY = DEFINE_MSGF("ptlrpc_body", 0, sizeof(struct ptlrpc_body), lustre_swab_ptlrpc_body, NULL); for a struct buffer. Other types of RMFs are possible (e.g., strings); see layout.c. So an MDS_SYNC RPC consists of a three-field (buffer) request and two-field reply. The request's fields are: PTLRPC_BODY, MDT_BODY, and CAPA1. The reply's fields are: PTLRPC_BODY and MDT_BODY. PTLRPC_BODY is a fixed-sized field containing a C structure, and that the swabber for this field is lustre_swab_ptlrpc_body(). And so on. If you look at Lustre 2.0's mdt_handler.c and ost_handler.c you'll find that one of the first things done is to initialize a "capsule", and that the expected message format of a request is decided based on its opcode. That is, the mapping of opcode to RQF is not given by some array, but decided as we go. Indeed, the RQF of a capsule can be changed mid-stream, with some constraints. So, with the new API you: - add C types to lustre_idl.h for on-the-wire data - add any swabbers to lustre/ptlrpc/pack_generic.c (declare them in lustre_idl.h) - add RQFs and, possibly, RMFs to layout.c - declare the RQFs/RMFs in lustre/include/lustre_req_layout.h - on the server-side: - Modify the relevant handler to add an arm to the existing switch on the request's opcode, call req_capsule_set() to set the capsule's format, then call a function that will use req_capsule_*get*() to get at the fields (buffers) (both, request and reply buffers) to read from (request) or write to (reply). - on the client-side: - You'll do something very similar, except that there's no handler function -- the pattern is less consistent, so you'll have to read mdc*.c and so on to get a flavor for this... Typically you'll allocate a request using ptlrpc_request_alloc_pack(), fill in its fields (again, using req_capsule_client_get() and friends), then you'll send it using, for example, ptlrpc_queue_wait(). Take a good look at mdc_request.c in 2.0 to get a better idea of how to build client stubs for your new RPCs. I haven't described the wirecheck part -- I can do that later, once you've made enough progress. (We have a wirecheck/wiretest program pair to check that only backwards interoperable changes are made to lustre_idl.h.) I hope that helps. Yes, it'd be nice to have something closer to an actual IDL. The RQF/RMF/wirecheck/wiretest stuff could be extended to: - auto-generate swabbers from lustre_idl.h structs - provide a default opcode->RQF mapping - provide more static type safety (by having req_capsule_*get() be macros that cast the buffer address to the right type) - auto-generate simple request constructors (that take pointers to values of an RQF's correct request field C types) Compared to the old thing, the new API is much closer to an IDL. It's a good thing. I strongly recommend that you use it, Nico -- From alexey.lyashkov at clusterstor.com Wed Oct 13 05:54:57 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 08:54:57 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013054233.GC1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: On Oct 13, 2010, at 08:42, Nicolas Williams wrote: > On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote: >> Thanks a lot Alexey for the reply.The information will be really useful. >> >> Since I am using 1.8.1.1 for my research project I will have to rely on the >> old API.Since in the source tree prior to 2.0 we do not have a >> mdt/mdt_handler.c and layout.c files will have to work with the low level >> buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a >> place or a function which make use of the old API which I can use as a >> reference to write the RPC for my task. > > > Compared to the old thing, the new API is much closer to an IDL. It's a > good thing. I strongly recommend that you use it, > main problem - lustre 1.8.1 don't have the new API :) -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From vilobh.meshram at gmail.com Wed Oct 13 06:07:06 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Wed, 13 Oct 2010 02:07:06 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013054233.GC1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: Amazing...Thanks Nicholas and Alexey for your time and detailed reply. I will try out the new API to create new RPC as per the steps mentioned by you for Lustre 2.0 (since I am using 1.8.1.1 right now) . Thanks again. Thanks, Vilobh *Graduate Research Associate* *Department of Computer Science* *The Ohio State University Columbus Ohio** * On Wed, Oct 13, 2010 at 1:42 AM, Nicolas Williams < Nicolas.Williams at oracle.com> wrote: > On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote: > > Thanks a lot Alexey for the reply.The information will be really useful. > > > > Since I am using 1.8.1.1 for my research project I will have to rely on > the > > old API.Since in the source tree prior to 2.0 we do not have a > > mdt/mdt_handler.c and layout.c files will have to work with the low level > > buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you > know a > > place or a function which make use of the old API which I can use as a > > reference to write the RPC for my task. > > The new API is _much_ easier to use than the old API. > > To add an RPC you must: > > - decide what it looks like > > Every PTLRPC has an opcode and one or more "buffers", with each > buffer containing a C struct, a string, whatever. If a buffer > contains a C struct, then it has to be fixed sized. The first buffer > is struct ptlrpc_body. > > A single RPC opcode can denote multiple different layouts, depending > on contents of various buffers. A single layout is called a > "layout". See below. > > - add any struct, enum, and other C types you need to lustre_idl.h > > You must make sure to use the base types we use in lustre_idl.h, such > as __u64. > > - create swabber functions for your data, if necessary > > - add handlers for the new RPC to mdt_handler.c (for the MDS) or > ost_handler.c (for the OST), and so on > > The handlers are responsible for knowing which buffers contain what, > and for swabbing them. You have to make sure that you don't swab a > buffer more than once. > > The new API allows you define formats quite nicely, and it takes care of > calling swabbers and ensuring that no buffer is swabbed more than once. > The formats are defined in lustre/ptlrpc/layout.c and look like this: > > struct req_format RQF_MDS_SYNC = > DEFINE_REQ_FMT0("MDS_SYNC", mdt_body_capa, mdt_body_only); > ... > static const struct req_msg_field *mdt_body_capa[] = { > &RMF_PTLRPC_BODY, > &RMF_MDT_BODY, > &RMF_CAPA1 > }; > static const struct req_msg_field *mdt_body_only[] = { > &RMF_PTLRPC_BODY, > &RMF_MDT_BODY > }; > ... > > An RPC consists of a request and reply, with their formats given in the > DEFINE_REQ_FMT0() macro (there's other macros). Each message format > defines a layout of buffers or, as we call them now, "fields", and each > field has a format definition as well, such as: > > struct req_msg_field RMF_PTLRPC_BODY = > DEFINE_MSGF("ptlrpc_body", 0, > sizeof(struct ptlrpc_body), lustre_swab_ptlrpc_body, > NULL); > > for a struct buffer. Other types of RMFs are possible (e.g., strings); > see layout.c. > > So an MDS_SYNC RPC consists of a three-field (buffer) request and > two-field reply. The request's fields are: PTLRPC_BODY, MDT_BODY, and > CAPA1. The reply's fields are: PTLRPC_BODY and MDT_BODY. PTLRPC_BODY > is a fixed-sized field containing a C structure, and that the swabber > for this field is lustre_swab_ptlrpc_body(). And so on. > > If you look at Lustre 2.0's mdt_handler.c and ost_handler.c you'll find > that one of the first things done is to initialize a "capsule", and that > the expected message format of a request is decided based on its opcode. > That is, the mapping of opcode to RQF is not given by some array, but > decided as we go. Indeed, the RQF of a capsule can be changed > mid-stream, with some constraints. > > So, with the new API you: > > - add C types to lustre_idl.h for on-the-wire data > - add any swabbers to lustre/ptlrpc/pack_generic.c (declare them in > lustre_idl.h) > - add RQFs and, possibly, RMFs to layout.c > - declare the RQFs/RMFs in lustre/include/lustre_req_layout.h > > - on the server-side: > - Modify the relevant handler to add an arm to the existing switch > on the request's opcode, call req_capsule_set() to set the > capsule's format, then call a function that will use > req_capsule_*get*() to get at the fields (buffers) (both, request > and reply buffers) to read from (request) or write to (reply). > > - on the client-side: > - You'll do something very similar, except that there's no handler > function -- the pattern is less consistent, so you'll have to read > mdc*.c and so on to get a flavor for this... Typically you'll > allocate a request using ptlrpc_request_alloc_pack(), fill in its > fields (again, using req_capsule_client_get() and friends), then > you'll send it using, for example, ptlrpc_queue_wait(). > > Take a good look at mdc_request.c in 2.0 to get a better idea of > how to build client stubs for your new RPCs. > > I haven't described the wirecheck part -- I can do that later, once > you've made enough progress. (We have a wirecheck/wiretest program pair > to check that only backwards interoperable changes are made to > lustre_idl.h.) > > I hope that helps. Yes, it'd be nice to have something closer to an > actual IDL. The RQF/RMF/wirecheck/wiretest stuff could be extended to: > > - auto-generate swabbers from lustre_idl.h structs > - provide a default opcode->RQF mapping > - provide more static type safety (by having req_capsule_*get() be > macros that cast the buffer address to the right type) > - auto-generate simple request constructors (that take pointers to > values of an RQF's correct request field C types) > > Compared to the old thing, the new API is much closer to an IDL. It's a > good thing. I strongly recommend that you use it, > > Nico > -- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Wed Oct 13 06:25:00 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 09:25:00 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013054233.GC1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: On Oct 13, 2010, at 08:42, Nicolas Williams wrote: > > > - add handlers for the new RPC to mdt_handler.c (for the MDS) or > ost_handler.c (for the OST), and so on > > The handlers are responsible for knowing which buffers contain what, > and for swabbing them. You have to make sure that you don't swab a > buffer more than once. > BTW. That not enough. Some of operations wants to have own recostructors for replay/resend. Some of operations want to have a return lock - as example MDS_GETATTR and MDS_REINT commands. .... -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From Nicolas.Williams at oracle.com Wed Oct 13 07:12:02 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 02:12:02 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: <20101013071201.GD1635@oracle.com> On Wed, Oct 13, 2010 at 09:25:00AM +0300, Alexey Lyashkov wrote: > > On Oct 13, 2010, at 08:42, Nicolas Williams wrote: > > > > > > - add handlers for the new RPC to mdt_handler.c (for the MDS) or > > ost_handler.c (for the OST), and so on > > > > The handlers are responsible for knowing which buffers contain what, > > and for swabbing them. You have to make sure that you don't swab a > > buffer more than once. > > > BTW. > That not enough. > Some of operations wants to have own recostructors for replay/resend. I glossed over replay/resend, mostly because I know little about them, but also because they are completely orthogonal to the message format details. If you want to add an RPC then the first step should be to get the RPC format designed and surrounding code up and running, then you can take care of replay/resend. > Some of operations want to have a return lock - as example MDS_GETATTR > and MDS_REINT commands. That too is orthogonal to the message formats. The message format has to have a buffer (field) declared to carry lock (or capability, or whatever) bits, and some function has to be invoked to populate the buffer in the reply. Nico -- From Nicolas.Williams at oracle.com Wed Oct 13 07:15:07 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 02:15:07 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: <20101013071507.GE1635@oracle.com> On Wed, Oct 13, 2010 at 08:54:57AM +0300, Alexey Lyashkov wrote: > On Oct 13, 2010, at 08:42, Nicolas Williams wrote: > > On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote: > >> Thanks a lot Alexey for the reply.The information will be really useful. > >> > >> Since I am using 1.8.1.1 for my research project I will have to rely on the > >> old API.Since in the source tree prior to 2.0 we do not have a > >> mdt/mdt_handler.c and layout.c files will have to work with the low level > >> buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a > >> place or a function which make use of the old API which I can use as a > >> reference to write the RPC for my task. > > > > > > Compared to the old thing, the new API is much closer to an IDL. It's a > > good thing. I strongly recommend that you use it, > > > main problem - lustre 1.8.1 don't have the new API :) You'll note that Vilobh did not provide any rationale for his/her choice of Lustre version. Without having any other good reason for picking 1.8 or 2.0, I strongly recommend 2.0. Now, perhaps Vilobh has a need to interop with an installed base of 1.8. That would be a good reason to do this in 1.8. But the work will have to be done for 2.0 as well. Nico -- From Nicolas.Williams at oracle.com Wed Oct 13 07:17:08 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 02:17:08 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> Message-ID: <20101013071708.GF1635@oracle.com> On Wed, Oct 13, 2010 at 02:07:06AM -0400, Vilobh Meshram wrote: > I will try out the new API to create new RPC as per the steps mentioned by > you for Lustre 2.0 (since I am using 1.8.1.1 right now) . The new API, incidentally, uses the old API under the hood. That might help guide you. To understand usage patterns for the old API it should help to look at 2.0 code, particularly layout.c code, then look at corresponding 1.8 code. Nico -- From alexey.lyashkov at clusterstor.com Wed Oct 13 07:27:55 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 10:27:55 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013071201.GD1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> Message-ID: <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> eh.. Nicolas, Format for messages which want to reconstructed after resend and don't want recontructed - is different. As quick example it is OPEN request (via MDS_REINT command), that type message need a have extra buffer to store LOV EA, which to be send to MDS in replay case (with additional flag in header). (client have a copy data from a mds reply after ptlrpc finish processing request). That is why i say about "Reconstruct/replay case" Also format is different is you want to use MDS_REINT + sub commands or you want to use something similar to MDS_SET_INFO. For MDS_SET_INFO you use single format for all messages (just simple key <> value) buffer, but for MDS_REINT you need two formats - one for generic MDS_REINT code (get opcode from command, get locks, and possible other) and own format for each opcode - such as open, unlink, setxattr, setattr. all of them have a different number of buffers (fields). On Oct 13, 2010, at 10:12, Nicolas Williams wrote: > On Wed, Oct 13, 2010 at 09:25:00AM +0300, Alexey Lyashkov wrote: >> >> On Oct 13, 2010, at 08:42, Nicolas Williams wrote: >>> >>> >>> - add handlers for the new RPC to mdt_handler.c (for the MDS) or >>> ost_handler.c (for the OST), and so on >>> >>> The handlers are responsible for knowing which buffers contain what, >>> and for swabbing them. You have to make sure that you don't swab a >>> buffer more than once. >>> >> BTW. >> That not enough. >> Some of operations wants to have own recostructors for replay/resend. > > I glossed over replay/resend, mostly because I know little about them, > but also because they are completely orthogonal to the message format > details. If you want to add an RPC then the first step should be to get > the RPC format designed and surrounding code up and running, then you > can take care of replay/resend. > >> Some of operations want to have a return lock - as example MDS_GETATTR >> and MDS_REINT commands. > > That too is orthogonal to the message formats. The message format has > to have a buffer (field) declared to carry lock (or capability, or > whatever) bits, and some function has to be invoked to populate the > buffer in the reply. > > Nico > -- From alexey.lyashkov at clusterstor.com Wed Oct 13 07:32:23 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 10:32:23 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013071507.GE1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071507.GE1635@oracle.com> Message-ID: <4D7ECBDD-DA79-4DB4-AAE1-B1164802FF86@clusterstor.com> MDS code in 1.8 is more simple, because they don't have a parts of clustered metadata project aka CMD3 ;-) same for client, they don't have many advantages - such as FID assignments on client side, or extra MD layer (LMV) and don't have a CLIO. so 1.8 is good start to learn :) On Oct 13, 2010, at 10:15, Nicolas Williams wrote: > On Wed, Oct 13, 2010 at 08:54:57AM +0300, Alexey Lyashkov wrote: >> On Oct 13, 2010, at 08:42, Nicolas Williams wrote: >>> On Wed, Oct 13, 2010 at 12:35:13AM -0400, Vilobh Meshram wrote: >>>> Thanks a lot Alexey for the reply.The information will be really useful. >>>> >>>> Since I am using 1.8.1.1 for my research project I will have to rely on the >>>> old API.Since in the source tree prior to 2.0 we do not have a >>>> mdt/mdt_handler.c and layout.c files will have to work with the low level >>>> buffer management structures(ptlrpc_request,lustre_msg_v2,etc).Do you know a >>>> place or a function which make use of the old API which I can use as a >>>> reference to write the RPC for my task. >>> >>> >>> Compared to the old thing, the new API is much closer to an IDL. It's a >>> good thing. I strongly recommend that you use it, >>> >> main problem - lustre 1.8.1 don't have the new API :) > > You'll note that Vilobh did not provide any rationale for his/her choice > of Lustre version. Without having any other good reason for picking 1.8 > or 2.0, I strongly recommend 2.0. > > Now, perhaps Vilobh has a need to interop with an installed base of 1.8. > That would be a good reason to do this in 1.8. But the work will have > to be done for 2.0 as well. > > Nico > -- -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From Nicolas.Williams at oracle.com Wed Oct 13 07:43:46 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 02:43:46 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> Message-ID: <20101013074345.GH1635@oracle.com> On Wed, Oct 13, 2010 at 10:27:55AM +0300, Alexey Lyashkov wrote: > eh.. Nicolas, > > Format for messages which want to reconstructed after resend and don't > want recontructed - is different. > > As quick example it is OPEN request (via MDS_REINT command), that type > message need a have extra buffer to store LOV EA, which to be send to > MDS in replay case (with additional flag in header). (client have a > copy data from a mds reply after ptlrpc finish processing request). > That is why i say about "Reconstruct/replay case" Sure, but this buffer needs to be declared a priori. If you won't know whether you'll need a buffer until later, that's OK: you declare it anyways and you set its size to zero if you don't need it. You can't change a capsule's format to add buffers; you can only set the size of unnecessary buffers to zero. This is because the header of a ptlrpc (not the ptlrpc_body, mind you) has a count of buffers then a variable length (64-bit aligned) set of that many 32-bit buffer lengths (I'm going from memory here), and adding buffers can put a reply over the expected max size on the client side, leading to it being dropped. You can change a capsule's format to change the definition of a field from one without a swabber to one with a swabber. You'll see in many cases that the presence of a field (meaning, whether it's checked for or whether it has a non-zero size) is dependent on a flag in the mdt or ost body, as you mention. Replays are not the only interesting case here. Capabilities are another. Some of these flags could be removed and replaced instead with checks of buffer size (0 -> flag not set, >0 -> flag set). > Also format is different is you want to use MDS_REINT + sub commands > or you want to use something similar to MDS_SET_INFO. For > MDS_SET_INFO you use single format for all messages (just simple key > <> value) buffer, but for MDS_REINT you need two formats - one for > generic MDS_REINT code (get opcode from command, get locks, and > possible other) and own format for each opcode - such as open, > unlink, setxattr, setattr. all of them have a different number of > buffers (fields). The SET_INFO RPCs are kinda gross. I should know, since I finished the conversion of ost_handler.c to the new API. You can see that I used req_capsule_extend() to handle some SET_INFO cases. No, I didn't cover this detail, nor others, because I figured Vilobh needed a starting point, and that's all I was going to provide tonight. Nico -- From alexey.lyashkov at clusterstor.com Wed Oct 13 07:51:21 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Wed, 13 Oct 2010 10:51:21 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101013074345.GH1635@oracle.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> Message-ID: <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> On Oct 13, 2010, at 10:43, Nicolas Williams wrote: > On Wed, Oct 13, 2010 at 10:27:55AM +0300, Alexey Lyashkov wrote: >> eh.. Nicolas, >> >> Format for messages which want to reconstructed after resend and don't >> want recontructed - is different. >> >> As quick example it is OPEN request (via MDS_REINT command), that type >> message need a have extra buffer to store LOV EA, which to be send to >> MDS in replay case (with additional flag in header). (client have a >> copy data from a mds reply after ptlrpc finish processing request). >> That is why i say about "Reconstruct/replay case" > > Sure, but this buffer needs to be declared a priori. If you won't know > whether you'll need a buffer until later, that's OK: you declare it > anyways and you set its size to zero if you don't need it. > > You can't change a capsule's format to add buffers; you can only set the > size of unnecessary buffers to zero. but you can reassing format for a message :) if you look to MDT code, you can see - for MDS_REINT you have first format for operation inside REINT lustre use second format static int mdt_reint(struct mdt_thread_info *info) { long opc; int rc; static const struct req_format *reint_fmts[REINT_MAX] = { [REINT_SETATTR] = &RQF_MDS_REINT_SETATTR, [REINT_CREATE] = &RQF_MDS_REINT_CREATE, [REINT_LINK] = &RQF_MDS_REINT_LINK, [REINT_UNLINK] = &RQF_MDS_REINT_UNLINK, [REINT_RENAME] = &RQF_MDS_REINT_RENAME, [REINT_OPEN] = &RQF_MDS_REINT_OPEN, [REINT_SETXATTR] = &RQF_MDS_REINT_SETXATTR }; .. understand me ? -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From jeremy.filizetti at gmail.com Wed Oct 13 16:04:26 2010 From: jeremy.filizetti at gmail.com (Jeremy Filizetti) Date: Wed, 13 Oct 2010 12:04:26 -0400 Subject: [Lustre-devel] Large readdir RPCs project In-Reply-To: <4CA3EE66.2040206@whamcloud.com> References: <4CA2EE55.7010209@whamcloud.com> <4CA3EE66.2040206@whamcloud.com> Message-ID: I've put together a small patch to modify ll_dir_readpage and mdc_readpage to read extra pages (if available) with each RPC. It is posted under the bug (https://bugzilla.lustre.org/show_bug.cgi?id=17833). I see you we're the original assignee when you were at Sun/Oracle. Jeremy 2010/9/29 Fan Yong > On 9/30/10 3:01 AM, Jeremy Filizetti wrote: > > >> On the other hand, Large readdir RPCs is basic of another metadata read >> performance improvement features - "readdir+", which is quite useful for "ls >> -l" operation (for large directory), and reduce lookup/getattr RPC as much >> as possible. In such feature, MDS will pack more dir-item's attribute (not >> only name/ino as does currently by readdir, but also mode/owner, and etc) >> information back to client-side in "readdir+" RPC. That means the dir-item >> count in one "readdir+" page is less than in the traditional readdir page, >> then more pages to be sent back to client. If without "Large readdir RPCs", >> the advantage of "readdir+" will be discounted. >> > > I'd be interested in working this as well but probably as a separate > effort since SOM isn't in 1.8 and that's my main focus. In our testing, SOM > had significant benefits over the WAN and I'd expect even better from > readdir+. I have tried Oleg's patch for asynchronous ll_glimpse_size but > oddly I've seen someone erradic performance where at times it was worse then > statahead and synchronous ll_glimpse_size. > > Jeremy > > > Yes, SOM is another important feature for metadata reading performance > improvement by bypassing the glimpse RPC between client and OSS. Engineers > from Lustre Group worked for that for some time, hope can be released soon. > > As for the asynchronous ll_glimpse_size maybe cause bad performance > occasionally, one possible reason is that: glimpse RPC maybe not obtain > extent lock(s) if some others in using such lock(s), so the file size > information obtained by asynchronous glimpse is invalid when it is really > used later, means the caller ("stat") has to send synchronous glimpse again. > Anyway, I did not study such patch, so it is just a guess. > > > Cheers, > Nasf > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yong.fan at whamcloud.com Wed Oct 13 16:32:54 2010 From: yong.fan at whamcloud.com (Fan Yong) Date: Thu, 14 Oct 2010 00:32:54 +0800 Subject: [Lustre-devel] Large readdir RPCs project In-Reply-To: References: <4CA2EE55.7010209@whamcloud.com> <4CA3EE66.2040206@whamcloud.com> Message-ID: <4CB5DF36.3020508@whamcloud.com> On 10/14/10 12:04 AM, Jeremy Filizetti wrote: > I've put together a small patch to modify ll_dir_readpage and > mdc_readpage to read extra pages (if available) with each RPC. It is > posted under the bug > (https://bugzilla.lustre.org/show_bug.cgi?id=17833). I see you we're > the original assignee when you were at Sun/Oracle. Thanks, I will study such patch. I added Lsy into such bug CC list who is interested in it also. -- Nasf > Jeremy > > 2010/9/29 Fan Yong > > > On 9/30/10 3:01 AM, Jeremy Filizetti wrote: >> >> >> On the other hand, Large readdir RPCs is basic of another >> metadata read performance improvement features - "readdir+", >> which is quite useful for "ls -l" operation (for large >> directory), and reduce lookup/getattr RPC as much as >> possible. In such feature, MDS will pack more dir-item's >> attribute (not only name/ino as does currently by readdir, >> but also mode/owner, and etc) information back to client-side >> in "readdir+" RPC. That means the dir-item count in one >> "readdir+" page is less than in the traditional readdir page, >> then more pages to be sent back to client. If without "Large >> readdir RPCs", the advantage of "readdir+" will be discounted. >> >> I'd be interested in working this as well but probably as a >> separate effort since SOM isn't in 1.8 and that's my main focus. >> In our testing, SOM had significant benefits over the WAN and I'd >> expect even better from readdir+. I have tried Oleg's patch for >> asynchronous ll_glimpse_size but oddly I've seen someone erradic >> performance where at times it was worse then statahead and >> synchronous ll_glimpse_size. >> Jeremy > Yes, SOM is another important feature for metadata reading > performance improvement by bypassing the glimpse RPC between > client and OSS. Engineers from Lustre Group worked for that for > some time, hope can be released soon. > > As for the asynchronous ll_glimpse_size maybe cause bad > performance occasionally, one possible reason is that: glimpse RPC > maybe not obtain extent lock(s) if some others in using such > lock(s), so the file size information obtained by asynchronous > glimpse is invalid when it is really used later, means the caller > ("stat") has to send synchronous glimpse again. Anyway, I did not > study such patch, so it is just a guess. > > > Cheers, > Nasf > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.murrell at oracle.com Wed Oct 13 18:39:21 2010 From: brian.murrell at oracle.com (Brian J. Murrell) Date: Wed, 13 Oct 2010 14:39:21 -0400 Subject: [Lustre-devel] How was lustre-1.8.4.tar.gz created? In-Reply-To: <4CAFC255.8090906@gmail.com> References: <4CAFC255.8090906@gmail.com> Message-ID: <1286995161.13574.1.camel@pc> On Fri, 2010-10-08 at 18:16 -0700, Jay Lan wrote: > So, how was lustre-1.8.4.tar.gz generated? [ assuming you have a git clone of prime ] $ git checkout 1.8.4 $ autogen.sh $ ./configure -disable-modules --disable-utils --disable-liblustre --disable-tests --disable-doc $ make dist That's it. b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From andreas.dilger at oracle.com Wed Oct 13 19:08:29 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 13 Oct 2010 13:08:29 -0600 Subject: [Lustre-devel] New lustre-community mailing list Message-ID: Hello All, I'd like to announce the creation of a new lustre-community at lists.lustre.org mailing list. After discussions with the various Lustre parties in the community, we thought there should be a new list to focus the "meta" discussion related to Lustre development in the community, such as how feature design, code development, patch contribution, and landing is coordinated, and how to test the resulting code, test harnesses for simplifying this, the mechanics of code inspections, etc. We'd like to keep this high-level discussion separate from the technical and code-specific discussions on lustre-devel, and the usage questions and general discussion on lustre-discuss. I think we all agree that maintaining the quality of the Lustre codebase going forward is important to everyone, so I hope we can have a productive discussion. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From mark.seger at hp.com Wed Oct 13 19:13:07 2010 From: mark.seger at hp.com (Seger, Mark) Date: Wed, 13 Oct 2010 19:13:07 +0000 Subject: [Lustre-devel] New lustre-community mailing list In-Reply-To: References: Message-ID: <9FCE3A46FE7C8045A6207AE4B42E9F9A4EADFEB346@GVW1119EXC.americas.hpqcorp.net> Good try. A lot of these people are unemployable and may with existing conditions so a million new jobs wouldn't have saved them. Good to hear you agree that nobody should die for lack of insurance. Perhaps you should explain it to seth some time -mark >-----Original Message----- >From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel- >bounces at lists.lustre.org] On Behalf Of Andreas Dilger >Sent: Wednesday, October 13, 2010 3:08 PM >To: lustre-announce at lists.lustre.org >Cc: Lustre-discuss discuss; lustre-devel List >Subject: [Lustre-devel] New lustre-community mailing list > >Hello All, >I'd like to announce the creation of a new lustre- >community at lists.lustre.org mailing list. After discussions with the >various Lustre parties in the community, we thought there should be a >new list to focus the "meta" discussion related to Lustre development in >the community, such as how feature design, code development, patch >contribution, and landing is coordinated, and how to test the resulting >code, test harnesses for simplifying this, the mechanics of code >inspections, etc. > >We'd like to keep this high-level discussion separate from the technical >and code-specific discussions on lustre-devel, and the usage questions >and general discussion on lustre-discuss. > >I think we all agree that maintaining the quality of the Lustre codebase >going forward is important to everyone, so I hope we can have a >productive discussion. > >Cheers, Andreas >-- >Andreas Dilger >Lustre Technical Lead >Oracle Corporation Canada Inc. > >_______________________________________________ >Lustre-devel mailing list >Lustre-devel at lists.lustre.org >http://lists.lustre.org/mailman/listinfo/lustre-devel From vilobh.meshram at gmail.com Wed Oct 13 23:51:37 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Wed, 13 Oct 2010 19:51:37 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> Message-ID: Hi Alexey/Nicholas/All, I had a look at the 2.0 side of the code and seems like there have been some significant modifications at the MDS side e.g. seems like the request processing part at MDS has been redefined. I need to stick to 1.8.1.1 since my most of the modifications are at the MDS side and the project also demands the same.So I will need to play around with the low-level message packing and unpacking stuff which is pretty complicated from my previous experience. Here is my understanding of the way the request are processed .Please correct me if I am wrong. 1) Seems like in Lustre code base for each RPC you have a static structure "size" and count defined which defines the way the message would be layed out (after doing all the rounding off operations etc) i.e the offset at which the buffers will be packed and so on. 2) At MDS side the swabber() function + some associated functions extract the buffer information. *What I need is :-* 1) Is it possible that without writing a new RPC in Lustre 1.8.1.1 I can append some string such as "Hello" to the exsisting message sent by the Client (with the buffer size set at client side by the count,size fields).I tried modifying the "size" of the request for one of the RPC in-built in Lustre __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; ---->> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS }; The thing I want to know is how can I send a buffer from the client side by modifying the static structure "size" mentioned above.What all main places do I need to consider to make this work. If the above step i.e appending a buffer in the "size" array is not possible then I can move to write a new RPC. Thanks, Vilobh *Graduate Research Associate* *Department of Computer Science* *The Ohio State University Columbus Ohio* On Wed, Oct 13, 2010 at 3:51 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > > On Oct 13, 2010, at 10:43, Nicolas Williams wrote: > > > On Wed, Oct 13, 2010 at 10:27:55AM +0300, Alexey Lyashkov wrote: > >> eh.. Nicolas, > >> > >> Format for messages which want to reconstructed after resend and don't > >> want recontructed - is different. > >> > >> As quick example it is OPEN request (via MDS_REINT command), that type > >> message need a have extra buffer to store LOV EA, which to be send to > >> MDS in replay case (with additional flag in header). (client have a > >> copy data from a mds reply after ptlrpc finish processing request). > >> That is why i say about "Reconstruct/replay case" > > > > Sure, but this buffer needs to be declared a priori. If you won't know > > whether you'll need a buffer until later, that's OK: you declare it > > anyways and you set its size to zero if you don't need it. > > > > You can't change a capsule's format to add buffers; you can only set the > > size of unnecessary buffers to zero. > but you can reassing format for a message :) > if you look to MDT code, you can see - for MDS_REINT you have first format > for operation inside REINT lustre use second format > > static int mdt_reint(struct mdt_thread_info *info) > { > long opc; > int rc; > > static const struct req_format *reint_fmts[REINT_MAX] = { > [REINT_SETATTR] = &RQF_MDS_REINT_SETATTR, > [REINT_CREATE] = &RQF_MDS_REINT_CREATE, > [REINT_LINK] = &RQF_MDS_REINT_LINK, > [REINT_UNLINK] = &RQF_MDS_REINT_UNLINK, > [REINT_RENAME] = &RQF_MDS_REINT_RENAME, > [REINT_OPEN] = &RQF_MDS_REINT_OPEN, > [REINT_SETXATTR] = &RQF_MDS_REINT_SETXATTR > }; > > .. > > understand me ? > > > -------------------------------------- > Alexey Lyashkov > alexey.lyashkov at clusterstor.com > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Williams at oracle.com Thu Oct 14 00:28:45 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 13 Oct 2010 19:28:45 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> Message-ID: <20101014002844.GY1635@oracle.com> On Wed, Oct 13, 2010 at 07:51:37PM -0400, Vilobh Meshram wrote: > 1) Is it possible that without writing a new RPC in Lustre 1.8.1.1 I can > append some string such as "Hello" to the exsisting message sent by the > Client (with the buffer size set at client side by the count,size fields).I > tried modifying the "size" of the request for one of the RPC in-built in > Lustre Yes, it's possible to add buffers to requests. It's not possible to add buffers to _replies_ to existing RPCs unless you know the client expects those additional buffers -- existing clients expect a given maxsize for each reply, and if your reply is bigger then it will get dropped. > __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct > ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct > ldlm_request) }; > > ---->> > __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct > ldlm_request) , > //how to add "char *str=Hello" ofcourse we > will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz > for a specific kind of RPC there are limited number of such MACROS > }; Add a buffer. Don't change the size of an existing buffer. > The thing I want to know is how can I send a buffer from the client side by > modifying the static structure "size" mentioned above.What all main places > do I need to consider to make this work. Add an element to the size[] array, then set it to the correct size when you know the length of the string. Look at the SET_INFO RPCs. > If the above step i.e appending a buffer in the "size" array is not possible > then I can move to write a new RPC. The size[] array is just a convenient place to store the sizes of the individual buffers while you construct them. Nico -- From vilobh.meshram at gmail.com Thu Oct 14 01:41:02 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Wed, 13 Oct 2010 21:41:02 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101014002844.GY1635@oracle.com> References: <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> Message-ID: Thanks Nicolas.I will try it out by today/tommorow . Seems like it will touch lot of places in the codebase :-) Thanks again. Thanks, Vilobh *Graduate Research Associate* *Department of Computer Science* *The Ohio State University Columbus Ohio* On Wed, Oct 13, 2010 at 8:28 PM, Nicolas Williams < Nicolas.Williams at oracle.com> wrote: > On Wed, Oct 13, 2010 at 07:51:37PM -0400, Vilobh Meshram wrote: > > 1) Is it possible that without writing a new RPC in Lustre 1.8.1.1 I can > > append some string such as "Hello" to the exsisting message sent by the > > Client (with the buffer size set at client side by the count,size > fields).I > > tried modifying the "size" of the request for one of the RPC in-built in > > Lustre > > Yes, it's possible to add buffers to requests. It's not possible to add > buffers to _replies_ to existing RPCs unless you know the client expects > those additional buffers -- existing clients expect a given maxsize for > each reply, and if your reply is bigger then it will get dropped. > > > __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct > > ptlrpc_body), > > [DLM_LOCKREQ_OFF] = sizeof(struct > > ldlm_request) }; > > > > ---->> > > __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct > ptlrpc_body), > > [DLM_LOCKREQ_OFF] = sizeof(struct > > ldlm_request) , > > //how to add "char *str=Hello" ofcourse > we > > will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF > bcz > > for a specific kind of RPC there are limited number of such MACROS > > }; > > Add a buffer. Don't change the size of an existing buffer. > > > The thing I want to know is how can I send a buffer from the client side > by > > modifying the static structure "size" mentioned above.What all main > places > > do I need to consider to make this work. > > Add an element to the size[] array, then set it to the correct size when > you know the length of the string. Look at the SET_INFO RPCs. > > > If the above step i.e appending a buffer in the "size" array is not > possible > > then I can move to write a new RPC. > > The size[] array is just a convenient place to store the sizes of the > individual buffers while you construct them. > > Nico > -- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Thu Oct 14 03:38:16 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 06:38:16 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101014002844.GY1635@oracle.com> References: <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> Message-ID: On Oct 14, 2010, at 03:28, Nicolas Williams wrote: > On Wed, Oct 13, 2010 at 07:51:37PM -0400, Vilobh Meshram wrote: >> 1) Is it possible that without writing a new RPC in Lustre 1.8.1.1 I can >> append some string such as "Hello" to the exsisting message sent by the >> Client (with the buffer size set at client side by the count,size fields).I >> tried modifying the "size" of the request for one of the RPC in-built in >> Lustre > > Yes, it's possible to add buffers to requests. It's not possible to add > buffers to _replies_ to existing RPCs unless you know the client expects > those additional buffers -- existing clients expect a given maxsize for > each reply, and if your reply is bigger then it will get dropped. It is wrong for last ~1year. ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. if (ev->mlength < ev->rlength ) { CDEBUG(D_RPCTRACE, "truncate req %p rpc %d - %d+%d\n", req, req->rq_replen, ev->rlength, ev->offset); req->rq_reply_truncate = 1; req->rq_replied = 1; req->rq_status = -EOVERFLOW; req->rq_nob_received = ev->rlength + ev->offset; ... if (req->rq_reply_truncate) { if (ptlrpc_no_resend(req)) { DEBUG_REQ(D_ERROR, req, "reply buffer overflow," " expected: %d, actual size: %d", req->rq_nob_received, req->rq_repbuf_len); RETURN(-EOVERFLOW); } sptlrpc_cli_free_repbuf(req); /* Pass the required reply buffer size (include * space for early reply). * NB: no need to roundup because alloc_repbuf * will roundup it */ req->rq_replen = req->rq_nob_received; req->rq_nob_received = 0; req->rq_resend = 1; RETURN(0); } -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From Nicolas.Williams at oracle.com Thu Oct 14 05:18:52 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Thu, 14 Oct 2010 00:18:52 -0500 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> Message-ID: <20101014051852.GZ1635@oracle.com> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: > On Oct 14, 2010, at 03:28, Nicolas Williams wrote: > > Yes, it's possible to add buffers to requests. It's not possible to add > > buffers to _replies_ to existing RPCs unless you know the client expects > > those additional buffers -- existing clients expect a given maxsize for > > each reply, and if your reply is bigger then it will get dropped. > It is wrong for last ~1year. > ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. Ah, I didn't know that was in 1.8. Are there interop issues (with older clients) though with sending larger replies than expected? From alexey.lyashkov at clusterstor.com Thu Oct 14 05:46:24 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 08:46:24 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101014051852.GZ1635@oracle.com> References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> Message-ID: On Oct 14, 2010, at 08:18, Nicolas Williams wrote: > On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>> Yes, it's possible to add buffers to requests. It's not possible to add >>> buffers to _replies_ to existing RPCs unless you know the client expects >>> those additional buffers -- existing clients expect a given maxsize for >>> each reply, and if your reply is bigger then it will get dropped. >> It is wrong for last ~1year. >> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. > > Ah, I didn't know that was in 1.8. that is added near of 1.8.1 and easy to check by grep -rn rq_reply_truncate in ptlrpc directory. Severity : normal Bugzilla : 19526 Description: can't stat file in some situation. Details : improve initialize osc date when target is added to mds and ability to resend too big getattr request is client isn't have info about ost. > Are there interop issues (with older > clients) though with sending larger replies than expected? I not clearly understand that question, but main propose of that change - problem with LOV EA buffer size for files with ACL (look to some conf-sanity tests). In some situation - MDS can have larger LOV EA buffer, when client expected (some files with wide striping have a reference to OST which removed from a cluster, or configuration lost, or new OST added but OST isn't connected to client or bad call shrink reply buffer, or other ... - you can find more references in bugzilla.) in that case MDS have send larger buffer to client. older client have infinity loop on connect or in stat syscall (because of messages without rq_no_resend flag) new client have a resend message - and adjust maximal size of LOV EA after got a valid reply, to avoid that's problem in future. -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From alexey.lyashkov at clusterstor.com Thu Oct 14 08:44:57 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 11:44:57 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <5A70DB00-9F04-47AF-A31C-01ADA3B87D5E@clusterstor.com> <17D943BA-FF0B-467F-8413-CB8C8184858C@clusterstor.com> <72D9946A-6452-4BC3-8C18-D2CA607D82DC@clusterstor.com> <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> Message-ID: <589D5419-E22E-4006-AEDA-22957F25290F@clusterstor.com> On Oct 14, 2010, at 02:51, Vilobh Meshram wrote: > > > 1) Is it possible that without writing a new RPC in Lustre 1.8.1.1 I can append some string such as "Hello" to the exsisting message sent by the Client (with the buffer size set at client side by the count,size fields).I tried modifying the "size" of the request for one of the RPC in-built in Lustre > > __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; > > ---->> > __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , > //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS > should be better if you complete describe - what you want to do, because some requests can't changed easy without compatibility lost, - like ELC (early lock cancel) feature which add extra buffer in messages and have special connect flag, to check request format changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at oracle.com Thu Oct 14 14:24:45 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Thu, 14 Oct 2010 08:24:45 -0600 Subject: [Lustre-devel] New lustre-community mailing list In-Reply-To: References: Message-ID: On 2010-10-13, at 13:08, Andreas Dilger wrote: > I'd like to announce the creation of a new lustre-community at lists.lustre.org mailing list. After discussions with the various Lustre parties in the community, we thought there should be a new list to focus the "meta" discussion related to Lustre development in the community, such as how feature design, code development, patch contribution, and landing is coordinated, and how to test the resulting code, test harnesses for simplifying this, the mechanics of code inspections, etc. Sorry not to have mentioned it in the original email, but to subscribe to this list visit http://lists.lustre.org/mailman/listinfo/lustre-community, or send an email to lustre-community-request at lists.lustre.org with the subject "subscribe". Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From andreas.dilger at oracle.com Thu Oct 14 14:31:26 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Thu, 14 Oct 2010 08:31:26 -0600 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <20101014051852.GZ1635@oracle.com> References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> Message-ID: On 2010-10-13, at 23:18, Nicolas Williams wrote: > On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>> Yes, it's possible to add buffers to requests. It's not possible to add >>> buffers to _replies_ to existing RPCs unless you know the client expects >>> those additional buffers -- existing clients expect a given maxsize for >>> each reply, and if your reply is bigger then it will get dropped. >> It is wrong for last ~1year. >> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. > > Ah, I didn't know that was in 1.8. Are there interop issues (with older > clients) though with sending larger replies than expected? Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer. Is that not true with the 2.x RPC handling? Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From alexey.lyashkov at clusterstor.com Thu Oct 14 14:40:14 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 17:40:14 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> Message-ID: <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> Andreas, On Oct 14, 2010, at 17:31, Andreas Dilger wrote: > On 2010-10-13, at 23:18, Nicolas Williams wrote: >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>>> Yes, it's possible to add buffers to requests. It's not possible to add >>>> buffers to _replies_ to existing RPCs unless you know the client expects >>>> those additional buffers -- existing clients expect a given maxsize for >>>> each reply, and if your reply is bigger then it will get dropped. >>> It is wrong for last ~1year. >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. >> >> Ah, I didn't know that was in 1.8. Are there interop issues (with older >> clients) though with sending larger replies than expected? > > Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer. > I think that question don't about rebalance buffers size in message, i think that sending large reply in smaller reply buffer. LNet don't able to put large reply to small buffer (without truncate flag, which is not exist in older ptlrpc version). without that flag you will see messages >> CERROR("Matching packet from %s, match "LPU64 " length %d too big: %d left, %d allowed\n", libcfs_id2str(src), match_bits, rlength, md->md_length - offset, mlength); >> and LNet will drop message without notify PtlRPC. > Is that not true with the 2.x RPC handling? > 2.x able to rebalance space between buffers (but looks by hand), and able adjust reply buffer after truncated reply. -------------------------------------- Alexey Lyashkov alexey.lyashkov at clusterstor.com From vilobh.meshram at gmail.com Thu Oct 14 15:04:37 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Thu, 14 Oct 2010 11:04:37 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> Message-ID: Hi Alexey, Thanks again for your reply. I am trying to embed a buffer in the RPC which will get filled in with some values which MDS is aware of which the client calling the RPC is not aware of.It has nothing to do with locking.I just want to fill in the buffer which I embedd in the RPC with some suitable data from the MDS end and then do operations on that data at the client side.So I think the approach suggested by you and Nicholas of just including the sizeof(str) [the size of the expected information from the MDS] in the size[] array should be fine as done below :- __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; ---->> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS *Please correct me if I am wrong or please guide me if I need to consider few corner cases to handle this use case. *Thanks again. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > Andreas, > > On Oct 14, 2010, at 17:31, Andreas Dilger wrote: > > > On 2010-10-13, at 23:18, Nicolas Williams wrote: > >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: > >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: > >>>> Yes, it's possible to add buffers to requests. It's not possible to > add > >>>> buffers to _replies_ to existing RPCs unless you know the client > expects > >>>> those additional buffers -- existing clients expect a given maxsize > for > >>>> each reply, and if your reply is bigger then it will get dropped. > >>> It is wrong for last ~1year. > >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, > and resend a request. > >> > >> Ah, I didn't know that was in 1.8. Are there interop issues (with older > >> clients) though with sending larger replies than expected? > > > > Nico, it has always been possible in the past to increase the size of any > buffer in a request, or in a reply (if the total reply size will fit into > the pre-allocated reply buffer). An older peer would just ignore the bytes > beyond the known part of the buffer. > > > I think that question don't about rebalance buffers size in message, > i think that sending large reply in smaller reply buffer. > LNet don't able to put large reply to small buffer (without truncate flag, > which is not exist in older ptlrpc version). > without that flag you will see messages > >> > CERROR("Matching packet from %s, match "LPU64 > " length %d too big: %d left, %d allowed\n", > libcfs_id2str(src), match_bits, rlength, > md->md_length - offset, mlength); > >> > and LNet will drop message without notify PtlRPC. > > > > Is that not true with the 2.x RPC handling? > > > 2.x able to rebalance space between buffers (but looks by hand), and able > adjust reply buffer after truncated reply. > > > > -------------------------------------- > Alexey Lyashkov > alexey.lyashkov at clusterstor.com > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Thu Oct 14 15:10:53 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 18:10:53 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> Message-ID: <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> Hi Vilobh, as i see, you touched code related to locking. struct ldm_request used to lock enqueue process - that why i say about interop issue in ELC code, which solved with export flag. for common mdc requests you can resolve interop issue with flags in mdc_body (mdt_body), but that not possible for ldlm requests. On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: > Hi Alexey, > > Thanks again for your reply. > > I am trying to embed a buffer in the RPC which will get filled in with some values which MDS is aware of which the client calling the RPC is not aware of.It has nothing to do with locking.I just want to fill in the buffer which I embedd in the RPC with some suitable data from the MDS end and then do operations on that data at the client side.So I think the approach suggested by you and Nicholas of just including the sizeof(str) [the size of the expected information from the MDS] in the size[] array should be fine as done below :- > > > > __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; > > ---->> > __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , > //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS > > > Please correct me if I am wrong or please guide me if I need to consider few corner cases to handle this use case. > > Thanks again. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov wrote: > Andreas, > > On Oct 14, 2010, at 17:31, Andreas Dilger wrote: > > > On 2010-10-13, at 23:18, Nicolas Williams wrote: > >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: > >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: > >>>> Yes, it's possible to add buffers to requests. It's not possible to add > >>>> buffers to _replies_ to existing RPCs unless you know the client expects > >>>> those additional buffers -- existing clients expect a given maxsize for > >>>> each reply, and if your reply is bigger then it will get dropped. > >>> It is wrong for last ~1year. > >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. > >> > >> Ah, I didn't know that was in 1.8. Are there interop issues (with older > >> clients) though with sending larger replies than expected? > > > > Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer. > > > I think that question don't about rebalance buffers size in message, > i think that sending large reply in smaller reply buffer. > LNet don't able to put large reply to small buffer (without truncate flag, which is not exist in older ptlrpc version). > without that flag you will see messages > >> > CERROR("Matching packet from %s, match "LPU64 > " length %d too big: %d left, %d allowed\n", > libcfs_id2str(src), match_bits, rlength, > md->md_length - offset, mlength); > >> > and LNet will drop message without notify PtlRPC. > > > > Is that not true with the 2.x RPC handling? > > > 2.x able to rebalance space between buffers (but looks by hand), and able adjust reply buffer after truncated reply. > > > > -------------------------------------- > Alexey Lyashkov > alexey.lyashkov at clusterstor.com > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Thu Oct 14 15:29:07 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Thu, 14 Oct 2010 11:29:07 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> Message-ID: Hi Alexey, Thanks again for the reply. Can you briefly give me some pointers about this interop issue and in which kind of RPC should this issue arise ? How should we resolve this what kind of flag needs to be set in ? I went through the bugzilla entry mentioned by you it seems like for RPCs dealing with LDLM may cause this issue.Please correct me if I am wrong. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > Hi Vilobh, > > as i see, you touched code related to locking. struct ldm_request used to > lock enqueue process - that why i say about interop issue in ELC code, which > solved with export flag. > for common mdc requests you can resolve interop issue with flags in > mdc_body (mdt_body), but that not possible for ldlm requests. > > > On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: > > Hi Alexey, > > Thanks again for your reply. > > I am trying to embed a buffer in the RPC which will get filled in with some > values which MDS is aware of which the client calling the RPC is not aware > of.It has nothing to do with locking.I just want to fill in the buffer > which I embedd in the RPC with some suitable data from the MDS end and then > do operations on that data at the client side.So I think the approach > suggested by you and Nicholas of just including the sizeof(str) [the size of > the expected information from the MDS] in the size[] array should be fine as > done below :- > > > > __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct > ldlm_request) }; > > ---->> > __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREQ_OFF] = sizeof(struct > ldlm_request) , > //how to add "char *str=Hello" ofcourse > we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF > bcz for a specific kind of RPC there are limited number of such MACROS > > > *Please correct me if I am wrong or please guide me if I need to consider > few corner cases to handle this use case. > > *Thanks again. > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > > > On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov < > alexey.lyashkov at clusterstor.com> wrote: > >> Andreas, >> >> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >> >> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >> >>>> Yes, it's possible to add buffers to requests. It's not possible to >> add >> >>>> buffers to _replies_ to existing RPCs unless you know the client >> expects >> >>>> those additional buffers -- existing clients expect a given maxsize >> for >> >>>> each reply, and if your reply is bigger then it will get dropped. >> >>> It is wrong for last ~1year. >> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, >> and resend a request. >> >> >> >> Ah, I didn't know that was in 1.8. Are there interop issues (with >> older >> >> clients) though with sending larger replies than expected? >> > >> > Nico, it has always been possible in the past to increase the size of >> any buffer in a request, or in a reply (if the total reply size will fit >> into the pre-allocated reply buffer). An older peer would just ignore the >> bytes beyond the known part of the buffer. >> > >> I think that question don't about rebalance buffers size in message, >> i think that sending large reply in smaller reply buffer. >> LNet don't able to put large reply to small buffer (without truncate flag, >> which is not exist in older ptlrpc version). >> without that flag you will see messages >> >> >> CERROR("Matching packet from %s, match "LPU64 >> " length %d too big: %d left, %d allowed\n", >> libcfs_id2str(src), match_bits, rlength, >> md->md_length - offset, mlength); >> >> >> and LNet will drop message without notify PtlRPC. >> >> >> > Is that not true with the 2.x RPC handling? >> > >> 2.x able to rebalance space between buffers (but looks by hand), and able >> adjust reply buffer after truncated reply. >> >> >> >> -------------------------------------- >> Alexey Lyashkov >> alexey.lyashkov at clusterstor.com >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Thu Oct 14 15:45:35 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Thu, 14 Oct 2010 18:45:35 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> Message-ID: <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Hi Vilobh, interop == interoperability between nodes with different version of software. in general we have two ways to solve that - for requests with mdc_body - you can set flag in body and analyze that flag in server/client side. if you want add new operation - better way add new flag into connect_data (look to OBD_CONNECT_* macroses handling) that flag can checked via export->connect_flags on client or server side for remote side features. as example 1.x and 2.0 have a different format for setattr requests : int mdc_setattr ... if (mdc_exp_is_2_0_server(exp)) { size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); size[REQ_REC_OFF + 1] = 0; /* capa */ size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); size[REQ_REC_OFF + 3] = ealen; size[REQ_REC_OFF + 4] = ea2len; size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); offset = REQ_REC_OFF + 5; bufcount = 6; replybufcount = 6; } else { bufcount = 4; } example of client features are checking version based recovery support for client mds_version_get_check ... if (inode == NULL || !exp_connect_vbr(req->rq_export)) I hope that help you. On Oct 14, 2010, at 18:29, Vilobh Meshram wrote: > Hi Alexey, > > Thanks again for the reply. > > Can you briefly give me some pointers about this interop issue and in which kind of RPC should this issue arise ? How should we resolve this what kind of flag needs to be set in ? > > I went through the bugzilla entry mentioned by you it seems like for RPCs dealing with LDLM may cause this issue.Please correct me if I am wrong. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov wrote: > Hi Vilobh, > > as i see, you touched code related to locking. struct ldm_request used to lock enqueue process - that why i say about interop issue in ELC code, which solved with export flag. > for common mdc requests you can resolve interop issue with flags in mdc_body (mdt_body), but that not possible for ldlm requests. > > > On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: > >> Hi Alexey, >> >> Thanks again for your reply. >> >> I am trying to embed a buffer in the RPC which will get filled in with some values which MDS is aware of which the client calling the RPC is not aware of.It has nothing to do with locking.I just want to fill in the buffer which I embedd in the RPC with some suitable data from the MDS end and then do operations on that data at the client side.So I think the approach suggested by you and Nicholas of just including the sizeof(str) [the size of the expected information from the MDS] in the size[] array should be fine as done below :- >> >> >> >> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; >> >> ---->> >> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , >> //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS >> >> >> Please correct me if I am wrong or please guide me if I need to consider few corner cases to handle this use case. >> >> Thanks again. >> >> Thanks, >> Vilobh >> Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio >> >> >> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov wrote: >> Andreas, >> >> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >> >> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >> >>>> Yes, it's possible to add buffers to requests. It's not possible to add >> >>>> buffers to _replies_ to existing RPCs unless you know the client expects >> >>>> those additional buffers -- existing clients expect a given maxsize for >> >>>> each reply, and if your reply is bigger then it will get dropped. >> >>> It is wrong for last ~1year. >> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. >> >> >> >> Ah, I didn't know that was in 1.8. Are there interop issues (with older >> >> clients) though with sending larger replies than expected? >> > >> > Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer. >> > >> I think that question don't about rebalance buffers size in message, >> i think that sending large reply in smaller reply buffer. >> LNet don't able to put large reply to small buffer (without truncate flag, which is not exist in older ptlrpc version). >> without that flag you will see messages >> >> >> CERROR("Matching packet from %s, match "LPU64 >> " length %d too big: %d left, %d allowed\n", >> libcfs_id2str(src), match_bits, rlength, >> md->md_length - offset, mlength); >> >> >> and LNet will drop message without notify PtlRPC. >> >> >> > Is that not true with the 2.x RPC handling? >> > >> 2.x able to rebalance space between buffers (but looks by hand), and able adjust reply buffer after truncated reply. >> >> >> >> -------------------------------------- >> Alexey Lyashkov >> alexey.lyashkov at clusterstor.com >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Thu Oct 14 16:25:41 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Thu, 14 Oct 2010 12:25:41 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Message-ID: Hi Alexey, That surely helps.Thanks for all the help till now. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > Hi Vilobh, > > interop == interoperability between nodes with different version of > software. > > in general we have two ways to solve that - for requests with mdc_body - > you can set flag in body and analyze that flag in server/client side. > if you want add new operation - better way add new flag into connect_data > (look to OBD_CONNECT_* macroses handling) > that flag can checked via export->connect_flags on client or server side > for remote side features. > as example 1.x and 2.0 have a different format for setattr requests : > int mdc_setattr > ... > if (mdc_exp_is_2_0_server(exp)) { > > size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); > > size[REQ_REC_OFF + 1] = 0; /* capa */ > > size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); > > size[REQ_REC_OFF + 3] = ealen; > > size[REQ_REC_OFF + 4] = ea2len; > > size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); > > offset = REQ_REC_OFF + 5; > > bufcount = 6; > > replybufcount = 6; > > } else { > > bufcount = 4; > > } > > > example of client features are checking version based recovery support for > client > mds_version_get_check > ... > if (inode == NULL || !exp_connect_vbr(req->rq_export)) > > > > I hope that help you. > > > On Oct 14, 2010, at 18:29, Vilobh Meshram wrote: > > Hi Alexey, > > Thanks again for the reply. > > Can you briefly give me some pointers about this interop issue and in which > kind of RPC should this issue arise ? How should we resolve this what kind > of flag needs to be set in ? > > I went through the bugzilla entry mentioned by you it seems like for RPCs > dealing with LDLM may cause this issue.Please correct me if I am wrong. > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > > > On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov < > alexey.lyashkov at clusterstor.com> wrote: > >> Hi Vilobh, >> >> as i see, you touched code related to locking. struct ldm_request used to >> lock enqueue process - that why i say about interop issue in ELC code, which >> solved with export flag. >> for common mdc requests you can resolve interop issue with flags in >> mdc_body (mdt_body), but that not possible for ldlm requests. >> >> >> On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: >> >> Hi Alexey, >> >> Thanks again for your reply. >> >> I am trying to embed a buffer in the RPC which will get filled in with >> some values which MDS is aware of which the client calling the RPC is not >> aware of.It has nothing to do with locking.I just want to fill in the >> buffer which I embedd in the RPC with some suitable data from the MDS end >> and then do operations on that data at the client side.So I think the >> approach suggested by you and Nicholas of just including the sizeof(str) >> [the size of the expected information from the MDS] in the size[] array >> should be fine as done below :- >> >> >> >> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >> [DLM_LOCKREQ_OFF] = sizeof(struct >> ldlm_request) }; >> >> ---->> >> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >> [DLM_LOCKREQ_OFF] = sizeof(struct >> ldlm_request) , >> //how to add "char *str=Hello" ofcourse >> we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF >> bcz for a specific kind of RPC there are limited number of such MACROS >> >> >> *Please correct me if I am wrong or please guide me if I need to consider >> few corner cases to handle this use case. >> >> *Thanks again. >> >> Thanks, >> Vilobh >> *Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio* >> >> >> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov < >> alexey.lyashkov at clusterstor.com> wrote: >> >>> Andreas, >>> >>> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >>> >>> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >>> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >>> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>> >>>> Yes, it's possible to add buffers to requests. It's not possible to >>> add >>> >>>> buffers to _replies_ to existing RPCs unless you know the client >>> expects >>> >>>> those additional buffers -- existing clients expect a given maxsize >>> for >>> >>>> each reply, and if your reply is bigger then it will get dropped. >>> >>> It is wrong for last ~1year. >>> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for >>> reply, and resend a request. >>> >> >>> >> Ah, I didn't know that was in 1.8. Are there interop issues (with >>> older >>> >> clients) though with sending larger replies than expected? >>> > >>> > Nico, it has always been possible in the past to increase the size of >>> any buffer in a request, or in a reply (if the total reply size will fit >>> into the pre-allocated reply buffer). An older peer would just ignore the >>> bytes beyond the known part of the buffer. >>> > >>> I think that question don't about rebalance buffers size in message, >>> i think that sending large reply in smaller reply buffer. >>> LNet don't able to put large reply to small buffer (without truncate >>> flag, which is not exist in older ptlrpc version). >>> without that flag you will see messages >>> >> >>> CERROR("Matching packet from %s, match "LPU64 >>> " length %d too big: %d left, %d allowed\n", >>> libcfs_id2str(src), match_bits, rlength, >>> md->md_length - offset, mlength); >>> >> >>> and LNet will drop message without notify PtlRPC. >>> >>> >>> > Is that not true with the 2.x RPC handling? >>> > >>> 2.x able to rebalance space between buffers (but looks by hand), and able >>> adjust reply buffer after truncated reply. >>> >>> >>> >>> -------------------------------------- >>> Alexey Lyashkov >>> alexey.lyashkov at clusterstor.com >>> >>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Fri Oct 15 00:58:02 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Thu, 14 Oct 2010 20:58:02 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Message-ID: Hi Alexey/Nicholas, I modified the code in following way in the way Nicholas suggested yesterday in-order to get some information filled in a fixed sized buffer sent from client side.Here I am sending a buffer called "str" (whose size is 16) which will be updated at the MDS side by the string "hello"(whose size is 7 much less than original size of buffer "str" i.e 16).But I am not able to perform the operation successfully and I am getting an error "LustreError: 4209:0:(file.c:3143:ll_inode_revalidate_fini()) failure -14 inode 31257" which seems to be related to DLM_REPLY_REC_OFF since I have modified this offset in my code.Can you please review my code and suggest me if I am making any mistake.I will be done with my task if I can resolve this problem. Following are the modifications .The text in BOLD and Italics (blue color) are my modification at Client and MDS side for *Lustre 1.8.1.1*:- *At Client side :- lustre/ldlm/ldlm_lockd.c** * 655 int ldlm_cli_enqueue(.........) 665 __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), 666 [DLM_LOCKREQ_OFF] = sizeof(*body), 667 [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : 668 sizeof(struct ost_lvb), * 669 16};* 717 if (reqp == NULL || *reqp == NULL) { *718 req = ldlm_prep_enqueue_req(exp, 4, size, NULL, 0); | | v 575 struct ptlrpc_request *ldlm_prep_elc_req(.......) 584 void *str=NULL; 585 char *bufs[4] = {NULL,NULL,NULL,str}; 616 req = ptlrpc_prep_req(class_exp2cliimp(exp), version, 617 opc, bufcount, size, bufs**); At MDS side :- lustre/ldlm/ldlm_lockd.c 992 int ldlm_handle_enqueue(.........) 996 { 1000 void *str; __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) 1009 char *org = "hello"; *1119 existing_lock: 1120 1121 if (flags & LDLM_FL_HAS_INTENT) { 1122 /* In this case, the reply buffer is allocated deep in 1123 * local_lock_enqueue by the policy function. */ 1124 cookie = req; 1125 } else { *1126 int buffers = 4;* 1127 1128 lock_res_and_lock(lock); 1129 if (lock->l_resource->lr_lvb_len) { * size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; buffers = 4;* 1132 } 1133 unlock_res_and_lock(lock); 1134 1135 if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) 1136 GOTO(out, rc = -ENOMEM); * str = lustre_msg_buf(req->rq_reqmsg, DLM_REPLY_REC_OFF+1, 1); memcpy ( str , org , 7); size[DLM_REPLY_REC_OFF + 1] = 16; * Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Thu, Oct 14, 2010 at 12:25 PM, Vilobh Meshram wrote: > Hi Alexey, > > That surely helps.Thanks for all the help till now. > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > > > On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov < > alexey.lyashkov at clusterstor.com> wrote: > >> Hi Vilobh, >> >> interop == interoperability between nodes with different version of >> software. >> >> in general we have two ways to solve that - for requests with mdc_body - >> you can set flag in body and analyze that flag in server/client side. >> if you want add new operation - better way add new flag into connect_data >> (look to OBD_CONNECT_* macroses handling) >> that flag can checked via export->connect_flags on client or server side >> for remote side features. >> as example 1.x and 2.0 have a different format for setattr requests : >> int mdc_setattr >> ... >> if (mdc_exp_is_2_0_server(exp)) { >> >> size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); >> >> size[REQ_REC_OFF + 1] = 0; /* capa */ >> >> size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); >> >> size[REQ_REC_OFF + 3] = ealen; >> >> size[REQ_REC_OFF + 4] = ea2len; >> >> size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); >> >> offset = REQ_REC_OFF + 5; >> >> bufcount = 6; >> >> replybufcount = 6; >> >> } else { >> >> bufcount = 4; >> >> } >> >> >> example of client features are checking version based recovery support for >> client >> mds_version_get_check >> ... >> if (inode == NULL || !exp_connect_vbr(req->rq_export)) >> >> >> >> I hope that help you. >> >> >> On Oct 14, 2010, at 18:29, Vilobh Meshram wrote: >> >> Hi Alexey, >> >> Thanks again for the reply. >> >> Can you briefly give me some pointers about this interop issue and in >> which kind of RPC should this issue arise ? How should we resolve this what >> kind of flag needs to be set in ? >> >> I went through the bugzilla entry mentioned by you it seems like for RPCs >> dealing with LDLM may cause this issue.Please correct me if I am wrong. >> >> Thanks, >> Vilobh >> *Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio* >> >> >> On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov < >> alexey.lyashkov at clusterstor.com> wrote: >> >>> Hi Vilobh, >>> >>> as i see, you touched code related to locking. struct ldm_request used to >>> lock enqueue process - that why i say about interop issue in ELC code, which >>> solved with export flag. >>> for common mdc requests you can resolve interop issue with flags in >>> mdc_body (mdt_body), but that not possible for ldlm requests. >>> >>> >>> On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: >>> >>> Hi Alexey, >>> >>> Thanks again for your reply. >>> >>> I am trying to embed a buffer in the RPC which will get filled in with >>> some values which MDS is aware of which the client calling the RPC is not >>> aware of.It has nothing to do with locking.I just want to fill in the >>> buffer which I embedd in the RPC with some suitable data from the MDS end >>> and then do operations on that data at the client side.So I think the >>> approach suggested by you and Nicholas of just including the sizeof(str) >>> [the size of the expected information from the MDS] in the size[] array >>> should be fine as done below :- >>> >>> >>> >>> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >>> [DLM_LOCKREQ_OFF] = sizeof(struct >>> ldlm_request) }; >>> >>> ---->> >>> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct >>> ptlrpc_body), >>> [DLM_LOCKREQ_OFF] = sizeof(struct >>> ldlm_request) , >>> //how to add "char *str=Hello" ofcourse >>> we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF >>> bcz for a specific kind of RPC there are limited number of such MACROS >>> >>> >>> *Please correct me if I am wrong or please guide me if I need to >>> consider few corner cases to handle this use case. >>> >>> *Thanks again. >>> >>> Thanks, >>> Vilobh >>> *Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio* >>> >>> >>> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov < >>> alexey.lyashkov at clusterstor.com> wrote: >>> >>>> Andreas, >>>> >>>> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >>>> >>>> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >>>> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >>>> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>>> >>>> Yes, it's possible to add buffers to requests. It's not possible >>>> to add >>>> >>>> buffers to _replies_ to existing RPCs unless you know the client >>>> expects >>>> >>>> those additional buffers -- existing clients expect a given maxsize >>>> for >>>> >>>> each reply, and if your reply is bigger then it will get dropped. >>>> >>> It is wrong for last ~1year. >>>> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for >>>> reply, and resend a request. >>>> >> >>>> >> Ah, I didn't know that was in 1.8. Are there interop issues (with >>>> older >>>> >> clients) though with sending larger replies than expected? >>>> > >>>> > Nico, it has always been possible in the past to increase the size of >>>> any buffer in a request, or in a reply (if the total reply size will fit >>>> into the pre-allocated reply buffer). An older peer would just ignore the >>>> bytes beyond the known part of the buffer. >>>> > >>>> I think that question don't about rebalance buffers size in message, >>>> i think that sending large reply in smaller reply buffer. >>>> LNet don't able to put large reply to small buffer (without truncate >>>> flag, which is not exist in older ptlrpc version). >>>> without that flag you will see messages >>>> >> >>>> CERROR("Matching packet from %s, match "LPU64 >>>> " length %d too big: %d left, %d allowed\n", >>>> libcfs_id2str(src), match_bits, rlength, >>>> md->md_length - offset, mlength); >>>> >> >>>> and LNet will drop message without notify PtlRPC. >>>> >>>> >>>> > Is that not true with the 2.x RPC handling? >>>> > >>>> 2.x able to rebalance space between buffers (but looks by hand), and >>>> able adjust reply buffer after truncated reply. >>>> >>>> >>>> >>>> -------------------------------------- >>>> Alexey Lyashkov >>>> alexey.lyashkov at clusterstor.com >>>> >>>> >>>> >>>> >>>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at clusterstor.com Fri Oct 15 07:39:25 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Fri, 15 Oct 2010 10:39:25 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Message-ID: can you please attach diff file ? On Oct 15, 2010, at 03:58, Vilobh Meshram wrote: > Hi Alexey/Nicholas, > > I modified the code in following way in the way Nicholas suggested yesterday in-order to get some information filled in a fixed sized buffer sent from client side.Here I am sending a buffer called "str" (whose size is 16) which will be updated at the MDS side by the string "hello"(whose size is 7 much less than original size of buffer "str" i.e 16).But I am not able to perform the operation successfully and I am getting an error > "LustreError: 4209:0:(file.c:3143:ll_inode_revalidate_fini()) failure -14 inode 31257" > > which seems to be related to DLM_REPLY_REC_OFF since I have modified this offset in my code.Can you please review my code and suggest me if I am making any mistake.I will be done with my task if I can resolve this problem. > > Following are the modifications .The text in BOLD and Italics (blue color) are my modification at Client and MDS side for Lustre 1.8.1.1:- > > At Client side :- lustre/ldlm/ldlm_lockd.c > > 655 int ldlm_cli_enqueue(.........) > 665 __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > 666 [DLM_LOCKREQ_OFF] = sizeof(*body), > 667 [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : > 668 sizeof(struct ost_lvb), > 669 16}; > > 717 if (reqp == NULL || *reqp == NULL) { > 718 req = ldlm_prep_enqueue_req(exp, 4, size, NULL, 0); > | > | > v > > 575 struct ptlrpc_request *ldlm_prep_elc_req(.......) > 584 void *str=NULL; > 585 char *bufs[4] = {NULL,NULL,NULL,str}; > 616 req = ptlrpc_prep_req(class_exp2cliimp(exp), version, > 617 opc, bufcount, size, bufs); > > > At MDS side :- lustre/ldlm/ldlm_lockd.c > > 992 int ldlm_handle_enqueue(.........) > 996 { > 1000 void *str; > __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), > [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) > 1009 char *org = "hello"; > > > 1119 existing_lock: > 1120 > 1121 if (flags & LDLM_FL_HAS_INTENT) { > 1122 /* In this case, the reply buffer is allocated deep in > 1123 * local_lock_enqueue by the policy function. */ > 1124 cookie = req; > 1125 } else { > 1126 int buffers = 4; > 1127 > 1128 lock_res_and_lock(lock); > 1129 if (lock->l_resource->lr_lvb_len) { > size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; > buffers = 4; > 1132 } > 1133 unlock_res_and_lock(lock); > 1134 > 1135 if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) > 1136 GOTO(out, rc = -ENOMEM); > str = lustre_msg_buf(req->rq_reqmsg, DLM_REPLY_REC_OFF+1, 1); > memcpy ( str , org , 7); > size[DLM_REPLY_REC_OFF + 1] = 16; > > > > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Thu, Oct 14, 2010 at 12:25 PM, Vilobh Meshram wrote: > Hi Alexey, > > That surely helps.Thanks for all the help till now. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov wrote: > Hi Vilobh, > > interop == interoperability between nodes with different version of software. > > in general we have two ways to solve that - for requests with mdc_body - you can set flag in body and analyze that flag in server/client side. > if you want add new operation - better way add new flag into connect_data (look to OBD_CONNECT_* macroses handling) > that flag can checked via export->connect_flags on client or server side for remote side features. > as example 1.x and 2.0 have a different format for setattr requests : > int mdc_setattr > ... > if (mdc_exp_is_2_0_server(exp)) { > size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); > size[REQ_REC_OFF + 1] = 0; /* capa */ > size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); > size[REQ_REC_OFF + 3] = ealen; > size[REQ_REC_OFF + 4] = ea2len; > size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); > offset = REQ_REC_OFF + 5; > bufcount = 6; > replybufcount = 6; > } else { > bufcount = 4; > } > > example of client features are checking version based recovery support for client > mds_version_get_check > ... > if (inode == NULL || !exp_connect_vbr(req->rq_export)) > > > I hope that help you. > > > On Oct 14, 2010, at 18:29, Vilobh Meshram wrote: > >> Hi Alexey, >> >> Thanks again for the reply. >> >> Can you briefly give me some pointers about this interop issue and in which kind of RPC should this issue arise ? How should we resolve this what kind of flag needs to be set in ? >> >> I went through the bugzilla entry mentioned by you it seems like for RPCs dealing with LDLM may cause this issue.Please correct me if I am wrong. >> >> Thanks, >> Vilobh >> Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio >> >> >> On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov wrote: >> Hi Vilobh, >> >> as i see, you touched code related to locking. struct ldm_request used to lock enqueue process - that why i say about interop issue in ELC code, which solved with export flag. >> for common mdc requests you can resolve interop issue with flags in mdc_body (mdt_body), but that not possible for ldlm requests. >> >> >> On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: >> >>> Hi Alexey, >>> >>> Thanks again for your reply. >>> >>> I am trying to embed a buffer in the RPC which will get filled in with some values which MDS is aware of which the client calling the RPC is not aware of.It has nothing to do with locking.I just want to fill in the buffer which I embedd in the RPC with some suitable data from the MDS end and then do operations on that data at the client side.So I think the approach suggested by you and Nicholas of just including the sizeof(str) [the size of the expected information from the MDS] in the size[] array should be fine as done below :- >>> >>> >>> >>> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >>> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) }; >>> >>> ---->> >>> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >>> [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request) , >>> //how to add "char *str=Hello" ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF bcz for a specific kind of RPC there are limited number of such MACROS >>> >>> >>> Please correct me if I am wrong or please guide me if I need to consider few corner cases to handle this use case. >>> >>> Thanks again. >>> >>> Thanks, >>> Vilobh >>> Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio >>> >>> >>> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov wrote: >>> Andreas, >>> >>> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >>> >>> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >>> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >>> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>> >>>> Yes, it's possible to add buffers to requests. It's not possible to add >>> >>>> buffers to _replies_ to existing RPCs unless you know the client expects >>> >>>> those additional buffers -- existing clients expect a given maxsize for >>> >>>> each reply, and if your reply is bigger then it will get dropped. >>> >>> It is wrong for last ~1year. >>> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for reply, and resend a request. >>> >> >>> >> Ah, I didn't know that was in 1.8. Are there interop issues (with older >>> >> clients) though with sending larger replies than expected? >>> > >>> > Nico, it has always been possible in the past to increase the size of any buffer in a request, or in a reply (if the total reply size will fit into the pre-allocated reply buffer). An older peer would just ignore the bytes beyond the known part of the buffer. >>> > >>> I think that question don't about rebalance buffers size in message, >>> i think that sending large reply in smaller reply buffer. >>> LNet don't able to put large reply to small buffer (without truncate flag, which is not exist in older ptlrpc version). >>> without that flag you will see messages >>> >> >>> CERROR("Matching packet from %s, match "LPU64 >>> " length %d too big: %d left, %d allowed\n", >>> libcfs_id2str(src), match_bits, rlength, >>> md->md_length - offset, mlength); >>> >> >>> and LNet will drop message without notify PtlRPC. >>> >>> >>> > Is that not true with the 2.x RPC handling? >>> > >>> 2.x able to rebalance space between buffers (but looks by hand), and able adjust reply buffer after truncated reply. >>> >>> >>> >>> -------------------------------------- >>> Alexey Lyashkov >>> alexey.lyashkov at clusterstor.com >>> >>> >>> >>> >>> >> >> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Fri Oct 15 16:25:02 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Fri, 15 Oct 2010 12:25:02 -0400 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Message-ID: Hi Alexey, I have attached the diff file .Please have a look at it and please let me know your comments /suggestions. Thanks again. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Fri, Oct 15, 2010 at 3:39 AM, Alexey Lyashkov < alexey.lyashkov at clusterstor.com> wrote: > can you please attach diff file ? > > On Oct 15, 2010, at 03:58, Vilobh Meshram wrote: > > Hi Alexey/Nicholas, > > I modified the code in following way in the way Nicholas suggested > yesterday in-order to get some information filled in a fixed sized buffer > sent from client side.Here I am sending a buffer called "str" (whose size is > 16) which will be updated at the MDS side by the string "hello"(whose size > is 7 much less than original size of buffer "str" i.e 16).But I am not able > to perform the operation successfully and I am getting an error > "LustreError: 4209:0:(file.c:3143:ll_inode_revalidate_fini()) failure -14 > inode 31257" > > which seems to be related to DLM_REPLY_REC_OFF since I have modified this > offset in my code.Can you please review my code and suggest me if I am > making any mistake.I will be done with my task if I can resolve this > problem. > > Following are the modifications .The text in BOLD and Italics (blue color) > are my modification at Client and MDS side for *Lustre 1.8.1.1*:- > > *At Client side :- lustre/ldlm/ldlm_lockd.c** > > * 655 int ldlm_cli_enqueue(.........) > 665 __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct > ptlrpc_body), > 666 [DLM_LOCKREQ_OFF] = sizeof(*body), > 667 [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : > 668 sizeof(struct > ost_lvb), > * 669 16};* > > 717 if (reqp == NULL || *reqp == NULL) { > *718 req = ldlm_prep_enqueue_req(exp, 4, size, NULL, 0); > | > | > v > > 575 struct ptlrpc_request *ldlm_prep_elc_req(.......) > 584 void *str=NULL; > 585 char *bufs[4] = {NULL,NULL,NULL,str}; > 616 req = > ptlrpc_prep_req(class_exp2cliimp(exp), version, > 617 opc, bufcount, > size, bufs**); > > > At MDS side :- lustre/ldlm/ldlm_lockd.c > > 992 int ldlm_handle_enqueue(.........) > 996 { > 1000 void *str; > __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct > ptlrpc_body), > [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) > 1009 char *org = "hello"; > > > *1119 existing_lock: > 1120 > 1121 if (flags & LDLM_FL_HAS_INTENT) { > 1122 /* In this case, the reply buffer is allocated deep in > 1123 * local_lock_enqueue by the policy function. */ > 1124 cookie = req; > 1125 } else { > *1126 int buffers = 4;* > 1127 > 1128 lock_res_and_lock(lock); > 1129 if (lock->l_resource->lr_lvb_len) { > * size[DLM_REPLY_REC_OFF] = > lock->l_resource->lr_lvb_len; > buffers = 4;* > 1132 } > 1133 unlock_res_and_lock(lock); > 1134 > 1135 if > (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) > 1136 GOTO(out, rc = -ENOMEM); > * str = lustre_msg_buf(req->rq_reqmsg, DLM_REPLY_REC_OFF+1, > 1); > memcpy ( str , org , 7); > size[DLM_REPLY_REC_OFF + 1] = 16; > > > * > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > > > On Thu, Oct 14, 2010 at 12:25 PM, Vilobh Meshram > wrote: > >> Hi Alexey, >> >> That surely helps.Thanks for all the help till now. >> >> Thanks, >> Vilobh >> *Graduate Research Associate >> Department of Computer Science >> The Ohio State University Columbus Ohio* >> >> >> On Thu, Oct 14, 2010 at 11:45 AM, Alexey Lyashkov < >> alexey.lyashkov at clusterstor.com> wrote: >> >>> Hi Vilobh, >>> >>> interop == interoperability between nodes with different version of >>> software. >>> >>> in general we have two ways to solve that - for requests with mdc_body - >>> you can set flag in body and analyze that flag in server/client side. >>> if you want add new operation - better way add new flag into >>> connect_data (look to OBD_CONNECT_* macroses handling) >>> that flag can checked via export->connect_flags on client or server side >>> for remote side features. >>> as example 1.x and 2.0 have a different format for setattr requests : >>> int mdc_setattr >>> ... >>> if (mdc_exp_is_2_0_server(exp)) { >>> >>> size[REQ_REC_OFF] = sizeof(struct mdt_rec_setattr); >>> >>> size[REQ_REC_OFF + 1] = 0; /* capa */ >>> >>> size[REQ_REC_OFF + 2] = 0; //sizeof (struct mdt_epoch); >>> >>> size[REQ_REC_OFF + 3] = ealen; >>> >>> size[REQ_REC_OFF + 4] = ea2len; >>> >>> size[REQ_REC_OFF + 5] = sizeof(struct ldlm_request); >>> >>> offset = REQ_REC_OFF + 5; >>> >>> bufcount = 6; >>> >>> replybufcount = 6; >>> >>> } else { >>> >>> bufcount = 4; >>> >>> } >>> >>> >>> example of client features are checking version based recovery support >>> for client >>> mds_version_get_check >>> ... >>> if (inode == NULL || !exp_connect_vbr(req->rq_export)) >>> >>> >>> >>> I hope that help you. >>> >>> >>> On Oct 14, 2010, at 18:29, Vilobh Meshram wrote: >>> >>> Hi Alexey, >>> >>> Thanks again for the reply. >>> >>> Can you briefly give me some pointers about this interop issue and in >>> which kind of RPC should this issue arise ? How should we resolve this what >>> kind of flag needs to be set in ? >>> >>> I went through the bugzilla entry mentioned by you it seems like for RPCs >>> dealing with LDLM may cause this issue.Please correct me if I am wrong. >>> >>> Thanks, >>> Vilobh >>> *Graduate Research Associate >>> Department of Computer Science >>> The Ohio State University Columbus Ohio* >>> >>> >>> On Thu, Oct 14, 2010 at 11:10 AM, Alexey Lyashkov < >>> alexey.lyashkov at clusterstor.com> wrote: >>> >>>> Hi Vilobh, >>>> >>>> as i see, you touched code related to locking. struct ldm_request used >>>> to lock enqueue process - that why i say about interop issue in ELC code, >>>> which solved with export flag. >>>> for common mdc requests you can resolve interop issue with flags in >>>> mdc_body (mdt_body), but that not possible for ldlm requests. >>>> >>>> >>>> On Oct 14, 2010, at 18:04, Vilobh Meshram wrote: >>>> >>>> Hi Alexey, >>>> >>>> Thanks again for your reply. >>>> >>>> I am trying to embed a buffer in the RPC which will get filled in with >>>> some values which MDS is aware of which the client calling the RPC is not >>>> aware of.It has nothing to do with locking.I just want to fill in the >>>> buffer which I embedd in the RPC with some suitable data from the MDS end >>>> and then do operations on that data at the client side.So I think the >>>> approach suggested by you and Nicholas of just including the sizeof(str) >>>> [the size of the expected information from the MDS] in the size[] array >>>> should be fine as done below :- >>>> >>>> >>>> >>>> __u32 size[2] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), >>>> [DLM_LOCKREQ_OFF] = >>>> sizeof(struct ldlm_request) }; >>>> >>>> ---->> >>>> __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct >>>> ptlrpc_body), >>>> [DLM_LOCKREQ_OFF] = sizeof(struct >>>> ldlm_request) , >>>> //how to add "char *str=Hello" >>>> ofcourse we will have sizeof(str) but how to choose the MACRO like DLM_LOCKREQ_OFF >>>> bcz for a specific kind of RPC there are limited number of such MACROS >>>> >>>> >>>> *Please correct me if I am wrong or please guide me if I need to >>>> consider few corner cases to handle this use case. >>>> >>>> *Thanks again. >>>> >>>> Thanks, >>>> Vilobh >>>> *Graduate Research Associate >>>> Department of Computer Science >>>> The Ohio State University Columbus Ohio* >>>> >>>> >>>> On Thu, Oct 14, 2010 at 10:40 AM, Alexey Lyashkov < >>>> alexey.lyashkov at clusterstor.com> wrote: >>>> >>>>> Andreas, >>>>> >>>>> On Oct 14, 2010, at 17:31, Andreas Dilger wrote: >>>>> >>>>> > On 2010-10-13, at 23:18, Nicolas Williams wrote: >>>>> >> On Thu, Oct 14, 2010 at 06:38:16AM +0300, Alexey Lyashkov wrote: >>>>> >>> On Oct 14, 2010, at 03:28, Nicolas Williams wrote: >>>>> >>>> Yes, it's possible to add buffers to requests. It's not possible >>>>> to add >>>>> >>>> buffers to _replies_ to existing RPCs unless you know the client >>>>> expects >>>>> >>>> those additional buffers -- existing clients expect a given >>>>> maxsize for >>>>> >>>> each reply, and if your reply is bigger then it will get dropped. >>>>> >>> It is wrong for last ~1year. >>>>> >>> ~1year ago i add code to ptlrpc layer which a adjust buffer for >>>>> reply, and resend a request. >>>>> >> >>>>> >> Ah, I didn't know that was in 1.8. Are there interop issues (with >>>>> older >>>>> >> clients) though with sending larger replies than expected? >>>>> > >>>>> > Nico, it has always been possible in the past to increase the size of >>>>> any buffer in a request, or in a reply (if the total reply size will fit >>>>> into the pre-allocated reply buffer). An older peer would just ignore the >>>>> bytes beyond the known part of the buffer. >>>>> > >>>>> I think that question don't about rebalance buffers size in message, >>>>> i think that sending large reply in smaller reply buffer. >>>>> LNet don't able to put large reply to small buffer (without truncate >>>>> flag, which is not exist in older ptlrpc version). >>>>> without that flag you will see messages >>>>> >> >>>>> CERROR("Matching packet from %s, match "LPU64 >>>>> " length %d too big: %d left, %d allowed\n", >>>>> libcfs_id2str(src), match_bits, rlength, >>>>> md->md_length - offset, mlength); >>>>> >> >>>>> and LNet will drop message without notify PtlRPC. >>>>> >>>>> >>>>> > Is that not true with the 2.x RPC handling? >>>>> > >>>>> 2.x able to rebalance space between buffers (but looks by hand), and >>>>> able adjust reply buffer after truncated reply. >>>>> >>>>> >>>>> >>>>> -------------------------------------- >>>>> Alexey Lyashkov >>>>> alexey.lyashkov at clusterstor.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- Index: lustre.spec =================================================================== --- lustre.spec (revision 8279) +++ lustre.spec (working copy) @@ -1,7 +1,7 @@ # lustre.spec %{!?version: %define version 1.8.1.1} -%{!?kversion: %define kversion } -%{!?release: %define release } +%{!?kversion: %define kversion 2.6.18-128.7.1.el5-lustre.1.8.1.1smp-cust} +%{!?release: %define release 2.6.18_128.7.1.el5_lustre.1.8.1.1smp_cust_201010141227} %{!?lustre_name: %define lustre_name lustre} %define is_client %(bash -c "if [[ %{lustre_name} = *-client ]]; then echo -n '1'; else echo -n '0'; fi") @@ -104,7 +104,7 @@ # Set an explicit path to our Linux tree, if we can. cd $RPM_BUILD_DIR/lustre-%{version} -./configure '--disable-modules' '--disable-utils' '--disable-liblustre' '--disable-tests' '--disable-doc' --with-lustre-hack --with-sockets %{?configure_flags:configure_flags} \ +./configure '--with-o2ib=/usr/local/ofed/src/ofa_kernel' '--with-linux=/lib/modules/2.6.18-128.7.1.el5-lustre.1.8.1.1smp-cust/build' --with-lustre-hack --with-sockets %{?configure_flags:configure_flags} \ --sysconfdir=%{_sysconfdir} \ --mandir=%{_mandir} \ --libdir=%{_libdir} Index: lustre/mds/handler.c =================================================================== --- lustre/mds/handler.c (revision 8279) +++ lustre/mds/handler.c (working copy) @@ -1687,7 +1687,7 @@ mds->mds_max_mdsize, mds->mds_max_cookiesize }; int bufcount; - + printk("Inside function %s a hit for case MDS_REINT",__func__); /* NB only peek inside req now; mds_reint() will swab it */ if (opcp == NULL) { CERROR ("Can't inspect opcode\n"); @@ -1704,15 +1704,18 @@ switch (opc) { case REINT_CREATE: + printk("Inside function %s a hit for case REINT_CREATE",__func__); op = PTLRPC_LAST_CNTR + MDS_REINT_CREATE; break; case REINT_LINK: op = PTLRPC_LAST_CNTR + MDS_REINT_LINK; break; case REINT_OPEN: + printk("Inside function %s a hit for case REINT_OPEN",__func__); op = PTLRPC_LAST_CNTR + MDS_REINT_OPEN; break; case REINT_SETATTR: + printk("Inside function %s a hit for case REINT_SETATTR",__func__); op = PTLRPC_LAST_CNTR + MDS_REINT_SETATTR; break; case REINT_RENAME: @@ -1745,8 +1748,9 @@ if (opc == REINT_UNLINK || opc == REINT_RENAME) size[DLM_REPLY_REC_OFF + 1] = 0; } - + printk("Inside function %s in case MDS_REINT before calling lustre_pack_reply",__func__); rc = lustre_pack_reply(req, bufcount, size, NULL); + printk("Inside function %s in case MDS_REINT after calling lustre_pack_reply",__func__); if (rc) break; @@ -1756,6 +1760,7 @@ } case MDS_CLOSE: + printk("Inside function %s in case MDS_CLOSE",__func__); DEBUG_REQ(D_INODE, req, "close"); OBD_FAIL_RETURN(OBD_FAIL_MDS_CLOSE_NET, 0); rc = mds_close(req, REQ_REC_OFF); @@ -1798,6 +1803,7 @@ break; #endif case OBD_PING: + printk("Inside function %s got a hit at case OBD_PING",__func__); DEBUG_REQ(D_INODE, req, "ping"); rc = target_handle_ping(req); if (req->rq_export->exp_delayed) @@ -1811,6 +1817,7 @@ break; case LDLM_ENQUEUE: + printk("\n Inside function %s got a hit at case LDLM_ENQUEUE",__func__); DEBUG_REQ(D_INODE, req, "enqueue"); OBD_FAIL_RETURN(OBD_FAIL_LDLM_ENQUEUE, 0); rc = ldlm_handle_enqueue(req, ldlm_server_completion_ast, Index: lustre/ldlm/ldlm_request.c =================================================================== --- lustre/ldlm/ldlm_request.c (revision 8279) +++ lustre/ldlm/ldlm_request.c (working copy) @@ -581,6 +581,8 @@ int flags, avail, to_free, pack = 0; struct ldlm_request *dlm = NULL; struct ptlrpc_request *req; + void *str=NULL; + char *bufs[4] = {NULL,NULL,NULL,str}; CFS_LIST_HEAD(head); ENTRY; @@ -609,8 +611,10 @@ size[bufoff] = ldlm_request_bufsize(pack, opc); } + printk("\n Inside function %s before calling ptlrpc_prep_req",__func__); + printk("\n OPC for LDLM_ENQUEUE is %d",opc); req = ptlrpc_prep_req(class_exp2cliimp(exp), version, - opc, bufcount, size, NULL); + opc, bufcount, size, bufs); req->rq_export = class_export_get(exp); if (exp_connect_cancelset(exp) && req) { if (canceloff) { @@ -658,10 +662,11 @@ struct ldlm_lock *lock; struct ldlm_request *body; struct ldlm_reply *reply; - __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), + __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(*body), [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : - sizeof(struct ost_lvb) }; + sizeof(struct ost_lvb), + 16}; int is_replay = *flags & LDLM_FL_REPLAY; int req_passed_in = 1, rc, err; struct ptlrpc_request *req; @@ -710,7 +715,7 @@ /* lock not sent to server yet */ if (reqp == NULL || *reqp == NULL) { - req = ldlm_prep_enqueue_req(exp, 2, size, NULL, 0); + req = ldlm_prep_enqueue_req(exp, 4, size, NULL, 0); if (req == NULL) { failed_lock_cleanup(ns, lock, lockh, einfo->ei_mode); LDLM_LOCK_PUT(lock); Index: lustre/ldlm/ldlm_lockd.c =================================================================== --- lustre/ldlm/ldlm_lockd.c (revision 8279) +++ lustre/ldlm/ldlm_lockd.c (working copy) @@ -997,13 +997,17 @@ struct obd_device *obddev = req->rq_export->exp_obd; struct ldlm_reply *dlm_rep; struct ldlm_request *dlm_req; - __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), - [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) }; + void *str; + __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), + [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) + }; int rc = 0; __u32 flags; ldlm_error_t err = ELDLM_OK; struct ldlm_lock *lock = NULL; void *cookie = NULL; + char *org = "hello"; + ENTRY; LDLM_DEBUG_NOLOCK("server-side enqueue handler START"); @@ -1119,19 +1123,24 @@ * local_lock_enqueue by the policy function. */ cookie = req; } else { - int buffers = 2; + int buffers = 4; lock_res_and_lock(lock); if (lock->l_resource->lr_lvb_len) { size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; - buffers = 3; + buffers = 4; } unlock_res_and_lock(lock); if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) GOTO(out, rc = -ENOMEM); + str = lustre_msg_buf(req->rq_reqmsg, DLM_REPLY_REC_OFF+1, 1); + memcpy ( str , org , 7); + size[DLM_REPLY_REC_OFF + 1] = 16; + printk("\n Inside function %s before calling 1.LUSTRE_PACK_REPLY",__func__); rc = lustre_pack_reply(req, buffers, size, NULL); + printk("\n Inside function %s after calling 1.LUSTRE_PACK_REPLY",__func__); if (rc) GOTO(out, rc); } @@ -1215,7 +1224,9 @@ out: req->rq_status = rc ?: err; /* return either error - bug 11190 */ if (!req->rq_packed_final) { + printk("\n Inside function %s before calling 2.LUSTRE_PACK_REPLY",__func__); err = lustre_pack_reply(req, 1, NULL, NULL); + printk("\n Inside function %s after calling 2.LUSTRE_PACK_REPLY",__func__); if (rc == 0) rc = err; } From alexey.lyashkov at clusterstor.com Fri Oct 15 17:22:01 2010 From: alexey.lyashkov at clusterstor.com (Alexey Lyashkov) Date: Fri, 15 Oct 2010 20:22:01 +0300 Subject: [Lustre-devel] Query to understand the Lustre request/reply message In-Reply-To: References: <20101013054233.GC1635@oracle.com> <20101013071201.GD1635@oracle.com> <49BEA69F-6931-473E-AA86-4A676A71607A@clusterstor.com> <20101013074345.GH1635@oracle.com> <023F45A1-E881-4FD2-AF75-93324876C909@clusterstor.com> <20101014002844.GY1635@oracle.com> <20101014051852.GZ1635@oracle.com> <401C408D-C677-48EC-93FE-03333CCD9478@clusterstor.com> <419E085D-61B9-4434-A172-1A1DAC99B070@clusterstor.com> <7E0BFE49-914F-44AC-9534-D0544F83878D@clusterstor.com> Message-ID: <1D568B58-37C5-4B8A-81EC-8451D8B5EB7D@clusterstor.com> first comment. please use diff -p to see what function has changed. second, please use CDEBUG() if need :) you can set debug level via sysctl -w lnet.debug=-1, sysctl -w lnet.debug_subsystem=-1 after it you can get very detail log via lctl dk (dump_kernel) > $log-file. other comments i say tomorrow. On Oct 15, 2010, at 19:25, Vilobh Meshram wrote: > Hi Alexey, > > I have attached the diff file .Please have a look at it and please let me know your comments /suggestions. > > Thanks again. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Mon Oct 18 21:07:42 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Mon, 18 Oct 2010 17:07:42 -0400 Subject: [Lustre-devel] LIBLUSTRE CLIENTS Message-ID: Hi, What do we mean by LIBLUSTRE CLIENTS ? What all things are clients of LIBLUSTRE ? Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at oracle.com Mon Oct 18 22:38:10 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Mon, 18 Oct 2010 16:38:10 -0600 Subject: [Lustre-devel] LIBLUSTRE CLIENTS In-Reply-To: References: Message-ID: <661B8D44-77A8-44A8-B282-7D5D5F4B3A00@oracle.com> On 2010-10-18, at 15:07, Vilobh Meshram wrote: > What do we mean by LIBLUSTRE CLIENTS ? What all things are clients of LIBLUSTRE ? Liblustre is a user-space library that allows "mounting" a lustre filesystem by linking it to an application. This is somewhat older code and probably not something that you need to use. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From vilobh.meshram at gmail.com Mon Oct 18 23:33:36 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Mon, 18 Oct 2010 19:33:36 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE Message-ID: Hi, Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the most frequently used RPC to communicate between the client and the MDS.I have few queries regarding the same :- 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind of request ; through which the client can interact with the MDS ? I tried couple of experiments and found out that LDLM_ENQUEUE comes into picture while mounting the FS as well as when we do a lookup,create or open a file.I was expecting the MDS_REINT RPC to get invoked in case of a CREATE/OPEN request via mdc_create() but it seems like Lustre invokes LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). Please correct me if I am wrong. 2) In which cases (which system calls) does the MDS_REINT RPC will get invoked ? Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* -------------- next part -------------- An HTML attachment was scrubbed... URL: From yong.fan at whamcloud.com Tue Oct 19 15:46:37 2010 From: yong.fan at whamcloud.com (Fan Yong) Date: Tue, 19 Oct 2010 23:46:37 +0800 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: <4CBDBD5D.9060600@whamcloud.com> On 10/19/10 7:33 AM, Vilobh Meshram wrote: > Hi, > > Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the > most frequently used RPC to communicate between the client and the > MDS.I have few queries regarding the same :- > > 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind > of request ; through which the client can interact with the MDS ? > > I tried couple of experiments and found out that LDLM_ENQUEUE comes > into picture while mounting the FS as well as when we do a > lookup,create or open a file.I was expecting the MDS_REINT RPC to get > invoked in case of a CREATE/OPEN request via mdc_create() but it seems > like Lustre invokes LDLM_ENQEUE even for CREATE/OPEN( by packing the > intent related data). > Please correct me if I am wrong. For OPEN_CREATE case, it is through LDLM_ENQUEUE interface to communicate with MDS. > > 2) In which cases (which system calls) does the MDS_REINT RPC will get > invoked ? You can try mkdir/symlink/mknode to trigger MDS_REINT. Cheers, -- Nasf > > > Thanks, > Vilobh > /Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio/ > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From vilobh.meshram at gmail.com Tue Oct 19 20:28:48 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Tue, 19 Oct 2010 16:28:48 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: Hi, >From my exploration it seems like for create/open kind of request LDLM_ENQUEUE is the RPC through which the client talks to MDS.Please confirm on this. Since I could figure out that LDLM_ENQUEUE is the only RPC to interface with MDS I am planning to send the LDLM_ENQUEUE RPC *with some additonal buffer*from the client to the MDS so that based on some specific condition the MDS can fill the information in the buffer sent from the client.I have made some modifications to the code for the LDLM_ENQUEUE RPC but I am getting kernel panics.Can someone please help me and suggest me what is a good way to tackle this problem.I am using Lustre 1.8.1.1 and I cannot upgrade to Lustre 2.0. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Mon, Oct 18, 2010 at 7:33 PM, Vilobh Meshram wrote: > Hi, > > Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the most > frequently used RPC to communicate between the client and the MDS.I have few > queries regarding the same :- > > 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind of > request ; through which the client can interact with the MDS ? > > I tried couple of experiments and found out that LDLM_ENQUEUE comes into > picture while mounting the FS as well as when we do a lookup,create or open > a file.I was expecting the MDS_REINT RPC to get invoked in case of a > CREATE/OPEN request via mdc_create() but it seems like Lustre invokes > LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). > Please correct me if I am wrong. > > 2) In which cases (which system calls) does the MDS_REINT RPC will get > invoked ? > > > Thanks, > Vilobh > *Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at oracle.com Tue Oct 19 22:53:18 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Tue, 19 Oct 2010 16:53:18 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: On 2010-10-19, at 14:28, Vilobh Meshram wrote: > From my exploration it seems like for create/open kind of request LDLM_ENQUEUE is the RPC through which the client talks to MDS.Please confirm on this. > > Since I could figure out that LDLM_ENQUEUE is the only RPC to interface with MDS I am planning to send the LDLM_ENQUEUE RPC with some additonal buffer from the client to the MDS so that based on some specific condition the MDS can fill the information in the buffer sent from the client. This isn't correct. LDLM_ENQUEUE is used for enqueueing locks. It just happens that when Lustre wants to create a new file it enqueues a lock on the parent directory with the "intent" to create a new file. The MDS currently always replies "you cannot have the lock for the directory, I created the requested file for you". Similarly, when the client is getting attributes on a file, it needs a lock on that file in order to cache the attributes, and to save RPCs the attributes are returned with the lock. > I have made some modifications to the code for the LDLM_ENQUEUE RPC but I am getting kernel panics.Can someone please help me and suggest me what is a good way to tackle this problem.I am using Lustre 1.8.1.1 and I cannot upgrade to Lustre 2.0. It would REALLY be a lot easier to have this discussion with you if you actually told us what it is you are working on. Not only could we focus on the higher-level issue that you are trying to solve (instead of possibly wasting a lot of time focussing in a small issue that may in fact be completely irrelevant), but with many ideas related to Lustre it has probably already been discussed at length by the Lustre developers sometime over the past 8 years that we've been working on it. I suspect that the readership of this list could probably give you a lot of assistance with whatever you are working on, if you will only tell us what it actually is you are trying to do. > On Mon, Oct 18, 2010 at 7:33 PM, Vilobh Meshram wrote: >> Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the most frequently used RPC to communicate between the client and the MDS.I have few queries regarding the same :- >> >> 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind of request ; through which the client can interact with the MDS ? >> >> I tried couple of experiments and found out that LDLM_ENQUEUE comes into picture while mounting the FS as well as when we do a lookup,create or open a file.I was expecting the MDS_REINT RPC to get invoked in case of a CREATE/OPEN request via mdc_create() but it seems like Lustre invokes LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). >> Please correct me if I am wrong. >> >> 2) In which cases (which system calls) does the MDS_REINT RPC will get invoked ? Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From vilobh.meshram at gmail.com Wed Oct 20 02:04:21 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Tue, 19 Oct 2010 22:04:21 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: Hi Andreas, Thanks for your e-mail. We are trying to do following things.Please let me know if things are not clear :- Say we have 2 client C1 and C2 and a MDS .Say C1 and C2 share a file. 1) When a client C1 performs a open/create kind of request to the MDS we want to follow the normal path which Lustre performs. 2) Now say C2 tries to open the same file which was opened by C1. 3) At the MDS end we maintain some data structure to scan and see if the file was already opened by some Client(in this case C1 has opened this file). 4) If MDS finds that some client(C1 here) has already opened the file then it send the new client(C2 here) with some information about the client which has initially opened the file. 5) Once C2 gets the information its upto C2 to take further actions. 6) By this process we can save the time spent in the locking mechanism for C2.Basically we aim to by-pass the locking scheme of Lustre for the files already opened by some client by maintaining some kind of data structure. Please let us know your thoughts on the above approach.Is this a feasible design moving ahead can we see any complications ? So considering the problem statement I need a way for C2 to extract the information from the data structure maintained at MDS.In order to do that , C2 will send a request with intent = create|open which will be a LDLM_ENQUEUE RPC.I need to modify this RPC such that :- 1) I can enclose some additional buffer whose size is known to me . 2) When we pack the reply at the MDS side we should be able to include this buffer in the reply message . 3) At the client side we should be able to extract the information from the reply message about the buffer. As of now , I need help in above three steps. Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Tue, Oct 19, 2010 at 6:53 PM, Andreas Dilger wrote: > On 2010-10-19, at 14:28, Vilobh Meshram wrote: > > From my exploration it seems like for create/open kind of request > LDLM_ENQUEUE is the RPC through which the client talks to MDS.Please confirm > on this. > > > > Since I could figure out that LDLM_ENQUEUE is the only RPC to interface > with MDS I am planning to send the LDLM_ENQUEUE RPC with some additonal > buffer from the client to the MDS so that based on some specific condition > the MDS can fill the information in the buffer sent from the client. > > This isn't correct. LDLM_ENQUEUE is used for enqueueing locks. It just > happens that when Lustre wants to create a new file it enqueues a lock on > the parent directory with the "intent" to create a new file. The MDS > currently always replies "you cannot have the lock for the directory, I > created the requested file for you". Similarly, when the client is getting > attributes on a file, it needs a lock on that file in order to cache the > attributes, and to save RPCs the attributes are returned with the lock. > > > I have made some modifications to the code for the LDLM_ENQUEUE RPC but I > am getting kernel panics.Can someone please help me and suggest me what is a > good way to tackle this problem.I am using Lustre 1.8.1.1 and I cannot > upgrade to Lustre 2.0. > > It would REALLY be a lot easier to have this discussion with you if you > actually told us what it is you are working on. Not only could we focus on > the higher-level issue that you are trying to solve (instead of possibly > wasting a lot of time focussing in a small issue that may in fact be > completely irrelevant), but with many ideas related to Lustre it has > probably already been discussed at length by the Lustre developers sometime > over the past 8 years that we've been working on it. I suspect that the > readership of this list could probably give you a lot of assistance with > whatever you are working on, if you will only tell us what it actually is > you are trying to do. > > > On Mon, Oct 18, 2010 at 7:33 PM, Vilobh Meshram < > vilobh.meshram at gmail.com> wrote: > >> Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the most > frequently used RPC to communicate between the client and the MDS.I have few > queries regarding the same :- > >> > >> 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind of > request ; through which the client can interact with the MDS ? > >> > >> I tried couple of experiments and found out that LDLM_ENQUEUE comes into > picture while mounting the FS as well as when we do a lookup,create or open > a file.I was expecting the MDS_REINT RPC to get invoked in case of a > CREATE/OPEN request via mdc_create() but it seems like Lustre invokes > LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). > >> Please correct me if I am wrong. > >> > >> 2) In which cases (which system calls) does the MDS_REINT RPC will get > invoked ? > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at oracle.com Wed Oct 20 07:55:43 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 01:55:43 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: On 2010-10-19, at 20:04, Vilobh Meshram wrote: > We are trying to do following things.Please let me know if things are not clear :- > > Say we have 2 client C1 and C2 and a MDS .Say C1 and C2 share a file. > 1) When a client C1 performs a open/create kind of request to the MDS we want to follow the normal path which Lustre performs. > 2) Now say C2 tries to open the same file which was opened by C1. > 3) At the MDS end we maintain some data structure to scan and see if the file was already opened by some Client(in this case C1 has opened this file). > 4) If MDS finds that some client(C1 here) has already opened the file then it send the new client(C2 here) with some information about the client which has initially opened the file. While I understand the basic concept, I don't really see how your proposal will actually improve performance. If C2 already has to contact the MDS and get a reply from it, then wouldn't it be about the same to simply perform the open as is done today? The number of MDS RPCs is the same, and in fact this would avoid further message overhead between C1 and C2. > 5) Once C2 gets the information its upto C2 to take further actions. > 6) By this process we can save the time spent in the locking mechanism for C2.Basically we aim to by-pass the locking scheme of Lustre for the files already opened by some client by maintaining some kind of data structure. > > Please let us know your thoughts on the above approach.Is this a feasible design moving ahead can we see any complications ? There is a separate proposal that has been underway in the Linux community for some time, to allow a user process to get a file handle (i.e. binary blob returned from a new name_to_handle() syscall) from the kernel for a given pathname, and then later use that file handle in another process to open a file descriptor without re-traversing the path. I've been thinking this would be very useful for Lustre (and MPI in general), and have tried to steer the Linux development in a direction that would allow this to happen. Is this in line with what you are investigating? While this wouldn't eliminate the actual MDS open RPC (i.e. the LDLM_ENQUEUE you have been discussing), it could avoid the path traversal from each client, possibly saving {path_elements * num_clients} additional RPCs, > So considering the problem statement I need a way for C2 to extract the information from the data structure maintained at MDS.In order to do that , C2 will send a request with intent = create|open which will be a LDLM_ENQUEUE RPC.I need to modify this RPC such that :- > 1) I can enclose some additional buffer whose size is known to me . > 2) When we pack the reply at the MDS side we should be able to include this buffer in the reply message . > 3) At the client side we should be able to extract the information from the reply message about the buffer. > > As of now , I need help in above three steps. > > Thanks, > Vilobh > Graduate Research Associate > Department of Computer Science > The Ohio State University Columbus Ohio > > > On Tue, Oct 19, 2010 at 6:53 PM, Andreas Dilger wrote: > On 2010-10-19, at 14:28, Vilobh Meshram wrote: > > From my exploration it seems like for create/open kind of request LDLM_ENQUEUE is the RPC through which the client talks to MDS.Please confirm on this. > > > > Since I could figure out that LDLM_ENQUEUE is the only RPC to interface with MDS I am planning to send the LDLM_ENQUEUE RPC with some additonal buffer from the client to the MDS so that based on some specific condition the MDS can fill the information in the buffer sent from the client. > > This isn't correct. LDLM_ENQUEUE is used for enqueueing locks. It just happens that when Lustre wants to create a new file it enqueues a lock on the parent directory with the "intent" to create a new file. The MDS currently always replies "you cannot have the lock for the directory, I created the requested file for you". Similarly, when the client is getting attributes on a file, it needs a lock on that file in order to cache the attributes, and to save RPCs the attributes are returned with the lock. > > > I have made some modifications to the code for the LDLM_ENQUEUE RPC but I am getting kernel panics.Can someone please help me and suggest me what is a good way to tackle this problem.I am using Lustre 1.8.1.1 and I cannot upgrade to Lustre 2.0. > > It would REALLY be a lot easier to have this discussion with you if you actually told us what it is you are working on. Not only could we focus on the higher-level issue that you are trying to solve (instead of possibly wasting a lot of time focussing in a small issue that may in fact be completely irrelevant), but with many ideas related to Lustre it has probably already been discussed at length by the Lustre developers sometime over the past 8 years that we've been working on it. I suspect that the readership of this list could probably give you a lot of assistance with whatever you are working on, if you will only tell us what it actually is you are trying to do. > > > On Mon, Oct 18, 2010 at 7:33 PM, Vilobh Meshram wrote: > >> Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the most frequently used RPC to communicate between the client and the MDS.I have few queries regarding the same :- > >> > >> 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind of request ; through which the client can interact with the MDS ? > >> > >> I tried couple of experiments and found out that LDLM_ENQUEUE comes into picture while mounting the FS as well as when we do a lookup,create or open a file.I was expecting the MDS_REINT RPC to get invoked in case of a CREATE/OPEN request via mdc_create() but it seems like Lustre invokes LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). > >> Please correct me if I am wrong. > >> > >> 2) In which cases (which system calls) does the MDS_REINT RPC will get invoked ? > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From bzzz.tomas at gmail.com Wed Oct 20 08:11:01 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 12:11:01 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: <4CBEA415.80307@gmail.com> On 10/20/10 11:55 AM, Andreas Dilger wrote: > There is a separate proposal that has been underway in the Linux community for some time, to allow a user process to get a file handle (i.e. binary blob returned from a new name_to_handle() syscall) from the kernel for a given pathname, and then later use that file handle in another process to open a file descriptor without re-traversing the path. > > I've been thinking this would be very useful for Lustre (and MPI in general), and have tried to steer the Linux development in a direction that would allow this to happen. Is this in line with what you are investigating? with FIDs is quite possible and even safe if application can learn it (using xattr_get or ioctl). then it should be trivial to export FID namespace on MDS via special .lustre-fids directory? another idea was to do whole path traversal on MDS within a single RPC. bug that'd require amount of changes to llite and/or VFS and keep MDS a bottleneck. thanks, z From andreas.dilger at oracle.com Wed Oct 20 08:24:26 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 02:24:26 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBEA415.80307@gmail.com> References: <4CBEA415.80307@gmail.com> Message-ID: <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> On 2010-10-20, at 02:11, bzzz.tomas at gmail.com wrote: > On 10/20/10 11:55 AM, Andreas Dilger wrote: >> There is a separate proposal that has been underway in the Linux community for some time, to allow a user process to get a file handle (i.e. binary blob returned from a new name_to_handle() syscall) from the kernel for a given pathname, and then later use that file handle in another process to open a file descriptor without re-traversing the path. >> >> I've been thinking this would be very useful for Lustre (and MPI in general), and have tried to steer the Linux development in a direction that would allow this to happen. Is this in line with what you are investigating? > > with FIDs is quite possible and even safe if application can learn it > (using xattr_get or ioctl). then it should be trivial to export FID > namespace on MDS via special .lustre-fids directory? I'm reluctant to expose the whole FID namespace to applications, since this completely bypasses all directory permissions and allows opening files only based on their inode permissions. If we require a name_to_handle() syscall to succeed first, before allowing open_by_handle() to work, then at least we know that one of the involved processes was able to do a full path traversal. > another idea was to do whole path traversal on MDS within a single RPC. > bug that'd require amount of changes to llite and/or VFS and keep MDS > a bottleneck. This was discussed a long time ago, and has the potential drawback that if one of the path components is over-mounted on the client (e.g. local RAM-based tmpfs on a Lustre root filesystem) then the MDS-side path traversal would be incorrect. It could return an entry underneath the mountpoint, instead of inside it. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From bzzz.tomas at gmail.com Wed Oct 20 08:30:33 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 12:30:33 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> Message-ID: <4CBEA8A9.9080802@gmail.com> On 10/20/10 12:24 PM, Andreas Dilger wrote: > I'm reluctant to expose the whole FID namespace to applications, since this completely bypasses all directory permissions and allows opening files only based on their inode permissions. If we require a name_to_handle() syscall to succeed first, before allowing open_by_handle() to work, then at least we know that one of the involved processes was able to do a full path traversal. yes, this is a good point. can be solved if you use FID + capability/signature ? >> another idea was to do whole path traversal on MDS within a single RPC. >> bug that'd require amount of changes to llite and/or VFS and keep MDS >> a bottleneck. > > This was discussed a long time ago, and has the potential drawback that if one of the path components is over-mounted on the client (e.g. local RAM-based tmpfs on a Lustre root filesystem) then the MDS-side path traversal would be incorrect. It could return an entry underneath the mountpoint, instead of inside it. yes, and that could be solved if server returns a series of FIDs, then client could check whether any of those is over-mounted? thanks, z From eeb at whamcloud.com Wed Oct 20 13:30:37 2010 From: eeb at whamcloud.com (Eric Barton) Date: Wed, 20 Oct 2010 15:30:37 +0200 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBEA8A9.9080802@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> Message-ID: <00d001cb705a$fd64cb80$f82e6280$@com> I do like the idea of a collective open, but I'm wondering if it can be implemented simply enough to be worth the effort. True, it avoids the O(n) load on the server of all the clients (re)populating their namespace caches, but it's only useful for parallel jobs - a scale-out NAS style workload can't benefit. Ultimately the O(n) will have to be replaced with something that scales O(log n) (e.g. with a fat tree of caching proxy servers). > On 10/20/10 12:24 PM, Andreas Dilger wrote: > > I'm reluctant to expose the whole FID namespace to applications, ??? It can just be opaque bytes to the app. > > since this completely bypasses all directory permissions and allows > > opening files only based on their inode permissions. If we require a > > name_to_handle() syscall to succeed first, before allowing > > open_by_handle() to work, then at least we know that one of the > > involved processes was able to do a full path traversal. I think this defeats the scalability objective - we trying to avoid having to pull the namespace into every client aren't we? > yes, this is a good point. can be solved if you use FID + > capability/signature ? Yes, I think capabilities are the only way collective open can be made secure "properly". And given the way we believe capabilities have to be implemented for scalability (i.e. to keep the capability cache down to a reasonable size on the server) any open by one node in a given client cluster may well have to confer the right to use the FID by any of its peers. > >> another idea was to do whole path traversal on MDS within a single > >> RPC. bug that'd require amount of changes to llite and/or VFS and > >> keep MDS a bottleneck. That's an optimization rather than a scalability feature. How much does it complicate the code? I'd hate to see something new tricky and fragile complicate further development. Cheers, Eric From bzzz.tomas at gmail.com Wed Oct 20 13:40:32 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 17:40:32 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <00d001cb705a$fd64cb80$f82e6280$@com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> Message-ID: <4CBEF150.9000007@gmail.com> On 10/20/10 5:30 PM, Eric Barton wrote: > I do like the idea of a collective open, but I'm wondering if it can be > implemented simply enough to be worth the effort. True, it avoids the O(n) > load on the server of all the clients (re)populating their namespace > caches, but it's only useful for parallel jobs - a scale-out NAS style > workload can't benefit. Ultimately the O(n) will have to be replaced with > something that scales O(log n) (e.g. with a fat tree of caching proxy > servers). in long-term I'd prefer proxy approach because this way we could improve number of cases, including existing POSIX apps doing open, stat, etc. >>>> another idea was to do whole path traversal on MDS within a single >>>> RPC. bug that'd require amount of changes to llite and/or VFS and >>>> keep MDS a bottleneck. > > That's an optimization rather than a scalability feature. How much does > it complicate the code? I'd hate to see something new tricky and fragile > complicate further development. yes, this is an optimization. good thing here is that single client can benefit a lot from this (replacing few RPCs with a single one). bad thing is that it can be quite quite complicated on the client side (the server side's part looks OK). thanks, z From Nikita_Danilov at xyratex.com Wed Oct 20 08:38:44 2010 From: Nikita_Danilov at xyratex.com (Nikita Danilov) Date: Wed, 20 Oct 2010 12:38:44 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBEA8A9.9080802@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> Message-ID: On 20 October 2010 12:30, wrote: > On 10/20/10 12:24 PM, Andreas Dilger wrote: > > I'm reluctant to expose the whole FID namespace to applications, since > this completely bypasses all directory permissions and allows opening files > only based on their inode permissions. If we require a name_to_handle() > syscall to succeed first, before allowing open_by_handle() to work, then at > least we know that one of the involved processes was able to do a full path > traversal. > > yes, this is a good point. can be solved if you use FID + > capability/signature ? > > >> another idea was to do whole path traversal on MDS within a single RPC. > >> bug that'd require amount of changes to llite and/or VFS and keep MDS > >> a bottleneck. > > > > This was discussed a long time ago, and has the potential drawback that > if one of the path components is over-mounted on the client (e.g. local > RAM-based tmpfs on a Lustre root filesystem) then the MDS-side path > traversal would be incorrect. It could return an entry underneath the > mountpoint, instead of inside it. > > yes, and that could be solved if server returns a series of FIDs, > then client could check whether any of those is over-mounted? > This is what sufficiently smart nfsv4 clients are supposed to do, by the way, I believe: issue a compound RPC with a sequence of LOOKUP requests and traverse returned sequence of file-id-s locally, checking for mount points. Nikita. > > thanks, z > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nicolas.Williams at oracle.com Wed Oct 20 14:45:25 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 20 Oct 2010 09:45:25 -0500 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> Message-ID: <20101020144524.GQ1635@oracle.com> On Wed, Oct 20, 2010 at 12:38:44PM +0400, Nikita Danilov wrote: > On 20 October 2010 12:30, wrote: > > yes, and that could be solved if server returns a series of FIDs, > > then client could check whether any of those is over-mounted? > > This is what sufficiently smart nfsv4 clients are supposed to do, by the > way, I believe: issue a compound RPC with a sequence of LOOKUP requests and > traverse returned sequence of file-id-s locally, checking for mount points. Yes. The detection and replication of server-side mountpoints on the client-side is called "mirror mounts" in Solaris, and it's quite handy. For clients the main issue is going to be whether the VFS allows plugins to resolve more than one path component at a time. From pauln at psc.edu Wed Oct 20 14:51:06 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 10:51:06 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <00d001cb705a$fd64cb80$f82e6280$@com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> Message-ID: <4CBF01DA.3090505@psc.edu> Eric Barton wrote: > I do like the idea of a collective open, but I'm wondering if it can be > implemented simply enough to be worth the effort. True, it avoids the O(n) > load on the server of all the clients (re)populating their namespace > caches, but it's only useful for parallel jobs - a scale-out NAS style > workload can't benefit. Ultimately the O(n) will have to be replaced with > something that scales O(log n) (e.g. with a fat tree of caching proxy > servers). Eric makes a good point in that only parallel jobs really need this feature. Unfortunately, at scale the system (both clients and servers) *really do* need something like this, especially if we continue pushing users to perform N-1 file I/O instead of 'file per process'. I too am in agreement that some sort of capability mechanism is the best approach. I wonder if this is something that could be done outside of POSIX and supported through a parallel I/O library? Perhaps a single application threads could make a special open call (/proc magic perhaps?) and obtain the glob of opaque bytes which are then broadcast to the rest of the client via mpi. Traversing the namespace would be avoided on all but one client. In such a scenario I don't feel that enforcing unix permissions at every level of the path is needed or sensible, the operation should be treated as a simple logical open. The question to the lustre experts - can enough state be packed into an opaque object such that the recv'ing client can construct the necessary cache state? > >> On 10/20/10 12:24 PM, Andreas Dilger wrote: >>> I'm reluctant to expose the whole FID namespace to applications, > > ??? It can just be opaque bytes to the app. > >>> since this completely bypasses all directory permissions and allows >>> opening files only based on their inode permissions. If we require a >>> name_to_handle() syscall to succeed first, before allowing >>> open_by_handle() to work, then at least we know that one of the >>> involved processes was able to do a full path traversal. > > I think this defeats the scalability objective - we trying to avoid having > to pull the namespace into every client aren't we? > >> yes, this is a good point. can be solved if you use FID + >> capability/signature ? > > Yes, I think capabilities are the only way collective open can be made > secure "properly". And given the way we believe capabilities have to be > implemented for scalability (i.e. to keep the capability cache down to a > reasonable size on the server) any open by one node in a given client > cluster may well have to confer the right to use the FID by any of its > peers. > >>>> another idea was to do whole path traversal on MDS within a single >>>> RPC. bug that'd require amount of changes to llite and/or VFS and >>>> keep MDS a bottleneck. > > That's an optimization rather than a scalability feature. How much does > it complicate the code? I'd hate to see something new tricky and fragile > complicate further development. > > Cheers, > Eric > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Nicolas.Williams at oracle.com Wed Oct 20 14:55:57 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 20 Oct 2010 09:55:57 -0500 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF01DA.3090505@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> Message-ID: <20101020145556.GR1635@oracle.com> On Wed, Oct 20, 2010 at 10:51:06AM -0400, Paul Nowoczynski wrote: > Eric makes a good point in that only parallel jobs really need this > feature. Unfortunately, at scale the system (both clients and servers) > *really do* need something like this, especially if we continue pushing > users to perform N-1 file I/O instead of 'file per process'. I too am in > agreement that some sort of capability mechanism is the best approach. I > wonder if this is something that could be done outside of POSIX and > supported through a parallel I/O library? Perhaps a single application > threads could make a special open call (/proc magic perhaps?) and obtain > the glob of opaque bytes which are then broadcast to the rest of the > client via mpi. Traversing the namespace would be avoided on all but one > client. In such a scenario I don't feel that enforcing unix permissions > at every level of the path is needed or sensible, the operation should > be treated as a simple logical open. The question to the lustre experts > - can enough state be packed into an opaque object such that the > recv'ing client can construct the necessary cache state? POSIX already has what you're asking for, and it's called openg() ;) From pauln at psc.edu Wed Oct 20 15:16:59 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 11:16:59 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <20101020145556.GR1635@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <20101020145556.GR1635@oracle.com> Message-ID: <4CBF07EB.3090001@psc.edu> Yes! I think I was at this HEC meeting a few years ago?? :) Here are the pointers to the manpages if anyone else is interested. http://www.opengroup.org/platform/hecewg/ So my question wasn't so much about the interface which is why I posed a scenario based on MPI. But rather, how feasible is it to import the necessary state the from the client issuing openg() to the rest? paul Nicolas Williams wrote: > On Wed, Oct 20, 2010 at 10:51:06AM -0400, Paul Nowoczynski wrote: > >> Eric makes a good point in that only parallel jobs really need this >> feature. Unfortunately, at scale the system (both clients and servers) >> *really do* need something like this, especially if we continue pushing >> users to perform N-1 file I/O instead of 'file per process'. I too am in >> agreement that some sort of capability mechanism is the best approach. I >> wonder if this is something that could be done outside of POSIX and >> supported through a parallel I/O library? Perhaps a single application >> threads could make a special open call (/proc magic perhaps?) and obtain >> the glob of opaque bytes which are then broadcast to the rest of the >> client via mpi. Traversing the namespace would be avoided on all but one >> client. In such a scenario I don't feel that enforcing unix permissions >> at every level of the path is needed or sensible, the operation should >> be treated as a simple logical open. The question to the lustre experts >> - can enough state be packed into an opaque object such that the >> recv'ing client can construct the necessary cache state? >> > > POSIX already has what you're asking for, and it's called openg() ;) > From bzzz.tomas at gmail.com Wed Oct 20 15:22:50 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 19:22:50 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF01DA.3090505@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> Message-ID: <4CBF094A.9020302@gmail.com> On 10/20/10 6:51 PM, Paul Nowoczynski wrote: > Eric makes a good point in that only parallel jobs really need this > feature. Unfortunately, at scale the system (both clients and servers) > *really do* need something like this, especially if we continue pushing > users to perform N-1 file I/O instead of 'file per process'. I too am in > agreement that some sort of capability mechanism is the best approach. I > wonder if this is something that could be done outside of POSIX and > supported through a parallel I/O library? Perhaps a single application > threads could make a special open call (/proc magic perhaps?) and obtain > the glob of opaque bytes which are then broadcast to the rest of the > client via mpi. Traversing the namespace would be avoided on all but one > client. In such a scenario I don't feel that enforcing unix permissions > at every level of the path is needed or sensible, the operation should > be treated as a simple logical open. The question to the lustre experts > - can enough state be packed into an opaque object such that the > recv'ing client can construct the necessary cache state? could you explain why is it so important to skip intermediate lookups? those are to be done once, then the clients will do them locally. is it because your nodes are getting new paths all the time or the nodes are rebooted very often and lose cache? thanks, z From andreas.dilger at oracle.com Wed Oct 20 16:07:35 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 10:07:35 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF07EB.3090001@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <20101020145556.GR1635@oracle.com> <4CBF07EB.3090001@psc.edu> Message-ID: <89C51724-8970-4D6E-B155-94E4668C6AE2@oracle.com> Note that I was in contact with the HECEWG also, and openg() was proposed to be renamed to be more understandable. The current Linux name_to_handle() proposal could be used to export an blob identifier (file handle that holds a FID, fs UUID, and a cookie/capability in the Lustre case) to userspace, then MPI-IO or some other mechanism can be used to distribute this to other client processes and open_by_handle() to convert this back into a file handle. One question is whether mpi_open() could be used for a collective operation (allowing this to be handled inside the Lustre ADIO layer) or if it would need specific application support? Cheers, Andreas On 2010-10-20, at 9:16, Paul Nowoczynski wrote: > Yes! I think I was at this HEC meeting a few years ago?? :) > Here are the pointers to the manpages if anyone else is interested. > http://www.opengroup.org/platform/hecewg/ > > So my question wasn't so much about the interface which is why I posed a > scenario based on MPI. But rather, how feasible is it to import the > necessary state the from the client issuing openg() to the rest? > paul > > > Nicolas Williams wrote: >> On Wed, Oct 20, 2010 at 10:51:06AM -0400, Paul Nowoczynski wrote: >> >>> Eric makes a good point in that only parallel jobs really need this >>> feature. Unfortunately, at scale the system (both clients and servers) >>> *really do* need something like this, especially if we continue pushing >>> users to perform N-1 file I/O instead of 'file per process'. I too am in >>> agreement that some sort of capability mechanism is the best approach. I >>> wonder if this is something that could be done outside of POSIX and >>> supported through a parallel I/O library? Perhaps a single application >>> threads could make a special open call (/proc magic perhaps?) and obtain >>> the glob of opaque bytes which are then broadcast to the rest of the >>> client via mpi. Traversing the namespace would be avoided on all but one >>> client. In such a scenario I don't feel that enforcing unix permissions >>> at every level of the path is needed or sensible, the operation should >>> be treated as a simple logical open. The question to the lustre experts >>> - can enough state be packed into an opaque object such that the >>> recv'ing client can construct the necessary cache state? >>> >> >> POSIX already has what you're asking for, and it's called openg() ;) >> > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From andreas.dilger at oracle.com Wed Oct 20 16:35:41 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 10:35:41 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <00d001cb705a$fd64cb80$f82e6280$@com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> Message-ID: <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> On 2010-10-20, at 7:30, Eric Barton wrote: >> On 10/20/10 12:24 PM, Andreas Dilger wrote: >>> I'm reluctant to expose the whole FID namespace to applications, > > ??? It can just be opaque bytes to the app. This was in reply to Alex Z's comments that we can just do open-by-FID from userspace. >>> since this completely bypasses all directory permissions and allows >>> opening files only based on their inode permissions. If we require a >>> name_to_handle() syscall to succeed first, before allowing >>> open_by_handle() to work, then at least we know that one of the >>> involved processes was able to do a full path traversal. > > I think this defeats the scalability objective - we trying to avoid having > to pull the namespace into every client aren't we? The name_to_handle() only needs to be called on a single node, and open_by_handle() is called on the other nodes. I agree that this doesn't avoid the full O(n) RPCs for the open itself but at least it does avoid the full path traversal from every client and on the MDS (replacing it with an MPI broadcast of the handle). > Cheers, Andreas -------------- next part -------------- An HTML attachment was scrubbed... URL: From pauln at psc.edu Wed Oct 20 16:43:46 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 12:43:46 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF094A.9020302@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> Message-ID: <4CBF1C42.1090109@psc.edu> bzzz.tomas at gmail.com wrote: > On 10/20/10 6:51 PM, Paul Nowoczynski wrote: > >> Eric makes a good point in that only parallel jobs really need this >> feature. Unfortunately, at scale the system (both clients and servers) >> *really do* need something like this, especially if we continue pushing >> users to perform N-1 file I/O instead of 'file per process'. I too am in >> agreement that some sort of capability mechanism is the best approach. I >> wonder if this is something that could be done outside of POSIX and >> supported through a parallel I/O library? Perhaps a single application >> threads could make a special open call (/proc magic perhaps?) and obtain >> the glob of opaque bytes which are then broadcast to the rest of the >> client via mpi. Traversing the namespace would be avoided on all but one >> client. In such a scenario I don't feel that enforcing unix permissions >> at every level of the path is needed or sensible, the operation should >> be treated as a simple logical open. The question to the lustre experts >> - can enough state be packed into an opaque object such that the >> recv'ing client can construct the necessary cache state? >> > > could you explain why is it so important to skip intermediate lookups? > those are to be done once, then the clients will do them locally. > is it because your nodes are getting new paths all the time or the nodes > are rebooted very often and lose cache? > It's for scalability reasons. When N clients traverse the namespace with the purpose of opening the same file the result is a storm of RPC requests which bear down on the metadata server. This type of activity becomes prohibitive especially when you start considering client counts > 10^4. An operation such as this is ripe for optimization because every client in the network is trying to build the same state. If you have a method for a single client to 'learn' the final state, i.e. the pathname -> fid translation, and broadcast it to its cohorts, it's a huge win because it eliminates an O(N) operation. paul > thanks, z > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From pauln at psc.edu Wed Oct 20 16:46:56 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 12:46:56 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> Message-ID: <4CBF1D00.9080402@psc.edu> > The name_to_handle() only needs to be called on a single node, and > open_by_handle() is called on the other nodes. I agree that this > doesn't avoid the full O(n) RPCs for the open itself but at least it > does avoid the full path traversal from every client and on the > MDS (replacing it with an MPI broadcast of the handle). Andreas, excuse my ignorance, but why does open_by_handle() need to issue an RPC? If it's to obtain the layout, couldn't the layout be encoded into the 'handle'? p > > Cheers, Andreas > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From bzzz.tomas at gmail.com Wed Oct 20 16:49:06 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 20:49:06 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF1C42.1090109@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> Message-ID: <4CBF1D82.60508@gmail.com> On 10/20/10 8:43 PM, Paul Nowoczynski wrote: > It's for scalability reasons. When N clients traverse the namespace with > the purpose of opening the same file the result is a storm of RPC > requests which bear down on the metadata server. This type of activity > becomes prohibitive especially when you start considering client counts > > 10^4. An operation such as this is ripe for optimization because > every client in the network is trying to build the same state. If you > have a method for a single client to 'learn' the final state, i.e. the > pathname -> fid translation, and broadcast it to its cohorts, it's a > huge win because it eliminates an O(N) operation. > paul clear enough, but what is the bottleneck here: MDS to handle lots of RPCs or network to pass RPCs ? thanks, z From andreas.dilger at oracle.com Wed Oct 20 17:00:53 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 11:00:53 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF1D00.9080402@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> <4CBF1D00.9080402@psc.edu> Message-ID: <269E1A0D-2117-4ADC-BCDE-67A60EA9B974@oracle.com> On 2010-10-20, at 10:46, Paul Nowoczynski wrote: > >> The name_to_handle() only needs to be called on a single node, and open_by_handle() is called on the other nodes. I agree that this doesn't avoid the full O(n) RPCs for the open itself but at least it does avoid the full path traversal from every client and on the MDS (replacing it with an MPI broadcast of the handle). > > excuse my ignorance, but why does open_by_handle() need to issue an RPC? If it's to obtain the layout, couldn't the layout be encoded into the 'handle'? In theory, yes. Practically, there is a size limit on the handle, and in large filesystems the layout is larger than this limit. Also, it depends on whether we want the MDS to have consistent behavior with the resulting open file descriptor or not. I suppose in many cases it would be possible to fake out an open file on the client without telling the MDS, but then there will be strange problems in some cases (e.g. stat() of the file, errors on close, etc.) that would result since the MDS won't know anything about the other openers. Maybe that is acceptable, I don't know. Cheers, Andreas From Nicolas.Williams at oracle.com Wed Oct 20 17:01:24 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 20 Oct 2010 12:01:24 -0500 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF1D00.9080402@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> <4CBF1D00.9080402@psc.edu> Message-ID: <20101020170124.GT1635@oracle.com> On Wed, Oct 20, 2010 at 12:46:56PM -0400, Paul Nowoczynski wrote: > > > The name_to_handle() only needs to be called on a single node, and > > open_by_handle() is called on the other nodes. I agree that this > > doesn't avoid the full O(n) RPCs for the open itself but at least it > > does avoid the full path traversal from every client and on the > > MDS (replacing it with an MPI broadcast of the handle). > Andreas, > excuse my ignorance, but why does open_by_handle() need to issue an > RPC? If it's to obtain the layout, couldn't the layout be encoded into > the 'handle'? If you don't mind having a huge handle, then yes, we could skip additional RPCs. A handle would have to consist of a {MGS address, FID, layout, access type, capability}, or so. From pauln at psc.edu Wed Oct 20 17:11:58 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 13:11:58 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF1D82.60508@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> Message-ID: <4CBF22DE.9080204@psc.edu> bzzz.tomas at gmail.com wrote: > On 10/20/10 8:43 PM, Paul Nowoczynski wrote: > >> It's for scalability reasons. When N clients traverse the namespace with >> the purpose of opening the same file the result is a storm of RPC >> requests which bear down on the metadata server. This type of activity >> becomes prohibitive especially when you start considering client counts >> > 10^4. An operation such as this is ripe for optimization because >> every client in the network is trying to build the same state. If you >> have a method for a single client to 'learn' the final state, i.e. the >> pathname -> fid translation, and broadcast it to its cohorts, it's a >> huge win because it eliminates an O(N) operation. >> paul >> > > clear enough, but what is the bottleneck here: MDS to handle lots of > RPCs or network to pass RPCs ? I could be wrong but my guess is that the network congestion caused by this communication pattern is a more serious problem. The mds should be able to easily service lookup rpc's since only the first few necessitate a read I/O from the disk. > thanks, z > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From Nicolas.Williams at Oracle.com Wed Oct 20 17:13:49 2010 From: Nicolas.Williams at Oracle.com (Nicolas Williams) Date: Wed, 20 Oct 2010 12:13:49 -0500 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <269E1A0D-2117-4ADC-BCDE-67A60EA9B974@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> <4CBF1D00.9080402@psc.edu> <269E1A0D-2117-4ADC-BCDE-67A60EA9B974@oracle.com> Message-ID: <20101020171348.GU1635@oracle.com> On Wed, Oct 20, 2010 at 11:00:53AM -0600, Andreas Dilger wrote: > On 2010-10-20, at 10:46, Paul Nowoczynski wrote: > >> The name_to_handle() only needs to be called on a single node, and > >> open_by_handle() is called on the other nodes. I agree that this > >> doesn't avoid the full O(n) RPCs for the open itself but at least > >> it does avoid the full path traversal from every client and on the > >> MDS (replacing it with an MPI broadcast of the handle). > > > > excuse my ignorance, but why does open_by_handle() need to issue an > > RPC? If it's to obtain the layout, couldn't the layout be encoded > > into the 'handle'? > > In theory, yes. Practically, there is a size limit on the handle, and > in large filesystems the layout is larger than this limit. > > Also, it depends on whether we want the MDS to have consistent > behavior with the resulting open file descriptor or not. > > I suppose in many cases it would be possible to fake out an open file > on the client without telling the MDS, but then there will be strange > problems in some cases (e.g. stat() of the file, errors on close, > etc.) that would result since the MDS won't know anything about the > other openers. Maybe that is acceptable, I don't know. Well, if we're going to add openg() (or whatever its name), we might as well add variants of stat() that don't require getting the size when the app doesn't need it, and forget about SOM, or forget about SOM when we know that a file might be open by unknown clients (recover issues here). Another possibility is that the handle encodes the current size, and that to write past that size requires an RPC to establish open state, but this ignores truncation. Another possibility is to say that a handle is only good as long as the original file descriptor remains open (recovery issues here), and that client can tell the MDS that it will be sharing its handle with other clients. Or that client could tell the MDS what all the clients are that will share that handle (recovery issues here too). Some sort of additional RPC seems hard to avoid here, but maybe it could be async for clients opening by handle. Nico -- From bzzz.tomas at gmail.com Wed Oct 20 17:18:59 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 21:18:59 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF22DE.9080204@psc.edu> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> Message-ID: <4CBF2483.7030805@gmail.com> On 10/20/10 9:11 PM, Paul Nowoczynski wrote: > I could be wrong but my guess is that the network congestion caused by > this communication pattern is a more serious problem. The mds should be > able to easily service lookup rpc's since only the first few necessitate > a read I/O from the disk. but then the network should be able to deal with storm of * <# clients> to read/write data? or it's a specific switch being the bottleneck to specific node? because if it isn't network, but MDS being a real bottleneck, then proxy might be a solution like Eric said above. not sure is this important in your case, but this would allow to use existing apps. of course, distribution tree for a handle may scale better. thanks, z From pauln at psc.edu Wed Oct 20 17:25:15 2010 From: pauln at psc.edu (Paul Nowoczynski) Date: Wed, 20 Oct 2010 13:25:15 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF2483.7030805@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> Message-ID: <4CBF25FB.5030708@psc.edu> have a look at this, it explains the type of problem networks have in dealing with these communication patterns. http://www.pdl.cmu.edu/Incast/ and yes, a proxy is a workable solution, and probably the most well rounded. The disadvantages is that it would presumably require more engineering to deploy. p bzzz.tomas at gmail.com wrote: > On 10/20/10 9:11 PM, Paul Nowoczynski wrote: > >> I could be wrong but my guess is that the network congestion caused by >> this communication pattern is a more serious problem. The mds should be >> able to easily service lookup rpc's since only the first few necessitate >> a read I/O from the disk. >> > > but then the network should be able to deal with storm of > * <# clients> to read/write data? > > or it's a specific switch being the bottleneck to specific node? > > because if it isn't network, but MDS being a real bottleneck, > then proxy might be a solution like Eric said above. not sure > is this important in your case, but this would allow to use > existing apps. > > of course, distribution tree for a handle may scale better. > > thanks, z > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From andreas.dilger at oracle.com Wed Oct 20 17:27:25 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 11:27:25 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF2483.7030805@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> Message-ID: <65A68391-7759-403D-A059-704410FFDA03@oracle.com> On 2010-10-20, at 11:18, bzzz.tomas at gmail.com wrote: > On 10/20/10 9:11 PM, Paul Nowoczynski wrote: >> I could be wrong but my guess is that the network congestion caused by >> this communication pattern is a more serious problem. The mds should be >> able to easily service lookup rpc's since only the first few necessitate >> a read I/O from the disk. > > but then the network should be able to deal with storm of > * <# clients> to read/write data? > > or it's a specific switch being the bottleneck to specific node? I think there is definitely non-trivial overhead of the MDS threads descending into the filesystem to do path lookup and permission checking than would be avoided. > because if it isn't network, but MDS being a real bottleneck, > then proxy might be a solution like Eric said above. not sure > is this important in your case, but this would allow to use > existing apps. > > of course, distribution tree for a handle may scale better. I don't think the actual distribution of the handle is a significant factor (this can be done via efficient broadcast in MPI layer). If we want to keep the MDS state consistent with N openers of the file then that may take more effort. However, I also just thought of a partial solution to the MDS state issue - if the original client doing name_to_handle() also gets the MDS open lock, then it can somewhat act as "proxy" for the remaining clients that are opening via handle. The MDS will know that the client with the MDS open lock may be doing other opens, and if the handle also contains the layout as Paul proposed, then it seems possible to get at least a reasonable representation of the file on each client w/o having an additional MDS RPC from each one. Those clients may still have issues if contacting the MDS for that file, but maybe not. Actually implementing this is left as an exercise for the reader... Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From Nicolas.Williams at oracle.com Wed Oct 20 17:29:57 2010 From: Nicolas.Williams at oracle.com (Nicolas Williams) Date: Wed, 20 Oct 2010 12:29:57 -0500 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF2483.7030805@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> Message-ID: <20101020172956.GV1635@oracle.com> On Wed, Oct 20, 2010 at 09:18:59PM +0400, bzzz.tomas at gmail.com wrote: > On 10/20/10 9:11 PM, Paul Nowoczynski wrote: > > I could be wrong but my guess is that the network congestion caused by > > this communication pattern is a more serious problem. The mds should be > > able to easily service lookup rpc's since only the first few necessitate > > a read I/O from the disk. > > but then the network should be able to deal with storm of > * <# clients> to read/write data? > > or it's a specific switch being the bottleneck to specific node? > > because if it isn't network, but MDS being a real bottleneck, > then proxy might be a solution like Eric said above. not sure > is this important in your case, but this would allow to use > existing apps. MDSes are typically CPU bound, so that's likely the issue. The problem though is that the MDS does need to track open file state for SOM and for dealing with unlinks. The semantics of open-by-handle might be such that unlinks of files opened by handle can cause the file to disappear and syscalls on FDs opened by handle could then return EBADF or EIO or some new error code. But open-by-handle semantics don't allow for that, then the MDS needs to track open file state, and it's hard to see how to avoid RPCs to the MDS to establish that state (the original client could tell the MDS about all the clients that will open-by-handle, but this seems unlikely to perform so much better than N smaller RPCs as to justify it, and the open-by-handle API suddenly gets much more complex). Nico -- From andreas.dilger at oracle.com Wed Oct 20 17:30:24 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 11:30:24 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <20101020171348.GU1635@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <90E83093-2655-4C70-ACEA-E75D7E8C5511@oracle.com> <4CBF1D00.9080402@psc.edu> <269E1A0D-2117-4ADC-BCDE-67A60EA9B974@oracle.com> <20101020171348.GU1635@oracle.com> Message-ID: <94C30482-7C88-4E67-89AE-CB9B43DF57FB@oracle.com> On 2010-10-20, at 11:13, Nicolas Williams wrote: > Well, if we're going to add openg() (or whatever its name), we might as > well add variants of stat() that don't require getting the size when the > app doesn't need it That is "stat_lite" (or various different names), and was also under discussion for adding to the Linux kernel, until it turned from being a sensible API to a Linux-designed-by-committee API from hell (IMHO, of course) and has stopped dead in its tracks. > Another possibility is to say that a handle is only good as long as the > original file descriptor remains open (recovery issues here), and that > client can tell the MDS that it will be sharing its handle with other > clients. That is partly what the MDS open lock does. It was intended for NFS servers to allow them to open and close a file locally for its clients w/o MDS RPCs. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From bzzz.tomas at gmail.com Wed Oct 20 17:40:11 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 21:40:11 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <20101020172956.GV1635@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> <20101020172956.GV1635@oracle.com> Message-ID: <4CBF297B.9070103@gmail.com> On 10/20/10 9:29 PM, Nicolas Williams wrote: > MDSes are typically CPU bound, so that's likely the issue. The problem > though is that the MDS does need to track open file state for SOM and > for dealing with unlinks. The semantics of open-by-handle might be such > that unlinks of files opened by handle can cause the file to disappear > and syscalls on FDs opened by handle could then return EBADF or EIO or > some new error code. But open-by-handle semantics don't allow for that, > then the MDS needs to track open file state, and it's hard to see how to > avoid RPCs to the MDS to establish that state (the original client could > tell the MDS about all the clients that will open-by-handle, but this > seems unlikely to perform so much better than N smaller RPCs as to > justify it, and the open-by-handle API suddenly gets much more complex). I guess for this purpose they may just disable SOM and do few steps away from POSIX. probably inter-client data consistency isn't that important any more ;) then get rid of MDS and namespace completely using some sort of FID. thanks, z From andreas.dilger at oracle.com Wed Oct 20 18:01:00 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 20 Oct 2010 12:01:00 -0600 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <4CBF297B.9070103@gmail.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> <20101020172956.GV1635@oracle.com> <4CBF297B.9070103@gmail.com> Message-ID: <51FFCB06-9335-42E4-8EDD-E2CA128F3CB7@oracle.com> On 2010-10-20, at 11:40, bzzz.tomas at gmail.com wrote: > On 10/20/10 9:29 PM, Nicolas Williams wrote: >> MDSes are typically CPU bound, so that's likely the issue. The problem >> though is that the MDS does need to track open file state for SOM and >> for dealing with unlinks. The semantics of open-by-handle might be such >> that unlinks of files opened by handle can cause the file to disappear >> and syscalls on FDs opened by handle could then return EBADF or EIO or >> some new error code. But open-by-handle semantics don't allow for that, >> then the MDS needs to track open file state, and it's hard to see how to >> avoid RPCs to the MDS to establish that state (the original client could >> tell the MDS about all the clients that will open-by-handle, but this >> seems unlikely to perform so much better than N smaller RPCs as to >> justify it, and the open-by-handle API suddenly gets much more complex). > > I guess for this purpose they may just disable SOM and do few steps away > from POSIX. probably inter-client data consistency isn't that important > any more ;) then get rid of MDS and namespace completely using some sort > of FID. I don't think that most customers want to drop POSIX and namespaces completely, because of the huge numbers of tools/apps that depend on this, but rather to have an API that can improve the performance of select applications that have a need for it. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From bzzz.tomas at gmail.com Wed Oct 20 18:09:51 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 20 Oct 2010 22:09:51 +0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: <51FFCB06-9335-42E4-8EDD-E2CA128F3CB7@oracle.com> References: <4CBEA415.80307@gmail.com> <9C26CBA7-8DBD-4875-8E14-FB663B749096@oracle.com> <4CBEA8A9.9080802@gmail.com> <00d001cb705a$fd64cb80$f82e6280$@com> <4CBF01DA.3090505@psc.edu> <4CBF094A.9020302@gmail.com> <4CBF1C42.1090109@psc.edu> <4CBF1D82.60508@gmail.com> <4CBF22DE.9080204@psc.edu> <4CBF2483.7030805@gmail.com> <20101020172956.GV1635@oracle.com> <4CBF297B.9070103@gmail.com> <51FFCB06-9335-42E4-8EDD-E2CA128F3CB7@oracle.com> Message-ID: <4CBF306F.7040308@gmail.com> On 10/20/10 10:01 PM, Andreas Dilger wrote: >> I guess for this purpose they may just disable SOM and do few steps away >> from POSIX. probably inter-client data consistency isn't that important >> any more ;) then get rid of MDS and namespace completely using some sort >> of FID. > > I don't think that most customers want to drop POSIX and namespaces completely, because of the huge numbers of tools/apps that depend on this, but rather to have an API that can improve the performance of select applications that have a need for it. oh, sorry for this sort of joke.. what I mean is that probably we could provide with another user-visible API which allows to bypass regular namespace, for example. thanks, z From vilobh.meshram at gmail.com Fri Oct 22 02:33:44 2010 From: vilobh.meshram at gmail.com (Vilobh Meshram) Date: Thu, 21 Oct 2010 22:33:44 -0400 Subject: [Lustre-devel] Queries regarding LDLM_ENQUEUE In-Reply-To: References: Message-ID: Thanks Andreas for the e-mail. I am trying to modify the LDLM_ENQUEUE rpc to get the the reply in the form of some buffer (say string "Hello World") filled in from MDS.I have explained the use case in my last e-mail.Please refer my e-mail sent on 10/19. I have attached the diff files. I am getting a kernel panic at the MDS end when I try to make the attached changes.Can someone please suggest me where I might be missing ? Thanks, Vilobh *Graduate Research Associate Department of Computer Science The Ohio State University Columbus Ohio* On Wed, Oct 20, 2010 at 3:55 AM, Andreas Dilger wrote: > On 2010-10-19, at 20:04, Vilobh Meshram wrote: > > We are trying to do following things.Please let me know if things are not > clear :- > > > > Say we have 2 client C1 and C2 and a MDS .Say C1 and C2 share a file. > > 1) When a client C1 performs a open/create kind of request to the MDS we > want to follow the normal path which Lustre performs. > > 2) Now say C2 tries to open the same file which was opened by C1. > > 3) At the MDS end we maintain some data structure to scan and see if the > file was already opened by some Client(in this case C1 has opened this > file). > > 4) If MDS finds that some client(C1 here) has already opened the file > then it send the new client(C2 here) with some information about the client > which has initially opened the file. > > While I understand the basic concept, I don't really see how your proposal > will actually improve performance. If C2 already has to contact the MDS and > get a reply from it, then wouldn't it be about the same to simply perform > the open as is done today? The number of MDS RPCs is the same, and in fact > this would avoid further message overhead between C1 and C2. > > > 5) Once C2 gets the information its upto C2 to take further actions. > > 6) By this process we can save the time spent in the locking mechanism > for C2.Basically we aim to by-pass the locking scheme of Lustre for the > files already opened by some client by maintaining some kind of data > structure. > > > > Please let us know your thoughts on the above approach.Is this a feasible > design moving ahead can we see any complications ? > > There is a separate proposal that has been underway in the Linux community > for some time, to allow a user process to get a file handle (i.e. binary > blob returned from a new name_to_handle() syscall) from the kernel for a > given pathname, and then later use that file handle in another process to > open a file descriptor without re-traversing the path. > > I've been thinking this would be very useful for Lustre (and MPI in > general), and have tried to steer the Linux development in a direction that > would allow this to happen. Is this in line with what you are > investigating? > > While this wouldn't eliminate the actual MDS open RPC (i.e. the > LDLM_ENQUEUE you have been discussing), it could avoid the path traversal > from each client, possibly saving {path_elements * num_clients} additional > RPCs, > > > So considering the problem statement I need a way for C2 to extract the > information from the data structure maintained at MDS.In order to do that , > C2 will send a request with intent = create|open which will be a > LDLM_ENQUEUE RPC.I need to modify this RPC such that :- > > 1) I can enclose some additional buffer whose size is known to me . > > 2) When we pack the reply at the MDS side we should be able to include > this buffer in the reply message . > > 3) At the client side we should be able to extract the information from > the reply message about the buffer. > > > > As of now , I need help in above three steps. > > > > Thanks, > > Vilobh > > Graduate Research Associate > > Department of Computer Science > > The Ohio State University Columbus Ohio > > > > > > On Tue, Oct 19, 2010 at 6:53 PM, Andreas Dilger < > andreas.dilger at oracle.com> wrote: > > On 2010-10-19, at 14:28, Vilobh Meshram wrote: > > > From my exploration it seems like for create/open kind of request > LDLM_ENQUEUE is the RPC through which the client talks to MDS.Please confirm > on this. > > > > > > Since I could figure out that LDLM_ENQUEUE is the only RPC to interface > with MDS I am planning to send the LDLM_ENQUEUE RPC with some additonal > buffer from the client to the MDS so that based on some specific condition > the MDS can fill the information in the buffer sent from the client. > > > > This isn't correct. LDLM_ENQUEUE is used for enqueueing locks. It just > happens that when Lustre wants to create a new file it enqueues a lock on > the parent directory with the "intent" to create a new file. The MDS > currently always replies "you cannot have the lock for the directory, I > created the requested file for you". Similarly, when the client is getting > attributes on a file, it needs a lock on that file in order to cache the > attributes, and to save RPCs the attributes are returned with the lock. > > > > > I have made some modifications to the code for the LDLM_ENQUEUE RPC but > I am getting kernel panics.Can someone please help me and suggest me what is > a good way to tackle this problem.I am using Lustre 1.8.1.1 and I cannot > upgrade to Lustre 2.0. > > > > It would REALLY be a lot easier to have this discussion with you if you > actually told us what it is you are working on. Not only could we focus on > the higher-level issue that you are trying to solve (instead of possibly > wasting a lot of time focussing in a small issue that may in fact be > completely irrelevant), but with many ideas related to Lustre it has > probably already been discussed at length by the Lustre developers sometime > over the past 8 years that we've been working on it. I suspect that the > readership of this list could probably give you a lot of assistance with > whatever you are working on, if you will only tell us what it actually is > you are trying to do. > > > > > On Mon, Oct 18, 2010 at 7:33 PM, Vilobh Meshram < > vilobh.meshram at gmail.com> wrote: > > >> Out of the many RPC's used in Lustre seems like LDLM_ENQUEUE is the > most frequently used RPC to communicate between the client and the MDS.I > have few queries regarding the same :- > > >> > > >> 1) Is LDLM_ENQUEUE the only interface(RPC here) for CREATE/OPEN kind > of request ; through which the client can interact with the MDS ? > > >> > > >> I tried couple of experiments and found out that LDLM_ENQUEUE comes > into picture while mounting the FS as well as when we do a lookup,create or > open a file.I was expecting the MDS_REINT RPC to get invoked in case of a > CREATE/OPEN request via mdc_create() but it seems like Lustre invokes > LDLM_ENQEUE even for CREATE/OPEN( by packing the intent related data). > > >> Please correct me if I am wrong. > > >> > > >> 2) In which cases (which system calls) does the MDS_REINT RPC will get > invoked ? > > > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Lustre Technical Lead > > Oracle Corporation Canada Inc. > > > > > > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- *** ./lustre/ldlm/ldlm_lockd.c 2010-10-21 17:49:05.000000000 -0400 --- ../fresh/lustre/ldlm/ldlm_lockd.c 2010-10-15 15:37:02.000000000 -0400 *************** int ldlm_handle_enqueue(struct ptlrpc_re *** 997,1017 **** struct obd_device *obddev = req->rq_export->exp_obd; struct ldlm_reply *dlm_rep; struct ldlm_request *dlm_req; ! __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) }; int rc = 0; __u32 flags; ldlm_error_t err = ELDLM_OK; struct ldlm_lock *lock = NULL; void *cookie = NULL; - int i; - char *str = "Hello World Sun"; - char *str_target; ENTRY; LDLM_DEBUG_NOLOCK("server-side enqueue handler START"); ! printk("\n Inside function %s server-side enqueue handler START",__func__); ! for(i=0;i<3;i++) printk("\n Inside function %s size[%d]:%d",__func__,i,size[i]); dlm_req = lustre_swab_reqbuf(req, DLM_LOCKREQ_OFF, sizeof(*dlm_req), lustre_swab_ldlm_request); if (dlm_req == NULL) { --- 997,1013 ---- struct obd_device *obddev = req->rq_export->exp_obd; struct ldlm_reply *dlm_rep; struct ldlm_request *dlm_req; ! __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREPLY_OFF] = sizeof(*dlm_rep) }; int rc = 0; __u32 flags; ldlm_error_t err = ELDLM_OK; struct ldlm_lock *lock = NULL; void *cookie = NULL; ENTRY; LDLM_DEBUG_NOLOCK("server-side enqueue handler START"); ! dlm_req = lustre_swab_reqbuf(req, DLM_LOCKREQ_OFF, sizeof(*dlm_req), lustre_swab_ldlm_request); if (dlm_req == NULL) { *************** existing_lock: *** 1126,1148 **** int buffers = 2; lock_res_and_lock(lock); - printk("\n Exsisting lock lock->l_resource->lr_lvb_len:%u",lock->l_resource->lr_lvb_len); if (lock->l_resource->lr_lvb_len) { - printk("\n Inside function %s , inside condition lock->l_resource->lr_lvb_len so buffers=3",__func__); size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; buffers = 3; } - //size[DLM_REPLY_REC_OFF] = 16; - //buffer = buffer + 1; - if(lock->l_resource->lr_lvb_len == 0) - { - buffers++; - size[DLM_REPLY_REC_OFF] = 0; - } - buffers++; - size[DLM_REPLY_REC_OFF+1] = 16; unlock_res_and_lock(lock); ! printk("\n Inside function %s , outside condition lock->l_resource->lr_lvb_len so buffers=2",__func__); if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) GOTO(out, rc = -ENOMEM); --- 1122,1133 ---- int buffers = 2; lock_res_and_lock(lock); if (lock->l_resource->lr_lvb_len) { size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; buffers = 3; } unlock_res_and_lock(lock); ! if (OBD_FAIL_CHECK_ONCE(OBD_FAIL_LDLM_ENQUEUE_EXTENT_ERR)) GOTO(out, rc = -ENOMEM); *************** existing_lock: *** 1156,1164 **** if (dlm_req->lock_desc.l_resource.lr_type == LDLM_EXTENT) lock->l_req_extent = lock->l_policy_data.l_extent; - printk("%s: \twill do lock-enq...\n", __func__); err = ldlm_lock_enqueue(obddev->obd_namespace, &lock, cookie, (int *)&flags); - printk("%s: \tafter lock-enq...\n", __func__); if (err) GOTO(out, err); --- 1141,1147 ---- *************** existing_lock: *** 1178,1185 **** dlm_rep->lock_flags |= dlm_req->lock_flags & LDLM_INHERIT_FLAGS; lock->l_flags |= dlm_req->lock_flags & LDLM_INHERIT_FLAGS; - str_target = lustre_msg_buf(req->rq_repmsg, DLM_REPLY_REC_OFF+1,16); - memcpy(str_target,str,16); /* Don't move a pending lock onto the export if it has already * been evicted. Cancel it now instead. (bug 5683) */ if (req->rq_export->exp_failed || --- 1161,1166 ---- *************** existing_lock: *** 1232,1238 **** EXIT; out: - printk("\n [VM] Inside function %s got a hit at out",__func__); req->rq_status = rc ?: err; /* return either error - bug 11190 */ if (!req->rq_packed_final) { err = lustre_pack_reply(req, 1, NULL, NULL); --- 1213,1218 ---- *************** existing_lock: *** 1248,1257 **** if (rc == 0) { lock_res_and_lock(lock); - printk("\n Inside function %s , inside if condition rc=0 the place where we do a memcpy for offset = DLM_REPLY_REC_OFF",__func__); size[DLM_REPLY_REC_OFF] = lock->l_resource->lr_lvb_len; - printk("\n Inside function %s , size[DLM_REPLY_REC_OFF] : %u , lock->l_resource->lr_lvb_len :%u",__func__,size[DLM_REPLY_REC_OFF],lock->l_resource->lr_lvb_len); - size[DLM_REPLY_REC_OFF+1]= 16; if (size[DLM_REPLY_REC_OFF] > 0) { void *lvb = lustre_msg_buf(req->rq_repmsg, DLM_REPLY_REC_OFF, --- 1228,1234 ---- *************** existing_lock: *** 1264,1270 **** } unlock_res_and_lock(lock); } else { - printk("\n Inside function %s , inside else condition rc=0 the place where we do a memcpy for offset = DLM_REPLY_REC_OFF",__func__); lock_res_and_lock(lock); ldlm_resource_unlink_lock(lock); ldlm_lock_destroy_nolock(lock); --- 1241,1246 ---- -------------- next part -------------- *** ./lustre/ldlm/ldlm_request.c 2010-10-21 22:26:28.000000000 -0400 --- ../fresh/lustre/ldlm/ldlm_request.c 2010-10-15 15:37:02.000000000 -0400 *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 389,395 **** int cleanup_phase = 1; ENTRY; - printk("\n Inside function %s",__func__); lock = ldlm_handle2lock(lockh); /* ldlm_cli_enqueue is holding a reference on this lock. */ if (!lock) { --- 389,394 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 401,407 **** LASSERT(!is_replay); LDLM_DEBUG(lock, "client-side enqueue END (%s)", rc == ELDLM_LOCK_ABORTED ? "ABORTED" : "FAILED"); - printk("\n Inside %s if client lock aborted or failed",__func__); if (rc == ELDLM_LOCK_ABORTED) { /* Before we return, swab the reply */ reply = lustre_swab_repbuf(req, DLM_LOCKREPLY_OFF, --- 400,405 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 433,440 **** GOTO(cleanup, rc = -EPROTO); } - printk("\n Inside function %s we have received a reply",__func__); - /* lock enqueued on the server */ cleanup_phase = 0; --- 431,436 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 463,469 **** * again. */ if ((*flags) & LDLM_FL_LOCK_CHANGED) { int newmode = reply->lock_desc.l_req_mode; - printk("\n Inside function %s in condition (*flags) & LDLM_FL_LOCK_CHANGED)",__func__); LASSERT(!is_replay); if (newmode && newmode != lock->l_req_mode) { LDLM_DEBUG(lock, "server returned different mode %s", --- 459,464 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 504,510 **** * because it cannot handle asynchronous ASTs robustly (see * bug 7311). */ (LIBLUSTRE_CLIENT && type == LDLM_EXTENT)) { - printk("\n Inside function %s in condition ((*flags) & LDLM_FL_AST_SENT ||(LIBLUSTRE_CLIENT && type == LDLM_EXTENT))",__func__); lock_res_and_lock(lock); lock->l_flags |= LDLM_FL_CBPENDING | LDLM_FL_BL_AST; unlock_res_and_lock(lock); --- 499,504 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 515,521 **** * clobber the LVB with an older one. */ if (lvb_len && (lock->l_req_mode != lock->l_granted_mode)) { void *tmplvb; - printk("\n Inside function %s in condition lvb_len && (lock->l_req_mode != lock->l_granted_mode) , lvb_len:%d",__func__,lvb_len); tmplvb = lustre_swab_repbuf(req, DLM_REPLY_REC_OFF, lvb_len, lvb_swabber); if (tmplvb == NULL) --- 509,514 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 524,530 **** } if (!is_replay) { - printk("\n Inside function %s in condition !is_replay",__func__); rc = ldlm_lock_enqueue(ns, &lock, NULL, flags); if (lock->l_completion_ast != NULL) { int err = lock->l_completion_ast(lock, *flags, NULL); --- 517,522 ---- *************** int ldlm_cli_enqueue_fini(struct obd_exp *** 536,542 **** } if (lvb_len && lvb != NULL) { - printk("\n Inside function %s in condition lvb_len && lvb != NULL",__func__); /* Copy the LVB here, and not earlier, because the completion * AST (if any) can override what we got in the reply */ memcpy(lvb, lock->l_lvb_data, lvb_len); --- 528,533 ---- *************** static inline int ldlm_req_handles_avail *** 560,578 **** __u32 *size, int bufcount, int off) { int avail = min_t(int, LDLM_MAXREQSIZE, CFS_PAGE_SIZE - 512); ! printk("\n Inside function %s",__func__); ! printk("\n avail--before = %d",avail); avail -= lustre_msg_size(class_exp2cliimp(exp)->imp_msg_magic, bufcount, size); ! printk("\n avail--after = %d",avail); ! if (likely(avail >= 0)){ avail /= (int)sizeof(struct lustre_handle); - printk("\n avail--likely = %d",avail); - } else avail = 0; avail += LDLM_LOCKREQ_HANDLES - off; ! printk("\n avail--lats = %d",avail); return avail; } --- 551,565 ---- __u32 *size, int bufcount, int off) { int avail = min_t(int, LDLM_MAXREQSIZE, CFS_PAGE_SIZE - 512); ! avail -= lustre_msg_size(class_exp2cliimp(exp)->imp_msg_magic, bufcount, size); ! if (likely(avail >= 0)) avail /= (int)sizeof(struct lustre_handle); else avail = 0; avail += LDLM_LOCKREQ_HANDLES - off; ! return avail; } *************** struct ptlrpc_request *ldlm_prep_elc_req *** 597,622 **** CFS_LIST_HEAD(head); ENTRY; - printk("\n Inside function %s, opc=%d",__func__, opc); if (cancels == NULL) cancels = &head; if (exp_connect_cancelset(exp)) { /* Estimate the amount of free space in the request. */ - printk("\n Inside exp_connect_cancelset(exp) in func %s",__func__); LASSERT(bufoff < bufcount); avail = ldlm_req_handles_avail(exp, size, bufcount, canceloff); - printk("\n In function %s avail = %d",__func__,avail); flags = ns_connect_lru_resize(ns) ? LDLM_CANCEL_LRUR : LDLM_CANCEL_AGED; - printk("\n In function %s ns_connect_lru_resize(ns) :%d",__func__,ns_connect_lru_resize(ns)); to_free = !ns_connect_lru_resize(ns) && opc == LDLM_ENQUEUE ? 1 : 0; /* Cancel lru locks here _only_ if the server supports * EARLY_CANCEL. Otherwise we have to send extra CANCEL * rpc, what will make us slower. */ - printk("\n In function %s count = %d",__func__,count); if (avail > count) count += ldlm_cancel_lru_local(ns, cancels, to_free, avail - count, 0, flags); --- 584,604 ---- *************** struct ptlrpc_request *ldlm_prep_elc_req *** 624,632 **** pack = count; else pack = avail; - printk("\n In function %s pack = %d",__func__,pack); size[bufoff] = ldlm_request_bufsize(pack, opc); - printk("\n In function %s , bufoff : %d , size[bufoff]= %u",__func__,bufoff,size[bufoff]); } req = ptlrpc_prep_req(class_exp2cliimp(exp), version, --- 606,612 ---- *************** struct ptlrpc_request *ldlm_prep_enqueue *** 657,663 **** struct list_head *cancels, int count) { - printk("\n Inside function %s \n",__func__); return ldlm_prep_elc_req(exp, LUSTRE_DLM_VERSION, LDLM_ENQUEUE, bufcount, size, DLM_LOCKREQ_OFF, LDLM_ENQUEUE_CANCEL_OFF, cancels, count); --- 637,642 ---- *************** int ldlm_cli_enqueue(struct obd_export * *** 679,697 **** struct ldlm_lock *lock; struct ldlm_request *body; struct ldlm_reply *reply; ! __u32 size[4] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(*body), [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : ! sizeof(struct ost_lvb), ! [DLM_REPLY_REC_OFF+1] = 16}; int is_replay = *flags & LDLM_FL_REPLAY; int req_passed_in = 1, rc, err; struct ptlrpc_request *req; - int i; ENTRY; - printk("\n Inside function %s \n",__func__); - for(i=0;i<4;i++) printk("\n size[%d] : %d",i,size[i]); LASSERT(exp != NULL); /* If we're replaying this lock, just check some invariants. --- 658,672 ---- struct ldlm_lock *lock; struct ldlm_request *body; struct ldlm_reply *reply; ! __u32 size[3] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(*body), [DLM_REPLY_REC_OFF] = lvb_len ? lvb_len : ! sizeof(struct ost_lvb) }; int is_replay = *flags & LDLM_FL_REPLAY; int req_passed_in = 1, rc, err; struct ptlrpc_request *req; ENTRY; LASSERT(exp != NULL); /* If we're replaying this lock, just check some invariants. *************** int ldlm_cli_enqueue(struct obd_export * *** 700,706 **** lock = ldlm_handle2lock(lockh); LASSERT(lock != NULL); LDLM_DEBUG(lock, "client-side enqueue START"); - printk("\n Client-side enqueue START in %s",__func__); LASSERT(exp == lock->l_conn_export); } else { lock = ldlm_lock_create(ns, res_id, einfo->ei_type, --- 675,680 ---- *************** int ldlm_cli_enqueue(struct obd_export * *** 736,742 **** /* lock not sent to server yet */ if (reqp == NULL || *reqp == NULL) { ! req = ldlm_prep_enqueue_req(exp,3, size, NULL, 0); if (req == NULL) { failed_lock_cleanup(ns, lock, lockh, einfo->ei_mode); LDLM_LOCK_PUT(lock); --- 710,716 ---- /* lock not sent to server yet */ if (reqp == NULL || *reqp == NULL) { ! req = ldlm_prep_enqueue_req(exp, 2, size, NULL, 0); if (req == NULL) { failed_lock_cleanup(ns, lock, lockh, einfo->ei_mode); LDLM_LOCK_PUT(lock); *************** int ldlm_cli_enqueue(struct obd_export * *** 746,752 **** if (reqp) *reqp = req; } else { - printk("\n [VM]got a hit at case where reqp is not NULL in %s",__func__); req = *reqp; LASSERTF(lustre_msg_buflen(req->rq_reqmsg, DLM_LOCKREQ_OFF) >= sizeof(*body), "buflen[%d] = %d, not %d\n", --- 720,725 ---- *************** int ldlm_cli_enqueue(struct obd_export * *** 768,774 **** /* Continue as normal. */ if (!req_passed_in) { size[DLM_LOCKREPLY_OFF] = sizeof(*reply); ! ptlrpc_req_set_repsize(req, 4, size); } /* --- 741,747 ---- /* Continue as normal. */ if (!req_passed_in) { size[DLM_LOCKREPLY_OFF] = sizeof(*reply); ! ptlrpc_req_set_repsize(req, 3, size); } /* *************** int ldlm_cli_enqueue(struct obd_export * *** 784,793 **** RETURN(0); } - printk("\n in --func-- %s SENDING REQUEST",__func__); LDLM_DEBUG(lock, "sending request"); rc = ptlrpc_queue_wait(req); - printk("\n in --func-- %s REQUEST SENT after ptlrpc_queue_wait",__func__); err = ldlm_cli_enqueue_fini(exp, req, einfo->ei_type, policy ? 1 : 0, einfo->ei_mode, flags, lvb, lvb_len, lvb_swabber, lockh, rc); --- 757,764 ---- -------------- next part -------------- *** ./lustre/mdc/mdc_locks.c 2010-10-20 20:58:51.000000000 -0400 --- ../fresh/lustre/mdc/mdc_locks.c 2010-10-15 15:37:15.000000000 -0400 *************** static struct ptlrpc_request *mdc_intent *** 252,264 **** int repbufcount = 5; int mode; int rc; - int i; ENTRY; - printk("\n Inside function %s",__func__); - for(i=0;i<6;i++) printk("\n size[%d] : %d",i,size[i]); - for(i=0;i<5;i++) printk("\n repsize[%d] : %d",i,repsize[i]); - it->it_create_mode = (it->it_create_mode & ~S_IFMT) | S_IFREG; if (mdc_exp_is_2_0_server(exp)) { size[DLM_INTENT_REC_OFF] = sizeof(struct mdt_rec_create); --- 252,259 ---- *************** static struct ptlrpc_request *mdc_intent *** 381,405 **** struct ptlrpc_request *req; struct ldlm_intent *lit; struct obd_device *obddev = class_exp2obd(exp); ! __u32 size[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request), [DLM_INTENT_IT_OFF] = sizeof(*lit), [DLM_INTENT_REC_OFF] = sizeof(struct mdt_body), [DLM_INTENT_REC_OFF+1]= data->namelen + 1, ! [DLM_INTENT_REC_OFF+2]= 0, ! [DLM_INTENT_REC_OFF+3]= 16 }; ! __u32 repsize[] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREPLY_OFF] = sizeof(struct ldlm_reply), [DLM_REPLY_REC_OFF] = sizeof(struct mdt_body), [DLM_REPLY_REC_OFF+1] = obddev->u.cli. cl_max_mds_easize, [DLM_REPLY_REC_OFF+2] = LUSTRE_POSIX_ACL_MAX_SIZE, ! [DLM_REPLY_REC_OFF+3] = 0, ! [DLM_REPLY_REC_OFF+4] = 16 }; obd_valid valid = OBD_MD_FLGETATTR | OBD_MD_FLEASIZE | OBD_MD_FLACL | OBD_MD_FLMODEASIZE | OBD_MD_FLDIREA; ! int bufcount = 6; ! int i=0; ENTRY; if (mdc_exp_is_2_0_server(exp)) { --- 376,397 ---- struct ptlrpc_request *req; struct ldlm_intent *lit; struct obd_device *obddev = class_exp2obd(exp); ! __u32 size[6] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREQ_OFF] = sizeof(struct ldlm_request), [DLM_INTENT_IT_OFF] = sizeof(*lit), [DLM_INTENT_REC_OFF] = sizeof(struct mdt_body), [DLM_INTENT_REC_OFF+1]= data->namelen + 1, ! [DLM_INTENT_REC_OFF+2]= 0 }; ! __u32 repsize[6] = { [MSG_PTLRPC_BODY_OFF] = sizeof(struct ptlrpc_body), [DLM_LOCKREPLY_OFF] = sizeof(struct ldlm_reply), [DLM_REPLY_REC_OFF] = sizeof(struct mdt_body), [DLM_REPLY_REC_OFF+1] = obddev->u.cli. cl_max_mds_easize, [DLM_REPLY_REC_OFF+2] = LUSTRE_POSIX_ACL_MAX_SIZE, ! [DLM_REPLY_REC_OFF+3] = 0 }; obd_valid valid = OBD_MD_FLGETATTR | OBD_MD_FLEASIZE | OBD_MD_FLACL | OBD_MD_FLMODEASIZE | OBD_MD_FLDIREA; ! int bufcount = 5; ENTRY; if (mdc_exp_is_2_0_server(exp)) { *************** static struct ptlrpc_request *mdc_intent *** 407,418 **** size[DLM_INTENT_REC_OFF+2] = data->namelen + 1; bufcount = 6; } - - printk("%s: prep-enq-req: bufcnt=%d\n", __func__, bufcount); - for(i=0; i= 0); /* Similarly, if we're going to replay this request, we don't want to * actually get a lock, just perform the intent. */ --- 441,446 ---- *************** static int mdc_finish_enqueue(struct obd *** 517,523 **** /* We know what to expect, so we do any byte flipping required here */ if (it->it_op & (IT_OPEN | IT_UNLINK | IT_LOOKUP | IT_GETATTR)) { struct mds_body *body; ! printk("\n Inside function %s inside condition IT_OPEN , IT_LOOKUP , IT_GETATTR",__func__); body = lustre_swab_repbuf(req, DLM_REPLY_REC_OFF, sizeof(*body), lustre_swab_mds_body); if (body == NULL) { --- 502,508 ---- /* We know what to expect, so we do any byte flipping required here */ if (it->it_op & (IT_OPEN | IT_UNLINK | IT_LOOKUP | IT_GETATTR)) { struct mds_body *body; ! body = lustre_swab_repbuf(req, DLM_REPLY_REC_OFF, sizeof(*body), lustre_swab_mds_body); if (body == NULL) { *************** int mdc_enqueue(struct obd_export *exp, *** 587,593 **** int rc; ENTRY; - printk("\n Inside function %s \n",__func__); fid_build_reg_res_name((void *)&data->fid1, &res_id); LASSERTF(einfo->ei_type == LDLM_IBITS,"lock type %d\n", einfo->ei_type); if (it->it_op & (IT_UNLINK | IT_GETATTR | IT_READDIR)) --- 572,577 ---- *************** int mdc_intent_getattr_async(struct obd_ *** 924,933 **** int flags = LDLM_FL_HAS_INTENT; ENTRY; - printk("%s: name: %.*s in inode "LPU64", intent: %s flags %#o\n",__func__, - op_data->namelen, op_data->name, op_data->fid1.id, - ldlm_it2str(it->it_op), it->it_flags); - CDEBUG(D_DLMTRACE,"name: %.*s in inode "LPU64", intent: %s flags %#o\n", op_data->namelen, op_data->name, op_data->fid1.id, ldlm_it2str(it->it_op), it->it_flags); --- 908,913 ---- From andreas.dilger at oracle.com Wed Oct 27 13:06:32 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Wed, 27 Oct 2010 21:06:32 +0800 Subject: [Lustre-devel] changelog for whole filesystem? Message-ID: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> I had an interesting idea today during a discussion on HSM. One of the issues with enabling HSM today is that a full-filesystem scan must be done initially to populate the policy engine database. I was thinking that it would be useful to have a virtual changelog that provides a feed from internally traversing the whole filesystem in an efficient manner. For bug 22741 we have implemented a virtual index in the OSD which will return all of the in-use ldiskfs inodes in inode-number order (which is the fastest way to read/stat all of the inodes). It probably wouldn't be too hard to hook this OSD callback into the changelog API, so that reading a special changelog file would return all of the files in the filesystem. It is possible to generate the full pathnames of these inodes via the "link" xattr, if that is needed. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From bzzz.tomas at gmail.com Wed Oct 27 13:17:31 2010 From: bzzz.tomas at gmail.com (bzzz.tomas at gmail.com) Date: Wed, 27 Oct 2010 17:17:31 +0400 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> Message-ID: <4CC8266B.8020903@gmail.com> On 10/27/10 5:06 PM, Andreas Dilger wrote: > I had an interesting idea today during a discussion on HSM. One of the issues with enabling HSM today is that a full-filesystem scan must be done initially to populate the policy engine database. > > I was thinking that it would be useful to have a virtual changelog that provides a feed from internally traversing the whole filesystem in an efficient manner. For bug 22741 we have implemented a virtual index in the OSD which will return all of the in-use ldiskfs inodes in inode-number order (which is the fastest way to read/stat all of the inodes). It probably wouldn't be too hard to hook this OSD callback into the changelog API, so that reading a special changelog file would return all of the files in the filesystem. It is possible to generate the full pathnames of these inodes via the "link" xattr, if that is needed. probably you meant to iterate over OI ? thanks, z From thomas.leibovici at cea.fr Wed Oct 27 15:28:09 2010 From: thomas.leibovici at cea.fr (LEIBOVICI Thomas) Date: Wed, 27 Oct 2010 17:28:09 +0200 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> Message-ID: <4CC84509.4050609@cea.fr> Would this special log have the same record structure as current changelogs, or a different structure with more information? Depending on how this iterator works, maybe we can avoid RPCs (for stat, fid2path, get_stripe, hsm_state_get...) if this info is available when the log record is generated. Anyhow, this feature sounds very interesting. We'll be glad to help, if you need taskforce for implementing such a feature. Thomas Andreas Dilger wrote: > I had an interesting idea today during a discussion on HSM. One of the issues with enabling HSM today is that a full-filesystem scan must be done initially to populate the policy engine database. > > I was thinking that it would be useful to have a virtual changelog that provides a feed from internally traversing the whole filesystem in an efficient manner. For bug 22741 we have implemented a virtual index in the OSD which will return all of the in-use ldiskfs inodes in inode-number order (which is the fastest way to read/stat all of the inodes). It probably wouldn't be too hard to hook this OSD callback into the changelog API, so that reading a special changelog file would return all of the files in the filesystem. It is possible to generate the full pathnames of these inodes via the "link" xattr, if that is needed. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > From andreas.dilger at oracle.com Thu Oct 28 09:04:02 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Thu, 28 Oct 2010 17:04:02 +0800 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <4CC8266B.8020903@gmail.com> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> <4CC8266B.8020903@gmail.com> Message-ID: <6E547A9E-287D-4358-9EDB-776B85619DCC@oracle.com> On 2010-10-27, at 21:17, bzzz.tomas at gmail.com wrote: > On 10/27/10 5:06 PM, Andreas Dilger wrote: >> >> I was thinking that it would be useful to have a virtual changelog that provides a feed from internally traversing the whole filesystem in an efficient manner. For bug 22741 we have implemented a virtual index in the OSD which will return all of the in-use ldiskfs inodes in inode-number order (which is the fastest way to read/stat all of the inodes). It probably wouldn't be too hard to hook this OSD callback into the changelog API, so that reading a special changelog file would return all of the files in the filesystem. It is possible to generate the full pathnames of these inodes via the "link" xattr, if that is needed. > > probably you meant to iterate over OI ? Yes, I was thinking to hook the OI iteration to the ChangeLog API so that it is easier to iterate over the Filesystem from userspace. Cheers, Andreas From andreas.dilger at oracle.com Thu Oct 28 09:15:34 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Thu, 28 Oct 2010 17:15:34 +0800 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <4CC84509.4050609@cea.fr> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> <4CC84509.4050609@cea.fr> Message-ID: <9759789F-8FD0-464F-9632-E31F0F45287C@oracle.com> On 2010-10-27, at 23:28, LEIBOVICI Thomas wrote: > Would this special log have the same record structure as current changelogs, or a different structure with more information? > Depending on how this iterator works, maybe we can avoid RPCs (for stat, fid2path, get_stripe, hsm_state_get...) if this info is available when the log record is generated. My thought was to use the same format for the changelog so that it would be easy to use the same API to use the "whole filesystem" traversal log and then transfer over to the standard "changes only" changelog. In fact, it might make sense to make this atomic so that this is a flag on a regular changelog open, and it will continue after the traversal is completed to the changelog for any changes that happened since the traversal started. > Anyhow, this feature sounds very interesting. We'll be glad to help, if you need taskforce for implementing such a feature. > > Thomas > > Andreas Dilger wrote: >> I had an interesting idea today during a discussion on HSM. One of the issues with enabling HSM today is that a full-filesystem scan must be done initially to populate the policy engine database. >> >> I was thinking that it would be useful to have a virtual changelog that provides a feed from internally traversing the whole filesystem in an efficient manner. For bug 22741 we have implemented a virtual index in the OSD which will return all of the in-use ldiskfs inodes in inode-number order (which is the fastest way to read/stat all of the inodes). It probably wouldn't be too hard to hook this OSD callback into the changelog API, so that reading a special changelog file would return all of the files in the filesystem. It is possible to generate the full pathnames of these inodes via the "link" xattr, if that is needed. >> >> Cheers, Andreas >> -- >> Andreas Dilger >> Lustre Technical Lead >> Oracle Corporation Canada Inc. >> >> > From thomas.leibovici at cea.fr Thu Oct 28 13:43:19 2010 From: thomas.leibovici at cea.fr (LEIBOVICI Thomas) Date: Thu, 28 Oct 2010 15:43:19 +0200 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <9759789F-8FD0-464F-9632-E31F0F45287C@oracle.com> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> <4CC84509.4050609@cea.fr> <9759789F-8FD0-464F-9632-E31F0F45287C@oracle.com> Message-ID: <4CC97DF7.8040206@cea.fr> Andreas Dilger wrote: > On 2010-10-27, at 23:28, LEIBOVICI Thomas wrote: > >> Would this special log have the same record structure as current changelogs, or a different structure with more information? >> Depending on how this iterator works, maybe we can avoid RPCs (for stat, fid2path, get_stripe, hsm_state_get...) if this info is available when the log record is generated. >> > > My thought was to use the same format for the changelog so that it would be easy to use the same API to use the "whole filesystem" traversal log and then transfer over to the standard "changes only" changelog. In fact, it might make sense to make this atomic so that this is a flag on a regular changelog open, and it will continue after the traversal is completed to the changelog for any changes that happened since the traversal started. > OK, I got it. So the idea is to have a switch in the policy engine that would be: - if it starts for the first time => open the changelog with a special flag to get all entries + changes in the meanwhile - else => open the changelog as usual "any changes that happened since the traversal started" A couple of comments about that: - With the current implementation, the ChangeLog transaction management starts after the "changelog_register" on MDT, then the log records start accumulating on MDT until they are read and acknowledged by the consummer. So, reporting only the "changes that happened since the traversal started" implies to voluntarily forget previous records that were waiting to be read. - if changes occur during the scan: do we skip/ignore records for entries that have not been listed yet? - If we want to make the "scan log" restartable from the last read entry, the client should be able to reopen the log by giving the last record id in argument and continue the scan and/or the standard log records where it stopped. So merging the 2 log streams (scan and standard changelog) may imply a common record id management. Distinguishing the two kind of logs depending on open flag makes it possible to manage log record index and scan record index separately, which would simplify the implementation: the record index for "scan log" will be something like the inode-number order, and the log consummer can use this index for restarting an aborted scan. Once the changelog consummer is registered on MDT, we are sure not to miss any change that occurs on the filesystem. So, for initializing the HSM policy engine DB, we can proceed the following way: 1) register a changelog consummer on MDT 2) open and process the "scan log" 3) open and process the standard changelog records that are accumlated since step 1) we are sure to know all entries in filesystem after those 3 steps. Policy engine can actually perform 3) at any time. The only contain is to have step 1) before step 2). Thomas. From jeremy.filizetti at gmail.com Fri Oct 29 03:07:46 2010 From: jeremy.filizetti at gmail.com (Jeremy Filizetti) Date: Thu, 28 Oct 2010 23:07:46 -0400 Subject: [Lustre-devel] releasing BKL in lustre_fill_super Message-ID: I've seen a lot of issues with mounting all of our OSTs on an OSS taking an excessive amount of time. Most of the individual OST mount time was related to bug 18456, but we still see mount times take minutes per OST with the relevant patches. At mount time the llog does a small write which ends up scanning nearly our entire 7+ TB OSTs to find the desired block and complete the write. To reduce startup time mounting multiple OSTs simultaneously would help, but during that process it looks like the code path is still holding the big kernel lock from the mount system call. During that time all other mount commands are in an uninterruptible sleep (D state). Based on the discussions from bug 23790 it doesn't appear that Lustre relies on the BKL so would it be reasonable to call unlock_kernel in lustre_fill_super or at least before lustre_start_mgc and lock it again before the return so multiple OSTs could be mounting at the same time? I think the same thing would apply to unmounting but I haven't looked at the code path there. Jeremy -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.dilger at oracle.com Fri Oct 29 15:26:15 2010 From: andreas.dilger at oracle.com (Andreas Dilger) Date: Fri, 29 Oct 2010 23:26:15 +0800 Subject: [Lustre-devel] layout lock bug with 118k In-Reply-To: <4CC82688.4050801@cea.fr> References: <4CC82688.4050801@cea.fr> Message-ID: <18180EC5-3EFB-4C92-8F6A-4169A1DBA2D7@oracle.com> On 2010-10-27, at 21:18, Jacques-Charles Lafoucriere wrote: > I have found a bug in layout lock (the bug was seen with test 118k, this is the last known). > > A simpler reproducer is to make an rm during a long file write. > > A lock timeout is trigged because during the writes the client hold the layout lock which is in the same lock as a lookup (muliple inode_bits in the same lock). So when the MDS try to get an LCK_EX on the object (before calling mdo_unlink), the lock is not freed because of the ref count. The client should only be holding a reference on the layout lock for 1MB chunks of IO. Between each IO the layout lock reference should be dropped, and if there was a blocking callback on the lock the client should also cancel the lock at that time. > A solution is the request a LCK_CR on the object before the mdo_unlink (the directory is still protected by a strong lock). Is it a good solution ? Do you have another one ? We discussed this issue recently, and the preferred solution is to release the layout lock as soon as the OST extent locks are referenced, since we don't actually require the layout lock once we hold the object extent lock(s). We discussed this before, and it is a bit tricky, because the ll_layout_lock_get() and ll_layout_lock_put() currently wrap the IO function. One proposal is to refcount the lsm structure under the layout lock, and then drop the last lsm reference in the LOV code after the object lock is held, and that would release the lsm lock. Cheers, Andreas -- Andreas Dilger Lustre Technical Lead Oracle Corporation Canada Inc. From eeb at whamcloud.com Fri Oct 29 16:50:02 2010 From: eeb at whamcloud.com (Eric Barton) Date: Fri, 29 Oct 2010 09:50:02 -0700 Subject: [Lustre-devel] changelog for whole filesystem? In-Reply-To: <4CC97DF7.8040206@cea.fr> References: <855B43E2-1D24-4095-ABE7-643A3ADAD67D@oracle.com> <4CC84509.4050609@cea.fr> <9759789F-8FD0-464F-9632-E31F0F45287C@oracle.com> <4CC97DF7.8040206@cea.fr> Message-ID: <022a01cb7789$55cc7a40$01656ec0$@com> Andreas, Thomas, I _do_ like the idea of opening the changelog to see changes either "from now" or "from empty". But I think the idea needs to worked out fully to support multiple changelog consumers - e.g. how to keep multiple placeholders in the object enumeration so that changes to objects yet to be enumerated for a particular consumer are not queued to that consumer. As ever, I'm concerned that what looks like "low hanging fruit" now later turns into technical debt later. Cheers, Eric > -----Original Message----- > From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf > Of LEIBOVICI Thomas > Sent: 28 October 2010 6:43 AM > To: Andreas Dilger > Cc: lustre-hsm-core-ext at Sun.COM; lustre-devel at lists.lustre.org List > Subject: Re: [Lustre-devel] changelog for whole filesystem? > > Andreas Dilger wrote: > > On 2010-10-27, at 23:28, LEIBOVICI Thomas wrote: > > > >> Would this special log have the same record structure as current changelogs, or a different > structure with more information? > >> Depending on how this iterator works, maybe we can avoid RPCs (for stat, fid2path, get_stripe, > hsm_state_get...) if this info is available when the log record is generated. > >> > > > > My thought was to use the same format for the changelog so that it would be easy to use the same API > to use the "whole filesystem" traversal log and then transfer over to the standard "changes only" > changelog. In fact, it might make sense to make this atomic so that this is a flag on a regular > changelog open, and it will continue after the traversal is completed to the changelog for any changes > that happened since the traversal started. > > > OK, I got it. So the idea is to have a switch in the policy engine that > would be: > - if it starts for the first time => open the changelog with a special > flag to get all entries + changes in the meanwhile > - else => open the changelog as usual > > "any changes that happened since the traversal started" > > A couple of comments about that: > - With the current implementation, the ChangeLog transaction management starts after the > "changelog_register" on MDT, > then the log records start accumulating on MDT until they are read and acknowledged by the consummer. > So, reporting only the "changes that happened since the traversal started" implies to voluntarily > forget previous records > that were waiting to be read. > - if changes occur during the scan: do we skip/ignore records for entries that have not been listed > yet? > - If we want to make the "scan log" restartable from the last read entry, the client should be able to > reopen the log > by giving the last record id in argument and continue the scan and/or the standard log records where > it stopped. > So merging the 2 log streams (scan and standard changelog) may imply a common record id management. > > Distinguishing the two kind of logs depending on open flag makes it possible > to manage log record index and scan record index separately, which would simplify the implementation: > the record index for "scan log" will be something like the inode-number order, > and the log consummer can use this index for restarting an aborted scan. > > Once the changelog consummer is registered on MDT, we are sure not to miss any change that occurs on > the filesystem. > So, for initializing the HSM policy engine DB, we can proceed the following way: > 1) register a changelog consummer on MDT > 2) open and process the "scan log" > 3) open and process the standard changelog records that are accumlated since step 1) > we are sure to know all entries in filesystem after those 3 steps. > Policy engine can actually perform 3) at any time. The only contain is to have step 1) before step 2). > > Thomas. > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Norman.Morse at OpenSFS.org Thu Oct 28 20:40:01 2010 From: Norman.Morse at OpenSFS.org (Norman Morse) Date: Thu, 28 Oct 2010 15:40:01 -0500 Subject: [Lustre-devel] OpenSFS Information Meeting In Parallel With SC10 Message-ID: *Inaugural OpenSFS Meeting At SC10 To Promote Open Parallel File System Advancement & Community Participation* HPC Industry Leaders To Host Session On The Lustre File System and Open Source Collaboration *Livermore, California, October 28, 2010 — * OpenSFS, a non-profit corporation focused on advancing the collaborative development of open-source parallel file system technologies, today announced its first annual SC Informational Meeting to be scheduled at this year’s SC10 International Conference for High Performance Computing, Networking, Storage and Analysis in New Orleans, LA. At this open session, OpenSFS members and prospective members are invited to participate in a forum designed to introduce this newly-formed and substantially-funded organization and its focus on the Lustre® open source parallel file system. At this first formal meeting of the OpenSFS organization, members of the Lustre community will engage in a structured series of discussions that will span several of current topics, including: · End-user perspective on the status of the Lustre support and development ecosystem · Organizational structure, objectives and development framework · Sustainable long-term file system evolution initiatives through coordinated investment and community development · Community engagement opportunities and consortium participation “The OpenSFS consortium has been designed from the ground up to provide a positive long-term benefit to Lustre users through the coordinated requirements gathering and investment from the worldwide user community.” said Norm Morse, CEO of OpenSFS. “The organization has been designed to very openly engage users of all sizes. With membership fees as low as $5,000 annually – stakeholders are provided a forum to influence the future of open-source file system technology while working within budgetary constraints.” Details of the meeting are as follows: What: 2010 Annual OpenSFS SC Informational Meeting When: Tuesday, Nov 16th – 2:30-4PM CST, 2010 Where: Ritz-Carlton, 921 Canal St. New Orleans, LA Broadmoor Room (1st floor) To Attend: Email event at opensfs.org with the number of attendees from your organization *About OpenSFS* Founded in 2010, OpenSFS is a non-profit California corporation focused on high-end, open-source file system technologies. The goals of the organization are to provide a forum for collaboration among entities deploying file systems on leading edge HPC systems, to communicate future requirements to open-source file system developers, and to support file system releases which have been designed to meet these goals. The group's initial focus is the Lustre parallel file system, which supports many of the requirements of leadership class HPC simulation environments, has a diverse development community and is open-source software. Founded by Cray Inc., Data Direct Networks (DDN) Inc., Lawrence Livermore National Laboratory (LLNL) and Oak Ridge National Laboratory (ORNL), OpenSFS draws from over $2 million of combined funding to ensure substantial and sustained community benefit through advancing the stability, usability and scalability of sophisticated open-source file system technology. More information can be found online at www.opensfs.org. All trademarks are the property of their respective owners. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacques-charles.lafoucriere at cea.fr Sat Oct 30 10:48:14 2010 From: jacques-charles.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Sat, 30 Oct 2010 12:48:14 +0200 Subject: [Lustre-devel] layout lock bug with 118k In-Reply-To: <18180EC5-3EFB-4C92-8F6A-4169A1DBA2D7@oracle.com> References: <4CC82688.4050801@cea.fr> <18180EC5-3EFB-4C92-8F6A-4169A1DBA2D7@oracle.com> Message-ID: <4CCBF7EE.2070904@cea.fr> On 10/29/2010 05:26 PM, Andreas Dilger wrote: > On 2010-10-27, at 21:18, Jacques-Charles Lafoucriere wrote: > >> I have found a bug in layout lock (the bug was seen with test 118k, this is the last known). >> >> A simpler reproducer is to make an rm during a long file write. >> >> A lock timeout is trigged because during the writes the client hold the layout lock which is in the same lock as a lookup (muliple inode_bits in the same lock). So when the MDS try to get an LCK_EX on the object (before calling mdo_unlink), the lock is not freed because of the ref count. >> > The client should only be holding a reference on the layout lock for 1MB chunks of IO. Between each IO the layout lock reference should be dropped, and if there was a blocking callback on the lock the client should also cancel the lock at that time. > > The client hold the layout lock only around the IO. So between I/O's, the lock should be canceled. The issue comes from that the same lock is also referenced because of the other inodes bits. >> A solution is the request a LCK_CR on the object before the mdo_unlink (the directory is still protected by a strong lock). Is it a good solution ? Do you have another one ? >> > We discussed this issue recently, and the preferred solution is to release the layout lock as soon as the OST extent locks are referenced, since we don't actually require the layout lock once we hold the object extent lock(s). > > We discussed this before, and it is a bit tricky, because the ll_layout_lock_get() and ll_layout_lock_put() currently wrap the IO function. One proposal is to refcount the lsm structure under the layout lock, and then drop the last lsm reference in the LOV code after the object lock is held, and that would release the lsm lock. > I will see how to do this > Cheers, Andreas > -- > Andreas Dilger > Lustre Technical Lead > Oracle Corporation Canada Inc. > > >