From dinatale2 at llnl.gov Thu Aug 13 22:43:14 2015 From: dinatale2 at llnl.gov (Di Natale, Giuseppe) Date: Thu, 13 Aug 2015 22:43:14 +0000 Subject: [lustre-devel] lctl conf_param vs set_param -P Message-ID: <974262C966B9F640899A48EB96313E9A2B3C90@PRDEXMBX-08.the-lab.llnl.gov> Greetings, In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior? Thanks, Giuseppe Di Natale -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: set_param_log Type: application/octet-stream Size: 3081 bytes Desc: set_param_log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: conf_param_log Type: application/octet-stream Size: 4383 bytes Desc: conf_param_log URL: From morrone2 at llnl.gov Fri Aug 14 20:41:03 2015 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Fri, 14 Aug 2015 13:41:03 -0700 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: <974262C966B9F640899A48EB96313E9A2B3C90@PRDEXMBX-08.the-lab.llnl.gov> References: <974262C966B9F640899A48EB96313E9A2B3C90@PRDEXMBX-08.the-lab.llnl.gov> Message-ID: <55CE525F.3020505@llnl.gov> I too am confused. And a bit dismayed that there is so little in the way of code comments to explain the intent. Is it the intent of "set_param -P" that the specified changes only take effect after components are restarted? And if so, why? How would a normal system administrator go about finding out what settings are currently set permanently? I read through LU-3155 and see the discussion about using a single llog file for all nodes, so I will withhold comment about that for now. From a user-interface standpoint though, presenting a single namespace for all nodes in the entire center seems like less than desirable choice. Might we not want to set settings differently on different clusters (be they client or server clusters)? Given that not all paths under /proc have differentiating strings in their path, there are some things that can only be set completely globally in this design. And what about sites that use an MGS per filesystem, rather than a single MGS for the entire site? If one MGS says that the debug level should be one value, and another says that the debug value should be another value, is it entirely random which debug setting will appear on any given node? Chris On 08/13/2015 03:43 PM, Di Natale, Giuseppe wrote: > Greetings, > > In an effort to change test-framework.sh to not utilize the deprecated > conf_param option in lctl, I stumbled upon what appears to be > inconsistent behavior between lctl's conf_param and set_param -P > options. The permanent option test-framework.sh is attempting to change > is jobid_var. When using conf_param, any changes to the property are > written to /proc/fs/lustre/jobid_var within a short period of time. This > is not the case with set_param -P. The change is never reflected nor is > it stored in some other file within /proc. I started digging into the > MGS logs and found that the behavior for both are different (the > relevant segments of the logs are attached to this email and are named > accordingly). In short, it appears that conf_param attempts to apply the > changes to all the targets while set_param does not (it does not > recognize it as a global property). Can someone offer any insight on why > the behavior appears to be different or provide insight on if this is > incorrect behavior? > > Thanks, > Giuseppe Di Natale > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From andreas.dilger at intel.com Sat Aug 15 00:07:22 2015 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Sat, 15 Aug 2015 00:07:22 +0000 Subject: [lustre-devel] lctl conf_param vs set_param -P Message-ID: Coincidentally (or maybe that was what drove your investigations?), I'd just filed https://jira.hpdd.intel.com/browse/LU-7004 about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" on behalf of dinatale2 at llnl.gov> wrote: Greetings, In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior? Thanks, Giuseppe Di Natale From morrone2 at llnl.gov Sat Aug 15 00:36:47 2015 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Fri, 14 Aug 2015 17:36:47 -0700 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: References: Message-ID: <55CE899F.5060403@llnl.gov> It was coincidental this time. We saw the deprecation message during llmount.sh and figured that might be an easy first task for Giuseppe to get familiarity with the patch submission process. But it turned out to be a little more difficult than I thought. :) We might want to just remove the conf_param deprecation message altogether. I think there is more than just a simple bug in the "set_param -P" implementation. From what I am seeing, it looks to me like we have some design issues. Without anyone committed to working out a better design and implementing it, set_param -P looks like technical debt with no clear resolution in sight. Maybe we should even consider stripping it back back out of the tree. Chris On 08/14/2015 05:07 PM, Dilger, Andreas wrote: > Coincidentally (or maybe that was what drove your investigations?), I'd just filed https://jira.hpdd.intel.com/browse/LU-7004 about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Software Architect > Intel High Performance Data Division > > On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" on behalf of dinatale2 at llnl.gov> wrote: > > Greetings, > > In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior? > > > Thanks, > Giuseppe Di Natale > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From artem.blagodarenko at seagate.com Mon Aug 17 16:10:59 2015 From: artem.blagodarenko at seagate.com (Artem Blagodarenko) Date: Mon, 17 Aug 2015 19:10:59 +0300 Subject: [lustre-devel] lctl conf_param vs set_param -P Message-ID: Hello, >Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior "set_param -P" command syntax is differ from "conf_param" by design. It is same as "set_param" syntax. This allows use "set_param" for temporary commands and "set_param -P" for permanent. "set_param -P" allows change the same parameters that "set_param". So command set is wider, BUT existing commands should be changed to qualify "set_param" syntax. Below is the example: [root at devvm-sl6-1 MRP-2661]# lctl get_param jobid_var jobid_var=disable [root at devvm-sl6-1 MRP-2661]# lctl set_param -P client.jobid_var=SLURM_JOB_ID [root at devvm-sl6-1 MRP-2661]# lctl get_param jobid_var jobid_var=SLURM_JOB_ID >When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. After "set_param -P" applied, lock for "params" file is canceled that initiate this log is acquired and "executed" by targets and clients. This requires some time. >The change is never reflected nor is it stored in some other file within /proc. Changes stored in "params" file in CONFIG directory. Can be viewed using log_reader. >In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Really "set_param -P" attempts to apply the new changes to all the targets too. Command is executed on all targets, and all commands have "general" as 0th parameter of command in llog file params. >Is it the intent of "set_param -P" that the specified changes only take effect after components are restarted? And if so, why? No, changes applied just after command executed, but some time is required to cancel lock for params file. >How would a normal system administrator go about finding out what settings are currently set permanently? Seme settings as "set_param" sets can be set using "set_param -P". Finally all command that stored using "set_param" executed using "lctl set_param" upcall on targets. >From a user-interface standpoint though, presenting a single namespace for all nodes in the entire center seems like less than desirable choice. Yes, probably. All commands are saved in one namespace. It called "general". But I believe we could fix this issue easy, changing "general" to filesystem name, so targets checks this namespace before executing upcall. Actually, command that have filesystem part (for example started as "testfs.") faintly failed on wrong nodes and succeed only on addressed nodes. So probably, we do not need change this, but we can use reserved field if we would. >Given that not all paths under /proc have differentiating strings in their path, there are some things that can only be set completely globally in this design. Yes, this is case there we could change "general" to filename. Or fix this issue changing procfs paths. >And what about sites that use an MGS per filesystem, rather than a single MGS for the entire site? If one MGS says that the debug level should be one value, and another says that the debug value should be another value, is it entirely random which debug setting will appear on any given node? Good example there we could use reserved field. Do you think this can solve the issue? If so, we could extend "set_param -P" now. >We might want to just remove the conf_param deprecation message altogether. I think there is more than just a simple bug in the "set_param -P" implementation. I believe we need move to "set_param -P". Yes, this require some scripts to be changed, but fixing this now we get method with this benefits instead of conf_param: 1. Identical format for permanent and temporary commands 2. wildcarding in strings 3. no unimplemented paths (e.g. ptlrpc services) 4. simpler implementation >From what I am seeing, it looks to me like we have some design issues. All mentioned issues can be fixed easily without changing design. The problem of namespace distinguish already is part of design. Special field is reserved. >Without anyone committed to working out a better design and implementing it, set_param -P looks like technical debt with no clear resolution in sight. Maybe we should even consider stripping it back back out of the tree. I belive we could solve found issues and move forward using "set_param -P" I offered "set_param -P topic to last LAD, but unfortunately it was not accepted. Lets' discuss this together. Moving conf_param discarding date forward gives us some time for this. Thanks. On Sat, Aug 15, 2015 at 11:04 PM, wrote: > Send lustre-devel mailing list submissions to > lustre-devel at lists.lustre.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e= > or, via email, send a message with subject or body 'help' to > lustre-devel-request at lists.lustre.org > > You can reach the person managing the list at > lustre-devel-owner at lists.lustre.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of lustre-devel digest..." > > > Today's Topics: > > 1. Re: lctl conf_param vs set_param -P (Christopher J. Morrone) > 2. Re: lctl conf_param vs set_param -P (Dilger, Andreas) > 3. Re: lctl conf_param vs set_param -P (Christopher J. Morrone) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 14 Aug 2015 13:41:03 -0700 > From: "Christopher J. Morrone" > To: lustre-devel at lists.lustre.org > Subject: Re: [lustre-devel] lctl conf_param vs set_param -P > Message-ID: <55CE525F.3020505 at llnl.gov> > Content-Type: text/plain; charset=windows-1252; format=flowed > > I too am confused. And a bit dismayed that there is so little in the > way of code comments to explain the intent. > > Is it the intent of "set_param -P" that the specified changes only take > effect after components are restarted? And if so, why? > > How would a normal system administrator go about finding out what > settings are currently set permanently? > > I read through LU-3155 and see the discussion about using a single llog > file for all nodes, so I will withhold comment about that for now. > > From a user-interface standpoint though, presenting a single namespace > for all nodes in the entire center seems like less than desirable > choice. Might we not want to set settings differently on different > clusters (be they client or server clusters)? Given that not all paths > under /proc have differentiating strings in their path, there are some > things that can only be set completely globally in this design. > > And what about sites that use an MGS per filesystem, rather than a > single MGS for the entire site? If one MGS says that the debug level > should be one value, and another says that the debug value should be > another value, is it entirely random which debug setting will appear on > any given node? > > Chris > > On 08/13/2015 03:43 PM, Di Natale, Giuseppe wrote: > > Greetings, > > > > In an effort to change test-framework.sh to not utilize the deprecated > > conf_param option in lctl, I stumbled upon what appears to be > > inconsistent behavior between lctl's conf_param and set_param -P > > options. The permanent option test-framework.sh is attempting to change > > is jobid_var. When using conf_param, any changes to the property are > > written to /proc/fs/lustre/jobid_var within a short period of time. This > > is not the case with set_param -P. The change is never reflected nor is > > it stored in some other file within /proc. I started digging into the > > MGS logs and found that the behavior for both are different (the > > relevant segments of the logs are attached to this email and are named > > accordingly). In short, it appears that conf_param attempts to apply the > > changes to all the targets while set_param does not (it does not > > recognize it as a global property). Can someone offer any insight on why > > the behavior appears to be different or provide insight on if this is > > incorrect behavior? > > > > Thanks, > > Giuseppe Di Natale > > > > > > _______________________________________________ > > lustre-devel mailing list > > lustre-devel at lists.lustre.org > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e= > > > > > > ------------------------------ > > Message: 2 > Date: Sat, 15 Aug 2015 00:07:22 +0000 > From: "Dilger, Andreas" > To: "Di Natale, Giuseppe" > Cc: "lustre-devel at lists.lustre.org" > Subject: Re: [lustre-devel] lctl conf_param vs set_param -P > Message-ID: > Content-Type: text/plain; charset="us-ascii" > > Coincidentally (or maybe that was what drove your investigations?), I'd > just filed > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e= > about this issue. It looks like the "lctl set_param -P" feature needs more > testing, and I've removed the deprecation warning for "lctl conf_param" for > 2.8. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Software Architect > Intel High Performance Data Division > > On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" < > lustre-devel-bounces at lists.lustre.org lustre-devel-bounces at lists.lustre.org> on behalf of dinatale2 at llnl.gov > > wrote: > > Greetings, > > In an effort to change test-framework.sh to not utilize the deprecated > conf_param option in lctl, I stumbled upon what appears to be inconsistent > behavior between lctl's conf_param and set_param -P options. The permanent > option test-framework.sh is attempting to change is jobid_var. When using > conf_param, any changes to the property are written to > /proc/fs/lustre/jobid_var within a short period of time. This is not the > case with set_param -P. The change is never reflected nor is it stored in > some other file within /proc. I started digging into the MGS logs and found > that the behavior for both are different (the relevant segments of the logs > are attached to this email and are named accordingly). In short, it appears > that conf_param attempts to apply the changes to all the targets while > set_param does not (it does not recognize it as a global property). Can > someone offer any insight on why the behavior appears to be different or > provide insight on if this is incorrect behavior? > > Thanks, > Giuseppe Di Natale > > > ------------------------------ > > Message: 3 > Date: Fri, 14 Aug 2015 17:36:47 -0700 > From: "Christopher J. Morrone" > To: lustre-devel at lists.lustre.org > Subject: Re: [lustre-devel] lctl conf_param vs set_param -P > Message-ID: <55CE899F.5060403 at llnl.gov> > Content-Type: text/plain; charset=windows-1252; format=flowed > > It was coincidental this time. We saw the deprecation message during > llmount.sh and figured that might be an easy first task for Giuseppe to > get familiarity with the patch submission process. But it turned out to > be a little more difficult than I thought. :) > > We might want to just remove the conf_param deprecation message > altogether. I think there is more than just a simple bug in the > "set_param -P" implementation. From what I am seeing, it looks to me > like we have some design issues. Without anyone committed to working > out a better design and implementing it, set_param -P looks like > technical debt with no clear resolution in sight. Maybe we should even > consider stripping it back back out of the tree. > > Chris > > On 08/14/2015 05:07 PM, Dilger, Andreas wrote: > > Coincidentally (or maybe that was what drove your investigations?), I'd > just filed > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.hpdd.intel.com_browse_LU-2D7004&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=h6zLIi4EW2KHxq37PO71vZIbOskaw7V_qJJGV9mplo4&e= > about this issue. It looks like the "lctl set_param -P" feature needs more > testing, and I've removed the deprecation warning for "lctl conf_param" for > 2.8. > > > > Cheers, Andreas > > -- > > Andreas Dilger > > Lustre Software Architect > > Intel High Performance Data Division > > > > On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" < > lustre-devel-bounces at lists.lustre.org lustre-devel-bounces at lists.lustre.org> on behalf of dinatale2 at llnl.gov > > wrote: > > > > Greetings, > > > > In an effort to change test-framework.sh to not utilize the deprecated > conf_param option in lctl, I stumbled upon what appears to be inconsistent > behavior between lctl's conf_param and set_param -P options. The permanent > option test-framework.sh is attempting to change is jobid_var. When using > conf_param, any changes to the property are written to > /proc/fs/lustre/jobid_var within a short period of time. This is not the > case with set_param -P. The change is never reflected nor is it stored in > some other file within /proc. I started digging into the MGS logs and found > that the behavior for both are different (the relevant segments of the logs > are attached to this email and are named accordingly). In short, it appears > that conf_param attempts to apply the changes to all the targets while > set_param does not (it does not recognize it as a global property). Can > someone offer any insight on why the behavior appears to be different or > provide insight on if this is incorrect behavio > r? > > > > > > Thanks, > > Giuseppe Di Natale > > _______________________________________________ > > lustre-devel mailing list > > lustre-devel at lists.lustre.org > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e= > > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=f-9bup-jYTTZb-zqZbdOzQTObV_VGrTP7-8xROGj35I&m=zc0hAFYXhRlbqxXHoLOSK0PoVteDFbf33MzwVDBMADY&s=NedsjGFvLB9S2GAZw1bC2c9WNT-N8YBC0YHvMimBTvw&e= > > > ------------------------------ > > End of lustre-devel Digest, Vol 103, Issue 2 > ******************************************** > -- Artem Blagodarenko Ph.D.*·* SW Developer on my.seagate.com Seagate Technology, LLC www.seagate.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Mon Aug 17 18:06:18 2015 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Mon, 17 Aug 2015 11:06:18 -0700 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: References: Message-ID: <55D2229A.10506@llnl.gov> On 08/17/2015 09:10 AM, Artem Blagodarenko wrote: > Hello, > > >Can someone offer any insight on why the behavior appears to be [cut] > After "set_param -P" applied, lock for "params" file is canceled that > initiate this log is acquired and "executed" by targets and clients. > This requires some time. How much time? In human terms, this should happen in very little time on a vm with llmount.sh created filesystem, should it not? > >Is it the intent of "set_param -P" that the specified changes only take > effect after components are restarted? And if so, why? > > No, changes applied just after command executed, but some time > is required to cancel lock for params file. Perhaps that is what is intended to happen, but is that working on master? > >How would a normal system administrator go about finding out what > settings are currently set permanently? > > Seme settings as "set_param" sets can be set using "set_param -P". > Finally all command that stored using "set_param" executed using "lctl > set_param" upcall on targets. That was not the question. I asked how a system administrator can get a listing of the parameters that have been permanently set. It would appear that the only current method is to mount the underlying filesystem, locate the log file, and use llog_reader on it. That is a software developer's interface, not an adequate interface for system administrators. > >And what about sites that use an MGS per filesystem, rather than a > single MGS for the entire site? If one MGS says that the debug level > should be one value, and another says that the debug value should be > another value, is it entirely random which debug setting will appear on > any given node? > > Good example there we could use reserved field. Do you think this can > solve the issue? If so, we could extend "set_param -P" now. I'm not so sure. I have a feeling that really we need to consider and classify all of the configurables. Another use case: In WAN usage, two completely different organizations (two different universities, for instance) with separate security and administration policies may control different parts of the filesystem. One organization runs the servers and some clients of its own. A second organization runs clients that mount the first organization's servers over the internet. The second organization probably doesn't want the first organization to be setting _most_ of the settings that are available under lnet and lustre /proc trees. But there are certainly some that are specific to the running of that filesystem that might be reasonable to pass to parts of the client that pertain to only that mount (keep in mind that the same client node may be mounting both local and remote lustre filesystems). > >We might want to just remove the conf_param deprecation message > altogether. I think there is more than just a simple bug in the > "set_param -P" implementation. > > I believe we need move to "set_param -P". Yes, this require some scripts > to be changed, but fixing this now we get method with this benefits > instead of conf_param: I would agree if there was no change other than than interface, but this also changes underlying semantics. That concerns me since there seems to be little information about the design and not much in the way of vetting said design. Is there more design information out there to which you can point us? > I belive we could solve found issues and move forward using "set_param -P" > I offered "set_param -P topic to last LAD, but unfortunately it was not > accepted. Lets' discuss this together. Moving conf_param discarding date > forward gives us some time for this. This mailing list reaches a greater number of Lustre developers than LAD or LUG can ever reach (travel is expensive, email is cheap). This list is very good place to work through design and implementation issues. Chris From oleg.drokin at intel.com Mon Aug 17 23:28:12 2015 From: oleg.drokin at intel.com (Drokin, Oleg) Date: Mon, 17 Aug 2015 23:28:12 +0000 Subject: [lustre-devel] New master tag 2.7.58 Message-ID: <6F059C6E-A6A4-416D-BEDA-4BD41E08D7B9@intel.com> Hello! I just tagged 2.7.58 in the community Lustre tree master branch. Bye, Oleg From nathan.rutman at seagate.com Tue Aug 18 03:37:35 2015 From: nathan.rutman at seagate.com (Nathan Rutman) Date: Mon, 17 Aug 2015 20:37:35 -0700 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: <55D2229A.10506@llnl.gov> References: <55D2229A.10506@llnl.gov> Message-ID: The goal of set_param -P (permanent, global) was to replace conf_param and harmonize with set_param (local). (conf_param semantics were the beginning of an aborted attempt to move away from /proc and toward a more rational parameter space.) Unfortunately, most of the conf_param items were "specials", that had to be re-implemented under set_param -P. Most of that work was done years ago, but some things like "jobid" were not around then -- and apparently were implemented using the older mechanism only. So the right way forward to my mind is to deprecate conf_param, and fix the broken things to work correctly with set_param -P. *--* *Nathan Rutman · Principal Systems ArchitectSeagate Technology** · *+1 503 877-9507* · *GMT-8 On Mon, Aug 17, 2015 at 11:06 AM, Christopher J. Morrone wrote: > On 08/17/2015 09:10 AM, Artem Blagodarenko wrote: > >> Hello, >> >> >Can someone offer any insight on why the behavior appears to be >> > [cut] > >> After "set_param -P" applied, lock for "params" file is canceled that >> initiate this log is acquired and "executed" by targets and clients. >> This requires some time. >> > > How much time? In human terms, this should happen in very little time on > a vm with llmount.sh created filesystem, should it not? > > >Is it the intent of "set_param -P" that the specified changes only take >> effect after components are restarted? And if so, why? >> >> No, changes applied just after command executed, but some time >> is required to cancel lock for params file. >> > > Perhaps that is what is intended to happen, but is that working on master? > > >How would a normal system administrator go about finding out what >> settings are currently set permanently? >> >> Seme settings as "set_param" sets can be set using "set_param -P". >> Finally all command that stored using "set_param" executed using "lctl >> set_param" upcall on targets. >> > > That was not the question. I asked how a system administrator can get a > listing of the parameters that have been permanently set. It would appear > that the only current method is to mount the underlying filesystem, locate > the log file, and use llog_reader on it. That is a software developer's > interface, not an adequate interface for system administrators. > > >And what about sites that use an MGS per filesystem, rather than a >> single MGS for the entire site? If one MGS says that the debug level >> should be one value, and another says that the debug value should be >> another value, is it entirely random which debug setting will appear on >> any given node? >> >> Good example there we could use reserved field. Do you think this can >> solve the issue? If so, we could extend "set_param -P" now. >> > > I'm not so sure. I have a feeling that really we need to consider and > classify all of the configurables. > > Another use case: > > In WAN usage, two completely different organizations (two different > universities, for instance) with separate security and administration > policies may control different parts of the filesystem. One organization > runs the servers and some clients of its own. A second organization runs > clients that mount the first organization's servers over the internet. > > The second organization probably doesn't want the first organization to be > setting _most_ of the settings that are available under lnet and lustre > /proc trees. But there are certainly some that are specific to the running > of that filesystem that might be reasonable to pass to parts of the client > that pertain to only that mount (keep in mind that the same client node may > be mounting both local and remote lustre filesystems). > > >We might want to just remove the conf_param deprecation message >> altogether. I think there is more than just a simple bug in the >> "set_param -P" implementation. >> >> I believe we need move to "set_param -P". Yes, this require some scripts >> to be changed, but fixing this now we get method with this benefits >> instead of conf_param: >> > > I would agree if there was no change other than than interface, but this > also changes underlying semantics. That concerns me since there seems to > be little information about the design and not much in the way of vetting > said design. > > Is there more design information out there to which you can point us? > > I belive we could solve found issues and move forward using "set_param -P" >> I offered "set_param -P topic to last LAD, but unfortunately it was not >> accepted. Lets' discuss this together. Moving conf_param discarding date >> forward gives us some time for this. >> > > This mailing list reaches a greater number of Lustre developers than LAD > or LUG can ever reach (travel is expensive, email is cheap). This list is > very good place to work through design and implementation issues. > > Chris > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=AwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=lPtum2eaUl7hXL6vefqvGWdMnCaUrX7yAIJeZQgwM1s&m=vsZ-zDT5iFOnt_-eNj7umzU-BW83CqhBtJkJkGQdboM&s=nlcfRkzHK6EveIuWcYWCpM_Zo9IudKKl_qjR5XBUL9s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From artem.blagodarenko at seagate.com Tue Aug 18 11:06:23 2015 From: artem.blagodarenko at seagate.com (Artem Blagodarenko) Date: Tue, 18 Aug 2015 14:06:23 +0300 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: <55D2229A.10506@llnl.gov> References: <55D2229A.10506@llnl.gov> Message-ID: Hello, How much time? In human terms, this should happen in very little time on a > vm with llmount.sh created filesystem, should it not? I am not ready to give exact time. I believe it depends from configuration. But the time is same as for parameters that set up using "lctl conf_param" and stored in llog file, because the same mechanism is used for log propagation. Difference is only that another lock is canceled (CONFIG_T_PARAMS). Nathan answered in previous message about exact "jobid" case. > Perhaps that is what is intended to happen, but is that working on master? It is designed such way. Please, report if any problem happened. > > > >How would a normal system administrator go about finding out what >> settings are currently set permanently? >> >> Seme settings as "set_param" sets can be set using "set_param -P". >> Finally all command that stored using "set_param" executed using "lctl >> set_param" upcall on targets. >> > > That was not the question. I asked how a system administrator can get a > listing of the parameters that have been permanently set. It would appear > that the only current method is to mount the underlying filesystem, locate > the log file, and use llog_reader on it. That is a software developer's > interface, not an adequate interface for system administrators. Yes, the problem is the same as for parameters that stored in llog using "lctl conf_param". Probably we need add functionality to show such permanent parameter. > > > >And what about sites that use an MGS per filesystem, rather than a >> single MGS for the entire site? If one MGS says that the debug level >> should be one value, and another says that the debug value should be >> another value, is it entirely random which debug setting will appear on >> any given node? >> >> Good example there we could use reserved field. Do you think this can >> solve the issue? If so, we could extend "set_param -P" now. >> > > I'm not so sure. I have a feeling that really we need to consider and > classify all of the configurables. > > Another use case: > > In WAN usage, two completely different organizations (two different > universities, for instance) with separate security and administration > policies may control different parts of the filesystem. One organization > runs the servers and some clients of its own. A second organization runs > clients that mount the first organization's servers over the internet. > > The second organization probably doesn't want the first organization to be > setting _most_ of the settings that are available under lnet and lustre > /proc trees. But there are certainly some that are specific to the running > of that filesystem that might be reasonable to pass to parts of the client > that pertain to only that mount (keep in mind that the same client node may > be mounting both local and remote lustre filesystems). Probably I don't understood this scheme, but first organization have access to MGS and can change any configuration it wants. It is non secure with our without "set_param -P" > > > >We might want to just remove the conf_param deprecation message >> altogether. I think there is more than just a simple bug in the >> "set_param -P" implementation. >> >> I believe we need move to "set_param -P". Yes, this require some scripts >> to be changed, but fixing this now we get method with this benefits >> instead of conf_param: >> > > I would agree if there was no change other than than interface, but this > also changes underlying semantics. That concerns me since there seems to > be little information about the design and not much in the way of vetting > said design. > > Is there more design information out there to which you can point us? It would be great if we could discuss all this problems during design phase, but this feature already landed. I believe we could show something like presentation about architecture of "set_param -P", because original arch doc is quite outdate because of changes during inspections and landing. > > > I belive we could solve found issues and move forward using "set_param -P" >> I offered "set_param -P topic to last LAD, but unfortunately it was not >> accepted. Lets' discuss this together. Moving conf_param discarding date >> forward gives us some time for this. >> > > This mailing list reaches a greater number of Lustre developers than LAD > or LUG can ever reach (travel is expensive, email is cheap). This list is > very good place to work through design and implementation issues. > > Yes, sure. Thank you for stating this thread. -- Artem Blagodarenko Ph.D.*·* SW Developer on seagate.com Seagate Technology, LLC www.seagate.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Tue Aug 18 18:00:23 2015 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 18 Aug 2015 11:00:23 -0700 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: References: <55D2229A.10506@llnl.gov> Message-ID: <55D372B7.1080700@llnl.gov> On 08/17/2015 08:37 PM, Nathan Rutman wrote: > The goal of set_param -P (permanent, global) was to replace conf_param > and harmonize with set_param (local). (conf_param semantics were the > beginning of an aborted attempt to move away from /proc and toward a > more rational parameter space.) Unfortunately, most of the conf_param > items were "specials", that had to be re-implemented under set_param -P. > Most of that work was done years ago, but some things like "jobid" were > not around then -- and apparently were implemented using the older > mechanism only. Could you elaborate on how jobid is special? Why, set_param -P not currently knowing that jobid is special, is it not simply triggering the upcall on all nodes to write a value to the proc file? How would a system administrator know which things are special and which are not, and how the behavior changes for special and non-special options? Chris From kit.westneat at gmail.com Wed Aug 19 02:37:26 2015 From: kit.westneat at gmail.com (Kit Westneat) Date: Tue, 18 Aug 2015 22:37:26 -0400 Subject: [lustre-devel] lctl conf_param vs set_param -P In-Reply-To: <55D2229A.10506@llnl.gov> References: <55D2229A.10506@llnl.gov> Message-ID: > > How much time? In human terms, this should happen in very little time on > a vm with llmount.sh created filesystem, should it not? If it uses the same timeout as the other config locks, it will take between 5-10 seconds (5 + 0 to 5 random seconds). It's all done in the mgc_requeue_thread function. - Kit On Mon, Aug 17, 2015 at 2:06 PM, Christopher J. Morrone wrote: > On 08/17/2015 09:10 AM, Artem Blagodarenko wrote: > >> Hello, >> >> >Can someone offer any insight on why the behavior appears to be >> > [cut] > >> After "set_param -P" applied, lock for "params" file is canceled that >> initiate this log is acquired and "executed" by targets and clients. >> This requires some time. >> > > How much time? In human terms, this should happen in very little time on > a vm with llmount.sh created filesystem, should it not? > > >Is it the intent of "set_param -P" that the specified changes only take >> effect after components are restarted? And if so, why? >> >> No, changes applied just after command executed, but some time >> is required to cancel lock for params file. >> > > Perhaps that is what is intended to happen, but is that working on master? > > >How would a normal system administrator go about finding out what >> settings are currently set permanently? >> >> Seme settings as "set_param" sets can be set using "set_param -P". >> Finally all command that stored using "set_param" executed using "lctl >> set_param" upcall on targets. >> > > That was not the question. I asked how a system administrator can get a > listing of the parameters that have been permanently set. It would appear > that the only current method is to mount the underlying filesystem, locate > the log file, and use llog_reader on it. That is a software developer's > interface, not an adequate interface for system administrators. > > >And what about sites that use an MGS per filesystem, rather than a >> single MGS for the entire site? If one MGS says that the debug level >> should be one value, and another says that the debug value should be >> another value, is it entirely random which debug setting will appear on >> any given node? >> >> Good example there we could use reserved field. Do you think this can >> solve the issue? If so, we could extend "set_param -P" now. >> > > I'm not so sure. I have a feeling that really we need to consider and > classify all of the configurables. > > Another use case: > > In WAN usage, two completely different organizations (two different > universities, for instance) with separate security and administration > policies may control different parts of the filesystem. One organization > runs the servers and some clients of its own. A second organization runs > clients that mount the first organization's servers over the internet. > > The second organization probably doesn't want the first organization to be > setting _most_ of the settings that are available under lnet and lustre > /proc trees. But there are certainly some that are specific to the running > of that filesystem that might be reasonable to pass to parts of the client > that pertain to only that mount (keep in mind that the same client node may > be mounting both local and remote lustre filesystems). > > >We might want to just remove the conf_param deprecation message >> altogether. I think there is more than just a simple bug in the >> "set_param -P" implementation. >> >> I believe we need move to "set_param -P". Yes, this require some scripts >> to be changed, but fixing this now we get method with this benefits >> instead of conf_param: >> > > I would agree if there was no change other than than interface, but this > also changes underlying semantics. That concerns me since there seems to > be little information about the design and not much in the way of vetting > said design. > > Is there more design information out there to which you can point us? > > I belive we could solve found issues and move forward using "set_param -P" >> I offered "set_param -P topic to last LAD, but unfortunately it was not >> accepted. Lets' discuss this together. Moving conf_param discarding date >> forward gives us some time for this. >> > > This mailing list reaches a greater number of Lustre developers than LAD > or LUG can ever reach (travel is expensive, email is cheap). This list is > very good place to work through design and implementation issues. > > Chris > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hornc at cray.com Wed Aug 19 16:31:04 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 16:31:04 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? Message-ID: A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends? Chris Horn From hornc at cray.com Wed Aug 19 18:24:41 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 18:24:41 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: Message-ID: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing? Chris Horn On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov > wrote: Chris, concurrent_sends is measurement about parallel operations, but credits is flow control artifact. each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes. On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn > wrote: A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends? Chris Horn _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at seagate.com Wed Aug 19 18:58:49 2015 From: alexey.lyashkov at seagate.com (Alexey Lyashkov) Date: Wed, 19 Aug 2015 21:58:49 +0300 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> Message-ID: Chris, In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message. So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages. But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation.. It's know bug for me but need large time to create a network model to create a correct credits distribution. On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn wrote: > They sure look pretty related to me. In kiblnd_post_tx_locked() we return > EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is > incremented on every send. So it looks like you couldn’t ever send more > than concurrent_sends messages to a single peer, which is exactly what > peer_credits is supposed to govern, no? What am I missing? > > Chris Horn > > > On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov > wrote: > > Chris, > > concurrent_sends is measurement about parallel operations, but credits is > flow control artifact. > each operations may eats different number credits and calculated in per > link and per destination basic, so it's completely different attributes. > > > > > On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn wrote: > >> A thread on HPDD-discuss made me think about this question. AFAICT, the >> o2iblnd driver code will not let you have more that concurrent_sends >> messages in flight at the same time(in fact, we LASSERT on this fact in >> kiblnd_check_sends). Thus peer_credits is effectively limited by >> concurrent_sends anyways. What’s the reasoning behind allowing peer_credits >> to be larger than concurrent_sends? >> >> Chris Horn >> _______________________________________________ >> lustre-devel mailing list >> lustre-devel at lists.lustre.org >> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> >> > > > > -- > Alexey Lyashkov *·* Technical lead for a Morpheus team > Seagate Technology, LLC > www.seagate.com > www.lustre.org > > > > -- Alexey Lyashkov *·* Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From hornc at cray.com Wed Aug 19 19:38:52 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 19:38:52 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> Message-ID: But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation.. Well, I think that depends on how the flow control is supposed to work. The code definitely prevents *more* than peer_credits sends to a single peer. And negative number of credits indeed indicates a queue which, I think, is fine. What is non-obvious is that it appears that the code may prevent fewer than peer_credits sends to a single peer, even if there aren’t any other outstanding sends to other peers. i.e. we hit the concurrent_sends limit before we hit the peer_credits limit. This is counterintuitive, and I don’t understand why the code was written this way. Chris Horn On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov > wrote: Chris, In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message. So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages. But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation.. It's know bug for me but need large time to create a network model to create a correct credits distribution. On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn > wrote: They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing? Chris Horn On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov > wrote: Chris, concurrent_sends is measurement about parallel operations, but credits is flow control artifact. each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes. On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn > wrote: A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends? Chris Horn _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey.lyashkov at seagate.com Wed Aug 19 19:54:57 2015 From: alexey.lyashkov at seagate.com (Alexey Lyashkov) Date: Wed, 19 Aug 2015 22:54:57 +0300 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> Message-ID: In credit based theory - credits should never to be less zero, as credit is resources to send. If credits set to zero we just stop send. My assumption base on book "Traffic Management for High-speed Networks International Science Lecture Series ; 4th Lecture". > This is counterintuitive, and I don’t understand why the code was written this way. if i understand correctly it code was originally written for a direct connected network, but credits distribution model forget to change when lnet routers was created. So we continue to count credits in peers base instead of distribute credits only in next hop base where a router should distribute a provided credits over connected clients. It provide a router overloading with huge parallel sends and new limit was added. On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn wrote: > But as i understand credit based flow control don't work now - i see > several situations when server have a negative number a credits, which > indicate we have sending queue more than limits and parallel sends limits > will work in that situation.. > > > Well, I think that depends on how the flow control is supposed to work. > The code definitely prevents *more* than peer_credits sends to a single > peer. And negative number of credits indeed indicates a queue which, I > think, is fine. What is non-obvious is that it appears that the code may > prevent fewer than peer_credits sends to a single peer, even if there > aren’t any other outstanding sends to other peers. i.e. we hit the > concurrent_sends limit before we hit the peer_credits limit. This is > counterintuitive, and I don’t understand why the code was written this way. > > Chris Horn > > On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov > wrote: > > Chris, > > In current code it's may be same. Let me finish some bugs before look to > code. But with "good" code - you should have a different cost (credits > count) for different messages. 1Mb bulk transfer should eats more credits > than simple 4kb message. So you "should" have a two limits first one for > parallel send with zero/low cost messages and second one for large messages. > But as i understand credit based flow control don't work now - i see > several situations when server have a negative number a credits, which > indicate we have sending queue more than limits and parallel sends limits > will work in that situation.. > > It's know bug for me but need large time to create a network model to > create a correct credits distribution. > > > On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn wrote: > >> They sure look pretty related to me. In kiblnd_post_tx_locked() we return >> EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is >> incremented on every send. So it looks like you couldn’t ever send more >> than concurrent_sends messages to a single peer, which is exactly what >> peer_credits is supposed to govern, no? What am I missing? >> >> Chris Horn >> >> >> On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov < >> alexey.lyashkov at seagate.com> wrote: >> >> Chris, >> >> concurrent_sends is measurement about parallel operations, but credits is >> flow control artifact. >> each operations may eats different number credits and calculated in per >> link and per destination basic, so it's completely different attributes. >> >> >> >> >> On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn wrote: >> >>> A thread on HPDD-discuss made me think about this question. AFAICT, the >>> o2iblnd driver code will not let you have more that concurrent_sends >>> messages in flight at the same time(in fact, we LASSERT on this fact in >>> kiblnd_check_sends). Thus peer_credits is effectively limited by >>> concurrent_sends anyways. What’s the reasoning behind allowing peer_credits >>> to be larger than concurrent_sends? >>> >>> Chris Horn >>> _______________________________________________ >>> lustre-devel mailing list >>> lustre-devel at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >>> >>> >> >> >> >> -- >> Alexey Lyashkov *·* Technical lead for a Morpheus team >> Seagate Technology, LLC >> www.seagate.com >> www.lustre.org >> >> >> >> > > > -- > Alexey Lyashkov *·* Technical lead for a Morpheus team > Seagate Technology, LLC > www.seagate.com > www.lustre.org > > > > -- Alexey Lyashkov *·* Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From hornc at cray.com Wed Aug 19 20:04:27 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 20:04:27 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> Message-ID: <92C8FFD7-63AB-4423-B280-EA48557F8268@cray.com> Fair enough. The code I’m talking about, that allows concurrent_sends to be set less than peer_credits, was apparently implemented as a “workaround” for bug 15983. Chris Horn On Aug 19, 2015, at 2:54 PM, Alexey Lyashkov > wrote: In credit based theory - credits should never to be less zero, as credit is resources to send. If credits set to zero we just stop send. My assumption base on book "Traffic Management for High-speed Networks International Science Lecture Series ; 4th Lecture". > This is counterintuitive, and I don’t understand why the code was written this way. if i understand correctly it code was originally written for a direct connected network, but credits distribution model forget to change when lnet routers was created. So we continue to count credits in peers base instead of distribute credits only in next hop base where a router should distribute a provided credits over connected clients. It provide a router overloading with huge parallel sends and new limit was added. On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn > wrote: But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation.. Well, I think that depends on how the flow control is supposed to work. The code definitely prevents *more* than peer_credits sends to a single peer. And negative number of credits indeed indicates a queue which, I think, is fine. What is non-obvious is that it appears that the code may prevent fewer than peer_credits sends to a single peer, even if there aren’t any other outstanding sends to other peers. i.e. we hit the concurrent_sends limit before we hit the peer_credits limit. This is counterintuitive, and I don’t understand why the code was written this way. Chris Horn On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov > wrote: Chris, In current code it's may be same. Let me finish some bugs before look to code. But with "good" code - you should have a different cost (credits count) for different messages. 1Mb bulk transfer should eats more credits than simple 4kb message. So you "should" have a two limits first one for parallel send with zero/low cost messages and second one for large messages. But as i understand credit based flow control don't work now - i see several situations when server have a negative number a credits, which indicate we have sending queue more than limits and parallel sends limits will work in that situation.. It's know bug for me but need large time to create a network model to create a correct credits distribution. On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn > wrote: They sure look pretty related to me. In kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on every send. So it looks like you couldn’t ever send more than concurrent_sends messages to a single peer, which is exactly what peer_credits is supposed to govern, no? What am I missing? Chris Horn On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov > wrote: Chris, concurrent_sends is measurement about parallel operations, but credits is flow control artifact. each operations may eats different number credits and calculated in per link and per destination basic, so it's completely different attributes. On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn > wrote: A thread on HPDD-discuss made me think about this question. AFAICT, the o2iblnd driver code will not let you have more that concurrent_sends messages in flight at the same time(in fact, we LASSERT on this fact in kiblnd_check_sends). Thus peer_credits is effectively limited by concurrent_sends anyways. What’s the reasoning behind allowing peer_credits to be larger than concurrent_sends? Chris Horn _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Wed Aug 19 20:48:49 2015 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Wed, 19 Aug 2015 13:48:49 -0700 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> Message-ID: <55D4EBB1.3000502@llnl.gov> As a quick aside to all developers: _This_ is why code comments are so important. We can probably all figure out _what_ this code does, but without comments explaining _why_ it is doing what it does, the reader is spending way too much time trying to fathom what possible purpose this could be for. LNet does stop sending LNet messages on a peer connection when that peer's credit count reaches zero. LNet chose to then relate the count of messages awaiting credits by using negative values of the same variable. It is just the convention chosen, and doesn't necessarily mean that there is a design problem there. I think the major difference between o2iblnd's usage of peer_credits and concurrent_sends (besides being inverses of one another) is that peer_credits only counts outstanding LNet level messages, whereas concurrent_sends tracks _all_ outstanding sends that the o2iblnd puts on the wire for that peer. The messages IBLND_MSG_PUT_NAK, IBLND_MSG_PUT_ACK, IBLND_MSG_PUT_DONE, and IBLND_MSG_GET_DONE do not use peer credits. NOOP messages also skip using peer credits if the device support out-of-band messages. So I would assume that there are some hardware devices out there with limitations that mean that we need to precisely count and limit the number of IB level sends in flight at one time. Since some o2iblnd messaging skips using the peer credits value, that value did not server as a precise limit on the number of IB messages in flight. As far as I can tell, routers don't have anything to do with this. Chris On 08/19/2015 12:54 PM, Alexey Lyashkov wrote: > In credit based theory - credits should never to be less zero, as credit > is resources to send. If credits set to zero we just stop send. > My assumption base on book "Traffic Management for High-speed Networks > International Science Lecture Series ; 4th Lecture". > >> This is counterintuitive, and I don’t understand why the code was written this way. > if i understand correctly it code was originally written for a direct > connected network, but credits distribution model forget to change when > lnet routers was created. So we continue to count credits in peers base > instead of distribute credits only in next hop base where a router > should distribute a provided credits over connected clients. It provide > a router overloading with huge parallel sends and new limit was added. > > > > On Wed, Aug 19, 2015 at 10:38 PM, Chris Horn > wrote: > >> But as i understand credit based flow control don't work now - i >> see several situations when server have a negative number a >> credits, which indicate we have sending queue more than limits and >> parallel sends limits will work in that situation.. > > Well, I think that depends on how the flow control is supposed to > work. The code definitely prevents *more* than peer_credits sends to > a single peer. And negative number of credits indeed indicates a > queue which, I think, is fine. What is non-obvious is that it > appears that the code may prevent fewer than peer_credits sends to a > single peer, even if there aren’t any other outstanding sends to > other peers. i.e. we hit the concurrent_sends limit before we hit > the peer_credits limit. This is counterintuitive, and I don’t > understand why the code was written this way. > > Chris Horn > >> On Aug 19, 2015, at 1:58 PM, Alexey Lyashkov >> > >> wrote: >> >> Chris, >> >> In current code it's may be same. Let me finish some bugs before >> look to code. But with "good" code - you should have a different >> cost (credits count) for different messages. 1Mb bulk transfer >> should eats more credits than simple 4kb message. So you "should" >> have a two limits first one for parallel send with zero/low cost >> messages and second one for large messages. >> But as i understand credit based flow control don't work now - i >> see several situations when server have a negative number a >> credits, which indicate we have sending queue more than limits and >> parallel sends limits will work in that situation.. >> >> It's know bug for me but need large time to create a network model >> to create a correct credits distribution. >> >> >> On Wed, Aug 19, 2015 at 9:24 PM, Chris Horn> >wrote: >> >> They sure look pretty related to me. In >> kiblnd_post_tx_locked() we return EAGAIN if ibc_nsends_posted >> == IBLND_CONCURRENT_SENDS. ibc_nsends_posted is incremented on >> every send. So it looks like you couldn’t ever send more than >> concurrent_sends messages to a single peer, which is exactly >> what peer_credits is supposed to govern, no? What am I missing? >> >> Chris Horn >> >> >>> On Aug 19, 2015, at 12:28 PM, Alexey Lyashkov >>> >> > wrote: >>> >>> Chris, >>> >>> concurrent_sends is measurement about parallel operations, >>> but credits is flow control artifact. >>> each operations may eats different number credits and >>> calculated in per link and per destination basic, so it's >>> completely different attributes. >>> >>> >>> >>> >>> On Wed, Aug 19, 2015 at 7:31 PM, Chris Horn>> >wrote: >>> >>> A thread on HPDD-discuss made me think about this >>> question. AFAICT, the o2iblnd driver code will not let >>> you have more that concurrent_sends messages in flight at >>> the same time(in fact, we LASSERT on this fact in >>> kiblnd_check_sends). Thus peer_credits is effectively >>> limited by concurrent_sends anyways. What’s the reasoning >>> behind allowing peer_credits to be larger than >>> concurrent_sends? >>> >>> Chris Horn >>> _______________________________________________ >>> lustre-devel mailing list >>> lustre-devel at lists.lustre.org >>> >>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >>> >>> >>> >>> >>> >>> -- >>> Alexey Lyashkov *·* Technical lead for a Morpheus team >>> Seagate Technology, LLC >>> www.seagate.com >>> www.lustre.org >>> >> >> >> >> >> -- >> Alexey Lyashkov *·* Technical lead for a Morpheus team >> Seagate Technology, LLC >> www.seagate.com >> www.lustre.org >> > > > > > -- > Alexey Lyashkov *·* Technical lead for a Morpheus team > Seagate Technology, LLC > www.seagate.com > www.lustre.org > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > From alexey.lyashkov at seagate.com Wed Aug 19 20:54:57 2015 From: alexey.lyashkov at seagate.com (Alexey Lyashkov) Date: Wed, 19 Aug 2015 23:54:57 +0300 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: <55D4EBB1.3000502@llnl.gov> References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> <55D4EBB1.3000502@llnl.gov> Message-ID: In my invested case, I have see large number tx in sending queue with negative credits. it's mean we don't able to resend these messages via different gateway until message expired. But if we stop to queue messages with reach a zero credits, we will able to send message via different gateway after peer dead event without any notifications to ptlrpc layer. So i think it's likely to be a bug as from my point view, we need to avoid ptlrpc reconnects as possible. On Wed, Aug 19, 2015 at 11:48 PM, Christopher J. Morrone wrote: > LNet does stop sending LNet messages on a peer connection when that peer's > credit count reaches zero. LNet chose to then relate the count of messages > awaiting credits by using negative values of the same variable. It is just > the convention chosen, and doesn't necessarily mean that there is a design > problem there. > -- Alexey Lyashkov *·* Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From hornc at cray.com Wed Aug 19 21:14:47 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 21:14:47 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> <55D4EBB1.3000502@llnl.gov> Message-ID: We could more easily help that situation by changing the lnet_compare_routes() method to look at the number of credits available when deciding which router peer to use as a next hop. Chris Horn On Aug 19, 2015, at 3:54 PM, Alexey Lyashkov > wrote: In my invested case, I have see large number tx in sending queue with negative credits. it's mean we don't able to resend these messages via different gateway until message expired. But if we stop to queue messages with reach a zero credits, we will able to send message via different gateway after peer dead event without any notifications to ptlrpc layer. So i think it's likely to be a bug as from my point view, we need to avoid ptlrpc reconnects as possible. On Wed, Aug 19, 2015 at 11:48 PM, Christopher J. Morrone > wrote: LNet does stop sending LNet messages on a peer connection when that peer's credit count reaches zero. LNet chose to then relate the count of messages awaiting credits by using negative values of the same variable. It is just the convention chosen, and doesn't necessarily mean that there is a design problem there. -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From hornc at cray.com Wed Aug 19 21:16:29 2015 From: hornc at cray.com (Chris Horn) Date: Wed, 19 Aug 2015 21:16:29 +0000 Subject: [lustre-devel] Why can you set concurrent_sends < peer_credits ? In-Reply-To: References: <04556C78-49F4-4A78-A477-ADE76C7773BB@cray.com> <55D4EBB1.3000502@llnl.gov> Message-ID: <0D0C00EC-C1C6-4360-BA73-730B97F2D85A@cray.com> To clarify, the lnet_compare_routes() does look at peer credits, but only if priority, hops and queued nob are all the same. It would probably be better to weight all of these things together as was suggested at one of the developer conferences recently. Chris Horn On Aug 19, 2015, at 4:14 PM, Chris Horn > wrote: We could more easily help that situation by changing the lnet_compare_routes() method to look at the number of credits available when deciding which router peer to use as a next hop. Chris Horn On Aug 19, 2015, at 3:54 PM, Alexey Lyashkov > wrote: In my invested case, I have see large number tx in sending queue with negative credits. it's mean we don't able to resend these messages via different gateway until message expired. But if we stop to queue messages with reach a zero credits, we will able to send message via different gateway after peer dead event without any notifications to ptlrpc layer. So i think it's likely to be a bug as from my point view, we need to avoid ptlrpc reconnects as possible. On Wed, Aug 19, 2015 at 11:48 PM, Christopher J. Morrone > wrote: LNet does stop sending LNet messages on a peer connection when that peer's credit count reaches zero. LNet chose to then relate the count of messages awaiting credits by using negative values of the same variable. It is just the convention chosen, and doesn't necessarily mean that there is a design problem there. -- Alexey Lyashkov · Technical lead for a Morpheus team Seagate Technology, LLC www.seagate.com www.lustre.org _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jeffrey.C.Becker at nasa.gov Thu Aug 20 20:01:38 2015 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Thu, 20 Aug 2015 13:01:38 -0700 Subject: [lustre-devel] [PATCH] staging/lustre: fix block comment formatting Message-ID: <1440100898-37695-1-git-send-email-Jeffrey.C.Becker@nasa.gov> Running checkpatch.pl on lnet/klnds/o2iblnd/o2iblnd.h produces several "Block comments use a trailing */ on a separate line" warnings. This patch fixes these. --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 90 ++++++++++++++-------- 1 file changed, 60 insertions(+), 30 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index f4b6c33..659d9cc 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -82,9 +82,11 @@ typedef struct { int *kib_dev_failover; /* HCA failover */ unsigned int *kib_service; /* IB service number */ int *kib_min_reconnect_interval; /* first failed connection - * retry... */ + * retry... + */ int *kib_max_reconnect_interval; /* ...exponentially increasing - * to this */ + * to this + */ int *kib_cksum; /* checksum kib_msg_t? */ int *kib_timeout; /* comms timeout (seconds) */ int *kib_keepalive; /* keepalive timeout (seconds) */ @@ -92,9 +94,11 @@ typedef struct { int *kib_credits; /* # concurrent sends */ int *kib_peertxcredits; /* # concurrent sends to 1 peer */ int *kib_peerrtrcredits; /* # per-peer router buffer - * credits */ + * credits + */ int *kib_peercredits_hiw; /* # when eagerly to return - * credits */ + * credits + */ int *kib_peertimeout; /* seconds to consider peer dead */ char **kib_default_ipif; /* default IPoIB interface */ int *kib_retry_count; @@ -103,13 +107,15 @@ typedef struct { int *kib_ib_mtu; /* IB MTU */ int *kib_map_on_demand; /* map-on-demand if RD has more * fragments than this value, 0 - * disable map-on-demand */ + * disable map-on-demand + */ int *kib_fmr_pool_size; /* # FMRs in pool */ int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ int *kib_fmr_cache; /* enable FMR pool cache? */ int *kib_require_priv_port; /* accept only privileged ports */ int *kib_use_priv_port; /* use privileged port for active - * connect */ + * connect + */ int *kib_nscheds; /* # threads on each CPT */ } kib_tunables_t; @@ -200,7 +206,8 @@ typedef struct { int ibd_failed_failover; /* # failover failures */ unsigned int ibd_failover; /* failover in progress */ unsigned int ibd_can_failover; /* IPoIB interface is a bonding - * master */ + * master + */ struct list_head ibd_nets; struct kib_hca_dev *ibd_hdev; } kib_dev_t; @@ -250,7 +257,8 @@ typedef struct kib_poolset { struct list_head ps_pool_list; /* list of pools */ struct list_head ps_failed_pool_list;/* failed pool list */ unsigned long ps_next_retry; /* time stamp for retry if - * failed to allocate */ + * failed to allocate + */ int ps_increasing; /* is allocating new pool */ int ps_pool_size; /* new pool size */ int ps_cpt; /* CPT id */ @@ -258,7 +266,8 @@ typedef struct kib_poolset { kib_ps_pool_create_t ps_pool_create; /* create a new pool */ kib_ps_pool_destroy_t ps_pool_destroy; /* destroy a pool */ kib_ps_node_init_t ps_node_init; /* initialize new allocated - * node */ + * node + */ kib_ps_node_fini_t ps_node_fini; /* finalize node */ } kib_poolset_t; @@ -269,7 +278,8 @@ typedef struct kib_pool { unsigned long po_deadline; /* deadline of this pool */ int po_allocated; /* # of elements in use */ int po_failed; /* pool is created on failed - * HCA */ + * HCA + */ int po_size; /* # of pre-allocated elements */ } kib_pool_t; @@ -296,7 +306,8 @@ typedef struct { int fps_flush_trigger; int fps_increasing; /* is allocating new pool */ unsigned long fps_next_retry; /* time stamp for retry if - * failed to allocate */ + * failed to allocate + */ } kib_fmr_poolset_t; typedef struct { @@ -348,23 +359,29 @@ typedef struct { int kib_shutdown; /* shut down? */ struct list_head kib_devs; /* IB devices extant */ struct list_head kib_failed_devs; /* list head of failed - * devices */ + * devices + */ wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ atomic_t kib_nthreads; /* # live threads */ rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn - * ops */ + * ops + */ struct list_head *kib_peers; /* hash table of all my known - * peers */ + * peers + */ int kib_peer_hash_size; /* size of kib_peers */ void *kib_connd; /* the connd task * (serialisation assertions) */ struct list_head kib_connd_conns; /* connections to - * setup/teardown */ + * setup/teardown + */ struct list_head kib_connd_zombies; /* connections with zero - * refcount */ + * refcount + */ wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps - * here */ + * here + */ spinlock_t kib_connd_lock; /* serialise */ struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ struct kib_sched_info **kib_scheds; /* percpt data for schedulers @@ -481,9 +498,11 @@ typedef struct { #define IBLND_REJECT_CONN_UNCOMPAT 4 /* incompatible version peer */ #define IBLND_REJECT_CONN_STALE 5 /* stale peer */ #define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match - * mine */ + * mine + */ #define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't - * match mine */ + * match mine + */ /***********************************************************************/ @@ -492,7 +511,8 @@ typedef struct kib_rx /* receive message */ struct list_head rx_list; /* queue for attention */ struct kib_conn *rx_conn; /* owning conn */ int rx_nob; /* # bytes received (-1 while - * posted) */ + * posted) + */ enum ib_wc_status rx_status; /* completion status */ kib_msg_t *rx_msg; /* message buffer (host vaddr) */ __u64 rx_msgaddr; /* message buffer (I/O addr) */ @@ -505,12 +525,14 @@ typedef struct kib_rx /* receive message */ #define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ #define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ #define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give myself back 1 reserved - * credit */ + * credit + */ typedef struct kib_tx /* transmit message */ { struct list_head tx_list; /* queue on idle_txs ibc_tx_queue - * etc. */ + * etc. + */ kib_tx_pool_t *tx_pool; /* pool I'm from */ struct kib_conn *tx_conn; /* owning conn */ short tx_sending; /* # tx callbacks outstanding */ @@ -520,7 +542,8 @@ typedef struct kib_tx /* transmit message */ unsigned long tx_deadline; /* completion deadline */ __u64 tx_cookie; /* completion cookie */ lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on - * completion */ + * completion + */ kib_msg_t *tx_msg; /* message buffer (host vaddr) */ __u64 tx_msgaddr; /* message buffer (I/O addr) */ DECLARE_PCI_UNMAP_ADDR(tx_msgunmap); /* for dma_unmap_single() */ @@ -566,18 +589,22 @@ typedef struct kib_conn { * kiblnd_check_conns only */ struct list_head ibc_early_rxs; /* rxs completed before - * ESTABLISHED */ + * ESTABLISHED + */ struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for - * IBLND_MSG_VERSION_1 */ + * IBLND_MSG_VERSION_1 + */ struct list_head ibc_tx_queue; /* sends that need a credit */ struct list_head ibc_tx_queue_nocred; /* sends that don't need a - * credit */ + * credit + */ struct list_head ibc_tx_queue_rsrvd; /* sends that need to * reserve an ACK/DONE msg */ struct list_head ibc_active_txs; /* active tx awaiting - * completion */ + * completion + */ spinlock_t ibc_lock; /* serialise */ kib_rx_t *ibc_rxs; /* the rx descs */ kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ @@ -586,7 +613,8 @@ typedef struct kib_conn { struct ib_cq *ibc_cq; /* completion queue */ kib_connvars_t *ibc_connvars; /* in-progress connection - * state */ + * state + */ } kib_conn_t; #define IBLND_CONN_INIT 0 /* being initialised */ @@ -781,7 +809,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q) } /* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the - * lowest bits of the work request id to stash the work item type. */ + * lowest bits of the work request id to stash the work item type. + */ #define IBLND_WID_TX 0 #define IBLND_WID_RDMA 1 @@ -930,7 +959,8 @@ static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev, /* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly * right because OFED1.2 defines it as const, to use it we have to add - * (void *) cast to overcome "const" */ + * (void *) cast to overcome "const" + */ #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) -- 2.4.5 From simmonsja at ornl.gov Thu Aug 20 20:07:11 2015 From: simmonsja at ornl.gov (Simmons, James A.) Date: Thu, 20 Aug 2015 20:07:11 +0000 Subject: [lustre-devel] [PATCH] staging/lustre: fix block comment formatting In-Reply-To: <1440100898-37695-1-git-send-email-Jeffrey.C.Becker@nasa.gov> References: <1440100898-37695-1-git-send-email-Jeffrey.C.Becker@nasa.gov> Message-ID: >Running checkpatch.pl on lnet/klnds/o2iblnd/o2iblnd.h produces several >"Block comments use a trailing */ on a separate line" warnings. This patch >fixes these. These fixes are just plain ugly. Is this change a requirement? .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 90 ++++++++++++++-------- 1 file changed, 60 insertions(+), 30 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index f4b6c33..659d9cc 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -82,9 +82,11 @@ typedef struct { int *kib_dev_failover; /* HCA failover */ unsigned int *kib_service; /* IB service number */ int *kib_min_reconnect_interval; /* first failed connection - * retry... */ + * retry... + */ int *kib_max_reconnect_interval; /* ...exponentially increasing - * to this */ + * to this + */ int *kib_cksum; /* checksum kib_msg_t? */ int *kib_timeout; /* comms timeout (seconds) */ int *kib_keepalive; /* keepalive timeout (seconds) */ @@ -92,9 +94,11 @@ typedef struct { int *kib_credits; /* # concurrent sends */ int *kib_peertxcredits; /* # concurrent sends to 1 peer */ int *kib_peerrtrcredits; /* # per-peer router buffer - * credits */ + * credits + */ int *kib_peercredits_hiw; /* # when eagerly to return - * credits */ + * credits + */ int *kib_peertimeout; /* seconds to consider peer dead */ char **kib_default_ipif; /* default IPoIB interface */ int *kib_retry_count; @@ -103,13 +107,15 @@ typedef struct { int *kib_ib_mtu; /* IB MTU */ int *kib_map_on_demand; /* map-on-demand if RD has more * fragments than this value, 0 - * disable map-on-demand */ + * disable map-on-demand + */ int *kib_fmr_pool_size; /* # FMRs in pool */ int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ int *kib_fmr_cache; /* enable FMR pool cache? */ int *kib_require_priv_port; /* accept only privileged ports */ int *kib_use_priv_port; /* use privileged port for active - * connect */ + * connect + */ int *kib_nscheds; /* # threads on each CPT */ } kib_tunables_t; @@ -200,7 +206,8 @@ typedef struct { int ibd_failed_failover; /* # failover failures */ unsigned int ibd_failover; /* failover in progress */ unsigned int ibd_can_failover; /* IPoIB interface is a bonding - * master */ + * master + */ struct list_head ibd_nets; struct kib_hca_dev *ibd_hdev; } kib_dev_t; @@ -250,7 +257,8 @@ typedef struct kib_poolset { struct list_head ps_pool_list; /* list of pools */ struct list_head ps_failed_pool_list;/* failed pool list */ unsigned long ps_next_retry; /* time stamp for retry if - * failed to allocate */ + * failed to allocate + */ int ps_increasing; /* is allocating new pool */ int ps_pool_size; /* new pool size */ int ps_cpt; /* CPT id */ @@ -258,7 +266,8 @@ typedef struct kib_poolset { kib_ps_pool_create_t ps_pool_create; /* create a new pool */ kib_ps_pool_destroy_t ps_pool_destroy; /* destroy a pool */ kib_ps_node_init_t ps_node_init; /* initialize new allocated - * node */ + * node + */ kib_ps_node_fini_t ps_node_fini; /* finalize node */ } kib_poolset_t; @@ -269,7 +278,8 @@ typedef struct kib_pool { unsigned long po_deadline; /* deadline of this pool */ int po_allocated; /* # of elements in use */ int po_failed; /* pool is created on failed - * HCA */ + * HCA + */ int po_size; /* # of pre-allocated elements */ } kib_pool_t; @@ -296,7 +306,8 @@ typedef struct { int fps_flush_trigger; int fps_increasing; /* is allocating new pool */ unsigned long fps_next_retry; /* time stamp for retry if - * failed to allocate */ + * failed to allocate + */ } kib_fmr_poolset_t; typedef struct { @@ -348,23 +359,29 @@ typedef struct { int kib_shutdown; /* shut down? */ struct list_head kib_devs; /* IB devices extant */ struct list_head kib_failed_devs; /* list head of failed - * devices */ + * devices + */ wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ atomic_t kib_nthreads; /* # live threads */ rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn - * ops */ + * ops + */ struct list_head *kib_peers; /* hash table of all my known - * peers */ + * peers + */ int kib_peer_hash_size; /* size of kib_peers */ void *kib_connd; /* the connd task * (serialisation assertions) */ struct list_head kib_connd_conns; /* connections to - * setup/teardown */ + * setup/teardown + */ struct list_head kib_connd_zombies; /* connections with zero - * refcount */ + * refcount + */ wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps - * here */ + * here + */ spinlock_t kib_connd_lock; /* serialise */ struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ struct kib_sched_info **kib_scheds; /* percpt data for schedulers @@ -481,9 +498,11 @@ typedef struct { #define IBLND_REJECT_CONN_UNCOMPAT 4 /* incompatible version peer */ #define IBLND_REJECT_CONN_STALE 5 /* stale peer */ #define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match - * mine */ + * mine + */ #define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't - * match mine */ + * match mine + */ /***********************************************************************/ @@ -492,7 +511,8 @@ typedef struct kib_rx /* receive message */ struct list_head rx_list; /* queue for attention */ struct kib_conn *rx_conn; /* owning conn */ int rx_nob; /* # bytes received (-1 while - * posted) */ + * posted) + */ enum ib_wc_status rx_status; /* completion status */ kib_msg_t *rx_msg; /* message buffer (host vaddr) */ __u64 rx_msgaddr; /* message buffer (I/O addr) */ @@ -505,12 +525,14 @@ typedef struct kib_rx /* receive message */ #define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ #define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ #define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give myself back 1 reserved - * credit */ + * credit + */ typedef struct kib_tx /* transmit message */ { struct list_head tx_list; /* queue on idle_txs ibc_tx_queue - * etc. */ + * etc. + */ kib_tx_pool_t *tx_pool; /* pool I'm from */ struct kib_conn *tx_conn; /* owning conn */ short tx_sending; /* # tx callbacks outstanding */ @@ -520,7 +542,8 @@ typedef struct kib_tx /* transmit message */ unsigned long tx_deadline; /* completion deadline */ __u64 tx_cookie; /* completion cookie */ lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on - * completion */ + * completion + */ kib_msg_t *tx_msg; /* message buffer (host vaddr) */ __u64 tx_msgaddr; /* message buffer (I/O addr) */ DECLARE_PCI_UNMAP_ADDR(tx_msgunmap); /* for dma_unmap_single() */ @@ -566,18 +589,22 @@ typedef struct kib_conn { * kiblnd_check_conns only */ struct list_head ibc_early_rxs; /* rxs completed before - * ESTABLISHED */ + * ESTABLISHED + */ struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for - * IBLND_MSG_VERSION_1 */ + * IBLND_MSG_VERSION_1 + */ struct list_head ibc_tx_queue; /* sends that need a credit */ struct list_head ibc_tx_queue_nocred; /* sends that don't need a - * credit */ + * credit + */ struct list_head ibc_tx_queue_rsrvd; /* sends that need to * reserve an ACK/DONE msg */ struct list_head ibc_active_txs; /* active tx awaiting - * completion */ + * completion + */ spinlock_t ibc_lock; /* serialise */ kib_rx_t *ibc_rxs; /* the rx descs */ kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ @@ -586,7 +613,8 @@ typedef struct kib_conn { struct ib_cq *ibc_cq; /* completion queue */ kib_connvars_t *ibc_connvars; /* in-progress connection - * state */ + * state + */ } kib_conn_t; #define IBLND_CONN_INIT 0 /* being initialised */ @@ -781,7 +809,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q) } /* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the - * lowest bits of the work request id to stash the work item type. */ + * lowest bits of the work request id to stash the work item type. + */ #define IBLND_WID_TX 0 #define IBLND_WID_RDMA 1 @@ -930,7 +959,8 @@ static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev, /* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly * right because OFED1.2 defines it as const, to use it we have to add - * (void *) cast to overcome "const" */ + * (void *) cast to overcome "const" + */ #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) -- 2.4.5 _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From bevans at cray.com Thu Aug 20 20:45:16 2015 From: bevans at cray.com (Ben Evans) Date: Thu, 20 Aug 2015 15:45:16 -0500 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans Message-ID: <20150820204516.GA26830@lusbld01.us.cray.com> I'm with James on the previous patch being ugly. Something like this is much cleaner. My opinion may not necessarily align with anyone elses. --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 +++++++++----------- 1 files changed, 26 insertions(+), 32 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index f5d1d9f..cc4872d 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -79,39 +79,33 @@ #define IBLND_N_SCHED_HIGH 4 typedef struct { - int *kib_dev_failover; /* HCA failover */ + int *kib_dev_failover; /* HCA failover */ unsigned int *kib_service; /* IB service number */ - int *kib_min_reconnect_interval; /* first failed connection - * retry... */ - int *kib_max_reconnect_interval; /* ...exponentially increasing - * to this */ - int *kib_cksum; /* checksum kib_msg_t? */ - int *kib_timeout; /* comms timeout (seconds) */ - int *kib_keepalive; /* keepalive timeout (seconds) */ - int *kib_ntx; /* # tx descs */ - int *kib_credits; /* # concurrent sends */ - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ - int *kib_peerrtrcredits; /* # per-peer router buffer - * credits */ - int *kib_peercredits_hiw; /* # when eagerly to return - * credits */ - int *kib_peertimeout; /* seconds to consider peer dead */ - char **kib_default_ipif; /* default IPoIB interface */ - int *kib_retry_count; - int *kib_rnr_retry_count; - int *kib_concurrent_sends; /* send work queue sizing */ - int *kib_ib_mtu; /* IB MTU */ - int *kib_map_on_demand; /* map-on-demand if RD has more - * fragments than this value, 0 - * disable map-on-demand */ - int *kib_pmr_pool_size; /* # physical MR in pool */ - int *kib_fmr_pool_size; /* # FMRs in pool */ - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ - int *kib_fmr_cache; /* enable FMR pool cache? */ - int *kib_require_priv_port; /* accept only privileged ports */ - int *kib_use_priv_port; /* use privileged port for active - * connect */ - int *kib_nscheds; /* # threads on each CPT */ + int *kib_min_reconnect_interval; /* first failed connection retry */ + int *kib_max_reconnect_interval; /* exponentially increase to this */ + int *kib_cksum; /* checksum kib_msg_t? */ + int *kib_timeout; /* comms timeout (seconds) */ + int *kib_keepalive; /* keepalive timeout (seconds) */ + int *kib_ntx; /* # tx descs */ + int *kib_credits; /* # concurrent sends */ + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ + int *kib_peercredits_hiw; /* # when eagerly to return credits */ + int *kib_peertimeout; /* seconds to consider peer dead */ + char **kib_default_ipif; /* default IPoIB interface */ + int *kib_retry_count; + int *kib_rnr_retry_count; + int *kib_concurrent_sends; /* send work queue sizing */ + int *kib_ib_mtu; /* IB MTU */ + int *kib_map_on_demand; /* map-on-demand if RD has more fragments + than this value, 0=disable */ + int *kib_pmr_pool_size; /* # physical MR in pool */ + int *kib_fmr_pool_size; /* # FMRs in pool */ + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ + int *kib_fmr_cache; /* enable FMR pool cache? */ + int *kib_require_priv_port; /* accept only privileged ports */ + int *kib_use_priv_port; /* use priv. port for active connect */ + int *kib_nscheds; /* # threads on each CPT */ } kib_tunables_t; extern kib_tunables_t kiblnd_tunables; -- 1.6.5.6 From jeffrey.c.becker at nasa.gov Thu Aug 20 21:39:41 2015 From: jeffrey.c.becker at nasa.gov (Jeff Becker) Date: Thu, 20 Aug 2015 14:39:41 -0700 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans In-Reply-To: <20150820204516.GA26830@lusbld01.us.cray.com> References: <20150820204516.GA26830@lusbld01.us.cray.com> Message-ID: <55D6491D.5080702@nasa.gov> On 08/20/2015 01:45 PM, Ben Evans wrote: > I'm with James on the previous patch being ugly. Something like this is much cleaner. My opinion may not necessarily align with anyone elses. I am happy to resubmit following this idea, if that is preferred. Thanks. -jeff > --- > .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 +++++++++----------- > 1 files changed, 26 insertions(+), 32 deletions(-) > > diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h > index f5d1d9f..cc4872d 100644 > --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h > +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h > @@ -79,39 +79,33 @@ > #define IBLND_N_SCHED_HIGH 4 > > typedef struct { > - int *kib_dev_failover; /* HCA failover */ > + int *kib_dev_failover; /* HCA failover */ > unsigned int *kib_service; /* IB service number */ > - int *kib_min_reconnect_interval; /* first failed connection > - * retry... */ > - int *kib_max_reconnect_interval; /* ...exponentially increasing > - * to this */ > - int *kib_cksum; /* checksum kib_msg_t? */ > - int *kib_timeout; /* comms timeout (seconds) */ > - int *kib_keepalive; /* keepalive timeout (seconds) */ > - int *kib_ntx; /* # tx descs */ > - int *kib_credits; /* # concurrent sends */ > - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ > - int *kib_peerrtrcredits; /* # per-peer router buffer > - * credits */ > - int *kib_peercredits_hiw; /* # when eagerly to return > - * credits */ > - int *kib_peertimeout; /* seconds to consider peer dead */ > - char **kib_default_ipif; /* default IPoIB interface */ > - int *kib_retry_count; > - int *kib_rnr_retry_count; > - int *kib_concurrent_sends; /* send work queue sizing */ > - int *kib_ib_mtu; /* IB MTU */ > - int *kib_map_on_demand; /* map-on-demand if RD has more > - * fragments than this value, 0 > - * disable map-on-demand */ > - int *kib_pmr_pool_size; /* # physical MR in pool */ > - int *kib_fmr_pool_size; /* # FMRs in pool */ > - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ > - int *kib_fmr_cache; /* enable FMR pool cache? */ > - int *kib_require_priv_port; /* accept only privileged ports */ > - int *kib_use_priv_port; /* use privileged port for active > - * connect */ > - int *kib_nscheds; /* # threads on each CPT */ > + int *kib_min_reconnect_interval; /* first failed connection retry */ > + int *kib_max_reconnect_interval; /* exponentially increase to this */ > + int *kib_cksum; /* checksum kib_msg_t? */ > + int *kib_timeout; /* comms timeout (seconds) */ > + int *kib_keepalive; /* keepalive timeout (seconds) */ > + int *kib_ntx; /* # tx descs */ > + int *kib_credits; /* # concurrent sends */ > + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ > + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ > + int *kib_peercredits_hiw; /* # when eagerly to return credits */ > + int *kib_peertimeout; /* seconds to consider peer dead */ > + char **kib_default_ipif; /* default IPoIB interface */ > + int *kib_retry_count; > + int *kib_rnr_retry_count; > + int *kib_concurrent_sends; /* send work queue sizing */ > + int *kib_ib_mtu; /* IB MTU */ > + int *kib_map_on_demand; /* map-on-demand if RD has more fragments > + than this value, 0=disable */ > + int *kib_pmr_pool_size; /* # physical MR in pool */ > + int *kib_fmr_pool_size; /* # FMRs in pool */ > + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ > + int *kib_fmr_cache; /* enable FMR pool cache? */ > + int *kib_require_priv_port; /* accept only privileged ports */ > + int *kib_use_priv_port; /* use priv. port for active connect */ > + int *kib_nscheds; /* # threads on each CPT */ > } kib_tunables_t; > > extern kib_tunables_t kiblnd_tunables; From bevans at cray.com Fri Aug 21 13:57:37 2015 From: bevans at cray.com (Ben Evans) Date: Fri, 21 Aug 2015 13:57:37 +0000 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans In-Reply-To: <55D6491D.5080702@nasa.gov> References: <20150820204516.GA26830@lusbld01.us.cray.com> <55D6491D.5080702@nasa.gov> Message-ID: I¹d get feedback from other parties first. On 8/20/15, 5:39 PM, "lustre-devel on behalf of Jeff Becker" wrote: >On 08/20/2015 01:45 PM, Ben Evans wrote: >> I'm with James on the previous patch being ugly. Something like this >>is much cleaner. My opinion may not necessarily align with anyone elses. > >I am happy to resubmit following this idea, if that is preferred. Thanks. > >-jeff >> --- >> .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 >>+++++++++----------- >> 1 files changed, 26 insertions(+), 32 deletions(-) >> >> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >> index f5d1d9f..cc4872d 100644 >> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >> @@ -79,39 +79,33 @@ >> #define IBLND_N_SCHED_HIGH 4 >> >> typedef struct { >> - int *kib_dev_failover; /* HCA failover */ >> + int *kib_dev_failover; /* HCA failover */ >> unsigned int *kib_service; /* IB service number */ >> - int *kib_min_reconnect_interval; /* first failed connection >> - * retry... */ >> - int *kib_max_reconnect_interval; /* ...exponentially >>increasing >> - * to this */ >> - int *kib_cksum; /* checksum kib_msg_t? */ >> - int *kib_timeout; /* comms timeout (seconds) */ >> - int *kib_keepalive; /* keepalive timeout (seconds) */ >> - int *kib_ntx; /* # tx descs */ >> - int *kib_credits; /* # concurrent sends */ >> - int *kib_peertxcredits; /* # concurrent sends to 1 peer >>*/ >> - int *kib_peerrtrcredits; /* # per-peer router buffer >> - * credits */ >> - int *kib_peercredits_hiw; /* # when eagerly to return >> - * credits */ >> - int *kib_peertimeout; /* seconds to consider peer dead >>*/ >> - char **kib_default_ipif; /* default IPoIB interface */ >> - int *kib_retry_count; >> - int *kib_rnr_retry_count; >> - int *kib_concurrent_sends; /* send work queue sizing */ >> - int *kib_ib_mtu; /* IB MTU */ >> - int *kib_map_on_demand; /* map-on-demand if RD has more >> - * fragments than this value, 0 >> - * disable map-on-demand */ >> - int *kib_pmr_pool_size; /* # physical MR in pool */ >> - int *kib_fmr_pool_size; /* # FMRs in pool */ >> - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >> - int *kib_fmr_cache; /* enable FMR pool cache? */ >> - int *kib_require_priv_port; /* accept only privileged ports >>*/ >> - int *kib_use_priv_port; /* use privileged port for active >> - * connect */ >> - int *kib_nscheds; /* # threads on each CPT */ >> + int *kib_min_reconnect_interval; /* first failed connection retry */ >> + int *kib_max_reconnect_interval; /* exponentially increase to this */ >> + int *kib_cksum; /* checksum kib_msg_t? */ >> + int *kib_timeout; /* comms timeout (seconds) */ >> + int *kib_keepalive; /* keepalive timeout (seconds) */ >> + int *kib_ntx; /* # tx descs */ >> + int *kib_credits; /* # concurrent sends */ >> + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >> + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ >> + int *kib_peercredits_hiw; /* # when eagerly to return credits */ >> + int *kib_peertimeout; /* seconds to consider peer dead */ >> + char **kib_default_ipif; /* default IPoIB interface */ >> + int *kib_retry_count; >> + int *kib_rnr_retry_count; >> + int *kib_concurrent_sends; /* send work queue sizing */ >> + int *kib_ib_mtu; /* IB MTU */ >> + int *kib_map_on_demand; /* map-on-demand if RD has more fragments >> + than this value, 0=disable */ >> + int *kib_pmr_pool_size; /* # physical MR in pool */ >> + int *kib_fmr_pool_size; /* # FMRs in pool */ >> + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >> + int *kib_fmr_cache; /* enable FMR pool cache? */ >> + int *kib_require_priv_port; /* accept only privileged ports */ >> + int *kib_use_priv_port; /* use priv. port for active connect */ >> + int *kib_nscheds; /* # threads on each CPT */ >> } kib_tunables_t; >> >> extern kib_tunables_t kiblnd_tunables; > >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From andreas.dilger at intel.com Fri Aug 21 20:53:13 2015 From: andreas.dilger at intel.com (Dilger, Andreas) Date: Fri, 21 Aug 2015 20:53:13 +0000 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans In-Reply-To: <20150820204516.GA26830@lusbld01.us.cray.com> References: <20150820204516.GA26830@lusbld01.us.cray.com> Message-ID: On 2015/08/20, 2:45 PM, "Ben Evans " wrote: >I'm with James on the previous patch being ugly. Something like this is >much cleaner. My opinion may not necessarily align with anyone elses. I'd agree this is much better than the other patch, but you need to send this to Greg KH and fix the summary line for it to be accepted instead. Cheers, Andreas >--- > .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 >+++++++++----------- > 1 files changed, 26 insertions(+), 32 deletions(-) > >diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >index f5d1d9f..cc4872d 100644 >--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >@@ -79,39 +79,33 @@ > #define IBLND_N_SCHED_HIGH 4 > > typedef struct { >- int *kib_dev_failover; /* HCA failover */ >+ int *kib_dev_failover; /* HCA failover */ > unsigned int *kib_service; /* IB service number */ >- int *kib_min_reconnect_interval; /* first failed connection >- * retry... */ >- int *kib_max_reconnect_interval; /* ...exponentially increasing >- * to this */ >- int *kib_cksum; /* checksum kib_msg_t? */ >- int *kib_timeout; /* comms timeout (seconds) */ >- int *kib_keepalive; /* keepalive timeout (seconds) */ >- int *kib_ntx; /* # tx descs */ >- int *kib_credits; /* # concurrent sends */ >- int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >- int *kib_peerrtrcredits; /* # per-peer router buffer >- * credits */ >- int *kib_peercredits_hiw; /* # when eagerly to return >- * credits */ >- int *kib_peertimeout; /* seconds to consider peer dead */ >- char **kib_default_ipif; /* default IPoIB interface */ >- int *kib_retry_count; >- int *kib_rnr_retry_count; >- int *kib_concurrent_sends; /* send work queue sizing */ >- int *kib_ib_mtu; /* IB MTU */ >- int *kib_map_on_demand; /* map-on-demand if RD has more >- * fragments than this value, 0 >- * disable map-on-demand */ >- int *kib_pmr_pool_size; /* # physical MR in pool */ >- int *kib_fmr_pool_size; /* # FMRs in pool */ >- int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >- int *kib_fmr_cache; /* enable FMR pool cache? */ >- int *kib_require_priv_port; /* accept only privileged ports */ >- int *kib_use_priv_port; /* use privileged port for active >- * connect */ >- int *kib_nscheds; /* # threads on each CPT */ >+ int *kib_min_reconnect_interval; /* first failed connection retry */ >+ int *kib_max_reconnect_interval; /* exponentially increase to this */ >+ int *kib_cksum; /* checksum kib_msg_t? */ >+ int *kib_timeout; /* comms timeout (seconds) */ >+ int *kib_keepalive; /* keepalive timeout (seconds) */ >+ int *kib_ntx; /* # tx descs */ >+ int *kib_credits; /* # concurrent sends */ >+ int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >+ int *kib_peerrtrcredits; /* # per-peer router buffer credits */ >+ int *kib_peercredits_hiw; /* # when eagerly to return credits */ >+ int *kib_peertimeout; /* seconds to consider peer dead */ >+ char **kib_default_ipif; /* default IPoIB interface */ >+ int *kib_retry_count; >+ int *kib_rnr_retry_count; >+ int *kib_concurrent_sends; /* send work queue sizing */ >+ int *kib_ib_mtu; /* IB MTU */ >+ int *kib_map_on_demand; /* map-on-demand if RD has more fragments >+ than this value, 0=disable */ >+ int *kib_pmr_pool_size; /* # physical MR in pool */ >+ int *kib_fmr_pool_size; /* # FMRs in pool */ >+ int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >+ int *kib_fmr_cache; /* enable FMR pool cache? */ >+ int *kib_require_priv_port; /* accept only privileged ports */ >+ int *kib_use_priv_port; /* use priv. port for active connect */ >+ int *kib_nscheds; /* # threads on each CPT */ > } kib_tunables_t; > > extern kib_tunables_t kiblnd_tunables; >-- >1.6.5.6 > >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division From bevans at cray.com Fri Aug 21 20:57:09 2015 From: bevans at cray.com (Ben Evans) Date: Fri, 21 Aug 2015 20:57:09 +0000 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans In-Reply-To: References: <20150820204516.GA26830@lusbld01.us.cray.com> Message-ID: Jeff, would you like to continue the patch along the lines I provided, or would you prefer I submit the patch, then you follow up? -Ben On 8/21/15, 4:53 PM, "Dilger, Andreas" wrote: >On 2015/08/20, 2:45 PM, "Ben Evans " wrote: > >>I'm with James on the previous patch being ugly. Something like this is >>much cleaner. My opinion may not necessarily align with anyone elses. > >I'd agree this is much better than the other patch, but you need to send >this to Greg KH and fix the summary line for it to be accepted instead. > >Cheers, Andreas > >>--- >> .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 >>+++++++++----------- >> 1 files changed, 26 insertions(+), 32 deletions(-) >> >>diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>index f5d1d9f..cc4872d 100644 >>--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>@@ -79,39 +79,33 @@ >> #define IBLND_N_SCHED_HIGH 4 >> >> typedef struct { >>- int *kib_dev_failover; /* HCA failover */ >>+ int *kib_dev_failover; /* HCA failover */ >> unsigned int *kib_service; /* IB service number */ >>- int *kib_min_reconnect_interval; /* first failed connection >>- * retry... */ >>- int *kib_max_reconnect_interval; /* ...exponentially >>increasing >>- * to this */ >>- int *kib_cksum; /* checksum kib_msg_t? */ >>- int *kib_timeout; /* comms timeout (seconds) */ >>- int *kib_keepalive; /* keepalive timeout (seconds) */ >>- int *kib_ntx; /* # tx descs */ >>- int *kib_credits; /* # concurrent sends */ >>- int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >>- int *kib_peerrtrcredits; /* # per-peer router buffer >>- * credits */ >>- int *kib_peercredits_hiw; /* # when eagerly to return >>- * credits */ >>- int *kib_peertimeout; /* seconds to consider peer dead >>*/ >>- char **kib_default_ipif; /* default IPoIB interface */ >>- int *kib_retry_count; >>- int *kib_rnr_retry_count; >>- int *kib_concurrent_sends; /* send work queue sizing */ >>- int *kib_ib_mtu; /* IB MTU */ >>- int *kib_map_on_demand; /* map-on-demand if RD has more >>- * fragments than this value, 0 >>- * disable map-on-demand */ >>- int *kib_pmr_pool_size; /* # physical MR in pool */ >>- int *kib_fmr_pool_size; /* # FMRs in pool */ >>- int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >>- int *kib_fmr_cache; /* enable FMR pool cache? */ >>- int *kib_require_priv_port; /* accept only privileged ports */ >>- int *kib_use_priv_port; /* use privileged port for active >>- * connect */ >>- int *kib_nscheds; /* # threads on each CPT */ >>+ int *kib_min_reconnect_interval; /* first failed connection retry */ >>+ int *kib_max_reconnect_interval; /* exponentially increase to this */ >>+ int *kib_cksum; /* checksum kib_msg_t? */ >>+ int *kib_timeout; /* comms timeout (seconds) */ >>+ int *kib_keepalive; /* keepalive timeout (seconds) */ >>+ int *kib_ntx; /* # tx descs */ >>+ int *kib_credits; /* # concurrent sends */ >>+ int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >>+ int *kib_peerrtrcredits; /* # per-peer router buffer credits */ >>+ int *kib_peercredits_hiw; /* # when eagerly to return credits */ >>+ int *kib_peertimeout; /* seconds to consider peer dead */ >>+ char **kib_default_ipif; /* default IPoIB interface */ >>+ int *kib_retry_count; >>+ int *kib_rnr_retry_count; >>+ int *kib_concurrent_sends; /* send work queue sizing */ >>+ int *kib_ib_mtu; /* IB MTU */ >>+ int *kib_map_on_demand; /* map-on-demand if RD has more fragments >>+ than this value, 0=disable */ >>+ int *kib_pmr_pool_size; /* # physical MR in pool */ >>+ int *kib_fmr_pool_size; /* # FMRs in pool */ >>+ int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >>+ int *kib_fmr_cache; /* enable FMR pool cache? */ >>+ int *kib_require_priv_port; /* accept only privileged ports */ >>+ int *kib_use_priv_port; /* use priv. port for active connect */ >>+ int *kib_nscheds; /* # threads on each CPT */ >> } kib_tunables_t; >> >> extern kib_tunables_t kiblnd_tunables; >>-- >>1.6.5.6 >> >>_______________________________________________ >>lustre-devel mailing list >>lustre-devel at lists.lustre.org >>http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >> > > >Cheers, Andreas >-- >Andreas Dilger > >Lustre Software Architect >Intel High Performance Data Division > > From jeffrey.c.becker at nasa.gov Fri Aug 21 21:02:46 2015 From: jeffrey.c.becker at nasa.gov (Jeff Becker) Date: Fri, 21 Aug 2015 14:02:46 -0700 Subject: [lustre-devel] [PATCH] Example of better reformatting Signed-off-by: Ben Evans In-Reply-To: References: <20150820204516.GA26830@lusbld01.us.cray.com> Message-ID: <55D791F6.6080802@nasa.gov> Hi Ben, On 08/21/2015 01:57 PM, Ben Evans wrote: > Jeff, would you like to continue the patch along the lines I provided, or > would you prefer I submit the patch, then you follow up? Let me give it another try. Thanks. -jeff > > -Ben > > On 8/21/15, 4:53 PM, "Dilger, Andreas" wrote: > >> On 2015/08/20, 2:45 PM, "Ben Evans " wrote: >> >>> I'm with James on the previous patch being ugly. Something like this is >>> much cleaner. My opinion may not necessarily align with anyone elses. >> I'd agree this is much better than the other patch, but you need to send >> this to Greg KH and fix the summary line for it to be accepted instead. >> >> Cheers, Andreas >> >>> --- >>> .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 58 >>> +++++++++----------- >>> 1 files changed, 26 insertions(+), 32 deletions(-) >>> >>> diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>> b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>> index f5d1d9f..cc4872d 100644 >>> --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>> +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >>> @@ -79,39 +79,33 @@ >>> #define IBLND_N_SCHED_HIGH 4 >>> >>> typedef struct { >>> - int *kib_dev_failover; /* HCA failover */ >>> + int *kib_dev_failover; /* HCA failover */ >>> unsigned int *kib_service; /* IB service number */ >>> - int *kib_min_reconnect_interval; /* first failed connection >>> - * retry... */ >>> - int *kib_max_reconnect_interval; /* ...exponentially >>> increasing >>> - * to this */ >>> - int *kib_cksum; /* checksum kib_msg_t? */ >>> - int *kib_timeout; /* comms timeout (seconds) */ >>> - int *kib_keepalive; /* keepalive timeout (seconds) */ >>> - int *kib_ntx; /* # tx descs */ >>> - int *kib_credits; /* # concurrent sends */ >>> - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >>> - int *kib_peerrtrcredits; /* # per-peer router buffer >>> - * credits */ >>> - int *kib_peercredits_hiw; /* # when eagerly to return >>> - * credits */ >>> - int *kib_peertimeout; /* seconds to consider peer dead >>> */ >>> - char **kib_default_ipif; /* default IPoIB interface */ >>> - int *kib_retry_count; >>> - int *kib_rnr_retry_count; >>> - int *kib_concurrent_sends; /* send work queue sizing */ >>> - int *kib_ib_mtu; /* IB MTU */ >>> - int *kib_map_on_demand; /* map-on-demand if RD has more >>> - * fragments than this value, 0 >>> - * disable map-on-demand */ >>> - int *kib_pmr_pool_size; /* # physical MR in pool */ >>> - int *kib_fmr_pool_size; /* # FMRs in pool */ >>> - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >>> - int *kib_fmr_cache; /* enable FMR pool cache? */ >>> - int *kib_require_priv_port; /* accept only privileged ports */ >>> - int *kib_use_priv_port; /* use privileged port for active >>> - * connect */ >>> - int *kib_nscheds; /* # threads on each CPT */ >>> + int *kib_min_reconnect_interval; /* first failed connection retry */ >>> + int *kib_max_reconnect_interval; /* exponentially increase to this */ >>> + int *kib_cksum; /* checksum kib_msg_t? */ >>> + int *kib_timeout; /* comms timeout (seconds) */ >>> + int *kib_keepalive; /* keepalive timeout (seconds) */ >>> + int *kib_ntx; /* # tx descs */ >>> + int *kib_credits; /* # concurrent sends */ >>> + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >>> + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ >>> + int *kib_peercredits_hiw; /* # when eagerly to return credits */ >>> + int *kib_peertimeout; /* seconds to consider peer dead */ >>> + char **kib_default_ipif; /* default IPoIB interface */ >>> + int *kib_retry_count; >>> + int *kib_rnr_retry_count; >>> + int *kib_concurrent_sends; /* send work queue sizing */ >>> + int *kib_ib_mtu; /* IB MTU */ >>> + int *kib_map_on_demand; /* map-on-demand if RD has more fragments >>> + than this value, 0=disable */ >>> + int *kib_pmr_pool_size; /* # physical MR in pool */ >>> + int *kib_fmr_pool_size; /* # FMRs in pool */ >>> + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >>> + int *kib_fmr_cache; /* enable FMR pool cache? */ >>> + int *kib_require_priv_port; /* accept only privileged ports */ >>> + int *kib_use_priv_port; /* use priv. port for active connect */ >>> + int *kib_nscheds; /* # threads on each CPT */ >>> } kib_tunables_t; >>> >>> extern kib_tunables_t kiblnd_tunables; >>> -- >>> 1.6.5.6 >>> >>> _______________________________________________ >>> lustre-devel mailing list >>> lustre-devel at lists.lustre.org >>> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org >>> >> >> Cheers, Andreas >> -- >> Andreas Dilger >> >> Lustre Software Architect >> Intel High Performance Data Division >> >> > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From dinatale2 at llnl.gov Fri Aug 21 22:05:04 2015 From: dinatale2 at llnl.gov (Di Natale, Giuseppe) Date: Fri, 21 Aug 2015 22:05:04 +0000 Subject: [lustre-devel] lctl conf_param vs set_param -P Message-ID: <974262C966B9F640899A48EB96313E9A2B92D7@PRDEXMBX-08.the-lab.llnl.gov> I just filed https://jira.hpdd.intel.com/browse/LU-7031 to keep track of the discussion on what to do regarding set_param -P so we don't lose it in the email chain. Giuseppe ________________________________________ From: lustre-devel [lustre-devel-bounces at lists.lustre.org] on behalf of lustre-devel-request at lists.lustre.org [lustre-devel-request at lists.lustre.org] Sent: Saturday, August 15, 2015 1:04 PM To: lustre-devel at lists.lustre.org Subject: lustre-devel Digest, Vol 103, Issue 2 Send lustre-devel mailing list submissions to lustre-devel at lists.lustre.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org or, via email, send a message with subject or body 'help' to lustre-devel-request at lists.lustre.org You can reach the person managing the list at lustre-devel-owner at lists.lustre.org When replying, please edit your Subject line so it is more specific than "Re: Contents of lustre-devel digest..." Today's Topics: 1. Re: lctl conf_param vs set_param -P (Christopher J. Morrone) 2. Re: lctl conf_param vs set_param -P (Dilger, Andreas) 3. Re: lctl conf_param vs set_param -P (Christopher J. Morrone) ---------------------------------------------------------------------- Message: 1 Date: Fri, 14 Aug 2015 13:41:03 -0700 From: "Christopher J. Morrone" To: lustre-devel at lists.lustre.org Subject: Re: [lustre-devel] lctl conf_param vs set_param -P Message-ID: <55CE525F.3020505 at llnl.gov> Content-Type: text/plain; charset=windows-1252; format=flowed I too am confused. And a bit dismayed that there is so little in the way of code comments to explain the intent. Is it the intent of "set_param -P" that the specified changes only take effect after components are restarted? And if so, why? How would a normal system administrator go about finding out what settings are currently set permanently? I read through LU-3155 and see the discussion about using a single llog file for all nodes, so I will withhold comment about that for now. From a user-interface standpoint though, presenting a single namespace for all nodes in the entire center seems like less than desirable choice. Might we not want to set settings differently on different clusters (be they client or server clusters)? Given that not all paths under /proc have differentiating strings in their path, there are some things that can only be set completely globally in this design. And what about sites that use an MGS per filesystem, rather than a single MGS for the entire site? If one MGS says that the debug level should be one value, and another says that the debug value should be another value, is it entirely random which debug setting will appear on any given node? Chris On 08/13/2015 03:43 PM, Di Natale, Giuseppe wrote: > Greetings, > > In an effort to change test-framework.sh to not utilize the deprecated > conf_param option in lctl, I stumbled upon what appears to be > inconsistent behavior between lctl's conf_param and set_param -P > options. The permanent option test-framework.sh is attempting to change > is jobid_var. When using conf_param, any changes to the property are > written to /proc/fs/lustre/jobid_var within a short period of time. This > is not the case with set_param -P. The change is never reflected nor is > it stored in some other file within /proc. I started digging into the > MGS logs and found that the behavior for both are different (the > relevant segments of the logs are attached to this email and are named > accordingly). In short, it appears that conf_param attempts to apply the > changes to all the targets while set_param does not (it does not > recognize it as a global property). Can someone offer any insight on why > the behavior appears to be different or provide insight on if this is > incorrect behavior? > > Thanks, > Giuseppe Di Natale > > > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > ------------------------------ Message: 2 Date: Sat, 15 Aug 2015 00:07:22 +0000 From: "Dilger, Andreas" To: "Di Natale, Giuseppe" Cc: "lustre-devel at lists.lustre.org" Subject: Re: [lustre-devel] lctl conf_param vs set_param -P Message-ID: Content-Type: text/plain; charset="us-ascii" Coincidentally (or maybe that was what drove your investigations?), I'd just filed https://jira.hpdd.intel.com/browse/LU-7004 about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8. Cheers, Andreas -- Andreas Dilger Lustre Software Architect Intel High Performance Data Division On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" on behalf of dinatale2 at llnl.gov> wrote: Greetings, In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavior? Thanks, Giuseppe Di Natale ------------------------------ Message: 3 Date: Fri, 14 Aug 2015 17:36:47 -0700 From: "Christopher J. Morrone" To: lustre-devel at lists.lustre.org Subject: Re: [lustre-devel] lctl conf_param vs set_param -P Message-ID: <55CE899F.5060403 at llnl.gov> Content-Type: text/plain; charset=windows-1252; format=flowed It was coincidental this time. We saw the deprecation message during llmount.sh and figured that might be an easy first task for Giuseppe to get familiarity with the patch submission process. But it turned out to be a little more difficult than I thought. :) We might want to just remove the conf_param deprecation message altogether. I think there is more than just a simple bug in the "set_param -P" implementation. From what I am seeing, it looks to me like we have some design issues. Without anyone committed to working out a better design and implementing it, set_param -P looks like technical debt with no clear resolution in sight. Maybe we should even consider stripping it back back out of the tree. Chris On 08/14/2015 05:07 PM, Dilger, Andreas wrote: > Coincidentally (or maybe that was what drove your investigations?), I'd just filed https://jira.hpdd.intel.com/browse/LU-7004 about this issue. It looks like the "lctl set_param -P" feature needs more testing, and I've removed the deprecation warning for "lctl conf_param" for 2.8. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Software Architect > Intel High Performance Data Division > > On 2015/08/13, 4:43 PM, "lustre-devel on behalf of Di Natale, Giuseppe" on behalf of dinatale2 at llnl.gov> wrote: > > Greetings, > > In an effort to change test-framework.sh to not utilize the deprecated conf_param option in lctl, I stumbled upon what appears to be inconsistent behavior between lctl's conf_param and set_param -P options. The permanent option test-framework.sh is attempting to change is jobid_var. When using conf_param, any changes to the property are written to /proc/fs/lustre/jobid_var within a short period of time. This is not the case with set_param -P. The change is never reflected nor is it stored in some other file within /proc. I started digging into the MGS logs and found that the behavior for both are different (the relevant segments of the logs are attached to this email and are named accordingly). In short, it appears that conf_param attempts to apply the changes to all the targets while set_param does not (it does not recognize it as a global property). Can someone offer any insight on why the behavior appears to be different or provide insight on if this is incorrect behavio r? > > > Thanks, > Giuseppe Di Natale > _______________________________________________ > lustre-devel mailing list > lustre-devel at lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > ------------------------------ Subject: Digest Footer _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org ------------------------------ End of lustre-devel Digest, Vol 103, Issue 2 ******************************************** From Jeffrey.C.Becker at nasa.gov Mon Aug 24 18:28:18 2015 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Mon, 24 Aug 2015 11:28:18 -0700 Subject: [lustre-devel] [PATCH v2] staging/lustre: fix block comment formatting Message-ID: <1440440898-49055-1-git-send-email-Jeffrey.C.Becker@nasa.gov> Running checkpatch.pl on lnet/klnds/o2iblnd/o2iblnd.h produces several "Block comments use a trailing */ on a separate line" warnings. This patch fixes these. Signed-off-by: Jeff Becker --- .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 284 +++++++++------------ 1 file changed, 127 insertions(+), 157 deletions(-) diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index f4b6c33..07e81cb 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -79,38 +79,33 @@ #define IBLND_N_SCHED_HIGH 4 typedef struct { - int *kib_dev_failover; /* HCA failover */ - unsigned int *kib_service; /* IB service number */ - int *kib_min_reconnect_interval; /* first failed connection - * retry... */ - int *kib_max_reconnect_interval; /* ...exponentially increasing - * to this */ - int *kib_cksum; /* checksum kib_msg_t? */ - int *kib_timeout; /* comms timeout (seconds) */ - int *kib_keepalive; /* keepalive timeout (seconds) */ - int *kib_ntx; /* # tx descs */ - int *kib_credits; /* # concurrent sends */ - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ - int *kib_peerrtrcredits; /* # per-peer router buffer - * credits */ - int *kib_peercredits_hiw; /* # when eagerly to return - * credits */ - int *kib_peertimeout; /* seconds to consider peer dead */ - char **kib_default_ipif; /* default IPoIB interface */ - int *kib_retry_count; - int *kib_rnr_retry_count; - int *kib_concurrent_sends; /* send work queue sizing */ - int *kib_ib_mtu; /* IB MTU */ - int *kib_map_on_demand; /* map-on-demand if RD has more - * fragments than this value, 0 - * disable map-on-demand */ - int *kib_fmr_pool_size; /* # FMRs in pool */ - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ - int *kib_fmr_cache; /* enable FMR pool cache? */ - int *kib_require_priv_port; /* accept only privileged ports */ - int *kib_use_priv_port; /* use privileged port for active - * connect */ - int *kib_nscheds; /* # threads on each CPT */ + int *kib_dev_failover; /* HCA failover */ + unsigned int *kib_service; /* IB service number */ + int *kib_min_reconnect_interval; /* first failed connection retry... */ + int *kib_max_reconnect_interval; /* exponentially increasing to this */ + int *kib_cksum; /* checksum kib_msg_t? */ + int *kib_timeout; /* comms timeout (seconds) */ + int *kib_keepalive; /* keepalive timeout (seconds) */ + int *kib_ntx; /* # tx descs */ + int *kib_credits; /* # concurrent sends */ + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ + int *kib_peercredits_hiw; /* # when eagerly to return credits */ + int *kib_peertimeout; /* seconds to consider peer dead */ + char **kib_default_ipif; /* default IPoIB interface */ + int *kib_retry_count; + int *kib_rnr_retry_count; + int *kib_concurrent_sends; /* send work queue sizing */ + int *kib_ib_mtu; /* IB MTU */ + int *kib_map_on_demand; /* map-on-demand if RD has more */ + /* fragments than this value, 0 */ + /* disable map-on-demand */ + int *kib_fmr_pool_size; /* # FMRs in pool */ + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ + int *kib_fmr_cache; /* enable FMR pool cache? */ + int *kib_require_priv_port; /* accept only privileged ports */ + int *kib_use_priv_port; /* use privileged port for active connect */ + int *kib_nscheds; /* # threads on each CPT */ } kib_tunables_t; extern kib_tunables_t kiblnd_tunables; @@ -199,8 +194,7 @@ typedef struct { unsigned long ibd_next_failover; int ibd_failed_failover; /* # failover failures */ unsigned int ibd_failover; /* failover in progress */ - unsigned int ibd_can_failover; /* IPoIB interface is a bonding - * master */ + unsigned int ibd_can_failover; /* IPoIB interface is a bonding master */ struct list_head ibd_nets; struct kib_hca_dev *ibd_hdev; } kib_dev_t; @@ -249,28 +243,26 @@ typedef struct kib_poolset { char ps_name[IBLND_POOL_NAME_LEN]; /* pool set name */ struct list_head ps_pool_list; /* list of pools */ struct list_head ps_failed_pool_list;/* failed pool list */ - unsigned long ps_next_retry; /* time stamp for retry if - * failed to allocate */ + unsigned long ps_next_retry; /* time stamp for retry if */ + /* failed to allocate */ int ps_increasing; /* is allocating new pool */ int ps_pool_size; /* new pool size */ int ps_cpt; /* CPT id */ kib_ps_pool_create_t ps_pool_create; /* create a new pool */ kib_ps_pool_destroy_t ps_pool_destroy; /* destroy a pool */ - kib_ps_node_init_t ps_node_init; /* initialize new allocated - * node */ + kib_ps_node_init_t ps_node_init; /* initialize new allocated node */ kib_ps_node_fini_t ps_node_fini; /* finalize node */ } kib_poolset_t; typedef struct kib_pool { - struct list_head po_list; /* chain on pool list */ - struct list_head po_free_list; /* pre-allocated node */ - kib_poolset_t *po_owner; /* pool_set of this pool */ - unsigned long po_deadline; /* deadline of this pool */ - int po_allocated; /* # of elements in use */ - int po_failed; /* pool is created on failed - * HCA */ - int po_size; /* # of pre-allocated elements */ + struct list_head po_list; /* chain on pool list */ + struct list_head po_free_list; /* pre-allocated node */ + kib_poolset_t *po_owner; /* pool_set of this pool */ + unsigned long po_deadline; /* deadline of this pool */ + int po_allocated; /* # of elements in use */ + int po_failed; /* pool is created on failed HCA */ + int po_size; /* # of pre-allocated elements */ } kib_pool_t; typedef struct { @@ -295,8 +287,8 @@ typedef struct { int fps_pool_size; int fps_flush_trigger; int fps_increasing; /* is allocating new pool */ - unsigned long fps_next_retry; /* time stamp for retry if - * failed to allocate */ + unsigned long fps_next_retry; /* time stamp for retry if*/ + /* failed to allocate */ } kib_fmr_poolset_t; typedef struct { @@ -344,31 +336,22 @@ struct kib_sched_info { }; typedef struct { - int kib_init; /* initialisation state */ - int kib_shutdown; /* shut down? */ - struct list_head kib_devs; /* IB devices extant */ - struct list_head kib_failed_devs; /* list head of failed - * devices */ - wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ - atomic_t kib_nthreads; /* # live threads */ - rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn - * ops */ - struct list_head *kib_peers; /* hash table of all my known - * peers */ - int kib_peer_hash_size; /* size of kib_peers */ - void *kib_connd; /* the connd task - * (serialisation assertions) - */ - struct list_head kib_connd_conns; /* connections to - * setup/teardown */ - struct list_head kib_connd_zombies; /* connections with zero - * refcount */ - wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps - * here */ - spinlock_t kib_connd_lock; /* serialise */ - struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ - struct kib_sched_info **kib_scheds; /* percpt data for schedulers - */ + int kib_init; /* initialisation state */ + int kib_shutdown; /* shut down? */ + struct list_head kib_devs; /* IB devices extant */ + struct list_head kib_failed_devs; /* list head of failed devices */ + wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ + atomic_t kib_nthreads; /* # live threads */ + rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn ops */ + struct list_head *kib_peers; /* hash table of all my known peers */ + int kib_peer_hash_size; /* size of kib_peers */ + void *kib_connd; /* the connd task (serialisation assertions) */ + struct list_head kib_connd_conns; /* connections to setup/teardown */ + struct list_head kib_connd_zombies; /* connections with zero refcount */ + wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps here */ + spinlock_t kib_connd_lock; /* serialise */ + struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ + struct kib_sched_info **kib_scheds; /* percpt data for schedulers */ } kib_data_t; #define IBLND_INIT_NOTHING 0 @@ -480,10 +463,10 @@ typedef struct { #define IBLND_REJECT_FATAL 3 /* Anything else */ #define IBLND_REJECT_CONN_UNCOMPAT 4 /* incompatible version peer */ #define IBLND_REJECT_CONN_STALE 5 /* stale peer */ -#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match - * mine */ -#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't - * match mine */ +#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match */ + /* mine */ +#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't */ + /* match mine */ /***********************************************************************/ @@ -491,8 +474,7 @@ typedef struct kib_rx /* receive message */ { struct list_head rx_list; /* queue for attention */ struct kib_conn *rx_conn; /* owning conn */ - int rx_nob; /* # bytes received (-1 while - * posted) */ + int rx_nob; /* # bytes received (-1 while posted) */ enum ib_wc_status rx_status; /* completion status */ kib_msg_t *rx_msg; /* message buffer (host vaddr) */ __u64 rx_msgaddr; /* message buffer (I/O addr) */ @@ -501,38 +483,35 @@ typedef struct kib_rx /* receive message */ struct ib_sge rx_sge; /* ...and its memory */ } kib_rx_t; -#define IBLND_POSTRX_DONT_POST 0 /* don't post */ -#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ -#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ -#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give myself back 1 reserved - * credit */ +#define IBLND_POSTRX_DONT_POST 0 /* don't post */ +#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ +#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ +#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give self back 1 reserved credit */ typedef struct kib_tx /* transmit message */ { - struct list_head tx_list; /* queue on idle_txs ibc_tx_queue - * etc. */ - kib_tx_pool_t *tx_pool; /* pool I'm from */ - struct kib_conn *tx_conn; /* owning conn */ - short tx_sending; /* # tx callbacks outstanding */ - short tx_queued; /* queued for sending */ - short tx_waiting; /* waiting for peer */ - int tx_status; /* LNET completion status */ - unsigned long tx_deadline; /* completion deadline */ - __u64 tx_cookie; /* completion cookie */ - lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on - * completion */ - kib_msg_t *tx_msg; /* message buffer (host vaddr) */ - __u64 tx_msgaddr; /* message buffer (I/O addr) */ + struct list_head tx_list; /* queue on idle_txs ibc_tx_queue etc. */ + kib_tx_pool_t *tx_pool; /* pool I'm from */ + struct kib_conn *tx_conn; /* owning conn */ + short tx_sending; /* # tx callbacks outstanding */ + short tx_queued; /* queued for sending */ + short tx_waiting; /* waiting for peer */ + int tx_status; /* LNET completion status */ + unsigned long tx_deadline; /* completion deadline */ + __u64 tx_cookie; /* completion cookie */ + lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on completion */ + kib_msg_t *tx_msg; /* message buffer (host vaddr) */ + __u64 tx_msgaddr; /* message buffer (I/O addr) */ DECLARE_PCI_UNMAP_ADDR(tx_msgunmap); /* for dma_unmap_single() */ - int tx_nwrq; /* # send work items */ - struct ib_send_wr *tx_wrq; /* send work items... */ - struct ib_sge *tx_sge; /* ...and their memory */ - kib_rdma_desc_t *tx_rd; /* rdma descriptor */ - int tx_nfrags; /* # entries in... */ - struct scatterlist *tx_frags; /* dma_map_sg descriptor */ - __u64 *tx_pages; /* rdma phys page addrs */ - kib_fmr_t fmr; /* FMR */ - int tx_dmadir; /* dma direction */ + int tx_nwrq; /* # send work items */ + struct ib_send_wr *tx_wrq; /* send work items... */ + struct ib_sge *tx_sge; /* ...and their memory */ + kib_rdma_desc_t *tx_rd; /* rdma descriptor */ + int tx_nfrags; /* # entries in... */ + struct scatterlist *tx_frags; /* dma_map_sg descriptor */ + __u64 *tx_pages; /* rdma phys page addrs */ + kib_fmr_t fmr; /* FMR */ + int tx_dmadir; /* dma direction */ } kib_tx_t; typedef struct kib_connvars { @@ -540,53 +519,44 @@ typedef struct kib_connvars { } kib_connvars_t; typedef struct kib_conn { - struct kib_sched_info *ibc_sched; /* scheduler information */ - struct kib_peer *ibc_peer; /* owning peer */ - kib_hca_dev_t *ibc_hdev; /* HCA bound on */ - struct list_head ibc_list; /* stash on peer's conn - * list */ - struct list_head ibc_sched_list; /* schedule for attention */ - __u16 ibc_version; /* version of connection */ - __u64 ibc_incarnation; /* which instance of the - * peer */ - atomic_t ibc_refcount; /* # users */ - int ibc_state; /* what's happening */ - int ibc_nsends_posted; /* # uncompleted sends */ - int ibc_noops_posted; /* # uncompleted NOOPs */ - int ibc_credits; /* # credits I have */ + struct kib_sched_info *ibc_sched; /* scheduler information */ + struct kib_peer *ibc_peer; /* owning peer */ + kib_hca_dev_t *ibc_hdev; /* HCA bound on */ + struct list_head ibc_list; /* stash on peer's conn list */ + struct list_head ibc_sched_list; /* schedule for attention */ + __u16 ibc_version; /* version of connection */ + __u64 ibc_incarnation; /* which instance of the peer */ + atomic_t ibc_refcount; /* # users */ + int ibc_state; /* what's happening */ + int ibc_nsends_posted; /* # uncompleted sends */ + int ibc_noops_posted; /* # uncompleted NOOPs */ + int ibc_credits; /* # credits I have */ int ibc_outstanding_credits; /* # credits to return */ int ibc_reserved_credits; /* # ACK/DONE msg credits */ - int ibc_comms_error; /* set on comms error */ - unsigned int ibc_nrx:16; /* receive buffers owned */ - unsigned int ibc_scheduled:1; /* scheduled for attention - */ - unsigned int ibc_ready:1; /* CQ callback fired */ - unsigned long ibc_last_send; /* time of last send */ - struct list_head ibc_connd_list; /* link chain for - * kiblnd_check_conns only - */ - struct list_head ibc_early_rxs; /* rxs completed before - * ESTABLISHED */ - struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for - * IBLND_MSG_VERSION_1 */ - struct list_head ibc_tx_queue; /* sends that need a credit - */ - struct list_head ibc_tx_queue_nocred; /* sends that don't need a - * credit */ - struct list_head ibc_tx_queue_rsrvd; /* sends that need to - * reserve an ACK/DONE msg - */ - struct list_head ibc_active_txs; /* active tx awaiting - * completion */ - spinlock_t ibc_lock; /* serialise */ - kib_rx_t *ibc_rxs; /* the rx descs */ - kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ - - struct rdma_cm_id *ibc_cmid; /* CM id */ - struct ib_cq *ibc_cq; /* completion queue */ - - kib_connvars_t *ibc_connvars; /* in-progress connection - * state */ + int ibc_comms_error; /* set on comms error */ + unsigned int ibc_nrx:16; /* receive buffers owned */ + unsigned int ibc_scheduled:1; /* scheduled for attention */ + unsigned int ibc_ready:1; /* CQ callback fired */ + unsigned long ibc_last_send; /* time of last send */ + struct list_head ibc_connd_list; /* link chain for */ + /* kiblnd_check_conns only */ + struct list_head ibc_early_rxs; /* rxs completed before ESTABLISHED */ + struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for */ + /* IBLND_MSG_VERSION_1 */ + struct list_head ibc_tx_queue; /* sends that need a credit */ + struct list_head ibc_tx_queue_nocred; /* sends that don't need a */ + /* credit */ + struct list_head ibc_tx_queue_rsrvd; /* sends that need to */ + /* reserve an ACK/DONE msg */ + struct list_head ibc_active_txs; /* active tx awaiting completion */ + spinlock_t ibc_lock; /* serialise */ + kib_rx_t *ibc_rxs; /* the rx descs */ + kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ + + struct rdma_cm_id *ibc_cmid; /* CM id */ + struct ib_cq *ibc_cq; /* completion queue */ + + kib_connvars_t *ibc_connvars; /* in-progress connection state */ } kib_conn_t; #define IBLND_CONN_INIT 0 /* being initialised */ @@ -780,8 +750,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q) return NULL; } -/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the - * lowest bits of the work request id to stash the work item type. */ +/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the */ +/* lowest bits of the work request id to stash the work item type. */ #define IBLND_WID_TX 0 #define IBLND_WID_RDMA 1 @@ -928,9 +898,9 @@ static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev, return ib_sg_dma_len(dev, sg); } -/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly - * right because OFED1.2 defines it as const, to use it we have to add - * (void *) cast to overcome "const" */ +/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly */ +/* right because OFED1.2 defines it as const, to use it we have to add */ +/* (void *) cast to overcome "const" */ #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) -- 2.4.5 From spitzcor at cray.com Wed Aug 26 16:25:57 2015 From: spitzcor at cray.com (Cory Spitz) Date: Wed, 26 Aug 2015 16:25:57 +0000 Subject: [lustre-devel] Community code review list Message-ID: If you are (or would like to be) a Lustre developer, please be sure to add your name to the list of community code reviewers at http://wiki.lustre.org/Code_Reviewers. Other developers will then select you as a reviewer when http://wiki.lustre.org/Submitting_Changes. We’d like more collaborative peer review of code amongst a wider set of developers, in part, to strengthen the Lustre developer community. Happy reviewing! -Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From simmonsja at ornl.gov Thu Aug 27 18:16:31 2015 From: simmonsja at ornl.gov (Simmons, James A.) Date: Thu, 27 Aug 2015 18:16:31 +0000 Subject: [lustre-devel] [PATCH v2] staging/lustre: fix block comment formatting In-Reply-To: <1440440898-49055-1-git-send-email-Jeffrey.C.Becker@nasa.gov> References: <1440440898-49055-1-git-send-email-Jeffrey.C.Becker@nasa.gov> Message-ID: <188827c902704a95bc479702f4454e40@EXCHCS32.ornl.gov> >Running checkpatch.pl on lnet/klnds/o2iblnd/o2iblnd.h produces several >"Block comments use a trailing */ on a separate line" warnings. This patch >fixes these. > >Signed-off-by: Jeff Becker >--- > .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 284 +++++++++------------ > 1 file changed, 127 insertions(+), 157 deletions(-) This original was done to avoid the 80 character limit with comments. I remember the discuss to make checkpatch.pl not complain in that case. Was that ever implemented? If so this patch could neatly tab the comments to make them line up. That would make it easier on the eyes. If that is not the case this patch is fine as it is. diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h index f4b6c33..07e81cb 100644 --- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h +++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h @@ -79,38 +79,33 @@ #define IBLND_N_SCHED_HIGH 4 typedef struct { - int *kib_dev_failover; /* HCA failover */ - unsigned int *kib_service; /* IB service number */ - int *kib_min_reconnect_interval; /* first failed connection - * retry... */ - int *kib_max_reconnect_interval; /* ...exponentially increasing - * to this */ - int *kib_cksum; /* checksum kib_msg_t? */ - int *kib_timeout; /* comms timeout (seconds) */ - int *kib_keepalive; /* keepalive timeout (seconds) */ - int *kib_ntx; /* # tx descs */ - int *kib_credits; /* # concurrent sends */ - int *kib_peertxcredits; /* # concurrent sends to 1 peer */ - int *kib_peerrtrcredits; /* # per-peer router buffer - * credits */ - int *kib_peercredits_hiw; /* # when eagerly to return - * credits */ - int *kib_peertimeout; /* seconds to consider peer dead */ - char **kib_default_ipif; /* default IPoIB interface */ - int *kib_retry_count; - int *kib_rnr_retry_count; - int *kib_concurrent_sends; /* send work queue sizing */ - int *kib_ib_mtu; /* IB MTU */ - int *kib_map_on_demand; /* map-on-demand if RD has more - * fragments than this value, 0 - * disable map-on-demand */ - int *kib_fmr_pool_size; /* # FMRs in pool */ - int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ - int *kib_fmr_cache; /* enable FMR pool cache? */ - int *kib_require_priv_port; /* accept only privileged ports */ - int *kib_use_priv_port; /* use privileged port for active - * connect */ - int *kib_nscheds; /* # threads on each CPT */ + int *kib_dev_failover; /* HCA failover */ + unsigned int *kib_service; /* IB service number */ + int *kib_min_reconnect_interval; /* first failed connection retry... */ + int *kib_max_reconnect_interval; /* exponentially increasing to this */ + int *kib_cksum; /* checksum kib_msg_t? */ + int *kib_timeout; /* comms timeout (seconds) */ + int *kib_keepalive; /* keepalive timeout (seconds) */ + int *kib_ntx; /* # tx descs */ + int *kib_credits; /* # concurrent sends */ + int *kib_peertxcredits; /* # concurrent sends to 1 peer */ + int *kib_peerrtrcredits; /* # per-peer router buffer credits */ + int *kib_peercredits_hiw; /* # when eagerly to return credits */ + int *kib_peertimeout; /* seconds to consider peer dead */ + char **kib_default_ipif; /* default IPoIB interface */ + int *kib_retry_count; + int *kib_rnr_retry_count; + int *kib_concurrent_sends; /* send work queue sizing */ + int *kib_ib_mtu; /* IB MTU */ + int *kib_map_on_demand; /* map-on-demand if RD has more */ + /* fragments than this value, 0 */ + /* disable map-on-demand */ + int *kib_fmr_pool_size; /* # FMRs in pool */ + int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ + int *kib_fmr_cache; /* enable FMR pool cache? */ + int *kib_require_priv_port; /* accept only privileged ports */ + int *kib_use_priv_port; /* use privileged port for active connect */ + int *kib_nscheds; /* # threads on each CPT */ } kib_tunables_t; extern kib_tunables_t kiblnd_tunables; @@ -199,8 +194,7 @@ typedef struct { unsigned long ibd_next_failover; int ibd_failed_failover; /* # failover failures */ unsigned int ibd_failover; /* failover in progress */ - unsigned int ibd_can_failover; /* IPoIB interface is a bonding - * master */ + unsigned int ibd_can_failover; /* IPoIB interface is a bonding master */ struct list_head ibd_nets; struct kib_hca_dev *ibd_hdev; } kib_dev_t; @@ -249,28 +243,26 @@ typedef struct kib_poolset { char ps_name[IBLND_POOL_NAME_LEN]; /* pool set name */ struct list_head ps_pool_list; /* list of pools */ struct list_head ps_failed_pool_list;/* failed pool list */ - unsigned long ps_next_retry; /* time stamp for retry if - * failed to allocate */ + unsigned long ps_next_retry; /* time stamp for retry if */ + /* failed to allocate */ int ps_increasing; /* is allocating new pool */ int ps_pool_size; /* new pool size */ int ps_cpt; /* CPT id */ kib_ps_pool_create_t ps_pool_create; /* create a new pool */ kib_ps_pool_destroy_t ps_pool_destroy; /* destroy a pool */ - kib_ps_node_init_t ps_node_init; /* initialize new allocated - * node */ + kib_ps_node_init_t ps_node_init; /* initialize new allocated node */ kib_ps_node_fini_t ps_node_fini; /* finalize node */ } kib_poolset_t; typedef struct kib_pool { - struct list_head po_list; /* chain on pool list */ - struct list_head po_free_list; /* pre-allocated node */ - kib_poolset_t *po_owner; /* pool_set of this pool */ - unsigned long po_deadline; /* deadline of this pool */ - int po_allocated; /* # of elements in use */ - int po_failed; /* pool is created on failed - * HCA */ - int po_size; /* # of pre-allocated elements */ + struct list_head po_list; /* chain on pool list */ + struct list_head po_free_list; /* pre-allocated node */ + kib_poolset_t *po_owner; /* pool_set of this pool */ + unsigned long po_deadline; /* deadline of this pool */ + int po_allocated; /* # of elements in use */ + int po_failed; /* pool is created on failed HCA */ + int po_size; /* # of pre-allocated elements */ } kib_pool_t; typedef struct { @@ -295,8 +287,8 @@ typedef struct { int fps_pool_size; int fps_flush_trigger; int fps_increasing; /* is allocating new pool */ - unsigned long fps_next_retry; /* time stamp for retry if - * failed to allocate */ + unsigned long fps_next_retry; /* time stamp for retry if*/ + /* failed to allocate */ } kib_fmr_poolset_t; typedef struct { @@ -344,31 +336,22 @@ struct kib_sched_info { }; typedef struct { - int kib_init; /* initialisation state */ - int kib_shutdown; /* shut down? */ - struct list_head kib_devs; /* IB devices extant */ - struct list_head kib_failed_devs; /* list head of failed - * devices */ - wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ - atomic_t kib_nthreads; /* # live threads */ - rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn - * ops */ - struct list_head *kib_peers; /* hash table of all my known - * peers */ - int kib_peer_hash_size; /* size of kib_peers */ - void *kib_connd; /* the connd task - * (serialisation assertions) - */ - struct list_head kib_connd_conns; /* connections to - * setup/teardown */ - struct list_head kib_connd_zombies; /* connections with zero - * refcount */ - wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps - * here */ - spinlock_t kib_connd_lock; /* serialise */ - struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ - struct kib_sched_info **kib_scheds; /* percpt data for schedulers - */ + int kib_init; /* initialisation state */ + int kib_shutdown; /* shut down? */ + struct list_head kib_devs; /* IB devices extant */ + struct list_head kib_failed_devs; /* list head of failed devices */ + wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ + atomic_t kib_nthreads; /* # live threads */ + rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn ops */ + struct list_head *kib_peers; /* hash table of all my known peers */ + int kib_peer_hash_size; /* size of kib_peers */ + void *kib_connd; /* the connd task (serialisation assertions) */ + struct list_head kib_connd_conns; /* connections to setup/teardown */ + struct list_head kib_connd_zombies; /* connections with zero refcount */ + wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps here */ + spinlock_t kib_connd_lock; /* serialise */ + struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ + struct kib_sched_info **kib_scheds; /* percpt data for schedulers */ } kib_data_t; #define IBLND_INIT_NOTHING 0 @@ -480,10 +463,10 @@ typedef struct { #define IBLND_REJECT_FATAL 3 /* Anything else */ #define IBLND_REJECT_CONN_UNCOMPAT 4 /* incompatible version peer */ #define IBLND_REJECT_CONN_STALE 5 /* stale peer */ -#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match - * mine */ -#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't - * match mine */ +#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't match */ + /* mine */ +#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size can't */ + /* match mine */ /***********************************************************************/ @@ -491,8 +474,7 @@ typedef struct kib_rx /* receive message */ { struct list_head rx_list; /* queue for attention */ struct kib_conn *rx_conn; /* owning conn */ - int rx_nob; /* # bytes received (-1 while - * posted) */ + int rx_nob; /* # bytes received (-1 while posted) */ enum ib_wc_status rx_status; /* completion status */ kib_msg_t *rx_msg; /* message buffer (host vaddr) */ __u64 rx_msgaddr; /* message buffer (I/O addr) */ @@ -501,38 +483,35 @@ typedef struct kib_rx /* receive message */ struct ib_sge rx_sge; /* ...and its memory */ } kib_rx_t; -#define IBLND_POSTRX_DONT_POST 0 /* don't post */ -#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ -#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ -#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give myself back 1 reserved - * credit */ +#define IBLND_POSTRX_DONT_POST 0 /* don't post */ +#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ +#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ +#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give self back 1 reserved credit */ typedef struct kib_tx /* transmit message */ { - struct list_head tx_list; /* queue on idle_txs ibc_tx_queue - * etc. */ - kib_tx_pool_t *tx_pool; /* pool I'm from */ - struct kib_conn *tx_conn; /* owning conn */ - short tx_sending; /* # tx callbacks outstanding */ - short tx_queued; /* queued for sending */ - short tx_waiting; /* waiting for peer */ - int tx_status; /* LNET completion status */ - unsigned long tx_deadline; /* completion deadline */ - __u64 tx_cookie; /* completion cookie */ - lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on - * completion */ - kib_msg_t *tx_msg; /* message buffer (host vaddr) */ - __u64 tx_msgaddr; /* message buffer (I/O addr) */ + struct list_head tx_list; /* queue on idle_txs ibc_tx_queue etc. */ + kib_tx_pool_t *tx_pool; /* pool I'm from */ + struct kib_conn *tx_conn; /* owning conn */ + short tx_sending; /* # tx callbacks outstanding */ + short tx_queued; /* queued for sending */ + short tx_waiting; /* waiting for peer */ + int tx_status; /* LNET completion status */ + unsigned long tx_deadline; /* completion deadline */ + __u64 tx_cookie; /* completion cookie */ + lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on completion */ + kib_msg_t *tx_msg; /* message buffer (host vaddr) */ + __u64 tx_msgaddr; /* message buffer (I/O addr) */ DECLARE_PCI_UNMAP_ADDR(tx_msgunmap); /* for dma_unmap_single() */ - int tx_nwrq; /* # send work items */ - struct ib_send_wr *tx_wrq; /* send work items... */ - struct ib_sge *tx_sge; /* ...and their memory */ - kib_rdma_desc_t *tx_rd; /* rdma descriptor */ - int tx_nfrags; /* # entries in... */ - struct scatterlist *tx_frags; /* dma_map_sg descriptor */ - __u64 *tx_pages; /* rdma phys page addrs */ - kib_fmr_t fmr; /* FMR */ - int tx_dmadir; /* dma direction */ + int tx_nwrq; /* # send work items */ + struct ib_send_wr *tx_wrq; /* send work items... */ + struct ib_sge *tx_sge; /* ...and their memory */ + kib_rdma_desc_t *tx_rd; /* rdma descriptor */ + int tx_nfrags; /* # entries in... */ + struct scatterlist *tx_frags; /* dma_map_sg descriptor */ + __u64 *tx_pages; /* rdma phys page addrs */ + kib_fmr_t fmr; /* FMR */ + int tx_dmadir; /* dma direction */ } kib_tx_t; typedef struct kib_connvars { @@ -540,53 +519,44 @@ typedef struct kib_connvars { } kib_connvars_t; typedef struct kib_conn { - struct kib_sched_info *ibc_sched; /* scheduler information */ - struct kib_peer *ibc_peer; /* owning peer */ - kib_hca_dev_t *ibc_hdev; /* HCA bound on */ - struct list_head ibc_list; /* stash on peer's conn - * list */ - struct list_head ibc_sched_list; /* schedule for attention */ - __u16 ibc_version; /* version of connection */ - __u64 ibc_incarnation; /* which instance of the - * peer */ - atomic_t ibc_refcount; /* # users */ - int ibc_state; /* what's happening */ - int ibc_nsends_posted; /* # uncompleted sends */ - int ibc_noops_posted; /* # uncompleted NOOPs */ - int ibc_credits; /* # credits I have */ + struct kib_sched_info *ibc_sched; /* scheduler information */ + struct kib_peer *ibc_peer; /* owning peer */ + kib_hca_dev_t *ibc_hdev; /* HCA bound on */ + struct list_head ibc_list; /* stash on peer's conn list */ + struct list_head ibc_sched_list; /* schedule for attention */ + __u16 ibc_version; /* version of connection */ + __u64 ibc_incarnation; /* which instance of the peer */ + atomic_t ibc_refcount; /* # users */ + int ibc_state; /* what's happening */ + int ibc_nsends_posted; /* # uncompleted sends */ + int ibc_noops_posted; /* # uncompleted NOOPs */ + int ibc_credits; /* # credits I have */ int ibc_outstanding_credits; /* # credits to return */ int ibc_reserved_credits; /* # ACK/DONE msg credits */ - int ibc_comms_error; /* set on comms error */ - unsigned int ibc_nrx:16; /* receive buffers owned */ - unsigned int ibc_scheduled:1; /* scheduled for attention - */ - unsigned int ibc_ready:1; /* CQ callback fired */ - unsigned long ibc_last_send; /* time of last send */ - struct list_head ibc_connd_list; /* link chain for - * kiblnd_check_conns only - */ - struct list_head ibc_early_rxs; /* rxs completed before - * ESTABLISHED */ - struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for - * IBLND_MSG_VERSION_1 */ - struct list_head ibc_tx_queue; /* sends that need a credit - */ - struct list_head ibc_tx_queue_nocred; /* sends that don't need a - * credit */ - struct list_head ibc_tx_queue_rsrvd; /* sends that need to - * reserve an ACK/DONE msg - */ - struct list_head ibc_active_txs; /* active tx awaiting - * completion */ - spinlock_t ibc_lock; /* serialise */ - kib_rx_t *ibc_rxs; /* the rx descs */ - kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ - - struct rdma_cm_id *ibc_cmid; /* CM id */ - struct ib_cq *ibc_cq; /* completion queue */ - - kib_connvars_t *ibc_connvars; /* in-progress connection - * state */ + int ibc_comms_error; /* set on comms error */ + unsigned int ibc_nrx:16; /* receive buffers owned */ + unsigned int ibc_scheduled:1; /* scheduled for attention */ + unsigned int ibc_ready:1; /* CQ callback fired */ + unsigned long ibc_last_send; /* time of last send */ + struct list_head ibc_connd_list; /* link chain for */ + /* kiblnd_check_conns only */ + struct list_head ibc_early_rxs; /* rxs completed before ESTABLISHED */ + struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for */ + /* IBLND_MSG_VERSION_1 */ + struct list_head ibc_tx_queue; /* sends that need a credit */ + struct list_head ibc_tx_queue_nocred; /* sends that don't need a */ + /* credit */ + struct list_head ibc_tx_queue_rsrvd; /* sends that need to */ + /* reserve an ACK/DONE msg */ + struct list_head ibc_active_txs; /* active tx awaiting completion */ + spinlock_t ibc_lock; /* serialise */ + kib_rx_t *ibc_rxs; /* the rx descs */ + kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ + + struct rdma_cm_id *ibc_cmid; /* CM id */ + struct ib_cq *ibc_cq; /* completion queue */ + + kib_connvars_t *ibc_connvars; /* in-progress connection state */ } kib_conn_t; #define IBLND_CONN_INIT 0 /* being initialised */ @@ -780,8 +750,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head *q) return NULL; } -/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the - * lowest bits of the work request id to stash the work item type. */ +/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the */ +/* lowest bits of the work request id to stash the work item type. */ #define IBLND_WID_TX 0 #define IBLND_WID_RDMA 1 @@ -928,9 +898,9 @@ static inline unsigned int kiblnd_sg_dma_len(struct ib_device *dev, return ib_sg_dma_len(dev, sg); } -/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly - * right because OFED1.2 defines it as const, to use it we have to add - * (void *) cast to overcome "const" */ +/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly */ +/* right because OFED1.2 defines it as const, to use it we have to add */ +/* (void *) cast to overcome "const" */ #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) -- 2.4.5 _______________________________________________ lustre-devel mailing list lustre-devel at lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org From bevans at cray.com Thu Aug 27 18:25:10 2015 From: bevans at cray.com (Ben Evans) Date: Thu, 27 Aug 2015 18:25:10 +0000 Subject: [lustre-devel] [PATCH v2] staging/lustre: fix block comment formatting In-Reply-To: <188827c902704a95bc479702f4454e40@EXCHCS32.ornl.gov> References: <1440440898-49055-1-git-send-email-Jeffrey.C.Becker@nasa.gov> <188827c902704a95bc479702f4454e40@EXCHCS32.ornl.gov> Message-ID: I believe the "*/ on a trailing line" is getting picked up by a doxygen parser, which wants them that way. If you remove the * beginning in a wrapped comment, I believe it treats it as a ³standard² comment. Ideally it would be nice to fully deoxygenate the whole thing, but that probably shares issues of large-scale changes with tabathon. -Ben Evans On 8/27/15, 2:16 PM, "lustre-devel on behalf of Simmons, James A." wrote: >>Running checkpatch.pl on lnet/klnds/o2iblnd/o2iblnd.h produces several >>"Block comments use a trailing */ on a separate line" warnings. This >>patch >>fixes these. >> >>Signed-off-by: Jeff Becker >>--- >> .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h | 284 >>+++++++++------------ >> 1 file changed, 127 insertions(+), 157 deletions(-) > >This original was done to avoid the 80 character limit with comments. I >remember >the discuss to make checkpatch.pl not complain in that case. Was that >ever implemented? >If so this patch could neatly tab the comments to make them line up. That >would make it >easier on the eyes. If that is not the case this patch is fine as it is. > >diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >index f4b6c33..07e81cb 100644 >--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h >@@ -79,38 +79,33 @@ > #define IBLND_N_SCHED_HIGH 4 > > typedef struct { >- int *kib_dev_failover; /* HCA failover */ >- unsigned int *kib_service; /* IB service number */ >- int *kib_min_reconnect_interval; /* first failed connection >- * retry... */ >- int *kib_max_reconnect_interval; /* ...exponentially increasing >- * to this */ >- int *kib_cksum; /* checksum kib_msg_t? */ >- int *kib_timeout; /* comms timeout (seconds) */ >- int *kib_keepalive; /* keepalive timeout (seconds) */ >- int *kib_ntx; /* # tx descs */ >- int *kib_credits; /* # concurrent sends */ >- int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >- int *kib_peerrtrcredits; /* # per-peer router buffer >- * credits */ >- int *kib_peercredits_hiw; /* # when eagerly to return >- * credits */ >- int *kib_peertimeout; /* seconds to consider peer dead */ >- char **kib_default_ipif; /* default IPoIB interface */ >- int *kib_retry_count; >- int *kib_rnr_retry_count; >- int *kib_concurrent_sends; /* send work queue sizing */ >- int *kib_ib_mtu; /* IB MTU */ >- int *kib_map_on_demand; /* map-on-demand if RD has more >- * fragments than this value, 0 >- * disable map-on-demand */ >- int *kib_fmr_pool_size; /* # FMRs in pool */ >- int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >- int *kib_fmr_cache; /* enable FMR pool cache? */ >- int *kib_require_priv_port; /* accept only privileged ports */ >- int *kib_use_priv_port; /* use privileged port for active >- * connect */ >- int *kib_nscheds; /* # threads on each CPT */ >+ int *kib_dev_failover; /* HCA failover */ >+ unsigned int *kib_service; /* IB service number */ >+ int *kib_min_reconnect_interval; /* first failed connection retry... */ >+ int *kib_max_reconnect_interval; /* exponentially increasing to this */ >+ int *kib_cksum; /* checksum kib_msg_t? */ >+ int *kib_timeout; /* comms timeout (seconds) */ >+ int *kib_keepalive; /* keepalive timeout (seconds) */ >+ int *kib_ntx; /* # tx descs */ >+ int *kib_credits; /* # concurrent sends */ >+ int *kib_peertxcredits; /* # concurrent sends to 1 peer */ >+ int *kib_peerrtrcredits; /* # per-peer router buffer credits */ >+ int *kib_peercredits_hiw; /* # when eagerly to return credits */ >+ int *kib_peertimeout; /* seconds to consider peer dead */ >+ char **kib_default_ipif; /* default IPoIB interface */ >+ int *kib_retry_count; >+ int *kib_rnr_retry_count; >+ int *kib_concurrent_sends; /* send work queue sizing */ >+ int *kib_ib_mtu; /* IB MTU */ >+ int *kib_map_on_demand; /* map-on-demand if RD has more */ >+ /* fragments than this value, 0 */ >+ /* disable map-on-demand */ >+ int *kib_fmr_pool_size; /* # FMRs in pool */ >+ int *kib_fmr_flush_trigger; /* When to trigger FMR flush */ >+ int *kib_fmr_cache; /* enable FMR pool cache? */ >+ int *kib_require_priv_port; /* accept only privileged ports */ >+ int *kib_use_priv_port; /* use privileged port for active connect */ >+ int *kib_nscheds; /* # threads on each CPT */ > } kib_tunables_t; > > extern kib_tunables_t kiblnd_tunables; >@@ -199,8 +194,7 @@ typedef struct { > unsigned long ibd_next_failover; > int ibd_failed_failover; /* # failover failures */ > unsigned int ibd_failover; /* failover in progress */ >- unsigned int ibd_can_failover; /* IPoIB interface is a bonding >- * master */ >+ unsigned int ibd_can_failover; /* IPoIB interface is a bonding master */ > struct list_head ibd_nets; > struct kib_hca_dev *ibd_hdev; > } kib_dev_t; >@@ -249,28 +243,26 @@ typedef struct kib_poolset { > char ps_name[IBLND_POOL_NAME_LEN]; /* pool set name */ > struct list_head ps_pool_list; /* list of pools */ > struct list_head ps_failed_pool_list;/* failed pool list */ >- unsigned long ps_next_retry; /* time stamp for retry if >- * failed to allocate */ >+ unsigned long ps_next_retry; /* time stamp for retry if */ >+ /* failed to allocate */ > int ps_increasing; /* is allocating new pool */ > int ps_pool_size; /* new pool size */ > int ps_cpt; /* CPT id */ > > kib_ps_pool_create_t ps_pool_create; /* create a new pool */ > kib_ps_pool_destroy_t ps_pool_destroy; /* destroy a pool */ >- kib_ps_node_init_t ps_node_init; /* initialize new allocated >- * node */ >+ kib_ps_node_init_t ps_node_init; /* initialize new allocated node */ > kib_ps_node_fini_t ps_node_fini; /* finalize node */ > } kib_poolset_t; > > typedef struct kib_pool { >- struct list_head po_list; /* chain on pool list */ >- struct list_head po_free_list; /* pre-allocated node */ >- kib_poolset_t *po_owner; /* pool_set of this pool */ >- unsigned long po_deadline; /* deadline of this pool */ >- int po_allocated; /* # of elements in use */ >- int po_failed; /* pool is created on failed >- * HCA */ >- int po_size; /* # of pre-allocated elements */ >+ struct list_head po_list; /* chain on pool list */ >+ struct list_head po_free_list; /* pre-allocated node */ >+ kib_poolset_t *po_owner; /* pool_set of this pool */ >+ unsigned long po_deadline; /* deadline of this pool */ >+ int po_allocated; /* # of elements in use */ >+ int po_failed; /* pool is created on failed HCA */ >+ int po_size; /* # of pre-allocated elements */ > } kib_pool_t; > > typedef struct { >@@ -295,8 +287,8 @@ typedef struct { > int fps_pool_size; > int fps_flush_trigger; > int fps_increasing; /* is allocating new pool */ >- unsigned long fps_next_retry; /* time stamp for retry if >- * failed to allocate */ >+ unsigned long fps_next_retry; /* time stamp for retry if*/ >+ /* failed to allocate */ > } kib_fmr_poolset_t; > > typedef struct { >@@ -344,31 +336,22 @@ struct kib_sched_info { > }; > > typedef struct { >- int kib_init; /* initialisation state */ >- int kib_shutdown; /* shut down? */ >- struct list_head kib_devs; /* IB devices extant */ >- struct list_head kib_failed_devs; /* list head of failed >- * devices */ >- wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ >- atomic_t kib_nthreads; /* # live threads */ >- rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn >- * ops */ >- struct list_head *kib_peers; /* hash table of all my known >- * peers */ >- int kib_peer_hash_size; /* size of kib_peers */ >- void *kib_connd; /* the connd task >- * (serialisation assertions) >- */ >- struct list_head kib_connd_conns; /* connections to >- * setup/teardown */ >- struct list_head kib_connd_zombies; /* connections with zero >- * refcount */ >- wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps >- * here */ >- spinlock_t kib_connd_lock; /* serialise */ >- struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ >- struct kib_sched_info **kib_scheds; /* percpt data for schedulers >- */ >+ int kib_init; /* initialisation state */ >+ int kib_shutdown; /* shut down? */ >+ struct list_head kib_devs; /* IB devices extant */ >+ struct list_head kib_failed_devs; /* list head of failed devices */ >+ wait_queue_head_t kib_failover_waitq; /* schedulers sleep here */ >+ atomic_t kib_nthreads; /* # live threads */ >+ rwlock_t kib_global_lock; /* stabilize net/dev/peer/conn ops */ >+ struct list_head *kib_peers; /* hash table of all my known peers */ >+ int kib_peer_hash_size; /* size of kib_peers */ >+ void *kib_connd; /* the connd task (serialisation assertions) */ >+ struct list_head kib_connd_conns; /* connections to setup/teardown */ >+ struct list_head kib_connd_zombies; /* connections with zero refcount */ >+ wait_queue_head_t kib_connd_waitq; /* connection daemon sleeps here */ >+ spinlock_t kib_connd_lock; /* serialise */ >+ struct ib_qp_attr kib_error_qpa; /* QP->ERROR */ >+ struct kib_sched_info **kib_scheds; /* percpt data for schedulers */ > } kib_data_t; > > #define IBLND_INIT_NOTHING 0 >@@ -480,10 +463,10 @@ typedef struct { > #define IBLND_REJECT_FATAL 3 /* Anything else */ > #define IBLND_REJECT_CONN_UNCOMPAT 4 /* incompatible version peer */ > #define IBLND_REJECT_CONN_STALE 5 /* stale peer */ >-#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't >match >- * mine */ >-#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size >can't >- * match mine */ >+#define IBLND_REJECT_RDMA_FRAGS 6 /* Fatal: peer's rdma frags can't >match */ >+ /* mine */ >+#define IBLND_REJECT_MSG_QUEUE_SIZE 7 /* Fatal: peer's msg queue size >can't */ >+ /* match mine */ > > /***********************************************************************/ > >@@ -491,8 +474,7 @@ typedef struct kib_rx /* >receive message */ > { > struct list_head rx_list; /* queue for attention */ > struct kib_conn *rx_conn; /* owning conn */ >- int rx_nob; /* # bytes received (-1 while >- * posted) */ >+ int rx_nob; /* # bytes received (-1 while posted) */ > enum ib_wc_status rx_status; /* completion status */ > kib_msg_t *rx_msg; /* message buffer (host vaddr) */ > __u64 rx_msgaddr; /* message buffer (I/O addr) */ >@@ -501,38 +483,35 @@ typedef struct kib_rx /* >receive message */ > struct ib_sge rx_sge; /* ...and its memory */ > } kib_rx_t; > >-#define IBLND_POSTRX_DONT_POST 0 /* don't post */ >-#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ >-#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ >-#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give myself back 1 >reserved >- * credit */ >+#define IBLND_POSTRX_DONT_POST 0 /* don't post */ >+#define IBLND_POSTRX_NO_CREDIT 1 /* post: no credits */ >+#define IBLND_POSTRX_PEER_CREDIT 2 /* post: give peer back 1 credit */ >+#define IBLND_POSTRX_RSRVD_CREDIT 3 /* post: give self back 1 reserved >credit */ > > typedef struct kib_tx /* transmit message */ > { >- struct list_head tx_list; /* queue on idle_txs ibc_tx_queue >- * etc. */ >- kib_tx_pool_t *tx_pool; /* pool I'm from */ >- struct kib_conn *tx_conn; /* owning conn */ >- short tx_sending; /* # tx callbacks outstanding */ >- short tx_queued; /* queued for sending */ >- short tx_waiting; /* waiting for peer */ >- int tx_status; /* LNET completion status */ >- unsigned long tx_deadline; /* completion deadline */ >- __u64 tx_cookie; /* completion cookie */ >- lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on >- * completion */ >- kib_msg_t *tx_msg; /* message buffer (host vaddr) */ >- __u64 tx_msgaddr; /* message buffer (I/O addr) */ >+ struct list_head tx_list; /* queue on idle_txs ibc_tx_queue etc. */ >+ kib_tx_pool_t *tx_pool; /* pool I'm from */ >+ struct kib_conn *tx_conn; /* owning conn */ >+ short tx_sending; /* # tx callbacks outstanding */ >+ short tx_queued; /* queued for sending */ >+ short tx_waiting; /* waiting for peer */ >+ int tx_status; /* LNET completion status */ >+ unsigned long tx_deadline; /* completion deadline */ >+ __u64 tx_cookie; /* completion cookie */ >+ lnet_msg_t *tx_lntmsg[2]; /* lnet msgs to finalize on completion */ >+ kib_msg_t *tx_msg; /* message buffer (host vaddr) */ >+ __u64 tx_msgaddr; /* message buffer (I/O addr) */ > DECLARE_PCI_UNMAP_ADDR(tx_msgunmap); /* for dma_unmap_single() */ >- int tx_nwrq; /* # send work items */ >- struct ib_send_wr *tx_wrq; /* send work items... */ >- struct ib_sge *tx_sge; /* ...and their memory */ >- kib_rdma_desc_t *tx_rd; /* rdma descriptor */ >- int tx_nfrags; /* # entries in... */ >- struct scatterlist *tx_frags; /* dma_map_sg descriptor */ >- __u64 *tx_pages; /* rdma phys page addrs */ >- kib_fmr_t fmr; /* FMR */ >- int tx_dmadir; /* dma direction */ >+ int tx_nwrq; /* # send work items */ >+ struct ib_send_wr *tx_wrq; /* send work items... */ >+ struct ib_sge *tx_sge; /* ...and their memory */ >+ kib_rdma_desc_t *tx_rd; /* rdma descriptor */ >+ int tx_nfrags; /* # entries in... */ >+ struct scatterlist *tx_frags; /* dma_map_sg descriptor */ >+ __u64 *tx_pages; /* rdma phys page addrs */ >+ kib_fmr_t fmr; /* FMR */ >+ int tx_dmadir; /* dma direction */ > } kib_tx_t; > > typedef struct kib_connvars { >@@ -540,53 +519,44 @@ typedef struct kib_connvars { > } kib_connvars_t; > > typedef struct kib_conn { >- struct kib_sched_info *ibc_sched; /* scheduler information */ >- struct kib_peer *ibc_peer; /* owning peer */ >- kib_hca_dev_t *ibc_hdev; /* HCA bound on */ >- struct list_head ibc_list; /* stash on peer's conn >- * list */ >- struct list_head ibc_sched_list; /* schedule for attention */ >- __u16 ibc_version; /* version of connection */ >- __u64 ibc_incarnation; /* which instance of the >- * peer */ >- atomic_t ibc_refcount; /* # users */ >- int ibc_state; /* what's happening */ >- int ibc_nsends_posted; /* # uncompleted sends */ >- int ibc_noops_posted; /* # uncompleted NOOPs */ >- int ibc_credits; /* # credits I have */ >+ struct kib_sched_info *ibc_sched; /* scheduler information */ >+ struct kib_peer *ibc_peer; /* owning peer */ >+ kib_hca_dev_t *ibc_hdev; /* HCA bound on */ >+ struct list_head ibc_list; /* stash on peer's conn list */ >+ struct list_head ibc_sched_list; /* schedule for attention */ >+ __u16 ibc_version; /* version of connection */ >+ __u64 ibc_incarnation; /* which instance of the peer */ >+ atomic_t ibc_refcount; /* # users */ >+ int ibc_state; /* what's happening */ >+ int ibc_nsends_posted; /* # uncompleted sends */ >+ int ibc_noops_posted; /* # uncompleted NOOPs */ >+ int ibc_credits; /* # credits I have */ > int ibc_outstanding_credits; /* # credits to return */ > int ibc_reserved_credits; /* # ACK/DONE msg credits */ >- int ibc_comms_error; /* set on comms error */ >- unsigned int ibc_nrx:16; /* receive buffers owned */ >- unsigned int ibc_scheduled:1; /* scheduled for attention >- */ >- unsigned int ibc_ready:1; /* CQ callback fired */ >- unsigned long ibc_last_send; /* time of last send */ >- struct list_head ibc_connd_list; /* link chain for >- * kiblnd_check_conns only >- */ >- struct list_head ibc_early_rxs; /* rxs completed before >- * ESTABLISHED */ >- struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for >- * IBLND_MSG_VERSION_1 */ >- struct list_head ibc_tx_queue; /* sends that need a credit >- */ >- struct list_head ibc_tx_queue_nocred; /* sends that don't need a >- * credit */ >- struct list_head ibc_tx_queue_rsrvd; /* sends that need to >- * reserve an ACK/DONE msg >- */ >- struct list_head ibc_active_txs; /* active tx awaiting >- * completion */ >- spinlock_t ibc_lock; /* serialise */ >- kib_rx_t *ibc_rxs; /* the rx descs */ >- kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ >- >- struct rdma_cm_id *ibc_cmid; /* CM id */ >- struct ib_cq *ibc_cq; /* completion queue */ >- >- kib_connvars_t *ibc_connvars; /* in-progress connection >- * state */ >+ int ibc_comms_error; /* set on comms error */ >+ unsigned int ibc_nrx:16; /* receive buffers owned */ >+ unsigned int ibc_scheduled:1; /* scheduled for attention */ >+ unsigned int ibc_ready:1; /* CQ callback fired */ >+ unsigned long ibc_last_send; /* time of last send */ >+ struct list_head ibc_connd_list; /* link chain for */ >+ /* kiblnd_check_conns only */ >+ struct list_head ibc_early_rxs; /* rxs completed before ESTABLISHED */ >+ struct list_head ibc_tx_noops; /* IBLND_MSG_NOOPs for */ >+ /* IBLND_MSG_VERSION_1 */ >+ struct list_head ibc_tx_queue; /* sends that need a credit */ >+ struct list_head ibc_tx_queue_nocred; /* sends that don't need a */ >+ /* credit */ >+ struct list_head ibc_tx_queue_rsrvd; /* sends that need to */ >+ /* reserve an ACK/DONE msg */ >+ struct list_head ibc_active_txs; /* active tx awaiting completion */ >+ spinlock_t ibc_lock; /* serialise */ >+ kib_rx_t *ibc_rxs; /* the rx descs */ >+ kib_pages_t *ibc_rx_pages; /* premapped rx msg pages */ >+ >+ struct rdma_cm_id *ibc_cmid; /* CM id */ >+ struct ib_cq *ibc_cq; /* completion queue */ >+ >+ kib_connvars_t *ibc_connvars; /* in-progress connection state */ > } kib_conn_t; > > #define IBLND_CONN_INIT 0 /* being initialised */ >@@ -780,8 +750,8 @@ kiblnd_queue2str(kib_conn_t *conn, struct list_head >*q) > return NULL; > } > >-/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the >- * lowest bits of the work request id to stash the work item type. */ >+/* CAVEAT EMPTOR: We rely on descriptor alignment to allow us to use the >*/ >+/* lowest bits of the work request id to stash the work item type. */ > > #define IBLND_WID_TX 0 > #define IBLND_WID_RDMA 1 >@@ -928,9 +898,9 @@ static inline unsigned int kiblnd_sg_dma_len(struct >ib_device *dev, > return ib_sg_dma_len(dev, sg); > } > >-/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly >- * right because OFED1.2 defines it as const, to use it we have to add >- * (void *) cast to overcome "const" */ >+/* XXX We use KIBLND_CONN_PARAM(e) as writable buffer, it's not strictly >*/ >+/* right because OFED1.2 defines it as const, to use it we have to add */ >+/* (void *) cast to overcome "const" */ > > #define KIBLND_CONN_PARAM(e) ((e)->param.conn.private_data) > #define KIBLND_CONN_PARAM_LEN(e) ((e)->param.conn.private_data_len) >-- >2.4.5 > >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > >_______________________________________________ >lustre-devel mailing list >lustre-devel at lists.lustre.org >http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org