From adilger at whamcloud.com  Wed Nov  2 19:17:29 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Wed, 2 Nov 2011 13:17:29 -0600
Subject: [Lustre-devel] Lustre  code style Git commit hooks and
Message-ID: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>

In order to help improve the quality of commits being submitted for
inclusion into Lustre, I've created some Git commit hooks that can
be used to detect problems with the format and coding style of the
patch before it is submitted to Gerrit.


The b1_8, b2_1, and master (2.2) branches of lustre-release now have
scripts called build/commit-msg and build/prepare-commit-msg that
should be copied into the .git/hooks/ subdirectory of any Lustre
checkout.  Currently the commit hooks are optional, but I would
recommend that everyone developing for Lustre would begin using them,
because I hope to make them mandatory in the future, once I'm sure
that they won't interfere with the development process.



The prepare-commit-msg hook will run the build/checkpatch.pl script
when "git commit" is run, and will append a comment with a summary
of any code style issues that it finds in the to-be-committed patch.
It is also possible to run checkpatch.pl manually before committing:

        git diff [--cached] | build/checkpatch.pl -

While checkpatch.pl isn't going to catch all cases that don't follow
the Lustre style style guidelines (as documented in the wiki page:

http://wiki.whamcloud.com/display/PUB/Coding+Guidelines)

This can catch a lot of trivial mistakes that avoid spending the time
of inspectors and test systems, and the need for patch resubmission.

Errors from checkpatch.pl currently won't prevent committing the patch,
but you shouldn't ignore the warnings or errors lightly, since they
can cause the patch to be rejected later after it has consumed test
and inspection resources.  The checkpatch.pl script also won't catch
all style problems, but it covers many of the basic problems.  If you
feel the script is not agreeing with the Lustre "Coding Guidelines"
page, or you would like to improve it, please email me with details
and/or file a bug with a patch to fix the issue.


The commit-msg hook verifies the format of the commit message itself,
so that it contains a Jira ticket number, a "subsystem:" field, a
commit body, and a Signed-off-by: line.  If no Gerrit "Change-Id:"
line is present it will automatically add this as well.  The new
"subsystem:" field is similar to those used for kernel commit messages
all have, so that it is easier to decide what part of the code a patch
affects.  Valid commit messages are of the form:

> LU-000 component: short description of change under 64 columns
> 
> A more detailed explanation of the change being made.  This can be
> as detailed as you'd like, possibly several paragraphs in length.
> 
> Please provide a detailed explanation of what problem was solved,
> a good high-level description of how it was solved, and which
> parts of the code were affected (including function names as
> needed).  Wrap lines at 72 columns or less.
> 
> Signed-off-by: Random J Developer <random at developer.example.org>
> Change-Id: Ica9ed1612eab0c4673dee088f8b441d806c64932

If you haven't looked at the "Using Gerrit" and "Submitting Changes"
wiki pages recently, or want more detailed information about the
patch submission process, they contain a more detailed description:

http://wiki.whamcloud.com/display/PUB/Using+Gerrit
http://wiki.whamcloud.com/display/PUB/Submitting+Changes


I hope that these small additions will help make the Lustre code more consistent and easier to read, and make our inspection, testing, and landing process more efficient as we move into 2.2 development.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.




From surya1 at llnl.gov  Thu Nov  3 22:28:31 2011
From: surya1 at llnl.gov (Prakash Surya)
Date: Thu, 3 Nov 2011 15:28:31 -0700
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
	and
In-Reply-To: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
Message-ID: <20111103222831.GB19258@llnl.gov>

I like the idea of cleaning up the commit messages in Lustre, although I
feel enforcing these rules is better suited at the server level.

Currently, the proposed hooks will not allow any local commits to occur
if the message does not fit the format required. This presents
unnecessary overhead if the commit in question is intended to be
squashed or reworded later down the road.

It also prevents downstream commits, which are not intended to make it
upstream, from using any other commit message format. Thus, making it
impossible to use downstream bug numbers in the message which don't
take the form LU-nnn.

I think until those two use cases are accounted for, it will be tough
for me to use the proposed commit hooks. Perhaps, simply emitting a
warning on offending messages, leaving enforcement to the upstream
repository would suffice? Or bypass the local checks altogether if the
first line summary does not begin with LU-nnn? Or both?

On Wed, Nov 02, 2011 at 12:17:29PM -0700, Andreas Dilger wrote:
> In order to help improve the quality of commits being submitted for
> inclusion into Lustre, I've created some Git commit hooks that can
> be used to detect problems with the format and coding style of the
> patch before it is submitted to Gerrit.
> 
> 
> The b1_8, b2_1, and master (2.2) branches of lustre-release now have
> scripts called build/commit-msg and build/prepare-commit-msg that
> should be copied into the .git/hooks/ subdirectory of any Lustre
> checkout.  Currently the commit hooks are optional, but I would
> recommend that everyone developing for Lustre would begin using them,
> because I hope to make them mandatory in the future, once I'm sure
> that they won't interfere with the development process.
> 
> 
> 
> The prepare-commit-msg hook will run the build/checkpatch.pl script
> when "git commit" is run, and will append a comment with a summary
> of any code style issues that it finds in the to-be-committed patch.
> It is also possible to run checkpatch.pl manually before committing:
> 
>         git diff [--cached] | build/checkpatch.pl -
> 
> While checkpatch.pl isn't going to catch all cases that don't follow
> the Lustre style style guidelines (as documented in the wiki page:
> 
> http://wiki.whamcloud.com/display/PUB/Coding+Guidelines)
> 
> This can catch a lot of trivial mistakes that avoid spending the time
> of inspectors and test systems, and the need for patch resubmission.
> 
> Errors from checkpatch.pl currently won't prevent committing the patch,
> but you shouldn't ignore the warnings or errors lightly, since they
> can cause the patch to be rejected later after it has consumed test
> and inspection resources.  The checkpatch.pl script also won't catch
> all style problems, but it covers many of the basic problems.  If you
> feel the script is not agreeing with the Lustre "Coding Guidelines"
> page, or you would like to improve it, please email me with details
> and/or file a bug with a patch to fix the issue.
> 
> 
> The commit-msg hook verifies the format of the commit message itself,
> so that it contains a Jira ticket number, a "subsystem:" field, a
> commit body, and a Signed-off-by: line.  If no Gerrit "Change-Id:"
> line is present it will automatically add this as well.  The new
> "subsystem:" field is similar to those used for kernel commit messages
> all have, so that it is easier to decide what part of the code a patch
> affects.  Valid commit messages are of the form:
> 
> > LU-000 component: short description of change under 64 columns
> > 
> > A more detailed explanation of the change being made.  This can be
> > as detailed as you'd like, possibly several paragraphs in length.
> > 
> > Please provide a detailed explanation of what problem was solved,
> > a good high-level description of how it was solved, and which
> > parts of the code were affected (including function names as
> > needed).  Wrap lines at 72 columns or less.
> > 
> > Signed-off-by: Random J Developer <random at developer.example.org>
> > Change-Id: Ica9ed1612eab0c4673dee088f8b441d806c64932
> 
> If you haven't looked at the "Using Gerrit" and "Submitting Changes"
> wiki pages recently, or want more detailed information about the
> patch submission process, they contain a more detailed description:
> 
> http://wiki.whamcloud.com/display/PUB/Using+Gerrit
> http://wiki.whamcloud.com/display/PUB/Submitting+Changes
> 
> 
> I hope that these small additions will help make the Lustre code more consistent and easier to read, and make our inspection, testing, and landing process more efficient as we move into 2.2 development.
> 
> Cheers, Andreas
> --
> Andreas Dilger 
> Principal Engineer
> Whamcloud, Inc.
> 
> 

-- 
Cheers,
Prakash

From adilger at whamcloud.com  Fri Nov  4 16:40:41 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Fri, 4 Nov 2011 10:40:41 -0600
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
	and
In-Reply-To: <20111103222831.GB19258@llnl.gov>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
Message-ID: <8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>

On 2011-11-03, at 4:28 PM, Prakash Surya <surya1 at llnl.gov> wrote:
> I like the idea of cleaning up the commit messages in Lustre, although I
> feel enforcing these rules is better suited at the server level.

There are already limited checks at the server level, and they may be tightened in the near future. The concerns about doing this included making sure that the verification scripts were working correctly first (i.e. as local hooks which can be bypassed if needed), to avoid too much disruption before developers got a chance to conform to the new requirements (i.e. some time after this initial announcement), and to catch the errors as early in the development process as possible (i.e. when the commit message is first created).

> Currently, the proposed hooks will not allow any local commits to occur
> if the message does not fit the format required. This presents
> unnecessary overhead if the commit in question is intended to be
> squashed or reworded later down the road.

True, but I've been using them for some time w/o undue effort for patches not intended for upstream use either. Note that there is a sample Signed-off-by: at the end of the commit comment that can be uncommented, and if you think it would be helpful a full template could be provided in the comments. 

> It also prevents downstream commits, which are not intended to make it
> upstream, from using any other commit message format. Thus, making it
> impossible to use downstream bug numbers in the message which don't
> take the form LU-nnn.
> 
> I think until those two use cases are accounted for, it will be tough
> for me to use the proposed commit hooks.

I thought about this previously, and tried (but did not commit) an environment variable that the commit-msg hook could could check to skip verification, something like:

COMCHECK=y git commit -a

> Perhaps, simply emitting a
> warning on offending messages, leaving enforcement to the upstream
> repository would suffice? Or bypass the local checks altogether if the
> first line summary does not begin with LU-nnn? Or both?

Since there are different Jira spaces (LU-nnn, ORI-nnn, MRP-nnn, etc.) this would be error prone. 

> On Wed, Nov 02, 2011 at 12:17:29PM -0700, Andreas Dilger wrote:
>> In order to help improve the quality of commits being submitted for
>> inclusion into Lustre, I've created some Git commit hooks that can
>> be used to detect problems with the format and coding style of the
>> patch before it is submitted to Gerrit.
>> 
>> 
>> The b1_8, b2_1, and master (2.2) branches of lustre-release now have
>> scripts called build/commit-msg and build/prepare-commit-msg that
>> should be copied into the .git/hooks/ subdirectory of any Lustre
>> checkout.  Currently the commit hooks are optional, but I would
>> recommend that everyone developing for Lustre would begin using them,
>> because I hope to make them mandatory in the future, once I'm sure
>> that they won't interfere with the development process.
>> 
>> 
>> 
>> The prepare-commit-msg hook will run the build/checkpatch.pl script
>> when "git commit" is run, and will append a comment with a summary
>> of any code style issues that it finds in the to-be-committed patch.
>> It is also possible to run checkpatch.pl manually before committing:
>> 
>>        git diff [--cached] | build/checkpatch.pl -
>> 
>> While checkpatch.pl isn't going to catch all cases that don't follow
>> the Lustre style style guidelines (as documented in the wiki page:
>> 
>> http://wiki.whamcloud.com/display/PUB/Coding+Guidelines)
>> 
>> This can catch a lot of trivial mistakes that avoid spending the time
>> of inspectors and test systems, and the need for patch resubmission.
>> 
>> Errors from checkpatch.pl currently won't prevent committing the patch,
>> but you shouldn't ignore the warnings or errors lightly, since they
>> can cause the patch to be rejected later after it has consumed test
>> and inspection resources.  The checkpatch.pl script also won't catch
>> all style problems, but it covers many of the basic problems.  If you
>> feel the script is not agreeing with the Lustre "Coding Guidelines"
>> page, or you would like to improve it, please email me with details
>> and/or file a bug with a patch to fix the issue.
>> 
>> 
>> The commit-msg hook verifies the format of the commit message itself,
>> so that it contains a Jira ticket number, a "subsystem:" field, a
>> commit body, and a Signed-off-by: line.  If no Gerrit "Change-Id:"
>> line is present it will automatically add this as well.  The new
>> "subsystem:" field is similar to those used for kernel commit messages
>> all have, so that it is easier to decide what part of the code a patch
>> affects.  Valid commit messages are of the form:
>> 
>>> LU-000 component: short description of change under 64 columns
>>> 
>>> A more detailed explanation of the change being made.  This can be
>>> as detailed as you'd like, possibly several paragraphs in length.
>>> 
>>> Please provide a detailed explanation of what problem was solved,
>>> a good high-level description of how it was solved, and which
>>> parts of the code were affected (including function names as
>>> needed).  Wrap lines at 72 columns or less.
>>> 
>>> Signed-off-by: Random J Developer <random at developer.example.org>
>>> Change-Id: Ica9ed1612eab0c4673dee088f8b441d806c64932
>> 
>> If you haven't looked at the "Using Gerrit" and "Submitting Changes"
>> wiki pages recently, or want more detailed information about the
>> patch submission process, they contain a more detailed description:
>> 
>> http://wiki.whamcloud.com/display/PUB/Using+Gerrit
>> http://wiki.whamcloud.com/display/PUB/Submitting+Changes
>> 
>> 
>> I hope that these small additions will help make the Lustre code more consistent and easier to read, and make our inspection, testing, and landing process more efficient as we move into 2.2 development.
>> 
>> Cheers, Andreas
>> --
>> Andreas Dilger 
>> Principal Engineer
>> Whamcloud, Inc.
>> 
>> 
> 
> -- 
> Cheers,
> Prakash

From brian at whamcloud.com  Fri Nov  4 21:47:42 2011
From: brian at whamcloud.com (Brian J. Murrell)
Date: Fri, 04 Nov 2011 17:47:42 -0400
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
 and
In-Reply-To: <8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
Message-ID: <1320443263.7466.26.camel@brian-laptop>

On Fri, 2011-11-04 at 10:40 -0600, Andreas Dilger wrote:
> On 2011-11-03, at 4:28 PM, Prakash Surya <surya1 at llnl.gov> wrote:
> > I like the idea of cleaning up the commit messages in Lustre, although I
> > feel enforcing these rules is better suited at the server level.
> 
> There are already limited checks at the server level, and they may be tightened in the near future.

And the most the server (which is Gerrit that we are talking about her)
can do is to negatively review the patch as it does currently.  There is
no mechanism to outright refuse to allow a push based on the content of
the patch, so client-side is still the best place to apply these checks
in an effort to prevent the patch from even being submitted if not
correct.

b.



From yong.fan at whamcloud.com  Tue Nov  8 07:30:59 2011
From: yong.fan at whamcloud.com (Fan Yong)
Date: Tue, 08 Nov 2011 15:30:59 +0800
Subject: [Lustre-devel] Reminder for China Lustre Workshop 2011
In-Reply-To: <4E7980BF.1050801@whamcloud.com>
References: <4E7980BF.1050801@whamcloud.com>
Message-ID: <4EB8DAB3.2030902@whamcloud.com>


??????????: 	2011?10?15?
????????: 	2011?11?10?
??????: 	2011?11?20?



This is the invitation for China Lustre Workshop 2011. For English 
users, please click here 
<http://lustrechina.csp.escience.cn/dct/page/65595>.


  ??Lustre????2011???


  http://lustrechina.csp.escience.cn/

???Lustre??:

      
????????,Lustre????????I/O??,?????????,???????????,????,POSIX?????????,? 
??????????????????????????????Top 500?????,??2011?6?,??Top 
10?????????8??????Lustre,?Top 100??????????70%??????Lustre?

      
????,??????????????,???????????????Lustre??????,Lustre?????????????????? 
??????????????,???????????Lustre?????????????????????????? 
?????"??Lustre????",?????Lustre????,????"??Lustre????",??? 
?Lustre??????,?Lustre??????????????

      ????????????????? <http://www.ihep.ac.cn/>?Whamcloud 
<http://www.whamcloud.com/>????????????Whamcloud????????????????????????,????? 
???Lustre???????Lustre???????????????????:

  *   Lustre??? roadmap
  *   Lustre??/???? ??,???/????
  *   Lustre??????? ????????
  *   ???Lustre???? ????
  *   Lustre??????

????????????Lustre???? 2011(??????),????????? 
<http://lustrechina.csp.escience.cn/dct/page/65554>,?????????????????? 
????? 
<http://lustrechina.csp.escience.cn/dct/page/65558>???,????????????? 
??????? 
<http://lustrechina.ihep.ac.cn:8080/enableq/q.php?qname=lustresurvey>,?????????????,?? 
??!


????: 2011?11?26?
????: ??????????????
? ? ?: 
??(138-1075-8632,Lu.Wang at ihep.ac.cn),??(138-1064-7466,yong.fan at whamcloud.com)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20111108/c2123aed/attachment.html>

From jsimmons at infradead.org  Wed Nov  9 16:27:57 2011
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 9 Nov 2011 16:27:57 +0000 (GMT)
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
 and
In-Reply-To: <1320443263.7466.26.camel@brian-laptop>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
Message-ID: <alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>


> > On 2011-11-03, at 4:28 PM, Prakash Surya <surya1 at llnl.gov> wrote:
> > > I like the idea of cleaning up the commit messages in Lustre, although I
> > > feel enforcing these rules is better suited at the server level.
> > 
> > There are already limited checks at the server level, and they may be tightened in the near future.
> 
> And the most the server (which is Gerrit that we are talking about her)
> can do is to negatively review the patch as it does currently.  There is
> no mechanism to outright refuse to allow a push based on the content of
> the patch, so client-side is still the best place to apply these checks
> in an effort to prevent the patch from even being submitted if not
> correct.

Speaking of I just attempted to push a patch and noticed that no Change-Id 
is being generated when I commit a new patch. This is with the default 
hooks that I copied form build directory.

From adilger at whamcloud.com  Wed Nov  9 17:04:58 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Wed, 9 Nov 2011 10:04:58 -0700
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
	and
In-Reply-To: <alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
Message-ID: <855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>

On 2011-11-09, at 9:27 AM, James Simmons <jsimmons at infradead.org> wrote:
> 
>>> 
> Speaking of I just attempted to push a patch and noticed that no Change-Id 
> is being generated when I commit a new patch. This is with the default 
> hooks that I copied form build directory.

Is this from the tip of the master branch?

An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.

If updating to the latest version (or removing "-t commit") my fix this for you, please file a bug assigned to me and we can work it out. 

The contents of the .git directory are not under revision control, so it isn't possible to automatically update the commit hooks. At some point I'd like to update the hooks via "make", but was holding off until I have more confidence that issues like this related to different development environments are fixed. 

Thanks for the feedback. 

Cheers, Andreas

From surya1 at llnl.gov  Wed Nov  9 17:40:13 2011
From: surya1 at llnl.gov (Prakash Surya)
Date: Wed, 9 Nov 2011 09:40:13 -0800
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
 and
In-Reply-To: <855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
Message-ID: <20111109174013.GA9136@llnl.gov>

On Wed, Nov 09, 2011 at 09:04:58AM -0800, Andreas Dilger wrote:
> On 2011-11-09, at 9:27 AM, James Simmons <jsimmons at infradead.org> wrote:
> > 
> >>> 
> > Speaking of I just attempted to push a patch and noticed that no Change-Id 
> > is being generated when I commit a new patch. This is with the default 
> > hooks that I copied form build directory.
> 
> Is this from the tip of the master branch?
> 
> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.
> 
> If updating to the latest version (or removing "-t commit") my fix this for you, please file a bug assigned to me and we can work it out. 
> 
> The contents of the .git directory are not under revision control, so it isn't possible to automatically update the commit hooks. At some point I'd like to update the hooks via "make", but was holding off until I have more confidence that issues like this related to different development environments are fixed. 

Alternatively, one could also use symlinks in .git to point to the files
in the lustre build directory. It's kind of a hack, but it keeps the
hooks up to date. And that way, the lustre build process doesn't have
to muck with the user's .git directory.

> 
> Thanks for the feedback. 
> 
> Cheers, Andreas
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

-- 
Cheers,
Prakash

From jsimmons at infradead.org  Wed Nov  9 19:23:13 2011
From: jsimmons at infradead.org (James Simmons)
Date: Wed, 9 Nov 2011 19:23:13 +0000 (GMT)
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
 and
In-Reply-To: <855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
Message-ID: <alpine.LFD.2.02.1111091918270.401@casper.infradead.org>


> > Speaking of I just attempted to push a patch and noticed that no Change-Id 
> > is being generated when I commit a new patch. This is with the default 
> > hooks that I copied form build directory.
> 
> Is this from the tip of the master branch?
> 
> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.

Yes your right. I updated the hooks and it works now. I also had to learn 
you need a subsystem: field for commits now. Info about the subsystem 
field is not the wiki.

From adilger at whamcloud.com  Wed Nov  9 22:00:37 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Wed, 9 Nov 2011 15:00:37 -0700
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
	<alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
Message-ID: <ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>

On 2011-11-09, at 12:23 PM, James Simmons wrote:
>> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.
> 
> Yes your right. I updated the hooks and it works now. I also had to learn 
> you need a subsystem: field for commits now. Info about the subsystem 
> field is not the wiki.

Hmm, which wiki page are you referring to?  The "Using Gerrit" page has had
the "component:" tag since june, long before I sent out the email,
and it was in the original email as well:

> The commit-msg hook verifies the format of the commit message itself,
> so that it contains a Jira ticket number, a "subsystem:" field, a
> commit body, and a Signed-off-by: line.  If no Gerrit "Change-Id:"
> line is present it will automatically add this as well.  The new
> "subsystem:" field is similar to those used for kernel commit messages
> all have, so that it is easier to decide what part of the code a patch
> affects.

If it is missing on some other wiki page, I'd like to correct it.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.




From bruce_korb at xyratex.com  Mon Nov 14 15:11:50 2011
From: bruce_korb at xyratex.com (Bruce Korb)
Date: Mon, 14 Nov 2011 07:11:50 -0800
Subject: [Lustre-devel] lustre-devel  packaging
Message-ID: <CAE66FB6.134A%bruce_korb@xyratex.com>

Hi,

We at Xyratex are needing to develop some auxiliary utilities that
need internal data structures from lustre.  To accomplish this, our
sources have been reaching into the lustre build tree for the
necessary headers.  Not being a particular fan of that approach,
I developed a patch for lustre that will create another RPM:
a lustre-devel package.


It builds the packages and drops the headers into a tree that looks like
this:

usr/include/lustre
usr/include/lustre/darwin
usr/include/lustre/linux
usr/include/lustre/lnet
usr/include/lustre/lnet/darwin
usr/include/lustre/lnet/linux
usr/include/lustre/lnet/winnt
usr/include/lustre/libcfs
usr/include/lustre/libcfs/darwin
usr/include/lustre/libcfs/linux
usr/include/lustre/libcfs/winnt
usr/include/lustre/libcfs/util
usr/include/lustre/libcfs/posix
usr/include/linux
usr/include/libcfs/posix


one file is dropped into include/libcfs/posix and one file into
include/linux.
I'd have preferred not doing that, but the old code already did.
186 headers get installed.


The other files installed as part of lustre-devel are object
archives and library interface man pages:

usr/lib64/libcfsutil.a
usr/lib64/liblustreapi.a
usr/lib64/liblustre.a
usr/lib64/libptlctl.a
usr/lib64/libiam.a
usr/share/lustre/mpich-1.2.6-lustre.patch
usr/share/man/man3/llapi_quotactl.3.gz
usr/share/man/man3/llapi_file_open.3.gz
usr/share/man/man3/llapi_file_create.3.gz
usr/share/man/man3/llapi_file_get_stripe.3.gz


The rest of the lustre installation is left with the main package.

Below is my git-commit message.  Under separate cover, I'll post the
git patch under the subject "[PATCH] lustre-devel packaging"

* autoMakefile.am
  Constrain lines to 80 columns (readability)
  Per automake maintainers, configurables should be set to a
  make file macro before use.  Enables "make" time replacement.

* build/autoMakefile.am.toplevel
  Constrain lines to 80 columns (readability)

* libcfs/include/Makefile.am
  Put every header under libcfs/include into nobase_pkginclude_HEADERS.

* lnet/autoconf/lustre-lnet.m4
  The body of an m4 (autoconf) macro is indented (readability).
  "LNET_MAX_PAYLOAD" is now a configured value
  lnet/include/lnet/types.h is now a configured file
  Alphabetize and indent the configured files

* lnet/include/Makefile.am
  Collect the neaders into nobase_pkginclude_HEADERS and
  nodist_nobase_pkginclude_HEADERS (just lnet/types.h)

* lnet/include/lnet/Makefile.am
  lnet/include/lnet/types.h is now a configured file

* lnet/include/lnet/types.h
  Renamed to:
* lnet/include/lnet/types.h.in
  Add configured #define for LNET_MAX_PAYLOAD and adjust CPP testing.

* lustre.spec.in
  354 columns in a single line is too much.  Trim the egregious ones.
  add a "%package devel" directive
  In %install section, separate "devel" files from non-devel and list
  them in lustre-devel.files and lustre.files.

* lustre/autoconf/lustre-core.m4
  Add config.h #define for OBD_MAX_IOCTL_BUFFER and make it a
  substitution value as well.
  lustre/include/lustre/Makefile.am is removed.  Do not configure.

* lustre/include/Makefile.am
  The lustre/include/lustre/*.h headers get installed in pkginclude.
  Drop the rest into nodist_pkginclude_HEADER (lustre_ver.h) or
  nobase_pkginclude_HEADER (everything else).

* lustre/include/linux/Makefile.am
  The EXTRA_DIST value is obsolete.

* lustre/include/linux/lustre_acl.h
  Fix spelling of "Should not include directly".

* lustre/include/lustre/Makefile.am
  Obsolete

* lustre/include/lustre_sec.h
  Prefer "__u32" to "uint32_t".  Eliminates need for inttypes.h.

* lustre/include/lustre_ver.h.in
  Add configured #define OBD_MAX_IOCTL_BUFFER

* lustre/utils/Makefile.am
  Move obd.c and lustre_cfg.c sources to convenience library.
---
 autoMakefile.am                   |   36 ++-
 build/autoMakefile.am.toplevel    |   17 +-
 libcfs/include/Makefile.am        |    8 +
 lnet/autoconf/lustre-lnet.m4      |  102 ++++----
 lnet/include/Makefile.am          |   10 +
 lnet/include/lnet/Makefile.am     |    2 +-
 lnet/include/lnet/types.h         |  509
-------------------------------------
 lnet/include/lnet/types.h.in      |  505
++++++++++++++++++++++++++++++++++++
 lustre.spec.in                    |  156 ++++++++----
 lustre/autoconf/lustre-core.m4    |   13 +-
 lustre/include/Makefile.am        |   71 +++++-
 lustre/include/linux/Makefile.am  |    7 -
 lustre/include/linux/lustre_acl.h |    2 +-
 lustre/include/lustre/Makefile.am |   42 ---
 lustre/include/lustre_sec.h       |    4 +-
 lustre/include/lustre_ver.h.in    |   23 +-
 lustre/utils/Makefile.am          |   12 +-
 17 files changed, 814 insertions(+), 705 deletions(-)



=== 500 insertions and 500 deletions from file move.
=== patch actually 300 inserts and 200 deletes.



From jsimmons at infradead.org  Mon Nov 14 16:08:49 2011
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 14 Nov 2011 16:08:49 +0000 (GMT)
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
	<alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
	<ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>
Message-ID: <alpine.LFD.2.02.1111141607030.3705@casper.infradead.org>


> >> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.
> > 
> > Yes your right. I updated the hooks and it works now. I also had to learn 
> > you need a subsystem: field for commits now. Info about the subsystem 
> > field is not the wiki.
> 
> Hmm, which wiki page are you referring to?  The "Using Gerrit" page has had
> the "component:" tag since june, long before I sent out the email,
> and it was in the original email as well:

http://wiki.whamcloud.com/display/PUB/Using+Gerrit

Looking at the page I do see the "component:" tag, but its because I 
looked for it. It needs to be more clear what is required to push a 
patch. Just a outsider's view.


From surya1 at llnl.gov  Mon Nov 14 18:10:24 2011
From: surya1 at llnl.gov (Prakash Surya)
Date: Mon, 14 Nov 2011 10:10:24 -0800
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <alpine.LFD.2.02.1111141607030.3705@casper.infradead.org>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
	<alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
	<ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>
	<alpine.LFD.2.02.1111141607030.3705@casper.infradead.org>
Message-ID: <20111114181024.GJ9136@llnl.gov>

On Mon, Nov 14, 2011 at 08:08:49AM -0800, James Simmons wrote:
> 
> > >> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.
> > > 
> > > Yes your right. I updated the hooks and it works now. I also had to learn 
> > > you need a subsystem: field for commits now. Info about the subsystem 
> > > field is not the wiki.
> > 
> > Hmm, which wiki page are you referring to?  The "Using Gerrit" page has had
> > the "component:" tag since june, long before I sent out the email,
> > and it was in the original email as well:
> 
> http://wiki.whamcloud.com/display/PUB/Using+Gerrit
> 
> Looking at the page I do see the "component:" tag, but its because I 
> looked for it. It needs to be more clear what is required to push a 
> patch. Just a outsider's view.

Would a more detailed explanation of each "part" of the commit message
help? For example, under the sample commit message:

 * LU-000: Issue number referencing an open issue at jira.whamcloud.com

 * component: The component(s) affected by this patch (i.e. osd-ldiskfs,
              lnet, etc.)

 * Short Description: ...

 * etc...

Looking at the wiki page referenced above, it's a little hard to
distinguish each section from one another. They all seem to blend into a
single wall of text, IMO. Splitting them up into separate pages, or
simply using a better color scheme to distinguish the sections apart,
might help the page's readability as a whole.

-- 
Cheers,
Prakash

From jsimmons at infradead.org  Mon Nov 14 19:14:23 2011
From: jsimmons at infradead.org (James Simmons)
Date: Mon, 14 Nov 2011 19:14:23 +0000 (GMT)
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <20111114181024.GJ9136@llnl.gov>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
	<alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
	<ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>
	<alpine.LFD.2.02.1111141607030.3705@casper.infradead.org>
	<20111114181024.GJ9136@llnl.gov>
Message-ID: <alpine.LFD.2.02.1111141913160.19436@casper.infradead.org>


> > > >> An earlier version of the commit-msg hook used "git hash-object -t commit" and it caused an empty Change-Id string for older versions of Git. The "-t commit" option was removed from the latest version and solved the problem for the people that used it.
> > > > 
> > > > Yes your right. I updated the hooks and it works now. I also had to learn 
> > > > you need a subsystem: field for commits now. Info about the subsystem 
> > > > field is not the wiki.
> > > 
> > > Hmm, which wiki page are you referring to?  The "Using Gerrit" page has had
> > > the "component:" tag since june, long before I sent out the email,
> > > and it was in the original email as well:
> > 
> > http://wiki.whamcloud.com/display/PUB/Using+Gerrit
> > 
> > Looking at the page I do see the "component:" tag, but its because I 
> > looked for it. It needs to be more clear what is required to push a 
> > patch. Just a outsider's view.
> 
> Would a more detailed explanation of each "part" of the commit message
> help? For example, under the sample commit message:
> 
>  * LU-000: Issue number referencing an open issue at jira.whamcloud.com
> 
>  * component: The component(s) affected by this patch (i.e. osd-ldiskfs,
>               lnet, etc.)
> 
>  * Short Description: ...
> 
>  * etc...
> 
> Looking at the wiki page referenced above, it's a little hard to
> distinguish each section from one another. They all seem to blend into a
> single wall of text, IMO. Splitting them up into separate pages, or
> simply using a better color scheme to distinguish the sections apart,
> might help the page's readability as a whole.

Absolutely agree. I like the idea of a color scheme with the above format 
under the simple commit message.


From adilger at whamcloud.com  Tue Nov 15 08:55:36 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Tue, 15 Nov 2011 00:55:36 -0800
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <alpine.LFD.2.02.1111141913160.19436@casper.infradead.org>
References: <E524F377-9B95-4736-92FF-22C711DD9EE8@whamcloud.com>
	<20111103222831.GB19258@llnl.gov>
	<8FB8F9D4-A179-490D-AE9F-61C731FD98AA@whamcloud.com>
	<1320443263.7466.26.camel@brian-laptop>
	<alpine.LFD.2.02.1111091624400.14374@casper.infradead.org>
	<855EC42A-3184-49A5-AC6E-9C5012166319@whamcloud.com>
	<alpine.LFD.2.02.1111091918270.401@casper.infradead.org>
	<ABD9E0C9-58A5-43D6-B382-6E5EE7FE6688@whamcloud.com>
	<alpine.LFD.2.02.1111141607030.3705@casper.infradead.org>
	<20111114181024.GJ9136@llnl.gov>
	<alpine.LFD.2.02.1111141913160.19436@casper.infradead.org>
Message-ID: <AC8A421C-129A-4E45-A9C7-FFF531DF236C@whamcloud.com>

On 2011-11-14, at 11:14 AM, James Simmons wrote:
>>> http://wiki.whamcloud.com/display/PUB/Using+Gerrit
>>> 
>>> Looking at the page I do see the "component:" tag, but its because I 
>>> looked for it. It needs to be more clear what is required to push a 
>>> patch. Just a outsider's view.
>> 
>> Would a more detailed explanation of each "part" of the commit message
>> help? For example, under the sample commit message:
>> 
>> * LU-000: Issue number referencing an open issue at jira.whamcloud.com
>> 
>> * component: The component(s) affected by this patch (i.e. osd-ldiskfs,
>>              lnet, etc.)
>> 
>> * Short Description: ...

Have a look at: http://wiki.whamcloud.com/display/PUB/Commit+Comments
It has an updated description of the commit comment, and why having a
good one is important, as well as an improved commit comment template.
I'm submitting this va change http://review.whamcloud.com/1688, and it
is reproduced below:


LU-nnn component: short description of change under 64 columns

The "component:" should be a lower-case single-word subsystem of the
Lustre code that best encompasses the change being made.  Examples of
components include modules like: llite, lov, lmv, osc, mdc, ldlm, lnet,
ptlrpc, mds, oss, osd, ldiskfs, libcfs, socklnd, o2iblnd; functional
subsystems like: recovery, quota, grant; and auxilliary areas like:
build, tests, docs.  This list is not exhaustive, but is a guideline.

The commit comment should contain a detailed explanation of the change
being made.  This can be as long as you'd like.  Please give details
of what problem was solved (including error messages or problems that
were seen), a good high-level description of how it was solved, and
which parts of the code were changed (including important functions
that were changed, if this is useful to understand the patch, and
for easier searching).  Wrap lines at/under 70 columns.


This is the error message that you would get if you try to submit an
incorrect commit.  If people think it would be more useful, the
prepare-commit-msg hook could go whole-hog and just replace the basic
commit comment (which basically shows "git status") with a full sample
commit template that has (commented out) the above template and a
Signed-off-by: line.

>> Looking at the wiki page referenced above, it's a little hard to
>> distinguish each section from one another. They all seem to blend into a
>> single wall of text, IMO. Splitting them up into separate pages, or
>> simply using a better color scheme to distinguish the sections apart,
>> might help the page's readability as a whole.
> 
> Absolutely agree. I like the idea of a color scheme with the above format 
> under the simple commit message.

Yes, we started to split up these pages and improve the content, but
SC has interrupted this process.


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.




From adilger at whamcloud.com  Tue Nov 15 09:47:07 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Tue, 15 Nov 2011 01:47:07 -0800
Subject: [Lustre-devel] lustre-devel  packaging
In-Reply-To: <CAE66FB6.134A%bruce_korb@xyratex.com>
References: <CAE66FB6.134A%bruce_korb@xyratex.com>
Message-ID: <8B892676-E029-4052-8A98-9DACFD170C2E@whamcloud.com>

On 2011-11-14, at 7:11 AM, Bruce Korb wrote:
> We at Xyratex are needing to develop some auxiliary utilities that
> need internal data structures from lustre.  To accomplish this, our
> sources have been reaching into the lustre build tree for the
> necessary headers.

That is messy, I agree.

> Not being a particular fan of that approach, I developed a patch for
> lustre that will create another RPM: a lustre-devel package.

I like the idea of having a separate lustre-devel package, since this
is fairly standard practise for other libraries and such.

My main question would be - do you _need_ to have access to all of the
headers that you included, or did you simply include all of the headers
because that was the easiest thing to do?  Doing a simple check on the
current master tree, it appears you just copied all of the headers in
libcfs/include, lnet/include, and lustre/include (which total 186 files).

By just copying everything into "public" headers, it is introducing a
maintenance nightmare, because it is no longer clear which APIs, structs,
constants, ioctls, etc. are private to Lustre or specific tools, and
which ones might possibly be used by external tools and would cause those
tools to break if they were changed for some reason.

> The other files installed as part of lustre-devel are object
> archives and library interface man pages:
> 
> usr/lib64/libcfsutil.a
> usr/lib64/liblustreapi.a
> usr/lib64/liblustre.a
> usr/lib64/libptlctl.a
> usr/lib64/libiam.a

Now that you post this list of libraries, it in fact appears that libiam
is not even used by userspace at all anymore.  From simple checking of
lustre/utils/Makefile.am, it only shows mkfs_lustre.c as a potential user,
and that doesn't seem to use it at all.  The creation of the object index
"oi.16", which is the only IAM-format index in the filesystem, is done in
the kernel since before Lustre 2.0 was released.


So, rather than simply copying everything that is available, it would be
much better to have a discussion about what APIs you are using (or which
ones you wish would be available), and then implement llapi_* wrappers
for those interfaces.  This would at least give some level of abstraction
from the low-level interfaces, and makes it much clearer which interfaces
are actually in use by tools that are not part of the Lustre package,
(and should try to remain relatively stable), and which interfaces, ioctls,
structures, etc. are for internal use only (or sometimes not in use at all).

> usr/share/lustre/mpich-1.2.6-lustre.patch

I think a significantly improved version this patch was already included
in the MPICH upstream release, and this one is obsolete.

> usr/share/man/man3/llapi_quotactl.3.gz
> usr/share/man/man3/llapi_file_open.3.gz
> usr/share/man/man3/llapi_file_create.3.gz
> usr/share/man/man3/llapi_file_get_stripe.3.gz

> 
> Below is my git-commit message.  Under separate cover, I'll post the
> git patch under the subject "[PATCH] lustre-devel packaging"

Looking at the descriptions, the patches look quite reasonable.  However,
you need to submit patches to Gerrit in order to get them inspected,
tested, and landed.  Please see:

http://wiki.whamcloud.com/display/PUB/Submitting+Changes

and in particular:

http://wiki.whamcloud.com/display/PUB/Using+Gerrit

There are a number of people at Xyratex already following this process
that you can likely ask for guidance.

Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.




From bruce_korb at xyratex.com  Tue Nov 15 14:23:31 2011
From: bruce_korb at xyratex.com (Bruce Korb)
Date: Tue, 15 Nov 2011 06:23:31 -0800
Subject: [Lustre-devel] [wc-discuss] Lustre code style Git commit hooks
In-Reply-To: <AC8A421C-129A-4E45-A9C7-FFF531DF236C@whamcloud.com>
Message-ID: <CAE7B53C.1381%bruce_korb@xyratex.com>

Cool:

On 11/15/11 12:55 AM, Andreas Dilger wrote:

>On 2011-11-14, at 11:14 AM, James Simmons wrote:
>This is the error message that you would get if you try to submit an
>incorrect commit.  If people think it would be more useful, the
>prepare-commit-msg hook could go whole-hog and just replace the basic
>commit comment (which basically shows "git status") with a full sample
>commit template that has (commented out) the above template and a
>Signed-off-by: line.

Just a brief synopsis of the "transgression" and a URL to the full
description would be excellent.  (I don't like having to Google an
error message to find the full docs and the full docs in an error
message is overly verbose.)

Thank you!



From ravibadiger24 at gmail.com  Tue Nov 15 12:50:42 2011
From: ravibadiger24 at gmail.com (ravi badiger)
Date: Tue, 15 Nov 2011 18:20:42 +0530
Subject: [Lustre-devel] http://jira.whamcloud.com/browse/LU-828
Message-ID: <CAKh5fBGma9yPky6T5-8rUQJdkON7-gjoaZrHQyZKXeTxXVNO-g@mail.gmail.com>

Hi Cliffw,Adilger,All,

We have uploaded all Lustre server syslogs for both /home /scratch lustre
filesystems also sample client Syslog where problem was noticed.
Can some one please look at the cause for the below reported issue at
whamcloud jira and how we can proceed to overcome it.

http://jira.whamcloud.com/browse/LU-828

Thanks & Regard
-Ravi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel_lists.lustre.org/attachments/20111115/9d98a66e/attachment.html>

From bruce_korb at xyratex.com  Tue Nov 15 15:22:17 2011
From: bruce_korb at xyratex.com (Bruce Korb)
Date: Tue, 15 Nov 2011 07:22:17 -0800
Subject: [Lustre-devel] lustre-devel  packaging - LU-482
In-Reply-To: <8B892676-E029-4052-8A98-9DACFD170C2E@whamcloud.com>
Message-ID: <CAE7B612.1389%bruce_korb@xyratex.com>

Hi Adreas,

On 11/15/11 1:47 AM, Andreas Dilger wrote:

>On 2011-11-14, at 7:11 AM, Bruce Korb wrote:
>My main question would be - do you _need_ to have access to all of the
>headers that you included, or did you simply include all of the headers
>because that was the easiest thing to do?  Doing a simple check on the
>current master tree, it appears you just copied all of the headers in
>libcfs/include, lnet/include, and lustre/include (which total 186 files).

Lets assume I did a minimal approach and only included the necessary
headers.  Then, someone oblivious to which headers get exported and
which do not, then added a new dependency into the headers.  Everything
builds and checks out so it looks right and you ship the new version.
Except it isn't because of the new inadvertent dependency.  Oops.

The correct and proper solution is for each component of lustre to
explicitly copy interface headers into an include staging area
with everything under that getting installed.  I'm not going there.
That is a future exercise.  Probably ought to be done, though.
OK, *surely* ought to be done.  :)

>By just copying everything into "public" headers, it is introducing a
>maintenance nightmare, because it is no longer clear which APIs, structs,
>constants, ioctls, etc. are private to Lustre or specific tools, and
>which ones might possibly be used by external tools and would cause those
>tools to break if they were changed for some reason.

Were the private vs. public headers separated in some way, then
the unwanted exports could be removed.  At the moment, I think
the assumption has to be that clients of the lustre-devel package
would have to be friendly clients.  Very friendly.  Right now, we
have actual clients grubbing around all over the lustre source
tree in a very unfriendly way.

>> The other files installed as part of lustre-devel areŠ..
>So, rather than simply copying everything that is available, it would be
>much better to have a discussion about what APIs you are using (or which
>ones you wish would be available), and then implement llapi_* wrappers

Ultimately, that is completely correct.  For now, I'm mostly interested in
getting headers and libraries installed in a way where I'm not dependent
upon the build tree layout.  It is already understood that if lustre
changes
internals these utilities have to adapt.  Adapting semi-private interfaces
to a coherent framework would be such a change.  This change simply
isolates
our auxiliary utilities from changes in build layout.  That's step 1.
I (we at Xyratex) would be completely okay with a usage caveat that the
interfaces exposed are subject to change while the process of working out
exported vs. completely private interfaces goes on.

>>usr/share/lustre/mpich-1.2.6-lustre.patch
>
>I think a significantly improved version this patch was already included
>in the MPICH upstream release, and this one is obsolete.

The .spec file needs to adapt to whatever gets staged into BUILDROOT.
Some of that may be hard coded and need changing, but the .spec file
ought to be as scripted as possible, minimizing the need for any changes
when the lustre subcomponents change the set of files they install.

>> 
>> Below is my git-commit message.  Under separate cover, I'll post the
>> git patch under the subject "[PATCH] lustre-devel packaging"
>
>Looking at the descriptions, the patches look quite reasonable.  However,
>you need to submit patches to Gerrit in order to get them inspected,
>tested, and landed.

I was actually starting with email before going to a formal review request.
You have already seen the review request for changes I consider less
controversial (see "P.S." below), even if the changes were too unfocused
as a single patch.

>Cheers, Andreas

Thank you!!  Regards, Bruce

P.S. the other issue (LU-483) got combined because of procedural issues.
Sorry about that.  I will do as you ask within a few days and break it
up into several independent patches, but still under LU-483, yes?



From Nathan_Rutman at xyratex.com  Tue Nov 15 15:43:31 2011
From: Nathan_Rutman at xyratex.com (Nathan Rutman)
Date: Tue, 15 Nov 2011 07:43:31 -0800
Subject: [Lustre-devel] [lustre-devel] lustre-devel packaging - LU-482
In-Reply-To: <CAE7B612.1389%bruce_korb@xyratex.com>
References: <CAE7B612.1389%bruce_korb@xyratex.com>
Message-ID: <9C6215E9-C385-44B7-BF35-2A20E85C8967@us.xyratex.com>



On Nov 15, 2011, at 7:22 AM, "Bruce Korb" <bruce_korb at xyratex.com> wrote:

> Hi Adreas,
> 
> On 11/15/11 1:47 AM, Andreas Dilger wrote:
> 
>> On 2011-11-14, at 7:11 AM, Bruce Korb wrote:
>> My main question would be - do you _need_ to have access to all of the
>> headers that you included, or did you simply include all of the headers
>> because that was the easiest thing to do?  Doing a simple check on the
>> current master tree, it appears you just copied all of the headers in
>> libcfs/include, lnet/include, and lustre/include (which total 186 files).
> 
> 
> Were the private vs. public headers separated in some way, then
> the unwanted exports could be removed.  At the moment, I think
> the assumption has to be that clients of the lustre-devel package
> would have to be friendly clients.  Very friendly.  Right now, we
> have actual clients grubbing around all over the lustre source
> tree in a very unfriendly way.
Actually I thought we were careful to only expose llapi_* functions. So what's doing the grubbing?


From bruce_korb at xyratex.com  Wed Nov 16 22:22:38 2011
From: bruce_korb at xyratex.com (Bruce Korb)
Date: Wed, 16 Nov 2011 14:22:38 -0800
Subject: [Lustre-devel] lustre-devel  packaging - LU-482
In-Reply-To: <CAE7B612.1389%bruce_korb@xyratex.com>
Message-ID: <CAE97569.14B4%bruce_korb@xyratex.com>

Hi Andreas,

I did do a little more due diligence:

On 11/15/11 7:22 AM, Bruce Korb wrote:
>>My main question would be - do you _need_ to have access to all of the
>>headers that you included, or did you simply include all of the headers
>>because that was the easiest thing to do?  Doing a simple check on the
>>current master tree, it appears you just copied all of the headers in
>>libcfs/include, lnet/include, and lustre/include (which total 186 files).
>
>Lets assume I did a minimal approach and only included the necessary


  libcfs/libcfsutil.h
  lnet/lnetctl.h
  lustre/liblustreapi.h
  lustre/lustre_idl.h
  test.h
  utils/obdctl.h

These are the headers directly #include-d by our "utility"

that live in the lustre tree.  They likely pull in several more.
As I said elsewhere somewhere, unless the libcfs, lnet, lustre and utils
components of lustre know that these are exported via an unambiguous
mechanism, installing only the minimal subset will be prone to problems.
A better way is to have them stage the headers and run a validation that
they are all both self-sufficient and idempotent, meaning that any
requisite headers are also staged and they are all guarded with duplicate
inclusion guards.  This can be done with a trivial script.

I can make this part of the "lustre-devel package" project.  I was trying
to limit scope by installing everything in sight.  At least for now.



From adilger at whamcloud.com  Thu Nov 17 01:35:56 2011
From: adilger at whamcloud.com (Andreas Dilger)
Date: Wed, 16 Nov 2011 17:35:56 -0800
Subject: [Lustre-devel] lustre-devel  packaging - LU-482
In-Reply-To: <CAE97569.14B4%bruce_korb@xyratex.com>
References: <CAE97569.14B4%bruce_korb@xyratex.com>
Message-ID: <ABDA628E-8DA2-4CB8-BC57-371942B013D5@whamcloud.com>

On 2011-11-16, at 14:22, Bruce Korb <bruce_korb at xyratex.com> wrote:
> 
> I did do a little more due diligence:
> 
> On 11/15/11 7:22 AM, Bruce Korb wrote:
>>> My main question would be - do you _need_ to have access to all of the
>>> headers that you included, or did you simply include all of the headers
>>> because that was the easiest thing to do?  Doing a simple check on the
>>> current master tree, it appears you just copied all of the headers in
>>> libcfs/include, lnet/include, and lustre/include (which total 186 files).
>> 
>> Lets assume I did a minimal approach and only included the necessary
> 
>  libcfs/libcfsutil.h
>  lnet/lnetctl.h
>  lustre/liblustreapi.h
>  lustre/lustre_idl.h
>  test.h
>  utils/obdctl.h
> 
> These are the headers directly #include-d by our "utility"

Including liblustrapi.h is expected, since this is the entry point for the Lustre wrappers, and this header is already installed. 

I did some work several months ago to make lustre_idl.h usable from userspace for lfsck, which works OK except for the use of __u32/__u64 and friends, which needs the "types.h" header to define.

I haven't looked at the other headers (just about to get on a plane) but I think they might be acceptable for low-level Lustre specific applications. That said, the only previous user of these headers is probably lctl, and I'm hesitant to expose them as an "API".

As Nathan wrote, we've tried in the past to do the right thing and create llapi wrappers for code that needs to poke into the Lustre kernel interfaces. This allows things like e.g. fixing the ioctl numbers, changing the data structures used with the kernel, etc. without having to modify the applications that are using these wrappers.

I'm the first one to admit that the llapi_* wrappers do not address a large number of use cases, but that also won't get better if nobody works to improve them. 

I guess a reasonable question at this point is what specifically you are trying to access?  Is it (essentially) trying to link directly into lctl, lfs, etc?  Access to wire protocol structures (that lustre_idl.h should handle), or something else entirely?

> As I said elsewhere somewhere, unless the libcfs, lnet, lustre and utils
> components of lustre know that these are exported via an unambiguous
> mechanism, installing only the minimal subset will be prone to problems.
> A better way is to have them stage the headers and run a validation that
> they are all both self-sufficient and idempotent, meaning that any
> requisite headers are also staged and they are all guarded with duplicate
> inclusion guards.  This can be done with a trivial script.
> 
> I can make this part of the "lustre-devel package" project.  I was trying
> to limit scope by installing everything in sight.  At least for now.
> 
> 

From alexey_lyashkov at xyratex.com  Thu Nov 17 12:08:39 2011
From: alexey_lyashkov at xyratex.com (Alexey Lyashkov)
Date: Thu, 17 Nov 2011 16:08:39 +0400
Subject: [Lustre-devel] lustre-devel  packaging - LU-482
In-Reply-To: <ABDA628E-8DA2-4CB8-BC57-371942B013D5@whamcloud.com>
References: <CAE97569.14B4%bruce_korb@xyratex.com>
	<ABDA628E-8DA2-4CB8-BC57-371942B013D5@whamcloud.com>
Message-ID: <84CBE853-0699-4AF2-B284-798CA99FD4C1@xyratex.com>


On Nov 17, 2011, at 05:35, Andreas Dilger wrote:

> On 2011-11-16, at 14:22, Bruce Korb <bruce_korb at xyratex.com> wrote:
>> 
>> I did do a little more due diligence:
>> 
>> On 11/15/11 7:22 AM, Bruce Korb wrote:
>>>> My main question would be - do you _need_ to have access to all of the
>>>> headers that you included, or did you simply include all of the headers
>>>> because that was the easiest thing to do?  Doing a simple check on the
>>>> current master tree, it appears you just copied all of the headers in
>>>> libcfs/include, lnet/include, and lustre/include (which total 186 files).
>>> 
>>> Lets assume I did a minimal approach and only included the necessary
>> 
>> libcfs/libcfsutil.h
>> lnet/lnetctl.h
>> lustre/liblustreapi.h
>> lustre/lustre_idl.h
>> test.h
>> utils/obdctl.h
>> 
>> These are the headers directly #include-d by our "utility"
> 
> Including liblustrapi.h is expected, since this is the entry point for the Lustre wrappers, and this header is already installed. 
> 
> I did some work several months ago to make lustre_idl.h usable from userspace for lfsck, which works OK except for the use of __u32/__u64 and friends, which needs the "types.h" header to define.
that should be done via libcfs.h i think ?