<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Andreas,</p>
<p>As I mentioned in an earlier email, this had been working for a
long time. I think that using an old header file is at the root
of the issue. On my development platform, which doesn't have
Lustre installed, nor did I have e2fsprogs installed, I had simply
copied the Lustre files I needed from the site I was working
with. The fiemap.h file I was using , the top of which is shown
below ( I see you mentioned ) has fe_device explicitly in the
structure. Was it this way before the #define fe_device was
implemented? The #define was using the fe_reserved[0], which
always had a 0 value. What puzzles me is why this ever worked at
all. That will have to wait for a rainy day to mess with. What
started me down this path at this time was getting my lustre
extents plotting program working with PFL.<br>
</p>
<p>Again, thanks much for your excellent/quick assistance in
tracking this down. <br>
</p>
<p>John<br>
</p>
<pre><font color="#3465A4">/*</font>
<font color="#3465A4"> * FS_IOC_FIEMAP ioctl infrastructure.</font>
<font color="#3465A4"> *</font>
<font color="#3465A4"> * Some portions copyright (C) 2007 Cluster File Systems, Inc</font>
<font color="#3465A4"> *</font>
<font color="#3465A4"> * Authors: Mark Fasheh <a class="moz-txt-link-rfc2396E" href="mailto:mfasheh@suse.com"><mfasheh@suse.com></a></font>
<font color="#3465A4"> * Kalpak Shah <a class="moz-txt-link-rfc2396E" href="mailto:kalpak.shah@sun.com"><kalpak.shah@sun.com></a></font>
<font color="#3465A4"> * Andreas Dilger <a class="moz-txt-link-rfc2396E" href="mailto:adilger@sun.com"><adilger@sun.com></a></font>
<font color="#3465A4"> */</font>
<font color="#75507B">#ifndef _LINUX_FIEMAP_H</font>
<font color="#75507B">#define _LINUX_FIEMAP_H</font>
<font color="#4E9A06">struct</font> fiemap_extent {
__u64 fe_logical; <font color="#3465A4">/* logical offset in bytes for the start of</font>
<font color="#3465A4"> * the extent from the beginning of the file */</font>
__u64 fe_physical; <font color="#3465A4">/* physical offset in bytes for the start</font>
<font color="#3465A4"> * of the extent from the beginning of the disk */</font>
__u64 fe_length; <font color="#3465A4">/* length in bytes for this extent */</font>
__u64 fe_reserved64[<font color="#CC0000">2</font>];
__u32 fe_flags; <font color="#3465A4">/* FIEMAP_EXTENT_* flags for this extent */</font>
__u32 fe_device; <font color="#3465A4">/* device number (fs-specific if FIEMAP_EXTENT_NET)*/</font>
__u32 fe_reserved[<font color="#CC0000">2</font>];
};
</pre>
<div class="moz-cite-prefix">On 8/18/22 23:44, Andreas Dilger wrote:<br>
</div>
<blockquote type="cite"
cite="mid:F46BFD2B-476D-4216-BD2B-5586EAEF5026@whamcloud.com">
The "fe_device" field is actually Lustre-specific, so it is a
macro that overlays on fe_reserved[0]:
<div class=""><br class="">
</div>
<div class=""> #define fe_device fe_reserved[0]</div>
<div class=""><br class="">
</div>
<div class="">but that shouldn't affect compiler alignment. On my
system, "pahole lustre/llite/lustre.ko" reports:</div>
<div class=""><br class="">
</div>
<div class="">struct fiemap_extent {<br class="">
__u64 fe_logical; /*
0 8 */<br class="">
__u64 fe_physical; /*
8 8 */<br class="">
__u64 fe_length; /*
16 8 */<br class="">
__u64 fe_reserved64[2]; /*
24 16 */<br class="">
__u32 fe_flags; /*
40 4 */<br class="">
__u32 fe_reserved[3]; /*
44 12 */<br class="">
<br class="">
/* size: 56, cachelines: 1, members: 6 */<br class="">
/* last cacheline: 56 bytes */<br class="">
};</div>
<div class=""><br class="">
</div>
<div class="">So there is definitely something going wrong with
the struct alignment for fe_reserved, even though there doesn't
need to be (all of the fields have "natural" alignment on their
4/8-byte sizes.</div>
<div class=""><br class="">
</div>
<div class="">The other thing that is strange is that you show
only 2 fe_reserved[] fields, when I have 3. Is there some other
field added to your version of struct fiemap_extent after
fe_flags? I don't see anything in the upstream kernel, nor in
the Lustre headers.</div>
<div class=""><br class="">
</div>
<div class="">You could try adding "__attribute__((packed))" at
the end of the struct definition to see if that fixes the
problem.<br class="">
<div><br class="">
</div>
<div>Cheers, Andreas</div>
<div><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">On Aug 18, 2022, at 21:54, John Bauer <<a
href="mailto:bauerj@iodoctors.com"
class="moz-txt-link-freetext" moz-do-not-send="true">bauerj@iodoctors.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">Andreas,</p>
<p class="">This is no longer Lustre related, but I hope
you can shed some light on this. It appears that my
compilier, gcc 8.5.0, which I upgraded to recently
when I upgraded my build system to Centos 8, is not
padding the struct fiemap_extent correctly. I put the
following prints in to see whats going on. The sizeof
the structure is good at 56, but notice that both
fe_device and fe_reserved[0] have an offset of 48
bytes into the structure. Odd that the sizeof
fe_flags is 4, but fe_device is 8 bytes away from it.
I traced the compile to ensure that I am getting the
lustre_include/ext2fs/fiemap.h and there is nothing
odd in the fiemap.h ( it's the one I've been using for
years ). Any thoughts on how to remedy this?</p>
<p class="">John<br class="">
</p>
<pre class="">fprintf(stderr,"%s() logical %d %ld\n", __func__, sizeof(fm_ext->fe_logical ), (char *)&fm_ext->fe_logical -(char *)fm_ext);
fprintf(stderr,"%s() physical %d %ld\n", __func__, sizeof(fm_ext->fe_physical), (char *)&fm_ext->fe_physical -(char *)fm_ext);
fprintf(stderr,"%s() length %d %ld\n", __func__, sizeof(fm_ext->fe_length ), (char *)&fm_ext->fe_length - (char *)fm_ext);
fprintf(stderr,"%s() res64[0] %d %ld\n", __func__, sizeof(fm_ext->fe_reserved64[0]), (char *)&fm_ext->fe_reserved64[0] - (char *)fm_ext);
fprintf(stderr,"%s() res64[1] %d %ld\n", __func__, sizeof(fm_ext->fe_reserved64[1]), (char *)&fm_ext->fe_reserved64[1] - (char *)fm_ext);
fprintf(stderr,"%s() flags %d %ld\n", __func__, sizeof(fm_ext->fe_flags ), (char *)&fm_ext->fe_flags -(char *)fm_ext);
fprintf(stderr,"%s() device %d %ld\n", __func__, sizeof(fm_ext->fe_device ), (char *)&fm_ext->fe_device -(char *)fm_ext);
fprintf(stderr,"%s() res32[0] %d %ld\n", __func__, sizeof(fm_ext->fe_reserved[0]), (char *)&fm_ext->fe_reserved[0] - (char *)fm_ext);
fprintf(stderr,"%s() res32[1] %d %ld\n", __func__, sizeof(fm_ext->fe_reserved[1]), (char *)&fm_ext->fe_reserved[1] - (char *)fm_ext);
</pre>
<pre class="">StripeChunks_get() fm_ext->fe_device=0 fe_logical=0 sizeof(struct fiemap_extent)56
StripeChunks_get() logical 8 0
StripeChunks_get() physical 8 8
StripeChunks_get() length 8 16
StripeChunks_get() res64[0] 8 24
StripeChunks_get() res64[1] 8 32
StripeChunks_get() flags 4 40
StripeChunks_get() device 4 48
StripeChunks_get() res32[0] 4 48
StripeChunks_get() res32[1] 4 52
</pre>
<div class="moz-cite-prefix">On 8/18/22 16:11, Andreas
Dilger wrote:<br class="">
</div>
<blockquote type="cite"
cite="mid:17DE058E-FEF9-43FD-B39B-3A8BE7583B8F@whamcloud.com"
class="">
<div class="">
<div class="">On Aug 18, 2022, at 14:28, John Bauer
<<a href="mailto:bauerj@iodoctors.com"
class="moz-txt-link-freetext"
moz-do-not-send="true">bauerj@iodoctors.com</a>>
wrote:</div>
<blockquote type="cite" class=""><br
class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">Andreas,</p>
<p class="">Thanks for the reply. I don't
think I'm accessing the Lustre filefrag (
see below ). Where would I normally find
that installed? I downloaded the
lustre-release git repository and can't find
filefrag stuff to build my own. Is that
somewhere else?</p>
</div>
</div>
</blockquote>
filefrag is part of the e2fsprogs package ("rpm -qf
$(which filefrag)"), so you need to download and
install the Lustre-patched e2fsprogs from <u
class=""><a
href="https://downloads.whamcloud.com/public/e2fsprogs/latest/"
class="moz-txt-link-freetext"
moz-do-not-send="true">https://downloads.whamcloud.com/public/e2fsprogs/latest/</a></u><br
class="">
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">
<p class="">More info: <br class="">
</p>
<pre class="">pfe27.jbauer2 334> cat /sys/fs/lustre/version
2.12.8_ddn12
</pre>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
You should really use "lctl get_param version",
since the Lustre /proc and /sys files move around on
occasion.</div>
<div class=""><br class="">
</div>
<div class="">The PFL/FLR change for FIEMAP is not
included in this version, but it _should_ be
irrelevant because the file you are testing is using
a plain layout, not PFL or FLR.<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">
<pre class="">pfe27.jbauer2 335> filefrag -v /nobackupp17/jbauer2/dd.dat
Filesystem type is: bd00bd0
File size of /nobackupp17/jbauer2/dd.dat is 104857600 (25600 blocks of 4096 bytes)
/nobackupp17/jbauer2/dd.dat: FIBMAP unsupported
pfe27.jbauer2 336> which filefrag
/usr/sbin/filefrag
</pre>
<div class="moz-cite-prefix">John<br class="">
</div>
<div class="moz-cite-prefix"><br class="">
</div>
<div class="moz-cite-prefix">On 8/18/22 14:57,
Andreas Dilger wrote:<br class="">
</div>
<blockquote type="cite"
cite="mid:E166F2A4-5F6B-43CC-8E8E-055B25B9A97C@whamcloud.com"
class="">
What version of Lustre are you using? Does
"filefrag -v" from a newer Lustre e2fsprogs
(1.45.6.wc3+) work properly?
<div class=""><br class="">
</div>
<div class="">There was a small change to
the Lustre FIEMAP handling in order to
handle overstriped files and PFL/FLR files
with many stripes and multiple components,
since the FIEMAP "restart" mechanism was
broken for files that had multiple objects
on the same OST index. See LU-11484 for
details. That change was included in the
2.14.0 release.</div>
<div class=""><br class="">
</div>
<div class="">In essence, the fe_device
field now encodes the absolute file stripe
number in the high 16 bits of fe_device,
and the device number in the low 16 bits
(as it did before). Since "filefrag -v"
prints fe_device in hex and would show as
"0x<stripe><device>" instead
of "0x0000<device>", this was
considered an acceptable tradeoff compared
to other "less compatible" changes that
would have been needed to implement
PFL/FLR handling.</div>
<div class=""><br class="">
</div>
<div class="">That said, I would have
expected this change to result in your
tool reporting very large values for
fe_device (e.g. OST index + N * 65536), so
returning all-zero values is somewhat
unexpected. </div>
<div class=""><br class="">
</div>
<div class="">Cheers, Andreas</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 18, 2022, at
06:27, John Bauer <<a
href="mailto:bauerj@iodoctors.com"
class="moz-txt-link-freetext"
moz-do-not-send="true">bauerj@iodoctors.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hi all,<br class="">
<br class="">
I am trying to get my llfie
program (which uses fiemap) going
again, but now the struct
fiemap_extent structures I get
back from the ioctl call, all have
fe_device=0. The output from lfs
getstripe indicates that the
devices are not all 0. The sum of
the fe_length members adds up to
the file size, so that is
working. The fe_physical members
look reasonable also. Has
something changed? This used to
work.<br class="">
<br class="">
Thanks, John<br class="">
<br class="">
pfe27.jbauer2 300> llfie
/nobackupp17/jbauer2/dd.dat<br
class="">
LustreStripeInfo_get()
lum->lmm_magic=0xbd30bd0<br
class="">
listExtents()
fe_physical=30643484360704
fe_device=0 fe_length=16777216<br
class="">
listExtents()
fe_physical=30646084829184
fe_device=0 fe_length=2097152<br
class="">
listExtents()
fe_physical=5705226518528
fe_device=0 fe_length=16777216<br
class="">
listExtents()
fe_physical=5710209351680
fe_device=0 fe_length=2097152<br
class="">
listExtents()
fe_physical=30621271326720
fe_device=0 fe_length=16777216<br
class="">
listExtents()
fe_physical=31761568366592
fe_device=0 fe_length=16777216<br
class="">
listExtents()
fe_physical=24757567225856
fe_device=0 fe_length=16777216<br
class="">
listExtents()
fe_physical=14196460748800
fe_device=0 fe_length=16777216<br
class="">
listExtents() nMapped=8
byteCount=104857600<br class="">
<br class="">
<br class="">
pfe27.jbauer2 301> lfs
getstripe
/nobackupp17/jbauer2/dd.dat<br
class="">
/nobackupp17/jbauer2/dd.dat<br
class="">
lmm_stripe_count: 6<br class="">
lmm_stripe_size: 2097152<br
class="">
lmm_pattern: raid0<br
class="">
lmm_layout_gen: 0<br class="">
lmm_stripe_offset: 126<br class="">
lmm_pool: ssd-pool<br
class="">
<span class="Apple-tab-span"></span>obdidx<span
class="Apple-tab-span"> </span><span
class="Apple-tab-span"></span>objid<span
class="Apple-tab-span">
</span><span
class="Apple-tab-span"></span>objid<span
class="Apple-tab-span"> </span>
<span class="Apple-tab-span"></span>group<br
class="">
<span class="Apple-tab-span"></span> 126<span
class="Apple-tab-span"> </span> 13930025<span
class="Apple-tab-span">
</span> 0xd48e29<span
class="Apple-tab-span"> </span> 0<br
class="">
<span class="Apple-tab-span"></span> 113<span
class="Apple-tab-span"> </span> 13115889<span
class="Apple-tab-span">
</span> 0xc821f1<span
class="Apple-tab-span"> </span> 0<br
class="">
<span class="Apple-tab-span"></span> 120<span
class="Apple-tab-span"> </span> 14003176<span
class="Apple-tab-span">
</span> 0xd5abe8<span
class="Apple-tab-span"> </span> 0<br
class="">
<span class="Apple-tab-span"></span> 109<span
class="Apple-tab-span"> </span> 12785483<span
class="Apple-tab-span">
</span> 0xc3174b<span
class="Apple-tab-span"> </span> 0<br
class="">
<span class="Apple-tab-span"></span> 102<span
class="Apple-tab-span"> </span> 13811117<span
class="Apple-tab-span">
</span> 0xd2bdad<span
class="Apple-tab-span"> </span> 0<br
class="">
<span class="Apple-tab-span"></span> 116<span
class="Apple-tab-span"> </span> 13377285<span
class="Apple-tab-span">
</span> 0xcc1f05<span
class="Apple-tab-span"> </span> 0<br
class="">
<br class="">
_______________________________________________<br class="">
lustre-discuss mailing list<br
class="">
<a
href="mailto:lustre-discuss@lists.lustre.org"
class="moz-txt-link-freetext"
moz-do-not-send="true">lustre-discuss@lists.lustre.org</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org"
moz-do-not-send="true">http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org</a><br
class="">
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Cheers,
Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal
Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div class="">Cheers, Andreas</div>
<div class="">--</div>
<div class="">Andreas Dilger</div>
<div class="">Lustre Principal Architect</div>
<div class="">Whamcloud</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
<div class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div dir="auto" class="">
<div>Cheers, Andreas</div>
<div>--</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</blockquote>
</body>
</html>