[lustre-discuss] llapi_layout_file_comp_del
John Bauer
bauerj at iodoctors.com
Mon Aug 1 13:46:56 PDT 2022
Andeas and others,
Thanks again for all the info. I guess its about time for me to attempt
to contribute to the Lustre code base. A question about
llapi_layout_dom_size(). I have code that is building up a multi
component layout. After switching to the second component with
llapi_layout_comp_use( layout, LLAPI_COMP_USE_NEXT ) I call
llapi_layout_dom_size() so I know where to set the extent start for
second component. Internal to ...dom_size() the current component gets
set back to the first component, see below. I then make calls thinking
I am modifying the second component, but I am actually modifying the
first. Is this behavior documented somewhere, that calls to
llapi_layout_... functions can change the current component without notice?
Is there a llapi function that returns the lustre file system's max dom
size? I mistakenly thought this was the purpose of
llapi_layout_dom_size(), but found out I made a bad assumption. Trying
to find the max dom_size is what sent me down this path.
John
int llapi_layout_dom_size(struct llapi_layout *layout,uint64_t *size)
{
uint64_t pattern, start;
int rc;
if (!layout || !llapi_layout_is_composite(layout)) {
*size =0;
return 0;
}
rc = llapi_layout_comp_use(layout, LLAPI_LAYOUT_COMP_USE_FIRST);
if (rc)
return -errno;
rc = llapi_layout_pattern_get(layout, &pattern);
if (rc)
return -errno;
if (pattern != LOV_PATTERN_MDT && pattern != LLAPI_LAYOUT_MDT) {
*size =0;
return 0;
}
rc = llapi_layout_comp_extent_get(layout, &start, size);
if (rc)
return -errno;
if (start)
return -ERANGE;
return 0;
}
On 7/28/22 19:42, Andreas Dilger wrote:
> John, you are probably right that allowing a passed fd would also be
> useful, but nobody has done it this way before because of the need to
> use O_LOV_DELAY_CREATE within the application code... and to be
> honest very few applications tune their IO to this extent, especially
> with PFL layouts being available to handle 99% of the usage.
>
> Please feel free to submit a patch that splits
> llapi_layout_file_open() into two functions:
> - llapi_layout_open_fd() (or ...fd_open()? not sure) that only opens
> the file with O_LOV_DELAY_CREATE and returns the fd
> - llapi_layout_set_by_fd() that sets the layout on the provided fd
> (whatever the source)
>
> Then llapi_layout_file_open() would just call those two functions
> internally, and you could use only the llapi_layout_set_by_fd()
> function, if available (you could add:
>
> #defeine HAVE_LLAPI_LAYOUT_SET_BY_FD
>
> in the lustreapi.h header to avoid the need for a separate configure
> check. Otherwise, your code would call your own internal wrapper of
> the same name that calls llapi_layout_file_open() and immediately
> closes the returned fd. That would be slightly less efficient than
> the new API (two opens and closes), but would allow you to migrate to
> the new (more efficient) implementation easily in the future.
>
> Cheers, Andreas
>
>> On Jul 28, 2022, at 14:03, John Bauer <bauerj at iodoctors.com> wrote:
>>
>> Andreas,
>>
>> Thanks for the info. A related question: I am using the
>> O_LOV_DELAY_CREATE open flag mechanism to open a file and then set
>> the composite layout with llapi_layout_file_open(). I was kind of
>> surprised this worked. This ends up opening the file twice and I
>> simply close the fd returned from llapi_layout_file_open(). It would
>> seem there should be an llapi function such as
>> llapi_layout_set_by_fd() to match the llapi_layout_get_by_fd(). I
>> need to use this mechanism to set striping for files where the
>> pathname is not necessarily known before the open, such as the
>> mkstemps() family of opens. It also makes it easier to handle
>> setting striping for files opened with openat(). It seems it would
>> be more straight forward for llapi to work with an fd than a pathname
>> if a valid fd already exists. Am I missing an easier way to do this?
>>
>> Thanks,
>>
>> John
>>
>> On 7/27/22 16:25, Andreas Dilger wrote:
>>> The HLD document was written before the feature was implemented, and
>>> is outdated. The lustreapi.h and llapi_layout_file_comp_del.3 man
>>> page are correct. Feel free to update the wiki to use the correct
>>> argument list.
>>>
>>> I believe that it is possible to delete multiple components that
>>> match the <flags> argument (e.g. LCME_FL_NEG|LCME_FL_INIT), but I
>>> haven't tested that.
>>>
>>>> On Jul 26, 2022, at 14:35, John Bauer <bauerj at iodoctors.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I would like to use the llapi_layout_file_comp_del() function. I
>>>> have found 2 prototypes in different places. One has the 3rd
>>>> argument, uint32_t flags, and the other doesn't. I suspect the
>>>> High Level Design document is incorrect. The one line of
>>>> documentation in lustreapi.h indicates I could delete multiple
>>>> components with one call. How does one do that? What are the
>>>> applicable flags?
>>>>
>>>> From version 2.12.8 lustreapi.h
>>>>
>>>> /**
>>>> * Delete component(s) by the specified component id or flags.
>>>> */
>>>> int llapi_layout_file_comp_del(const char *path, uint32_t id,
>>>> uint32_t flags);
>>>>
>>>>
>>>> From https://wiki.lustre.org/PFL2_High_Level_Design
>>>>
>>>> A new interface llapi_layout_file_comp_del(3) to delete
>>>> component(s) by the specified component id (accepting LCME_ID_*
>>>> wildcards also) from an existing file:
>>>>
>>>> int llapi_layout_file_comp_del(const char *path, uint32_t id);
>>>>
>>>> John
>>>>
>>>> _______________________________________________
>>>> lustre-discuss mailing list
>>>> lustre-discuss at lists.lustre.org
>>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>> Cheers, Andreas
>>> --
>>> Andreas Dilger
>>> Lustre Principal Architect
>>> Whamcloud
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20220801/d3778203/attachment.htm>
More information about the lustre-discuss
mailing list