[lustre-discuss] Hole Identification in file

[Dilger, Andreas]

The FIEMAP ioctl is implemented slightly differently for multi-stripe files in Lustre - it essentially does the FIEMAP separately for each stripe so that it is easier to see whether the file is fragmented. See the filefrag program in the Lustre-patched e2fsprogs.

Cheers, Andreas

On Mar 9, 2018, at 06:01, lokesh jaliminche <lokesh.jaliminche at gmail.com<mailto:lokesh.jaliminche at gmail.com>> wrote:

Ahh got it!!
Meanwhile, I found one implicit method to find holes on the file i.e. using fiemap ioctl.
I have tested it and works fine

Logs :
=====
dd if=/dev/urandom  of=sparse_file seek=10 bs=1M  count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0969376 s, 10.8 MB/s
[lokesh]# dd if=/dev/urandom  of=sparse_file seek=20 bs=1M  count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.087068 s, 12.0 MB/s
[lokesh]# dd if=/dev/urandom  of=sparse_file seek=30 bs=1M  count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0918352 s, 11.4 MB/s
[lokesh]# dd if=/dev/urandom  of=sparse_file seek=40 bs=1M  count=10
^[[A^[[D10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.946796 s, 11.1 MB/s
[lokesh]# dd if=/dev/urandom  of=sparse_file seek=50 bs=1M  count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.978465 s, 10.7 MB/s
[lokesh]# ./a.out sparse_file
Extents in file "sparse_file":    4
Extents returned: 4
Logical: ###[10485760]    Ext length: ###[1048576]    Physical: ###[104666759168]    flags: 2147483648
Logical: ###[20971520]    Ext length: ###[1048576]    Physical: ###[104677244928]    flags: 2147483648
Logical: ###[31457280]    Ext length: ###[1048576]    Physical: ###[104687730688]    flags: 2147483648
Logical: ###[41943040]    Ext length: ###[20971520]    Physical: ###[104689827840]    flags: 2147483649

On Fri, Mar 9, 2018 at 2:12 PM, Dilger, Andreas <andreas.dilger at intel.com<mailto:andreas.dilger at intel.com>> wrote:
On Mar 6, 2018, at 00:12, lokesh jaliminche <lokesh.jaliminche at gmail.com<mailto:lokesh.jaliminche at gmail.com>> wrote:
>
> Hi,
>
> Does lustre support SEEK_HOLE/SEEK_DATA flag for lseek?
>
> I did some experiment with the below program to find out if lustre supports SEEK_HOLE and SEEK_DATA flag. I found that lustre always returns the end of the file for SEEK_HOLE and 0 for SEEK_DATA and as per the man page this is the simplest implementation that file system can have(If they dont want to support these flags). So just wanted to confirm.

Lustre does not directly support the SEEK_HOLE/SEEK_DATA interface currently.
What you are seeing is the default/minimum implementation provided by the
lseek interface.

Implementing these seek options might be quite complex for a Lustre file
because of multiple stripes in the layout.  You would need to locate holes
in the stripes, and then map the hole offset on the object to the file offset.

If this is something you are interested to implement I would be happy to
discuss it with you further.  Probably the best place to start is to file
an LU ticket with details, and then interested parties can discuss the
implementation there.

Cheers, Andreas

> Program :
> ========
> #include <unistd.h>
> #include <sys/types.h>
> #include <stdio.h>
> #include <fcntl.h>
> #include <stdlib.h>
> #define SEEK_OFF 10485760
> int main()
> {
>     int fd;
>     char buffer[80];
>     int i = 0;
>     static char message[]="Hello world";
>     fd=open("myfile", O_RDWR);
>     if (fd != -1)
>         return 1;
>     printf("creating hole by writing at each of %d strides\n", SEEK_OFF);
>     for (i = 1; i < 10; i++)
>     {
>         int seek_off = i * SEEK_OFF;
>         int sz;
>         printf("seek_off %ld\n", lseek(fd, seek_off, SEEK_SET));
>         sz = write(fd,message,sizeof(message));
>         printf("write size = %d\n", sz);
>         printf("String : %s\n", message);
>     }
>     printf("Demonstrating SEEK_HOLE and SEEK_DATA %d \n", SEEK_OFF);
>     int start_off = 0;
>     lseek(fd, 0, SEEK_SET);
>     printf("after SEEK_HOLE start_off %ld\n", lseek(fd, 0, SEEK_HOLE));
>     printf("after SEEK_DATA start_off %ld\n", lseek(fd, start_off, SEEK_DATA));
>     printf("after SEEK_HOLE start_off %ld\n", lseek(fd, 10485760, SEEK_HOLE));
>     printf("after SEEK_DATA start_off %ld\n", lseek(fd, (10485760 *2 ), SEEK_DATA));
>     close(fd);
> }
>
> output:
> =====
> after SEEK_HOLE start_off 94372142
> after SEEK_DATA start_off 0
> after SEEK_HOLE start_off 94372142
> after SEEK_DATA start_off 0
>
> Regards,
> Lokesh
> _______________________________________________
> lustre-discuss mailing list
> lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel Corporation








_______________________________________________
lustre-discuss mailing list
lustre-discuss at lists.lustre.org<mailto:lustre-discuss at lists.lustre.org>
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20180311/d999aa59/attachment.html>