[lustre-discuss] File Level Redundancy - Data Movement

Indivar Nair indivar.nair at techterra.in
Sun Feb 21 06:33:53 PST 2021


Okay. Got it.

Regards,


Indivar Nair

On Sun, Feb 21, 2021 at 7:55 PM Spitz, Cory James <cory.spitz at hpe.com>
wrote:

> Right, except I should point out that a mirror sync operation doesn’t have
> to be from one pool to another.  The FLR replica can use any specified
> layout.  Using an OST Pool is just a convenience.
>
>
>
> -Cory
>
>
>
> On 2/20/21, 12:10 PM, "Indivar Nair" <indivar.nair at techterra.in> wrote:
>
>
>
> Thanks, Cory.
>
>
>
> So -
>
>
>
> 1. When the "lfs mirror" command is run, the file(s) are copied by the
> client from one pool to another
>
>     Pool A -> Client Machine -> Pool B (mirror)
>
> 2. If a mirrored file is modified, the modifications are saved only to the
> primary mirror
>
>     We will then have to run the "lfs mirror resync" command to
> synchronize all the mirror copies
>
>     Pool A (modified file) -> Client Machine -> Pool B (mirror)
>
>
>
> So for this feature to be really effective, we will need a dedicated
> client  that does this copying / re-copying continuously.
>
> And if there are a lot of mirrored files, the dedicated client will have
> to be very fast too.
>
>
>
> 3. In future it may be possible to move these files using HSM agents.
>
>
>
> Regards,
>
>
>
>
>
> Indivar Nair
>
>
>
> On Fri, Feb 19, 2021 at 1:51 AM Spitz, Cory James <cory.spitz at hpe.com>
> wrote:
>
> Good questions, Indivar.
>
>
>
> 1) The Lustre Operations Manual doesn’t make this clear at
> https://doc.lustre.org/lustre_manual.xhtml#flr.operations.resyncmirror ,
> but a mirror sync will be completed in the context of the client that it
> was executed on.
>
>
>
> 2) Mirror resync is entirely manual today.  Again, it is done in the
> context of the client from which the lfs command was executed.  Automatic
> or “Immediate” mirror is on the https://www.lustre.org/roadmap/ for
> 2.16.0.  However, as a side note, there are also plans to “externalize” and
> leverage the HSM coordinator to be able to hand-off the data movement to
> multiple data movers.  See LU-6081, LU-12890, and LU-13384 for an outline
> of that work.  I only mention it here because you’ve hinted at the need for
> this improvement, but it is not committed to the roadmap yet.
>
>
>
> 3) One way could use lfs find to find files needing a mirror update.
> There are multiple options to zero in on the mirror count and/or state.
> Consult the lfs-find man page for details.
>
>
>
> -Cory
>
>
>
> On 2/17/21, 6:43 AM, "lustre-discuss" <
> lustre-discuss-bounces at lists.lustre.org> wrote:
>
>
>
> Hi All,
>
>
>
> I was going through the FLR feature in the Lustre Manual and had some
> questions.
>
>
>
> 1. If we run the "lfs mirror" command on an existing file or directory,
> how is the file ' directory copied from one OST to another?
>
>     Is the copying done by the Lustre client on which this command was run?
>
>
>
> 2. And, in case of delayed mirroring of newly created files, who does the
> copying?
>
>
>
> 3. How can we automate this process to automatically mirror certain file
> types?
>
>
>
> Regards,
>
>
>
>
>
> Indivar Nair
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20210221/69017a92/attachment.html>


More information about the lustre-discuss mailing list