<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Sven,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
The "combining layouts without any data movement" part isn't currently possible. It's probably possible in theory, but it's never been implemented. (I'm curious what your use case is?)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Even allowing for data movement, there's no tool to do this for you. Depending what you mean by combining, it's possible to do this with Linux tools (see the end of my note), but you're going to have data copying.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
It's a bit of an odd requirement, with some inherent questions - For example, file layouts generally go to infinity, because if they don't, you will get IO errors when you 'run off the end', ie, go past the defined layout, so the last component is usually defined
to go to infinity.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
That poses obvious questions when combining files.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
If you're looking to combine files with layouts that do not go to infinity, then it's at least straightforward to see how you'd concatenate them. But presumably the data in each file doesn't go to the very end of the layout? So do you want the empty parts
of the layout included?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Say file 1 is 10 MiB in size but the layout goes to 20 MiB (again, layouts normally should go to infinity) and file 2 is also 10 MiB in size but the layout goes to, say, 15 MiB. Should the result look like this?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Layout: 1 1 1 1 1 1 1 ... 20 MiB 2 2 2 2 2 2 .... 35 MiB<br>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
With data from 0-10 MiB and 20 - 30 MiB.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
That's something you'd have to write a tool for, so it could write the data at your specified offset for putting in the second file (and third, etc...). You could also do something like:<br>
<br>
lfs setstripe [your layout] combined file; cat file 1 > combined file; truncate [combined file] 20 MiB (the end of the file 1 layout); cat file 2 > combined_file", etc.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
So, you definitely can't avoid data copying here. But that's how you could do it with simple Linux tools (which you could probably have drawn up yourself :)).</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
-Patrick</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of Sven Willner <sven.willner@mpimet.mpg.de><br>
<b>Sent:</b> Wednesday, March 29, 2023 7:58 AM<br>
<b>To:</b> lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><br>
<b>Subject:</b> [lustre-discuss] Joining files</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">[You don't often get email from sven.willner@mpimet.mpg.de. Learn why this is important at
<a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
https://aka.ms/LearnAboutSenderIdentification</a> ]<br>
<br>
Dear all,<br>
<br>
I am looking for a way to join/merge/concatenate several files into one, whose layout is just the concatenation of the layouts of the respective files - ideally without any copying/moving on the data side (even if this would result in "holes" in the joined
file).<br>
<br>
I would very much appreciate any hints to tools or ideas of how to achieve such a join. As I understand that has been a `join` command for `lfs`, which is now deprecated (however, I am not sure if a use case like mine has been its purpose or why it has been
deprecated).<br>
<br>
Thanks a lot!<br>
Best regards,<br>
Sven<br>
<br>
--<br>
Dr. Sven Willner<br>
Scientific Computing Lab (SCLab)<br>
Max Planck Institute for Meteorology<br>
Bundesstraße 53, D-20146 Hamburg, Germany<br>
</div>
</span></font></div>
</body>
</html>