<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hello, Nathan<br>
<br>
On 10/03/2011 01:15 PM, Nathan Rutman wrote:
<blockquote
cite="mid:73AED5C780AE05478241DB067651A9210237883C@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial">Oracle BZ-4424 (continued in WC LU-80) adds
support for larger OST stripe counts via increased EXT4 EA
sizes.</font></div>
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial">Some problems with this are:</font></div>
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial">1) increased MDT storage and network loading for
transmitting the object list </font></div>
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial">2) relative low new limit (1350 up from 160)</font></div>
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial"><br>
</font></div>
<div><font class="Apple-style-span" style="font-size: 15px;"
face="Arial">We have been thinking about a different
wide-striping method that doesn't have these problems. </font><span
class="Apple-style-span" style="font-family: Arial; font-size:
15px; white-space: pre-wrap;">The basic idea is to create a
new stripe type that encodes the list of OSTs compactly, and
then using the same (or a calculable) object identifier (or
FID) on all these OSTs.</span></div>
<div>
<div style="background-color: transparent;"><font
class="Apple-style-span" size="4"><span style="font-size:
11pt; font-family: Arial; color: rgb(0, 0, 0);
font-weight: normal; font-style: normal; font-variant:
normal; text-decoration: none; vertical-align: baseline;
white-space: pre-wrap; background-color: transparent;"></span></font><font
class="Apple-style-span" face="Times"><img
moz-do-not-send="true"
src="https://lh4.googleusercontent.com/mxm5R4Yd000I_v5qNcpYH6ZzHBvryGEE6pjxOBWz6ysHUNK0Yjh1J81kmP-5zVaoCiOU8RJv04WMhNoe1JqipOOmtRd7otrZ0saWKUnNyNVvaWvLRD8"
height="163px;" width="404px;"></font><br>
<font class="Apple-style-span" face="Arial"><span
class="Apple-style-span" style="font-size: 15px;
white-space: pre-wrap;"><br>
</span></font><span style="font-size: 11pt; font-family:
Arial; color: rgb(0, 0, 0); background-color: transparent;
font-weight: normal; font-style: normal; font-variant:
normal; text-decoration: none; vertical-align: baseline;
white-space: pre-wrap;">Our version of widestriping does not
involve increasing the EA size at all, but instead utilizes
a new stripe pattern. (This will not be understandable by
older Lustre versions, which will generate an error locally,
or potentially we can convert into the BZ-4424 form if the
layout fits in that format). A bitmap will identify which
OSTs hold a stripe of this file. The bitmap should probably
fit into current ext4 EA size limit, giving us ~32k stripes.</span><br>
<font class="Apple-style-span" size="4"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;
background-color: transparent;"></span></font><br>
<span style="font-size: 11pt; font-family: Arial; color:
rgb(0, 0, 0); background-color: transparent; font-weight:
normal; font-style: normal; font-variant: normal;
text-decoration: none; vertical-align: baseline;
white-space: pre-wrap;">Some OST’s may be down at file
creation time, or new OSTs added later; hence there will
likely be holes in the bitmap (but relatively few). Start
index will still be used, but stripe order will be strictly
round-robin (we will wrap around). </span><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0,
0); background-color: transparent; font-weight: normal;
font-style: normal; font-variant: normal; text-decoration:
none; vertical-align: baseline; white-space: pre-wrap;">In
other words, the stripe sequence will always be in linear
OST order, starting from start_index, maybe skipping some
holes, wrapping around to start_index-1.</span><br>
<font class="Apple-style-span" size="4"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;
background-color: transparent;"></span></font><br>
<span style="font-size: 11pt; font-family: Arial; color:
rgb(0, 0, 0); background-color: transparent; font-weight:
normal; font-style: normal; font-variant: normal;
text-decoration: none; vertical-align: baseline;
white-space: pre-wrap;">Widestripe objects do not need a
special sequence number (fid_seq); the MDT knows the file
was created as widestriped and marks it as such
(LOV_PATTERN_BITMAP). There are two options for OST object
identification: common object ID and FID-on-OST.</span><br>
</div>
</div>
</blockquote>
<br>
Actually, we also discussed to use real object (IAM or other index
format) to store the stripe pattern, instead of using EA. Of course
it would use more space, but it would give us the potential to
explore the stripe pattern.<br>
<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A9210237883C@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div>
<div style="background-color: transparent;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0,
0); background-color: transparent; font-weight: normal;
font-style: normal; font-variant: normal; text-decoration:
none; vertical-align: baseline; white-space: pre-wrap;">Common
Object ID</span><br>
<div style="font-family: Times; margin-left: 36pt; margin-top:
0pt; margin-bottom: 0pt;"><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); background-color:
transparent; font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;">The MDT
tracks a special range of OST object ID’s (“wide stripe
objectid” = WSO) that are used on all OSTs. The MDT
assigns the next available WSO to the file, and this
objectid is used on all the OSTs. The OSTs must never use
these objects for regular striped files. A special
precreation group for these objects is probably necessary,
as well as orphan cleanup (the MDT should purge "hole"
objects that aren’t allocated from a particular OST). The
MDT should track the last assigned WSO; this will be the
starting point for new wide striped files after recovery.</span><span
class="Apple-style-span" style="font-family: Helvetica;"><span
style="font-size: 11pt; font-family: Arial; color:
rgb(0, 0, 0); background-color: transparent;
font-weight: normal; font-style: normal; font-variant:
normal; text-decoration: none; vertical-align: baseline;
white-space: pre-wrap;"> Objects cannot be migrated
from one OST to another, since this would result in
out-of-order access. Similarly, s</span></span><span
class="Apple-style-span" style="font-family: Helvetica;"><span
class="Apple-style-span" style="font-family: Arial;
font-size: 15px; white-space: pre-wrap;">tripes can
never be added to holes.</span></span></div>
<span style="font-size: 11pt; font-family: Arial; color:
rgb(0, 0, 0); background-color: transparent; font-weight:
normal; font-style: normal; font-variant: normal;
text-decoration: none; vertical-align: baseline;
white-space: pre-wrap;">FID-on-OST</span><br>
<div style="font-family: Times; margin-left: 36pt; margin-top:
0pt; margin-bottom: 0pt;"><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); background-color:
transparent; font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;">Use a
mapping of the MDT FID to uniquely determine an OST
object. The clients and MDT add in the OST number to the
MDT FID (probably just reserve one sequence per OST).
(This allows the objects to potentially migrate to
different OSTs). The OSTs then internally must map the
FID to a local object id. Note this allows OST-local
precreation pools, getting the MDT out of the
precreate/orphan cleanup business and potentially
improving create speeds, and also facilitates "create on
write" semantics. The FID can be assigned during the
first access to OST object.</span></div>
</div>
</div>
</blockquote>
<br>
I am not sure I follow your idea here. You mean the OST needs
internally map MDT FID(added in OST number) to object id (or inode
ino) ? So there are no real OST FID? But you also said "<span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0);
background-color: transparent; font-weight: normal; font-style:
normal; font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;"></span><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0);
background-color: transparent; font-weight: normal; font-style:
normal; font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;">The FID can be
assigned during the first access to OST object.", Could you please
explain more here? </span><br>
<br>
<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A9210237883C@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div>
<div style="background-color: transparent;">
<div style="font-family: Times; margin-left: 36pt; margin-top:
0pt; margin-bottom: 0pt;"><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); background-color:
transparent; font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;">The big
problem here is that FID>OBJID ( or better
FID->inode id ) translation is absent from the OSTs
today. See </span><a moz-do-not-send="true"
href="http://wiki.lustre.org/images/e/e9/SC09-FID-on-OST.pdf"><span
style="font-size: 11pt; font-family: Arial; color:
rgb(0, 0, 153); background-color: transparent;
font-weight: normal; font-style: normal; font-variant:
normal; text-decoration: underline; vertical-align:
baseline; white-space: pre-wrap;">http://wiki.lustre.org/images/e/e9/SC09-FID-on-OST.pdf</span></a><span
style="font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); background-color: transparent; font-weight: normal;
font-style: normal; font-variant: normal; text-decoration:
none; vertical-align: baseline; white-space: pre-wrap;">
(what is the current state of this?) There is also some
work in this direction in the OST restructuring work
(“Orion” WC branch, ORI-300(?), scheduled for Lustre 2.4).
</span></div>
<div style="margin-left: 36pt; margin-top: 0pt; margin-bottom:
0pt;"><font class="Apple-style-span" face="Arial"><span
class="Apple-style-span" style="font-size: 15px;
white-space: pre-wrap;"><br>
</span></font></div>
</div>
<div style="background-color: transparent;"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0, 0,
0); background-color: transparent; font-weight: normal;
font-style: normal; font-variant: normal; text-decoration:
none; vertical-align: baseline; white-space: pre-wrap;">There's
a few questions here, probably the first of which is "</span><span
class="Apple-style-span" style="font-family: Arial;
font-size: 15px; white-space: pre-wrap;">is it worthwhile to
spend effort on this, or is BZ4424 good enough?" Then there
is the question of object identification, where FID-on-OST
is more flexible, but also significantly more work (and
risk). Also, I thought I understood from the EOFS Summit
that WC also has a separate FID-on-OST project (separate
from Orion that is) -- can someone tell me the state of
that?</span></div>
</div>
</blockquote>
<br>
FID-on-OST is actually part of DNE(dirtribute name space) phase I.
It basically follows current fid client server infrastructure.<br>
<br>
1. MDT is the fid client, which requests fid from the OST and
allocates fids for the object during pre-creation. <br>
2. OST is the fid server, which will allocate the FIDs to MDTs and
requests super fid sequence from fid control server (root MDT).<br>
3. Similar as MDT FID, there will be OI to map FID to object inside
OST.<br>
<br>
The code will be release with DNE sometime next year.<br>
<br>
Thanks<br>
WangDi<br>
<br>
<br>
<br>
<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A9210237883C@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div>
<div style="background-color: transparent;"><span
class="Apple-style-span" style="font-family: Arial;
font-size: 15px; white-space: pre-wrap;"><br>
</span></div>
<div style="background-color: transparent;"><font
class="Apple-style-span" face="Arial"><span
class="Apple-style-span" style="font-size: 15px;
white-space: pre-wrap;"><br>
</span></font><font class="Apple-style-span" size="4"><span
style="font-size: 11pt; font-family: Arial; color: rgb(0,
0, 0); font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;
background-color: transparent;"></span></font><br>
<font class="Apple-style-span" face="Arial"><span
class="Apple-style-span" style="font-size: 15px;
white-space: pre-wrap;"><br>
</span></font>
<div style="font-family: Times; margin-left: 36pt; margin-top:
0pt; margin-bottom: 0pt;"><span style="font-size: 11pt;
font-family: Arial; color: rgb(0, 0, 0); background-color:
transparent; font-weight: normal; font-style: normal;
font-variant: normal; text-decoration: none;
vertical-align: baseline; white-space: pre-wrap;"><br>
</span></div>
</div>
</div>
<pre>______________________________________________________________________
This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
______________________________________________________________________
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Lustre-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Lustre-devel@lists.lustre.org">Lustre-devel@lists.lustre.org</a>
<a class="moz-txt-link-freetext" href="http://lists.lustre.org/mailman/listinfo/lustre-devel">http://lists.lustre.org/mailman/listinfo/lustre-devel</a>
</pre></pre>
</blockquote>
<br>
</body>
</html>