<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Oct 3, 2011, at 3:15 PM, Nathan Rutman wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><font class="Apple-style-span" face="Arial"><span class="Apple-style-span" style="font-size: 15px;">... snip...</span></font></div><div><font class="Apple-style-span" face="Arial" style="font-size: 15px;">We have been thinking about a different wide-striping method that doesn't have these problems. </font><span class="Apple-style-span" style="font-family: Arial; font-size: 15px; white-space: pre-wrap; ">The basic idea is to create a new stripe type that encodes the list of OSTs compactly, and then using the same (or a calculable) object identifier (or FID) on all these OSTs.</span></div><div><div style="background-color: transparent; "><font class="Apple-style-span" size="4"><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; background-color: transparent;"></span></font><font class="Apple-style-span" face="Times"><img src="https://lh4.googleusercontent.com/mxm5R4Yd000I_v5qNcpYH6ZzHBvryGEE6pjxOBWz6ysHUNK0Yjh1J81kmP-5zVaoCiOU8RJv04WMhNoe1JqipOOmtRd7otrZ0saWKUnNyNVvaWvLRD8" width="404px;" height="163px;"></font><br><font class="Apple-style-span" face="Arial"><span class="Apple-style-span" style="font-size: 15px; white-space: pre-wrap;"><br></span></font><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Our version of widestriping does not involve increasing the EA size at all, but instead utilizes a new stripe pattern. (This will not be understandable by older Lustre versions, which will generate an error locally, or potentially we can convert into the BZ-4424 form if the layout fits in that format). A bitmap will identify which OSTs hold a stripe of this file. The bitmap should probably fit into current ext4 EA size limit, giving us ~32k stripes.</span><br><font class="Apple-style-span" size="4"><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; background-color: transparent;"></span></font><br><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; font-weight: normal; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap; ">Some OST’s may be down at file creation time, or new OSTs added later; hence there will likely be holes in the bitmap (but relatively few). </span></div></div></div></blockquote><div><br></div><div>1) There will be holes when OST pools used: if the file can be written only to the set of OST from specific OST POOL and if by the virtue of configuration OSTs in the pool do not represent continuous set then there will be holes in OST bit map even if all OSTs are online.</div><div><br></div><div>2) "relatively few holes [in bitmap]" - did you consider compressing bitmap? Like BBC or WAH described at en.wikipedia.org/wiki/Bitmap_index#Compression ? Reportedly you can do bitwise operations without decompression. This way you can go up in number of stripes (well, 32k is big number). But it may help control RPC size - you may represent wide striping with few integers effectively representing continuous blocks and OST holes, the size of the descriptor is the function of # of blocks and holes and to the less extent function of number of stripes.</div><div>More:</div><div>It is possible to have two bitmaps: </div><div>0000000111111111000000111111 - one describing general "blocks" of OST = ((beg1,end1),(beg2,end2))</div><div>0000000000000010100100100000 - other describing "corrections" - drop two OST, add two OST ; here 4 bits, compressed to X bytes </div><div>0000000111111101100100011111 - OST map, computed on client as bitwise XOR to uncompressed maps (1) and (2)</div><div>Each of two maps is compressed for transfer, thus shall not take much space.</div><div><br></div><div>3) If metadata file format going to be changed, is it right time to reserve descriptors to have few replicas of the file data?</div><div> </div><div>In such case we need to have number of replicas, and layout descriptor for each replica. Each replica may have different number of stripes, thus you can have widely striped file replica on SAS disks (or in flash) and replicate it to slower disk storage with one or "few" stripes for further tape archival.</div><div> I assume after initial writes file has more or less "stable" content. Replicas can be on different media type, like flash/ SAS/ SATA, fast / cheap disks, effectively Hierarchical Storage.</div><div>I'm thinking about "lazy" replication as you implemented to replicate data to another file system but in this case replication is within the same lustre file system. Client became aware of multiple replicas and can chose what file replica to use (e.g when some OSTs down). It eliminates OST as single point of failure.</div><div><br></div><div>Alex.</div><br><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="background-color: transparent; "><span class="Apple-style-span" style="font-family: monospace; white-space: pre; ">______________________________________________________________________</span></div></div></div><pre>This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it.
Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses.
Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.
The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan.
______________________________________________________________________
_______________________________________________<br>Lustre-devel mailing list<br><a href="mailto:Lustre-devel@lists.lustre.org">Lustre-devel@lists.lustre.org</a><br>http://lists.lustre.org/mailman/listinfo/lustre-devel<br></pre></blockquote></div><br></body></html>