<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:GulimChe;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Malgun Gothic";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"\@Malgun Gothic";}
@font-face
{font-family:"\@GulimChe";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1554805949;
mso-list-type:hybrid;
mso-list-template-ids:259968820 1097607636 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Andreas,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">An interesting thought on this, as the same limitation came up recently in discussions with a Cray customer. Strictly honoring the direct I/O expectations around data copying is apparently optional. GPFS is a notable example – It allows
non page-aligned/page-size direct I/O, but it apparently (This is second hand from a GPFS knowledgeable person, so take with a grain of salt) uses the buffered path (data copy, page cache, etc) and flushes it, O_SYNC style. My understanding from conversations
is this is the general approach taken by file systems that support unaligned direct I/O – they cheat a little and do buffered I/O in those cases.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So rather than refusing to perform unaligned direct I/O, we could emulate the approach taken by (some) other file systems. There’s no clear standard here, but this is an option others have taken that might improve the user experience.
(I believe we persuaded our particular user to switch their code away from direct I/O, since they had no real reason to be using it.)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Patrick<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">lustre-discuss <lustre-discuss-bounces@lists.lustre.org> on behalf of
</span><span style="font-size:12.0pt;font-family:"Malgun Gothic",sans-serif;color:black">김형근</span><span style="font-size:12.0pt;color:black"> <okok102928@fusiondata.co.kr><br>
<b>Date: </b>Sunday, October 28, 2018 at 11:40 PM<br>
<b>To: </b>Andreas Dilger <adilger@whamcloud.com><br>
<b>Cc: </b>"lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org><br>
<b>Subject: </b>Re: [lustre-discuss] dd oflag=direct error (512 byte Direct I/O)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">The software I use is RedHat Virtualization. When using Posix compatible FS, it seems to perform direct I / O with a block size of 256512 bytes.<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">If I can't resolve the issue with my storage configuration, I will contact RedHat.<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">Your answer was very helpful.<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">Thank you.<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">보내는사람 : Andreas Dilger <adilger@whamcloud.com><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">받는사람 : 김형근 <okok102928@fusiondata.co.kr><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">참조 : lustre-discuss@lists.lustre.org <lustre-discuss@lists.lustre.org><o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">보낸 날짜 : 2018-10-25 16:47:58<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">제목 : Re: [lustre-discuss] dd oflag=direct error (512 byte Direct I/O)<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"GulimChe",serif;color:black">On Oct 25, 2018, at 15:05, 김형근
<o:p></o:p></span></p>
<p class="MsoNormal">wrote: <br>
> <br>
> Hi. <br>
> It's a pleasure to meet you, the lustre specialists. <br>
> (I do not speak English well ... Thank you for your understanding!) <br>
<br>
Your english is better than my Korean. :-) <br>
<br>
> I used the dd command in lustre mount point. (using the oflag = direct option) <br>
> <br>
> ------------------------------------------------------------ <br>
> dd if = / dev / zero of = / mnt / testfile oflag = direct bs = 512 count = 1 <br>
> ------------------------------------------------------------ <br>
> <br>
> I need direct I / O with 512 byte block size. <br>
> This is a required check function on the software I use. <br>
<br>
What software is it? Is it possible to change the application to use <br>
4096-byte alignment? <br>
<br>
> But unfortunately, If the direct option is present, <br>
> bs must be a multiple of 4K (4096) (for 8K, 12K, 256K, 1M, 8M, etc.) for operation.
<br>
> For example, if you enter a value such as 512 or 4095, it will not work. The error message is as follows.
<br>
> <br>
> 'error message: dd: error writing [filename]: invalid argument' <br>
> <br>
> My test system is all up to date. (RHEL, lustre-server, client) <br>
> I have used both ldiskfs and zfs for backfile systems. The result is same. <br>
> <br>
> <br>
> My question is simply two. <br>
> <br>
> 1. Why does DirectIO work only in 4k multiples block size? <br>
<br>
The client PAGE_SIZE on an x86 system is 4096 bytes. The Lustre client <br>
cannot cache data smaller than PAGE_SIZE, so the current implementation <br>
is limited to have O_DIRECT read/write being a multiple of PAGE_SIZE. <br>
<br>
I think the same would happen if you try to use O_DIRECT on a disk with <br>
4096-byte native sector drive (https://en.wikipedia.org/w/index.php?title=Advanced_Format§ion=5#4K_native )?
<br>
<br>
> 2. Can I change the settings of the server and client to enable 512bytes of DirectIO?
<br>
<br>
This would not be possible without changing the Lustre client code. <br>
I don't know how easily this is possible to do and still ensure that <br>
the 512-byte writes are handled correctly. <br>
<br>
So far we have not had other requests to change this limitation, so <br>
it is not a high priority to change on our side, especially since <br>
applications will have to deal with 4096-byte sectors in any case. <br>
<br>
Cheers, Andreas <br>
--- <br>
Andreas Dilger <br>
Principal Lustre Architect <br>
Whamcloud <br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>