<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content=text/html;charset=big5>
<META content="MSHTML 6.00.6000.16705" name=GENERATOR></HEAD>
<BODY id=MailContainerBody
style="PADDING-RIGHT: 10px; PADDING-LEFT: 10px; PADDING-TOP: 15px"
bgColor=#ffffff leftMargin=0 topMargin=0 CanvasTabStop="true"
name="Compose message area">
<DIV><FONT face=s\u04e9 size=2>Hi, all,</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>I got a problem when I testing lustre-1.6.5.1 on
CentOS-5.2.</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>I have four machines(PCs), they are MGS
co-located with MDT, OSS-1, OSS-2 and CLT. </FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>OSS-1 have two disks which are formatted as
ost01(40GB) and ost02(15GB).</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>OSS-2 have two disks which are formatted as
ost03(23GB) and ost04(5GB).</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>At first, I reformat the MGS/MDT and mount it
to /mnt/ as follow:</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@MGS ~]# mkfs.lustre --reformat --fsname=testfs
--mgs --mdt /dev/hdb</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@MGS ~]# mount -t lustre /dev/hdb
/mnt/mgs</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>Second, I reformat OSTs and mount it to /mnt as
follow:</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-1 ~]# mkfs.lustre --reformat
--fsname=testfs --ost <A title="mailto:--mgsnode=MGS@tcp CTRL + Click }\u04b3s"
href="mailto:--mgsnode=MGS@tcp">--mgsnode=MGS@tcp</A> /dev/hdc</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-1 ~]# mkfs.lustre --reformat
--fsname=testfs --ost <A title="mailto:--mgsnode=MGS@tcp CTRL + Click }\u04b3s"
href="mailto:--mgsnode=MGS@tcp">--mgsnode=MGS@tcp</A> /dev/hdd</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-1 ~]# mount -t lustre /dev/hdc
/mnt/ost01</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-1 ~]# mount -t lustre /dev/hdd
/mnt/ost02</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-2 ~]# mkfs.lustre --reformat
--fsname=testfs --ost <A title="mailto:--mgsnode=MGS@tcp CTRL + Click }\u04b3s"
href="mailto:--mgsnode=MGS@tcp">--mgsnode=MGS@tcp</A> /dev/hdc</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-2 ~]# mkfs.lustre --reformat
--fsname=testfs --ost <A title="mailto:--mgsnode=MGS@tcp CTRL + Click }\u04b3s"
href="mailto:--mgsnode=MGS@tcp">--mgsnode=MGS@tcp</A> /dev/hdd</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-2 ~]# mount -t lustre /dev/hdc
/mnt/ost03</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@OSS-2 ~]# mount -t lustre /dev/hdd
/mnt/ost04</FONT></DIV>
<DIV> </DIV>
<DIV>Third, I mounted lustre file system at CLT like this:</DIV>
<DIV>[root@CLT ~] # mount -t lustre <A
href="mailto:MGS@tcp:/testfs">MGS@tcp:/testfs</A> /mnt/lfs<BR>[root@CLT mnt]# df
-h<BR>Filesystem Capacity Used Available
Use% Mounted on:<BR>/dev/mapper/VolGroup00-LogVol00 4.3G 1.9G
2.2G 46%
/<BR>/dev/hda1
99M 67M 28M 72%
/boot<BR>tmpfs 252M
0 252M 0% /dev/shm<BR><A
href="mailto:MGS@tcp:/testfs">MGS@tcp:/testfs</A>
82G 1.6G 77G 2%
/mnt/lfs<BR></DIV></FONT><FONT face=s\u04e9 size=2></FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>Fourth, I try to use "lfs" command to
set stripe parameters at CLT.</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>[root@CLT mnt]# lfs setstripe lfs -s 8m -c
-1</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>Fifth, I use "dd" command to test lustre file
system.</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>Then, it gives <EM>error</EM> "<EM>no space
left</EM>" since just ost04(5GB) get full.</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>[root@CLT lfs]# dd if=/dev/zero of=testfile001
bs=128M count=24<BR>24+0 records in<BR>24+0 records out<BR>3221225472 bytes (3.2
GB) copiedA164.585 secondsA19.6 MB/s<BR>[root@CLT lfs]# dd if=/dev/zero
of=testfile002 bs=128M count=24<BR>24+0 records in<BR>24+0 records
out<BR>3221225472 bytes (3.2 GB) copiedA164.836 secondsA19.5 MB/s<BR>[root@CLT
lfs]# dd if=/dev/zero of=testfile003 bs=128M count=48<BR>48+0 records in<BR>48+0
records out<BR>6442450944 bytes (6.4 GB) copiedA383.2 secondsA16.8
MB/s<BR>[root@CLT lfs]# dd if=/dev/zero of=testfile004 bs=128M
count=48<BR>dd: write error: testfile004: No space left on this
device.<BR>47+0 records in<BR>46+0 records out<BR>6301048832 bytes (6.3 GB)
copiedA418.321 A15.1 MB/s</DIV></FONT>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>[root@CLT lfs]# df
-h<BR>Filesystem Capacity Used Available
Use% Mounted on<BR>/dev/mapper/VolGroup00-LogVol00 4.3G 1.9G
2.2G 46%
/<BR>/dev/hda1
99M 67M 28M 72%
/boot<BR>tmpfs
252M 0 252M 0%
/dev/shm<BR><A
href="mailto:MGS@tcp:/testfs">MGS@tcp:/testfs</A>
82G 20G 59G 25% /mnt/lfs<BR>[root@CLT
lfs]# lfs
df<BR> UUID
1K-blocks Used Available Use% Mounted
on<BR>testfs-MDT0000_UUID 2752272
127844 2467144 4%
/mnt/lfs[MDT:0]<BR>testfs-OST0000_UUID 41284928
5145080 34042632 12%
/mnt/lfs[OST:0]<BR>testfs-OST0001_UUID 15481840
5134432 9560912 33%
/mnt/lfs[OST:1]<BR>testfs-OST0002_UUID 23738812
5141040 17391848 21%
/mnt/lfs[OST:2]<BR>testfs-OST0003_UUID 5160576
4898364 4 94%
/mnt/lfs[OST:3]</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2></FONT> </DIV>
<DIV><FONT face=s\u04e9 size=2>filesystem summary: 85666156
20318916 60995396 23% /mnt/lfs<BR></FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>
<DIV><FONT face=s\u04e9 size=2>I have no idea about this error. </FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>Is there anyone could tell me about that how to
config lustre to avoid this error?</FONT></DIV>
<DIV><FONT face=s\u04e9 size=2>didn't Lustre put file into the OSTs which still
have free spaces instead of those full ones ?</FONT></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>Regards,</DIV>
<DIV> </DIV>
<DIV>Chris</DIV></FONT></DIV></BODY></HTML>