From Jingjing.Li at Sun.COM Fri Nov 2 07:06:40 2007 From: Jingjing.Li at Sun.COM (Karen Li) Date: Fri, 02 Nov 2007 15:06:40 +0800 Subject: [Lustre-devel] Lustre Level I training sessions next week! Only a few seats remaining! Message-ID: <472ACC80.9060205@sun.com> Lustre Administration & Support training will be offered next week, on November 7-9, in Reno, Nevada. Only a few seats remaining! Click here to register today! http://www.acteva.com/go/clusterfilesystems Price: US$3,000 for 3 days of in-depth technical hands-on instruction. Location * Circus Circus Hotel * 500 N. Sierra Street, Reno, NV 89503 * Phone: 1-775-329-0711 * Toll-free: 1-800-648-5010 * http://www.circusreno.com/ ENROLL NOW! http://www.acteva.com/go/clusterfilesystems Questions regarding Lustre courses can be sent to training at clusterfs.com Visit www.clusterfs.com/events.html for more information. From rs2006ts at hotmail.com Tue Nov 6 16:59:26 2007 From: rs2006ts at hotmail.com (RS RS) Date: Tue, 6 Nov 2007 11:59:26 -0500 Subject: [Lustre-devel] Checksum Algorithm Message-ID: Hi, We have seen a huge performance drop in 1.6.3, due to the checksum being enabled by default. I looked at the algorithm being used, and it is actually a CRC32, which is a very strong algorithm for detecting all sorts of problems, such as single bit errors, swapped bytes, and missing bytes. I've been experimenting with using a simple XOR algorithm. I've been able to recover most of the lost performance. This algorithm will detected corrupted bytes and words. This algorithm will not detect swapped bytes errors, but I think that these are pretty rare. This algorithm will not detect missing bytes, but I suspect that other things in Lustre or LNET will detect this problem. This algorithm will not detect two errors that offset each other, such as a single bit error in two words that are a multiple of 4 bytes apart. Should we consider using a more efficient checksum algorithm, in order to regain performance? Should the algorithm be configurable? -Roger _________________________________________________________________ Boo! Scare away worms, viruses and so much more! Try Windows Live OneCare! http://onecare.live.com/standard/en-us/purchase/trial.aspx?s_cid=wl_hotmailnews -------------- next part -------------- An HTML attachment was scrubbed... URL: From behlendorf1 at llnl.gov Tue Nov 6 19:30:28 2007 From: behlendorf1 at llnl.gov (Brian Behlendorf) Date: Tue, 6 Nov 2007 11:30:28 -0800 Subject: [Lustre-devel] Checksum Algorithm In-Reply-To: References: Message-ID: <200711061130.32694.behlendorf1@llnl.gov> Roger, We've been running with checksums enabled in our release for some time now and have seen the exact same impact on performance. In our case single node performance is impacted but aggregate FS performance remains good when enough clients are involved. We are tracking the performance issue under bug 13805 and would love any input/insight you might have on the issue. Bug13805 My view on the issue is that it is madness to run with checksums disabled and we need to investigate more efficient checksum algorithms. The current crc32 algorithm may be too heavy weight but the simple XOR algorithm you propose I fear is not strong enough. I've seen to many cases now of various network components corrupting data in all sorts of interesting ways. Happily we have a lot of other choices for algorithms to investigate. If you have the time I'd encourage you to investigate an assortment of algorithms and see which work best. Making this a runtime option via proc I think is also an excellent idea. -- Thanks, Brian > Hi, > > We have seen a huge performance drop in 1.6.3, due to the checksum being > enabled by default. I looked at the algorithm being used, and it is > actually a CRC32, which is a very strong algorithm for detecting all sorts > of problems, such as single bit errors, swapped bytes, and missing bytes. > > I've been experimenting with using a simple XOR algorithm. I've been able > to recover most of the lost performance. This algorithm will detected > corrupted bytes and words. This algorithm will not detect swapped bytes > errors, but I think that these are pretty rare. This algorithm will not > detect missing bytes, but I suspect that other things in Lustre or LNET > will detect this problem. This algorithm will not detect two errors that > offset each other, such as a single bit error in two words that are a > multiple of 4 bytes apart. > > Should we consider using a more efficient checksum algorithm, in order to > regain performance? Should the algorithm be configurable? > > -Roger -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From pauln at psc.edu Tue Nov 6 19:57:50 2007 From: pauln at psc.edu (Paul Nowoczynski) Date: Tue, 06 Nov 2007 14:57:50 -0500 Subject: [Lustre-devel] Checksum Algorithm In-Reply-To: <200711061130.32694.behlendorf1@llnl.gov> References: <200711061130.32694.behlendorf1@llnl.gov> Message-ID: <4730C73E.3080606@psc.edu> Brian, How does the crc mechanism work? I assume that the crc is done at the client, does the server verify the crc? Also are the crc's stored on disk? thanks, paul Brian Behlendorf wrote: > Roger, > > We've been running with checksums enabled in our release for some time now > and have seen the exact same impact on performance. In our case single node > performance is impacted but aggregate FS performance remains good when enough > clients are involved. We are tracking the performance issue under bug 13805 > and would love any input/insight you might have on the issue. > > Bug13805 > > My view on the issue is that it is madness to run with checksums disabled > and we need to investigate more efficient checksum algorithms. The current > crc32 algorithm may be too heavy weight but the simple XOR algorithm you > propose I fear is not strong enough. I've seen to many cases now of various > network components corrupting data in all sorts of interesting ways. > Happily we have a lot of other choices for algorithms to investigate. > > If you have the time I'd encourage you to investigate an assortment of > algorithms and see which work best. Making this a runtime option via > proc I think is also an excellent idea. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > From Niklas.Edmundsson at hpc2n.umu.se Wed Nov 7 08:39:55 2007 From: Niklas.Edmundsson at hpc2n.umu.se (Niklas Edmundsson) Date: Wed, 7 Nov 2007 09:39:55 +0100 (MET) Subject: [Lustre-devel] Checksum Algorithm In-Reply-To: <200711061130.32694.behlendorf1@llnl.gov> References: <200711061130.32694.behlendorf1@llnl.gov> Message-ID: On Tue, 6 Nov 2007, Brian Behlendorf wrote: > My view on the issue is that it is madness to run with checksums disabled > and we need to investigate more efficient checksum algorithms. The current > crc32 algorithm may be too heavy weight but the simple XOR algorithm you > propose I fear is not strong enough. I've seen to many cases now of various > network components corrupting data in all sorts of interesting ways. > Happily we have a lot of other choices for algorithms to investigate. > > If you have the time I'd encourage you to investigate an assortment of > algorithms and see which work best. Making this a runtime option via > proc I think is also an excellent idea. I'd strongly recommend looking at which algorithms are used for checksumming in ZFS, they have done rather extensive investigations on the subject. If I remember correctly ZFS is using fletcher by default with good performance, with sha256 as an option for those who wants it. Anyhow, given the nowadays tight bond between CFS and SUN finding the relevant info on the subject should be a no-brainer :) /Nikke -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke at hpc2n.umu.se --------------------------------------------------------------------------- OH NO, my wife burned the rice crispies--AGAIN!! =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= From eng_amiri20 at hotmail.com Wed Nov 7 19:06:53 2007 From: eng_amiri20 at hotmail.com (Rami Amiri) Date: Wed, 7 Nov 2007 19:06:53 +0000 Subject: [Lustre-devel] Lustre and PVFS2 Message-ID: Hey Everybody, I am trying to write a module for Lustre, which can read a metadata for a file (inode) then translate it to PVFS2 metadata. just I need that for read op. I 'd love to hear your suggestions. thanks _________________________________________________________________ Peek-a-boo FREE Tricks & Treats for You! http://www.reallivemoms.com?ocid=TXT_TAGHM&loc=us -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at sun.com Wed Nov 7 20:05:20 2007 From: adilger at sun.com (Andreas Dilger) Date: Wed, 7 Nov 2007 13:05:20 -0700 Subject: [Lustre-devel] Checksum Algorithm In-Reply-To: References: Message-ID: <20071107200520.GD3966@webber.adilger.int> On Nov 06, 2007 11:59 -0500, RS RS wrote: > We have seen a huge performance drop in 1.6.3, due to the checksum > being enabled by default. I looked at the algorithm being used, and it is > actually a CRC32, which is a very strong algorithm for detecting all sorts > of problems, such as single bit errors, swapped bytes, and missing bytes. > I've been experimenting with using a simple XOR algorithm. I've > been able to recover most of the lost performance. This algorithm > will detected corrupted bytes and words. This algorithm will not > detect swapped bytes errors, but I think that these are pretty rare. > This algorithm will not detect missing bytes, but I suspect that other > things in Lustre or LNET will detect this problem. This algorithm will > not detect two errors that offset each other, such as a single bit error > in two words that are a multiple of 4 bytes apart. Note that it is possible to disable checksums to get the previous behaviour back at runtime with (on all clients that should skip checksums): for C in /proc/fs/lustre/osc/*/checksums; do echo 0 > $C done in the lustre configuration: mgs> lctl conf_param testfs-OST0001.osc.checksums=0 or at compile time with "configure --disable-checksum ..." Cheers, Andreas -- Andreas Dilger Sr. Software Engineer, Lustre Group Sun Microsystems of Canada, Inc. From dsternkopf at hpce.nec.com Mon Nov 12 16:10:22 2007 From: dsternkopf at hpce.nec.com (Danny Sternkopf) Date: Mon, 12 Nov 2007 17:10:22 +0100 Subject: [Lustre-devel] Lustre - HA: lc_hb - ha.cf configuration In-Reply-To: <4721B252.1010808@hpce.nec.com> References: <4721B252.1010808@hpce.nec.com> Message-ID: <47387AEE.3000203@hpce.nec.com> Hi, I dont' understand how the ha.cf configuration works. The file is created as follows: python /usr/lib64/heartbeat/haresources2cib.py --stdout /tmp/heartbeat/haresources.lustre > /tmp/heartbeat/cib.xml.lustre Then it is transferred to the corresponding node. haresources2cib.py uses /etc/ha.d/ha.cf which is different on other nodes. I think the cib.xml must be created on the local host. Or there should be the possibility to specify a node-specific ha.cf for HA configuration creation. Best regards, Danny -- Danny Sternkopf dsternkopf at hpce.nec.com NEC HPC Europe GmbH, http://www.teraflop-workbench.de Stuttgart, Germany phone: +49-711-68770-35 fax: +49-711-6877145 From Ricardo.M.Correia at Sun.COM Tue Nov 13 11:48:38 2007 From: Ricardo.M.Correia at Sun.COM (Ricardo Correia) Date: Tue, 13 Nov 2007 11:48:38 +0000 Subject: [Lustre-devel] ZFS/DMU benchmarks In-Reply-To: References: Message-ID: <47398F16.5000504@Sun.COM> Hi everyone, Here are some current ZFS/DMU benchmark results that might be of interest to you. These results were obtained on a Sunfire x4500 with these specifications: - 2 dual-core AMD Opteron processors - 16 GB main memory - 48 SATA II HDD (7200 RPM, 500 GB each) - Solaris 10 update 3 - Userspace DMU from OpenSolaris build 74 The tool used in this particular benchmark was PIOS, which simulates a parallel I/O load typically experienced by an OSS. Since PIOS links directly with the DMU, it should give a good estimation of the maximum throughput of the userspace DMU-OSS. For comparison, you can also see the results of running the same PIOS benchmark with normal POSIX I/O on the Solaris 10u3 ZFS implementation. See the attached PDF for the results. First page provides PIOS throughput information on striped pools with 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, the kernel ZFS implementation scales better to 24 and 46 disks. Surprisingly enough, disabling checksums has a negative impact on the throughput with a 10-disk striped pool. This impact can be attributed to the differences in the uDMU IO pipeline which are probably causing the IOs to not being parallelized when checksumming is disabled. This issue, however, should not be hard to fix. Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and mirrored pools. In these configurations, the kernel ZFS currently has a better throughput than the userspace DMU. These results were obtained with only minimal tuning of the DMU (setting a maximum cache size and increasing the number of I/O threads). Work is already underway to improve the performance of the DMU and we expect to see better throughput in the coming months. Best regards, Ricardo -- * Ricardo Manuel Correia * Lustre Engineering Group *Sun Microsystems, Inc.* Portugal Ricardo.M.Correia at Sun.COM -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Lustre-uDMU.pdf Type: application/pdf Size: 66695 bytes Desc: not available URL: From Peter.Bojanic at Sun.COM Tue Nov 13 20:14:09 2007 From: Peter.Bojanic at Sun.COM (Peter Bojanic) Date: Tue, 13 Nov 2007 12:14:09 -0800 Subject: [Lustre-devel] Open CVS Repository for Lustre Message-ID: <17414268-DFDE-4CEA-9ED3-BE8FECAC1CFD@sun.com> Lustre community, It's my pleasure to announce the availability of an open CVS repository for Lustre source code. You can now get the latest sources from this anonymous, read-only pserver, which is synchronized against the master source tree once every hour. Details on how to check out Lustre are available on Lustre.org: http://wiki.lustre.org/index.php?title=Open_CVS Details on how to build Lustre are available in the Operations Manual: http://manual.lustre.org/ Cheers, Bojanic From scjody at sun.com Wed Nov 14 21:58:04 2007 From: scjody at sun.com (Jody McIntyre) Date: Wed, 14 Nov 2007 16:58:04 -0500 Subject: [Lustre-devel] [Lustre-discuss] ZFS/DMU benchmarks In-Reply-To: <47398F16.5000504@Sun.COM> References: <47398F16.5000504@Sun.COM> Message-ID: <20071114215804.GJ6168@modernduck.com> Hi Ricardo, On Tue, Nov 13, 2007 at 11:48:38AM +0000, Ricardo Correia wrote: > Here are some current ZFS/DMU benchmark results that might be of > interest to you. It's nice to see these results - thanks for posting them. When you have time, could you do a comparison with Lustre 1.6.x on ldiskfs + software RAID 5? It would be good to know how ZFS stands up (probably quite favourably, even at this early stage.) Cheers, Jody > These results were obtained on a Sunfire x4500 with these specifications: > > - 2 dual-core AMD Opteron processors > - 16 GB main memory > - 48 SATA II HDD (7200 RPM, 500 GB each) > - Solaris 10 update 3 > - Userspace DMU from OpenSolaris build 74 > > The tool used in this particular benchmark was PIOS, which simulates a > parallel I/O load typically experienced by an OSS. Since PIOS links > directly with the DMU, it should give a good estimation of the maximum > throughput of the userspace DMU-OSS. For comparison, you can also see > the results of running the same PIOS benchmark with normal POSIX I/O on > the Solaris 10u3 ZFS implementation. > > See the attached PDF for the results. > > First page provides PIOS throughput information on striped pools with > 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, > the kernel ZFS implementation scales better to 24 and 46 disks. > Surprisingly enough, disabling checksums has a negative impact on the > throughput with a 10-disk striped pool. This impact can be attributed to > the differences in the uDMU IO pipeline which are probably causing the > IOs to not being parallelized when checksumming is disabled. This issue, > however, should not be hard to fix. > > Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and > mirrored pools. In these configurations, the kernel ZFS currently has a > better throughput than the userspace DMU. > > These results were obtained with only minimal tuning of the DMU (setting > a maximum cache size and increasing the number of I/O threads). Work is > already underway to improve the performance of the DMU and we expect to > see better throughput in the coming months. > > Best regards, > Ricardo > > -- > * Ricardo Manuel Correia * > > Lustre Engineering Group > *Sun Microsystems, Inc.* > Portugal > > Ricardo.M.Correia at Sun.COM > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss -- From garlick at llnl.gov Thu Nov 15 00:14:32 2007 From: garlick at llnl.gov (Jim Garlick) Date: Wed, 14 Nov 2007 16:14:32 -0800 (PST) Subject: [Lustre-devel] [Lustre-discuss] ZFS/DMU benchmarks In-Reply-To: <20071114215804.GJ6168@modernduck.com> References: <47398F16.5000504@Sun.COM> <20071114215804.GJ6168@modernduck.com> Message-ID: It would also be nice to see the tests repeated on Linux on the same hardware. (Obviously not the kernel ZFS stuff though!) Jim p.s. I should add that the efforts at Livermore to test the ZFS/PIOS stuff are on hold until we can get 1.6.2 stablized at scale for our production environment, likely another two months. Sorry we were not of more help. On Wed, 14 Nov 2007, Jody McIntyre wrote: > Hi Ricardo, > > On Tue, Nov 13, 2007 at 11:48:38AM +0000, Ricardo Correia wrote: > >> Here are some current ZFS/DMU benchmark results that might be of >> interest to you. > > It's nice to see these results - thanks for posting them. When you have > time, could you do a comparison with Lustre 1.6.x on ldiskfs + software > RAID 5? It would be good to know how ZFS stands up (probably quite > favourably, even at this early stage.) > > Cheers, > Jody > >> These results were obtained on a Sunfire x4500 with these specifications: >> >> - 2 dual-core AMD Opteron processors >> - 16 GB main memory >> - 48 SATA II HDD (7200 RPM, 500 GB each) >> - Solaris 10 update 3 >> - Userspace DMU from OpenSolaris build 74 >> >> The tool used in this particular benchmark was PIOS, which simulates a >> parallel I/O load typically experienced by an OSS. Since PIOS links >> directly with the DMU, it should give a good estimation of the maximum >> throughput of the userspace DMU-OSS. For comparison, you can also see >> the results of running the same PIOS benchmark with normal POSIX I/O on >> the Solaris 10u3 ZFS implementation. >> >> See the attached PDF for the results. >> >> First page provides PIOS throughput information on striped pools with >> 10, 24 and 46 disks. uDMU outperforms kernel ZFS with 10 disks, however, >> the kernel ZFS implementation scales better to 24 and 46 disks. >> Surprisingly enough, disabling checksums has a negative impact on the >> throughput with a 10-disk striped pool. This impact can be attributed to >> the differences in the uDMU IO pipeline which are probably causing the >> IOs to not being parallelized when checksumming is disabled. This issue, >> however, should not be hard to fix. >> >> Second page shows the results of running PIOS on RAID-Z, RAID-Z2 and >> mirrored pools. In these configurations, the kernel ZFS currently has a >> better throughput than the userspace DMU. >> >> These results were obtained with only minimal tuning of the DMU (setting >> a maximum cache size and increasing the number of I/O threads). Work is >> already underway to improve the performance of the DMU and we expect to >> see better throughput in the coming months. >> >> Best regards, >> Ricardo >> >> -- >> * Ricardo Manuel Correia * >> >> Lustre Engineering Group >> *Sun Microsystems, Inc.* >> Portugal >> >> Ricardo.M.Correia at Sun.COM >> > > >> _______________________________________________ >> Lustre-discuss mailing list >> Lustre-discuss at clusterfs.com >> https://mail.clusterfs.com/mailman/listinfo/lustre-discuss > > > -- > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at clusterfs.com > https://mail.clusterfs.com/mailman/listinfo/lustre-devel > From Ricardo.M.Correia at Sun.COM Tue Nov 20 17:01:38 2007 From: Ricardo.M.Correia at Sun.COM (Ricardo M. Correia) Date: Tue, 20 Nov 2007 17:01:38 +0000 Subject: [Lustre-devel] ZFS/DMU benchmarks In-Reply-To: <47398F16.5000504@Sun.COM> References: <47398F16.5000504@Sun.COM> Message-ID: <474312F2.5010809@Sun.COM> Hi, If you want to run the ZFS/DMU benchmarks yourself in Linux or Solaris, you will find instructions in the following wiki page: http://wiki.lustre.org/index.php?title=PIOS-DMU There are also instructions for running the Lustre uOSS alpha/demo on both operating systems here: http://wiki.lustre.org/index.php?title=Lustre_OSS/MDS_with_ZFS_DMU Let me know if you run into any problems. Best regards, Ricardo -- *Ricardo Manuel Correia* Lustre Engineering *Sun Microsystems, Inc.* Portugal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 6g_top.gif Type: image/gif Size: 1257 bytes Desc: not available URL: From chas at cmf.nrl.navy.mil Mon Nov 26 17:03:06 2007 From: chas at cmf.nrl.navy.mil (chas williams - CONTRACTOR) Date: Mon, 26 Nov 2007 12:03:06 -0500 Subject: [Lustre-devel] parallelism across oss or ost? Message-ID: <200711261703.lAQH369U028247@cmf.nrl.navy.mil> if i have a single oss with multiple ost's do i get any parallelism when striping across the ost's? i.e. would a client form multiple connections on whatever transport between the itself the oss? From Brian.Murrell at Sun.COM Mon Nov 26 17:55:48 2007 From: Brian.Murrell at Sun.COM (Brian J. Murrell) Date: Mon, 26 Nov 2007 12:55:48 -0500 Subject: [Lustre-devel] parallelism across oss or ost? In-Reply-To: <200711261703.lAQH369U028247@cmf.nrl.navy.mil> References: <200711261703.lAQH369U028247@cmf.nrl.navy.mil> Message-ID: <1196099748.31044.4.camel@pc.ilinx> On Mon, 2007-11-26 at 12:03 -0500, chas williams - CONTRACTOR wrote: > if i have a single oss with multiple ost's do i get any parallelism > when striping across the ost's? If the hardware can handle it. i.e. as long as the bandwidth between the clients and the OSS is capable and as long as the bandwidth and/or connect paths to the disk can handle it. We have many installations with >1 OST/OSS simply because usually, the bandwidth of a single disk is much less than the next slowest component in and out of a single OSS. > i.e. would a client form multiple > connections on whatever transport between the itself the oss? If you are asking what I think you are, yes. Maybe not as literally as you are asking, but logically, yes. A client will utilize multiple OSTs in an OSS, in parallel if the file access patterns dictate such. b. From rs2006ts at hotmail.com Wed Nov 28 14:57:22 2007 From: rs2006ts at hotmail.com (RS RS) Date: Wed, 28 Nov 2007 09:57:22 -0500 Subject: [Lustre-devel] ZFS in User Space In-Reply-To: <474312F2.5010809@Sun.COM> References: <47398F16.5000504@Sun.COM> <474312F2.5010809@Sun.COM> Message-ID: It appears, from what I’ve read, that the future of Lustre will be based on ZFS running in User space. What are the advantages to doing this in User space? I’ve also read that this will be available in Lustre 1.8. What is the timeframe for this? Thanks. -Roger _________________________________________________________________ Connect and share in new ways with Windows Live. http://www.windowslive.com/connect.html?ocid=TXT_TAGLM_Wave2_newways_112007 -------------- next part -------------- An HTML attachment was scrubbed... URL: