[Lustre-discuss] Lustre Storage Sizing- How?
Atul Vidwansa
Atul.Vidwansa at Sun.COM
Fri Jan 8 01:38:59 PST 2010
Hi Deval,
Lustre storage sizing is largely driven by:
* Capacity required
* Performance required
* Type of workload
Lustre 1.8.1.1 has a limit of 8 TB for an individual OST. Lets say you
are using SATA disks for OST. A Seagate enterprise 1TB SATA disk can do
around 90 MB/Sec with 1 MB blocksize using dd (can go upto 110 MB/Sec if
blocksize is really large). Assuming that you are looking for RAID6
protection for OST, you need 10 SATA disks to form a 8 TB lun.
You will need 4 such OSTs to give you 32 TB unformatted space.
Lets consider performance:
Ideally, you should get 720 MB/Sec/OST [ 90 MB/sec/disk X 8 data disks
in (8+2) RAID6 set]. But you have to cater for overhead of
software/hardware RAID and limits of SAS PCIe HCA (or FC hardware RAID
HCA). A 4gbps FC HCA tops out at 500 MB/Sec so you need 5-6 FC HCAs to
utilize storage bandwidth of 4 RAID6 OSTs [Total bandwidth = 4 X 720
MB/Sec/OST = 2.8 GB/Sec].
So, now you have a storage system that delivers 32 TB unformatted space
and 2.8 GB/Sec of performance for large sequential read/write workload.
If you are planning to have mixed or small io workload and still want to
achieve 2 GB/Sec throughput, you have to double the specs. Small, random
IO (think of home directories) kills storage performance.
Lets size MDS now.
There is no direct relation between size of OST and that of MDT. MDTs
are purely based on number of files required. It is a good idea to use
FC or SAS disks for MDS as they spin at higher rate and have better IOPS
performance. For example, lets consider Seagate enterprise 15 K rpm 300
GB SAS disks. You can put 4 such SAS disks in RAID10 configuration for
MDT which will give you 600 GB of unformatted space.
Lustre needs 4 KB of metadata for each file created, so you can store
about 150 Million files in 600 GB MDT. In reality, this number would be
much smaller depending on your average file size [no of files = total
size of OST/average file size].
Hope this helps.
Cheers,
_Atul
Deval kulshrestha wrote:
> Hi
> I am considering a new storage of 30 TB usable space with a 2 GB/s sustained
> read write performance in clustered mode. But not able to figure out sizing
> part of it like what OSS, what OST and what MDS.
> Urgent help would be highly appreciable
>
> Thanks and Regards
> Deval
>
> -----Original Message-----
> From: lustre-discuss-bounces at lists.lustre.org
> [mailto:lustre-discuss-bounces at lists.lustre.org] On Behalf Of
> lustre-discuss-request at lists.lustre.org
> Sent: Friday, January 08, 2010 12:30 AM
> To: lustre-discuss at lists.lustre.org
> Subject: Lustre-discuss Digest, Vol 48, Issue 13
>
> Send Lustre-discuss mailing list submissions to
> lustre-discuss at lists.lustre.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
> or, via email, send a message with subject or body 'help' to
> lustre-discuss-request at lists.lustre.org
>
> You can reach the person managing the list at
> lustre-discuss-owner at lists.lustre.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Lustre-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: Lustre-discuss Digest, Vol 48, Issue 11 (Jim Garlick)
> 2. Error on restarted Lustre disk--follow-up (Ms. Megan Larko)
> 3. Re: Lustre Monitoring Tools (Erik Froese)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 7 Jan 2010 09:13:55 -0800
> From: Jim Garlick <garlick at llnl.gov>
> Subject: Re: [Lustre-discuss] Lustre-discuss Digest, Vol 48, Issue 11
> To: Dam Thanh Tung <tungdt at isds.vn>
> Cc: lustre-discuss at lists.lustre.org
> Message-ID: <20100107171355.GA32305 at llnl.gov>
> Content-Type: text/plain; charset=us-ascii
>
> Actually lmt is not web-based. Tools for viewing lustre performance
> are included: "ltop" is curses based, and "lwatch" is X based.
>
> http://code.google.com/p/lmt/
>
> Jim
>
> On Thu, Jan 07, 2010 at 09:08:09AM +0700, Dam Thanh Tung wrote:
>
>> You can try collectl <http://*collectl.sourceforge.net/>, i see it from
>>
> the
>
>> 1.8 manual, maybe it's options is not really rich, but i think it's quite
>> good. If you need a web-based monitor tools, you can try
>> lmt<http://*sourceforge.net/projects/lmt/>,
>> i haven't tried this yet. If you feel well with it, let me know please :)
>>
>> Hope this helps
>>
>> Hi Guys,
>>
>>> I would like to monitor the performance and usage of my Lustre
>>>
> filesystem
>
>>> and was wondering what are the commonly used monitoring tools for this?
>>> Cacti? Nagios? Any input would be greatly appreciated.
>>>
>>> Regards,
>>> -Simran
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>>
>>>
> http://*lists.lustre.org/pipermail/lustre-discuss/attachments/20100106/c45e5
> a90/attachment-0001.html
>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://*lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>>
>>> End of Lustre-discuss Digest, Vol 48, Issue 11
>>> **********************************************
>>>
>>>
>
>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://*lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 7 Jan 2010 12:19:09 -0500
> From: "Ms. Megan Larko" <dobsonunit at gmail.com>
> Subject: [Lustre-discuss] Error on restarted Lustre disk--follow-up
> To: Lustre User Discussion Mailing List
> <lustre-discuss at lists.lustre.org>
> Message-ID:
> <9e24b8301001070919q6b4a24fcm9a5d2fa72d125999 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello,
>
> Replying to my own subject line, I had to reboot the
> Lustre-1.6.7.1smp client for reasons completely unrelated to Lustre
> (very related to NFS). After the reboot, the error messages regarding
> the handle change went away. The Lustre disk mounted correctly and
> is usable after the client reboot.
>
> Just FYI,
> megan
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 7 Jan 2010 12:27:39 -0500
> From: Erik Froese <erik.froese at gmail.com>
> Subject: Re: [Lustre-discuss] Lustre Monitoring Tools
> To: Jim Garlick <garlick at llnl.gov>
> Cc: Cliff White <Cliff.White at sun.com>,
> "lustre-discuss at lists.lustre.org" <lustre-discuss at lists.lustre.org>
> Message-ID:
> <f9ba03e41001070927u1888f7edh29a063703b958dc5 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I have it running on 1.8. 1.1. It works well. I had to edit the SQL it
> generated though.
> Erik
>
> On Wed, Jan 6, 2010 at 2:37 PM, Jim Garlick <garlick at llnl.gov> wrote:
>
>
>> I'm using LMT with 1.8 on our test system and it seems to be OK.
>> We're still 1.6.6 in production so it hasn't been extensively tested with
>> 1.8.
>>
>> Jim
>>
>> On Wed, Jan 06, 2010 at 11:23:54AM -0800, Cliff White wrote:
>>
>>> Jeffrey Bennett wrote:
>>>
>>>> Last time I checked, LMT was designed for Lustre 1.4. LLNL stopped
>>>>
>> development of LMT some time ago. Not sure if LMT will work with Lustre
>>
> 1.8.
>
>> If somebody has tried, please let everyone know.
>>
>>> Ah, it has moved to Google:
>>> http://code.google.com/p/lmt/
>>>
>>> "The current release has been tested with Lustre 1.6.6."
>>> So, yup, seems a bit old. But might be worth looking into.
>>> cliffw
>>>
>>>
>>>> jab
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: lustre-discuss-bounces at lists.lustre.org [mailto:
>>>>
>> lustre-discuss-bounces at lists.lustre.org] On Behalf Of Cliff White
>>
>>>> Sent: Wednesday, January 06, 2010 11:12 AM
>>>> To: Jagga Soorma
>>>> Cc: lustre-discuss at lists.lustre.org
>>>> Subject: Re: [Lustre-discuss] Lustre Monitoring Tools
>>>>
>>>> Jagga Soorma wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> I would like to monitor the performance and usage of my Lustre
>>>>> filesystem and was wondering what are the commonly used monitoring
>>>>>
>> tools
>>
>>>>> for this? Cacti? Nagios? Any input would be greatly appreciated.
>>>>>
>>>>> Regards,
>>>>> -Simran
>>>>>
>>>>>
>>>> LLNL's LMT tool is very good. It's available on Sourceforge, afaik.
>>>> cliffw
>>>>
>>>>
>> ------------------------------------------------------------------------
>>
>>>>> _______________________________________________
>>>>> Lustre-discuss mailing list
>>>>> Lustre-discuss at lists.lustre.org
>>>>> http://*lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>>
>>>> _______________________________________________
>>>> Lustre-discuss mailing list
>>>> Lustre-discuss at lists.lustre.org
>>>> http://*lists.lustre.org/mailman/listinfo/lustre-discuss
>>>>
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> Lustre-discuss at lists.lustre.org
>>> http://*lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>> _______________________________________________
>> Lustre-discuss mailing list
>> Lustre-discuss at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100107/3063de
> ce/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> End of Lustre-discuss Digest, Vol 48, Issue 13
> **********************************************
>
> ===========================================================
> Privileged or confidential information may be contained
> in this message. If you are not the addressee indicated
> in this message (or responsible for delivery of the
> message to such person), please delete this message and
> kindly notify the sender by an emailed reply. Opinions,
> conclusions and other information in this message that
> do not relate to the official business of Progression
> and its associate entities shall be understood as neither
> given nor endorsed by them.
>
>
> -----------------------------------------------------------------------
> Progression Infonet Private Limited, Gurgaon (Haryana), India
> Authorised dealer of PostMaster, by QuantumLink Communications Pvt. Ltd
> Get your free copy of PostMaster at http://www.postmaster.co.in/
>
>
>
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
More information about the lustre-discuss
mailing list