[Lustre-discuss] "ls -l" command in the Lustre Filesystem

[Brian J. Murrell]

On Mon, 2008-07-07 at 03:29 -0700, Johnlya wrote:
> I found that executing "ls" command would need more time
> in the Lustre filesystem than in the ext3 filesystem.

This is not terribly surprising if you think about what Lustre is doing.

An ls on ext3 involves lots of lookup(s) of (meta)data to a very local
device, namely a disk on a SATA/SCSI/IDE bus local in the machine.  The
latency of this lookup is very low.  Those same ls lookups on Lustre has
to go out on a network to another machine to do the same metadata
lookups plus it has to go to the OST(s) that hold the file to get the
size and has to return that data all on the network again, with higher
latencies.

Indeed, there are features in Lustre, such as "statahead" to try to
streamline the data lookups and data movement and reduce the round-trip
latencies, but you are still not working with "like" technologies.

Instead you are paying a small price for huge gains.  For people who
need Lustre, using a local ext3 filesystem simply won't scale for them.
Lustre will.  Their workload is not usually "ls" either though and they
appreciate the scalability they get for the small cost of a slower "ls".

That said, the future will bring clustered metadata (CMD) storage, where
an ls of a large directory could actually be faster than an ls to the
same sized directory on a local ext3 filesystem due to parallelizing
metadata the way we can with object data over many OSTs today.

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20080708/c910166a/attachment.pgp>