[Lustre-discuss] What's the human translation for: ost_write operation failed with -28

[Christian Becker]

Hi Thomas,

On 12/05/2011 08:30 AM, Thomas Guthmann wrote:
> Hi,
>
> Over the week-end on the client sides we started to see a lot of:
>
> LustreError: 11-0: an error occurred while communicating with
> 192.168.1.32 at tcp. The ost_write operation failed with -28
> LustreError: Skipped 23528 previous similar messages

The negative number in this line means the usual linux error code, in 
this case:

# grep 28 /usr/include/asm-generic/errno-base.h
#define ENOSPC          28      /* No space left on device */

best regards,
Christian

>
> What do they mean or imply, I guess -28 means a specific error ?
>
> The host 192.168.1.32 is up and provides other lustre filesystems which
> don't have this problem (finger crossed:).Once this error happened, I
> couldn't write to the filesystem anymore despite 250GB free (lfs df -h).
> Unmount / remount the lustrefs fixed the issue. But then the error came
> back later. As usual everything went fine for 2 years until today ;)
>
> Any ideas, leads I can follow to investigate more this issue ?
>
> BTW on the server side we 'only' have the usual messages we had before
> the disaster where xxxx is our lustrefs having the above issue.
>
> Lustre: Skipped 2 previous similar messages
> Lustre: xxxx-OST0004: slow direct_io 73s due to heavy IO load
> Lustre: xxxx-OST0004: slow journal start 72s due to heavy IO load
> Lustre: xxxx-OST0004: slow commitrw commit 72s due to heavy IO load
> Lustre: xxxx-OST0003: slow journal start 146s due to heavy IO load
> Lustre: xxxx-OST0003: slow brw_start 163s due to heavy IO load
> Lustre: Skipped 1 previous similar message
> Lustre: xxxx-OST0003: slow journal start 164s due to heavy IO load
> Lustre: xxxx-OST0003: slow commitrw commit 164s due to heavy IO load
> Lustre: xxxx-OST0003: slow direct_io 164s due to heavy IO load
>
> centos5# rpm -qa |grep lustre
> lustre-modules-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5
> lustre-ldiskfs-3.1.4-2.6.18_194.17.1.el5_lustre.1.8.5
> kernel-devel-2.6.18-194.17.1.el5_lustre.1.8.5
> kernel-2.6.18-194.17.1.el5_lustre.1.8.5
> lustre-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5
>
> Cheers,
> Thomas
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss