[Lustre-discuss] Nodes claim error with files, then say everything is fine.

[Chris Worley]

Going back further in the logs, there seem to be lots of this message:

Lustre: 5594:0:(client.c:1587:ptlrpc_unregister_reply()) @@@
Unexpectedly long timeout rvcng=1 unlnk=1  req at ffff810270717b68
x1187993/t0 o400->MGS at 36.102.29.1@o2ib:26/25 lens 128/256 e 1 to 40 dl
1217828500 ref 1 fl Rpc:EN/0/0 rc -4/0


On Wed, Aug 6, 2008 at 8:24 AM, Chris Worley <worleys at gmail.com> wrote:
> On Wed, Aug 6, 2008 at 8:19 AM, Brian J. Murrell <Brian.Murrell at sun.com> wrote:
>>
>> Are there any Lustre messages in your syslog on the clients which are
>> returning unexpected results?
>
> Indeed.  These clients seem to be timing out and getting evicted:
>
> LustreError: 11-0: an error occurred while communicating with
> 36.102.29.1 at o2ib. The ldlm_enqueue operation failed with -107
> LustreError: Skipped 36 previous similar messages
> Lustre: lfs-MDT0000-mdc-ffff81027cca8d20: Connection to service
> lfs-MDT0000 via nid 36.102.29.1 at o2ib was lost; in progress operations
> using this service will wait for recovery to complete.
> Lustre: Skipped 36 previous similar messages
> LustreError: 167-0: This client was evicted by lfs-MDT0000; in
> progress operations using this service will fail.
> LustreError: Skipped 36 previous similar messages
> LustreError: 3143:0:(mdc_locks.c:586:mdc_enqueue()) ldlm_cli_enqueue: -5
> LustreError: 3143:0:(mdc_locks.c:586:mdc_enqueue()) Skipped 1 previous
> similar message
> LustreError: 3143:0:(file.c:2664:ll_inode_revalidate_fini()) failure
> -5 inode 66781185
> LustreError: 3143:0:(client.c:716:ptlrpc_import_delay_req()) @@@
> IMP_INVALID  req at ffff81016f0f0988 x2285852/t0
> o101->lfs-MDT0000_UUID at 36.102.29.1@o2ib:12/10 lens 432/2048 e 0 to 40
> dl 0 ref 1 fl Rpc:/0/0 rc 0/0
> LustreError: 3143:0:(client.c:716:ptlrpc_import_delay_req()) Skipped 5
> previous similar messages
> LustreError: 3143:0:(file.c:2664:ll_inode_revalidate_fini()) failure
> -108 inode 66781185
> LustreError: 3188:0:(client.c:716:ptlrpc_import_delay_req()) @@@
> IMP_INVALID  req at ffff8104bfc93238 x2285905/t0
> o101->lfs-MDT0000_UUID at 36.102.29.1@o2ib:12/10 lens 432/2048 e 0 to 40
> dl 0 ref 1 fl Rpc:/0/0 rc 0/0
> LustreError: 3188:0:(client.c:716:ptlrpc_import_delay_req()) Skipped 1
> previous similar message
> LustreError: 3188:0:(mdc_locks.c:586:mdc_enqueue()) ldlm_cli_enqueue: -108
> LustreError: 3188:0:(mdc_locks.c:586:mdc_enqueue()) Skipped 2 previous
> similar messages
> LustreError: 3188:0:(file.c:2664:ll_inode_revalidate_fini()) failure
> -108 inode 66781185
> LustreError: 3188:0:(file.c:2664:ll_inode_revalidate_fini()) Skipped 1
> previous similar message
> LustreError: 3190:0:(file.c:2664:ll_inode_revalidate_fini()) failure
> -108 inode 66781185
> LustreError: 3190:0:(file.c:2664:ll_inode_revalidate_fini()) Skipped 1
> previous similar message
>