[Lustre-discuss] Fwd: Reg /// OSS rebooted automatically

[Kevin Van Maren]

Daniel Raj wrote:
> Hi Jeff,
>
>
> *messages :*

> Dec 19 11:42:26 cluster kernel: LustreError: 
> 23330:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 3 previous 
> similar messages
> Dec 19 11:42:26 cluster kernel: LustreError: Skipped 3 previous 
> similar messages
> Dec 19 11:44:05 cluster kernel: LustreError: 137-5: UUID 
> 'cluster-ost8_UUID' is not available  for connect (no target)
> Dec 19 11:44:05 cluster kernel: LustreError: 
> 23292:0:(ldlm_lib.c:1892:target_send_reply_msg()) @@@ processing error 
> (-19)  req at ffff810283dee000 x1353488904620989/t0 o8-><?>@<?>:0/0 lens 
> 368/0 e 0 to 0 dl 1292739345 ref 1 fl Interpret:/0/0 rc -19/0
> Dec 19 11:44:05 cluster kernel: LustreError: 
> 23292:0:(ldlm_lib.c:1892:target_send_reply_msg()) Skipped 3 previous 
> similar messages
> Dec 19 11:44:05 cluster kernel: LustreError: Skipped 3 previous 
> similar messages

Looks like here the system rebooted on its own.  Kernel crashes are not 
always able to be sent to syslog, so you would need the serial console 
output.
You might also want to check the IPMI SEL (system event log) to see if 
there were any hardware errors recorded for your node.

> Dec 19 11:48:11 cluster syslogd 1.4.1: restart.
> Dec 19 11:48:11 cluster kernel: klogd 1.4.1, log source = /proc/kmsg 
> started.
> Dec 19 11:48:11 cluster kernel: Linux version 
> 2.6.18-164.11.1.el5_lustre.1.8.3 (lbuild at x86-build-1) (gcc version 
> 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Fri Apr 9 18:00:39 MDT 2010
> Dec 19 11:48:11 cluster kernel: Command line: ro root=LABEL=/ rhgb 
> quiet  irqpoll maxcpus=1 reset_devices  memmap=exactmap memmap=640K at 0K 
> memmap=5276K at 16384K memmap=125140K at 22300K elfcorehdr=147440K 
> memmap=32K#3144016K
> Dec 19 11:48:11 cluster kernel: BIOS-provided physical RAM map:
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 0000000000010000 - 
> 000000000009ec00 (usable)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 000000000009ec00 - 
> 00000000000a0000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 0000000000100000 - 
> 00000000bfe54000 (usable)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000bfe54000 - 
> 00000000bfe5c000 (ACPI data)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000bfe5c000 - 
> 00000000bfe5d000 (usable)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000bfe5d000 - 
> 00000000c0000000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000e0000000 - 
> 00000000f0000000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000fec00000 - 
> 00000000fed00000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000fee00000 - 
> 00000000fee10000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 00000000ffc00000 - 
> 0000000100000000 (reserved)
> Dec 19 11:48:11 cluster kernel:  BIOS-e820: 0000000100000000 - 
> 000000043ffff000 (usable)

Here someone initiated a reboot:

> Dec 19 11:51:25 cluster init: Switching to runlevel: 6
> Dec 19 11:51:27 cluster rpc.statd[5886]: Caught signal 15, 
> un-registering and exiting.
> Dec 19 11:51:27 cluster multipathd: --------shut down-------
> Dec 19 11:51:27 cluster auditd[5740]: The audit daemon is exiting.
> Dec 19 11:51:27 cluster kernel: audit(1292739687.831:17): audit_pid=0 
> old=5740 by auid=4294967295
> Dec 19 11:51:27 cluster kernel: Kernel logging (proc) stopped.
> Dec 19 11:51:27 cluster kernel: Kernel log daemon terminating.
> Dec 19 11:51:29 cluster exiting on signal 15
> Dec 19 11:55:48 cluster syslogd 1.4.1: restart.
> Dec 19 11:55:48 cluster kernel: klogd 1.4.1, log source = /proc/kmsg 
> started.
> Dec 19 11:55:48 cluster kernel: Linux version 
> 2.6.18-164.11.1.el5_lustre.1.8.3 (lbuild at x86-build-1) (gcc version 
> 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Fri Apr 9 18:00:39 MDT 2010
> Dec 19 11:55:48 cluster kernel: Command line: ro root=LABEL=/ rhgb 
> quiet crashkernel=128M at 16M
> Dec 19 11:55:48 cluster kernel: BIOS-provided physical RAM map: