[Lustre-discuss] Kernel crash

Chan Ching Yu, Patrick cychan at clustertech.com
Fri Jul 19 20:05:54 PDT 2013


Hi,

We have been using Lustre-2.1.5 for few months. LNET device of all the MGS, MDS and OSS is a bonding interface bond1 with ib0 and ib1 as the slave interface (they are Infiniband). However, one client was rebooted with a kernel dump.

Anyone knows the problem?  Thanks in advance.

For your reference, I attach the backtrace and system message buffer in crash as follows:

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.19.1.el6_lustre.x86_64/vmlinux
    DUMPFILE: vmcore-machine-2013-07-20-08:48:56  [PARTIAL DUMP]
        CPUS: 16
        DATE: Sat Jul 20 08:47:51 2013
      UPTIME: 17 days, 21:11:15
LOAD AVERAGE: 16.02, 16.02, 15.12
       TASKS: 676
     RELEASE: 2.6.32-279.19.1.el6_lustre.x86_64
     VERSION: #1 SMP Wed Mar 20 16:37:18 PDT 2013
     MACHINE: x86_64  (2599 Mhz)
      MEMORY: 64 GB
       PANIC: "kernel BUG at mm/filemap.c:129!"
         PID: 12891
     COMMAND: "ptlrpcd-rcv"
        TASK: ffff880863c8eaa0  [THREAD_INFO: ffff880869ef0000]
         CPU: 10
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 12891  TASK: ffff880863c8eaa0  CPU: 10  COMMAND: "ptlrpcd-rcv"
#0 [ffff880869ef14a0] machine_kexec at ffffffff81031f7b
#1 [ffff880869ef1500] crash_kexec at ffffffff810b8c22
#2 [ffff880869ef15d0] oops_end at ffffffff814ed980
#3 [ffff880869ef1600] die at ffffffff8100f19b
#4 [ffff880869ef1630] do_trap at ffffffff814ed274
#5 [ffff880869ef1690] do_invalid_op at ffffffff8100cdb5
#6 [ffff880869ef1730] invalid_op at ffffffff8100be5b
    [exception RIP: __remove_from_page_cache+213]
    RIP: ffffffff81110c85  RSP: ffff880869ef17e0  RFLAGS: 00010046
    RAX: 00c0000000000065  RBX: ffffea0030e5f6b0  RCX: 00000000ffffffd9
    RDX: 0000000000000000  RSI: 0000000000000009  RDI: ffff880880010dc0
    RBP: ffff880869ef17f0   R8: 000000000000005a   R9: 0000000000000001
    R10: 0000000000000357  R11: d000000000000000  R12: ffff881047e3e318
    R13: ffff881047e3e330  R14: ffff8810370c5f50  R15: ffff8810450b9980
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#7 [ffff880869ef17f8] remove_from_page_cache at ffffffff81110ce4
#8 [ffff880869ef1828] vvp_page_discard at ffffffffa0bfa2d0 [lustre]
#9 [ffff880869ef1858] cl_page_invoid at ffffffffa061cb08 [obdclass]
#10 [ffff880869ef18a8] cl_page_discard at ffffffffa061cc13 [obdclass]
#11 [ffff880869ef18b8] check_and_discard_cb at ffffffffa0626b0e [obdclass]
#12 [ffff880869ef1908] cl_page_gang_lookup at ffffffffa0621a34 [obdclass]
#13 [ffff880869ef19b8] cl_lock_page_out at ffffffffa06235db [obdclass]
#14 [ffff880869ef1a28] osc_lock_flush at ffffffffa0a9a9cf [osc]
#15 [ffff880869ef1a78] osc_lock_cancel at ffffffffa0a9aadf [osc]
#16 [ffff880869ef1ac8] cl_lock_cancel0 at ffffffffa0622265 [obdclass]
#17 [ffff880869ef1af8] cl_lock_cancel at ffffffffa0622f4b [obdclass]
#18 [ffff880869ef1b18] osc_ldlm_blocking_ast at ffffffffa0a9bbba [osc]
#19 [ffff880869ef1b88] ldlm_cancel_callback at ffffffffa0729cc0 [ptlrpc]
#20 [ffff880869ef1ba8] ldlm_lock_cancel at ffffffffa0729db5 [ptlrpc]
#21 [ffff880869ef1bc8] ldlm_cli_cancel_list_local at ffffffffa07469d8 [ptlrpc]
#22 [ffff880869ef1c28] ldlm_cancel_lru_local at ffffffffa0747895 [ptlrpc]
#23 [ffff880869ef1c48] ldlm_replay_locks at ffffffffa0747a0a [ptlrpc]
#24 [ffff880869ef1cc8] ptlrpc_import_recovery_state_machine at ffffffffa0790187 [ptlrpc]
#25 [ffff880869ef1d08] ptlrpc_connect_interpret at ffffffffa0793288 [ptlrpc]
#26 [ffff880869ef1d98] ptlrpc_check_set at ffffffffa0764cab [ptlrpc]
#27 [ffff880869ef1e38] ptlrpcd_check at ffffffffa0795f30 [ptlrpc]
#28 [ffff880869ef1e68] ptlrpcd at ffffffffa07962c3 [ptlrpc]
#29 [ffff880869ef1f48] kernel_thread at ffffffff8100c0ca
-------------------------------------------------------------------------------------------------------------------

crash> log

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-279.19.1.el6_lustre.x86_64 (jenkins at builder-2-sdd1-el6-x8664.lab.whamcloud.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Wed Mar 20 16:37:18 PDT 2013
Command line: ro root=/dev/mapper/vg_system-lv_root nomodeset rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_system/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet biosdevname=0
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009d000 (usable)
BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000783d7000 (usable)
BIOS-e820: 00000000783d7000 - 00000000797d7000 (reserved)
BIOS-e820: 00000000797d7000 - 000000007e2ff000 (usable)
BIOS-e820: 000000007e2ff000 - 000000007eaff000 (reserved)
BIOS-e820: 000000007eaff000 - 000000007eeff000 (ACPI NVS)
BIOS-e820: 000000007eeff000 - 000000007efff000 (ACPI data)
BIOS-e820: 000000007efff000 - 000000007f000000 (usable)
BIOS-e820: 000000007f000000 - 0000000090000000 (reserved)
BIOS-e820: 00000000feb00000 - 00000000feb04000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
BIOS-e820: 00000000fed18000 - 00000000fed19000 (reserved)
BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000001080000000 (usable)
DMI 2.7 present.
SMBIOS version 2.7 @ 0xFE120
DMI: Dell Inc. PowerEdge C6220/0TTH1R, BIOS 1.1.19 02/25/2013
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x1080000 max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR fixed ranges enabled:
  00000-9FFFF write-back
  A0000-BFFFF uncachable
  C0000-EFFFF write-protect
  F0000-FFFFF write-combining
MTRR variable ranges enabled:
  0 base 000000000000 mask 3FFFC0000000 write-back
  1 base 0000FFC00000 mask 3FFFFFC00000 write-protect
  2 base 000040000000 mask 3FFFC0000000 write-back
  3 disabled
  4 base 000100000000 mask 3FFF00000000 write-back
  5 base 000200000000 mask 3FFE00000000 write-back
  6 base 000400000000 mask 3FFC00000000 write-back
  7 base 000800000000 mask 3FF800000000 write-back
  8 base 001000000000 mask 3FFF80000000 write-back
  9 disabled
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
last_pfn = 0x7f000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
Using GB pages for direct mapping
init_memory_mapping: 0000000000000000-000000007f000000
0000000000 - 0040000000 page 1G
0040000000 - 007f000000 page 2M
kernel direct mapping tables up to 7f000000 @ 8000-a000
init_memory_mapping: 0000000100000000-0000001080000000
0100000000 - 1080000000 page 1G
kernel direct mapping tables up to 1080000000 @ 9000-a000
RAMDISK: 349ab000 - 37fef925
ACPI: RSDP 00000000000fe020 00024 (v02 DELL  )
ACPI: XSDT 000000007effe170 0009C (v01 DELL   DCSRADON 00000001      01000013)
ACPI: FACP 000000007effa000 000F4 (v04 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: DSDT 000000007efec000 09B24 (v01 DELL   DCSRADON 00000000 DELL 00040000)
ACPI: FACS 000000007eda3000 00040
ACPI: UEFI 000000007effd000 00236 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SPMI 000000007effc000 00040 (v05 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: BOOT 000000007effb000 00028 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: APIC 000000007eff8000 00676 (v02 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: MCFG 000000007eff7000 0003C (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SLIC 000000007eff6000 00176 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SRAT 000000007efea000 00580 (v02 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SLIT 000000007efe9000 00030 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: BDAT 000000007efe8000 00030 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: FPDT 000000007efe6000 00044 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SSDT 000000007ef6c000 79E84 (v02 DELL   DCSRADON 00004000 DELL 00040000)
ACPI: HPET 000000007eff9000 00038 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: WDAT 000000007ef6b000 00194 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: SPCR 000000007efeb000 00050 (v01 DELL   DCSRADON 00000001 DELL 00040000)
ACPI: Local APIC address 0xfee00000
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 2 -> Node 0
SRAT: PXM 0 -> APIC 4 -> Node 0
SRAT: PXM 0 -> APIC 6 -> Node 0
SRAT: PXM 0 -> APIC 8 -> Node 0
SRAT: PXM 0 -> APIC 10 -> Node 0
SRAT: PXM 0 -> APIC 12 -> Node 0
SRAT: PXM 0 -> APIC 14 -> Node 0
SRAT: PXM 1 -> APIC 32 -> Node 1
SRAT: PXM 1 -> APIC 34 -> Node 1
SRAT: PXM 1 -> APIC 36 -> Node 1
SRAT: PXM 1 -> APIC 38 -> Node 1
SRAT: PXM 1 -> APIC 40 -> Node 1
SRAT: PXM 1 -> APIC 42 -> Node 1
SRAT: PXM 1 -> APIC 44 -> Node 1
SRAT: PXM 1 -> APIC 46 -> Node 1
SRAT: Node 0 PXM 0 0-880000000
SRAT: Node 1 PXM 1 880000000-1080000000
NUMA: Using 31 for the hash shift.
Bootmem setup node 0 0000000000000000-0000000880000000
  NODE_DATA [0000000000009000 - 000000000003cfff]
  bootmap [0000000000100000 -  000000000020ffff] pages 110
(8 early reservations) ==> bootmem [0000000000 - 0880000000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0001000000 - 0002011024]    TEXT DATA BSS ==> [0001000000 - 0002011024]
  #3 [00349ab000 - 0037fef925]          RAMDISK ==> [00349ab000 - 0037fef925]
  #4 [000009d000 - 0000100000]    BIOS reserved ==> [000009d000 - 0000100000]
  #5 [0002012000 - 000201231b]              BRK ==> [0002012000 - 000201231b]
  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
  #7 [0000001000 - 0000001030]        ACPI SLIT ==> [0000001000 - 0000001030]
Bootmem setup node 1 0000000880000000-0000001080000000
  NODE_DATA [0000000880000040 - 000000088003403f]
  bootmap [0000000880035000 -  0000000880134fff] pages 100
(8 early reservations) ==> bootmem [0880000000 - 1080000000]
  #0 [0000000000 - 0000001000]   BIOS data page
  #1 [0000006000 - 0000008000]       TRAMPOLINE
  #2 [0001000000 - 0002011024]    TEXT DATA BSS
  #3 [00349ab000 - 0037fef925]          RAMDISK
  #4 [000009d000 - 0000100000]    BIOS reserved
  #5 [0002012000 - 000201231b]              BRK
  #6 [0000008000 - 0000009000]          PGTABLE
  #7 [0000001000 - 0000001030]        ACPI SLIT
found SMP MP-table at [ffff8800000fe1b0] fe1b0
Reserving 133MB of memory at 48MB for crashkernel (System RAM: 67584MB)
[ffffea0000000000-ffffea000ddfffff] PMD -> [ffff880028600000-ffff8800347fffff] on node 0
[ffffea000de00000-ffffea001dbfffff] PMD -> [ffff880038000000-ffff880047dfffff] on node 0
[ffffea001dc00000-ffffea0039bfffff] PMD -> [ffff880880200000-ffff88089c1fffff] on node 1
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x01080000
Movable zone start PFN for each node
early_node_map[6] active PFN ranges
    0: 0x00000001 -> 0x0000009d
    0: 0x00000100 -> 0x000783d7
    0: 0x000797d7 -> 0x0007e2ff
    0: 0x0007efff -> 0x0007f000
    0: 0x00100000 -> 0x00880000
    1: 0x00880000 -> 0x01080000
On node 0 totalpages: 8375964
  DMA zone: 56 pages used for memmap
  DMA zone: 103 pages reserved
  DMA zone: 3837 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 493368 pages, LIFO batch:31
  Normal zone: 107520 pages used for memmap
  Normal zone: 7756800 pages, LIFO batch:31
On node 1 totalpages: 8388608
  Normal zone: 114688 pages used for memmap
  Normal zone: 8273920 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x408
ACPI: Local APIC address 0xfee00000
ACPI: X2APIC (apic_id[0xffffffff] uid[0x00] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x01] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x02] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x03] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x04] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x05] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x06] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x07] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x08] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x09] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0a] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0b] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0c] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0d] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0e] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x0f] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x10] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x11] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x12] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x13] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x14] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x15] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x16] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x17] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x18] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x19] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1a] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1b] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1c] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1d] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1e] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x1f] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x20] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x21] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x22] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x23] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x24] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x25] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x26] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x27] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x28] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x29] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2a] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2b] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2c] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2d] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2e] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x2f] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x30] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x31] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x32] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x33] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x34] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x35] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x36] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x37] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x38] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x39] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3a] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3b] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3c] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3d] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3e] disabled)
ACPI: X2APIC (apic_id[0xffffffff] uid[0x3f] disabled)
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled)
ACPI: LAPIC (acpi_id[0x10] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x11] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x12] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x13] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x14] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x15] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x16] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x17] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x18] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x19] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1a] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1b] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1c] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1d] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1e] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x1f] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled)
ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled)
ACPI: X2APIC_NMI (uid[0xffffffff] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47
ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48])
IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a701 base: 0xfed00000
SMP: Allowing 128 CPUs, 112 hotplug CPUs
nr_irqs_gsi: 72
PM: Registered nosave memory: 000000000009d000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
PM: Registered nosave memory: 00000000783d7000 - 00000000797d7000
PM: Registered nosave memory: 000000007e2ff000 - 000000007eaff000
PM: Registered nosave memory: 000000007eaff000 - 000000007eeff000
PM: Registered nosave memory: 000000007eeff000 - 000000007efff000
PM: Registered nosave memory: 000000007f000000 - 0000000090000000
PM: Registered nosave memory: 0000000090000000 - 00000000feb00000
PM: Registered nosave memory: 00000000feb00000 - 00000000feb04000
PM: Registered nosave memory: 00000000feb04000 - 00000000fec00000
PM: Registered nosave memory: 00000000fec00000 - 00000000fec01000
PM: Registered nosave memory: 00000000fec01000 - 00000000fed18000
PM: Registered nosave memory: 00000000fed18000 - 00000000fed19000
PM: Registered nosave memory: 00000000fed19000 - 00000000fed1c000
PM: Registered nosave memory: 00000000fed1c000 - 00000000fed90000
PM: Registered nosave memory: 00000000fed90000 - 00000000fee00000
PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
PM: Registered nosave memory: 00000000fee01000 - 00000000ffc00000
PM: Registered nosave memory: 00000000ffc00000 - 0000000100000000
Allocating PCI resources starting at 90000000 (gap: 90000000:6eb00000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:128 nr_cpu_ids:128 nr_node_ids:2
PERCPU: Embedded 31 pages/cpu @ffff880047e00000 s94424 r8192 d24360 u131072
pcpu-alloc: s94424 r8192 d24360 u131072 alloc=1*2097152
pcpu-alloc: [0] 000 001 002 003 004 005 006 007 016 018 020 022 024 026 028 030 
pcpu-alloc: [0] 032 034 036 038 040 042 044 046 048 050 052 054 056 058 060 062 
pcpu-alloc: [0] 064 066 068 070 072 074 076 078 080 082 084 086 088 090 092 094 
pcpu-alloc: [0] 096 098 100 102 104 106 108 110 112 114 116 118 120 122 124 126 
pcpu-alloc: [1] 008 009 010 011 012 013 014 015 017 019 021 023 025 027 029 031 
pcpu-alloc: [1] 033 035 037 039 041 043 045 047 049 051 053 055 057 059 061 063 
pcpu-alloc: [1] 065 067 069 071 073 075 077 079 081 083 085 087 089 091 093 095 
pcpu-alloc: [1] 097 099 101 103 105 107 109 111 113 115 117 119 121 123 125 127 
Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16527925
Policy zone: Normal
Kernel command line: ro root=/dev/mapper/vg_system-lv_root nomodeset rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=133M at 0M rd_LVM_LV=vg_system/lv_swap rd_LVM_LV=vg_system/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet biosdevname=0
PID hash table entries: 4096 (order: 3, 32768 bytes)
xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Checking aperture...
No AGP bridge found
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 65849756k/69206016k available (5073k kernel code, 2147728k absent, 1208532k reserved, 7245k data, 1256k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:2248
Extended CMOS year: 2000
Console: colour VGA+ 80x25
console [tty0] enabled
allocated 536870912 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
hpet clockevent registered
Fast TSC calibration using PIT
Detected 2599.956 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5199.91 BogoMIPS (lpj=2599956)
pid_max: default: 131072 minimum: 1024
Security Framework initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes)
Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 20 MCE banks
CPU0: Thermal monitoring enabled (TM1)
using mwait in idle threads.
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20990 entries in 83 pages
x2apic not enabled, IRQ remapping init failed
Setting APIC routing to physical flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz stepping 07
Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
... version:                3
... bit width:              48
... generic registers:      8
... value mask:             0000ffffffffffff
... max period:             000000007fffffff
... fixed-purpose events:   3
... event mask:             00000007000000ff
NMI watchdog enabled, takes one hw-pmu counter.
Booting Node   0, Processors  #1 #2 #3 #4 #5 #6 #7 Ok.
Booting Node   1, Processors  #8 #9 #10 #11 #12 #13 #14 #15
Brought up 16 CPUs
Total of 16 processors activated (83193.48 BogoMIPS).
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2648 bytes
devtmpfs: initialized
PM: Registering ACPI NVS region at 7eaff000 (4194304 bytes)
regulator: core version 0.5
NET: Registered protocol family 16
ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
ACPI: bus type pci registered
PCI: MCFG configuration 0: base 80000000 segment 0 buses 0 - 255
PCI: MCFG area at 80000000 reserved in E820
PCI: Using MMCONFIG at 80000000 - 8fffffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI: Interpreter enabled
ACPI: (supports S0 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
HEST: Table not found.
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])
pci_root PNP0A08:00: host bridge window [io  0x0000-0x0cf7]
pci_root PNP0A08:00: host bridge window [io  0x1000-0xdfff]
pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff]
pci_root PNP0A08:00: host bridge window [mem 0x90000000-0xebffefff]
pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:00.0: PME# disabled
pci 0000:00:01.0: PME# supported from D0 D3hot D3cold
pci 0000:00:01.0: PME# disabled
pci 0000:00:01.1: PME# supported from D0 D3hot D3cold
pci 0000:00:01.1: PME# disabled
pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
pci 0000:00:02.0: PME# disabled
pci 0000:00:04.0: reg 10: [mem 0x91300000-0x91303fff 64bit]
pci 0000:00:04.1: reg 10: [mem 0x91304000-0x91307fff 64bit]
pci 0000:00:04.2: reg 10: [mem 0x91308000-0x9130bfff 64bit]
pci 0000:00:04.3: reg 10: [mem 0x9130c000-0x9130ffff 64bit]
pci 0000:00:04.4: reg 10: [mem 0x91310000-0x91313fff 64bit]
pci 0000:00:04.5: reg 10: [mem 0x91314000-0x91317fff 64bit]
pci 0000:00:04.6: reg 10: [mem 0x91318000-0x9131bfff 64bit]
pci 0000:00:04.7: reg 10: [mem 0x9131c000-0x9131ffff 64bit]
pci 0000:00:05.4: reg 10: [mem 0x91327000-0x91327fff]
pci 0000:00:11.0: PME# supported from D0 D3hot D3cold
pci 0000:00:11.0: PME# disabled
pci 0000:00:16.0: reg 10: [mem 0x91320000-0x9132000f 64bit]
pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
pci 0000:00:16.0: PME# disabled
pci 0000:00:16.1: reg 10: [mem 0x91321000-0x9132100f 64bit]
pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
pci 0000:00:16.1: PME# disabled
pci 0000:00:1a.0: reg 10: [mem 0x91326000-0x913263ff]
pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1a.0: PME# disabled
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: PME# disabled
pci 0000:00:1d.0: reg 10: [mem 0x91325000-0x913253ff]
pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.0: PME# disabled
pci 0000:00:1f.2: reg 10: [io  0x4048-0x404f]
pci 0000:00:1f.2: reg 14: [io  0x4054-0x4057]
pci 0000:00:1f.2: reg 18: [io  0x4040-0x4047]
pci 0000:00:1f.2: reg 1c: [io  0x4050-0x4053]
pci 0000:00:1f.2: reg 20: [io  0x4020-0x403f]
pci 0000:00:1f.2: reg 24: [mem 0x91324000-0x913247ff]
pci 0000:00:1f.2: PME# supported from D3hot
pci 0000:00:1f.2: PME# disabled
pci 0000:00:1f.3: reg 10: [mem 0x91322000-0x913220ff 64bit]
pci 0000:00:1f.3: reg 20: [io  0x4000-0x401f]
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:00:01.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:01.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:02:00.0: reg 10: [mem 0x91220000-0x9123ffff]
pci 0000:02:00.0: reg 18: [io  0x3020-0x303f]
pci 0000:02:00.0: reg 1c: [mem 0x91244000-0x91247fff]
pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
pci 0000:02:00.0: PME# disabled
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 10: [mem 0x91200000-0x9121ffff]
pci 0000:02:00.3: reg 18: [io  0x3000-0x301f]
pci 0000:02:00.3: reg 1c: [mem 0x91240000-0x91243fff]
pci 0000:02:00.3: PME# supported from D0 D3hot D3cold
pci 0000:02:00.3: PME# disabled
pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:00:01.1: PCI bridge to [bus 02-02]
pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
pci 0000:00:01.1:   bridge window [mem 0x91200000-0x912fffff]
pci 0000:00:01.1:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:03:00.0: reg 10: [io  0x2000-0x20ff]
pci 0000:03:00.0: reg 14: [mem 0x91140000-0x91143fff 64bit]
pci 0000:03:00.0: reg 1c: [mem 0x91100000-0x9113ffff 64bit]
pci 0000:03:00.0: reg 30: [mem 0xfffe0000-0xffffffff pref]
pci 0000:03:00.0: supports D1 D2
pci 0000:00:02.0: PCI bridge to [bus 03-03]
pci 0000:00:02.0:   bridge window [io  0x2000-0x2fff]
pci 0000:00:02.0:   bridge window [mem 0x91100000-0x911fffff]
pci 0000:00:02.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:00:11.0: PCI bridge to [bus 04-04]
pci 0000:00:11.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:00:11.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:11.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:05:00.0: supports D1 D2
pci 0000:05:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:05:00.0: PME# disabled
pci 0000:00:1c.0: PCI bridge to [bus 05-06]
pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
pci 0000:00:1c.0:   bridge window [mem 0x91000000-0x910fffff]
pci 0000:00:1c.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
pci 0000:06:00.0: reg 10: [mem 0x90000000-0x90ffffff pref]
pci 0000:06:00.0: reg 14: [mem 0x91000000-0x9101ffff]
pci 0000:06:00.0: reg 18: [io  0x1000-0x107f]
pci 0000:06:00.0: supports D1 D2
pci 0000:06:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:06:00.0: PME# disabled
pci 0000:05:00.0: PCI bridge to [bus 06-06]
pci 0000:05:00.0:   bridge window [io  0x1000-0x1fff]
pci 0000:05:00.0:   bridge window [mem 0x91000000-0x910fffff]
pci 0000:05:00.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
pci 0000:00:1e.0: PCI bridge to [bus 07-07] (subtractive decode)
pci 0000:00:1e.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:00:1e.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:00:1e.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:00:1e.0:   bridge window [io  0x0000-0x0cf7] (subtractive decode)
pci 0000:00:1e.0:   bridge window [io  0x1000-0xdfff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0x000a0000-0x000bffff] (subtractive decode)
pci 0000:00:1e.0:   bridge window [mem 0x90000000-0xebffefff] (subtractive decode)
pci_bus 0000:00: on NUMA node 0 (pxm 0)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE01._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE02._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE03._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EVRP._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.RP01._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
pci0000:00: Requesting ACPI _OSC control (0x1d)
pci0000:00: ACPI _OSC control (0x1d) granted
ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f])
pci0000:7f: Requesting ACPI _OSC control (0x1d)
Unable to assume _OSC PCIe control. Disabling ASPM
ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe])
pci_root PNP0A08:01: host bridge window [io  0x0000-0x0cf7]
pci_root PNP0A08:01: host bridge window [io  0xe000-0xffff]
pci_root PNP0A08:01: host bridge window [mem 0xec000000-0xfbffffff]
pci 0000:80:02.0: PME# supported from D0 D3hot D3cold
pci 0000:80:02.0: PME# disabled
pci 0000:80:02.2: PME# supported from D0 D3hot D3cold
pci 0000:80:02.2: PME# disabled
pci 0000:80:04.0: reg 10: [mem 0xee100000-0xee103fff 64bit]
pci 0000:80:04.1: reg 10: [mem 0xee104000-0xee107fff 64bit]
pci 0000:80:04.2: reg 10: [mem 0xee108000-0xee10bfff 64bit]
pci 0000:80:04.3: reg 10: [mem 0xee10c000-0xee10ffff 64bit]
pci 0000:80:04.4: reg 10: [mem 0xee110000-0xee113fff 64bit]
pci 0000:80:04.5: reg 10: [mem 0xee114000-0xee117fff 64bit]
pci 0000:80:04.6: reg 10: [mem 0xee118000-0xee11bfff 64bit]
pci 0000:80:04.7: reg 10: [mem 0xee11c000-0xee11ffff 64bit]
pci 0000:80:05.4: reg 10: [mem 0xee120000-0xee120fff]
pci 0000:80:02.0: PCI bridge to [bus 81-81]
pci 0000:80:02.0:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:80:02.0:   bridge window [mem 0xfff00000-0x000fffff] (disabled)
pci 0000:80:02.0:   bridge window [mem 0xfff00000-0x000fffff pref] (disabled)
pci 0000:82:00.0: reg 10: [mem 0xee000000-0xee0fffff 64bit]
pci 0000:82:00.0: reg 18: [mem 0xec000000-0xedffffff 64bit pref]
pci 0000:80:02.2: PCI bridge to [bus 82-82]
pci 0000:80:02.2:   bridge window [io  0xf000-0x0000] (disabled)
pci 0000:80:02.2:   bridge window [mem 0xee000000-0xee0fffff]
pci 0000:80:02.2:   bridge window [mem 0xec000000-0xedffffff 64bit pref]
pci_bus 0000:80: on NUMA node 1 (pxm 1)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PE13._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PE15._PRT]
pci0000:80: Requesting ACPI _OSC control (0x1d)
pci0000:80: ACPI _OSC control (0x1d) granted
ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff])
pci0000:ff: Requesting ACPI _OSC control (0x1d)
Unable to assume _OSC PCIe control. Disabling ASPM
ACPI: PCI Interrupt Link [LNKA] (IRQs 1 3 *4 5 6 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 1 3 4 *5 6 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 1 3 4 5 *6 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 1 3 4 5 6 10 11 12 14 15) *9
ACPI: PCI Interrupt Link [LNKE] (IRQs 1 3 4 5 6 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 1 3 4 5 6 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 1 3 4 5 6 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 1 3 4 5 6 *10 11 12 14 15)
vgaarb: device added: PCI:0000:06:00.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:06:00.0
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
PCI: pci_cache_line_size set to 64 bytes
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
  alloc irq_desc for 72 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 73 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 74 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 75 on node 0
  alloc kstat_irqs on node 0
  alloc irq_desc for 76 on node 0
  alloc kstat_irqs on node 0
HPET: 8 timers in total, 5 timers will be used for per-cpu timer
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 72, 73, 74, 75, 76, 0
hpet0: 8 comparators, 64-bit 14.318180 MHz counter
hpet: hpet2 irq 72 for MSI
hpet: hpet3 irq 73 for MSI
hpet: hpet4 irq 74 for MSI
hpet: hpet5 irq 75 for MSI
hpet: hpet6 irq 76 for MSI
Switching to clocksource hpet
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:00: [io  0x0cf8-0x0cff]
pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
pnp 00:01: [mem 0xfed1c000-0xfed1ffff]
pnp 00:01: [mem 0xebfff000-0xebffffff]
pnp 00:01: [mem 0xfed20000-0xfed3ffff]
pnp 00:01: [mem 0x00000000-0xffffffffffffffff disabled]
pnp 00:01: [mem 0xff000000-0xffffffff]
pnp 00:01: [mem 0xfee00000-0xfeefffff]
pnp 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:02: [io  0x0ca8]
pnp 00:02: [io  0x0cac]
pnp 00:02: Plug and Play ACPI device, IDs IPI0001 (active)
pnp 00:03: [io  0x0000-0x001f]
pnp 00:03: [io  0x0081-0x0091]
pnp 00:03: [io  0x0093-0x009f]
pnp 00:03: [io  0x00c0-0x00df]
pnp 00:03: [dma 4]
pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:04: [mem 0xff000000-0xffffffff]
pnp 00:04: Plug and Play ACPI device, IDs INT0800 (active)
pnp 00:05: [mem 0xfed00000-0xfed003ff]
pnp 00:05: Plug and Play ACPI device, IDs PNP0103 (active)
pnp 00:06: [io  0x00f0]
pnp 00:06: [irq 13]
pnp 00:06: Plug and Play ACPI device, IDs PNP0c04 (active)
pnp 00:07: [io  0x002e-0x002f]
pnp 00:07: [io  0x004e-0x004f]
pnp 00:07: [io  0x0061]
pnp 00:07: [io  0x0063]
pnp 00:07: [io  0x0065]
pnp 00:07: [io  0x0067]
pnp 00:07: [io  0x0070]
pnp 00:07: [io  0x0080]
pnp 00:07: [io  0x0092]
pnp 00:07: [io  0x00b2-0x00b3]
pnp 00:07: [io  0x0680-0x069f]
pnp 00:07: [io  0x0400-0x0453]
pnp 00:07: [io  0x0458-0x047f]
pnp 00:07: [io  0x0500-0x057f]
pnp 00:07: [io  0x164e-0x164f]
pnp 00:07: disabling [io  0x164e-0x164f] because it overlaps 0000:00:1c.0 BAR 13 [io  0x1000-0x1fff]
pnp 00:07: disabling [io  0x164e-0x164f disabled] because it overlaps 0000:05:00.0 BAR 13 [io  0x1000-0x1fff]
pnp 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:08: [io  0x0070-0x0077]
pnp 00:08: [irq 8]
pnp 00:08: Plug and Play ACPI device, IDs PNP0b00 (active)
pnp 00:09: [io  0x0454-0x0457]
pnp 00:09: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active)
pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (disabled)
pnp 00:0b: Plug and Play ACPI device, IDs PNP0a03 (active)
pnp 00:0c: [io  0x0cf8-0x0cff]
pnp 00:0c: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active)
pnp 00:0d: [mem 0x00000000-0xffffffffffffffff disabled]
pnp 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active)
pnp 00:0e: Plug and Play ACPI device, IDs PNP0a03 (active)
pnp: PnP ACPI: found 15 devices
ACPI: ACPI bus type pnp unregistered
system 00:01: [mem 0xfed1c000-0xfed1ffff] has been reserved
system 00:01: [mem 0xebfff000-0xebffffff] has been reserved
system 00:01: [mem 0xfed20000-0xfed3ffff] has been reserved
system 00:01: [mem 0xff000000-0xffffffff] could not be reserved
system 00:01: [mem 0xfee00000-0xfeefffff] could not be reserved
system 00:07: [io  0x0680-0x069f] has been reserved
system 00:07: [io  0x0400-0x0453] has been reserved
system 00:07: [io  0x0458-0x047f] has been reserved
system 00:07: [io  0x0500-0x057f] has been reserved
system 00:09: [io  0x0454-0x0457] has been reserved
pci 0000:03:00.0: no compatible bridge window for [mem 0xfffe0000-0xffffffff pref]
PCI: max bus depth: 2 pci_try_num: 3
pci 0000:00:02.0: BAR 15: assigned [mem 0x91400000-0x914fffff pref]
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0: PCI bridge to [bus 01-01]
pci 0000:00:01.0:   bridge window [io  disabled]
pci 0000:00:01.0:   bridge window [mem disabled]
pci 0000:00:01.0:   bridge window [mem pref disabled]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: BAR 7: assigned [mem 0x91248000-0x91267fff]
pci 0000:02:00.0: BAR 7: set to [mem 0x91248000-0x91267fff] (PCI address [0x91248000-0x91267fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.0: BAR 10: assigned [mem 0x91268000-0x91287fff]
pci 0000:02:00.0: BAR 10: set to [mem 0x91268000-0x91287fff] (PCI address [0x91268000-0x91287fff]
pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 184: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: BAR 7: assigned [mem 0x91288000-0x912a7fff]
pci 0000:02:00.3: BAR 7: set to [mem 0x91288000-0x912a7fff] (PCI address [0x91288000-0x912a7fff]
pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: reg 190: [mem 0x00000000-0x00003fff]
pci 0000:02:00.3: BAR 10: assigned [mem 0x912a8000-0x912c7fff]
pci 0000:02:00.3: BAR 10: set to [mem 0x912a8000-0x912c7fff] (PCI address [0x912a8000-0x912c7fff]
pci 0000:00:01.1: PCI bridge to [bus 02-02]
pci 0000:00:01.1: PCI bridge to [bus 02-02]
pci 0000:00:01.1:   bridge window [io  0x3000-0x3fff]
pci 0000:00:01.1:   bridge window [mem 0x91200000-0x912fffff]
pci 0000:00:01.1:   bridge window [mem pref disabled]
pci 0000:03:00.0: BAR 6: assigned [mem 0x91400000-0x9141ffff pref]
pci 0000:00:02.0: PCI bridge to [bus 03-03]
pci 0000:00:02.0: PCI bridge to [bus 03-03]
pci 0000:00:02.0:   bridge window [io  0x2000-0x2fff]
pci 0000:00:02.0:   bridge window [mem 0x91100000-0x911fffff]
pci 0000:00:02.0:   bridge window [mem 0x91400000-0x914fffff pref]
pci 0000:00:11.0: PCI bridge to [bus 04-04]
pci 0000:00:11.0: PCI bridge to [bus 04-04]
pci 0000:00:11.0:   bridge window [io  disabled]
pci 0000:00:11.0:   bridge window [mem disabled]
pci 0000:00:11.0:   bridge window [mem pref disabled]
pci 0000:05:00.0: PCI bridge to [bus 06-06]
pci 0000:05:00.0: PCI bridge to [bus 06-06]
pci 0000:05:00.0:   bridge window [io  0x1000-0x1fff]
pci 0000:05:00.0:   bridge window [mem 0x91000000-0x910fffff]
pci 0000:05:00.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
pci 0000:00:1c.0: PCI bridge to [bus 05-06]
pci 0000:00:1c.0: PCI bridge to [bus 05-06]
pci 0000:00:1c.0:   bridge window [io  0x1000-0x1fff]
pci 0000:00:1c.0:   bridge window [mem 0x91000000-0x910fffff]
pci 0000:00:1c.0:   bridge window [mem 0x90000000-0x90ffffff 64bit pref]
pci 0000:00:1e.0: PCI bridge to [bus 07-07]
pci 0000:00:1e.0: PCI bridge to [bus 07-07]
pci 0000:00:1e.0:   bridge window [io  disabled]
pci 0000:00:1e.0:   bridge window [mem disabled]
pci 0000:00:1e.0:   bridge window [mem pref disabled]
pci 0000:80:02.0: PCI bridge to [bus 81-81]
pci 0000:80:02.0: PCI bridge to [bus 81-81]
pci 0000:80:02.0:   bridge window [io  disabled]
pci 0000:80:02.0:   bridge window [mem disabled]
pci 0000:80:02.0:   bridge window [mem pref disabled]
pci 0000:80:02.2: PCI bridge to [bus 82-82]
pci 0000:80:02.2: PCI bridge to [bus 82-82]
pci 0000:80:02.2:   bridge window [io  disabled]
pci 0000:80:02.2:   bridge window [mem 0xee000000-0xee0fffff]
pci 0000:80:02.2:   bridge window [mem 0xec000000-0xedffffff 64bit pref]
  alloc irq_desc for 26 on node 0
  alloc kstat_irqs on node 0
pci 0000:00:01.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pci 0000:00:01.0: setting latency timer to 64
pci 0000:00:01.1: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pci 0000:00:01.1: setting latency timer to 64
  alloc irq_desc for 32 on node 0
  alloc kstat_irqs on node 0
pci 0000:00:02.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
pci 0000:00:02.0: setting latency timer to 64
  alloc irq_desc for 16 on node 0
  alloc kstat_irqs on node 0
pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:11.0: setting latency timer to 64
  alloc irq_desc for 17 on node 0
  alloc kstat_irqs on node 0
pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
pci 0000:00:1c.0: setting latency timer to 64
pci 0000:05:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:05:00.0: setting latency timer to 64
pci 0000:00:1e.0: setting latency timer to 64
  alloc irq_desc for 56 on node 1
  alloc kstat_irqs on node 1
pci 0000:80:02.0: PCI INT A -> GSI 56 (level, low) -> IRQ 56
pci 0000:80:02.0: setting latency timer to 64
pci 0000:80:02.2: PCI INT A -> GSI 56 (level, low) -> IRQ 56
pci 0000:80:02.2: setting latency timer to 64
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:00: resource 5 [io  0x1000-0xdfff]
pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:00: resource 7 [mem 0x90000000-0xebffefff]
pci_bus 0000:02: resource 0 [io  0x3000-0x3fff]
pci_bus 0000:02: resource 1 [mem 0x91200000-0x912fffff]
pci_bus 0000:03: resource 0 [io  0x2000-0x2fff]
pci_bus 0000:03: resource 1 [mem 0x91100000-0x911fffff]
pci_bus 0000:03: resource 2 [mem 0x91400000-0x914fffff pref]
pci_bus 0000:05: resource 0 [io  0x1000-0x1fff]
pci_bus 0000:05: resource 1 [mem 0x91000000-0x910fffff]
pci_bus 0000:05: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
pci_bus 0000:06: resource 0 [io  0x1000-0x1fff]
pci_bus 0000:06: resource 1 [mem 0x91000000-0x910fffff]
pci_bus 0000:06: resource 2 [mem 0x90000000-0x90ffffff 64bit pref]
pci_bus 0000:07: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:07: resource 5 [io  0x1000-0xdfff]
pci_bus 0000:07: resource 6 [mem 0x000a0000-0x000bffff]
pci_bus 0000:07: resource 7 [mem 0x90000000-0xebffefff]
pci_bus 0000:80: resource 4 [io  0x0000-0x0cf7]
pci_bus 0000:80: resource 5 [io  0xe000-0xffff]
pci_bus 0000:80: resource 6 [mem 0xec000000-0xfbffffff]
pci_bus 0000:82: resource 1 [mem 0xee000000-0xee0fffff]
pci_bus 0000:82: resource 2 [mem 0xec000000-0xedffffff 64bit pref]
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1a.0: PCI INT A disabled
  alloc irq_desc for 23 on node 0
  alloc kstat_irqs on node 0
pci 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
pci 0000:00:1d.0: PCI INT A disabled
pci 0000:06:00.0: Boot video device
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 55570k freed
Simple Boot Flag at 0x44 set to 0x1
audit: initializing netlink socket (disabled)
type=2000 audit(1372764995.403:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
SELinux:  Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key CE0DADC0C406F3A8
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert at redhat.com>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered (default)
io scheduler cfq registered
pcieport 0000:00:01.0: setting latency timer to 64
  alloc irq_desc for 77 on node 0
  alloc kstat_irqs on node 0
pcieport 0000:00:01.0: irq 77 for MSI/MSI-X
pcieport 0000:00:01.1: setting latency timer to 64
  alloc irq_desc for 78 on node 0
  alloc kstat_irqs on node 0
pcieport 0000:00:01.1: irq 78 for MSI/MSI-X
pcieport 0000:00:02.0: setting latency timer to 64
  alloc irq_desc for 79 on node 0
  alloc kstat_irqs on node 0
pcieport 0000:00:02.0: irq 79 for MSI/MSI-X
pcieport 0000:00:11.0: setting latency timer to 64
  alloc irq_desc for 80 on node 0
  alloc kstat_irqs on node 0
pcieport 0000:00:11.0: irq 80 for MSI/MSI-X
pcieport 0000:00:1c.0: setting latency timer to 64
  alloc irq_desc for 81 on node 0
  alloc kstat_irqs on node 0
pcieport 0000:00:1c.0: irq 81 for MSI/MSI-X
pcieport 0000:80:02.0: setting latency timer to 64
  alloc irq_desc for 82 on node 1
  alloc kstat_irqs on node 1
pcieport 0000:80:02.0: irq 82 for MSI/MSI-X
pcieport 0000:80:02.2: setting latency timer to 64
  alloc irq_desc for 83 on node 1
  alloc kstat_irqs on node 1
pcieport 0000:80:02.2: irq 83 for MSI/MSI-X
aer 0000:00:01.0:pcie02: service driver aer loaded
aer 0000:00:01.1:pcie02: service driver aer loaded
aer 0000:00:02.0:pcie02: service driver aer loaded
aer 0000:00:11.0:pcie02: service driver aer loaded
aer 0000:80:02.0:pcie02: service driver aer loaded
aer 0000:80:02.2:pcie02: service driver aer loaded
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
intel_idle: MWAIT substates: 0x21120
intel_idle: v0.4 model 0x2D
intel_idle: lapic_timer_reliable_states 0xffffffff
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
ACPI: acpi_idle yielding to intel_idle
ERST: Table is not found!
GHES: HEST is not enabled!
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
serial 00:0a: [io  0x03f8-0x03ff]
serial 00:0a: [irq 10]
serial 00:0a: activated
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ehci_hcd 0000:00:1a.0: setting latency timer to 64
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1a.0: irq 16, io mem 0x91326000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-279.19.1.el6_lustre.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.0: setting latency timer to 64
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1d.0: irq 23, io mem 0x91325000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-279.19.1.el6_lustre.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos 00:08: RTC can wake from S4
rtc_cmos 00:08: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 242 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:08: setting system clock to 2013-07-02 11:36:37 UTC (1372764997)
Initalizing network drop monitor service
Freeing unused kernel memory: 1256k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1052k freed
Freeing unused kernel memory: 1732k freed
dracut: dracut-004-283.el6
dracut: rd_NO_LUKS: removing cryptoluks activation
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.22.6-ioctl (2011-10-19) initialised: dm-devel at redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
dracut: rd_NO_DM: removing DM RAID activation
dracut: rd_NO_MD: removing MD RAID activation
megasas: 00.00.06.14-rh1 Fri. Jan. 6 17:00:00 PDT 2012
megasas: 0x1000:0x005b:0x1000:0x9265: bus 3:slot 0:func 0
megaraid_sas 0000:03:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
megaraid_sas 0000:03:00.0: setting latency timer to 64
megasas: FW now in Ready state
  alloc irq_desc for 84 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 84 for MSI/MSI-X
  alloc irq_desc for 85 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 85 for MSI/MSI-X
  alloc irq_desc for 86 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 86 for MSI/MSI-X
  alloc irq_desc for 87 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 87 for MSI/MSI-X
  alloc irq_desc for 88 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 88 for MSI/MSI-X
  alloc irq_desc for 89 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 89 for MSI/MSI-X
  alloc irq_desc for 90 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 90 for MSI/MSI-X
  alloc irq_desc for 91 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 91 for MSI/MSI-X
  alloc irq_desc for 92 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 92 for MSI/MSI-X
  alloc irq_desc for 93 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 93 for MSI/MSI-X
  alloc irq_desc for 94 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 94 for MSI/MSI-X
  alloc irq_desc for 95 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 95 for MSI/MSI-X
  alloc irq_desc for 96 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 96 for MSI/MSI-X
  alloc irq_desc for 97 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 97 for MSI/MSI-X
  alloc irq_desc for 98 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 98 for MSI/MSI-X
  alloc irq_desc for 99 on node 0
  alloc kstat_irqs on node 0
megaraid_sas 0000:03:00.0: irq 99 for MSI/MSI-X
megasas:IOC Init cmd success
megasas: INIT adapter done
usb 1-1: new high speed USB device number 2 using ehci_hcd
scsi0 : LSI SAS based MegaRAID driver
scsi 0:0:0:0: Direct-Access     LSI      MR9265-8i        3.14 PQ: 0 ANSI: 5
scsi 0:2:0:0: Direct-Access     LSI      MR9265-8i        3.14 PQ: 0 ANSI: 5
ahci 0000:00:1f.2: version 3.0
  alloc irq_desc for 19 on node 0
  alloc kstat_irqs on node 0
ahci 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
  alloc irq_desc for 100 on node 0
  alloc kstat_irqs on node 0
ahci 0000:00:1f.2: irq 100 for MSI/MSI-X
ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst 
ahci 0000:00:1f.2: setting latency timer to 64
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
scsi6 : ahci
ata1: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324100 irq 100
ata2: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324180 irq 100
ata3: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324200 irq 100
ata4: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324280 irq 100
ata5: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324300 irq 100
ata6: SATA max UDMA/133 abar m2048 at 0x91324000 port 0x91324380 irq 100
usb 1-1: New USB device found, idVendor=8087, idProduct=0024
usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1: configuration #1 chosen from 1 choice
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
usb 2-1: new high speed USB device number 2 using ehci_hcd
Refined TSC clocksource calibration: 2599.999 MHz.
Switching to clocksource tsc
usb 2-1: New USB device found, idVendor=8087, idProduct=0024
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
ata5: SATA link down (SStatus 0 SControl 300)
ata1: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 0 SControl 300)
ata2: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata6: SATA link down (SStatus 0 SControl 300)
usb 1-1.6: new high speed USB device number 3 using ehci_hcd
ACPI: WMI: Mapper loaded
usb 1-1.6: New USB device found, idVendor=0624, idProduct=0248
usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-1.6: Product: Gadget USB HUB
usb 1-1.6: Manufacturer: no manufacturer
usb 1-1.6: SerialNumber: 0123456789
usb 1-1.6: configuration #1 chosen from 1 choice
hub 1-1.6:1.0: USB hub found
hub 1-1.6:1.0: 5 ports detected
sd 0:2:0:0: [sda] 285671424 512-byte logical blocks: (146 GB/136 GiB)
sd 0:2:0:0: [sda] Write Protect is off
sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
sd 0:2:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2
sd 0:2:0:0: [sda] Attached SCSI disk
usb 1-1.6.1: new high speed USB device number 4 using ehci_hcd
dracut: Scanning devices sda2  for LVM logical volumes vg_system/lv_swap vg_system/lv_root 
dracut: inactive '/dev/vg_system/lv_root' [71.73 GiB] inherit
dracut: inactive '/dev/vg_system/lv_swap' [64.00 GiB] inherit
usb 1-1.6.1: New USB device found, idVendor=0624, idProduct=0249
usb 1-1.6.1: New USB device strings: Mfr=4, Product=5, SerialNumber=6
usb 1-1.6.1: Product: Keyboard/Mouse Function
usb 1-1.6.1: Manufacturer: Avocent
usb 1-1.6.1: SerialNumber: 20121018
usb 1-1.6.1: configuration #1 chosen from 1 choice
input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.0/input/input2
generic-usb 0003:0624:0249.0001: input,hidraw0: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input0
input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.1/input/input3
generic-usb 0003:0624:0249.0002: input,hidraw1: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input1
input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.2/input/input4
generic-usb 0003:0624:0249.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input2
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
dracut: Mounted root filesystem /dev/mapper/vg_system-lv_root
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
type=1404 audit(1372764999.065:2): selinux=0 auid=4294967295 ses=4294967295
dracut: 
dracut: Switching root
readahead: starting
udev: starting version 147
dca service started, version 1.12.1
igb: Intel(R) Gigabit Ethernet Network Driver - version 3.2.10-k
igb: Copyright (c) 2007-2011 Intel Corporation.
  alloc irq_desc for 30 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: PCI INT B -> GSI 30 (level, low) -> IRQ 30
igb 0000:02:00.0: setting latency timer to 64
  alloc irq_desc for 101 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 101 for MSI/MSI-X
  alloc irq_desc for 102 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 102 for MSI/MSI-X
  alloc irq_desc for 103 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 103 for MSI/MSI-X
  alloc irq_desc for 104 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 104 for MSI/MSI-X
  alloc irq_desc for 105 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 105 for MSI/MSI-X
  alloc irq_desc for 106 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 106 for MSI/MSI-X
  alloc irq_desc for 107 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 107 for MSI/MSI-X
  alloc irq_desc for 108 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 108 for MSI/MSI-X
  alloc irq_desc for 109 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.0: irq 109 for MSI/MSI-X
igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 78:45:c4:f9:7c:8a
igb 0000:02:00.0: eth0: PBA No: 105000-000
igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
  alloc irq_desc for 27 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: PCI INT A -> GSI 27 (level, low) -> IRQ 27
igb 0000:02:00.3: setting latency timer to 64
  alloc irq_desc for 110 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 110 for MSI/MSI-X
  alloc irq_desc for 111 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 111 for MSI/MSI-X
  alloc irq_desc for 112 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 112 for MSI/MSI-X
  alloc irq_desc for 113 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 113 for MSI/MSI-X
  alloc irq_desc for 114 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 114 for MSI/MSI-X
  alloc irq_desc for 115 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 115 for MSI/MSI-X
  alloc irq_desc for 116 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 116 for MSI/MSI-X
  alloc irq_desc for 117 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 117 for MSI/MSI-X
  alloc irq_desc for 118 on node 0
  alloc kstat_irqs on node 0
igb 0000:02:00.3: irq 118 for MSI/MSI-X
igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection
igb 0000:02:00.3: eth1: (PCIe:5.0Gb/s:Width x4) 78:45:c4:f9:7c:8b
igb 0000:02:00.3: eth1: PBA No: 105000-000
igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
ioatdma: Intel(R) QuickData Technology Driver 4.00
  alloc irq_desc for 31 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.0: PCI INT A -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.0: setting latency timer to 64
  alloc irq_desc for 119 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.0: irq 119 for MSI/MSI-X
igb 0000:02:00.0: DCA enabled
igb 0000:02:00.3: DCA enabled
  alloc irq_desc for 39 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.1: PCI INT B -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.1: setting latency timer to 64
  alloc irq_desc for 120 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.1: irq 120 for MSI/MSI-X
ioatdma 0000:00:04.2: PCI INT C -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.2: setting latency timer to 64
  alloc irq_desc for 121 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.2: irq 121 for MSI/MSI-X
ioatdma 0000:00:04.3: PCI INT D -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.3: setting latency timer to 64
  alloc irq_desc for 122 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.3: irq 122 for MSI/MSI-X
ioatdma 0000:00:04.4: PCI INT A -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.4: setting latency timer to 64
  alloc irq_desc for 123 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.4: irq 123 for MSI/MSI-X
ioatdma 0000:00:04.5: PCI INT B -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.5: setting latency timer to 64
  alloc irq_desc for 124 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.5: irq 124 for MSI/MSI-X
ioatdma 0000:00:04.6: PCI INT C -> GSI 31 (level, low) -> IRQ 31
ioatdma 0000:00:04.6: setting latency timer to 64
  alloc irq_desc for 125 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.6: irq 125 for MSI/MSI-X
ioatdma 0000:00:04.7: PCI INT D -> GSI 39 (level, low) -> IRQ 39
ioatdma 0000:00:04.7: setting latency timer to 64
  alloc irq_desc for 126 on node 0
  alloc kstat_irqs on node 0
ioatdma 0000:00:04.7: irq 126 for MSI/MSI-X
  alloc irq_desc for 55 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.0: PCI INT A -> GSI 55 (level, low) -> IRQ 55
ioatdma 0000:80:04.0: setting latency timer to 64
  alloc irq_desc for 127 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.0: irq 127 for MSI/MSI-X
  alloc irq_desc for 63 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.1: PCI INT B -> GSI 63 (level, low) -> IRQ 63
ioatdma 0000:80:04.1: setting latency timer to 64
  alloc irq_desc for 128 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.1: irq 128 for MSI/MSI-X
ioatdma 0000:80:04.2: PCI INT C -> GSI 55 (level, low) -> IRQ 55
ioatdma 0000:80:04.2: setting latency timer to 64
  alloc irq_desc for 129 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.2: irq 129 for MSI/MSI-X
ioatdma 0000:80:04.3: PCI INT D -> GSI 63 (level, low) -> IRQ 63
ioatdma 0000:80:04.3: setting latency timer to 64
  alloc irq_desc for 130 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.3: irq 130 for MSI/MSI-X
ioatdma 0000:80:04.4: PCI INT A -> GSI 55 (level, low) -> IRQ 55
ioatdma 0000:80:04.4: setting latency timer to 64
  alloc irq_desc for 131 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.4: irq 131 for MSI/MSI-X
ioatdma 0000:80:04.5: PCI INT B -> GSI 63 (level, low) -> IRQ 63
ioatdma 0000:80:04.5: setting latency timer to 64
  alloc irq_desc for 132 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.5: irq 132 for MSI/MSI-X
ioatdma 0000:80:04.6: PCI INT C -> GSI 55 (level, low) -> IRQ 55
ioatdma 0000:80:04.6: setting latency timer to 64
  alloc irq_desc for 133 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.6: irq 133 for MSI/MSI-X
ioatdma 0000:80:04.7: PCI INT D -> GSI 63 (level, low) -> IRQ 63
ioatdma 0000:80:04.7: setting latency timer to 64
  alloc irq_desc for 134 on node 1
  alloc kstat_irqs on node 1
ioatdma 0000:80:04.7: irq 134 for MSI/MSI-X
sd 0:2:0:0: Attached scsi generic sg0 type 0
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh
iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS
i801_smbus 0000:00:1f.3: PCI INT C -> GSI 19 (level, low) -> IRQ 19
ACPI: resource 0000:00:1f.3 [io  0x4000-0x401f] conflicts with ACPI region SMBI [io 0x4000-0x400f]
ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
EDAC MC: Ver: 2.1.0 Mar 20 2013
EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0
EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8
EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71
EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa
EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab
EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac
EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad
EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8
EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4
EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6
EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5
EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0
EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0
EDAC sbridge: Driver loaded.
mlx4_core: Mellanox ConnectX core driver v1.0-mlnx_ofed1.5.3 (November 3, 2011)
mlx4_core: Initializing 0000:82:00.0
  alloc irq_desc for 58 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: PCI INT A -> GSI 58 (level, low) -> IRQ 58
mlx4_core 0000:82:00.0: setting latency timer to 64
  alloc irq_desc for 135 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 135 for MSI/MSI-X
  alloc irq_desc for 136 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 136 for MSI/MSI-X
  alloc irq_desc for 137 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 137 for MSI/MSI-X
  alloc irq_desc for 138 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 138 for MSI/MSI-X
  alloc irq_desc for 139 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 139 for MSI/MSI-X
  alloc irq_desc for 140 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 140 for MSI/MSI-X
  alloc irq_desc for 141 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 141 for MSI/MSI-X
  alloc irq_desc for 142 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 142 for MSI/MSI-X
  alloc irq_desc for 143 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 143 for MSI/MSI-X
  alloc irq_desc for 144 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 144 for MSI/MSI-X
  alloc irq_desc for 145 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 145 for MSI/MSI-X
  alloc irq_desc for 146 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 146 for MSI/MSI-X
  alloc irq_desc for 147 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 147 for MSI/MSI-X
  alloc irq_desc for 148 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 148 for MSI/MSI-X
  alloc irq_desc for 149 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 149 for MSI/MSI-X
  alloc irq_desc for 150 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 150 for MSI/MSI-X
  alloc irq_desc for 151 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 151 for MSI/MSI-X
  alloc irq_desc for 152 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 152 for MSI/MSI-X
  alloc irq_desc for 153 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 153 for MSI/MSI-X
  alloc irq_desc for 154 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 154 for MSI/MSI-X
  alloc irq_desc for 155 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 155 for MSI/MSI-X
  alloc irq_desc for 156 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 156 for MSI/MSI-X
  alloc irq_desc for 157 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 157 for MSI/MSI-X
  alloc irq_desc for 158 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 158 for MSI/MSI-X
  alloc irq_desc for 159 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 159 for MSI/MSI-X
  alloc irq_desc for 160 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 160 for MSI/MSI-X
  alloc irq_desc for 161 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 161 for MSI/MSI-X
  alloc irq_desc for 162 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 162 for MSI/MSI-X
  alloc irq_desc for 163 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 163 for MSI/MSI-X
  alloc irq_desc for 164 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 164 for MSI/MSI-X
  alloc irq_desc for 165 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 165 for MSI/MSI-X
  alloc irq_desc for 166 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 166 for MSI/MSI-X
  alloc irq_desc for 167 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 167 for MSI/MSI-X
  alloc irq_desc for 168 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 168 for MSI/MSI-X
  alloc irq_desc for 169 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 169 for MSI/MSI-X
  alloc irq_desc for 170 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 170 for MSI/MSI-X
  alloc irq_desc for 171 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 171 for MSI/MSI-X
  alloc irq_desc for 172 on node 1
  alloc kstat_irqs on node 1
mlx4_core 0000:82:00.0: irq 172 for MSI/MSI-X
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.8.3 (June 2012)
microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU12 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU13 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU14 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
microcode: CPU15 sig=0x206d7, pf=0x1, revision=0x70d
platform microcode: firmware: requesting intel-ucode/06-2d-07
Microcode Update Driver: v2.00 <tigran at aivazian.fsnet.co.uk>, Peter Oruba
dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Adding 67108856k swap on /dev/mapper/vg_system-lv_swap.  Priority:-1 extents:1 across:67108856k 
mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-mlnx_ofed1.5.3 (November 3, 2011)
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Default coalesing params for mtu:2044 - rx_frames:88 rx_usecs:16
Default coalesing params for mtu:2044 - rx_frames:88 rx_usecs:16
802.1Q VLAN Support v1.8 Ben Greear <greearb at candelatech.com>
All bugs added by David S. Miller <davem at redhat.com>
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
bonding: MII link monitoring set to 100 ms
Loading kernel module for a network device with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias netdev-bond1 instead
bonding: bond1: doing slave updates when interface is down.
bonding: bond1: Adding slave ib0.
bonding bond1: master_dev is not up in bond_enslave
bonding: bond1: Warning: enslaved VLAN challenged slave ib0. Adding VLANs will be blocked as long as ib0 is part of bond bond1
bonding: bond1: Warning: The first slave device specified does not support setting the MAC address. Setting fail_over_mac to active.
ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
New slave device ib0 does not support netpoll
Disabling netpoll support for bond1
bonding: bond1: enslaving ib0 as a backup interface with a down link.
ib0: enabling connected mode will cause multicast packet drops
ib0: mtu > 2044 will cause multicast packet drops.
ib0: mtu > 2044 will cause multicast packet drops.
bonding: bond1: doing slave updates when interface is down.
bonding: bond1: Adding slave ib1.
bonding bond1: master_dev is not up in bond_enslave
bonding: bond1: Warning: enslaved VLAN challenged slave ib1. Adding VLANs will be blocked as long as ib1 is part of bond bond1
ib1: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
ib1: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
bonding: bond1: enslaving ib1 as a backup interface with a down link.
ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
ib1: enabling connected mode will cause multicast packet drops
ib1: mtu > 2044 will cause multicast packet drops.
ib1: mtu > 2044 will cause multicast packet drops.
ip_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ADDRCONF(NETDEV_UP): bond0: link is not ready
8021q: adding VLAN 0 to HW filter on device bond0
bonding: bond0: Adding slave eth0.
8021q: adding VLAN 0 to HW filter on device eth0
bonding: bond0: enslaving eth0 as a backup interface with a down link.
bonding: bond0: Adding slave eth1.
8021q: adding VLAN 0 to HW filter on device eth1
bonding: bond0: enslaving eth1 as a backup interface with a down link.
ib1: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth0, 1000 Mbps full duplex.
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth1, 1000 Mbps full duplex.
ADDRCONF(NETDEV_UP): bond1: link is not ready
8021q: adding VLAN 0 to HW filter on device bond1
ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
ib1: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Lustre: Lustre: Build Version: RC1-MLNX_OFED_LINUX_1.5.3_3.1.0_rhel6.3--PRISTINE-2.6.32-279.19.1.el6_lustre.x86_64
Lustre: Added LNI 192.168.127.106 at tcp [8/256/0/180]
Lustre: Accept secure, port 988
Lustre: Lustre OSC module (ffffffffa0a696a0).
Lustre: Lustre LOV module (ffffffffa0afba60).
Lustre: Lustre client module (ffffffffa0be5140).
bond0: no IPv6 routers present
Lustre: 2741:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1372736217/real 1372736220]  req at ffff88106393a000 x1439418250428417/t0(0) o250->MGC192.168.127.14 at tcp@192.168.127.14 at tcp:26/25 lens 368/512 e 0 to 1 dl 1372736222 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
LustreError: 2677:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired   req at ffff881064880000 x1439418250428418/t0(0) o101->MGC192.168.127.14 at tcp@192.168.127.14 at tcp:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
bonding: bond1: first active interface up!
bond1: link status definitely up for interface ib1, 4294967295 Mbps full duplex.
ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready
LustreError: 2763:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired   req at ffff881057fbc000 x1439418250428420/t0(0) o101->MGC192.168.127.14 at tcp@192.168.127.14 at tcp:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
bond1: no IPv6 routers present
LustreError: 2763:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired   req at ffff8810583c5000 x1439418250428421/t0(0) o101->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Lustre: 2741:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1372736242/real 1372736242]  req at ffff881057e8a800 x1439418250428422/t0(0) o250->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 368/512 e 0 to 1 dl 1372736247 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
LustreError: 2763:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired   req at ffff8810583c5000 x1439418250428423/t0(0) o101->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
LustreError: 2763:0:(client.c:1049:ptlrpc_import_delay_req()) @@@ send limit expired   req at ffff881064221000 x1439418250428424/t0(0) o101->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:W/0/ffffffff rc 0/-1
Lustre: MGC192.168.127.14 at tcp: Reactivating import
Lustre: Mounted modelfs-client
Lustre: Mounted preposfs-client
LustreError: 125620:0:(ldlm_request.c:1174:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
LustreError: 125620:0:(ldlm_request.c:1801:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
Lustre: Unmounted modelfs-client
LustreError: 125645:0:(ldlm_request.c:1174:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
LustreError: 125645:0:(ldlm_request.c:1174:ldlm_cli_cancel_req()) Skipped 103 previous similar messages
LustreError: 125645:0:(ldlm_request.c:1801:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
LustreError: 125645:0:(ldlm_request.c:1801:ldlm_cli_cancel_list()) Skipped 103 previous similar messages
Lustre: Unmounted preposfs-client
Slow work thread pool: Starting up
Slow work thread pool: Ready
FS-Cache: Loaded
Registering the id_resolver key type
FS-Cache: Netfs 'nfs' registered for caching
Lustre: Removed LNI 192.168.127.106 at tcp
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
xhpl_intel64[1022]: segfault at 61ac020 ip 00000000007eca81 sp 00007f4af0254d50 error 4 in xhpl_intel64[400000+618000]
xhpl_intel64[1030]: segfault at 5ef7048 ip 00000000007eca81 sp 00007fa1b9b38d50 error 4 in xhpl_intel64[400000+618000]
xhpl_intel64[1035]: segfault at 6099008 ip 00000000007eca81 sp 00007f5c7c293d50 error 4 in xhpl_intel64[400000+618000]
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
xhpl_intel64[2963]: segfault at 5b4b058 ip 00000000007eca81 sp 00007fe6f4b8cd50 error 4 in xhpl_intel64[400000+618000]
xhpl_intel64[2971]: segfault at 49ce028 ip 00000000007eca81 sp 00007f0416c90d50 error 4 in xhpl_intel64[400000+618000]
xhpl_intel64[2974]: segfault at 4d29048 ip 00000000007eca81 sp 00007f9d95e51d50 error 4
in xhpl_intel64[400000+618000]
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
reached max retry count. status=-22  .Giving up
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
reached max retry count. status=-22  .Giving up
bonding: bond1: link status definitely down for interface ib0, disabling it
bonding: bond1: making interface ib1 the new active one.
bond1: link status definitely up for interface ib0, 4294967295 Mbps full duplex.
bonding: bond1: making interface ib0 the new active one.
Lustre: Lustre: Build Version: RC1-MLNX_OFED_LINUX_1.5.3_3.1.0_rhel6.3--PRISTINE-2.6.32-279.19.1.el6_lustre.x86_64
Lustre: Added LNI 192.168.127.106 at tcp [8/256/0/180]
Lustre: Accept secure, port 988
Lustre: Lustre OSC module (ffffffffa0aae6a0).
Lustre: Lustre LOV module (ffffffffa0b40a60).
Lustre: Lustre client module (ffffffffa0c2a140).
Lustre: MGC192.168.127.14 at tcp: Reactivating import
Lustre: Mounted modelfs-client
Lustre: Mounted preposfs-client
INFO: task ungrib.exe:94999 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000000     0 94999  94977 0x00000080
ffff881036799b38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff88076e689600 ffff88086a2c7c00 ffff881036799b18 0000000000000282
ffff881065036638 ffff881036799fd8 000000000000fb88 ffff881065036638
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95001 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000001     0 95001  94977 0x00000080
ffff8810290e9b38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff880785e0a400 ffff88086a2c7c00 ffff8810290e9b18 0000000000000286
ffff8810657c8638 ffff8810290e9fd8 000000000000fb88 ffff8810657c8638
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95002 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 000000000000000a     0 95002  94977 0x00000080
ffff881057dd7b38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff88105c9ba000 ffff88086a2c7c00 ffff881057dd7b18 0000000000000286
ffff881063079ab8 ffff881057dd7fd8 000000000000fb88 ffff881063079ab8
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95004 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000009     0 95004  94977 0x00000080
ffff8810360e1b38 0000000000000086 0000000000000000 ffff88105c6ac588
ffff88105dd50a00 ffff88086a2c7c00 ffff8810360e1b18 0000000000000286
ffff881063ab3ab8 ffff8810360e1fd8 000000000000fb88 ffff881063ab3ab8
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95005 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 000000000000000d     0 95005  94977 0x00000080
ffff8810657c5b38 0000000000000086 0000000000000000 ffff88105c6ac588
ffff881047834400 ffff88086a2c7c00 ffff8810657c5b18 0000000000000282
ffff8810650125f8 ffff8810657c5fd8 000000000000fb88 ffff8810650125f8
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95006 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000004     0 95006  94977 0x00000080
ffff8810581f3b38 0000000000000086 0000000000000000 ffff88105c6ac588
ffff8807e191ac00 ffff88086a2c7c00 ffff8810581f3b18 0000000000000286
ffff8810654c7ab8 ffff8810581f3fd8 000000000000fb88 ffff8810654c7ab8
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95007 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000009     0 95007  94977 0x00000080
ffff8810564d7b38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff881027004e00 ffff88086a2c7c00 ffff8810564d7b18 0000000000000286
ffff8810654c7058 ffff8810564d7fd8 000000000000fb88 ffff8810654c7058
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95008 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000007     0 95008  94977 0x00000080
ffff88105b52db38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff880863e03c00 ffff88086a2c7c00 ffff88105b52db18 0000000000000282
ffff881064253ab8 ffff88105b52dfd8 000000000000fb88 ffff881064253ab8
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95009 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 000000000000000b     0 95009  94977 0x00000080
ffff881058a6fb38 0000000000000086 0000000000000000 ffff88105c6ac588
ffff88105cb25200 ffff88086a2c7c00 ffff881058a6fb18 0000000000000286
ffff8810641b0638 ffff881058a6ffd8 000000000000fb88 ffff8810641b0638
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff8100c535>] ? math_state_restore+0x45/0x60
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
INFO: task ungrib.exe:95010 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ungrib.exe    D 0000000000000006     0 95010  94977 0x00000080
ffff881058a61b38 0000000000000082 0000000000000000 ffff88105c6ac588
ffff88076e539400 ffff88086a2c7c00 ffff881058a61b18 0000000000000286
ffff8810641b1098 ffff881058a61fd8 000000000000fb88 ffff8810641b1098
Call Trace:
[<ffffffff814ec485>] rwsem_down_failed_common+0x95/0x1d0
[<ffffffff814ec5e3>] rwsem_down_write_failed+0x23/0x30
[<ffffffff812754e3>] call_rwsem_down_write_failed+0x13/0x20
[<ffffffff814ebae2>] ? down_write+0x32/0x40
[<ffffffffa0bc801c>] ll_setattr_raw+0x48c/0xf30 [lustre]
[<ffffffffa0bc8b1f>] ll_setattr+0x5f/0x100 [lustre]
[<ffffffff811923a8>] notify_change+0x168/0x340
[<ffffffff81174c74>] do_truncate+0x64/0xa0
[<ffffffff811873b9>] do_filp_open+0x829/0xd60
[<ffffffff81140540>] ? unmap_region+0x110/0x130
[<ffffffff81193282>] ? alloc_fd+0x92/0x160
[<ffffffff81173a39>] do_sys_open+0x69/0x140
[<ffffffff81173b50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
nfs: server avmmst1a not responding, still trying
nfs: server avmmst1a OK
ipmi message handler version 39.2
IPMI System Interface driver.
ipmi_si: Adding SMBIOS-specified kcs state machine
ipmi_si: Adding ACPI-specified kcs state machine: duplicate interface
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi: Found new BMC (man_id: 0x002b99,  prod_id: 0x0083, dev_id: 0x20)
IPMI kcs interface initialized
ipmi device interface
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373509742/real 1373509742]  req at ffff881055d9e400 x1439531544675803/t0(0) o400->MGC192.168.127.14 at tcp@192.168.127.14 at tcp:26/25 lens 192/192 e 0 to 1 dl 1373509749 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
LustreError: 166-1: MGC192.168.127.14 at tcp: Connection to MGS (at 192.168.127.14 at tcp) was lost; in progress operations using this service will fail
Lustre: modelfs-MDT0000-mdc-ffff8808641b8400: Connection to modelfs-MDT0000 (at 192.168.127.14 at tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373509749/real 1373509749]  req at ffff880f712d9000 x1439531544675818/t0(0) o250->MGC192.168.127.14 at tcp@192.168.127.14 at tcp:26/25 lens 368/512 e 0 to 1 dl 1373509755 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: Evicted from MGS (at 192.168.127.15 at tcp) after server handle changed from 0xbb9b8e06bf5d917c to 0x71c19737ef0868b4
Lustre: MGC192.168.127.14 at tcp: Reactivating import
Lustre: MGC192.168.127.14 at tcp: Connection restored to MGS (at 192.168.127.15 at tcp)
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373509774/real 1373509774]  req at ffff881065606800 x1439531544675822/t0(0) o38->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.15@tcp:12/10 lens 368/512 e 0 to 1 dl 1373509780 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373509799/real 1373509799]  req at ffff8810461a3000 x1439531544675848/t0(0) o38->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.14@tcp:12/10 lens 368/512 e 0 to 1 dl 1373509810 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 1 previous similar message
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req at ffff88105cf81400 x1439531543885079/t73014918113(73014918113) o101->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.15@tcp:12/10 lens 752/544 e 0 to 0 dl 1373509872 ref 2 fl Interpret:RP/4/0 rc 301/301
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req at ffff881036340400 x1439531543930893/t73015503064(73015503064) o101->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.15@tcp:12/10 lens 504/544 e 0 to 0 dl 1373509872 ref 2 fl Interpret:RP/4/0 rc 301/301
Lustre: modelfs-MDT0000-mdc-ffff8808641b8400: Connection restored to modelfs-MDT0000 (at 192.168.127.15 at tcp)
Lustre: preposfs-MDT0000-mdc-ffff880868cf8000: Connection restored to preposfs-MDT0000 (at 192.168.127.15 at tcp)
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373537099/real 1373537099]  req at ffff880447119400 x1439531544709159/t0(0) o400->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 192/192 e 0 to 1 dl 1373537106 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 1 previous similar message
LustreError: 166-1: MGC192.168.127.14 at tcp: Connection to MGS (at 192.168.127.15 at tcp) was lost; in progress operations using this service will fail
Lustre: modelfs-MDT0000-mdc-ffff8808641b8400: Connection to modelfs-MDT0000 (at 192.168.127.15 at tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1373537106/real 1373537106]  req at ffff88086917d000 x1439531544709174/t0(0) o250->MGC192.168.127.14 at tcp@192.168.127.15 at tcp:26/25 lens 368/512 e 0 to 1 dl 1373537112 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: Evicted from MGS (at 192.168.127.14 at tcp) after server handle changed from 0x71c19737ef0868b4 to 0xbb9b8e06cdc26fc6
Lustre: MGC192.168.127.14 at tcp: Reactivating import
Lustre: MGC192.168.127.14 at tcp: Connection restored to MGS (at 192.168.127.14 at tcp)
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) @@@ status 301, old was 0  req at ffff880869a3b400 x1439531544676036/t77309411388(77309411388) o101->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.14@tcp:12/10 lens 504/544 e 0 to 0 dl 1373537174 ref 2 fl Interpret:RP/4/0 rc 301/301
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) Skipped 51 previous similar messages
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) @@@ status -116, old was 0  req at ffff8810419f8c00 x1439531544708969/t77309662829(77309662829) o35->modelfs-MDT0000-mdc-ffff8808641b8400 at 192.168.127.14@tcp:23/10 lens 360/424 e 0 to 0 dl 1373537150 ref 2 fl Interpret:R/4/0 rc -116/-116
LustreError: 12891:0:(client.c:2631:ptlrpc_replay_interpret()) Skipped 111 previous similar messages
Lustre: modelfs-MDT0000-mdc-ffff8808641b8400: Connection restored to modelfs-MDT0000 (at 192.168.127.14 at tcp)
Lustre: preposfs-MDT0000-mdc-ffff880868cf8000: Connection restored to preposfs-MDT0000 (at 192.168.127.14 at tcp)
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
nfs: server avmmst1a not responding, timed out
nfs: server avmmst1a not responding, timed out
Error: state manager failed on NFSv4 server avmmst1a with error 5
sendrecv-blocki[19369]: segfault at 7fff9c4e0288 ip 0000000000405220 sp 00007fff9c4e0290 error 6 in sendrecv-blocking.x[400000+1da000]
sendrecv-blocki[19470]: segfault at 7fff47ae8308 ip 0000000000405220 sp 00007fff47ae8310 error 6 in sendrecv-blocking.x[400000+1da000]
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374141957/real 1374141957]  req at ffff88103bfc3800 x1439531545346894/t0(0) o400->modelfs-OST0001-osc-ffff8808641b8400 at 192.168.127.18@tcp:28/4 lens 192/192 e 0 to 1 dl 1374141964 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: modelfs-OST0001-osc-ffff8808641b8400: Connection to modelfs-OST0001 (at 192.168.127.18 at tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374141964/real 1374141964]  req at ffff88103e45d400 x1439531545346906/t0(0) o8->modelfs-OST0001-osc-ffff8808641b8400 at 192.168.127.18@tcp:28/4 lens 368/512 e 0 to 1 dl 1374141970 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
LustreError: 11-0: an error occurred while communicating with 192.168.127.17 at tcp. The ost_connect operation failed with -19
LustreError: 11-0: an error occurred while communicating with 192.168.127.17 at tcp. The ost_connect operation failed with -19
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374142014/real 1374142017]  req at ffff88102708e000 x1439531545346927/t0(0) o8->modelfs-OST0001-osc-ffff8808641b8400 at 192.168.127.18@tcp:28/4 lens 368/512 e 0 to 1 dl 1374142025 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
LustreError: 11-0: an error occurred while communicating with 192.168.127.17 at tcp. The ost_connect operation failed with -19
LustreError: Skipped 1 previous similar message
Lustre: preposfs-OST0000-osc-ffff880868cf8000: Connection restored to preposfs-OST0000 (at 192.168.127.17 at tcp)
Lustre: modelfs-OST0001-osc-ffff8808641b8400: Connection restored to modelfs-OST0001 (at 192.168.127.17 at tcp)
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374142064/real 1374142067]  req at ffff881039cde800 x1439531545347499/t0(0) o8->modelfs-OST0005-osc-ffff8808641b8400 at 192.168.127.18@tcp:28/4 lens 368/512 e 0 to 1 dl 1374142080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: modelfs-OST0005-osc-ffff8808641b8400: Connection restored to modelfs-OST0005 (at 192.168.127.17 at tcp)
Lustre: 12890:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281089/real 1374281089]  req at ffff88105be87400 x1439531545475212/t0(0) o400->modelfs-OST0000-osc-ffff8808641b8400 at 192.168.127.17@tcp:28/4 lens 192/192 e 0 to 1 dl 1374281096 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: modelfs-OST0000-osc-ffff8808641b8400: Connection to modelfs-OST0000 (at 192.168.127.17 at tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281096/real 1374281096]  req at ffff881051ba7c00 x1439531545475225/t0(0) o8->modelfs-OST0000-osc-ffff8808641b8400 at 192.168.127.17@tcp:28/4 lens 368/512 e 0 to 1 dl 1374281102 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: There was an unexpected network error while writing to 192.168.127.17: -110.
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281121/real 1374281121]  req at ffff88105838c800 x1439531545475233/t0(0) o8->modelfs-OST0000-osc-ffff8808641b8400 at 192.168.127.18@tcp:28/4 lens 368/512 e 0 to 1 dl 1374281127 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281146/real 1374281148]  req at ffff88103feb0800 x1439531545475248/t0(0) o8->modelfs-OST0000-osc-ffff8808641b8400 at 192.168.127.17@tcp:28/4 lens 368/512 e 0 to 1 dl 1374281157 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
LustreError: 11-0: an error occurred while communicating with 192.168.127.18 at tcp. The ost_connect operation failed with -19
Lustre: preposfs-OST0000-osc-ffff880868cf8000: Connection restored to preposfs-OST0000 (at 192.168.127.18 at tcp)
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281196/real 1374281196]  req at ffff88104cc68000 x1439531545475714/t0(0) o8->modelfs-OST0004-osc-ffff8808641b8400 at 192.168.127.17@tcp:28/4 lens 368/512 e 0 to 1 dl 1374281212 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Lustre: modelfs-OST0000-osc-ffff8808641b8400: Connection restored to modelfs-OST0000 (at 192.168.127.18 at tcp)
LustreError: 11-0: an error occurred while communicating with 192.168.127.18 at tcp. The ost_connect operation failed with -19
LustreError: Skipped 2 previous similar messages
Lustre: modelfs-OST0001-osc-ffff8808641b8400: Connection restored to modelfs-OST0001 (at 192.168.127.18 at tcp)
Lustre: modelfs-OST0005-osc-ffff8808641b8400: Connection restored to modelfs-OST0005 (at 192.168.127.18 at tcp)
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request  sent has timed out for slow reply: [sent 1374281246/real 1374281249]  req at ffff88105cbe0800 x1439531545476171/t0(0) o8->modelfs-OST0008-osc-ffff8808641b8400 at 192.168.127.17@tcp:28/4 lens 368/512 e 0 to 1 dl 1374281267 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Lustre: 12891:0:(client.c:1817:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
------------[ cut here ]------------
kernel BUG at mm/filemap.c:129!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:80/0000:80:02.2/0000:82:00.0/infiniband_mad/umad1/port
CPU 10 
Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables rdma_ucm(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) bonding 8021q garp stp llc ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U) dcdbas microcode mlx4_en(U) mlx4_core(U) sb_edac edac_core i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma igb dca ext4 mbcache jbd2 sd_mod crc_t10dif wmi ahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]

Pid: 12891, comm: ptlrpcd-rcv Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Dell Inc. PowerEdge C6220/0TTH1R
RIP: 0010:[<ffffffff81110c85>]  [<ffffffff81110c85>] __remove_from_page_cache+0xd5/0xe0
RSP: 0018:ffff880869ef17e0  EFLAGS: 00010046
RAX: 00c0000000000065 RBX: ffffea0030e5f6b0 RCX: 00000000ffffffd9
RDX: 0000000000000000 RSI: 0000000000000009 RDI: ffff880880010dc0
RBP: ffff880869ef17f0 R08: 000000000000005a R09: 0000000000000001
R10: 0000000000000357 R11: d000000000000000 R12: ffff881047e3e318
R13: ffff881047e3e330 R14: ffff8810370c5f50 R15: ffff8810450b9980
FS:  00007fd238319700(0000) GS:ffff88089c440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000033430abc30 CR3: 0000001050a99000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ptlrpcd-rcv (pid: 12891, threadinfo ffff880869ef0000, task ffff880863c8eaa0)
Stack:
ffffea0030e5f6b0 0000000000000000 ffff880869ef1820 ffffffff81110ce4
<d> ffffc9002cd0a000 ffffea0030e5f6b0 ffff8810370c5f50 ffff881047e3e318
<d> ffff880869ef1850 ffffffffa0bfa2d0 ffff8810450b98c0 ffff88046f3f8428
Call Trace:
[<ffffffff81110ce4>] remove_from_page_cache+0x54/0x90
[<ffffffffa0bfa2d0>] vvp_page_discard+0x70/0x130 [lustre]
[<ffffffffa061cb08>] cl_page_invoid+0x78/0x170 [obdclass]
[<ffffffffa061d058>] ? cl_page_invoke+0x158/0x1c0 [obdclass]
[<ffffffffa061cc13>] cl_page_discard+0x13/0x20 [obdclass]
[<ffffffffa0626b0e>] check_and_discard_cb+0x13e/0x160 [obdclass]
[<ffffffffa0621a34>] cl_page_gang_lookup+0x1d4/0x3e0 [obdclass]
[<ffffffffa06269d0>] ? check_and_discard_cb+0x0/0x160 [obdclass]
[<ffffffffa06235db>] cl_lock_page_out+0x15b/0x330 [obdclass]
[<ffffffffa06269d0>] ? check_and_discard_cb+0x0/0x160 [obdclass]
[<ffffffffa0a9a9cf>] osc_lock_flush+0x4f/0x90 [osc]
[<ffffffffa0a9aadf>] osc_lock_cancel+0xcf/0x1b0 [osc]
[<ffffffffa061c48d>] ? cl_env_nested_get+0x5d/0xc0 [obdclass]
[<ffffffffa0622265>] cl_lock_cancel0+0x75/0x160 [obdclass]
[<ffffffffa0622f4b>] cl_lock_cancel+0x13b/0x140 [obdclass]
[<ffffffffa0a9bbba>] osc_ldlm_blocking_ast+0x13a/0x380 [osc]
[<ffffffffa0729cc0>] ldlm_cancel_callback+0x60/0x100 [ptlrpc]
[<ffffffffa0729db5>] ldlm_lock_cancel+0x55/0x1b0 [ptlrpc]
[<ffffffffa07469d8>] ldlm_cli_cancel_list_local+0x78/0x1f0 [ptlrpc]
[<ffffffffa0747895>] ldlm_cancel_lru_local+0x35/0x40 [ptlrpc]
[<ffffffffa0747a0a>] ldlm_replay_locks+0x16a/0x6e0 [ptlrpc]
[<ffffffffa0790187>] ptlrpc_import_recovery_state_machine+0x8e7/0xc20 [ptlrpc]
[<ffffffffa0793288>] ptlrpc_connect_interpret+0x578/0x1f20 [ptlrpc]
[<ffffffffa0760710>] ? after_reply+0x730/0xe60 [ptlrpc]
[<ffffffffa0764cab>] ptlrpc_check_set+0x29b/0x1b00 [ptlrpc]
[<ffffffff814ead1a>] ? schedule_timeout+0x19a/0x2e0
[<ffffffff8107cb50>] ? process_timeout+0x0/0x10
[<ffffffffa0795f30>] ptlrpcd_check+0x1a0/0x230 [ptlrpc]
[<ffffffffa07962c3>] ptlrpcd+0x303/0x370 [ptlrpc]
[<ffffffff8105fa40>] ? default_wake_function+0x0/0x20
[<ffffffffa0795fc0>] ? ptlrpcd+0x0/0x370 [ptlrpc]
[<ffffffff8100c0ca>] child_rip+0xa/0x20
[<ffffffffa0795fc0>] ? ptlrpcd+0x0/0x370 [ptlrpc]
[<ffffffffa0795fc0>] ? ptlrpcd+0x0/0x370 [ptlrpc]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: 00 00 e8 2f c8 16 00 48 89 df 57 9d 0f 1f 44 00 00 5b 41 5c c9 c3 be 16 00 00 00 48 89 df e8 f3 1c 02 00 48 8b 03 e9 71 ff ff ff <0f> 0b eb fe 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 
RIP  [<ffffffff81110c85>] __remove_from_page_cache+0xd5/0xe0
RSP <ffff880869ef17e0>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20130720/ed8c071e/attachment.htm>


More information about the lustre-discuss mailing list