[rds-devel] FW: RDS -- hanging kernel

Tang, Changqing changquing.tang at hp.com
Fri Apr 9 09:40:35 PDT 2010


Tina,
        Eventually I get the vmcore for the hanging kernel. I run MPI pallas test over
RDS, two nodes, 5 rank on the first node, 6 ranks on the second node.

        After running a while, during MPI_Allreduce() test, the first node hangs. We
Managed to get the vmcore from this hanging node.

        It shows following:

crash> bt
PID: 9353   TASK: ffff810416f8c7b0  CPU: 1   COMMAND: "pmb.x.32"
 #0 [ffff81042ff33e68] die_nmi at ffffffff8006634d
 #1 [ffff81042ff33e80] die_nmi at ffffffff80066393
 #2 [ffff81042ff33ea0] nmi_watchdog_tick at ffffffff80066af9
 #3 [ffff81042ff33ef0] default_do_nmi at ffffffff80066717
 #4 [ffff81042ff33f40] do_nmi at ffffffff80066984
 #5 [ffff81042ff33f50] nmi at ffffffff80065fd7
    [exception RIP: .text.lock.spinlock+17]
    RIP: ffffffff80065cf3  RSP: ffff810407383af0  RFLAGS: 00000082
    RAX: 0000000000000246  RBX: ffff810405c21b80  RCX: 0000000000002000
    RDX: ffff810407383ed8  RSI: ffff810407383ed8  RDI: ffff810405c21df8
    RBP: 0000000000000000   R8: 0000000080000040   R9: 0000000000002000
    R10: 0000000000002000  R11: 00000000ffb2a618  R12: 0000000000000001
    R13: ffff810407383ed8  R14: ffff810407383ed8  R15: ffff810405c21df8
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <exception stack> ---
 #6 [ffff810407383af0] .text.lock.spinlock at ffffffff80065cf3 (via _spin_lock_i
rqsave)
 #7 [ffff810407383af0] rds_notify_queue_get at ffffffff8868c6a0
#8 [ffff810407383b60] rds_recvmsg at ffffffff8868cbb2
 #9 [ffff810407383c10] sock_recvmsg at ffffffff80030ab8
#10 [ffff810407383db0] sys_recvmsg at ffffffff8003d77f
#11 [ffff810407383f50] compat_sys_socketcall at ffffffff8022b44b
#12 [ffff810407383f80] cstar_do_call at ffffffff80062618

What else do I need to provide you?

--CQ




-----Original Message-----
From: Tina Yang [mailto:tina.yang at oracle.com]
Sent: Monday, April 05, 2010 3:38 PM
To: Tang, Changqing
Cc: Andy Grover; RDS Devel
Subject: Re: [rds-devel] FW: RDS -- how to detect peer is gone ?

Apr  4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.64.33 version 3.1
Apr  4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.80.240 version 3.1
Apr  4 08:18:40 sq0n32 syslogd 1.4.1: restart.
Apr  4 08:18:40 sq0n32 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Apr  4 08:18:40 sq0n32 kernel: Linux version 2.6.18-128.1.14.el5.8hp (root at stsdbo01.ind.hp.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Sep 9 03:25:05 EDT 2009

Ok, it doesn't look like an orderly system shutdown.
What's the value of 'sysctl kernel.panic' ?  Setting it to 0
should prevent reboot on panic to allow you the chance
to examine the console.  Or you can set up 'netconsole'
to capture the fault on a remote system.



Tang, Changqing wrote:
> Here is the file /var/log/messages, the system is running OK without MPI's heartbeat.
>
> Apr  4 07:42:54 sq0n32 syslogd 1.4.1: restart.
> Apr  4 07:42:54 sq0n32 kernel: klogd 1.4.1, log source = /proc/kmsg started.
> Apr  4 07:42:54 sq0n32 kernel: Linux version 2.6.18-128.1.14.el5.8hp (root at stsdbo01.ind.hp.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Sep 9 03:25:05 EDT 2009
> Apr  4 07:42:54 sq0n32 kernel: Command line: ro root=LABEL=/  console=ttyS0 clock=pmtmr pci=nommconf        nmi_watchdog=0 crashkernel=128M at 16M
> Apr  4 07:42:54 sq0n32 kernel: BIOS-provided physical RAM map:
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 0000000000100000 - 00000000cfe4e000 (usable)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000cfe4e000 - 00000000cfe56000 (ACPI data)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000cfe56000 - 00000000cfe57000 (usable)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000cfe57000 - 00000000e0000000 (reserved)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
> Apr  4 07:42:54 sq0n32 audispd: starting audispd
> Apr  4 07:42:54 sq0n32 kernel:  BIOS-e820: 0000000100000000 - 000000042ffff000 (usable)
> Apr  4 07:42:55 sq0n32 rpc.statd[3768]: Version 1.0.9 Starting
> Apr  4 07:42:55 sq0n32 kernel: DMI 2.4 present.
> Apr  4 07:42:55 sq0n32 rpc.statd[3768]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
> Apr  4 07:42:55 sq0n32 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
> Apr  4 07:42:55 sq0n32 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
> Apr  4 07:42:55 sq0n32 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
> Apr  4 07:42:55 sq0n32 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
> Apr  4 07:42:55 sq0n32 kernel: SRAT: Node 0 PXM 0 0-a0000
> Apr  4 07:42:55 sq0n32 kernel: SRAT: Node 0 PXM 0 0-d0000000
> Apr  4 07:42:55 sq0n32 kernel: SRAT: Node 0 PXM 0 0-430000000
> Apr  4 07:42:55 sq0n32 kernel: Bootmem setup node 0 0000000000000000-000000042ffff000
> Apr  4 07:42:55 sq0n32 kernel: ACPI: PM-Timer IO Port: 0x920
> Apr  4 07:42:55 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> Apr  4 07:42:55 sq0n32 kernel: Processor #0 0:2 APIC version 16
> Apr  4 07:42:55 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> Apr  4 07:42:55 sq0n32 kernel: Processor #1 0:2 APIC version 16
> Apr  4 07:42:55 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> Apr  4 07:42:55 sq0n32 kernel: Processor #2 0:2 APIC version 16
> Apr  4 07:42:55 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> Apr  4 07:42:55 sq0n32 kernel: Processor #3 0:2 APIC version 16
> Apr  4 07:42:55 sq0n32 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
> Apr  4 07:42:55 sq0n32 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
> Apr  4 07:42:55 sq0n32 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
> Apr  4 07:42:55 sq0n32 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
> Apr  4 07:42:55 sq0n32 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
> Apr  4 07:42:55 sq0n32 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
> Apr  4 07:42:55 sq0n32 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> Apr  4 07:42:55 sq0n32 kernel: Setting APIC routing to physical flat
> Apr  4 07:42:56 sq0n32 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
> Apr  4 07:42:56 sq0n32 kernel: Using ACPI (MADT) for SMP configuration information
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
> Apr  4 07:42:56 sq0n32 kdump: kexec: loaded kdump kernel
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
> Apr  4 07:42:56 sq0n32 kdump: started up
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000cfe4e000 - 00000000cfe56000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000cfe57000 - 00000000e0000000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
> Apr  4 07:42:56 sq0n32 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
> Apr  4 07:42:56 sq0n32 kernel: Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
> Apr  4 07:42:56 sq0n32 kernel: SMP: Allowing 4 CPUs, 0 hotplug CPUs
> Apr  4 07:42:56 sq0n32 kernel: Built 1 zonelists.  Total pages: 4132463
> Apr  4 07:42:56 sq0n32 kernel: Kernel command line: ro root=LABEL=/  console=ttyS0 clock=pmtmr pci=nommconf        nmi_watchdog=0 crashkernel=128M at 16M
> Apr  4 07:42:56 sq0n32 kernel: Warning: clock=pmtmr is deprecated. Use clocksource=acpi_pm.
> Apr  4 07:42:56 sq0n32 kernel: Initializing CPU#0
> Apr  4 07:42:56 sq0n32 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
> Apr  4 07:42:56 sq0n32 kernel: Console: colour VGA+ 80x25
> Apr  4 07:42:57 sq0n32 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
> Apr  4 07:42:57 sq0n32 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> Apr  4 07:42:57 sq0n32 kernel: Checking aperture...
> Apr  4 07:42:57 sq0n32 kernel: CPU 0: aperture @ 410a000000 size 32 MB
> Apr  4 07:42:57 sq0n32 pcscd: pcscdaemon.c:464:main() pcsc-lite 1.3.1 daemon ready.
> Apr  4 07:42:57 sq0n32 kernel: Aperture too small (32 MB)
> Apr  4 07:42:57 sq0n32 kernel: No AGP bridge found
> Apr  4 07:42:57 sq0n32 kernel: Your BIOS doesn't leave a aperture memory hole
> Apr  4 07:42:57 sq0n32 kernel: Please enable the IOMMU option in the BIOS setup
> Apr  4 07:42:57 sq0n32 kernel: This costs you 64 MB of RAM
> Apr  4 07:42:57 sq0n32 kernel: Mapping aperture over 65536 KB of RAM @ c000000
> Apr  4 07:42:57 sq0n32 kernel: Nosave address range: 000000000c000000 - 0000000010000000
> Apr  4 07:42:57 sq0n32 kernel: Memory: 16304976k/17563644k available (2501k kernel code, 470116k reserved, 1264k data, 200k init)
> Apr  4 07:42:57 sq0n32 kernel: Calibrating delay using timer specific routine.. 4403.51 BogoMIPS (lpj=2201755)
> Apr  4 07:42:57 sq0n32 kernel: Security Framework v1.0.0 initialized
> Apr  4 07:42:57 sq0n32 kernel: SELinux:  Initializing.
> Apr  4 07:42:57 sq0n32 kernel: selinux_register_security:  Registering secondary module capability
> Apr  4 07:42:57 sq0n32 kernel: Capability LSM initialized as secondary
> Apr  4 07:42:57 sq0n32 kernel: Mount-cache hash table entries: 256
> Apr  4 07:42:57 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 07:42:57 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 07:42:57 sq0n32 kernel: CPU 0/0 -> Node 0
> Apr  4 07:42:57 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 07:42:57 sq0n32 kernel: CPU: Processor Core ID: 0
> Apr  4 07:42:57 sq0n32 kernel: SMP alternatives: switching to UP code
> Apr  4 07:42:57 sq0n32 kernel: ACPI: Core revision 20060707
> Apr  4 07:42:57 sq0n32 kernel: Using local APIC timer interrupts.
> Apr  4 07:42:57 sq0n32 kernel: result 12500474
> Apr  4 07:42:57 sq0n32 kernel: Detected 12.500 MHz APIC timer.
> Apr  4 07:42:57 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 07:42:57 sq0n32 kernel: Booting processor 1/4 APIC 0x1
> Apr  4 07:42:57 sq0n32 kernel: Initializing CPU#1
> Apr  4 07:42:57 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.72 BogoMIPS (lpj=2200361)
> Apr  4 07:42:57 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 07:42:57 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 07:42:58 sq0n32 kernel: CPU 1/1 -> Node 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Processor Core ID: 1
> Apr  4 07:42:58 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 07:42:58 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 07:42:58 sq0n32 kernel: Booting processor 2/4 APIC 0x2
> Apr  4 07:42:58 sq0n32 kernel: Initializing CPU#2
> Apr  4 07:42:58 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.10 BogoMIPS (lpj=2200050)
> Apr  4 07:42:58 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 07:42:58 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 07:42:58 sq0n32 kernel: CPU 2/2 -> Node 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Processor Core ID: 2
> Apr  4 07:42:58 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 07:42:58 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 07:42:58 sq0n32 kernel: Booting processor 3/4 APIC 0x3
> Apr  4 07:42:58 sq0n32 kernel: Initializing CPU#3
> Apr  4 07:42:58 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.17 BogoMIPS (lpj=2200088)
> Apr  4 07:42:58 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 07:42:58 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 07:42:58 sq0n32 kernel: CPU 3/3 -> Node 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 07:42:58 sq0n32 kernel: CPU: Processor Core ID: 3
> Apr  4 07:42:58 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 07:42:58 sq0n32 kernel: Brought up 4 CPUs
> Apr  4 07:42:58 sq0n32 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
> Apr  4 07:42:58 sq0n32 kernel: time.c: Detected 2200.080 MHz processor.
> Apr  4 07:42:58 sq0n32 kernel: migration_cost=253
> Apr  4 07:42:58 sq0n32 kernel: checking if image is initramfs... it is
> Apr  4 07:42:58 sq0n32 kernel: Freeing initrd memory: 3120k freed
> Apr  4 07:42:58 sq0n32 kernel: NET: Registered protocol family 16
> Apr  4 07:42:58 sq0n32 kernel: No dock devices found.
> Apr  4 07:42:58 sq0n32 kernel: ACPI: bus type pci registered
> Apr  4 07:42:58 sq0n32 kernel: PCI: Using configuration type 1
> Apr  4 07:42:58 sq0n32 kernel: ACPI: Interpreter enabled
> Apr  4 07:42:58 sq0n32 kernel: ACPI: Using IOAPIC for interrupt routing
> Apr  4 07:42:58 sq0n32 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
> Apr  4 07:42:58 sq0n32 kernel: PCI: Enabling HT MSI Mapping on 0000:00:05.0
> Apr  4 07:42:58 sq0n32 kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:06.1
> Apr  4 07:42:58 sq0n32 kernel: ACPI: PCI Root Bridge [PCI1] (0000:40)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [ISF0] (IRQs 14) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 *10 11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 *10 11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 *11)
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled.
> Apr  4 07:42:59 sq0n32 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
> Apr  4 07:43:00 sq0n32 kernel: pnp: PnP ACPI init
> Apr  4 07:43:00 sq0n32 kernel: pnp: PnP ACPI: found 11 devices
> Apr  4 07:43:00 sq0n32 kernel: usbcore: registered new driver usbfs
> Apr  4 07:43:00 sq0n32 kernel: usbcore: registered new driver hub
> Apr  4 07:43:00 sq0n32 kernel: PCI: Using ACPI for IRQ routing
> Apr  4 07:43:00 sq0n32 kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
> Apr  4 07:43:00 sq0n32 kernel: NetLabel: Initializing
> Apr  4 07:43:00 sq0n32 kernel: NetLabel:  domain hash size = 128
> Apr  4 07:43:00 sq0n32 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
> Apr  4 07:43:00 sq0n32 kernel: NetLabel:  unlabeled traffic allowed by default
> Apr  4 07:43:00 sq0n32 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
> Apr  4 07:43:00 sq0n32 kernel: hpet0: 3 64-bit timers, 14318180 Hz
> Apr  4 07:43:00 sq0n32 kernel: PCI-DMA: Disabling AGP.
> Apr  4 07:43:00 sq0n32 kernel: PCI-DMA: aperture base @ c000000 size 65536 KB
> Apr  4 07:43:00 sq0n32 kernel: PCI-DMA: using GART IOMMU.
> Apr  4 07:43:00 sq0n32 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> Apr  4 07:43:00 sq0n32 kernel: pnp: 00:01: ioport range 0x379-0x37a has been reserved
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:01:0d.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:00:05.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:40:0f.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: fdf00000-fdffffff
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: f7800000-f7ffffff
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:40:10.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:41:00.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: f8000000-f9ffffff
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: e2100000-e21fffff
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:40:11.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:00 sq0n32 kernel:   MEM window: f8000000-f9ffffff
> Apr  4 07:43:00 sq0n32 kernel:   PREFETCH window: e2100000-e21fffff
> Apr  4 07:43:00 sq0n32 kernel: PCI: Bridge: 0000:43:00.0
> Apr  4 07:43:00 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:01 sq0n32 kernel:   MEM window: fa000000-fbffffff
> Apr  4 07:43:01 sq0n32 kernel:   PREFETCH window: e2200000-e22fffff
> Apr  4 07:43:01 sq0n32 kernel: PCI: Bridge: 0000:40:12.0
> Apr  4 07:43:01 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:01 sq0n32 kernel:   MEM window: fa000000-fbffffff
> Apr  4 07:43:01 sq0n32 kernel:   PREFETCH window: e2200000-e22fffff
> Apr  4 07:43:01 sq0n32 kernel: PCI: Bridge: 0000:46:04.0
> Apr  4 07:43:01 sq0n32 kernel:   IO window: disabled.
> Apr  4 07:43:01 sq0n32 kernel:   MEM window: disabled.
> Apr  4 07:43:01 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 07:43:01 sq0n32 kernel: PCI: Bridge: 0000:45:00.0
> Apr  4 07:43:01 sq0n32 kernel:   IO window: 4000-4fff
> Apr  4 07:43:01 sq0n32 kernel:   MEM window: fde00000-fdefffff
> Apr  4 07:43:01 sq0n32 kernel:   PREFETCH window: e2300000-e23fffff
> Apr  4 07:43:01 sq0n32 kernel: PCI: Bridge: 0000:40:13.0
> Apr  4 07:43:01 sq0n32 kernel:   IO window: 4000-4fff
> Apr  4 07:43:01 sq0n32 kernel:   MEM window: fde00000-fdefffff
> Apr  4 07:43:01 sq0n32 kernel:   PREFETCH window: e2300000-e23fffff
> Apr  4 07:43:01 sq0n32 kernel: GSI 16 sharing vector 0xA9 and IRQ 16
> Apr  4 07:43:01 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:0f.0[A] -> GSI 16 (level, low) -> IRQ 169
> Apr  4 07:43:01 sq0n32 kernel: GSI 17 sharing vector 0xB1 and IRQ 17
> Apr  4 07:43:01 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:10.0[A] -> GSI 20 (level, low) -> IRQ 177
> Apr  4 07:43:01 sq0n32 kernel: GSI 18 sharing vector 0xB9 and IRQ 18
> Apr  4 07:43:01 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:11.0[A] -> GSI 19 (level, low) -> IRQ 185
> Apr  4 07:43:01 sq0n32 kernel: GSI 19 sharing vector 0xC1 and IRQ 19
> Apr  4 07:43:01 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:12.0[A] -> GSI 28 (level, low) -> IRQ 193
> Apr  4 07:43:01 sq0n32 kernel: GSI 20 sharing vector 0xC9 and IRQ 20
> Apr  4 07:43:01 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:13.0[A] -> GSI 17 (level, low) -> IRQ 201
> Apr  4 07:43:01 sq0n32 kernel: NET: Registered protocol family 2
> Apr  4 07:43:01 sq0n32 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Apr  4 07:43:01 sq0n32 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
> Apr  4 07:43:01 sq0n32 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> Apr  4 07:43:01 sq0n32 kernel: TCP: Hash tables configured (established 262144 bind 65536)
> Apr  4 07:43:01 sq0n32 kernel: TCP reno registered
> Apr  4 07:43:01 sq0n32 kernel: audit: initializing netlink socket (disabled)
> Apr  4 07:43:01 sq0n32 kernel: type=2000 audit(1270366605.304:1): initialized
> Apr  4 07:43:01 sq0n32 kernel: Total HugeTLB memory allocated, 0
> Apr  4 07:43:01 sq0n32 kernel: VFS: Disk quotas dquot_6.5.1
> Apr  4 07:43:01 sq0n32 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> Apr  4 07:43:01 sq0n32 kernel: Initializing Cryptographic API
> Apr  4 07:43:01 sq0n32 kernel: alg: No test for crc32c (crc32c-generic)
> Apr  4 07:43:02 sq0n32 kernel: ksign: Installing public key data
> Apr  4 07:43:02 sq0n32 kernel: Loading keyring
> Apr  4 07:43:02 sq0n32 kernel: io scheduler noop registered
> Apr  4 07:43:02 sq0n32 kernel: io scheduler anticipatory registered
> Apr  4 07:43:02 sq0n32 kernel: io scheduler deadline registered
> Apr  4 07:43:02 sq0n32 kernel: io scheduler cfq registered (default)
> Apr  4 07:43:02 sq0n32 kernel: pci 0000:00:04.4: HCRESET not completed yet!
> Apr  4 07:43:02 sq0n32 kernel: assign_interrupt_mode Found MSI capability
> Apr  4 07:43:02 sq0n32 last message repeated 4 times
> Apr  4 07:43:02 sq0n32 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> Apr  4 07:43:02 sq0n32 kernel: Real Time Clock Driver v1.12ac
> Apr  4 07:43:02 sq0n32 kernel: Non-volatile memory driver v1.2
> Apr  4 07:43:02 sq0n32 kernel: Linux agpgart interface v0.101 (c) Dave Jones
> Apr  4 07:43:02 sq0n32 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
> Apr  4 07:43:02 sq0n32 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> Apr  4 07:43:02 sq0n32 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
> Apr  4 07:43:02 sq0n32 kernel: 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> Apr  4 07:43:02 sq0n32 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
> Apr  4 07:43:02 sq0n32 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> Apr  4 07:43:02 sq0n32 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> Apr  4 07:43:02 sq0n32 kernel: SvrWks HT1000: IDE controller at PCI slot 0000:00:06.1
> Apr  4 07:43:02 sq0n32 kernel: SvrWks HT1000: chipset revision 0
> Apr  4 07:43:02 sq0n32 kernel: SvrWks HT1000: not 100% native mode: will probe irqs later
> Apr  4 07:43:02 sq0n32 kernel:     ide0: BM-DMA at 0x0500-0x0507, BIOS settings: hda:DMA, hdb:pio
> Apr  4 07:43:02 sq0n32 kernel: hda: DV-28E-V, ATAPI CD/DVD-ROM drive
> Apr  4 07:43:02 sq0n32 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> Apr  4 07:43:02 sq0n32 kernel: ide-floppy driver 0.99.newide
> Apr  4 07:43:02 sq0n32 kernel: usbcore: registered new driver hiddev
> Apr  4 07:43:02 sq0n32 kernel: usbcore: registered new driver usbhid
> Apr  4 07:43:02 sq0n32 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
> Apr  4 07:43:02 sq0n32 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
> Apr  4 07:43:02 sq0n32 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
> Apr  4 07:43:02 sq0n32 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
> Apr  4 07:43:02 sq0n32 kernel: mice: PS/2 mouse device common for all mice
> Apr  4 07:43:02 sq0n32 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
> Apr  4 07:43:02 sq0n32 kernel: md: bitmap version 4.39
> Apr  4 07:43:02 sq0n32 kernel: TCP bic registered
> Apr  4 07:43:02 sq0n32 kernel: Initializing IPsec netlink socket
> Apr  4 07:43:02 sq0n32 kernel: NET: Registered protocol family 1
> Apr  4 07:43:02 sq0n32 kernel: NET: Registered protocol family 17
> Apr  4 07:43:02 sq0n32 kernel: ACPI: (supports S0 S4 S5)
> Apr  4 07:43:02 sq0n32 kernel: Freeing unused kernel memory: 200k freed
> Apr  4 07:43:03 sq0n32 kernel: Write protecting the kernel read-only data: 483k
> Apr  4 07:43:03 sq0n32 kernel: USB Universal Host Controller Interface driver v3.0
> Apr  4 07:43:03 sq0n32 kernel: GSI 21 sharing vector 0x42 and IRQ 21
> Apr  4 07:43:03 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.4[B] -> GSI 26 (level, low) -> IRQ 66
> Apr  4 07:43:03 sq0n32 kernel: uhci_hcd 0000:00:04.4: UHCI Host Controller
> Apr  4 07:43:03 sq0n32 kernel: uhci_hcd 0000:00:04.4: new USB bus registered, assigned bus number 1
> Apr  4 07:43:03 sq0n32 kernel: uhci_hcd 0000:00:04.4: port count misdetected? forcing to 2 ports
> Apr  4 07:43:03 sq0n32 kernel: uhci_hcd 0000:00:04.4: HCRESET not completed yet!
> Apr  4 07:43:03 sq0n32 kernel: uhci_hcd 0000:00:04.4: irq 66, io base 0x00001800
> Apr  4 07:43:03 sq0n32 kernel: usb usb1: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: hub 1-0:1.0: USB hub found
> Apr  4 07:43:03 sq0n32 kernel: hub 1-0:1.0: 2 ports detected
> Apr  4 07:43:03 sq0n32 kernel: ACPI: PCI Interrupt Link [IUSB] enabled at IRQ 5
> Apr  4 07:43:03 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.0: OHCI Host Controller
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.0: new USB bus registered, assigned bus number 2
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.0: irq 5, io mem 0xf76e0000
> Apr  4 07:43:03 sq0n32 kernel: usb 1-1: new full speed USB device using uhci_hcd and address 2
> Apr  4 07:43:03 sq0n32 kernel: usb 1-1: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: usb usb2: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: input: HP Virtual Keyboard as /class/input/input0
> Apr  4 07:43:03 sq0n32 kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:00:04.4-1
> Apr  4 07:43:03 sq0n32 kernel: input: HP Virtual Keyboard as /class/input/input1
> Apr  4 07:43:03 sq0n32 kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:00:04.4-1
> Apr  4 07:43:03 sq0n32 kernel: usb 1-2: new full speed USB device using uhci_hcd and address 3
> Apr  4 07:43:03 sq0n32 kernel: usb 1-2: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: hub 1-2:1.0: USB hub found
> Apr  4 07:43:03 sq0n32 kernel: hub 1-2:1.0: 7 ports detected
> Apr  4 07:43:03 sq0n32 kernel: hub 2-0:1.0: USB hub found
> Apr  4 07:43:03 sq0n32 kernel: hub 2-0:1.0: 2 ports detected
> Apr  4 07:43:03 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.1[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.1: OHCI Host Controller
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.1: new USB bus registered, assigned bus number 3
> Apr  4 07:43:03 sq0n32 kernel: ohci_hcd 0000:00:07.1: irq 5, io mem 0xf76d0000
> Apr  4 07:43:03 sq0n32 kernel: usb usb3: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: hub 3-0:1.0: USB hub found
> Apr  4 07:43:03 sq0n32 kernel: hub 3-0:1.0: 2 ports detected
> Apr  4 07:43:03 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.2[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 07:43:03 sq0n32 kernel: ehci_hcd 0000:00:07.2: EHCI Host Controller
> Apr  4 07:43:03 sq0n32 kernel: ehci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 4
> Apr  4 07:43:03 sq0n32 kernel: ehci_hcd 0000:00:07.2: irq 5, io mem 0xf76c0000
> Apr  4 07:43:03 sq0n32 kernel: ehci_hcd 0000:00:07.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> Apr  4 07:43:03 sq0n32 kernel: usb usb4: configuration #1 chosen from 1 choice
> Apr  4 07:43:03 sq0n32 kernel: hub 4-0:1.0: USB hub found
> Apr  4 07:43:03 sq0n32 kernel: hub 4-0:1.0: 4 ports detected
> Apr  4 07:43:04 sq0n32 kernel: SCSI subsystem initialized
> Apr  4 07:43:04 sq0n32 kernel: HP CISS Driver (v 3.6.20-RH2)
> Apr  4 07:43:04 sq0n32 kernel: ACPI: PCI Interrupt 0000:46:08.0[A] -> GSI 17 (level, low) -> IRQ 201
> Apr  4 07:43:04 sq0n32 kernel: cciss: requesting IRQ
> Apr  4 07:43:04 sq0n32 kernel: cciss0: <0x3238> at PCI 0000:46:08.0 IRQ 74 using DAC
> Apr  4 07:43:04 sq0n32 kernel: ohci_hcd 0000:00:07.1: wakeup
> Apr  4 07:43:04 sq0n32 kernel:       blocks= 286677120 block_size= 512
> Apr  4 07:43:04 sq0n32 kernel:       heads= 255, sectors= 32, cylinders= 35132
> Apr  4 07:43:04 sq0n32 kernel:
> Apr  4 07:43:04 sq0n32 kernel:       blocks= 286677120 block_size= 512
> Apr  4 07:43:04 sq0n32 kernel:       heads= 255, sectors= 32, cylinders= 35132
> Apr  4 07:43:04 sq0n32 kernel:
> Apr  4 07:43:04 sq0n32 kernel: usb 3-1: new full speed USB device using ohci_hcd and address 3
> Apr  4 07:43:04 sq0n32 kernel:  cciss/c0d0: p1 p2 p3 p4 < p5 >
> Apr  4 07:43:04 sq0n32 kernel: Fusion MPT base driver 3.04.07
> Apr  4 07:43:04 sq0n32 kernel: Copyright (c) 1999-2008 LSI Corporation
> Apr  4 07:43:04 sq0n32 kernel: usb 3-1: not running at top speed; connect to a high speed hub
> Apr  4 07:43:04 sq0n32 kernel: usb 3-1: configuration #1 chosen from 1 choice
> Apr  4 07:43:04 sq0n32 kernel: hub 3-1:1.0: USB hub found
> Apr  4 07:43:04 sq0n32 kernel: hub 3-1:1.0: 4 ports detected
> Apr  4 07:43:04 sq0n32 kernel: Fusion MPT SPI Host driver 3.04.07
> Apr  4 07:43:04 sq0n32 kernel: Fusion MPT FC Host driver 3.04.07
> Apr  4 07:43:04 sq0n32 kernel: Fusion MPT SAS Host driver 3.04.07
> Apr  4 07:43:04 sq0n32 kernel: Adaptec aacraid driver 1.1-5[2455]
> Apr  4 07:43:04 sq0n32 kernel: qla2xxx: Unknown parameter `ql2xfailover'
> Apr  4 07:43:04 sq0n32 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
> Apr  4 07:43:04 sq0n32 kernel: EXT3-fs: write access will be enabled during recovery.
> Apr  4 07:43:04 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 07:43:04 sq0n32 kernel: EXT3-fs: recovery complete.
> Apr  4 07:43:04 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 07:43:04 sq0n32 kernel: SELinux:  Disabled at runtime.
> Apr  4 07:43:04 sq0n32 kernel: type=1404 audit(1270366618.954:2): selinux=0 auid=4294967295 ses=4294967295
> Apr  4 07:43:04 sq0n32 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> Apr  4 07:43:04 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.2[B] -> GSI 26 (level, low) -> IRQ 66
> Apr  4 07:43:04 sq0n32 kernel: hda: ATAPI 24X DVD-ROM drive, 198kB Cache, UDMA(33)
> Apr  4 07:43:04 sq0n32 kernel: Uniform CD-ROM driver Revision: 3.20
> Apr  4 07:43:04 sq0n32 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.7.9-1 (July 18, 2008)
> Apr  4 07:43:04 sq0n32 kernel: ACPI: PCI Interrupt 0000:42:00.0[A] -> GSI 19 (level, low) -> IRQ 185
> Apr  4 07:43:04 sq0n32 kernel: input: PC Speaker as /class/input/input2
> Apr  4 07:43:04 sq0n32 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.1 (April 4, 2008)
> Apr  4 07:43:04 sq0n32 kernel: mlx4_core: Initializing 0000:4b:00.0
> Apr  4 07:43:04 sq0n32 kernel: ACPI: PCI Interrupt 0000:4b:00.0[A] -> GSI 16 (level, low) -> IRQ 169
> Apr  4 07:43:05 sq0n32 kernel: piix4_smbus 0000:00:06.0: Found 0000:00:06.0 device
> Apr  4 07:43:05 sq0n32 kernel: Floppy drive(s): fd0 is 1.44M
> Apr  4 07:43:05 sq0n32 kernel: eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 185, node addr 00215a4c8d58
> Apr  4 07:43:05 sq0n32 kernel: ACPI: PCI Interrupt 0000:44:00.0[A] -> GSI 28 (level, low) -> IRQ 193
> Apr  4 07:43:05 sq0n32 kernel: eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem fa000000, IRQ 193, node addr 00215a4c8d50
> Apr  4 07:43:05 sq0n32 kernel: floppy0: no floppy controllers found
> Apr  4 07:43:05 sq0n32 kernel: Floppy drive(s): fd0 is 1.44M
> Apr  4 07:43:05 sq0n32 kernel: floppy0: no floppy controllers found
> Apr  4 07:43:05 sq0n32 kernel: lp: driver loaded but no devices found
> Apr  4 07:43:05 sq0n32 kernel: ACPI: Power Button (FF) [PWRF]
> Apr  4 07:43:05 sq0n32 kernel: md: Autodetecting RAID arrays.
> Apr  4 07:43:05 sq0n32 kernel: md: autorun ...
> Apr  4 07:43:05 sq0n32 kernel: md: ... autorun DONE.
> Apr  4 07:43:05 sq0n32 kernel: device-mapper: uevent: version 1.0.3
> Apr  4 07:43:05 sq0n32 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel at redhat.com
> Apr  4 07:43:05 sq0n32 kernel: device-mapper: multipath: version 1.0.5 loaded
> Apr  4 07:43:05 sq0n32 kernel: EXT3 FS on cciss/c0d0p2, internal journal
> Apr  4 07:43:05 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 07:43:05 sq0n32 kernel: EXT3 FS on cciss/c0d0p1, internal journal
> Apr  4 07:43:05 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 07:43:05 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 07:43:05 sq0n32 kernel: EXT3 FS on cciss/c0d0p3, internal journal
> Apr  4 07:43:05 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 07:43:05 sq0n32 kernel: Adding 15675464k swap on /dev/cciss/c0d0p5.  Priority:-1 extents:1 across:15675464k
> Apr  4 07:43:05 sq0n32 kernel: NET: Registered protocol family 10
> Apr  4 07:43:05 sq0n32 kernel: lo: Disabled Privacy Extensions
> Apr  4 07:43:05 sq0n32 kernel: IPv6 over IPv4 tunneling driver
> Apr  4 07:43:05 sq0n32 kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.lo.retrans_time; Use net.ipv6.neigh.lo.retrans_time_ms instead.
> Apr  4 07:43:05 sq0n32 kernel: bnx2: eth0: using MSI
> Apr  4 07:43:05 sq0n32 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
> Apr  4 07:43:05 sq0n32 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
> Apr  4 07:43:05 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> Apr  4 07:43:05 sq0n32 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.1 (April 4, 2008)
> Apr  4 07:43:05 sq0n32 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.1 (Feb 2010)
> Apr  4 07:43:05 sq0n32 kernel: mlx4_en 0000:4b:00.0: UDP RSS is not supported on this device.
> Apr  4 07:43:05 sq0n32 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
> Apr  4 07:43:05 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
> Apr  4 07:43:05 sq0n32 kernel: ib0: enabling connected mode will cause multicast packet drops
> Apr  4 07:43:05 sq0n32 kernel: ib0: mtu > 2044 will cause multicast packet drops.
> Apr  4 07:43:05 sq0n32 kernel: ib0: mtu > 2044 will cause multicast packet drops.
> Apr  4 07:43:05 sq0n32 kernel: ib1: enabling connected mode will cause multicast packet drops
> Apr  4 07:43:05 sq0n32 kernel: ib1: mtu > 2044 will cause multicast packet drops.
> Apr  4 07:43:05 sq0n32 kernel: ib1: mtu > 2044 will cause multicast packet drops.
> Apr  4 07:43:05 sq0n32 kernel: NET: Registered protocol family 27
> Apr  4 07:43:05 sq0n32 kernel: NET: Registered protocol family 21
> Apr  4 07:43:06 sq0n32 kernel: Registered RDS/iwarp transport
> Apr  4 07:43:06 sq0n32 kernel: Registered RDS/infiniband transport
> Apr  4 07:43:06 sq0n32 kernel: Registered RDS/tcp transport
> Apr  4 07:43:06 sq0n32 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
> Apr  4 07:43:06 sq0n32 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
> Apr  4 07:43:06 sq0n32 kernel: Netfilter messages via NETLINK v0.30.
> Apr  4 07:43:06 sq0n32 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
> Apr  4 07:43:06 sq0n32 kernel: bnx2: eth0: using MSI
> Apr  4 07:43:06 sq0n32 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
> Apr  4 07:43:06 sq0n32 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
> Apr  4 07:43:06 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> Apr  4 07:43:06 sq0n32 kernel: RPC: Registered udp transport module.
> Apr  4 07:43:06 sq0n32 kernel: RPC: Registered tcp transport module.
> Apr  4 07:43:06 sq0n32 kernel: FS-Cache: Loaded
> Apr  4 07:43:06 sq0n32 kernel: FS-Cache: netfs 'nfs' registered for caching
> Apr  4 07:43:06 sq0n32 kernel: svc: failed to register lockdv1 RPC service (errno 97).
> Apr  4 07:48:02 sq0n32 MOND: Bound to socket 38154
> Apr  4 07:48:02 sq0n32 xinetd[4097]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
> Apr  4 07:48:02 sq0n32 xinetd[4097]: Started working: 2 available services
> Apr  4 07:48:03 sq0n32 cmsnmpd[4113]: Initializing
> Apr  4 07:48:03 sq0n32 ntpd[4126]: ntpd 4.2.2p1 at 1.1570-o Mon Feb 25 17:01:02 UTC 2008 (1)
> Apr  4 07:48:03 sq0n32 ntpd[4127]: precision = 1.000 usec
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface wildcard, 0.0.0.0#123 Disabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface wildcard, ::#123 Disabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface lo, ::1#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface ib0, fe80::216:35ff:ffbf:aba5#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface eth0, fe80::221:5aff:fe4c:8d58#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface lo, 127.0.0.1#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface eth0, 172.31.0.32#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: Listening on interface ib0, 172.31.64.32#123 Enabled
> Apr  4 07:48:03 sq0n32 ntpd[4127]: kernel time sync status 0040
> Apr  4 07:48:03 sq0n32 ntpd[4127]: frequency initialized 15.798 PPM from /var/lib/ntp/drift
> Apr  4 07:48:04 sq0n32 exportfs[4149]: can't open /var/lib/nfs/rmtab for reading
> Apr  4 07:48:04 sq0n32 kernel: Installing knfsd (copyright (C) 1996 okir at monad.swb.de).
> Apr  4 07:48:04 sq0n32 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> Apr  4 07:48:04 sq0n32 kernel: NFSD: unable to find recovery directory /var/lib/nfs/v4recovery
> Apr  4 07:48:04 sq0n32 kernel: NFSD: starting 90-second grace period
> Apr  4 07:48:04 sq0n32 xinetd[4097]: START: hacl-cfgudp pid=4188 from=127.0.0.1
> Apr  4 07:48:04 sq0n32 gpm[4217]: *** info [startup.c(95)]:
> Apr  4 07:48:04 sq0n32 gpm[4217]: Started gpm successfully. Entered daemon mode.
> Apr  4 07:48:05 sq0n32 cmclconfd[4188]: Unable to open /dev/deadman for reading to get time since boot: No such file or directory
> Apr  4 07:48:05 sq0n32 xinetd[4097]: EXIT: hacl-cfgudp status=1 pid=4188 duration=1(sec)
> Apr  4 07:48:06 sq0n32 kernel: ipmi message handler version 39.0
> Apr  4 07:48:06 sq0n32 kernel: ipmi message handler version 39.0
> Apr  4 07:48:06 sq0n32 kernel: ipmi device interface
> Apr  4 07:48:06 sq0n32 kernel: IPMI System Interface driver.
> Apr  4 07:48:06 sq0n32 kernel: ipmi_si: Trying SMBIOS-specified KCS state machine at I/O address 0xca2, slave address 0x20, irq 0
> Apr  4 07:48:06 sq0n32 kernel: ipmi: Found new BMC (man_id: 0x00000b,  prod_id: 0x0000, dev_id: 0x11)
> Apr  4 07:48:06 sq0n32 kernel:  IPMI KCS interface initialized
> Apr  4 07:48:06 sq0n32 kernel: ipmi_si: Trying ACPI-specified KCS state machine at I/O address 0xca2, slave address 0x0, irq 0
> Apr  4 07:48:06 sq0n32 kernel: ipmi_si: duplicate interface
> Apr  4 07:48:06 sq0n32 kernel: GSI 22 sharing vector 0x8A and IRQ 22
> Apr  4 07:48:06 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.6[A] -> GSI 25 (level, low) -> IRQ 138
> Apr  4 07:48:06 sq0n32 kernel: ipmi_si: Trying PCI-specified KCS state machine at memory address 0xf76f0000, slave address 0x0, irq 138
> Apr  4 07:48:06 sq0n32 kernel: ipmi_si: Using PCI Plug and Play interrupt 138
> Apr  4 07:48:06 sq0n32 kernel: ipmi: interfacing existing BMC (man_id: 0x00000b, prod_id: 0x0000, dev_id: 0x11)
> Apr  4 07:48:07 sq0n32 kernel:  IPMI KCS interface initialized
> Apr  4 07:48:21 sq0n32 kernel: Fusion MPT misc device (ioctl) driver 3.04.07
> Apr  4 07:48:21 sq0n32 kernel: mptctl: Registered with Fusion MPT base driver
> Apr  4 07:48:21 sq0n32 kernel: mptctl: /dev/mptctl @ (major,minor=10,220)
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: Successfully dropped root privileges.
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: avahi-daemon 0.6.16 starting up.
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: dbus_bus_get(): Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> Apr  4 07:48:24 sq0n32 avahi-daemon[4684]: WARNING: Failed to contact D-Bus daemon.
> Apr  4 07:48:26 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 33234280
> Apr  4 07:48:29 sq0n32 nrpe[4828]: Starting up daemon
> Apr  4 07:48:29 sq0n32 nrpe[4828]: Listening for connections on port 5666
> Apr  4 07:48:29 sq0n32 nrpe[4828]: Allowing connections from: sq0n0,sq0n31,172.31.0.32,127.0.0.1
> Apr  4 07:49:00 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 59896248
> Apr  4 07:49:00 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9887448
> Apr  4 07:49:04 sq0n32 cmsnmpd[4113]: Ready
> Apr  4 07:49:04 sq0n32 cmsnmpd[4113]: Ready
> Apr  4 07:49:10 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989792
> Apr  4 07:49:20 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989311
> Apr  4 07:49:30 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9988829
> Apr  4 07:49:40 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989356
> Apr  4 07:49:50 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989863
> Apr  4 07:50:00 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989380
> Apr  4 07:50:10 sq0n32 kernel: setitimer: collectl (pid = 4782) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9987892
> Apr  4 07:52:20 sq0n32 ntpd[4127]: synchronized to 172.31.80.240, stratum 4
> Apr  4 07:52:20 sq0n32 ntpd[4127]: kernel time sync enabled 0001
> Apr  4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.64.33 version 3.1
> Apr  4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.80.240 version 3.1
> Apr  4 08:18:40 sq0n32 syslogd 1.4.1: restart.
> Apr  4 08:18:40 sq0n32 kernel: klogd 1.4.1, log source = /proc/kmsg started.
> Apr  4 08:18:40 sq0n32 kernel: Linux version 2.6.18-128.1.14.el5.8hp (root at stsdbo01.ind.hp.com) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Wed Sep 9 03:25:05 EDT 2009
> Apr  4 08:18:40 sq0n32 kernel: Command line: ro root=LABEL=/  console=ttyS0 clock=pmtmr pci=nommconf        nmi_watchdog=0 crashkernel=128M at 16M
> Apr  4 08:18:40 sq0n32 kernel: BIOS-provided physical RAM map:
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 0000000000100000 - 00000000cfe4e000 (usable)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000cfe4e000 - 00000000cfe56000 (ACPI data)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000cfe56000 - 00000000cfe57000 (usable)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000cfe57000 - 00000000e0000000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
> Apr  4 08:18:40 sq0n32 kernel:  BIOS-e820: 0000000100000000 - 000000042ffff000 (usable)
> Apr  4 08:18:41 sq0n32 kernel: DMI 2.4 present.
> Apr  4 08:18:41 sq0n32 rpc.statd[3771]: Version 1.0.9 Starting
> Apr  4 08:18:41 sq0n32 audispd: starting audispd
> Apr  4 08:18:41 sq0n32 rpc.statd[3771]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
> Apr  4 08:18:41 sq0n32 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
> Apr  4 08:18:41 sq0n32 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
> Apr  4 08:18:41 sq0n32 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
> Apr  4 08:18:41 sq0n32 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
> Apr  4 08:18:41 sq0n32 kernel: SRAT: Node 0 PXM 0 0-a0000
> Apr  4 08:18:41 sq0n32 kernel: SRAT: Node 0 PXM 0 0-d0000000
> Apr  4 08:18:41 sq0n32 kernel: SRAT: Node 0 PXM 0 0-430000000
> Apr  4 08:18:41 sq0n32 kernel: Bootmem setup node 0 0000000000000000-000000042ffff000
> Apr  4 08:18:41 sq0n32 kernel: ACPI: PM-Timer IO Port: 0x920
> Apr  4 08:18:41 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> Apr  4 08:18:41 sq0n32 kernel: Processor #0 0:2 APIC version 16
> Apr  4 08:18:41 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> Apr  4 08:18:41 sq0n32 kernel: Processor #1 0:2 APIC version 16
> Apr  4 08:18:41 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> Apr  4 08:18:41 sq0n32 kernel: Processor #2 0:2 APIC version 16
> Apr  4 08:18:41 sq0n32 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> Apr  4 08:18:41 sq0n32 kernel: Processor #3 0:2 APIC version 16
> Apr  4 08:18:41 sq0n32 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
> Apr  4 08:18:41 sq0n32 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
> Apr  4 08:18:41 sq0n32 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
> Apr  4 08:18:41 sq0n32 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
> Apr  4 08:18:41 sq0n32 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
> Apr  4 08:18:41 sq0n32 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
> Apr  4 08:18:41 sq0n32 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> Apr  4 08:18:42 sq0n32 kernel: Setting APIC routing to physical flat
> Apr  4 08:18:42 sq0n32 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
> Apr  4 08:18:42 sq0n32 kernel: Using ACPI (MADT) for SMP configuration information
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
> Apr  4 08:18:42 sq0n32 kdump: kexec: loaded kdump kernel
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000cfe4e000 - 00000000cfe56000
> Apr  4 08:18:42 sq0n32 kdump: started up
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000cfe57000 - 00000000e0000000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
> Apr  4 08:18:42 sq0n32 kernel: Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
> Apr  4 08:18:42 sq0n32 kernel: SMP: Allowing 4 CPUs, 0 hotplug CPUs
> Apr  4 08:18:42 sq0n32 kernel: Built 1 zonelists.  Total pages: 4132463
> Apr  4 08:18:42 sq0n32 kernel: Kernel command line: ro root=LABEL=/  console=ttyS0 clock=pmtmr pci=nommconf        nmi_watchdog=0 crashkernel=128M at 16M
> Apr  4 08:18:42 sq0n32 kernel: Warning: clock=pmtmr is deprecated. Use clocksource=acpi_pm.
> Apr  4 08:18:42 sq0n32 kernel: Initializing CPU#0
> Apr  4 08:18:42 sq0n32 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
> Apr  4 08:18:42 sq0n32 kernel: Console: colour VGA+ 80x25
> Apr  4 08:18:42 sq0n32 kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
> Apr  4 08:18:42 sq0n32 kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> Apr  4 08:18:42 sq0n32 kernel: Checking aperture...
> Apr  4 08:18:42 sq0n32 kernel: CPU 0: aperture @ 10a000000 size 32 MB
> Apr  4 08:18:42 sq0n32 kernel: Aperture too small (32 MB)
> Apr  4 08:18:42 sq0n32 kernel: No AGP bridge found
> Apr  4 08:18:42 sq0n32 kernel: Your BIOS doesn't leave a aperture memory hole
> Apr  4 08:18:42 sq0n32 kernel: Please enable the IOMMU option in the BIOS setup
> Apr  4 08:18:42 sq0n32 kernel: This costs you 64 MB of RAM
> Apr  4 08:18:42 sq0n32 kernel: Mapping aperture over 65536 KB of RAM @ c000000
> Apr  4 08:18:42 sq0n32 kernel: Nosave address range: 000000000c000000 - 0000000010000000
> Apr  4 08:18:42 sq0n32 kernel: Memory: 16304976k/17563644k available (2501k kernel code, 470116k reserved, 1264k data, 200k init)
> Apr  4 08:18:42 sq0n32 kernel: Calibrating delay using timer specific routine.. 4403.51 BogoMIPS (lpj=2201757)
> Apr  4 08:18:42 sq0n32 kernel: Security Framework v1.0.0 initialized
> Apr  4 08:18:42 sq0n32 kernel: SELinux:  Initializing.
> Apr  4 08:18:42 sq0n32 kernel: selinux_register_security:  Registering secondary module capability
> Apr  4 08:18:42 sq0n32 kernel: Capability LSM initialized as secondary
> Apr  4 08:18:43 sq0n32 kernel: Mount-cache hash table entries: 256
> Apr  4 08:18:43 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU 0/0 -> Node 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Processor Core ID: 0
> Apr  4 08:18:43 sq0n32 kernel: SMP alternatives: switching to UP code
> Apr  4 08:18:43 sq0n32 kernel: ACPI: Core revision 20060707
> Apr  4 08:18:43 sq0n32 kernel: Using local APIC timer interrupts.
> Apr  4 08:18:43 sq0n32 kernel: result 12500465
> Apr  4 08:18:43 sq0n32 kernel: Detected 12.500 MHz APIC timer.
> Apr  4 08:18:43 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 08:18:43 sq0n32 kernel: Booting processor 1/4 APIC 0x1
> Apr  4 08:18:43 sq0n32 kernel: Initializing CPU#1
> Apr  4 08:18:43 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.64 BogoMIPS (lpj=2200320)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU 1/1 -> Node 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Processor Core ID: 1
> Apr  4 08:18:43 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 08:18:43 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 08:18:43 sq0n32 kernel: Booting processor 2/4 APIC 0x2
> Apr  4 08:18:43 sq0n32 kernel: Initializing CPU#2
> Apr  4 08:18:43 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.21 BogoMIPS (lpj=2200106)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU 2/2 -> Node 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Processor Core ID: 2
> Apr  4 08:18:43 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 08:18:43 sq0n32 kernel: SMP alternatives: switching to SMP code
> Apr  4 08:18:43 sq0n32 kernel: Booting processor 3/4 APIC 0x3
> Apr  4 08:18:43 sq0n32 kernel: Initializing CPU#3
> Apr  4 08:18:43 sq0n32 kernel: Calibrating delay using timer specific routine.. 4400.72 BogoMIPS (lpj=2200361)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU: L2 Cache: 512K (64 bytes/line)
> Apr  4 08:18:43 sq0n32 kernel: CPU 3/3 -> Node 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Physical Processor ID: 0
> Apr  4 08:18:43 sq0n32 kernel: CPU: Processor Core ID: 3
> Apr  4 08:18:43 sq0n32 kernel: Quad-Core AMD Opteron(tm) Processor 2354 stepping 03
> Apr  4 08:18:43 sq0n32 kernel: Brought up 4 CPUs
> Apr  4 08:18:43 sq0n32 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
> Apr  4 08:18:44 sq0n32 kernel: time.c: Detected 2200.081 MHz processor.
> Apr  4 08:18:44 sq0n32 kernel: migration_cost=257
> Apr  4 08:18:44 sq0n32 kernel: checking if image is initramfs... it is
> Apr  4 08:18:44 sq0n32 kernel: Freeing initrd memory: 3120k freed
> Apr  4 08:18:44 sq0n32 kernel: NET: Registered protocol family 16
> Apr  4 08:18:44 sq0n32 kernel: No dock devices found.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: bus type pci registered
> Apr  4 08:18:44 sq0n32 kernel: PCI: Using configuration type 1
> Apr  4 08:18:44 sq0n32 kernel: ACPI: Interpreter enabled
> Apr  4 08:18:44 sq0n32 kernel: ACPI: Using IOAPIC for interrupt routing
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
> Apr  4 08:18:44 sq0n32 kernel: PCI: Enabling HT MSI Mapping on 0000:00:05.0
> Apr  4 08:18:44 sq0n32 kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:06.1
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Root Bridge [PCI1] (0000:40)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [ISF0] (IRQs 14) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 *10 11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 *10 11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 *11)
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:44 sq0n32 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:45 sq0n32 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:45 sq0n32 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:45 sq0n32 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:45 sq0n32 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled.
> Apr  4 08:18:45 sq0n32 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
> Apr  4 08:18:45 sq0n32 kernel: pnp: PnP ACPI init
> Apr  4 08:18:45 sq0n32 kernel: pnp: PnP ACPI: found 11 devices
> Apr  4 08:18:45 sq0n32 kernel: usbcore: registered new driver usbfs
> Apr  4 08:18:45 sq0n32 kernel: usbcore: registered new driver hub
> Apr  4 08:18:45 sq0n32 kernel: PCI: Using ACPI for IRQ routing
> Apr  4 08:18:45 sq0n32 kernel: PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
> Apr  4 08:18:45 sq0n32 kernel: NetLabel: Initializing
> Apr  4 08:18:45 sq0n32 kernel: NetLabel:  domain hash size = 128
> Apr  4 08:18:45 sq0n32 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
> Apr  4 08:18:45 sq0n32 kernel: NetLabel:  unlabeled traffic allowed by default
> Apr  4 08:18:45 sq0n32 kernel: hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
> Apr  4 08:18:45 sq0n32 kernel: hpet0: 3 64-bit timers, 14318180 Hz
> Apr  4 08:18:45 sq0n32 kernel: PCI-DMA: Disabling AGP.
> Apr  4 08:18:45 sq0n32 kernel: PCI-DMA: aperture base @ c000000 size 65536 KB
> Apr  4 08:18:45 sq0n32 kernel: PCI-DMA: using GART IOMMU.
> Apr  4 08:18:45 sq0n32 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> Apr  4 08:18:46 sq0n32 kernel: pnp: 00:01: ioport range 0x379-0x37a has been reserved
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:01:0d.0
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:00:05.0
> Apr  4 08:18:46 sq0n32 pcscd: pcscdaemon.c:464:main() pcsc-lite 1.3.1 daemon ready.
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:40:0f.0
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: fdf00000-fdffffff
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: f7800000-f7ffffff
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:40:10.0
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:41:00.0
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: f8000000-f9ffffff
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: e2100000-e21fffff
> Apr  4 08:18:46 sq0n32 kernel: PCI: Bridge: 0000:40:11.0
> Apr  4 08:18:46 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:46 sq0n32 kernel:   MEM window: f8000000-f9ffffff
> Apr  4 08:18:46 sq0n32 kernel:   PREFETCH window: e2100000-e21fffff
> Apr  4 08:18:47 sq0n32 kernel: PCI: Bridge: 0000:43:00.0
> Apr  4 08:18:47 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:47 sq0n32 kernel:   MEM window: fa000000-fbffffff
> Apr  4 08:18:47 sq0n32 kernel:   PREFETCH window: e2200000-e22fffff
> Apr  4 08:18:47 sq0n32 kernel: PCI: Bridge: 0000:40:12.0
> Apr  4 08:18:47 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:47 sq0n32 kernel:   MEM window: fa000000-fbffffff
> Apr  4 08:18:47 sq0n32 kernel:   PREFETCH window: e2200000-e22fffff
> Apr  4 08:18:47 sq0n32 kernel: PCI: Bridge: 0000:46:04.0
> Apr  4 08:18:47 sq0n32 kernel:   IO window: disabled.
> Apr  4 08:18:47 sq0n32 kernel:   MEM window: disabled.
> Apr  4 08:18:47 sq0n32 kernel:   PREFETCH window: disabled.
> Apr  4 08:18:47 sq0n32 kernel: PCI: Bridge: 0000:45:00.0
> Apr  4 08:18:47 sq0n32 kernel:   IO window: 4000-4fff
> Apr  4 08:18:47 sq0n32 kernel:   MEM window: fde00000-fdefffff
> Apr  4 08:18:47 sq0n32 kernel:   PREFETCH window: e2300000-e23fffff
> Apr  4 08:18:47 sq0n32 kernel: PCI: Bridge: 0000:40:13.0
> Apr  4 08:18:47 sq0n32 kernel:   IO window: 4000-4fff
> Apr  4 08:18:47 sq0n32 kernel:   MEM window: fde00000-fdefffff
> Apr  4 08:18:47 sq0n32 kernel:   PREFETCH window: e2300000-e23fffff
> Apr  4 08:18:47 sq0n32 kernel: GSI 16 sharing vector 0xA9 and IRQ 16
> Apr  4 08:18:47 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:0f.0[A] -> GSI 16 (level, low) -> IRQ 169
> Apr  4 08:18:47 sq0n32 kernel: GSI 17 sharing vector 0xB1 and IRQ 17
> Apr  4 08:18:47 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:10.0[A] -> GSI 20 (level, low) -> IRQ 177
> Apr  4 08:18:47 sq0n32 kernel: GSI 18 sharing vector 0xB9 and IRQ 18
> Apr  4 08:18:47 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:11.0[A] -> GSI 19 (level, low) -> IRQ 185
> Apr  4 08:18:47 sq0n32 kernel: GSI 19 sharing vector 0xC1 and IRQ 19
> Apr  4 08:18:47 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:12.0[A] -> GSI 28 (level, low) -> IRQ 193
> Apr  4 08:18:47 sq0n32 kernel: GSI 20 sharing vector 0xC9 and IRQ 20
> Apr  4 08:18:47 sq0n32 kernel: ACPI: PCI Interrupt 0000:40:13.0[A] -> GSI 17 (level, low) -> IRQ 201
> Apr  4 08:18:47 sq0n32 kernel: NET: Registered protocol family 2
> Apr  4 08:18:47 sq0n32 kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Apr  4 08:18:47 sq0n32 kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
> Apr  4 08:18:47 sq0n32 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> Apr  4 08:18:47 sq0n32 kernel: TCP: Hash tables configured (established 262144 bind 65536)
> Apr  4 08:18:47 sq0n32 kernel: TCP reno registered
> Apr  4 08:18:47 sq0n32 kernel: audit: initializing netlink socket (disabled)
> Apr  4 08:18:47 sq0n32 kernel: type=2000 audit(1270368752.291:1): initialized
> Apr  4 08:18:47 sq0n32 kernel: Total HugeTLB memory allocated, 0
> Apr  4 08:18:47 sq0n32 kernel: VFS: Disk quotas dquot_6.5.1
> Apr  4 08:18:48 sq0n32 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> Apr  4 08:18:48 sq0n32 kernel: Initializing Cryptographic API
> Apr  4 08:18:48 sq0n32 kernel: alg: No test for crc32c (crc32c-generic)
> Apr  4 08:18:48 sq0n32 kernel: ksign: Installing public key data
> Apr  4 08:18:48 sq0n32 kernel: Loading keyring
> Apr  4 08:18:48 sq0n32 kernel: io scheduler noop registered
> Apr  4 08:18:48 sq0n32 kernel: io scheduler anticipatory registered
> Apr  4 08:18:48 sq0n32 kernel: io scheduler deadline registered
> Apr  4 08:18:48 sq0n32 kernel: io scheduler cfq registered (default)
> Apr  4 08:18:48 sq0n32 kernel: pci 0000:00:04.4: HCRESET not completed yet!
> Apr  4 08:18:48 sq0n32 kernel: assign_interrupt_mode Found MSI capability
> Apr  4 08:18:48 sq0n32 last message repeated 4 times
> Apr  4 08:18:48 sq0n32 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> Apr  4 08:18:48 sq0n32 kernel: Real Time Clock Driver v1.12ac
> Apr  4 08:18:48 sq0n32 kernel: Non-volatile memory driver v1.2
> Apr  4 08:18:48 sq0n32 kernel: Linux agpgart interface v0.101 (c) Dave Jones
> Apr  4 08:18:48 sq0n32 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
> Apr  4 08:18:48 sq0n32 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> Apr  4 08:18:48 sq0n32 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
> Apr  4 08:18:48 sq0n32 kernel: 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
> Apr  4 08:18:48 sq0n32 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
> Apr  4 08:18:48 sq0n32 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
> Apr  4 08:18:48 sq0n32 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
> Apr  4 08:18:48 sq0n32 kernel: SvrWks HT1000: IDE controller at PCI slot 0000:00:06.1
> Apr  4 08:18:48 sq0n32 kernel: SvrWks HT1000: chipset revision 0
> Apr  4 08:18:48 sq0n32 kernel: SvrWks HT1000: not 100% native mode: will probe irqs later
> Apr  4 08:18:48 sq0n32 kernel:     ide0: BM-DMA at 0x0500-0x0507, BIOS settings: hda:DMA, hdb:pio
> Apr  4 08:18:48 sq0n32 kernel: hda: DV-28E-V, ATAPI CD/DVD-ROM drive
> Apr  4 08:18:48 sq0n32 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> Apr  4 08:18:48 sq0n32 kernel: ide-floppy driver 0.99.newide
> Apr  4 08:18:48 sq0n32 kernel: usbcore: registered new driver hiddev
> Apr  4 08:18:48 sq0n32 kernel: usbcore: registered new driver usbhid
> Apr  4 08:18:48 sq0n32 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
> Apr  4 08:18:48 sq0n32 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
> Apr  4 08:18:48 sq0n32 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
> Apr  4 08:18:48 sq0n32 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
> Apr  4 08:18:48 sq0n32 kernel: mice: PS/2 mouse device common for all mice
> Apr  4 08:18:48 sq0n32 kernel: md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
> Apr  4 08:18:48 sq0n32 kernel: md: bitmap version 4.39
> Apr  4 08:18:48 sq0n32 kernel: TCP bic registered
> Apr  4 08:18:48 sq0n32 kernel: Initializing IPsec netlink socket
> Apr  4 08:18:48 sq0n32 kernel: NET: Registered protocol family 1
> Apr  4 08:18:48 sq0n32 kernel: NET: Registered protocol family 17
> Apr  4 08:18:48 sq0n32 kernel: ACPI: (supports S0 S4 S5)
> Apr  4 08:18:49 sq0n32 kernel: Freeing unused kernel memory: 200k freed
> Apr  4 08:18:49 sq0n32 kernel: Write protecting the kernel read-only data: 483k
> Apr  4 08:18:49 sq0n32 kernel: USB Universal Host Controller Interface driver v3.0
> Apr  4 08:18:49 sq0n32 kernel: GSI 21 sharing vector 0x42 and IRQ 21
> Apr  4 08:18:49 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.4[B] -> GSI 26 (level, low) -> IRQ 66
> Apr  4 08:18:49 sq0n32 kernel: uhci_hcd 0000:00:04.4: UHCI Host Controller
> Apr  4 08:18:49 sq0n32 kernel: uhci_hcd 0000:00:04.4: new USB bus registered, assigned bus number 1
> Apr  4 08:18:49 sq0n32 kernel: uhci_hcd 0000:00:04.4: port count misdetected? forcing to 2 ports
> Apr  4 08:18:49 sq0n32 kernel: uhci_hcd 0000:00:04.4: HCRESET not completed yet!
> Apr  4 08:18:49 sq0n32 kernel: uhci_hcd 0000:00:04.4: irq 66, io base 0x00001800
> Apr  4 08:18:49 sq0n32 kernel: usb usb1: configuration #1 chosen from 1 choice
> Apr  4 08:18:49 sq0n32 kernel: hub 1-0:1.0: USB hub found
> Apr  4 08:18:49 sq0n32 kernel: hub 1-0:1.0: 2 ports detected
> Apr  4 08:18:49 sq0n32 kernel: ACPI: PCI Interrupt Link [IUSB] enabled at IRQ 5
> Apr  4 08:18:49 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.0: OHCI Host Controller
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.0: new USB bus registered, assigned bus number 2
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.0: irq 5, io mem 0xf76e0000
> Apr  4 08:18:49 sq0n32 kernel: usb 1-1: new full speed USB device using uhci_hcd and address 2
> Apr  4 08:18:49 sq0n32 kernel: usb 1-1: configuration #1 chosen from 1 choice
> Apr  4 08:18:49 sq0n32 kernel: usb usb2: configuration #1 chosen from 1 choice
> Apr  4 08:18:49 sq0n32 kernel: input: HP Virtual Keyboard as /class/input/input0
> Apr  4 08:18:49 sq0n32 kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:00:04.4-1
> Apr  4 08:18:49 sq0n32 kernel: input: HP Virtual Keyboard as /class/input/input1
> Apr  4 08:18:49 sq0n32 kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:00:04.4-1
> Apr  4 08:18:49 sq0n32 kernel: usb 1-2: new full speed USB device using uhci_hcd and address 3
> Apr  4 08:18:49 sq0n32 kernel: usb 1-2: configuration #1 chosen from 1 choice
> Apr  4 08:18:49 sq0n32 kernel: hub 1-2:1.0: USB hub found
> Apr  4 08:18:49 sq0n32 kernel: hub 1-2:1.0: 7 ports detected
> Apr  4 08:18:49 sq0n32 kernel: hub 2-0:1.0: USB hub found
> Apr  4 08:18:49 sq0n32 kernel: hub 2-0:1.0: 2 ports detected
> Apr  4 08:18:49 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.1[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.1: OHCI Host Controller
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.1: new USB bus registered, assigned bus number 3
> Apr  4 08:18:49 sq0n32 kernel: ohci_hcd 0000:00:07.1: irq 5, io mem 0xf76d0000
> Apr  4 08:18:49 sq0n32 kernel: usb usb3: configuration #1 chosen from 1 choice
> Apr  4 08:18:49 sq0n32 kernel: hub 3-0:1.0: USB hub found
> Apr  4 08:18:49 sq0n32 kernel: hub 3-0:1.0: 2 ports detected
> Apr  4 08:18:49 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:07.2[A] -> Link [IUSB] -> GSI 5 (level, low) -> IRQ 5
> Apr  4 08:18:49 sq0n32 kernel: ehci_hcd 0000:00:07.2: EHCI Host Controller
> Apr  4 08:18:49 sq0n32 kernel: ehci_hcd 0000:00:07.2: new USB bus registered, assigned bus number 4
> Apr  4 08:18:49 sq0n32 kernel: ehci_hcd 0000:00:07.2: irq 5, io mem 0xf76c0000
> Apr  4 08:18:49 sq0n32 kernel: ehci_hcd 0000:00:07.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> Apr  4 08:18:50 sq0n32 kernel: usb usb4: configuration #1 chosen from 1 choice
> Apr  4 08:18:50 sq0n32 kernel: hub 4-0:1.0: USB hub found
> Apr  4 08:18:50 sq0n32 kernel: hub 4-0:1.0: 4 ports detected
> Apr  4 08:18:50 sq0n32 kernel: SCSI subsystem initialized
> Apr  4 08:18:50 sq0n32 kernel: HP CISS Driver (v 3.6.20-RH2)
> Apr  4 08:18:50 sq0n32 kernel: ACPI: PCI Interrupt 0000:46:08.0[A] -> GSI 17 (level, low) -> IRQ 201
> Apr  4 08:18:50 sq0n32 kernel: ohci_hcd 0000:00:07.1: wakeup
> Apr  4 08:18:50 sq0n32 kernel: cciss: requesting IRQ
> Apr  4 08:18:50 sq0n32 kernel: cciss0: <0x3238> at PCI 0000:46:08.0 IRQ 74 using DAC
> Apr  4 08:18:50 sq0n32 kernel:       blocks= 286677120 block_size= 512
> Apr  4 08:18:50 sq0n32 kernel:       heads= 255, sectors= 32, cylinders= 35132
> Apr  4 08:18:50 sq0n32 kernel:
> Apr  4 08:18:50 sq0n32 kernel:       blocks= 286677120 block_size= 512
> Apr  4 08:18:50 sq0n32 kernel: usb 3-1: new full speed USB device using ohci_hcd and address 3
> Apr  4 08:18:50 sq0n32 kernel:       heads= 255, sectors= 32, cylinders= 35132
> Apr  4 08:18:50 sq0n32 kernel:
> Apr  4 08:18:50 sq0n32 kernel:  cciss/c0d0: p1 p2 p3 p4 < p5 >
> Apr  4 08:18:50 sq0n32 kernel: Fusion MPT base driver 3.04.07
> Apr  4 08:18:50 sq0n32 kernel: Copyright (c) 1999-2008 LSI Corporation
> Apr  4 08:18:50 sq0n32 kernel: usb 3-1: not running at top speed; connect to a high speed hub
> Apr  4 08:18:50 sq0n32 kernel: usb 3-1: configuration #1 chosen from 1 choice
> Apr  4 08:18:50 sq0n32 kernel: hub 3-1:1.0: USB hub found
> Apr  4 08:18:50 sq0n32 kernel: hub 3-1:1.0: 4 ports detected
> Apr  4 08:18:50 sq0n32 kernel: Fusion MPT SPI Host driver 3.04.07
> Apr  4 08:18:50 sq0n32 kernel: Fusion MPT FC Host driver 3.04.07
> Apr  4 08:18:50 sq0n32 kernel: Fusion MPT SAS Host driver 3.04.07
> Apr  4 08:18:50 sq0n32 kernel: Adaptec aacraid driver 1.1-5[2455]
> Apr  4 08:18:50 sq0n32 kernel: qla2xxx: Unknown parameter `ql2xfailover'
> Apr  4 08:18:50 sq0n32 kernel: EXT3-fs: INFO: recovery required on readonly filesystem.
> Apr  4 08:18:50 sq0n32 kernel: EXT3-fs: write access will be enabled during recovery.
> Apr  4 08:18:50 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 08:18:50 sq0n32 kernel: EXT3-fs: recovery complete.
> Apr  4 08:18:50 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 08:18:50 sq0n32 kernel: SELinux:  Disabled at runtime.
> Apr  4 08:18:50 sq0n32 kernel: type=1404 audit(1270368765.600:2): selinux=0 auid=4294967295 ses=4294967295
> Apr  4 08:18:50 sq0n32 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
> Apr  4 08:18:50 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.2[B] -> GSI 26 (level, low) -> IRQ 66
> Apr  4 08:18:50 sq0n32 kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.7.9-1 (July 18, 2008)
> Apr  4 08:18:50 sq0n32 kernel: ACPI: PCI Interrupt 0000:42:00.0[A] -> GSI 19 (level, low) -> IRQ 185
> Apr  4 08:18:51 sq0n32 kernel: hda: ATAPI 24X DVD-ROM drive, 198kB Cache, UDMA(33)
> Apr  4 08:18:51 sq0n32 kernel: Uniform CD-ROM driver Revision: 3.20
> Apr  4 08:18:51 sq0n32 kernel: input: PC Speaker as /class/input/input2
> Apr  4 08:18:51 sq0n32 kernel: Floppy drive(s): fd0 is 1.44M
> Apr  4 08:18:51 sq0n32 kernel: piix4_smbus 0000:00:06.0: Found 0000:00:06.0 device
> Apr  4 08:18:51 sq0n32 kernel: eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 185, node addr 00215a4c8d58
> Apr  4 08:18:51 sq0n32 kernel: ACPI: PCI Interrupt 0000:44:00.0[A] -> GSI 28 (level, low) -> IRQ 193
> Apr  4 08:18:51 sq0n32 kernel: eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem fa000000, IRQ 193, node addr 00215a4c8d50
> Apr  4 08:18:51 sq0n32 kernel: mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.1 (April 4, 2008)
> Apr  4 08:18:51 sq0n32 kernel: mlx4_core: Initializing 0000:4b:00.0
> Apr  4 08:18:51 sq0n32 kernel: ACPI: PCI Interrupt 0000:4b:00.0[A] -> GSI 16 (level, low) -> IRQ 169
> Apr  4 08:18:51 sq0n32 kernel: floppy0: no floppy controllers found
> Apr  4 08:18:51 sq0n32 kernel: Floppy drive(s): fd0 is 1.44M
> Apr  4 08:18:51 sq0n32 kernel: floppy0: no floppy controllers found
> Apr  4 08:18:51 sq0n32 kernel: lp: driver loaded but no devices found
> Apr  4 08:18:51 sq0n32 kernel: ACPI: Power Button (FF) [PWRF]
> Apr  4 08:18:51 sq0n32 kernel: md: Autodetecting RAID arrays.
> Apr  4 08:18:51 sq0n32 kernel: md: autorun ...
> Apr  4 08:18:51 sq0n32 kernel: md: ... autorun DONE.
> Apr  4 08:18:51 sq0n32 kernel: device-mapper: uevent: version 1.0.3
> Apr  4 08:18:51 sq0n32 kernel: device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel at redhat.com
> Apr  4 08:18:51 sq0n32 kernel: device-mapper: multipath: version 1.0.5 loaded
> Apr  4 08:18:51 sq0n32 kernel: EXT3 FS on cciss/c0d0p2, internal journal
> Apr  4 08:18:51 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 08:18:51 sq0n32 kernel: EXT3 FS on cciss/c0d0p1, internal journal
> Apr  4 08:18:51 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 08:18:51 sq0n32 kernel: kjournald starting.  Commit interval 5 seconds
> Apr  4 08:18:51 sq0n32 kernel: EXT3 FS on cciss/c0d0p3, internal journal
> Apr  4 08:18:51 sq0n32 kernel: EXT3-fs: mounted filesystem with ordered data mode.
> Apr  4 08:18:51 sq0n32 kernel: Adding 15675464k swap on /dev/cciss/c0d0p5.  Priority:-1 extents:1 across:15675464k
> Apr  4 08:18:51 sq0n32 kernel: NET: Registered protocol family 10
> Apr  4 08:18:51 sq0n32 kernel: lo: Disabled Privacy Extensions
> Apr  4 08:18:51 sq0n32 kernel: IPv6 over IPv4 tunneling driver
> Apr  4 08:18:51 sq0n32 kernel: process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.lo.retrans_time; Use net.ipv6.neigh.lo.retrans_time_ms instead.
> Apr  4 08:18:51 sq0n32 kernel: bnx2: eth0: using MSI
> Apr  4 08:18:51 sq0n32 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
> Apr  4 08:18:51 sq0n32 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
> Apr  4 08:18:51 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> Apr  4 08:18:51 sq0n32 kernel: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0-ofed1.5.1 (April 4, 2008)
> Apr  4 08:18:51 sq0n32 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.1 (Feb 2010)
> Apr  4 08:18:51 sq0n32 kernel: mlx4_en 0000:4b:00.0: UDP RSS is not supported on this device.
> Apr  4 08:18:51 sq0n32 kernel: ADDRCONF(NETDEV_UP): ib0: link is not ready
> Apr  4 08:18:51 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
> Apr  4 08:18:51 sq0n32 kernel: ib0: enabling connected mode will cause multicast packet drops
> Apr  4 08:18:51 sq0n32 kernel: ib0: mtu > 2044 will cause multicast packet drops.
> Apr  4 08:18:52 sq0n32 kernel: ib0: mtu > 2044 will cause multicast packet drops.
> Apr  4 08:18:52 sq0n32 kernel: ib1: enabling connected mode will cause multicast packet drops
> Apr  4 08:18:52 sq0n32 kernel: ib1: mtu > 2044 will cause multicast packet drops.
> Apr  4 08:18:52 sq0n32 kernel: ib1: mtu > 2044 will cause multicast packet drops.
> Apr  4 08:18:52 sq0n32 kernel: NET: Registered protocol family 27
> Apr  4 08:18:52 sq0n32 kernel: NET: Registered protocol family 21
> Apr  4 08:18:52 sq0n32 kernel: Registered RDS/iwarp transport
> Apr  4 08:18:52 sq0n32 kernel: Registered RDS/infiniband transport
> Apr  4 08:18:52 sq0n32 kernel: Registered RDS/tcp transport
> Apr  4 08:18:52 sq0n32 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
> Apr  4 08:18:52 sq0n32 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
> Apr  4 08:18:52 sq0n32 kernel: Netfilter messages via NETLINK v0.30.
> Apr  4 08:18:52 sq0n32 kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) - 304 bytes per conntrack
> Apr  4 08:18:52 sq0n32 kernel: bnx2: eth0: using MSI
> Apr  4 08:18:52 sq0n32 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
> Apr  4 08:18:52 sq0n32 kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex
> Apr  4 08:18:52 sq0n32 kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> Apr  4 08:18:52 sq0n32 kernel: RPC: Registered udp transport module.
> Apr  4 08:18:52 sq0n32 kernel: RPC: Registered tcp transport module.
> Apr  4 08:18:52 sq0n32 kernel: FS-Cache: Loaded
> Apr  4 08:18:52 sq0n32 kernel: FS-Cache: netfs 'nfs' registered for caching
> Apr  4 08:18:52 sq0n32 kernel: svc: failed to register lockdv1 RPC service (errno 97).
> Apr  4 08:23:51 sq0n32 MOND: Bound to socket 38154
> Apr  4 08:23:51 sq0n32 xinetd[4100]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
> Apr  4 08:23:51 sq0n32 xinetd[4100]: Started working: 2 available services
> Apr  4 08:23:53 sq0n32 cmsnmpd[4116]: Initializing
> Apr  4 08:23:53 sq0n32 ntpd[4129]: ntpd 4.2.2p1 at 1.1570-o Mon Feb 25 17:01:02 UTC 2008 (1)
> Apr  4 08:23:53 sq0n32 ntpd[4130]: precision = 1.000 usec
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface wildcard, 0.0.0.0#123 Disabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface wildcard, ::#123 Disabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface lo, ::1#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface ib0, fe80::216:35ff:ffbf:aba5#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface eth0, fe80::221:5aff:fe4c:8d58#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface lo, 127.0.0.1#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface eth0, 172.31.0.32#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: Listening on interface ib0, 172.31.64.32#123 Enabled
> Apr  4 08:23:53 sq0n32 ntpd[4130]: kernel time sync status 0040
> Apr  4 08:23:53 sq0n32 ntpd[4130]: frequency initialized 15.798 PPM from /var/lib/ntp/drift
> Apr  4 08:23:53 sq0n32 exportfs[4152]: can't open /var/lib/nfs/rmtab for reading
> Apr  4 08:23:53 sq0n32 kernel: Installing knfsd (copyright (C) 1996 okir at monad.swb.de).
> Apr  4 08:23:53 sq0n32 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> Apr  4 08:23:54 sq0n32 kernel: NFSD: unable to find recovery directory /var/lib/nfs/v4recovery
> Apr  4 08:23:54 sq0n32 kernel: NFSD: starting 90-second grace period
> Apr  4 08:23:54 sq0n32 xinetd[4100]: START: hacl-cfgudp pid=4200 from=127.0.0.1
> Apr  4 08:23:54 sq0n32 gpm[4221]: *** info [startup.c(95)]:
> Apr  4 08:23:54 sq0n32 gpm[4221]: Started gpm successfully. Entered daemon mode.
> Apr  4 08:23:54 sq0n32 cmclconfd[4200]: Unable to open /dev/deadman for reading to get time since boot: No such file or directory
> Apr  4 08:23:54 sq0n32 xinetd[4100]: EXIT: hacl-cfgudp status=1 pid=4200 duration=0(sec)
> Apr  4 08:23:55 sq0n32 kernel: ipmi message handler version 39.0
> Apr  4 08:23:56 sq0n32 kernel: ipmi message handler version 39.0
> Apr  4 08:23:56 sq0n32 kernel: ipmi device interface
> Apr  4 08:23:56 sq0n32 kernel: IPMI System Interface driver.
> Apr  4 08:23:56 sq0n32 kernel: ipmi_si: Trying SMBIOS-specified KCS state machine at I/O address 0xca2, slave address 0x20, irq 0
> Apr  4 08:23:56 sq0n32 kernel: ipmi: Found new BMC (man_id: 0x00000b,  prod_id: 0x0000, dev_id: 0x11)
> Apr  4 08:23:56 sq0n32 kernel:  IPMI KCS interface initialized
> Apr  4 08:23:56 sq0n32 kernel: ipmi_si: Trying ACPI-specified KCS state machine at I/O address 0xca2, slave address 0x0, irq 0
> Apr  4 08:23:56 sq0n32 kernel: ipmi_si: duplicate interface
> Apr  4 08:23:56 sq0n32 kernel: GSI 22 sharing vector 0x8A and IRQ 22
> Apr  4 08:23:56 sq0n32 kernel: ACPI: PCI Interrupt 0000:00:04.6[A] -> GSI 25 (level, low) -> IRQ 138
> Apr  4 08:23:56 sq0n32 kernel: ipmi_si: Trying PCI-specified KCS state machine at memory address 0xf76f0000, slave address 0x0, irq 138
> Apr  4 08:23:56 sq0n32 kernel: ipmi_si: Using PCI Plug and Play interrupt 138
> Apr  4 08:23:56 sq0n32 kernel: ipmi: interfacing existing BMC (man_id: 0x00000b, prod_id: 0x0000, dev_id: 0x11)
> Apr  4 08:23:56 sq0n32 kernel:  IPMI KCS interface initialized
> Apr  4 08:24:11 sq0n32 kernel: Fusion MPT misc device (ioctl) driver 3.04.07
> Apr  4 08:24:11 sq0n32 kernel: mptctl: Registered with Fusion MPT base driver
> Apr  4 08:24:11 sq0n32 kernel: mptctl: /dev/mptctl @ (major,minor=10,220)
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: Successfully dropped root privileges.
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: avahi-daemon 0.6.16 starting up.
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: dbus_bus_get(): Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
> Apr  4 08:24:13 sq0n32 avahi-daemon[4687]: WARNING: Failed to contact D-Bus daemon.
> Apr  4 08:24:16 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 43120287
> Apr  4 08:24:19 sq0n32 nrpe[4830]: Starting up daemon
> Apr  4 08:24:19 sq0n32 nrpe[4830]: Listening for connections on port 5666
> Apr  4 08:24:19 sq0n32 nrpe[4830]: Allowing connections from: sq0n0,sq0n31,172.31.0.32,127.0.0.1
> Apr  4 08:24:54 sq0n32 cmsnmpd[4116]: Ready
> Apr  4 08:24:54 sq0n32 cmsnmpd[4116]: Ready
> Apr  4 08:25:00 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 59896517
> Apr  4 08:25:00 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9887871
> Apr  4 08:25:10 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989078
> Apr  4 08:25:20 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9988589
> Apr  4 08:25:30 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9990115
> Apr  4 08:25:40 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989618
> Apr  4 08:25:50 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989146
> Apr  4 08:26:00 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989651
> Apr  4 08:26:10 sq0n32 kernel: setitimer: collectl (pid = 4784) provided invalid timeval it_value: tv_sec = 0 tv_usec = 9989158
> Apr  4 08:28:09 sq0n32 ntpd[4130]: synchronized to 172.31.80.240, stratum 4
> Apr  4 08:28:09 sq0n32 ntpd[4130]: kernel time sync enabled 0001
> Apr  4 20:40:25 sq0n32 kernel: RDS/IB: connected to 172.31.80.240 version 3.1
> Apr  4 20:40:25 sq0n32 kernel: RDS/IB: connected to 172.31.64.33 version 3.1
> Apr  4 21:19:52 sq0n32 kernel: RDS/IB: connected to 172.31.64.33 version 3.1
> Apr  5 06:53:08 sq0n32 kernel: RDS/IB: recv completion on 172.31.80.240 had status 4, disconnecting and reconnecting
> Apr  5 06:53:08 sq0n32 kernel: RDS/IB: connected to 172.31.80.240 version 3.1
> Apr  5 07:01:32 sq0n32 kernel: RDS/IB: recv completion on 172.31.80.240 had status 4, disconnecting and reconnecting
> Apr  5 07:01:32 sq0n32 kernel: RDS/IB: connected to 172.31.80.240 version 3.1
> Apr  5 07:03:03 sq0n32 kernel: RDS/IB: connected to 172.31.64.32 version 3.1
>
> -----Original Message-----
> From: Tina Yang [mailto:tina.yang at oracle.com]
> Sent: Monday, April 05, 2010 12:29 PM
> To: Tang, Changqing
> Cc: Andy Grover; RDS Devel
> Subject: Re: [rds-devel] FW: RDS -- how to detect peer is gone ?
>
> On 4/5/2010 7:51 AM, Tang, Changqing wrote:
>
>> I still don't know the silent crash of the kernel, but I figure out what causes it.
>>
>> My MPI code has a heartbeat logic, when there is contention and a message is not
>> Received after some period, a heartbeat is sent to peer process.
>>
>> In two ranks two nodes case, the heartbeat is not triggered.
>>
>> In three ranks three nodes case, heartbeat is sent to the rank on the crashed node.
>> Heartbeat is a 48 bytes message. Maybe too many heartbeat messages flood the target
>> Process ? But no matter how bad my code logic is, it should not crash RDS kernel.
>>
>>
>
>
>      Can you send us the /var/log/messages ?  From what you've described,
>      I can't determine whether there was a real crash or the machine just
>      simply rebooted itself (maybe due to some built-in features of the
> applications).
>      If it looks like a crash, we'll probably need additional
> information for
>      further investigation.  At any rate, I think it would be a good
> practice to
>      set up 'netconsole' or preferably 'kdump' especially if you don't have
>      an easy access to the console.
>
>
>
>> After I disable the heartbeat, my MPI work works fine now.
>>
>>
>> --CQ
>>
>>
>>
>> -----Original Message-----
>> From: Andy Grover [mailto:andy.grover at oracle.com]
>> Sent: Sunday, April 04, 2010 2:07 PM
>> To: Tang, Changqing
>> Cc: Tina Yang; RDS Devel
>> Subject: Re: [rds-devel] FW: RDS -- how to detect peer is gone ?
>>
>> Do you have a way of capturing the oops? Perhaps via serial or
>> management console?
>>
>> Regards -- Andy
>>
>> Tang, Changqing wrote:
>>
>>
>>> Hi, Tina,
>>>
>>> I implement RDS as our MPI's interconnect, it works fine when I run
>>> IMB benchmark two ranks over two nodes.
>>>
>>> However, if I run 3 ranks over three nodes, it runs for a while and
>>> then silently crashes one of the machine, it is a repeatable crash of
>>> the same node.
>>>
>>>
>>>
>>> When I go to /var/log/messages after the node reboot, I don't find
>>> any useful info from RDS. The only RDS message
>>>
>>> Between the last two 'restart' are:
>>>
>>> .....
>>>
>>> Apr 4 07:43:05 sq0n32 kernel: Registered RDS/iwarp transport
>>>
>>> Apr 4 07:43:05 sq0n32 kernel: Registered RDS/infiniband transport
>>>
>>> Apr 4 07:43:05 sq0n32 kernel: Registered RDS/tcp transport
>>>
>>> .....
>>>
>>> Apr 4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.64.33
>>> version 3.1
>>>
>>> Apr 4 07:56:14 sq0n32 kernel: RDS/IB: connected to 172.31.80.240
>>> version 3.1
>>>
>>> .....
>>>
>>>
>>>
>>> When the crash occurs, the other two ranks just send a message to
>>> this rank on the crashed node, asking it to
>>>
>>> Send a RDS_CMSG_RDMA_MAP message back, the message size is 32K. But I
>>> don't know what this crashed rank is doing.
>>>
>>> I always use RDS_CMSG_RDMA_MAP control message with RDS_RDMA_USE_ONCE
>>> flag, and doing rdma write.
>>>
>>>
>>>
>>> Can you tell how to turn on RDS verbose mode so that I can get more
>>> message from /var/log/messages to figure
>>>
>>> Out what's wrong ?
>>>
>>>
>>>
>>> I am using OFED 1.5.1 on 2.6.18-128 kernel.
>>>
>>>
>>>
>>> Thanks
>>>
>>> --CQ
>>>
>>>
>>>
>>>
>>>
>>> From: Tina Yang [mailto:tina.yang at oracle.com] Sent: Friday, April 02,
>>> 2010 5:02 PM To: Andy Grover Cc: Tang, Changqing; RDS Devel Subject:
>>> Re: [rds-devel] FW: RDS -- how to detect peer is gone ?
>>>
>>>
>>>
>>> Andy Grover wrote:
>>>
>>> I went ahead and opened bug 2006 with this analysis.
>>>
>>> https://bugs.openfabrics.org/show_bug.cgi?id=2006
>>>
>>> Thanks! -- Regards -- Andy
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> Andy, Thank you, I will try to open a bug and provide a patch if I
>>> could.
>>>
>>> After reading the rds_recvmsg() function in recv.c (RDS source code),
>>>   I find the msg.msg_controllen processing does not follow the Linux
>>> recvmsg() man page.
>>>
>>> The Linux recvmsg() man page says that, upon return from recvmsg,
>>> msg.msg_controllen should contain the length of control message
>>> sequence. So if there is no control message, msg_controllen should be
>>>   set to zero.
>>>
>>> However, from the rds_recvmsg() code, if we receive rdma notification
>>>   control message, put_cmsg() is used on 'msghdr', in turn, put_cmsg()
>>>   just advance msg_control to next control message space, and
>>> msg_controllen is decreased to the size of available space.
>>> Eventually msg_controllen will be zero (if input length is multiple
>>> of control message length). The same thing for receiving
>>> RDS_CMSG_RDMA_DEST control message.
>>>
>>> Also if there is no rdma notification control message, or other
>>> control message, msg_controllen is not touched by RDS code.
>>>
>>> In another words, upon return from recvmsg(), msg_controllen is not
>>> the buffer length RDS code filled in.
>>>
>>>
>>>
>>>
>>>
>>> You seem to overlook the fact that the 'msghdr' rds_recvmsg() (and
>>> others like put_cmsg()) are manipulating is actually a kernel copy of
>>> the user-passed structure. To see what's returned after recvmsg(),
>>> you should go to sys_recvmsg() below,
>>>
>>> 1955         if (MSG_CMSG_COMPAT&  flags) 1956                 err =
>>> __put_user((unsigned long)msg_sys.msg_control-cmsg_ptr, 1957
>>> &msg_compat->msg_controllen); 1958         else 1959
>>> err = __put_user((unsigned long)msg_sys.msg_control-cmsg_ptr, 1960
>>> &msg->msg_controllen);
>>>
>>> Here, you see the msg_controllen is indeed set to whatever it says it
>>> would in the linux recvmsg() man page.
>>>
>>>
>>>
>>>
>>>
>>> Thanks for your comment.
>>>
>>> --CQ
>>>
>>>
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Wednesday, March 31, 2010 6:43
>>> PM To: Tang, Changqing Cc: RDS Devel Subject: Re: [rds-devel] FW: RDS
>>>   -- how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> We strongly ask the ability to run both 32bit and 64bit RDS code on
>>> 64bit kernel.
>>>
>>> --CQ
>>>
>>>
>>> Please open a bug at bugs.openfabrics.org.
>>>
>>> This is more likely to get fixed faster if you also attach a patch.
>>>
>>> Thanks -- Regards -- Andy
>>>
>>>
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Wednesday, March 31, 2010 4:40
>>> PM To: Tang, Changqing Cc: RDS Devel Subject: Re: [rds-devel] FW: RDS
>>> -- how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> Why not ? even IB verbs support both 32bit and 64bit apps.
>>>
>>>
>>> We support 32bit apps on a 32bit kernel and 64bit apps on a 64bit
>>> kernel. You are talking about some kind of 32bit userspace on a 64bit
>>> kernel. Nobody does that.
>>>
>>> -- Andy
>>>
>>>
>>>
>>> --CQ
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Wednesday, March 31, 2010 1:33
>>> PM To: Tang, Changqing Cc: RDS Devel Subject: Re: [rds-devel] FW: RDS
>>> -- how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> Andy, Thank you for your confirmation, when do you have a fix for
>>> this 32bit RDS problem on x86_64 system ?
>>>
>>> --CQ
>>>
>>>
>>> Running 32 bit apps on 64bit kernel is not supported.
>>>
>>> -- Andy
>>>
>>>
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Tuesday, March 30, 2010 8:00 PM
>>> To: Tang, Changqing Cc: RDS Devel Subject: Re: [rds-devel] FW: RDS --
>>> how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> Andy, I looked 'man cmsg', 'struct rds_get_mr_args' is always 32
>>> bytes.  Here is my test code:
>>>
>>> #include<stdio.h>  #include<stdlib.h>  #include<sys/socket.h>
>>>
>>> int main ()
>>>
>>> { struct cmsghdr *cmsg; char    cmsgbuf[CMSG_SPACE(32)];  /* using
>>> struct rds_get_mr_args size */
>>>
>>> cmsg = (struct cmsghdr *)cmsgbuf;
>>>
>>> cmsg->cmsg_len = CMSG_SPACE(32); cmsg->cmsg_type = 0;
>>> cmsg->cmsg_level = 1;
>>>
>>> fprintf(stderr, "offset %d\n", (char*)CMSG_DATA(cmsg)-(char*)cmsg); }
>>>
>>>
>>>
>>> The offset for 64bit is 16 and for 32bit is 12.
>>>
>>> So if my code is 32bit, I put 'struct rds_get_mr_args' on 12 bytes
>>> offset, but RDS kernel code will get it from 16 bytes offset.
>>>
>>> Am I wrong ?  Thank you again.
>>>
>>>
>>> Hi CQ,
>>>
>>> First, please always CC rds-devel so this discussion may be archived,
>>> and maybe help someone else in the future.
>>>
>>> Regarding your question -- I think you're correct that 32bit userland
>>> will not work with 64bit kernel.
>>>
>>> Regards -- Andy
>>>
>>>
>>>
>>> --CQ
>>>
>>>
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Tuesday, March 30, 2010 1:41 PM
>>> To: Tang, Changqing; RDS Devel Subject: Re: [rds-devel] FW: RDS --
>>> how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> Andy, One simple question, does 32bit rds-rdma code work on x86_64
>>> machine ? I noticed that the size of 'struct cmsghdr' is different
>>> between 32bit and 64bit, If the kernel code is always 64bit, how does
>>> the RDS kernel code figure out The control message buffer is passed
>>> as 32bit format?
>>>
>>> Do I miss something here ?
>>>
>>>
>>> See "man cmsg", it describes the various macros that resolve 32/64
>>> differences.
>>>
>>> Regards -- Andy
>>>
>>>
>>>
>>> Thank you. --CQ
>>>
>>> -----Original Message----- From: Andy Grover
>>> [mailto:andy.grover at oracle.com] Sent: Tuesday, March 16, 2010 5:44 PM
>>> To: Tang, Changqing Cc:
>>> rds-devel at oss.oracle.com<mailto:rds-devel at oss.oracle.com>  Subject:
>>> Re: [rds-devel] FW: RDS -- how to detect peer is gone ?
>>>
>>> Tang, Changqing wrote:
>>>
>>>
>>> [CQ] yes, the node is up and the process may corrupted. If you can
>>> extend the rds ping message a little bit to process as optional, that
>>> would be wonderful.
>>>
>>>
>>> I don't see why rds's ping functionality as-is is insufficient for
>>> what you want to do.
>>>
>>> [CQ] What do you mean ? how can I use rds ping function as-is to
>>> identify process down ?
>>>
>>>
>>> Like I said, if the process doesn't respond but the rds ping does,
>>> then you know the machine is alive but the process is not.
>>>
>>> -- Andy
>>>
>>>
>>>
>>>
>>> _______________________________________________ rds-devel mailing
>>> list rds-devel at oss.oracle.com<mailto:rds-devel at oss.oracle.com>
>>> http://oss.oracle.com/mailman/listinfo/rds-devel
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
>




More information about the rds-devel mailing list