[Ocfs2-users] High Load Average - Now the server doesn´t load

Thu Dec 18 14:17:45 PST 2008

Hello Sunil and all,

I was testing a new kernel in another server, to simulate the migration and my both servers crashed, ie, they don't load anymore. The linux load, but when the ocfs2 load, appears the follow in screen:

Dec 18 08:06:16 paramana kernel: ----------- [cut here ] --------- [please bite here ] ---------
Dec 18 08:06:16 paramana kernel: CPU 0 
Dec 18 08:06:16 paramana kernel: Modules linked in: ocfs2 ocfs2_dlmfs ocfs2_dlm ocfs2_nodemanager configfs qla2xxx reiserfs dm_snapshot dm_mirror dm_mod loop serio_raw floppy psmouse shpchp pci_hotplug pcspkr tsdev evdev joydev sg ext3 jbd mbcache ide_cd cdrom usbhid piix sd_mod generic ide_core ehci_hcd uhci_hcd firmware_class scsi_transport_fc megaraid_mbox scsi_mod megaraid_mm tg3 thermal processor fan
Dec 18 08:06:16 paramana kernel: Pid: 4014, comm: ocfs2_wq Not tainted 2.6.18-4-amd64 #1
Dec 18 08:06:16 paramana kernel: RIP: 0010:[<ffffffff88271360>] [<ffffffff88271360>] :ocfs2:ocfs2_commit_truncate+0x550/0x1537
Dec 18 08:06:16 paramana kernel: RSP: 0000:ffff810223dedb40 EFLAGS: 00010297
Dec 18 08:06:16 paramana kernel: RAX: 0000000000000000 RBX: ffff8102220c90c0 RCX: 0000000000000002
Dec 18 08:06:16 paramana kernel: RDX: 0000000000f30000 RSI: 0000000000000000 RDI: 0000000000000000
Dec 18 08:06:16 paramana kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: ffff810226afc080
Dec 18 08:06:16 paramana kernel: R10: ffff810226482e40 R11: 0000000000000060 R12: ffff81021fe82780
Dec 18 08:06:16 paramana kernel: R13: ffff810220305c48 R14: ffff81021fe82b48 R15: ffff81021fe82b48
Dec 18 08:06:16 paramana kernel: FS: 0000000000000000(0000) GS:ffffffff80521000(0000) knlGS:0000000000000000
Dec 18 08:06:16 paramana kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 18 08:06:16 paramana kernel: CR2: 00002b2119d43210 CR3: 0000000223acf000 CR4: 00000000000006e0
Dec 18 08:06:16 paramana kernel: Process ocfs2_wq (pid: 4014, threadinfo ffff810223dec000, task ffff810226afc080)
Dec 18 08:06:16 paramana kernel: Stack: ffff8101ffe20b20 ffff81021ff2ac10 ffff81022500c000 ffff810224070b88
Dec 18 08:06:16 paramana kernel: ffff81021fe82a88 0000000025088ac0 ffff810200000000 ffffffff881192f7
Dec 18 08:06:16 paramana kernel: 0000000000000000 ffff81021fa42000 ffff81021fa420c0 ffff8102265330d8
Dec 18 08:06:16 paramana kernel: Call Trace:
Dec 18 08:06:16 paramana kernel: [<ffffffff881192f7>] :jbd:__journal_file_buffer+0x14a/0x24f
Dec 18 08:06:16 paramana kernel: [<ffffffff802b641b>] alternate_node_alloc+0x70/0x8c
Dec 18 08:06:16 paramana kernel: [<ffffffff8826f784>] :ocfs2:ocfs2_prepare_truncate+0x184/0x4f8
Dec 18 08:06:16 paramana kernel: [<ffffffff882851d6>] :ocfs2:ocfs2_wipe_inode+0x466/0xb23
Dec 18 08:06:16 paramana kernel: [<ffffffff882a11bc>] :ocfs2:ocfs2_delete_response_cb+0x0/0x17f
Dec 18 08:06:16 paramana kernel: [<ffffffff88288122>] :ocfs2:ocfs2_delete_inode+0x623/0x7b1
Dec 18 08:06:16 paramana kernel: [<ffffffff8022b39f>] wake_up_bit+0x11/0x23
Dec 18 08:06:16 paramana kernel: [<ffffffff882780e3>] :ocfs2:ocfs2_cluster_unlock+0x243/0x2e1
Dec 18 08:06:16 paramana kernel: [<ffffffff88287aff>] :ocfs2:ocfs2_delete_inode+0x0/0x7b1
Dec 18 08:06:16 paramana kernel: [<ffffffff8022d395>] generic_delete_inode+0xc6/0x143
Dec 18 08:06:16 paramana kernel: [<ffffffff8828753a>] :ocfs2:ocfs2_drop_inode+0x117/0x16e
Dec 18 08:06:16 paramana kernel: [<ffffffff8828b798>] :ocfs2:ocfs2_complete_recovery+0xa1d/0xb65
Dec 18 08:06:16 paramana kernel: [<ffffffff8025cc4e>] thread_return+0x0/0xe7
Dec 18 08:06:16 paramana kernel: [<ffffffff8828ad7b>] :ocfs2:ocfs2_complete_recovery+0x0/0xb65
Dec 18 08:06:16 paramana kernel: [<ffffffff80249495>] run_workqueue+0x94/0xe5
Dec 18 08:06:16 paramana kernel: [<ffffffff80245e96>] worker_thread+0x0/0x122
Dec 18 08:06:16 paramana kernel: [<ffffffff802901da>] keventd_create_kthread+0x0/0x61
Dec 18 08:06:16 paramana kernel: [<ffffffff80245f86>] worker_thread+0xf0/0x122
Dec 18 08:06:16 paramana kernel: [<ffffffff8027d299>] default_wake_function+0x0/0xe
Dec 18 08:06:16 paramana kernel: [<ffffffff802901da>] keventd_create_kthread+0x0/0x61
Dec 18 08:06:16 paramana kernel: [<ffffffff80230575>] kthread+0xd4/0x107
Dec 18 08:06:16 paramana kernel: [<ffffffff80259360>] child_rip+0xa/0x12
Dec 18 08:06:16 paramana kernel: [<ffffffff802901da>] keventd_create_kthread+0x0/0x61
Dec 18 08:06:16 paramana kernel: [<ffffffff802304a1>] kthread+0x0/0x107
Dec 18 08:06:16 paramana kernel: [<ffffffff80259356>] child_rip+0x0/0x12
Dec 18 08:06:16 paramana kernel: 
Dec 18 08:06:16 paramana kernel: 
Dec 18 08:06:16 paramana kernel: Code: 0f 0b 68 f6 56 2a 88 c2 b9 01 66 85 d2 0f 95 c2 66 ff ce 0f 
Dec 18 08:06:16 paramana kernel: RSP <ffff810223dedb40>
Dec 18 08:06:18 paramana heartbeat[4240]: info: **************************
Dec 18 08:06:18 paramana heartbeat[4240]: info: Configuration validated. Starting heartbeat 1.2.5
Dec 18 08:06:18 paramana heartbeat[4245]: info: heartbeat: version 1.2.5
Dec 18 08:06:28 paramana kernel: <3>BUG: soft lockup detected on CPU#13!
Dec 18 08:06:28 paramana kernel: 
Dec 18 08:06:28 paramana kernel: Call Trace:
Dec 18 08:06:28 paramana kernel: <IRQ> [<ffffffff802a4008>] softlockup_tick+0xdb/0xed
Dec 18 08:06:28 paramana kernel: [<ffffffff802881fb>] update_process_times+0x42/0x68
Dec 18 08:06:28 paramana kernel: [<ffffffff8026cbd8>] smp_local_timer_interrupt+0x23/0x47
Dec 18 08:06:28 paramana kernel: [<ffffffff8026d2cc>] smp_apic_timer_interrupt+0x41/0x47
Dec 18 08:06:28 paramana kernel: [<ffffffff8025904a>] apic_timer_interrupt+0x66/0x6c
Dec 18 08:06:28 paramana kernel: <EOI> [<ffffffff8020ac31>] __find_get_block+0x8c/0x16c
Dec 18 08:06:28 paramana kernel: [<ffffffff8829f09d>] :ocfs2:ocfs2_buffer_cached+0x9a/0x10e
Dec 18 08:06:28 paramana kernel: [<ffffffff8021797b>] __getblk+0x1d/0x223
Dec 18 08:06:28 paramana kernel: [<ffffffff8827448d>] :ocfs2:ocfs2_read_blocks+0x255/0x65f
Dec 18 08:06:28 paramana kernel: [<ffffffff88296f99>] :ocfs2:ocfs2_search_chain+0x275/0xfab
Dec 18 08:06:28 paramana kernel: [<ffffffff88298585>] :ocfs2:ocfs2_claim_suballoc_bits+0x8b6/0xba6
Dec 18 08:06:28 paramana kernel: [<ffffffff8829a19f>] :ocfs2:ocfs2_claim_new_inode+0xda/0x1d7
Dec 18 08:06:28 paramana kernel: [<ffffffff882907a3>] :ocfs2:ocfs2_mknod_locked+0xd2/0x731
Dec 18 08:06:28 paramana kernel: [<ffffffff8829ee23>] :ocfs2:ocfs2_get_system_file_inode+0x3b/0x1b8
Dec 18 08:06:28 paramana kernel: [<ffffffff8811a12f>] :jbd:journal_start+0xc9/0x100
Dec 18 08:06:28 paramana kernel: [<ffffffff88291313>] :ocfs2:ocfs2_mknod+0x511/0xc6c
Dec 18 08:06:28 paramana kernel: [<ffffffff88291f87>] :ocfs2:ocfs2_create+0x7f/0xda
Dec 18 08:06:28 paramana kernel: [<ffffffff80238208>] vfs_create+0xe7/0x12c
Dec 18 08:06:28 paramana kernel: [<ffffffff80218f16>] open_namei+0x18d/0x69c
Dec 18 08:06:28 paramana kernel: [<ffffffff8022530c>] do_filp_open+0x1c/0x3d
Dec 18 08:06:28 paramana kernel: [<ffffffff80217bc5>] do_sys_open+0x44/0xc5
Dec 18 08:06:28 paramana kernel: [<ffffffff802584d6>] system_call+0x7e/0x83
Dec 18 08:06:28 paramana kernel: 
Dec 18 08:06:28 paramana kernel: 
Dec 18 08:06:28 paramana kernel: Call Trace:
Dec 18 08:06:28 paramana kernel: <IRQ> [<ffffffff802a4008>] softlockup_tick+0xdb/0xed
Dec 18 08:06:28 paramana kernel: [<ffffffff802881fb>] update_process_times+0x42/0x68
Dec 18 08:06:28 paramana kernel: [<ffffffff8026cbd8>] smp_local_timer_interrupt+0x23/0x47
Dec 18 08:06:28 paramana kernel: [<ffffffff8026d2cc>] smp_apic_timer_interrupt+0x41/0x47
Dec 18 08:06:28 paramana kernel: [<ffffffff8025904a>] apic_timer_interrupt+0x66/0x6c
Dec 18 08:06:28 paramana kernel: <EOI> [<ffffffff88298c47>] :ocfs2:ocfs2_block_group_find_clear_bits+0xff/0x151
Dec 18 08:06:28 paramana kernel: [<ffffffff88298c44>] :ocfs2:ocfs2_block_group_find_clear_bits+0xfc/0x151
Dec 18 08:06:28 paramana kernel: [<ffffffff8829a7bb>] :ocfs2:ocfs2_cluster_group_search+0xff/0x13b
Dec 18 08:06:28 paramana kernel: [<ffffffff882970d4>] :ocfs2:ocfs2_search_chain+0x3b0/0xfab
Dec 18 08:06:28 paramana kernel: [<ffffffff882986df>] :ocfs2:ocfs2_claim_suballoc_bits+0xa10/0xba6
Dec 18 08:06:28 paramana kernel: [<ffffffff882989ec>] :ocfs2:ocfs2_claim_clusters+0x177/0x2d3
Dec 18 08:06:28 paramana kernel: [<ffffffff8828dff6>] :ocfs2:ocfs2_reserve_local_alloc_bits+0x965/0xe10
Dec 18 08:06:28 paramana kernel: [<ffffffff88299c70>] :ocfs2:ocfs2_reserve_clusters+0x11c/0x317
Dec 18 08:06:28 paramana kernel: [<ffffffff88281705>] :ocfs2:ocfs2_extend_file+0x4b9/0xf7c
Dec 18 08:06:28 paramana kernel: [<ffffffff882780e3>] :ocfs2:ocfs2_cluster_unlock+0x243/0x2e1
Dec 18 08:06:28 paramana kernel: [<ffffffff88284070>] :ocfs2:ocfs2_file_aio_write+0x7d3/0x98b
Dec 18 08:06:28 paramana kernel: [<ffffffff80264926>] do_gettimeofday+0x50/0x94
Dec 18 08:06:28 paramana kernel: [<ffffffff80215ebc>] do_sync_write+0xc7/0x104
Dec 18 08:06:28 paramana kernel: [<ffffffff8023a038>] hrtimer_start+0xbb/0xcd
Dec 18 08:06:28 paramana kernel: [<ffffffff8029039d>] autoremove_wake_function+0x0/0x2e
Dec 18 08:06:28 paramana kernel: [<ffffffff80231459>] do_setitimer+0x16d/0x4bf
Dec 18 08:06:28 paramana kernel: [<ffffffff80226b4c>] do_sigaction+0x7a/0x19e
Dec 18 08:06:28 paramana kernel: [<ffffffff80214966>] vfs_write+0xce/0x174
Dec 18 08:06:28 paramana kernel: [<ffffffff802151f0>] sys_write+0x45/0x6e
Dec 18 08:06:28 paramana kernel: [<ffffffff802584d6>] system_call+0x7e/0x83

I have to reboot with ctrl + alt + prt scr + s + u + b. It happens when I turn on one or both servers. I need help to make things be as before crash.

Can I attach this disk/partition to the new server with the new kernel 2.6.24 and make fsck with there without be worried?

I´m terrible sorry to insist in this topic, but I was planning to migrate saturday morning, and the crash happens today :(

Thank a lot

Jeronimo 

      Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20081218/a53a8ea2/attachment.html 