<div dir="ltr"><div>Nope, the node that crashed is not the NFS server.</div>
<div> </div>
<div>How should I proceed?</div>
<div> </div>
<div>What do you suggest?</div>
<div> </div>
<div>Could this happen again?<br><br></div>
<div class="gmail_quote">On Wed, Oct 7, 2009 at 8:16 PM, Sunil Mushran <span dir="ltr"><<a href="mailto:sunil.mushran@oracle.com">sunil.mushran@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">And does the node exporting the volume encounter the oops?<br><br>If so, the likeliest candidate would be:<br>
<a href="http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6ca497a83e592d64e050c4d04b6dedb8c915f39a" target="_blank">http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6ca497a83e592d64e050c4d04b6dedb8c915f39a</a><br>
<br>If it is on another node, I am currently unsure whether a nfs<br>export on one node could cause this to occur on another. Need more<br>coffee.<br><br>The problem in short is due to how nfs bypasses the normal fs lookup<br>
to access files. It uses the file handle to directly access the inode,<br>bypassing the locking. Normally that is not a problem. The race window<br>is if the file is deleted (on any node in the cluster) and nfs reads that<br>
inode without the lock. In the oops we see the disk generation is greater<br>than the in-memory inode generation. That means the inode was deleted and<br>reused. The fix closes the race window.<br><br>Sunil<br><br>Laurence Mayer wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div class="im">Yes.<br>We have setup 10 node cluster, with one of the nodes exporting the NFS to the workstations.<br> Please expand your answer.<br> Thanks<br>Laurence<br><br><br></div>
<div>
<div></div>
<div class="h5"> On Wed, Oct 7, 2009 at 7:12 PM, Sunil Mushran <<a href="mailto:sunil.mushran@oracle.com" target="_blank">sunil.mushran@oracle.com</a> <mailto:<a href="mailto:sunil.mushran@oracle.com" target="_blank">sunil.mushran@oracle.com</a>>> wrote:<br>
<br> Are you exporting this volume via nfs? We fixed a small race (in<br> the nfs<br> access path) that could lead to this oops.<br><br> Laurence Mayer wrote:<br><br> Hi again,<br> OS: Ubuntu 8.04 x64<br>
Kern: Linux n1 2.6.24-24-server #1 SMP Tue Jul 7 19:39:36 UTC<br> 2009 x86_64 GNU/Linux<br> 10 Node Cluster<br> OCFS2 Version: 1.3.9-0ubuntu1<br> I received this panic on the 5th Oct, I cannot work out why<br>
this has started to happen.<br> Please please can you provide directions.<br> Let me know if you require any further details or information.<br> Oct 5 10:21:22 n1 kernel: [1006473.993681]<br> (1387,3):ocfs2_meta_lock_update:1675 ERROR: bug expression:<br>
inode->i_generation != le32_to_cpu(fe->i_generation)<br> Oct 5 10:21:22 n1 kernel: [1006473.993756]<br> (1387,3):ocfs2_meta_lock_update:1675 ERROR: Invalid dinode<br> 3064741 disk generation: 1309441612 inode->i_generation: 13<br>
09441501<br> Oct 5 10:21:22 n1 kernel: [1006473.993865] ------------[ cut<br> here ]------------<br> Oct 5 10:21:22 n1 kernel: [1006473.993896] kernel BUG at<br> /build/buildd/linux-2.6.24/fs/ocfs2/dlmglue.c:1675!<br>
Oct 5 10:21:22 n1 kernel: [1006473.993949] invalid opcode:<br> 0000 [3] SMP<br> Oct 5 10:21:22 n1 kernel: [1006473.993982] CPU 3<br> Oct 5 10:21:22 n1 kernel: [1006473.994008] Modules linked in:<br>
ocfs2 crc32c libcrc32c nfsd auth_rpcgss exportfs ipmi_devintf<br> ipmi_si ipmi_msghandler ipv6 ocfs2_dlmfs ocfs2_dlm<br> ocfs2_nodemanager configfs iptable_filter ip_tables x_tables<br> xfs ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr<br>
iscsi_tcp libiscsi scsi_transport_iscsi nfs lockd nfs_acl<br> sunrpc parport_pc lp parport loop serio_raw psmouse i2c_piix4<br> i2c_core dcdbas evdev button k8temp shpchp pci_hotplug pcspkr<br> ext3 jbd mbcache sg sr_mod cdrom sd_mod ata_generic pata_acpi<br>
usbhid hid ehci_hcd tg3 sata_svw pata_serverworks ohci_hcd<br> libata scsi_mod usbcore thermal processor fan fbcon tileblit<br> font bitblit softcursor fuse<br> Oct 5 10:21:22 n1 kernel: [1006473.994445] Pid: 1387, comm: R<br>
Tainted: G D 2.6.24-24-server #1<br> Oct 5 10:21:22 n1 kernel: [1006473.994479] RIP:<br> 0010:[<ffffffff8856c404>] [<ffffffff8856c404>]<br> :ocfs2:ocfs2_meta_lock_full+0x6a4/0xec0<br>
Oct 5 10:21:22 n1 kernel: [1006473.994558] RSP:<br> 0018:ffff8101238f9d58 EFLAGS: 00010296<br> Oct 5 10:21:22 n1 kernel: [1006473.994590] RAX:<br> 0000000000000093 RBX: ffff8102eaf03000 RCX: 00000000ffffffff<br>
Oct 5 10:21:22 n1 kernel: [1006473.994642] RDX:<br> 00000000ffffffff RSI: 0000000000000000 RDI: ffffffff8058ffa4<br> Oct 5 10:21:22 n1 kernel: [1006473.994694] RBP:<br> 0000000100080000 R08: 0000000000000000 R09: 00000000ffffffff<br>
Oct 5 10:21:22 n1 kernel: [1006473.994746] R10:<br> 0000000000000000 R11: 0000000000000000 R12: ffff81012599ee00<br> Oct 5 10:21:22 n1 kernel: [1006473.994799] R13:<br> ffff81012599ef08 R14: ffff81012599f2b8 R15: ffff81012599ef08<br>
Oct 5 10:21:22 n1 kernel: [1006473.994851] FS:<br> 00002b3802fed670(0000) GS:ffff810418022c80(0000)<br> knlGS:00000000f546bb90<br> Oct 5 10:21:22 n1 kernel: [1006473.994906] CS: 0010 DS: 0000<br>
ES: 0000 CR0: 000000008005003b<br> Oct 5 10:21:22 n1 kernel: [1006473.994938] CR2:<br> 00007f5db5542000 CR3: 0000000167ddf000 CR4: 00000000000006e0<br> Oct 5 10:21:22 n1 kernel: [1006473.994990] DR0:<br>
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br> Oct 5 10:21:22 n1 kernel: [1006473.995042] DR3:<br> 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br> Oct 5 10:21:22 n1 kernel: [1006473.995095] Process R (pid:<br>
1387, threadinfo ffff8101238f8000, task ffff8104110cc000)<br> Oct 5 10:21:22 n1 kernel: [1006473.995148] Stack:<br> 000000004e0c7e4c ffff81044e0c7ddd ffff8101a3b4d2b8<br> 00000000802c34c0<br> Oct 5 10:21:22 n1 kernel: [1006473.995212] 0000000000000000<br>
0000000100000000 ffffffff80680c00 00000000804715e2<br> Oct 5 10:21:22 n1 kernel: [1006473.995272] 0000000100000000<br> ffff8101238f9e48 ffff810245558b80 ffff81031e358680<br> Oct 5 10:21:22 n1 kernel: [1006473.995313] Call Trace:<br>
Oct 5 10:21:22 n1 kernel: [1006473.995380]<br> [<ffffffff8857d03f>] :ocfs2:ocfs2_inode_revalidate+0x5f/0x290<br> Oct 5 10:21:22 n1 kernel: [1006473.995427]<br> [<ffffffff88577fe6>] :ocfs2:ocfs2_getattr+0x56/0x1c0<br>
Oct 5 10:21:22 n1 kernel: [1006473.995470]<br> [vfs_stat_fd+0x46/0x80] vfs_stat_fd+0x46/0x80<br> Oct 5 10:21:22 n1 kernel: [1006473.995514]<br> [<ffffffff88569634>] :ocfs2:ocfs2_meta_unlock+0x1b4/0x210<br>
Oct 5 10:21:22 n1 kernel: [1006473.995553]<br> [filldir+0x0/0xf0] filldir+0x0/0xf0<br> Oct 5 10:21:22 n1 kernel: [1006473.995594]<br> [<ffffffff8856799e>] :ocfs2:ocfs2_readdir+0xce/0x230<br>
Oct 5 10:21:22 n1 kernel: [1006473.995631]<br> [sys_newstat+0x27/0x50] sys_newstat+0x27/0x50<br> Oct 5 10:21:22 n1 kernel: [1006473.995664]<br> [vfs_readdir+0xa5/0xd0] vfs_readdir+0xa5/0xd0<br>
Oct 5 10:21:22 n1 kernel: [1006473.995699]<br> [sys_getdents+0xcf/0xe0] sys_getdents+0xcf/0xe0<br> Oct 5 10:21:22 n1 kernel: [1006473.997568]<br> [system_call+0x7e/0x83] system_call+0x7e/0x83<br>
Oct 5 10:21:22 n1 kernel: [1006473.997605]<br> Oct 5 10:21:22 n1 kernel: [1006473.997627]<br> Oct 5 10:21:22 n1 kernel: [1006473.997628] Code: 0f 0b eb fe<br> 83 fd fe 0f 84 73 fc ff ff 81 fd 00 fe ff ff 0f<br>
Oct 5 10:21:22 n1 kernel: [1006473.997745] RIP<br> [<ffffffff8856c404>] :ocfs2:ocfs2_meta_lock_full+0x6a4/0xec0<br> Oct 5 10:21:22 n1 kernel: [1006473.997808] RSP<br> <ffff8101238f9d58><br>
Thanks<br> Laurence<br> ------------------------------------------------------------------------<br><br> _______________________________________________<br> Ocfs2-users mailing list<br></div>
</div> <a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a> <mailto:<a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a>>
<div class="im"><br> <a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br><br><br><br></div></blockquote><br></blockquote></div><br>
</div>