<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<tt>Hello everyone,<br>
<br>
We have a problem in a 3 member OCFS2 cluster used to serve an
web/php application that access (read and/or write) files located
in the OCFS2 volume.<br>
The problem appears only some times (apparently during high load
periods).<br>
<br>
SYMPTOMS:<br>
- access to OCFS2 content becomes more an more slow until stalls<br>
* a "ls" command that normally takes <=1s takes 30s, 40s,
1m,...<br>
- load average of the system grows to 150, 200 or even more<br>
<br>
- high iowait values: 70-90%<br>
* but CPU usage is low<br>
<br>
- in the syslog appears a lot of messages like:<br>
(httpd,XXXXX,Y):ocfs2_rename:1474 ERROR: status = -13<br>
or<br>
(httpd,XXXXX,Y):ocfs2_unlink:951 ERROR: status = -2<br>
<br>
and the more "worrying":<br>
kernel: INFO: task httpd:3488 blocked for more than 120
seconds.<br>
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.<br>
kernel: httpd D c6fe5d74 0 3488 1616
0x00000080 <br>
kernel: c6fe5e04 00000082 00000000 c6fe5d74 c6fe5d74 000041fd
c6fe5d88 c0439b18<br>
kernel: c0b976c0 c0b976c0 c0b976c0 c0b976c0 ed0f0ac0 c6fe5de8
c0b976c0 f75ac6c0<br>
kernel: f2f0cd60 c0a95060 00000001 c6fe5dbc c0874b8d c6fe5de8
f8fd9a86 00000001<br>
kernel: Call Trace:<br>
kernel: [<c0439b18>] ? default_spin_lock_flags+0x8/0x10<br>
kernel: [<c0874b8d>] ? _raw_spin_lock+0xd/0x10<br>
kernel: [<f8fd9a86>] ?
ocfs2_dentry_revalidate+0xc6/0x2d0 [ocfs2]<br>
kernel: [<f8ff17be>] ? ocfs2_permission+0xfe/0x110
[ocfs2]<br>
kernel: [<f905b6f0>] ? ocfs2_acl_chmod+0xd0/0xd0
[ocfs2]<br>
kernel: [<c0873105>] schedule+0x35/0x50<br>
kernel: [<c0873b2e>] __mutex_lock_slowpath+0xbe/0x120<br>
....<br>
<br>
<br>
(UNACCEPTABLE) WORKAROUND:<br>
stop httpd (really slow)<br>
stop ocfs2 service (really slow)<br>
start ocfs2 an httpd<br>
<br>
MORE INFO:<br>
- OS information:<br>
Oracle Linux 6.4 32bit<br>
4GB RAM<br>
uname -a: 2.6.39-400.109.6.el6uek.i686 #1 SMP Wed Aug 28
09:55:10 PDT 2013 i686 i686 i386 GNU/Linux<br>
* anyway: we have another 5 nodes cluster with Oracle Linux
7.1 (so 64bit OS) serving a newer version of the same application
and the problems are similar, so it appears not to be a OS problem
but a more specific OCFS2 problem (bug? some tuning? other?)<br>
<br>
- standard configuration<br>
* if you want I can show the cluster.conf configuration but is
the "expected configuration"<br>
<br>
- standard configuration in o2cb:<br>
Driver for "configfs": Loaded<br>
Filesystem "configfs": Mounted<br>
Stack glue driver: Loaded<br>
Stack plugin "o2cb": Loaded<br>
Driver for "ocfs2_dlmfs": Loaded<br>
Filesystem "ocfs2_dlmfs": Mounted<br>
Checking O2CB cluster "MoodleOCFS2": Online<br>
Heartbeat dead threshold: 31<br>
Network idle timeout: 30000<br>
Network keepalive delay: 2000<br>
Network reconnect delay: 2000<br>
Heartbeat mode: Local<br>
Checking O2CB heartbeat: Active<br>
<br>
- mount options: _netdev,rw,noatime<br>
* so other options (commit, data, ...) have their default
values<br>
<br>
<br>
Any ideas/suggestion?<br>
<br>
Regards.<br>
<br>
</tt>
<div class="moz-signature">-- <br>
<hr>
<img src="cid:part1.06060203.08010504@uva.es">
<p class="MsoNormal"><b><font face="Franklin Gothic Book"
color="gray" size="1"><span
style="font-size:8.0pt;font-family:"Franklin Gothic
Book";color:gray; font-weight:bold">
Area de Sistemas<br>
Servicio de las Tecnologias de la Informacion y
Comunicaciones (STIC)<br>
Universidad de Valladolid<br>
Edificio Alfonso VIII, C/Real de Burgos s/n. 47011,
Valladolid - ESPAÑA<br>
Telefono: 983 18-6410, Fax: 983 423271<br>
E-mail: <a class="moz-txt-link-abbreviated" href="mailto:sistemas@uva.es">sistemas@uva.es</a><br>
</span></font></b></p>
<b><font face="Franklin Gothic Book" color="gray" size="1">
<hr>
</font></b></div>
</body>
</html>