<div class="gmail_extra">1.5 ms per inode. Times 900K files equals 22 mins.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Large dirs are a problem is all file systems. The degree of problem</div><div class="gmail_extra">
depends on the overhead. An easy solution around is to shard the</div><div class="gmail_extra">files into multilevel dirs. Like a 2 level structure of a 1000 files in</div><div class="gmail_extra">1000 dirs. Or, a 3 level structure with even fewer files per dir.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Or you could use the other approach suggested. Avoids stat()</div><div class="gmail_extra">by disabling color-ls. Or just use plain find.</div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Dec 4, 2012 at 3:16 PM, Erik Schwartz <span dir="ltr"><<a href="mailto:schwartz.erik.c@gmail.com" target="_blank">schwartz.erik.c@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Amaury, you can see in strace output that it's performing a stat on<br>
every file.<br>
<br>
Try simply:<br>
<br>
$ /bin/ls<br>
<br>
My guess is you're using a system where "ls" is aliased to use options<br>
that are more expensive.<br>
<br>
Best regards -<br>
<br>
Erik<br>
<div><div class="h5"><br>
<br>
On 12/4/12 5:12 PM, Amaury Francois wrote:<br>
> The strace looks like this (on all files) :<br>
><br>
><br>
><br>
> 1354662591.755319<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P069_F01589.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001389><br>
><br>
> 1354662591.756775<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P035_F01592.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001532><br>
><br>
> 1354662591.758376<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P085_F01559.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001429><br>
><br>
> 1354662591.759873<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P027_F01569.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001377><br>
><br>
> 1354662591.761317<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P002_F01581.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001420><br>
><br>
> 1354662591.762804<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P050_F01568.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001345><br>
><br>
> 1354662591.764216<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P089_F01567.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001541><br>
><br>
> 1354662591.765828<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P010_F01594.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001358><br>
><br>
> 1354662591.767252<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P045_F01569.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001396><br>
><br>
> 1354662591.768715<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P036_F01592.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.002072><br>
><br>
> 1354662591.770854<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P089_F01568.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001722><br>
><br>
> 1354662591.772643<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P009_F01600.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001281><br>
><br>
> 1354662591.773992<br>
> lstat64("TEW_STRESS_TEST_VM.1K_100P_10000F.P022_F01583.txt",<br>
> {st_mode=S_IFREG|0664, st_size=1000, ...}) = 0 <0.001413><br>
><br>
><br>
><br>
> We are using a 32 bits architecture, can it be the cause of the kernel<br>
> not having enough memory ? Any possibility to change this behavior ?<br>
><br>
><br>
><br>
</div></div><div class="im">> Description : Description : Description : cid:image001.png@01CD01F3.35091200<br>
><br>
><br>
><br>
</div>> * *<br>
><br>
> *Amaury FRANCOIS* • *Ingénieur*<br>
<div class="im">><br>
> Mobile <a href="tel:%2B33%20%280%296%2088%20%2012%2062%2054" value="+33688126254">+33 (0)6 88 12 62 54</a><br>
><br>
</div>> *<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a> <mailto:<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a>>***<br>
><br>
> * *<br>
><br>
> *Siège Social – 66 rue du Marché Gare – 67200 STRASBOURG*<br>
<div class="im">><br>
> Tél : 0 820 200 217 - <a href="tel:%2B33%20%280%293%2088%2010%2049%2020" value="+33388104920">+33 (0)3 88 10 49 20</a><br>
><br>
><br>
><br>
</div>> Description : test<br>
><br>
><br>
><br>
><br>
><br>
> *De :*Sunil Mushran [mailto:<a href="mailto:sunil.mushran@gmail.com">sunil.mushran@gmail.com</a>]<br>
> *Envoyé :* mardi 4 décembre 2012 18:29<br>
> *À :* Amaury Francois<br>
> *Cc :* <a href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>
> *Objet :* Re: [Ocfs2-users] "ls" taking ages on a directory containing<br>
<div class="im">> 900000 files<br>
><br>
><br>
><br>
> strace -p PID -ttt -T<br>
><br>
><br>
><br>
> Attach and get some timings. The simplest guess is that the system lacks<br>
> memory to cache all the inodes<br>
><br>
> and thus has to hit disk (and more importantly take cluster locks) for<br>
> the same inode repeatedly. The user<br>
><br>
> guide has a section in NOTES explaining this.<br>
><br>
><br>
><br>
><br>
><br>
> On Tue, Dec 4, 2012 at 8:54 AM, Amaury Francois<br>
</div><div><div class="h5">> <<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a> <mailto:<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a>>> wrote:<br>
><br>
> Hello,<br>
><br>
><br>
><br>
> We are running OCFS2 1.8 and on a kernel UEK2. An ls on a directory<br>
> containing approx. 1 million of files is very long (1H). The features<br>
> we have activated on the filesystem are the following :<br>
><br>
><br>
><br>
> [root@pa-oca-app10 ~]# debugfs.ocfs2 -R "stats" /dev/sdb1<br>
><br>
> Revision: 0.90<br>
><br>
> Mount Count: 0 Max Mount Count: 20<br>
><br>
> State: 0 Errors: 0<br>
><br>
> Check Interval: 0 Last Check: Fri Nov 30 19:30:17 2012<br>
><br>
> Creator OS: 0<br>
><br>
> Feature Compat: 3 backup-super strict-journal-super<br>
><br>
> Feature Incompat: 32592 sparse extended-slotmap inline-data<br>
> metaecc xattr indexed-dirs refcount discontig-bg clusterinfo<br>
><br>
> Tunefs Incomplete: 0<br>
><br>
> Feature RO compat: 1 unwritten<br>
><br>
> Root Blknum: 5 System Dir Blknum: 6<br>
><br>
> First Cluster Group Blknum: 3<br>
><br>
> Block Size Bits: 12 Cluster Size Bits: 12<br>
><br>
> Max Node Slots: 8<br>
><br>
> Extended Attributes Inline Size: 256<br>
><br>
> Label: exchange2<br>
><br>
> UUID: 2375EAF4E4954C4ABB984BDE27AC93D5<br>
><br>
> Hash: 2880301520 (0xabade9d0)<br>
><br>
> DX Seeds: 1678175851 1096448356 79406012 (0x6406ee6b 0x415a7964<br>
> 0x04bba3bc)<br>
><br>
> Cluster stack: o2cb<br>
><br>
> Cluster name: appcluster<br>
><br>
> Cluster flags: 1 Globalheartbeat<br>
><br>
> Inode: 2 Mode: 00 Generation: 3567595533 (0xd4a5300d)<br>
><br>
> FS Generation: 3567595533 (0xd4a5300d)<br>
><br>
> CRC32: 0c996202 ECC: 0819<br>
><br>
> Type: Unknown Attr: 0x0 Flags: Valid System Superblock<br>
><br>
> Dynamic Features: (0x0)<br>
><br>
> User: 0 (root) Group: 0 (root) Size: 0<br>
><br>
> Links: 0 Clusters: 5242635<br>
><br>
> ctime: 0x508eac6b 0x0 -- Mon Oct 29 17:18:51.0 2012<br>
><br>
> atime: 0x0 0x0 -- Thu Jan 1 01:00:00.0 1970<br>
><br>
> mtime: 0x508eac6b 0x0 -- Mon Oct 29 17:18:51.0 2012<br>
><br>
> dtime: 0x0 -- Thu Jan 1 01:00:00 1970<br>
><br>
> Refcount Block: 0<br>
><br>
> Last Extblk: 0 Orphan Slot: 0<br>
><br>
> Sub Alloc Slot: Global Sub Alloc Bit: 65535<br>
><br>
><br>
><br>
><br>
><br>
> May inline-data or xattr be the source of the problem ?<br>
><br>
><br>
><br>
> Thank you.<br>
><br>
><br>
><br>
</div></div><div class="im">> Description : Description : Description : cid:image001.png@01CD01F3.35091200<br>
><br>
><br>
><br>
</div>> * *<br>
><br>
> *Amaury FRANCOIS* • *Ingénieur*<br>
<div class="im">><br>
> Mobile <a href="tel:%2B33%20%280%296%2088%20%2012%2062%2054" value="+33688126254">+33 (0)6 88 12 62 54</a><br>
</div>> <tel:%2B33%20%280%296%2088%C2%A0%2012%2062%2054><br>
><br>
> *<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a> <mailto:<a href="mailto:amaury.francois@digora.com">amaury.francois@digora.com</a>>*<br>
><br>
> * *<br>
><br>
> *Siège Social – 66 rue du Marché Gare – 67200 STRASBOURG*<br>
<div class="im">><br>
> Tél : 0 820 200 217 - <a href="tel:%2B33%20%280%293%2088%2010%2049%2020" value="+33388104920">+33 (0)3 88 10 49 20</a><br>
</div>> <tel:%2B33%20%280%293%2088%2010%2049%2020><br>
<div class="im">><br>
><br>
><br>
> Description : test<br>
><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Ocfs2-users mailing list<br>
</div>> <a href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a> <mailto:<a href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a>><br>
> <a href="https://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br>
<div class="HOEnZb"><div class="h5">><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Ocfs2-users mailing list<br>
> <a href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><br>
> <a href="https://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br>
><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Erik Schwartz <<a href="mailto:schwartz.erik.c@gmail.com">schwartz.erik.c@gmail.com</a>> | GPG key 14F1139B<br>
<br>
</font></span></blockquote></div><br></div>