[Ocfs2-users] No space left on device

[Brad Plant]

On Sat, Feb 28, 2009 at 18:13:12 PST 2009, Joel Becker wrote:
> On Sat, Feb 28, 2009 at 12:09:37PM +0000, Nuno Fernandes wrote:
> > > 	That's rather odd.  What is the blocksize and cluster size of
> > > this filesystem?  Can you send me the output of 'debugfs.ocfs2 -R "stat
> > > /.zbr" /dev/hda3'?
> > >
> > > Joel
> > 
> > Hi,
> > 
> > Well.. we've deleted over 275 000 files and the problem disappeared... at 
> > least for now. We'll just have to wait for that amount of files to be added.
> 
> 	I wanted to see what .zbr looked like, because my hunch was that
> you were very fragmented and couldn't allocate enough contiguous space
> to create more extent blocks.  Deleting files would perhaps have freed
> some in-use extent blocks, thus making it work again.
> 	If you run into the problem again, we'll revisit.

I have this happen on different file systems running different mainline kernel versions created with different versions of ocfs2-tools. I've seen 1 file system "fill up" with only 40% odd in use according to df. What's interesting is that while 1 node can't write anymore, another node (only 2 nodes mounted rw) will still be able to write to the FS, but neither will be able to completely fill the FS.

I have created new ocfs2 file systems and the problem doesn't seem to occur (or it gets very close be actually being full), but weeks/months later after being in use, the problem develops. The FS I mentioned above that filled up at ~40% use was about 6 months old and you could say it did it's fair share of "work".

I just dumped some 1GB files in a directory using dd to fill up a FS (full at 87%) to that I could collect some debugfs.ocfs2 information. Upon not being able to create any further 1GB files (operations on the FS got really slow), I successfully created some 10MB files. I then created 100+ 1MB files using dd. What seemed strange is that dd kept returning "no space left on device", but it would always write 995328 bytes of data to the file.

dd if=/dev/zero of=2156 bs=1k count=1k
dd: writing `2156': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0551648 s, 18.0 MB/s

dd if=/dev/zero of=2157 bs=1k count=1k
dd: writing `2157': No space left on device
973+0 records in
972+0 records out
995328 bytes (995 kB) copied, 0.0545104 s, 18.3 MB/s



echo "stats" | debugfs.ocfs2 /dev/xvdd1 |cat
debugfs.ocfs2 1.4.3
debugfs: 	Revision: 0.90
	Mount Count: 0   Max Mount Count: 20
	State: 0   Errors: 0
	Check Interval: 0   Last Check: Thu Oct  8 20:23:54 2009
	Creator OS: 0
	Feature Compat: 0 
	Feature Incompat: 0 
	Tunefs Incomplete: 0 
	Feature RO compat: 0 
	Root Blknum: 5   System Dir Blknum: 6
	First Cluster Group Blknum: 3
	Block Size Bits: 12   Cluster Size Bits: 12
	Max Node Slots: 16
	Extended Attributes Inline Size: 0
	Label: 
	UUID: BD88053E263B4CEAABC941FE3413B4EA
	UUID_hash: 0 (0x0)
	Cluster stack: classic o2cb
	Inode: 2   Mode: 00   Generation: 964566170 (0x397e1c9a)
	FS Generation: 964566170 (0x397e1c9a)
	CRC32: 00000000   ECC: 0000
	Type: Unknown   Attr: 0x0   Flags: Valid System Superblock 
	Dynamic Features: (0x0) 
	User: 0 (root)   Group: 0 (root)   Size: 0
	Links: 0   Clusters: 13107199
	ctime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
	atime: 0x0 -- Thu Jan  1 10:00:00 1970
	mtime: 0x48f80fe8 -- Fri Oct 17 15:09:12 2008
	dtime: 0x0 -- Thu Jan  1 10:00:00 1970
	ctime_nsec: 0x00000000 -- 0
	atime_nsec: 0x00000000 -- 0
	mtime_nsec: 0x00000000 -- 0
	Last Extblk: 0
	Sub Alloc Slot: Global   Sub Alloc Bit: 65535
debugfs: 



echo "stat /brad/aaaaa" | debugfs.ocfs2 /dev/xvdd1 |cat                 
debugfs.ocfs2 1.4.3
debugfs: 	Inode: 11539619   Mode: 0750   Generation: 3859452415 (0xe60a91ff)
	FS Generation: 964566170 (0x397e1c9a)
	CRC32: 00000000   ECC: 0000
	Type: Directory   Attr: 0x0   Flags: Valid 
	Dynamic Features: (0x0) 
	User: 1000 (brad)   Group: 2000 (unknown)   Size: 4096
	Links: 2   Clusters: 1
	ctime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
	atime: 0x4b28b03b -- Wed Dec 16 21:02:35 2009
	mtime: 0x4b28b899 -- Wed Dec 16 21:38:17 2009
	dtime: 0x0 -- Thu Jan  1 10:00:00 1970
	ctime_nsec: 0x29bd9e6e -- 700292718
	atime_nsec: 0x31a68876 -- 832997494
	mtime_nsec: 0x29bd9e6e -- 700292718
	Last Extblk: 0
	Sub Alloc Slot: 2   Sub Alloc Bit: 87
	Tree Depth: 0   Count: 243   Next Free Rec: 1
	## Offset        Clusters       Block#          Flags
	0  0             1              9343355         0x0
debugfs: 



Hopefully this sheds some light on what's going on. Please let me know if you need any further info.

Cheers,

Brad