File slack space calculator
I wanted to know whether I could use some tricks to make more efficient use of the 4GB Compact Flash storage i have in my Alix (running Gentoo Linux). I would like to keep a local portage tree on it — it currently mounts the tree over NFS due to space concerns.
Why no harddisk? Because that goes against the idea of having a small, reliable, always-on, energy-efficient home server.
Techies like pictures of hardware with the cover off so here’s my Alix:
Such a setup can lead to unfortunate situations, such as needing a package to restore connectivity to the network on which the fileserver resides that contains the package that I need to restore connectivity to the network on which… see where I’m going? Nowhere! ;-)
The portage tree fulfills the role of a package database for us Gentoo ricers. I need it locally.
Diego, one of the Gentoo devs, wrote a blog post about space inefficiency incurred through the use of many small files in the portage tree. He puts the Portage tree on different filesystems to arrive at an accurate and detailed picture of incurred overhead.
I wanted to quickly find out what filesystem block size would suit different parts of my filesystem (such as /etc/, /usr/src/linux/, /usr/portage/, /var/db/pkg/) best, and how much I could save. Testing all block sizes with all parts of my FS was not very appealing, so I decided to simply calculate the file slack and wrote this simple Python script:
#!/usr/bin/env python3 """ slacktastic.py - calculate filesystem file slack for different block sizes. Invoke me thusly: find /path/to/tree -xdev -type f -printf "%s\\n" | slacktastic.py 4096 for a calculation using the contents below /path/to/tree with a block size of 4096 bytes. """ import sys, functools, math blksz = None try: blksz = int(sys.argv[1]) except (IndexError,ValueError): print(__doc__, file=sys.stderr) print('I need a blocksize as the first argument.\n', file=sys.stderr) sys.exit(1) sizes = [int(strsize) for strsize in sys.stdin.read().strip().split('\n')] sumslack = functools.reduce(lambda sumslack, sz: sumslack + (blksz - (sz % blksz)), sizes, 0) sumblks = functools.reduce(lambda sumblks, sz: sumblks + (math.ceil(sz / blksz)), sizes, 0) sumsizes = sum(sizes) print('{:n} total slack'.format(sumslack)) print('{:n} bytes in files'.format(sumsizes)) print('{0:n} total blocks of {1} bytes'.format(sumblks,blksz)) print('{:.2%} inefficiency'.format( sumslack / (sumblks*blksz) ))This doesn’t take into account things such as tail packing (à la ReiserFS), compression (Btrfs), or directory slack. Just file slack.
On my laptop filesystem, with a 4096 block size, this leads to the following observation:
My /var/db/pkg, Gentoo’s ‘database’ of installed packages (containing their build environment and all kinds of stuff you wouldn’t need on a binary distro) contains 65% air! That’s 170 megs of waste which I don’t want that on my Alix’s 4GB CF card. With a 1024 byte block size — Ext4’s minimum — the situation is better, but it’s still over 40 megs of hot air.
I ended up choosing btrfs with compression. It has a fixed 4096 byte leaf/node size but it does tail packing (good for small files) and compression (for my /var/log). My script is useless for estimations on such a filesystem so I ran some actual tests and it turns out I can fit my /usr/src/linux, /var/db/pkg, /var/log and /usr/portage on a 1GB btrfs filesystem. They didn’t fit on an bs=1024b Ext4 FS.
Vrouwelijke arbeidsparticipatie in Legoland
Legoland. Je kunt er van alles worden. Behalve als je vrouw bent. Dan ben je dat gewoon, “vrouw”.
“Hee, vrouw!” zegt de beroepsbevolking dan op straat tegen je. Of: “Zeg meisje daar! Word jij ook vrouw als je groot bent? Zal wel hé?”
