Jim's Depository

I installed SLOCCount today on my development machine. You can run it from one of your source directories like… sloccount *.[hc] … and it will tell you not only how many lines of code you have but what it may have cost to write them. So far I’ve written \$15k of code today. I can’t find the right command line option to get it to deposit that in my account.

Since I last wrote on the topic, Into the world of Virtual Hosting, I’ve split my server into two with this part being on a new virtual hosting operation. I chose RapidXen because it has IPv6 support, and is cheap. I’m using a 128M RAM/10G disk package for \$83/year, which is about the same as my 64M/2.5G package over at VPSLink.  

• The RAM change is a little deceptive, the RapidXen box is a 64 bit machine, so a good part of the extra RAM just goes to cover that bloat.
  
• The RapidXen machine is many times faster.
• The VPSLink machine has become much slower as time has gone on. I can’t tell if they are loading the machine more heavily, or if some IO pig has moved into my server, but life is not as good as it used to be.
• RapidXen is a smaller operation, I had to wait for a human during the setup.
• My IPv4 address at RapidXen was pre-blocked as a spam source by AT&T, but no other black lists I could find. Sometime worth checking when you get a recycled IP.
• You apparently have to open a trouble ticket and ask for console access to your server at RapidXen, I haven’t completed that yet. At VPSLink it is available by default with a Java based SSH client.

One person regularly corresponds with me using Hotmail. I’m frequently amused at the odd non-sequiturs at the end of his sometimes delicate email.

Lately though, the Hotmail graffiti ads don’t even make sense.

For example:

Send e-mail anywhere. No map, no compass. Get your Hotmail account now.

Because I need maps and a compass to use gmail? My Safari icon is a compass, should I not use Safari for Hotmail? Or is it a reference to paper mail where I used a map and a compass to post letters? 

Or perhaps:

Send e-mail faster without improving your typing skills. Get your Hotmail account now.

Faster? My bits already move at about light speed, that can’t be it. Perhaps the delay between pressing SEND and when it leaves the sending computer could be a few milliseconds shorter. How many people sit down to deliberately improve their typing speed so they can send email faster? Who, other than spammers, even cares how fast they send email?

Bus error - founder dumped

That was a nice sixteen years.

I needed a machine to do some DNS server tests. I settled on a \$280 EEE PC 900A (stripped of webcam and half of its storage) from Best Buy. That gets me a 1.6GHz x86 server with 1G of ram that only burns 10 watts and comes with its own little console for when I need it. Not a bad deal. 

Only 4G of storage, but I’m only using about 60% even with a bunch of heavy eyecandy gnome and compiz stuff I installed to see what would happen (it is pretty fast, lower end graphic accelerator, but not many pixels comes out well).

I wiped the friendly linux it came with and installed Debian Lenny and all is good, except I kept noticing intermittent disk hangs lasting several seconds. I think I finally tracked this down to the kernel syncing out written pages. The fix is to not write so much. By mounting the partitions noatime most of my writes go away and I don’t notice hangs anymore.

Reading the first byte of every file in /usr went from 131 seconds to 92 seconds with the change (after a fresh boot each time), that is about a 30% speedup.

I’m pleased with the EEE. My code builds from clean in 1.6 seconds. I rarely use more than 10% of the RAM doing development which leaves plenty of RAM for caches to mitigate the slow flash disk. 

At last, I can put my /boot partition in LVM.

• Get the Debian box up to Lenny.
• Note that I accidentally trashed my MBR and had to boot into rescue mode while working out these steps. You shouldn’t do this if you follow all of the instructions, but you ought to have media handy.
• aptitude install grub-pc (Note: this will remove the old grub package and offer to chain load grub2 from your existing grub. Do this. If you have problems you can still boot.)
• Verify you can reboot.
• Remove the old grub MBR and put in the grub2 one with upgrade-from-grub-legacy
• Hide your /boot/grub/menu.lst so you aren’t tempted to edit it.
• Your basic configuration, like kernel command line parameters is now in /etc/defaults/grub, there is also /etc/grub.d/* which I hope to never touch.
• Move your /boot into the LVM. You could tar up your /boot partition, unmount it, and extract it onto the root partition. You could also make a new LVM managed /boot if you like it on its own partition. I was out of space in the volume groups, so I went with /. If you didn’t make a new /boot, remember to take /boot out of /etc/fstab.
• dd zeros onto your old boot partition to make sure you aren’t deluding yourself.
• Edit /etc/defaults/grub to add GRUB_PRELOAD_MODULES=lvm
• Go back and make certain you did the previous step. I made an unbootable system before I learned that little tidbit.
• Do an update-grub and a grub-install /dev/sda or whatever your disk is.
• Go back and make sure you did the grub-install… Just update-grub is not enough to pick up the lvm module.
• Reboot and rejoice.

I am left wondering what silliness lead to GRUB-2 being version 1.96, but I am happy.

Little known and not terribly documented, but true:

You can tell your Debian machine to do an apt update and and download any files that you might need by adding two lines to your apt config (which I’ll bet you didn’t know you had.)

cat > /etc/apt/apt.conf.d/50autoupdate APT::Periodic::Update-Package-Lists “1”; APT::Periodic::Download-Upgradeable-Packages “1”;

Then the script /etc/cron.daily/apt will keep you all up to date savings you seconds to minutes every time you decide to upgrade.

Cacheing common reports from a database inside the database itself is surprisingly easy and makes a huge difference for Ajax style web pages.

I have several systems that record data at regular intervals. For the sake of example let us consider a weather station which reports temperature, humidity, pressure, and wind every 10 minutes. If I want to graph this data for a three day period, I have to query out 432 of these rows and send them down to my browser. Unfortunately this is not fast enough.

Step 1: Get the browser cache working for me.

If I break the request in to midnight aligned 24 hour periods, then I can cache the result for any completed day. This way I only need to pull new days of data. 

This helps, but it turns out I don’t revisit days often during a session.

Step 2: Tune the database indices.

Fail miserably. It turns out SQLite does my query faster without indicies, so I took them off completely. (Sequential read on my virtual server is much faster than random.)

Step 3: Server side cacheing. In the database.

Now we get to the meat. I can cache pre-compressed reports for each of my daily periods. There are a couple of wrinkles though. I need some way to invalidate a report when the underlying data changes. (Sometimes some observations can be delayed and trickle in later.) I can’t think of a good way to have the database delete a cache file, so instead I store the cached copies in the database.

This turns out to be surprisingly clean to code, and a simple set of triggers on the underlying data can remove any affected report.

It would be more efficient to keep the cached reports as files, or better let in a httpd cache, but then they could not be invalidated by the database. 

Step 4: Clean up the little PHP wrapper and stick it here.

I’ll get to that.

Certain corrupt JPEG files will explode Quicktime Player if it encounters them in an “Import Image Sequence…” operation. This bug has been present for many years and shows no sign of going away.

If you are archiving images from cheap webcams you will encounter these.

I am not aware of anything included with Mac OS X Leopard that validates a JPEG file, so I built jpeginfo for Leopard. 

With that, you can do things like… for v in *.jpg ;do jpeginfo -c \$v || rm \$v ;done … to delete all the corrupted JPEGs.

You can find the sources at http://www.iki.fi/tjko/projects.html You will also need to build libjpeg, from ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz

Build libjpeg first (./configure ; make ), then build jpeginfo (./configure –with-libjpeg=THE-RIGHT_DIRECTORY ; make )

But you don’t need to do that… I’ve attached the copy I built.

Safari is odd with contentEditable divs. It doesn’t assume you will have a pre-formatted div as a container so it puts each line into its own div… sometimes marked with code class, sometimes not.

That makes a mess but is tolerable, until my HTML sanitizer burps out the clean version with newlines between the divs and causes accidental double spacing.

I could rewrite the ->saveHTML() method of the PHP DOMDocument to not put newlines in between divs, but that would give awful looking HTML.

For now I added a function to the sanitizer to remove extra divs from inside code formated divs. It tries to be smart about inserting newline characters, but it may not be smart enough.

Someone should revisit the whole contentEditable thing and specify precisely what is to meant by all of the operations.