Jim's Depository

I’m entering into Virtual Hosting. Many of the pages and services I’ve run out of spare gear at FSG are now moved or moving to a virtual host. I chose VPSLink for their \$7/mo tiny server, then while moving my domain name pointers discovered that Gandi.net is starting a virtual hosting business with more RAM and disk for about the same price.

Living in a tiny slice of a virtual host is certainly different from living in a leftover 2GHz/1GB/500GB PC, fortunately I remember when 64M of RAM was huge so I think I’ll get by just fine.

• postfix/dovecot in TLS, lighttpd+fastcgi+php5 all fits nicely.
• TLS incurs a fair bit of RAM use.
• CPU use is odd. I don’t have visibility to know if I am CPU bound. I don’t think I am, but I can’t see from inside my Xen box.
• The machine is noticeably sluggish at things like an initial rsync backup offsite. Slow disk? Slow network? I’m not sure.
• The total annual bill will cost less than the electricity to keep a PC powered up. Happy Earth Day!

I guess if you find the site gone and just this at archive.org you’ll know I’ve had a virtual hosting disaster.  I don’t foresee a problem. I don’t know what the hosting company does if a machine fails, but I keep a full backup every night in my closet so I’ll be ok. 

VHD is a nice format for exchanging disks. It doesn’t take up a lot of bytes with the empty space of a filesystem. But if you want to access the data without a virtual machine it can be a bit of a pain.

The attached program will expand most VHD files into a sparse image of the disk. You can then do whatever you wish with that, such as:

• Use it with a virtual machine that takes raw images.
• Convert it to a VMDK with qemu-img from the qemu folk. (Note: some people say qemu-img can read VHD format already. It didn’t work for me and the man page doesn’t mention it.)

• Mount it on your Linux box, though you have to skip some bytes to get over the MBR and down to the first partition. Go look at Creating and using disk images mini-howto | Marc’s Realm and find “The dirty way”. Basically it boils down to trying all the plausible offsets until you find the right one:

  for ((i=0 ; $i < 10000 ; i=$i + 1)) ; do    mount -o loop,offset=$(($i * 512)) image.bootable /mnt && breakdone
  

• Give it a sound grepping.

Note: I don’t support this program or even intend to run it ever again. I converted my data. If you need to convert your data, then enjoy.

Sometimes you will find data structures to read that are encoded little-endian. You may find these functions useful. They aren’t as nightmarish to read as macros and presumably your compiler will do right by them.

#include <stdio.h>
#include <arpa/inet.h>/*
** Little endian to host, short
*/
static unsigned short ltohs( unsigned short v)
{
    if ( htons(1) == 1) {
    return ((v>>8)&0xff) | ((v<<8)&0xff00);
    } else return v;
}/*
** Little endian to host, Long
*/
static unsigned long ltohl( unsigned long v)
{
    if ( htons(1) == 1) {
    return ((v>>24)&0xff) | ((v>>8)&0xff00) | 
               ((v<<8)&0xff0000) | ((v << 24)&0xff000000);
    } else return v;
}/*
** Little endian to host, "double" i.e. long long
*/
static unsigned long long ltohd( unsigned long long v)
{
    if ( htons(1) == 1) {
    return (unsigned long long)ltohl( v&0x00000000ffffffff) << 32 | 
        (unsigned long long)ltohl( (v>>32)&0x00000000ffffffff);
    } else return v;
}int main( int argc, char **argv)
{
    unsigned char t1[] = { 0x01, 0x02, 0x03, 0x04, 
                           0x05, 0x06, 0x07, 0x08 };
    unsigned char t2[] = { 0xff, 0xfe, 0xfd, 0xfc, 
                           0xfb, 0xfa, 0xf9, 0xf8 };    printf("ltohs(01,02) = %04x\n", ltohs(*(unsigned short *)t1));
    printf("ltohs(ff,fe) = %04x\n", ltohs(*(unsigned short *)t2));
    printf("ltohl(01,02,03,04) = %08x\n", ltohl(*(unsigned long *)t1));
    printf("ltohl(ff,fe,fd,fc) = %08x\n", ltohl(*(unsigned long *)t2));
    printf("ltohd(01,02,03,04,05,06,07,08) = %016llx\n", 
           ltohd(*(unsigned long long *)t1));
    printf("ltohd(ff,fe,fd,fc,fb,fa,f9,f8) = %016llx\n", 
           ltohd(*(unsigned long long *)t2));
}

Temperature is the enemy of hard disks. Hot disks fail sooner.

If you have a well designed server with lightly loaded disks in a cool room your drives are probably running around 25°C(77°F). If you have a 4U box that you chucked full of drives you may find some of your drives running much hotter. For instance, I just found two running at 41°C(105°F) in a box with otherwise cool drives.

Drives are typically specified to operate up to 55°C(131°F), but their lives are shortened. Estimating from Google data it looks like your failure rate within three years is about triple for 45°C versus 25°C. In the third year, about 1 in 6 of the hot drives will fail.

So protect your servers:

1. Know the problem: In debian land, install hddtemp and run it to see which drives are hot. Windows users might use DTemp.

   `` vev# for v in a b c d ;do hddtemp /dev/sd\$v ;done /dev/sda: ST3750640AS: 34°C /dev/sdb: ST3750640AS: 27°C /dev/sdc: ST3750640AS: 41°C /dev/sdd: ST3750640AS: 40°C vev# # I have two hot drives.

2. Mitigate: If you have hot drives, move them around or adjust airflow, perhaps by making little cardstock air dams inside the server to cool the drives. Add a fan blowing on them if you have a large case.

3. Monitor: Record their temperatures and check in on them once in a while to make sure things aren’t going badly.

Debian users might also want to install smartmontools which will track your S.M.A.R.T. data and notify you of problems.

Note: for SATA drives you will need a “-d ata” or it will misaddress them as SCSI drives and you need a “-m foo@example.com” if you want to be email notified.

Somewhere along the line the GNU C library and the BSD folks added variants of sprintf() that malloc() the resulting string. Very handy for avoiding an overflow situation and also makes for cleaner reading code.

`` int asprintf( char **ret, const char *format, …); int vasprintf( char **ret, const char *format, va_list ap);

Before:

`` char buf[32]; sprintf(buf, “s:%d m:%s”, code, msg); // stack overflow when msg is too long

… or …

``

char buf[1024];
snprintf( buf, sizeof(buf)-1, “s:%d m:%s”, code, msg); // fail oddly if we guessed wrong on the maximum buf size

After:

``

char *buf;
asprintf( &buf, “s:%d m:%s”, code, msg);
… free(buf);

Danger danger DANGER: This will not work on the tiny libc in OpenWRT and similar embedded systems. But at least you will get a link error during the build.

Now if only we had a scanf() that would do a realloc() on its pointers.

Femtoblogger has just been rolled out to a corporate community of 150 users. Let the bug reports and feature requests roll.

Perhaps it isn’t fair, firefox 2 is the oldest of the browsers I am supporting, but could they make the editable html interface any less user friendly? 

The editable region has to be in an IFRAME, I’d rather not restrict people to a fixed region of screen, but ok. Now the user clicks in the IFRAME to edit, but it won’t place the insertion point unless they happen to click on an element, so if you click in the bottom of the IFRAME you get no insertion point. That makes me create a special onfocus handler to place the insertion point someplace to get them started. Then in one final gesture of hostility, the insertion point placing code doesn’t work on an empty IFRAME, so I have to insert a nonbreaking space to open up a line for the insertion point (because a normal space gets an insertion point about 2 pixels tall.) All told, most of a day wasted to support  bad decisions from the firefox coders. They should have tried to use it after they coded it.

I recently needed to install 10 motion detecting cameras on 5 doorways. My first thought was to use ethernet cameras, but at \$300 each that gets expensive. My second plan was to use Linksys WRTSL54GS units running OpenWrt and cheap web cameras with the ‘motion’ package. That would sort of work, but it takes far more CPU than those little boxes have.

The problem is that all that JPEG decoding, motion detection on 640x480 pixels, and JPEG encoding is just too much computation. Fortunately we can be simpler.

Many cheap web cameras have JPEG encoders built in and deliver their images as JPEGs, though sometimes missing a critical table. By keeping a copy of the original camera data it is possible to avoid the JPEG encoding step and just insert the missing data table.

We can do better though. We can avoid most of the decoding and also reduce the data for motion detection by a factor of 64. The key is that luminance of JPEG files is encoded in 8x8 pixel blocks. Each 8x8 pixel block has 64 coefficients which are used to regenerate the 64 pixels of data (more on these coefficients later). The first coefficient is the ‘DC’ coefficient, the average luminosity of the 8x8 block. This is outstanding for motion detection! We get a low pass filter, a factor of 64 data reduction, and all we have to do is not perform a bunch of multiplication and addition in the decoding process.

With a process like the following the tiny router hardware can each support two cameras at 640x480, color, 10 frames per second, motion detection on 5 frames per second.

1. Read JPEG image from camera over USB.
2. Hold a copy of the image.
3. Decode the image enough to extract the DC coefficients of the luminance channel.
4. Compare to the last frame’s coefficients and decide if there is motion.
5. If we are going to save the image, then insert the missing JPEG tables into the image if needed and write it out to storage (NFS in my case).
6. If motion has stopped, then write a sentinel file to independent processes examining the image streams know that the motion event is complete.
7. Repeat forever.

Astute readers will notice that I can only afford to motion check every other frame with the two camera setup. I’m not happy about this. Essentially all of the CPU time is used in the Huffman decoding of the JPEG data. A long time ago in the age when Vax 8530s roamed my world and I was busy trying to move X-rays and MRIs to radiologists I wrote a state machine based Huffman decoder that could process 3mbits/second of input data, the fastest we could get over Ethernet at the time. Those were 4 MIP machines, these little routers are something like 200MHz. Each camera is generating about 4mbits/second. I have high hopes that this will be doable.

I have other fish to fry and probably won’t get around to the faster huffman code anytime soon. After I run these things for a week or so I’ll release the code in case anyone else wants to read it or use it. I attached the man page in case you wanted to peek at it.

Update: Looks like I have an evil twin in Alexander K. Seewald who has written autofocus software using the AC coefficients. I like his Huffman decoder. I must benchmark it against the slow one I’m currently using to see the range of differences.

When writing daemons there is always the question of “How will I check on the status as it runs?“. Solutions include:

• Have a debug flag and write to stderr.
• Periodically write a file in a known place.
• On a particular signal, write a file in a known place.
• Keep a socket open to write status if someone connects.

I add one more:

• Have an HTTP interface and let people talk to you with a browser.

libmicrohttpd is just the thing for embedding an HTTP server in your daemon. It adds about 25k to your code, a couple lines to start it and then a dispatching function you write to handle the requests.

It has three models, the least obtrusive of which it is to just let it make some threads and live in its own threads. The others are useful if you can’t tolerate threads, but you have to cooperate to get select() calls made.

I don’t think you’d want to let it take a slashdotting, so either keep it behind a firewall or check the incoming IPs.

It seems to be solid and well written, but I did notice that it raises a SIGPIPE that kills the host application when a connection closes in the middle of a request. (3 word fix in the source. Add MSG_NOSIGNAL to the SEND() calls) That was found in the first minutes of testing, so it doesn’t get the “bullet proof - use and forget” stamp of trust. Maybe after 6 months of daily use it can earn that.

openload, which is called openwebload at sourceforge, is a simple URL load tester. Just the right sized tool for checking if your spiffy new web page is efficient enough to be tolerated.

\$ openload localhost 10 URL: http://localhost:80/ Clients: 10 MaTps 355.11, Tps 355.11, Resp Time 0.015, Err 0%, Count 511 MaTps 339.50, Tps 199.00, Resp Time 0.051, Err 0%, Count 711 MaTps 343.72, Tps 381.68, Resp Time 0.032, Err 0%, Count 1111 MaTps 382.04, Tps 727.00, Resp Time 0.020, Err 0%, Count 1838 MaTps 398.54, Tps 547.00, Resp Time 0.018, Err 0%, Count 2385 MaTps 425.78, Tps 670.90, Resp Time 0.014, Err 0%, Count 3072
<You press enter to stop the run> Total TPS: 452.90 Avg. Response time: 0.021 sec. Max Response time: 0.769 sec

There is a Debian package.

Update: I couldn’t leave it alone. The attached patch will add bits/second reports. It is also worth noting that you can add  ”-l number” to sample for a certain number of seconds, and a “-o csv” to put out a CSV line of the data at the end for easier scripting. Perhaps it needs a man page too.

Update: If you are an especially trusting sort of person you could install the Debian package attached and not have to build it yourself.

Update: Best wait on this. There is something odd going on with openload that is giving unstable results for bytes transfered. More diagnosis required.