Jim's Depository

I have written a program and some javascript to add alpha channels to JPEG images and render them correctly in modern browsers. 

The exposition requires some Javascript which femtoblogger does not allow, so I have documented the techniques on their own page: On adding alpha channels to JPEG images.

The code isn’t pretty yet, but it works.

I apologize for not testing on IE. The canvas based method should work with FlashCanvas per http://blog.jackadam.net/2010/alpha-jpegs/, but I don’t have the licenses to run the IE test images. (Wine will unpack the self extracting .EXE they insist on using as a shipping container, but VirtualBox has different emulated hardware from what the images expect and that leads to no network which cascades into avalanches of dialogs asking to download network drivers over the network that doesn’t exist and demands to register over the network with does not exist.)

… and how to fix for Debian x86 users…

Anyone with a Western Digital Green Power drive should immediately check the Load_Cycle_Count in their S.M.A.R.T. data. This is the number of head parks. The drives are rated for 300000 over their lifetime, but the firmware parks the head after 8 seconds of idle time. If your computer wakes it up regularly you could wear out in a matter of weeks. Don’t panic a lot if yours is high. Mine drives varied from 30k to 1.5M before I figured this all out and stopped them from parking.

I know this because one of my rather new WD10EADS drives crapped out, causing a fairly unpleasant interruption in my mirrored root partition.

You can not use the standard ATA mechanisms for tuning drives to adjust this parameter.

There is a utility from Western Digital that you can use to change the timeout from 8 seconds to 300 seconds, which is probably enough to keep it from parking the head. Now, if you could run this on any operating system sold this millennium it would be pretty handy. Sadly, it runs only under DOS.

I’ve fixed the drives in three different computers, (and failed on three other drives), and it is an ugly process.

For people running Debian servers, these instructions will guide you through making a USB flash drive to boot DOS and run the WD utility.

• Go to wdidle3, download the zip and unpack it to get the .exe.
• Get a flash drive. I used a 256M one.
• Put a MBR on it. Install syslinux-common, then dd /usr/lib/syslinux/mbr.bin onto the first block of your device.
• Download a bootable image of freedos, perhaps this one: balder10.img
• Boot a fake dos machine to format the usb key…
  qemu -boot a -fda balder10.img -hda /dev/sdX -k en-us
• Use fdisk to make a single partition on the device. Then use format, and finally sys c:
• Quit qemu, mount the /dev/sdX1 partition somewhere and copy the wdidle3.exe program on there too.
• Now the ugly part comes up: Your BIOS is not your friend here. You will need to turn your SATA controllers to Legacy IDE mode. You will need to force your USB device to be handled as a hard disk, not a floppy. And of course make it boot off the USB.
• wdidle3 /s300
• wdidle3 /r
• Dance, you deserve it.
• Now remember to undo all the damage you wrought on your BIOS before you boot back to Debian.

Now, where you will fail is where you have a drive in a USB sled. Even my fancy USB sleds with S.M.A.R.T. support won’t let the drives be fixed. I suppose I need to take all that gear apart and temporarily rotate the drives through a server to repair them.

After piddling around with jQuery,CSS3, and the <canvas> tag for a week using only Safari 5 I decided to see how bad it was in Firefox. The comforting result was “not bad at all”.

Initially nothing rendered right, but that was all the first problem.

• I was using a <base> with a relative URL, nice if you test in a subdirectory of your real server, but a mortal sin in the eyes of the W3C. Firefox caught me and ignored the directive.
• My round corners and drop shadows were missing, but that is just because I didn’t add the -mozilla- variants. (I suppose I’ll make a CSS preprocessor to handle that as well as stripping my comments and white space.)

• My @font-face fonts didn’t render in the <canvas> elements. The problem here is that there is no good way to tell a <canvas> tag to make sure it has the font loaded before it tries to get draw, and no way for your drawing code to check if the font is ready. (Safari can fail this race condition too, but it wasn’t happening to me so I didn’t notice it until I provoked it.)

  My solution is to defer any <canvas> drawing that uses text until after the $(window).load() fires. (Which you also can’t test for, and unlike .ready() won’t fire if the window is already loaded, so you have to add one to set your own ‘loaded’ flag.) This seems robust.

All in all I was pleasantly surprised. Two years ago the first browser change could be counted on to be a nightmare. Good work browser vendors!

The page in question looks like this:

• Those meters are fully vector drawn in <canvas> at twice screen resolution to control pixelation artifacts and ensure a nice experience on \~300dpi devices. (\~12kbytes unminified and uncompressed, but that includes the pushbuttons and indicators as well which share code but aren’t on this page. Maybe 4k minified and compressed.)
• The fonts are Droid and Molengo out of the Google Font Directory. A nice way to get some new fonts in play, but also have a high likelihood of a user already having them cached and serving them off a different hostname for more parallelism if not cached. (90k bytes)
• I spent about an hour looking for a nice tileable brushed metal before giving up (after finding one only to discover the author didn’t know what tileable meant). I made this one in 5 minutes with gimp. (3.8k bytes)
• Those meters also have color coding, but the sun is up and the batteries are full, so none of them are yellow or red.
• Drawing the meters in <canvas> takes less bandwidth than sending images. For even a single meter, the code is smaller than the image would be, plus the code caches, so subsequent meter updates are nearly bandwidth free.
• All told, the sources to that page are larger than the PNG screenshot, but the bulk of that is the fonts (90k) and jquery (22k). The uncached portion is 2.5k(800 on wire) for the index.html and 600 bytes for the AJAX query for the current data. Each of these comfortably fits in a single datagram.

My new Western Digital Green 2TB (WDC WD20 EARS-00MVWB0) drives have 4k sectors, but they report as 512 byte for compatibility.  Linux 2.6.32 and the fdisk on squeeze will do the wrong thing by default and place partitions on bad boundaries.

fdisk -b 4096 -u /dev/XXXX

… is the right command to force the proper sector size. It will then die with a floating point exception unless you type a ‘c’ to disable MS-DOS compatibility mode once you are in fdisk.

(If you are just going to use them with LVM, maybe skip the silly label anyway and use the whole disk for LVM. Be careful though, currently on squeeze the lvm scan does not happen after the USB scan, so they won’t be found at reboot unless you do another scan, like in rc.local.)

Today, my gift to you: After extensive research and testing, I have found a 3.5” SATA⇒USB enclosure that will work with smartmontools to allow you to monitor the S.M.A.R.T. data of your drive. This one may be unique in that I have an actual link you can click and buy the enclosure! 

You can get them from Amazon for \$30: Amazon.com: Sabrent eSATA to Serial ATA (SATA) and IDE(PATA) USB 2.0 3.5-Inch Aluminum Hard Drive Enclosure Case

Ugly Caveat: There are at least two different designs sold under the same model number. Some, possibly older ones though 2 of the 3 I got from Amazon today are this variant, have silver screws and absolutely awful power adapters with white labels. The part where the 120v cord enters is badly made and the cord will not make contact or make intermittent contact unless you cram it in very hard. The variant with black screws has a different model of power supply and seems to be ok. The box art work matches the unit inside. Both variants have the JMicron JM20337 chipset so all is good.

I’ve got a Western Digital Green 1TB (older model, 512 byte sectors) running in there and smartctl -d usbjmicron -a /dev/sdg is working perfectly on my Linux box.

(smartmontools 5.38 from Debian Lenny was not new enough. I think 5.39 would be, but I jumped straight to the dev edge.)

I couldn’t get S.M.A.R.T. to work with this drive on Mac OS X, same version of smartmontools, but it doesn’t want to talk to the drive.

I was experimenting with HTML-5 Web Storage when I realized Safari wasn’t showing me my storage use under the Safari⇒Preferences…⇒Security⇒Show Databases panel. A little find action shows that they live in databases named things like ~/Library/Safari/LocalStorage/http_SITENAME_0.localstorage, and good heavens there are a lot of them!

Most of them are for some sort personal information harvesting operation named loomia. They seem to be about personalized recommendations, but I don’t remember authorizing them to collect information on me, and I sure didn’t ask them to tag me with a tracking code tucked away in a location I can’t see that will survive cookie clearing.

Most of the rest are of the form IXAIInvited{number} = true. I have no idea what that is about, but the number isn’t long enough to be a unique identifier for me.

So Safari guys: I know you’ve been busy, but how about letting me see and manage my localStorage from the user interface?

(And while you’re at it, either make localStorage.setItem() work for objects, or update the documentation to mention that it is strings only. And give me my hour of debugging time back.)

I ordered a pair of Kodak Pulse 7" wifi picture frames from Provantage about 90 days ago which was long enough to forget, so it was like a little surprise gift to myself.

I intend to rip them open and see if I can replace their firmware with something I can control, but I fired one up to see if it could be made to work as a remote webcam viewer with its intended firmware.

I have to say I am impressed. The picture quality is quite nice. The web site for controlling the content is very nice (except that their applet signature does not have a valid signature). You can feed it from a web page, email, facebook, or some Kodak online gallery.

But here is the thing that makes me feel best about it:

On the support system they have an actual, honest, system status display. Very classy.

The 30 minute hacking report is:

• I don't think I can make it display a once/{timeunit} update of a webcam using the intended APIs. I could mail a new image to it regularly, but it doesn't display the latest one by preference. There is not a way to email a "delete". - It contacts Kodak home base (running on Amazon's EC2, S3, and CloudFront) once per minute where it does:
  1. Some secret SSL shrouded query that they don't let me see. 2. An NTP lookup from a random server on the planet. 3. An HTTP query to see if it needs an update, nice XML response. 4. If needed, the files are then requested by a UUID and come down by HTTP. - Blocking the SSL shrouded query stops the others, but I'm pretty sure I could make a proxy to replace the content with my own. This however only works for picture frames in my domains where I control the routers and can intercept. - I captured its firmware upgrade. The file will be useful.

Looking forward:

• I need to look at Amazon data rates and see if it would be abusive to use their browser API to upload and delete pictures, say once a minute all day. I'm sure that's more traffic than they expect, but perhaps the total cost is small enough to be ethical. - I'm itching to pop apart the 2nd unit, but I will wait until I have time to photodocument to see what is on the inside of these things.

I posted my code optimization with explanations regarding assembler etc at http://geekofpassage.blogspot.com/2010/05/win2k3r2ee-virtual-appliance-running-in.html

The unified diffs are available there too.

Cheers!

The available pages I can google about this seem overly complicated. This one documents setting up nginx to run lua scripts on Debian Squeeze. These instructions are derived from http://www.unreliablepollution.net/p/other/howto-nginx-and-lua and a dozen remedial googleings.

1. aptitude install nginx
   The config in /etc/nginx is good in squeeze. We will need to add the fastcgi handler, but that is to be expected. (I edited it to pull the root up to /var/www instead of in an nginx-default subdirectory.)
2. aptitude install liblua5.1-wsapi-fcgi-1
   There is also a ‘0’ version, but I took 1 on the theory that newer is better.
3. aptitude install liblua5.1-coxpcall0
   liblua5.1-wsapi-fcgi-1 probably needs a dependency on this. Things will go poorly for you without it.
4. aptitude install lua5.1
   Can’t forget that.
5. aptitude install spawn-fcgi
   Older instructions talk about libfastcgi-dev, but it isn’t available and I think they just wanted spawn-fcgi out of it anyway.

6. Put a Lua file in /var/www/hello.lua
   The WSAPI is a little strange. Apparently you don’t just print your output, but rather coroutine.yield() it. I used this example program from http://keplerproject.github.com/wsapi/manual.html …

   module(…, package.seeall)

   function run(wsapi_env) local headers = { [“Content-type”] = “text/html” }

   local function hello_text() coroutine.yield(”<html><body>”) coroutine.yield(”<p>Hello Wsapi!</p>”) coroutine.yield(”<p>PATH_INFO: “ .. wsapi_env.PATH_INFO .. “</p>”) coroutine.yield(”<p>SCRIPT_NAME: “ .. wsapi_env.SCRIPT_NAME .. “</p>”) coroutine.yield(”</body></html>”) end

   return 200, headers, coroutine.wrap(hello_text) end

7. spawn-fcgi -a 127.0.0.1 -p 9100 -F 4 -- /usr/bin/wsapi.fcgi
   Now we have handlers on port 9100. This will need to be in /etc/init.d/blahblahblah but this will do for now.
   Note: spawn-fcgi silently fails if you get the command wrong. Do a “ps” and make sure they are there. You will have a section like this if it succeeds…
   18554 ?        Ss     0:00 lua5.1 /usr/bin/wsapi.fcgi
   18555 ?        Ss     0:00 lua5.1 /usr/bin/wsapi.fcgi
   18556 ?        Ss     0:00 lua5.1 /usr/bin/wsapi.fcgi
   18557 ?        Ss     0:00 lua5.1 /usr/bin/wsapi.fcgi

8. Edit /etc/nginx/sites-available/default to know how we want .lua files handled. I put in this section
   location \~ \^(.+\.lua)(.*)\$ { root /var/www/; fastcgi_pass 127.0.0.1:9100; fastcgi_index index.lua; fastcgi_split_path_info \^(.+\.lua)(.*)\$; fastcgi_param PATH_INFO \$fastcgi_path_info; fastcgi_param SCRIPT_FILENAME \$document_root\$fastcgi_script_name; include fastcgi_params; } Be careful! Make ‘root’ match your root!

9. Restart nginx. /etc/init.d/nginx restart

10. Load http://YOURSERVER/hello.lua in your browser. You should get something along the lines of:

    Hello Wsapi!

    PATH_INFO: / SCRIPT_NAME: /hello.lua

11. Celebrate!

Now my thoughts: This may be more than I wanted. I have to think about the implications of a scrutinizer layer above my Lua program. I really wanted to just blast my standard output back up the fastcgi socket. All this yielding seems strange.

3:14am, spring daylight savings time day. The church bells across the street have been ringing for 14 minutes.