Jim's Depository

The Swift server packages are documented with the inline comment documentation system. I went ahead and used it for my recent test mule project (femtoblogger, you're soaking in it).

It's easy enough to add documentation, Xcode has a CMD-click Add Documentation menu that will jam in a template for your function with spaces for all the parameters and return types. You just type markdown after that. If you add an argument, click it again. Xcode will add more template.

Now people can Option-click on your function name (or whatever) and get a lovely Quick Help pop up.

I appreciate this greatly for libraries.

But…

There doesn't seem to be any pathway from this snippets to a real document where the big picture is explained, and all the functions listed in well organized categories. You know,… like the good old Apple documentation.

"Good enough" is the enemy of "Good".

This isn't a special flaw for the Swift comment based documentation. Essentially all in code systems I've ever used have suffered from this. (Halfway excepting one I wrote, but you can't use.)

The Moral

• Do your "in code" comments.

• Update your "in code" comments when you change the code.

• Write book like documentation somewhere! and put the links to it in HUGE type in the middle of your project page.

Well that was a fair bit of gap in between posts.

Femtoblogger is now on its 4th implementation. I feel the need to try out Swift on the server, so here we are. 100% pure Swift Femtoblogger.

The previous incarnation was Go. Before that was PHP. There was probably a never deployed Python version in there too.

Thoughts on Swift and the design of the server this time around…

• I very much like the type safety of Swift. When I code in Swift there are no runtime explosions, and very few bugs discovered in testing.

• The Swift code is more information dense than the Go code was. The Go code felt like a great sea of text with a little bit of content. The Swift code packs a lot of content into a screen of code.

• Performance appears to be a non-issue. SQLite backed pages are serving in 3-12ms. For comparison I added a caching layer for most popular queries and dropped that into the 80µs range for the query handling. That's not real, I handle them by doing a X-Accel-Redirect header back to Nginx and letting it serve a file out of a RAM disk directly. But it means my process is not going to be buried.

• I haven't bothered with any compiler optimization. I want the checks to stay in place while I see if I've screwed up somewhere.

• It's a small thing, but Swift Argument Parser really got things right. Very easy and natural to use.

The Network End of Things

• The Swift server network library, NIO, really wants to push you into a futures and promises style of programming. I resisted because it is so clumsy at the source code level. Your code begins to vanish under a sea of ritual. That said, I probably would have been better off to embrace it. A request needs to get out of the network queues to free them up while it gets in a non-concurrent model queue to work with the data, then out of there as soon as possible and into a render queue to do template expansion and back into the network queues to send the data out. It's a lot to coordinate.

• I suppose I'll continue avoiding futures and promises in femtoblogger until Swift 6 and the beginning of its new concurrency model make it into NIO.

• I didn't use Vapor or similar web frameworks, I build my server directly on NIOHTTP1 so I'd understand the architecture. The first 90% was simple! Then I needed to parse some form data and couldn't find the calls in the API. Eventually I found an email in a filing cabinet behind a sign that said "Beware of the Leopard" where the NIO team proclaimed that "out of scope". Maybe I'll make a post on parsing "multipart/form-data" (which can't be a String because it might not be valid) and parsing HTTP header values.

Using Other People's Packages

• I did reach out for 'random dude' packages for a higher level SQLite API, Markdown to HTML conversion, and template expansion. It's a bit of a mixed bag there. I'm glad I did it, but there were issues. As always it seems to take a lot of research time to figure out which solution is well supported, not horrible, and will likely last a while.

• I used SQLite.swift for accessing SQLite. The tutorial writing crowd on the internet likes it. It gives you a type safe way of interacting with SQLite without too much ritual. The package is a few years old and has been dormant for most of a year. I found a design flaw where it would try to read a mismatched datatype and explode your process. (Just because you called a SQLite column TEXT doesn't mean it isn't going to return a BLOB for one of the rows.) It looked to be an involved fix to use the receiver's datatype instead of the provider's, and in the absence of an active maintainer I didn't want to invest in making the changes. It was easier to rewrite all the rows to have their TEXT really be TEXT.

• I used Ink for Markdown to HTML conversion. It is simple to use. Out of the box it choked on CRLF line endings. The patch ended up being a , "\r\n" in just the right place, plus about 16 lines of test cases. Hint: "\r\n" is a single Character in Swift… grapheme clusters and all. The maintainer is active so I sent that one in.

• I used Stencil for template expansion. I've used it before and the rough edges I used to have to hammer off are now addressed. Note: Stencil could really use a SafeExpansion mode which escapes HTML for when you have untrusted users. It has user created filters, I ended up making one instead of forking and adding a safe expansion feature.

When your friend's laptop is at full disk, stop them from trying to make space by going through their tiny documents and deleting things that will be missed or finding that pesky Library folder that contains a bunch of gunk they never wanted and deleting that.

Go to the Mac App Store and download Free Disk Space. It will show what is really taking up all the space and should be dealt with.

Personally, I've used Space Gremlin for years, but Free Disk Space is free and does the job.

Update: Apparently Free Disk Space decided money is good. They are $10 now,

I’ve been using USB cameras on tiny repurposed routers, beagle bones, and raspberry pis for some time. I’d like to deploy more cameras, but I am on a tight power budget where every watt matters and my DC power controller that allocates the power is out of ports and I don’t really want to build another one.

Enter “power over ethernet” (PoE) cameras. These hang on an ethernet cable and receive their power from that cable so I only have to run out a cat5 and I’m set. I can buy ethernet switches with PoE and control which devices are powered at any given time.

Danger: most lower cost “smart” switches allow you to control the port power, but only through their silly web interfaces, they don’t allow it over SNMP until you pay for an enterprise device. It appears SMC has a model that will allow SNMP control.

In the made up land of retail pricing, PoE cameras are a little pricey compared to their less capable friends. Fortunately China is a really big place and lots of things are made and sold there. I found 720P PoE cameras for $25 each delivered half a planet away.

Good: They have pretty good picture quality, aluminum “weather proof” enclosures, IR lights for night time, and you can order with four different focal lengths. H.264 over rstp.

Bad: They really sad documentation, on a mini-CD, like I own a tray loading CD drive. Are hardwired to boot on 192.168.0.103 despite documentation that gives a different address, have an open telnet port with a root account backdoor with fixed password, respond with a web interface only in Chinese that doesn’t let you configure the camera in Chrome and Safari. Chrome will offer to guess at translations and turn the text back near English.

That “Bad” entry took me about 3 hours to work out. I do not have the root password, but apparently some tech support responses give it to you.

Proceeding onward, I got Microsoft to give me a demo copy of Windows 8.1 and spun that up in a VirtualBox. If you hit the camera with Internet Explorer, enable Active-X downloads from unknown and untrusted developers and basically give total control of your machine over to unabashedly anonymous people, then you get a web interface that can be used in English and can tell the camera to use DHCP or some static address of your choosing. (I use DHCP for everything and then tell my DHCP server to give a static address to certain MACs.)

The rstp streams can be found at rtsp://camera.example.com:554/user=ACCOUNT&password=PASSWORD&channel=1&stream=0.sdp?real_stream--rtp-caching=100 for the high res stream, and stream=1 for the low res.

Unresolved: I’d like to find a URL to pull a single frame. Perhaps if I figure out how to query the ONVIF interface enough it will tell me.

Further thoughts: It is hard to fathom how someone can sell me a computer, with a camera for $25, in a nice aluminum case and ship it half a planet on demand. Because I need individual control of the power on the switch ports, I’m spending almost that much on the hole in the switch!

Update:

These were garbage. They weren’t weatherproof in any meaningful way and rapidly succumbed. They also have hard coded root backdoors in them.

The following modules need to be added to the Debian init ramdisk to support root on a 9p disk:

./kernel/fs/9p/9p.ko
./kernel/net/9p/9pnet.ko
./kernel/net/9p/9pnet_rdma.ko
./kernel/net/9p/9pnet_virtio.ko

To add these modules, you will extract an init ramdisk (which is just a compressed cpio archive), add the files, and run a depmod -a in it. Then re-archive it.

The commands I used to make mine are:

#!/bin/sh

set -e

mkdir /tmp/9p-ramdisk
cd /tmp/9p-ramdisk
zcat /boot/initrd.img-3.16.0-4-amd64 | cpio -id

mkdir lib/modules/3.16.0-4-amd64/kernel/fs/9p
mkdir lib/modules/3.16.0-4-amd64/kernel/net/9p

cp /lib/modules/3.16.0-4-amd64/kernel/fs/9p/9p.ko lib/modules/3.16.0-4-amd64/kernel/fs/9p/9p.ko 
cp /lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet.ko lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet.ko 
cp /lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet_rdma.ko lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet_rdma.ko 
cp /lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet_virtio.ko lib/modules/3.16.0-4-amd64/kernel/net/9p/9pnet_virtio.ko 

depmod -b /tmp/9p-ramdisk

find . | cpio -o -H newc | gzip > /boot/initrd.img-3.16.0-4-amd64-9p

Update: This works, but seems to be a bad idea. When I try to do aptitude update it fails with lines like this…

E: Unable to determine file size for fd 7 - fstat (2: No such file or directory)
E: Problem opening /var/lib/apt/lists/ftp.debian.org_debian_dists_jessie_contrib_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.
E: Unable to determine file size for fd 8 - fstat (2: No such file or directory)
E: Problem opening /var/lib/apt/lists/ftp.debian.org_debian_dists_jessie_contrib_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.

… which seems to say that 9p has a problem with some operation aptitude needs.

Back to disk images for me.

If you have an old HP plotter, you will find that the Mac OS X drivers available from HP’s support site can not be installed because they are in an old VISE .hqx format installer.

If you go and get a generic PPD for the printer, and stick it gzipped in /Library/Printers/PPDs/Contents/Resources then you will be able to select it as a printer type when you add the printer.

Attached is the one I found and used. It needs more work, I am getting postscript errors at the end of prints which can use a lot of paper. But it is working.

Update: It looks like you can disable postscript error printing with…

defaults write com.apple.print DoPostScriptErrorHandler NO

…and re-enable it with…

defaults delete com.apple.print DoPostScriptErrorHandler

Update2: The defaults thing does not appear to work.

Update3: Using HP JetDirect as the protocol instead of LPR seems to have fixed the PostScript error problem.

I’d just like to mention that when you use QEMU from a Macintosh running OS X, using the regular Terminal.app terminal, the terminal type will be listed as xterm-256. This will make your monitor console be black on black no mater what you do, except that making the background transparent will make the text somewhat visible.

The “solution” is to change your terminal type to regular old xterm. I’m doing something likes this on my QEMU commands…

$ TERM=xterm qemu-system-i386 ...the-rest-of-my-arguments…

I made a video of my cat chasing the shadow of her tail. It got posted to a social media site. I have contributed to videos of kittens.

If you find yourself making ajax requests for HTML pages, then using jquery to massage them or extract data before displaying it, you should be aware that you may be attempting to load the <img> resources from the page. This can be a performance killer or information leak.

Consider…

function parseSomeDudesPage( pageText) {
    var page = $(pageText);   // <<<<------   This is the problem
    ... do a bunch of stuff with page...
}

...
   $.get('http://somedudeswebsite.com/blah/blah.html', parseSomeDudesPage);
...

That line marked as the problem will parse the text into DOM nodes which will be in your main document. The browser may decide to start fetching the images contained in pageText, (Safari 8.0.2 does). Depending how you fetched the document you could be getting images, broken links, or security domain violations for pulling http: resources into your https: page.

I think the solution is to keep the nodes out of your main document by creating a new one. Reading this stackoverflow comment and not caring about old browsers in my application I went with…

function parseSomeDudesPage( pageText) {
    var page = (new DOMParser).parseFromString( pageText, 'text/html');
    ... do a bunch of stuff with page which is a new document and doesn't trigger <img> loading.
}

Since you have a new document, you won’t be able to move nodes from page to your main document, but I tend to create nodes anyway, so I don’t miss the ability.

A quick warning: When you upgrade your Debian installation to Jessie, you may find the grub2 can no longer install itself and you are left with an unbootable system. (Been there, done that, managed to save myself.)

Preconditions:

• Your disks have been MBR partitioned so far in the past that there isn’t a megabyte hiding between the MBR and your first partition.

• Your /boot is sitting on something which requires more than a tiny amount of code to access. In my case ext3 on an LVM partition.

When grub2 tries to update you will get messages about core.img being too big. At this point you may be screwed.

Am I safe?

If you are MBR partitioned, then check where your first partition is. If it starts at block 2048, you have plenty of space. If it is 63, you are in trouble.

If you are GUID partitioned, make sure you have a 1MB partition for boot loader.

How I recovered:

Since I was using LVM I plowed all the data off of my /dev/sda spindle using pvmove (many times), that involved de-mirroring things since I only had three spindles in the machine. Then I could repartition the drive. I went to GUID partitioning, where you explicitly make a boot loader partition for the use of grub2.

Then I could create an LVM physical volume on the remaining space of /dev/sda and shuffle all my data back into mirrors. The process was more complicated since I went through and reformatted each of the drives in case one blows and the BIOS chooses another, but all told it is possible to perform without a reboot.