Jim's Depository

this code is not yet written
 

libmicrohttpd - A web interface for your daemons.

When writing daemons there is always the question of “How will I check on the status as it runs?“. Solutions include:

  • Have a debug flag and write to stderr.
  • Periodically write a file in a known place.
  • On a particular signal, write a file in a known place.
  • Keep a socket open to write status if someone connects.

I add one more:

  • Have an HTTP interface and let people talk to you with a browser.

libmicrohttpd is just the thing for embedding an HTTP server in your daemon. It adds about 25k to your code, a couple lines to start it and then a dispatching function you write to handle the requests.

It has three models, the least obtrusive of which it is to just let it make some threads and live in its own threads. The others are useful if you can’t tolerate threads, but you have to cooperate to get select() calls made.

I don’t think you’d want to let it take a slashdotting, so either keep it behind a firewall or check the incoming IPs.

It seems to be solid and well written, but I did notice that it raises a SIGPIPE that kills the host application when a connection closes in the middle of a request. (3 word fix in the source. Add MSG_NOSIGNAL to the SEND() calls) That was found in the first minutes of testing, so it doesn’t get the “bullet proof - use and forget” stamp of trust. Maybe after 6 months of daily use it can earn that.

I can't help but wonder though, can it be simpler? I'll probably have to read the HTTP 1.1 spec and see how simple a daemon I can write.
Why are there so many packages ruined by people using autoconfigure and libtool? Just write a simple Makefile and let it be. 

The application where I was going to use libmicrohttpd requires me to crosscompile and after hours of thrashing about I still can't get it to build.

Back to the bit heap with it. I'll write my own httpd code.

Well that wasn't half hard. wc reports 279 lines of code weighing in at 7.5kb source and just under 4k of binary for an HTTP/1.0 and HTTP/1.1 compliant httpd function. (Well, still a few more lines to enforce a maximum concurrent thread limit and a thread timeout so I needn't fear nefarious people… but it is nearly done.)

I thought going in that getting HTTP/1.1 pipelining right was going to be the trickiest part, but on further investigation none of the major browsers use it. Apparently enough web servers screw it up to prevent it.

In the absence of pipelining I decided to forgo keep-alive entirely in favor of simplicity. By careful use of TCP options I only need 3 outgoing packets for each request (up to 1.mumble kbytes). The SYN-ACK, an ACK-Data-FIN, and a FIN-ACK.

An interesting performance issue: Safari shotguns me with many simultaneous connections, to which my httpd responds quickly. If I were supporting keep-alive I think Safari would be encouraged to only use two connections and serialize the requests over them. I wonder which is faster? I may have to add keep-alive support just to answer this question.

Another interesting tidbit: Some people on the web maintain that TCP_QUICKACK and TCP_DEFER_ACCEPT are inherited from the listener socket to the accepted socket. I don't think so. At least the only way I can get QUICKACK turned off is to not use TCP_DEFER_ACCEPT on the listening socket  and slam TCP_QUICKACK off on the accepted socket before the first data arrives. Otherwise I end up sending an ACK before my first data packet.

And a last tidbit: You can keep your TCP_CORK in all the to shutdown(), that gets your FIN piggybacked on your last data packet.

Added the maximum concurrent connection support. It took -3 lines of code. sem_init(), sem_post(), sem_wait() is nicer for this than using pthread mutexes on variables.
Bleh, timeout was harder than I had hoped. I use a simple watchdog thread per request scheme, but even that takes a mutex and some care to get everything deallocated safely.

Worse, if thread A is in an fgets() on file F when it is phtread_canceled, then when thread B tries to fclose(F) it hangs. I suppose there is a lock inside the FILE *. I punted stdio and just did my input at the socket level. I was already doing output at the socket level to avoid a copy operation.

Now to add some comments, forget all about this code, and move on with the actual problem.

openload

openload, which is called openwebload at sourceforge, is a simple URL load tester. Just the right sized tool for checking if your spiffy new web page is efficient enough to be tolerated.

\$ openload localhost 10 URL: http://localhost:80/ Clients: 10 MaTps 355.11, Tps 355.11, Resp Time 0.015, Err 0%, Count 511 MaTps 339.50, Tps 199.00, Resp Time 0.051, Err 0%, Count 711 MaTps 343.72, Tps 381.68, Resp Time 0.032, Err 0%, Count 1111 MaTps 382.04, Tps 727.00, Resp Time 0.020, Err 0%, Count 1838 MaTps 398.54, Tps 547.00, Resp Time 0.018, Err 0%, Count 2385 MaTps 425.78, Tps 670.90, Resp Time 0.014, Err 0%, Count 3072
<You press enter to stop the run> Total TPS: 452.90 Avg. Response time: 0.021 sec. Max Response time: 0.769 sec

There is a Debian package.

Update: I couldn’t leave it alone. The attached patch will add bits/second reports. It is also worth noting that you can add  ”-l number” to sample for a certain number of seconds, and a “-o csv” to put out a CSV line of the data at the end for easier scripting. Perhaps it needs a man page too.

Update: If you are an especially trusting sort of person you could install the Debian package attached and not have to build it yourself.

Update: Best wait on this. There is something odd going on with openload that is giving unstable results for bytes transfered. More diagnosis required.

Attachments

openload_0.1.2-1.1-jws1_i386.deb 15708 bytes
openload.patch 3971 bytes

Really Learn Javascript

I, like most people, started using Javascript without first learning it. That was sort of fine back when all we did was stick a little statement in an onclick. When you find yourself writing more than 60 lines of JavaScript it is time to actually learn the language.

The development folk at Yahoo have a Yahoo! UI Library: YUI Theater with some videos to straighten out your understanding of JavaScript, wrap yellow warning tape around the broken corners of the language, and generally get you in shape to write real software in JavaScript.

I recommend these:

  • Douglas Crockford - “The JavaScript Programming Language”
  • Douglas Crockford - “Advanced Javascript”
  • Douglas Crockford - “An Inconvenient API: The Theory of the DOM”

That is about 3 hours of video. If improving your productivity and product quality isn’t worth three hours, you might try the 40 minute “JavaScript: The Good Stuff”.

Things I May Add To Femtoblogger

A list of things that I might add to femtoblogger:

  • [DONE] Attachments.
  • Some mechanism for fiddling with MIME types of attachments.
  • Local copies of images. Pull it in, resize it to display size, click through to original size.
  • {NO} Some sort of Todo list.
  • [DONE] The concept of a draft copy of an article.
  • I’d like to use PATH_INFO to have more meaningful URLs, but then I’m committed to .htaccess files working to keep sane URLs. Or maybe I just let the ‘.php’ show in the middle of the path.
  • [DONE] Some time based archive structure in the nav section.
  • An export format, perhaps based on Atom.
  • [DONE] Support for external httpd authentication.
  • [DONE] A global configuration page.
  • [DONE] CSS takes variables from database now. Color skinning.
  • [DONE] Use <META NAME=“ROBOTS” CONTENT=“NONE”> to keep robots out of the non-content areas.

ifstat - network interface bandwidth numbers

ifstat - A little like a vmstat for network interfaces. This is pretty handy for a quick look at network saturation without have to break out some massive GUIfied tool with a hideous interface.

asbestos:\~# ifstat -b -i eth0,eth1,eth2,eth3
       eth0                eth1                eth2                eth3      
 Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out
   13.44     19.06    914.15    953.60      0.00      0.00      0.00      0.00
    9.73     10.71    632.22    630.49    300.84    286.86      5.27      4.72
   21.01     23.73    211.98    226.27      0.00      0.00      0.00      0.00
    2.23      3.33    260.78    281.29    293.32    315.72      5.80      4.72
    3.47      6.03    180.47    227.07      0.00      0.00      0.00      0.00

There is a debian package. And note that “-b” option to report in bits/second instead of bytes. What kind of person measures network use in bytes per second?

And in the same vein: iostat is hiding over in the systat package on debian. Use it too.

The Linux Category

In this category I make notes of useful linux tools and utilities that are perhaps newer than 1997. If, like me, you tend to use your existing knowledge to get the job done, this list might be useful to see some of the things developed in the past 10 years.

Now Showing on IE7: femtoblogger

Femtoblogger now shows on IE7. Not too bad. It only took 2 hours to work out the layout problems.

The most severe is that max-width: 600px in the CSS file brings up a bug when used inside a table cell. The cell gets expanded as if there was no max-width so the table ends up expanding to fill the screen. All the articles were cowering against the lefthand border of the screen with the nav stuff still on the right. (Firefox does this too but only if you are clicking a link in a table cell. There went 120 minutes of my life for that one.)

My ultimate solution was to make IE use a width: 600px instead. There is now an IE only CSS file that is loaded after the real one and has a rule to patch that up. Thankfully the IE blokes must have known they were being a menace to society and they built a special comment hack into their HTML parser to let people cope.

<!--[if IE]> <link rel=stylesheet type=text/css href=femtoblogger-ie.css> <![endif]-->

… it would have been nice to have something similar in CSS, that would save a file and a download.

You can read more about this hack at CSS Hacks- The good, the bad, and the ugly- Conditional comments

You still can't post comments or articles from IE7. I'll tackle that tomorrow.

Things fixed so far:
  1. IE does not have contentDocument, use contentWindow instead.
  2. addEventListenter/attachEvent silliness.
  3. The focus and blur events go on the IFRAME's contentWindow, except Firefox needs it on the contentWindow.document. (Not an IE problem, but my first IE fix broke this.)
  4. The absolute DIV needed an explicit .style.left as it was positioning to the right of the parent.
  5. Odd problem where a delete n.moseyTimer was leaving the value there. I had to set it to false explicitly.
  6. Lucky break: accidentally found a problem with which document I use to make the .selection() call while wresting the fleeing tool bar.
  7. Wash me. Unable to keep the toolbar from taking focus, I have resorted to a timer to notice if a toolbar hide is a result of the toolbar being clicked and suppressing the hide.
  8. IE is still doing amusing things with the spaces around selections when it wraps them. More hackery to come.
Went to do the CSS alterations for iPhones. Elapsed time: 0 seconds. Just worked.

No more CSS columns

I have abandoned CSS columns. They are an ongoing time sink. If you are willing to specify hard physical pixel widths for the columns they almost work like I want, but not quite. Their background colors end early, and when you get a wider than planned element inside one it spills out on to neighboring content instead of resizing.

Back to the grand one row table for me. The CSS zealots may scoff, but my columns work right all the time.

On the plus side, I may now be web 2.0. I added a couple rounded corners and a drop shadow. Unless you are using IE.

femtoblogger.com

I went ahead and registered femtoblogger.com which reminds me how much easier it is for me to write code than prose. Filling a page is just painful. I’ll have to make filling in that outline be a long term project.

For now it serves as a placeholder for people that might guess at the URL and will send them back to this blog for femtoblogger news. I suppose I need to rename this blog now which means more code. Blast.

WYSIWYG Articles and Comments

This is half baked, but I’m going to roll it out and try to live with it. Articles and comments will now use a WYSIWYG interface for styling. I hope to remove the How do I insert-a-link/bold/blockquote? type of questions that interrupt people’s thought process when posting in a new venue.

The downsides are many:

  • This uses the designMode and the execCommand() interface. execCommand() was probably designed by children and is functionally incomplete and inadequately specified. It promises to be a festering source of browser quirks and user bafflement for years to come.
  • The HTML sanitizer will need to normalize as well to unquirk the postings.
  • I need to also support a raw HTML mode for people with odd browsers.
  • I will need to make decisions about copying in images and possibly resizing.
  • I will probably need onscreen hinting ofsome sort to let people know how design mode works, e.g. They can drag in links and images.
  • I will probably screw up and butcher comments at least several times.

But… what would life be without a little excitement, so here we go.

Ok, comments seem to be at least minimally functional with the new editor code in Safari.
  1. Switch new post editing over to the new style.
  2. Make sure I didn't break Firefox.
  3. Patch things up so IE works.
  4. Make a safe fallback to HTML for browsers without designMode.

more articles