Jim's Depository

I've finished rewriting femtoblogger again. Now it is in Swift 6 concurrency using the Hummingbird 2 web framework.

Thoughts:

• Swift 6 is surprisingly good at identifying concurrency foot guns. Sometimes it is obtuse about corrective action, but it caught a number of cases where I would have risked uncontrolled concurrent access.
• Hummingbird is ok, but I find it missing some features which should really be included. There are places to hook them in, and they generally write what you need in the documentation, but it would be nicer if they were included. (e.g.: access to the remote IP in the request context, or parsing multipart/form-data.
• For building on Debian, the binaries are tied to the Swift Stdlib which might not be synced on your deploy machine, so I've resorted to swift build --static-swift-stdlib in my Makefile and keeping the Swift Stdlib embedded.
• For simple packages, you can organize all the parts in a directoy at build time and use dpkg-deb --build --root-owner-group to make a Debian package with minimal fuss. If you need to screw with file owners other than root then you'll have to get into a fakeroot or something. The dpkg-deb man page alludes to them adding a owner/mode manifest, so I'd hold out for that.
• brew can install dpkg on your Mac so you can test building the package there. You'll get your macos binary, so it isn't usable. If you used the Swift Static SDK you could build your Debian packages on the mac, but I didn't want to mess with that.
• I used Lighter to access the database, but I'm not sure I like it. The API has like three ways of doing everything and doesn't really talk about them, so it's kind of random exploration to get something to work. It makes a set of struct to map to your SQLite tables which is kind of its thing, but I kept finding myself working with partial data so bypassing that layer on insert, update, or delete. For reading I ended up making my own structs anyway for reasons on about half the tables. Still, I like the strongly typed access.

So, maybe I'll write more here now that the software works again.

If you have had to do something heinous in your code to avoid triggering a Swift compiler bug, you might consider…

#if swift(>=7)
#warning("Check if Swift bug is fixed and we can remove the '@unchecked Sendable'.")
#endif

In my case, certain presentations of indirect enums which reference each other (and that's totally legal!) trip a circular reference error in the analysis. The work around is to jiggle the order of your declarations, or dodge the analysis by marking it as an @unchecked Sendable.

The bug has been there since sometime in version 5, we are up in 6 now, so safe to say it isn't a priority. Setting a reminder that someday when you hit Swift 7 it is worth a review let's you forget about it now.

I know my daily readers have been wondering what's up these past 700 or so days…

• There will be no more OS work. I rolled a critical fumble on COVID and can no longer work with complex abstractions in my head. I was kind of hoping for a recovery, but it appears there will not be one.

• An OS which makes limited use of system calls and primarily operates by streaming requests from user space to the OS and responses make to the user space asynchronously was looking extremely promising and anyone looking to build one sadly only gets that anecdote from my work.

• Most of the entities which the user space communicates with are ordinairy, though trusted, processes in their own right, the kernel is quite small and really about connecting things. Breaking traditional OS subsystems off into their own processes has a nice synergy with high core count processors by putting more L1/L2 cache to work.

• Popular programming languages can be made to work in this environment, but there is a lot of friction. A language with builtin light weight concurrency is more ideal. I was in the middle of a sort of Swift subset which fulfilled that and looked very promising.

Not so OS related…

• I'm ok. I've been doing a lot of physical manufacturing skill work and exploring what I can create. I think I can do small software bits, but I have to finish them in one push, if I have to pick them up later it makes no sense to me.

• So I guess I'll take a little pivot here and start documenting small things and maybe just showing stuff I make.

It is worth knowing that libicu likes to append version strings to its API symbols behind your back. If your header files do not match your libraries, like say you have the Swift compiler installed on Debian and it has replaced /usr/include/unicode/ with its own nailed down version, then you try to link symbols like utext_close_swift_65 or similar and you will be unhappy.

I removed the swift compiler and reinstalled the libicu-dev package and all is good now.

Sometimes I want to work on my experimental operating system when I'm away from my Linux monsters. To that end I keep the ability to build it on macOS. Broadly speaking that means a qemu built to support SDL because the Cocoa one in brew hasn't worked in years.

But the wrinkle… you can't use the qemu -kernel xxx flags with 64 bit multiboot kernels. It appears to be a "won't fix" from qemu, so I need to make ISO images, but making a working, modern grub on macOS cross compiling from ARM is not anything the grub guys envisioned.

Note: You will find people using legacy grub with just an eltorito.bin. This won't work well for 64 bit x86, you have to relink to elf32 then your symbol entries are all in the wrong format and insanity ensues.

So this is how I tackle it:

• Use mkisofs from cdrtools from brew to build the ISO image.

• Rip the guts out of an ISO I made on a Linux machine using grub and stuff those into my ISO.

If you mount an ISO you have made, you will find a boot/grub directory. Take that and stick it into your ISO, add your own boot/grub/grub.cfg, pack it all up and you are ready to go.

The relevant part of my Makefile looks a bit like this…

iso/boot/grub/grub.cfg : $(srcdir)/scripts/grub.cfg | iso/boot/grub
        cp $< $@

iso/boot/grub/i386-pc/eltorito.img : $(srcdir)/scripts/grub.tar.gz | iso/boot/grub
        tar -C iso/boot -xf $<

os.iso: iso/boot/kernel.elf iso/boot/grub/grub.cfg iso/boot/grub/i386-pc/eltorito.img
        mkisofs -R -b boot/grub/i386-pc/eltorito.img -no-emul-boot -boot-load-size 4 -boot-info-table -o $@ iso

You've noticed I grabbed a grub.tar.gz from someplace you can't see. I've added it as an attachment to this post. I'm using a single file (eltorito.img) as a proxy for the whole thing being unpacked. Don't fiddle with the pieces and expect make to notice.

The pinouts diagram for the Raspberry Pi Pico is miserable when printed on a monochrome laser printer. Fortunately is was in SVG so I took a copy, drug it into Graphic (the tragic rename by Autocad) and made it boring but legible. (Assuming you are on a white background. It is displayed here on the view default, and in dark mode looks like garbage).

I don't print enough color to keep the ink live in my inkjet printers and HP abandoned my networked color laser printer (then a mouse ate part of it).

If you find yourself in the same position, then enjoy! Here it its PDF and SVG glory.

The PDF is down in the attachments.

The Google scanned indicator has been updated to this decade's aesthetic. Also, it and the verified user indicator have been brought into the CSS to prevent loading image resources.

I'm not totally happy about that. They are spans with images embedded, I'd rather have them be images with shared content for the sake of alt tags. But oh well.

Beginning now, well, slightly ago… all post authors get a blue checkmark by their name.

Hithertofore I have caught all port 80 requests to my web sites and helpfully redirected them to https on port 443. But not everyone wants to have an SSL capable browser, and SSL on a tiny embedded device which you expect to run for more than 10 years without a firmware replacement is a mess.

Enter Upgrade-Insecure-Requests

Some modern browsers will send a Upgrade-Insecure-Requests: 1 header when they request a non-SSL resource. This instructs the server that the browser would be happy to use SSL if only the server could redirect it to an appropriate URL.

I'm using nginx for all my web servers these days. (I tried caddy and was largely liking it, but got tripped up on X-Accel-Redirect support which is required by this blog software among other things.) So, here is a sketch of my nginx configuration file for a simple web site.

server {
    server_name  yourserver.example.com;

    listen 80;
    listen [::]:80;
    listen 443 ssl;
    listen  [::]:443;

    … A bunch of site configuration which is not germane …

    # Concatenating two values to get cheap logic: "on1" "on" "1" "" are possible here.                                                  
    set $do_http_upgrade "$https$http_upgrade_insecure_requests";

    location / {
        if ($do_http_upgrade = "1") {
            add_header Vary Upgrade-Insecure-Requests;
            return 307 https://$host$request_uri;
        }

        index  index.html;
    }
}

The important takeaways here are:

• I'm doing all my listen in the same server block

• That $do_http_upgrade variable is making a cheap and function by concatenating two values so when I check for "1" I am testing not https and upgrade_insecure_requests present.

• Down in the location I do the redirect if requested and needed.

• That vary header comes from an MDN example. Maybe it will keep some forsaken middleware box from inappropriately caching the upgrade, but I'm sure there are ones where it won't. Their problem. Not mine. You might also try a never cache header to try to keep the middleware boxes from breaking your site.

Wasted time: 10 hours.

I wanted to include a library, which I maintain, inside an ESP-IDF project. There is a nasty set of interactions between idf_component_register() and ExternalProject_Add() which make this brutally fragile.

To find your .h files in your project, you are going to have to have a INCLUDE_DIRS in your idf_component_register(). This is going to prevent you from using the download support built into ExternalProject_Add() since the files won't exist when they are checked. – So you are going to put your library in as a git submodule.

I settled on this layout to balancing sane paths in the component's CMakeLists.txt file and polluting the library. (For purposes of this example, pretend my library is named tomlbed, since it is.)

MyProject
   components
      tomlbed
         CMakeLists.txt
         tomlbed-upstream   <<---- this is my submodule

This keeps the ESP-IDF stuff out of my library. Now for the results of 10 hours of googling and whacking about, the CMakeLists.txt file…

#
# We have to make the component know where its .h files are
# and where to find its .a file.
#
idf_component_register( INCLUDE_DIRS ${COMPONENT_DIR}/tomlbed-upstream/include )
target_link_libraries( ${COMPONENT_LIB} INTERFACE ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a )

#
# Declare our external project.
# I believe the BUILD_BYPRODUCTS interacts with the
# 'target_link_libraries' above to force this to build.
#
ExternalProject_Add( tomlbed_build
                     PREFIX ${COMPONENT_DIR}
                     SOURCE_DIR ${COMPONENT_DIR}/tomlbed-upstream
                     DOWNLOAD_COMMAND ""
                     CONFIGURE_COMMAND ""
                     BUILD_IN_SOURCE 1
                     BUILD_COMMAND make CC=${CMAKE_C_COMPILER} CFLAGS=${CMAKE_C_FLAGS} AR=${CMAKE_AR} lib
                     INSTALL_COMMAND make CC=${CMAKE_C_COMPILER} CFLAGS=${CMAKE_C_FLAGS} AR=${CMAKE_AR} install libdir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed includedir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed
                     BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/include
                     BUILD_ALWAYS 1
                     )

#
# Get that SOURCE_DIR variable hauled out so I can use it
#
ExternalProject_Get_Property( tomlbed_build SOURCE_DIR )

#
# Make our local 'build' directory get wiped on a Cmake 'clean'
#
set_directory_properties( PROPERTIES ADDITIONAL_CLEAN_FILES "${SOURCE_DIR}/build")

There's a lot in there that is finicky.

• DOWNLOAD_COMMAND and CONFIGURE_COMMAND are disabled with the empty strings.

• You must get the CC, CFLAGS, AR, and any other binutil type command and flag dredged out of CMake and sent down to your build command (and install if it could use them) or you will not be cross compiling. One symptom is your final link says it can't find your symbols, even though you can see them in the .a file and see the .a file passed in to the link.

• BUILD_BYPRODUCTS is telling CMake that you will produce these files. It lets you hook into the idf_component_register() and its target.

• COMPONENT_LIB is the CMake target for the idf_component_register().

• I am building in the source tree as far as ESP-IDF knows. The library has its own build tree support and uses that, then hauls its build products out into the ESP-IDF locations with its INSTALL_COMMAND.

• I didn't find a place for a "clean" command. The ADDITIONAL_CLEAN_FILES lets me at least get my build directory wiped. Not a great solution. Be aware, there is also an obsolete variant of that name with "MAKE" in it, which silently fails since ninja ignores it. Don't copy and paste that one from other sources.

• If you do try to get the idf_component_register git download commands to work, be careful. It keeps getting into a detached head state for me, and I couldn't convince myself it wouldn't wipe out my work. For a while I was using DOWNLOAD_COMMAND, but it get trying to overwrite my work. Submodule seems safer since CMake will keep its fingers off.