Jim's Depository

this code is not yet written

Hithertofore I have caught all port 80 requests to my web sites and helpfully redirected them to https on port 443. But not everyone wants to have an SSL capable browser, and SSL on a tiny embedded device which you expect to run for more than 10 years without a firmware replacement is a mess.

Enter Upgrade-Insecure-Requests

Some modern browsers will send a Upgrade-Insecure-Requests: 1 header when they request a non-SSL resource. This instructs the server that the browser would be happy to use SSL if only the server could redirect it to an appropriate URL.

I'm using nginx for all my web servers these days. (I tried caddy and was largely liking it, but got tripped up on X-Accel-Redirect support which is required by this blog software among other things.) So, here is a sketch of my nginx configuration file for a simple web site.

server {
    server_name  yourserver.example.com;

    listen 80;
    listen [::]:80;
    listen 443 ssl;
    listen  [::]:443;

    … A bunch of site configuration which is not germane …

    # Concatenating two values to get cheap logic: "on1" "on" "1" "" are possible here.                                                  
    set $do_http_upgrade "$https$http_upgrade_insecure_requests";

    location / {
        if ($do_http_upgrade = "1") {
            add_header Vary Upgrade-Insecure-Requests;
            return 307 https://$host$request_uri;

        index  index.html;

The important takeaways here are:

  • I'm doing all my listen in the same server block

  • That $do_http_upgrade variable is making a cheap and function by concatenating two values so when I check for "1" I am testing not https and upgrade_insecure_requests present.

  • Down in the location I do the redirect if requested and needed.

  • That vary header comes from an MDN example. Maybe it will keep some forsaken middleware box from inappropriately caching the upgrade, but I'm sure there are ones where it won't. Their problem. Not mine. You might also try a never cache header to try to keep the middleware boxes from breaking your site.

Wasted time: 10 hours.

I wanted to include a library, which I maintain, inside an ESP-IDF project. There is a nasty set of interactions between idf_component_register() and ExternalProject_Add() which make this brutally fragile.

To find your .h files in your project, you are going to have to have a INCLUDE_DIRS in your idf_component_register(). This is going to prevent you from using the download support built into ExternalProject_Add() since the files won't exist when they are checked. – So you are going to put your library in as a git submodule.

I settled on this layout to balancing sane paths in the component's CMakeLists.txt file and polluting the library. (For purposes of this example, pretend my library is named tomlbed, since it is.)

         tomlbed-upstream   <<---- this is my submodule

This keeps the ESP-IDF stuff out of my library. Now for the results of 10 hours of googling and whacking about, the CMakeLists.txt file…

# We have to make the component know where its .h files are
# and where to find its .a file.
idf_component_register( INCLUDE_DIRS ${COMPONENT_DIR}/tomlbed-upstream/include )
target_link_libraries( ${COMPONENT_LIB} INTERFACE ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a )

# Declare our external project.
# I believe the BUILD_BYPRODUCTS interacts with the
# 'target_link_libraries' above to force this to build.
ExternalProject_Add( tomlbed_build
                     PREFIX ${COMPONENT_DIR}
                     SOURCE_DIR ${COMPONENT_DIR}/tomlbed-upstream
                     DOWNLOAD_COMMAND ""
                     CONFIGURE_COMMAND ""
                     BUILD_IN_SOURCE 1
                     INSTALL_COMMAND make CC=${CMAKE_C_COMPILER} CFLAGS=${CMAKE_C_FLAGS} AR=${CMAKE_AR} install libdir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed includedir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed
                     BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/include
                     BUILD_ALWAYS 1

# Get that SOURCE_DIR variable hauled out so I can use it
ExternalProject_Get_Property( tomlbed_build SOURCE_DIR )

# Make our local 'build' directory get wiped on a Cmake 'clean'
set_directory_properties( PROPERTIES ADDITIONAL_CLEAN_FILES "${SOURCE_DIR}/build")

There's a lot in there that is finicky.

  • DOWNLOAD_COMMAND and CONFIGURE_COMMAND are disabled with the empty strings.

  • You must get the CC, CFLAGS, AR, and any other binutil type command and flag dredged out of CMake and sent down to your build command (and install if it could use them) or you will not be cross compiling. One symptom is your final link says it can't find your symbols, even though you can see them in the .a file and see the .a file passed in to the link.

  • BUILD_BYPRODUCTS is telling CMake that you will produce these files. It lets you hook into the idf_component_register() and its target.

  • COMPONENT_LIB is the CMake target for the idf_component_register().

  • I am building in the source tree as far as ESP-IDF knows. The library has its own build tree support and uses that, then hauls its build products out into the ESP-IDF locations with its INSTALL_COMMAND.

  • I didn't find a place for a "clean" command. The ADDITIONAL_CLEAN_FILES lets me at least get my build directory wiped. Not a great solution. Be aware, there is also an obsolete variant of that name with "MAKE" in it, which silently fails since ninja ignores it. Don't copy and paste that one from other sources.

  • If you do try to get the idf_component_register git download commands to work, be careful. It keeps getting into a detached head state for me, and I couldn't convince myself it wouldn't wipe out my work. For a while I was using DOWNLOAD_COMMAND, but it get trying to overwrite my work. Submodule seems safer since CMake will keep its fingers off.

Mostly this is just notes to myself, but I'll document the process. It took me 30 minutes. It will take you less since you will read my third bullet point.

  • Go see ESP32's Standard Toolchain Setup for Linux and macOS

  • You are going to install pip and brew to get packages. I would prefer not to have brew on my system, but it is the cost of using the ESP-IDF.

  • The first instruction is to sudo easy_install pip. This does not work. There is not easy_install command. No answer found to that in a quick googling. I ultimately skipped it and everything seems to be working fine.

  • Install Homebrew. Be aware that if their site is every hacked you will give control of your Mac to the attackers by following the installation instructions. Also, this wants to install the Xcode command line tools, so that can take a while. Even if you already have Xcode. Pay attention at the end, there are two commands you need to execute in your terminal.

  • Get the tools used to build ESP-IDF from brew. brew install cmake ninja dfu-util

  • Python3 checks… my clean Monterey system comes with python3 install and not python2. Sounds ok.

  • Get ESP-IDF. You will make a directory first. The documents suggest ~/esp, but I put mine in ~/coding/esp. We'll see if that strike me dead later. This is about a ½ GB download.

  • Hop on down into esp-idf and ./install.sh esp32 to install the cross compilers and linkers.

  • Sent your environment variables. You need to do this each time you want to do do ESP32 work. or put it in your .zprofile or whatever your login script is… . ~/WHERE_YOU_PUT_ESP-IDF/export.sh. Don't miss the first . in that command. You need to execute it in your top level shell process, not as a subshell so the environment variables stick.

  • See if things are working… cd examples/get-started/hello_world then idf.py build. It should end making a ".bin" file and suggesting a flash command.

  • Congratulations! You are ready to develop. The above steps took me 30 minutes, but a good chunk of that was trying to find easy_install. It turns out you don't need it.

The ESP32 HTTPS over the air update mechanism requires you to know the SSL certificate used by the web server. This is problematic in a letsencrypt, fast expiring certificate world, but also for devices which will be deployed for long time frames.

It is possible to disable the SSL using the CONFIG_OTA_ALLOW_HTTP option. The SDK will tell you this should only be used for development, but if you also used signed firmware it is safe for deployed use.

Rather than protect the pipe the firmware traverses and blindly accepting anything coming down that pipe, you will instead not trust the pipe and validate the firmware as it arrives.

See the Secure OTA Updates Without Secure boot section. In a nutshell, you will make a private signing key and the OTA updates will be checked against that key.

Notes Thee Well!

If your firmware itself is sensitive, then don't do this. It can be snooped in transit. On the other hand, in the regular HTTPS scheme there is a URL which provides a copy of your firmware, so you are probably already working on something for that.

Apparently you can use methods names as first class values in Swift. It must not be a popular feature, I can't find it in the official documentation, but it is there and is just the trick when you want to pass a method selector into a function to, say operate on a complex graph of similarly base classed instances.

The short answer is the "SomeClass.someMethod" gives you a function, which when applied to an instance gives you another function, which when you call that one with the methods arguments (without the labels) invokes the method on the object.

Playground Example

All output is what you'd expect.

// A base class and a derived class.
class Base {
    func blurt( capitalized:Bool) -> String {
        return "I am \( capitalized ? "Base" : "base")"

class Derived : Base {
    override func blurt( capitalized:Bool) -> String {
        return "I am \( capitalized ? "Derived" : "derived")"

// … an instance of each …
let base = Base()
let derived = Derived( )

// … they do what you expect …
print( base.blurt( capitalized: true ))
print( derived.blurt( capitalized: true ))

// HERE BE MAGIC: we grab a value for the method itself,
// it gives us a function which produces a function to
// invoke the method on an instance.
// (You can infer the type, I just put it in so
//  you can see it.)
let m : (Base) -> (Bool) -> String = Base.blurt

// Get a function for each of our instances which invokes
// our method.
let fDerived : (Bool) -> String = m(derived)
let fBase : (Bool) -> String = m(base)

// And invoke them. Notice, we lost our argument names.
print( fBase( false) )
print( fDerived( false) )

// Once you understand the extra layer of function here,
// you can invoke them like this.
print( m(base)(true) )

// Limitation: I was unable to tease out a syntax to
// work with polymorphic methods.

Open Questions

  • Where is this in the documents?

  • Is there a way to do this with polymorphic methods? By which I mean something like let m = SomeClass.someMethod( onArray:[Array]) or some such if I have someMethod for both arrays and strings or something.

There is an internal temperature sensor hiding in the RP2040 processor. It is just ever so slightly tricky to read. I notice that a lot of the help on the web skips an important step where you enable the temperature sensor, and glosses over the ADC reference voltage.

It isn't a high precision sensor, about ½ degree Celsius is as good as you are going to read, but it will let you know if your enclosure is cooking your project.

Enable the temperature sensor

In addition to adc_init() you will need to adc_set_temp_sensor_enabled(true). I kind of think there is a period of time before you would not trust the reading. It seems like it creeps up for a second or two. So if you are using this, just turn it on and leave it on.

Understand the ADC reference voltage

The ADC system is all relative to a reference voltage. If you haven't connected anything to the ADC_VREF pin of a Pi Pico, then it is nominally 3.3v, and that constant is present in everyone's sample code. It is worth noting that it is probably lower than that, so if you want to calibrate, check the pin with a multimeter.

If you are interested in accuracy with other analog inputs, you will probably put a 'real' voltage reference on ADC_VREF, then you need to change the constants in the code to match your chosen hardware.

My Example

In any event, my example looks like this.

** NOTE THEE WELL:                                                                                                                    
** You must do these before using this function...                                                                                    
**  adc_init();                                                                                                                       
**  adc_set_temp_sensor_enabled(true);                                                                                                
** You will get ridiculous temperature values if you don't enable                                                                     
** the temperature sensor. It also might take a little bit to converge                                                                
** on something close to right, so don't be tricky and snap it on and                                                                 
** off to save power.                                                                                                                 


void print_info(void) {
    uint16_t bits = adc_read();
    float voltage = bits * REFERENCE_VOLTAGE / 4095;
    float temperature = 27.0 - ( voltage - 0.706) / 0.001721;

    printf("Internal temperature: %4.1fC\n", temperature);

I put this little Makefile in the top of my Pico project directory. It handles making the build directory and invoking cmake with the right flags to build my actual make files.

Mostly I just do make install to build and flash onto my device. (Presuming the device is in UF2 mode, that's a different post.)

# Path to my SDK installation
PICO_SDK_PATH ?= ~/pico-sdk

# Path to the mount point for my pico
PICO_MOUNT ?= /rpi-rp2

# Path to my UF2 image in the build directory
IMAGE = build/sbi-weather.uf2

all : $(IMAGE)

install : all
	mount $(PICO_MOUNT)
	cp $(IMAGE) $(PICO_MOUNT)/foo

clean :
	rm -rf build pico_sdk_import.cmake

build :
	mkdir build

$(IMAGE): build/Makefile
	( cd build ; make all )

build/Makefile : CMakeLists.txt pico_sdk_import.cmake | build
	( cd build ; cmake -DPICO_SDK_PATH=$(PICO_SDK_PATH) .. )

pico_sdk_import.cmake : $(PICO_SDK_PATH)/external/pico_sdk_import.cmake
	cp $< $@

.PHONY : all install clean $(IMAGE)


I'm using a Raspberry Pi running Raspbian as my host. In the install target the sync command is helpful to push the dirty blocks out right now.

The mount command also works fine for my user account because I have this in my /etc/fstab

LABEL=RPI-RP2 /rpi-rp2 msdos defaults,noauto,user 0 1


Find the reset_usb_boot() function in pico/bootrom.h instead.

But for historical reasons, and maybe you want to render it unbootable…

I want to be able to erase and reprogram my Pi Pico without having to touch it and only having a USB cable to it.

I built my program to use USB for the console. If the console receives a "reset" command then it erases the secondary boot loader from flash and forces a restart. This will make it boot into the UF2 loader.

The function looks like this…

#include "wipe.h"
#include "hardware/watchdog.h"
#include "hardware/flash.h"
#include "hardware/sync.h"
#include "pico/multicore.h"

// Remember: Put hardware_flash and pico_multicore in the CMakeLists.txt target_link_libraries

void wipe(void) {
    const uint8_t blank[256] = {0};

    // Stop core1. We better be core0.                                                                                                

    uint32_t saved = save_and_disable_interrupts();
    // Wipe the secondary boot loader                                                                                                 
    flash_range_program( 0, blank, 256);

    // this will reset us in 1 millisecond.                                                                                           
    watchdog_enable( 1, false);

    // await our doom                                                                                                                 
    while( true);

No more physical access required while developing. Until I do something stupid to hang the CPU.

Subtle Note:

I am using flash_range_program instead of flash_range_erase because I want to do only 256 bytes. Otherwise I clobber the code I'm executing and everything ends in tears. In the world of this flash chip, "erase" means "set to all ones" and "program" means "set the zeros to zero". So I can turn the 256 bytes into all zeros without erasing.


If you decide you want to erase the whole flash, there is going to be trouble. You will end up erasing the code you are executing and hang. I would suggest you move the watchdog_enable call above your flash_range_erase (which you will use instead of program so you don't need a buffer) and give it a timeout long enough to accomplish the erase.

Ostensibly you can mark your function as being copied to RAM, but I didn't have success with that. If I used the CMakeLists.txt to make the whole program be in RAM I could erase flash willy nilly without problems.

I'm going to try the Raspberry Pi Pico for some of my small projects. I'm primarily attracted by the deterministic timing available with the Programmable I/O state machines.

I'll mostly be building devices which present as USB peripherals to a host. As such I probably won't bother with the UART for debugging messages during development. I'd also like to avoid the Serial Wire Debug (SWD) so I don't have to wire that up either.

Developer Ergonomics

That presents me with a little development conundrum. I don't like doing the unplug/press/plug/release dance, and the human quadrature dance with a reset button added isn't much better.

My current plan is to build a "destroy yourself" function into my USB interface. That will erase the secondary boot loader block in flash then do a reset from software to force a USB Mass Storage mode boot.

Mounting on Debian

I'm using a Raspberry Pi 4 as a development machine, we'll see. It takes it about 3 seconds to build my 15kB test program. That's really pretty sad. It is mostly cmake generated abominations faffing about. Hopefully it doesn't get much longer as I add real code.

I want a target in my Makefile to kill a running device, flash new, and restart it. That means mounting the mass storage device when it becomes available. For now I've got a line in /etc/fstab to make that easy...

LABEL=RPI-RP2 /rpi-rp2 msdos defaults,noauto,user 0 1

With that my untrusted user account can mount /rpi-rp2 and do what it needs.

If you are thinking things changed, you are probably getting the dark mode support. It's your browser's choice. I send the same stuff to you, but the CSS is now littered with @media (prefers-color-scheme: dark) { sections to change things up for dark mode.

I wish Safari didn't put all the editable text, radio buttons, and buttons in blinding white. I wonder if I gorked something up there.

Update: Yeah. It was me. Somewhere in the process of switching to use CSS variables for my colors the problem went away. I probably had a background forced to something unfortunate.

more articles