Jim's Depository

Wasted time: 10 hours.

I wanted to include a library, which I maintain, inside an ESP-IDF project. There is a nasty set of interactions between idf_component_register() and ExternalProject_Add() which make this brutally fragile.

To find your .h files in your project, you are going to have to have a INCLUDE_DIRS in your idf_component_register(). This is going to prevent you from using the download support built into ExternalProject_Add() since the files won't exist when they are checked. – So you are going to put your library in as a git submodule.

I settled on this layout to balancing sane paths in the component's CMakeLists.txt file and polluting the library. (For purposes of this example, pretend my library is named tomlbed, since it is.)

MyProject
   components
      tomlbed
         CMakeLists.txt
         tomlbed-upstream   <<---- this is my submodule

This keeps the ESP-IDF stuff out of my library. Now for the results of 10 hours of googling and whacking about, the CMakeLists.txt file…

#
# We have to make the component know where its .h files are
# and where to find its .a file.
#
idf_component_register( INCLUDE_DIRS ${COMPONENT_DIR}/tomlbed-upstream/include )
target_link_libraries( ${COMPONENT_LIB} INTERFACE ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a )

#
# Declare our external project.
# I believe the BUILD_BYPRODUCTS interacts with the
# 'target_link_libraries' above to force this to build.
#
ExternalProject_Add( tomlbed_build
                     PREFIX ${COMPONENT_DIR}
                     SOURCE_DIR ${COMPONENT_DIR}/tomlbed-upstream
                     DOWNLOAD_COMMAND ""
                     CONFIGURE_COMMAND ""
                     BUILD_IN_SOURCE 1
                     BUILD_COMMAND make CC=${CMAKE_C_COMPILER} CFLAGS=${CMAKE_C_FLAGS} AR=${CMAKE_AR} lib
                     INSTALL_COMMAND make CC=${CMAKE_C_COMPILER} CFLAGS=${CMAKE_C_FLAGS} AR=${CMAKE_AR} install libdir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed includedir=${CMAKE_BINARY_DIR}/esp-idf/tomlbed
                     BUILD_BYPRODUCTS ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/libtomlbed.a ${CMAKE_BINARY_DIR}/esp-idf/tomlbed/include
                     BUILD_ALWAYS 1
                     )

#
# Get that SOURCE_DIR variable hauled out so I can use it
#
ExternalProject_Get_Property( tomlbed_build SOURCE_DIR )

#
# Make our local 'build' directory get wiped on a Cmake 'clean'
#
set_directory_properties( PROPERTIES ADDITIONAL_CLEAN_FILES "${SOURCE_DIR}/build")

There's a lot in there that is finicky.

• DOWNLOAD_COMMAND and CONFIGURE_COMMAND are disabled with the empty strings.

• You must get the CC, CFLAGS, AR, and any other binutil type command and flag dredged out of CMake and sent down to your build command (and install if it could use them) or you will not be cross compiling. One symptom is your final link says it can't find your symbols, even though you can see them in the .a file and see the .a file passed in to the link.

• BUILD_BYPRODUCTS is telling CMake that you will produce these files. It lets you hook into the idf_component_register() and its target.

• COMPONENT_LIB is the CMake target for the idf_component_register().

• I am building in the source tree as far as ESP-IDF knows. The library has its own build tree support and uses that, then hauls its build products out into the ESP-IDF locations with its INSTALL_COMMAND.

• I didn't find a place for a "clean" command. The ADDITIONAL_CLEAN_FILES lets me at least get my build directory wiped. Not a great solution. Be aware, there is also an obsolete variant of that name with "MAKE" in it, which silently fails since ninja ignores it. Don't copy and paste that one from other sources.

• If you do try to get the idf_component_register git download commands to work, be careful. It keeps getting into a detached head state for me, and I couldn't convince myself it wouldn't wipe out my work. For a while I was using DOWNLOAD_COMMAND, but it get trying to overwrite my work. Submodule seems safer since CMake will keep its fingers off.

Mostly this is just notes to myself, but I'll document the process. It took me 30 minutes. It will take you less since you will read my third bullet point.

• Go see ESP32's Standard Toolchain Setup for Linux and macOS

• You are going to install pip and brew to get packages. I would prefer not to have brew on my system, but it is the cost of using the ESP-IDF.

• The first instruction is to sudo easy_install pip. This does not work. There is not easy_install command. No answer found to that in a quick googling. I ultimately skipped it and everything seems to be working fine.

• Install Homebrew. Be aware that if their site is every hacked you will give control of your Mac to the attackers by following the installation instructions. Also, this wants to install the Xcode command line tools, so that can take a while. Even if you already have Xcode. Pay attention at the end, there are two commands you need to execute in your terminal.

• Get the tools used to build ESP-IDF from brew. brew install cmake ninja dfu-util

• Python3 checks… my clean Monterey system comes with python3 install and not python2. Sounds ok.

• Get ESP-IDF. You will make a directory first. The documents suggest ~/esp, but I put mine in ~/coding/esp. We'll see if that strike me dead later. This is about a ½ GB download.

• Hop on down into esp-idf and ./install.sh esp32 to install the cross compilers and linkers.

• Sent your environment variables. You need to do this each time you want to do do ESP32 work. or put it in your .zprofile or whatever your login script is… . ~/WHERE_YOU_PUT_ESP-IDF/export.sh. Don't miss the first . in that command. You need to execute it in your top level shell process, not as a subshell so the environment variables stick.

• See if things are working… cd examples/get-started/hello_world then idf.py build. It should end making a ".bin" file and suggesting a flash command.

• Congratulations! You are ready to develop. The above steps took me 30 minutes, but a good chunk of that was trying to find easy_install. It turns out you don't need it.