Cross-Compiling MicroPython for Kindle Paperwhite 3

Cross-Compiling MicroPython for Kindle Paperwhite 3

MicroPython is (according to their website) “a lean and efficient implementation of the Python 3 programming language that includes a small subset of the Python standard library and is optimised to run on microcontrollers and in constrained environments”.

I’ve used it for a couple of projects on the ESP-8266 and ESP32 platforms and have always been impressed with the speed in which you can go from basic idea to working prototype. The FAT based filesystem and inclusion of a REPL mean that it is possible to iterate at a much faster pace compared to Arduino or ESP-IDF, but there are a few flaws - MicroPython is more resource intensive than other languages (get used to peppering your code with gc.collect()) and the community has a long way to go before it can match the sheer variety of hardware support and external libraries offered by the Arduino ecosystem.

However, MicroPython is not limited to the embedded domain. A port for Unix exists and despite the lack of hardware capabilities found on ports for microcontrollers, the tiny binary size and minimal dependencies are well suited to the constrained Linux environment found on the Kindle.

Setting up the toolchain

Before we can do anything else, we need to build an ARM cross compiler toolchain. The Kindle uses an extensively patched kernel and ancient glibc release, which means that we cannot use the cross-compile toolchains found within the repositories of most Linux distributions. Amazon does include instructions for setting up a Poky environment to build binaries within their GPL releases, but as far as I know, nobody has been able to successfully build it.

Luckily, @NiLuJe has put in a significant amount of time and effort into maintaining Kindle compatible toolchains based upon crosstool-ng. To get started, clone the koxtoolchain repository:

git clone https://github.com/NiLuJe/koxtoolchain.git

We will also need to install all of the dependencies needed to compile the toolchain:

sudo apt-get install build-essential gperf help2man bison texinfo flex gawk autoconf automake wget curl file libncurses-dev

We now need to establish which toolchain that we need to build. This needs to match the specific Kindle model that we want to build binaries for:

TC Supported Devices Target
kindle Kindle 2, DX, DXg, 3 kindle-legacy
kindle5 Kindle 4, Touch, PW1 kindle
kindlepw2 Kindle PW2 & everything since kindlepw2

In this case, we’re using a Kindle Paperwhite 3, so we need the kindlepw2 toolchain. To start the compilation of the toolchain, we need to run these commands:

cd koxtoolchain
./gen-tc.sh kindlepw2

Note that this will take a while to complete - on my old-ish laptop with an Intel i5-7300U CPU, it took just under 40 minutes. When compiled, the toolchain will be located within the ~/x-tools folder.

Setting up Micropython build enviroment

  • First, clone a copy of the MicroPython source code
git clone https://github.com/micropython/micropython/
  • We now need to build mpy-cross - this is used to pre-compile MicroPython scripts so that they can be embedded (or frozen, in MicroPython parlence) into the MicroPython binary.
cd mpy-cross
make
  • Next, add the /bin directory containing the toolchain to your $PATH variable - I use this bash snippet within my ./bash_aliases file:
# In ~/.bash_aliases
tc_pw2() {
    export PATH=~/x-tools/arm-kindlepw2-linux-gnueabi/bin:$PATH
}
  • Once the prior steps are completed, enter the Unix port directory and build the dependencies required by MicroPython using the cross-compile toolchain.
cd ports/unix
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- make submodules
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- make deplibs
  • We are ready to begin building MicroPython. I ran into a few minor issues when doing this and have included the problems and solutions as they occurred.
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- make
  • As we are cross-compiling instead of building for the host platform, we immediately run into a problem:
user@ubuntu:~/Git/micropython/ports/unix$ CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- make
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
mkdir -p build-standard/genhdr
GEN build-standard/genhdr/mpversion.h
GEN build-standard/genhdr/moduledefs.h
GEN build-standard/genhdr/qstr.i.last
modffi.c:32:10: fatal error: ffi.h: No such file or directory
 #include <ffi.h>
          ^~~~~~~
compilation terminated.
../../py/mkrules.mk:88: recipe for target 'build-standard/genhdr/qstr.i.last' failed
make: *** [build-standard/genhdr/qstr.i.last] Error 1
make: *** Deleting file 'build-standard/genhdr/qstr.i.last'
  • The libffi headers are not included as part of the cross-compile toolchain build environment, so we need to set the PKG_CONFIG_PATH variable to point to the version included with MicroPython:
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- PKG_CONFIG_PATH=~/Git/micropython/ports/unix/build-standard/lib/libffi/ make
  • This time round, we run into a linker problem when running make:
LINK micropython
/home/user/x-tools/arm-kindlepw2-linux-gnueabi/lib/gcc/arm-kindlepw2-linux-gnueabi/7.5.0/../../../../arm-kindlepw2-linux-gnueabi/bin/ld.bfd: build-standard/unix_mphal.o: in function `mp_hal_ticks_ms':
/home/user/Git/micropython/ports/unix/unix_mphal.c:197: undefined reference to `clock_gettime'
/home/user/x-tools/arm-kindlepw2-linux-gnueabi/lib/gcc/arm-kindlepw2-linux-gnueabi/7.5.0/../../../../arm-kindlepw2-linux-gnueabi/bin/ld.bfd: build-standard/unix_mphal.o: in function `mp_hal_ticks_us':
/home/user/Git/micropython/ports/unix/unix_mphal.c:209: undefined reference to `clock_gettime'
collect2: error: ld returned 1 exit status
../../py/mkrules.mk:168: recipe for target 'micropython' failed
make: *** [micropython] Error 1
  • In theory, the clock_* functions are available on the Kindle as part of glibc (they were moved from librt to glibc in version 2.17; the PW3 uses 2.20). However, as the cross-compiler toolchain uses 2.12 for compatibility with other Kindle models, we need to link against librt. We can do this by setting the LDFLAGS_EXTRA variable to -lrt:
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- PKG_CONFIG_PATH=~/Git/micropython/ports/unix/build-standard/lib/libffi/ LDFLAGS_EXTRA="-lrt" make
  • The build should now be successful:
LINK micropython
   text	   data	    bss	    dec	    hex	filename
 219128	   4568	   1248	 224944	  36eb0	micropython
  • At this point, we should copy the binary over to the Kindle so that we can test it:
scp micropython root@$KINDLE_IP:/mnt/us
[root@kindle us]# ./micropython 
./micropython: error while loading shared libraries: libffi.so.6: cannot open shared object file: No such file or directory
  • An error is thrown due to the absense of a suitable version of libffi. There’s a few ways that we can resolve this issue:
    • Creating a symlink to the existing libffi version already on the Kindle. This isn’t a great idea as MicroPython might be using functionality that is only found in a recent version of the library.
    • Adding the MicroPython version of libffi to /usr/lib on the Kindle. This would work well, but I prefer to avoid making changes to the root filesystem of the Kindle unless I have a way to track and revert changes.
    • Rebuilding MicroPython with an extra shared library search path that points to a location on /mnt/us. This would also work well, but we would need to transfer our libffi build separately.
    • Statically linking MicroPython. This gives us a single binary that runs without the need to perform any additional tasks, at the expense of significantly increased file size.

Out of these approaches, the last 2 are preferable.

Building with extra shared library path

  • Delete the existing MicroPython build:
rm micropython
  • Re-link MicroPython, with -Wl,-rpath=/mnt/us/micropython/lib appended to LDFLAGS_EXTRA:
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- PKG_CONFIG_PATH=~/Git/micropython/ports/unix/build-standard/lib/libffi/ LDFLAGS_EXTRA="-lrt -Wl,-rpath=/mnt/us/micropython/lib" make
  • Create a directory to contain libffi on the Kindle:
mkdir -p /mnt/us/micropython/lib
  • Copy the cross-compiled libraries to the Kindle:
scp build-standard/lib/libffi/out/lib/libffi.so* root@$KINDLE_IP:/mnt/us/micropython/lib
  • Copy the MicroPython binary to the Kindle
scp micropython root@$KINDLE_IP:/mnt/us/micropython/
  • We should verify that the Kindle is able to load the shared library from the additional search path - the output from ldd should look similar to this:
[root@kindle micropython]# ldd micropython 
	/usr/lib/libenvload.so (0x40128000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x400c3000)
	libffi.so.6 => /mnt/us/micropython/lib/libffi.so.6 (0x4004e000)
	libdl.so.2 => /lib/libdl.so.2 (0x400e7000)
	libm.so.6 => /lib/libm.so.6 (0x40131000)
	librt.so.1 => /lib/librt.so.1 (0x4008e000)
	libc.so.6 => /lib/libc.so.6 (0x401a5000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x400f2000)
	/lib/ld-linux.so.3 (0x40065000)
  • If libffi is found, MicroPython will run successfully:
[root@kindle micropython]# ./micropython 
MicroPython v1.13-95-g0fff2e03f on 2020-10-04; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> print("Hello, world")
Hello, world

Building with statically linked libraries

  • Delete the existing MicroPython build:
rm micropython
  • Re-link MicroPython with LDFLAGS_EXTRA set to -static -lrt -lffi
CROSS_COMPILE=arm-kindlepw2-linux-gnueabi- PKG_CONFIG_PATH=~/Git/micropython/ports/unix/build-standard/lib/libffi/ LDFLAGS_EXTRA="-static -lrt -lffi" make
  • This will emit a few warnings, but they can be safely ignored:
LINK micropython
/home/user/x-tools/arm-kindlepw2-linux-gnueabi/lib/gcc/arm-kindlepw2-linux-gnueabi/7.5.0/../../../../arm-kindlepw2-linux-gnueabi/bin/ld.bfd: build-standard/modffi.o: in function `ffimod_make_new':
/home/user/Git/micropython/ports/unix/modffi.c:329: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/home/user/x-tools/arm-kindlepw2-linux-gnueabi/lib/gcc/arm-kindlepw2-linux-gnueabi/7.5.0/../../../../arm-kindlepw2-linux-gnueabi/bin/ld.bfd: build-standard/modusocket.o: in function `mod_socket_getaddrinfo':
/home/user/Git/micropython/ports/unix/modusocket.c:574: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
  • Copy the MicroPython binary to the Kindle:
scp micropython root@192.168.1.111:/mnt/us
  • MicroPython should run successfully:
[root@kindle us]# ./micropython 
MicroPython v1.13-95-g0fff2e03f on 2020-10-04; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> print("Hello, world")
Hello, world
  • Statically linking the binary has increased the size by around 700KB, but this isn’t a major problem as the total size is still under 1MB:
[root@kindle us]# ls -lh micropython*
-rwxrwxrwx    1 root     root      914.9K Oct  4 19:28 micropython
-rwxrwxrwx    1 root     root      221.4K Oct  4 19:35 micropython-dynamic

Installing additional libraries with upip

MicroPython includes a package manager that is vaguely similar to pip. The package selection is limited (with many being stub packages that offer no functionality) and packages need to be created specifically for MicroPython; you cannot use packages for other implementations like CPython without adapting them beforehand. These libraries are available on micropython.org and PyPI - the full listing can be found here.

By default, packages are installed in the home directory of the current user. This presents a problem in the case of the Kindle - the default home directory is /tmp/root, which means that any installed packages will be deleted when the device is rebooted. According to the documentation, it is possible to specify an installation path for upip using the -p flag:

./micropython -m upip install -p /mnt/us/.micropython/lib urequests

However, it’s easier to change the install path so that we don’t need to specify it each time we want to install a package.

  • To do this, open micropython/tools/upip.py on your host system. The function that we need to alter is get_install_path:
def get_install_path():
    global install_path
    if install_path is None:
        # sys.path[0] is current module's path
        install_path = sys.path[1]
    install_path = expandhome(install_path)
    return install_path
  • Change install_path from sys.path[1] to /mnt/us/.micropython/lib:
def get_install_path():
    global install_path
    if install_path is None:
        install_path = "/mnt/us/.micropython/lib"
    install_path = expandhome(install_path)
    return install_path
  • Once you have made this change, rebuild MicroPython using one of the processes above, then transfer the binary to the Kindle. When a package is installed, the installation directory will be shown:
[root@kindle us]# ./micropython -m upip install urequests
Installing to: /mnt/us/.micropython/lib/
Warning: micropython.org SSL certificate is not validated
Installing urequests 0.6 from https://micropython.org/pi/urequests/urequests-0.6.tar.gz

You should now have a working version of MicroPython on your Kindle and hopefully, a clearer idea of how basic cross compilation works. Although the utility of MicroPython is limited by the lack of deeper hardware access found on microcontroller ports, the ability to access a lightweight scripting language with support for extra packages can be incredibly useful, especially once you understand the concept of freezing modules into a build.

If you found this post interesting, it’s definitely worth checking out the MobileRead Kindle Developers forum to see what else you can do with your Kindle, and the MicroPython forums to get a sense of other things you can can achieve with this minimal language.

kindle micropython cross-compiling