Native machine code in .mpy files

This section describes how to build and work with .mpy files that contain native machine code from a language other than Python. This allows you to write code in a language like C, compile and link it into a .mpy file, and then import this file like a normal Python module. This can be used for implementing functionality which is performance critical, or for including an existing library written in another language.

One of the main advantages of using native .mpy files is that native machine code can be imported by a script dynamically, without the need to rebuild the main MicroPython firmware. This is in contrast to MicroPython external C modules which also allows defining custom modules in C but they must be compiled into the main firmware image.

The focus here is on using C to build native modules, but in principle any language which can be compiled to stand-alone machine code can be put into a .mpy file.

A native .mpy module is built using the mpy_ld.py tool, which is found in the tools/ directory of the project. This tool takes a set of object files (.o files) and links them together to create a native .mpy files. It requires CPython 3 and the library pyelftools v0.25 or greater.

Supported features and limitations

A .mpy file can contain MicroPython bytecode and/or native machine code. If it contains native machine code then the .mpy file has a specific architecture associated with it. Current supported architectures are (these are the valid options for the ARCH variable, see below):

  • x86 (32 bit)

  • x64 (64 bit x86)

  • armv7m (ARM Thumb 2, eg Cortex-M3)

  • armv7emsp (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)

  • armv7emdp (ARM Thumb 2, double precision float, eg Cortex-M7)

  • xtensa (non-windowed, eg ESP8266)

  • xtensawin (windowed with window size 8, eg ESP32)

When compiling and linking the native .mpy file the architecture must be chosen and the corresponding file can only be imported on that architecture. For more details about .mpy files see MicroPython .mpy files.

Native code must be compiled as position independent code (PIC) and use a global offset table (GOT), although the details of this varies from architecture to architecture. When importing .mpy files with native code the import machinery is able to do some basic relocation of the native code. This includes relocating text, rodata and BSS sections.

Supported features of the linker and dynamic loader are:

  • executable code (text)

  • read-only data (rodata), including strings and constant data (arrays, structs, etc)

  • zeroed data (BSS)

  • pointers in text to text, rodata and BSS

  • pointers in rodata to text, rodata and BSS

The known limitations are:

  • data sections are not supported; workaround: use BSS data and initialise the data values explicitly

  • static BSS variables are not supported; workaround: use global BSS variables

So, if your C code has writable data, make sure the data is defined globally, without an initialiser, and only written to within functions.

Linker limitation: the native module is not linked against the symbol table of the full MicroPython firmware. Rather, it is linked against an explicit table of exported symbols found in mp_fun_table (in py/nativeglue.h), that is fixed at firmware build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system function, for example.

New symbols can be added to the end of the table and the firmware rebuilt. The symbols also need to be added to tools/mpy_ld.py’s fun_table dict in the same location. This allows mpy_ld.py to be able to pick the new symbols up and provide relocations for them when the mpy is imported. Finally, if the symbol is a function, a macro or stub should be added to py/dynruntime.h to make it easy to call the function.

Defining a native module

A native .mpy module is defined by a set of files that are used to build the .mpy. The filesystem layout consists of two main parts, the source files and the Makefile:

  • In the simplest case only a single C source file is required, which contains all the code that will be compiled into the .mpy module. This C source code must include the py/dynruntime.h file to access the MicroPython dynamic API, and must at least define a function called mpy_init. This function will be the entry point of the module, called when the module is imported.

    The module can be split into multiple C source files if desired. Parts of the module can also be implemented in Python. All source files should be listed in the Makefile, by adding them to the SRC variable (see below). This includes both C source files as well as any Python files which will be included in the resulting .mpy file.

  • The Makefile contains the build configuration for the module and list the source files used to build the .mpy module. It should define MPY_DIR as the location of the MicroPython repository (to find header files, the relevant Makefile fragment, and the mpy_ld.py tool), MOD as the name of the module, SRC as the list of source files, optionally specify the machine architecture via ARCH, and then include py/dynruntime.mk.

Minimal example

This section provides a fully working example of a simple module named factorial. This module provides a single function factorial.factorial(x) which computes the factorial of the input and returns the result.

Directory layout:

factorial/
├── factorial.c
└── Makefile

The file factorial.c contains:

// Include the header file to get access to the MicroPython API
#include "py/dynruntime.h"

// Helper function to compute factorial
STATIC mp_int_t factorial_helper(mp_int_t x) {
    if (x == 0) {
        return 1;
    }
    return x * factorial_helper(x - 1);
}

// This is the function which will be called from Python, as factorial(x)
STATIC mp_obj_t factorial(mp_obj_t x_obj) {
    // Extract the integer from the MicroPython input object
    mp_int_t x = mp_obj_get_int(x_obj);
    // Calculate the factorial
    mp_int_t result = factorial_helper(x);
    // Convert the result to a MicroPython integer object and return it
    return mp_obj_new_int(result);
}
// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);

// This is the entry point and is called when the module is imported
mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
    // This must be first, it sets up the globals dict and other things
    MP_DYNRUNTIME_INIT_ENTRY

    // Make the function available in the module's namespace
    mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));

    // This must be last, it restores the globals dict
    MP_DYNRUNTIME_INIT_EXIT
}

The file Makefile contains:

# Location of top-level MicroPython directory
MPY_DIR = ../../..

# Name of module
MOD = factorial

# Source files (.c or .py)
SRC = factorial.c

# Architecture to build for (x86, x64, armv7m, xtensa, xtensawin)
ARCH = x64

# Include to get the rules for compiling and linking the module
include $(MPY_DIR)/py/dynruntime.mk

Compiling the module

The prerequisite tools needed to build a native .mpy file are:

  • The MicroPython repository (at least the py/ and tools/ directories).

  • CPython 3, and the library pyelftools (eg pip install 'pyelftools>=0.25').

  • GNU make.

  • A C compiler for the target architecture (if C source is used).

  • Optionally mpy-cross, built from the MicroPython repository (if .py source is used).

Be sure to select the correct ARCH for the target you are going to run on. Then build with:

$ make

Without modifying the Makefile you can specify the target architecture via:

$ make ARCH=armv7m

Module usage in MicroPython

Once the module is built there should be a file called factorial.mpy. Copy this so it is accessible on the filesystem of your MicroPython system and can be found in the import path. The module can now be accessed in Python just like any other module, for example:

import factorial
print(factorial.factorial(10))
# should display 3628800

Further examples

See examples/natmod/ for further examples which show many of the available features of native .mpy modules. Such features include:

  • using multiple C source files

  • including Python code alongside C code

  • rodata and BSS data

  • memory allocation

  • use of floating point

  • exception handling

  • including external C libraries