docs/develop: Add documentation on how to build native .mpy modules.

Author: dpgeorge

docs/develop/cmodules.rst

@@ -1,3 +1,5 @@
+.. _cmodules:
+
 MicroPython external C modules
 ==============================
 
@@ -17,6 +19,10 @@ more sense to keep this external to the main MicroPython repository.
 This chapter describes how to compile such external modules into the
 MicroPython executable or firmware image.
 
+An alternative approach is to use :ref:`natmod` which allows writing custom C
+code that is placed in a .mpy file, which can be imported dynamically in to
+a running MicroPython system without the need to recompile the main firmware.
+
 
 Structure of an external C module
 ---------------------------------

docs/develop/index.rst

@@ -11,3 +11,4 @@ See the `getting started guide
 
    cmodules.rst
    qstr.rst
+   natmod.rst

docs/develop/natmod.rst

@@ -0,0 +1,202 @@
+.. _natmod:
+
+Native machine code in .mpy files
+=================================
+
+This section describes how to build and work with .mpy files that contain native
+machine code from a language other than Python.  This allows you to
+write code in a language like C, compile and link it into a .mpy file, and then
+import this file like a normal Python module.  This can be used for implementing
+functionality which is performance critical, or for including an existing
+library written in another language.
+
+One of the main advantages of using native .mpy files is that native machine code
+can be imported by a script dynamically, without the need to rebuild the main
+MicroPython firmware.  This is in contrast to :ref:`cmodules` which also allows
+defining custom modules in C but they must be compiled into the main firmware image.
+
+The focus here is on using C to build native modules, but in principle any
+language which can be compiled to stand-alone machine code can be put into a
+.mpy file.
+
+A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the
+``tools/`` directory of the project.  This tool takes a set of object files
+(.o files) and links them together to create a native .mpy files.
+
+Supported features and limitations
+----------------------------------
+
+A .mpy file can contain MicroPython bytecode and/or native machine code.  If it
+contains native machine code then the .mpy file has a specific architecture
+associated with it.  Current supported architectures are (these are the valid
+options for the ``ARCH`` variable, see below):
+
+* ``x86`` (32 bit)
+* ``x64`` (64 bit x86)
+* ``armv7m`` (ARM Thumb 2, eg Cortex-M3)
+* ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)
+* ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7)
+* ``xtensa`` (non-windowed, eg ESP8266)
+* ``xtensawin`` (windowed with window size 8, eg ESP32)
+
+When compiling and linking the native .mpy file the architecture must be chosen
+and the corresponding file can only be imported on that architecture.  For more
+details about .mpy files see :ref:`mpy_files`.
+
+Native code must be compiled as position independent code (PIC) and use a global
+offset table (GOT), although the details of this varies from architecture to
+architecture.  When importing .mpy files with native code the import machinery
+is able to do some basic relocation of the native code.  This includes
+relocating text, rodata and BSS sections.
+
+Supported features of the linker and dynamic loader are:
+
+* executable code (text)
+* read-only data (rodata), including strings and constant data (arrays, structs, etc)
+* zeroed data (BSS)
+* pointers in text to text, rodata and BSS
+* pointers in rodata to text, rodata and BSS
+
+The known limitations are:
+
+* data sections are not supported; workaround: use BSS data and initialise the
+  data values explicitly
+
+* static BSS variables are not supported; workaround: use global BSS variables
+
+So, if your C code has writable data, make sure the data is defined globally,
+without an initialiser, and only written to within functions.
+
+Defining a native module
+------------------------
+
+A native .mpy module is defined by a set of files that are used to build the .mpy.
+The filesystem layout consists of two main parts, the source files and the Makefile:
+
+* In the simplest case only a single C source file is required, which contains all
+  the code that will be compiled into the .mpy module.  This C source code must
+  include the ``py/dynruntime.h`` file to access the MicroPython dynamic API, and
+  must at least define a function called ``mpy_init``.  This function will be the
+  entry point of the module, called when the module is imported.
+
+  The module can be split into multiple C source files if desired.  Parts of the
+  module can also be implemented in Python.  All source files should be listed in
+  the Makefile, by adding them to the ``SRC`` variable (see below).  This includes
+  both C source files as well as any Python files which will be included in the
+  resulting .mpy file.
+
+* The ``Makefile`` contains the build configuration for the module and list the
+  source files used to build the .mpy module.  It should define ``MPY_DIR`` as the
+  location of the MicroPython repository (to find header files, the relevant Makefile
+  fragment, and the ``mpy_ld.py`` tool), ``MOD`` as the name of the module, ``SRC``
+  as the list of source files, optionally specify the machine architecture via ``ARCH``,
+  and then include ``py/dynruntime.mk``.
+
+Minimal example
+---------------
+
+This section provides a fully working example of a simple module named ``factorial``.
+This module provides a single function ``factorial.factorial(x)`` which computes the
+factorial of the input and returns the result.
+
+Directory layout::
+
+    factorial/
+    ├── factorial.c
+    └── Makefile
+
+The file ``factorial.c`` contains:
+
+.. code-block:: c
+
+    // Include the header file to get access to the MicroPython API
+    #include "py/dynruntime.h"
+
+    // Helper function to compute factorial
+    STATIC mp_int_t factorial_helper(mp_int_t x) {
+        if (x == 0) {
+            return 1;
+        }
+        return x * factorial_helper(x - 1);
+    }
+
+    // This is the function which will be called from Python, as factorial(x)
+    STATIC mp_obj_t factorial(mp_obj_t x_obj) {
+        // Extract the integer from the MicroPython input object
+        mp_int_t x = mp_obj_get_int(x_obj);
+        // Calculate the factorial
+        mp_int_t result = factorial_helper(x);
+        // Convert the result to a MicroPython integer object and return it
+        return mp_obj_new_int(result);
+    }
+    // Define a Python reference to the function above
+    STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);
+
+    // This is the entry point and is called when the module is imported
+    mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
+        // This must be first, it sets up the globals dict and other things
+        MP_DYNRUNTIME_INIT_ENTRY
+
+        // Make the function available in the module's namespace
+        mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));
+
+        // This must be last, it restores the globals dict
+        MP_DYNRUNTIME_INIT_EXIT
+    }
+
+The file ``Makefile`` contains:
+
+.. code-block:: make
+
+    # Location of top-level MicroPython directory
+    MPY_DIR = ../../..
+
+    # Name of module
+    MOD = features0
+
+    # Source files (.c or .py)
+    SRC = features0.c
+
+    # Architecture to build for (x86, x64, armv7m, xtensa, xtensawin)
+    ARCH = x64
+
+    # Include to get the rules for compiling and linking the module
+    include $(MPY_DIR)/py/dynruntime.mk
+
+Compiling the module
+--------------------
+
+Be sure to select the correct ``ARCH`` for the target you are going to run on.
+Then build with::
+
+    $ make
+
+Without modifying the Makefile you can specify the target architecture via::
+
+    $ make ARCH=armv7m
+
+Module usage in MicroPython
+---------------------------
+
+Once the module is built there should be a file called ``factorial.mpy``.  Copy
+this so it is accessible on the filesystem of your MicroPython system and can be
+found in the import path.  The module con now be accessed in Python just like any
+other module, for example::
+
+    import factorial
+    print(factorial.factorial(10))
+    # should display 3628800
+
+Further examples
+----------------
+
+See ``examples/natmod/`` for further examples which show many of the available
+features of native .mpy modules.  Such features include:
+
+* using multiple C source files
+* including Python code alongside C code
+* rodata and BSS data
+* memory allocation
+* use of floating point
+* exception handling
+* including external C libraries