// -*- mode:doc; -*- // vim: set syntax=asciidoc: === Infrastructure for Python packages This infrastructure applies to Python packages that use the standard Python setuptools or pep517 mechanisms as their build system, generally recognizable by the usage of a +setup.py+ script or +pyproject.toml+ file. [[python-package-tutorial]] ==== +python-package+ tutorial First, let's see how to write a +.mk+ file for a Python package, with an example : ------------------------ 01: ################################################################################ 02: # 03: # python-foo 04: # 05: ################################################################################ 06: 07: PYTHON_FOO_VERSION = 1.0 08: PYTHON_FOO_SOURCE = python-foo-$(PYTHON_FOO_VERSION).tar.xz 09: PYTHON_FOO_SITE = http://www.foosoftware.org/download 10: PYTHON_FOO_LICENSE = BSD-3-Clause 11: PYTHON_FOO_LICENSE_FILES = LICENSE 12: PYTHON_FOO_ENV = SOME_VAR=1 13: PYTHON_FOO_DEPENDENCIES = libmad 14: PYTHON_FOO_SETUP_TYPE = distutils 15: 16: $(eval $(python-package)) ------------------------ On line 7, we declare the version of the package. On line 8 and 9, we declare the name of the tarball (xz-ed tarball recommended) and the location of the tarball on the Web. Buildroot will automatically download the tarball from this location. On line 10 and 11, we give licensing details about the package (its license on line 10, and the file containing the license text on line 11). On line 12, we tell Buildroot to pass custom options to the Python +setup.py+ script when it is configuring the package. On line 13, we declare our dependencies, so that they are built before the build process of our package starts. On line 14, we declare the specific Python build system being used. In this case the +distutils+ Python build system is used. The four supported ones are +distutils+, +flit+, +pep517+ and +setuptools+. Finally, on line 16, we invoke the +python-package+ macro that generates all the Makefile rules that actually allow the package to be built. [[python-package-reference]] ==== +python-package+ reference As a policy, packages that merely provide Python modules should all be named +python-+ in Buildroot. Other packages that use the Python build system, but are not Python modules, can freely choose their name (existing examples in Buildroot are +scons+ and +supervisor+). The main macro of the Python package infrastructure is +python-package+. It is similar to the +generic-package+ macro. It is also possible to create Python host packages with the +host-python-package+ macro. Just like the generic infrastructure, the Python infrastructure works by defining a number of variables before calling the +python-package+ or +host-python-package+ macros. All the package metadata information variables that exist in the xref:generic-package-reference[generic package infrastructure] also exist in the Python infrastructure: +PYTHON_FOO_VERSION+, +PYTHON_FOO_SOURCE+, +PYTHON_FOO_PATCH+, +PYTHON_FOO_SITE+, +PYTHON_FOO_SUBDIR+, +PYTHON_FOO_DEPENDENCIES+, +PYTHON_FOO_LICENSE+, +PYTHON_FOO_LICENSE_FILES+, +PYTHON_FOO_INSTALL_STAGING+, etc. Note that: * It is not necessary to add +python+ or +host-python+ in the +PYTHON_FOO_DEPENDENCIES+ variable of a package, since these basic dependencies are automatically added as needed by the Python package infrastructure. * Similarly, it is not needed to add +host-setuptools+ to +PYTHON_FOO_DEPENDENCIES+ for setuptools-based packages, since it's automatically added by the Python infrastructure as needed. One variable specific to the Python infrastructure is mandatory: * +PYTHON_FOO_SETUP_TYPE+, to define which Python build system is used by the package. The four supported values are +distutils+, +flit+, +pep517+ and +setuptools+. If you don't know which one is used in your package, look at the +setup.py+ or +pyproject.toml+ file in your package source code, and see whether it imports things from the +distutils+, +flit+ module or the +setuptools+ module. If the package is using a +pyproject.toml+ file without any build-system requires and with a local in-tree backend-path one should use +pep517+. A few additional variables, specific to the Python infrastructure, can optionally be defined, depending on the package's needs. Many of them are only useful in very specific cases, typical packages will therefore only use a few of them, or none. * +PYTHON_FOO_SUBDIR+ may contain the name of a subdirectory inside the package that contains the main +setup.py+ or +pyproject.toml+ file. This is useful, if for example, the main +setup.py+ or +pyproject.toml+ file is not at the root of the tree extracted by the tarball. If +HOST_PYTHON_FOO_SUBDIR+ is not specified, it defaults to +PYTHON_FOO_SUBDIR+. * +PYTHON_FOO_ENV+, to specify additional environment variables to pass to the Python +setup.py+ script (for distutils/setuptools packages) or the +support/scripts/pyinstaller.py+ script (for flit/pep517 packages) for both the build and install steps. Note that the infrastructure is automatically passing several standard variables, defined in +PKG_PYTHON_DISTUTILS_ENV+ (for distutils target packages), +HOST_PKG_PYTHON_DISTUTILS_ENV+ (for distutils host packages), +PKG_PYTHON_SETUPTOOLS_ENV+ (for setuptools target packages), +HOST_PKG_PYTHON_SETUPTOOLS_ENV+ (for setuptools host packages), +PKG_PYTHON_PEP517_ENV+ (for flit/pep517 target packages) and +HOST_PKG_PYTHON_PEP517_ENV+ (for flit/pep517 host packages). * +PYTHON_FOO_BUILD_OPTS+, to specify additional options to pass to the Python +setup.py+ script during the build step, this generally only makes sense to use for distutils/setuptools based packages as flit/pep517 based packages do not pass these options to a +setup.py+ script but instead pass them to +support/scripts/pyinstaller.py+. For target distutils packages, the +PKG_PYTHON_DISTUTILS_BUILD_OPTS+ options are already passed automatically by the infrastructure. * +PYTHON_FOO_INSTALL_TARGET_OPTS+, +PYTHON_FOO_INSTALL_STAGING_OPTS+, +HOST_PYTHON_FOO_INSTALL_OPTS+ to specify additional options to pass to the Python +setup.py+ script (for distutils/setuptools packages) or +support/scripts/pyinstaller.py+ (for flit/pep517 packages) during the target installation step, the staging installation step or the host installation, respectively. Note that the infrastructure is automatically passing some options, defined in +PKG_PYTHON_DISTUTILS_INSTALL_TARGET_OPTS+ or +PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS+ (for target distutils packages), +HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS+ (for host distutils packages), +PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS+ or +PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS+ (for target setuptools packages), +HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS+ (for host setuptools packages) and +PKG_PYTHON_PEP517_INSTALL_TARGET_OPTS+ or +PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS+ (for target flit/pep517 packages). With the Python infrastructure, all the steps required to build and install the packages are already defined, and they generally work well for most Python-based packages. However, when required, it is still possible to customize what is done in any particular step: * By adding a post-operation hook (after extract, patch, configure, build or install). See xref:hooks[] for details. * By overriding one of the steps. For example, even if the Python infrastructure is used, if the package +.mk+ file defines its own +PYTHON_FOO_BUILD_CMDS+ variable, it will be used instead of the default Python one. However, using this method should be restricted to very specific cases. Do not use it in the general case. [[scanpypi]] ==== Generating a +python-package+ from a PyPI repository If the Python package for which you would like to create a Buildroot package is available on PyPI, you may want to use the +scanpypi+ tool located in +utils/+ to automate the process. You can find the list of existing PyPI packages https://pypi.python.org[here]. +scanpypi+ requires Python's +setuptools+ package to be installed on your host. When at the root of your buildroot directory just do : ----------------------- utils/scanpypi foo bar -o package ----------------------- This will generate packages +python-foo+ and +python-bar+ in the package folder if they exist on https://pypi.python.org. Find the +external python modules+ menu and insert your package inside. Keep in mind that the items inside a menu should be in alphabetical order. Please keep in mind that you'll most likely have to manually check the package for any mistakes as there are things that cannot be guessed by the generator (e.g. dependencies on any of the python core modules such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the license and license files are guessed and must be checked. You also need to manually add the package to the +package/Config.in+ file. If your Buildroot package is not in the official Buildroot tree but in a br2-external tree, use the -o flag as follows: ----------------------- utils/scanpypi foo bar -o other_package_dir ----------------------- This will generate packages +python-foo+ and +python-bar+ in the +other_package_directory+ instead of +package+. Option +-h+ will list the available options: ----------------------- utils/scanpypi -h ----------------------- [[python-package-cffi-backend]] ==== +python-package+ CFFI backend C Foreign Function Interface for Python (CFFI) provides a convenient and reliable way to call compiled C code from Python using interface declarations written in C. Python packages relying on this backend can be identified by the appearance of a +cffi+ dependency in the +install_requires+ field of their +setup.py+ file. Such a package should: * add +python-cffi+ as a runtime dependency in order to install the compiled C library wrapper on the target. This is achieved by adding +select BR2_PACKAGE_PYTHON_CFFI+ to the package +Config.in+. ------------------------ config BR2_PACKAGE_PYTHON_FOO bool "python-foo" select BR2_PACKAGE_PYTHON_CFFI # runtime ------------------------ * add +host-python-cffi+ as a build-time dependency in order to cross-compile the C wrapper. This is achieved by adding +host-python-cffi+ to the +PYTHON_FOO_DEPENDENCIES+ variable. ------------------------ ################################################################################ # # python-foo # ################################################################################ ... PYTHON_FOO_DEPENDENCIES = host-python-cffi $(eval $(python-package)) ------------------------