mirror of
https://github.com/mozilla/gecko-dev.git
synced 2024-11-23 21:01:08 +00:00
Bug 1755562: Document Mach dependency management r=ahal
* Restructure "Using third-party Python packages" page to focus on the "Mach commands"/"adding a Python package" use case since that's why most people will be looking at these docs. * Document the `<site>_virtualenv_packages.txt` behaviour and how it relates to a Mach command's definition. * Simplify the information around using a non-PyPI index to reference the RelEng docs directly. It's a shame that the existing docs don't explain how to identify tasks that need to use the internal mirror, because I'm not sure either. There's existing cases of ad-hoc `pip` installs //not// using the mirror, but the pattern isn't clear to me. * Remove the "specify hashes" information, since the centralized solution (will) automatically manage this internally. * Arguably, it's still beneficial instructions for ad-hoc `pip install` usages, but those are frowned upon today anyways - use the centralized solution! Differential Revision: https://phabricator.services.mozilla.com/D138931
This commit is contained in:
Mitchell Hentges
parent
7ae4626035
commit
1ba4a4ea2d
@ -2,62 +2,134 @@
|
||||
Using third-party Python packages
|
||||
=================================
|
||||
|
||||
When using third-party Python packages, there are two options:
|
||||
Mach and its associated commands have a variety of 3rd-party Python dependencies. Many of these
|
||||
are vendored in ``third_party/python``, while others are installed at runtime via ``pip``.
|
||||
Dependencies with "native code" are handled on a machine-by-machine basis.
|
||||
|
||||
#. Install/use a vendored version of the package.
|
||||
#. Install the package from a package index, such as PyPI or our internal
|
||||
mirror.
|
||||
The dependencies of Mach itself can be found at ``build/mach_virtualenv_packages.txt``. Mach commands
|
||||
may have additional dependencies which are specified at ``build/<site>_virtualenv_packages.txt``.
|
||||
|
||||
Vendoring Python packages
|
||||
=========================
|
||||
For example, the following Mach command would have its 3rd-party dependencies declared at
|
||||
``build/foo_virtualenv_packages.txt``.
|
||||
|
||||
If the Python package is to be used in the building of Firefox itself, then we
|
||||
**MUST** use a vendored version. This ensures that to build Firefox we only
|
||||
require a checkout of the source, and do not depend on a package index. This
|
||||
ensures that building Firefox is deterministic and dependable, avoids packages
|
||||
from changing out from under us, and means we’re not affected when 3rd party
|
||||
services are offline. We don't want a DoS against PyPI or a random package
|
||||
maintainer removing an old tarball to delay a Firefox chemspill.
|
||||
.. code:: python
|
||||
|
||||
Where possible, the following policy applies to **ALL** vendored packages:
|
||||
|
||||
* Vendored libraries **SHOULD NOT** be modified except as required to
|
||||
successfully vendor them.
|
||||
* Vendored libraries **SHOULD** be released copies of libraries available on
|
||||
PyPI.
|
||||
@Command(
|
||||
"foo-it",
|
||||
virtualenv_name="foo",
|
||||
)
|
||||
# ...
|
||||
def foo_it_command():
|
||||
command_context.activate_virtualenv()
|
||||
import specific_dependency
|
||||
|
||||
The format of ``<site>_virtualenv_requirements.txt`` files are documented further in the
|
||||
:py:class:`~mach.requirements.MachEnvRequirements` class.
|
||||
|
||||
Adding a Python package
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
=======================
|
||||
|
||||
There's two ways of using 3rd-party Python dependencies:
|
||||
|
||||
* :ref:`pip install the packages <python-pip-install>`. Python dependencies with native code must
|
||||
be installed using ``pip``. This is the recommended technique for adding new Python dependencies.
|
||||
* :ref:`Vendor the source of the Python package in-tree <python-vendor>`. Dependencies of the Mach
|
||||
core logic or of building Firefox itself must be vendored.
|
||||
|
||||
.. note::
|
||||
|
||||
If encountering an ``ImportError``, even after following either of the above techniques,
|
||||
then the issue could be that the package is being imported too soon.
|
||||
Move the import to after ``.activate()``/``.activate_virtualenv()`` to resolve the issue.
|
||||
|
||||
This will be fixed by `bug 1717104 <https://bugzilla.mozilla.org/show_bug.cgi?id=1717104>`__.
|
||||
|
||||
.. _python-pip-install:
|
||||
|
||||
``pip install`` the package
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To add a ``pip install``-d package dependency, add it to your site's
|
||||
``build/<site>_virtualenv_packages.txt`` manifest file:
|
||||
|
||||
.. code::
|
||||
|
||||
...
|
||||
pypi:new-package==<version>
|
||||
...
|
||||
|
||||
.. note::
|
||||
|
||||
Some tasks are not permitted to use external resources, and for those we can
|
||||
publish packages to an internal PyPI mirror.
|
||||
See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_
|
||||
for more details.
|
||||
|
||||
.. _python-vendor:
|
||||
|
||||
Vendoring Python packages
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To vendor a Python package, add it to ``third_party/python/requirements.in``
|
||||
and then run ``mach vendor python``. This will update the tree of pinned
|
||||
dependencies in ``third_party/python/requirements.txt`` and download them all
|
||||
into the ``third_party/python`` directory.
|
||||
|
||||
What if the package isn't on PyPI?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Next, add that package and any new transitive dependencies (you'll see them added in
|
||||
``third_party/python/requirements.txt``) to the associated site's dependency manifest in
|
||||
``build/<site>_virtualenv_packages.txt``:
|
||||
|
||||
If the package isn't available on any Python package index, then you can
|
||||
manually copy the source distribution into the ``third_party/python`` directory.
|
||||
.. code::
|
||||
|
||||
Using a Python package index
|
||||
============================
|
||||
...
|
||||
vendored:third_party/python/new-package
|
||||
vendored:third_party/python/new-package-dependency-foo
|
||||
vendored:third_party/python/new-package-dependency-bar
|
||||
...
|
||||
|
||||
If the Python package is not used in the building of Firefox then it can be
|
||||
installed from a package index. Some tasks are not permitted to use external
|
||||
resources, and for those we can publish packages to an internal PyPI mirror.
|
||||
See `how to upload to internal PyPI <https://wiki.mozilla.org/ReleaseEngineering/How_To/Upload_to_internal_Pypi>`_
|
||||
for more details. If you are not restricted, you can install packages from PyPI
|
||||
or another package index.
|
||||
.. note::
|
||||
|
||||
All packages installed from a package index **MUST** specify hashes to ensure
|
||||
compatibility and protect against remote tampering. Hash-checking mode can be
|
||||
forced on when using ``pip`` be specifying the ``--require-hashes``
|
||||
command-line option. See `hash-checking mode <https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode>`_ for
|
||||
more details.
|
||||
The following policy applies to **ALL** vendored packages:
|
||||
|
||||
Note that when using a Python package index there is a risk that the service
|
||||
could be unavailable, or packages may be updated or even pulled without notice.
|
||||
These issues are less likely with our internal PyPI mirror, but still possible.
|
||||
If this is undesirable, then consider vendoring the package.
|
||||
* Vendored PyPI libraries **MUST NOT** be modified
|
||||
* Vendored libraries **SHOULD** be released copies of libraries available on
|
||||
PyPI.
|
||||
|
||||
* When considering manually vendoring a package, discuss the situation with
|
||||
the ``#build`` team to ensure that other, more maintainable options are exhausted.
|
||||
|
||||
* ``mach vendor python`` **MUST** be run on Linux with Python 3.6.
|
||||
This restriction will be lifted when
|
||||
`bug 1659593 <https://bugzilla.mozilla.org/show_bug.cgi?id=1659593>`_
|
||||
is resolved.
|
||||
|
||||
.. note::
|
||||
|
||||
We require that it is possible to build Firefox using only a checkout of the source,
|
||||
without depending on a package index. This ensures that building Firefox is
|
||||
deterministic and dependable, avoids packages from changing out from under us,
|
||||
and means we’re not affected when 3rd party services are offline. We don't want a
|
||||
DoS against PyPI or a random package maintainer removing an old tarball to delay
|
||||
a Firefox chemspill. Therefore, packages required by Mach core logic or for building
|
||||
Firefox itself must be vendored.
|
||||
|
||||
Package compatibility
|
||||
=====================
|
||||
|
||||
Mach requires that all commands' package requirements be compatible with those of Mach itself.
|
||||
(This is because functions and state created by Mach are still usable from within the commands, and
|
||||
they may still need access to their associated 3rd-party modules).
|
||||
|
||||
However, it is OK for Mach commands to have package requirements which are incompatible with each
|
||||
other. This allows the flexibility for some Mach commands to depend on modern dependencies while
|
||||
other, more mature commands may still only be compatible with a much older version.
|
||||
|
||||
.. note::
|
||||
|
||||
Only one version of a package may be vendored at any given time. If two Mach commands need to
|
||||
have conflicting packages, then at least one of them must ``pip install`` the package instead
|
||||
of vendoring.
|
||||
|
||||
If a Mach command's dependency conflicts with a vendored package, and that vendored package
|
||||
isn't needed by Mach itself, then that vendored dependency should be moved from
|
||||
``mach_virtualenv_packages.txt`` to its associated environment.
|
||||
|
||||
@ -98,6 +98,10 @@ mach command:
|
||||
implement ancillary functionality in Python. If you do implement something
|
||||
in Python, hooking it up to ``mach`` is often trivial.
|
||||
|
||||
How do I use 3rd-party Python packages in my ``mach`` command?
|
||||
--------------------------------------------------------------
|
||||
|
||||
See :ref:`Using third-party Python packages`.
|
||||
|
||||
How does ``mach`` fit into the modules system?
|
||||
----------------------------------------------
|
||||
|
||||
Loading…
Reference in New Issue
Block a user