Using Tox with Poetry dependency groups

fievelk · December 26, 2024

How To   Python

Poetry is one of the most famous dependency management tools for Python, and I frequently use it in place of the basic requirements.txt to simplify version management in my libraries. Despite its widespread adoption, Poetry is not fully compliant with Python’s PEPs and this sometimes leads to challenges during the development process. One such challenge I recently encountered was integrating Poetry’s “dev dependencies” with Tox.

In my projects, I usually specify two groups of dependencies with Poetry:

  • the main dependency group: these dependencies will be installed with my library as part of the regular installation process (for example: pip install my-project).
  • An additional dependency group called dev: these additional dependencies are only used during development and testing and can only be installed through Poetry (for example: poetry install --with dev).

Assuming this simple setup, the initial pyproject.toml file would look like this:

[tool.poetry]
name = "my-project"
version = "0.1.0"
authors = ["me"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = "^3.11"

[tool.poetry.group.dev.dependencies]
pytest = "^8.3.2"
tox = "^4.23.2"

As you can see, pytest and tox are both part of the tool.poetry.group.dev.dependencies group: we don’t want them to be installed by default with our library (since most users won’t need them) but we still need to declare them, as they are necessary for development and testing.

Tox is a “generic virtual environment management and test command line tool” that can be used to run tests in multiple, isolated environments (for example different Python versions with different dependencies). For this example, let’s pretend we want Tox to run pytest on two environments: Python 3.11 and 3.12.

The following tox.ini configuration looks reasonable, but it won’t work:

[tox]
envlist =
    py312

[testenv]
description = Run tests using pytest
allowlist_externals = pytest
passenv = *
commands =
    pytest

The problem with this configuration is that Tox will not automatically install the dev dependencies we declared in pyproject.toml. Unfortunately, in fact, Poetry’s dependency groups (like our tool.poetry.group.dev.dependencies above) are not a standard and do not comply with existing PEPs, so Tox won’t detect tool.poetry.group.dev.dependencies and won’t install them.

One simple solution would be to just re-install all the dependencies needed by Tox (pytest, in our example) in tox.ini:

[tox]
envlist =
    py312

[testenv]
allowlist_externals = pytest
passenv = *
commands_pre =
    pip install pytest==8.3.4  # Install pytest explicitly before running the command below
commands =
    pytest

However, this means that we will duplicate our dependency declarations (pytest and its version are now declared both in pyproject.toml and tox.ini). This could introduce bugs and unexpected inconsistencies between our Tox environment and the one defined by pyproject.toml.

Another possible (but discouraged) way to do it would be to directly use Poetry in tox.ini to install the dev dependencies:

[tox]
envlist =
    py311
    py312

[testenv]
allowlist_externals = poetry, pytest
passenv = *
commands =
    poetry install --with dev
    pytest

While this might work, it will also make Tox aware of Poetry, which is suboptimal and unnecessary: Tox shouldn’t care about how dependencies are installed, nor should it explicitly install them. In fact, by default Tox looks into the [build-system] section of pyproject.toml and uses its value to delegate the dependencies’ build process to the correct backend (poetry.core.masonry.api in our case). This means that we don’t need to explicitly mention poetry in our tox.ini, because Tox already knows about its build system.

The solution

So Tox will use Poetry’s build-backend to install dependencies. But how do we make Tox aware of Poetry’s development dependency group? Simply put: we don’t. As I mentioned, Poetry dependency groups are non-standard and Tox is not aware of them. However, we can use another field that is, in fact, defined in the standard specification for Python packaging (PEP 508): extras.

The pyproject.toml then becomes:

[tool.poetry]
name = "my-project"
version = "0.1.0"
authors = ["me"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = "^3.11"
# Specify the optional dependencies that will be used below as "extras"
pytest = { version = "^8.3.2", optional = true }

# List the extra dependencies from the dependencies we defined above.
# This is used in place of tool.poetry.group.dev.dependencies.
[tool.poetry.extras]
dev = ["pytest"]

# Define dependencies that will be installed by developers using Poetry
[tool.poetry.group.dev.dependencies]
tox = "^4.23.2"

Final notes

Poetry’s popularity reflects the the broad appreciation of Python’s community. However, its widespread adoption also creates constrains regarding backward compatibility, making it challenging for the team to support new PEP standards especially when this involves dropping existing features that users rely on. At the same time, I would feel more confident using tools that fully comply with current standards. Package managers like Hatch or PDM seem to offer this, although their user bases seem significantly smaller than Poetry’s.

On the bright side, PEP 735, a new standard for dependency groups, was recently accepted. Hopefully, this can be a step towards moving Python dependency management tools in the direction of further standardization.

Resources

Twitter, Facebook