AssertionError when importing pip before setuptools

[jaraco]

    The following triggers the same error:

$ docker run -it python:3.9 bash
root@ccdb7166de24:/# pip install setuptools==65.4.1 pip==22.2.2
...
Successfully installed pip-22.2.2 setuptools-65.4.1

root@ccdb7166de24:/# python -c "import pip; import setuptools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/usr/local/lib/python3.9/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 77, in do_override
    ensure_local_distutils()
  File "/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 64, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/local/lib/python3.9/distutils/core.py

Though perhaps not quite the reproducing case you are looking for, as it doesn't include get-pip?

Dockerfile form:

FROM python:3.9
RUN pip install setuptools==65.4.1 pip==22.2.2
RUN python -c "import pip; import setuptools"

Incidentally, a workaround is to import setuptools first, and filter warnings if you then want to import pip. This looks like:

python -c "import setuptools; import warnings; warnings.filterwarnings('ignore', 'Setuptools is replacing distutils'); import pip; import setuptools"

(I care about the latter setuptools because it is something that I don't have full control of in my real-world case)

Originally posted by @pelson in #3439 (comment)

[jaraco]

[importing pip then setuptools] triggers the same error.

Yes. It seems that's the case, and I'm tempted to say that usage is unsupported. pip is meant to be used as a command-line tool, so if it's imported that should generally only happen in an isolated interpreter, and furthermore pip should never import setuptools.

The distutils shim intentionally disables the distutils override when it detects that it's being imported by pip. That function references pypa/pip#8761 as the rationale. Thereafter, when setuptools is imported, it checks to ensure that the override was applied, but since it was disabled by the presence of an import pip, it fails the check. This behavior is by design to prevent edge use cases like these.

I also don't think it's wise to use the workaround either. By importing setuptools then pip, you're causing setuptools to get distutils from its own local copy, but pip and subsequent importers (including possibly late importers in setuptools) to get it from stdlib. This situation isn't great.

Is there a good reason to "import pip" against the guidance of that project?

[pelson]

Is there a good reason to "import pip" against the guidance of that project?

Yes, but not a supported reason. In my case, I need to patch pip. I do so at runtime during Python startup, rather than actually forking the codebase.

Until this issue occurred, I did this patching unconditionally (making sure to keep the patching quick to avoid any significant startup overhead). I have just updated this to only patch pip when it is first imported (by attaching a lazy module in sys.modules) - the resulting code is certainly more complex, but has the advantage of only affecting and importing pip if it is actually used (i.e. by pip), thus making the patching compatible with a process that imports setuptools.

It is unusual that one can't import two packages in the same process though. Thank you for providing the links on the history of the underlying reasons - I have one question that wasn't clear from what I read though (please feel free to link to a discussion elsewhere):

Why is it that setuptools knows that it needs to have a patched version of distutils, and yet cannot use a (setuptools) namespaced version of distutils? Is there fundamentally some design limitation of distutils that forced this upon you? This issue is fundamentally caused by setuptools calling its patched version distutils from what I can see, so would be interesting to know if that could be avoided.

In terms of this issue, I am happy in principle to resolve this as a wont-fix - import pip is clearly documented as not supported, and therefore no need for import pip, setuptools to be supported.