distutils: distro config file not copied to overlay directory

[0-wiz-0]

Description

setuptools recently added a distro-config file for distutils (see e.g. pypa/setuptools#2896). pkgsrc uses this feature.
pip does not copy this file to its overlay directory, causing 'pip install' to fail. 'pip install --no-build-isolation' works.

Example output of the error:

pip install 'PyYAML<6'                                            
Defaulting to user installation because normal site-packages is not writeable                                                                            
Collecting PyYAML<6                   
  Downloading PyYAML-5.4.1.tar.gz (175 kB)                                                                                                               
     |████████████████████████████████| 175 kB 3.9 MB/s            
  Installing build dependencies ... done                                                                                                                 
  Getting requirements to build wheel ... error            
  ERROR: Command errored out with exit status 1:                                                                                                         
   command: /usr/pkg/bin/python3.10 /usr/pkg/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/
tmp7uz3x8rj                                                                                                                                              
       cwd: /tmp/pip-install-q0d2qju0/pyyaml_cb0b2188e0a243029f850480e1a32cef
  Complete output (31 lines):                                                                                                                            
  Traceback (most recent call last):                                        
    File "/tmp/pip-build-env-o87pew3o/overlay/lib/python3.10/site-packages/setuptools/_distutils/sysconfig.py", line 454, in _init_posix    
      _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
  ModuleNotFoundError: No module named '_sysconfigdata__netbsd9_'

(This particular error happens because the pkgsrc distro config file overrides _sysconfig_name_tmpl to _sysconfigdata_${platform}.)

Expected behavior

pip should provide the _distutils_system_mod.py in its overlay directory, if the file exists.

pip version

21.3.1

Python version

3.10.2

OS

NetBSD

How to Reproduce

pip install 'PyYAML<6' though probably any package works.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

A PR to fix this would be welcome. It should be noted, however, that NetBSD is not a supported platform so a reproducer for this on a supported platform would also be helpful here.

[0-wiz-0]

@pfmoore I don't know the code base, can you please give me some pointers where this could be added?
Btw, when you follow the first link above, you'll see that e.g. Fedora will also be affected. I don't have a Fedora system available though.

[pfmoore]

Class BuildEnvironment in src/pip/_internal/build_env.py would be the place to look. You'll need to be able to reproduce the issue on supported platforms in order to create a test, which will be part of creating a PR (the CI will provide the necessary environments, you "just" need to write suitable test code).

[0-wiz-0]

Here's the shell equivalent of what needs to happen:

cp ${PREFIX}/lib/python${PYVERSION}/site-packages/_distutils_system_mod.py ${OVERLAY}/lib/python${PYVERSION}/site-packages  || true

Can you please add this in the appropriate python code?
I don't have access to Fedora, and I don't know the code base.

[q0w]

@0-wiz-0 it works on fedora with overridden _sysconfig_name_tmpl

[pradyunsg]

If you could provide a way to reproduce this, on a Linux system (eg: Fedora, RHEL, Ubuntu, Debian etc) that'd be great.

[0-wiz-0]

This patch fixes the problem for me:

--- src/pip/_internal/build_env.py.orig 2021-10-22 15:15:54.000000000 +0000
+++ src/pip/_internal/build_env.py
@@ -9,6 +9,7 @@ import sys
 import textwrap
 import zipfile
 from collections import OrderedDict
+from shutil import copy
 from sysconfig import get_paths
 from types import TracebackType
 from typing import TYPE_CHECKING, Iterable, Iterator, List, Optional, Set, Tuple, Type
@@ -93,6 +94,9 @@ class BuildEnvironment:
         self._site_dir = os.path.join(temp_dir.path, "site")
         if not os.path.exists(self._site_dir):
             os.mkdir(self._site_dir)
+        distutils_distro_config = get_paths()["platlib"] + "/_distutils_system_mod.py"
+        if os.path.exists(distutils_distro_config):
+            copy(distutils_distro_config, self._site_dir)
         with open(os.path.join(self._site_dir, "sitecustomize.py"), "w") as fp:
             fp.write(
                 textwrap.dedent(

[0-wiz-0]

build_env.py stopped importing syconfig, so the patch is now:

--- src/pip/_internal/build_env.py.orig 2023-02-17 18:31:10.000000000 +0000
+++ src/pip/_internal/build_env.py
@@ -8,6 +8,8 @@ import site
 import sys
 import textwrap
 from collections import OrderedDict
+from shutil import copy
+from sysconfig import get_paths
 from types import TracebackType
 from typing import TYPE_CHECKING, Iterable, List, Optional, Set, Tuple, Type, Union
 
@@ -102,6 +104,9 @@ class BuildEnvironment:
         self._site_dir = os.path.join(temp_dir.path, "site")
         if not os.path.exists(self._site_dir):
             os.mkdir(self._site_dir)
+        distutils_distro_config = get_paths()["platlib"] + "/_distutils_system_mod.py"
+        if os.path.exists(distutils_distro_config):
+            copy(distutils_distro_config, self._site_dir)
         with open(
             os.path.join(self._site_dir, "sitecustomize.py"), "w", encoding="utf-8"
         ) as fp:

[uranusjr]

Why does the file need to be copied? The point of having an isolated environment is so it has a fresh set of build tools and does not need to rely on the outside environment. This is conceptually against the grains of environment isolation.

[0-wiz-0]

pkgsrc has patches for Python to handle different operating systems the same way, to make the list of installed files the same. This is a property of Python itself on platforms using pkgsrc.
distutils needs overrides for this to work, since distutils does not use sysconfig but its own version, distutils.sysconfig.
More details can be found here and here.

[uranusjr]

This sounds like something pip should not do. Unless the distro override is standardised, there’s no reason why pip needs to implement a special treatment for a setuptools-specific mechanism. Environment isolation and PEP 517 is designed exactly to abstract away build tools specifics in the first place.

[pfmoore]

I think I agree with @uranusjr here. The linked setuptools issue describes the override as a short term solution. It also appears that there is a similar request for virtualenv. Does the stdlib venv module need this too? That’s a lot of work to handle a short term problem in setuptools.

I think that either the mechanism being used needs to be standardised, or setuptools needs to come up with a fix that doesn’t require multiple other tools to add special cases for it.

[0-wiz-0]

@jaraco writes about a long-term solution in distutils, true, but as far as I understand, distutils development is dead, so that will not be coming. Perhaps they can chime in about distutils development and if that change will ever come?

@jaraco also writes that pip previously copied distutils, so copying one file instead is less work and equivalent.

[jaraco]

Distutils development is alive and well insofar as it’s used as a basis for Setuptools. Other uses are discouraged but still supported.