Skip to content

virtualenv's develop branch fails to create virtualenvs with a different Python interpreter #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mgedmin opened this issue Dec 2, 2014 · 17 comments

Comments

@mgedmin
Copy link

mgedmin commented Dec 2, 2014

Steps to reproduce:

  1. git clone -b develop https://github.com/pypa/virtualenv
  2. cd virtualenv
  3. python2.7 virtualenv.py -p python3.4 /tmp/py34

The error is

Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

I get the same error if I use -p python2.6, or -p python3.3, or -p python3.2, or -p python3.1.

I don't get the error if I use -p python2.7 or -p pypy.

I don't get the error if I switch to the master branch and use -p python3.4.

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

Reverting commit dde220e fixes this for me.

mgedmin added a commit to mgedmin/virtualenv that referenced this issue Dec 2, 2014
@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

The irony is that dde220e was supposed to fix the same issue as this one, only in a different way.

Pinging @nvie, @saulshanabrook: can you test if this PR doesn't regress your test case (#625)?

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

Apologies, "this PR" is #672. I meant to post the comment there, not here. Too many open tabs!

@nvie
Copy link

nvie commented Dec 2, 2014

The funny thing is that I cannot reproduce the behaviour you mention:

$ git clone -b develop https://github.com/pypa/virtualenv
$ cd virtualenv
$ python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /Users/nvie/.pythonz/pythons/CPython-3.4.2/bin/python3.4
Using base prefix '/Users/nvie/.pythonz/pythons/CPython-3.4.2'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
Installing setuptools, pip...done.
$ /tmp/py34/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/tmp/py34

@nvie
Copy link

nvie commented Dec 2, 2014

I've tried Python versions 2.6, 2.7, 3.2, 3.3, 3.4, but none of them pose this problem. They all work.

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

I'm on Ubuntu 14.10.

$ python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

$ /tmp/py34/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/usr

$ tree /tmp/py34
/tmp/py34
├── bin
│   ├── python -> python3.4
│   ├── python3 -> python3.4
│   └── python3.4
├── include
│   └── python3.4m -> /usr/include/python3.4m
└── lib
    └── python3.4
        ├── config-3.4dm-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4dm-x86_64-linux-gnu
        ├── config-3.4m-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4m-x86_64-linux-gnu
        ├── lib-dynload -> /usr/lib/python3.4/lib-dynload
        ├── no-global-site-packages.txt
        ├── orig-prefix.txt
        ├── plat-x86_64-linux-gnu -> /usr/lib/python3.4/plat-x86_64-linux-gnu
        ├── site-packages
        └── site.py

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

As opposed to

$ python3.4 virtualenv.py /tmp/py34good --no-setuptools
Using base prefix '/usr'
New python executable in /tmp/py34good/bin/python3.4
Also creating executable in /tmp/py34good/bin/python

$ /tmp/py34good/bin/python -c 'import sys; print(sys.version_info); print(sys.prefix)'
sys.version_info(major=3, minor=4, micro=2, releaselevel='final', serial=0)
/tmp/py34good

$ tree /tmp/py34good
/tmp/py34good
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── python -> python3.4
│   ├── python3 -> python3.4
│   └── python3.4
├── include
│   └── python3.4m -> /usr/include/python3.4m
└── lib
    └── python3.4
        ├── abc.py -> /usr/lib/python3.4/abc.py
        ├── base64.py -> /usr/lib/python3.4/base64.py
        ├── bisect.py -> /usr/lib/python3.4/bisect.py
        ├── _bootlocale.py -> /usr/lib/python3.4/_bootlocale.py
        ├── codecs.py -> /usr/lib/python3.4/codecs.py
        ├── collections -> /usr/lib/python3.4/collections
        ├── _collections_abc.py -> /usr/lib/python3.4/_collections_abc.py
        ├── config-3.4dm-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4dm-x86_64-linux-gnu
        ├── config-3.4m-x86_64-linux-gnu -> /usr/lib/python3.4/config-3.4m-x86_64-linux-gnu
        ├── copy.py -> /usr/lib/python3.4/copy.py
        ├── copyreg.py -> /usr/lib/python3.4/copyreg.py
        ├── distutils
        │   ├── distutils.cfg
        │   └── __init__.py
        ├── _dummy_thread.py -> /usr/lib/python3.4/_dummy_thread.py
        ├── encodings -> /usr/lib/python3.4/encodings
        ├── fnmatch.py -> /usr/lib/python3.4/fnmatch.py
        ├── functools.py -> /usr/lib/python3.4/functools.py
        ├── __future__.py -> /usr/lib/python3.4/__future__.py
        ├── genericpath.py -> /usr/lib/python3.4/genericpath.py
        ├── hashlib.py -> /usr/lib/python3.4/hashlib.py
        ├── heapq.py -> /usr/lib/python3.4/heapq.py
        ├── hmac.py -> /usr/lib/python3.4/hmac.py
        ├── importlib -> /usr/lib/python3.4/importlib
        ├── imp.py -> /usr/lib/python3.4/imp.py
        ├── io.py -> /usr/lib/python3.4/io.py
        ├── keyword.py -> /usr/lib/python3.4/keyword.py
        ├── lib-dynload -> /usr/lib/python3.4/lib-dynload
        ├── linecache.py -> /usr/lib/python3.4/linecache.py
        ├── locale.py -> /usr/lib/python3.4/locale.py
        ├── no-global-site-packages.txt
        ├── ntpath.py -> /usr/lib/python3.4/ntpath.py
        ├── operator.py -> /usr/lib/python3.4/operator.py
        ├── orig-prefix.txt
        ├── os.py -> /usr/lib/python3.4/os.py
        ├── __pycache__
        │   ├── abc.cpython-34.pyc
        │   ├── _bootlocale.cpython-34.pyc
        │   ├── codecs.cpython-34.pyc
        │   ├── _collections_abc.cpython-34.pyc
        │   ├── genericpath.cpython-34.pyc
        │   ├── io.cpython-34.pyc
        │   ├── os.cpython-34.pyc
        │   ├── posixpath.cpython-34.pyc
        │   ├── site.cpython-34.pyc
        │   ├── stat.cpython-34.pyc
        │   └── _weakrefset.cpython-34.pyc
        ├── plat-x86_64-linux-gnu -> /usr/lib/python3.4/plat-x86_64-linux-gnu
        ├── posixpath.py -> /usr/lib/python3.4/posixpath.py
        ├── random.py -> /usr/lib/python3.4/random.py
        ├── re.py -> /usr/lib/python3.4/re.py
        ├── reprlib.py -> /usr/lib/python3.4/reprlib.py
        ├── rlcompleter.py -> /usr/lib/python3.4/rlcompleter.py
        ├── shutil.py -> /usr/lib/python3.4/shutil.py
        ├── site-packages
        ├── site.py
        ├── sre_compile.py -> /usr/lib/python3.4/sre_compile.py
        ├── sre_constants.py -> /usr/lib/python3.4/sre_constants.py
        ├── sre_parse.py -> /usr/lib/python3.4/sre_parse.py
        ├── stat.py -> /usr/lib/python3.4/stat.py
        ├── struct.py -> /usr/lib/python3.4/struct.py
        ├── tarfile.py -> /usr/lib/python3.4/tarfile.py
        ├── tempfile.py -> /usr/lib/python3.4/tempfile.py
        ├── types.py -> /usr/lib/python3.4/types.py
        ├── tokenize.py -> /usr/lib/python3.4/tokenize.py
        ├── token.py -> /usr/lib/python3.4/token.py
        ├── warnings.py -> /usr/lib/python3.4/warnings.py
        ├── weakref.py -> /usr/lib/python3.4/weakref.py
        └── _weakrefset.py -> /usr/lib/python3.4/_weakrefset.py

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

I suspect what happens here is that sys.path[0] is now deleted twice: once at the top level and once again in copy_required_modules, which makes the latter be unable to find the required bootstrap modules.

My sys.path looks like this:

$ python3.4 -c 'import sys; print("\n".join(map(repr, sys.path)))'
''
'/usr/lib/python3.4'
'/usr/lib/python3.4/plat-x86_64-linux-gnu'
'/usr/lib/python3.4/lib-dynload'
'/home/mg/.local/lib/python3.4/site-packages'
'/usr/local/lib/python3.4/dist-packages'
'/usr/lib/python3/dist-packages'

and if I run virtualenv with the -v flag, I see the expected errors about "Cannot import bootstrap module":

$ python2.7 virtualenv.py -p python3.4 /tmp/py34 -v
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
Directory /tmp/py34/lib/python3.4 already exists
Symlinking Python bootstrap modules
  Cannot import bootstrap module: os
  Ignoring built-in bootstrap module: posix
  Cannot import bootstrap module: posixpath
  Cannot import bootstrap module: nt
  Cannot import bootstrap module: ntpath
  Cannot import bootstrap module: genericpath
  Cannot import bootstrap module: fnmatch
  Cannot import bootstrap module: locale
  Cannot import bootstrap module: encodings
  Cannot import bootstrap module: codecs
  Cannot import bootstrap module: stat
  Cannot import bootstrap module: UserDict
  Cannot import bootstrap module: copy_reg
  Cannot import bootstrap module: types
  Cannot import bootstrap module: re
  Cannot import bootstrap module: sre
  Cannot import bootstrap module: sre_parse
  Cannot import bootstrap module: sre_constants
  Cannot import bootstrap module: sre_compile
  Ignoring built-in bootstrap module: zlib
  Cannot import bootstrap module: _abcoll
  Cannot import bootstrap module: warnings
  Cannot import bootstrap module: linecache
  Cannot import bootstrap module: abc
  Cannot import bootstrap module: io
  Cannot import bootstrap module: _weakrefset
  Cannot import bootstrap module: copyreg
  Cannot import bootstrap module: tempfile
  Cannot import bootstrap module: random
  Cannot import bootstrap module: __future__
  Cannot import bootstrap module: collections
  Cannot import bootstrap module: keyword
  Cannot import bootstrap module: tarfile
  Cannot import bootstrap module: shutil
  Cannot import bootstrap module: struct
  Cannot import bootstrap module: copy
  Cannot import bootstrap module: tokenize
  Cannot import bootstrap module: token
  Cannot import bootstrap module: functools
  Cannot import bootstrap module: heapq
  Cannot import bootstrap module: bisect
  Cannot import bootstrap module: weakref
  Cannot import bootstrap module: reprlib
  Cannot import bootstrap module: base64
  Cannot import bootstrap module: _dummy_thread
  Cannot import bootstrap module: hashlib
  Cannot import bootstrap module: hmac
  Cannot import bootstrap module: imp
  Cannot import bootstrap module: importlib
  Cannot import bootstrap module: rlcompleter
  Cannot import bootstrap module: operator
  Cannot import bootstrap module: _collections_abc
  Cannot import bootstrap module: _bootlocale
Directory /tmp/py34/lib/python3.4/site-packages already exists
Content /tmp/py34/lib/python3.4/site.py already in place
Content /tmp/py34/lib/python3.4/orig-prefix.txt already in place
Content /tmp/py34/lib/python3.4/no-global-site-packages.txt already in place
Directory /tmp/py34/bin already exists
New python executable in /tmp/py34/bin/python3.4
Changed mode of /tmp/py34/bin/python3.4 to 0o775
Not overwriting existing python script /tmp/py34/bin/python (you must use /tmp/py34/bin/python3.4)
Testing executable with /tmp/py34/bin/python3.4 -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
ERROR: The executable /tmp/py34/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/usr' (should be '/tmp/py34')
ERROR: virtualenv is not compatible with this system or executable

I'm curious to know why virtualenv doesn't fail for you.

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

Is PYTHONPATH set in your environment? Because in that case there are extra items in front of the stdlib path in sys.path and this error does not occur.

$ PYTHONPATH=something:extra python3.4 -c 'import sys; print("\n".join(map(repr, sys.path)))'
''
'/home/mg/src/virtualenv/something'
'/home/mg/src/virtualenv/extra'
'/usr/lib/python3.4'
'/usr/lib/python3.4/plat-x86_64-linux-gnu'
'/usr/lib/python3.4/lib-dynload'
'/home/mg/.local/lib/python3.4/site-packages'
'/usr/local/lib/python3.4/dist-packages'
'/usr/lib/python3/dist-packages'

$ PYTHONPATH=somethingextra python2.7 virtualenv.py -p python3.4 /tmp/py34
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Not overwriting existing python script /tmp/py34/bin/python (you must use /tmp/py34/bin/python3.4)
Installing setuptools, pip...done.

@nvie
Copy link

nvie commented Dec 2, 2014

I suspect what happens here is that sys.path[0] is now deleted twice: once at the top level and once again in copy_required_modules […]

Ohhh. I wasn't even aware this was happening when I added my patch. Therefore I might indeed have broken something. Removing it at the top level is required to make sure the following imports works as expected. I think we can safely remove the sys.path modification from the copy_required_modules() function (in favor of the top-level one). What happens if you simply remove that part?

@mgedmin
Copy link
Author

mgedmin commented Dec 2, 2014

Here's the patch that I tested on top of current develop (commit b7e489f):

diff --git a/virtualenv.py b/virtualenv.py
index 2f43427..dd23ff3 100755
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -1113,8 +1113,8 @@ def copy_required_modules(dst_prefix, symlink):
     # virtualenv.py is installed under (which might lead to py2/py3
     # incompatibility issues)
     _prev_sys_path = sys.path
-    if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
-        sys.path = sys.path[1:]
+##  if os.environ.get('VIRTUALENV_INTERPRETER_RUNNING'):
+##      sys.path = sys.path[1:]
     try:
         for modname in REQUIRED_MODULES:
             if modname in sys.builtin_module_names:

With it the following works:

$ python2.7 virtualenv.py -p python3.4 /tmp/py34 
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /tmp/py34/bin/python3.4
Also creating executable in /tmp/py34/bin/python
Installing setuptools, pip...done.

but #671 remains a problem for me. Digging deeper, the del sys.path[0] is on line 30, but the import error I get happens before that, on line 8 (import base64). To fix that you'd have to reorder the imports (put import sys at the top) and then do the sys.path deletion before importing the rest (especially base64).

My patch in #672 fixes both this bug and #671. I ike it conceptually (the wrong Python's site-packages never get a chance to show up in sys.path) but dislike the implementation (the one-liner that exec()s the contents of a script is grotty).

@nvie
Copy link

nvie commented Dec 3, 2014

I've given the exec() call some thought myself when I implemented my patch, but I refrained from it because it does depend on implementation details of the runtime (i.e. you're setting __file__ and __name__, but __loader__ or __package__ may also exist. Do they need to be set too? Are there more? I wanted to stay away from a solution that would force decisions there.)

Have you tried simply moving the base64 import down until after the path modification? That sounds like the correct thing to do anyway, and it may also be the simplest solution.

@mgedmin
Copy link
Author

mgedmin commented Dec 3, 2014

Yes, moving the base64 import down fixes both issues (this one and #671). Would you like me to submit a pull request?

mgedmin added a commit to mgedmin/virtualenv that referenced this issue Dec 3, 2014
Fixes pypa#671, pypa#673.

base64 should be imported *after* sys.path manipulation -- not doing
this causes pypa#671.

sys.path is fixed up at the top level (since commit dde220e),
doing it again in copy_required_modules() breaks things -- this is pypa#673.
This makes the try/finally to restore the old sys.path unnecessary,
which produces the gnarly diff :/
@nvie
Copy link

nvie commented Dec 3, 2014

So the complete fix then boils down to (1) moving the import base64 down, and (2) removing this part, right? Sounds simple enough. I'd say go for it (even though I'm not a maintainer, I'm just a mere contributor) :)

@mgedmin
Copy link
Author

mgedmin commented Dec 22, 2014

This should be fixed by the merge of #674 now.

@mgedmin mgedmin closed this as completed Dec 22, 2014
@mgedmin
Copy link
Author

mgedmin commented Jan 7, 2015

This one also regressed when #674 was reverted.

@mgedmin mgedmin reopened this Jan 7, 2015
@mgedmin
Copy link
Author

mgedmin commented Jan 7, 2015

Actually never mind, I cannot reproduce with 12.0.5.

@mgedmin mgedmin closed this as completed Jan 7, 2015
@mgedmin mgedmin mentioned this issue Jan 12, 2015
5 tasks
@pypa pypa locked and limited conversation to collaborators Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants