Multiple versions of Python on a Linux machine

[Mario F.]

What is the best way to manage multiple versions of Python on a Linux machine?

On my Linux Mint machine at the office, I have:

• Python 3.4, system installation (installed with the distro)
• Python 2.7, system installation (installed with the distro)
• Python 3.5, with make altinstall

I never install packages to the system python versions (or even upgrade them). I just create a virtualenv and install them there.

I did do "sudo pip" for a few packages in the python 3.5 altinstall installation though:

• virtualenv
• iPython + jupyter
• pytest
• coverage
• pylint

The rest I just install to the virtualenvs of my projects.

Do you consider this a good practice, despite "sudo pip" being highly unrecommended?

[Epy]

Interested in this as well. On Windows, they install in different places; I simply have a python3.bat in the python27 folder to redirect to the Python 3.5 installation. That way both Python directories are not in PATH and conflicting.

[phantomotap]

O_o

Why did you need to install `virtualenv` for the newest version?

*shrug*

Anyways, I do the mechanical equivalent of `sudo pip` for packages related to my own work on my own machines for the current Python versions. I wouldn't really call the approach "good practice", but I have the hooks setup for system and Python packages managers so that running my own administration tools installs everything automatically for relevant `chroot` or overlay environment. Basically, a convenience I have no intention of giving up in any event.

Soma

[Mario F.]

Why did you need to install `virtualenv` for the newest version?

You wouldn't believe how old the version of virtualenv is on the default python 2 installation that comes with Linux Mint. And since I make a point of never messing with the package list on the default installations, it's just better to install virtualenv again on one of the altinstall versions.

...

I'll be soon moving away from Mint at the office. I can't stand this nonsense distro anymore. It's just not good for development environments or any other environment where we must follow development cycles of a number of applications and system tools. We spend more time fighting against the aggressively conservative repositories than anything else.

My concern is what other alternative is there for a proper Python installation of multiple versions, and helps me avoid 'sudo pip'?
pyenv won't do. I don't even understand its purpose. To change my global python (even if temporarily) is a great way to break existing tools that rely on a python version I just hid.

[phantomotap]

To change my global python (even if temporarily) is a great way to break existing tools that rely on a python version I just hid.

O_o

I may have been, or may still be, on the wrong page.

Are you asking to use, for example, a `tool1 --dosomestuff` and a `tool2 --dosomestuff` at a terminal without needing to remember the virtual environment binary location or dancing around with scripts to change the current environment stack to account for the dependencies of each application?

I eventually put all of my `virtualenv` directories in the same relative location (`~/.local/opt/$ENV`) and hooked `pip` to create a script handling the environment stack for any binaries (`~/.local/opt/$ENV/bin/application`) because I found myself forgetting which virtual environment owned which projects.

I don't know of any project which creates such a serviceable shim automatically on install or upgrade, but you shouldn't need more than a couple of hours if you don't mind a few restrictions.

*shrug*

If the issue is still just the "recommended" nature of `sudo pip`, you could remove the various global installs and setup three `virtualenv` directories before again installing everything. You would then have no global repository which could ever need a `sudo pip` install, but I wouldn't recommend that so I guess I'll stop talking and maybe you'll get a response from someone knowing more about the state of `virtualenv` wrappers.

Soma

[Mario F.]

Are you asking to use, for example, a `tool1 --dosomestuff` and a `tool2 --dosomestuff` at a terminal without needing to remember the virtual environment binary location or dancing around with scripts to change the current environment stack to account for the dependencies of each application?

I can do this already with 'make altinstall' when installing a new python version. Currently, I execute python, python3 and python3.5 for the python 2.7, python 3.4 and python 3.5 interpreters, respectively. And pip follows the same pattern. That is not a problem.

However 'make altinstall' install python on /usr/local. hence the need for 'sudo pip'.

I eventually put all of my `virtualenv` directories in the same relative location (`~/.local/opt/$ENV`) and hooked `pip` to create a script handling the environment stack for any binaries (`~/.local/opt/$ENV/bin/application`) because I found myself forgetting which virtual environment owned which projects.

It's why sometime ago I gave up on a common folder for virtual envs. Now I just install a virtual environment inside each project folder and add it to .gitignore. I don't have to remember nothing about anything anymore.

If the issue is still just the "recommended" nature of `sudo pip`,

That is precisely the issue
I have 3 global installations (more on the way). 2 of them are installed with the system and the third one was installed by me with 'make altinstall' so it won't try to overwrite the default ones. All is working. But this also means I am now forced to sudo pip when I wish to install packages to the global python installation. It's not really a big deal, because global packages should be avoided anyways. And I could always create a "system env" on my home folder exactly to keep those packages I listed above. But I was still wondering if there was a better way to install multiple global versions of python without ending up with sudo pip.

you could remove the various global installs and setup three `virtualenv` directories before again installing everything.

(note: I could only remove one python version above; 3.5. Any of the other two would break my system, because the distro itself uses them for core system functionality.)

What you propose is possible with a 3rd party tool called pyenv (previously much better named as pythonbrew). It creates global virtual environments, and downloads and builds the python version there. However there is a downside. Those virtual envs are global, so in order to switch to python 3.5 installed by pyenv I have to activate it, which deactivates all other pythons. I don't want to lose access to py2 when I am working for instance on a py3 env.

[phantomotap]

I can do this already with 'make altinstall' when installing a new python version. [...] a py3 env.

O_o

I think that, given the post as a whole, you didn't understand what my setup is doing... or the various `virtualenv` tools have gotten better while I wasn't looking.

My setup doesn't hide things by mucking about with the terminal environment.

I guess I'd describe my setup as, in a way, continuing the naming pattern you reference but with any `virtualenv` you've setup.

[Edit]

I don't have to throw a `pyenv -whatever py27` (I don't know the application so you'll have to insert whatever `pyenv` command would setup paths for a generic Python27 virtual environment... if `pyenv` can do anything of the sort.) or `source ~/whatever/activate` and then `sometool` at my terminal. I can simply throw a `sometool.py27` or a `sometool.py33` at my terminal, or I can throw `sometool` at my terminal and expect the default environment stack.

I can similarly throw `python.https somecode.py` at my terminal to see how the code handles the relevant, here `https` having things like `jsonschema` and `flup` installed, `virtualenv` environment.

[/Edit]

If `virtualenv` and `pip` already accomplish similar mechanism, I can only say I didn't know about the feature.

*shrug*

None of which is even to say that my setup is something you would want.

That said, I don't really see much of an option if anything like `sudo pip` and `./configure --prefix=$HOME/whatever && make altinstall` are problems for you.

[Edit]

Wait. I read the thread again, and I do understand "global versions", but I didn't see anything specific.

You talked about `pyenv` building versions, but you can build a `virtualenv` against the 3.5 version with the same suffix naming you were otherwise expecting if you don't mind a little work manually building the package.

Do you really need to avoid the `./configure --prefix=$HOME/whatever && make altinstall` approach?

I guess I'm asking if you are just developing for a different branch or if you are deploying a different branch on a server!?

[/Edit]

Soma

[Mario F.]

Do you really need to avoid the `./configure --prefix=$HOME/whatever && make altinstall` approach?

Damn! You are right. I don't actually need to avoid it at all. I'm just developing for that branch. And all development computers are single-user, anyways.

One last question: Suppose my system has Python 2.7 installed by its own installer and depends on it for core functionality. The conservative repositories don't normally offer upgrades. So it comes with 2.7.6, but I'm not offered an official way to upgrade to 2.7.7 and beyond. How do you suggest I can ensure a parallel 2.7 installation that follows upstream releases?

[phantomotap]

How do you suggest I can ensure a parallel 2.7 installation that follows upstream releases?

O_o

My experience with package managers supporting relocation (You could think in terms of `apt-get --user package`, as if that was really a thing, or similar.) can be summed as "Don't." as most are aimed towards staging rather than a user.

In my case, my account has the same name on different computers so I can just `tar` the tree and reuse.

With the assumptions about your environment, I don't have any real suggestions.

Soma

[Mario F.]

O_o

My experience with package managers supporting relocation (You could think in terms of `apt-get --user package`, as if that was really a thing, or similar.) can be summed as "Don't." as most are aimed towards staging rather than a user.

In my case, my account has the same name on different computers so I can just `tar` the tree and reuse.

With the assumptions about your environment, I don't have any real suggestions.

Soma

I did some tests on a VM this weekend with Debian, Antergos, and Fedora. I had a sort of idea and just wanted to make sure it would work. Which it did.

Python 2.X and 3.X comes installed on these distributions as python2 and python3. So it was in fact possible to install python 2.7.11 and 3.5.1 with 'make altinstall' on $HOME, which gives python2.7 and python3.5 executables and isolates them from the core python installations. So I do have a path for manual upgrades. As long as the distribution respects the python2 and python3 naming scheme, it works.

I didn't want to test on Mint because we are planning to move away from it. Just not sure yet what to use to replace it. Possibly CentOs. Not sure yet.