Thread: Multiple versions of Python on a Linux machine

  1. #1
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Angola
    Posts
    8,445

    Multiple versions of Python on a Linux machine

    What is the best way to manage multiple versions of Python on a Linux machine?

    On my Linux Mint machine at the office, I have:
    • Python 3.4, system installation (installed with the distro)
    • Python 2.7, system installation (installed with the distro)
    • Python 3.5, with make altinstall

    I never install packages to the system python versions (or even upgrade them). I just create a virtualenv and install them there.

    I did do "sudo pip" for a few packages in the python 3.5 altinstall installation though:

    • virtualenv
    • iPython + jupyter
    • pytest
    • coverage
    • pylint


    The rest I just install to the virtualenvs of my projects.

    Do you consider this a good practice, despite "sudo pip" being highly unrecommended?
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  2. #2
    Make Fortran great again
    Join Date
    Sep 2009
    Posts
    1,413
    Interested in this as well. On Windows, they install in different places; I simply have a python3.bat in the python27 folder to redirect to the Python 3.5 installation. That way both Python directories are not in PATH and conflicting.

  3. #3
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    O_o

    Why did you need to install `virtualenv` for the newest version?

    *shrug*

    Anyways, I do the mechanical equivalent of `sudo pip` for packages related to my own work on my own machines for the current Python versions. I wouldn't really call the approach "good practice", but I have the hooks setup for system and Python packages managers so that running my own administration tools installs everything automatically for relevant `chroot` or overlay environment. Basically, a convenience I have no intention of giving up in any event.

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  4. #4
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Angola
    Posts
    8,445
    Quote Originally Posted by phantomotap View Post
    Why did you need to install `virtualenv` for the newest version?
    You wouldn't believe how old the version of virtualenv is on the default python 2 installation that comes with Linux Mint. And since I make a point of never messing with the package list on the default installations, it's just better to install virtualenv again on one of the altinstall versions.

    Quote Originally Posted by phantomotap View Post
    ...
    I'll be soon moving away from Mint at the office. I can't stand this nonsense distro anymore. It's just not good for development environments or any other environment where we must follow development cycles of a number of applications and system tools. We spend more time fighting against the aggressively conservative repositories than anything else.

    My concern is what other alternative is there for a proper Python installation of multiple versions, and helps me avoid 'sudo pip'?
    pyenv won't do. I don't even understand its purpose. To change my global python (even if temporarily) is a great way to break existing tools that rely on a python version I just hid.
    Last edited by Mario F.; 06-09-2016 at 04:37 PM.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  5. #5
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    To change my global python (even if temporarily) is a great way to break existing tools that rely on a python version I just hid.
    O_o

    I may have been, or may still be, on the wrong page.

    Are you asking to use, for example, a `tool1 --dosomestuff` and a `tool2 --dosomestuff` at a terminal without needing to remember the virtual environment binary location or dancing around with scripts to change the current environment stack to account for the dependencies of each application?

    I eventually put all of my `virtualenv` directories in the same relative location (`~/.local/opt/$ENV`) and hooked `pip` to create a script handling the environment stack for any binaries (`~/.local/opt/$ENV/bin/application`) because I found myself forgetting which virtual environment owned which projects.

    I don't know of any project which creates such a serviceable shim automatically on install or upgrade, but you shouldn't need more than a couple of hours if you don't mind a few restrictions.

    *shrug*

    If the issue is still just the "recommended" nature of `sudo pip`, you could remove the various global installs and setup three `virtualenv` directories before again installing everything. You would then have no global repository which could ever need a `sudo pip` install, but I wouldn't recommend that so I guess I'll stop talking and maybe you'll get a response from someone knowing more about the state of `virtualenv` wrappers.

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  6. #6
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Angola
    Posts
    8,445
    Quote Originally Posted by phantomotap View Post
    Are you asking to use, for example, a `tool1 --dosomestuff` and a `tool2 --dosomestuff` at a terminal without needing to remember the virtual environment binary location or dancing around with scripts to change the current environment stack to account for the dependencies of each application?
    I can do this already with 'make altinstall' when installing a new python version. Currently, I execute python, python3 and python3.5 for the python 2.7, python 3.4 and python 3.5 interpreters, respectively. And pip follows the same pattern. That is not a problem.

    However 'make altinstall' install python on /usr/local. hence the need for 'sudo pip'.

    I eventually put all of my `virtualenv` directories in the same relative location (`~/.local/opt/$ENV`) and hooked `pip` to create a script handling the environment stack for any binaries (`~/.local/opt/$ENV/bin/application`) because I found myself forgetting which virtual environment owned which projects.
    It's why sometime ago I gave up on a common folder for virtual envs. Now I just install a virtual environment inside each project folder and add it to .gitignore. I don't have to remember nothing about anything anymore.

    If the issue is still just the "recommended" nature of `sudo pip`,
    That is precisely the issue
    I have 3 global installations (more on the way). 2 of them are installed with the system and the third one was installed by me with 'make altinstall' so it won't try to overwrite the default ones. All is working. But this also means I am now forced to sudo pip when I wish to install packages to the global python installation. It's not really a big deal, because global packages should be avoided anyways. And I could always create a "system env" on my home folder exactly to keep those packages I listed above. But I was still wondering if there was a better way to install multiple global versions of python without ending up with sudo pip.

    you could remove the various global installs and setup three `virtualenv` directories before again installing everything.
    (note: I could only remove one python version above; 3.5. Any of the other two would break my system, because the distro itself uses them for core system functionality.)

    What you propose is possible with a 3rd party tool called pyenv (previously much better named as pythonbrew). It creates global virtual environments, and downloads and builds the python version there. However there is a downside. Those virtual envs are global, so in order to switch to python 3.5 installed by pyenv I have to activate it, which deactivates all other pythons. I don't want to lose access to py2 when I am working for instance on a py3 env.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  7. #7
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    I can do this already with 'make altinstall' when installing a new python version. [...] a py3 env.
    O_o

    I think that, given the post as a whole, you didn't understand what my setup is doing... or the various `virtualenv` tools have gotten better while I wasn't looking.

    My setup doesn't hide things by mucking about with the terminal environment.

    I guess I'd describe my setup as, in a way, continuing the naming pattern you reference but with any `virtualenv` you've setup.

    [Edit]

    I don't have to throw a `pyenv -whatever py27` (I don't know the application so you'll have to insert whatever `pyenv` command would setup paths for a generic Python27 virtual environment... if `pyenv` can do anything of the sort.) or `source ~/whatever/activate` and then `sometool` at my terminal. I can simply throw a `sometool.py27` or a `sometool.py33` at my terminal, or I can throw `sometool` at my terminal and expect the default environment stack.

    I can similarly throw `python.https somecode.py` at my terminal to see how the code handles the relevant, here `https` having things like `jsonschema` and `flup` installed, `virtualenv` environment.

    [/Edit]

    If `virtualenv` and `pip` already accomplish similar mechanism, I can only say I didn't know about the feature.

    *shrug*

    None of which is even to say that my setup is something you would want.

    That said, I don't really see much of an option if anything like `sudo pip` and `./configure --prefix=$HOME/whatever && make altinstall` are problems for you.

    [Edit]

    Wait. I read the thread again, and I do understand "global versions", but I didn't see anything specific.

    You talked about `pyenv` building versions, but you can build a `virtualenv` against the 3.5 version with the same suffix naming you were otherwise expecting if you don't mind a little work manually building the package.

    Do you really need to avoid the `./configure --prefix=$HOME/whatever && make altinstall` approach?

    I guess I'm asking if you are just developing for a different branch or if you are deploying a different branch on a server!?

    [/Edit]

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  8. #8
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Angola
    Posts
    8,445
    Quote Originally Posted by phantomotap View Post
    Do you really need to avoid the `./configure --prefix=$HOME/whatever && make altinstall` approach?
    Damn! You are right. I don't actually need to avoid it at all. I'm just developing for that branch. And all development computers are single-user, anyways.

    One last question: Suppose my system has Python 2.7 installed by its own installer and depends on it for core functionality. The conservative repositories don't normally offer upgrades. So it comes with 2.7.6, but I'm not offered an official way to upgrade to 2.7.7 and beyond. How do you suggest I can ensure a parallel 2.7 installation that follows upstream releases?
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  9. #9
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    How do you suggest I can ensure a parallel 2.7 installation that follows upstream releases?
    O_o

    My experience with package managers supporting relocation (You could think in terms of `apt-get --user package`, as if that was really a thing, or similar.) can be summed as "Don't." as most are aimed towards staging rather than a user.

    In my case, my account has the same name on different computers so I can just `tar` the tree and reuse.

    With the assumptions about your environment, I don't have any real suggestions.

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  10. #10
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Angola
    Posts
    8,445
    Quote Originally Posted by phantomotap View Post
    O_o

    My experience with package managers supporting relocation (You could think in terms of `apt-get --user package`, as if that was really a thing, or similar.) can be summed as "Don't." as most are aimed towards staging rather than a user.

    In my case, my account has the same name on different computers so I can just `tar` the tree and reuse.

    With the assumptions about your environment, I don't have any real suggestions.

    Soma
    I did some tests on a VM this weekend with Debian, Antergos, and Fedora. I had a sort of idea and just wanted to make sure it would work. Which it did.

    Python 2.X and 3.X comes installed on these distributions as python2 and python3. So it was in fact possible to install python 2.7.11 and 3.5.1 with 'make altinstall' on $HOME, which gives python2.7 and python3.5 executables and isolates them from the core python installations. So I do have a path for manual upgrades. As long as the distribution respects the python2 and python3 naming scheme, it works.

    I didn't want to test on Mint because we are planning to move away from it. Just not sure yet what to use to replace it. Possibly CentOs. Not sure yet.
    Last edited by Mario F.; 06-13-2016 at 04:26 AM.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 5
    Last Post: 09-18-2015, 03:01 PM
  2. SSH'ing from a windows machine to a linux machine
    By BinaryProgamer in forum Networking/Device Communication
    Replies: 2
    Last Post: 12-11-2014, 10:56 PM
  3. Older versions of linux
    By überfuzz in forum Linux Programming
    Replies: 2
    Last Post: 01-27-2014, 06:50 AM
  4. Multiple versions of a library
    By jhr in forum C Programming
    Replies: 1
    Last Post: 02-17-2012, 09:24 AM
  5. Run time differences in Linux gcc versions
    By circuitbreaker in forum C++ Programming
    Replies: 7
    Last Post: 02-14-2008, 11:09 PM

Tags for this Thread