Using Python on Red Hat 8

What’s Changing?

Not all versions of Python currently installed on ARC clusters will be available after the update from CentOS 7 to Red Hat 8 in August.

Python 2 will no longer be provided and is no longer supported on ARC resources. Code written for Python 2 is most often not directly compatible with Python 3. Changes to code written for Python 2 will likely be required.

Why that matters

Any scripts that load a version-specific Python module will likely have to be updated to specify a version that is available on the updated cluster.

Python packages are installed for the specific version of Python that is in use during installation. If you switch from using a module for one version of Python to a different one with where either the major or minor version changes (for python3.7, the major version is 3 and the minor version is 7), then you will have to re-install any packages/libraries in order to make them available in the library of the new version of Python.

You only need to install Python packages once for each cluster on which you wish to use the library and, separately, for each version of Python that you use.

Please note, Python packages should be installed using the command line from a login node, not from within Jupyter Notebook or the JupyterLab app.

What should I do?

Check available versions

Log onto the Preview cluster prior to maintenance and use the following command (the $ is the prompt; do not type it):

$ module available python

The output from that command will show available options (versions subject to change) as follows:

python/3.9.12 python/3.10.4 (D) python3.9-anaconda/2021.11

Note that the lowest Python version is 3.9, so if libraries were previously installed with a Python of a lower version, you will need to reinstall those libraries. The (D) next to version 3.10.4 indicates that it is the default Python version. If you load the Python module without specifying a version number, version 3.10.4 will load by default.

Update version of Python module being loaded

You should modify version numbers in any of your scripts that load Python modules, as appropriate. ARC recommends that you take advantage of the Preview cluster during the pre-maintenance period to review changes in available module versions and to run some test jobs with your revised scripts. Because /home and /nfs directories are shared, we highly recommend that when you test you use copies of data and scripts.

Re-install user-installed libraries/packages

If you have previously been using any version of Python that is 3.8 or earlier, you will need to re-install Python libraries after the update. There are several ways you might have additional Python libraries and programs configured on the Great Lakes, Armis, or Lighthouse clusters: 1) installed into the default private library collection (~/.local), for example with pip --user; 2) using venv or virtualenv environments; or 3) using conda virtual environments and/or Miniconda installations. The recommended testing strategy varies depending on which of those you used.

Testing with ~/.local

Because your $HOME directory is common to both the old clusters and the preview clusters and the default path for a private library collection is in a user’s $HOME directory (~/.local), there is potential for conflict. We recommend that users take the following steps to avoid issues that could arise due to incompatible libraries that might result otherwise.

Probably the easiest way to test if you have software installed in ~/.local is to simply move aside what you have installed now and start with an empty directory. Then, at the end of a testing session, move it back. First, create a directory for the new libraries

$ mkdir ~/new-local

Then use this command after you log onto the preview cluster for a testing session.

$ mv ~/.local ~/old-local
$ mv ~/new-local ~/.local

You can then install any libraries that you need and verify that they work. To install a Python package into your personal library using pip, enter the following command, replacing <package_name> with the actual package name:

$ pip install --user <package_name>

The --user tag will, by default, place packages in
$HOME/.local/lib/python?.?/site-packages

where ?.? indicates the versioning of the Python release. The library will then be available to you for this and future sessions.

Before logging out of the preview cluster, you would use these commands to change back.

$ mv ~/.local ~/new-local
$ mv ~/old-local ~/.local

Testing with virtual environments

If you used one of the Python Anaconda modules to get conda, we recommend that you create fresh virtual environments on the preview cluster using the same method you used to create them on the current cluster. Similarly, if you used virtualenv or venv to do so, you should also recreate fully. That will ensure that everything is correct.

Miniconda installations

Miniconda installations are fully self-contained and should work without modification. To test these, simply run in the normal way.

Port Python 2 code to Python 3

If you have code that was written for Python 2, you will need to update it so that it is compatible with Python 3. Please see the following links for further information and some suggested tools that you may find helpful: