Oscar
HomeServicesDocumentation
  • Overview
  • Quickstart
  • Getting Started
  • System Hardware
  • Account Information
  • Short "How to" Videos
  • Quick Reference
    • Common Acronyms and Terms
    • Managing Modules
    • Common Linux Commands
  • Getting Help
    • ❓FAQ
  • Citing CCV
  • CCV Account Information
  • Student Accounts
  • Offboarding
  • Connecting to Oscar
    • SSH (Terminal)
      • SSH Key Login (Passwordless SSH)
        • Mac/Linux/Windows(PowerShell)
        • Windows(PuTTY)
      • SSH Configuration File
      • X-Forwarding
      • SSH Agent Forwarding
        • Mac/Linux
        • Windows (PuTTY)
      • Arbiter2
    • Open OnDemand
      • Using File Explorer on OOD
      • Web-based Terminal App
      • Interactive Apps on OOD
      • Using Python or Conda environments in the Jupyter App
      • Using RStudio
      • Desktop App (VNC)
    • SMB (Local Mount)
    • Remote IDE (VS Code)
      • From Non-compliant Networks (2-FA)
      • Setup virtual environment and debugger
  • Managing files
    • Oscar's Filesystem
    • Transferring Files to and from Oscar
    • Transferring Files between Oscar and Campus File Storage (Replicated and Non-Replicated)
    • Resolving quota issues
      • Understanding Disk Quotas
    • Inspecting Disk Usage (Ncdu)
    • Restoring Deleted Files
    • Best Practices for I/O
    • Version Control
  • Submitting jobs
    • Running Jobs
    • Slurm Partitions
    • Interactive Jobs
    • Batch Jobs
    • Managing Jobs
    • Job Arrays
    • MPI Jobs
    • Condo/Priority Jobs
    • Dependent Jobs
    • Associations & Quality of Service (QOS)
  • GPU Computing
    • GPUs on Oscar
      • Grace Hopper GH200 GPUs
      • H100 NVL Tensor Core GPUs
      • Ampere Architecture GPUs
    • Submitting GPU Jobs
    • Intro to CUDA
    • Compiling CUDA
    • Installing Frameworks (PyTorch, TensorFlow, Jax)
      • Installing JAX
      • Installing TensorFlow
    • Mixing MPI and CUDA
  • Large Memory Computing
    • Large Memory Nodes on Oscar
  • Software
    • Software on Oscar
    • Using Modules
    • Migration of MPI Apps to Slurm 22.05.7
    • Python on Oscar
    • Python in batch jobs
    • Installing Python Packages
    • Installing R Packages
    • Using CCMake
    • Intro to Parallel Programming
    • Anaconda
    • Conda and Mamba
    • DMTCP
    • Screen
    • VASP
    • Gaussian
    • IDL
    • MPI4PY
  • Jupyter Notebooks/Labs
    • Jupyter Notebooks on Oscar
    • Jupyter Labs on Oscar
    • Tunneling into Jupyter with Windows
  • Debugging
    • Arm Forge
      • Configuring Remote Launch
      • Setting Job Submission Settings
  • MATLAB
    • Matlab GUI
    • Matlab Batch Jobs
    • Improving Performance and Memory Management
    • Parallel Matlab
  • Visualization 🕶
    • ParaView Remote Rendering
  • Singularity Containers
    • Intro to Apptainer
    • Building Images
    • Running Images
    • Accessing Oscar Filesystem
      • Example Container (TensorFlow)
    • Singularity Tips and Tricks
  • Installing Software Packages Locally
    • Installing your own version of Quantum Espresso
    • Installing your own version of Qmcpack
  • dbGaP
    • dbGaP Architecture
    • dbGaP Data Transfers
    • dbGaP Job Submission
  • RHEL9 Migration
    • RHEL-9 Migration
    • LMOD - New Module System
    • Module Changes
    • Testing Jupyter Notebooks on RHEL9 mini-cluster
  • Large Language Models
    • Ollama
Powered by GitBook
On this page
  • Using Python Enviroments (venv)
  • Reinstalling environment
  • Step 1: Generate a list of installed packages
  • Step 2: Create a new environment and install packages
  • Install into your home directory
  • Install at custom location
  • Installing from source

Was this helpful?

Export as PDF
  1. Software

Installing Python Packages

PreviousPython in batch jobsNextInstalling R Packages

Last updated 1 year ago

Was this helpful?

For Python 3, we recommend using the system Python. You do not need to load any Python module to use system Python3

Python modules do not include other common Python packages (e.g., SciPy, NumPy). This affords individual users complete control over the packages they are using.

There are several ways for users to install python packages on Oscar

  • using a Python environment

  • using

  • into their home directory

  • into a custom location

  • from source into a custom location

We recommend using a Python environment for your workflow if you preferpip. If you are a conda user we recommend managing your workflow with You can load an module and then use conda.

In this document, we use angular brackets <> to denote command line options that you should replace with an appropriate value

Intel provides optimized packages for numerical and scientific work that you can install through or .

Using Python Enviroments (venv)

Python environments are a cleaner way to install python packages for a specific workflow. In the example below, a virtual environment called my_cool_science is set up in your home directory:

cd ~
python -m venv my_cool_science
source ~/my_cool_science/bin/activate
pip install <your package>
deactivate

line 1: load the version of python you want to use

line 2: change directory to home

line 3: create the Python environment

line 4: activate the Python environment

line 5: install any packages you need for the Python environment

line 6: deactivate the environment

When you want to use the environment, e.g. in a batch script or an interactive session

source ~/my_cool_science/bin/activate

When your work is finished, deactivate the environment with

deactivate

Reinstalling environment

Step 1: Generate a list of installed packages

Activate the environment and print the list of installed packages to a file

source ~/old_env/bin/activate
pip freeze > ~/old_env_req.txt

Step 2: Create a new environment and install packages

Here, we create a new environment and install packages inside it from old_env_req.txt file.

cd ~
python -m venv new_env
source ~/new_env/bin/activate
pip install -r ~/old_env_req.txt
deactivate

Install into your home directory

The --user flag will instruct pip to install to you home directory

pip install --user <package>

This will install the package under the following path in user's HOME directory:

~/.local/lib/python<version>/site-packages

If you omit the --user flag you will see

    IOError: [Errno 13] Permission denied: '/gpfs/runtime/opt/python/2.7.3/lib/python2.7/site-packages/ordereddict.py'

This is because users do not have access to the default locations where software is installed.

Python packages can often have conflicting dependencies. For workflows that require a lot of python packages, we recommend using virtual environments.

Install at custom location

Users have a limit of 20GB for their home directories on Oscar. Hence, users might want to use their data directory instead for installing software. Another motivation to do that is to have shared access to the software among the whole research group.

 pip install --target=</path/to/install/location> <package>

This path to install location will have to be added to the PYTHONPATH environment variable so that python can find the python modules to be used. This is not necessary for software installed using the --user option.

export PYTHONPATH=</path/to/install/location>:$PYTHONPATH

This can be added at the end of your .bashrc file in your home directory. This will update the PYTHONPATH environment variable each time during startup. Alternatively, you can update PYTHONPATH in your batch script as required. This can be cleaner as compared to the former method. If you have a lot of python installs at different locations, adding everything to PYTHONPATH can create conflicts and other issues.

A caveat of using this method is that pip will install the packages (along with its requirements) even if the package required is already installed under the global install or the default local install location. Hence, this is more of a brute force method and not the most efficient one.

For example, if your package depends on numpy or scipy, you might want to use the numpy and scipy under our global install as those have been compiled with MKL support. Using the --target option will reinstall numpy with default optimizations and without MKL support at the specified location.

Installing from source

Sometimes, python software is not packaged by the developers to be installed by pip. Or, you may want to use the development version which has not been packaged. In this case, the python package can be installed by downloading the source code itself. Most python packages can be installed by running the setup.py script that should be included in the downloaded files.

You will need to provide a "prefix path" for the install location

python setup.py install --prefix=</path/to/install/location>

This will create the sub-directories bin, lib, etc. at the location provided above and install the packages there. The environment will have to be set up accordingly to use the package:

export PATH=</path/to/install/location>/bin:$PATH
export PYTHONPATH=</path/to/install/location>/lib/python<version>/site-packages:$PYTHONPATH
conda
conda environments .
anaconda
pip
anaconda