# Installing Python Packages For Python 3, we recommend using the system Python. You do not need to load any Python module to use system Python3 {% hint style="info" %} Python modules do not include other common Python packages (e.g., SciPy, NumPy). This affords individual users complete control over the packages they are using. {% endhint %} There are several ways for users to install python packages on Oscar * using a **Python environment** * using [conda](https://docs.conda.io/en/latest/) * into their home directory * into a custom location * from source into a custom location {% hint style="warning" %} We recommend using a Python environment for your workflow if you prefer`pip.` If you are a conda user we recommend managing your workflow with [conda environments . ](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)You can load an [anaconda](https://docs.ccv.brown.edu/oscar/software/anaconda) module and then use conda. {% endhint %} {% hint style="info" %} In this document, we use angular brackets `<>` to denote command line options that you should replace with an appropriate value {% endhint %} {% hint style="info" %} Intel provides optimized packages for numerical and scientific work that you can install through[ pip](https://software.intel.com/en-us/articles/installing-the-intel-distribution-for-python-and-intel-performance-libraries-with-pip-and) or [anaconda](https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda). {% endhint %} ## Using Python Enviroments (venv) Python environments are a cleaner way to install python packages for a specific workflow. In the example below, a virtual environment called `my_cool_science` is set up in your home directory: ```bash cd ~ python -m venv my_cool_science source ~/my_cool_science/bin/activate pip install deactivate ``` line 1: load the version of python you want to use line 2: change directory to home line 3: create the Python environment line 4: activate the Python environment line 5: install any packages you need for the Python environment line 6: deactivate the environment When you want to use the environment, e.g. in a batch script or an interactive session `source ~/my_cool_science/bin/activate` When your work is finished, deactivate the environment with `deactivate` ## Reinstalling environment ### Step 1: Generate a list of installed packages Activate the environment and print the list of installed packages to a file ```bash source ~/old_env/bin/activate pip freeze > ~/old_env_req.txt ``` ### Step 2: Create a new environment and install packages Here, we create a new environment and install packages inside it from `old_env_req.txt` file. ```bash cd ~ python -m venv new_env source ~/new_env/bin/activate pip install -r ~/old_env_req.txt deactivate ``` ### Install into your home directory The `--user` flag will instruct pip to install to you home directory ```bash pip install --user ``` This will install the package under the following path in user's HOME directory: ``` ~/.local/lib/python/site-packages ``` {% hint style="warning" %} If you omit the `--user` flag you will see ``` IOError: [Errno 13] Permission denied: '/oscar/runtime/opt/python/2.7.3/lib/python2.7/site-packages/ordereddict.py' ``` This is because users do not have access to the default locations where software is installed. {% endhint %} Python packages can often have conflicting dependencies. For workflows that require a lot of python packages, we recommend using virtual environments. ## Install at custom location Users have a limit of 20GB for their home directories on Oscar. Hence, users might want to use their data directory instead for installing software. Another motivation to do that is to have shared access to the software among the whole research group. ```bash pip install --target= ``` This path to install location will have to be added to the **PYTHONPATH** environment variable so that python can find the python modules to be used. This is not necessary for software installed using the `--user` option. ```bash export PYTHONPATH=:$PYTHONPATH ``` This can be added at the end of your `.bashrc` file in your home directory. This will update the PYTHONPATH environment variable each time during startup. Alternatively, you can update PYTHONPATH in your batch script as required. This can be cleaner as compared to the former method. If you have a lot of python installs at different locations, adding everything to PYTHONPATH can create conflicts and other issues. A caveat of using this method is that pip will install the packages (along with its requirements) even if the package required is already installed under the global install or the default local install location. Hence, this is more of a brute force method and not the most efficient one. For example, if your package depends on numpy or scipy, you might want to use the numpy and scipy under our global install as those have been compiled with MKL support. Using the `--target` option will reinstall numpy with default optimizations and without MKL support at the specified location. ## Installing from source Sometimes, python software is not packaged by the developers to be installed by pip. Or, you may want to use the development version which has not been packaged. In this case, the python package can be installed by downloading the source code itself. Most python packages can be installed by running the `setup.py` script that should be included in the downloaded files. {% hint style="warning" %} You will need to provide a "prefix path" for the install location ```bash python setup.py install --prefix= ``` {% endhint %} This will create the sub-directories `bin`, `lib`, etc. at the location provided above and install the packages there. The environment will have to be set up accordingly to use the package: ```bash export PATH=/bin:$PATH export PYTHONPATH=/lib/python/site-packages:$PYTHONPATH ```