The simplest way to connect to Oscar is via Open OnDemand (OOD). To connect to OOD, go to and log in using your Brown credentials. For more details, see the .

Alternatively, you can connect to OSCAR via SSH (Terminal):

You can submit a job using sbatch:

You can confirm that your job ran successfully by running:

For more detailed information on submitting jobs, see the .

To get specific files on to / off of Oscar, read through the .

If you encounter problems while using Oscar, check out the , or read through the Overview page.

Nightly snaphots of the file system are available for the last 30 days.

Restore a file from a snapshot in the last 30 days

Nightly snapshots of the file system are available for the last 30 days can be found in the following directories.

Home directory snapshot

/oscar/home/.snapshot/Oscar_<yyyy-mm-dd>_00_00_00_UTC/<username>/<path_to_file>

Data directory snapshot

Scratch directory snapshot

To restore a file, copy the file from the snapshot to your directory.

e.g.:

1. If you have an existing virtual environment, proceed to step 2. Otherwise, to create a new virtual environment:

   Copy

   python3 -m venv my_env
   source my_env/bin/activate
   # Install packages manually or from requirements.txt file
   pip install -r requirements.txt

2. Search for Python.VenvPath as shown in the picture below:
3. VSCode expects you to have multiple virtual environments for each of your different python projects, and it expects you to put them all in the same directory. Pointing to the parent directory lets it scan and find all expected virtual environments, and then you can easily toggle between them in interface.
4. Once you have the virtual environment selected, the debugging capabilities should work.

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. An Oscar account is required to access Open OnDemand. Visit this link in a web browser and sign in with your Brown username and password to access this portal.

OOD provides with a several resources for interacting with Oscar.

• Use the in the portal to view, copy, download or delete files on Oscar.

• Launch , like Matlab and Jupyter Notebook, inside your web browser.

SSH provides a method of sharing the ssh keys on your local machine with Oscar. This feature is called Agent Forwarding and can be useful for instance when working with version control or other services that authenticate via ssh keys. Below are instructions on how to configure your SSH connection to forward ssh-agent for diffeent operating systems

These pages list some command commands and terms you will come across while using Oscar.

Common Acronyms and Terms

Managing Modules

Common Linux Commands

If you have an installation of X11 on your local system, you can access Oscar with X forwarding enabled, so that the windows, menus, cursor, etc. of any X applications running on Oscar are all forwarded to your local X11 server. Here are some resources for setting up X11:

• Mac OS - X Quartz

• Windows - Xming

Mac/Linux

Once your X11 server is running locally, open a terminal and use

to establish the X forwarding connection. Then, you can launch GUI applications from Oscar and they will be displayed locally on your X11 server.

Windows (PuTTY)

For Windows users using PuTTY, enable X forwarding under Connections->SSH->X11:

If you publish research that benefited from the use of CCV services or resources, we would greatly appreciate an acknowledgment that states:

This research [Part of this research] was conducted using [computational/visualization]
resources and services at the Center for Computation and Visualization, Brown University.

The Oscar GPUs are in a separate partition to the regular compute nodes. The partition is called gpu. To see how many jobs are running and pending in the gpu partition, use

Interactive use

To start an session on a GPU node, use the interact command and specify the gpu partition. You also need to specify the requested number of GPUs using the -g option:

Batch jobs

Here is an example batch script for a cuda job that uses 1 gpu and 1 cpu for 5 minutes

To submit this script:

DGX GPU Nodes in the GPU-HE Partition

All the nodes in the gpu-he partition have V100 GPUs. However, two of them are DGX nodes (gpu1404/1405) which have 8 GPUs. When a gpu-he job requests for more than 4 GPUs, the job will automatically be allocated to the DGX nodes.

The other non-DGX nodes actually have a better NVLink interconnect topology as all of them have direct links to the other. So the non-DGX nodes are better for a gpu-he job if the job does not require more than 4 GPUs.

This is a quick guide for resolving issues related to file system quotas. To read more details about these quotas, refer to this page.

Step 1: Identify the directory

Run the checkquota command and identify the line that shows the warning status message.

If this directory is either /oscar/home or /oscar/scratch , you will have to take the subsequent steps to resolve this issue. If the directory is data+<group> you should inform others in your group and take collective action to resolve this issue.

Step 2: Disk Space or Inodes

Check whether you have exceeded your disk space quota or your inodes quota. Disk space usage is specified in GB or TB while inodes usage is just numerical count.

Step 3: Remove files

You will need to take the following steps based on the quota you have exceeded.

Disk Space quota:

The fastest way to reduce this usage is identifying large and unnecessary files. Load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. If they are not sorted by size, press lowercase s to sort them by size. You can navigate the directory tree using the arrow keys and delete any files or directories that are unnecessary.

Some programs leave a lot of temporary files on the disk that may not be necessary.

• Apptainer: Run the command to clear the apptainer cache. This will clear up the cache in your home directory without affecting any container images. However, pulling a new image from a repository may be slower in the future.

• Conda: Run the command to delete any tarballs downloaded by conda. This does not affect any existing conda or python virtual environments. However, it may slow down the installation of some packages in the future

• Core Dump Files: This files are typically named core.<number> A core dump file is generated when a program crashes. It contains the state of the system and it is useful for debugging purposes. You can safely delete any core dump files if you know the reason behind the crash. Old core dump files can take up a lot of disk space and they can be safely deleted.

Inode usage can be reduced by removing any files and directories OR tarring up large nested directories. When a directory is converted to a tar ball, it uses a single inode instead of one inode per directory or file. This can drastically decrease your inode usage. Identify directories that contain a large number of files or a very large nested tree of directories with a lot of files.

To identify such directories, load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. Press uppercase C to switch the sorting criteria to "number of items". You can navigate the directory tree using the arrow keys and delete or tar any files or directories that are unnecessary.

To create a tar ball of a directory:

If your usage has exceeded quota and you cannot write to the directory, you can tar ball in another directory. Using this command, you can create a tar ball in the scratch directory:

Checkquota

Use the command checkquota to view your current disk usage and quotas. Here's an example output of this command

In the output of the checkquota command, each line represents a top level directory that you have access to such us /oscar/home, /oscar/scratch /data, and /jobtmp (a temporary storage directory for your jobs). The columns of the output show different types of quotas and limits that we explain below.

Types of Quota:

Disk usage and quotas are calculated separately for top level directories. Two types of quotas are calculated for each of these directories:

Disk space usage

This usage is expressed in Gigabytes (G) or Terabytes (T) . This is the total size of all the files in that directory and it does not depend upon the number of files. Run the command checkquota to see your disk usage and quota. Here's an example:

This is the total number of files and directories in the particular directory. This number does not depend upon the size of the files. Run the command checkquota to see your inode usage and quota. Here's an example:

All quotas have a soft limit (SLimit) and hard limit (HLimit). When usage exceeds the soft limit, a grace period associated with this limit begins. During the grace period, the usage is allowed to increase up to the hard limit. When the usage reaches the hard limit or when the grace period expires, the user is not allowed to write any files to that particular directory.

The "Usage State" column shows the status of the grace period for a particular directory. Here are some of the status messages:

This indicates that your usage of the disk space or inodes has exceeded the soft limit and you are still within the grace period. Check the Grace_Period column to see the number of days left in the grace period. You may continue writing data into this directory until the end of the grace period, as long as you do not exceed the hard limit

This indicates that your usage has exceeded the soft limit AND the grace period has expired. You will not be able to write data into that directory, but you can remove files.

This indicates that your usage has reached the hard limit. You will not be able to write data into that directory, but you can remove data.

This indicates that your usage of the disk space as well as inodes in within the soft quota.

This page serves as a guide for application developers getting started with parallel programming, or users wanting to know more about the working of parallel programs/software they are using.

Although there are several ways to classify parallel programming models, a basic classification is:

1. Shared Memory Programming

2. Distributed Memory Programming

Shared Memory Parallelism

This model is useful when all threads/processes have access to a common memory space. The most basic form of shared memory parallelism is Multithreading. According to Wikipedia, a of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler (Operating System).

Note that most compilers have inherent support for multithreading up to some level. Multithreading comes into play when the compiler converts your code to a set of instructions such that they are divided into several independent instruction sequences (threads) which can be executed in parallel by the Operating System. Apart from multithreading, there are other features like "vectorized instructions" which the compiler uses to optimize the use of compute resources. In some programming languages, the way of writing the sequential code can significantly affect the level of optimization the compiler can induce. However, this is not the focus here.

Multithreading can also be induced at code level by the application developer and this is what we are interested in. If programmed correctly, it can also be the most "efficient" way of parallel programming as it is managed at the Operating System level and ensures optimum use of "available" resources. Here too, there are different parallel programming constructs which support multithreading.

POSIX threads is a standardized C language threads programming interface. It is a widely accepted standard because of being lightweight, highly efficient and portable. The routine to create Pthreads in a C program is called pthread_create and an "entry point" function is defined which is to be executed by the threads created. There are mechanisms to synchronize the threads, create "locks and mutexes", etc. Help pages:

• Comprehensive tutorial page on

OpenMP is a popular directive based construct for shared memory programming. Like POSIX threads, OpenMP is also just a "standard" interface which can be implemented in different ways by different vendors.

Compiler directives appear as comments in your source code and are ignored by compilers unless you tell them otherwise - usually by specifying the appropriate compiler flag (). This makes the code more portable and easier to parallelize. you can parallelize loop iterations and code segments by inserting these directives. OpenMP also makes it simpler to tune the application during run time using environment variables. for example, you can set the number of threads to be used by setting the environment variable OMP_NUM_THREADS before running the program. Help pages:

We have provided templates for you to use for job submission settings. These templates are in/oscar/runtime/opt/forge/19.1.2/templates

Click Run and debug a program to open the following menu

Click Configure next to Submit to Queue and enter /oscar/runtime/opt/forge/19.1.2/templates/slurm-ccv.qtf as the Submission template file

Agent Forwarding with PuTTY

1. Once adding your private key to Pageant, open PuTTY and navigate to the Auth menu.

2. Check the 'Allow agent forwarding' checkbox, and return to the Session menu.

3. Enter the Host Name you usually use to connect to Oscar, and click 'Open'.

4. Entering your password. If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

ssh -T git@github.com

Hi JaneDoe! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.

Many scientific and HPC software packages are already installed on Oscar, and additional packages can be requested by submitting a ticket to support@ccv.brown.edu. If you want a particular version of the software, do mention it in the email along with a link to the web page from where it can be downloaded. You can also install your own software on Oscar.

CCV cannot, however, supply funding for the purchase of commercial software. This is normally attributed as a direct cost of research, and should be purchased with research funding. CCV can help in identifying other potential users of the software to potentially share the cost of purchase and maintenance. Several commercial software products that are licensed campus-wide at Brown are available on Oscar.

For software that requires a Graphical User Interface (GUI) we recommend using CCV's rather than X-Forwarding.

Compiling with CUDA

To compile a CUDA program on Oscar, first load the CUDA module with:

$ module load cuda

The CUDA compiler is called nvcc, and for compiling a simple CUDA program it uses syntax simlar to gcc:

$ nvcc -o program source.cu

Optimizations for Fermi

The Oscar GPU nodes feature NVIDIA M2050 cards with the Fermi architecture, which supports CUDA's "compute capability" 2.0. To fully utilize the hardware optimizations available in this architecture, add the -arch=sm_20 flag to your compile line:

$ nvcc -arch=sm_20 -o program source.cu

This means that the resulting executable will not be backwards-compatible with earlier GPU architectures, but this should not be a problem since CCV nodes only use the M2050.

Memory caching

The Fermi architecture has two levels of memory cache similar to the L1 and L2 caches of a CPU. The 768KB L2 cache is shared by all multiprocessors, while the L1 cache by default uses only 16KB of the available 64KB shared memory on each multiprocessor.

You can increase the amount of L1 cache to 48KB at compile time by adding the flags -Xptxas -dlcm=ca to your compile line:

If your kernel primarily accesses global memory and uses less than 16KB of shared memory, you may see a benefit by increasing the L1 cache size.

If your kernel has a simple memory access pattern, you may have better results by explicitly caching global memory into shared memory from within your kernel. You can turn off the L1 cache using the flags –Xptxas –dlcm=cg.

Anaconda provides Python, R and other packages for scientific computing including data sciences, machine learning, etc.

There is one anaconda module:

$ module avail anaconda

-------- /oscar/runtime/software/spack/0.20.1/share/spack/lmod/linux-rhel9-x86_64/Core -------
   anaconda/2023.09-0-7nso27y

• If you are at Brown and have requested a regular CCV account, your Oscar login can be authenticated using your Brown credentials itself, i.e. the same username and password that you use to login to any Brown service such as "canvas".

• If you are an external user, you will have to get a sponsored ID at Brown through the department with which you are associated, before requesting an account on Oscar. Once you have the sponsored ID at Brown, you can on Oscar and use your Brown username and password to login.

Oscar users should use their Brown passwords to log into Oscar. Users should change their Brown passwords at .

Oscar has two Grace Hopper GH200 GPU nodes. Each node combines and .

Each GH200 node has 72 Arm cores with 550G memory. Both CPU and GPU threads on GH200 nodes can now .

The two GH200 nodes are in the gracehopper partition.

A gk-condo user can submit jobs to the GH200 nodes with their gk-gh200-gcondo account, i.e.,

For users who are not a gk-condo user, a High End GPU priority account is required for accessing the gracehopper partition and GH200 nodes. All users with access to the GH200 nodes need to submit jobs to the nodes with the ccv-gh200-gcondo account, i.e.

NGC containers provide the best performance from the GH200 nodes. is an example for running NGC containers.

The two nodes have Arm CPUs. So Oscar modules

First, start your ssh-agent with the command below.

You should see an output similar to this:

Next, add your ssh private keys to the running agent (using the ssh-add command on line 1). This step may be repeated for every key pair you use to connect to different git servers. For most, this file is called id_rsa and will live in ~/.ssh/id_rsa. If you set a password for your ssh keys, the agent will prompt you to enter them.

Confirm the ssh keys have been loaded into the agent with ssh-add -L:

You may use either Globus (recommended) or smbclient to transfer data between Oscar and Campus File Storage.

Follow the instructions .

You can transfer files between Campus File Storage and Oscar using .

1) Log into Oscar:

2) Start a screen session. This will allow you to reattach to your terminal window if you disconnect.

3) To use Oscar's high-speed connection to Campus File Storage - Replicated:

Similarly to access Campus File Storage - Non-Replicated ( LRS: Locally Redundant Share)

Replace SHARE_NAME, DIRECTORY_NAME, and BROWN_ID. DIRECTORY_NAME is an optional parameter. The password required is your Brown password.

4) Upload/download your data using the FTP "put"/"get" commands. Replace DIRECTORY_NAME with the folder you'd like to upload.

To get started with NCDU, load the module using the following command:

Once the module has been loaded, it can be used to easily show the size of all files within a directory:

The line above uses Ncdu to rank all of the files within the my_directory directory. Your window should change to show a loading screen (if the directory doesn't have a lot in it, you may not even see this screen):

Once Ncdu has finished loading, you will see a result like this:

The files will be ordered with the largest file at the top and the smallest file at the bottom. The bottom left corner shows the Total disk usage

Oscar is Brown University's high performance computing cluster for both research and classes. Oscar is maintained and supported by .

Please contact if there are any questions on Oscar.

If you do not have an Oscar account, you can request one by clicking the following link:

Anyone with a Brown account can get a free Exploratory account on Oscar, or pay for priority accounts.

More details can be found at the .

Individuals external to Brown can get access to Oscar by having a sponsored Brown account. Please work with your department to for any external collaborators.

Authorized users must comply with the following Brown University policies:

screen is a "terminal multiplexer", it enables a number of terminals (or windows) to be accessed and controlled from a single terminal. screen is a great way to save an interactive session between connections to oscar. You can reconnect to the session from anywhere!

Common commands are:

• start a new screen session with session name: screen -S <name>

Open OnDemand offers a browser-based terminal app to access Oscar. Windows users who do not want to install an SSH client like Putty will find this app very useful.

1. Log in to

2. In the top menu, click Clusters -> >_OSCAR Shell Access

If you are using Windows, you can use any of the following options to open a terminal on your machine (ranked in order of least difficult to set up and use):

You can launch several different apps on the Open OnDemand (OOD) interface. All of these apps start of a Slurm batch job on the Oscar cluster with the requested amount of resources. These jobs can access the filesystem on Oscar and all output files are written to the Oscar's file system.

1. Open on any browser of the your choice

2. If prompted, enter your Brown username and password.

1. Open PuTTYgen (this comes as part of the package), change the 'Number of bits in a generated key:' to 4096 (recommended), then click 'Generate'
2. Move your cursor around randomly in order to "salt" your key, while the key is being generated. Once the key is generated, you should see something like this:
3. Replace the text in the 'Key comment:' field with something recognizable and enter a passphrase in the two fields below.

You will need to configure remote launch for Oscar

1. Open the client on your machine

2. Click 'Remote Launch' -> Configure

3. Enter the following details in each field:

Here is an example script for running dependent jobs on Oscar.

There are 3 batch jobs. Each job has it's own batch script: job1.sh, job2,sh, jobs.sh. The script above (script.sh) submits the three jobs.

line 4: job1 is submitted.

line 7: job2 depends on job1 finishing successfully.

line 10: job3 depends on job2 finishing successfully.

For details on the types of dependencies you can use in slurm see the manual page.

RStudio is an IDE for R that can be run on Oscar.

Open the . Select RStudio (under "Default GUI's"). Fill in the form to allocate the required resources, and optionally select your R modules. Finally, click the "Launch Session" button.

Plotting figures may not work within RStudio. If this is the case, save the plots to a file, and view them through the Open On Demand Desktop App. If plots are required for your task, launch RStudio through the Desktop App.

Lists all modules that are currently loaded in your software environment.

Lists all available modules on the system. Note that a module can have multiple versions. Use module avail <name> to list available modules which start with <name>

Prints additional information about the given software.

Adds a module to your current environment. If you load using just the name of a module, you will get the default version. To load a specific version, load the module using its full name with the version: "module load gcc/10.2"

Removes a module from your current environment.

Oscar is a shared machine used by hundreds of users at once. User requests are called jobs. A job is the combination of the resource requested and the program you want to run on the compute nodes of the Oscar cluster. On Oscar, is used to schedule and manage jobs.

Jobs can be run on Oscar in two different ways:

• Interactive jobs allow the user to interact with programs (e.g., by entering input manually, using a GUI) while they are running. However, if your connection to the system is interrupted, the job will abort. Small jobs with short run times and jobs that require the use of a GUI are best-suited for running interactively.

When connecting from a campus network to sshcampus.ccv.brown.edu you can set up SSH keys as a form of authentication instead of having to enter your password interactively. Follow the insctructions below that correspond to your operating system/connection method.

Oscar users will keep their access to Oscar as long as their Brown account are still active. To be able to access Oscar account after a user's Brown account is deactivated, the user needs to through the department the user is associated.

Your data (directories and files) will stay in Oscar for one years after your Brown account is deactivated. After that your data will be archived.

You may delete your data when you leave Brown University. Or you may request that CCV delete your data on Oscar, especially if you have lots of data.

You can from Oscar. Globus is recommended for large data transfer.

If you are a PI and want to keep your priority accounts and/or data directories after leaving Brown University, please contact support@ccv.brown.edu to update your billing information.

Oscar has two H100 nodes. is based on the that accelerates the training of AI models. The two DGX nodes provides better performance when multiple GPUS are used, in particular with Nvidia software like .

Each DGX H100 node has 112 Intel CPUs with 2TB memory, and 8 Nvidia H100 GPUs. Each H100 GPU has 80G memory.

The two DGX H100 nodes are in the gpu-he partition. To access H100 GPUs, users need to submit jobs to the gpu-he partition and request the h100 feature, i.e.

NGC containers provide the best performance from the DGX H100 nodes. is an example for running NGC containers.

The two nodes have Intel CPUs. So Oscar modules can still be loaded and run on the two DGX nodes.

Efficient I/O is essential for good performance in data-intensive applications. Often, the file system is a substantial bottleneck on HPC systems, because CPU and memory technology has improved much more drastically in the last few decades than I/O technology.

Parallel I/O libraries such as MPI-IO, HDF5 and netCDF can help parallelize, aggregate and efficiently manage I/O operations. HDF5 and netCDF also have the benefit of using self-describing binary file formats that support complex data models and provide system portability. However, some simple guidelines can be used for almost any type of I/O on Oscar:

• Try to aggregate small chunks of data into larger reads and writes.

  For the GPFS file systems, reads and writes in multiples of 512KB

  provide the highest bandwidth.

By default, print in Python is buffered. When running Python in a batch job in SLURM you may see output less often than you would when running interactively. This is because the output is being buffered - the print statements are collected until there is a large amount to print, then the messages are all printed at once. For debugging or checking that a Python script is producing the correct output, you may want to switch off buffering.

Switch off buffering

For a single python script you can use the -u option, e.g.

python -u my_script.py

The -u stands for "unbuffered". You can use the environment variable PYTHONUNBUFFERED to set unbuffered I/O for your whole batch script.

#!/bin/bash
#SBATCH -n 1

export PYTHONUNBUFFERED=TRUE
python my_script.py

There is some performance penalty for having unbuffered print statements, so you may want to reduce the number of print statements, or run buffered for production runs.

Git Overview

Version Control refers to the management of changes made to source code or any such large amount of information in a robust manner by multiple collaborators. Git is by far the most popular version control system.

Git enables effective collaboration among developers. In a team setting, multiple developers often work on the same project simultaneously. With Git, each developer can work on their own local copy of the project, making changes and experimenting freely without affecting the main codebase. Git allows developers to merge their changes seamlessly, ensuring that modifications made by different individuals can be consolidated efficiently. It provides mechanisms to track who made specific changes, making it easier to understand the evolution of the project and identify potential issues.

Git Workflow

Nearly all operations that are performed by Git are in you local computing environment,except for a few used purely to synchronize with a remote. Some of the most common git operations are depicted below. In summary a typical flow consists of making changes to your files, staging them via git add, marking a save point via git commit, then finally syncing to your remote (e.g., GitHub) via git push. If you are pushing changes to your remote from multiple places, you can bring changes your most recent version using git pull, which is the equivalent of doing git fetch followed by a git merge operation

Below are some of the most commonly used Git commands. You can also get much more information by running git --help. And if you'd like to learn more there is an

While using Git on Oscar, make sure that you to have your correct Name and Email ID to avoid confusion while working with remote repositories (e.g., GitHub, GitLab, BitBucket).

Git can sometimes be a bit tricky. And we all eventually find ourselves in a place where we want to undo something or fix a mistake we made with Git. contains a number of excellent solutions to common problems we sometimes run in to with Git.

To use your condo account to submit jobs, please follow the steps below to check the association of your Oscar account and include condo information in your batch script or command line.

Step 1 - Check your account associations to find your condo Account and Partition information by running the following command:

sacctmgr -p list assoc where user=$USER | grep -E 'condo|Account|Partition'

In the example below, the user has access to two condos, where their Account and Partition are highlighted.

Step 2 - Choose the correct way to submit jobs to a condo according to the condo's Account column:

To see the running and pending jobs in a condo:

condo <condo-name>

If you have a premium account, that should be your default QOS for submitting jobs. You can check if you have a premium account with the command groups. If you have a priority account you will see priority in your the output form groups.

You can check the qos for a running job by running the command myq. The QOS column should show "pri-<username>"

If you are interested in seeing all your accounts and associations, you can use the following command:

In this example, we will install Jax.

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

interact -q gpu -g 1 -f ampere -m 20g -n 4

Here, -f = feature. We only need to build on Ampere once.

Step 2: Once your session has started on a compute node, run nvidia-smi to verify the GPU and then load the appropriate modules

module purge 
unset LD_LIBRARY_PATH
module load cuda cudnn

Step 3: Create and activate the virtual environment

python -m venv jax.venv
source jax.venv/bin/activate

Step 4: Install the required packages

pip install --upgrade pip
pip  install  --upgrade  "jax[cuda12_pip]"  -f  https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

Step 5: Test that JAX is able to detect GPUs

python
>>> from jax.lib import xla_bridge
>>> print(xla_bridge.get_backend().platform)
gpu

If the above function returns gpu, then it's working correctly. You are all set, now you can install other necessary packages.

Step 1 : Check for existing SSH key pair

Before generating new SSH key pair first check if you have an SSH key on your local machine.

ls ~/.ssh/id_*.pub

If there are existing keys, please move to Step 3

Step 2 : Generate a new SSH Keypair

ssh-keygen -t rsa

ssh-keygen.exe

Press Enter to accept the default file location and file name.

The ssh-keygen will ask you to type a secure passphrase. This is optional. If you don't want to use a passphrase just press Enter

Verify the SSH keys are generated correctly, you should see two files id_rsa and id_rsa.pub under ~/.ssh directory.

You will now need to copy your public key to Oscar. There are two ways to acomplish this.

If your OS comes with the ssh-copy-id utility, then you'll be able to copy your public key into Oscar as follows:

You will be prompted for a Password. The public key will be appended to the authorized_keys file on Oscar.

If you used a custom name for your key instead of the default id_rsa then you'll need pass the name of your key to ssh-copy-id i.e.,

If your system does not come with the ssh-copy-id utility installed, then you'll need to copy your public key by hand.

1. Get the contents of id_rsa.pub file. One option is to use cat in your teminal cat id_rsa.pub.

2. Copy the contents of this file to your clipboard, as we need to upload it to Oscar.

3. Login into Oscar via regular ssh ssh <username>@ssh.ccv.brown.edu

If everything went well, you will be logged in immediately withouth prompting you for a password.

Apptainer Using NGC Containers (Our #1 Recommendation)

There are multiple ways to install and run TensorFlow. Our recommended approach is via NGC containers. The containers are available via NGC Registry. In this example we will pull TensorFlow NGC container

1. Build the container:

   Copy

   apptainer build tensorflow-24.03-tf2-py3.simg docker://nvcr.io/nvidia/tensorflow:24.03-tf2-py3

   This will take some time, and once it completes you should see a .simg file.

   For your convenience, the pre-built container images are located in directory:

   You can choose either to build your own or use one of the pre-downloaded images.

   Working with Apptainer images requires lots of storage space. By default Apptainer will use ~/.apptainer as a cache directory which can cause you to go over your Home quota.

2. Once the container is ready, request an interactive session with a GPU

   Copy

   interact -q gpu -g 1 -f ampere -m 20g -n 4

3. Run a container wih GPU support

4. Or, if you're executing a specific command inside the container:

5. Make sure your Tensorflow image is able to detect GPUs

6. If you need to install more custom packages, the containers itself are non-writable but we can use the --user flag to install packages inside .local Example:

Here is how you can submit a SLURM job script by using the srun command to run your container. Here is a basic example:

The new Ampere architecture GPUs do not support older CUDA modules. Users must re-compile their applications with the newer CUDA/11 or older modules. Here are detailed instructions to compile major frameworks such as PyTorch, and TensorFlow.

PyTorch

Users can install PyTorch from a pip virtual environment or use pre-built singularity containers provided by Nvidia NGC.

To install via virtual environment:

# Make sure none of the LMOD modules are loaded
module purge 
module list

# create and activate the environment
python -m venv pytorch.venv
source pytorch.venv/bin/activate
pip install torch torchvision torchaudio

# test if it can detect GPUs 

To use NGC containers via Singularity :

• Pull the image from NGC

singularity build pytorch:21.06-py3 docker://nvcr.io/nvidia/pytorch:21.06-py3

• Export PATHs to mount the Oscar file system

export SINGULARITY_BINDPATH="/oscar/home/$USER,/oscar/scratch/$USER,/oscar/data/"

• To use the image interactively

• To submit batch jobs

cd

Moves the user into the specified directory. Change Directory.

cd .. to move one directory up

cd by itself to move to home directory

cd - to move to previous directory

cd <directory-path> to move to a directory (can be an absolute path or relative path)

cp <old_filepath> <new directory path>

Copies the file into the specified directory

clear

Clears the terminal

Lists the contents of a file. Concatenate files.

List contents within the current directory

Searches for the string / within the specified file and prints the line(s) with the result

Displays the path of the current directory that you are in. Present Working Directory

Displays the help manual instruction for the given command

Moves a file into a new directory.

mv <old_file_name> <new_file_name> to rename a file

Creates a new directory

Deletes a file

Deletes directories and the contents within them. -r stands for recursive

Removes the specified directory (must be empty)

Creates a blank new file

Arm Forge is available on Oscar. There are two products, DDT (debugger) and MAP (performance reports).

We recommend you use the Arm Forge remote client to launch your debugging jobs on Oscar. The first time you set up Arm Forge you will need to configure the client with the following steps:

1. Download the arm forge remote client on your machine.

   1. MacOS client

Arm DDT is a powerful graphical debugger suitable for many different development environments, including:

• Single process and multithreaded software.

• OpenMP.

• Parallel (MPI) software.

Arm MAP is a parallel profiler that shows you which lines of code took the most time to run, and why. Arm MAP does not require any complicated configuration, and you do not need to have experience with profiling tools to use it.

Arm MAP supports:

• MPI, OpenMP and single-threaded programs.

• Small data files. All data is aggregated on the cluster and only a few megabytes written to disk, regardless of the size or duration of the run.

• Sophisticated source code view, enabling you to analyze performance across individual functions.

To start an interactive session for running serial or threaded programs on an Oscar compute node, simply run the command interact from the login node:

interact

By default, this will create an interactive session that reserves 1 core and 4GB of memory for a period of 30 minutes. You can change the resources reserved for the session from these default limits by modifying the interact command:

usage: interact [-n cores] [-t walltime] [-m memory] [-q queue]
                [-o outfile] [-X] [-f featurelist] [-h hostname] [-g ngpus]

Starts an interactive job by wrapping the SLURM 'salloc' and 'srun' commands.

options:
  -n cores        (default: 1)
  -t walltime     as hh:mm:ss (default: 30:00)
  -m memory       as #[k|m|g] (default: 4g)
  -q queue        (default: 'batch')
  -o outfile      save a copy of the sessions output to outfile (default: off)
  -X              enable X forwarding (default: no)
  -f featurelist  CCV-defined node features (e.g., 'e5-2600'),
                  combined with '&' and '|' (default: none)
  -h hostname     only run on the specific node 'hostname'
                  (default: none, use any available node)
  -a account      user SLURM accounting account name
  -g ngpus        number of GPUs   

For example, the command

$ interact -n 20 -t 01:00:00 -m 10g

requests an interactive session with 20 cores and 10 GB of memory (per node) for a period of 1 hour.

In this example, we will install PyTorch (refer to sub-pages for TensorFlow and Jax).

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

interact -q gpu -g 1 -f ampere -m 20g -n 4

Here, -f = feature. We only need to build on Ampere once.

Step 2: Once your session has started on a compute node, run nvidia-smi to verify the GPU and then load the appropriate modules

Step 3: Create and activate the virtual environment, unload the pre-loaded modules then load cudnn and cuda dependencies

module purge
unset LD_LIBRARY_PATH
module load cudnn cuda

Step 4: Create a new virtual environment

Step 5: Install the required packages

The aforementioned will install the latest version of PyTorch with cuda11 compatibility, for older versions you can specify the version by:

Step 6: Test that PyTorch is able to detect GPUs

If the above functions return True and GPU model, then it's working correctly. You are all set, now you can install other necessary packages.

Filing a Support Ticket

Filing a good support ticket makes it much easier for CCV staff to deal with your request

When you email support@ccv.brown.edu aim to include the following:

• State the problem/request in the subject of the email

• Describe which software and with version you are using

• Error message (if there was one)

• The job number

• How you were running, e.g. batch, interactively, vnc

• Give as as small an example as possible that reproduces the problem

Ask questions and search for previous problems at our .

Join our CCV-Share Slack workspace to discuss your questions with CCV Staff in the #oscar channel.

CCV holds weekly office hours. These are drop in sessions where we'll have one or more CCV staff members available to answer questions and help with any problems you have. Please visit for upcoming office hours and events.

You can arrange to meet with a CCV staff member in person to go over difficult problems, or to discuss how best to use Oscar. Email support@ccv.brown.edu to arrange a consultation.

CCV provides access to HPC resources for classes, workshops, demonstrations, and other instructional uses. In general, the system is available for most types of instructional use at Brown where HPC resources are required, and we will do what we can to provide the resources necessary to help teach your class. We do ask that you follow these guidelines to help us better support your class.

Account Requests and Software Needs

Requests for class accounts should be made in writing to support@ccv.brown.edu two weeks prior to the beginning of class, and should be made in bulk. Please provide the Brown username (required), name and Brown Email address for the students, TAs and instructor as well as the course number and the semester. Requests for specific software should also be made two weeks before the start of the semester, and should be properly licensed, tested and verified to work by an instructor or TA.

Usage Expectations and System Utilization

Unless prior arrangements are made, student class accounts will have the same priority and access as free accounts on the CCV system. Access can be provided to specialized hardware or higher cores if needed provided it does not impact research use of the CCV systems. Be aware that usage of the CCV system is unpredictable, and high utilization of the system could impact a student's ability to finish assignments in a specific time period. We also encourage instructors to give an overview of the system and discuss computing policies before students use the system. CCV can provide resources (slides, documentation and in class workshops) to help prepare students to use HPC system. CCV staff are always available to meet directly with instructors and TAs to help prepare for classes and help setup specific software or environments for the class.

Support

It is expected that any class being taught using CCV resources will have its own TA. The TA should be the first line of support for any problems or questions the students may have regarding the use of the CCV system. CCV staff may not know specifics about how to use or run the programs the class is using, and can’t provide direct support to students for that software.

CCV will provide limited duration guests accounts that are custom tailored for the class use of the system. These accounts will have a username of “ccvws###”, and each account is associated with an individual student, instructor, or TA. Guest accounts are temporary and are only active for the duration of the class, and are deactivated at the conclusion of the semester/workshop. Account data is kept intact on our system for one semester after the conclusion of the class, and is then permanently deleted from the CCV system.

CCV uses all-flash parallel filesystem (Vast Data). Users have a home, data, and scratch space.

home ~

• 100GB of space

• Optimized for many small files

• 30 days snapshots

• The quota is per individual user

• A grace period of 14 days

data ~/data

• Each PI gets 256GB for free

• Optimized for reading large files

• 30 days snapshots

• The quota is by group

scratch ~/scratch

• 512G (soft-quota): 12T (hard-quota)

• Optimized for reading/writing large files

• 30 days snapshots

• Purging: Files not accessed for 30 days may be deleted

A good practice is to configure your application to read any initial input data from ~/data and write all output into ~/scratch. Then, when the application has finished, move or copy data you would like to save from ~/scratch to ~/data.

To see how much space on your directories, you can use the command checkquota. Below is an example output

Setting Up Gaussian

In order to use Gaussian on Oscar, you must be a part of the ccv-g09 group. To check your groups, run the groups command in the terminal.

You must first choose a Gaussian module to load. To see available Gaussian modules, run module avail gauss. You can load a Gaussian module using the command module load <module-name>.

Available Versions

• Gaussian 9 (g09)

• Gaussian 16 (g16)

NOTE: There are three versions of g09, you can load any one of those, but the newer version g16 is preferred now. If using g09 just replace g16 below with g09.

Gaussian can be run either interactively or within a batch script using one of two command styles:

• g16 job-name

• g16 <input-file >output-file

In the first form, the program reads input from job-name.gjf and writes its output to job-name.log. When no job-name has been specified, the program will read from standard input and write to standard output

Given a valid .gjf file (we'll call it test-file.gjf), we can use the following simple batch script to run Gaussian:

Then queue the script using

Once the job has been completed, you should have a g16-test.out, a g16-test.err, and a test-file.out.

1. Install the Remote Development extension pack for VSCode

2. Open VSCode settings

   • On Windows/Linux - File > Preferences > Settings

   • On macOS - Code > Preferences > Settings

   Search for symlink and make sure the symlink searching is unchecked
3. Under VSCode settings, search for remote ssh timeout and manually enter a timeout value i.e. 50s. It should give you enough time to complete 2-Factor Authentication.
4. Edit the ~/.ssh/config file on your local machine, add the following lines. Replace <username> with your Oscar username.

5. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select ccv-vscode-node
6. When prompted in VSCode, please enter your Brown password and complete the DUO authentication. After that, wait about 30 seconds and VSCode should connect to Oscar.

Combining CUDA and MPI

Mixing MPI (C) and CUDA (C++) code requires some care during linking because of differences between the C and C++ calling conventions and runtimes. One option is to compile and link all source files with a C++ compiler, which will enforce additional restrictions on C code. Alternatively, if you wish to compile your MPI/C code with a C compiler and call CUDA kernels from within an MPI task, you can wrap the appropriate CUDA-compiled functions with the extern keyword, as in the following example.

These two source files can be compiled and linked with both a C and C++ compiler into a single executable on Oscar using:

module load mpi cuda
mpicc -c main.c -o main.o
nvcc -c multiply.cu -o multiply.o
mpicc main.o multiply.o -lcudart

The CUDA/C++ compiler nvcc is used only to compile the CUDA source file, and the MPI C compiler mpicc is used to compile the C code and to perform the linking.

multiply.cu

#include <cuda.h>
#include <cuda_runtime.h>

__global__ void __multiply__ (const float *a, float *b)
{
    const int i = threadIdx.x + blockIdx.x * blockDim.x;
    b[i] *= a[i];
}

extern "C" void launch_multiply(const float *a, const *b)
{
    /* ... load CPU data into GPU buffers a_gpu and b_gpu */

    __multiply__ <<< ...block configuration... >>> (a_gpu, b_gpu);

    safecall(cudaThreadSynchronize());
    safecall(cudaGetLastError());

    /* ... transfer data from GPU to CPU */
}

Note the use of extern "C" around the function launch_multiply, which instructs the C++ compiler (nvcc in this case) to make that function callable from the C runtime. The following C code shows how the function could be called from an MPI task.

Listing running and queued jobs

The squeue command will list all jobs scheduled in the cluster. We have also written wrappers for squeue on Oscar that you may find more convenient:

myq                   List only your own jobs.
allq                  List all jobs, but organized by partition, and a summary of the nodes in use in the
                      partition.
allq <partition>      List all jobs in a single partition.
myjobinfo            Get the time and memory used for your jobs.

Viewing estimated time until completion for pending jobs

squeue -u <your-username> -t PENDING --start

This command will list all of your pending jobs and the estimated time until completion.

Canceling jobs

scancel <jobid>

View details about completed jobs

sacct

The sacct command will list all of your running, queued and completed jobs since midnight of the previous day. To pick an earlier start date, specify it with the -S option:

sacct -S 2012-01-01

To find out more information about a specific job, such as its exit status or the amount of runtime or memory it used, specify the -l ("long" format) and -j options with the job ID:

sacct -lj <jobid>

(example)

The myjobinfo command uses the sacct command to display "Elapsed Time", "Requested Memory" and "Maximum Memory used on any one Node" for your jobs. This can be used to optimize the requested time and memory to have the job started as early as possible. Make sure you request a conservative amount based on how much was used.

ReqMem shows the requested memory: A c at the end of number represents Memory Per CPU, a n represents Memory Per Node. MaxRSS is the maximum memory used on any one node. Note that memory specified to sbatch using --mem is Per Node.

The 'jobstats' utility is now available for analyzing recently completed jobs, comparing the resources used to those requested in the job script, including CPU, GPU, and memory. If email notifications are enabled, 'jobstats' sends an email with the results and includes a prompt to contact support for help with resource requests.

Run this command in a bash shell on Oscar. No additional module needs to be loaded.

To send this output to your email after the job is completed, make sure that these lines are in your job submit script

A job array is a collection of jobs that all run the same program, but on different values of a parameter. It is very useful for running parameter sweeps, since you don't have to write a separate batch script for each parameter setting.

To use a job array, add the option:

#SBATCH --array=<range>

in your batch script. The range can be a comma separated list of integers, along with ranges separated by a dash. For example:

1-20
1-10,12,14,16-20

A job will be submitted for each value in the range. The values in the range will be substituted for the variable $SLURM_ARRAY_TASK_ID in the remainder of the script. Here is an example of a script for running a serial Matlab script on 16 different parameters by submitting 16 different jobs as an array:

#!/bin/bash
#SBATCH -J MATLAB
#SBATCH -t 1:00:00
#SBATCH --array=1-16

# Use '%A' for array-job ID, '%J' for job ID and '%a' for task ID
#SBATCH -e arrayjob-%a.err
#SBATCH -o arrayjob-%a.out

echo "Starting job $SLURM_ARRAY_TASK_ID on $HOSTNAME"
matlab -r "MyMatlabFunction($SLURM_ARRAY_TASK_ID); quit;"

You can then submit the multiple jobs using a single sbatch command:

$ sbatch <jobscript>

The $SLURM_ARRAY_TASK_ID can be manipulated as needed. For example, you can generate a fixed length number form it. The following example generates a number of length of 3 from $SLURM_ARRAY_TASK_ID.

For more info:

Several versions of Python are available on Oscar as modules. However, we recommend using the system Python available at /usr/bin/python . You do not need to load any module to use this version of Python.

$ which python
/usr/bin/python
$ python --version
Python 3.9.16

pip is also installed as a system package, but other common Python packages (e.g., SciPy, NumPy) are not installed on the system. This affords individual users complete control over the packages they are using, thereby avoiding issues that can arise when code written in Python requires specific versions of Python packages.

Users can install any Python package they require by following the instructions given on the Installing Python Packages page.

Setting up VASP

In order to use VASP, you must be a part of the vasp group on Oscar. To check your groups, run the groups command in the terminal.

First, you must choose which VASP module to load. You can see the available modules using module avail vasp. You can load your preferred VASP module using module load <module-name>.

Available Versions

• VASP 5.4.1

• VASP 5.4.4

• VASP 6.1.1

Within a batch job, you should specify the number of MPI tasks as

If you would like 40 cores for your calculation, you would include the following in your batch script:

OSCAR

Oscar is the shared compute cluster operated by CCV.

Oscar runs the Linux Red Hat 9 operating system. General Linux documentation is available from The Linux Documentation Project. We recommend you read up on basic Linux commands before using Oscar. Some of the most common commands you'll be using in Oscar can also be found on our Quick Reference page.

Oscar has two login nodes and several hundred compute nodes. When users log in through Secure Shell (SSH), they are first put on one of the login nodes which are shared among several users at a time. You can use the login nodes to compile your code, manage files, and launch jobs on the compute nodes from your own computer. Running computationally intensive or memory intensive programs on the login node slows down the system for all users. Any processes taking up too much CPU or memory on a login node will be killed. Please do not run Matlab on the login nodes.

What username and password should I be using?

• If you are at Brown and have requested a regular CCV account, your Oscar login will be authenticated using your Brown credentials, i.e. the same username and password that you use to log into any Brown service such as "canvas".

• If you are an external user, you will have to get a sponsored ID at Brown through the department with which you are associated before requesting an account on Oscar. Once you have the sponsored ID at Brown, you can on Oscar and use your Brown username and password to log in.

To log in to Oscar you need Secure Shell (SSH) on your computer. Mac and Linux machines normally have SSH available. To login in to Oscar, open a terminal and type

Windows users need to install an SSH client. We recommend , a free SSH client for Windows. Once you've installed PuTTY, open the client and use <username>@ssh.ccv.brown.edufor the Host Name and click Open. The configuration should look similar to the screenshot below.

The first time you connect to Oscar you will see a message like:

You can type yes . You will be prompted for your password. Note that nothing will show up on the screen when you type in your password; just type it in and press enter. You will now be in your home directory on Oscar. In your terminal you will see a prompt like this:

Congratulations, you are now on one of the Oscar login nodes.

Users on Oscar have three places to store files:

• home

• scratch

• data

Note that class accounts may not have a data directory. Users who are members of more than one research group may have access to multiple data directories.

From the home directory, you can use the command ls to see your scratch directory and your data directory (if you have one) and use cd to navigate into them if needed.

To see how much space in your directories, use the command checkquota. Below is an example output:

A good practice is to configure your application to read any initial input data from ~/data and write all output into ~/scratch. Then, when the application has finished, move or copy data you would like to save from ~/scratch to ~/data. For more information on which directories are backed up and best practices for reading/writing files, see and . You can go over your quota up to the hard limit for a grace period. This grace period is to give you time to manage your files. When the grace period expires you will be unable to write any files until you are back under quota.

You can also transfer files to and from the Oscar Filesystem from your own computer. See .

CCV uses the package for managing the software environment on OSCAR. To see the software available on Oscar, use the command module avail. You can load any one of these software modules using module load <module>. The command module list shows what modules you have loaded. Below is an example of checking which versions of the module 'workshop' are available and loading a given version.

For a list of all Lmod commands, see . If you have a request for software to be installed on Oscar, email support@ccv.brown.edu.

You can connect remotely to a graphical desktop environment on Oscar using . The OOD Desktop integrates with the scheduling system on Oscar to create dedicated, persistent VNC sessions that are tied to a single user.

Using VNC, you can run graphical user interface (GUI) applications like Matlab, Mathematica, etc. while having access to Oscar's compute power and file system.

You are on Oscar's login nodes when you log in through SSH. You should not (and would not want to) run your programs on these nodes as these are shared by all active users to perform tasks like managing files and compiling programs.

With so many active users, a shared cluster has to use a "job scheduler" to assign compute resources to users for running programs. When you submit a job (a set of commands) to the scheduler along with the resources you need, it puts your job in a queue. The job is run when the required resources (cores, memory, etc.) become available. Note that since Oscar is a shared resource, you must be prepared to wait for your job to start running, and it can't be expected to start running straight away.

Oscar uses the SLURM job scheduler. Batch jobs are the preferred mode of running programs, where all commands are mentioned in a "batch script" along with the required resources (number of cores, wall-time, etc.). However, there is also a way to run programs interactively.

For information on how to submit jobs on Oscar, see .

There is also extensive documentation on the web on using SLURM ().

• Online resources: , , ,

• CCV's page detailing you might face on Oscar

• Email

For Python 3, we recommend using the system Python. You do not need to load any Python module to use system Python3

There are several ways for users to install python packages on Oscar

• using a Python environment

• using

First load the IDL module that you want to use with module load idl/version_number:

As indicated by the system dialogue, you will need to enter the following command to set up the environment for IDL:

Once you've set up IDL in the way outlined above, you can open the IDL command line by simply using the command idl:

As is stated in the , IDL in command-line mode "uses a text-only interface and sends output to your terminal screen or shell window." Thus, this is a mode in which you can enter commands and see their results in real time, but it is not where one should write full IDL programs.

To write an IDL program, you can use any of the text editors on Oscar (such as vim, emacs, and nano) or you can create the program in a file on your own computer and then copy that file to Oscar when you are finished. Here is an example (hello world) IDL program

We recommend all users to install Python packages within an environment. This can be a . Follow these steps to use such environments in the .

1. Open a terminal on Oscar.

2. Load the relevant python module and create and/or activate the environment. See this page for more information about creating .

Oscar users are not permitted to:

• Share their accounts or passwords with others or enable unauthorized users to access Center for Computation and Visualization resources

• Use Center for Computation and Visualization resources for personal economic gain

• Engage in unauthorized activity (e.g., cryto currency mining etc.) that intentionally impacts integrity of resources

CCV provides preemptible Slurm partitions that allow condo account holders to submit jobs beyond their standard per-account resource limits by utilizing idle capacity across the cluster.

Available Partitions