These pages list some command commands and terms you will come across while using Oscar.

Common Acronyms and Terms

Managing Modules

Common Linux Commands

module list

Lists all modules that are currently loaded in your software environment.

module avail

Lists all available modules on the system. Note that a module can have multiple versions. Use module avail <name> to list available modules which start with <name>

module help <name>

Prints additional information about the given software.

module load <name>

Adds a module to your current environment. If you load using just the name of a module, you will get the default version. To load a specific version, load the module using its full name with the version: "module load gcc/10.2"

module unload <name>

Removes a module from your current environment.

Efficient I/O is essential for good performance in data-intensive applications. Often, the file system is a substantial bottleneck on HPC systems, because CPU and memory technology has improved much more drastically in the last few decades than I/O technology.

Parallel I/O libraries such as MPI-IO, HDF5 and netCDF can help parallelize, aggregate and efficiently manage I/O operations. HDF5 and netCDF also have the benefit of using self-describing binary file formats that support complex data models and provide system portability. However, some simple guidelines can be used for almost any type of I/O on Oscar:

• Try to aggregate small chunks of data into larger reads and writes.

  For the GPFS file systems, reads and writes in multiples of 512KB

  provide the highest bandwidth.

• Avoid using ASCII representations of your data. They will usually

  require much more space to store, and require conversion to/from

  binary when reading/writing.

• Avoid creating directory hierarchies with thousands or millions of

  files in a directory. This causes a significant overhead in managing

  file metadata.

While it may seem convenient to use a directory hierarchy for managing large sets of very small files, this causes severe performance problems due to the large amount of file metadata. A better approach might be to implement the data hierarchy inside a single HDF5 file using HDF5's grouping and dataset mechanisms. This single data file would exhibit better I/O performance and would also be more portable than the directory approach.

If you publish research that benefited from the use of CCV services or resources, we would greatly appreciate an acknowledgment that states:

This research [Part of this research] was conducted using [computational/visualization]
resources and services at the Center for Computation and Visualization, Brown University.

Connect to OSCAR

The simplest way to connect to Oscar is via Open OnDemand (OOD). To connect to OOD, go to https://ood.ccv.brown.eduand log in using your Brown credentials. For more details on Open OnDemand click .

Alternatively, you can connect to OSCAR via SSH (Terminal):

Submit a Job

You can submit a job using sbatch:

You can confirm that your job ran successfully by running:

For more detailed information on submitting jobs, see the section of the documentation.

Transfer Files

To get specific files on to / off of Oscar, read through the page of the documentation.

Get Help

If you encounter problems while using Oscar, check out the documentation, or read through the Overview page.

RStudio is an IDE for R that can be run on Oscar.

Launching RStudio

Open the Open On Demand Dashboard by following this link. Select RStudio (under "Default GUI's"). Fill in the form to allocate the required resources, and optionally select your R modules. Finally, click the "Launch Session" button.

Known Issues

Plotting figures may not work within RStudio. If this is the case, save the plots to a file, and view them through the Open On Demand Desktop App. If plots are required for your task, launch RStudio through the Desktop App.

Oscar is Brown University's high performance computing cluster for both research and classes. Oscar is maintained and supported by .

Please contact if there are any questions on Oscar.

Accounts

If you do not have an Oscar account, you can request one by clicking the following link:

Anyone with a Brown account can get a free Exploratory account on Oscar, or pay for priority accounts

cd

Moves the user into the specified directory. Change Directory.

cd .. to move one directory up

cd by itself to move to home directory

Account and Access

Oscar users will keep their access to Oscar as long as their Brown account are still active. To be able to access Oscar account after a user's Brown account is deactivated, the user needs to through the department the user is associated.

When connecting from a campus network to sshcampus.ccv.brown.edu you can set up SSH keys as a form of authentication instead of having to enter your password interactively. Follow the insctructions below that correspond to your operating system/connection method.

If you have an installation of X11 on your local system, you can access Oscar with X forwarding enabled, so that the windows, menus, cursor, etc. of any X applications running on Oscar are all forwarded to your local X11 server. Here are some resources for setting up X11:

• Mac OS -

• Windows -

SSH provides a method of sharing the ssh keys on your local machine with Oscar. This feature is called Agent Forwarding and can be useful for instance when working with version control or other services that authenticate via ssh keys. Below are instructions on how to configure your SSH connection to forward ssh-agent for diffeent operating systems

Many scientific and HPC software packages are already installed on Oscar, and additional packages can be requested by submitting a ticket to [email protected]. If you want a particular version of the software, do mention it in the email along with a link to the web page from where it can be downloaded. You can also install your own software on Oscar.

CCV cannot, however, supply funding for the purchase of commercial software. This is normally attributed as a direct cost of research, and should be purchased with research funding. CCV can help in identifying other potential users of the software to potentially share the cost of purchase and maintenance. Several commercial software products that are licensed campus-wide at Brown are available on Oscar.

For software that requires a Graphical User Interface (GUI) we recommend using CCV's rather than X-Forwarding.

By default, print in Python is buffered. When running Python in a batch job in SLURM you may see output less often than you would when running interactively. This is because the output is being buffered - the print statements are collected until there is a large amount to print, then the messages are all printed at once. For debugging or checking that a Python script is producing the correct output, you may want to switch off buffering.

Switch off buffering

For a single python script you can use the -u option, e.g.

python -u my_script.py

Anaconda provides Python, R and other packages for scientific computing including data sciences, machine learning, etc.

There is one anaconda module:

Account Usage

Oscar users are not permitted to:

• Share their accounts or passwords with others or enable unauthorized users to access Center for Computation and Visualization resources

• Use Center for Computation and Visualization resources for personal economic gain

• Engage in unauthorized activity (e.g., cryto currency mining etc.) that intentionally impacts integrity of resources

Storage

Each user (premium or exploratory) gets 20GB Home Directory, 512GB short-term Scratch, and 256G Data directory (shared amongst the members of group)

• Files in Scratch Directory not accessed for last 30 days are automatically purged. CCV only stores snapshots for 7 days after that files will be automatically deleted.

• PI has the ultimate access to Data Directory - if a student leaves Brown the files in Data directory will be owned by the PI.

Software and Data

All software and data stored or used on Center hosted systems must be appropriately and legally acquired and must be used in compliance with applicable licensing terms. Unauthorized misuse or copying of copyrighted materials is prohibited.

Data Retention

CCV reserves the right to remove any data at any time and/or transfer data or other individuals (such as Principal Investigators working on a same or similar project) after a user account is deleted is no longer affiliated with Brown University.

Accounts Validity

• Once created, Oscar accounts are valid for duration of one's Brown AD credentials

CCV provides access to HPC resources for classes, workshops, demonstrations, and other instructional uses. In general, the system is available for most types of instructional use at Brown where HPC resources are required, and we will do what we can to provide the resources necessary to help teach your class. We do ask that you follow these guidelines to help us better support your class.

Account Requests and Software Needs

Requests for class accounts should be made in writing to [email protected] two weeks prior to the beginning of class, and should be made in bulk. Please provide the Brown username (required), name and Brown Email address for the students, TAs and instructor as well as the course number and the semester. Requests for specific software should also be made two weeks before the start of the semester, and should be properly licensed, tested and verified to work by an instructor or TA.

Usage Expectations and System Utilization

Unless prior arrangements are made, student class accounts will have the same priority and access as free accounts on the CCV system. Access can be provided to specialized hardware or higher cores if needed provided it does not impact research use of the CCV systems. Be aware that usage of the CCV system is unpredictable, and high utilization of the system could impact a student's ability to finish assignments in a specific time period. We also encourage instructors to give an overview of the system and discuss computing policies before students use the system. CCV can provide resources (slides, documentation and in class workshops) to help prepare students to use HPC system. CCV staff are always available to meet directly with instructors and TAs to help prepare for classes and help setup specific software or environments for the class.

Support

It is expected that any class being taught using CCV resources will have its own TA. The TA should be the first line of support for any problems or questions the students may have regarding the use of the CCV system. CCV staff may not know specifics about how to use or run the programs the class is using, and can’t provide direct support to students for that software.

Class Guest Accounts

CCV will provide limited duration guests accounts that are custom tailored for the class use of the system. These accounts will have a username of “ccvws###”, and each account is associated with an individual student, instructor, or TA. Guest accounts are temporary and are only active for the duration of the class, and are deactivated at the conclusion of the semester/workshop. Account data is kept intact on our system for one semester after the conclusion of the class, and is then permanently deleted from the CCV system.

Oscar is a shared machine used by hundreds of users at once. User requests are called jobs. A job is the combination of the resource requested and the program you want to run on the compute nodes of the Oscar cluster. On Oscar, Slurm is used to schedule and manage jobs.

Jobs can be run on Oscar in two different ways:

• Interactive jobs allow the user to interact with programs (e.g., by entering input manually, using a GUI) while they are running. However, if your connection to the system is interrupted, the job will abort. Small jobs with short run times and jobs that require the use of a GUI are best-suited for running interactively.

• Batch jobs allow you to submit a script that tells the cluster how to run your program. Your program can run for long periods of time in the background, so you don't need to be connected to Oscar. The output of your program is continuously written to an output file that you can view both during and after your program runs.

Jobs are scheduled to run on the cluster according to your account priority and the resources you request (i.e., cores, memory, and runtime). In general, the fewer resources you request, the less time your job will spend waiting in the queue.

Users can check the nodes in a partition using this command nodes . As of July 2025 the Oscar cluster currently has following nodes in bigmem partition.

$ nodeinfo|grep bigmem
node1609  32-core  770GB 32core,intel,scalable,cascade,edr         idle~      0     0%    766GB  99.6%   bigmem                       
node1610  32-core  770GB 32core,intel,scalable,cascade,edr           mix     17  53.1%    633GB  82.4%   bigmem                       
node1611  32-core  770GB 32core,intel,scalable,cascade,edr         idle~      0     0%    712GB  92.6%   bigmem                       
node1612  32-core  770GB 32core,intel,scalable,cascade,edr           mix     18  56.2%    577GB  75.1%   bigmem                       
node2412 192-core 1540GB 192core,amd,genoa,edr                       mix    100    52%   1061GB    69%   bigmem                       
node2415 192-core 1540GB 192core,amd,genoa,edr                       mix    153  79.6%   1231GB    80%   bigmem    

All Oscar users have access to this partition, and can submit jobs to it. To submit batch jobs to large memory nodes, include the following in your batch script:

#SBATCH -p bigmem

To run an interactive job on large memory node, launch the interact command with the following flag:

$ interact -q bigmem

CCV uses all-flash parallel filesystem (Vast Data). Users have a home, data, and scratch space.

home ~

• 100GB of space

• Optimized for many small files

• 30 days snapshots

• The quota is per individual user

• A grace period of 14 days

data ~/data

• Each PI gets 256GB for free

• Optimized for reading large files

• 30 days snapshots

• The quota is by group

scratch ~/scratch

• 512G (soft-quota): 12T (hard-quota)

• Optimized for reading/writing large files

• 30 days snapshots

• Purging: Files not accessed for 30 days may be deleted

A good practice is to configure your application to read any initial input data from ~/data and write all output into ~/scratch. Then, when the application has finished, move or copy data you would like to save from ~/scratch to ~/data.

To see how much space on your directories, you can use the command checkquota. Below is an example output

This is a quick guide for resolving issues related to file system quotas. To read more details about these quotas, refer to this page.

Step 1: Identify the directory

Run the checkquota command and identify the line that shows the warning status message.

If this directory is either /oscar/home or /oscar/scratch , you will have to take the subsequent steps to resolve this issue. If the directory is data+<group> you should inform others in your group and take collective action to resolve this issue.

Step 2: Disk Space or Inodes

Check whether you have exceeded your disk space quota or your inodes quota. Disk space usage is specified in GB or TB while inodes usage is just numerical count.

Step 3: Remove files

You will need to take the following steps based on the quota you have exceeded.

Disk Space quota:

The fastest way to reduce this usage is identifying large and unnecessary files. Load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. If they are not sorted by size, press lowercase s to sort them by size. You can navigate the directory tree using the arrow keys and delete any files or directories that are unnecessary.

Some programs leave a lot of temporary files on the disk that may not be necessary.

• Apptainer: Run the command to clear the apptainer cache. This will clear up the cache in your home directory without affecting any container images. However, pulling a new image from a repository may be slower in the future.

• Conda: Run the command to delete any tarballs downloaded by conda. This does not affect any existing conda or python virtual environments. However, it may slow down the installation of some packages in the future

• Core Dump Files: This files are typically named core.<number> A core dump file is generated when a program crashes. It contains the state of the system and it is useful for debugging purposes. You can safely delete any core dump files if you know the reason behind the crash. Old core dump files can take up a lot of disk space and they can be safely deleted.

Inodes quota:

Inode usage can be reduced by removing any files and directories OR tarring up large nested directories. When a directory is converted to a tar ball, it uses a single inode instead of one inode per directory or file. This can drastically decrease your inode usage. Identify directories that contain a large number of files or a very large nested tree of directories with a lot of files.

To identify such directories, load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. Press uppercase C to switch the sorting criteria to "number of items". You can navigate the directory tree using the arrow keys and delete or tar any files or directories that are unnecessary.

To create a tar ball of a directory:

If your usage has exceeded quota and you cannot write to the directory, you can tar ball in another directory. Using this command, you can create a tar ball in the scratch directory:

Filing a Support Ticket

Filing a good support ticket makes it much easier for CCV staff to deal with your request

When you email [email protected] aim to include the following:

• State the problem/request in the subject of the email

• Describe which software and with version you are using

• Error message (if there was one)

• The job number

• How you were running, e.g. batch, interactively, vnc

• Give as as small an example as possible that reproduces the problem

Q&A Forum

Ask questions and search for previous problems at our .

Slack

Join our CCV-Share Slack workspace to discuss your questions with CCV Staff in the #oscar channel.

Office Hours

CCV holds weekly office hours. These are drop in sessions where we'll have one or more CCV staff members available to answer questions and help with any problems you have. Please visit for upcoming office hours and events.

Arrange a Meeting

You can arrange to meet with a CCV staff member in person to go over difficult problems, or to discuss how best to use Oscar. Email [email protected] to arrange a consultation.

• Putty Installation

• SSH to Oscar from Linux

• SSH to Oscar from Mac

You can launch several different apps on the Open OnDemand (OOD) interface. All of these apps start of a Slurm batch job on the Oscar cluster with the requested amount of resources. These jobs can access the filesystem on Oscar and all output files are written to the Oscar's file system.

Launching an App on OOD

1. Open https://ood.ccv.brown.edu on any browser of the your choice

2. If prompted, enter your Brown username and password.

3. Click on the "Interactive Apps" tab at the top of the screen to see the list of available apps. This will open the form to enter the details of the job.

4. Follow the instructions on the form to complete it. Some of fields can be left blank and OOD will choose the default option for you.

5. Click Launch to submit an OOD job. This will open a new tab on the browser It may take a few minutes for this job to start.

6. Click "Launch <APP>" again if prompted in the next tab.

You may use either Globus (recommended) or smbclient to transfer data between Oscar and Campus File Storage.

Globus

Follow the instructions here for transferring data between files.brown.edu and Oscar.

smbclient

You can transfer files between Campus File Storage and Oscar using .

Transfer Instructions

1) Log into Oscar:

2) Start a screen session. This will allow you to reattach to your terminal window if you disconnect.

3) To use Oscar's high-speed connection to Campus File Storage - Replicated:

Similarly to access Campus File Storage - Non-Replicated ( LRS: Locally Redundant Share)

Replace SHARE_NAME, DIRECTORY_NAME, and BROWN_ID. DIRECTORY_NAME is an optional parameter. The password required is your Brown password.

4) Upload/download your data using the FTP "put"/"get" commands. Replace DIRECTORY_NAME with the folder you'd like to upload.

5) You can detach from the screen session with a "CTRL+A D" keypress. To reattach to your session:

smbclient basics

• put is upload to Campus File Storage

Usage: put <local_file> [remote file name]

Copy <local_file> from Oscar to Campus File Storage. The remote file name is optional (use if you want to rename the file)

• get is download to Oscar

Usage: get <remote_file> [local file name] Copy <remote_file> from the Campus File Storage to Oscar. The local file name is optional (use if you want to rename the file)

Moving more than one file:

To move more than one file at once use mput or mget. By default:

recurse is OFF. smbclient will not recurse into any subdirectories when copying files

prompt is ON. smbclient will ask for confirmation for each file in the subdirectories

You can toggle recursion ON/OFF with:

You can toggle prompt OFF/ON with:

1. If you have an existing virtual environment, proceed to step 2. Otherwise, to create a new virtual environment:

$ python3 -m venv my_env
$ source my_env/bin/activate
#Install packages manually or from requirements.txt file
$ pip install -r requirements.txt

2. Search for Python.VenvPath as shown in the picture below:

3. VSCode expects you to have multiple virtual environments for each of your different python projects, and it expects you to put them all in the same directory. Pointing to the parent directory lets it scan and find all expected virtual environments, and then you can easily toggle between them in interface.

4. Once you have the virtual environment selected, the debugging capabilities should work.

The Oscar GPUs are in a separate partition to the regular compute nodes. The partition is called gpu. To see how many jobs are running and pending in the gpu partition, use

allq gpu

Interactive use

To start an session on a GPU node, use the interact command and specify the gpu partition. You also need to specify the requested number of GPUs using the -g option:

interact -q gpu -g 1

Batch jobs

Here is an example batch script for a cuda job that uses 1 gpu and 1 cpu for 5 minutes

To submit this script:

DGX GPU Nodes in the GPU-HE Partition

All the nodes in the gpu-he partition have V100 GPUs. However, two of them are DGX nodes (gpu1404/1405) which have 8 GPUs. When a gpu-he job requests for more than 4 GPUs, the job will automatically be allocated to the DGX nodes.

The other non-DGX nodes actually have a better NVLink interconnect topology as all of them have direct links to the other. So the non-DGX nodes are better for a gpu-he job if the job does not require more than 4 GPUs.

Several versions of Python are available on Oscar as modules. However, we recommend using the system Python available at /usr/bin/python . You do not need to load any module to use this version of Python.

$ which python
/usr/bin/python
$ python --version
Python 3.9.16

pip is also installed as a system package, but other common Python packages (e.g., SciPy, NumPy) are not installed on the system. This affords individual users complete control over the packages they are using, thereby avoiding issues that can arise when code written in Python requires specific versions of Python packages.

Users can install any Python package they require by following the instructions given on the Installing Python Packages page.

screen is a "terminal multiplexer", it enables a number of terminals (or windows) to be accessed and controlled from a single terminal. screen is a great way to save an interactive session between connections to oscar. You can reconnect to the session from anywhere!

Screen commands

Common commands are:

• start a new screen session with session name: screen -S <name>

• list running sessions/screens: screen -ls

• attach to session by name: screen -r <name>

• detach: Ctrl+a d

• detach and logout (quick exit): Ctrl+a d d

• kill a screen session: screen -XS session_name quit

Reconnecting to your screen session

In order to reconnect to a running screen session, you need to be connected to the same login node that you launched your screen session from. In order to locate and identify your screen sessions correctly, we recommed the following:

• Create a directory to store the information of your screen sessions. You only need do this once.

• Put the following line into your /.bashrc. This tells the screen program to save the information of your screen sessions in the directory created in the previous step . This allows you to query your screen sessions across different login nodes. To make this change effective in your current sessions, you need run 'source /.bashrc' in each of your current session . However, you do not need to run 'source /bashrc' in your new sessions.

• Name your new screen session using the name of the login node. For instance, start your screen with a commnd similar to

Setting up VASP

In order to use VASP, you must be a part of the vasp group on Oscar. To check your groups, run the groups command in the terminal.

First, you must choose which VASP module to load. You can see the available modules using module avail vasp. You can load your preferred VASP module using module load <module-name>.

Available Versions

• VASP 5.4.1

• VASP 5.4.4

• VASP 6.1.1

Running VASP

Within a batch job, you should specify the number of MPI tasks as

If you would like 40 cores for your calculation, you would include the following in your batch script:

Compiling with CUDA

To compile a CUDA program on Oscar, first load the CUDA module with:

$ module load cuda

The CUDA compiler is called nvcc, and for compiling a simple CUDA program it uses syntax simlar to gcc:

Optimizations for Fermi

The Oscar GPU nodes feature NVIDIA M2050 cards with the Fermi architecture, which supports CUDA's "compute capability" 2.0. To fully utilize the hardware optimizations available in this architecture, add the -arch=sm_20 flag to your compile line:

This means that the resulting executable will not be backwards-compatible with earlier GPU architectures, but this should not be a problem since CCV nodes only use the M2050.

Memory caching

The Fermi architecture has two levels of memory cache similar to the L1 and L2 caches of a CPU. The 768KB L2 cache is shared by all multiprocessors, while the L1 cache by default uses only 16KB of the available 64KB shared memory on each multiprocessor.

You can increase the amount of L1 cache to 48KB at compile time by adding the flags -Xptxas -dlcm=ca to your compile line:

If your kernel primarily accesses global memory and uses less than 16KB of shared memory, you may see a benefit by increasing the L1 cache size.

If your kernel has a simple memory access pattern, you may have better results by explicitly caching global memory into shared memory from within your kernel. You can turn off the L1 cache using the flags –Xptxas –dlcm=cg.

To start an interactive session for running serial or threaded programs on an Oscar compute node, simply run the command interact from the login node:

interact

By default, this will create an interactive session that reserves 1 core and 4GB of memory for a period of 30 minutes. You can change the resources reserved for the session from these default limits by modifying the interact command:

usage: interact [-n cores] [-t walltime] [-m memory] [-q queue]
                [-o outfile] [-X] [-f featurelist] [-h hostname] [-g ngpus]

Starts an interactive job by wrapping the SLURM 'salloc' and 'srun' commands.

options:
  -n cores        (default: 1)
  -t walltime     as hh:mm:ss (default: 30:00)
  -m memory       as #[k|m|g] (default: 4g)
  -q queue        (default: 'batch')
  -o outfile      save a copy of the sessions output to outfile (default: off)
  -X              enable X forwarding (default: no)
  -f featurelist  CCV-defined node features (e.g., 'e5-2600'),
                  combined with '&' and '|' (default: none)
  -h hostname     only run on the specific node 'hostname'
                  (default: none, use any available node)
  -a account      user SLURM accounting account name
  -g ngpus        number of GPUs   

For example, the command

$ interact -n 20 -t 01:00:00 -m 10g

requests an interactive session with 20 cores and 10 GB of memory (per node) for a period of 1 hour.

We have provided templates for you to use for job submission settings. These templates are in/gpfs/runtime/opt/forge/19.1.2/templates

Click Run and debug a program to open the following menu

Click Configure next to Submit to Queue and enter /gpfs/runtime/opt/forge/19.1.2/templates/slurm-ccv.qtf as the Submission template file

Here is an example script for running dependent jobs on Oscar.

#!/bin/bash

# first job - no dependencies
jobID_1=$(sbatch  job1.sh | cut -f 4 -d' ')

# second job - depends on job1
jobID_2=$(sbatch --dependency=afterok:$jobID_1 job2.sh | cut -f 4 -d' ')

# third job - depends on job2
sbatch  --dependency=afterany:$jobID_2  job3.sh

There are 3 batch jobs. Each job has it's own batch script: job1.sh, job2,sh, jobs.sh. The script above (script.sh) submits the three jobs.

line 4: job1 is submitted.

line 7: job2 depends on job1 finishing successfully.

line 10: job3 depends on job2 finishing successfully.

For details on the types of dependencies you can use in slurm see the manual page.

Configuring Remote Launch from the client

You will need to configure remote launch for Oscar

1. Open the client on your machine

2. Click 'Remote Launch' -> Configure

3. Add [email protected] as the Host Name

4. Add /gpfs/runtime/opt/forge/19.1.2 as the Remote Installation Directory

5. Test Remote Launch. You should enter the password used for Oscar. If successful you should see the message Remote Launch test completed successfully

Nightly snaphots of the file system are available for the last 30 days.

Restore a file from a snapshot in the last 30 days

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. An Oscar account is required to access Open OnDemand. Visit this link in a web browser and sign in with your Brown username and password to access this portal.

Agent Forwarding with PuTTY

1. Once adding your private key to Pageant, open PuTTY and navigate to the Auth menu.

2. Check the 'Allow agent forwarding' checkbox, and return to the Session menu.

Oscar has two Grace Hopper GH200 GPU nodes. Each node combines and .

Hardware Specifications

Each GH200 node has 72 Arm cores with 550G memory. Multiple-Install GPU (MIG) is enabled on only one GH200 node that has 4 MIGs. The other GH200 node doesn't have MIGs and only one GPU. Both CPU and GPU threads on GH200 nodes can now .

Listing running and queued jobs

The squeue command will list all jobs scheduled in the cluster. We have also written wrappers for squeue on Oscar that you may find more convenient:

To get started with NCDU, load the module using the following command:

Once the module has been loaded, it can be used to easily show the size of all files within a directory:

The line above uses Ncdu to rank all of the files within the my_directory directory. Your window should change to show a loading screen (if the directory doesn't have a lot in it, you may not even see this screen):

Oscar has two H100 nodes. is based on the that accelerates the training of AI models. The two DGX nodes provides better performance when multiple GPUS are used, in particular with Nvidia software like .

Hardware Specifications

Installing R packages

Users should install R packages for themselves locally. This documentation shows you how to install R packages locally (without root access) on Oscar.

If the package you want to install has operating-system-level dependencies (i.e. the package depends on core libraries), then we can install it as a module.

Software that makes it easy

If you are using Windows, you can use any of the following options to open a terminal on your machine (ranked in order of least difficult to set up and use):

Open OnDemand offers a browser-based terminal app to access Oscar. Windows users who do not want to install an SSH client like Putty will find this app very useful.

Accessing the terminal

1. Log in to

In this example, we will install Jax.

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

Here, -f = feature. We only need to build on Ampere once.

Step 2: Once your session has started on a compute node, run nvidia-smi to verify the GPU and then load the appropriate modules

Step 3: Create and activate the virtual environment

Step 4: Install the required packages

Step 5: Test that JAX is able to detect GPUs

A job array is a collection of jobs that all run the same program, but on different values of a parameter. It is very useful for running parameter sweeps, since you don't have to write a separate batch script for each parameter setting.

To use a job array, add the option:

in your batch script. The range can be a comma separated list of integers, along with ranges separated by a dash. For example:

A job will be submitted for each value in the range. The values in the range will be substituted for the variable $SLURM_ARRAY_TASK_ID in the remainder of the script. Here is an example of a script for running a serial Matlab script on 16 different parameters by submitting 16 different jobs as an array:

You can then submit the multiple jobs using a single sbatch command:

The $SLURM_ARRAY_TASK_ID can be manipulated as needed. For example, you can generate a fixed length number form it. The following example generates a number of length of 3 from

In this example, we will install PyTorch (refer to sub-pages for TensorFlow and Jax).

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

interact -q gpu -g 1 -f ampere -m 20g -n 4

We recommend all users to install Python packages within an environment. This can be a Conda to a python virtual environment. More information can be found here. Follow these steps to use such environments in the Jupyter app.

Python Environments:

One Time Setup:

1. Open a terminal on Oscar.

2. Load the relevant python module and create and/or activate the environment. See this page for more information about creating .

3. Run pip install notebook to install Jupyter notebook, if not already installed.

Launching Jupyter Notebook

1. Open the "Basic Jupyter Notebook for Python Environments" app on the Open OnDemand interface

2. Under "Python Module on Oscar", choose the python module you loaded when the environment was created.

3. Under "Python Virtual Environment", add the name of the Virtual Environment you created. Note: If your virtual environment is not at the top level of your home directory, you should input the absolute path to the environment directory.

Conda Environments

One Time Setup:

1. Open a terminal on Oscar.

2. Activate the conda environment.

3. Run pip install notebook to install Jupyter notebook, if not already installed.

4. Run pip install ipykernel

Launching Jupyter Notebook

1. Open the "Basic Jupyter Notebook with Anaconda" app on the Open OnDemand interface

2. Under "Oscar Anaconda module", choose "anaconda/2020.02"

3. Enter the name of the conda environment in "Conda Env"

4. Choose the other options as required.

Git Overview

Version Control refers to the management of changes made to source code or any such large amount of information in a robust manner by multiple collaborators. Git is by far the most popular version control system.

Git enables effective collaboration among developers. In a team setting, multiple developers often work on the same project simultaneously. With Git, each developer can work on their own local copy of the project, making changes and experimenting freely without affecting the main codebase. Git allows developers to merge their changes seamlessly, ensuring that modifications made by different individuals can be consolidated efficiently. It provides mechanisms to track who made specific changes, making it easier to understand the evolution of the project and identify potential issues.

Git Workflow

Nearly all operations that are performed by Git are in you local computing environment, for the exception of few used purely to synchronize with a remote. Some of the most common git operations are depicted below. In summary a typical flow consists of making changes to your files, staging them via git add, marking a save point via git commit, then finally syncing to your remote (e.g., GitHub) via git push. If you are pushing changes to your remote from multiple places, you can bring changes your most recent version using git pull, which is the equivalent of doing git fetch followed by a git merge operation

Cheatsheet

Below are some of the most commonly used Git commands. You can also get much more information by running git --help. And if you'd like to learn more there is an

Git Configuration

While using Git on Oscar, make sure that you to have your correct Name and Email ID to avoid confusion while working with remote repositories (e.g., GitHub, GitLab, BitBucket).

Getting Out of Trouble

Git can sometimes be a bit tricky. And we all eventually find ourselves in a place where we want to undo something or fix a mistake we made with Git. (pardon the profanity) has a bunch of really excellent solutions to common problems we sometimes run in to with Git.

1. Install the Remote Development extension pack for VSCode

2. Open VSCode settings

• On Windows/Linux - File > Preferences > Settings

• On macOS - Code > Preferences > Settings

Search for symlink and make sure the symlink searching is unchecked

3. Under VSCode settings, search for remote ssh timeout and manually enter a timeout value i.e. 50s. It should give you enough time to complete 2-Factor Authentication.

4. Edit the ~/.ssh/config file on your local machine, add the following lines. Replace <username> with your Oscar username.

6. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select ccv-vscode-node

1. When prompted in VSCode, please enter your Brown password and complete the DUO authentication. After that, wait about 30 seconds and VSCode should connect to Oscar.

Start the SSH-Agent

First, start your ssh-agent with the command below.

$ eval $(ssh-agent)

You should see an output similar to this:

Add Key(s)

Next, add your ssh private keys to the running agent (using the ssh-add command on line 1). This step may be repeated for every key pair you use to connect to different git servers. For most, this file is called id_rsa and will live in ~/.ssh/id_rsa. If you set a password for your ssh keys, the agent will prompt you to enter them.

Confirm the ssh keys have been loaded into the agent with ssh-add -L:

Connect to Oscar

Now ssh into Oscar with the -A option as shown on the first line below (replace username with your Oscar username). -A will forward your ssh-agent to Oscar, enabling you to use the ssh keys on your laptop while logged into Oscar.

If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Always connecting with Agent Forwarding

To make these changes permanent, you can add the ForwardAgent yes option to your ssh configuration file. To learn more about configuring your ssh connections, visit

Key Generation & Setup

1. Open PuTTYgen (this comes as part of the PuTTY package), change the 'Number of bits in a generated key:' to 4096 (recommended), then click 'Generate'

2. Move your cursor around randomly in order to "salt" your key, while the key is being generated. Once the key is generated, you should see something like this:

3. Replace the text in the 'Key comment:' field with something recognizable and enter a passphrase in the two fields below.

4. Copy the text in the 'Public key for pasting...' field (the text continues past what is displayed) and paste it wherever the public key is needed. If you are using GitHub, you can now create a new SSH key in your Personal Settings and paste this text into the 'Key' field.

5. Click on 'Save private key' and select a logical/recognizable name and directory for the file. Your private key is saved in the selected file.

6. Open Pageant (also part of the PuTTY package). If a message saying "Pageant is already running" is displayed, open your system tray and double click on the Pageant icon.

7. Click on 'Add Key' and select the file you saved when generating your key earlier (Step 5). If it is requested, enter the passphrase you created at Step 3 to complete the process.

The new Ampere architecture GPUs do not support older CUDA modules. Users must re-compile their applications with the newer CUDA/11 or older modules. Here are detailed instructions to compile major frameworks such as PyTorch, and TensorFlow.

PyTorch

Users can install PyTorch from a pip virtual environment or use pre-built singularity containers provided by Nvidia NGC.

To install via virtual environment:

To use NGC containers via Singularity :

• Pull the image from NGC

• Export PATHs to mount the Oscar file system

• To use the image interactively

• To submit batch jobs

Checkquota

Use the command checkquota to view your current disk usage and quotas. Here's an example output of this command

Each line represents a top level directory that you have access to.

Each column represents a usage or quota for these directories.

Types of Quota:

Disk usage and quotas are calculated separately for . Two types of quotas are calculated for each of these directories:

Disk space usage

This usage is expressed in Gigabytes (G) or Terabytes (T) . This is the total size of all the files in that directory and it does not depend upon the number of files. Run the command checkquota to see your disk usage and quota. Here's an example:

Inode usage

This is the total number of files and directories in the particular directory. This number does not depend upon the size of the files. Run the command checkquota to see your inode usage and quota. Here's an example:

Soft Limits vs Hard Limits

All quotas have a soft limit (SLimit) and hard limit (HLimit). When usage exceeds the soft limit, a grace period associated with this limit begins. During the grace period, the usage is allowed to increase up to the hard limit. When the usage reaches the hard limit or when the grace period expires, the user is not allowed to write any files to that particular directory.

Usage State

The "Usage State" column shows the status of the grace period for a particular directory. Here are some of the status messages:

SOFT_EXCEEDED

This indicates that your usage of the disk space or inodes has exceeded the soft limit and you are still within the grace period. Check the Grace_Period column to see the number of days left in the grace period. You may continue writing data into this directory until the end of the grace period, as long as you do not exceed the hard limit

GRACE_EXPIRED

This indicates that your usage has exceeded the soft limit AND the grace period has expired. You will not be able to write data into that directory, but you can remove files.

HARD_EXCEEDED

This indicates that your usage has reached the hard limit. You will not be able to write data into that directory, but you can remove data.

OK

This indicates that your usage of the disk space as well as inodes in within the soft quota.

Introduction to CUDA

CUDA is an extension of the C language, as well as a runtime library, to facilitate general-purpose programming of NVIDIA GPUs. If you already program in C, you will probably find the syntax of CUDA programs familiar. If you are more comfortable with C++, you may consider instead using the higher-level Thrust library, which resembles the Standard Template Library and is included with CUDA.

In either case, you will probably find that because of the differences between GPU and CPU architectures, there are several new concepts you will encounter that do not arise when programming serial or threaded programs for CPUs. These are mainly to do with how CUDA uses threads and how memory is arranged on the GPU, both described in more detail below.

There are several useful documents from NVIDIA that you will want to consult as you become more proficient with CUDA:

There are also many CUDA tutorials available online:

• from NVIDIA

• from The Supercomputing Blog

Threads in CUDA

CUDA uses a data-parallel programming model, which allows you to program at the level of what operations an individual thread performs on the data that it owns. This model works best for problems that can be expressed as a few operations that all threads apply in parallel to an array of data. CUDA allows you to define a thread-level function, then execute this function by mapping threads to the elements of your data array.

A thread-level function in CUDA is called a kernel. To launch a kernel on the GPU, you must specify a grid, and a decomposition of the grid into smaller thread blocks. A thread block usually has around 32 to 512 threads, and the grid may have many thread blocks totalling thousands of threads. The GPU uses this high thread count to help it hide the latency of memory references, which can take 100s of clock cycles.

Conceptually, it can be useful to map the grid onto the data you are processing in some meaningful way. For instance, if you have a 2D image, you can create a 2D grid where each thread in the grid corresponds to a pixel in the image. For example, you may have a 512x512 pixel image, on which you impose a grid of 512x512 threads that are subdivided into thread blocks with 8x8 threads each, for a total of 64x64 thread blocks. If your data does not allow for a clean mapping like this, you can always use a flat 1D array for the grid.

The CUDA runtime dynamically schedules the thread blocks to run on the multiprocessors of the GPU. The M2050 GPUs available on Oscar each have 14 multiprocessors. By adjusting the size of the thread block, you can control how much work is done concurrently on each multiprocessor.

Memory on the GPU

The GPU has a separate memory subsystem from the CPU. The M2050 GPUs have GDDR5 memory, which is a higher bandwidth memory than the DDR2 or DDR3 memory used by the CPU. The M2050 can deliver a peak memory bandwidth of almost 150 GB/sec, while a multi-core Nehalem CPU is limited to more like 25 GB/sec.

The trade-off is that there is usually less memory available on a GPU. For instance, on the Oscar GPU nodes, each M2050 has only 3 GB of memory shared by 14 multiprocessors (219 MB per multiprocessor), while the dual quad-core Nehalem CPUs have 24 GB shared by 8 cores (3 GB per core).

Another bottleneck is transferring data between the GPU and CPU, which happens over the PCI Express bus. For a CUDA program that must process a large dataset residing in CPU memory, it may take longer to transfer that data to the GPU than to perform the actual computation. The GPU offers the largest benefit over the CPU for programs where the input data is small, or there is a large amount of computation relative to the size of the input data.

CUDA kernels can access memory from three different locations with very different latencies: global GDDR5 memory (100s of cycles), shared memory (1-2 cycles), and constant memory (1 cycle). Global memory is available to all threads across all thread blocks, and can be transferred to and from CPU memory. Shared memory can only be shared by threads within a thread block and is only accessible on the GPU. Constant memory is accessible to all threads and the CPU, but is limited in size (64KB).

Step 1 : Check for existing SSH key pair

Before generating new SSH key pair first check if you have an SSH key on your local machine.

If there are existing keys, please move to Step 3

Step 2 : Generate a new SSH Keypair

Press Enter to accept the default file location and file name.

The ssh-keygen will ask you to type a secure passphrase. This is optional. If you don't want to use a passphrase just press Enter

Verify the SSH keys are generated correctly, you should see two files id_rsa and id_rsa.pub under ~/.ssh directory.

Step 3 : Copy the public key to Oscar

You will now need to copy your public key to Oscar. There are two ways to acomplish this.

With ssh-copy-id

If your OS comes with the ssh-copy-id utility, then you'll be able to copy your public key into Oscar as follows:

You will be prompted for a Password. The public key will be appended to the authorized_keys file on Oscar.

If you used a custom name for your key instead of the default id_rsa then you'll need pass the name of your key to ssh-copy-id i.e.,

Without ssh-copy-id

If your system does not come with the ssh-copy-id utility installed, then you'll need to copy your public key by hand.

1. Get the contents of id_rsa.pub file. One option is to use cat in your teminal cat id_rsa.pub.

2. Copy the contents of this file to your clipboard, as we need to upload it to Oscar.

3. Login into Oscar via regular ssh ssh <username>@ssh.ccv.brown.edu

Step 4 : Login to Oscar using your SSH keys

If everything went well, you will be logged in immediately withouth prompting you for a password.

Apptainer Using NGC Containers (Our #1 Recommendation)

There are multiple ways to install and run TensorFlow. Our recommended approach is via NGC containers. The containers are available via NGC Registry. In this example we will pull TensorFlow NGC container

1. Build the container:

This will take some time, and once it completes you should see a .simg file.

1. Once the container is ready, request an interactive session with a GPU

1. Run a container wih GPU support

1. Or, if you're executing a specific command inside the container:

1. Make sure your Tensorflow image is able to detect GPUs

1. If you need to install more custom packages, the containers itself are non-writable but we can use the --user flag to install packages inside .local Example:

Slurm Script:

Here is how you can submit a SLURM job script by using the srun command to run your container. Here is a basic example:

Setting Up Gaussian

In order to use Gaussian on Oscar, you must be a part of the ccv-g09 group. To check your groups, run the groups command in the terminal.

You must first choose a Gaussian module to load. To see available Gaussian modules, run module avail gauss. You can load a Gaussian module using the command module load <module-name>.

Available Versions

• Gaussian 9 (g09)

• Gaussian 16 (g16)

NOTE: There are three versions of g09, you can load any one of those, but the newer version g16 is preferred now. If using g09 just replace g16 below with g09.

Running Gaussian

Gaussian can be run either interactively or within a batch script using one of two command styles:

• g16 job-name

• g16 <input-file >output-file

In the first form, the program reads input from job-name.gjf and writes its output to job-name.log. When no job-name has been specified, the program will read from standard input and write to standard output

Given a valid .gjf file (we'll call it test-file.gjf), we can use the following simple batch script to run Gaussian:

g16-test.sh

Then queue the script using

Once the job has been completed, you should have a g16-test.out, a g16-test.err, and a test-file.out.

Arm Forge is available on Oscar. There are two products, DDT (debugger) and MAP (performance reports).

We recommend you use the Arm Forge remote client to launch your debugging jobs on Oscar. The first time you set up Arm Forge you will need to configure the client with the following steps:

1. Download the arm forge remote client on your machine.

2. Configuring Remote Launch from the client

Arm DDT

Arm DDT is a powerful graphical debugger suitable for many different development environments, including:

• Single process and multithreaded software.

• OpenMP.

• Parallel (MPI) software.

Arm MAP

Arm MAP is a parallel profiler that shows you which lines of code took the most time to run, and why. Arm MAP does not require any complicated configuration, and you do not need to have experience with profiling tools to use it.

Arm MAP supports:

• MPI, OpenMP and single-threaded programs.

• Small data files. All data is aggregated on the cluster and only a few megabytes written to disk, regardless of the size or duration of the run.

• Sophisticated source code view, enabling you to analyze performance across individual functions.

• Both interactive and batch modes for gathering profile data.

Setting Up IDL

First load the IDL module that you want to use with module load idl/version_number:

As indicated by the system dialogue, you will need to enter the following command to set up the environment for IDL:

IDL Command Line

Once you've set up IDL in the way outlined above, you can open the IDL command line by simply using the command idl:

As is stated in the , IDL in command-line mode "uses a text-only interface and sends output to your terminal screen or shell window." Thus, this is a mode in which you can enter commands and see their results in real time, but it is not where one should write full IDL programs.

IDL Programs

To write an IDL program, you can use any of the text editors on Oscar (such as vim, emacs, and nano) or you can create the program in a file on your own computer and then copy that file to Oscar when you are finished. Here is an example (hello world) IDL program idl_hello_world.pro:

Once you have the .pro file on Oscar, you can then run this file using a batch script. Here is a bare bones version of a batch script (called idl_hello_world.sh)that will run the script idl_hello_world.pro (note that the .pro is omitted in the script).

We can then run the batch file by using the sbatch command:

To log in to Oscar you need Secure Shell (SSH) on your computer.

There are two options for signing into Oscar: with or without VPN.

When regularly connecting to multiple remote systems over SSH, you’ll find that remembering all the hosts and various command-line options becomes tedious. OpenSSH allows setting up a configuration file to store different SSH options for each remote machine you connect t.

SSH Config File Location

OpenSSH client-side (in this case your personal computer) configuration file is named config, and it is stored in the hidded .sshdirectory under your user’s home directory (i.e., ~/.ssh)

CCV uses thepackage for managing the software environment on OSCAR. The advantage of the modules approach is that it allows multiple versions of the same software to be installed at the same time. With the modules approach, you can "load'' and "unload'' modules to dynamically control your environment.

Combining CUDA and MPI

Mixing MPI (C) and CUDA (C++) code requires some care during linking because of differences between the C and C++ calling conventions and runtimes. One option is to compile and link all source files with a C++ compiler, which will enforce additional restrictions on C code. Alternatively, if you wish to compile your MPI/C code with a C compiler and call CUDA kernels from within an MPI task, you can wrap the appropriate CUDA-compiled functions with the extern keyword, as in the following example.

These two source files can be compiled and linked with both a C and C++ compiler into a single executable on Oscar using:

The CUDA/C++ compiler nvcc is used only to compile the CUDA source file, and the MPI C compiler mpicc

Open-source software refers to any program whose source code is available for use or modification as users or other developers see fit. This is usually developed as a public collaboration and made freely available.

CMake and CCMake

Due to the complexity of some software, we often have to link to third party or external libraries. When working with software that has complicated building and linking steps, it is often impractical to use GCC (or your favorite compiler) directly. GNU Make is a build system that can simplify things somewhat, but "makefiles" can become unwieldy in their own way. Thankfully for us, there is a tool that simplifies this process.

CMake is a build system generator that one can use to facilitate the software build process. CMake allows one to specify—at a higher level than GNU Make—the instructions for compiling and linking our software. Additionally, CMake comes packaged with CCMake, which is an easy-to-use interactive tool that will let us provide build instructions to the compiler and the linker for projects written in C, Fortran, or C++. For more information about CMake and CCMake, please click

This page serves as a guide for application developers getting started with parallel programming, or users wanting to know more about the working of parallel programs/software they are using.

Although there are several ways to classify parallel programming models, a basic classification is:

2. Distributed Memory Programming

To use your condo account to submit jobs, please follow the steps below to check the association of your Oscar account and include condo information in your batch script or command line.

Step 1 - Check your account associations to find your condo Account and Partition information by running the following command:

In the example below, the user has access to two condos, where their

Using MPI4PY in a Python Script

To use MPI in a python script through

Partition Overview

Oscar has the following partitions. The number and size of jobs allowed on Oscar vary with both partition and type of user account. You can email [email protected] if you need advice on which partitions to use‌.

To list partitions on Oscar available to your account, run the following command:

To view all partitions (including ones you don't have access to), replace the -O in the command above with -aO.

What username and password should I be using?

• If you are at Brown and have requested a regular CCV account, your Oscar login can be authenticated using your Brown credentials itself, i.e. the same username and password that you use to login to any Brown service such as "canvas".

• If you are an external user, you will have to get a sponsored ID at Brown through the department with which you are associated, before requesting an account on Oscar. Once you have the sponsored ID at Brown, you can on Oscar and use your Brown username and password to login.

Changing Passwords

Oscar users should use their Brown passwords to log into Oscar. Users should change their Brown passwords at .

Exploratory Account

• Exploratory accounts are available to all members of the Brown community for free.

• See thefor detailed description of the resources

• Jobs are submitted to the batch partition. See the page for available hardware

Priority Accounts

The following accounts are billed quarterly and offer more computational resources than the exploratory accounts. See thefor pricing and detailed description of the resources

HPC Priority

• Intended for users running CPU-intensive jobs. These offer more CPU and memory resources than an exploratory account

• Two types of accounts:

  • HPC Priority

Standard GPU Priority

• Intended for users running GPU intensive jobs. These accounts offer fewer CPU and memory resources but more GPU resources than an exploratory account.

• Two types of accounts:

  • Standard GPU Priority

High End GPU Priority

• Intended for GPU jobs required high-end gpus. These offer the same number of CPUS as Standard GPU priority accounts

• High end GPUS like A40, v100 and a6000 are available

• See thefor pricing and detailed description of the resources

• Jobs are submitted to the

Large Memory Priority

• Intended for jobs requiring large amounts of memory.

• These accounts offer 2TB of memory and twice the wall-time of exploratory accounts.

• See thefor pricing and detailed description of the resources

Condo

PIs who purchase hardware (compute nodes) for the CCV machine get a Condo account. Condo account users have the highest priority on the number of cores equivalent to the hardware they purchased. Condo accounts last for five years and give their owners access to 25% more CPU cores than they purchase for the first three years of their lifespan. GPU resources do not decrease over the lifetime of the condo.

Investigators may also purchase condos to grant access to computing resources for others working with them. After a condo is purchased, they can have users request to join the condo group through the "Request Access to Existing Condo" option on the on the CCV homepage.

Anaconda / Conda

A distribution of Python and R used for scientific computing that is meant to simplify package management and deployment. Conda is used for installing packages and managing their dependencies. [Related Page - Anaconda]

Association

Within Oscar, an association refers to a combination of four factors: Cluster, Account, User, and Partition. Associations are used to control job submissions for users. []

Batch Jobs

Put simply, batch jobs are scheduled programs that are assigned to run on a computer without further user interaction. []

CCV

Brown University's Center for Computation and Visualization. Provides software, expertise, and other services for Brown's research community. See for more information.

CESM

Stands for Community Earth System Model. "CESM is a fully-coupled, community, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states." () []

Condo

PIs can purchase condos that have a significant amount of computing resources which can be shared with others. []

CUDA

" is an extension of the C language, as well as a runtime library, to facilitate general-purpose programming of NVIDIA GPUs." () []

Desktop App

This app on Open OnDemand allows users to launch a Desktop GUI on Oscar. This app is based on VNC which is a desktop sharing system that allows you to remotely control another desktop.[]

HPC

Stands for High Performance Computing. HPC is the ability to process data and perform highly complex calculations at an accelerated rate. Oscar is the service that CCV offers to the Brown community for their High Performance Computing needs.

Job Array

A job array is a collection of jobs that all run the same program but on different values of a parameter. []

Jupyter Notebook

"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text." []

Interactive Jobs

Jobs that allow the user to interact in real time with applications within Oscar, often from the command line. This differs from batch jobs in that each command to be run must be put in one at a time. []

Modules

Modules are software components that can easily be loaded or unloaded into Oscar. For instance, a user can load the Python 3 module using a module load command. []

MPI

Stands for . MPI is a system that aims to be the standard for portable and efficient message passing. is a technique often used in object-oriented programming and parallel programming []

Open OnDemand (OOD)

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. It can be used to launch a Desktop session on Oscar []

OOD app

OOD app is a web application that runs on the Open OnDemand web portal. It allows users to launch interactive applications like Jupyter Notebook, RStudio, Matlab or Desktop. []

Partition

Partitions are essentially groupings of nodes that allocate resources for specific types of tasks. On Oscar, partitions are based on job submissions through the Slurm workload manager. []

PI

Stands for Principal Investigator. Mainly used to refer to the individual responsible for conducting and administrating a research grant. Within Oscar, PIs have their own data directories that can be shared to students. PIs may also purchase condos. []

PuTTY

A client for SSH for Windows and Unix that emulates a terminal []

Python

An object-oriented, high-level, and popular programming language []

Quality of Service (QOS)

The job limits that are linked to a given association. For instance, Priority Accounts will generally have a higher quality of service than Exploratory Accounts. []

Slurm

A workload manager used within Oscar to schedule jobs []

SSH

Stands for Secure Shell Protocol. Used to communicate securely between computers and often used within a command-line interface (CLI) for connections to remote servers []

SMB

The Server Message Block (SMB) protocol is a network protocol that allows users to communicate with remote computers for file-sharing and other uses. It is one of the versions of the Common Internet File System (CIFS). Within Oscar, SMB is mainly used for file transfer. []

The filesystem on Oscar can be accessed through the file explorer on this web portal. The file explorer allows you

• List files

• Create a directory

• Rename files

• Copy/Move files

To access the file explorer, click "Files" -> "Home Directory" at the top of the screen.

Check the documentation below for some of these services below:

Changing directories on File explorer

To access a directory, click "Change directory" and enter the path name

• To access your home directory, click the "Home Directory" link on the left. The path name at the top of the page should change to "/users/<username>"

• To access your scratch directory, click the "scratch" directory in your home directory OR click "Change directory" and enter "/users/<username>/scratch"

• To access your data directory, click the "data" directory in your home directory OR click "Change directory" and enter "/users/<username>/data"

Edit plain-text files

1. that contains the plain-text file.

2. Click the icon with the three dots -> Edit

3. The file will open in a text editor in a new tab

Download files or directories

1. that contains the file or directory.

2. Click the icon with the three dots -> Download

To download multiples files:

1. Click the check-box to the left of the file name.

2. Scroll to the top of the page and click "Download"

Directories are downloaded as zipped files on your computer.

Upload files or directories

1. where you need to upload the files.

2. Click the "Upload" button.

3. Follow the instructions on the screen. You can click the "Browse" buttons or drag and drop files.

Launch a terminal

1. where you would like to open the terminal

2. Click "Open in Terminal" at the top of the page.

3. A web-based terminal will open in a new tab of your browser. You will be logged into one of the login nodes.

Associations

Oscar uses associations to control job submissions from users. An association refers to a combination of four factors: Cluster, Account, User, and Partition. For a user to submit jobs to a partition, an association for the user and partition is required in Oscar.

To view a table of association data for a specific user (thegrouch in the example), enter the following command in Oscar: