These pages list some command commands and terms you will come across while using Oscar.

Common Acronyms and Terms

Managing Modules

Common Linux Commands

module list

Lists all modules that are currently loaded in your software environment.

module avail

Lists all available modules on the system. Note that a module can have multiple versions. Use module avail <name> to list available modules which start with <name>

module help <name>

Prints additional information about the given software.

module load <name>

Adds a module to your current environment. If you load using just the name of a module, you will get the default version. To load a specific version, load the module using its full name with the version: "module load gcc/10.2"

module unload <name>

Removes a module from your current environment.

Connect to OSCAR

Alternatively, you can connect to OSCAR via SSH (Terminal):

ssh <username>@ssh.ccv.brown.edu

Submit a Job

You can submit a job using sbatch:

sbatch batch_scripts/hello.sh

You can confirm that your job ran successfully by running:

cat hello-*.out

Transfer Files

Get Help

If you encounter problems while using Oscar, check out the Getting Help documentation, or read through the Overview page.

Accounts

If you do not have an Oscar account, you can request one by clicking the following link:

Anyone with a Brown account can get a free Exploratory account on Oscar, or pay for priority accounts.

Authorized users must comply with the following Brown University policies:

Hardware

Users can run their computing-intensive and/or long runtime jobs/program in Oscar to take advantage of high performance computing resources there, as highlighted below:

• 2 Login nodes

• 8 PB of storage

• Red Hat Enterprise Linux 9.2 (Linux)

• Mellanox InfiniBand network

• Slurm Workload manager

Scheduler

Users should not run computations or simulations on the login nodes, because they are shared with other users. You can use the login nodes to compile your codes, manage files, and launch jobs on the compute nodes.

To allow users sharing access to Oscar, there are limits on the maximum number of pending and running jobs a user account may have/submit:

• 1200 for a priority account

• 1000 for an exploratory account

Software

• Operating systems of all Oscar nodes: Red Hat 9.2

• CCV Staff install software upon user requests or help users on software installation

Storage

Access and User Accounts - User accounts are controlled via central authentication and directories on Oscar are only deleted on the request of the user, PI, or departmental chair.

Files not accessed for 30 days will be deleted from your ~/scratch directory. Use ~/data for files you wish to keep long term.

Connecting to Oscar

Oscar users can connect to Oscar by

Maintenance Schedule

• Non-disruptive Maintenance:

  • non-disruptive work, including software changes, maintenance, and testing

  • may occur at any time

  • no notification provided

• Monthly Scheduled Maintenance:

  • no downtime expected, but there may be limited degradation of performance

  • first Tuesday of the month, 8:00 am - 12:00 noon

  • no notification provided

• Unscheduled Maintenance:

  • maximum 1 day downtime

  • occurs very rarely and includes any unplanned emergency issues that arise

  • Prior notification provided (depending on the issue, 1 day to 4 weeks advance notice provided)

• Major Upgrade Maintenance:

  • service may be brought down for 3-5 days

  • occurs annually

  • 4-week prior notification provided

Unplanned Outage

• During Business Hours:
• During Non-Business Hours:

  • Call CIS Operations Center at (401) 863-7562. A ticket will get created and CCV staff will be contacted to address the issue.

User and Research Support

CCV staff support for researchers seeking help with statistical modeling, machine learning, data mining, data visualization, computational biology, high-performance computing, and software engineering.

CCV provides short videos (coming soon) for users to learn as well.

What username and password should I be using?

• If you are at Brown and have requested a regular CCV account, your Oscar login can be authenticated using your Brown credentials itself, i.e. the same username and password that you use to login to any Brown service such as "canvas".

Changing Passwords

Exploratory Account

• Exploratory accounts are available to all members of the Brown community for free.

Priority Accounts

HPC Priority

• Intended for users running CPU-intensive jobs. These offer more CPU and memory resources than an exploratory account

• Two types of accounts:

  • HPC Priority

  • HPC Priority+ (Twice the resources of HPC Priority)

Standard GPU Priority

• Intended for users running GPU intensive jobs. These accounts offer fewer CPU and memory resources but more GPU resources than an exploratory account.

• Two types of accounts:

  • Standard GPU Priority

  • Standard GPU Priority+ (Twice the resources of Standard GPU Priority)

High End GPU Priority

• Intended for GPU jobs required high-end gpus. These offer the same number of CPUS as Standard GPU priority accounts

• High end GPUS like A40, v100 and a6000 are available

Large Memory Priority

• Intended for jobs requiring large amounts of memory.

• These accounts offer 2TB of memory and twice the wall-time of exploratory accounts.

Condo

PIs who purchase hardware (compute nodes) for the CCV machine get a Condo account. Condo account users have the highest priority on the number of cores equivalent to the hardware they purchased. Condo accounts last for five years and give their owners access to 25% more CPU cores than they purchase for the first three years of their lifespan. GPU resources do not decrease over the lifetime of the condo.

Anaconda / Conda

Association

Batch Jobs

CCV

CESM

Condo

CUDA

Desktop App

HPC

Stands for High Performance Computing. HPC is the ability to process data and perform highly complex calculations at an accelerated rate. Oscar is the service that CCV offers to the Brown community for their High Performance Computing needs.

Job Array

Jupyter Notebook

Interactive Jobs

Modules

MPI

Open OnDemand (OOD)

OOD app

Partition

PI

PuTTY

Python

Quality of Service (QOS)

Slurm

SSH

SMB

cd

Moves the user into the specified directory. Change Directory.

cd .. to move one directory up

cd by itself to move to home directory

cd - to move to previous directory

cd <directory-path> to move to a directory (can be an absolute path or relative path)

cp <old_filepath> <new directory path>

Copies the file into the specified directory

clear

Clears the terminal

cat <filename>

Lists the contents of a file. Concatenate files.

ls

List contents within the current directory

grep <string_to_match> <filename>

pwd

Displays the path of the current directory that you are in. Present Working Directory

man <command>

Displays the help manual instruction for the given command

mv <file_name> <new_directory>

Moves a file into a new directory.

mv <old_file_name> <new_file_name> to rename a file

mkdir <directory_name>

Creates a new directory

rm <file_name>

Deletes a file

rm -r <directory_name>

Deletes directories and the contents within them. -r stands for recursive

rmdir <directory_name>

Removes the specified directory (must be empty)

touch

Creates a blank new file

Oscar Specifications

Compute Nodes

Oscar has compute nodes in the partitions listed below.

• batch - The batch partition is for programs/jobs which need neither GPUs nor large memory.

• bigmem - The bigmem partition is for programs/jobs which require large memory.

• debug - The debug partition is for users to debug programs/jobs.

• gpu - The gpu partition is for programs/jobs which require GPUs.

• gpu-debug - The gpu-debug partition is for users to debug gpu programs/jobs.

• gpu-he -The gpu-he partition is for programs/jobs which need to access high-end GPUs.

• vnc - The vnc partition is for users to run programs/jobs in an graphical desktop environment.

Below are node details including cores and memory for all partitions.

Hardware details

Hardware details for all partitions. The Features column shows the features available for the --constraint option for SLURM. This includes the available CPU types as well GPUs.

GPU Features and GPU Memory

If you publish research that benefited from the use of CCV services or resources, we would greatly appreciate an acknowledgment that states:

This research [Part of this research] was conducted using [computational/visualization]
resources and services at the Center for Computation and Visualization, Brown University.

Account Usage

Oscar users are not permitted to:

• Share their accounts or passwords with others or enable unauthorized users to access Center for Computation and Visualization resources

• Use Center for Computation and Visualization resources for personal economic gain

• Engage in unauthorized activity (e.g., cryto currency mining etc.) that intentionally impacts integrity of resources

Storage

Each user (premium or exploratory) gets 20GB Home Directory, 512GB short-term Scratch, and 256G Data directory (shared amongst the members of group)

• Files in Scratch Directory not accessed for last 30 days are automatically purged. CCV only stores snapshots for 7 days after that files will be automatically deleted.

• PI has the ultimate access to Data Directory - if a student leaves Brown the files in Data directory will be owned by the PI.

Software and Data

All software and data stored or used on Center hosted systems must be appropriately and legally acquired and must be used in compliance with applicable licensing terms. Unauthorized misuse or copying of copyrighted materials is prohibited.

Data Retention

CCV reserves the right to remove any data at any time and/or transfer data or other individuals (such as Principal Investigators working on a same or similar project) after a user account is deleted is no longer affiliated with Brown University.

Accounts Validity

• Once created, Oscar accounts are valid for duration of one's Brown AD credentials

CCV provides access to HPC resources for classes, workshops, demonstrations, and other instructional uses. In general, the system is available for most types of instructional use at Brown where HPC resources are required, and we will do what we can to provide the resources necessary to help teach your class. We do ask that you follow these guidelines to help us better support your class.

Account Requests and Software Needs

Requests for class accounts should be made in writing to support@ccv.brown.edu two weeks prior to the beginning of class, and should be made in bulk. Please provide the Brown username (required), name and Brown Email address for the students, TAs and instructor as well as the course number and the semester. Requests for specific software should also be made two weeks before the start of the semester, and should be properly licensed, tested and verified to work by an instructor or TA.

Usage Expectations and System Utilization

Unless prior arrangements are made, student class accounts will have the same priority and access as free accounts on the CCV system. Access can be provided to specialized hardware or higher cores if needed provided it does not impact research use of the CCV systems. Be aware that usage of the CCV system is unpredictable, and high utilization of the system could impact a student's ability to finish assignments in a specific time period. We also encourage instructors to give an overview of the system and discuss computing policies before students use the system. CCV can provide resources (slides, documentation and in class workshops) to help prepare students to use HPC system. CCV staff are always available to meet directly with instructors and TAs to help prepare for classes and help setup specific software or environments for the class.

Support

It is expected that any class being taught using CCV resources will have its own TA. The TA should be the first line of support for any problems or questions the students may have regarding the use of the CCV system. CCV staff may not know specifics about how to use or run the programs the class is using, and can’t provide direct support to students for that software.

Class Guest Accounts

CCV will provide limited duration guests accounts that are custom tailored for the class use of the system. These accounts will have a username of “ccvws###”, and each account is associated with an individual student, instructor, or TA. Guest accounts are temporary and are only active for the duration of the class, and are deactivated at the conclusion of the semester/workshop. Account data is kept intact on our system for one semester after the conclusion of the class, and is then permanently deleted from the CCV system.

Filing a Support Ticket

Filing a good support ticket makes it much easier for CCV staff to deal with your request

When you email support@ccv.brown.edu aim to include the following:

• State the problem/request in the subject of the email

• Describe which software and with version you are using

• Error message (if there was one)

• The job number

• How you were running, e.g. batch, interactively, vnc

• Give as as small an example as possible that reproduces the problem

Q&A Forum

Slack

Office Hours

Arrange a Meeting

You can arrange to meet with a CCV staff member in person to go over difficult problems, or to discuss how best to use Oscar. Email support@ccv.brown.edu to arrange a consultation.

General

How do I request help?

Most inquiries can be directed to CCV’s support address, support@ccv.brown.edu, which will create a support ticket with one of our staff.

What are the fees for CCV services?

How do I acknowledge CCV in a research publication?

We greatly appreciate acknowledgements in research publications that benefited from the use of CCV services or resources.

Oscar

What is Oscar?

Oscar is our primary research computing cluster with several hundred multi-core nodes sharing a high-performance interconnect and file system. Applications can be run interactively or scheduled as batch jobs.

How do I request an account on Oscar?

How do I run a job on Oscar?

Can I use Oscar for teaching?

How do I find out when the system is down?

We post updates to our user mailing list, ccv@listserv.brown.edu which you are automatically subscribed to when setting up an account with CCV. If you need to be added to the mailing list, please submit a support ticket to support@ccv.brown.edu. We also have an announcement mailing list for office hours, workshops and other events relevant to CCV users, ccv-announce.listserve.brown.edu.

How do I run a job array on Oscar?

How do I run a MPI job on Oscar?

I have some MPI-enabled source code. How can I compile it on Oscar?

Load an mpi module module load mpi. For a list of mpi modules available, module avail mpi

What applications are available on Oscar?

What compilers are available on Oscar?

How do I get information about finished jobs?

The sacct command will list all of your completed jobs since midnight of the previous day (as well as running and queued jobs). You can pick an earlier start date with the -S option, e.g. sacct -S 2012-01-01.

How much storage am I using?

My job keeps terminating unexpectedly with a "Killed" message, or without any errors. What happened?

How do I request a certain amount of memory per CPU?

Specify the SLURM option --mem-per-cpu= in your script.

How do I link against a BLAS and LAPACK library?

I am getting a "WARNING: Remote HOST IDENTIFICATION HAS CHANGED?

We have recently updated the login and VSCode node hardware to improve performance, security, and reliability. As a result of this migration, the SSH host keys for our servers have been updated. To fix this:

• On MacOS:

sed -i '' -e '/^oscar/d' -e '/^vscode/d' ~/.ssh/known_hosts

• On Linux:

sed -i -e '/^oscar/d' -e '/^vscode/d' ~/.ssh/known_hosts

• On Windows: from VSCode's internal terminal Window:

vi ~/.ssh/known_hosts 

and delete the lines starting with Oscar and delete lines starting with vscode and oscarHopefully, this will make things easier.

• OpenOnDemand (OOD) Shell Access: either get a Desktop session or login via regular terminal into 'ssh.ccv.brown.edu' and run

sed -i -e '/^oscar/d' -e '/^vscode/d' ~/.ssh/known_hosts

Then login again via OOD > Clusters

RUNNING JOBS

How is a job identified?

By a unique JobID, e.g. 1318013

Which of my jobs are running/pending?

Use the command myq

How do I check the progress of my running job?

You can look at the output file. The default output file is slurm-%j.out" where %j is the JobID. If you specified and output file using #SBATCH -o output_filename and/or an error file #SBATCH -e error_filename you can check these files for any output from your job. You can view the contents of a text file using the program less , e.g.

less output_filename

Use the spacebar to move down the file, b to move back up the file, and q to quit.

My job is not running how I intended it to. How do I cancel the job?

scancel <JobID> where <JobID> is the job allocation number, e.g. 13180139

How do I save a copy of an interactive session?

You can use interact -o outfile to save a copy of the session's output to "outfile"

I've submitted a bunch of jobs. How do I tell which one is which? myq will list the running and pending jobs with their JobID and the name of the job. The name of the job is set in the batch script with #SBATCH -J jobname. For jobs that are in the queue (running or pending) you can use the command scontrol show job <JobID> where <JobID> is the job allocation number, e.g.13180139 to give you more detail about what was submitted.

How do I ask for a haswell node?

Use the --constraint (or -C) option:

#SBATCH --constraint=haswell

You can use the --constraint option restrict your allocation according to other features too. The nodes command provides a list of "features" for each type of node.

Why won't my job start?

When your job is pending (PD) in the queue, SLURM will display a reason why your job is pending. The table below shows some common reasons for which jobs are kept pending.

Why is my job taking so long to start? Just waiting in (Priority) or (Resources)

1. Overall system busy: when tens of thousands of jobs are submitted it total by all users, the time it takes SLURM to process these into the system may increase from the normal almost instantly to a half-hour or more.

2. Specific resource busy: if you request very specific resources (e.g., a specific processor) you then have to wait for that specific resource to become available while other similar resources may be going unused.

3. Specified resource not available: if you request something that is not or may never be available, your job will simply wait in the queue. E.g., requesting 64 GB of RAM on a 64 GB node will never run because the system needs at least 1 GB for itself so you should reduce your request to less than 64.

TRANSFERRING FILES

How do I transfer big files to/from Oscar?

Please use the server transfer.ccv.brown.edu

1. Transfer local file to Oscar:

sftp <username>@transfer.ccv.brown.edu 
put /path/local_file

2. Transfer remote file on Oscar to the local system:

sftp <username>@transfer.ccv.brown.edu 
get -r filename.txt 

Cloud HPC Options

The use of cloud resources for HPC varies according to your demands and circumstances. Cloud options are changing rapidly both in service providers and various services being offered. For those who have short-term needs that don't demand the highest of computational performance, a cloud option might be appropriate. For others, a local option customized to individual needs may be better. The cost of cloud services also varies quite a bit and includes not only compute time but data transfer charges. Other issues involved licensing, file synchronization, etc.

We are actively investigating a number of options to connect Brown users seamlessly to suitable cloud options. We are collecting such information for publishing on the CIS website as part of research services available. At this point, the best course of action is to request an individual consultation to help address your specific needs. Please send email to support@ccv. brown.edu.

When connecting from a campus network to sshcampus.ccv.brown.edu you can set up SSH keys as a form of authentication instead of having to enter your password interactively. Follow the insctructions below that correspond to your operating system/connection method.

Step 1 : Check for existing SSH key pair

Before generating new SSH key pair first check if you have an SSH key on your local machine.

If there are existing keys, please move to Step 3

Step 2 : Generate a new SSH Keypair

Press Enter to accept the default file location and file name.

The ssh-keygen will ask you to type a secure passphrase. This is optional. If you don't want to use a passphrase just press Enter

Verify the SSH keys are generated correctly, you should see two files id_rsa and id_rsa.pub under ~/.ssh directory.

Step 3 : Copy the public key to Oscar

You will now need to copy your public key to Oscar. There are two ways to acomplish this.

With ssh-copy-id

If your OS comes with the ssh-copy-id utility, then you'll be able to copy your public key into Oscar as follows:

You will be prompted for a Password. The public key will be appended to the authorized_keys file on Oscar.

If you used a custom name for your key instead of the default id_rsa then you'll need pass the name of your key to ssh-copy-id i.e.,

Without ssh-copy-id

If your system does not come with the ssh-copy-id utility installed, then you'll need to copy your public key by hand.

1. Get the contents of id_rsa.pub file. One option is to use cat in your teminal cat id_rsa.pub.

2. Copy the contents of this file to your clipboard, as we need to upload it to Oscar.

3. Login into Oscar via regular ssh ssh <username>@ssh.ccv.brown.edu. Once you are on the login node, open the authorized_keys file with your text editor of choice e.g., vim ~/.ssh/authorized_keys or nano ~/.ssh/authorized_keys Add your public keys to end of this file. Save and exit.

Step 4 : Login to Oscar using your SSH keys

If everything went well, you will be logged in immediately withouth prompting you for a password.

Key Generation & Setup

2. Move your cursor around randomly in order to "salt" your key, while the key is being generated. Once the key is generated, you should see something like this:

3. Replace the text in the 'Key comment:' field with something recognizable and enter a passphrase in the two fields below.

4. Copy the text in the 'Public key for pasting...' field (the text continues past what is displayed) and paste it wherever the public key is needed. If you are using GitHub, you can now create a new SSH key in your Personal Settings and paste this text into the 'Key' field.

5. Click on 'Save private key' and select a logical/recognizable name and directory for the file. Your private key is saved in the selected file.

6. Open Pageant (also part of the PuTTY package). If a message saying "Pageant is already running" is displayed, open your system tray and double click on the Pageant icon.

7. Click on 'Add Key' and select the file you saved when generating your key earlier (Step 5). If it is requested, enter the passphrase you created at Step 3 to complete the process.

Account and Access

Data

Data Retention

Your data (directories and files) will stay in Oscar for one years after your Brown account is deactivated. After that your data will be archived.

Date Deletion

You may delete your data when you leave Brown University. Or you may request that CCV delete your data on Oscar, especially if you have lots of data.

Retrieve Data

Billing

If you are a PI and want to keep your priority accounts and/or data directories after leaving Brown University, please contact support@ccv.brown.edu to update your billing information.

Agent Forwarding with PuTTY

1. Once adding your private key to Pageant, open PuTTY and navigate to the Auth menu.

2. Check the 'Allow agent forwarding' checkbox, and return to the Session menu.

3. Enter the Host Name you usually use to connect to Oscar, and click 'Open'.

4. Entering your password. If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Start the SSH-Agent

First, start your ssh-agent with the command below.

You should see an output similar to this:

Add Key(s)

Next, add your ssh private keys to the running agent (using the ssh-add command on line 1). This step may be repeated for every key pair you use to connect to different git servers. For most, this file is called id_rsa and will live in ~/.ssh/id_rsa. If you set a password for your ssh keys, the agent will prompt you to enter them.

Confirm the ssh keys have been loaded into the agent with ssh-add -L:

Connect to Oscar

Now ssh into Oscar with the -A option as shown on the first line below (replace username with your Oscar username). -A will forward your ssh-agent to Oscar, enabling you to use the ssh keys on your laptop while logged into Oscar.

If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Always connecting with Agent Forwarding

To make these changes permanent, you can add the ForwardAgent yes option to your ssh configuration file. To learn more about configuring your ssh connections, visit

• login009

• login010

• vscode1

Status and Limits

Arbiter2 applies different limits to a user's processes depending on the user's status: normal, penalty1, and penalty2.

Normal Status and Limits

Upon first log in, the user is in the normal status. These normal limits apply to all the user's processes on the node:

Penalty1 Status and Limits

When a user's processes consume CPU time more than the default CPU time limit for a period of time, the user's status is changed to the penalty1 status. These penalty1 limits are applied:

While a user is in penalty1 status, their processes are throttled if they consume more CPU time than penalty1 limit. However, if a user's processes exceed penalty1 memory limit, the processes (PIDs) will be terminated by cgroups.

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty1 limit for 30 minutes.

Penalty2 Status and Limits

When a user's processes consume more CPU time than the penalty1 limit for a period of time, the user is put in the penalty2 status, and the penalty2 limits apply to the user's processes.

In penalty2 status, the user's processes will be throttled if they consume more CPU time than penalty2 limit. However, if a user's processes exceed penalty2 memory limit, the processes (PIDs) will be terminated by cgroups.

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty2 limit for one hour.

Penalty3 Status and Limits

When a user's processes consume more CPU time than the penalty2 limit for a period of time, the user is put in the penalty3 status. These penalty3 limits apply to the user's processes.

In penalty3 status, the user's processes will be throttled if they consume more CPU time than penalty3 limit. If a user's processes exceed penalty3 memory limit, the processes (PIDs) will be terminated by cgroups

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty3 limit for two hours.

Email Notification

A user receives an email notification upon each violation. Below is a example email:

Violation of usage policy

This may indicate that you are running computationally-intensive work on the interactive/login node (when it should be run on compute nodes instead). Please utilize the 'interact' command to initiate a SLURM session on a compute node and run your workloads there.

You now have the status penalty1 because your usage has exceeded the thresholds for appropriate usage on the node. Your CPU usage is now limited to 80% of your original limit (8.0 cores) for the next 30 minutes. In addition, your memory limit is 80% of your original limit (40.0 GB) for the same period of time.

These limits will apply on login006.

High-impact processes

Usage values are recent averages. Instantaneous usage metrics may differ. The processes listed are probable suspects, but there may be some variation in the processes responsible for your impact on the node. Memory usage is expressed in GB and CPU usage is relative to one core (and may exceed 100% as a result).

Recent system usage

*This process is generally permitted on interactive nodes and is only counted against you when considering memory usage (regardless of the process, too much memory usage is still considered bad; it cannot be throttled like CPU). The process is included in this report to show usage holistically.

**This accounts for the difference between the overall usage and the collected PID usage (which can be less accurate). This may be large if there are a lot of short-lived processes (such as compilers or quick commands) that account for a significant fraction of the total usage. These processes are whitelisted as defined above.

Required User Actions

When a user receives an alert email that the user is put in a penalty status, the user should

• kill the processes that use too much resources on the shared node listed in the alert email, and/or reduce the resources used by the processes

Exempt Processes

Essential Linux utilities, such as rsync, cp, scp, SLURM commands, creating Singularity images, and code compilation, are exempt. To obtain a comprehensive list, please get in touch with us

SSH provides a method of sharing the ssh keys on your local machine with Oscar. This feature is called Agent Forwarding and can be useful for instance when working with version control or other services that authenticate via ssh keys. Below are instructions on how to configure your SSH connection to forward ssh-agent for diffeent operating systems

When regularly connecting to multiple remote systems over SSH, you’ll find that remembering all the hosts and various command-line options becomes tedious. OpenSSH allows setting up a configuration file to store different SSH options for each remote machine you connect t.

SSH Config File Location

OpenSSH client-side (in this case your personal computer) configuration file is named config, and it is stored in the hidded .sshdirectory under your user’s home directory (i.e., ~/.ssh)

This file must be readable and writable only by the user and not accessible by others:

SSH Config File Structure Basics

The SSH Config File takes the following structure:

The contents of the SSH config file is organized into sections. Each section starts with the Host directive and contains specific SSH options used when establishing a connection with the remote SSH server.

Oscar Hosts

Here we peovide a list of Oscar hosts and typical SSH configuration options. You have two options

1. Copy the list of hosts below directly into your SSH Config File (i.e., ~/.ssh/config)

2. Keep this content in a spearate file for Oscar hosts, lets say ~/.ssh/config.oscar and include that file in your main configuration file. In this case, the first line of ~/.ssh/config will be Include "~/.ssh/config.oscar"

Connecting to your preconfigured host

You may now connect using the shortchut notation provided by your configuration file. That is, all you need to type is:

According to the configuration above, this is equivalent to

Much shorter. Enjoy!

CCV uses all-flash parallel filesystem (Vast Data). Users have a home, data, and scratch space.

home ~

• 100GB of space

• Optimized for many small files

• 30 days snapshots

• The quota is per individual user

• A grace period of 14 days

data ~/data

• Each PI gets 256GB for free

• Optimized for reading large files

• 30 days snapshots

• The quota is by group

• A grace period of 14 days

scratch ~/scratch

• 512G (soft-quota): 12T (hard-quota)

• Optimized for reading/writing large files

• 30 days snapshots

• Purging: Files not accessed for 30 days may be deleted

• The quota is per individual user

• A grace period of 21 days

A good practice is to configure your application to read any initial input data from ~/data and write all output into ~/scratch. Then, when the application has finished, move or copy data you would like to save from ~/scratch to ~/data.

To see how much space on your directories, you can use the command checkquota. Below is an example output

$ checkquota
Name       Path                 Used(G)    (%) Used   SLIMIT(G)  H-LIMIT(G) Used_Inodes     SLIMIT     HLIMIT     Usage_State  Grace_Period  
ccvdemo1   /oscar/home          3.72       2          100        140        63539           2000000    3000000    OK           None          
ccvdemo1   /oscar/scratch       0.00       0          512        10240      1               4000000    16000000   OK           None          
Now fetching Data directory quotas...
Name        Used(T)   (%) Used   SLIMIT(T)   HLIMIT(T)   Used_Inodes   SLIMIT    HLIMIT    Usage_State   Grace_Period  
data+nopi   0.0       0          0.88        0.98        466           4194304   6291456   OK            None 

Efficient I/O is essential for good performance in data-intensive applications. Often, the file system is a substantial bottleneck on HPC systems, because CPU and memory technology has improved much more drastically in the last few decades than I/O technology.

Parallel I/O libraries such as MPI-IO, HDF5 and netCDF can help parallelize, aggregate and efficiently manage I/O operations. HDF5 and netCDF also have the benefit of using self-describing binary file formats that support complex data models and provide system portability. However, some simple guidelines can be used for almost any type of I/O on Oscar:

• Try to aggregate small chunks of data into larger reads and writes.

  For the GPFS file systems, reads and writes in multiples of 512KB

  provide the highest bandwidth.

• Avoid using ASCII representations of your data. They will usually

  require much more space to store, and require conversion to/from

  binary when reading/writing.

• Avoid creating directory hierarchies with thousands or millions of

  files in a directory. This causes a significant overhead in managing

  file metadata.

While it may seem convenient to use a directory hierarchy for managing large sets of very small files, this causes severe performance problems due to the large amount of file metadata. A better approach might be to implement the data hierarchy inside a single HDF5 file using HDF5's grouping and dataset mechanisms. This single data file would exhibit better I/O performance and would also be more portable than the directory approach.

RStudio is an IDE for R that can be run on Oscar.

Launching RStudio

Known Issues

Plotting figures may not work within RStudio. If this is the case, save the plots to a file, and view them through the Open On Demand Desktop App. If plots are required for your task, launch RStudio through the Desktop App.

The filesystem on Oscar can be accessed through the file explorer on this web portal. The file explorer allows you

• List files

• Create a directory

• Rename files

• Copy/Move files

To access the file explorer, click "Files" -> "Home Directory" at the top of the screen.

Check the documentation below for some of these services below:

Changing directories on File explorer

To access a directory, click "Change directory" and enter the path name

• To access your home directory, click the "Home Directory" link on the left. The path name at the top of the page should change to "/users/<username>"

• To access your scratch directory, click the "scratch" directory in your home directory OR click "Change directory" and enter "/users/<username>/scratch"

• To access your data directory, click the "data" directory in your home directory OR click "Change directory" and enter "/users/<username>/data"

Edit plain-text files

2. Click the icon with the three dots -> Edit

3. The file will open in a text editor in a new tab

Download files or directories

2. Click the icon with the three dots -> Download

To download multiples files:

1. Click the check-box to the left of the file name.

2. Scroll to the top of the page and click "Download"

Directories are downloaded as zipped files on your computer.

Upload files or directories

2. Click the "Upload" button.

3. Follow the instructions on the screen. You can click the "Browse" buttons or drag and drop files.

Launch a terminal

2. Click "Open in Terminal" at the top of the page.

3. A web-based terminal will open in a new tab of your browser. You will be logged into one of the login nodes.

If you have an installation of X11 on your local system, you can access Oscar with X forwarding enabled, so that the windows, menus, cursor, etc. of any X applications running on Oscar are all forwarded to your local X11 server. Here are some resources for setting up X11:

Mac/Linux

Once your X11 server is running locally, open a terminal and use

$ ssh -X <user>@ssh.ccv.brown.edu

to establish the X forwarding connection. Then, you can launch GUI applications from Oscar and they will be displayed locally on your X11 server.

Windows (PuTTY)

For Windows users using PuTTY, enable X forwarding under Connections->SSH->X11:

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. An Oscar account is required to access Open OnDemand. Visit this link in a web browser and sign in with your Brown username and password to access this portal.

Intro to Open OnDemand Slides

OOD provides with a several resources for interacting with Oscar.

Features:

1. No installation needed. Just use your favorite browser!

3. No need to use two-factor authentication multiple times. Just do it once, when you log into OOD.

4. Use it with, or without, VPN. Your workflow remains the same.

Open OnDemand offers a browser-based terminal app to access Oscar. Windows users who do not want to install an SSH client like Putty will find this app very useful.

Accessing the terminal

2. In the top menu, click Clusters -> >_OSCAR Shell Access

A new tab will open and the web-based terminal app will be launched in it. The shell will be launched on one of the login nodes.

Features:

1. No installation needed. Just use your favorite browser!

2. No need to enter your password again. SSH into Oscar in seconds!

3. No need to use two factor authentication again. Just do it once, when you log into OOD.

4. Use it with, or without, VPN. Your workflow remains the same.

You can launch several different apps on the Open OnDemand (OOD) interface. All of these apps start of a Slurm batch job on the Oscar cluster with the requested amount of resources. These jobs can access the filesystem on Oscar and all output files are written to the Oscar's file system.

Launching an App on OOD

2. If prompted, enter your Brown username and password.

3. Click on the "Interactive Apps" tab at the top of the screen to see the list of available apps. This will open the form to enter the details of the job.

4. Follow the instructions on the form to complete it. Some of fields can be left blank and OOD will choose the default option for you.

5. Click Launch to submit an OOD job. This will open a new tab on the browser It may take a few minutes for this job to start.

6. Click "Launch <APP>" again if prompted in the next tab.

There are several ways to move files between your machine and Oscar. Which method you choose will depend on how much data you need to move and your personal preference for each method.

1. SMB

2. Command line

Mac and Linux

SCP

You can use scp to transfer files. For example to copy a file from your computer to Oscar:

scp /path/to/source/file <username>@ssh.ccv.brown.edu:/path/to/destination/file

To copy a file from Oscar to your computer:

scp <username>@ssh.ccv.brown.edu:/path/to/source/file /path/to/destination/file

RSYNC

You can use rsync to sync files across your local computer to Oscar:

rsync -azvp --progress path/to/source/directory <username>@ssh.ccv.brown.edu:/path/to/destination/directory

Windows On Windows, if you have PuTTY installed, you can use it's pscp function from the terminal.

3. GUI programs for transferring files using the sftp protocol and transfer.ccv.brown.edu hostname

• DUO is required if you are not connected to approved networks, e.g., home network

  • There is no interactive terminal message but your Phone will get a prompt automatically

• DUO is NOT required if you are connected to approved Brown networks

In general, you can specify the following for your GUI programs:

• Protocol: SFTP

• Host: transfer.ccv.brown.edu

• User: your Oscar username

• Password: your Brown password

3.1.1 Limit Concurrent Transfer and Change Reconnect Options

Click the Optionsand then Preferences menu in WinsCP. In the poped up window, click Transfer and then Background to (Figure 1)

• change Maximal number of transfers at the same time to 1

• uncheck Use multiple connections for single transfer

click Endurace to (Figure 2)

• set Automatically reconnect session to 5 seconds

• uncheck Automatically reconnect session, if it stalls

• set Keep reconnection for to 10 seconds

3.1.2 Add a New Site

3.2 FileZilla

3.2.1. Disable Timeout

Click the Edit menu and then select the Settings submenu, and then change the Timeout in seconds to 0 to disable, as shown in Figure 2

3.2.2 Add a New Site

Open the Site Manager as show in Figure 5.

Click the 'New Site' button to add a new site, as shown in Figure 4:

Limit the number of simultaneous connections to 1, as shown in Figure 5.

Click the 'Connect' button to connect to Oscar and transfer files.

3.3 Cyberduck

You may see a popup window on 'Unknown Fingerprint'. You just need to check the 'Always' option and click 'Allow'. This is windows should not pop up again unless the transfer server is changed again.

4. Globus online

BrownU_CCV_Oscar

5. LFTP

module load lftp  # To load the LFTP module from Oscar
lftp -u login,passwd MyAwesomeUrl  # To connect to your (S)FTP server
ls   # To list files on the (S)FTP server
!ls  # To list files in your directory on Oscar
get MyAwesomeFile  # To download a single file
mirror # To download everything as is from the server
mirror --directory=/name_of_directory/ # To download a specific directory

VS Code one-time setup

2. Open VS Code settings and uncheck symlink:

Search for symlink and make sure the symlink searching is unchecked

4. Edit the config file:

C:\Users\<uname>\.ssh\config

Edit the config file on your local machine, add the following lines. Replace <username> with your Oscar username.

# Jump box with public IP address
Host jump-box
    HostName poodcit4.services.brown.edu
    User <username>

# Target machine with private IP address
Host ccv-vscode-node
    HostName vscode1
    User <username>
    ProxyCommand ssh -q -W %h:%p jump-box

1. September 10, 2023: Some users have reported issues while connecting to the Oscar VSCode remote extension. This is due to a recent change introduced by VSCode. To address this issue

Ctrl (cmd on Mac) + Shift + P > Remote-SSH: Settings
Disable the Remote.SSH: Use Exec Server option

1. In VS Code, select Remote-SSH: Connect to Host… and after the list populates select ccv-vscode-node

6. Install and set up of VS Code

1. Configure VS Code

Connect to VS Code first.

You can either create a symlink via the ln command below,

cp -v /gpfs/runtime/opt/vscode-server/ccv-vscode-config/settings.json /users/$USER/.vscode-server/data/Machine/settings.json

or manually create /users/$USER/.vscode-server/data/Machine/settings.json file with following contents

{
    "files.watcherExclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/**": true,
        "/usr/local/**": true,
        "/gpfs/home/**": true,
        "/gpfs/data/**": true,
        "/gpfs/scratch/**": true
    },
    "search.followSymlinks": false,
    "search.exclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/**": true,
        "/usr/local/**": true,
        "/gpfs/home/**": true,
        "/gpfs/data/**": true,
        "/gpfs/scratch/**": true
    }
}

Reconnect to VS Code

1. Click the green icon "Open a Remote Window" in the bottom left corner of VS Code Window. Then click "Connect to Host" in the drop down list.

2. Select the ccv-vscode-node option to connect to Oscar.

You may use either Globus (recommended) or smbclient to transfer data between Oscar and Campus File Storage.

Globus

smbclient

Transfer Instructions

1) Log into Oscar:

   ssh ssh.ccv.brown.edu

2) Start a screen session. This will allow you to reattach to your terminal window if you disconnect.

    screen

3) To use Oscar's high-speed connection to Campus File Storage - Replicated:

    smbclient "//smb.isi.ccv.brown.edu/SHARE_NAME" -D DIRECTORY_NAME -U "ad\BROWN_ID" -m SMB3

Similarly to access Campus File Storage - Non-Replicated ( LRS: Locally Redundant Share)

smbclient "//smblrs.ccv.brown.edu/Research" -D DIRECTORY_NAME -U "ad\BROWN_ID" -m SMB3

Replace SHARE_NAME, DIRECTORY_NAME, and BROWN_ID. DIRECTORY_NAME is an optional parameter. The password required is your Brown password.

4) Upload/download your data using the FTP "put"/"get" commands. Replace DIRECTORY_NAME with the folder you'd like to upload.

   put DIRECTORY_NAME

5) You can detach from the screen session with a "CTRL+A D" keypress. To reattach to your session:

   screen -r

smbclient basics

• put is upload to Campus File Storage

Usage: put <local_file> [remote file name]

Copy <local_file> from Oscar to Campus File Storage. The remote file name is optional (use if you want to rename the file)

• get is download to Oscar

Usage: get <remote_file> [local file name] Copy <remote_file> from the Campus File Storage to Oscar. The local file name is optional (use if you want to rename the file)

Moving more than one file:

To move more than one file at once use mput or mget. By default:

recurse is OFF. smbclient will not recurse into any subdirectories when copying files

prompt is ON. smbclient will ask for confirmation for each file in the subdirectories

You can toggle recursion ON/OFF with:

recurse

You can toggle prompt OFF/ON with:

prompt

Checkquota

Use the command checkquota to view your current disk usage and quotas. Here's an example output of this command

[ccvdemo2@login010 ~]$ checkquota
Name       Path                 Used(G)    (%) Used   SLIMIT(G)  H-LIMIT(G) Used_Inodes     SLIMIT     HLIMIT     Usage_State  Grace_Period
ccvdemo2   /oscar/home          23.29      18         100        125        188001          2000000    3000000    OK           None
ccvdemo2   /oscar/scratch       0.00       0          512        10240      2               4000000    16000000   OK           None
Now fetching Data directory quotas...
Name            Used(T)   (%) Used   SLIMIT(T)   HLIMIT(T)   Used_Inodes   SLIMIT    HLIMIT    Usage_State   Grace_Period
data+nopi       0.0       0          0.88        0.98        549           4194304   6291456   OK            None
data+ccvinter   0.015     1          0.50        1.00        122281        4194304   6291456   OK            None
==========================================================================================
Jobtmp Quotas: /jobtmp/$USER is the ultra-fast parallel storage system only for jobs. Not meant for long-term use
==========================================================================================
Block Limits                                    |     File Limits
Filesystem type         blocks      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
jobtmp     USR               0         1T        12T          0     none |        1 1000000  2000000        0     none sss6k.oscar.ccv.brown.edu
Got more Questions?
Read documentation: https://docs.ccv.brown.edu/oscar/managing-files/filesystem
Email: support@ccv.brown.edu
--------------------------------------------------------------------------------

Each line represents a top level directory that you have access to.

Each column represents a usage or quota for these directories.

Types of Quota:

Disk space usage

This usage is expressed in Gigabytes (G) or Terabytes (T) . This is the total size of all the files in that directory and it does not depend upon the number of files. Run the command checkquota to see your disk usage and quota. Here's an example:

Inode usage

This is the total number of files and directories in the particular directory. This number does not depend upon the size of the files. Run the command checkquota to see your inode usage and quota. Here's an example:

Soft Limits vs Hard Limits

All quotas have a soft limit (SLimit) and hard limit (HLimit). When usage exceeds the soft limit, a grace period associated with this limit begins. During the grace period, the usage is allowed to increase up to the hard limit. When the usage reaches the hard limit or when the grace period expires, the user is not allowed to write any files to that particular directory.

Usage State

The "Usage State" column shows the status of the grace period for a particular directory. Here are some of the status messages:

SOFT_EXCEEDED

This indicates that your usage of the disk space or inodes has exceeded the soft limit and you are still within the grace period. Check the Grace_Period column to see the number of days left in the grace period. You may continue writing data into this directory until the end of the grace period, as long as you do not exceed the hard limit

GRACE_EXPIRED

This indicates that your usage has exceeded the soft limit AND the grace period has expired. You will not be able to write data into that directory, but you can remove files.

HARD_EXCEEDED

This indicates that your usage has reached the hard limit. You will not be able to write data into that directory, but you can remove data.

OK

This indicates that your usage of the disk space as well as inodes in within the soft quota.

Step 1: Identify the directory

Run the checkquota command and identify the line that shows the warning status message.

If this directory is either /oscar/home or /oscar/scratch , you will have to take the subsequent steps to resolve this issue. If the directory is data+<group> you should inform others in your group and take collective action to resolve this issue.

Step 2: Disk Space or Inodes

Check whether you have exceeded your disk space quota or your inodes quota. Disk space usage is specified in GB or TB while inodes usage is just numerical count.

Step 3: Remove files

You will need to take the following steps based on the quota you have exceeded.

Disk Space quota:

The fastest way to reduce this usage is identifying large and unnecessary files. Load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. If they are not sorted by size, press lowercase s to sort them by size. You can navigate the directory tree using the arrow keys and delete any files or directories that are unnecessary.

Some programs leave a lot of temporary files on the disk that may not be necessary.

• Core Dump Files: This files are typically named core.<number> A core dump file is generated when a program crashes. It contains the state of the system and it is useful for debugging purposes. You can safely delete any core dump files if you know the reason behind the crash. Old core dump files can take up a lot of disk space and they can be safely deleted.

Inodes quota:

Inode usage can be reduced by removing any files and directories OR tarring up large nested directories. When a directory is converted to a tar ball, it uses a single inode instead of one inode per directory or file. This can drastically decrease your inode usage. Identify directories that contain a large number of files or a very large nested tree of directories with a lot of files.

To identify such directories, load the module ncdu using the command module load ncdu and run ncdu in the offending directory. This utility will scan that directory and show you all the directories and files, sorted by their size. Press uppercase C to switch the sorting criteria to "number of items". You can navigate the directory tree using the arrow keys and delete or tar any files or directories that are unnecessary.

To create a tar ball of a directory:

tar -cvf <directory_name>.tar.gz <directory_name>

If your usage has exceeded quota and you cannot write to the directory, you can tar ball in another directory. Using this command, you can create a tar ball in the scratch directory:

tar -cvf /oscar/scratch/$USER/<directory_name>.tar.gz <directory_name>

2. Open VSCode settings

• On Windows/Linux - File > Preferences > Settings

• On macOS - Code > Preferences > Settings

Search for symlink and make sure the symlink searching is unchecked

3. Under VSCode settings, search for remote ssh timeout and manually enter a timeout value i.e. 50s. It should give you enough time to complete 2-Factor Authentication.

4. Edit the ~/.ssh/config file on your local machine, add the following lines. Replace <username> with your Oscar username.

# Jump box with public IP address
Host jump-box
  HostName ssh8.ccv.brown.edu
  User <username>
# Target machine with private IP address
Host ccv-vscode-node
  HostName vscode1
  User <username>
  ProxyCommand ssh -q -W %h:%p jump-box

6. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select login-node

Nightly snaphots of the file system are available for the last 30 days.

Restore a file from a snapshot in the last 30 days

Nightly snapshots of the file system are available for the last 30 days can be found in the following directories.

Home directory snapshot

Data directory snapshot

Scratch directory snapshot

To restore a file, copy the file from the snapshot to your directory.

e.g.:

Jobs can be run on Oscar in two different ways:

• Interactive jobs allow the user to interact with programs (e.g., by entering input manually, using a GUI) while they are running. However, if your connection to the system is interrupted, the job will abort. Small jobs with short run times and jobs that require the use of a GUI are best-suited for running interactively.

• Batch jobs allow you to submit a script that tells the cluster how to run your program. Your program can run for long periods of time in the background, so you don't need to be connected to Oscar. The output of your program is continuously written to an output file that you can view both during and after your program runs.

Jobs are scheduled to run on the cluster according to your account priority and the resources you request (i.e., cores, memory, and runtime). In general, the fewer resources you request, the less time your job will spend waiting in the queue.

Git Overview

Version Control refers to the management of changes made to source code or any such large amount of information in a robust manner by multiple collaborators. Git is by far the most popular version control system.

Git enables effective collaboration among developers. In a team setting, multiple developers often work on the same project simultaneously. With Git, each developer can work on their own local copy of the project, making changes and experimenting freely without affecting the main codebase. Git allows developers to merge their changes seamlessly, ensuring that modifications made by different individuals can be consolidated efficiently. It provides mechanisms to track who made specific changes, making it easier to understand the evolution of the project and identify potential issues.

Git Workflow

Nearly all operations that are performed by Git are in you local computing environment, for the exception of few used purely to synchronize with a remote. Some of the most common git operations are depicted below. In summary a typical flow consists of making changes to your files, staging them via git add, marking a save point via git commit, then finally syncing to your remote (e.g., GitHub) via git push. If you are pushing changes to your remote from multiple places, you can bring changes your most recent version using git pull, which is the equivalent of doing git fetch followed by a git merge operation

Cheatsheet

Git Configuration

Getting Out of Trouble

Partition Overview

To list partitions on Oscar available to your account, run the following command:

To view all partitions (including ones you don't have access to), replace the -O in the command above with -aO.

batch is the default partition.

Partition Details

batch

• General purpose computing

• Priority is determined by account type (from highest

  to lowest: condo, priority, exploratory)

Condo limits apply to the group (i.e., they reflect the sum of all users on the condo). Condo users can check the limits on their condo with the command condos.

debug

• Short wait time, short run time access for debugging

• All users have the same limits and priority on the debug partition

vnc

• These nodes are for running VNC sessions/jobs

• Account type may affect Priority

gpu

• For GPU-based jobs

• GPU Priority users get higher priority and more resources than free users on the GPU partition

• Condo users submit to the gpu partition with normal or priority access (if they have a priority account in addition to their condo)

gpu-he

• For GPU-based jobs

• Uses Tesla V100 GPUs

• Restricted to High End GPU Priority users

gpu-debug

• Short wait time, short run time gpu access for debugging

• All users have the same limits and priority on the gpu-debug partition

bigmem

• For jobs requiring large amounts of memory

• Priority users get higher priority and more resources than free users on the bigmem partition

• Condo users submit to the bigmem partition with normal or priority access (if they have a priority account in addition to their condo)

• Premium users get higher priority and more resources than free users on the SMP partition

• Condo users submit to the SMP partition with normal or priority access (if they have a priority account in addition to their condo)

To start an interactive session for running serial or threaded programs on an Oscar compute node, simply run the command interact from the login node:

By default, this will create an interactive session that reserves 1 core and 4GB of memory for a period of 30 minutes. You can change the resources reserved for the session from these default limits by modifying the interact command:

For example, the command

requests an interactive session with 20 cores and 10 GB of memory (per node) for a period of 1 hour.

A job array is a collection of jobs that all run the same program, but on different values of a parameter. It is very useful for running parameter sweeps, since you don't have to write a separate batch script for each parameter setting.

To use a job array, add the option:

in your batch script. The range can be a comma separated list of integers, along with ranges separated by a dash. For example:

A job will be submitted for each value in the range. The values in the range will be substituted for the variable $SLURM_ARRAY_TASK_ID in the remainder of the script. Here is an example of a script for running a serial Matlab script on 16 different parameters by submitting 16 different jobs as an array:

You can then submit the multiple jobs using a single sbatch command:

The $SLURM_ARRAY_TASK_ID can be manipulated as needed. For example, you can generate a fixed length number form it. The following example generates a number of length of 3 from $SLURM_ARRAY_TASK_ID.

Listing running and queued jobs

The squeue command will list all jobs scheduled in the cluster. We have also written wrappers for squeue on Oscar that you may find more convenient:

Viewing estimated time until completion for pending jobs

This command will list all of your pending jobs and the estimated time until completion.

Canceling jobs

View details about completed jobs

sacct

The sacct command will list all of your running, queued and completed jobs since midnight of the previous day. To pick an earlier start date, specify it with the -S option:

To find out more information about a specific job, such as its exit status or the amount of runtime or memory it used, specify the -l ("long" format) and -j options with the job ID:

(example)

myjobinfo

The myjobinfo command uses the sacct command to display "Elapsed Time", "Requested Memory" and "Maximum Memory used on any one Node" for your jobs. This can be used to optimize the requested time and memory to have the job started as early as possible. Make sure you request a conservative amount based on how much was used.

ReqMem shows the requested memory: A c at the end of number represents Memory Per CPU, a n represents Memory Per Node. MaxRSS is the maximum memory used on any one node. Note that memory specified to sbatch using --mem is Per Node.

jobstats

The 'jobstats' utility is now available for analyzing recently completed jobs, comparing the resources used to those requested in the job script, including CPU, GPU, and memory. If email notifications are enabled, 'jobstats' sends an email with the results and includes a prompt to contact support for help with resource requests.

Run this command in a bash shell on Oscar. No additional module needs to be loaded.

To send this output to your email after the job is completed, make sure that these lines are in your job submit script

To use your condo account to submit jobs, please follow the steps below to check the association of your Oscar account and include condo information in your batch script or command line.

Step 1 - Check your account associations to find your condo Account and Partition information by running the following command:

In the example below, the user has access to two condos, where their Account and Partition are highlighted.

Step 2 - Choose the correct way to submit jobs to a condo according to the condo's Account column:

To see the running and pending jobs in a condo:

condo <condo-name>

Premium Account (priority) jobs

If you have a premium account, that should be your default QOS for submitting jobs. You can check if you have a premium account with the command groups. If you have a priority account you will see priority in your the output form groups.

You can check the qos for a running job by running the command myq. The QOS column should show "pri-<username>"

If you are interested in seeing all your accounts and associations, you can use the following command:

Associations

Oscar uses associations to control job submissions from users. An association refers to a combination of four factors: Cluster, Account, User, and Partition. For a user to submit jobs to a partition, an association for the user and partition is required in Oscar.

To view a table of association data for a specific user (thegrouch in the example), enter the following command in Oscar:

If thegrouch has an exploratory account, you should see an output similar to this:

Note that the first four columns correspond to the four factors that form an association. Each row of the table corresponds to a unique association (i.e., a unique combination of Cluster, Account, User, and Partition values). Each association is assigned a Quality of Service (see QOS section below for more details).

Some associations have a value for GrpTRESRunMins. This value indicates a limit on the total number of Trackable RESource (TRES) minutes that can be used by jobs running with this association at any given time. The cpu=110000 for the association with the batch partition indicates that all of the jobs running with this association can have at most an accumulated 110,000 core-minute cost. If this limit is reached, new jobs will be delayed until other jobs have completed and freed up resources.

Example of GrpTRESRunMins Limit

Here is an example file that incurs a significant core-minute cost:

If this file is named too_many_cpu_minutes.sh, a user withthegrouch's QOS might experience something like this:

Note that the REASON the job is pending and not yet running is AssocGrpCPURunMinutesLimit. This is because the program requests 30 cores for 90 hours, which is more than the oscar/default/thegrouch/batch association allows (30 cores * 90 hours * 60 minutes/hour = 162,000 core-minutes > 110,000 core-minutes). In fact, this job could be pending indefinitely, so it would be a good idea for thegrouch to run scancel 12345678 and make a less demanding job request (or use an association that allows for that amount of resources).

Account Quality of Service (QoS) and Resources

myaccount - To list the QoS & Resources

The myaccount command serves as a comprehensive tool for users to assess the resources associated with their accounts. By utilizing this command, individuals can gain insights into critical parameters such as Max Resources Per User and Max Jobs Submit Per User.

Resources from the web on getting started with MPI:

MPI is a standard that dictates the semantics and features of "message passing". There are different implementations of MPI. Those installed on Oscar are

• hpcx-mpi

• OpenMPI

We recommend using hpcx-mpi as it is integrated with the SLURM scheduler and optimized for the Infiniband network.

MPI modules on Oscar

Currently, the two available mpi implementations on Oscar are hpcx-mpi and openmpi. You can check the available versions by running these commands

hpcx-mpi/4.1.5rc2s-yflad4v is the recommend version of MPI on Oscar. It can be loaded by running

srun instead of mpirun

Use srun --mpi=pmix to run MPI programs. All MPI implementations are built with SLURM support. Hence, the programs need to be run using SLURM's srun command.

The --mpi=pmix flag is also required to match the configuration with which MPI is installed on Oscar.

Running MPI programs - Interactive

To run an MPI program interactively, first create an allocation from the login nodes using the salloc command:

For example, to request 4 cores to run 4 tasks (MPI processes):

Once the allocation is fulfilled, you can run MPI programs with the srun command:

When you are finished running MPI commands, you can release the allocation by exiting the shell:

Also, if you only need to run a single MPI program, you can skip the salloc command and specify the resources in a single sruncommand:

This will create the allocation, run the MPI program, and release the allocation.

Note: It is not possible to run MPI programs on compute nodes by using the interact command.

Running MPI programs - Batch Jobs

Here is a sample batch script to run an MPI program:

Hybrid MPI+OpenMP

If your program has multi-threading capability using OpenMP, you can have several cores attached with a single MPI task using the --cpus-per-task or -c option with sbatch or salloc. The environment variable OMP_NUM_THREADS governs the number of threads that will be used.

The above batch script will launch 4 MPI tasks - 2 on each node - and allocate 4 CPUs for each task (total 16 cores for the job). Setting OMP_NUM_THREADS governs the number of threads to be used, although this can also be set in the program.

Performance Scaling

The maximum theoretical speedup that can be achieved by a parallel program is governed by the proportion of sequential part in the program (Amdahl's law). Moreover, as the number of MPI processes increases, the communication overhead increases i.e. the amount of time spent in sending and receiving messages among the processes increases. For more than a certain number of processes, this increase starts dominating over the decrease in computational run time. This results in the overall program slowing down instead of speeding up as number of processes are increased.

Hence, MPI programs (or any parallel program) do not run faster as the number of processes are increased beyond a certain point.

If you intend to carry out a lot of runs for a program, the correct approach would be to find out the optimum number of processes which will result in the least run time or a reasonably less run time. Start with a small number of processes like 2 or 4 and first verify the correctness of the results by comparing them with the sequential runs. Then increase the number of processes gradually to find the optimum number beyond which the run time flattens out or starts increasing.

Maximum Number of Nodes for MPI Programs

An MPI program is allowed to run on at most 32 nodes. When a user requests more than 32 nodes for an MPI program/job, the user will receive the following error:

Batch job submission failed: Requested node configuration is not available

Submitting jobs using batch scripts

To run a batch job on Oscar, you first have to write a script that describes what resources you need and how your program will run. Some example batch scripts are available in your home directory on Oscar, in the directory:

A batch script starts by specifying the bash shell as its interpreter with the line:

By default, a batch job will reserve 1 core and 2.8GB of memory per core for your job. You can customize the amount of resources allocated for your job by explicitly requesting them in your batch script with a series of lines starting with #SBATCH, e.g.,

The above lines request 4 cores (-n), 16GB of memory per node (--mem), and one hour of runtime (-t). After you have described the resources you want allocated for the job, you then give the commands that you want to be executed.

Once you have your batch script, you can submit a batch job to the queue using the sbatch command:

Submitting jobs from the command line

As an alternative to requesting resources within your batch script, it is possible to define the resources requested as command-line options to sbatch. For example, the command below requests 4 cores (-n), 16GB of memory per node (--mem), and one hour of runtime (-t) to run the job defined in the batch script.

Note that command-line options passed to sbatch will override the resources specified in the script, so this is a handy way to reuse an existing batch script when you just want to change a few of the resource values.

Output from batch jobs

The sbatch command will return a number, which is your Job ID. You can view the output of your job in the file slurm-<jobid>.out in the directory where you invoked the sbatch command. For instance, you can view the last 10 lines of output with:

Alternatively, you can mention the file names where you want to dump the standard output and errors using the -o and -e flags. You can use %j within the output/error filenames to add the id of the job. If you would like to change your output file to be MyOutput-<job-id>, you can add the following line to your batch job:

sbatch command options

The table below summarizes some of the more useful options forsbatch .

Passing environment variables to a batch job

When a user logs into Oscar, there are pre-set environment variables such as HOME, which are the user's login environment variables. A user may modify an existing enviornmet variable, or add a new environment variable. So when a user submits a slurm batch job, the user's current environment variables may differ from the user's login environment. By default, a user's current environment variables, instead of the user's login environment variables, are accessible to the user's batch jobs on Oscar.

To modify or add an environment variable, run the following command:

• run the following command in your shell

• or have the following line in your batch script

After the step above to modify or add an environment variable, your batch job can access the environment variable my_variable whose value is my_value.

To export more than one environment variables, just list all the name=value pairs separated by commas:

Here is an example that a batch script loops over an input file and submits a job for each directory in the input file, where a directory is passed to a batch job for processing.

The input file test.txt has multiple lines where each line is a directory:

The loop.sh script reads each line (directory) from the input file and passes the directory as an environment variable to a batch job:

The test.job is a job script, which runs the test.sh to process the directory passed as an environment variable:

The test.sh is a bash script which simply echoes the directory:

If you run ./loop.sh, then three jobs are submitted. Each job generates an output like the following:

Using variables to set slurm job name, output filename, and error filename

Variables can be passed at the sbatch command line to set the job name, output and error file names, as shown in the following example:

The Oscar GPUs are in a separate partition to the regular compute nodes. The partition is called gpu. To see how many jobs are running and pending in the gpu partition, use

Interactive use

To start an session on a GPU node, use the interact command and specify the gpu partition. You also need to specify the requested number of GPUs using the -g option:

Batch jobs

Here is an example batch script for a cuda job that uses 1 gpu and 1 cpu for 5 minutes

To submit this script:

DGX GPU Nodes in the GPU-HE Partition

All the nodes in the gpu-he partition have V100 GPUs. However, two of them are DGX nodes (gpu1404/1405) which have 8 GPUs. When a gpu-he job requests for more than 4 GPUs, the job will automatically be allocated to the DGX nodes.

The other non-DGX nodes actually have a better NVLink interconnect topology as all of them have direct links to the other. So the non-DGX nodes are better for a gpu-he job if the job does not require more than 4 GPUs.

Hardware Specifications

Access

The two GH200 nodes are in the gracehopper partition.

gk-condo Account

A gk-condo user can submit jobs to the GH200 nodes with their gk-gh200-gcondo account, i.e.,

CCV Account

For users who are not a gk-condo user, a High End GPU priority account is required for accessing the gracehopper partition and GH200 nodes. All users with access to the GH200 nodes need to submit jobs to the nodes with the ccv-gh200-gcondo account, i.e.

MIG Access

To request a MIG, the feature mig needs be specified, i.e.

Running NGC Containers

Running Modules

The two nodes have Arm CPUs. So Oscar modules do not run on the two GH200 nodes. Please contact support@ccv.brown.edu about installing and running modules on GH200 nodes.

Compiling with CUDA

The CUDA compiler is called nvcc, and for compiling a simple CUDA program it uses syntax simlar to gcc:

Optimizations for Fermi

The Oscar GPU nodes feature NVIDIA M2050 cards with the Fermi architecture, which supports CUDA's "compute capability" 2.0. To fully utilize the hardware optimizations available in this architecture, add the -arch=sm_20 flag to your compile line:

This means that the resulting executable will not be backwards-compatible with earlier GPU architectures, but this should not be a problem since CCV nodes only use the M2050.

Memory caching

The Fermi architecture has two levels of memory cache similar to the L1 and L2 caches of a CPU. The 768KB L2 cache is shared by all multiprocessors, while the L1 cache by default uses only 16KB of the available 64KB shared memory on each multiprocessor.

You can increase the amount of L1 cache to 48KB at compile time by adding the flags -Xptxas -dlcm=ca to your compile line:

If your kernel primarily accesses global memory and uses less than 16KB of shared memory, you may see a benefit by increasing the L1 cache size.

If your kernel has a simple memory access pattern, you may have better results by explicitly caching global memory into shared memory from within your kernel. You can turn off the L1 cache using the flags –Xptxas –dlcm=cg.

The new Ampere architecture GPUs do not support older CUDA modules. Users must re-compile their applications with the newer CUDA/11 or older modules. Here are detailed instructions to compile major frameworks such as PyTorch, and TensorFlow.

PyTorch

Users can install PyTorch from a pip virtual environment or use pre-built singularity containers provided by Nvidia NGC.

To install via virtual environment: