• Putty Installation

• SSH to Oscar from Linux

• SSH to Oscar from Mac

• SSH to Oscar from Windows

• Batch Job Submission on Oscar

• Linux Basics for Oscar

• How to Use Modules in Oscar

• Remote Rendering with ParaView in Oscar

Oscar Specifications

Compute Nodes

Oscar has compute nodes in the partitions listed below.

• batch - The batch partition is for programs/jobs which need neither GPUs nor large memory.

• bigmem - The bigmem partition is for programs/jobs which require large memory.

• debug - The debug partition is for users to debug programs/jobs.

• gpu - The gpu partition is for programs/jobs which require GPUs.

• gpu-debug - The gpu-debug partition is for users to debug gpu programs/jobs.

• gpu-he -The gpu-he partition is for programs/jobs which need to access high-end GPUs.

• vnc - The vnc partition is for users to run programs/jobs in an graphical desktop environment.

Below are node details including cores and memory for all partitions.

Hardware details

Hardware details for all partitions. The Features column shows the features available for the --constraint option for SLURM. This includes the available CPU types as well GPUs.

GPU Features and GPU Memory

module list

Lists all modules that are currently loaded in your software environment.

module avail

Lists all available modules on the system. Note that a module can have multiple versions. Use module avail <name> to list available modules which start with <name>

module help <name>

Prints additional information about the given software.

module load <name>

Adds a module to your current environment. If you load using just the name of a module, you will get the default version. To load a specific version, load the module using its full name with the version: "module load gcc/10.2"

module unload <name>

Removes a module from your current environment.

Anaconda / Conda

A distribution of Python and R used for scientific computing that is meant to simplify package management and deployment. Conda is used for installing packages and managing their dependencies. [Related Page - Anaconda]

Association

Within Oscar, an association refers to a combination of four factors: Cluster, Account, User, and Partition. Associations are used to control job submissions for users. [Related Page - Associations & Quality of Service]

Batch Jobs

Put simply, batch jobs are scheduled programs that are assigned to run on a computer without further user interaction. [Related Page - Batch Jobs]

CCV

Brown University's Center for Computation and Visualization. Provides software, expertise, and other services for Brown's research community. See our website for more information.

CESM

Stands for Community Earth System Model. "CESM is a fully-coupled, community, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states." (Source) [Related Page - Using a CESM module]

Condo

PIs can purchase condos that have a significant amount of computing resources which can be shared with others. [Related Page - Account Types]

CUDA

"CUDA is an extension of the C language, as well as a runtime library, to facilitate general-purpose programming of NVIDIA GPUs." (Source) [Related Page - Intro to CUDA]

Desktop App

This app on Open OnDemand allows users to launch a Desktop GUI on Oscar. This app is based on VNC which is a desktop sharing system that allows you to remotely control another desktop.[Related Page- Desktop App (VNC)]

HPC

Stands for High Performance Computing. HPC is the ability to process data and perform highly complex calculations at an accelerated rate. Oscar is the service that CCV offers to the Brown community for their High Performance Computing needs.

Job Array

A job array is a collection of jobs that all run the same program but on different values of a parameter. [Related Page - Job Arrays]

Jupyter Notebook

"The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text." [Related Page - Jupyter Notebooks on Oscar]

Interactive Jobs

Jobs that allow the user to interact in real time with applications within Oscar, often from the command line. This differs from batch jobs in that each command to be run must be put in one at a time. [Related Page - Interactive Jobs]

Modules

Modules are software components that can easily be loaded or unloaded into Oscar. For instance, a user can load the Python 3 module using a module load command. [Related Page - Using Modules]

MPI

Stands for Message Passing Interface. MPI is a system that aims to be the standard for portable and efficient message passing. Message passing is a technique often used in object-oriented programming and parallel programming [Related Page - MPI Jobs]

Open OnDemand (OOD)

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. It can be used to launch a Desktop session on Oscar [Related Page - Open OnDemand]

OOD app

OOD app is a web application that runs on the Open OnDemand web portal. It allows users to launch interactive applications like Jupyter Notebook, RStudio, Matlab or Desktop. [Related Page - Interactive Apps on OOD]

Partition

Partitions are essentially groupings of nodes that allocate resources for specific types of tasks. On Oscar, partitions are based on job submissions through the Slurm workload manager. [Related Page - Slurm Partitions]

PI

Stands for Principal Investigator. Mainly used to refer to the individual responsible for conducting and administrating a research grant. Within Oscar, PIs have their own data directories that can be shared to students. PIs may also purchase condos. [Related Page - Account Types]

PuTTY

A client for SSH for Windows and Unix that emulates a terminal [Related Page - SSH (Terminal)]

Python

An object-oriented, high-level, and popular programming language [Related Page - Python on Oscar]

Quality of Service (QOS)

The job limits that are linked to a given association. For instance, Priority Accounts will generally have a higher quality of service than Exploratory Accounts. [Related Page - Associations & Quality of Service (QOS)]

Slurm

A workload manager used within Oscar to schedule jobs [Related Page - Slurm Partitions]

SSH

Stands for Secure Shell Protocol. Used to communicate securely between computers and often used within a command-line interface (CLI) for connections to remote servers [Related Page - SSH (Terminal)]

SMB

The Server Message Block (SMB) protocol is a network protocol that allows users to communicate with remote computers for file-sharing and other uses. It is one of the versions of the Common Internet File System (CIFS). Within Oscar, SMB is mainly used for file transfer. [Related Page - SMB (Local Mount)]

cd

Moves the user into the specified directory. Change Directory.

cd .. to move one directory up

cd by itself to move to home directory

cd - to move to previous directory

cd <directory-path> to move to a directory (can be an absolute path or relative path)

cp <old_filepath> <new directory path>

Copies the file into the specified directory

clear

Clears the terminal

cat <filename>

Lists the contents of a file. Concatenate files.

ls

List contents within the current directory

grep <string_to_match> <filename>

pwd

Displays the path of the current directory that you are in. Present Working Directory

man <command>

Displays the help manual instruction for the given command

mv <file_name> <new_directory>

Moves a file into a new directory.

mv <old_file_name> <new_file_name> to rename a file

mkdir <directory_name>

Creates a new directory

rm <file_name>

Deletes a file

rm -r <directory_name>

Deletes directories and the contents within them. -r stands for recursive

rmdir <directory_name>

Removes the specified directory (must be empty)

touch

Creates a blank new file

Oscar is Brown University's high performance computing cluster for both research and classes. Oscar is maintained and supported by Center for Computation and Visualization (CCV).

Please contact support@ccv.brown.edu if there are any questions on Oscar.

Accounts

If you do not have an Oscar account, you can request one by clicking the following link:

Anyone with a Brown account can get a free Exploratory account on Oscar, or pay for priority accounts.

More details can be found at the CCV Rates page.

Individuals external to Brown can get access to Oscar by having a sponsored Brown account. Please work with your department to get sponsored Brown accounts for any external collaborators.

Authorized users must comply with the following Brown University policies:

• Acceptable Use Policy

• Computing Passwords Policy.

• Computing Policies.

Hardware

Users can run their computing-intensive and/or long runtime jobs/program in Oscar to take advantage of high performance computing resources there, as highlighted below:

• 2 Login nodes

• 8 PB of storage

• Red Hat Enterprise Linux 9.2 (Linux)

• Mellanox InfiniBand network

• Slurm Workload manager

Please refer to the details at Oscar hardware.

Scheduler

Hundreds of users can share computing resources in Oscar. Slurm is used in Oscar to manage user jobs and computing resources such as cores and GPUs.

Users should not run computations or simulations on the login nodes, because they are shared with other users. You can use the login nodes to compile your codes, manage files, and launch jobs on the compute nodes.

To allow users sharing access to Oscar, there are limits on the maximum number of pending and running jobs a user account may have/submit:

• 1200 for a priority account

• 1000 for an exploratory account

Software

• Operating systems of all Oscar nodes: Red Hat 9.2

• More than 500 software modules

• CCV Staff install software upon user requests or help users on software installation

Storage

Oscar has 8 PB of all-flash storage from VAST, which provides high-performance access to storage. Users have ~/home, ~/scratch, and ~/data directories as their storage with quota in Oscar. Please refer to the details at Oscar's filesystem.

Access and User Accounts - User accounts are controlled via central authentication and directories on Oscar are only deleted on the request of the user, PI, or departmental chair.

Files not accessed for 30 days will be deleted from your ~/scratch directory. Use ~/data for files you wish to keep long term.

Users can transfer files from and to Oscar filesystem. In particular, users can transfer files between Oscar filesystem and Campus File Storage.

Connecting to Oscar

Oscar users can connect to Oscar by

• SSH

• Open OnDemand

• VS Code Remote IDE

Maintenance Schedule

• Non-disruptive Maintenance:

  • non-disruptive work, including software changes, maintenance, and testing

  • may occur at any time

  • no notification provided

• Monthly Scheduled Maintenance:

  • no downtime expected, but there may be limited degradation of performance

  • first Tuesday of the month, 8:00 am - 12:00 noon

  • no notification provided

• Unscheduled Maintenance:

  • maximum 1 day downtime

  • occurs very rarely and includes any unplanned emergency issues that arise

  • Prior notification provided (depending on the issue, 1 day to 4 weeks advance notice provided)

• Major Upgrade Maintenance:

  • service may be brought down for 3-5 days

  • occurs annually

  • 4-week prior notification provided

Unplanned Outage

• During Business Hours:

  • Send email to support@ccv.brown.edu. A ticket will get created and CCV staff will attempt to address the issue as soon as possible.

• During Non-Business Hours:

  • Send email to support@ccv.brown.edu.

  • Call CIS Operations Center at (401) 863-7562. A ticket will get created and CCV staff will be contacted to address the issue.

User and Research Support

CCV staff support for researchers seeking help with statistical modeling, machine learning, data mining, data visualization, computational biology, high-performance computing, and software engineering.

CCV staff provides tutorials on using Oscar for classes, groups and individual. Please check CCV Events for upcoming trainings and office hours.

CCV provides short videos (coming soon) for users to learn as well.

OSCAR

Oscar is the shared compute cluster operated by CCV.

Oscar runs the Linux RedHat7 operating system. General Linux documentation is available from . We recommend you read up on basic Linux commands before using Oscar. Some of the most common commands you'll be using in Oscar can also be found on our .

Oscar has two login nodes and several hundred compute nodes. When users log in through Secure Shell (SSH), they are first put on one of the login nodes which are shared among several users at a time. You can use the login nodes to compile your code, manage files, and launch jobs on the compute nodes from your own computer. Running computationally intensive or memory intensive programs on the login node slows down the system for all users. Any processes taking up too much CPU or memory on a login node will be killed. Please do not run Matlab on the login nodes.

What username and password should I be using?

• If you are at Brown and have requested a regular CCV account, your Oscar login will be authenticated using your Brown credentials, i.e. the same username and password that you use to log into any Brown service such as "canvas". We have seen login problems with the Brown credentials for some users so accounts moved to the RedHat7 system after September 1st 2018 can also log into RedHat7 with their CCV password.

• If you are an external user, you will have to get a sponsored ID at Brown through the department with which you are associated before requesting an account on Oscar. Once you have the sponsored ID at Brown, you can on Oscar and use your Brown username and password to log in.

Connecting to Oscar for the first time

To log in to Oscar you need Secure Shell (SSH) on your computer. Mac and Linux machines normally have SSH available. To login in to Oscar, open a terminal and type

Windows users need to install an SSH client. We recommend , a free SSH client for Windows. Once you've installed PuTTY, open the client and use <username>@ssh.ccv.brown.edufor the Host Name and click Open. The configuration should look similar to the screenshot below.

The first time you connect to Oscar you will see a message like:

You can type yes . You will be prompted for your password. Note that nothing will show up on the screen when you type in your password; just type it in and press enter. You will now be in your home directory on Oscar. In your terminal you will see a prompt like this:

Congratulations, you are now on one of the Oscar login nodes.

File system

Users on Oscar have three places to store files:

• home

• scratch

• data

Note that class accounts may not have a data directory. Users who are members of more than one research group may have access to multiple data directories.

From the home directory, you can use the command ls to see your scratch directory and your data directory (if you have one) and use cd to navigate into them if needed.

To see how much space in your directories, use the command checkquota. Below is an example output:

Software modules

Using a Desktop on Oscar

Using VNC, you can run graphical user interface (GUI) applications like Matlab, Mathematica, etc. while having access to Oscar's compute power and file system.

Running Jobs

You are on Oscar's login nodes when you log in through SSH. You should not (and would not want to) run your programs on these nodes as these are shared by all active users to perform tasks like managing files and compiling programs.

With so many active users, a shared cluster has to use a "job scheduler" to assign compute resources to users for running programs. When you submit a job (a set of commands) to the scheduler along with the resources you need, it puts your job in a queue. The job is run when the required resources (cores, memory, etc.) become available. Note that since Oscar is a shared resource, you must be prepared to wait for your job to start running, and it can't be expected to start running straight away.

Oscar uses the SLURM job scheduler. Batch jobs are the preferred mode of running programs, where all commands are mentioned in a "batch script" along with the required resources (number of cores, wall-time, etc.). However, there is also a way to run programs interactively.

Where to get help

General

How do I request help?

Most inquiries can be directed to CCV’s support address, support@ccv.brown.edu, which will create a support ticket with one of our staff.

What are the fees for CCV services?

All CCV services are billed quarterly, and rates can be found (requires a Brown authentication to view). Questions about rates should be directed to support@ccv.brown.edu.

How do I acknowledge CCV in a research publication?

We greatly appreciate acknowledgements in research publications that benefited from the use of CCV services or resources.

Oscar

What is Oscar?

Oscar is our primary research computing cluster with several hundred multi-core nodes sharing a high-performance interconnect and file system. Applications can be run interactively or scheduled as batch jobs.

How do I request an account on Oscar?

To request an account, please fill out a All accounts are subject to our .

How do I run a job on Oscar?

Can I use Oscar for teaching?

How do I find out when the system is down?

We post updates to our user mailing list, ccv@listserv.brown.edu which you are automatically subscribed to when setting up an account with CCV. If you need to be added to the mailing list, please submit a support ticket to support@ccv.brown.edu. We also have an announcement mailing list for office hours, workshops and other events relevant to CCV users, ccv-announce.listserve.brown.edu.

How do I run a job array on Oscar?

How do I run a MPI job on Oscar?

I have some MPI-enabled source code. How can I compile it on Oscar?

Load an mpi module module load mpi. For a list of mpi modules available, module avail mpi

What applications are available on Oscar?

What compilers are available on Oscar?

How do I get information about finished jobs?

The sacct command will list all of your completed jobs since midnight of the previous day (as well as running and queued jobs). You can pick an earlier start date with the -S option, e.g. sacct -S 2012-01-01.

How much storage am I using?

My job keeps terminating unexpectedly with a "Killed" message, or without any errors. What happened?

How do I request a certain amount of memory per CPU?

Specify the SLURM option --mem-per-cpu= in your script.

How do I link against a BLAS and LAPACK library?

I am getting a "WARNING: Remote HOST IDENTIFICATION HAS CHANGED?

We have recently updated the login and VSCode node hardware to improve performance, security, and reliability. As a result of this migration, the SSH host keys for our servers have been updated. To fix this:

• On MacOS:

• On Linux:

• On Windows: from VSCode's internal terminal Window:

and delete the lines starting with Oscar and delete lines starting with vscode and oscarHopefully, this will make things easier.

• OpenOnDemand (OOD) Shell Access: either get a Desktop session or login via regular terminal into 'ssh.ccv.brown.edu' and run

Then login again via OOD > Clusters

RUNNING JOBS

How is a job identified?

By a unique JobID, e.g. 1318013

Which of my jobs are running/pending?

Use the command myq

How do I check the progress of my running job?

You can look at the output file. The default output file is slurm-%j.out" where %j is the JobID. If you specified and output file using #SBATCH -o output_filename and/or an error file #SBATCH -e error_filename you can check these files for any output from your job. You can view the contents of a text file using the program less , e.g.

Use the spacebar to move down the file, b to move back up the file, and q to quit.

My job is not running how I intended it to. How do I cancel the job?

scancel <JobID> where <JobID> is the job allocation number, e.g. 13180139

How do I save a copy of an interactive session?

You can use interact -o outfile to save a copy of the session's output to "outfile"

I've submitted a bunch of jobs. How do I tell which one is which? myq will list the running and pending jobs with their JobID and the name of the job. The name of the job is set in the batch script with #SBATCH -J jobname. For jobs that are in the queue (running or pending) you can use the command scontrol show job <JobID> where <JobID> is the job allocation number, e.g.13180139 to give you more detail about what was submitted.

How do I ask for a haswell node?

Use the --constraint (or -C) option:

You can use the --constraint option restrict your allocation according to other features too. The nodes command provides a list of "features" for each type of node.

Why won't my job start?

When your job is pending (PD) in the queue, SLURM will display a reason why your job is pending. The table below shows some common reasons for which jobs are kept pending.

Why is my job taking so long to start? Just waiting in (Priority) or (Resources)

1. Overall system busy: when tens of thousands of jobs are submitted it total by all users, the time it takes SLURM to process these into the system may increase from the normal almost instantly to a half-hour or more.

2. Specific resource busy: if you request very specific resources (e.g., a specific processor) you then have to wait for that specific resource to become available while other similar resources may be going unused.

3. Specified resource not available: if you request something that is not or may never be available, your job will simply wait in the queue. E.g., requesting 64 GB of RAM on a 64 GB node will never run because the system needs at least 1 GB for itself so you should reduce your request to less than 64.

TRANSFERRING FILES

How do I transfer big files to/from Oscar?

Please use the server transfer.ccv.brown.edu

1. Transfer local file to Oscar:

2. Transfer remote file on Oscar to the local system:

Cloud HPC Options

The use of cloud resources for HPC varies according to your demands and circumstances. Cloud options are changing rapidly both in service providers and various services being offered. For those who have short-term needs that don't demand the highest of computational performance, a cloud option might be appropriate. For others, a local option customized to individual needs may be better. The cost of cloud services also varies quite a bit and includes not only compute time but data transfer charges. Other issues involved licensing, file synchronization, etc.

We are actively investigating a number of options to connect Brown users seamlessly to suitable cloud options. We are collecting such information for publishing on the CIS website as part of research services available. At this point, the best course of action is to request an individual consultation to help address your specific needs. Please send email to support@ccv. brown.edu.

Filing a Support Ticket

Filing a good support ticket makes it much easier for CCV staff to deal with your request

When you email support@ccv.brown.edu aim to include the following:

• State the problem/request in the subject of the email

• Describe which software and with version you are using

• Error message (if there was one)

• The job number

• How you were running, e.g. batch, interactively, vnc

• Give as as small an example as possible that reproduces the problem

Q&A Forum

Ask questions and search for previous problems at our .

Slack

Join our CCV-Share Slack workspace to discuss your questions with CCV Staff in the #oscar channel.

Office Hours

CCV holds weekly office hours. These are drop in sessions where we'll have one or more CCV staff members available to answer questions and help with any problems you have. Please visit for upcoming office hours and events.

Arrange a Meeting

You can arrange to meet with a CCV staff member in person to go over difficult problems, or to discuss how best to use Oscar. Email support@ccv.brown.edu to arrange a consultation.

Connect to OSCAR

The simplest way to connect to Oscar is via Open OnDemand (OOD). To connect to OOD, go to and log in using your Brown credentials. For more details on Open OnDemand click .

Alternatively, you can connect to OSCAR via SSH (Terminal):

Submit a Job

You can submit a job using sbatch:

You can confirm that your job ran successfully by running:

For more detailed information on submitting jobs, see the section of the documentation.

Transfer Files

To get specific files on to / off of Oscar, read through the page of the documentation.

Get Help

These pages list some command commands and terms you will come across while using Oscar.

Common Acronyms and Terms

Managing Modules

Common Linux Commands

If you publish research that benefited from the use of CCV services or resources, we would greatly appreciate an acknowledgment that states:

This research [Part of this research] was conducted using [computational/visualization]
resources and services at the Center for Computation and Visualization, Brown University.

Account Usage

Oscar users are not permitted to:

• Share their accounts or passwords with others or enable unauthorized users to access Center for Computation and Visualization resources

• Use Center for Computation and Visualization resources for personal economic gain

• Engage in unauthorized activity (e.g., cryto currency mining etc.) that intentionally impacts integrity of resources

Storage

Each user (premium or exploratory) gets 20GB Home Directory, 512GB short-term Scratch, and 256G Data directory (shared amongst the members of group)

• Files in Scratch Directory not accessed for last 30 days are automatically purged. CCV only stores snapshots for 7 days after that files will be automatically deleted.

• PI has the ultimate access to Data Directory - if a student leaves Brown the files in Data directory will be owned by the PI.

Software and Data

All software and data stored or used on Center hosted systems must be appropriately and legally acquired and must be used in compliance with applicable licensing terms. Unauthorized misuse or copying of copyrighted materials is prohibited.

Data Retention

CCV reserves the right to remove any data at any time and/or transfer data or other individuals (such as Principal Investigators working on a same or similar project) after a user account is deleted is no longer affiliated with Brown University.

Accounts Validity

• Once created, Oscar accounts are valid for duration of one's Brown AD credentials

Account and Access

Oscar users will keep their access to Oscar as long as their Brown account are still active. To be able to access Oscar account after a user's Brown account is deactivated, the user needs to get an affiliate account through the department the user is associated.

Data

Data Retention

Your data (directories and files) will stay in Oscar for one years after your Brown account is deactivated. After that your data will be archived.

Date Deletion

You may delete your data when you leave Brown University. Or you may request that CCV delete your data on Oscar, especially if you have lots of data.

Retrieve Data

You can download data from Oscar following the instructions here. Globus is recommended for large data transfer.

Billing

If you are a PI and want to keep your priority accounts and/or data directories after leaving Brown University, please contact support@ccv.brown.edu to update your billing information.

Step 1 : Check for existing SSH key pair

Before generating new SSH key pair first check if you have an SSH key on your local machine.

ls ~/.ssh/id_*.pub

If there are existing keys, please move to Step 3

Step 2 : Generate a new SSH Keypair

ssh-keygen -t rsa

ssh-keygen.exe

Press Enter to accept the default file location and file name.

The ssh-keygen will ask you to type a secure passphrase. This is optional. If you don't want to use a passphrase just press Enter

Verify the SSH keys are generated correctly, you should see two files id_rsa and id_rsa.pub under ~/.ssh directory.

Step 3 : Copy the public key to Oscar

You will now need to copy your public key to Oscar. There are two ways to acomplish this.

With ssh-copy-id

If your OS comes with the ssh-copy-id utility, then you'll be able to copy your public key into Oscar as follows:

ssh-copy-id <username>@ssh.ccv.brown.edu

You will be prompted for a Password. The public key will be appended to the authorized_keys file on Oscar.

If you used a custom name for your key instead of the default id_rsa then you'll need pass the name of your key to ssh-copy-id i.e.,

ssh-copy-id -i ~/.ssh/<keyname> <username>@ssh.ccv.brown.edu

Without ssh-copy-id

If your system does not come with the ssh-copy-id utility installed, then you'll need to copy your public key by hand.

1. Get the contents of id_rsa.pub file. One option is to use cat in your teminal cat id_rsa.pub.

2. Copy the contents of this file to your clipboard, as we need to upload it to Oscar.

3. Login into Oscar via regular ssh ssh <username>@ssh.ccv.brown.edu. Once you are on the login node, open the authorized_keys file with your text editor of choice e.g., vim ~/.ssh/authorized_keys or nano ~/.ssh/authorized_keys Add your public keys to end of this file. Save and exit.

Step 4 : Login to Oscar using your SSH keys

ssh <username>@sshcampus.ccv.brown.edu

If everything went well, you will be logged in immediately withouth prompting you for a password.

To log in to Oscar you need Secure Shell (SSH) on your computer.

There are two options for signing into Oscar: with or without VPN.

Summary of SSH Hosts

• ssh.ccv.brown.edu You can connect from anywhere. You will need Two Factor Authentication

• sshcampus.ccv.brown.edu You can connect when whithin Brown Wifi, Network or VPN. You will need to set up passwordless authentication.

• desktop.ccv.brown.edu This is the host to be used when connecting from a remote IDE, i.e., Visual Studio Code.

• transfer.ccv.brown.edu This host is used to transfer files to/from oscar using SFTP protocol

macOS and Linux

To log in to Oscar, open a terminal and

• If you are not connected to the Brown VPN, use the following command:

ssh -X username@ssh.ccv.brown.edu

• If you are connected to the Brown VPN, use the following command:

ssh -X username@sshcampus.ccv.brown.edu

The -X allows Oscar to display windows on your machine. This allows you to open and use GUI-based applications, such as the text editor gedit.

Windows

Windows users need to install an SSH client. We recommend PuTTY, a free SSH client for Windows.

• If you are not connected to the Brown VPN, use username@ssh.ccv.brown.edu as the Host Name and click Open.

• If you are connected to the Brown VPN, use username@sshvpn.ccv.brown.edu as the Host Name and click Open.

Connecting to Oscar for the First Time

The first time you connect to Oscar you will see a message about the authenticity of the host:

The authenticity of host 'ssh.ccv.brown.edu (138.16.172.8)' can't be established.
RSA key fingerprint is SHA256:Nt***************vL3cH7A.
Are you sure you want to continue connecting (yes/no)?

You can type yes and press return. On subsequent logins you should not see this message.

You will then be prompted for your password.

You will now be in your home directory on Oscar. In your terminal you will see a prompt like this:

[username@login004 ~]$

Congratulations, you are now on one of the Oscar login nodes! The login nodes are for administrative tasks such as editing files and compiling code. To use Oscar for computation you will need to use the compute nodes. To get to the compute nodes from the login nodes you can either start an interactive session on a compute node, or submit a batch job.

Start the SSH-Agent

First, start your ssh-agent with the command below.

You should see an output similar to this:

Add Key(s)

Next, add your ssh private keys to the running agent (using the ssh-add command on line 1). This step may be repeated for every key pair you use to connect to different git servers. For most, this file is called id_rsa and will live in ~/.ssh/id_rsa. If you set a password for your ssh keys, the agent will prompt you to enter them.

Confirm the ssh keys have been loaded into the agent with ssh-add -L:

Connect to Oscar

Now ssh into Oscar with the -A option as shown on the first line below (replace username with your Oscar username). -A will forward your ssh-agent to Oscar, enabling you to use the ssh keys on your laptop while logged into Oscar.

If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Always connecting with Agent Forwarding

To make these changes permanent, you can add the ForwardAgent yes option to your ssh configuration file. To learn more about configuring your ssh connections, visit

Agent Forwarding with PuTTY

1. Once adding your private key to Pageant, open PuTTY and navigate to the Auth menu.

2. Check the 'Allow agent forwarding' checkbox, and return to the Session menu.

3. Enter the Host Name you usually use to connect to Oscar, and click 'Open'.

4. Entering your password. If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Key Generation & Setup

1. Open PuTTYgen (this comes as part of the package), change the 'Number of bits in a generated key:' to 4096 (recommended), then click 'Generate'

2. Move your cursor around randomly in order to "salt" your key, while the key is being generated. Once the key is generated, you should see something like this:

3. Replace the text in the 'Key comment:' field with something recognizable and enter a passphrase in the two fields below.

4. Copy the text in the 'Public key for pasting...' field (the text continues past what is displayed) and paste it wherever the public key is needed. If you are using GitHub, you can now create a new SSH key in your Personal Settings and paste this text into the 'Key' field.

5. Click on 'Save private key' and select a logical/recognizable name and directory for the file. Your private key is saved in the selected file.

6. Open Pageant (also part of the PuTTY package). If a message saying "Pageant is already running" is displayed, open your system tray and double click on the Pageant icon.

7. Click on 'Add Key' and select the file you saved when generating your key earlier (Step 5). If it is requested, enter the passphrase you created at Step 3 to complete the process.

When regularly connecting to multiple remote systems over SSH, you’ll find that remembering all the hosts and various command-line options becomes tedious. OpenSSH allows setting up a configuration file to store different SSH options for each remote machine you connect t.

SSH Config File Location

OpenSSH client-side (in this case your personal computer) configuration file is named config, and it is stored in the hidded .sshdirectory under your user’s home directory (i.e., ~/.ssh)

When you use the command for the first time. The ~/.ssh directory is automatically created. If the directory doesn’t exist on your system, create it using the command below:

By default, the SSH configuration file may not exist, so you may need to create it using the :

This file must be readable and writable only by the user and not accessible by others:

SSH Config File Structure Basics

The SSH Config File takes the following structure:

The contents of the SSH config file is organized into sections. Each section starts with the Host directive and contains specific SSH options used when establishing a connection with the remote SSH server.

Oscar Hosts

Here we peovide a list of Oscar hosts and typical SSH configuration options. You have two options

1. Copy the list of hosts below directly into your SSH Config File (i.e., ~/.ssh/config)

2. Keep this content in a spearate file for Oscar hosts, lets say ~/.ssh/config.oscar and include that file in your main configuration file. In this case, the first line of ~/.ssh/config will be Include "~/.ssh/config.oscar"

Connecting to your preconfigured host

You may now connect using the shortchut notation provided by your configuration file. That is, all you need to type is:

According to the configuration above, this is equivalent to

Much shorter. Enjoy!

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. An Oscar account is required to access Open OnDemand. Visit this link in a web browser and sign in with your Brown username and password to access this portal.

Intro to Open OnDemand Slides

OOD provides with a several resources for interacting with Oscar.

Features:

1. No installation needed. Just use your favorite browser!

3. No need to use two-factor authentication multiple times. Just do it once, when you log into OOD.

4. Use it with, or without, VPN. Your workflow remains the same.

You can launch several different apps on the Open OnDemand (OOD) interface. All of these apps start of a Slurm batch job on the Oscar cluster with the requested amount of resources. These jobs can access the filesystem on Oscar and all output files are written to the Oscar's file system.

Launching an App on OOD

1. Open on any browser of the your choice

2. If prompted, enter your Brown username and password.

3. Click on the "Interactive Apps" tab at the top of the screen to see the list of available apps. This will open the form to enter the details of the job.

4. Follow the instructions on the form to complete it. Some of fields can be left blank and OOD will choose the default option for you.

5. Click Launch to submit an OOD job. This will open a new tab on the browser It may take a few minutes for this job to start.

6. Click "Launch <APP>" again if prompted in the next tab.

Open OnDemand offers a browser-based terminal app to access Oscar. Windows users who do not want to install an SSH client like Putty will find this app very useful.

Accessing the terminal

1. Log in to

2. In the top menu, click Clusters -> >_OSCAR Shell Access

A new tab will open and the web-based terminal app will be launched in it. The shell will be launched on one of the login nodes.

Features:

1. No installation needed. Just use your favorite browser!

2. No need to enter your password again. SSH into Oscar in seconds!

3. No need to use two factor authentication again. Just do it once, when you log into OOD.

4. Use it with, or without, VPN. Your workflow remains the same.

We recommend all users to install Python packages within an environment. This can be a Conda to a python virtual environment. More information can be found . Follow these steps to use such environments in the .

Python Environments:

One Time Setup:

1. Open a terminal on Oscar.

2. Load the relevant python module and create and/or activate the environment. See this page for more information about creating .

3. Run pip install notebook to install Jupyter notebook, if not already installed.

4. Run pip install ipykernel to install ipykernel in this environment.

5. Run python -m ipykernel install --user --name=<myenv> where <myenv> is the name of the environment.

Launching Jupyter Notebook

1. Open the "Basic Jupyter Notebook for Python Environments" app on the Open OnDemand interface

2. Under "Python Module on Oscar", choose the python module you loaded when the environment was created.

3. Under "Python Virtual Environment", add the name of the Virtual Environment you created. Note: If your virtual environment is not at the top level of your home directory, you should input the absolute path to the environment directory.

4. Under the "Modules" , enter the name of the python module used to create the environment. Add any additional modules you may need separated with a space.

5. Choose the other options as required.

6. Click "Launch" to start the job

7. Click "Connect to Jupyter" on the next screen.

8. To start a new notebook, click "New" -> <myenv> where <myenv> is the environment.

9. For starting a pre-existing notebook, open the notebook. In the Jupyter interface, click "Kernel" -> "Change Kernel" -> <myenv> where myenv is the name of the environment.

Conda Environments

One Time Setup:

1. Open a terminal on Oscar.

2. Activate the conda environment.

3. Run pip install notebook to install Jupyter notebook, if not already installed.

4. Run pip install ipykernel to install ipykernel in this environment.

5. Run python -m ipykernel install --user --name=<myenv> where <myenv> is the name of the environment.

Launching Jupyter Notebook

1. Open the "Basic Jupyter Notebook with Anaconda" app on the Open OnDemand interface

2. Under "Oscar Anaconda module", choose "anaconda/2020.02"

3. Enter the name of the conda environment in "Conda Env"

4. Choose the other options as required.

5. Click "Launch" to start the job

6. Click "Connect to Jupyter" on the next screen.

7. To start a new notebook, click "New" -> <myenv> where <myenv> is the environment.

8. For starting a pre-existing notebook, open the notebook. In the Jupyter interface, click "Kernel" -> "Change Kernel" -> <myenv> where myenv is the name of the environment.

is a cgroups-based mechanism that is designed to prevent the misuse of login nodes and VSCode node, which are scarce, shared resources. It is installed on shared nodes listed below:

• login005

• login006

• node1103 (VSCode)

Status and Limits

Arbiter2 applies different limits to a user's processes depending on the user's status: normal, penalty1, and penalty2.

Normal Status and Limits

Upon first log in, the user is in the normal status. These normal limits apply to all the user's processes on the node:

Penalty1 Status and Limits

When a user's processes consume CPU time more than the default CPU time limit for a period of time, the user's status is changed to the penalty1 status. These penalty1 limits are applied:

While a user is in penalty1 status, their processes are throttled if they consume more CPU time than penalty1 limit. However, if a user's processes exceed penalty1 memory limit, the processes (PIDs) will be terminated by cgroups.

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty1 limit for 30 minutes.

Penalty2 Status and Limits

When a user's processes consume more CPU time than the penalty1 limit for a period of time, the user is put in the penalty2 status, and the penalty2 limits apply to the user's processes.

In penalty2 status, the user's processes will be throttled if they consume more CPU time than penalty2 limit. However, if a user's processes exceed penalty2 memory limit, the processes (PIDs) will be terminated by cgroups.

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty2 limit for one hour.

Penalty3 Status and Limits

When a user's processes consume more CPU time than the penalty2 limit for a period of time, the user is put in the penalty3 status. These penalty3 limits apply to the user's processes.

In penalty3 status, the user's processes will be throttled if they consume more CPU time than penalty3 limit. If a user's processes exceed penalty3 memory limit, the processes (PIDs) will be terminated by cgroups

The user's status returns to the normal status after a user's processes consume CPU time less than the penalty3 limit for two hours.

Email Notification

A user receives an email notification upon each violation. Below is a example email:

Violation of usage policy

This may indicate that you are running computationally-intensive work on the interactive/login node (when it should be run on compute nodes instead). Please utilize the 'interact' command to initiate a SLURM session on a compute node and run your workloads there.

You now have the status penalty1 because your usage has exceeded the thresholds for appropriate usage on the node. Your CPU usage is now limited to 80% of your original limit (8.0 cores) for the next 30 minutes. In addition, your memory limit is 80% of your original limit (40.0 GB) for the same period of time.

These limits will apply on login006.

High-impact processes

Usage values are recent averages. Instantaneous usage metrics may differ. The processes listed are probable suspects, but there may be some variation in the processes responsible for your impact on the node. Memory usage is expressed in GB and CPU usage is relative to one core (and may exceed 100% as a result).

Recent system usage

*This process is generally permitted on interactive nodes and is only counted against you when considering memory usage (regardless of the process, too much memory usage is still considered bad; it cannot be throttled like CPU). The process is included in this report to show usage holistically.

**This accounts for the difference between the overall usage and the collected PID usage (which can be less accurate). This may be large if there are a lot of short-lived processes (such as compilers or quick commands) that account for a significant fraction of the total usage. These processes are whitelisted as defined above.

Required User Actions

When a user receives an alert email that the user is put in a penalty status, the user should

• kill the processes that use too much resources on the shared node listed in the alert email, and/or reduce the resources used by the processes

Exempt Processes

Essential Linux utilities, such as rsync, cp, scp, SLURM commands, creating Singularity images, and code compilation, are exempt. To obtain a comprehensive list, please get in touch with us

CCV users can access their home, data, and scratch directories as a local mount on their own Windows, Mac, or Linux system using the Common Internet File System (CIFS) protocol (also called Samba). This allows you to use applications on your machine to open files stored on Oscar. It is also a convenient way to move files between Oscar and your own machine, as you can drag and drop files.

Users should ensure that the date and time are set correctly on their machine. Now you are ready to mount your CCV directories locally. Instructions for each of the various operating systems are given below.

macOS

1. In the Finder, press "Command + K" or select "Connect to Server..."

   from the "Go" menu.

2. For "Server Address", enter smb://smb.ccv.brown.edu/<volume>/

   and click "Connect".

   • To access your Home directory, entersmb://smb.ccv.brown.edu/home/

   • To access your Scratch space, entersmb://smb.ccv.brown.edu/scratch/

   • To access your Data directory, entersmb://smb.ccv.brown.edu/data/<pi_group>/

     • To check your PI group run 'groups' command.

3. Enter your AD username and password. If you have trouble connecting, enter <username>@ad.brown.edu as your Username

4. You may choose to add your login credentials to your keychain so you will not need to enter this again.

Optional. If you would like to automatically connect to the share at startup:

1. Open "System Preferences" (leave the Finder window open).

2. Go to "Accounts" > "(your account name)".

3. Select "Login Items".

4. Drag your data share from the "Finder" window to the "Login Items" window.

Linux

1. Install the cifs-utils package:
2. Make a directory to mount the share into:
3. Create a credentials file and add your AD account information:
4. Allow only root access to the credentials files:
5. Add an entry to thefstab:
6. Thefstabentry should be following:
7. Replace<localUser>to the login used on your Linux workstation, and replace <user> and <pi_group> with your Oscar username and PI group, respectively.

8. Mount the share:

Windows

1. Right-click "Computer" and select "Map Network Drive"

2. Select an unassigned drive letter

3. To mount specific volumes:

• For Home directory, enter\\smb.ccv.brown.edu\home\

• For Scratch space, enter\\smb.ccv.brown.edu\scratch\

• For Data directory, enter\\smb.ccv.brown.edu\data\<pi_group>\

  • To check your<pi_group>run 'groups' command.

1. Check "Connect using different credentials"

2. Click "Finish"

3. Enter your AD user name. If your computer is not in Active Directory (AD), you should enter your username in the format ad\username

4. Enter your AD password and click "OK"

You can now access your home directory through Windows Explorer with the assigned drive letter. Your data and scratch directories are available as the subdirectories (~/data and ~/scratch) of your home directory.

1. If you have an existing virtual environment, proceed to step 2. Otherwise, to create a new virtual environment:

2. Search for Python.VenvPath as shown in the picture below:

3. VSCode expects you to have multiple virtual environments for each of your different python projects, and it expects you to put them all in the same directory. Pointing to the parent directory lets it scan and find all expected virtual environments, and then you can easily toggle between them in interface.

4. Once you have the virtual environment selected, the debugging capabilities should work.

RStudio is an IDE for R that can be run on Oscar.

Launching RStudio

Open the Open On Demand Dashboard by following this link. Select RStudio (under "Default GUI's"). Fill in the form to allocate the required resources, and optionally select your R modules. Finally, click the "Launch Session" button.

Known Issues

Plotting figures may not work within RStudio. If this is the case, save the plots to a file, and view them through the Open On Demand Desktop App. If plots are required for your task, launch RStudio through the Desktop App.

CCV uses all-flash parallel filesystem (Vast Data). Users have a home, data, and scratch space.

home ~

• 100GB of space

• Optimized for many small files

• 30 days snapshots

• The quota is per individual user

• A grace period of 14 days

data ~/data

• Each PI gets 256GB for free

• Optimized for reading large files

• 30 days snapshots

• The quota is by group

• A grace period of 14 days

scratch ~/scratch

• 512G (soft-quota): 12T (hard-quota)

• Optimized for reading/writing large files

• 30 days snapshots

• Purging: Files not accessed for 30 days may be deleted

• The quota is per individual user

• A grace period of 21 days

A good practice is to configure your application to read any initial input data from ~/data and write all output into ~/scratch. Then, when the application has finished, move or copy data you would like to save from ~/scratch to ~/data.

To see how much space on your directories, you can use the command checkquota. Below is an example output

$ checkquota
Name       Path                 Used(G)    (%) Used   SLIMIT(G)  H-LIMIT(G) Used_Inodes     SLIMIT     HLIMIT     Usage_State  Grace_Period  
ccvdemo1   /oscar/home          3.72       2          100        140        63539           2000000    3000000    OK           None          
ccvdemo1   /oscar/scratch       0.00       0          512        10240      1               4000000    16000000   OK           None          
Now fetching Data directory quotas...
Name        Used(T)   (%) Used   SLIMIT(T)   HLIMIT(T)   Used_Inodes   SLIMIT    HLIMIT    Usage_State   Grace_Period  
data+nopi   0.0       0          0.88        0.98        466           4194304   6291456   OK            None 

Nightly snaphots of the file system are available for the last 30 days.

Restore a file from a snapshot in the last 30 days

Nightly snapshots of the file system are available for the last 30 days can be found in the following directories.

Home directory snapshot

/oscar/home/.snapshot/Oscar_<yyyy-mm-dd>_00_00_00_UTC/<username>/<path_to_file>

Data directory snapshot

/oscar/data/.snapshot/Oscar_<yyyy-mm-dd>_00_00_00_UTC/<groupname>/<username>/path_to_file>

Scratch directory snapshot

/oscar/scratch/.snapshot/Oscar_Daily_<yyyy-mm-dd>_00_00_00_UTC/<username>/<path_to_file>

To restore a file, copy the file from the snapshot to your directory.

e.g.:

ls -l /oscar/home/.snapshot/Osar_2023-06-22_00_00_00_UTC/ghopper/data
lrwxrwxrwx 1 ghopper navy 22 Mar  1  2016 /oscar/home/.snapshot/Osar_2023-06-22_00_00_00_UTC/ghopper/data -> /oscar/data/navy

Oscar is a shared machine used by hundreds of users at once. User requests are called jobs. A job is the combination of the resource requested and the program you want to run on the compute nodes of the Oscar cluster. On Oscar, Slurm is used to schedule and manage jobs.

Jobs can be run on Oscar in two different ways:

• Interactive jobs allow the user to interact with programs (e.g., by entering input manually, using a GUI) while they are running. However, if your connection to the system is interrupted, the job will abort. Small jobs with short run times and jobs that require the use of a GUI are best-suited for running interactively.

• Batch jobs allow you to submit a script that tells the cluster how to run your program. Your program can run for long periods of time in the background, so you don't need to be connected to Oscar. The output of your program is continuously written to an output file that you can view both during and after your program runs.

Jobs are scheduled to run on the cluster according to your account priority and the resources you request (i.e., cores, memory, and runtime). In general, the fewer resources you request, the less time your job will spend waiting in the queue.

A job array is a collection of jobs that all run the same program, but on different values of a parameter. It is very useful for running parameter sweeps, since you don't have to write a separate batch script for each parameter setting.

To use a job array, add the option:

#SBATCH --array=<range>

in your batch script. The range can be a comma separated list of integers, along with ranges separated by a dash. For example:

1-20
1-10,12,14,16-20

A job will be submitted for each value in the range. The values in the range will be substituted for the variable $SLURM_ARRAY_TASK_ID in the remainder of the script. Here is an example of a script for running a serial Matlab script on 16 different parameters by submitting 16 different jobs as an array:

#!/bin/bash
#SBATCH -J MATLAB
#SBATCH -t 1:00:00
#SBATCH --array=1-16

# Use '%A' for array-job ID, '%J' for job ID and '%a' for task ID
#SBATCH -e arrayjob-%a.err
#SBATCH -o arrayjob-%a.out

echo "Starting job $SLURM_ARRAY_TASK_ID on $HOSTNAME"
matlab -r "MyMatlabFunction($SLURM_ARRAY_TASK_ID); quit;"

You can then submit the multiple jobs using a single sbatch command:

$ sbatch <jobscript>

The $SLURM_ARRAY_TASK_ID can be manipulated as needed. For example, you can generate a fixed length number form it. The following example generates a number of length of 3 from $SLURM_ARRAY_TASK_ID.

#!/bin/bash
#SBATCH -J MATLAB
#SBATCH -t 1:00:00
#SBATCH --array=1-16

# Use '%A' for array-job ID, '%J' for job ID and '%a' for task ID
#SBATCH -e arrayjob-%a.err
#SBATCH -o arrayjob-%a.out

echo "Starting job $SLURM_ARRAY_TASK_ID on $HOSTNAME"
t=`printf "%03d" $SLURM_ARRAY_TASK_ID`
matlab -r "MyMatlabFunction($t); quit;"

For more info: https://slurm.schedmd.com/job_array.html

Listing running and queued jobs

The squeue command will list all jobs scheduled in the cluster. We have also written wrappers for squeue on Oscar that you may find more convenient:

myq                   List only your own jobs.
allq                  List all jobs, but organized by partition, and a summary of the nodes in use in the
                      partition.
allq <partition>      List all jobs in a single partition.
myjobinfo            Get the time and memory used for your jobs.

Viewing estimated time until completion for pending jobs

squeue -u <your-username> -t PENDING --start

This command will list all of your pending jobs and the estimated time until completion.

Canceling jobs

scancel <jobid>

View details about completed jobs

sacct

The sacct command will list all of your running, queued and completed jobs since midnight of the previous day. To pick an earlier start date, specify it with the -S option:

sacct -S 2012-01-01

To find out more information about a specific job, such as its exit status or the amount of runtime or memory it used, specify the -l ("long" format) and -j options with the job ID:

sacct -lj <jobid>

(example)

myjobinfo

The myjobinfo command uses the sacct command to display "Elapsed Time", "Requested Memory" and "Maximum Memory used on any one Node" for your jobs. This can be used to optimize the requested time and memory to have the job started as early as possible. Make sure you request a conservative amount based on how much was used.

myjobinfo

Info about jobs for user 'mdave' submitted since 2017-05-19T00:00:00
Use option '-S' for a different date or option '-j' for a specific Job ID.

JobID    JobName                  Submit      State        Elapsed     ReqMem     MaxRSS
1861     ior 2017-05-19T08:31:01  COMPLETED   00:00:09     2800Mc      1744K
1862     ior 2017-05-19T08:31:11  COMPLETED   00:00:54     2800Mc     22908K
1911     ior 2017-05-19T15:02:01  COMPLETED   00:00:06     2800Mc      1748K
1912     ior 2017-05-19T15:02:07  COMPLETED   00:00:21     2800Mc      1744K

ReqMem shows the requested memory: A c at the end of number represents Memory Per CPU, a n represents Memory Per Node. MaxRSS is the maximum memory used on any one node. Note that memory specified to sbatch using --mem is Per Node.

jobstats

The 'jobstats' utility is now available for analyzing recently completed jobs, comparing the resources used to those requested in the job script, including CPU, GPU, and memory. If email notifications are enabled, 'jobstats' sends an email with the results and includes a prompt to contact support for help with resource requests.

Run this command in a bash shell on Oscar. No additional module needs to be loaded.

jobstats <jobid>

To send this output to your email after the job is completed, make sure that these lines are in your job submit script

#SBATCH --mail-type=END
#SBATCH --mail-user=<email>

Partition Overview

Oscar has the following slurm partitions. The number and size of jobs allowed on Oscar vary with both partition and type of user account. You can email support@ccv.brown.edu if you need advice on which partitions to use‌.

To list partitions on Oscar available to your account, run the following command:

$ sinfo -O "partition"     

To view all partitions (including ones you don't have access to), replace the -O in the command above with -aO.

batch is the default partition.

Partition Details

Below are brief summary of partitions. For the details of nodes in partitions, please see here.

batch

• General purpose computing

• Priority is determined by account type (from highest

  to lowest: condo, priority, exploratory)

Condo limits apply to the group (i.e., they reflect the sum of all users on the condo). Condo users can check the limits on their condo with the command condos.

debug

• Short wait time, short run time access for debugging

• All users have the same limits and priority on the debug partition

vnc

• These nodes are for running VNC sessions/jobs

• Account type may affect Priority

gpu

• For GPU-based jobs

• GPU Priority users get higher priority and more resources than free users on the GPU partition

• Condo users submit to the gpu partition with normal or priority access (if they have a priority account in addition to their condo)

gpu-he

• For GPU-based jobs

• Uses Tesla V100 GPUs

• Restricted to High End GPU Priority users

gpu-debug

• Short wait time, short run time gpu access for debugging

• All users have the same limits and priority on the gpu-debug partition

bigmem

• For jobs requiring large amounts of memory

• Priority users get higher priority and more resources than free users on the bigmem partition

• Condo users submit to the bigmem partition with normal or priority access (if they have a priority account in addition to their condo)

• Premium users get higher priority and more resources than free users on the SMP partition

• Condo users submit to the SMP partition with normal or priority access (if they have a priority account in addition to their condo)

There are several ways to move files between your machine and Oscar. Which method you choose will depend on how much data you need to move and your personal preference for each method.

1. SMB

2. Command line (scp)

3. GUI application

4. Globus online (best for large transfers)

5. LFTP

1. SMB

You can drag and drop files from your machine to the Oscar filesystem via SMB. This is an easy method for a small number of files. Please refer to this page for mounting filesystem via SMB.

2. Command line

Mac and Linux

SCP

You can use scp to transfer files. For example to copy a file from your computer to Oscar:

scp /path/to/source/file <username>@ssh.ccv.brown.edu:/path/to/destination/file

To copy a file from Oscar to your computer:

scp <username>@ssh.ccv.brown.edu:/path/to/source/file /path/to/destination/file

RSYNC

You can use rsync to sync files across your local computer to Oscar:

rsync -azvp --progress path/to/source/directory <username>@ssh.ccv.brown.edu:/path/to/destination/directory

Windows On Windows, if you have PuTTY installed, you can use it's pscp function from the terminal.

3. GUI programs for transferring files using the sftp protocol and transfer.ccv.brown.edu hostname

• DUO is required if you are not connected to approved networks, e.g., home network

  • There is no interactive terminal message but your Phone will get a prompt automatically

• DUO is NOT required if you are connected to approved Brown networks

  • A personal Windows computer must have CrowdStrike installed in order to be on approved Brown networks.

In general, you can specify the following for your GUI programs:

• Protocol: SFTP

• Host: transfer.ccv.brown.edu

• User: your Oscar username

• Password: your Brown password

3.1 WinSCP for Windows

3.1.1 Limit Concurrent Transfer and Change Reconnect Options

Click the Optionsand then Preferences menu in WinsCP. In the poped up window, click Transfer and then Background to (Figure 1)

• change Maximal number of transfers at the same time to 1

• uncheck Use multiple connections for single transfer

click Endurace to (Figure 2)

• set Automatically reconnect session to 5 seconds

• uncheck Automatically reconnect session, if it stalls

• set Keep reconnection for to 10 seconds

3.1.2 Add a New Site

3.2 FileZilla

3.2.1. Disable Timeout

Click the Edit menu and then select the Settings submenu, and then change the Timeout in seconds to 0 to disable, as shown in Figure 2

3.2.2 Add a New Site

Open the Site Manager as show in Figure 5.

Click the 'New Site' button to add a new site, as shown in Figure 4:

Limit the number of simultaneous connections to 1, as shown in Figure 5.

Click the 'Connect' button to connect to Oscar and transfer files.

3.3 Cyberduck

You may see a popup window on 'Unknown Fingerprint'. You just need to check the 'Always' option and click 'Allow'. This is windows should not pop up again unless the transfer server is changed again.

4. Globus online

Globus is a secure, reliable research data management service. You can move data directly to Oscar from anothe Globus endpoint. Oscar has one Globus endpoint:

BrownU_CCV_Oscar

If you want to use Globus Online to move data to/from you own machine, you can install Globus Connect Personal. For more instructions on how to use Globus, see the Oscar section in the Globus documentation.

5. LFTP

LFTP is a sophisticated file transfer program supporting a number of network protocols (ftp, http, sftp, fish, torrent). It has bookmarks, a built-in mirror command, can transfer several files in parallel and was designed with reliability in mind. You can use the LFTP module from Oscar to transfer data from any (S)FTP server you have access to directly to Oscar. Below are the main LFTP commands to get you started:

module load lftp  # To load the LFTP module from Oscar
lftp -u login,passwd MyAwesomeUrl  # To connect to your (S)FTP server
ls   # To list files on the (S)FTP server
!ls  # To list files in your directory on Oscar
get MyAwesomeFile  # To download a single file
mirror # To download everything as is from the server
mirror --directory=/name_of_directory/ # To download a specific directory

1. Install the Remote Development extension pack for VSCode

2. Open VSCode settings

• On Windows/Linux - File > Preferences > Settings

• On macOS - Code > Preferences > Settings

Search for symlink and make sure the symlink searching is unchecked

3. Under VSCode settings, search for remote ssh timeout and manually enter a timeout value i.e. 50s. It should give you enough time to complete 2-Factor Authentication.

4. Edit the ~/.ssh/config file on your local machine, add the following lines. Replace <username> with your Oscar username.

# Jump box with public IP address
Host jump-box
  HostName ssh8.ccv.brown.edu
  User <username>
# Target machine with private IP address
Host ccv-vscode-node
  HostName vscode1
  User <username>
  ProxyCommand ssh -q -W %h:%p jump-box

6. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select login-node

The Desktop app on Open OnDemand is a replacement for the older VNC Java client. This app allows you to launch a Desktop GUI on Oscar.

Launching Desktop App (VNC)

0. Launch Open OnDemand

Click here to launch Open OnDemand (OOD) and log in with you Brown Credentials.

1. Select the Desktop option in Interactive Apps dropdown list:

2. Choose the resource option:

3. Wait and Launch!

Reconnecting to session

A session may get disconnected if it is not active for a while:

If the session disconnects as shown above, please don't click the "Connect" button on the screen. You may go to Open OnDemand page and click “My Interactive Sessions” to find the session again:

Copying and pasting text

Desktop (Advanced)

If you need more or different resources than those available from the default Desktop session, you should use the Advanced Desktop app. Resources requested here count against the resources allowed for your Oscar account.

1. Select the Desktop (Advanced) app under Interactive Apps.

2. Choose required resources

Fill out the form with your required resources.

• Account: Enter your condo account name. If you are not a member of a condo, leave this blank

• Desktop Environment: Choose XFCE. KDE works for CPU jobs, but may not be able to use GPU acceleration correctly.

• Number of hours: Choose appropriately. Your Desktop session will end abruptly after this time has lapsed. Requesting a very long session will result in a lower job priority.

• Partition: Equivalent to #SBATCH -p option. The desktop session will run on this partition.

• Num Cores: Equivalent to the #SBATCH -n option.

• Num GPUs: Equivalent to the #SBATCH --gres=gpu: option. This field is ignored if the partition does not have any GPU nodes, e.g. batch

• Memory (GB): Equivalent to the #SBATCH --mem= option.

• Reservation: Equivalent to the #SBATCH --reservation= option. Leave blank if you are not using a reservation.

3. Wait and Launch!

Wait and launch this session like the regular Desktop session.

Modify the Terminal App

Inside the Desktop session , click on Applications in the top left

Applications -> Settings -> Default Applications

In the new Window, click on the "Utilities" tab and choose "Gnome Terminal" in the drop down menu under "Terminal Emulator"

Then click on "Applications -> Terminal Emulator" to launch the terminal:

If the steps mentioned above do not work:

1. Close the Desktop session

2. Inside a terminal (outside the Desktop session), run this command:

rm -r ~/.ood_config

1. Start a new desktop session .

Change the Terminal icon for launcher panel

Please drag and drop the "Terminal Emulator" icon from the "Applications" menu to the launcher panel at the bottom of the screen, it will be inserted to the launcher panel:

Then click on "Create Launcher":

You may remove the old terminal icon after adding the new icon:

The filesystem on Oscar can be accessed through the file explorer on this web portal. The file explorer allows you

• List files

• Create a directory

• Rename files

• Edit plain-text files

• Upload files or directories

• Download files or directories

• Copy/Move files

• Launch a terminal from the current directory

To access the file explorer, click "Files" -> "Home Directory" at the top of the screen.

Check the documentation below for some of these services below:

Changing directories on File explorer

To access a directory, click "Change directory" and enter the path name

• To access your home directory, click the "Home Directory" link on the left. The path name at the top of the page should change to "/users/<username>"

• To access your scratch directory, click the "scratch" directory in your home directory OR click "Change directory" and enter "/users/<username>/scratch"

• To access your data directory, click the "data" directory in your home directory OR click "Change directory" and enter "/users/<username>/data"

Edit plain-text files

1. Navigate to the directory that contains the plain-text file.

2. Click the icon with the three dots -> Edit

3. The file will open in a text editor in a new tab

Download files or directories

1. Navigate to the directory that contains the file or directory.

2. Click the icon with the three dots -> Download

To download multiples files:

1. Click the check-box to the left of the file name.

2. Scroll to the top of the page and click "Download"

Directories are downloaded as zipped files on your computer.

Upload files or directories

1. Navigate to the directory where you need to upload the files.

2. Click the "Upload" button.

3. Follow the instructions on the screen. You can click the "Browse" buttons or drag and drop files.

Launch a terminal

1. Navigate to the directory where you would like to open the terminal

2. Click "Open in Terminal" at the top of the page.

3. A web-based terminal will open in a new tab of your browser. You will be logged into one of the login nodes.

VSCode one-time setup

To use VSCode you will need to be connected to the VPN. Please install the Brown VPN client before proceeding.

1. Install the Remote Development extension pack for VSCode:

2. Open VSCode settings and uncheck symlink:

Search for symlink and make sure the symlink searching is unchecked

3. Make sure you have set up passwordless SSH authentication to Oscar. If you haven't, please refer to this documentation page.

4. Edit the config file:

C:\Users\<uname>\.ssh\config

Edit the config file on your local machine, add the following lines. Replace <username> with your Oscar username.

# Jump box with public IP address
Host jump-box
    HostName poodcit4.services.brown.edu
    User <username>

# Target machine with private IP address
Host ccv-vscode-node
    HostName vscode1
    User <username>
    ProxyCommand ssh -q -W %h:%p jump-box

1. September 10, 2023: Some users have reported issues while connecting to the Oscar VSCode remote extension. This is due to a recent change introduced by VSCode. To address this issue

Ctrl (cmd on Mac) + Shift + P > Remote-SSH: Settings
Disable the Remote.SSH: Use Exec Server option

1. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select ccv-vscode-node

6. Install and set up of VSCode

1. Configure VSCode

Connect to VSCode first.

You can either create a symlink via the ln command below,

cp -v /gpfs/runtime/opt/vscode-server/ccv-vscode-config/settings.json /users/$USER/.vscode-server/data/Machine/settings.json

or manually create /users/$USER/.vscode-server/data/Machine/settings.json file with following contents

{
    "files.watcherExclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/**": true,
        "/usr/local/**": true,
        "/gpfs/home/**": true,
        "/gpfs/data/**": true,
        "/gpfs/scratch/**": true
    },
    "search.followSymlinks": false,
    "search.exclude": {
        "**/.git/objects/**": true,
        "**/.git/subtree-cache/**": true,
        "**/node_modules/**": true,
        "/usr/local/**": true,
        "/gpfs/home/**": true,
        "/gpfs/data/**": true,
        "/gpfs/scratch/**": true
    }
}

Reconnect to VSCode

1. Click the green icon "Open a Remote Window" in the bottom left corner of VSCode Window. Then click "Connect to Host" in the drop down list.

2. Select the ccv-vscode-node option to connect to Oscar.

Git Overview

Version Control refers to the management of changes made to source code or any such large amount of information in a robust manner by multiple collaborators. Git is by far the most popular version control system.

Git enables effective collaboration among developers. In a team setting, multiple developers often work on the same project simultaneously. With Git, each developer can work on their own local copy of the project, making changes and experimenting freely without affecting the main codebase. Git allows developers to merge their changes seamlessly, ensuring that modifications made by different individuals can be consolidated efficiently. It provides mechanisms to track who made specific changes, making it easier to understand the evolution of the project and identify potential issues.

Git Workflow

Nearly all operations that are performed by Git are in you local computing environment, for the exception of few used purely to synchronize with a remote. Some of the most common git operations are depicted below. In summary a typical flow consists of making changes to your files, staging them via git add, marking a save point via git commit, then finally syncing to your remote (e.g., GitHub) via git push. If you are pushing changes to your remote from multiple places, you can bring changes your most recent version using git pull, which is the equivalent of doing git fetch followed by a git merge operation

Cheatsheet

Below are some of the most commonly used Git commands. You can also get much more information by running git --help. And if you'd like to learn more there is an excellent and thorough tutorial on Atlassian's website.

Git Configuration

While using Git on Oscar, make sure that you configure Git to have your correct Name and Email ID to avoid confusion while working with remote repositories (e.g., GitHub, GitLab, BitBucket).

$ git config --global user.name "John Smith“
$ git config --global user.email john@example.com

Getting Out of Trouble

Git can sometimes be a bit tricky. And we all eventually find ourselves in a place where we want to undo something or fix a mistake we made with Git. This website (pardon the profanity) has a bunch of really excellent solutions to common problems we sometimes run in to with Git.

To get started with NCDU, load the module using the following command:

module load ncdu/1.14

Once the module has been loaded, it can be used to easily show the size of all files within a directory:

ncdu my_directory