What username and password should I be using?

• If you are at Brown and have requested a regular CCV account, your Oscar login can be authenticated using your Brown credentials itself, i.e. the same username and password that you use to login to any Brown service such as "canvas".

• If you are an external user, you will have to get a sponsored ID at Brown through the department with which you are associated, before requesting an account on Oscar. Once you have the sponsored ID at Brown, you can on Oscar and use your Brown username and password to login.

Changing Passwords

Oscar users should use their Brown passwords to log into Oscar. Users should change their Brown passwords at .

Exploratory Account

• Exploratory accounts are available to all members of the Brown community for free.

• See thefor detailed description of the resources

• Jobs are submitted to the batch partition. See the page for available hardware

Priority Accounts

The following accounts are billed quarterly and offer more computational resources than the exploratory accounts. See thefor pricing and detailed description of the resources

HPC Priority

• Intended for users running CPU-intensive jobs. These offer more CPU and memory resources than an exploratory account

• Two types of accounts:

  • HPC Priority

Standard GPU Priority

• Intended for users running GPU intensive jobs. These accounts offer fewer CPU and memory resources but more GPU resources than an exploratory account.

• Two types of accounts:

  • Standard GPU Priority

High End GPU Priority

• Intended for GPU jobs required high-end gpus. These offer the same number of CPUS as Standard GPU priority accounts

• High end GPUS like A40, v100 and a6000 are available

• See thefor pricing and detailed description of the resources

Large Memory Priority

• Intended for jobs requiring large amounts of memory.

• These accounts offer 2TB of memory and twice the wall-time of exploratory accounts.

• See thefor pricing and detailed description of the resources

Condo

PIs who purchase hardware (compute nodes) for the CCV machine get a Condo account. Condo account users have the highest priority on the number of cores equivalent to the hardware they purchased. Condo accounts last for five years and give their owners access to 25% more CPU cores than they purchase for the first three years of their lifespan. GPU resources do not decrease over the lifetime of the condo.

Investigators may also purchase condos to grant access to computing resources for others working with them. After a condo is purchased, they can have users request to join the condo group through the "Request Access to Existing Condo" option on the on the CCV homepage.

Oscar - Ocean State Center for Advanced Resources - is Brown University's high performance computing cluster for both research and classes. Oscar is maintained and supported by .

Please contact if there are any questions on Oscar.

Accounts

If you do not have an Oscar account, you can request one by clicking the following link

If you publish research that benefited from the use of CCV services or resources, we would greatly appreciate an acknowledgment that states:

Efficient I/O is essential for good performance in data-intensive applications. Often, the file system is a substantial bottleneck on HPC systems, because CPU and memory technology has improved much more drastically in the last few decades than I/O technology.

Parallel I/O libraries such as MPI-IO, HDF5 and netCDF can help parallelize, aggregate and efficiently manage I/O operations. HDF5 and netCDF also have the benefit of using self-describing binary file formats that support complex data models and provide system portability. However, some simple guidelines can be used for almost any type of I/O on Oscar:

• Try to aggregate small chunks of data into larger reads and writes.

  For the GPFS file systems, reads and writes in multiples of 512KB

  provide the highest bandwidth.

Many scientific and HPC software packages are already installed on Oscar, and additional packages can be requested by submitting a ticket to [email protected]. If you want a particular version of the software, do mention it in the email along with a link to the web page from where it can be downloaded. You can also install your own software on Oscar.

CCV cannot, however, supply funding for the purchase of commercial software. This is normally attributed as a direct cost of research, and should be purchased with research funding. CCV can help in identifying other potential users of the software to potentially share the cost of purchase and maintenance. Several commercial software products that are licensed campus-wide at Brown are available on Oscar.

For software that requires a Graphical User Interface (GUI) we recommend using CCV's VNC Client rather than X-Forwarding.

SSH provides a method of sharing the ssh keys on your local machine with Oscar. This feature is called Agent Forwarding and can be useful for instance when working with version control or other services that authenticate via ssh keys. Below are instructions on how to configure your SSH connection to forward ssh-agent for diffeent operating systems

Matlab can be used within a batch script. Here is an example batch script for running a serial Matlab program on an Oscar compute node:

#!/bin/bash

# Request an hour of runtime:
#SBATCH --time=1:00:00

# Default resources are 1 core with 2.8GB of memory.

# Use more memory (4GB):
#SBATCH --mem=4G

# Specify a job name:
#SBATCH -J MyMatlabJob

# Specify an output file
#SBATCH -o MyMatlabJob-%j.out
#SBATCH -e MyMatlabJob-%j.out

# Run a matlab function called 'foo.m' in the same directory as this batch script.
matlab -r "run foo.m; exit"

This is also available in your home directory as the file:

~/batch_scripts/matlab-serial.sh

Note the exit command at the end which is very important to include either there or in the Matlab function/script itself. If you don't make Matlab exit the interpreter, it will keep waiting for the next command until SLURM cancels the job after running out of requested walltime. So for example, if you requested 4 hours of walltime and your actual program completes in 1 hour, the SLURM job will not complete until the designated 4 hours which results in idle cores and wastage of resources and also blocks up your other jobs.

If the name of your batch script file is matlab-serial.sh, the batch job can be submitted using the following command:

sbatch matlab-serial.sh

• Putty Installation

• SSH to Oscar from Linux

• SSH to Oscar from Mac

Agent Forwarding with PuTTY

1. Once adding your private key to Pageant, open PuTTY and navigate to the Auth menu.

2. Check the 'Allow agent forwarding' checkbox, and return to the Session menu.

3. Enter the Host Name you usually use to connect to Oscar, and click 'Open'.

4. Entering your password. If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Open OnDemand (OOD) is a web portal to the Oscar computing cluster. An Oscar account is required to access Open OnDemand. Visit this link in a web browser and sign in with your Brown username and password to access this portal.

Open on Demand

Intro to Open OnDemand Slides

OOD provides with a several resources for interacting with Oscar.

• Use the in the portal to view, copy, download or delete files on Oscar.

• Launch , like Matlab and Jupyter Notebook, inside your web browser.

• with your browser without needing a separate terminal emulator. This is especially handy for Windows users, since you do not need to install a separate program.

Features:

1. No installation needed. Just use your favorite browser!

2. No need to enter your password again. in seconds!

3. No need to use two-factor authentication multiple times. Just do it once, when you log into OOD.

Filing a Support Ticket

Filing a good support ticket makes it much easier for CCV staff to deal with your request

When you email [email protected] aim to include the following:

• State the problem/request in the subject of the email

• Describe which software and with version you are using

• Error message (if there was one)

• The job number

• How you were running, e.g. batch, interactively, vnc

• Give as as small an example as possible that reproduces the problem

Q&A Forum

Ask questions and search for previous problems at our .

Office Hours

CCV holds weekly office hours. These are drop in sessions where we'll have one or more CCV staff members available to answer questions and help with any problems you have. Please visit for upcoming office hours and events.

Arrange a Meeting

You can arrange to meet with a CCV staff member in person to go over difficult problems, or to discuss how best to use Oscar. Email [email protected] to arrange a consultation.

You can launch several different apps on the Open OnDemand (OOD) interface. All of these apps start of a Slurm batch job on the Oscar cluster with the requested amount of resources. These jobs can access the filesystem on Oscar and all output files are written to the Oscar's file system.

Launching an App on OOD

1. Open on any browser of the your choice

2. If prompted, enter your Brown username and password.

3. Click on the "Interactive Apps" tab at the top of the screen to see the list of available apps. This will open the form to enter the details of the job.

4. Follow the instructions on the form to complete it. Some of fields can be left blank and OOD will choose the default option for you.

5. Click Launch to submit an OOD job. This will open a new tab on the browser It may take a few minutes for this job to start.

6. Click "Launch <APP>" again if prompted in the next tab.

Modules

RStudio is an IDE for R that can be run on Oscar.

Launching RStudio

Open the Open On Demand Dashboard by following this link. Select RStudio (under "Default GUI's"). Fill in the form to allocate the required resources, and optionally select your R modules. Finally, click the "Launch Session" button.

Known Issues

Plotting figures may not work within RStudio. If this is the case, save the plots to a file, and view them through the Open On Demand Desktop App. If plots are required for your task, launch RStudio through the Desktop App.

Modules

The newest module miniconda/4.12.0 is recommeded.

$ module avail miniconda
miniconda/4.10    miniconda/4.12.0  

Conda

Please refere to for using a conda environment.

Mamba

Mamba is a drop-in replacement of conda, and is faster than conda.

Only activating and deactivating a conda environment still requires conda

For all other commands, conda can be replaced with mamba.

We have provided templates for you to use for job submission settings. These templates are in/gpfs/runtime/opt/forge/19.1.2/templates

Click Run and debug a program to open the following menu

Click Configure next to Submit to Queue and enter /gpfs/runtime/opt/forge/19.1.2/templates/slurm-ccv.qtf as the Submission template file

Step 1 : Check for existing SSH key pair

Before generating new SSH key pair first check if you have an SSH key on your local machine.

If there are existing keys, please move to Step 3

We recommend all users to install Python packages within an environment. This can be a Conda to a python virtual environment. More information can be found . Follow these steps to use such environments in the .

Python Environments:

Listing running and queued jobs

The squeue command will list all jobs scheduled in the cluster. We have also written wrappers for squeue on Oscar that you may find more convenient:

In this example, we will install Jax.

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

Here, -f = feature. We only need to build on Ampere once.

Step 2: Once your session has started on a compute node, run nvidia-smi to verify the GPU and then load the appropriate modules

Step 3: Create and activate the virtual environment

1. If you have an existing virtual environment, proceed to step 2. Otherwise, to create a new virtual environment:

2. Search for Python.VenvPath as shown in the picture below:

3. VSCode expects you to have multiple virtual environments for each of your different python projects, and it expects you to put them all in the same directory. Pointing to the parent directory lets it scan and find all expected virtual environments, and then you can easily toggle between them in interface.

Compiling with CUDA

To compile a CUDA program on Oscar, first load the CUDA with:

The CUDA compiler is called nvcc, and for compiling a simple CUDA program it uses syntax simlar to gcc:

Several versions of Python are available on Oscar as modules. For Python 2, we recommend using the python/2.7.16 module. For Python 3, we recommend using the python/3.7.4 module. These modules include the pip and virtualenv commands, but do not include other common Python packages (e.g., SciPy, NumPy). This affords individual users complete control over the packages they are using, thereby avoiding issues that can arise when code written in Python requires specific versions of Python packages.

To use the recommend Python modules, use the following commands to load the relevant module:

module load python/2.7.16

module load python/3.7.4

Nightly snaphots of the file system are available for the last 30 days.

In this example, we will install PyTorch (refer to sub-pages for TensorFlow and Jax).

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

CCV uses IBM's General Parallel File System (GPFS). Users have a home, data, and scratch space.

home ~

• 20GB of space

• Optimized for many small files

In this example, we will install TensorFlow.

Step 1: Request an interactive session on a GPU node with Ampere architecture GPUs

Here, -f = feature. We only need to build on Ampere once.

Step 2: Once your session has started on a compute node, run nvidia-smi to verify the GPU and then load the appropriate modules

Step 3: Create and activate the virtual environment

Here is an example script for running dependent jobs on Oscar.

There are 3 batch jobs. Each job has it's own batch script: job1.sh, job2,sh, jobs.sh. The script above (script.sh) submits the three jobs.

line 4: job1 is submitted.

line 7: job2 depends on job1 finishing successfully.

line 10: job3 depends on job2 finishing successfully.

Setting Up Gaussian

In order to use Gaussian on Oscar, you must be a part of the g09 group. To check your groups, run the groups command in the terminal.

You must first choose a Gaussian module to load. To see available Gaussian modules, run module avail gauss

You may use either Globus (recommended) or smbclient to transfer data between Oscar and Campus File Storage.

Globus

Follow .

Open OnDemand offers a browser-based terminal app to access Oscar. Windows users who do not want to install an SSH client like Putty will find this app very useful.

Accessing the terminal

Key Generation & Setup

1. Open PuTTYgen (this comes as part of the package), change the 'Number of bits in a generated key:' to 4096 (recommended), then click 'Generate'

You can explore GPU computing through Matlab if you think your program can benefit from massively parallel computations:

Finally, parallel computing features like parfor and spmd can be used by launching a pool of workers on a node.

Matlab programs often suffer from poor performance and running out of memory. Among other things, you can refer the following web pages for best practices for an efficient code:

The first step to speeding up Matlab applications is identifying the part which takes up most of the run time. Matlab's "Profiling" tool can be very helpful in doing that:

When connecting from a campus network to sshcampus.ccv.brown.edu you can set up SSH keys as a form of authentication instead of having to enter your password interactively. Follow the insctructions below that correspond to your operating system/connection method.

Oscar is a shared machine used by hundreds of users at once. User requests are called jobs. A job is the combination of the resource requested and the program you want to run on the compute nodes of the Oscar cluster. On Oscar, Slurm is used to schedule and manage jobs.

Jobs can be run on Oscar in two different ways:

• Interactive jobs allow the user to interact with programs (e.g., by entering input manually, using a GUI) while they are running. However, if your connection to the system is interrupted, the job will abort. Small jobs with short run times and jobs that require the use of a GUI are best-suited for running interactively.

• Batch jobs allow you to submit a script that tells the cluster how to run your program. Your program can run for long periods of time in the background, so you don't need to be connected to Oscar. The output of your program is continuously written to an output file that you can view both during and after your program runs.

Jobs are scheduled to run on the cluster according to your account priority and the resources you request (i.e., cores, memory, and runtime). In general, the fewer resources you request, the less time your job will spend waiting in the queue.

Git Overview

Version Control refers to the management of changes made to source code or any such large amount of information in a robust manner by multiple collaborators. Git is by far the most popular version control system.

Git enables effective collaboration among developers. In a team setting, multiple developers often work on the same project simultaneously. With Git, each developer can work on their own local copy of the project, making changes and experimenting freely without affecting the main codebase. Git allows developers to merge their changes seamlessly, ensuring that modifications made by different individuals can be consolidated efficiently. It provides mechanisms to track who made specific changes, making it easier to understand the evolution of the project and identify potential issues.

Git Workflow

Nearly all operations that are performed by Git are in you local computing environment, for the exception of few used purely to synchronize with a remote. Some of the most common git operations are depicted below. In summary a typical flow consists of making changes to your files, staging them via git add, marking a save point via git commit, then finally syncing to your remote (e.g., GitHub) via git push. If you are pushing changes to your remote from multiple places, you can bring changes your most recent version using git pull, which is the equivalent of doing git fetch followed by a git merge operation

Cheatsheet

Below are some of the most commonly used Git commands. You can also get much more information by running git --help. And if you'd like to learn more there is an

Git Configuration

While using Git on Oscar, make sure that you to have your correct Name and Email ID to avoid confusion while working with remote repositories (e.g., GitHub, GitLab, BitBucket).

Getting Out of Trouble

Git can sometimes be a bit tricky. And we all eventually find ourselves in a place where we want to undo something or fix a mistake we made with Git. (pardon the profanity) has a bunch of really excellent solutions to common problems we sometimes run in to with Git.

A job array is a collection of jobs that all run the same program, but on different values of a parameter. It is very useful for running parameter sweeps, since you don't have to write a separate batch script for each parameter setting.

To use a job array, add the option:

#SBATCH --array=<range>

in your batch script. The range can be a comma separated list of integers, along with ranges separated by a dash. For example:

1-20
1-10,12,14,16-20

A job will be submitted for each value in the range. The values in the range will be substituted for the variable $SLURM_ARRAY_TASK_ID in the remainder of the script. Here is an example of a script for running a serial Matlab script on 16 different parameters by submitting 16 different jobs as an array:

#!/bin/bash
#SBATCH -J MATLAB
#SBATCH -t 1:00:00
#SBATCH --array=1-16

# Use '%A' for array-job ID, '%J' for job ID and '%a' for task ID
#SBATCH -e arrayjob-%a.err
#SBATCH -o arrayjob-%a.out

echo "Starting job $SLURM_ARRAY_TASK_ID on $HOSTNAME"
matlab -r "MyMatlabFunction($SLURM_ARRAY_TASK_ID); quit;"

You can then submit the multiple jobs using a single sbatch command:

The $SLURM_ARRAY_TASK_ID can be manipulated as needed. For example, you can generate a fixed length number form it. The following example generates a number of length of 3 from $SLURM_ARRAY_TASK_ID.

For more info:

Installing R packages

Users should install R packages for themselves locally. This documentation shows you how to install R packages locally (without root access) on Oscar.

If the package you want to install has operating-system-level dependencies (i.e. the package depends on core libraries), then we can install it as a module.

Installing an R package

First load the R version that you want to use the package with:

Start an R session

Note some packages will require code to be compiled so it is best to do R packages installs on the login node.

To install the package 'wordcloud':

You will see a warning:

Answer y . If you have not installed any R packages before you will see the following message:

Answer y . The package will then be installed. If the install is successful you will see a message like:

If the installation was not successful you will see a message like:

There is normally information in the message that gives the reason why the install failed. Look for the word ERROR in the message.

Possible reasons for an installation failing include:

• Other software is needed to build the R package, e.g. the R package rgdal needs gdal so you have to do module load gdal

• A directory needs deleting from a previous failed installation.

Removing an R package

Start an R session:

To remove the 'wordcloud' package:

The new Ampere architecture GPUs do not support older CUDA modules. Users must re-compile their applications with the newer CUDA/11 or older modules. Here are detailed instructions to compile major frameworks such as PyTorch, and TensorFlow.

PyTorch

Users can install PyTorch from a pip virtual environment or use pre-built singularity containers provided by Nvidia NGC.

To install via virtual environment:

To use NGC containers via Singularity :

• Pull the image from NGC

• Export PATHs to mount the Oscar file system

• To use the image interactively

• To submit batch jobs

screen is a "terminal multiplexer", it enables a number of terminals (or windows) to be accessed and controlled from a single terminal. screen is a great way to save an interactive session between connections to oscar. You can reconnect to the session from anywhere!

Screen commands

Common commands are:

• start a new screen session with session name: screen -S <name>

• list running sessions/screens: screen -ls

• attach to session by name: screen -r <name>

• detach: Ctrl+a d

• detach and logout (quick exit): Ctrl+a d d

• kill a screen session: screen -XS session_name quit

Reconnecting to your screen session

In order to reconnect to a running screen session, you need to be connected to the same login node that you launched your screen session from. In order to locate and identify your screen sessions correctly, we recommed the following:

• Create a directory to store the information of your screen sessions. You only need do this once.

• Put the following line into your /.bashrc. This tells the screen program to save the information of your screen sessions in the directory created in the previous step . This allows you to query your screen sessions across different login nodes. To make this change effective in your current sessions, you need run 'source /.bashrc' in each of your current session . However, you do not need to run 'source /bashrc' in your new sessions.

• Name your new screen session using the name of the login node. For instance, start your screen with a commnd similar to

On Oscar, the command matlab is actually a wrapper that sets up MATLAB to run as a single-threaded, command-line program, which is the optimal way to pack multiple Matlab scripts onto the Oscar compute nodes.

To run the actual multi-threaded version with JVM and Display enabled, use:

Similarly, to run this without the display enabled:

VNC

The client provided by CCV is the best way to launch GUI applications on Oscar, including Matlab. From the terminal emulator in VNC use the matlab-threaded command to launch the Matlab GUI. For example,

Here is a snapshot of what it looks like:

X11 Forwarding

You can also run the MATLAB GUI in an X-forwarded interactive session. This requires installing an X server on your workstation/PC and logging in to Oscar with enabled. Use the interact command to get interactive access to a compute node. Again, for launching the GUI, you need to use the matlab-threaded command, which enables the display and JVM. You may however experience a lag in response from the Matlab GUI in an X forwarded session. Note that if Matlab does not find the X window system available, it will launch in command line mode (next section).

CIFS

A workaround in some situations may be to use to mount the Oscar filesystem on your PC and using the Matlab installation on your computer. For example, if you have your simulation results residing on Oscar, this might be a quick way to do post-processing on the data instead of having to move the data to your computer or using the Matlab GUI on Oscar. Note that users can connect to CIFS only from Brown computers or on Brown WiFi.

Start the SSH-Agent

First, start your ssh-agent with the command below.

You should see an output similar to this:

Add Key(s)

Next, add your ssh private keys to the running agent (using the ssh-add command on line 1). This step may be repeated for every key pair you use to connect to different git servers. For most, this file is called id_rsa and will live in ~/.ssh/id_rsa. If you set a password for your ssh keys, the agent will prompt you to enter them.

Confirm the ssh keys have been loaded into the agent with ssh-add -L:

Connect to Oscar

Now ssh into Oscar with the -A option as shown on the first line below (replace username with your Oscar username). -A will forward your ssh-agent to Oscar, enabling you to use the ssh keys on your laptop while logged into Oscar.

If you have ssh keys setup on your local computer to connect to GitHub, you can confirm your ssh-agent was properly forwarded by checking GitHub . If the ssh command fails, your agent has not been properly forwarded.

Always connecting with Agent Forwarding

To make these changes permanent, you can add the ForwardAgent yes option to your ssh configuration file. To learn more about configuring your ssh connections, visit

Combining CUDA and MPI

Mixing MPI (C) and CUDA (C++) code requires some care during linking because of differences between the C and C++ calling conventions and runtimes. One option is to compile and link all source files with a C++ compiler, which will enforce additional restrictions on C code. Alternatively, if you wish to compile your MPI/C code with a C compiler and call CUDA kernels from within an MPI task, you can wrap the appropriate CUDA-compiled functions with the extern keyword, as in the following example.

These two source files can be compiled and linked with both a C and C++ compiler into a single executable on Oscar using:

is available on Oscar. There are two products, DDT (debugger) and MAP (performance reports).

We recommend you use the Arm Forge remote client to launch your debugging jobs on Oscar. The first time you set up Arm Forge you will need to configure the client with the following steps:

1. Download the on your machine.

Oscar Specifications

Compute Nodes

Oscar has compute nodes in the partitions listed below.

• batch - The batch partition is for programs/jobs which need neither GPUs nor large memory.

• bigmem - The bigmem partition is for programs/jobs which require large memory.

• debug - The debug partition is for users to debug programs/jobs.

Below are node details including cores and memory for all partitions.

Hardware details

Hardware details for all partitions. The Features column shows the features available for the --constraint option for SLURM. This includes the available CPU types as well GPUs.

VSCode one-time setup

To use VSCode you will need to be connected to the VPN. Please install the before proceeding.

1. Install the for VSCode:

2. Open VSCode settings and uncheck symlink:

Search for symlink and make sure the symlink searching is unchecked

3. Make sure you have set up passwordless SSH authentication to Oscar. If you haven't, please refer to this .

4. Edit the config file:

Edit the config file on your local machine, add the following lines. Replace <username> with your Oscar username.

5. In VSCode, select Remote-SSH: Connect to Host… and after the list populates select ccv-vscode-node

6. Install and set up of VSCode

1. Configure VSCode

Connect to VSCode first.

You can either create a symlink via the ln command below,

or manually create /users/$USER/.vscode-server/data/Machine/settings.json file with following contents

Reconnect to VSCode

1. Click the green icon "Open a Remote Window" in the bottom left corner of VSCode Window. Then click "Connect to Host" in the drop down list.

2. Select the ccv-vscode-node option to connect to Oscar.

Common Linux Commands

The filesystem on Oscar can be accessed through the file explorer on this web portal. The file explorer allows you

• List files

• Create a directory

• Rename files

• Copy/Move files

To access the file explorer, click "Files" -> "Home Directory" at the top of the screen.

Check the documentation below for some of these services below:

Changing directories on File explorer

To access a directory, click "Change directory" and enter the path name

• To access your home directory, click the "Home Directory" link on the left. The path name at the top of the page should change to "/users/<username>"

• To access your scratch directory, click the "scratch" directory in your home directory OR click "Change directory" and enter "/users/<username>/scratch"

• To access your data directory, click the "data" directory in your home directory OR click "Change directory" and enter "/users/<username>/data"

Edit plain-text files

1. that contains the plain-text file.

2. Click the icon with the three dots -> Edit

3. The file will open in a text editor in a new tab

Download files or directories

1. that contains the file or directory.

2. Click the icon with the three dots -> Download

To download multiples files:

1. Click the check-box to the left of the file name.

2. Scroll to the top of the page and click "Download"

Directories are downloaded as zipped files on your computer.

Upload files or directories

1. where you need to upload the files.

2. Click the "Upload" button.

3. Follow the instructions on the screen. You can click the "Browse" buttons or drag and drop files.

Launch a terminal

1. where you would like to open the terminal

2. Click "Open in Terminal" at the top of the page.

3. A web-based terminal will open in a new tab of your browser. You will be logged into one of the login nodes.

Partition Overview

Oscar has the following slurm partitions. The number and size of jobs allowed on Oscar vary with both partition and type of user account. You can email [email protected] if you need advice on which partitions to use‌.

To list partitions on Oscar available to your account, run the following command:

To view all partitions (including ones you don't have access to), replace the -O in the command above with -aO.

batch is the default partition.

Partition Details

Below are brief summary of partitions. For the details of nodes in partitions, please see .

batch

• General purpose computing

• Priority is determined by account type (from highest

  to lowest: condo, priority, exploratory)

Condo limits apply to the group (i.e., they reflect the sum of all users on the condo). Condo users can check the limits on their condo with the command condos.

debug

• Short wait time, short run time access for debugging

• All users have the same limits and priority on the debug partition

vnc

• These nodes are for running VNC sessions/jobs

• Account type may affect Priority

gpu

• For GPU-based jobs

• GPU Priority users get higher priority and more resources than free users on the GPU partition

• Condo users submit to the gpu partition with normal or priority access (if they have a priority account in addition to their condo)

gpu-he

• For GPU-based jobs

• Uses Tesla V100 GPUs

• Restricted to High End GPU Priority users

gpu-debug

• Short wait time, short run time gpu access for debugging

• All users have the same limits and priority on the gpu-debug partition

bigmem

• For jobs requiring large amounts of memory

• Priority users get higher priority and more resources than free users on the bigmem partition

• Condo users submit to the bigmem partition with normal or priority access (if they have a priority account in addition to their condo)

Open-source software refers to any program whose source code is available for use or modification as users or other developers see fit. This is usually developed as a public collaboration and made freely available.

CMake and CCMake

Due to the complexity of some software, we often have to link to third party or external libraries. When working with software that has complicated building and linking steps, it is often impractical to use GCC (or your favorite compiler) directly. GNU Make is a build system that can simplify things somewhat, but "makefiles" can become unwieldy in their own way. Thankfully for us, there is a tool that simplifies this process.

CMake is a build system generator that one can use to facilitate the software build process. CMake allows one to specify—at a higher level than GNU Make—the instructions for compiling and linking our software. Additionally, CMake comes packaged with CCMake, which is an easy-to-use interactive tool that will let us provide build instructions to the compiler and the linker for projects written in C, Fortran, or C++. For more information about CMake and CCMake, please click .

Getting the source code from a Git Repository

Much of the time, source code is available on platforms such as GitHub, GitLab or BitBucket. Cloning (or downloading) the project from any of those is the same process. First, you need to get the URL from the repository. It usually looks like this:

GitHub repository

Bitbucket repository

Where username indicates the GitHub (or BitBucket, etc) account of the owner of the project, and project_name indicates, well, try to guess.

GitHub and BitBucket have a button at the top right side of the repository web page labeled "clone". Copy that URL

Clone The Repository

Create a new folder on a path with the necessary read/write permissions

Go inside inside that folder

Clone the repository:

URL is the repository's link mentioned above.

Getting the source code from a .tar or .zip file

In case you downloaded the project from a different source, and it is contained in a .tar or .zip file. Just extract the source code on a folder with the necessary read/write permissions.

Build the Project

Create a new folder and name it build

Go inside inside that folder

Execute CCMake pointing to the root folder which has a CMakeLists.txt file

The CCMake text interface will pop up with all the necessary attributes to build the software.

Set up the paths to the required libraries and press "c" to configure the project. Some errors might come up about CMake unable to find some specific libraries. This could be because that library does not exist in the system or you have not loaded the right module. Please contact CCV staff on how to fix this type of errors.

Make sure the attribute CMAKE_INSTALL_PREFIX points to a path with the necessary read/write permissions. By default it is set to the folder /usr/bin/ , which most of the users have no access to.

Once the configuration process has ended successfully, press "g" to generate the project. Generate the project does not mean compile or execute the program, please continue reading.

Compile the Project

Compile the project using the command make

You might want to increase the number of jobs compiling the software.

Once it is done, your project will be installed in the path set in the CMAKE_INSTALL_PREFIX attribute as explained above.

If you have any questions or need help please email [email protected].

The Desktop app on Open OnDemand is a replacement for the older VNC Java client. This app allows you to launch a Desktop GUI on Oscar.

When regularly connecting to multiple remote systems over SSH, you’ll find that remembering all the hosts and various command-line options becomes tedious. OpenSSH allows setting up a configuration file to store different SSH options for each remote machine you connect t.

SSH Config File Location

OpenSSH client-side (in this case your personal computer) configuration file is named config, and it is stored in the hidded .ssh

CCV users can access their home, data, and scratch directories as a local mount on their own Windows, Mac, or Linux system using the Common Internet File System (CIFS) protocol (also called Samba). This allows you to use applications on your machine to open files stored on Oscar. It is also a convenient way to move files between Oscar and your own machine, as you can drag and drop files.

For Python 2, we recommend using the python/2.7.16 module. For Python 3, we recommend using the python/3.7.4 module.

There are several ways for users to install python packages on Oscar

CCV uses the package for managing the software environment on OSCAR. The advantage of the modules approach is that it allows multiple versions of the same software to be installed at the same time. With the modules approach, you can "load'' and "unload'' modules to dynamically control your environment.

Using MPI4PY in a Python Script