Step-wise via Interact Session
🛑 We highly recommend using the Oscar utility script instead of this older approach.
Running XNAT2BIDS
The xnat-tools package provides a convenient xnat2bids script to facilitate data export and conversion to BIDS. The script is documented here, and the package documentation is useful for knowing the full list of inputs and defaults. We will demonstrate how to call that script within Oscar using the BNC's demo dataset.
0. Summary of commands
Before getting started, we have grouped all the commands executed in this page for your reference
interact -n 2 -t 01:00:00 -m 8g
version=v1.1.1
bids_root_dir=${HOME}/xnat-exports
mkdir -m 775 ${bids_root_dir} || echo "Output directory already exists"
simg=/oscar/data/bnc/simgs/brownbnc/xnat-tools-${version}.sif
XNAT_USER=${USER}
XNAT_SESSION="XNAT_E00114"
singularity exec --no-home --bind ${bids_root_dir} \
${simg} \
xnat2bids ${XNAT_SESSION} ${bids_root_dir} \
-u ${XNAT_USER} \
-i 71. Start an interactive session
1.1 Desktop app on Open OnDemand
Connecting via the Desktop app on Open OnDemand is a friendly way to request an graphical interactive session in Brown's supercomputer - Oscar. When you request a new Desktop session, you will be asked to specify the necessary resources. For this example, you can choose the basic job with 2 Cores and 7GB Memory. Once logged in, you are already inside an interactive session.
Once your requested session is running, you can launch it by clicking the Launch Desktop button.
At this point, simply open the terminal
1.2 SSH
To connect via SSH, you type ssh [email protected]. If this is your first time connecting via ssh, you will be asked to trust the remote computer (Oscar), your Brown credentials, and unless you are connected to VPN, you will be required to use DUO.
At this point you arrive at a login node. We will need to start an interactive session/job by typing
This starts an interactive job for one hour.
2. Define variables
We will now define a series of shell variables that will allow us to keep our commands more reusable. This will also be useful if you later decide to move to batch scripts, which provide a better way to run jobs simultaneously and without constant interaction.
When defining shell Variables, make sure to not have any spaces in the variable name, the assigned value or in between the equal sign
The variable will only be defined during the current shell session. If you close your terminal or interactive session, the variables will need to be redefined
If you want to copy-paste from these docs to the terminal in the OOD Desktop, click on the Copy to Clipboard Icon on the right of the code snippets. To paste into the terminal you will need to use the right click of your mouse and choose "Paste", or press Ctrl+Shift+V.
2.1 Define the version of xnat-tools
xnat-toolsWe recommend using the latest available version. You can get a list of the released versions here. The version specified here is likely the latest we have tested. If you test a newer version, we'd love your contributions to this documentation!
2.2 Set up paths
BIDS Root directory
This is the directory where data will be written. If the directory does not exist then it needs to be created.
For this sample walkthrough, you can use the path exactly as shown below. The ${USER} variable is a systemwide variable and it is automatically interpreted as your OSCAR user. However, once you are using your own data, you should export to directories in your PI's data folder, which typically follows the pattern /gpfs/data/<PI oscar user>
Path to Singularity Image
This is maintained by bnc and we will be pointing to the version defined above
XNAT USER and SESSION
Typically, your XNAT user is the same as your Brown user. Finding the session ID was explained in an earlier section. In this example we leverage the $USER variable to set your XNAT user. This is possible because both oscar username and XNAT username are typically the same (i.e your Brown username). For the session, we are using the accession number for participant 005 of the demo dataset
3. Understanding Singularity Containers
In the following section we will demonstrate how to run our software. Instead of directly installing the Python package xnat-tools we are going to run it from inside a container, which we have been referring to as "the singularity image/container". If you are new to containers, in a nutshell, a container allows for packaging all OS and package dependencies together so it can run in any computer. Docker containers have become very popular, you can learn a bit more about them here. Singularity is a type of container that has specialized on running on shared HPC clusters. You can learn little more about Singularity here. We referred to docker containers and singularity containers interchangeably, as they can easily be converted from one format to another.
In the specific case of xnat-tools , the wrapping container has Python , dcm2niix and heudiconv installed which are all needed by our software. The Dockerfile for xnat-tools provides some insight into how containers are built.
Understanding the file system of a container
Containers have their own file system, which is completely independent and isolated from the host where the container is run
Because a container does not have the same directory structure as the host, we have to remember that paths like /oscar/data/<PI> only exist in Oscar, but not inside the container
Sharing paths between a container and the host computer (OSCAR)
If we want a directory/file that exists in OSCAR to be available inside our container, we need to tell Singularity that. We do so, by binding a volume. This achieved by passing the flag --bind <oscar_path>:<container_path>. If we want the path inside the container to be exactly as the path in OSCAR, we can omit the destination path, that is --bind <oscar_path>.
The --bind flag can also be passed as -B
Singularity default behavior when it comes to sharing paths and environment variables
While generally speaking, a container is mostly isolated from the host, there are few exceptions. And these exceptions and default behaviors are different for Singularity and Docker. We will focus on Singularity rules here. By default Singularity binds the following locations
(Pro-Tip): Generally speaking, the default behavior of Singularity works great. Sometimes however, some of the configurations included in your $HOME or environment variables set in your $HOME/.bashrc may create conflicts. Singularity offers several flags that can be passed to further isolate the container from the local host. These include
If you see such flags in our examples and want to learn more about them visit this doc
4. Running the executable
Printing the help
Let's start by making sure that we can successfully run our xnat2bids executable inside the container.
Let's expand on the above command:
singularity: invokes singularity
exec: tells singularity we will be executing a command, in this case the command is xnat2bids
${simg}: is the singularity image/container that we will be using. We are passing the value of the variable we defined in Step 2. In our case, this is interpreted/evaluated as /oscar/data/bnc/simgs/brownbnc/xnat-tools-v1.1.1.sif
xnat2bids: is the command to be executed, and it is followed by any necessary inputs. In this case --help
If the above command is successful, you'll be seeing the following output in your terminal
Running XNAT2BIDS (test only on one series)
The following command will run the executable xnat2bids (via singularity) command to extract DICOMs from XNAT and export to BIDS.
Once again, let's expand on the command above:
singularity: invokes singularity
exec: tells singularity we will be executing a command, in this case the command is xnat2bids
${simg}: is the singularity image/container that we will be using. We are passing the value of the variable we defined in Step 2. In our case, this is interpreted/evaluated as /oscar/data/bnc/simgs/brownbnc/xnat-tools-v1.1.1.sif
xnat2bids: is the command to be executed, and it is followed by any necessary inputs. In this case we are passing it the positional arguments ${XNAT_SESSION} and ${bids_root_dir} and we are also passing the arguments -u ${XNAT_USER} and -i 7. The -i is asking to only process the first sequence. For a full list of inputs, please see the xnat-tools documentation
After running the command, you'll be asked to interactively type your Brown/XNAT password.
A successful run will print out the following output:
After confirming that XNAT2BIDS is behaving as expected we will run the program on the full dataset. To do so, we invoke it as follows:
The command above is almost identical to the one executed earlier, the only new argument is -s 6 . In this instance we are running xnat2bids on all scan sequences, except series 6. Series 6 corresponds to the individual echoes of a multi-echo MPRAGE and it's typically not used in BIDS apps. We only export the RMS of the echos (#7). We explain this a bit more in the BIDS Ready Protocols section. After successful execution, near the bottom of your log should be the line:
5. Checking XNAT2BIDS results
While running xnat2bids singularity container in an interactive session it is important to keep the session alive throughout the run of the container. However, sometimes due to connection drops this might not always be possible. So here we provide a heuristic way of checking if xnat2bids ran successfully by checking the directory structure.
1. Checking logs
Upon successful completion of the xnat2bids pipeline, you should have 2 log files in your ${HOME}/xnat-exports/bnc/study-demodat/logs directory - export-<date>-<time>.log
and heudiconv-<date>-<time>.log
You can check these logs for any errors or warning messages.
2. Checking the file structure
If xnat2bids ran successfully, you should have 2 folders relating to the 2 steps in the pipeline xnat-exports/ relating to downloading the data from XNAT server and bids relating to bidsification of the data. One easy way of checking the directory structure is to run the tree command -
Which should result in a structure similar to the output shown below:
6. Validate the BIDS output
After successfully running xnat2bids you'll need to make sure that BIDS validation passes. This process is explained in the BIDS Validation Section
Last updated
Was this helpful?