# First-Level Analysis Using afniproc.py

## Step 1: Download data from XNAT and automatically convert to BIDS format using xnat2bids

Running xnat2bids on the command line in Oscar will download the data we need, convert it to BIDS format, and run the BIDS validator to check for any issues. We will be using the xnat-tools Oscar utility script explained [here](https://docs.ccv.brown.edu/bnc-user-manual/xnat-to-bids-intro/using-oscar/oscar-utility-script). The Demodat2 dataset has 3 subjects with 2 sessions each. We will be exporting this entire project. 

#### Create a configuration file 

The configuration .toml file contains information that xnat-tools needs to download the correct data and put it where we want. Name this file x2b\_demodat2\_config.toml and place wherever you'd like. Paste the following into your .toml file, and change `mail-user` to your email address. The script will default to placing the downloaded and BIDS-converted data in a folder called "bids-export" in your home directory; if you'd like to change this location, add a new line at the bottom with your desired path, i.e.: `bids_root="/oscar/home/<yourusername>/data/<projectname>/xnat-export"`. Make sure to save this .toml file when you are done editing.&#x20;

```toml
# Configuring arguments here will override default parameters.
[slurm-args]
mail-user = "example-user@brown.edu"
mail-type = "ALL"
output = "/oscar/home/example-user/logs/%x-%J.txt"

[xnat2bids-args]
bids_root = "/oscar/home/<example-user>/data/Demodat2"
project="BNC_DEMODAT2"
verbose=2
```

#### Run the xnat2bids command

To run the xnat-tools export and BIDS conversion, ensure that you are working in a terminal in the same directory where you stored this .toml file (and if not, then give the full path to the .toml file in your command). On the command line, type:

`module load anaconda3`

`python /oscar/data/bnc/scripts/run_xnat2bids.py --config x2b_demodat2_config.toml`

Enter your XNAT username and password when prompted.

If you entered your email address, you should receive an email when each of the xnat2bids jobs begin, and another when they finish.&#x20;

This will create a sourcedata folder for each subject within `$bids_root/bnc/study-demodat2/xnat-export` and a BIDS-compatible data directory within `$bids_root/bnc/study-demodat2/bids/`.

{% hint style="info" %}
We will call this output BIDS-compatible folder `$bidsdir` for the remainder of the tutorial.&#x20;
{% endhint %}

## Step 2: Convert psychopy timing files to be used by AFNI

The aim of this step is to make our behavioral data BIDS compatible and facilitate future data sharing. To do this, we first need to create events.tsv files that correspond to each of our functional runs and contain information about each stimulus event of interest (onset time, condition, etc.). Then, in order to use AFNI for a regression, those tsv files must be converted into AFNI 1D files and saved in the bids derivatives directory.&#x20;

{% hint style="info" %}
How will these timing files differ from those in a single subject pipeline?&#x20;

* The tsv files are the same- they are still created per subject, session, and run. They are separated by task (checks, keypress), but not by condition (left, right).
* For the first level analysis, we only input data from session 1 into the AFNI regression. Since there are two runs per session, the 1D files had 2 rows. These files are separated by both task and condition (e.g. left\_keypress.1D, right\_checks.1D).
* For the second level analysis, the AFNI 1D files will contain 4 rows. These rows correspond to the two runs from session 1 and the two runs from session 2. By including all runs from both sessions in one file, we can include all data per subject in the regression.
  {% endhint %}

#### Download the psychopy behavioral timing files (csv)

First, download the participant's behavioral timing files (in our case, created by PsychoPy) and manually place them in a directory named 'beh', in the `sourcedata` subfolder of your BIDS directory. For each individual subject and session, the full path should be: `$bidsdir/sourcedata/sub-xxx/ses-xx/beh`.&#x20;

Repeat this step for each subject and session, placing the files in their respective `beh` folders. Files are zipped per subject (unzip the file to view ses-01 and ses-02 separately).

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2Fg9AmBf0diP4LAGj7Pwtq%2Fsub-101-beh.zip?alt=media&token=6ad80903-9f8f-4099-acc0-3e54a9f2bbe2>" %}

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FWIvXDsy74USYE1LV24Va%2Fsub-102-beh.zip?alt=media&token=d378dd36-21d0-4e1f-a6b6-8c38f17e4e4c>" %}

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FRSPu8yoQ4OFpBaCMKG3j%2Fsub-103-beh.zip?alt=media&token=6250ea6d-f84d-417a-953a-4464cd52d58e>" %}

#### Convert the timing files using a batch script&#x20;

Batch scripting improves the efficiency of data processing because it allows you to automatically launch the same job on all individual subjects/sessions, rather than running the script/command one by one. It takes advantage of the many resources available on our HPC (CPU, cores, etc) and runs the same script on all your subjects in parallel (rather than sequentially).

<details>

<summary>The components of a batch job on Oscar</summary>

Batch jobs remove much of the tedious work and allow you to process your data much faster!&#x20;

&#x20;Typically, a batch job requires 3 components:&#x20;

1. **The batch script:** A script detailing what resources you will request for your job, what processing script you will be running, and in our case, what subjects we will be input into that pipeline/script. This is written in bash and uses oscar's internal resource allocator/job manager, [SLURM](https://slurm.schedmd.com/overview.html). Information about the default SLURM settings can be found [here.](https://docs.ccv.brown.edu/oscar/submitting-jobs/batch?q=cache)
2. **The processing script**: The script that will be applied to your list of subjects. This can be written in any language.&#x20;
3. **A text file containing the subjects:** The batch script will iterate over this list and use it to fill in the subject ID variable in your processing script.&#x20;

</details>

In this tutorial, we will include the data from both runs of each session (4 runs total per subject) in our afniproc.py command. To do so, our example script `preprocess_behavior.py` will 1) convert and move the psychopy csv files to BIDS-organized tsv files, and then 2) convert and move those tsv files to 1D files that will be used by AFNI in the bids derivatives folder. &#x20;

Here you can download our example python processing script (`preprocess_behavior.py` ), the batch script that will be used to run it (run\_`preprocess_behavior.py` ), and the text file containing subject IDs (`subjects.txt`). Save these so they can be accessed on your oscar account. Open the batch script run\_preprocess\_behavior.sh and fill in your email and the path to your bids directory.&#x20;

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2Fgsnb9dZmarAsMxulYFSR%2Frun_preprocess_behavior.sh?alt=media&token=b2faf124-6626-47ff-9cdf-0a1fba1a142f>" %}

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FShsmmGrKG5gWtT0ovimB%2Fpreprocess_behavior.py?alt=media&token=2c34a815-b201-49c6-bd2e-d5db12bd0a6b>" %}

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FvLmlIM58uRfUTlqRRY6P%2Fsubjects.txt?alt=media&token=1375c2d6-8c2e-48af-a61f-cf63c3b30717>" %}

#### Run the batch script&#x20;

To run the batch script, which will then launch multiple iterations of the preprocess\_behavior.py script (one job/launch per subject), navigate to the directory where you saved these 3 files. On the command line, type `sbatch run_preprocess_behavior.sh`. Since you filled in your email, you should receive a message when each of the 3 jobs have launched.    &#x20;

If you are unable to run this script for any reason, you can download the events.tsv output files here, and manually place them in each respective directory, i.e.: `$bidsdir/sub-xxx/ses-xx/func/` . Similarly, you can download the .1D files and manually place them in a new folder called `$bidsdir/derivatives/afni/sub-xxx/stimtimes/`.&#x20;

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FwNUUzcGuInfAncuoVmig%2FDemodat2_events_tsv_files.zip?alt=media&token=c7bf335c-6316-4c92-8946-72bc0c0b7373>" %}

{% file src="<https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2FS2S5lhG1LnlpOjv7Gf93%2FDemodat2_events_1D_files.zip?alt=media&token=bc59c959-6dc9-4082-95dc-05f849f73906>" %}

## Step 3: Prepare fMRI data for preprocessing by warping to standard space

It is possible to quickly warp your data into standard space within the afniproc.py script, but this does not provide the most accurate warping, since it is done via a linear transformation. To perform a nonlinear transformation into standard space, there is a separate AFNI command called [sswarper2](https://afni.nimh.nih.gov/pub/dist/doc/program_help/sswarper2.html) (the successor of @SSwarper). This command takes roughly an hour and half to run on Oscar, and is used on each individual subject's T1 anatomical file. You only need to warp one of the T1s per subject- in the next step, all functional data from session 1 and 2 will be aligned to it.&#x20;

Copy and paste this script into a file called `SSW.sh`. Change the example email to your own, and edit the path to your bids directory. Ensure that it is saved in the same location as your subjects file (`subjects.txt`). To launch this script on all 3 subjects sequentially, navigate to the directory on the command line and type `sbatch SSW.sh`.&#x20;

{% code title="SSW\.sh <- This is the filename and should not be copy/pasted in the script" %}

```bash
#!/bin/bash
#SBATCH -N 1
#SBATCH -c 8
#SBATCH --mem=10G
#SBATCH --time 3:00:00
#SBATCH -J SSW
#SBATCH --output=logs/SSW-%A_%a.out
#SBATCH --mail-user example_user@brown.edu
#SBATCH --mail-type ALL
#SBATCH --array=1-3

# To run this, type on the terminal: sbatch SSW.sh

bidsdir=/path/to/bids # <-- Edit with your bids path
subID=$(sed -n "${SLURM_ARRAY_TASK_ID}p" subjects.txt)

echo "Running subject: $subID"

mkdir $bidsdir/derivatives/afni/$subID
outdir=$bidsdir/derivatives/afni/$subID
cd $outdir

# Run sswarper2 on ses-01 data for each subject

sswarper2             \
        -input $bidsdir/$subID/ses-01/anat/${subID}_ses-01_acq-memprageRMS_T1w.nii.gz    \
        -base MNI152_2009_template_SSW.nii.gz             \
        -subid ${subID} -odir ${outdir} 

```

{% endcode %}

<figure><img src="https://4276957606-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LtSg7ZEM6EHbi9iE84a%2Fuploads%2F7YKUHO6aAXUedndFhR06%2Fsswarper_image.png?alt=media&#x26;token=81bf7779-0ac4-4568-b33d-afe95c9ce34a" alt=""><figcaption></figcaption></figure>

## Step 4: Use afni\_proc.py to create a preprocessing stream and run the general linear model per subject

This basic example of a univariate analysis with AFNI is based on the [example 6b](https://afni.nimh.nih.gov/pub/dist/doc/program_help/afni_proc.py.html) for afni\_proc.py.&#x20;

#### Run the afniproc.py batch script

{% hint style="info" %}
afniproc.py will create its output folders in the directory that it is run from. To ensure all outputs are organized in the appropriate BIDS derivatives folder, the batch script will navigate to an output directory before launching afniproc.py. However, the log file for each job is created by SLURM, not by the script itself. It is recommend that you launch the batch script from the directory where you want your log files to be saved (for example, within a scripts directory).&#x20;
{% endhint %}

This `afniproc.sh` script will iterate over your list of subjects and launch afniproc.py  over them individually. Afniproc.py is a metascript, meaning that it will create your actual preprocessing script `proc.sub-xxx` for each subject. Since we included the `-execute` flag at the bottom, `proc.sub-xxx` will be automatically launched when you run afniproc.py. Looking at the `proc.sub-xxx` script is the best way to gain a deeper understanding of each of AFNI's processing steps.&#x20;

Copy the text in the box below into a file editor on Oscar.  Change your email in the beginning section, and change the value of the `bidsdir` variable to your own location (path should end in `/bids`). Save this script as a file called `afniproc.sh`, and then execute it on the command line with `sbatch demodat2_afniproc.sh`. It will launch as a batch script, similar to how we ran `run_preprocess_behavior.sh`. You will receive an email when the job has completed.&#x20;

{% hint style="info" %}
The next step after afniproc.py is to do a group analysis using afni's 3dMEMA. 3dMEMA requires that the linear regression was performed using the afni command 3dREMLfit. To do that, the -regress\_reml\_exec flag is included in our afniproc.py script.&#x20;
{% endhint %}

{% code title="afniproc.sh" %}

```bash
#!/bin/bash
#SBATCH -N 1
#SBATCH -c 8
#SBATCH --mem=10G
#SBATCH --time 4:00:00
#SBATCH -J afniproc
#SBATCH --output=logs/afniproc-%A_%a.out
#SBATCH --mail-user example_user@brown.edu
#SBATCH --mail-type ALL
#SBATCH --array=1-3

# To run this, type on the terminal: sbatch afniproc.sh

bidsdir=/path/to/bids # <---- FILL THIS IN
subID=$(sed -n "${SLURM_ARRAY_TASK_ID}p" subjects.txt)

echo "Running subject: $subID"

mkdir $bidsdir/derivatives/afni/$subID
cd $bidsdir/derivatives/afni/$subID

# Run afniproc
afni_proc.py \
    -subj_id ${subID} \
    -out_dir $outdir/$subID.results \
    -copy_anat $outdir/anatSS.${subID}.nii   \
    -anat_has_skull no \
    -dsets $bidsdir/$subID/ses-01/func/*checks*nii* \
           $bidsdir/$subID/ses-02/func/*checks*nii* \
    -blocks tshift align tlrc volreg blur mask scale regress \
    -tcat_remove_first_trs 0 \
    -align_opts_aea -cost lpc+ZZ -giant_move -check_flip \
    -tlrc_base MNI152_2009_template_SSW.nii.gz \
    -tlrc_NL_warp                                                    \
    -tlrc_NL_warped_dsets       $outdir/anatQQ.$subID.nii \
                                $outdir/anatQQ.$subID.aff12.1D   \
                                $outdir/anatQQ.${subID}_WARP.nii   \
    -volreg_align_to MIN_OUTLIER \
    -volreg_align_e2a \
    -volreg_tlrc_warp \
    -mask_epi_anat yes \
    -blur_size 4.0 \
    -regress_stim_times \
      $bidsdir/derivatives/afni/$subID/stimtimes/${subID}_checks_left_dualsession_stimtimes.1D \
      $bidsdir/derivatives/afni/$subID/dualsession/stimtimes/${subID}_checks_right_dualsession_stimtimes.1D \
      $bidsdir/derivatives/afni/$subID/dualsession/stimtimes/${subID}_keypress_left_dualsession_stimtimes.1D \
      $bidsdir/derivatives/afni/$subID/dualsession/stimtimes/${subID}_keypress_right_dualsession_stimtimes.1D \
    -regress_stim_labels leftchx rightchx leftpress rightpress \
    -regress_basis_multi 'BLOCK(12,1)' 'BLOCK(12,1)' 'GAM' 'GAM' \
    -regress_opts_3dD -jobs 2 \
        -gltsym 'SYM: leftchx -rightchx' -glt_label 1 left_vs_right_chx \
        -gltsym 'SYM: leftpress -rightpress' -glt_label 2 left_vs_right_press \
    -regress_motion_per_run \
    -regress_censor_motion 0.3 \
    -regress_censor_outliers 0.05 \
    -regress_reml_exec \
    -regress_compute_fitts \
    -regress_make_ideal_sum sum_ideal.1D \
    -regress_est_blur_epits \
    -regress_est_blur_errts \
    -regress_run_clustsim yes \
    -html_review_style pythonic \
    -execute
```

{% endcode %}

<details>

<summary>More information about the afniproc.py flags </summary>

The -blocks flag lists the processing blocks that will be executed, in order:&#x20;

`tshift`: slice time correction

`align`: aligning the EPIs to the anatomical scan

`volreg`: motion correction within each functional run

`blur`: spatial smoothing with a 4mm FWHM smoothing kernel

`mask`: create a "brain" mask from the functional data, restricted by the anatomy

`scale`: scale each voxel to have a mean of 100 per run

`regress`: build a general linear model and execute with [3dREMLfit](https://afni.nimh.nih.gov/pub/dist/doc/htmldoc/programs/3dREMLfit_sphx.html)&#x20;

**To point afniproc.py to the output of sswarper2:**

`-copy_anat` the argument of this flag is the skull stripped (but not warped/standardized) anatomical file. This is included in the output of sswarper2.&#x20;

`-anat_has_skull` is set to `no`, since we are inputting the anatSS file

`-tlrc_NL_warp`  performs a non-linear transformation into template space, using the files listed in the flag `-tlrc_NL_warped_dsets`

`-tlrc_NL_warped_dsets` lists the relevant datasets output from sswarper2. These include:&#x20;

1. `anatQQ.$subID.nii` (skull-stripped dataset nonlinearly warped to the base template space)
2. `anatQQ.$subID.aff12.1D` (affine matrix to transform original dataset to base template space)
3. `anatQQ.$subID_WARP.nii` (incremental warp from affine transformation to nonlinearly aligned dataset)

**For the visual hemifield localizer regression, we use:**

`-regress_stim_times` to provide the stimulus timing files for this participant (sub-xxx\_checks\_left\_stimtimes.1D and sub-xxx\_checks\_right\_stimtimes.1D)

`-regress_stim_labels` to assign those conditions the labels of "leftchx" and "rightchx" respectively

`-regress_basis` to model each stimulus as a block lasting 12 seconds

`-regress_opts_3dD` to specify our contrasts. Here, we do a "left\_vs\_right\_chx" contrast to find voxels whose activity is greater for the left hemifield stimulation than for the right

**For the motor task (keypress) regression, we use:**

`-regress_stim_times` to provide the stimulus timing files for this participant (sub-xxx\_keypress\_left\_stimtimes.1D and sub-xxx\_keypress\_right\_stimtimes.1D)

`-regress_stim_labels` to assign those conditions the labels of "leftpress" and "rightpress" respectively

`-regress_basis` to model each stimulus as an instantaneous event (indicated by using AFNI's 'GAM' function)

`-regress_opts_3dD` to specify our contrasts. Here, we do a "left\_vs\_right\_press" contrast to find voxels whose activity is greater for the left index finger motor activation than for the right&#x20;

</details>

Now that each subject's data has been preprocessed (both sessions included), we are ready to move on to the second-level/group analysis!&#x20;