# CytobankAPI advanced analysis guide

The `CytobankAPI` package is designed to make interacting with Cytobank API endpoints easy via R. This document is an accompanying overview of the package to learn concepts and see basic examples. View the Cytobank API Endpoint Documentation for a comprehensive list of API endpoints for Cytobank.

Within the `CytobankAPI` package, there are endpoints to interact with advanced analyses via R. This documentation is an overview of the different ways to utilize advanced analyses. To find more general documentation on using the `CytobankAPI` package, view the Cytobank quickstart guide.

All advanced analyses are encapsulated within an object. This guide will be an overview of advanced analyses object structures:

1. Advanced Analyses Objects: What are advanced analyses objects
2. Interactions: How to interact with the advanced analyses objects

# 1 Advanced Analyses Objects

## 1.1 Representation

Every advanced analysis is represented as an object. Creating a new advanced analysis will return an object that is passed to all of their other respective advanced analysis endpoints.

Important information to note:

1. Each advanced analysis object returned can be edited directly
2. Each advanced analysis object is a representation, but does not necessarily mean it is the same as what is seen in the GUI. In order to get the most current settings, utilize the `show` endpoint, or update the advanced analysis with the current settings using the `update` endpoint
3. When the update method is called on the advanced analysis object, the existing record for the run on the Cytobank is overwritten according to the settings in the object

    viSNE_analysis <- visne.show(cyto_session, experiment_id=22, visne_id=214)
    viSNE_analysis@name
    #>  [1] "My viSNE analysis example"

    # Update the viSNE analysis object name directly
    viSNE_analysis@name <- "My updated viSNE analysis name"
    # Update the viSNE analysis using the 'visne.update' endpoint
    updated_viSNE <- visne.update(cyto_session, viSNE_analysis)
    updated_viSNE@name
    #>  [1] "My updated viSNE analysis name"

## 1.2 Common features

There are common features for all advanced analyses:

1. Name: The name of the advanced analysis

2. Compensation ID: The compensation ID used for the advanced analysis

   - uncompensated compensation_id = -1
   - file-internal compensation_id = -2
   - external compensation matrix compensation_id listed in `compensations.list`

3. Channels: The channels being analyzed within the algorithm (clustering or for general analysis)

4. Source experiment: The experiment the advanced analysis belongs to (all advanced analyses belong to an experiment)

5. Status: The state of the advanced analysis (new, running, done, canceled, etc.)

6. Available FCS files: FCS files available for the advanced analysis

7. Available channels: Channels available for the advanced analysis

8. Available populations: Populations available for the advanced analysis

## 1.3 Unique features for each advanced analysis method

There are special settings that pertain to each advanced analysis algorithm. These settings affect how the advanced analysis algorithm is ran. For each advanced analysis, you can view their respective settings and slots as shown below.

    CITRUS_object <- citrus.new(cyto_session, experiment_id, citrus_name="My new Cytobank CITRUS analysis")

    slotNames(CITRUS_object)
    #>  [1] "citrus_id"                "population_id"
    #>  [3] "file_grouping"            "association_models"
    #>  [5] "cluster_characterization" "statistics_channels"
    #>  [7] "event_sampling_method"    "events_per_file"
    #>  [9] "minimum_cluster_size"     "cross_validation_folds"
    #> [11] "false_discovery_rate"     "normalize_scales"
    #> [13] "plot_theme"               "attachment_id"
    #> [15] "channels"                 "compensation_id"
    #> [17] "name"                     "source_experiment"
    #> [19] "status"                   ".available_channels"
    #> [21] ".available_files"         ".available_populations"

Learn more about CITRUS settings.

    FlowSOM_object <- flowsom.new(cyto_session, experiment_id, flowsom_name="My new Cytobank FlowSOM analysis")

    slotNames(FlowSOM_object)
    #>  [1] "author"
    #>  [2] "type"
    #>  [3] "flowsom_id"
    #>  [4] "selected_population_name"
    #>  [5] "population_id"
    #>  [6] "num_fcs_files"
    #>  [7] "fcs_files"
    #>  [8] "event_sampling_method"
    #>  [9] "desired_events_per_file"
    #> [10] "desired_total_events"
    #> [11] "sampled_event_total"
    #> [12] "num_events_to_actually_sample"
    #> [13] "random_seed"
    #> [14] "som_creation_method"
    #> [15] "clustering_method"
    #> [16] "expected_metaclusters"
    #> [17] "expected_clusters"
    #> [18] "iterations"
    #> [19] "normalize_scales"
    #> [20] "created_experiment"
    #> [21] "attachment_id"
    #> [22] "auto_seed"
    #> [23] "external_som_analysis_info"
    #> [24] "external_som_analysis_id"
    #> [25] "channels_to_plot"
    #> [26] "cluster_size_type"
    #> [27] "fixed_cluster_size"
    #> [28] "gate_set_names_to_label"
    #> [29] "max_relative_cluster_size"
    #> [30] "output_file_type"
    #> [31] "show_background_on_legend"
    #> [32] "show_background_on_channel_colored_msts"
    #> [33] "show_background_on_population_pies"
    #> [34] "final_result"
    #> [35] "completed"
    #> [36] "canceled"
    #> [37] "channels"
    #> [38] "compensation_id"
    #> [39] "name"
    #> [40] "source_experiment"
    #> [41] "status"
    #> [42] ".available_channels"
    #> [43] ".available_files"
    #> [44] ".available_populations"

Learn more about FlowSOM settings.

    SPADE_object <- spade.new(cyto_session, experiment_id=22, spade_name="My new Cytobank SPADE analysis")

    slotNames(SPADE_object)
    #>  [1] "created_experiment"         "down_sampled_events_target"
    #>  [3] "down_sampled_events_type"   "fold_change_groups"
    #>  [5] "population_id"              "spade_id"
    #>  [7] "target_number_nodes"        "channels"
    #>  [9] "compensation_id"            "name"
    #> [11] "source_experiment"          "status"
    #> [13] ".available_channels"        ".available_files"
    #> [15] ".available_populations"

Learn more about SPADE settings.

    viSNE_object <- visne.new(cyto_session, experiment_id, visne_name="My new Cytobank viSNE analysis")

    slotNames(viSNE_object)
    #>  [1] "iterations"             "perplexity"             "channels"
    #>  [4] "compensation_id"        "population_selections"  "sampling_total_count"
    #>  [7] "sampling_target_type"   "seed"                   "theta"
    #> [10] "visne_id"               "analysis_id"            "type"
    #> [13] "name"                   "status"                 "source_experiment"
    #> [16] "created_experiment"     ".available_channels"    ".available_files"
    #> [19] ".available_populations"

Learn more about viSNE settings.

    slotNames(dimensionality_reduction_object)
    #> [1] "analysis_id"            "type"                   "name"
    #> [4] "status"                 "source_experiment"      "created_experiment"
    #> [7] ".available_channels"    ".available_files"       ".available_populations"

Learn more about dimensionality reduction settings.

# 2 Interacting with advanced analyses objects

See each section below for instructions on how to interact with the object for each advanced analysis.

## 2.1 CITRUS

### 2.1.1 Updating general CITRUS settings

Directly update CITRUS settings via their slot names.

The following slots can be updated directly:

- Required settings:

  - population_id
    - `gateSetId` from `CITRUS_object@.available_populations`
  - channels: can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)
  - statistics_channels (for medians mode only)
    - file_grouping (see the next section for how to update CITRUS file grouping)

- Optional settings (defaults in parentheses):

  - compensation_id: (experiment-wide compensation)
  - association_models: (“pamr”), sam”, and/or “glmnet”
  - cluster_characterization: (“abundances”) or “medians”
  - event_sampling_method: (“equal”) or “max-per-file”
  - events_per_file (5000 or the number in the smallest file, if less than 5000)
  - minimum_cluster_size (5)
  - cross_validation_folds (5)
  - false_discovery_rate (1)
  - normalize_scales: TRUE or (FALSE)
  - plot_theme: (“white”) or “black”

    # Set association models, plot theme and compensation
    CITRUS_object@association_models <- c("sam", "pamr", "glmnet")
    CITRUS_object@plot_theme <- "black"
    CITRUS_object@compensation_id <- 22

    # Bulk update the changes made to the CITRUS object
    CITRUS_object <- citrus.update(cyto_session, CITRUS_object)

### 2.1.2 Updating CITRUS file grouping

The core functionality of CITRUS is establishing biological explanations for why samples between two or more groups differ from each other. CITRUS file grouping is used to categorize different files into these groups. There is 1 important setting to pay attention to:

- group_name: the group that each file is associated with

  - The minimum number of samples per group is three. However, for more robust statistical analysis and to avoid spurious results, at least eight samples are recommended per group (see documentation for more information here)

  - There must be at least 2 groups in order to run a CITRUS analysis

Directly update CITRUS file grouping data.

    # Set 'file1.fcs' through 'file4.fcs' to 'Group 1' and 'file5.fcs' through 'file8.fcs' to 'Group 2', exclude 'file9.fcs' from analysis
    CITRUS_object@file_grouping[CITRUS_object@file_grouping$id <= 44856,]$group_name <- "Group 1"
    CITRUS_object@file_grouping[is.element(c(44857, 44858, 44859, 44860), CITRUS_object@file_grouping$id),]$group_name <- "Group 2"

    View(CITRUS_object@file_grouping)

id | name | group_name
--- | --- | ---
44853 | file1.fcs | Group 1
44854 | file2.fcs | Group 1
44855 | file3.fcs | Group 1
44856 | file4.fcs | Group 1
44857 | file5.fcs | Group 2
44858 | file6.fcs | Group 2
44859 | file7.fcs | Group 2
44860 | file8.fcs | Group 2
44861 | file9.fcs | Unassigned

Learn more about CITRUS file grouping.

## 2.2 FlowSOM

### 2.2.1 Updating general FlowSOM settings

Directly update FlowSOM settings via their slot names.

The following slots can be updated directly:

- Basic settings:
  - population_id: `gateSetId` from `FlowSOM_object@.available_populations`
  - channels: can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)
  - fcs_files
  - random_seed
  - compensation_id: must match experiment-wide compensation
- Event sampling settings:
  - event_sampling_method: “equal”, “proportional”, or “all”
  - desired_events_per_file: for equal sampling
  - desired_total_events: for proportional sampling
- Algorithm settings (default in parentheses):
  - som_creation_method: (“create_new”) or “import_existing”
    - external_som_analysis_id: if importing existing SOM, character of source FlowSOM ID
  - clustering_method: (“consensus”), “hclust”, or “kmeans”
  - expected_metaclusters (10)
  - expected_clusters (100)
  - iterations (10)
- Transformations settings (default in parentheses):
  - normalize_scales: TRUE or (FALSE)
- PDF output settings (default in parentheses):
  - channels_to_plot: channels can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)
  - cluster_size_type: (“relative”), “fixed” or “both”
  - fixed_cluster_size (8)
  - max_relative_cluster_size (15)
  - gate_set_names_to_label: populations for pie charts
  - output_file_type: (“pdf”), “png” or “both”
  - Toggle metacluster background on plots: TRUE or FALSE
    - show_background_on_legend: (FALSE)
    - show_background_on_channel_colored_msts (FALSE)
    - show_background_on_population_pies (TRUE)

If the required `channels`, `fcs_files` and `random_seed` slots are not present, updates will not occur to the FlowSOM analysis.

    # Set a clustering method and target number of nodes
    FlowSOM_object@clustering_method <- "kmeans"
    FlowSOM_object@num_expected_clusters <- 144

    # Update FCS file selection to the first 5 files
    FlowSOM_object@fcs_files <- FlowSOM_object@.available_files$id[1:5]

    # Update channel selection
    FlowSOM_object@channels <- list("CD3", "CD4")

    # Bulk update the changes made to the FlowSOM object
    FlowSOM_object <- flowsom.update(cyto_session, FlowSOM_object)

## 2.3 SPADE

### 2.3.1 Updating general SPADE settings

Directly update SPADE settings via their slot names.

The following slots can be updated directly:

- Required settings:
  - population_id
  - channels: channels can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)
- Additional settings if changing from default (in parentheses):
  - compensation (experiment-wide compensation)
  - target_number_nodes (200)
  - down_sampled_events_target (10)
  - down_sampled_events_type (“percent”) or “absoluteNumber”
  - fold_change_groups: see the next section for how to update SPADE fold change groups

    # Set a new population, target number of nodes, and compensation
    SPADE_object@population_id <- 2
    SPADE_object@target_number_nodes <- 150
    SPADE_object@compensation_id <- 22

    # Update channels
    channel_ids_list <- list(2, 3, 5, 8)
    SPADE_object@channels <- channel_ids_list

    # Update channels by long channel names
    channel_names_list <- list("channel1", "channel2", "channel3")
    SPADE_object@channels <- channel_names_list

    # Bulk update the changes made to the SPADE object
    SPADE_object <- spade.update(cyto_session, SPADE_object)

    SPADE_object@population_id
    #> [1] 2
    SPADE_object@target_number_nodes
    #> [1] 150
    SPADE_object@compensation_id
    #> [1] 22
    SPADE_object@channels
    #> [[1]]
    #> [1] "channel1"
    #>
    #> [[2]]
    #> ...

### 2.3.2 Updating SPADE fold change groups

SPADE fold change groups are used to categorize different files into separate collections that will be compared amongst each other. There are 2 important settings to pay attention to:

1. group_name: The group a specific file belongs to
2. baseline: The file(s) used as the baseline in order to calculate fold change

Directly update SPADE fold change groups data.

    # Set 'file6.fcs' and 'file7.fcs' as the baseline for 'Group 1'
    SPADE_object@fold_change_groups[grep("my_file6|my_file7",
        SPADE_object@fold_change_groups$name),]$baseline <- TRUE

    # Set 'file2.fcs', 'file4.fcs', and 'file8.fcs' as part of 'Group 2', and set 'file2.fcs' as the baseline
    SPADE_object@fold_change_groups[grep("file2|file4|file8",
        SPADE_object@fold_change_groups$name),]$group_name <- "Group 2"
    SPADE_object@fold_change_groups[SPADE_object@fold_change_groups$name=="file2.fcs",]$baseline <- TRUE

    View(SPADE_object@fold_change_groups)

id | name | baseline | group_name
--- | --- | --- | ---
44853 | file1.fcs | FALSE | Group 1
44854 | file2.fcs | TRUE | Group 2
44855 | file3.fcs | FALSE | Group 1
44856 | file4.fcs | FALSE | Group 2
44857 | file5.fcs | FALSE | Group 1
44858 | file6.fcs | TRUE | Group 1
44859 | file7.fcs | TRUE | Group 1
44860 | file8.fcs | FALSE | Group 2

Learn more about SPADE fold change groups.

## 2.4 viSNE

### 2.4.1 Updating general viSNE settings

Directly update viSNE settings via their slot names.

The following slots can be updated directly:

- sampling_target_type

- sampling_total_count

- channels

  - channels can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)

- compensation_id

- iterations

- perplexity

- theta

- seed

The following slots must be updated via helper functions:

- population_selections (see the next section for how to update viSNE population selections)

  - visne.helper.set_populations

    # Set a new sampling target type, sampling total count, and compensation
    viSNE_object@sampling_target_type <- "equal"
    viSNE_object@sampling_total_count <- 150000
    viSNE_object@compensation_id <- 22

    # Bulk update the changes made to the viSNE object
    viSNE_object <- visne.update(cyto_session, viSNE_object)

### 2.4.2 Updating viSNE population selections

Adding viSNE population selections is slightly more difficult because the same file can be used in the analysis in combination with multiple populations. Because of this complexity, the `visne.helper.set_populations` helper function is used to set files for a selected population.

Parameters for `visne.helper.set_populations`:

1. visne: The viSNE object to set populations for
2. population_id: The gate set ID for the specified population (different than the actual population ID, and can be obtained by looking at the `.available_populations` slot)
3. fcs_files: A vector/list of FCS files to set for the population

Set files for a specific population through the `visne.helper.set_populations` helper function.

Setting files for a specific population will overwrite the files previously set for the population in question.

    # Set files for different populations
    viSNE_object <- visne.helper.set_populations(viSNE_object, population_id=1, fcs_files=c(44853))
    viSNE_object <- visne.helper.set_populations(viSNE_object, population_id=2, fcs_files=c(44867,44868))
    viSNE_object <- visne.helper.set_populations(viSNE_object, population_id=4, fcs_files=unlist(visne@.available_files[grep("file4|file5|file6", visne@.available_files$filename),]$id))
    # Overwrite 'population_id=2' FCS file selection, note that 'file1.fcs' and 'file2.fcs' are in both 'Population 1', as well as 'Population 2'
    viSNE_object <- visne.helper.set_populations(viSNE_object, population_id=2, fcs_files=c(44854,44855, 44853, 44867))

    # Update the changes made to viSNE population selections
    viSNE_object <- visne.update(cyto_session, viSNE_object)

    View(viSNE_object@population_selections)

id | name | samplingCount | eventCount | populationId | populationName
--- | --- | --- | --- | --- | ---
44853 | file1.fcs | NA | NA | 1 | Population 1
44856 | file4.fcs | NA | NA | 4 | Population 4
44857 | file5.fcs | NA | NA | 4 | Population 4
44858 | file6.fcs | NA | NA | 4 | Population 4
44854 | file2.fcs | NA | NA | 2 | Population 2
44855 | file3.fcs | NA | NA | 2 | Population 2
44853 | file1.fcs | NA | NA | 2 | Population 2
44856 | file4.fcs | NA | NA | 2 | Population 2

Learn more about selecting viSNE populations.

## 2.5 Dimensionality Reduction Suite

### 2.5.1 Setting up a new analysis run using one of the algorithms that are available in Dimensionality Reduction Suite

The following example shows how to set up and launch a tSNE-CUDA analysis run.

    # create a new dimensionality reduction analysis object and set the analysis type to be tSNE-CUDA
    dimensionality_reduction_object<-dimensionality_reduction.new(cyto_session, experiment_id=22, analysis_name = 'use API to launch a tSNEcuda run', analysis_type = 'tSNE-CUDA')

    # select clustering channels
    dimensionality_reduction_object@clustering_channels<-list('CD4','CD8','CD3')

    # Bulk update the change
    dimensionality_reduction_object<-dimensionality_reduction.update(cyto_session,dimensionality_reduction_object)

    # Launch the configured analysis run
    dimensionality_reduction.run(cyto_session,dimensionality_reduction_object)

    # Check the status of the launched analysis run
    dimensionality_reduction.status(cyto_session,cyto_tsnecuda)

### 2.5.2 Updating general settings for the algorithms in Dimensionality Reduction Suite

Directly update settings via their slot names.

The following slots can be updated directly:

- tSNE-CUDA, opt-SNE and UMAP:
  - event_sampling_method
  - desired_total_events
  - clustering_channels
    - channels can be set as a list of channel IDs or a list of channel long names (long names must correlate to a unique short channel name)
    - channel names can be obtained by looking at the `.available_channels` slot
  - normalize_scales
- Additional slots for tSNE-CUDA:
  - perplexity
  - iterations
  - auto_iterations
  - learning_rate
  - auto_learning_rate
  - early_exaggeration
- Additional slots for opt-SNE:
  - perplexity
  - auto_learning_rate
  - learning_rate
  - early_exaggeration
  - max_iterations
  - random_seed
- Additional slots for UMAP:
  - collapse_outliers
  - min_distance
  - num_neighbors

    # Set a new sampling target type, sampling total count, and compensation
    dimensionality_reduction_object@event_sampling_method <- "equal"
    dimensionality_reduction_object@desired_total_events <- 150000

    # Bulk update the changes made to the dimensionality reduction object
    dimensionality_reduction_object <- dimensionality_reduction.update(cyto_session, dimensionality_reduction_object)