---
title: "Introduction To The auctestr Package"
author: "Josh Gardner"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction To The auctestr Package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
## Introduction 

This is an introduction to the `auctestr` package, a package for statistical testing of the AUC (also known as Area Under the Receiver Operating Characteristic Curve, or A') statistic. The AUC has some useful statistical properties that make it especially simple to apply statistical tests. Furthermore, `auctestr` implements some basic statistical procedures for applying these statistical tests even when you have several observations of the AUC of a given model, even over different datasets, and within datasets when there is some kind of dependency (such as when there are observations within a dataset over time, or across multiple randomized resamples or cross-validation folds).

`auctestr` is useful if you:
- Are evaluating predictive models.
- Need to conduct pairwise comparisons of the performance of those models.
- Are using AUC (or A') to evaluate the performance of those models (note that there are multiclass versions of AUC that can also be used for this).

For the remainder of this document, we refer to the statistic of interest only as AUC. Note that the unique statistical properties used in this package *only* apply to the AUC statistic, and cannot be used to evaluate other model performance metrics (i.e., accuracy, F-1 score, etc). 

## Functions

`auctestr` currently contains only four simple functions, which is all that is required for complete statistical testing of the AUC. An example dataset would consist of one or more observations of at least two different predictive models:

```{r, echo=FALSE, results='hide'}
library(auctestr)
```


```{r}
data("sample_experiment_data", package="auctestr")
head(sample_experiment_data, 15)
```

Conducting statistical comparisons of models, including over time, can be completed in a single call to `auc_compare()`:

```{r}
# compare model A and model B, only evaluating VariantC of both models
z_score = auc_compare(sample_experiment_data, compare_values = c("ModelA", "ModelB"), filter_value = c("VariantC"), time_col = "time", outcome_col = "auc", compare_col = "model_id", over_col = "dataset", filter_col = "model_variant")
z_score
# fetch p-value of this comparison
pnorm(-abs(z_score))
```

`auctestr` also allows for flexible adjustment of which pairwise comparisons are conducted, and which elements are held fixed (the fixed values are set using `filter_value` and `filter_col` parameters):

```{r}
z_score = auc_compare(sample_experiment_data, compare_values = c("VariantA", "VariantB"), filter_value = c("ModelC"), time_col = "time", outcome_col = "auc", compare_col = "model_variant", over_col = "dataset", filter_col = "model_id")
z_score
pnorm(-abs(z_score))
```


The model comparisons are conducted using a method described in detail in: Fogarty, James, Ryan S. Baker, and Scott E. Hudson. "Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction." Proceedings of Graphics Interface 2005. Canadian Human-Computer Communications Society, 2005.

Note that these comparisons assume that there is a dataset-dependent column that needs to be statistically averaged over, and it uses Stouffer's method to do so:

$Z \sim \frac{\sum_{i=1}^kZ_i}{\sqrt{k}}$

This is a conservative adjustment and more powerful, less conservative adjustments may be added in future versions. For more information, see Stouffer, S.A.; Suchman, E.A.; DeVinney, L.C.; Star, S.A.; Williams, R.M. Jr. (1949). The American Soldier, Vol.1: Adjustment during Army Life. Princeton University Press, Princeton, or [wikipedia]{https://en.wikipedia.org/wiki/Fisher%27s_method#Relation_to_Stouffer.27s_Z-score_method}

We hope to implement more features in a future version, but this is a fully-featured package to allow for more principled statistical model selection based on the unique statistical properties of the AUC metric; we hope it improves your research and modeling.
