---
title: "Introduction to mobilityIndexR"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Introduction to mobilityIndexR}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

This vignette provides a guide for using the `mobilityIndexR` package. `mobilityIndexR` allows users to both calculate transition matrices (relative, mixed, and absolute) as well as calculate and compare mobility indices (*Prais-Bibby*, *Average Movement*, *Weighted Group Mobility*, and *Origin Specific*). Below are examples and brief explanations of each.


```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r setup, include = FALSE}
library(mobilityIndexR)
```

# Transition Matrices

Two dimensional transition matrices are a tool for representing mobility within a group between two points in time with respect to some observable value such as income of workers or grades of students. Generally, this approach in some way ranks the observable value at each point in time and displays a contingency table with aggregated counts or proportions of each combination of ranks. `mobilityIndexR` offers three methods of constructing ranks using the `getTMatrix` function: relative, mixed, and absolute. 

As example data, we include three datasets: `mobilityIndexR::incomeMobility`, `mobilityIndexR::incomeZeroInfMobility`, and `mobilityIndexR::gradeMobility`. These datasets each contain 125 records, an "id" column, and observations of some value at ten points in time, denoted "t0", ..., "t9". `mobilityIndexR::incomeMobility` simulates income data over ten years; `mobilityIndexR::incomeZeroInfMobility` simulates income data over ten years that has an inflated number of zeros; and `mobilityIndexR::gradeMobility` simulates student grade data over ten assignments.

```{r data}
head(incomeMobility)
```

## Relative

Types of transition matrices differ by how values are binned into ranks. With relative transition matrices, values in each of the two specified columns (`col_x`, `col_y`) are binned into ranks as quantiles. The number of quantiles is specified in the `num_ranks` argument. Note: if one value in the data occurs more often than the size of the quantile ranks, then `getTMatrix` will throw an error; see [Exclude Value](#exclude-value).

`getTMatrix` returns a list containing the transition matrix as well as the bounds used to bin the data into ranks. 

```{r relative_probsF}
getTMatrix(dat = incomeMobility, col_x = "t0", col_y = "t5", 
           type = "relative", num_ranks = 5, probs = FALSE)
```

The above example produces a $5 \times 5$ contingency table. Setting the argument `probs = TRUE`, we obtain a transition matrix with unconditional probabilities rather than counts. 

```{r relative_probsT}
getTMatrix(dat = incomeMobility, col_x = "t0", col_y = "t5", 
           type = "relative", num_ranks = 5, probs = TRUE)
```

## Mixed

With mixed transition matrices, values in `col_x` are first binned into ranks as quantiles, then the bounds for `col_x` are used to bin `col_y` into ranks. In the example below, observe that `col_x_bounds` and `col_y_bounds` are equal except for the minimum and maximum values. 

```{r mixed}
getTMatrix(dat = incomeMobility, col_x = "t0", col_y = "t5", 
           type = "mixed", num_ranks = 5, probs = FALSE)
```

## Absolute

With absolute transition matrices, values in `col_x` and `col_y` are binned into ranks using user-specified bounds with the `bounds` argument.

```{r absolute}
getTMatrix(dat = gradeMobility, col_x = "t0", col_y = "t5", 
           type = "absolute", probs = FALSE, bounds = c(0, 0.6, 0.7, 0.8, 0.9, 1.0))
```

# Mobility Indices

Mobility indices are measures of mobility within a group between two points in time with respect to some observable value such as income of workers or grades of students. `mobilityIndexR` calculates indices using the `getMobilityIndices` function. By default, the `getMobilityIndices` returns all available indices. Note that `getMobilityIndices` can additionally return bootstrapped intervals for each of the indices; see [Bootstrapped Intervals](#bootstrapped-intervals).

```{r indices}
getMobilityIndices(dat = incomeMobility, col_x = "t0", col_y = "t5", 
                   type = "relative", num_ranks = 5)
```

Below is a brief description of the indices. Let $s(i)$ be the event that one is in rank $i$ at the first or starting point in time and $e(j)$ be the event that one is in rank $j$ at the second or ending point in time. Suppose $k$ is the largest rank in the data. 

The *Prais-Bibby* index (`type = "prais_bibby"`) is the proportion of records that change ranks, i.e. $\Sigma_{i = 1}^kPr[s(i) \neq e(i)]$.

The *Average Mobility* index (`type = "average_movement"`) is the average number of ranks records move.

The *Weighted Group Mobility* index (`type = "wgm"`) is a weighted version of the proportion of records that change ranks, i.e. $k^{-1} \Sigma_{i = 1}^k \frac{Pr[\neg e(i) | s(i)]}{Pr[\neg e(i)]}$.

There are four *Origin Specific* indices (`type = "origin_specific"`): top, bottom, top far, and bottom far. The top (bottom) index is the proportion of records that begin in the top (bottom) and end outside of the top (bottom). The far versions of these indices are the proportions of records that begin in the top (bottom) and end at least two ranks away from the top (bottom). 

```{r one_index}
getMobilityIndices(dat = incomeMobility, col_x = "t0", col_y = "t5", 
                   type = "relative", num_ranks = 5, indices = "wgm")
```

A single index or subset of the indices can be returned by passing a string or vector of strings in the `indices` argument.

# Hypothesis Testing

The function `getHypothesisTest` compares mobility indices across datasets. The user specifies two datasets (`dat_A`, `dat_B`) as well as a pair of columns from each dataset as a vector (`cols_A`, `cols_B`). This function performs a non-parametric one-sided hypothesis test that the index value for `dat_A` is greater than the index value for `dat_B`. The parameter `bootstrap_iter` specifies the number of bootstrap iterations; the default value is `100`. 

```{r hypothesis_test}
getHypothesisTest(dat_A = incomeMobility, dat_B = incomeMobility, 
                  cols_A = c("t0", "t3"), cols_B = c("t5", "t8"),
                  type = "relative", num_ranks = 5, bootstrap_iter = 100)
```

By default, all available indices are returned. A single index or subset of the indices can be returned by passing a string or vector of strings in the `indices` argument.

# Advanced Options

## Bootstrapped Intervals

The function `getMobilityIndices` produces non-parametric bootstrapped confidence intervals using the percentile method by setting `intervals = TRUE`. By default, `intervals = FALSE`. The parameter `interval_pct` specifies the size of the confidence interval in terms of the proportion of the data covered; the default value is `0.95`. The parameter `bootstrap_iter` specifies the number of bootstrap iterations; the default value is `100`.

```{r indices_intervals}
getMobilityIndices(dat = incomeMobility, col_x = "t0", col_y = "t5", 
                   type = "relative", num_ranks = 5, indices = "wgm",
                   intervals = TRUE, interval_pct = 0.95, bootstrap_iter = 100)
```

## Exclude Value

For `type = "relative"` or `type = "mixed"`, if one value in the data occurs more often than the size of the ranks then an error will result. This error will occur with `getTMatrix`, `getMobilityIndices`, and `getHypothesisTest` since binning into quantile ranks is not well defined in this case. See the example below where $0$ makes up more than $20\%$ of the values in the "t0" column.

```{r matrices_error, error=TRUE}
getTMatrix(dat = incomeZeroInfMobility, col_x = "t0", col_y = "t5",
           type = "relative", num_ranks = 5, probs = FALSE)
table(incomeZeroInfMobility$t0)
```

One approach is to bin some elements of the problematic value in one rank and the others in another rank. This is done by setting the parameter `strict = FALSE`. By default, `strict = TRUE`. This setting adds a negligible amount of random noise to the problematic values to make binning the values into quantile ranks well defined. 

```{r matrices_strict, error=TRUE}
getTMatrix(dat = incomeZeroInfMobility, col_x = "t0", col_y = "t5",
           type = "relative", num_ranks = 5, probs = FALSE, strict = FALSE)
```

A second approach is to exclude the problematic value from binning the values into quantile ranks then either give the problematic value its own rank, add the problematic value to an existing rank, or exclude the problematic value altogether. A problematic value can be specified in the `exclude_value` argument. Note: currently, `mobilityIndexR` only supports one exclude value.

If an exclude value is specified, the user must also specify how to handle the exclude value using the `rerank_exclude_value` parameter. Setting `rerank_exclude_value = "as_new_rank"` creates an new rank for the exclude value and modifies the other ranks to remain ordinally consistent. Setting `rerank_exclude_value = "as_existing_rank"` bins the exclude value into an existing rank. Setting `rerank_exclude_value = "exclude"` drops any row that has the exclude value from the calculation. 

```{r matrices_exclude_value, error=TRUE}
getTMatrix(dat = incomeZeroInfMobility, col_x = "t0", col_y = "t5",
           type = "relative", num_ranks = 5, probs = FALSE,
           exclude_value = 0, rerank_exclude_value = "as_new_rank")

getTMatrix(dat = incomeZeroInfMobility, col_x = "t0", col_y = "t5",
           type = "relative", num_ranks = 5, probs = FALSE,
           exclude_value = 0, rerank_exclude_value = "as_existing_rank")

getTMatrix(dat = incomeZeroInfMobility, col_x = "t0", col_y = "t5",
           type = "relative", num_ranks = 5, probs = FALSE,
           exclude_value = 0, rerank_exclude_value = "exclude")
```
