This vignette walks through a complete workflow using AutoTab on simulated tabular data containing a mix of continuous, binary, and categorical variables. We cover setting up the required Python/TensorFlow environment, preprocessing, training a variational autoencoder (VAE), and sampling synthetic data from the trained model.
The workflow below is organized into five training examples, each
highlighting a different set of hyperparameter options available in
VAE_train(). These examples are not intended to represent
an optimized model, they are meant to demonstrate how each
hyperparameter is specified and what its effect looks like during
training.
| Example | Description |
|---|---|
| 1 | A VAE executing simple functionalities |
| 2 | A VAE executing a weighted wTAB-ELBO and separate examination of by variable reconstruction loss. |
| 3 | A VAE executing cyclical beta annealing. |
| 4 | A VAE executing temperature annealing. |
| 5 | A VAE executing L2 regularization and Batch Normalization. |
After training, we walk through how to rebuild a trained encoder/decoder from saved weights and sample synthetic observations from the model.
The simulated data contains 15 variables and 10,000 rows. There are five variables of each type: continuous, binary, and categorical. This dataset is accessible in the AutoTab package.
| Distribution | Variable Name | Range / Levels |
|---|---|---|
| Continuous | Age | (18,87) |
| Continuous | Height | (59,79) |
| Continuous | Weight | (100,328) |
| Continuous | SBP (Systolic Blood Pressure) | (90,188) |
| Continuous | Cholesterol | (100,361) |
| Binary | Female | 1=Yes; 0=No |
| Binary | History (family history of heart attacks) | |
| Binary | Smoke (current smoker) | |
| Binary | Hyperten (family history of hypertension) | |
| Binary | Heart (personal history of heart attack) | |
| Categorical | Socio (Socioeconomic Status) | Low; Medium; High |
| Categorical | Marital (Marital Status) | Single; Married; Divorced; Widowed |
| Categorical | Employ (Employment Status) | Unemployed; Student; Employed; Retired |
| Categorical | Race | Black; White; Hispanic; Other |
| Categorical | Edu (Education) | Some College; High School; Bachelors; Graduate |
data("data_example")
str(as.data.frame(data_example))
#> 'data.frame': 10000 obs. of 15 variables:
#> $ age : num 40.5 44.5 36.6 26.5 46 ...
#> $ height : num 67.8 70.4 68.8 70.7 71.1 ...
#> $ weight : num 137 198 229 217 202 ...
#> $ SBP : num 110 125 112 114 113 ...
#> $ cholesterol: num 219 189 275 196 209 ...
#> $ female : num 1 0 0 1 1 0 0 0 1 1 ...
#> $ history : num 0 0 1 0 0 0 1 1 0 1 ...
#> $ smoke : num 1 1 1 0 1 1 1 1 0 1 ...
#> $ hyperten : num 0 1 1 1 1 1 1 0 0 1 ...
#> $ heart : num 0 0 1 1 1 1 1 0 1 1 ...
#> $ socio : chr "High" "Medium" "Medium" "Low" ...
#> $ marital : chr "Married" "Divorced" "Married" "Married" ...
#> $ employ : chr "Employed" "Employed" "Employed" "Student" ...
#> $ race : chr "White" "Black" "White" "Hispanic" ...
#> $ edu : chr "BS" "HS" "BS" "HS" ...Preprocessing of the data includes removing missing values (or using an imputation method on the dataset), one-hot-coding, variable scaling, and ensuring all columns are numeric. Some recommended packages for handling missing data prior to preprocessing include mice and missForest (Buuren and Groothuis-Oudshoorn 2011; Stekhoven and Bühlmann 2012). The user may also decide to use the complete case version of the intended dataset. We use min-max-scaling for all continuous variables, and one-hot-coding for the categorical variables. Out of the total one-hot-coded columns for a given variable only one can be true in each row of data. This column will be 1 and all others receive a 0 for that observation. After one-hot-coding, socioeconomic status goes from 1 to 3 columns, marital status goes from 1 to 4 columns, and so on. This increases the number of columns from 15 to 29.
The code below creates the one-hot-coded categorical variables from the the categorical variables shown above, using the caret package.
library("caret")
#> Loading required package: ggplot2
#> Loading required package: lattice
encoded_data = dummyVars(~ socio + marital + employ + race + edu,
data = data_example)
one_hot_encoded = as.data.frame(predict(encoded_data,
newdata = data_example))The code below is used to min-max-scale the continuous variables, the function min_max_scale is used from the AutoTab package.
data_cont = subset(data_example,
select = c(age, height, weight, SBP, cholesterol))
data_scaled = as.data.frame(
lapply(data_cont, min_max_scale))The code below binds the one-hot-coded, min-max-scaled, and binary variables together. In doing so, continuous variables are now in the range of [0,1] and the categorical variables are one-hot-coded. Additionally, all columns are numeric.
data_bin = subset(data_example,
select = c(female, history, smoke, hyperten, heart))
data = cbind(data_scaled, data_bin, one_hot_encoded)
head(data)
#> age height weight SBP cholesterol female history smoke hyperten heart socioHigh socioLow
#> 1 0.3281284 0.4440304 0.1619451 0.2011243 0.4554580 1 0 1 0 0 1 0
#> 2 0.3868740 0.5768439 0.4274326 0.3602980 0.3399162 0 0 1 1 0 0 0
#> 3 0.2710934 0.4963822 0.5672755 0.2279109 0.6694594 0 1 1 1 1 0 0
#> 4 0.1237089 0.5940458 0.5119936 0.2475795 0.3671093 1 0 0 1 1 0 1
#> 5 0.4092805 0.6123429 0.4464353 0.2369313 0.4173742 1 0 1 1 1 0 0
#> 6 0.3760423 0.4976137 0.5650082 0.2733449 0.7009214 0 0 1 1 1 0 0
#> socioMedium maritalDivorced maritalMarried maritalSingle maritalWidowed employEmployed employRetired
#> 1 0 0 1 0 0 1 0
#> 2 1 1 0 0 0 1 0
#> 3 1 0 1 0 0 1 0
#> 4 0 0 1 0 0 0 0
#> 5 1 0 1 0 0 0 0
#> 6 1 0 1 0 0 1 0
#> employStudent employUnemployed raceBlack raceHispanic raceOther raceWhite eduBS eduGraduate eduHS
#> 1 0 0 0 0 0 1 1 0 0
#> 2 0 0 1 0 0 0 0 0 1
#> 3 0 0 0 0 0 1 1 0 0
#> 4 1 0 0 1 0 0 0 0 1
#> 5 1 0 0 0 0 1 1 0 0
#> 6 0 0 0 1 0 0 0 0 1
#> eduSome college
#> 1 0
#> 2 0
#> 3 0
#> 4 0
#> 5 0
#> 6 0In the preprocessed dataset there are 29 columns. With this final preprocessed dataset AutoTab can be applied.
In order to use the functionalities within AutoTab that are directly related to training the VAE, a reticulate environment must be set up in the live R session. The underlying neural network components in AutoTab are implemented via the Keras and TensorFlow Python libraries. Managing the version requirements at the system level can lead to conflicts across Python packages.
The recommended workflow is for the user to establish a Conda environment that has the dependencies needed for the implementation of AutoTab. A Conda environment is a self-contained environment that allows the user to have control over Python and package versions. The package reticulate is then used to point to the correct Conda environment. AutoTab is built and tested using Python 3.11 and TensorFlow 2.13.
AutoTab includes a dedicated function,
install_autotab_env(), that automatically builds this Conda
environment using a bundled environment specification file. This removes
the need for the user to manually download or create the environment
file themselves.
Running this function will create a Conda environment named
r-autotab-env containing the exact Python, TensorFlow, and
Keras versions AutoTab was built and tested against. If the environment
already exists, install_autotab_env() will detect this and
skip rebuilding it.
Once the environment has been created, it must be activated within the R session using reticulate.
The above will not be run when the vingette is ran. The user must run the above code themselves to activate the conda environment to have the remaining portions of the vingette run. Without an active conda environment the vingette will not run portions of the code that require the conda enviorment.
AutoTab was built using TensorFlow version 2.13 and may be incompatible with some system configurations, particularly older macOS software environments. If compatibility issues arise, we recommend using a virtual environment with the required dependencies or running AutoTab in a cloud-based environment such as Google Colab or a computing cluster.
Now that the preprocessed data and training environment is ready, the next step is to build feat_dist and make it available to AutoTab.
feat_dist = extracting_distribution(data_example)
print(feat_dist)
#> column_name distribution num_params
#> 1 age gaussian 2
#> 2 height gaussian 2
#> 3 weight gaussian 2
#> 4 SBP gaussian 2
#> 5 cholesterol gaussian 2
#> 6 female bernoulli 1
#> 7 history bernoulli 1
#> 8 smoke bernoulli 1
#> 9 hyperten bernoulli 1
#> 10 heart bernoulli 1
#> 11 socio categorical 3
#> 12 marital categorical 4
#> 13 employ categorical 4
#> 14 race categorical 4
#> 15 edu categorical 4The output above is in the same order, from row 1 to row 15 as the input data created during preprocessing. Hence, the additional function in AutoTab, feat_reorder, is not needed in this example.
To show its utility, we will manually reorder feat_dist_temp and then resolve the mismatch order. The code below swaps the rows of age and female in feat_dist_temp.
This results in a new feat_dist that does not match the order of the input data.
To resolve this mis-ordering and force the same order as the preprocessed data the code below can be implemented.
feat_dist_fixed = feat_reorder(feat_dist_shuffled, data)
identical(feat_dist, feat_dist_fixed)
#> [1] TRUENow, the order of feat_dist_fixed matches that of the preprocessed data and is ready to use in AutoTab. Given that the original feat_dist already matched in ordering, we will use this version. To make it accessible to AutoTab the following code must be run:
With the data preprocessed and feat_dist set within the environment, a training statement can be run within AutoTab.
Example One will run a simple VAE train statement. First, encoder_info and decoder_info need to be created. These two lists will describe the structure of the VAE encoder and decoder, along with applying any L2 regularization and BN. Below we show the code to create encoder_info and decoder_info in this example.
encoder_info = list(list("dense", 50, "tanh"), list("dense", 100, "tanh"))
decoder_info = list(list("dense", 100, "tanh"), list("dense", 50, "tanh"))The above example is a simple two layer VAE with hyperbolic tangent activations. Next, we will run a simple VAE train statement. First, we use the AutoTab function reset_seeds. This ensures reproducibility across R and Python. The seed 2026 is chosen as an example, the user may choose their own seed.
The code below will run a VAE step with a beta value of 0.2, a latent space with 10 dimensions, an early stopping criterion of 10 epochs with the default min_delta of 0.001, a batch size of 32, a temperature of 1.2, a learning rate of 0.00001, no spectral normalization, and will train the VAE across 10 epochs.
example1 = VAE_train(data, encoder_info = encoder_info,
decoder_info = decoder_info, latent_dim = 10,
batchsize = 32, epoch = 10, beta = 0.2,
temperature = 1.2, Lip_en = 0, lip_dec = 0,
wait = 10, lr = 0.00001)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - Processing feature 1 with distribution: gaussian and num_params: 2 with index starting at 0
#> Loss - Processing feature 2 with distribution: gaussian and num_params: 2 with index starting at 2
#> Loss - Processing feature 3 with distribution: gaussian and num_params: 2 with index starting at 4
#> Loss - Processing feature 4 with distribution: gaussian and num_params: 2 with index starting at 6
#> Loss - Processing feature 5 with distribution: gaussian and num_params: 2 with index starting at 8
#> Loss - Processing feature 6 with distribution: bernoulli and num_params: 1 with index starting at 10
#> Loss - Processing feature 7 with distribution: bernoulli and num_params: 1 with index starting at 11
#> Loss - Processing feature 8 with distribution: bernoulli and num_params: 1 with index starting at 12
#> Loss - Processing feature 9 with distribution: bernoulli and num_params: 1 with index starting at 13
#> Loss - Processing feature 10 with distribution: bernoulli and num_params: 1 with index starting at 14
#> Loss - Processing feature 11 with distribution: categorical and num_params: 3 with index starting at 15
#> Loss - Processing feature 12 with distribution: categorical and num_params: 4 with index starting at 18
#> Loss - Processing feature 13 with distribution: categorical and num_params: 4 with index starting at 22
#> Loss - Processing feature 14 with distribution: categorical and num_params: 4 with index starting at 26
#> Loss - Processing feature 15 with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - total KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='tf.math.reduce_sum_6/Sum:0', description="created by layer 'tf.math.reduce_sum_6'")
#> Epoch 1/10
#> 250/250 - 7s - loss: 5.1730 - kl_loss: 0.2322 - recon_loss: 4.9409 - val_loss: 4.7679 - val_kl_loss: 0.2145 - val_recon_loss: 4.5534 - 7s/epoch - 28ms/step
#> Epoch 2/10
#> 250/250 - 2s - loss: 4.4627 - kl_loss: 0.2147 - recon_loss: 4.2480 - val_loss: 4.1469 - val_kl_loss: 0.2230 - val_recon_loss: 3.9239 - 2s/epoch - 7ms/step
#> Epoch 3/10
#> 250/250 - 2s - loss: 3.8698 - kl_loss: 0.2352 - recon_loss: 3.6347 - val_loss: 3.6299 - val_kl_loss: 0.2509 - val_recon_loss: 3.3790 - 2s/epoch - 8ms/step
#> Epoch 4/10
#> 250/250 - 2s - loss: 3.4816 - kl_loss: 0.2653 - recon_loss: 3.2163 - val_loss: 3.2958 - val_kl_loss: 0.2797 - val_recon_loss: 3.0161 - 2s/epoch - 7ms/step
#> Epoch 5/10
#> 250/250 - 2s - loss: 3.2085 - kl_loss: 0.2874 - recon_loss: 2.9212 - val_loss: 3.0808 - val_kl_loss: 0.2964 - val_recon_loss: 2.7844 - 2s/epoch - 7ms/step
#> Epoch 6/10
#> 250/250 - 2s - loss: 3.0247 - kl_loss: 0.3000 - recon_loss: 2.7247 - val_loss: 2.8818 - val_kl_loss: 0.3056 - val_recon_loss: 2.5761 - 2s/epoch - 7ms/step
#> Epoch 7/10
#> 250/250 - 2s - loss: 2.8655 - kl_loss: 0.3107 - recon_loss: 2.5548 - val_loss: 2.7858 - val_kl_loss: 0.3151 - val_recon_loss: 2.4707 - 2s/epoch - 7ms/step
#> Epoch 8/10
#> 250/250 - 2s - loss: 2.7531 - kl_loss: 0.3194 - recon_loss: 2.4338 - val_loss: 2.7084 - val_kl_loss: 0.3238 - val_recon_loss: 2.3845 - 2s/epoch - 7ms/step
#> Epoch 9/10
#> 250/250 - 2s - loss: 2.6556 - kl_loss: 0.3279 - recon_loss: 2.3277 - val_loss: 2.6069 - val_kl_loss: 0.3324 - val_recon_loss: 2.2745 - 2s/epoch - 7ms/step
#> Epoch 10/10
#> 250/250 - 2s - loss: 2.5915 - kl_loss: 0.3374 - recon_loss: 2.2541 - val_loss: 2.5001 - val_kl_loss: 0.3443 - val_recon_loss: 2.1558 - 2s/epoch - 6ms/stepAutoTab will print summary information at each epoch. When run in an interactive R session, the user will also see the loss curves update live within the plot viewer, where the y axis is the loss value. This is not displayed in this document.
This viewer displays three curves: the top curve is the overall loss, this is AutoTab’s wTAB-ELBO. The middle curve is the regularization portion of wTAB-ELBO, and the bottom curve is the reconstruction portion. As training progresses, the reconstruction loss typically decreases while the regularization pressure increases, indicating that the latent space is being smoothed towards the prior while the reconstruction of the output data improves. Within each plot the blue line represents the loss on the training data and the green line represents the loss on the validation set.
Example two will show how the user can output the loss per variable type with seperate = 1 added to the training statement and add optional weighting with weighted = 1. This feature of AutoTab can be helpful in identifying whether all three variable types are being trained properly or if any of the variable types are isolated due to gradient imbalance. If an imbalance is identified, the user can add recon_weights to the VAE_train statement with weighted = 1.
reset_seeds(2026)
#> Random seeds reset
recon_weights = c(1, 3, 1)
example2 = VAE_train(data, encoder_info = encoder_info,
decoder_info = decoder_info, latent_dim = 10,
batchsize = 32, epoch = 10, beta = 0.2,
temperature = 1.2, seperate = 1,
weighted = 1, recon_weights = recon_weights,
Lip_en = 0, lip_dec = 0, wait = 10, lr = 0.00001)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - Processing feature 1 with distribution: gaussian and num_params: 2 with index starting at 0
#> Loss - Processing feature 2 with distribution: gaussian and num_params: 2 with index starting at 2
#> Loss - Processing feature 3 with distribution: gaussian and num_params: 2 with index starting at 4
#> Loss - Processing feature 4 with distribution: gaussian and num_params: 2 with index starting at 6
#> Loss - Processing feature 5 with distribution: gaussian and num_params: 2 with index starting at 8
#> Loss - Processing feature 6 with distribution: bernoulli and num_params: 1 with index starting at 10
#> Loss - Processing feature 7 with distribution: bernoulli and num_params: 1 with index starting at 11
#> Loss - Processing feature 8 with distribution: bernoulli and num_params: 1 with index starting at 12
#> Loss - Processing feature 9 with distribution: bernoulli and num_params: 1 with index starting at 13
#> Loss - Processing feature 10 with distribution: bernoulli and num_params: 1 with index starting at 14
#> Loss - Processing feature 11 with distribution: categorical and num_params: 3 with index starting at 15
#> Loss - Processing feature 12 with distribution: categorical and num_params: 4 with index starting at 18
#> Loss - Processing feature 13 with distribution: categorical and num_params: 4 with index starting at 22
#> Loss - Processing feature 14 with distribution: categorical and num_params: 4 with index starting at 26
#> Loss - Processing feature 15 with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - total KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='tf.math.reduce_sum_6/Sum:0', description="created by layer 'tf.math.reduce_sum_6'")
#> Epoch 1/10
#> 250/250 - 9s - loss: 7.1226 - kl_loss: 0.2340 - recon_loss: 6.8887 - cont_loss: 0.5941 - bin_loss: 0.9656 - cat_loss: 3.3977 - val_loss: 6.5549 - val_kl_loss: 0.2181 - val_recon_loss: 6.3368 - val_cont_loss: 0.5768 - val_bin_loss: 0.8772 - val_cat_loss: 3.1284 - 9s/epoch - 35ms/step
#> Epoch 2/10
#> 250/250 - 2s - loss: 6.1246 - kl_loss: 0.2225 - recon_loss: 5.9021 - cont_loss: 0.5547 - bin_loss: 0.8099 - cat_loss: 2.9176 - val_loss: 5.6986 - val_kl_loss: 0.2355 - val_recon_loss: 5.4632 - val_cont_loss: 0.5315 - val_bin_loss: 0.7496 - val_cat_loss: 2.6827 - 2s/epoch - 6ms/step
#> Epoch 3/10
#> 250/250 - 2s - loss: 5.3389 - kl_loss: 0.2561 - recon_loss: 5.0827 - cont_loss: 0.5100 - bin_loss: 0.7086 - cat_loss: 2.4468 - val_loss: 5.0168 - val_kl_loss: 0.2807 - val_recon_loss: 4.7361 - val_cont_loss: 0.4885 - val_bin_loss: 0.6665 - val_cat_loss: 2.2481 - 2s/epoch - 6ms/step
#> Epoch 4/10
#> 250/250 - 2s - loss: 4.8412 - kl_loss: 0.3040 - recon_loss: 4.5371 - cont_loss: 0.4666 - bin_loss: 0.6490 - cat_loss: 2.1235 - val_loss: 4.6268 - val_kl_loss: 0.3255 - val_recon_loss: 4.3013 - val_cont_loss: 0.4380 - val_bin_loss: 0.6322 - val_cat_loss: 1.9666 - 2s/epoch - 6ms/step
#> Epoch 5/10
#> 250/250 - 2s - loss: 4.5095 - kl_loss: 0.3385 - recon_loss: 4.1711 - cont_loss: 0.4224 - bin_loss: 0.6166 - cat_loss: 1.8989 - val_loss: 4.3631 - val_kl_loss: 0.3519 - val_recon_loss: 4.0112 - val_cont_loss: 0.3994 - val_bin_loss: 0.6054 - val_cat_loss: 1.7955 - 2s/epoch - 6ms/step
#> Epoch 6/10
#> 250/250 - 2s - loss: 4.2875 - kl_loss: 0.3570 - recon_loss: 3.9305 - cont_loss: 0.3810 - bin_loss: 0.5951 - cat_loss: 1.7644 - val_loss: 4.1337 - val_kl_loss: 0.3641 - val_recon_loss: 3.7696 - val_cont_loss: 0.3561 - val_bin_loss: 0.5917 - val_cat_loss: 1.6384 - 2s/epoch - 7ms/step
#> Epoch 7/10
#> 250/250 - 2s - loss: 4.0997 - kl_loss: 0.3713 - recon_loss: 3.7285 - cont_loss: 0.3404 - bin_loss: 0.5831 - cat_loss: 1.6387 - val_loss: 4.0039 - val_kl_loss: 0.3763 - val_recon_loss: 3.6276 - val_cont_loss: 0.3231 - val_bin_loss: 0.5757 - val_cat_loss: 1.5774 - 2s/epoch - 7ms/step
#> Epoch 8/10
#> 250/250 - 2s - loss: 3.9643 - kl_loss: 0.3828 - recon_loss: 3.5815 - cont_loss: 0.2997 - bin_loss: 0.5726 - cat_loss: 1.5641 - val_loss: 3.9136 - val_kl_loss: 0.3889 - val_recon_loss: 3.5246 - val_cont_loss: 0.2803 - val_bin_loss: 0.5715 - val_cat_loss: 1.5297 - 2s/epoch - 6ms/step
#> Epoch 9/10
#> 250/250 - 1s - loss: 3.8471 - kl_loss: 0.3947 - recon_loss: 3.4524 - cont_loss: 0.2539 - bin_loss: 0.5644 - cat_loss: 1.5051 - val_loss: 3.7936 - val_kl_loss: 0.4003 - val_recon_loss: 3.3933 - val_cont_loss: 0.2327 - val_bin_loss: 0.5612 - val_cat_loss: 1.4772 - 1s/epoch - 6ms/step
#> Epoch 10/10
#> 250/250 - 2s - loss: 3.7668 - kl_loss: 0.4074 - recon_loss: 3.3594 - cont_loss: 0.2117 - bin_loss: 0.5560 - cat_loss: 1.4797 - val_loss: 3.6628 - val_kl_loss: 0.4157 - val_recon_loss: 3.2471 - val_cont_loss: 0.1874 - val_bin_loss: 0.5501 - val_cat_loss: 1.4094 - 2s/epoch - 6ms/stepIn this output there are 6 more tracked losses, a training and validation loss for each variable type. When run interactively, the plot viewer will also display 3 additional plots (6 in total), one for each variable type.
Example Three will show how the user can apply β annealing or cyclical annealing. The hyperparameters in the VAE_train statement to indicate cyclical annealing are kl_warm, kl_cyclical, n_cycles, and ratio. The code below will train the VAE with cyclical annealing across 2 cycles with a ratio of 0.4.
reset_seeds(2026)
#> Random seeds reset
example3 = VAE_train(data, encoder_info = encoder_info,
decoder_info = decoder_info,
latent_dim = 10, batchsize = 32,
epoch = 10, beta = 0.2,
temperature = 1.2, kl_warm = TRUE,
kl_cyclical = TRUE, n_cycles = 2, ratio = 0.4,
Lip_en = 0, lip_dec = 0, wait = 10, lr = 0.00001)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - Processing feature 1 with distribution: gaussian and num_params: 2 with index starting at 0
#> Loss - Processing feature 2 with distribution: gaussian and num_params: 2 with index starting at 2
#> Loss - Processing feature 3 with distribution: gaussian and num_params: 2 with index starting at 4
#> Loss - Processing feature 4 with distribution: gaussian and num_params: 2 with index starting at 6
#> Loss - Processing feature 5 with distribution: gaussian and num_params: 2 with index starting at 8
#> Loss - Processing feature 6 with distribution: bernoulli and num_params: 1 with index starting at 10
#> Loss - Processing feature 7 with distribution: bernoulli and num_params: 1 with index starting at 11
#> Loss - Processing feature 8 with distribution: bernoulli and num_params: 1 with index starting at 12
#> Loss - Processing feature 9 with distribution: bernoulli and num_params: 1 with index starting at 13
#> Loss - Processing feature 10 with distribution: bernoulli and num_params: 1 with index starting at 14
#> Loss - Processing feature 11 with distribution: categorical and num_params: 3 with index starting at 15
#> Loss - Processing feature 12 with distribution: categorical and num_params: 4 with index starting at 18
#> Loss - Processing feature 13 with distribution: categorical and num_params: 4 with index starting at 22
#> Loss - Processing feature 14 with distribution: categorical and num_params: 4 with index starting at 26
#> Loss - Processing feature 15 with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - total KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='tf.math.reduce_sum_6/Sum:0', description="created by layer 'tf.math.reduce_sum_6'")
#> Cyclical Beta updated to: 0.00000 (epoch 0)
#> Epoch 1/10
#> 250/250 - 9s - loss: 4.9319 - kl_loss: 0.0000e+00 - recon_loss: 4.9319 - val_loss: 4.5290 - val_kl_loss: 0.0000e+00 - val_recon_loss: 4.5290 - 9s/epoch - 35ms/step
#> Cyclical Beta updated to: 0.10000 (epoch 1)
#> Epoch 2/10
#> 250/250 - 1s - loss: 4.3439 - kl_loss: 0.1292 - recon_loss: 4.2147 - val_loss: 4.0182 - val_kl_loss: 0.1368 - val_recon_loss: 3.8815 - 1s/epoch - 6ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 2)
#> Epoch 3/10
#> 250/250 - 2s - loss: 3.8762 - kl_loss: 0.2734 - recon_loss: 3.6028 - val_loss: 3.6342 - val_kl_loss: 0.2788 - val_recon_loss: 3.3554 - 2s/epoch - 7ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 3)
#> Epoch 4/10
#> 250/250 - 2s - loss: 3.4874 - kl_loss: 0.2871 - recon_loss: 3.2003 - val_loss: 3.3016 - val_kl_loss: 0.2965 - val_recon_loss: 3.0050 - 2s/epoch - 7ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 4)
#> Epoch 5/10
#> 250/250 - 2s - loss: 3.2134 - kl_loss: 0.3010 - recon_loss: 2.9124 - val_loss: 3.0846 - val_kl_loss: 0.3074 - val_recon_loss: 2.7771 - 2s/epoch - 7ms/step
#> Cyclical Beta updated to: 0.00000 (epoch 5)
#> Epoch 6/10
#> 250/250 - 1s - loss: 2.6647 - kl_loss: 0.0000e+00 - recon_loss: 2.6647 - val_loss: 2.4679 - val_kl_loss: 0.0000e+00 - val_recon_loss: 2.4679 - 1s/epoch - 5ms/step
#> Cyclical Beta updated to: 0.10000 (epoch 6)
#> Epoch 7/10
#> 250/250 - 2s - loss: 2.6673 - kl_loss: 0.2416 - recon_loss: 2.4257 - val_loss: 2.5749 - val_kl_loss: 0.2510 - val_recon_loss: 2.3239 - 2s/epoch - 6ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 7)
#> Epoch 8/10
#> 250/250 - 2s - loss: 2.7717 - kl_loss: 0.4445 - recon_loss: 2.3273 - val_loss: 2.7143 - val_kl_loss: 0.4066 - val_recon_loss: 2.3077 - 2s/epoch - 7ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 8)
#> Epoch 9/10
#> 250/250 - 2s - loss: 2.6580 - kl_loss: 0.3924 - recon_loss: 2.2656 - val_loss: 2.6063 - val_kl_loss: 0.3844 - val_recon_loss: 2.2219 - 2s/epoch - 7ms/step
#> Cyclical Beta updated to: 0.20000 (epoch 9)
#> Epoch 10/10
#> 250/250 - 2s - loss: 2.5882 - kl_loss: 0.3827 - recon_loss: 2.2055 - val_loss: 2.4955 - val_kl_loss: 0.3843 - val_recon_loss: 2.1112 - 2s/epoch - 7ms/stepIn this output, the beta value that is cyclically updated is tracked.
The next example will focus on temperature warming. The structure from Example 1 will be used with the addition of two hyperparameters: temp_warm and temp_epoch.
reset_seeds(2026)
#> Random seeds reset
example4 = VAE_train(data, encoder_info = encoder_info,
decoder_info = decoder_info, latent_dim = 10,
batchsize = 32, epoch = 10, beta = 0.2,
temperature = 1.2, temp_warm = TRUE,
temp_epoch = 10, Lip_en = 0, lip_dec = 0,
wait = 10, lr = 0.00001)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - Processing feature 1 with distribution: gaussian and num_params: 2 with index starting at 0
#> Loss - Processing feature 2 with distribution: gaussian and num_params: 2 with index starting at 2
#> Loss - Processing feature 3 with distribution: gaussian and num_params: 2 with index starting at 4
#> Loss - Processing feature 4 with distribution: gaussian and num_params: 2 with index starting at 6
#> Loss - Processing feature 5 with distribution: gaussian and num_params: 2 with index starting at 8
#> Loss - Processing feature 6 with distribution: bernoulli and num_params: 1 with index starting at 10
#> Loss - Processing feature 7 with distribution: bernoulli and num_params: 1 with index starting at 11
#> Loss - Processing feature 8 with distribution: bernoulli and num_params: 1 with index starting at 12
#> Loss - Processing feature 9 with distribution: bernoulli and num_params: 1 with index starting at 13
#> Loss - Processing feature 10 with distribution: bernoulli and num_params: 1 with index starting at 14
#> Loss - Processing feature 11 with distribution: categorical and num_params: 3 with index starting at 15
#> Loss - Processing feature 12 with distribution: categorical and num_params: 4 with index starting at 18
#> Loss - Processing feature 13 with distribution: categorical and num_params: 4 with index starting at 22
#> Loss - Processing feature 14 with distribution: categorical and num_params: 4 with index starting at 26
#> Loss - Processing feature 15 with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - total KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='tf.math.reduce_sum_6/Sum:0', description="created by layer 'tf.math.reduce_sum_6'")
#> Temperature updated to: 1.2
#> Epoch 1/10
#> 250/250 - 10s - loss: 5.1730 - kl_loss: 0.2322 - recon_loss: 4.9409 - val_loss: 4.7679 - val_kl_loss: 0.2145 - val_recon_loss: 4.5534 - 10s/epoch - 38ms/step
#> Temperature updated to: 1.08
#> Epoch 2/10
#> 250/250 - 2s - loss: 4.7144 - kl_loss: 0.2164 - recon_loss: 4.4980 - val_loss: 4.3693 - val_kl_loss: 0.2262 - val_recon_loss: 4.1430 - 2s/epoch - 7ms/step
#> Temperature updated to: 0.96
#> Epoch 3/10
#> 250/250 - 2s - loss: 4.3118 - kl_loss: 0.2411 - recon_loss: 4.0706 - val_loss: 4.0221 - val_kl_loss: 0.2591 - val_recon_loss: 3.7630 - 2s/epoch - 6ms/step
#> Temperature updated to: 0.84
#> Epoch 4/10
#> 250/250 - 1s - loss: 4.1006 - kl_loss: 0.2773 - recon_loss: 3.8233 - val_loss: 3.8472 - val_kl_loss: 0.2952 - val_recon_loss: 3.5519 - 1s/epoch - 6ms/step
#> Temperature updated to: 0.72
#> Epoch 5/10
#> 250/250 - 2s - loss: 4.0233 - kl_loss: 0.3091 - recon_loss: 3.7142 - val_loss: 3.8276 - val_kl_loss: 0.3239 - val_recon_loss: 3.5037 - 2s/epoch - 6ms/step
#> Temperature updated to: 0.6
#> Epoch 6/10
#> 250/250 - 2s - loss: 4.1107 - kl_loss: 0.3367 - recon_loss: 3.7740 - val_loss: 3.8605 - val_kl_loss: 0.3516 - val_recon_loss: 3.5089 - 2s/epoch - 7ms/step
#> Temperature updated to: 0.5
#> Epoch 7/10
#> 250/250 - 2s - loss: 4.2286 - kl_loss: 0.3686 - recon_loss: 3.8600 - val_loss: 4.0652 - val_kl_loss: 0.3840 - val_recon_loss: 3.6812 - 2s/epoch - 7ms/step
#> Temperature updated to: 0.5
#> Epoch 8/10
#> 250/250 - 1s - loss: 4.0174 - kl_loss: 0.3969 - recon_loss: 3.6205 - val_loss: 3.9407 - val_kl_loss: 0.4086 - val_recon_loss: 3.5322 - 1s/epoch - 6ms/step
#> Temperature updated to: 0.5
#> Epoch 9/10
#> 250/250 - 2s - loss: 3.8597 - kl_loss: 0.4180 - recon_loss: 3.4417 - val_loss: 3.7813 - val_kl_loss: 0.4267 - val_recon_loss: 3.3546 - 2s/epoch - 7ms/step
#> Temperature updated to: 0.5
#> Epoch 10/10
#> 250/250 - 1s - loss: 3.7730 - kl_loss: 0.4362 - recon_loss: 3.3369 - val_loss: 3.6164 - val_kl_loss: 0.4461 - val_recon_loss: 3.1704 - 1s/epoch - 6ms/stepThe output now tracks the temperature at each epoch. This value is decreasing after each additional epoch.
The last example shows additional options within the encoder_info and decoder_info statements.
encoder_info5 = list(list("dense", 50, "tanh", 0, 0, TRUE),
list("dense", 100, "tanh", 0, 0, TRUE))
decoder_info5 = list(list("dense", 100, "tanh", 1),
list("dense", 50, "tanh"))reset_seeds(2026)
#> Random seeds reset
example5 = VAE_train(data, encoder_info = encoder_info5,
decoder_info = decoder_info5, latent_dim = 10,
batchsize = 32, epoch = 10, beta = 0.2,
temperature = 1.2, Lip_en = 0, lip_dec = 0,
wait = 10, lr = 0.00001)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - Processing feature 1 with distribution: gaussian and num_params: 2 with index starting at 0
#> Loss - Processing feature 2 with distribution: gaussian and num_params: 2 with index starting at 2
#> Loss - Processing feature 3 with distribution: gaussian and num_params: 2 with index starting at 4
#> Loss - Processing feature 4 with distribution: gaussian and num_params: 2 with index starting at 6
#> Loss - Processing feature 5 with distribution: gaussian and num_params: 2 with index starting at 8
#> Loss - Processing feature 6 with distribution: bernoulli and num_params: 1 with index starting at 10
#> Loss - Processing feature 7 with distribution: bernoulli and num_params: 1 with index starting at 11
#> Loss - Processing feature 8 with distribution: bernoulli and num_params: 1 with index starting at 12
#> Loss - Processing feature 9 with distribution: bernoulli and num_params: 1 with index starting at 13
#> Loss - Processing feature 10 with distribution: bernoulli and num_params: 1 with index starting at 14
#> Loss - Processing feature 11 with distribution: categorical and num_params: 3 with index starting at 15
#> Loss - Processing feature 12 with distribution: categorical and num_params: 4 with index starting at 18
#> Loss - Processing feature 13 with distribution: categorical and num_params: 4 with index starting at 22
#> Loss - Processing feature 14 with distribution: categorical and num_params: 4 with index starting at 26
#> Loss - Processing feature 15 with distribution: categorical and num_params: 4 with index starting at 30
#> Loss - total KerasTensor(type_spec=TensorSpec(shape=(), dtype=tf.float32, name=None), name='tf.math.reduce_sum_6/Sum:0', description="created by layer 'tf.math.reduce_sum_6'")
#> Epoch 1/10
#> 250/250 - 10s - loss: 9.1136 - kl_loss: 3.1167 - recon_loss: 5.9951 - val_loss: 6.8337 - val_kl_loss: 1.5032 - val_recon_loss: 5.3288 - 10s/epoch - 39ms/step
#> Epoch 2/10
#> 250/250 - 2s - loss: 7.6085 - kl_loss: 2.4993 - recon_loss: 5.1074 - val_loss: 6.7814 - val_kl_loss: 2.1216 - val_recon_loss: 4.6579 - 2s/epoch - 7ms/step
#> Epoch 3/10
#> 250/250 - 2s - loss: 6.3006 - kl_loss: 2.0061 - recon_loss: 4.2928 - val_loss: 5.7638 - val_kl_loss: 1.8045 - val_recon_loss: 3.9576 - 2s/epoch - 7ms/step
#> Epoch 4/10
#> 250/250 - 1s - loss: 5.3509 - kl_loss: 1.6181 - recon_loss: 3.7311 - val_loss: 4.9019 - val_kl_loss: 1.4617 - val_recon_loss: 3.4384 - 1s/epoch - 6ms/step
#> Epoch 5/10
#> 250/250 - 2s - loss: 4.6167 - kl_loss: 1.3125 - recon_loss: 3.3025 - val_loss: 4.2798 - val_kl_loss: 1.1813 - val_recon_loss: 3.0967 - 2s/epoch - 8ms/step
#> Epoch 6/10
#> 250/250 - 1s - loss: 4.0958 - kl_loss: 1.0726 - recon_loss: 3.0214 - val_loss: 3.7911 - val_kl_loss: 0.9642 - val_recon_loss: 2.8251 - 1s/epoch - 6ms/step
#> Epoch 7/10
#> 250/250 - 2s - loss: 3.7213 - kl_loss: 0.8894 - recon_loss: 2.8301 - val_loss: 3.5066 - val_kl_loss: 0.8057 - val_recon_loss: 2.6991 - 2s/epoch - 7ms/step
#> Epoch 8/10
#> 250/250 - 2s - loss: 3.4720 - kl_loss: 0.7574 - recon_loss: 2.7129 - val_loss: 3.3319 - val_kl_loss: 0.6858 - val_recon_loss: 2.6443 - 2s/epoch - 7ms/step
#> Epoch 9/10
#> 250/250 - 2s - loss: 3.2895 - kl_loss: 0.6577 - recon_loss: 2.6300 - val_loss: 3.1610 - val_kl_loss: 0.5995 - val_recon_loss: 2.5598 - 2s/epoch - 7ms/step
#> Epoch 10/10
#> 250/250 - 1s - loss: 3.1677 - kl_loss: 0.5867 - recon_loss: 2.5793 - val_loss: 3.0014 - val_kl_loss: 0.5374 - val_recon_loss: 2.4623 - 1s/epoch - 6ms/stepAutoTab has a dedicated help page for applying MoG priors and therefore these examples are excluded.
Once the VAE has been trained, the next step is sampling from the VAE to understand how well the generated data represents the input data. AutoTab has three functions that the user can utilize to rebuild a trained VAE and sample from it: Encoder_weights, encoder_latent and Decoder_weights.
Sampling from the VAE can be done in five steps:
After these five steps are executed, the user can determine how the generated data will be transformed back to its original form (e.g., reversing min-max-scaling). Step 5 provides access to the generative distributional parameters learned by the VAE. At this stage, the user may choose how to sample from these distributions to produce synthetic observations. Suggestions are provided in this section; however, alternative approaches may also be used.
Below we outline the execution from step 1 to 5 using the trained VAE from Example one. Example one was a simple VAE with a set beta at 0.2, a latent dimension of 10, and two hidden layers in both the encoder and decoder.
Step One: Extract the trained model weights and biases.
library("dplyr")
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library("keras")
trained_model = example1$trained_modelThe above code is pulling the trained_model from the saved training object, example1. The other saved information in example1 is the loss history, this is not needed for sampling from a trained VAE.
Step Two: Rebuild the trained encoder.
weights_encoder = Encoder_weights(encoder_layers = 2,
trained_model, lip_enc = 0, pi_enc = 0,
BNenc_layers = 0, learn_BN = 0)
latent_encoder = encoder_latent(encoder_input = data,
encoder_info = encoder_info, latent_dim = 10,
Lip_en = 0, power_iterations = 0)
print(latent_encoder)
#> Model: "encoder"
#> _____________________________________________________________________________________________________________
#> Layer (type) Output Shape Param # Connected to
#> =============================================================================================================
#> input_3 (InputLayer) [(None, 29)] 0 []
#> dense_4 (Dense) (None, 50) 1500 ['input_3[0][0]']
#> dense_5 (Dense) (None, 100) 5100 ['dense_4[0][0]']
#> z_mean (Dense) (None, 10) 1010 ['dense_5[0][0]']
#> z_log_var (Dense) (None, 10) 1010 ['dense_5[0][0]']
#> =============================================================================================================
#> Total params: 8620 (33.67 KB)
#> Trainable params: 8620 (33.67 KB)
#> Non-trainable params: 0 (0.00 Byte)
#> _____________________________________________________________________________________________________________The output above shows that the encoder has been rebuilt. The input layer has 29 columns, the two dense layers have 50 and 100 nodes, and the results of the encoder are z_mean and z_log_var. Latent_encoder can be thought of as the skeleton for the encoder. Now, we need to apply weights to the encoder skeleton.
Now, the encoder is built with the learned weights and biases. These are the typical weights and biases seen in a neural network.
Step Three: Rebuild the trained decoder.
weights_decoder = Decoder_weights(encoder_layers = 2,
trained_model, lip_enc = 0, pi_enc = 0,
prior_learn = "fixed", BNenc_layers = 0,
learn_BN = 0)
decoder = decoder_model(decoder_input = NULL,
decoder_info = decoder_info, latent_dim = 10,
feat_dist = feat_dist, lip_dec = 0, pi_dec = 0)
#> Output- Processing feature age with distribution: gaussian and num_params: 2 with index starting at 0
#> Output- Processing feature height with distribution: gaussian and num_params: 2 with index starting at 2
#> Output- Processing feature weight with distribution: gaussian and num_params: 2 with index starting at 4
#> Output- Processing feature SBP with distribution: gaussian and num_params: 2 with index starting at 6
#> Output- Processing feature cholesterol with distribution: gaussian and num_params: 2 with index starting at 8
#> Output- Processing feature female with distribution: bernoulli and num_params: 1 with index starting at 10
#> Output- Processing feature history with distribution: bernoulli and num_params: 1 with index starting at 11
#> Output- Processing feature smoke with distribution: bernoulli and num_params: 1 with index starting at 12
#> Output- Processing feature hyperten with distribution: bernoulli and num_params: 1 with index starting at 13
#> Output- Processing feature heart with distribution: bernoulli and num_params: 1 with index starting at 14
#> Output- Processing feature socio with distribution: categorical and num_params: 3 with index starting at 15
#> Output- Processing feature marital with distribution: categorical and num_params: 4 with index starting at 18
#> Output- Processing feature employ with distribution: categorical and num_params: 4 with index starting at 22
#> Output- Processing feature race with distribution: categorical and num_params: 4 with index starting at 26
#> Output- Processing feature edu with distribution: categorical and num_params: 4 with index starting at 30
print(decoder)Model: "model_2"
_____________________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
=============================================================================================================
input_4 (InputLayer) [(None, 10)] 0 []
dense_6 (Dense) (None, 100) 1100 ['input_4[0][0]']
dense_7 (Dense) (None, 50) 5050 ['dense_6[0][0]']
decoder_output_linear (Dense) (None, 34) 1734 ['dense_7[0][0]']
tf.compat.v1.shape_57 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_59 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_61 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_63 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_65 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_55 ( () 0 ['tf.compat.v1.shape_57[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_58 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_57 ( () 0 ['tf.compat.v1.shape_59[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_60 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_59 ( () 0 ['tf.compat.v1.shape_61[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_62 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_61 ( () 0 ['tf.compat.v1.shape_63[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_64 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_63 ( () 0 ['tf.compat.v1.shape_65[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_66 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_72 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_73 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_74 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_75 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_76 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.slice_55 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_55[0][
0]']
tf.__operators__.getitem_56 ( () 0 ['tf.compat.v1.shape_58[0][0]']
SlicingOpLambda)
tf.slice_57 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_57[0][
0]']
tf.__operators__.getitem_58 ( () 0 ['tf.compat.v1.shape_60[0][0]']
SlicingOpLambda)
tf.slice_59 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_59[0][
0]']
tf.__operators__.getitem_60 ( () 0 ['tf.compat.v1.shape_62[0][0]']
SlicingOpLambda)
tf.slice_61 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_61[0][
0]']
tf.__operators__.getitem_62 ( () 0 ['tf.compat.v1.shape_64[0][0]']
SlicingOpLambda)
tf.slice_63 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_63[0][
0]']
tf.__operators__.getitem_64 ( () 0 ['tf.compat.v1.shape_66[0][0]']
SlicingOpLambda)
tf.compat.v1.shape_67 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_68 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_69 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_70 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.compat.v1.shape_71 (TFOpLa (2,) 0 ['decoder_output_linear[0][0]']
mbda)
tf.__operators__.getitem_70 ( () 0 ['tf.compat.v1.shape_72[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_71 ( () 0 ['tf.compat.v1.shape_73[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_72 ( () 0 ['tf.compat.v1.shape_74[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_73 ( () 0 ['tf.compat.v1.shape_75[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_74 ( () 0 ['tf.compat.v1.shape_76[0][0]']
SlicingOpLambda)
tf.math.tanh_10 (TFOpLambda) (None, 1) 0 ['tf.slice_55[0][0]']
tf.slice_56 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_56[0][
0]']
tf.math.tanh_11 (TFOpLambda) (None, 1) 0 ['tf.slice_57[0][0]']
tf.slice_58 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_58[0][
0]']
tf.math.tanh_12 (TFOpLambda) (None, 1) 0 ['tf.slice_59[0][0]']
tf.slice_60 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_60[0][
0]']
tf.math.tanh_13 (TFOpLambda) (None, 1) 0 ['tf.slice_61[0][0]']
tf.slice_62 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_62[0][
0]']
tf.math.tanh_14 (TFOpLambda) (None, 1) 0 ['tf.slice_63[0][0]']
tf.slice_64 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_64[0][
0]']
tf.__operators__.getitem_65 ( () 0 ['tf.compat.v1.shape_67[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_66 ( () 0 ['tf.compat.v1.shape_68[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_67 ( () 0 ['tf.compat.v1.shape_69[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_68 ( () 0 ['tf.compat.v1.shape_70[0][0]']
SlicingOpLambda)
tf.__operators__.getitem_69 ( () 0 ['tf.compat.v1.shape_71[0][0]']
SlicingOpLambda)
tf.slice_70 (TFOpLambda) (None, 3) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_70[0][
0]']
tf.slice_71 (TFOpLambda) (None, 4) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_71[0][
0]']
tf.slice_72 (TFOpLambda) (None, 4) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_72[0][
0]']
tf.slice_73 (TFOpLambda) (None, 4) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_73[0][
0]']
tf.slice_74 (TFOpLambda) (None, 4) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_74[0][
0]']
tf.math.add_22 (TFOpLambda) (None, 1) 0 ['tf.math.tanh_10[0][0]']
tf.math.softplus_5 (TFOpLambd (None, 1) 0 ['tf.slice_56[0][0]']
a)
tf.math.add_23 (TFOpLambda) (None, 1) 0 ['tf.math.tanh_11[0][0]']
tf.math.softplus_6 (TFOpLambd (None, 1) 0 ['tf.slice_58[0][0]']
a)
tf.math.add_24 (TFOpLambda) (None, 1) 0 ['tf.math.tanh_12[0][0]']
tf.math.softplus_7 (TFOpLambd (None, 1) 0 ['tf.slice_60[0][0]']
a)
tf.math.add_25 (TFOpLambda) (None, 1) 0 ['tf.math.tanh_13[0][0]']
tf.math.softplus_8 (TFOpLambd (None, 1) 0 ['tf.slice_62[0][0]']
a)
tf.math.add_26 (TFOpLambda) (None, 1) 0 ['tf.math.tanh_14[0][0]']
tf.math.softplus_9 (TFOpLambd (None, 1) 0 ['tf.slice_64[0][0]']
a)
tf.slice_65 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_65[0][
0]']
tf.slice_66 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_66[0][
0]']
tf.slice_67 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_67[0][
0]']
tf.slice_68 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_68[0][
0]']
tf.slice_69 (TFOpLambda) (None, 1) 0 ['decoder_output_linear[0][0]',
'tf.__operators__.getitem_69[0][
0]']
tf.math.tanh_15 (TFOpLambda) (None, 3) 0 ['tf.slice_70[0][0]']
tf.math.tanh_16 (TFOpLambda) (None, 4) 0 ['tf.slice_71[0][0]']
tf.math.tanh_17 (TFOpLambda) (None, 4) 0 ['tf.slice_72[0][0]']
tf.math.tanh_18 (TFOpLambda) (None, 4) 0 ['tf.slice_73[0][0]']
tf.math.tanh_19 (TFOpLambda) (None, 4) 0 ['tf.slice_74[0][0]']
tf.math.truediv_10 (TFOpLambd (None, 1) 0 ['tf.math.add_22[0][0]']
a)
tf.clip_by_value_17 (TFOpLamb (None, 1) 0 ['tf.math.softplus_5[0][0]']
da)
tf.math.truediv_11 (TFOpLambd (None, 1) 0 ['tf.math.add_23[0][0]']
a)
tf.clip_by_value_18 (TFOpLamb (None, 1) 0 ['tf.math.softplus_6[0][0]']
da)
tf.math.truediv_12 (TFOpLambd (None, 1) 0 ['tf.math.add_24[0][0]']
a)
tf.clip_by_value_19 (TFOpLamb (None, 1) 0 ['tf.math.softplus_7[0][0]']
da)
tf.math.truediv_13 (TFOpLambd (None, 1) 0 ['tf.math.add_25[0][0]']
a)
tf.clip_by_value_20 (TFOpLamb (None, 1) 0 ['tf.math.softplus_8[0][0]']
da)
tf.math.truediv_14 (TFOpLambd (None, 1) 0 ['tf.math.add_26[0][0]']
a)
tf.clip_by_value_21 (TFOpLamb (None, 1) 0 ['tf.math.softplus_9[0][0]']
da)
tf.math.multiply_35 (TFOpLamb (None, 1) 0 ['tf.slice_65[0][0]']
da)
tf.math.multiply_36 (TFOpLamb (None, 1) 0 ['tf.slice_66[0][0]']
da)
tf.math.multiply_37 (TFOpLamb (None, 1) 0 ['tf.slice_67[0][0]']
da)
tf.math.multiply_38 (TFOpLamb (None, 1) 0 ['tf.slice_68[0][0]']
da)
tf.math.multiply_39 (TFOpLamb (None, 1) 0 ['tf.slice_69[0][0]']
da)
tf.math.multiply_40 (TFOpLamb (None, 3) 0 ['tf.math.tanh_15[0][0]']
da)
tf.math.multiply_41 (TFOpLamb (None, 4) 0 ['tf.math.tanh_16[0][0]']
da)
tf.math.multiply_42 (TFOpLamb (None, 4) 0 ['tf.math.tanh_17[0][0]']
da)
tf.math.multiply_43 (TFOpLamb (None, 4) 0 ['tf.math.tanh_18[0][0]']
da)
tf.math.multiply_44 (TFOpLamb (None, 4) 0 ['tf.math.tanh_19[0][0]']
da)
tf.concat_5 (TFOpLambda) (None, 2) 0 ['tf.math.truediv_10[0][0]',
'tf.clip_by_value_17[0][0]']
tf.concat_6 (TFOpLambda) (None, 2) 0 ['tf.math.truediv_11[0][0]',
'tf.clip_by_value_18[0][0]']
tf.concat_7 (TFOpLambda) (None, 2) 0 ['tf.math.truediv_12[0][0]',
'tf.clip_by_value_19[0][0]']
tf.concat_8 (TFOpLambda) (None, 2) 0 ['tf.math.truediv_13[0][0]',
'tf.clip_by_value_20[0][0]']
tf.concat_9 (TFOpLambda) (None, 2) 0 ['tf.math.truediv_14[0][0]',
'tf.clip_by_value_21[0][0]']
tf.math.sigmoid_5 (TFOpLambda (None, 1) 0 ['tf.math.multiply_35[0][0]']
)
tf.math.sigmoid_6 (TFOpLambda (None, 1) 0 ['tf.math.multiply_36[0][0]']
)
tf.math.sigmoid_7 (TFOpLambda (None, 1) 0 ['tf.math.multiply_37[0][0]']
)
tf.math.sigmoid_8 (TFOpLambda (None, 1) 0 ['tf.math.multiply_38[0][0]']
)
tf.math.sigmoid_9 (TFOpLambda (None, 1) 0 ['tf.math.multiply_39[0][0]']
)
r_layer_5 (RLayer) (None, 3) 0 ['tf.math.multiply_40[0][0]']
r_layer_6 (RLayer) (None, 4) 0 ['tf.math.multiply_41[0][0]']
r_layer_7 (RLayer) (None, 4) 0 ['tf.math.multiply_42[0][0]']
r_layer_8 (RLayer) (None, 4) 0 ['tf.math.multiply_43[0][0]']
r_layer_9 (RLayer) (None, 4) 0 ['tf.math.multiply_44[0][0]']
concatenate_1 (Concatenate) (None, 34) 0 ['tf.concat_5[0][0]',
'tf.concat_6[0][0]',
'tf.concat_7[0][0]',
'tf.concat_8[0][0]',
'tf.concat_9[0][0]',
'tf.math.sigmoid_5[0][0]',
'tf.math.sigmoid_6[0][0]',
'tf.math.sigmoid_7[0][0]',
'tf.math.sigmoid_8[0][0]',
'tf.math.sigmoid_9[0][0]',
'r_layer_5[0][0]',
'r_layer_6[0][0]',
'r_layer_7[0][0]',
'r_layer_8[0][0]',
'r_layer_9[0][0]']
=============================================================================================================
Total params: 7884 (30.80 KB)
Trainable params: 7884 (30.80 KB)
Non-trainable params: 0 (0.00 Byte)
_____________________________________________________________________________________________________________Based on the output above it is clear that decoder takes a sample from the latent space with the correct dimension of 10, followed by dense hidden layers with 100 and 50 nodes.
Now that the decoder skeleton is built, the learned weights need to be applied.
Now, the decoder is built with the learned weights.
Step Four: Sample the latent space or prior.
Now that the VAE is built, the user can either sample from the chosen prior, or sample from the learned posterior (the latent space). In this example, a sample from the prior will be used, in this case N(0, 1).
This latent sample can be used in step 5.
If the user desired to sample directly from the latent space, or to visualize the latent space from a latent sample, the steps below could be used.
input_data <- as.matrix(data)
latent_space = predict(latent_encoder, input_data)
#> 313/313 - 1s - 562ms/epoch - 2ms/step
dim(latent_space[[1]])
#> [1] 10000 10
dim(latent_space[[2]])
#> [1] 10000 10The output in the console shows 313 due to our batch size of 32 over 10,000 samples (10,000/32 ≈ 313). This is an automatic output from Keras by using the function predict.
The resulting latent_space is a list with two objects, one that represents the mean and one that represents that log variance that are learned through the encoder. Each one has a dimension of 10,000 by 10. This is the size of the data by the chosen latent dimension.
Now that the latent space is separated into its respective parts the
reparameterization trick can be applied, as implemented in
Latent_sample().
Step Five: Pass the sampled latent vectors through the trained decoder.
This step will use the sample from the latent space and put it through the trained decoder to create a generative output.
decoder_sample now has the generative parameters from the trained VAE.
The dimension of this dataset is the total sample size by the decoder output layer size (in this example 10000 by 34). In our example, the input data has 29 columns, 5 of which are continuous. The decoder outputs 2 columns for each continuous variable, a mean and SD. Hence, there are 5 additional columns in the decoder output.
This section discusses possible options for creating the synthetic dataset based on the generative results from the decoder. Some recommendations are provided for AutoTab; however, the final choice is up to the user, and multiple approaches are possible. If additional preprocessing steps were applied to the data, they must be reversed accordingly. For this reason, the generative step is application dependent, and careful consideration should be given when implementing it.
The order of the generative sample will match the order of feat_dist. Based on the previous output of feat_dist, the first 5 variables were continuous. That means the first 10 columns belong to the continuous variables. To generate a sample from each output we sample from the following distribution:
\[x \sim N\left(\mu_\theta(z), \sigma^2_\theta(z)\right),\]
where \(\theta\) are the parameters of the decoder and \(z\) is the latent sample. The parameters for the distribution are within the R object decoder_sample created in Step Five.
The generative parameters for a continuous variable are the mean and SD. The user may decide to sample from a distribution using these values or first undo the min-max-scaling and sample from that distribution. In this example, using the variable age, we first undo the min-max-scaling and then sample.
Let:
\[x = R * x_{scaled} + min,\]
where \(R = max - min\), min is the minimum of age in the original data, max is the maximum of age in the original data, and \(x\) is the unscaled decoder output. Then:
\[E[x] = E[R * x_{scaled} + min] = R \, E[x_{scaled}] + min\]
and
\[Var(x) = Var(R * x_{scaled} + min) = R^2 Var(x_{scaled})\]
so,
\[\sigma = R * \sigma_{scaled}, \quad \mu = R * \mu_{scaled} + min\]
The code to implement this, on the variable age as an illustrative example, is shown below.
scaling_min = min(data_example$age)
scaling_max = max(data_example$age)
decoder_sample[,1] = (decoder_sample[,1] * (scaling_max - scaling_min)
+ scaling_min)
decoder_sample[,2] = decoder_sample[,2] * (scaling_max - scaling_min)Now the first row of age can be sampled from N(mean, SD²), the unscaled generative Gaussian distribution using the following code:
For binary variables, the user can decide to either sample from a Bernoulli distribution or choose the maximum a posteriori (MAP). A MAP estimate corresponds to the most probable class label given the probabilistic output (Murphy 2012). Choosing between these approaches is left to the user. The code below displays how MAP can be applied to the first row of the 11th column, which is a binary variable based on feat_dist.
The code above takes the predicted value and maps it to 1 since the probability was greater than 0.5.
For categorical variables, the user can choose to sample from a multinomial distribution with one draw based on the column probabilities or use another MAP approach.
Choosing between another generative sample from the categorical distribution versus taking the most probable category from the decoder output is again a design choice and is up to the user. The code below displays how argmax can be applied to the first row of columns 16–18, which represents a categorical variable based on feat_dist.
After applying argmax, the column will indicate the one-hot-coded vector.
This vignette walked through a complete AutoTab workflow: simulating and preprocessing mixed-type tabular data, setting up the required Python/ TensorFlow environment, training a variational autoencoder under five different hyperparameter configurations, and rebuilding a trained encoder and decoder to sample synthetic observations. The five training examples demonstrated core AutoTab features including weighted reconstruction loss, Beta annealing, Gumbel-softmax temperature annealing, and regularization options, though they are not intended to represent an optimized model for this dataset.
For guidance on Mixture-of-Gaussians priors, see the
mog_prior help page. For further details on any individual
function discussed here, consult the corresponding function
documentation (e.g., ?VAE_train,
?Encoder_weights, ?Latent_sample).