Semi-Supervised Learning with Deep Generative Models

Chainer implementation of Variational AutoEncoder(VAE) model M1, M2, M1+M2

Requirements

• Chainer 1.8+

• sklearn

To visualize results, you need

• matplotlib.patches

• PIL

• pandas

Download MNIST

run mnist-tools.py to download and extract MNIST.

How to label my own dataset?

You can provide label information by filename.

format:

{label_id}_{unique_filename}.{extension}

regex:

([0-9]+)_.+.(bmp|png|jpg)

e.g. MNIST

M1

Parameters

| params | value | |:-----------|------------:| | OS | Windows 7 | | GPU | GeForce GTX 970M | | ndim_z | 2 | | encoder_apply_dropout | False | | decoder_apply_dropout | False | | encoder_apply_batchnorm | True | | decoder_apply_batchnorm | True | | encoder_apply_batchnorm_to_input | True | | decoder_apply_batchnorm_to_input | True | | encoder_units | [600, 600] | | decoder_units | [600, 600] | | gradient_clipping | 1.0 | | learning_rate | 0.0003 | | gradient_momentum | 0.9 | | gradient_clipping | 1.0 | | nonlinear | softplus|

Result

Latent space

M2

Parameters

| params | value | |:-----------|------------:| | OS | Windows 7 | | GPU | GeForce GTX 970M | | ndim_z | 50 | | encoder_xy_z_apply_dropout | False | | encoder_x_y_apply_dropout | False | | decoder_apply_dropout | False | | encoder_xy_z_apply_batchnorm_to_input | True | | encoder_x_y_apply_batchnorm_to_input | True | | decoder_apply_batchnorm_to_input | True | | encoder_xy_z_apply_batchnorm | True | | encoder_x_y_apply_batchnorm | True | | decoder_apply_batchnorm | True | | encoder_xy_z_hidden_units | [500] | | encoder_x_y_hidden_units | [500] | | decoder_hidden_units | [500] | | batchnorm_before_activation | True | | gradient_clipping | 5.0 | | learning_rate | 0.0003 | | gradient_momentum | 0.9 | | gradient_clipping | 1.0 | | nonlinear | softplus|

Result

Classification

Training details

| data | # | |:-----------|------------:| | labeled | 100 | | unlabeled | 49900 | | validation | 10000 | | test | 10000 |

| * | # | |:-----------|------------:| | epochs | 490 | | minutes | 1412 | | weight updates per epoch | 2000 |

Validation accuracy:

Test accuracy: 0.9018

Analogies

run analogy.py after training

Model was trained with...

| data | # | |:-----------|------------:| | labeled | 100 | | unlabeled | 49900 |

| data | # | |:-----------|------------:| | labeled | 10000 | | unlabeled | 40000 |

| data | # | |:-----------|------------:| | labeled | 50000 | | unlabeled | 0 |

M1+M2

Parameters

M1

| params | value | |:-----------|------------:| | OS | Windows 7 | | GPU | GeForce GTX 970M | | ndim_z | 2 | | encoder_apply_dropout | False | | decoder_apply_dropout | False | | encoder_apply_batchnorm | True | | decoder_apply_batchnorm | True | | encoder_apply_batchnorm_to_input | True | | decoder_apply_batchnorm_to_input | True | | encoder_units | [600, 600] | | decoder_units | [600, 600] | | gradient_clipping | 1.0 | | learning_rate | 0.0003 | | gradient_momentum | 0.9 | | gradient_clipping | 1.0 | | nonlinear | softplus|

We trained M1 for 500 epochs before starting training of M2.

| * | # | |:-----------|------------:| | epochs | 500 | | minutes | 860 | | weight updates per epoch | 2000 |

M2

| params | value | |:-----------|------------:| | OS | Windows 7 | | GPU | GeForce GTX 970M | | ndim_z | 50 | | encoder_xy_z_apply_dropout | False | | encoder_x_y_apply_dropout | False | | decoder_apply_dropout | False | | encoder_xy_z_apply_batchnorm_to_input | True | | encoder_x_y_apply_batchnorm_to_input | True | | decoder_apply_batchnorm_to_input | True | | encoder_xy_z_apply_batchnorm | True | | encoder_x_y_apply_batchnorm | True | | decoder_apply_batchnorm | True | | encoder_xy_z_hidden_units | [500] | | encoder_x_y_hidden_units | [500] | | decoder_hidden_units | [500] | | batchnorm_before_activation | True | | gradient_clipping | 5.0 | | learning_rate | 0.0003 | | gradient_momentum | 0.9 | | gradient_clipping | 1.0 | | nonlinear | softplus|

Result

Classification

Training details

| data | # | |:-----------|------------:| | labeled | 100 | | unlabeled | 49900 | | validation | 10000 | | test | 10000 |

| * | # | |:-----------|------------:| | epochs | 600 | | minutes | 4920 | | weight updates per epoch | 5000 |

Validation accuracy:

Test accuracy

seed1: 0.954

seed2: 0.951