Neural Arithmetic Logic Units

[WIP]

This is a PyTorch implementation of Neural Arithmetic Logic Units by Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer and Phil Blunsom.

API

from models import *# single layer modulesNeuralAccumulatorCell(in_dim, out_dim)NeuralArithmeticLogicUnitCell(in_dim, out_dim)# stacked layersNAC(num_layers, in_dim, hidden_dim, out_dim)NALU(num_layers, in_dim, hidden_dim, out_dim)

Experiments

To reproduce "Numerical Extrapolation Failures in Neural Networks" (Section 1.1), run:

python failures.py

This should generate the following plot:

To reproduce "Simple Function Learning Tasks" (Section 4.1), run:

python function_learning.py

This should generate a text file called interpolation.txt with the following results. (Currently only supports interpolation, I'm working on the rest)

| | Relu6 | None | NAC | NALU | |---------|----------|----------|----------|--------| | a + b | 4.472 | 0.132 | 0.154 | 0.157 | | a - b | 85.727 | 2.224 | 2.403 | 34.610 | | a * b | 89.257 | 4.573 | 5.382 | 1.236 | | a / b | 97.070 | 60.594 | 5.730 | 3.042 | | a ^ 2 | 89.987 | 2.977 | 4.718 | 1.117 | | sqrt(a) | 5.939 | 40.243 | 7.263 | 1.119 |

Notes

• RMSprop works really well with NAC and NALU

• high learning rate (0.01) does a good job as well