mctorch
McTorch is a Python library that adds manifold optimization functionality to PyTorch.
McTorch:
Leverages tensor computation and GPU acceleration from PyTorch.
Enables optimization on manifold constrained tensors to address nonlinear optimization problems.
Facilitates constrained weight tensors in deep learning layers.
Sections:
McTorch builds on top of PyTorch and supports all PyTorch functions in addition to Manifold optimization. This is done to ensure researchers and developers using PyTorch can easily experiment with McTorch functions. McTorch's manifold implementations and optimization methods are derived from the Matlab toolbox Manopt and the Python toolbox Pymanopt.
Initialize Parameter - McTorch manifold parameters are same as PyTorch parameters (torch.nn.Parameter
) and requires just addition of one property to parameter initialization to constrain the parameter values.
Define Cost - Cost function can be any PyTorch function using the above parameter mixed with non constrained parameters.
Optimize - Any optimizer from torch.optim
can be used to optimize the cost function using same functionality as any PyTorch code.
PCA Example
import torchimport torch.nn as nn# Random data with high variance in first two dimensionX = torch.diag(torch.FloatTensor([3,2,1])).matmul(torch.randn(3,200))# 1. Initialize Parametermanifold_param = nn.Parameter(manifold=nn.Stiefel(3,2))# 2. Define Cost - squared reconstruction errordef cost(X, w): wTX = torch.matmul(w.transpose(1,0), X) wwTX = torch.matmul(w, wTX) return torch.sum((X - wwTX)**2)# 3. Optimizeoptimizer = torch.optim.Adagrad(params = [manifold_param], lr=1e-2)for epoch in range(30): cost_step = cost(X, manifold_param) print(cost_step) cost_step.backward() optimizer.step() optimizer.zero_grad()
Multi Layer Perceptron Example
import torchimport torch.nn as nnimport torch.nn.functional as F# a torch module using constrained linear layersclass ManifoldMLP(nn.Module): def __init__(self): super(ManifoldMLP, self).__init__() self.layer1 = nn.Linear(in_features=28*28, out_features=100, weight_manifold=nn.Stiefel) self.layer2 = nn.Linear(in_features=100, out_features=100, weight_manifold=nn.PositiveDefinite) self.output = nn.Linear(in_features=100, out_features=10, weight_manifold=nn.Stiefel) def forward(self, x): x = F.relu(self.layer1(x)) x = F.relu(self.layer2(x)) x = F.log_softmax(self.output(x), dim=0) return x# create module object and compute cost by applying module on inputsmlp_module = ManifoldMLP() cost = mlp_module(inputs)
This would be an ever increasing list of features. McTorch currently supports:
Stiefel
Positive Definite
All manifolds support k multiplier as well.
SGD
Adagrad
ConjugateGradient
Linear
Conv1d, Conv2d, Conv3d
Conv1d_transpose, Conv2d_transpose, Conv3d_transpose
This is same as PyTorch installation from source. I have tried the installation with Python 3.7 and it will be preferred version for installation.
If you are installing from source, we highly recommend installing an Anaconda environment.
Download anaconda installer from here - Anaconda and follow GUI steps to install
Create an environment, from anaconda console prompt: conda create -n myenv python=3.7
Activate the environment: source activate myenv
Install current release branch i.e. mctorch_1_1
source activate myenv conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing pip install git+git://github.com/mctorch/mctorch.git@mctorch_1_1
source activate myenv conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ pip install git+git://github.com/mctorch/mctorch.git@mctorch_1_1
ModuleNotFoundError: No module named 'torch._C'
: After installation when using McTorch make sure you have activate the conda environment and not in the same folder as McTorch as import torch
tries to refer to the module in torch folder and throws an error.
For other os and optional dependencies go through Installation.
McTorch is currently under development and any contributions, suggestions and feature requests are welcome. We'd closely follow PyTorch stable versions to keep the base updated and will have our own versions for other additions.
McTorch is released under the open source 3-clause BSD License.
Please cite [1] if you found this code useful.
[1] M. Meghawanshi, P. Jawanpuria, A. Kunchukuttan, H. Kasai, and B. Mishra, McTorch, a manifold optimization library for deep learning
@techreport{meghwanshi2018mctorch, title={McTorch, a manifold optimization library for deep learning}, author={Meghwanshi, Mayank and Jawanpuria, Pratik and Kunchukuttan, Anoop and Kasai, Hiroyuki and Mishra, Bamdev}, institution={arXiv preprint arXiv:1810.01811}, year={2018} }
上一篇:osqpth
还没有评论,说两句吧!
热门资源
Keras-ResNeXt
Keras ResNeXt Implementation of ResNeXt models...
seetafaceJNI
项目介绍 基于中科院seetaface2进行封装的JAVA...
spark-corenlp
This package wraps Stanford CoreNLP annotators ...
capsnet-with-caps...
CapsNet with capsule-wise convolution Project ...
inferno-boilerplate
This is a very basic boilerplate example for pe...
智能在线
400-630-6780
聆听.建议反馈
E-mail: support@tusaishared.com