Do you want >72% top-1 accuracy on a large video dataset? Are you tired of Kinetics videos disappearing from YouTube every day? Do you have recurring nightmares about Caffe2? Then this is the repo for you!
This is a PyTorch implementation of the Caffe2 I3D ResNet Nonlocal model from the video-nonlocal-net repo. The weights are directly ported from the caffe2 model (See checkpoints). This should be a good starting point to extract features, finetune on another dataset etc. without the hassle of dealing with Caffe2, and with all the benefits of a very carefully trained Kinetics model.
It's only a matter of time before FAIR releases a good PyTorch version of their nonlocal-net codebase, but until then, at least you have this ¯_(ツ)_/¯
Amazing features: - Only a single model (ResNet50-I3D). Parameters hardcoded with love. - Only the evaluation script for Kinetics (training from scratch or ftuning has not been tested yet.) - No nonlocal versions yet. One exciting NL version to choose from.
Kinetics Evaluation
The code has been tested with Python 3.7 + PyTorch 1.1.
Pretrained Weights Download pretrained weights for I3D and I3D-NL models from the nonlocal repo
from models import resnet
net = resnet.i3_res50() # vanilla I3D ResNet50net = resnet.i3_res50_nl() # Nonlocal version
Data Download videos using the official crawler and extract frames. This repo has a script to do this. Then create softlinks for frames and annotations:
Evaluate Run the evaluation script to generate scores on the validation set.
# Evaluate using 3 random spatial crops per frame + 10 uniformly sampled clips per video# Model = I3D ResNet50 Nonlocalpython eval.py --batch_size 8 --mode video --model r50_nl# Evaluate using a single, center crop and a single, centered clip of 32 frames # Model = I3D ResNet50python eval.py --batch_size 8 --mode clip --model r50# Use --parallel for multiple GPUspython eval.py --batch_size 16 --mode clip --model r50_nl --parallel
Model
clip (top1/top5)
video (top1/top5)
I3D Res50
0.647 / 0.853
0.721 / 0.902
I3D Res50 NL
0.664 / 0.864
0.737 / 0.912
You should get around 72.1% top-1 accuracy for the video using I3D Res50, and around 73.7% using the non-local version. Note that these numbers are on whatever is left of the Kinetics val set these days (~18434 videos).