Abstract
Images are formed by counting how many phorons travel-ing from a given set of directions hit an image sensor during a given time intenval.When photons are few and far in be-tween,the concept of 'image'breaks down and it is best to consider directly the flow of photons.Computer vision in this regime,which we call 'scotopic,is radically different from the classical image-based paradigm in that visual com- putations(classification,control,search)have to take place while the stream of photons is captured and decisions may be taken as soon as enough information is available.The scotopic regime is important for biomedical imaging,secu-rity,astronomy and many other fields.Here we develop a framework that allows a machine to ciassify objects with as few photons as possible,while maintaining the error rate below an acceptable threshold.A dynamic and asymptoti-cally optimal speed-accuracy tradeof is a key feature of this framework.We propose and study an algorithm to optimize the tradeoff of a convolutional network directly from low-light images and evaluate on simulated images from stan-dard datasets.Surprisingly,scotopic systems can achieve comparable classification performance as traditional vision systems while using less than 0.1% of the photons in a con-ventional image.In addition,we demonstrate that our algo-rithms work even when the illuminance of the environment is unknown and varying.Last,we outline a spiking neural netwvork coupled with photon-counting sensors as a power-efficient hardware realization of scotopic algorithms.