Abstract Applications based on stereo vision are becoming increasingly common, ranging from gaming over robotics to driver assistance. While stereo algorithms have been investigated heavily both on the pixel and the application level, far less attention has been dedicated to the use of stereo confifidence cues. Mostly, a threshold is applied to the con- fifidence values for further processing, which is essentially a sparsifified disparity map. This is straightforward but it does not take full advantage of the available information. In this paper, we make full use of the stereo confifidence cues by propagating all confifidence values along with the measured disparities in a Bayesian manner. Before using this information, a mapping from confifidence values to disparity outlier probability rate is performed based on gathered disparity statistics from labeled video data. We present an extension of the so called Stixel World, a generic 3D intermediate representation that can serve as input for many of the applications mentioned above. This scheme is modifified to directly exploit stereo confifidence cues in the underlying sensor model during a maximum a posteriori estimation process. The effectiveness of this step is verifified in an in-depth evaluation on a large real-world traffific data base of which parts are made publicly available. We show that using stereo confifidence cues allows both reducing the number of false object detections by a factor of six while keeping the detection rate at a near constant level.