Abstract In this paper we address the problem of semantic analysis of structured/unstructured crowded video scenes. Our proposed approach relies on tracklets for motion representation. Each extracted tracklet is abstracted as a directed line segment, and a novel tracklet similarity measure is formulated based on line geometry. For analysis, we apply non-parametric clustering on the extracted tracklets. Particularly, we adapt the Distance Dependent Chinese Restaurant Process (DD-CRP) to leverage the computed similarities between pairs of tracklets, which ensures the spatial coherence among tracklets in the same cluster. By analyzing the clustering results, we can identify semantic regions in the scene, particularly, the common pathways and their sources/sinks, without any prior information about the scene layout. Qualitative and quantitative experimental evaluation on multiple crowded scenes datasets, principally, the challenging New York Grand Central Station video, demonstrate the state of the art performance of our method.