Abstract
Selectional Preference (SP) is a commonly observed language phenomenon and proved to
be useful in many natural language processing
tasks. To provide a better evaluation method
for SP models, we introduce SP-10K, a largescale evaluation set that provides human ratings for the plausibility of 10,000 SP pairs
over five SP relations, covering 2,500 most frequent verbs, nouns, and adjectives in American English. Three representative SP acquisition methods based on pseudo-disambiguation
are evaluated with SP-10K. To demonstrate
the importance of our dataset, we investigate the relationship between SP-10K and the
commonsense knowledge in ConceptNet5 and
show the potential of using SP to represent
the commonsense knowledge. We also use the
Winograd Schema Challenge to prove that the
proposed new SP relations are essential for the
hard pronoun coreference resolution problem.