Incorporating Syntactic and Semantic Information in
Word Embeddings using Graph Convolutional Networks
Abstract
Word embeddings have been widely adopted
across several NLP applications. Most existing word embedding methods utilize sequential context of a word to learn its embedding.
While there have been some attempts at utilizing syntactic context of a word, such methods result in an explosion of the vocabulary
size. In this paper, we overcome this problem by proposing SynGCN, a flexible Graph
Convolution based method for learning word
embeddings. SynGCN utilizes the dependency
context of a word without increasing the vocabulary size. Word embeddings learned by
SynGCN outperform existing methods on various intrinsic and extrinsic tasks and provide
an advantage when used with ELMo. We also
propose SemGCN, an effective framework for
incorporating diverse semantic knowledge for
further enhancing learned word representations. We make the source code of both
models available to encourage reproducible research.