Unsupervised Acquisition of Comprehensive Multiword Lexicons using Competition in an n-gram Lattice
Published
2017-12-11
Julian Brooke
,
Jan Snajder
,
Timothy Baldwin
Julian Brooke
The University of Melbourne
Jan Snajder
The University of Zagreb
Timothy Baldwin
The University of Melbourne
Abstract
We present a new model for acquiring comprehensive multiword lexicons from large corpora based on competition among n-gram candidates. In contrast to the standard approach of simple ranking by association measure, in our model n-grams are arranged in a lattice structure based on subsumption and overlap relationships, with nodes inhibiting other nodes in their vicinity when they are selected as a lexical item. We show how the configuration of such a lattice can be optimized tractably, and demonstrate using annotations of sampled n-grams that our method consistently outperforms alternatives by at least 0.05 F-score across several corpora and languages.
PDF (presented at NAACL 2018)