Analysis and Methods for Knowledge Graph Embeddings

Abstract

Knowledge Graphs (KGs) are multi-relational graphs where nodes represent entities, and typed edges represent relationships among entities. These graphs store real-world facts such as (Lionel Messi, plays-for-team, Barcelona) as edges, called triples. KGs such as NELL, YAGO, Freebase, and WikiData have been very popular and support many applications such as Web Search, Query Recommendation, and Question Answering. Although popular, these KGs suffer from incompleteness. Learning Knowledge Graph Embeddings (KGE) is a common approach for predicting missing edges (i.e., link prediction) and representing entities and relations in downstream tasks. While numerous KGE methods have been proposed in the past decade, our understanding and analysis of such embeddings have been limited. Further, such methods only work well with ontological KGs. In this thesis, we address these gaps.

Firstly, we study various KGE methods and present an extensive analysis of these methods, resulting in many insights. Next, we address an under-explored problem of link prediction in Open Knowledge Graphs (OpenKGs) and present a novel approach that improves the type compatibility of predicted edges. Lastly, we present an adaptive interaction framework for learning KG embeddings that generalizes many existing methods.

In the first part, we present a macro and a micro analysis of embeddings learned by various KGE methods. Despite the popularity and effectiveness of KG embeddings, their geometric understanding (i.e., arrangement of entity and relation vectors in vector space) is unexplored. We initiate a study to analyze the geometry of KG embeddings and correlate it with task performance and other hyper-parameters. Firstly, we present a set of metrics (e.g., Conicity, ATM) to analyze the geometry of a group of vectors. Using these metrics, we find sharp differences between the geometry of embeddings learned by different classes of KGE methods. The vectors learned by a multiplicative model lie in a narrow cone, unlike additive models where the vectors are spread out in the space. This behavior of multiplicative models is amplified by increasing the number of negative samples used for training. Further, a very high Conicity value is negatively correlated with the performance on the link prediction task. We also study the problem of understanding KG embeddings’ semantics and propose an approach to learn more coherent dimensions. A dimension is coherent if the top entities have similar types (e.g., person). In this work, we formalize the notion of coherence using entity co-occurrence statistics and propose a regularizer term that maximizes coherence while learning KG embeddings. The proposed approach significantly improves coherence while having a comparable performance with baseline in the link prediction and triple classification tasks. Further, based on the human evaluation, we demonstrate that the proposed approach learns more coherent dimensions than the baseline.

In the second part, we address the problem of learning KG embeddings for Open Knowledge Graphs (OpenKGs), focusing on improving link prediction. An OpenKG refers to a set of (head noun phrase, relation phrase, tail noun phrase) triples such as (tesla, return to, new york) extracted from a text corpus using OpenIE tools. While OpenKGs are easy to bootstrap for a domain, they are very sparse. Therefore, link prediction becomes an important step while using these graphs in downstream tasks. Learning OpenKG embeddings is one approach for link prediction that has received some attention lately. However, on careful examination, we find that current algorithms often predict noun phrases (NPs) with incompatible types for given noun and relation phrases. We address this problem and propose OKGIT that improves OpenKG link prediction using novel type compatibility score and type regularization. With extensive experiments on multiple datasets, we show that the proposed method achieves state-of-the-art performance while producing type compatible NPs in the link prediction task.

In the third part, we address the problem of improving the KGE models. Firstly, we show that the performance of existing approaches vary across different datasets, and a simple neural network-based method can consistently achieve better performance on these datasets. Upon analysis, we find that KGE models depend on fixed sets of interactions among the dimensions of entity and relation vectors. Therefore, we investigate ways to learn such interactions automatically during training. We propose an adaptive interaction framework for learning KG embeddings, which can learn appropriate interactions while training. We show that some of the existing models could be seen as special cases of the proposed framework. Based on this framework, we also present two new models, which outperform the baseline models on the link prediction task. Further analysis demonstrates that the proposed approach can adapt to different datasets by learning appropriate interactions.