PhD thesis: Enhancing Semantic Parsing in the Age of Pre-trained Language Models, chapter by chapter

PhD thesis · Abelardo Carlos Martínez Lorenzo
Sapienza University of Rome · Sapienza NLP · XXXVI cycle, 2024
Advisor: Prof. Roberto Navigli · Marie Skłodowska-Curie fellow (KnowGraphs)

Semantic parsing turns a sentence into a graph of its meaning: who did what to whom. This thesis improves both sides of that task: the formalism the graph is written in and the parsers that write it.

Use ← → or swipe. Each chapter links to the interactive story of its paper.

Pre-trained language models made parsers far better, but two things held semantic parsing back: formalisms that only really work for English, and parsers that are expensive, blind to structure, and hungry for data that is locked behind licences.

The thesis is organised around seven research questions. Click one to jump to its chapter.

Abstract Meaning Representation became the most used formalism because it abstracts away from syntax and ships with a large annotated corpus. But its predicates come from the English PropBank and its other nodes are English words.

Adapting AMR to another language produces a different, unconnected graph: Spanish frames, Spanish lemmas, and no way to reuse the English corpus. That is not an interlingua.

The BabelNet Meaning Representation replaces words with BabelNet synsets, multilingual concepts covering 500 languages, and English frames with VerbAtlas frames and semantic roles.

The result is one graph for a sentence and all its translations. Language only returns when the graph is read out loud.

Published as · AAAI 2022

BMR · The Formalism

Full story

A formalism without data cannot train a model. We converted all 59,255 AMR 3.0 graphs into BMR with graph rules, word sense disambiguation and linguistic mappings, then trained the first BMR parsers and generators in English, German, Spanish and Italian.

Text → BMR → text preserves more of the original sentence than AMR does, and works across languages.

Published as · ACL 2022

BMR · Parsing & Generation

Full story

AMR parsing went from graph-based systems to transition systems, and then to sequence-to-sequence models such as SPRING, which read the sentence and write a linearised graph, token by token.

SPRING-style parsers set the state of the art, but they left open problems. Part II takes them one by one.

A seq2seq parser pays for every token it writes. A compact linearisation drops variables, slashes, quotes and redundant brackets, cutting sequences by about 40–50%.

On top of it, CLAP is a modular parser that works with T5 or BART and trains 82% faster than SPRING at the same accuracy. It became the base framework for the next chapters.

Published as · LREC-COLING 2024

CLAP

Full story

Language models read text, not graphs. We align the AMR graph to the sentence's words and feed it to small GNN adapters inside the encoder. With the graph leaked, the parser reaches 89.6 SMATCH.

At test time there is no graph. So the same model runs twice during training, with and without it, and self-distillation makes the text-only pass imitate the other: 85.7 SMATCH on AMR 2.0, state of the art at the time.

Published as · Findings of ACL 2023

LeakDistill

Full story

Seq2seq parsers write the graph but don't say which words each node came from, so separate, English-only aligners were needed afterwards.

While writing each token, the decoder's cross-attention looks at the input. We showed that this attention is an alignment: extracted while parsing, with no rules, it beats previous aligners and works in any language the parser reads. A light supervision of a few heads makes it sharper.

Published as · Findings of ACL 2023

Cross-lingual AMR Aligner

Full story

Merging several parsers' graphs raises SMATCH, but SMATCH only counts triples. Graph-mining mergers like Graphene produce graphs that break AMR's structural rules.

We wrote a validator for those rules and two ensemblers built on language models: one that merges and one that selects the most likely graph. Same or better SMATCH, a fraction of the corrupted graphs, and faster.

Published as · ACL 2023

AMRs Assemble!

Full story

A formalism that works across languages, its first dataset and parsers, and a parsing framework that is efficient, structure-aware, aligned and reliable, with open data to train it.

Limits: BabelNet does not cover every language or concept; encoder–decoders stay costly and propagate errors; silver data can carry biases.