Explainable and Efficient Neural Models for Natural Language to Bash Command Translation

Abstract

One of the key goals of Natural Language Processing is to make computers understand natural language. Semantic Parsing has been one of the driving tasks for Natural Language Understanding. It is formally defined as the task of generating meaning representation from natural language input. In this work, we focus on using the Bash command as the meaning representation. Bash is a Unix command language used for interacting with the Operating System. Recent works on natural language to Bash command translation have made significant advances on this problem. The best performing solutions employ a neural network architecture called the Transformer. In this work, we explore the aspects of explainability and efficiency for this task and use the Transformer as one of the baselines for comparing the proposed approaches.

In the first part, we utilize documentation data from Linux manual pages and the Abstract Syntax Tree for Bash to generate explanations for the translated Bash command. We propose a novel architecture that incorporates tree structure information in the Transformer and provides explanations for its predictions via alignment matrices between user invocation and manual page text. We find that the proposed method performs on par with the Transformer performance. Our method performs better than fine-tuned T5, a Transformer-based neural model pre-trained on a large amount of text data in a self-supervised manner.

In the second part, we use the problems inherent synchronous structure and propose the Segmented Invocation Transformer (SIT) that utilizes the information from the constituency parse tree of the natural language invocation. Our method is motivated by the alignment between segments in the natural language text and Bash command components. By utilizing this structure, the proposed method outperforms the state-of-the-art approach while achieving a 1.8x improvement in the inference time (as measured on a CPU) and a 5x reduction in model parameters. We also conduct an attribution analysis using Integrated Gradients to empirically confirm the identified structure and the ability of SIT to capture it.