Abstract Syntax Trees - The Magic Behind Compilers

A React application written with JSX. A style guide written with SASS. E-mail templates written with Pug. Such projects involve a compilation step that takes source code written in a language that a browser cannot understand and turns it into HTML/CSS/JavaScript code that a browser can parse and execute.

Anytime you tell the compiler to build a React application, you expect the compiler to process and transform the JSX source code into pure JavaScript code with React function calls. Often, we treat the compiler as a black box, and very rarely do we look underneath its hood to see how exactly it performs this transformation. The magic behind the compiler lies in the data structure it uses to convey the structural patterns of the source code: 

By analyzing the syntax of the source code and breaking it down into its constituent tokens (i.e. keywords, literals, operations, etc.), we can represent the code as a tree data structure. Being able to generalize the constructs and rules of the source code with an abstract syntax tree gives the compiler a high-level blueprint to follow when transforming the code. It's up to the compiler to traverse the abstract syntax tree (via the depth-first search traversal algorithm), extract information from it and output code of equivalent functionality, but written in a different language and/or optimized for performance. In some respects, you may say abstract syntax trees for compilers serve a similar role as pseudocode for algorithms.

Abstract syntax trees are not limited to compilers. In fact, they can be applied in a variety of use cases, such as:

Turning JSX Code into an Abstract Syntax Tree

JSX in React describes a component's UI by using syntax that resembles HTML markup. Like other JavaScript-based templating languages (e.g., Handlerbars and Pug), JSX requires a compiler to compile down to JavaScript for the code to run on browsers. 

For example, a browser cannot recognize code that looks like this:

<ul>
  {props.items.map(({ id, name }) => (
    <li key={id}>{name}</li>
  ))}
</ul>

Browsers only natively support the syntax of the JavaScript language (i.e. official features of the ECMAScript specification). Therefore, markup syntax within JavaScript code is not valid and will result in a runtime error. To avoid this problem, we must compile the above code down to pure JavaScript code that the browser understands:

React.createElement("ul", {}, props.items.map(({ id, name }) => (
  React.createElement("li", { key: id }, name)
)));

 method. For larger components with lots of nested elements, JSX provides greater readability and clarity. For the compiler to convert JSX to JavaScript, the original source code must be...

If we want to preview the abstract syntax tree generated by different parsers, then we can type the above JSX code into the 

Most JavaScript-based abstract syntax trees follow the 

, which defines properties categorized grammars. Below is an abbreviated version of the abstract syntax tree (in JSON format) generated by a parser for the JSX code.

Check out the complete abstract syntax tree 

Below is a visualization of this abstract syntax tree:

: The leaf nodes represent actual identifiers, keywords and literals from the code itself. The remaining parent nodes represent the token types discovered by the parser.

Here's a diagram that illustrates how the compiler works:

The parser responsible for outputting the above abstract syntax tree for the JSX code is the 

. Babel uses an abstract syntax tree based on the ESTree specification. React applications written with next-generation JavaScript and JSX rely on the Babel compiler to transform code into JavaScript that is compatible with the browsers you choose to target. 

Transforming JSX Code with the Babel Plugin 

With plugins, Babel applies specific code transformations to the source code. For example, if you want to use the arrow function syntax, but need your application to run in Internet Explorer, then enable the Babel plugin 

 in the configuration file. Whenever Babel encounters any instances of the arrow function syntax, it will transform them with this plugin.

 parses and transforms JSX code to React function calls. 

To understand how the plugin works, we must first see how Babel parses JSX code. Babel is not just one single package. Rather, its core functionality is divided up into several different packages:

 method that reads source code (as a string) and generates an abstract syntax tree from it. 

Here's a simple program for printing the abstract syntax tree (in JSON format) of a snippet of JSX code: 

const { parse } = require("@babel/parser");

const code = `
  <ul>
    {props.items.map(({ id, name }) => (
      <li key={id}>{name}</li>
    ))}
  </ul>
`;

const ast = parse(code, {
  plugins: ["jsx"]
});

console.log(JSON.stringify(ast, null, 2));

 tells the Babel parser which plugins ("jsx," "flow" and/or "typescript") to enable. The Babel parser supports JSX, Flow and TypeScript out of the box.

 parses the JSX code in this exact way. It delegates the responsibility of parsing the JSX code to the Babel plugin 

import { declare } from "@babel/helper-plugin-utils";

export default declare(api => {
  api.assertVersion(7);

  return {
    name: "syntax-jsx",

    manipulateOptions(opts, parserOpts) {
      if (
        parserOpts.plugins.some(
          p => (Array.isArray(p) ? p[0] : p) === "typescript",
        )
      ) {
        return;
      }

      parserOpts.plugins.push("jsx");
    },
  };
});

All this plugin does is modify the parser's options. The 

 helper method maintains backwards compatibility with Babel v6. The plugin checks whether or not the TypeScript plugin (

) has already ran. If so, then the plugin does nothing since the code (presumably written in TSX, the TypeScript variant of JSX) will have already been parsed and transformed by the TypeScript plugin. If there is no TypeScript plugin, then the plugin simply adds the "jsx" plugin to the parser's 

 preset, which is the most commonly used Babel preset for compiling React applications.

 is a collection of plugins. Instead of manually listing the individual plugins you want to enable for the Babel compiler, you just need to list one preset that contains these plugins.

 uses this preset for compiling your React application.

To transform JSX Code with the Babel plugin 

$ npm install --save-dev @babel/core @babel/cli

$ npm install --save-dev @babel/plugin-transform-react-jsx # Or @babel/preset-react.

{
  // Uncomment the below if you use the preset.
  // "presets": [
  //   "@babel/preset-react"
  // ],
  "plugins": [
    "@babel/plugin-transform-react-jsx"
  ]
}

{
  "scripts": {
    "build": "babel <file_with_jsx_code.jsx> > build.js"
  }
}

When you run this npm script for the following JSX code:

 file will contain only React function calls. Since the output contains only valid JavaScript, the code can be run within any browser (as long as the browser already has the React library loaded).

/*#__PURE__*/
React.createElement("ul", null, props.items.map(({
  id,
  name
}) => /*#__PURE__*/React.createElement("li", {
  key: id
}, name)));

As the JavaScript language continues to evolve and new specifications and proposals are introduced, Babel comes out with new plugins to add support these features even if they are not yet present in the target browsers.

Think about ways you can use abstract syntax trees to automate tasks like code auditing/linting. 

 within your project's today regardless of browser support.

The newline Guide to Practical Abstract Syntax Trees

, for the practical techniques you need to maintain any size codebase.

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

Abstract Syntax Trees - The Magic Behind Compilers

Responses (0)

Masterclasses

Tutorials

Fullstack React with TypeScript