Tutorials on Abstract Syntax Trees

Maintaining and refactoring a large codebase requires lots of development time and effort. Issues, such as inconsistencies in variable naming or passing incorrect arguments in function calls, tend to happen more frequently in larger codebases. As you scan your codebase for these issues, you may observe some of them emerging repeatedly in different parts of your codebase. Manually sifting through tens and hundreds of files to apply the same set of changes in multiple places is simply not feasible. Anytime you have to manually make many repetitive changes, the likelihood of a mistake occurring increases. To automate this entire process, you need tooling that draws upon abstract syntax trees to audit code, report the findings and, if warranted, immediately resolve them. Abstract syntax trees are not only used for compilers; they are also present in other types of development tooling. By parsing source code into an abstract syntax tree, we can traverse the nodes of the tree to interact directly with each and every literal, identifier, etc. in the source code. For React applications, auditing with abstract syntax trees lets us understand the current state of our components' source code and get information about our components' contents. For example, if we notice <button /> elements scattered throughout our application, then we can write a tool to list the CSS classes assigned to their className attributes. Using this list, we can check if there are any incorrectly spelled CSS classes and/or invalid CSS classes and fix them. Additionally, we can refactor the <button /> elements into a <Button /> component and consolidate the CSS classes within this single component. Therefore, if we decide to rename/remove any of the CSS classes, then we only need to visit and edit one component instead of many. Writing development tooling based on abstract syntax trees takes little time thanks to Babel's large ecosystem of packages and plugins. Babel comes with packages for parsing source code ( @babel/parser ) and traversing abstract syntax trees ( @babel/traverse ). Below, I'm going to show you how to write an auditing tool for React applications using Babel. To get started, initialize a new Create React App project: For this tutorial, we will be writing an auditing tool that generates a report of the <button /> elements used within the React application. The report will list the attributes (i.e. type , className and onClick ) set on these elements and the values assigned to them. If the report shows many many <button /> elements sharing the same CSS classes, then we can refactor them into a single <Button /> component that renders a <button /> element with those CSS classes. Within the new project, create three components that contain <button /> elements with type , className and onClick attributes: ( src/components/Modal.tsx ) ( src/components/LoginForm.tsx ) ( src/components/Card.tsx ) While the components are not actively used within the application, you can still audit them since the auditing tool statically analyzes the components' source code. It does not run the React application. The auditing tool uses two Babel packages: Let's install these two packages: To find all the relevant component files, the auditing tool uses a glob pattern to search for files by their name. Let's install the glob package: Since this tool will be written with TypeScript, we will need to install ts-node to run it and type definition files for the glob and Babel packages. Note : @babel/parser provides its own type definitions. Therefore, you don't need to install any extra dependency for its type definitions. At the root of the project directory, create a new directory named audits . Within this directory, create a buttons-audit.ts file. Within this file, start by synchronously glob searching for the component files and reading their content. ( audits/buttons-audit.ts ) Add an npm script to package.json to execute this script: ( package.json ) Note : The module compiler option suppresses the warning message Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension. . For larger projects, you should define compiler options within a tsconfig.json file. Run this npm script to verify that ts-node properly runs the audits/buttons-audit.ts file. Now, parse the source code of each file into an abstract syntax tree. ( audits/buttons-audit.ts ) Call the traverse function from the @babel/traverse package to traverse the abstract syntax tree. Pass it the abstract syntax tree generated by the parser and an object with the nodes to execute code when visited. In this case, we would like to execute code whenever Babel finds a <button /> JSX element. Since an element's attributes are assigned to its opening, not closing, tag, we should execute code on JSXOpeningElement nodes. A JSXOpeningElement node represents any opening JSX element (i.e. opening <a /> element tags), so how do we narrow down to opening <button /> element tags? Anytime Babel encounters a node of type JSXIdentifier with a name of button , this indicates that we have reached a node that represents an opening <button /> element tag. This is where we want to execute code to retrieve the <button /> element's attributes and corresponding values. ( audits/buttons-audit.ts ) To check whether an opening <button /> tag has the className attribute, we must iterate the node's attributes for a className attribute: ( audits/buttons-audit.ts ) If the <button /> element does not have a className attribute, then we should skip it. ( audits/buttons-audit.ts ) Next, record the button's attributes and their corresponding values to an object named propsAndValues , which we declare in the outer scope of the script: ( audits/buttons-audit.ts ) Once the audit finishes, we can log our findings. ( audits/buttons-audit.ts ) Altogether... ( audits/buttons-audit.ts ) If you re-run the audit, then you will see that there are four buttons with the CSS class btn . Since these buttons either have a btn--primary or btn--secondary CSS class, we may consider refactoring these buttons into a <Button /> component that centralizes these CSS classes in one file. If we ever decide to rename these CSS classes during a redesign, then we would only need to make the changes within this file only. For larger codebases, this approach allows you to quickly audit your code, saving you lots of development time that can be allocated towards other important tasks. Best of all, you can export this report into a separate file to share these results with other team members. Try auditing your own React applications with abstract syntax trees. You can also check out our new course,  The newline Guide to Practical Abstract Syntax Trees , for more practical techniques you need to maintain any size codebase. 

You may have seen the term "abstract syntax trees", "AST", or maybe even learned about them in a computer science curriculum, but chalked them up as being irrelevant to the work you need to do as a frontend engineer. On the contrary, abstract syntax trees are ubiquitous in the frontend ecosystem. Understanding abstract syntax trees isn't a requirement for being a productive or successful frontend engineer, but it can unlock a new skill set that has many practical applications in frontend development. First, what are abstract syntax trees? In the simplest form, an abstract syntax tree is a way to represent code so a computer can understand it. The code we write is a giant string of characters that will only do something if a computer can understand and interpret the code. An abstract syntax tree is a  tree data structure . A tree data structure starts with a root value. The root can then point to other values, and those values to others, and so on. This begins to create an implicit hierarchy, and also happens to be a great way to represent source code in a way computers can easily interpret. For example, say we had the code snippet  2 + (4 * 10) . To evaluate this code, multiplication is performed first, followed by addition. Since the addition is the last thing to happen or the highest in this hierarchy it will be the root. It then points to two other values, on the left is the number 2 , but on the right is another equation. This time it's multiplication, and it also has two values, the number  4  on the left and 10  on the right. A common example of using abstract syntax trees is in compilers. A compiler accepts source code as input, and then outputs another language. This is often from a high-level programming language to something low-level, like machine code. A common example in frontend development is transpilation, where modern JavaScript is transpiled to an older version of JavaScript syntax. But how does this impact you as a frontend engineer? First, you likely rely on tooling built on top of abstract syntax trees on a daily basis. Some common examples of frontend build tools or compilers that rely on abstract syntax trees are webpack ,  babel , and swc. However, they aren't isolated to build tools. Tools like Prettier (code formatter) , ESLint (code linter) , or jscodeshift (code transformer) have different purposes but they all rely on abstract syntax trees since they all need to understand and work with source code directly. It's possible to use most of these tools without an understanding of abstract syntax trees, but some have an expectation that you understand ASTs for the more advanced uses. For example, to create a custom linting rule with ESLint, or a custom code transform with jscodeshift requires traversing and mutating ASTs. Trying to use these tools without this understanding can be a challenging and confusing experience. One of the biggest benefits is using tools like babel or swc directly to generate, traverse, and mutate ASTs. This enables interacting with code in a reliable and automated way and many of the tools mentioned earlier use these or equivalent tools internally. This allows creating completely custom functionality to statically analyze/audit code, make dynamic code transformations, or whatever problem you might be solving in a large codebase. While it's not necessary to understand abstract syntax trees to be a productive and successful frontend engineer, having a basic understanding can uplevel your ability to maintain continuously evolving large codebases and more easily interact with common tools that rely on them.  Abstract syntax trees enable interacting with a codebase "at scale." For more on how to make sweeping changes in a safe and reliable way in any size codebase, check out our course  The newline Guide to Practical Abstract Syntax Trees.

Learn the practical techniques you need today to modify any large-scale codebase without the hassle of manual, one line at a time refactors.  We use real world tools such as Facebook's jscodeshift to apply these powerful concepts on actual codebases. The course goes beyond just theory to practical hands on coding , with a sample codebase we provide that you will modify as you apply the techniques you learn in the course. With  Practical Abstract Syntax Trees  you  unlock the ability to make sweeping changes in a safe and reliable way in any size codebase . We'll tackle:  It's taught by Spencer Miskoviak, who's an engineer at WealthFront, the leading automated investing services firm with over $20 billion in assets under management (AUM). Spencer is a recognized expert on ASTs and JavaScript. He presented on ASTs at React Conf in 2019, showing advanced ways to optimize a JavaScript codebase. Access The newline Guide to Practical Abstract Syntax Trees for all of the practical techniques you need to maintain any size codebase.