Visual regression in Chromatic
Using Chromatic and Storybook for visual regression testing.
In the Testing module, we learned testing strategies that are incredibly useful for a range of scenarios. However, there is one thing that most of them lack: "eyes". Not literally, of course! Sometimes, you make small CSS changes that might not be picked up in tests and might affect the style/structure of multiple elements, which could be an unintended bug! This brings us to visual regression testing.
What's visual regression testing?#
In an attempt to prevent visual bugs, visual regression testing brings an approach to give confidence when making code changes by comparing screenshots of components. It works by taking an initial set of screenshots (called snapshots) and calling them baseline. Once changes are made, new snapshots are taken based on these changes, and then the baseline and the new changes are compared by image processing tools. As a result, if there are visual changes, they are flagged as visual regression. Here's an example of regression from a button that changed its border-radius and affected a page that used it:
Of course, the visual change might be intentional, but more often than not, you might see non-intentional changes happening in your project. Usually, these changes are only noticed in production, which is too late.
The challenges of setting up visual regression testing#
In the Storybook test runner lesson we setup automated tests for our stories. We also learned that there's a custom hook that would allow us to set up visual regression testing! But should we do that? There are quite a few infrastructural challenges when it comes to setting these up:
We need to store the snapshots in the repo. That can become cumbersome
Components with dynamic content and animations are difficult to handle
Components that do network request are tricky to handle
We'd need to make sure every test runs against the same browser and version regardless of environment
Thankfully, Chromatic offers visual regression testing as a service and they handle all of those things for us, so let's just use it instead!
Identifying our first regression with Chromatic#
Following up with the previous lesson, let's see Chromatic in action by forcing some visual changes. Go to components/Badge.tsx and change the size property of the component from S to XS:
// src/components/Badge/Badge.tsx// ... rest of the codeexport const Badge = ({ text, className }: BadgeProps) => (
<Container className={className}>
<Body type="span" size="XS"> {/* <----- change this from S to XS */}
{text}</Body>
</Container>
)
Now let's create a new branch for this change:
git checkout -b chromatic-test
And commit and push the changes:
git add .
git commit -m "change font size in badge to xs"
git push
Now create a pull request from chromatic-test to main:
Once created, you will see that the Chromatic GitHub Action will run and start deploying the Storybook for this PR:
After the action runs (it might take a few minutes), you will see that the checks changed in GitHub, and you now have links to the published Storybook, the UI tests and the UI review:
Chromatic UI Tests#
Now, see the results of the UI tests by clicking on Details which will open "Build 2", a new Chromatic build for our latest changes. In there we see all of the changes that were caught! You notice how a simple change in a basic component as the Badge actually ended up affecting many other components!
This page is a preview of Storybook for React Apps