Workaround for turbo cache symlink bug

[00:00 - 00:24] In this lesson, unfortunately, we need to work around a bug in Turborepo that's exhibited when used in combination with Preconstruct. To show you the bug, I'm going to go to packages/ui and I'm going to first do "pnpm preconstruct dev". Then let's look at our dist folder.

[00:25 - 00:45] So the only thing that we have in the dist folder is our button, because our preconstruct config defines entry points to be all index files. And in our UI package, the only index file that we have created is our src/button/index.ts file.

[00:46 - 01:11] The content of this file in dist is a symlink, so the file in dist points to the actual underlying source file. This way we get bundlers to be able to load our TypeScript source files without needing to build them, and this can be used by Next.js, Vite, and other bundlers to directly run TypeScript.

[01:12 - 01:36] In contrast, when we do a build with "pnpm preconstruct build" what's going to be created in our dist folder is a full render of the file. It's no longer a symlink—if we open "dist/monorepo-ui-button.js" this is now bundled JavaScript. It looks quite different from the original source code.

[01:37 - 01:51] Now I'm going to run "pnpm turbo build". What this is going to internally do is it's going to run "preconstruct build" and overwrite any files in the dist folder.

[01:52 - 02:01] So it will just create that JavaScript file again. If I open it, it's going to be just the same JavaScript file.

[02:02 - 02:32] However, if I first do pnpm dev which runs "preconstruct dev" and creates our symlink to the source code, and after that I run "pnpm turbo build", which is going to overwrite the dist folder with turbo's cache, it actually did the wrong thing. It didn't update this file.

[02:33 - 02:54] It updated the real target of the symlink, so we can actually see that our source file for the button got overwritten with the compiled version of the code. It's no longer the original code that we had with JSX. Instead, it's the compiled JSX.

[02:56 - 03:24] So the problem that turbo has is that if there is a symlink in the folder where it does a cache restore, it is going to follow that symlink and write in its location, rather than delete the symlink and replace its contents, which is what it should be doing. I'm going to check out my src/button/index.tsx file to return its original contents. And then let's think about how we can solve this.

[03:25 - 03:52] Basically the problem that we have is that when we have first run preconstruct in dev mode, so it created a symlink and then turbo does a restore on top of it, it's going to follow the symlink and overwrite the source file. One way to solve this would be to delete the symlink ourselves before we do the Turbo testore. To do that, we could use something like rimraf.

[03:53 - 04:26] rimraff is a popular package that replaces the rm utility, and the reason why you would use this instead of rm directly is because you might want your code to also be able to run on Windows, and rm does not exist on Windows. However, there is a better way to remove directories without depending on an external package. Node can take an -e argument, which means evaluate, and whatever we give in that argument, it's just going to run as JavaScript.

[04:27 - 04:34] So I can say "node -e console.log('hello')". And it's going to log hello.

[04:35 - 05:06] What we could also do is this—we can say "node -e" which means evaluate, we can require the node fs module and run removeSync on the "dist" folder with options of recursive: true and force: true. This basically is the exact same thing as running rm -rf on Unix systems, and if I execute this I can see that the dist folder has been removed.

[05:07 - 05:38] So we can create a clean script in our package.json, that's just going to be this node script, and that way we don't need to install an additional package. And we have something that uses node's Nolde's own better portable implementation of rm compared to that package. So with this in place, we can also open our root up turbo.json file.

[05:39 - 05:55] And we need to do two things. We need to create a "clean" task, and we're going to say that this task should not be cached. And we can also say that "build" depends on the "clean" task.

[05:56 - 06:15] Like this. So now every time we tell turbo to execute "build" it's first going to execute the "clean" task. The clean task would not be cached, so it would be executed every time. And only after it has executed the clean task, it's going to execute the actual build task.

[06:16 - 07:09] Now that we have clean in here in our package.json, I can go back run "pnpm install" one time because this automatically runs preconstruct dev. I could have run "pnpm dev" as well and look at my dist folder. It is correctly a symlink right now and if I then run "pnpm turbo build" it's going to first run the UI clean task which will delete the dist folder and only then execute the build. Now as we saw before, to reproduce the problem, we needed to first run pnpm dev again and then restore from turbo cache for the problem to show up. And if we do that now, we did restore the build from cache.

[07:10 - 07:24] This was a cache hit, but but our dist folder is now correct. It's not a symlink that overwrites the actual source code, but only the file in dist that got overwritten.

[07:25 - 07:43] It's an unfortunate thing that we need to write this workaround and we need to do it everywhere where we have preconstruct. To quickly find where we're using preconstruct, we can do "pnpm ls -r @preconstruct/cli".

[07:44 - 07:59] And this will tell us that we have preconstruct in the monorepo root, in the logger package, and in the UI package. I'm currently in the UI package, so that only means that I need to add my clean task in the logger package.

[08:00 - 08:22] Let me quickly do that. I'm going to go to my scripts and I'm going to add the "clean" script exit and just verify that "pnpm turbo build" in the logger package is going to correctly execute the "clean" script, and it does.

[08:23 - 08:34] So with this workaround, we now have a fix for the problem in turbo. Unfortunately, this problem has been present for more than a year.

[08:35 - 08:54] Maybe by the time that this course gets released it might be fixed, in which case I will update the lesson. But at least for now, if you're using anything that writes symlinks and then restore in the same folder from Turbo cache, you must manually clean the folder yourself before doing the restore.

[08:55 - 09:40] With that said, I'm going to go to the root do "git add ." "git diff --cached", and we have added a clean script to both our logger and our UI packages, and we have updated our root turbo.json to include a dependency on the clean script for build tasks, and to specify the clean script as something that is not cached and must be executed every time. I'm going to commit this as "Workaround to issue #8476", which is the ticket number in the turbo GitHub repository. And with that I will see you in the next lesson.