How pnpm organizes node_modules

[00:00 - 00:10] In this lesson, we're going to explore how pnpm works and how it differs from NPM. Specifically in the way that it organizes its node_modules folder.

[00:11 - 00:40] To do this, I'm going to create an example package, and I'm going to create a minimal package.json that just has a "name" and a "version" field. Then for our example I'm going to use a package called tiny-truncate. It's a smart truncate function that will make sure that the text outputted to the terminal fits within its current window width, and it will also correctly handle ANSI escape characters for colors, for example.

[00:41 - 00:59] I'm going to use a website called https://npmgraph.js.org to look at the dependency graph of tiny-truncate. As you can see, it only has one direct dependency called ansi-truncate and it has one indirect dependency.

[01:00 - 01:18] Let's install it using npm and let's create a small script as an example. We're going to print out the characters "abc_" repeated 100 times to the console directly, first to see the behavior without using the truncate function.

[01:19 - 01:40] And as you can see, it wraps to three lines. Now let's add the tiny-truncate function, and let's wrap our text in it and see what the difference is. As you can see, the text gets correctly limited to a single line. Now let's look at our node_modules folder.

[01:41 - 02:02] With npm, all three modules that are part of our dependency graph are available for us to require. It means that even though we never specified ansi-truncate as a dependency, we can in fact require it instead of tiny-truncate. It has a bit of a different signature.

[02:03 - 02:08] It needs a second argument, that's the maximum length of the string. So let's give it that.

[02:09 - 02:17] And when we run it it runs no problem. This is despite the fact that as you can see, we never referenced this dependency.

[02:18 - 02:35] Let's see how this differs from pnpm. I'm going to delete node_modules and add a "packageManager" field to my package.json that sets pnpm to version 9.6, and I'm going to run pnpm install.

[02:36 - 03:05] When we look at our node_modules, we can see a very different picture. Now the only package that's available in node_modules is the package that's defined in our dependencies. And indeed, if we try to run our script, it no longer runs. It fails with "Cannot find package ansi-truncate." As you probably noticed, pnpm created a symlink pointing to a folder called ".pnpm" in node_modules, so let's look at that hidden folder.

[03:06 - 03:20] This is what's called the pnpm Virtual Store. It's a folder layout that's built in such a way that it forces npm's resolution to constrain what packages can import.

[03:21 - 03:37] It creates a nested dependency graph, where each package can only import either itself or its direct dependencies. It has access to no other packages in the dependency graph.

[03:38 - 03:46] So let's look at how this works. This tiny-truncate folder is the folder that the node_modules symlink points to.

[03:47 - 04:31] When there is an import or a require inside tiny-truncate, the node module resolution algorithm will traverse folders up until it finds a folder named node_modules. So in this node_modules folder, the only two packages that are available for import are tiny-truncate so it can import itself and ansi-truncate - a symlink to here. Now the same way if we look at ansi-truncate one folder up we have a folder called node_modules, and that means that the only two packages that ansi-truncate can import are ansi-truncate itself and fast-string-truncated-width.

[04:32 - 04:54] And finally, when we go to fast-string-truncated-width, because it's node_modules folder has only itself, it cannot import any other package in the dependency graph. So through this folder structure pnpm enforces the same dependency graph that we saw in https://npmgraph.js.org.

[04:55 - 05:15] tiny-truncate only sees ansi-truncate and ansi-truncate only sees fast-string-truncated-width. This strictness of pnpm is actually highly desirable, and it means that you always control the packages that are available to your code.

[05:16 - 05:36] If a package is not defined in dependencies, it's not available to you. This differs from npm, where you can have phantom dependencies, where one of your dependencies might be pulling some package, and then you can require that package, but you don't have control over it.

[05:37 - 06:01] If your dependency changes the version that it uses or it drops it completely, then suddenly your code will stop working, even though you didn't make any changes related to that phantom dependency. Still, in the interest of education, let's see how we can get the same behavior out of pnpm that npm has.

[06:02 - 06:30] pnpm refers to the packages that are visible in node_modules as hoisted packages, as they are hoisted out of the pnpm virtual store through a symlink, we can force pnpm to match npm's behavior and make all packages visible by setting the .npmrc option "shamefully-hoist = true". When we look at our node_modules.

[06:31 - 06:44] Now, all packages in the dependency graph are fully available for us to use, the same as what npm was doing. So now obviously our script that uses ansi-truncate will again work.

[06:45 - 06:56] In practice, you should never do this. If you must work around a broken package that has phantom dependencies, then you can use an option called "public-hoist-pattern".

[06:57 - 07:10] We will discuss this option more in the following lesson. In the general case, "shamefully-hoist" should always be set to false, and it should always explicitly declare any dependency that you're using in your code.