De-phantoming the npm ecosystem

Nub ships a default-on global virtual store with a built-in phantom detector. The store shares installed dependency trees across projects; the detector helps keep packages with undeclared dependencies working in that layout.

I've now run that detector across a corpus of 10,000 high-download npm packages and published the results as @nubjs/extensions, with package extensions for 791 packages. The source and scan results are on GitHub, and the database includes every rule in Yarn's existing catalog.

  import {
    packageExtensions,
} from '@yarnpkg/extensions';
} from '@nubjs/extensions';

The package uses the same export name and data format as Yarn's database.

The global virtual store

A shared download cache is not a global virtual store. Most package managers cache package files globally but still construct an installed tree inside each project, copying or hardlinking those files into place.

Nub can share the installed tree itself. A warm install links packages into the project instead of materializing every file again:

project-a/node_modules/example
              │
              ▼
 shared-store/example/<graph>/
              ▲
              │
project-b/node_modules/example

Two projects can share an entry when the package and its dependency context match. Matching the package version alone is not sufficient.

Node normally follows a package symlink to its real path before resolving the package's imports. Inside a machine-global store, that path no longer passes through the project's node_modules.

An undeclared import that resolved in a flat tree can fail in the global store:

// This package imports react
// but never declared it:
const React = require('react');

This undeclared import is a phantom dependency. It can resolve when another dependency installs React in an ancestor's node_modules, but fail when the importing package moves to the global store.

Bun enabled a global store for its isolated linker, then made it opt-in again after realpath and phantom-resolution failures. Aube has restored the default for Vite/VitePress and Nuxt/Parcel after excluding those projects.

Install-time detection

I built an on-the-fly dependency detector that reads the code being installed. It compares statically identifiable imports with the dependencies declared in that package's manifest.

Doing that during installation is only practical if the analysis is fast. Nub already includes Oxc, the JavaScript and TypeScript parser behind its transpiler. The detector uses that parser during package extraction:

• Analysis overlaps with the download phase rather than waiting for every package to arrive.
• Results are cached by package content and scanner version, so unchanged packages do not need another scan.
• The linker uses those results to move affected dependency subtrees back into the project.

That last step restores ordinary dependency lookup through a project-local hidden hoist tree. It makes packages already present in the installed graph reachable again; it does not guess and download every possible missing dependency. Importers move with the affected package so the application does not load both a shared and a project-local instance of it.

Unrelated packages remain in the shared store. The global-store post covers the layout and ejection mechanism in more detail.

The limits of a hardcoded list

For automatic manifest repairs, Yarn and pnpm rely on a curated database of package extensions. An extension adds information the published manifest omitted:

packageExtensions:
  "@nrwl/devkit@*":
    dependencies:
      tslib: "*"

This entry comes from the new database. The scanned package imports tslib without declaring it as a runtime dependency.

Yarn maintains the shared @yarnpkg/extensions database. Its extension source has received only a handful of updates since 2023. pnpm also carries a few additions of its own.

A hardcoded list cannot cover a new or obscure package nobody has added to it. Users can write their own extensions, but that requires someone to hit the failure, diagnose it, and work out the repair. I don't think that should be the user's job.

The install-time detector does not need the package's name to appear in a list. It examines the version being installed, including packages outside any popularity ranking.

A larger extensions database

The detector can also run outside an install. I used it to scan a 10,000-package corpus selected from the npm-high-impact download ranking, then turned the findings into a database other tools can consume.

The September 8 run successfully scanned 9,982 packages; 18 could not be scanned. The generated rules combine those findings with Yarn's existing catalog:

These counts include each package name once, even when the database has separate rules for different version ranges. They include type references and optional integrations, not just runtime failures.

The database combines generated findings with curation:

• Every Yarn rule is preserved as a fixed seed. The scan adds cases rather than replacing the existing catalog. Hand-written additions can be submitted by pull request and are included even when the detector finds nothing for that package.
• Most additions are optional peers. They cover type references, guarded imports, and framework adapters as well as runtime dependencies, making consumer-supplied packages visible without installing them unconditionally.
• Computed imports still need curation. A call such as require(name) may never name its target in the source. Yarn's hand-written rules cover cases the scanner cannot discover.

The repository includes the corpus, detector revision, scan failures, per-entry evidence, and ready-to-use configuration. The ranking is a pinned snapshot, and scanning a package version does not prove that every historical or future version behaves identically.

The corpus is rescanned on a daily schedule. A separate daily publication job releases an npm update when the extension rules change. Each dataset records its date so consumers can check whether the scheduled jobs have kept up.

The package documentation covers using the database with pnpm and Yarn, declaring optional integrations, and contributing corrections. The repository includes weekly npm download estimates alongside the rules and evidence.

Adoption

Nub already bundles a snapshot in its resolver. These precomputed rules repair manifests during resolution, alongside the install-time detector.

For tools importing Yarn's database, the replacement has the same packageExtensions export and Array<[selector, data]> shape, with CommonJS and ESM entry points and no dependencies:

const { packageExtensions } =
  require('@nubjs/extensions');

Installing the package alone does not replace a package manager's bundled database. Package-manager authors can integrate the export; project maintainers can use the supplied Yarn and pnpm configuration files.

I strongly recommend other package managers incorporate it so users receive these repairs without configuring extensions themselves.