js-dependency-mining

# js-dependency-mining Read-only **mining** codemod: scans `package.json` manifests (single-package repos and monorepos) together with `package-lock.json`, `pnpm-lock.yaml`, or `yarn.lock`, classifies each dependency, and emits **`js-dependency-mining`** metrics. It does **not** modify source files. **No `fs` capability required** — no Node `fs` or `path` imports. The workflow only targets `**/package.json`. Lockfiles are loaded with **`jssgTransform`** and small JSON/YAML transforms that capture `root.root().text()` (same mechanism as cross-file edits in the JSSG docs). ## Classifications Each finding is keyed by **owner package** (workspace-relative path), **package name**, and **edge type** (`direct` | `transitive`). Metric dimensions include: | Field | Values | |--------|--------| | `packageName` | npm package name | | `declaredVersion` | Range or spec from the manifest (empty for transitive-only rows) | | `resolvedVersion` | Version from the workspace lockfile when resolvable (empty if missing or unknown) | | `lockfileKind` | `npm`, `pnpm`, `yarn`, or `none` when no lockfile was loaded | | `overridePin` | Best-effort pins from npm `overrides`, Yarn `resolutions`, or pnpm `pnpm.overrides` for this dependency in the **current** manifest (empty for transitive rows and when absent) | | `requiredBy` | **Transitive only:** immediate parent package name(s) in the lockfile graph that depend on this package (sorted, joined with ` \| `). Empty for direct rows or when unknown. | | `ownerPackage` | Workspace-relative owning package path | | `manifestPath` | Path to `package.json` (relative to workspace root) | | `edgeType` | direct, transitive | | `purpose` | runtime, dev, test, build, optional, unknown | | `origin` | registry, vcs, local, workspace, vendored, internal, unknown | | `versionPolicy` | pinned, ranged, prerelease, unknown | ## Workspace root While processing a manifest, the codemod walks **up** the directory path and uses **`jssgTransform`** to probe for `package-lock.json` (language `json`), `pnpm-lock.yaml` / `yarn.lock` / `pnpm-workspace.yaml` (language `yaml`), and reads sibling `package.json` files to detect npm **`workspaces`**. The first directory that qualifies as a workspace root wins; the manifest is then resolved relative to that root. ## Pipeline 1. **Workspace root** — Walk upward from the manifest; at each level, try lockfiles and workspace markers via `jssgTransform` until a root is found. 2. **Lockfile** — From that root, load the preferred lockfile in order **npm → pnpm → yarn** using `jssgTransform` + capture transforms (`json` for `package-lock.json`, `yaml` for pnpm/yarn). Parse with `JSON.parse`, tree-sitter YAML AST helpers for pnpm (with a text fallback for very large files), and the built-in Yarn Classic line parser. The loaded lockfile for a given workspace root is **cached** for the rest of the run so each manifest does not re-read the same file. 3. **Metrics** — For each dependency block in the manifest, emit direct rows; subtract directs from the lockfile transitive closure and emit transitive rows. While walking the lockfile graph, record **immediate** parent → child edges so transitive rows can list **who** pulled that package in (`requiredBy`). **Workspace package names** (for `origin=workspace` heuristics) are accumulated in a **module-level registry** as each `package.json` in the run is seen — no `codemod:workflow` state. ## Configuration Parameters are declared in `workflow.yaml` (`params.schema`) and **merge with built-in defaults** (extra test/build lists, default vendored fragments, etc.). **Where to edit them** - **Codemod Insight UI** — When you run this workflow in Insight, each parameter appears as a labeled field. You can change values there without editing YAML; that is the easiest way to tune classification for a run or saved job. - **CLI** — Pass `--param key=value` to `codemod workflow run` (repeat for multiple keys), e.g. `--param internal_scopes=@acme --param test_packages=playwright`. **Parameters and examples** (values are comma-separated unless noted): | Parameter | Example | What it does | |-----------|---------|----------------| | `internal_scopes` | `@acme,@calcom` | Scope prefixes for **`origin=internal`**. Packages whose names start with these (e.g. `@acme/foo`) are treated as internal when not better classified as workspace/VCS/local. | | `test_packages` | `storybook,@storybook/react` | Extra names merged into the test-tool list. If listed in **`devDependencies`**, **`purpose`** can be **`test`** (instead of generic `dev`). | | `build_packages` | `turbo,tsc-alias` | Extra names merged into the build-tool list. If listed in **`devDependencies`**, **`purpose`** can be **`build`**. | | `vendored_path_fragments` | `patches/,/vendor/` | Extra path substrings. For `file:` / `link:` dependency specs, if the path contains one of these (in addition to defaults like `vendor/`), **`origin`** can be **`vendored`**. | ## Implementation notes - **No `fs` / `path` imports.** Path operations use pure-string helpers. **`jssgTransform`** reads lockfiles under the workflow target; paths must stay within that target (per JSSG rules). - **`jssgTransform` in `jssg test` is a no-op** — lockfiles are not loaded in unit tests, so snapshots typically show **direct** dependencies only. Real runs (`codemod workflow run` / Insight) load lockfiles and emit **transitive** rows. - **Yarn classic** lockfiles are read with language `yaml` for parsing; the custom line parser runs on the captured text. If ast-grep cannot parse a particular `yarn.lock`, that lock may be skipped (catch and continue). Berry-style dependency lines (`name: "npm:…"`) can make naive key captures include a trailing colon; those are stripped when building package-name metrics. - **Transitive** dependencies are derived from the lockfile closure minus direct declarations (npm and pnpm v9 snapshots supported; yarn classic is best-effort). - **Conservative**: ambiguous cases use **`unknown`** where classification would be a guess (e.g. transitive **`purpose`**). - **pnpm** `pnpm-lock.yaml` is deserialized from the **tree-sitter YAML AST** (inline helpers in `scripts/codemod.ts`) inside a `jssgTransform` callback — no third-party YAML library, so Insight bundling stays simple. - **Overrides** — `overridePin` reflects override/resolution fields on the manifest being scanned; root-level overrides in another `package.json` are not merged in (scan the workspace root manifest for a full picture).