npm_tarballs

This is an expermental feature inspired by external package fetching in rules_go and others.

See the design doc: https://hackmd.io/gu2Nj0TKS068LKAf8KanuA

Rules

translate_package_lock.bzl takes a package-lock.json file and produces a Starlark representation of downloader rules for each package listed.

Currently this is implemented only for npm v7 produced lockfiles (version 2 of the spec) but it could be ported to any other lockfile format.

For example, for https://github.com/bazelbuild/rules_nodejs/blob/stable/packages/node-patches/package-lock.json we produce an index.bzl file like:

"Generated by package_lock.bzl from //packages/node-patches:package-lock.json"

load("@build_bazel_rules_nodejs//internal/common:download.bzl", "bazel_download")

def npm_repositories():
    """Define external repositories to fetch each tarball individually from npm on-demand.
    """

# [...]

    bazel_download(
        name = "npm_typescript-3.5.3",
        output = "typescript-3.5.3.tgz",
        integrity = "sha512-ACzBtm/PhXBDId6a6sDJfroT2pOWt/oOnk4/dElG5G33ZL776N3Y6/6bKZJBFpd+b05F3Ct9qDjMeJmRWtE2/g==",
        url = ["https://registry.npmjs.org/typescript/-/typescript-3.5.3.tgz"],
        build_file_content = """"Generated by package_lock.bzl"

load("@build_bazel_rules_nodejs//internal/npm_tarballs:npm_tarball.bzl", "npm_tarball")

npm_tarball(
    name = "npm_typescript-3.5.3",
    src = "typescript-3.5.3.tgz",
    package_name = "typescript",
    deps = [],
    visibility = ["//visibility:public"],
)

"""
    )

# [...]

This generated index.bzl can then be loaded in the WORKSPACE and the npm_repositories macro called. This then declares bazel_download rules that are themselves able to fetch packages on-demand. We also supply a BUILD file content for each of these packages, using a minimal npm_tarball rule that represents the location and dependencies of the downloaded .tgz file.

In addition, we give some syntax sugar. In the repo produced by translate_package_lock we provide “catch-all” targets //:dependencies and //:devDependencies that depend on all tarballs so listed in the package-lock.json. For direct dependencies, we also produce a //somepackage target that aliases the version of somepackage depended on. In the above example, that means the user can dep on @npm_repositories//typescript rather than @npm_typescript-3.5.3 because we know the package depends on version 3.5.3.

Future work

So far the resulting tarballs aren‘t used by anything in rules_nodejs (nothing consumes NpmTarballInfo). In later work we’ll explore what other rules might want to use the tarballs, such as a pnpm_install rule that uses pnpm semantics to just symlink things into a tree. Or maybe an npm_install rule, one for each package, that unpacks the tarballs and runs the postinstall logic on each. We believe some experimentation will be required to find a good path forward that uses the download-as-needed semantics here, while keeping most existing semantics of rules_nodejs rules working.