fix(pypi): resolve self-referencing extras to a real fixed point (#4039)

## Summary

`_resolve_extras` decides that its fixed-point loop has converged by
comparing `num_extras_before` — the size of the extras set at the
*start* of the round — against `len(new_extras)`, the number of extras
discovered *during* that round:

```python
num_extras_before = len(extras)
extras = extras | new_extras
num_extras_after = len(new_extras)   # the delta, not the merged size

if num_extras_before == num_extras_after:
    break
```

Those are unrelated quantities, and the mismatch breaks in two separate
ways.

### 1. Extras are silently dropped

The loop exits early whenever the two counts happen to coincide while
the set is still growing, so extras reachable only through a further
round are never resolved —- and every dependency gated on them silently
disappears from the generated target.

The smallest reproducer is a two-hop self-extras chain:

| `Requires-Dist` | requested | resolved today | expected |
|---|---|---|---|
| `foo[b]; extra == 'all'`, `foo[c]; extra == 'b'` | `foo[all]` | `{all,
b}` | `{all, b, c}` |

Anything behind `extra == 'c'` is lost, with no error. This is not
limited to the first round: a chain that branches before it deepens
(`all → {p, q}`, `p → r`, `r → t`) hits the same equality on round two
and drops `t`.

### 2. The loop never terminates early for ordinary packages

For a package with no self-referencing extras — the overwhelmingly
common case — `self_reqs` is empty, so `new_extras` is always `{}` while
`extras` holds at least one entry. The condition can never hold, and the
loop runs all 10000 rounds, allocating a dict each time, while
evaluating the generated `BUILD` file of every wheel in the build.

On a ~52k-package repository this dominated loading-phase Starlark CPU:

| Metric | Before | After |
|---|---|---|
| `_resolve_extras` self-time | 430–442 CPU-s | below profiler threshold
|
| Total Starlark user-function CPU | 747–759 CPU-s | 275–298 CPU-s |
| Cold loading+analysis wall time (16 cores) | 82–84 s | 73–75 s |

## The fix

Compare the size of the *merged* set, which is what the `before`/`after`
naming already implied. The loop is monotonic, so the converged set is
unchanged wherever it previously terminated correctly — this only stops
it terminating too early, or not at all.

## Tests

Three regression tests are added to `tests/pypi/pep508/deps_tests.bzl`,
covering the three shapes that trigger the early exit: a two-hop chain,
multiple requested extras, and a chain where the counts only coincide
after the first round.

All three fail on `main` with exactly the dropped dependency, and pass
with the fix:

```
//tests/pypi/pep508:test_self_extras_chain_is_fully_resolved              FAILED   1 missing: c_dep
//tests/pypi/pep508:test_self_extras_chain_with_multiple_requested_extras FAILED   1 missing: c_dep
//tests/pypi/pep508:test_self_extras_chain_resolved_beyond_the_first_round FAILED  1 missing: t_dep
```

The three existing tests that exercise self-extras chains
(`test_self_is_ignored`, `test_self_dependencies_can_come_in_any_order`,
`test_self_include_deps_from_previously_visited`) pass either way, which
is why this went unnoticed.

`bazel test //tests/pypi/...` is green (249/249) with the fix applied.

## Notes

The condition was introduced in #3527, which replaced the previous
double loop with this fixed-point loop.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 files changed
tree: bea05393dcf162eb8d476d65afd12d144414ee59
  1. .agents/
  2. .bazelci/
  3. .bcr/
  4. .ci/
  5. .github/
  6. command_line_option/
  7. dev/
  8. docs/
  9. examples/
  10. gazelle/
  11. news/
  12. private/
  13. python/
  14. sphinxdocs/
  15. tests/
  16. tools/
  17. .bazelignore
  18. .bazelrc
  19. .bazelrc.deleted_packages
  20. .bazelversion
  21. .editorconfig
  22. .git-blame-ignore-revs
  23. .gitattributes
  24. .gitignore
  25. .pre-commit-config.yaml
  26. .python-version
  27. .readthedocs.yml
  28. addlicense.sh
  29. AGENTS.md
  30. AUTHORS
  31. BUILD.bazel
  32. BZLMOD_SUPPORT.md
  33. CHANGELOG.md
  34. CONTRIBUTING.md
  35. CONTRIBUTORS
  36. downloader_config.cfg
  37. GEMINI.md
  38. internal_dev_deps.bzl
  39. internal_dev_setup.bzl
  40. LICENSE
  41. MODULE.bazel
  42. README.md
  43. RELEASING.md
  44. replicate_ci
  45. ruff.toml
  46. specialized_configs.bazelrc
  47. version.bzl
  48. WORKSPACE
  49. WORKSPACE.bzlmod
  50. workspace_bazel9.bzl
README.md

Python Rules for Bazel

Build status

Overview

This repository is the home of the core Python rules -- py_library, py_binary, py_test, and related symbols that provide the basis for Python support in Bazel. It also contains package installation rules for integrating with PyPI and other indices.

Documentation for rules_python is at https://rules-python.readthedocs.io and in the Bazel Build Encyclopedia.

Examples live in the examples directory.

The core rules are stable. Their implementation is subject to Bazel's backward compatibility policy. This repository aims to follow semantic versioning.

The Bazel community maintains this repository. Neither Google nor the Bazel team provides support for the code. However, this repository is part of the test suite used to vet new Bazel releases. See How to contribute page for information on our development workflow.

Design

  • Supported OSes - as per our supported platform policy, we strive for support on all of the platforms that we have CI for. Some platforms do not have the same backwards compatibility guarantees, but we hope the community can step in where needed to make the support more robust.
  • requirements.txt is how users have been defining dependencies for a long time. We support this to support legacy usecases or package managers that we don't support directly. Any additional information that we need will be retrieved from the SimpleAPI during the bzlmod extension evaluation phase. Then it will be written to the MODULE.bazel.lock file for future reuse. We have plans to support uv.lock file directly. uv is recommended for generating a fully locked requirements.txt file and we do provide a rule for it.
  • The py_binary, py_test rules should scale to large monorepos and we work hard to minimize the work done during analysis and build phase. What is more, the space requirements for should be minimal, so we strive to use symlinks rather than extracting wheels at build time. This means that for different configurations of the same build, we are not extracting the wheel multiple times thus scaling better over the time. From 2.0 onwards we are creating a virtual env for each target by creating an actual minimal virtual environment using symlinks. We plan on creating the traditional site-packages layout in the future by default.
  • Support for standards - we strive to first implement any standards needed within rules_python and this has resulted in a few PEPs supported within pure starlark - PEP440, PEP509.

Common misconceptions:

  • rules_python has to keep backwards compatibility with google3. Whilst this might have been true in the past, rules_python is an open source project and any compatibility needs should come from the community - we have no requirement to keep this compatibility and are allowed to make our decisions. However, we do want to keep backwards compatibility as long as possible to not upset users with never ending migrations.
  • rules_python is not caching pip downloads. With 2.0, we use Bazel's downloader by default and rely on bazel to provide the repository caching mechanisms. This means that for simpler setups this should result in transparent and scalable caching with the most recent bazel versions unless there are issues in the bazel itself.

Documentation

For detailed documentation, see https://rules-python.readthedocs.io

Bzlmod support

See Bzlmod support for more details.