docs: document lazy indexing (#2188)
**What type of PR is this?**
> Documentation
**What package or component does this PR mostly affect?**
> all
**What does this PR do? Why is it needed?**
In README.rst, added a subsection under `fix` and `update` explaining
how to use lazy indexing from a user's perspective. This is mentioned in
a few other places with the `-r` and `-index` flags and the `go_search`
and `proto_search` directives, but it's hard to put it all together. A
larger documentation refactoring may be needed to separate reference
material from "how to" (following [Diataxis](https://diataxis.fr/)), but
this adds the needed material for now.
In extend.md, added a section on how to implement lazy indexing in a
language extension, using Go as an example. I didn't point to code here;
not sure how useful it would be.
**Which issues(s) does this PR fix?**
For #1891
**Other notes for review**
diff --git a/README.rst b/README.rst
index 164b677..5998237 100644
--- a/README.rst
+++ b/README.rst
@@ -44,6 +44,7 @@
.. _rules_rust: https://github.com/bazelbuild/rules_rust
.. _Verbs Tutorial: https://bazel.build/rules/verbs-tutorial
.. _cc_search: https://github.com/EngFlow/gazelle_cc?tab=readme-ov-file#-gazellecc_search-strip_include_prefix-include_prefix
+.. _proto_library: https://bazel.build/reference/be/protocol-buffer#proto_library
.. role:: cmd(code)
.. role:: flag(code)
@@ -651,6 +652,62 @@
.. _Predefined plugins: https://github.com/bazelbuild/rules_go/blob/master/proto/core.rst#predefined-plugins
+Lazy indexing in ``fix`` and ``update``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, ``fix`` and ``update`` read all build files in a repo to build an
+index of library rules (see `Dependency resolution`_) when Gazelle starts.
+This can take a long time on a large repo. To avoid this problem, Gazelle can
+lazily index specific directories, with help from extensions that support
+lazy indexing.
+
+To configure lazy indexing with Go, add ``go_search`` directives like this:
+
+.. code:: bzl
+ # gazelle:go_search third_party/go
+ # gazelle:go_search replace/b example.com/b
+
+These directives point to directories that contain Go code outside the current
+module, with an optional package prefix. ``go_search`` directives are not
+necessary if you're following regular Go module conventions or are using
+a Go ``vendor`` directory.
+
+To configure lazy indexing with protobuf, add ``proto_search`` directives
+like this:
+
+.. code:: bzl
+ # gazelle:proto_search third_party/proto api
+
+The two arguments are a prefix to remove from the import path and a prefix to
+add. These correspond to the `strip_import_prefix`_ and `import_prefix`_
+attributes of `proto_library`_. They tell Gazelle how to transform an import
+path read from a .proto source file into a repo-root-relative path to a
+directory that may contain the imported file.
+
+To use Gazelle with lazy indexing, run with ``-r=false -index=lazy``, and pass
+the directories to update on the command line.
+
+.. code:: bzl
+ gazelle -r=false -index=lazy path/to/dir1 path/to/dir2
+
+You can configure your ``gazelle`` Bazel target to pass these flags
+automatically:
+
+.. code:: bzl
+ load("@gazelle//:def.bzl", "gazelle", "gazelle_binary")
+
+ gazelle(
+ name = "gazelle",
+ command = "fix",
+ extra_args = ["-r=false", "-index=lazy"],
+ gazelle = ":gazelle_binary",
+ )
+
+ gazelle_binary(
+ name = "gazelle_binary",
+ ...
+ )
+
``update-repos``
~~~~~~~~~~~~~~~~
diff --git a/extend.md b/extend.md
index 3ac0a18..35e73a6 100644
--- a/extend.md
+++ b/extend.md
@@ -77,6 +77,60 @@
You can run this with `bazel run //:gazelle`.
+Lazy indexing
+-------------
+
+Lazy indexing lets Gazelle run quickly without needing to read all build files
+in a repo while still supporting index-based dependency resolution. This allows
+Gazelle to update build files for specific directories in milliseconds rather
+than seconds or minutes.
+
+Lazy indexing requires a small amount of user configuration, pointing to
+directories that may contain libraries based on import strings read from source
+files. It also requires support from language extensions to interpret that
+configuration. Most language extensions should implement this, though the
+implementation will be a bit different for each language.
+
+As an extended example, suppose a Go user has copied their module dependency
+`example.com/b` into the directory `replace/b`. From some other directory `a`,
+they import the package `example.com/b/c`. They then run
+`gazelle update -r=false -index=lazy a` to generate a build file for `a` with
+lazy indexing enabled.
+
+In this configuration, Gazelle doesn't automatically index `replace/b`, so
+the user must add a directive to their top-level build file:
+
+```starlark
+# gazelle:go_search replace/b example.com/b
+```
+
+This directive is interpreted by the Go extension. The user is telling Gazelle
+to index the directory `replace/b/<suffix>` when it sees a Go package imported
+with the string `example.com/b/<suffix>`. So in our example, when Gazelle sees
+`example.com/b/c`, it indexes `replace/b/c` and makes all library targets
+available for dependency resolution, including non-Go libraries that happen to
+be there.
+
+To support this, the Go extension needs to:
+
+1. Support the `go_search` directive in the `KnownDirectives` and `Configure`
+methods of the
+[`Configurer`](https://pkg.go.dev/github.com/bazelbuild/bazel-gazelle/config#Configurer)
+implementation. This directive may be repeated and applies in subdirectories.
+1. Convert an import string like `example.com/b/c` into a list of directory
+paths like `replace/b/c`.
+1. Return the directory paths through
+[`GenerateResult.RelsToIndex`](https://pkg.go.dev/github.com/bazelbuild/bazel-gazelle/language#GenerateResult)
+in the
+[`Language.GenerateRules`](https://pkg.go.dev/github.com/bazelbuild/bazel-gazelle/language#Language)
+method.
+
+Other extensions should follow a similar approach, though there are likely to
+be differences in how `*_search` directives and import strings are interpreted.
+For example, the protobuf and C++ extensions have `proto_search` and `cc_search`
+directives which are similar to each other but not to Go: both languages
+import libraries by file name and have similar conventions.
+
Interacting with protos
-----------------------
