checkout: Use new approach to caching

Disable use of the upstream caching functionality. It works for single
repositories but does not cache submodules. Instead, do a full non-bare
checkout of the repository in question, with submodules. Not doing a
bare checkout because bare checkouts don't work with submodules
(http://b/110807286). There's also no good way to clone with submodules
from a local path, so copying the entire tree at the suggestion of the
git-core team.

This is heavily based on the caching section of
https://fuchsia.googlesource.com/infra/recipes/+/cffb534560ac7cc539398ac186e19ca1e6004540/recipe_modules/git/api.py#276

Tested with one downstream project that usually took 3-4 minutes to
checkout before this change. With this change the cold checkout time
remained unchanged but with a warm cache checkout time was 30-40
seconds.

Change-Id: I9db1833fe31290533e45087d0c5bb60170fa8544
Bug: b/217449281
Reviewed-on: https://pigweed-review.googlesource.com/c/infra/recipes/+/84744
Reviewed-by: Marc-Antoine Ruel <maruel@google.com>
Reviewed-by: Ted Pudlik <tpudlik@google.com>
Commit-Queue: Rob Mohr <mohrr@google.com>
89 files changed
tree: 724146a24c3d3a0923fd869b9bca18e339b634c5
  1. infra/
  2. recipe_modules/
  3. recipes/
  4. scripts/
  5. .gitignore
  6. .style.yapf
  7. AUTHORS
  8. cipd.ensure
  9. LICENSE
  10. OWNERS
  11. presubmit.sh
  12. pyproject.toml
  13. README.md
  14. recipes.py
README.md

Pigweed Recipes

This repository contains recipes for Pigweed.

A recipe is a Python script that runs a series of commands, using the recipe engine framework from the LUCI project. We use recipes to automatically check out, build, and test Pigweed and downstream projects using Pigweed in continuous integration jobs. The commands the recipes use are very similar to the ones you would use as a developer to check out, build, and test Pigweed in your local environment.

See go/pigweed-recipe-docs for complete documentation and a guide for getting started with writing recipes.

Getting the Code

The recommended way to get the source code is with git.

git clone https://pigweed.googlesource.com/infra/recipes

In most cases you will need a Chromium depot_tools checkout in your PATH as well.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools ~/depot_tools
echo 'export PATH="$PATH:$HOME/depot_tools"' >> ~/.bashrc

Running Tests

./presubmit.sh runs three different sets of tests:

  • Recipe expectation tests (./recipes.py test train)
  • Formatting (./black --diff --check .)
  • Dependencies (.recipe_deps/fuchsia/scripts/cleanup_deps.py --check)

The formatting check will tell you what‘s wrong but not fix it. For that you need to run ./black .. Similarly, the dependencies check will tell you what’s wrong but you'll need to edit the files to fix issues.

If not using ./presubmit.sh you'll need to run ./scripts/ensure_black.sh before ./black is present.