Pass the chunk length to `NewReader()` when reading chunks.

Chunk reads create a child reader with `NewReader(offset)` and read the chunk
with `MaskedReader`. They now use `NewReader(offset, length_hint)` with the
length being read, so that `riegeli::GcsReader` sends one request for exactly
this range instead of an open-ended request which is cancelled after the chunk
is read. Sources which do not use the hint, e.g. `FileReader`, `FdReader`, and
`StringReader`, read as before.

With `index_storage_option:offloaded`, each chunk offset lookup reads the
footer chunk with `DefaultChunkReader`. Its length, measured when the footer is
read by `Initialize()`, is passed too, so these lookups no longer send an
open-ended request each which is cancelled before the footer padding and
postscript are read.

PiperOrigin-RevId: 1009621243
4 files changed
tree: 9f0ce1220522284424da104e588b0d35bda4557c
  1. .github/
  2. doc/
  3. python/
  4. riegeli/
  5. tf_dependency/
  6. .bazelrc
  7. configure
  8. CONTRIBUTING.md
  9. LICENSE
  10. MANIFEST.in
  11. MODULE.bazel
  12. README.md
README.md

Riegeli

Riegeli/records is a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding.

See documentation.

Status

Riegeli file format will only change in a backward compatible way (i.e. future readers will understand current files, but current readers might not understand files using future features).

Riegeli C++ API might change in incompatible ways.