Summary

RFC: #2789
Implementation: #10470
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#sparse-registry
Issues: A-sparse-registry Area: http sparse registries

This is a tracking issue for RFC #2789, an experimental extension to serve the index over HTTP instead of via git.

Unresolved issues

• • Cache invalidation

    • Currently 1 minute TTL. Too long for publish, too short for effective caching.
    • Minor concern about how long an invalidation takes (Cloudfront says 10 to 100 seconds)
    • The cost is expected to negligible.
    • Send invalidations for index.crates.io when updating crates crates.io#4913

    To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.
  • Make sparse+ and registry+ more consistent. HTTP registry implementation #10470 (comment) (resolved Add new SourceKind::SparseRegistry to differentiate sparse registries #11209)
  • Find a better name (and structure) for index_version HTTP registry implementation #10470 (comment)
  • Handle user urls that start with http. They currently imply registry+. But if sparse+ are more common then it would be good to auto detect. RFC: Serve crates-io registry over HTTP as static files rfcs#2789 (comment)
  • Have a plan for lockfiles.
  • Implement support in crates.io Upload index metadata to index/ when publishing new crates crates.io#4661
  • Check that crates.io is willing to maintain their current system at production levels of traffic.
  • Teach cargo that crates.io has two equivalent indexes. Teach cargo that crates.io has two equivalent indexes. #10722
  • Make sure we are future compatible with a merkle tree design. Sparse registry indexes should be viewable atomically #10928
  • The RFC is intentionally sparse on details, it is almost an eRFC. Do we need another RFC/PR to discuss the details we have chosen?
  • Does not support fuzzy queries. Is that a blocker? Only used by error messages. This was a misunderstanding, indexes don't support name-based typo suggestions at all. Sparse supports the same case-folding and -/_ fuzzy comparison as git indexes. Sparse fundamentally won't be able to support typo suggestions, but since cargo hasn't handled that in the past, this isn't a regression, and more like a "can't do that in the future" thing.
  • Investigate publish issues. 1 minute update is quite slow if publishing multiple packages (and cargo does not yet have retry logic).
  To pick up a draggable item, press the space bar. While dragging, use the arrow keys to move the item. Press space again to drop the item in its new position, or press escape to cancel.

Current best way to test

To try it out, add the -Z sparse-registry flag on nightly-2022-06-20 or newer build
of Cargo. For example, to update dependencies:

rustup update nightly
cargo +nightly -Z sparse-registry update

The feature can also be enabled by setting the environment variable
CARGO_UNSTABLE_SPARSE_REGISTRY=true. Setting this variable will have no effect on stable
Cargo, making it easy to opt-in for CI jobs.

If you see any issues please report them in new tickets here in the Cargo repo. The output of Cargo
with the environment variable CARGO_LOG=cargo::sources::registry::http_remote=trace set
will be helpful in debugging.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Implementation history

• Registry functions return Poll to enable parallel fetching of index data #10064

• HTTP registry implementation #10470

• Refactor RegistryData::load to handle management of the index cache #10482

• Require http-registry URLs to end with a '/' #10698

• Make -Z http-registry use index.crates.io when accessing crates-io #10725

• Improve testing framework for http registries #10738

• Make is_yanked return Poll<> #10830

• Fix publishing to crates.io with -Z sparse-registry #10831

• Refactor check_yanked to avoid some duplication #10835

• Change progress indicator for sparse registries #11068

• Add retry support to sparse registries #11069

• Fix sparse registry lockfile urls containing 'registry+sparse+' #11177

• Add new SourceKind::SparseRegistry to differentiate sparse registries #11209

• Add configuration option for controlling crates.io protocol #11215

• Upload index metadata to index/ when publishing new crates crates.io#4661

• Drop index/ prefix when uploading crate metadata. crates.io#4826