| //! A `Source` for registry-based packages. |
| //! |
| //! # What's a Registry? |
| //! |
| //! [Registries] are central locations where packages can be uploaded to, |
| //! discovered, and searched for. The purpose of a registry is to have a |
| //! location that serves as permanent storage for versions of a crate over time. |
| //! |
| //! Compared to git sources (see [`GitSource`]), a registry provides many |
| //! packages as well as many versions simultaneously. Git sources can also |
| //! have commits deleted through rebasings where registries cannot have their |
| //! versions deleted. |
| //! |
| //! In Cargo, [`RegistryData`] is an abstraction over each kind of actual |
| //! registry, and [`RegistrySource`] connects those implementations to |
| //! [`Source`] trait. Two prominent features these abstractions provide are |
| //! |
| //! * A way to query the metadata of a package from a registry. The metadata |
| //! comes from the index. |
| //! * A way to download package contents (a.k.a source files) that are required |
| //! when building the package itself. |
| //! |
| //! We'll cover each functionality later. |
| //! |
| //! [Registries]: https://doc.rust-lang.org/nightly/cargo/reference/registries.html |
| //! [`GitSource`]: super::GitSource |
| //! |
| //! # Different Kinds of Registries |
| //! |
| //! Cargo provides multiple kinds of registries. Each of them serves the index |
| //! and package contents in a slightly different way. Namely, |
| //! |
| //! * [`LocalRegistry`] --- Serves the index and package contents entirely on |
| //! a local filesystem. |
| //! * [`GitRegistry`] --- Serves the index ahead of time from a Git |
| //! repository, and package contents are downloaded as needed. |
| //! * [`HttpRegistry`] --- Serves both the index and package contents on demand |
| //! over a HTTP-based registry API. This is the default starting from 1.70.0. |
| //! |
| //! Each registry has its own [`RegistryData`] implementation, and can be |
| //! created from either [`RegistrySource::local`] or [`RegistrySource::remote`]. |
| //! |
| //! [`LocalRegistry`]: local::LocalRegistry |
| //! [`GitRegistry`]: git_remote::GitRegistry |
| //! [`HttpRegistry`]: http_remote::HttpRegistry |
| //! |
| //! # The Index of a Registry |
| //! |
| //! One of the major difficulties with a registry is that hosting so many |
| //! packages may quickly run into performance problems when dealing with |
| //! dependency graphs. It's infeasible for cargo to download the entire contents |
| //! of the registry just to resolve one package's dependencies, for example. As |
| //! a result, cargo needs some efficient method of querying what packages are |
| //! available on a registry, what versions are available, and what the |
| //! dependencies for each version is. |
| //! |
| //! To solve the problem, a registry must provide an index of package metadata. |
| //! The index of a registry is essentially an easily query-able version of the |
| //! registry's database for a list of versions of a package as well as a list |
| //! of dependencies for each version. The exact format of the index is |
| //! described later. |
| //! |
| //! See the [`index`] module for topics about the management, parsing, caching, |
| //! and versioning for the on-disk index. |
| //! |
| //! ## The Format of The Index |
| //! |
| //! The index is a store for the list of versions for all packages known, so its |
| //! format on disk is optimized slightly to ensure that `ls registry` doesn't |
| //! produce a list of all packages ever known. The index also wants to ensure |
| //! that there's not a million files which may actually end up hitting |
| //! filesystem limits at some point. To this end, a few decisions were made |
| //! about the format of the registry: |
| //! |
| //! 1. Each crate will have one file corresponding to it. Each version for a |
| //! crate will just be a line in this file (see [`cargo_util_schemas::index::IndexPackage`] for its |
| //! representation). |
| //! 2. There will be two tiers of directories for crate names, under which |
| //! crates corresponding to those tiers will be located. |
| //! (See [`cargo_util::registry::make_dep_path`] for the implementation of |
| //! this layout hierarchy.) |
| //! |
| //! As an example, this is an example hierarchy of an index: |
| //! |
| //! ```notrust |
| //! . |
| //! ├── 3 |
| //! │ └── u |
| //! │ └── url |
| //! ├── bz |
| //! │ └── ip |
| //! │ └── bzip2 |
| //! ├── config.json |
| //! ├── en |
| //! │ └── co |
| //! │ └── encoding |
| //! └── li |
| //! ├── bg |
| //! │ └── libgit2 |
| //! └── nk |
| //! └── link-config |
| //! ``` |
| //! |
| //! The root of the index contains a `config.json` file with a few entries |
| //! corresponding to the registry (see [`RegistryConfig`] below). |
| //! |
| //! Otherwise, there are three numbered directories (1, 2, 3) for crates with |
| //! names 1, 2, and 3 characters in length. The 1/2 directories simply have the |
| //! crate files underneath them, while the 3 directory is sharded by the first |
| //! letter of the crate name. |
| //! |
| //! Otherwise the top-level directory contains many two-letter directory names, |
| //! each of which has many sub-folders with two letters. At the end of all these |
| //! are the actual crate files themselves. |
| //! |
| //! The purpose of this layout is to hopefully cut down on `ls` sizes as well as |
| //! efficient lookup based on the crate name itself. |
| //! |
| //! See [The Cargo Book: Registry Index][registry-index] for the public |
| //! interface on the index format. |
| //! |
| //! [registry-index]: https://doc.rust-lang.org/nightly/cargo/reference/registry-index.html |
| //! |
| //! ## The Index Files |
| //! |
| //! Each file in the index is the history of one crate over time. Each line in |
| //! the file corresponds to one version of a crate, stored in JSON format (see |
| //! the [`cargo_util_schemas::index::IndexPackage`] structure). |
| //! |
| //! As new versions are published, new lines are appended to this file. **The |
| //! only modifications to this file that should happen over time are yanks of a |
| //! particular version.** |
| //! |
| //! # Downloading Packages |
| //! |
| //! The purpose of the index was to provide an efficient method to resolve the |
| //! dependency graph for a package. After resolution has been performed, we need |
| //! to download the contents of packages so we can read the full manifest and |
| //! build the source code. |
| //! |
| //! To accomplish this, [`RegistryData::download`] will "make" an HTTP request |
| //! per-package requested to download tarballs into a local cache. These |
| //! tarballs will then be unpacked into a destination folder. |
| //! |
| //! Note that because versions uploaded to the registry are frozen forever that |
| //! the HTTP download and unpacking can all be skipped if the version has |
| //! already been downloaded and unpacked. This caching allows us to only |
| //! download a package when absolutely necessary. |
| //! |
| //! # Filesystem Hierarchy |
| //! |
| //! Overall, the `$HOME/.cargo` looks like this when talking about the registry |
| //! (remote registries, specifically): |
| //! |
| //! ```notrust |
| //! # A folder under which all registry metadata is hosted (similar to |
| //! # $HOME/.cargo/git) |
| //! $HOME/.cargo/registry/ |
| //! |
| //! # For each registry that cargo knows about (keyed by hostname + hash) |
| //! # there is a folder which is the checked out version of the index for |
| //! # the registry in this location. Note that this is done so cargo can |
| //! # support multiple registries simultaneously |
| //! index/ |
| //! registry1-<hash>/ |
| //! registry2-<hash>/ |
| //! ... |
| //! |
| //! # This folder is a cache for all downloaded tarballs (`.crate` file) |
| //! # from a registry. Once downloaded and verified, a tarball never changes. |
| //! cache/ |
| //! registry1-<hash>/<pkg>-<version>.crate |
| //! ... |
| //! |
| //! # Location in which all tarballs are unpacked. Each tarball is known to |
| //! # be frozen after downloading, so transitively this folder is also |
| //! # frozen once its unpacked (it's never unpacked again) |
| //! # CAVEAT: They are not read-only. See rust-lang/cargo#9455. |
| //! src/ |
| //! registry1-<hash>/<pkg>-<version>/... |
| //! ... |
| //! ``` |
| //! |
| |
| use crate::util::data_structures::HashSet; |
| use std::cell::RefCell; |
| use std::fs; |
| use std::fs::{File, OpenOptions}; |
| use std::io; |
| use std::io::Read; |
| use std::io::Write; |
| use std::path::{Path, PathBuf}; |
| |
| use anyhow::Context as _; |
| use cargo_util::paths; |
| use cargo_util_terminal::report::Level; |
| use flate2::read::GzDecoder; |
| use futures::FutureExt as _; |
| use serde::Deserialize; |
| use serde::Serialize; |
| use tar::{Archive, EntryType}; |
| use tracing::debug; |
| |
| use crate::sources::PathSource; |
| use crate::sources::source::MaybePackage; |
| use crate::sources::source::QueryKind; |
| use crate::sources::source::Source; |
| use crate::util::cache_lock::CacheLockMode; |
| use crate::util::interning::InternedString; |
| use crate::util::{CargoResult, Filesystem, GlobalContext, LimitErrorReader, restricted_names}; |
| use crate::util::{VersionExt, hex}; |
| use crate::workspace::dependency::Dependency; |
| use crate::workspace::global_cache_tracker; |
| use crate::workspace::{Package, PackageId, SourceId}; |
| |
| pub use cargo_util_schemas::index::RegistryConfig; |
| |
| /// The `.cargo-ok` file is used to track if the source is already unpacked. |
| /// See [`RegistrySource::unpack_package`] for more. |
| /// |
| /// Not to be confused with `.cargo-ok` file in git sources. |
| const PACKAGE_SOURCE_LOCK: &str = ".cargo-ok"; |
| |
| pub const CRATES_IO_INDEX: &str = "https://github.com/rust-lang/crates.io-index"; |
| pub const CRATES_IO_HTTP_INDEX: &str = "sparse+https://index.crates.io/"; |
| pub const CRATES_IO_REGISTRY: &str = "crates-io"; |
| pub const CRATES_IO_DOMAIN: &str = "crates.io"; |
| |
| /// The content inside `.cargo-ok`. |
| /// See [`RegistrySource::unpack_package`] for more. |
| #[derive(Deserialize, Serialize)] |
| #[serde(rename_all = "kebab-case")] |
| struct LockMetadata { |
| /// The version of `.cargo-ok` file |
| v: u32, |
| } |
| |
| /// A [`Source`] implementation for a local or a remote registry. |
| /// |
| /// This contains common functionality that is shared between each registry |
| /// kind, with the registry-specific logic implemented as part of the |
| /// [`RegistryData`] trait referenced via the `ops` field. |
| /// |
| /// For general concepts of registries, see the [module-level documentation](crate::sources::registry). |
| pub struct RegistrySource<'gctx> { |
| /// A unique name of the source (typically used as the directory name |
| /// where its cached content is stored). |
| name: InternedString, |
| /// The unique identifier of this source. |
| source_id: SourceId, |
| /// The path where crate files are extracted (`$CARGO_HOME/registry/src/$REG-HASH`). |
| src_path: Filesystem, |
| /// Local reference to [`GlobalContext`] for convenience. |
| gctx: &'gctx GlobalContext, |
| /// Abstraction for interfacing to the different registry kinds. |
| ops: Box<dyn RegistryData + 'gctx>, |
| /// Interface for managing the on-disk index. |
| index: index::RegistryIndex<'gctx>, |
| /// Yanked versions that have already been selected during queries. |
| /// |
| /// As of this writing, this is for not emitting the `--precise <yanked>` |
| /// warning twice, with the assumption of (`dep.package_name()` + `--precise` |
| /// version) being sufficient to uniquely identify the same query result. |
| selected_precise_yanked: RefCell<HashSet<(InternedString, semver::Version)>>, |
| } |
| |
| /// Result from loading data from a registry. |
| #[derive(Debug, Clone)] |
| pub enum LoadResponse { |
| /// The cache is valid. The cached data should be used. |
| CacheValid, |
| |
| /// The cache is out of date. Returned data should be used. |
| Data { |
| raw_data: Vec<u8>, |
| /// Version of this data to determine whether it is out of date. |
| index_version: Option<String>, |
| }, |
| |
| /// The requested crate was found. |
| NotFound, |
| } |
| |
| /// An abstract interface to handle both a local and remote registry. |
| /// |
| /// This allows [`RegistrySource`] to abstractly handle each registry kind. |
| /// |
| /// For general concepts of registries, see the [module-level documentation](crate::sources::registry). |
| #[async_trait::async_trait(?Send)] |
| pub trait RegistryData { |
| /// Performs initialization for the registry. |
| /// |
| /// This should be safe to call multiple times, the implementation is |
| /// expected to not do any work if it is already prepared. |
| fn prepare(&self) -> CargoResult<()>; |
| |
| /// Returns the path to the index. |
| /// |
| /// Note that different registries store the index in different formats |
| /// (remote = git, http & local = files). |
| fn index_path(&self) -> &Filesystem; |
| |
| /// Returns the path of the directory that stores the cache of `.crate` files. |
| /// |
| /// The directory is currently expected to contain a flat list of all `.crate` files, |
| /// named `<package-name>-<version>.crate`. |
| fn cache_path(&self) -> &Filesystem; |
| |
| /// Loads the JSON for a specific named package from the index. |
| /// |
| /// * `root` is the root path to the index. |
| /// * `path` is the relative path to the package to load (like `ca/rg/cargo`). |
| /// * `index_version` is the version of the requested crate data currently |
| /// in cache. This is useful for checking if a local cache is outdated. |
| async fn load( |
| &self, |
| root: &Path, |
| path: &Path, |
| index_version: Option<&str>, |
| ) -> CargoResult<LoadResponse>; |
| |
| /// Loads the `config.json` file and returns it. |
| /// |
| /// Local registries don't have a config, and return `None`. |
| async fn config(&self) -> CargoResult<Option<RegistryConfig>>; |
| |
| /// Invalidates locally cached data. |
| fn invalidate_cache(&self); |
| |
| /// If quiet, the source should not display any progress or status messages. |
| fn set_quiet(&mut self, quiet: bool); |
| |
| /// Is the local cached data up-to-date? |
| fn is_updated(&self) -> bool; |
| |
| /// Prepare to start downloading a `.crate` file. |
| /// |
| /// Despite the name, this doesn't actually download anything. If the |
| /// `.crate` is already downloaded, then it returns [`MaybeLock::Ready`]. |
| /// If it hasn't been downloaded, then it returns [`MaybeLock::Download`] |
| /// which contains the URL to download. The [`crate::workspace::package::Downloads`] |
| /// system handles the actual download process. After downloading, it |
| /// calls [`Self::finish_download`] to save the downloaded file. |
| /// |
| /// `checksum` is currently only used by local registries to verify the |
| /// file contents (because local registries never actually download |
| /// anything). Remote registries will validate the checksum in |
| /// `finish_download`. For already downloaded `.crate` files, it does not |
| /// validate the checksum, assuming the filesystem does not suffer from |
| /// corruption or manipulation. |
| async fn download(&self, pkg: PackageId, checksum: &str) -> CargoResult<MaybeLock>; |
| |
| /// Finish a download by saving a `.crate` file to disk. |
| /// |
| /// After [`crate::workspace::package::Downloads`] has finished a download, |
| /// it will call this to save the `.crate` file. This is only relevant |
| /// for remote registries. This should validate the checksum and save |
| /// the given data to the on-disk cache. |
| /// |
| /// Returns a [`File`] handle to the `.crate` file, positioned at the start. |
| async fn finish_download( |
| &self, |
| pkg: PackageId, |
| checksum: &str, |
| data: &[u8], |
| ) -> CargoResult<File>; |
| |
| /// Returns whether or not the `.crate` file is already downloaded. |
| fn is_crate_downloaded(&self, _pkg: PackageId) -> bool { |
| true |
| } |
| |
| /// Validates that the global package cache lock is held. |
| /// |
| /// Given the [`Filesystem`], this will make sure that the package cache |
| /// lock is held. If not, it will panic. See |
| /// [`GlobalContext::acquire_package_cache_lock`] for acquiring the global lock. |
| /// |
| /// Returns the [`Path`] to the [`Filesystem`]. |
| fn assert_index_locked<'a>(&self, path: &'a Filesystem) -> &'a Path; |
| } |
| |
| /// The status of [`RegistryData::download`] which indicates if a `.crate` |
| /// file has already been downloaded, or if not then the URL to download. |
| pub enum MaybeLock { |
| /// The `.crate` file is already downloaded. [`File`] is a handle to the |
| /// opened `.crate` file on the filesystem. |
| Ready(File), |
| /// The `.crate` file is not downloaded, here's the URL to download it from. |
| /// |
| /// `descriptor` is just a text string to display to the user of what is |
| /// being downloaded. |
| Download { |
| url: String, |
| descriptor: String, |
| authorization: Option<String>, |
| }, |
| } |
| |
| mod download; |
| mod http_remote; |
| pub(crate) mod index; |
| pub use index::IndexSummary; |
| mod git_remote; |
| mod local; |
| |
| /// Generates a unique name for [`SourceId`] to have a unique path to put their |
| /// index files. |
| fn short_name(id: SourceId, is_shallow: bool) -> String { |
| // CAUTION: This should not change between versions. If you change how |
| // this is computed, it will orphan previously cached data, forcing the |
| // cache to be rebuilt and potentially wasting significant disk space. If |
| // you change it, be cautious of the impact. See `test_cratesio_hash` for |
| // a similar discussion. |
| let hash = hex::short_hash(&id); |
| let ident = id.url().host_str().unwrap_or("").to_string(); |
| let mut name = format!("{}-{}", ident, hash); |
| if is_shallow { |
| name.push_str("-shallow"); |
| } |
| name |
| } |
| |
| impl<'gctx> RegistrySource<'gctx> { |
| /// Creates a [`Source`] of a "remote" registry. |
| /// It could be either an HTTP-based [`http_remote::HttpRegistry`] or |
| /// a Git-based [`git_remote::GitRegistry`]. |
| pub fn remote( |
| source_id: SourceId, |
| gctx: &'gctx GlobalContext, |
| ) -> CargoResult<RegistrySource<'gctx>> { |
| assert!(source_id.is_remote_registry()); |
| let name = short_name( |
| source_id, |
| gctx.cli_unstable() |
| .git |
| .map_or(false, |features| features.shallow_index) |
| && !source_id.is_sparse(), |
| ); |
| let ops = if source_id.is_sparse() { |
| Box::new(http_remote::HttpRegistry::new(source_id, gctx, &name)?) as Box<_> |
| } else { |
| Box::new(git_remote::GitRegistry::new(source_id, gctx, &name)) as Box<_> |
| }; |
| |
| Ok(RegistrySource::new(source_id, gctx, &name, ops)) |
| } |
| |
| /// Creates a [`Source`] of a local registry, with [`local::LocalRegistry`] under the hood. |
| /// |
| /// * `path` --- The root path of a local registry on the file system. |
| pub fn local( |
| source_id: SourceId, |
| path: &Path, |
| gctx: &'gctx GlobalContext, |
| ) -> RegistrySource<'gctx> { |
| let name = short_name(source_id, false); |
| let ops = local::LocalRegistry::new(path, gctx, &name); |
| RegistrySource::new(source_id, gctx, &name, Box::new(ops)) |
| } |
| |
| /// Creates a source of a registry. This is a inner helper function. |
| /// |
| /// * `name` --- Name of a path segment which may affect where `.crate` |
| /// tarballs, the registry index and cache are stored. Expect to be unique. |
| /// * `ops` --- The underlying [`RegistryData`] type. |
| fn new( |
| source_id: SourceId, |
| gctx: &'gctx GlobalContext, |
| name: &str, |
| ops: Box<dyn RegistryData + 'gctx>, |
| ) -> RegistrySource<'gctx> { |
| // Before starting to work on the registry, make sure that |
| // `<cargo_home>/registry` is marked as excluded from indexing and |
| // backups. Older versions of Cargo didn't do this, so we do it here |
| // regardless of whether `<cargo_home>` exists. |
| // |
| // This does not use `create_dir_all_excluded_from_backups_atomic` for |
| // the same reason: we want to exclude it even if the directory already |
| // exists. |
| // |
| // IO errors in creating and marking it are ignored, e.g. in case we're on a |
| // read-only filesystem. |
| let registry_base = gctx.registry_base_path(); |
| let _ = registry_base.create_dir(); |
| cargo_util::paths::exclude_from_backups_and_indexing(®istry_base.into_path_unlocked()); |
| |
| RegistrySource { |
| name: name.into(), |
| src_path: gctx.registry_source_path().join(name), |
| gctx, |
| source_id, |
| index: index::RegistryIndex::new(source_id, ops.index_path(), gctx), |
| ops, |
| selected_precise_yanked: RefCell::new(HashSet::default()), |
| } |
| } |
| |
| /// Decode the [configuration](RegistryConfig) stored within the registry. |
| /// |
| /// This requires that the index has been at least checked out. |
| pub async fn config(&self) -> CargoResult<Option<RegistryConfig>> { |
| self.ops.config().await |
| } |
| |
| /// Unpacks a downloaded package into a location where it's ready to be |
| /// compiled. |
| /// |
| /// No action is taken if the source looks like it's already unpacked. |
| /// |
| /// # History of interruption detection with `.cargo-ok` file |
| /// |
| /// Cargo has always included a `.cargo-ok` file ([`PACKAGE_SOURCE_LOCK`]) |
| /// to detect if extraction was interrupted, but it was originally empty. |
| /// |
| /// In 1.34, Cargo was changed to create the `.cargo-ok` file before it |
| /// started extraction to implement fine-grained locking. After it was |
| /// finished extracting, it wrote two bytes to indicate it was complete. |
| /// It would use the length check to detect if it was possibly interrupted. |
| /// |
| /// In 1.36, Cargo changed to not use fine-grained locking, and instead used |
| /// a global lock. The use of `.cargo-ok` was no longer needed for locking |
| /// purposes, but was kept to detect when extraction was interrupted. |
| /// |
| /// In 1.49, Cargo changed to not create the `.cargo-ok` file before it |
| /// started extraction to deal with `.crate` files that inexplicably had |
| /// a `.cargo-ok` file in them. |
| /// |
| /// In 1.64, Cargo changed to detect `.crate` files with `.cargo-ok` files |
| /// in them in response to [CVE-2022-36113], which dealt with malicious |
| /// `.crate` files making `.cargo-ok` a symlink causing cargo to write "ok" |
| /// to any arbitrary file on the filesystem it has permission to. |
| /// |
| /// In 1.71, `.cargo-ok` changed to contain a JSON `{ v: 1 }` to indicate |
| /// the version of it. A failure of parsing will result in a heavy-hammer |
| /// approach that unpacks the `.crate` file again. This is in response to a |
| /// security issue that the unpacking didn't respect umask on Unix systems. |
| /// |
| /// This is all a long-winded way of explaining the circumstances that might |
| /// cause a directory to contain a `.cargo-ok` file that is empty or |
| /// otherwise corrupted. Either this was extracted by a version of Rust |
| /// before 1.34, in which case everything should be fine. However, an empty |
| /// file created by versions 1.36 to 1.49 indicates that the extraction was |
| /// interrupted and that we need to start again. |
| /// |
| /// Another possibility is that the filesystem is simply corrupted, in |
| /// which case deleting the directory might be the safe thing to do. That |
| /// is probably unlikely, though. |
| /// |
| /// To be safe, we delete the directory and start over again if an empty |
| /// `.cargo-ok` file is found. |
| /// |
| /// [CVE-2022-36113]: https://blog.rust-lang.org/2022/09/14/cargo-cves.html#arbitrary-file-corruption-cve-2022-36113 |
| fn unpack_package(&self, pkg: PackageId, tarball: &File) -> CargoResult<PathBuf> { |
| let package_dir = format!("{}-{}", pkg.name(), pkg.version()); |
| let dst = self.src_path.join(&package_dir); |
| let path = dst.join(PACKAGE_SOURCE_LOCK); |
| let path = self |
| .gctx |
| .assert_package_cache_locked(CacheLockMode::DownloadExclusive, &path); |
| let unpack_dir = path.parent().unwrap(); |
| match fs::read_to_string(path) { |
| Ok(ok) => match serde_json::from_str::<LockMetadata>(&ok) { |
| Ok(lock_meta) if lock_meta.v == 1 => { |
| self.gctx |
| .deferred_global_last_use()? |
| .mark_registry_src_used(global_cache_tracker::RegistrySrc { |
| encoded_registry_name: self.name, |
| package_dir: package_dir.into(), |
| size: None, |
| }); |
| return Ok(unpack_dir.to_path_buf()); |
| } |
| _ => { |
| if ok == "ok" { |
| tracing::debug!("old `ok` content found, clearing cache"); |
| } else { |
| tracing::warn!("unrecognized .cargo-ok content, clearing cache: {ok}"); |
| } |
| // See comment of `unpack_package` about why removing all stuff. |
| paths::remove_dir_all(dst.as_path_unlocked())?; |
| } |
| }, |
| Err(e) if e.kind() == io::ErrorKind::NotFound => {} |
| Err(e) => anyhow::bail!("unable to read .cargo-ok file at {path:?}: {e}"), |
| } |
| dst.create_dir()?; |
| |
| let bytes_written = unpack(self.gctx, tarball, unpack_dir, &|_| true)?; |
| update_mtime_for_generated_files(unpack_dir); |
| |
| // Now that we've finished unpacking, create and write to the lock file to indicate that |
| // unpacking was successful. |
| let mut ok = OpenOptions::new() |
| .create_new(true) |
| .read(true) |
| .write(true) |
| .open(&path) |
| .with_context(|| format!("failed to open `{}`", path.display()))?; |
| |
| let lock_meta = LockMetadata { v: 1 }; |
| write!(ok, "{}", serde_json::to_string(&lock_meta).unwrap())?; |
| |
| self.gctx |
| .deferred_global_last_use()? |
| .mark_registry_src_used(global_cache_tracker::RegistrySrc { |
| encoded_registry_name: self.name, |
| package_dir: package_dir.into(), |
| size: Some(bytes_written), |
| }); |
| |
| Ok(unpack_dir.to_path_buf()) |
| } |
| |
| /// Unpacks the `.crate` tarball of the package in a given directory. |
| /// |
| /// Returns the path to the crate tarball directory, |
| /// which is always `<unpack_dir>/<pkg>-<version>`. |
| /// |
| /// This holds some assumptions |
| /// |
| /// * The associated tarball already exists |
| /// * If this is a local registry, |
| /// the package cache lock must be externally synchronized. |
| /// Cargo does not take care of it being locked or not. |
| pub fn unpack_package_in( |
| &self, |
| pkg: &PackageId, |
| unpack_dir: &Path, |
| include: &dyn Fn(&Path) -> bool, |
| ) -> CargoResult<PathBuf> { |
| let path = self.ops.cache_path().join(pkg.tarball_name()); |
| let path = self.ops.assert_index_locked(&path); |
| let dst = unpack_dir.join(format!("{}-{}", pkg.name(), pkg.version())); |
| let tarball = |
| File::open(path).with_context(|| format!("failed to open {}", path.display()))?; |
| unpack(self.gctx, &tarball, &dst, include)?; |
| update_mtime_for_generated_files(&dst); |
| Ok(dst) |
| } |
| |
| /// Turns the downloaded `.crate` tarball file into a [`Package`]. |
| /// |
| /// This unconditionally sets checksum for the returned package, so it |
| /// should only be called after doing integrity check. That is to say, |
| /// you need to call either [`RegistryData::download`] or |
| /// [`RegistryData::finish_download`] before calling this method. |
| async fn get_pkg(&self, package: PackageId, path: &File) -> CargoResult<Package> { |
| let path = self |
| .unpack_package(package, path) |
| .with_context(|| format!("failed to unpack package `{}`", package))?; |
| let src = PathSource::new(&path, self.source_id, self.gctx); |
| src.load()?; |
| let mut pkg = match src.download(package).await? { |
| MaybePackage::Ready(pkg) => pkg, |
| MaybePackage::Download { .. } => unreachable!(), |
| }; |
| |
| // After we've loaded the package configure its summary's `checksum` |
| // field with the checksum we know for this `PackageId`. |
| let cksum = self |
| .index |
| .hash(package, &*self.ops) |
| .now_or_never() |
| .expect("a downloaded dep now pending!?") |
| .expect("summary not found"); |
| pkg.manifest_mut() |
| .summary_mut() |
| .set_checksum(cksum.to_string()); |
| |
| Ok(pkg) |
| } |
| } |
| |
| #[async_trait::async_trait(?Send)] |
| impl<'gctx> Source for RegistrySource<'gctx> { |
| async fn query( |
| &self, |
| dep: &Dependency, |
| kind: QueryKind, |
| f: &mut dyn FnMut(IndexSummary), |
| ) -> CargoResult<()> { |
| let mut req = dep.version_req().clone(); |
| |
| // Handle `cargo update --precise` here. |
| if let Some((_, requested)) = self |
| .source_id |
| .precise_registry_version(dep.package_name().as_str()) |
| .filter(|(c, to)| { |
| if to.is_prerelease() && self.gctx.cli_unstable().unstable_options { |
| req.matches_prerelease(c) |
| } else { |
| req.matches(c) |
| } |
| }) |
| { |
| req.precise_to(&requested); |
| } |
| |
| let mut called = false; |
| let callback = &mut |s| { |
| called = true; |
| f(s); |
| }; |
| |
| // If this is a locked dependency, then it came from a lock file and in |
| // theory the registry is known to contain this version. If, however, we |
| // come back with no summaries, then our registry may need to be |
| // updated, so we fall back to performing a lazy update. |
| if kind == QueryKind::Exact && req.is_locked() && !self.ops.is_updated() { |
| debug!("attempting query without update"); |
| self.index |
| .query_inner(dep.package_name(), &req, &*self.ops, &mut |is| { |
| match &is { |
| IndexSummary::Candidate(s) | IndexSummary::Yanked(s) if dep.matches(&s) => { |
| // We are looking for a package from a lock file so we do not care about yank |
| callback(is) |
| } |
| _ => {} |
| } |
| }) |
| .await?; |
| if called { |
| return Ok(()); |
| } else { |
| debug!("falling back to an update"); |
| self.invalidate_cache(); |
| } |
| } |
| |
| let mut called = false; |
| let callback = &mut |s| { |
| called = true; |
| f(s); |
| }; |
| |
| let mut precise_yanked_in_use = false; |
| self.index |
| .query_inner(dep.package_name(), &req, &*self.ops, &mut |s| { |
| let matched = match kind { |
| QueryKind::Exact | QueryKind::RejectedVersions => { |
| let s = match &s { |
| IndexSummary::Candidate(s) |
| | IndexSummary::Yanked(s) |
| | IndexSummary::Offline(s) |
| | IndexSummary::Unsupported(s, _) |
| | IndexSummary::Invalid(s) => s, |
| }; |
| if req.is_precise() && self.gctx.cli_unstable().unstable_options { |
| dep.matches_prerelease(&s) |
| } else { |
| dep.matches(&s) |
| } |
| } |
| QueryKind::AlternativeNames => true, |
| QueryKind::Normalized => true, |
| }; |
| if !matched { |
| return; |
| } |
| match s { |
| s @ _ if kind == QueryKind::RejectedVersions => callback(s), |
| s @ IndexSummary::Candidate(_) => callback(s), |
| s @ IndexSummary::Yanked(_) => { |
| // HACK: While source knows nothing about yank policy, |
| // We still detect `cargo update --precise <yanked>` |
| // so we can warn about the user-visible selection. |
| // |
| // We should consider also move this out from source query. |
| if req.is_precise() { |
| precise_yanked_in_use = true; |
| } |
| callback(s); |
| } |
| IndexSummary::Unsupported(summary, v) => { |
| tracing::debug!( |
| "unsupported schema version {} ({} {})", |
| v, |
| summary.name(), |
| summary.version() |
| ); |
| } |
| IndexSummary::Invalid(summary) => { |
| tracing::debug!("invalid ({} {})", summary.name(), summary.version()); |
| } |
| IndexSummary::Offline(summary) => { |
| tracing::debug!("offline ({} {})", summary.name(), summary.version()); |
| } |
| } |
| }) |
| .await?; |
| if precise_yanked_in_use { |
| let name = dep.package_name(); |
| let version = req |
| .precise_version() |
| .expect("--precise <yanked-version> in use"); |
| if self |
| .selected_precise_yanked |
| .borrow_mut() |
| .insert((name, version.clone())) |
| { |
| let mut shell = self.gctx.shell(); |
| shell.print_report( |
| &[Level::WARNING |
| .secondary_title(format!( |
| "selected package `{name}@{version}` was yanked by the author" |
| )) |
| .element( |
| Level::HELP.message("if possible, try a compatible non-yanked version"), |
| )], |
| false, |
| )?; |
| } |
| } |
| if called { |
| return Ok(()); |
| } |
| if kind == QueryKind::AlternativeNames || kind == QueryKind::Normalized { |
| // Attempt to handle misspellings by searching for a chain of related |
| // names to the original name. The resolver will later |
| // reject any candidates that have the wrong name, and with this it'll |
| // have enough information to offer "a similar crate exists" suggestions. |
| // For now we only try canonicalizing `-` to `_` and vice versa. |
| // More advanced fuzzy searching become in the future. |
| for name_permutation in [ |
| dep.package_name().replace('-', "_"), |
| dep.package_name().replace('_', "-"), |
| ] { |
| let name_permutation = name_permutation.into(); |
| if name_permutation == dep.package_name() { |
| continue; |
| } |
| self.index |
| .query_inner(name_permutation, &req, &*self.ops, &mut |s| f(s)) |
| .await?; |
| } |
| } |
| Ok(()) |
| } |
| |
| fn supports_checksums(&self) -> bool { |
| true |
| } |
| |
| fn requires_precise(&self) -> bool { |
| false |
| } |
| |
| fn source_id(&self) -> SourceId { |
| self.source_id |
| } |
| |
| fn invalidate_cache(&self) { |
| self.index.clear_summaries_cache(); |
| self.ops.invalidate_cache(); |
| } |
| |
| fn set_quiet(&mut self, quiet: bool) { |
| self.ops.set_quiet(quiet); |
| } |
| |
| async fn download(&self, package: PackageId) -> CargoResult<MaybePackage> { |
| let hash = self.index.hash(package, &*self.ops).await?; |
| match self.ops.download(package, &hash).await? { |
| MaybeLock::Ready(file) => self.get_pkg(package, &file).await.map(MaybePackage::Ready), |
| MaybeLock::Download { |
| url, |
| descriptor, |
| authorization, |
| } => Ok(MaybePackage::Download { |
| url, |
| descriptor, |
| authorization, |
| }), |
| } |
| } |
| |
| async fn finish_download(&self, package: PackageId, data: Vec<u8>) -> CargoResult<Package> { |
| let hash = self.index.hash(package, &*self.ops).await?; |
| let file = self.ops.finish_download(package, &hash, &data).await?; |
| self.get_pkg(package, &file).await |
| } |
| |
| fn fingerprint(&self, pkg: &Package) -> CargoResult<String> { |
| Ok(pkg.package_id().version().to_string()) |
| } |
| |
| fn describe(&self) -> String { |
| self.source_id.display_index() |
| } |
| } |
| |
| /// Get the maximum unpack size that Cargo permits |
| /// based on a given `size` of your compressed file. |
| /// |
| /// Returns the larger one between `size * max compression ratio` |
| /// and a fixed max unpacked size. |
| /// |
| /// In reality, the compression ratio usually falls in the range of 2:1 to 10:1. |
| /// We choose 20:1 to cover almost all possible cases hopefully. |
| /// Any ratio higher than this is considered as a zip bomb. |
| /// |
| /// In the future we might want to introduce a configurable size. |
| /// |
| /// Some of the real world data from common compression algorithms: |
| /// |
| /// * <https://www.zlib.net/zlib_tech.html> |
| /// * <https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf> |
| /// * <https://blog.cloudflare.com/results-experimenting-brotli/> |
| /// * <https://tukaani.org/lzma/benchmarks.html> |
| fn max_unpack_size(gctx: &GlobalContext, size: u64) -> u64 { |
| const SIZE_VAR: &str = "__CARGO_TEST_MAX_UNPACK_SIZE"; |
| const RATIO_VAR: &str = "__CARGO_TEST_MAX_UNPACK_RATIO"; |
| const MAX_UNPACK_SIZE: u64 = 512 * 1024 * 1024; // 512 MiB |
| const MAX_COMPRESSION_RATIO: usize = 20; // 20:1 |
| |
| let max_unpack_size = if cfg!(debug_assertions) && gctx.get_env(SIZE_VAR).is_ok() { |
| // For integration test only. |
| gctx.get_env(SIZE_VAR) |
| .unwrap() |
| .parse() |
| .expect("a max unpack size in bytes") |
| } else { |
| MAX_UNPACK_SIZE |
| }; |
| let max_compression_ratio = if cfg!(debug_assertions) && gctx.get_env(RATIO_VAR).is_ok() { |
| // For integration test only. |
| gctx.get_env(RATIO_VAR) |
| .unwrap() |
| .parse() |
| .expect("a max compression ratio in bytes") |
| } else { |
| MAX_COMPRESSION_RATIO |
| }; |
| |
| u64::max(max_unpack_size, size * max_compression_ratio as u64) |
| } |
| |
| /// Set the current [`umask`] value for the given tarball. No-op on non-Unix |
| /// platforms. |
| /// |
| /// On Windows, tar only looks at user permissions and tries to set the "read |
| /// only" attribute, so no-op as well. |
| /// |
| /// [`umask`]: https://man7.org/linux/man-pages/man2/umask.2.html |
| #[allow(unused_variables)] |
| fn set_mask<R: Read>(tar: &mut Archive<R>) { |
| #[cfg(unix)] |
| tar.set_mask(crate::util::get_umask()); |
| } |
| |
| /// Unpack a tarball with zip bomb and overwrite protections. |
| fn unpack( |
| gctx: &GlobalContext, |
| tarball: &File, |
| unpack_dir: &Path, |
| include: &dyn Fn(&Path) -> bool, |
| ) -> CargoResult<u64> { |
| let mut tar = { |
| let size_limit = max_unpack_size(gctx, tarball.metadata()?.len()); |
| let gz = GzDecoder::new(tarball); |
| let gz = LimitErrorReader::new(gz, size_limit); |
| let mut tar = Archive::new(gz); |
| set_mask(&mut tar); |
| tar |
| }; |
| let mut bytes_written = 0; |
| let prefix = unpack_dir.file_name().unwrap(); |
| let parent = unpack_dir.parent().unwrap(); |
| for entry in tar.entries()? { |
| let mut entry = entry.context("failed to iterate over archive")?; |
| let entry_path = entry |
| .path() |
| .context("failed to read entry path")? |
| .into_owned(); |
| |
| if let Ok(path) = entry_path.strip_prefix(prefix) { |
| if !include(path) { |
| continue; |
| } |
| } else { |
| // We're going to unpack this tarball into the global source |
| // directory, but we want to make sure that it doesn't accidentally |
| // (or maliciously) overwrite source code from other crates. Cargo |
| // itself should never generate a tarball that hits this error, and |
| // crates.io should also block uploads with these sorts of tarballs, |
| // but be extra sure by adding a check here as well. |
| anyhow::bail!( |
| "invalid tarball downloaded, contains \ |
| a file at {entry_path:?} which isn't under {prefix:?}", |
| ) |
| } |
| |
| // Prevent unpacking symlinks and other unexpected entry types |
| match entry.header().entry_type() { |
| EntryType::Regular | EntryType::Directory => {} |
| t => anyhow::bail!( |
| "invalid tarball downloaded, contains an entry at {entry_path:?} with invalid type {t:?}", |
| ), |
| } |
| |
| // Prevent unpacking the lockfile from the crate itself. |
| if entry_path |
| .file_name() |
| .map_or(false, |p| p == PACKAGE_SOURCE_LOCK) |
| { |
| continue; |
| } |
| // Unpacking failed |
| bytes_written += entry.size(); |
| let mut result = entry.unpack_in(parent).map_err(anyhow::Error::from); |
| if cfg!(windows) && restricted_names::is_windows_reserved_path(&entry_path) { |
| result = result.with_context(|| { |
| format!( |
| "`{}` appears to contain a reserved Windows path, \ |
| it cannot be extracted on Windows", |
| entry_path.display() |
| ) |
| }); |
| } |
| result.with_context(|| format!("failed to unpack entry at `{}`", entry_path.display()))?; |
| } |
| |
| Ok(bytes_written) |
| } |
| |
| /// Workaround for rust-lang/cargo#16237 |
| /// |
| /// Generated files should have the same deterministic mtime as other files. |
| /// However, since we forgot to set mtime for those files when uploading, they |
| /// always have older mtime (1973-11-29) that prevents zip from packing (requiring >1980) |
| /// |
| /// This workaround updates mtime after we unpack the tarball at the destination. |
| fn update_mtime_for_generated_files(pkg_root: &Path) { |
| const GENERATED_FILES: &[&str] = &["Cargo.lock", "Cargo.toml", ".cargo_vcs_info.json"]; |
| // Hardcoded value be removed once alexcrichton/tar-rs#420 is merged and released. |
| // See also rust-lang/cargo#16237 |
| const DETERMINISTIC_TIMESTAMP: i64 = 1153704088; |
| |
| for file in GENERATED_FILES { |
| let path = pkg_root.join(file); |
| let mtime = filetime::FileTime::from_unix_time(DETERMINISTIC_TIMESTAMP, 0); |
| if let Err(e) = filetime::set_file_mtime(&path, mtime) { |
| tracing::trace!("failed to set deterministic mtime for {path:?}: {e}"); |
| } |
| } |
| } |
