aboutsummaryrefslogtreecommitdiff
path: root/tests/substitute.scm
AgeCommit message (Expand)Author
2023-12-04substitute: Do not exit when failing to find a nar....Fixes <https://issues.guix.gnu.org/67575>. * guix/scripts/substitute.scm (process-substitution/fallback): Use ‘report-error’ instead of ‘leave’. Write status line to PORT. * tests/substitute.scm ("substitute, narinfo is available but nar is missing"): Adjust accordingly. Change-Id: Ic7297dbd563c007111ec2167c8d52505a07d4822 Ludovic Courtès
2023-06-08substitute: Gracefully retry after failed partial downloads....Fixes <https://issues.guix.gnu.org/63443>. Reported by Attila Lendvai <attila@lendvai.name>. * guix/scripts/substitute.scm (catch-system-error): New macro. (download-nar): Add call to 'delete-file-recursively'. * tests/substitute.scm ("substitute, previous partial download around"): New test. Ludovic Courtès
2023-05-30substitute: If a server's nar URL is 404, try the next one(s)....If a substitute server advertises in its narinfo, for example, both a /zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip URL. Fixes <https://issues.guix.gnu.org/63634>. * guix/narinfo.scm (narinfo-preferred-uris): New procedure. (narinfo-best-uri): Rebase on top of it. * guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure. Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of NARINFO. * tests/substitute.scm (request-substitution): Remove 'parameterize'. Delete DESTINATION. ("substitute, preferred nar URL is 404, other is 200"): New test. Ludovic Courtès
2022-09-28substitute: Retry downloading when a nar is unavailable....Fixes <https://issues.guix.gnu.org/57978> Reported by Attila Lendvai <attila@lendvai.name>. Previously, if a narinfo was available but its corresponding nar was missing (for instance because the narinfo was cached and the server became unreachable in the meantime), 'guix substitute --substitute' would try to download the nar from its preferred location and abort when that fails. This change forces one retry with each of the URLs. * guix/scripts/substitute.scm (download-nar): Do not catch 'http-get-error?' exceptions. (system-error?, network-error?, process-substitution/fallback): New procedures. (process-substitution): Call 'process-substitution/fallback' upon 'network-error?'. * tests/substitute.scm ("substitute, first URL has narinfo but lacks nar, second URL unauthorized") ("substitute, first URL has narinfo but nar is 404, both URLs authorized") ("substitute, first URL has narinfo but nar is 404, one URL authorized") ("substitute, narinfo is available but nar is missing"): New tests. Ludovic Courtès
2022-09-24substitute: Test behavior with unroutable substitute server addresses....* tests/substitute.scm (%unroutable-substitute-url): New variable. ("query narinfo signed with authorized key, unroutable URL first") ("substitute, authorized key, first substitute URL is unroutable"): New tests. Ludovic Courtès
2022-02-14publish: Do not sign the URL/Compression/FileSize narinfo fields....This will allow mirror operators to alter these non-normative bits of a narinfo without having to resign narinfos. * guix/scripts/publish.scm (narinfo-string): Remove URL/Compression/FileSize from BASE-INFO. Move them after "Signature". * tests/publish.scm ("/*.narinfo") ("/*.narinfo with properly encoded '+' sign") ("/*.narinfo with lzip + gzip") ("with cache, lzip + gzip"): Adjust accordingly. * tests/substitute.scm ("query narinfo with signature over relevant subset"): New test. Ludovic Courtès
2021-04-09daemon: 'guix substitute' replies on FD 4....This avoids the situation where error messages would unintentionally go to stderr and be wrongfully interpreted as a reply by the daemon. Fixes <https://bugs.gnu.org/46362>. This is a followup to ee3226e9d54891c7e696912245e4904435be191c. * guix/scripts/substitute.scm (display-narinfo-data): Add 'port' parameter and honor it. (process-query): Likewise. (process-substitution): Likewise. (%error-to-file-descriptor-4?, with-redirected-error-port): Remove. (%reply-file-descriptor): New variable. (guix-substitute): Remove use of 'with-redirected-error-port'. Define 'reply-port' and pass it to 'process-query' and 'process-substitution'. * nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap 'builderOut' and 'fromAgent'. * nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter): Likewise. * tests/substitute.scm <top level>: Set '%reply-file-descriptor' rather than '%error-to-file-descriptor-4?'. Ludovic Courtès
2021-01-16guix: Move narinfo code from substitute script to module....This separation between the code for dealing with narinfos from the code doing that for a purpose should make things clearer, and better support components other that the substitute script in using this code. This is just moving the code around, no code should have been significantly changed. * guix/scripts/substitute.scm (<narinfo>): Move record type to (guix narinfo). (fields->alist, narinfo-hash-algorithm+value, narinfo-hash->sha256, narinfo-signature->canonical-sexp, narinfo-maker, read-narinfo, narinfo-sha256, valid-narinfo?, write-narinfo, narinfo->string, string->narinfo, equivalent-narinfo?, supported-compression?, compresses-better?, narinfo-best-uri): Move procedures to (guix narinfo). (%compression-methods): Move variable to (guix narinfo). * guix/narinfo.scm: New file. * Makefile.am (MODULES): Add it. * po/guix/POTFILES.in: Add 'guix/narinfo.scm'. Christopher Baines
2020-12-19tests: Check the mtime and permissions of substituted items....* tests/store.scm ("substitute") ("substitute + build-things with output path") ("substitute + build-things with specific output"): Call 'canonical-file?'. * tests/substitute.scm ("substitute, authorized key"): Check the mtime and permissions of "substitute-retrieved". Ludovic Courtès
2020-12-19daemon: Let 'guix substitute' perform hash checks....This way, the hash of the store item can be computed as it is restored, thereby avoiding an additional file tree traversal ('hashPath' call) later on in the daemon. Consequently, it should reduce latency between subsequent substitute downloads. This is a followup to 5ff521452b9ec2aae9ed8e4bb7bdc250a581f203. * guix/scripts/substitute.scm (narinfo-hash-algorithm+value): New procedure. (process-substitution): Wrap INPUT into a hash input port, 'hashed', and read from it. Compare the actual and expected hashes, and print a "hash-mismatch" status line when they differ. When they match, print not just "success" but also the nar hash and size. * nix/libstore/build.cc (class SubstitutionGoal)[expectedHashStr]: Remove. (SubstitutionGoal::finished): Tokenize 'status'. Parse it and handle "success" and "hash-mismatch" accordingly. Call 'hashPath' only when the returned hash is not SHA256. (SubstitutionGoal::handleChildOutput): Remove 'expectedHashStr' handling. * tests/substitute.scm ("substitute, invalid hash"): Rename to... ("substitute, invalid narinfo hash"): ... this. ("substitute, invalid hash"): New test. Ludovic Courtès
2020-12-08daemon: Run 'guix substitute --substitute' as an agent....This avoids spawning one substitute process per substitution. * nix/libstore/build.cc (class Worker)[substituter]: New field. [outPipe, logPipe, pid]: Remove. (class SubstitutionGoal)[expectedHashStr, status, substituter]: New fields. (SubstitutionGoal::timedOut): Adjust to check 'substituter'. (SubstitutionGoal::tryToRun): Remove references to 'outPipe' and 'logPipe'. Run "guix substitute --substitute" as an 'Agent'. Send the request with 'writeLine'. (SubstitutionGoal::finished): Likewise. (SubstitutionGoal::handleChildOutput): Change to fill in 'expectedHashStr' and 'status'. (SubstitutionGoal::handleEOF): Call 'wakeUp' unconditionally. (SubstitutionGoal::~SubstitutionGoal): Adjust to check 'substituter'. * guix/scripts/substitute.scm (process-substitution): Write "success\n" to stdout upon success. (%error-to-file-descriptor-4?): New variable. (guix-substitute): Set 'current-error-port' to file descriptor 4 unless (%error-to-file-descriptor-4?) is false. Remove "--substitute" arguments. Loop reading line from stdin. * tests/substitute.scm <top level>: Call '%error-to-file-descriptor-4?'. (request-substitution): New procedure. ("substitute, no signature") ("substitute, invalid hash") ("substitute, unauthorized key") ("substitute, authorized key") ("substitute, unauthorized narinfo comes first") ("substitute, unsigned narinfo comes first") ("substitute, first narinfo is unsigned and has wrong hash") ("substitute, first narinfo is unsigned and has wrong refs") ("substitute, two invalid narinfos") ("substitute, narinfo with several URLs"): Adjust to new "guix substitute --substitute" calling convention. Ludovic Courtès
2020-12-08daemon: Use 'Agent' to spawn 'guix substitute --query'....* nix/libstore/local-store.hh (RunningSubstituter): Remove. (LocalStore)[runningSubstituter]: Change to unique_ptr<Agent>. [setSubstituterEnv, didSetSubstituterEnv]: Remove. [getLineFromSubstituter, getIntLineFromSubstituter]: Take an 'Agent'. * nix/libstore/local-store.cc (LocalStore::~LocalStore): Remove reference to 'runningSubstituter'. (LocalStore::setSubstituterEnv, LocalStore::startSubstituter): Remove. (LocalStore::getLineFromSubstituter): Adjust to 'run' being an 'Agent'. (LocalStore::querySubstitutablePaths): Spawn substituter agent if needed. Adjust to 'Agent' interface. (LocalStore::querySubstitutablePathInfos): Likewise. * nix/libstore/build.cc (SubstitutionGoal::tryToRun): Remove call to 'setSubstituterEnv' and add 'setenv' call for "_NIX_OPTIONS" instead. (SubstitutionGoal::finished): Remove 'readLine' call for 'dummy'. * guix/scripts/substitute.scm (%allow-unauthenticated-substitutes?): Remove second argument to 'make-parameter'. (process-query): Call 'warn-about-missing-authentication' when (%allow-unauthenticated-substitutes?) is #t. (guix-substitute): Wrap body in 'parameterize'. Set 'guix-warning-port' too. No longer exit when 'substitute-urls' returns the empty list. No longer print newline initially. * tests/substitute.scm (test-quit): Parameterize 'current-error-port' to account for the port changes in 'guix-substitute'. Ludovic Courtès
2020-08-24Use "guile-zlib" and "guile-lzlib" instead of (guix config)....* Makefile.am (MODULES): Remove guix/zlib.scm and guix/lzlib.scm, (SCM_TESTS): remove tests/zlib.scm, tests/lzlib.scm. * build-aux/build-self.scm (make-config.scm): Remove unused %libz variable. * configure.ac: Remove LIBZ and LIBLZ variables and check instead for Guile-zlib and Guile-lzlib. * doc/guix.texi ("Requirements"): Remove zlib requirement and add Guile-zlib and Guile-lzlib instead. * gnu/packages/package-management.scm (guix)[native-inputs]: Add "guile-zlib" and "guile-lzlib", [inputs]: remove "zlib" and "lzlib", [propagated-inputs]: ditto, [arguments]: add "guile-zlib" and "guile-lzlib" to Guile load path. * guix/config.scm.in (%libz, %liblz): Remove them. * guix/lzlib.scm: Remove it. * guix/man-db.scm: Use (zlib) instead of (guix zlib). * guix/profiles.scm (manual-database): Do not stub (guix config) in imported modules list, instead add "guile-zlib" to the extension list. * guix/scripts/publish.scm: Use (zlib) instead of (guix zlib) and (lzlib) instead of (guix lzlib), (string->compression-type, effective-compression): do not check for zlib and lzlib availability. * guix/scripts/substitute.scm (%compression-methods): Do not check for lzlib availability. * guix/self.scm (specification->package): Add "guile-zlib" and "guile-lzlib" and remove "zlib" and "lzlib", (compiled-guix): remove "zlib" and "lzlib" arguments and add guile-zlib and guile-lzlib to the dependencies, also do not pass "zlib" and "lzlib" to "make-config.scm" procedure, (make-config.scm): remove "zlib" and "lzlib" arguments as well as %libz and %liblz variables. * guix/utils.scm (lzip-port): Use (lzlib) instead of (guix lzlib) and do not check for lzlib availability. * guix/zlib.scm: Remove it. * m4/guix.m4 (GUIX_LIBZ_LIBDIR, GUIX_LIBLZ_FILE_NAME): Remove them. * tests/lzlib.scm: Use (zlib) instead of (guix zlib) and (lzlib) instead of (guix lzlib), and do not check for zlib and lzlib availability. * tests/publish.scm: Ditto. * tests/substitute.scm: Do not check for lzlib availability. * tests/utils.scm: Ditto. * tests/zlib.scm: Remove it. Mathieu Othacehe
2019-11-27substitute: Make '%allow-unauthenticated-substitutes?' public....* guix/scripts/substitute.scm (warn-about-missing-authentication): New procedure. (%allow-unauthenticated-substitutes?): Turn into a public parameter and use 'warn-about-missing-authentication'. (valid-narinfo?): Adjust accordingly. * tests/substitute.scm (call-with-narinfo): Likewise. Ludovic Courtès
2019-06-02substitute: Select the best compression methods....When a server publishes several URLs with different compression methods, 'guix substitute' can now choose the best one among the compression methods that it supports. * guix/scripts/substitute.scm (<narinfo>)[uri]: Replace with... [uris]: ... this. [compression]: Replace with... [compressions]: ... this. [file-size]: Replace with... [file-sizes]: ... this. [file-hash]: Replace with... [file-hashes]: ... this. (narinfo-maker): Adjust accordingly. Ensure 'file-sizes' and 'file-hashes' have the right length. (assert-valid-signature, valid-narinfo?): Use the first element of 'narinfo-uris' in error messages. (read-narinfo): Expect "URL", "Compression", "FileSize", and "FileHash" to occur multiple times. (display-narinfo-data): Call 'select-uri' to determine the file size. (%compression-methods): New variable. (supported-compression?, compresses-better?, select-uri): New procedures. (process-substitution): Call 'select-uri' to select the URI and compression. * guix/scripts/weather.scm (report-server-coverage): Account for all the values returned by 'narinfo-file-sizes'. * tests/substitute.scm ("substitute, narinfo with several URLs"): New test. Ludovic Courtès
2018-12-14substitute: Ignore irrelevant narinfo signatures....Fixes <https://bugs.gnu.org/33733>. Fixes a bug whereby 'guix substitute' would accept narinfos whose signature does not cover the StorePath/NarHash/References tuple. * guix/scripts/substitute.scm (narinfo-sha256)[%mandatory-fields]: New variable. Compute SIGNED-FIELDS; return #f unless each of the %MANDATORY-FIELDS is among SIGNED-FIELDS. * tests/substitute.scm ("query narinfo with signature over nothing") ("query narinfo with signature over irrelevant bits"): New tests. Ludovic Courtès
2018-09-04Switch to Guile-Gcrypt....This removes (guix hash) and (guix pk-crypto), which now live as part of Guile-Gcrypt (version 0.1.0.) * guix/gcrypt.scm, guix/hash.scm, guix/pk-crypto.scm, tests/hash.scm, tests/pk-crypto.scm: Remove. * configure.ac: Test for Guile-Gcrypt. Remove LIBGCRYPT and LIBGCRYPT_LIBDIR assignments. * m4/guix.m4 (GUIX_ASSERT_LIBGCRYPT_USABLE): Remove. * README: Add Guile-Gcrypt to the dependencies; move libgcrypt as "required unless --disable-daemon". * doc/guix.texi (Requirements): Likewise. * gnu/packages/bash.scm, guix/derivations.scm, guix/docker.scm, guix/git.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/cran.scm, guix/import/crate.scm, guix/import/elpa.scm, guix/import/gnu.scm, guix/import/hackage.scm, guix/import/texlive.scm, guix/import/utils.scm, guix/nar.scm, guix/pki.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/download.scm, guix/scripts/hash.scm, guix/scripts/pack.scm, guix/scripts/publish.scm, guix/scripts/refresh.scm, guix/scripts/substitute.scm, guix/store.scm, guix/store/deduplication.scm, guix/tests.scm, tests/base32.scm, tests/builders.scm, tests/challenge.scm, tests/cpan.scm, tests/crate.scm, tests/derivations.scm, tests/gem.scm, tests/nar.scm, tests/opam.scm, tests/pki.scm, tests/publish.scm, tests/pypi.scm, tests/store-deduplication.scm, tests/store.scm, tests/substitute.scm: Adjust imports. * gnu/system/vm.scm: Likewise. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (expression->derivation-in-linux-vm)[config]: Remove. (iso9660-image)[config]: Remove. (qemu-image)[config]: Remove. (system-docker-image)[config]: Remove. * guix/scripts/pack.scm: Adjust imports. (guile-sqlite3&co): Rename to... (gcrypt-sqlite3&co): ... this. Add GUILE-GCRYPT. (self-contained-tarball)[build]: Call 'make-config.scm' without #:libgcrypt argument. (squashfs-image)[libgcrypt]: Remove. [build]: Call 'make-config.scm' without #:libgcrypt. (docker-image)[config, json]: Remove. [build]: Add GUILE-GCRYPT to the extensions Remove (guix config) from the imported modules. * guix/self.scm (specification->package): Remove "libgcrypt", add "guile-gcrypt". (compiled-guix): Remove #:libgcrypt. [guile-gcrypt]: New variable. [dependencies]: Add it. [*core-modules*]: Remove #:libgcrypt from 'make-config.scm' call. Add #:extensions. [*config*]: Remove #:libgcrypt from 'make-config.scm' call. (%dependency-variables): Remove %libgcrypt. (make-config.scm): Remove #:libgcrypt. * build-aux/build-self.scm (guile-gcrypt): New variable. (make-config.scm): Remove #:libgcrypt. (build-program)[fake-gcrypt-hash]: New variable. Add (gcrypt hash) to the imported modules. Adjust load path assignments. * gnu/packages/package-management.scm (guix)[propagated-inputs]: Add GUILE-GCRYPT. [arguments]: In 'wrap-program' phase, add GUILE-GCRYPT to the search path. Ludovic Courtès
2017-09-11substitute: Download from unauthorized sources that provide the right content....This allows substitutes to be downloaded from unauthorized servers, as long as they advertise the same hash and references as one of the authorized servers. * guix/scripts/substitute.scm (assert-valid-narinfo): Remove. (valid-narinfo?): Add #:verbose?. Handle each case of 'signature-case'. (equivalent-narinfo?): New procedure. (lookup-narinfos/diverse): Add 'authorized?' parameter and honor it. [select-hit]: New procedure. (lookup-narinfo): Add 'authorized?' parameter and pass it. (process-query): Adjust callers accordingly. (process-substitution): Remove call to 'assert-valid-narinfo'. Check whether 'lookup-narinfo' returns true and call 'leave' if not. * tests/substitute.scm (%main-substitute-directory) (%alternate-substitute-directory): New variables. (call-with-narinfo): Make 'narinfo-directory' a parameter. Call 'mkdir-p' to create it. Change unwind handler to check whether CACHE-DIRECTORY exists before deleting it. (with-narinfo*): New macro. ("substitute, no signature") ("substitute, invalid hash") ("substitute, unauthorized key"): Change expected error message to "no valid substitute". ("substitute, unauthorized narinfo comes first") ("substitute, unsigned narinfo comes first") ("substitute, first narinfo is unsigned and has wrong hash") ("substitute, first narinfo is unsigned and has wrong refs") ("substitute, unsigned narinfo comes first") ("substitute, two invalid narinfos"): New tests. * doc/guix.texi (Substitutes): Explain the new behavior. Ludovic Courtès
2017-09-11substitute: Make substitute URLs a SRFI-39 parameter....* guix/scripts/substitute.scm (%cache-urls): Rename to... (%default-substitute-urls): ... this. (substitute-urls): New variable. (guix-substitute): Use it instead of %CACHE-URLS. * tests/substitute.scm: Likewise. Ludovic Courtès
2016-04-03build: Add a Guile custom test driver using SRFI-64....Before that '.log' files for scheme tests were fragmented and not included in test-suite.log. This unifies the semantics of SRFI-64 API with Automake test suite. * build-aux/test-driver.scm: New file. * Makefile.am (SCM_LOG_DRIVER, AM_SCM_LOG_DRIVER_FLAGS): New variables. (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables. (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0. * test-env.in: Silence guix-daemon. * doc/guix.texi (Running the Test Suite): Describe how to display the detailed results. Bug reports require only 'test-suite.log' file. * tests/base32.scm, tests/build-utils.scm, tests/builders.scm, tests/challenge.scm, tests/cpan.scm, tests/cpio.scm, tests/cran.scm, tests/cve.scm, tests/derivations.scm, tests/elpa.scm, tests/file-systems.scm, tests/gem.scm, tests/gexp.scm, tests/gnu-maintenance.scm, tests/grafts.scm, tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm, tests/import-utils.scm, tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm, tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm, tests/publish.scm, tests/pypi.scm, tests/records.scm, tests/scripts-build.scm, tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm, tests/snix.scm, tests/store.scm, tests/substitute.scm, tests/syscalls.scm, tests/system.scm, tests/ui.scm, tests/union.scm, tests/upstream.scm, tests/utils.scm: Don't exit at the end of test groups. * tests/containers.scm: Likewise. Use 'test-skip' instead of exiting with error code 77. Mathieu Lirzin
2015-10-28substitute: Honor all the specified server URLs....* guix/scripts/substitute.scm (lookup-narinfos/diverse): New procedure. (lookup-narinfo): Use it. (process-query): Change #:cache-url to #:cache-urls. [valid?]: Remove 'narinfo?' check, which is no longer necessary. Use 'lookup-narinfos/diverse' instead of 'lookup-narinfos'. (process-substitution): Change #:cache-url to #:cache-urls. (%cache-url): Rename to... (%cache-urls): ... this. Turn into a list. (guix-substitute): Remove 'getaddrinfo' test with early exit. Adjust calls to 'process-query' and 'process-substitution'. * tests/substitute.scm: Change '%cache-url' to '%cache-urls'. Ludovic Courtès
2015-03-25substitute: Rename cache directory from "substitute-binary" to "substitute"....* guix/scripts/substitute.scm (%narinfo-cache-directory): Change "substitute-binary" to "substitute". * tests/store.scm ("substitute query"): Likewise. * tests/substitute.scm (call-with-narinfo): Likewise. Ludovic Courtès
2015-03-25Rename 'guix substitute-binary' to 'guix substitute'....* guix/scripts/substitute-binary.scm: Rename to... * guix/scripts/substitute.scm: ... this. Adjust module name, entry point, comments, and help string accordingly. * nix/scripts/substitute-binary.in: Rename to... * nix/scripts/substitute.in: ... this. * pre-inst-env.in (NIX_SUBSTITUTERS): Adjust accordingly. * tests/substitute-binary.scm: Rename to... * tests/substitute.scm: ... this. Adjust references to (guix scripts substitute) accordingly. * guix/ui.scm (show-guix-help)[internal?]: Change "substitute-binary" to "substitute". * Makefile.am (MODULES, SCM_TESTS): Adjust to file renames. * daemon.am (nodist_pkglibexec_SCRIPTS): Likewise. * config-daemon.ac: Likewise. * guix/tests.scm (call-with-derivation-narinfo): Adjust comments and docstring. Ludovic Courtès