aboutsummaryrefslogtreecommitdiff


 
                         
generated by
                             
                                 cgit
                             
                         

                        
 


lass='left'>Commit message (Expand)Author
2021-11-16daemon: Do not deduplicate files smaller than 8 KiB....Files smaller than 8 KiB typically represent ~70% of the entries in
/gnu/store/.links but only contribute to ~4% of the space savings
afforded by deduplication.

Not considering these files for deduplication speeds up file insertion
in the store and, more importantly, leaves 'removeUnusedLinks' with
fewer entries to traverse, thereby speeding it up proportionally.

Partly fixes <https://issues.guix.gnu.org/24937>.

* config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK
definition.
* guix/store/deduplication.scm (%deduplication-minimum-size): New
variable.
(deduplicate)[loop]: Do not recurse when FILE's size is below
%DEDUPLICATION-MINIMUM-SIZE.
(dump-port): New procedure.
(dump-file/deduplicate)[hash]: Turn into...
[dump-and-compute-hash]: ... this thunk.
Call 'deduplicate' only when SIZE is greater than
%DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where
st.st_size < deduplicationMinSize.
* nix/libstore/local-store.hh (deduplicationMinSize): New declaration.
* nix/libstore/optimise-store.cc (deduplicationMinSize): New variable.
(LocalStore::optimisePath_): Return when PATH is a symlink or smaller
than 'deduplicationMinSize'.
* tests/derivations.scm ("identical files are deduplicated"): Produce
files bigger than %DEDUPLICATION-MINIMUM-SIZE.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
Likewise.
* tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"):
New test.
("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than
%DEDUPLICATION-MINIMUM-SIZE.
* tests/store.scm ("substitute, deduplication"): Likewise.
Ludovic Courtès
2021-06-18Start enabling substitutes from bordeaux.guix.gnu.org....In addition to substitutes from ci.guix.gnu.org.  There are more changes that
can be made in the future, but these changes seem like a good start.

* config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org.
* guix/scripts/substitute.scm (%default-substitute-urls): Add
http://bordeaux.guix.gnu.org.
* guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org.
* doc/guix.texi: Adjust accordingly.
* doc/contributing.texi: Adjust accordingly.
Christopher Baines
2020-12-11maint: Remove unused 'NIX_VERSION' macro....* config-daemon.ac: Do not define NIX_VERSION.
Ludovic Courtès
2020-12-11maint: Avoid macros obsolete in Autoconf 2.70....* configure.ac: Require Autoconf 2.69.  Use 'AS_HELP_STRING' instead of
'AC_HELP_STRING'.
* m4/guix.m4: Likewise.
* config-daemon.ac: Use 'AC_CONFIG_HEADERS' instead of the singular
variant.
Ludovic Courtès
2019-11-27daemon: GC remove-unused-links phase uses 'statx' when available....* config-daemon.ac: Check for 'statx'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks) [HAVE_STATX]: Use
'statx' instead of 'lstat'.
Ludovic Courtès
2019-11-13daemon: Don't include <linux/fs.h>....As of GNU libc 2.29, <sys/mount.h> declares all the constants and
functions we need, so there's no use in including <linux/fs.h> anymore.
This silences annoying warnings like this one:

  In file included from nix/libstore/local-store.cc:32:0:
  /gnu/store/…-linux-libre-headers-4.19.56/include/linux/fs.h:108:0: warning: "MS_RDONLY" redefined
   #define MS_RDONLY  1 /* Mount read-only */

  In file included from nix/libstore/local-store.cc:28:0:
  /gnu/store/…-glibc-2.29/include/sys/mount.h:36:0: note: this is the location of the previous definition
   #define MS_RDONLY MS_RDONLY

* config-daemon.ac: Remove check for <linux/fs.h>.
* nix/libstore/build.cc: Remove conditional inclusion of <linux/fs.h>.
* nix/libstore/local-store.cc: Remove "#if HAVE_LINUX_FS_H" and
inclusion of <linux/fs.h>.
Ludovic Courtès
2019-09-08daemon: Run 'guix substitute' directly and assume a single substituter....The daemon had a mechanism that allows it to handle a list of
substituters and try them sequentially; this removes it.

* nix/scripts/substitute.in: Remove.
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove.
* config-daemon.ac: Don't output 'nix/scripts/substitute'.
* nix/libstore/build.cc (SubstitutionGoal)[subs, sub, hasSubstitute]:
Remove.
[tryNext]: Make private.
(SubstitutionGoal::SubstitutionGoal, SubstitutionGoal::init): Remove now
unneeded initializers.
(SubstitutionGoal::tryNext): Adjust to assume a single substituter: call
'amDone' upfront when we couldn't find substitutes.
(SubstitutionGoal::tryToRun): Adjust to run 'guix substitute' via
'settings.guixProgram'.
(SubstitutionGoal::finished): Call 'amDone(ecFailed)' upon failure
instead of setting 'state' to 'tryNext'.
* nix/libstore/globals.hh (Settings)[substituters]: Remove.
* nix/libstore/local-store.cc (LocalStore::~LocalStore): Adjust to
handle a single substituter.
(LocalStore::startSubstituter): Remove 'path' parameter.  Adjust to
invoke 'settings.guixProgram'.  Don't refer to 'run.program', which no
longer exists.
(LocalStore::querySubstitutablePaths): Adjust for 'runningSubstituters'
being a singleton instead of a list.
(LocalStore::querySubstitutablePathInfos): Likewise, and remove
'substituter' parameter.
* nix/libstore/local-store.hh (RunningSubstituter)[program]: Remove.
(LocalStore)[runningSubstituters]: Remove.
[runningSubstituter]: New field.
[querySubstitutablePathInfos]: Remove 'substituter' parameter.
[startSubstituter]: Remove 'substituter' parameter.
* nix/nix-daemon/guix-daemon.cc (main): Remove references to
'settings.substituters'.
* nix/nix-daemon/nix-daemon.cc (performOp): Ignore the user's
"build-use-substitutes" value when 'settings.useSubstitutes' is false.
Ludovic Courtès
2019-09-08daemon: Run 'guix offload' directly....* nix/scripts/offload.in: Remove.
* nix/local.mk (nodist_pkglibexec_SCRIPTS) [BUILD_DAEMON_OFFLOAD]:
Remove 'scripts/offload'.
* config-daemon.ac: Don't output 'nix/scripts/offload'.
* build-aux/pre-inst-env.in: Don't set 'NIX_BUILD_HOOK'.
* nix/libstore/build.cc (HookInstance::HookInstance): Run 'guix
offload'.
(DerivationGoal::tryBuildHook): Remove reference to 'NIX_BUILD_HOOK'.
* nix/nix-daemon/guix-daemon.cc (main) [HAVE_DAEMON_OFFLOAD_HOOK]: Don't
set 'NIX_BUILD_HOOK'.
* nix/nix-daemon/nix-daemon.cc (performOp) [!HAVE_DAEMON_OFFLOAD_HOOK]:
Leave 'settings.useBuildHook' unchanged.
Ludovic Courtès
2019-09-08daemon: Run 'guix perform-download' directly....* nix/scripts/download.in: Remove.
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove 'scripts/download'.
* config-daemon.ac: Don't output 'nix/scripts/download'.
* nix/libstore/builtins.cc (builtinDownload): Invoke 'guix
perform-download' directly.
Ludovic Courtès
2019-09-08daemon: Run 'guix authenticate' directly....* nix/scripts/authenticate.in: Remove.
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove
scripts/authenticate.
* config-daemon.ac: Don't output 'nix/scripts/authenticate'.
* nix/libstore/local-store.cc (runAuthenticationProgram): Run 'guix
authenticate'.
Ludovic Courtès
2019-09-08daemon: Invoke 'guix gc --list-busy' instead of 'list-runtime-roots'....* nix/scripts/list-runtime-roots.in: Remove.
* guix/store/roots.scm (%proc-directory): New variable.
(proc-file-roots, proc-exe-roots, proc-cwd-roots)
(proc-fd-roots, proc-maps-roots, proc-environ-roots)
(referenced-files, canonicalize-store-item, busy-store-items): New
procedures, taken from 'list-runtime-roots.in'.
* nix/libstore/globals.hh (Settings)[guixProgram]: New field.
* nix/libstore/globals.cc (Settings::processEnvironment): Initialize
'guixProgram'.
* nix/libstore/gc.cc (addAdditionalRoots): Drop code related to
'NIX_ROOT_FINDER'.  Run "guix gc --list-busy".
* nix/local.mk (nodist_pkglibexec_SCRIPTS): Remove
'scripts/list-runtime-roots'.
* config-daemon.ac: Don't output nix/scripts/list-runtime-roots.
* build-aux/pre-inst-env.in: Don't set 'NIX_ROOT_FINDER'.
Set 'GUIX'.
* doc/guix.texi (Invoking guix gc): Document '--list-busy'.
* guix/scripts/gc.scm (show-help, %options): Add "--list-busy".
(guix-gc)[list-busy]: New procedure.
Handle the 'list-busy' action.
Ludovic Courtès
2019-05-01build: Change default substitute server to "ci.guix.gnu.org"....* config-daemon.ac: Replace "ci.guix.info" with "ci.guix.gnu.org".
* doc/guix.texi (SUBSTITUTE-SERVER): Likewise.
* etc/substitutes/ci.guix.gnu.org.pub: New file.
* Makefile.am (dist_pkgdata_DATA): Add it.
* guix/scripts/build.scm (%default-log-urls): Update.
* guix/scripts/substitute.scm (%default-substitute-urls): Likewise.
* guix/store.scm (%default-substitute-urls): Likewise.
* guix/self.scm (miscellaneous-files): Add "ci.guix.gnu.org".
Ludovic Courtès
2018-12-04build: Default to https://ci.guix.info for substitutes....* config-daemon.ac (guix_substitute_urls): Always default to
"https://ci.guix.info".
* doc/guix.texi (SUBSTITUTE-SERVER): Switch to ci.guix.info.
* guix/scripts/build.scm (%default-log-urls): Likewise.
* guix/scripts/substitute.scm (%default-substitute-urls): Likewise.
* guix/store.scm (%default-substitute-urls): Likewise.
Ludovic Courtès
2018-11-14daemon: Install 'authenticate' script under LIBEXECDIR/guix....That way it is handled in the same way as other helper scripts.

* nix/scripts/guix-authenticate.in: Rename to...
* nix/scripts/authenticate.in: ... this.
* config-daemon.ac: Adjust accordingly.
* nix/local.mk (libstore_a_CPPFLAGS): Remove -DOPENSSL_PATH.
(nodist_libexec_SCRIPTS): Remove.
(nodist_pkglibexec_SCRIPTS): New variable.
* nix/nix-daemon/guix-daemon.cc (main): Remove 'setenv' call for
"PATH".
* nix/libstore/local-store.cc (runAuthenticationProgram): New function.
(LocalStore::exportPath, LocalStore::importPath): Use it instead of
'runProgram' and OPENSSL_PATH.
Ludovic Courtès
2018-09-06build: Remove -L flag when $LIBGCRYPT_LIBDIR is empty....Reported by Alex Vong <alexvong1995@gmail.com>.

* config-daemon.ac: Do not add "-L$LIBGCRYPT_LIBDIR" to LIBGCRYPT_LIBS
when "$LIBGCRYPT_LIBDIR" is empty.
Ludovic Courtès
2018-03-05build: Default to berlin.guixsd.org substitutes on aarch64....Suggested by Efraim Flashner <efraim@flashner.co.il>.

* config-daemon.ac: Set 'guix_substitute_urls' to berlin.guixsd.org on
aarch64.
Ludovic Courtès
2018-03-05build: Always use https substitute URLs....* config-daemon.ac: Remove GUILE_MODULE_AVAILABLE check for (gnutls).
Ludovic Courtès
2018-02-27build: Really build 'guix offload' when possible....This fixes a regression introduced in
1d84d7bf6052c0c80bd212d4524876576e9817d4, whereby
HAVE_DAEMON_OFFLOAD_HOOK would never be defined.

* config-daemon.ac: Do not check for $ac_cv_guix_cbips_support_setvbuf,
which no longer exists.
Ludovic Courtès
2018-02-26build: Require Guile >= 2.0.13....* README, configure.ac, doc/guix.texi (Requirements): Increase minimum
Guile version from 2.0.9 to 2.0.13.
* config-daemon.ac: Remove use of 'GUIX_CHECK_UNBUFFERED_CBIP'.
* m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): Remove.
* guix/build/download.scm (current-http-proxy): Remove.
* guix/build/syscalls.scm (%libc-errno-pointer, errno): Remove.
(syscall->procedure): Use #:return-errno unconditionally.
* guix/hash.scm (open-sha256-input-port)[unbuffered]: Remove outdated
comment.
* guix/http-client.scm (when-guile<=2.0.5-or-otherwise-broken): Remove.
<top level>: Remove 'when-guile<=2.0.5-or-otherwise-broken' block.
* guix/scripts/substitute.scm (fetch): Remove 'guile-version>?'
conditional.
* tests/hash.scm (supports-unbuffered-cbip?): Remove.
<top level>: Remove 'test-skip' call.
Ludovic Courtès
2018-01-07daemon: Make libbz2 an optional dependency....* config-daemon.ac: Don't bail out when libbz2 is missing.  Define
'HAVE_LIBBZ2' Automake conditional.
* nix/libstore/build.cc: Wrap relevant bits in '#if HAVE_BZLIB_H'.
* nix/libstore/globals.cc (Settings::Settings): 'logCompression'
defaults to COMPRESSION_GZIP when HAVE_BZLIB_H is false.
* nix/libstore/globals.hh (CompressionType): Make 'COMPRESSION_BZIP2'
conditional on HAVE_BZLIB_H.
* nix/local.mk (guix_register_LDADD, guix_daemon_LDADD): Add -lbz2 only
when HAVE_LIBBZ2.
* nix/nix-daemon/guix-daemon.cc (parse_opt): Ignore "bzip2" when not
HAVE_BZLIB_H.
Ludovic Courtès
2018-01-07daemon: Add gzip log compression....* nix/nix-daemon/guix-daemon.cc (GUIX_OPT_LOG_COMPRESSION): New macro.
(options): Mark "disable-log-compression" as hidden and add
"log-compression".
(parse_opt): Handle GUIX_OPT_LOG_COMPRESSION.
* nix/libstore/build.cc (DerivationGoal): Add 'gzLogFile'.
(openLogFile): Initialize it when 'logCompression' is COMPRESSION_GZIP.
(closeLogFile, handleChildOutput): Honor 'gzLogFile'.
* nix/libstore/globals.hh (Settings)[compressLog]: Remove.
[logCompression]: New field.
(CompressionType): New enum.
* nix/libstore/globals.cc (Settings::Settings): Initialize it.
(update): Remove '_get' call for 'compressLog'.
* nix/local.mk (guix_daemon_LDADD, guix_register_LDADD): Add -lz.
* guix/store.scm (log-file): Handle '.gz' log files.
* tests/guix-daemon.sh: Add test with '--log-compression=gzip'.
* doc/guix.texi (Invoking guix-daemon): Adjust accordingly.
* config-daemon.ac: Check for libz and zlib.h.
Ludovic Courtès
2017-02-06config-daemon.ac: detect host AR...The problem is seen when we try to use explicit host:

    ./configure --prefix=/usr --localstatedir=/var/lib --host=x86_64-pc-linux-gnu
    make V=1

Before the change:
    ar cru libstore.a nix/libstore/libstore_a-gc.o
After the change:
    x86_64-pc-linux-gnu-ar cru libstore.a

* config-daemon.ac: use AM_PROG_AR to detect host AR

Signed-off-by: Sergei Trofimovich <slyfox@inbox.ru>
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Sergei Trofimovich
2016-11-25offload: Use Guile-SSH instead of GNU lsh....* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove.
[host-key, host-key-type]: New fields.
(%lsh-command, %lshg-command, user-lsh-private-key): Remove.
(user-openssh-private-key, private-key-from-file*): New procedures.
(host-key->type+key, open-ssh-session): New procedures.
(remote-pipe): Remove 'mode' parameter.  Rewrite in terms of
'open-ssh-session' etc.  Update users.
(send-files)[missing-files]: Rewrite using the bidirectional channel
port.
Remove call to 'call-with-compressed-output-port'.
(retrieve-files): Remove call to 'call-with-decompressed-port'.
(machine-load): Remove exit status logic.
* doc/guix.texi (Requirements): Mention Guile-SSH.
(Daemon Offload Setup): Document 'host-key' and 'private-key'.  Show the
default value on each @item line.
* m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro.
* config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'.  Set
'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
Ludovic Courtès
2016-11-16daemon: Add "builtin:download" derivation builder....This ensures that 1) the derivation doesn't change when Guix changes;
2) the derivation closure doesn't contain Guix and its dependencies; 3)
we don't have to rely on ugly chroot hacks.

Adapted from Nix commit 0a2bee307b20411f5b0dda0c662b1f9bb9e0e131.

* nix/libstore/build.cc (DerivationGoal::runChild): Add special case for
'isBuiltin(drv)'.  Disable chroot when 'isBuiltin(drv)'.
* nix/libstore/builtins.cc, nix/libstore/builtins.hh,
nix/scripts/download.in, guix/scripts/perform-download.scm: New files.
* guix/ui.scm (show-guix-help)[internal?]: Add 'perform-download'.
* nix/local.mk (libstore_a_SOURCES): Add builtins.cc.
(libstore_headers): Add builtins.hh.
(nodist_pkglibexec_SCRIPTS): Add 'scripts/download'.
* config-daemon.ac: Emit 'scripts/download'.
* Makefile.am (MODULES): Add 'guix/scripts/perform-download.scm'.
* tests/derivations.scm ("unknown built-in builder")
("'download' built-in builder")
("'download' built-in builder, invalid hash")
("'download' built-in builder, not found")
("'download' built-in builder, not fixed-output"): New tests.

Co-authored-by: Eelco Dolstra <eelco.dolstra@logicblox.com>
Ludovic Courtès