index-state is not respected in the presence of revisions

[pranaysashank]

Describe the bug

When index-state or cabalProjectFile is passed to haskell-nix.project or haskell-nix.tool, bumping the hackageSrc pin causes rebuilds because of new revisions in the new updated index.

Steps To Reproduce

I have created a repo to reproduce the issue at https://github.com/pranaysashank/haskell-nix-index-state-bug-repro . fOld uses hackageNix with index-state 2023-12-15T23:46:08Z and fNew uses a newer index state which contains the revisions

1. Clone the repo
2. nix-build -A fOld --dry-run
3. nix-build -A fNew --dry-run
4. nix-diff the .drv for fourmolu in both the runs, this shows the difference is because of the presence of new revisions

nix-diff /nix/store/g5z6w89qh54lagzhnxfsmdz5hccrldvh-fourmolu-exe-fourmolu-0.14.1.0.drv /nix/store/168bjqffj01sbn2rvihbvh147ya00gcg-fourmolu-exe-fourmolu-0.14.1.0.drv
- /nix/store/g5z6w89qh54lagzhnxfsmdz5hccrldvh-fourmolu-exe-fourmolu-0.14.1.0.drv:{out}
+ /nix/store/168bjqffj01sbn2rvihbvh147ya00gcg-fourmolu-exe-fourmolu-0.14.1.0.drv:{out}
• The set of input derivation names do not match:
    - fourmolu-0.14.1.0-src
    + fourmolu-0.14.1.0-r1-src
• The input derivation named `fourmolu-lib-fourmolu-0.14.1.0` differs
  - /nix/store/qd04ny3s0vlahlmcgc26jmqp6whflxkl-fourmolu-lib-fourmolu-0.14.1.0.drv:{out}
  + /nix/store/707ikcvyxrw2i6c5ihjlwn9xicqn4dab-fourmolu-lib-fourmolu-0.14.1.0.drv:{out}
  • The set of input derivation names do not match:
      - fourmolu-0.14.1.0-src
      + fourmolu-0.14.1.0-r1-src
  • The input derivation named `ghc-lib-parser-lib-ghc-lib-parser-9.8.1.20231121` differs
    - /nix/store/pnsp2alwyw3va5hkbypchc2khm79pn0m-ghc-lib-parser-lib-ghc-lib-parser-9.8.1.20231121.drv:{out}
    + /nix/store/8xs9bl4qaxnr4qz2x9xw95gg93s08cll-ghc-lib-parser-lib-ghc-lib-parser-9.8.1.20231121.drv:{out}
    • The input derivation named `process-lib-process-1.6.18.0` differs
      - /nix/store/826ih8qw631a3xbsbvxcm5i0mcfnj4vs-process-lib-process-1.6.18.0.drv:{out}
      + /nix/store/ijh4zdk7ijys0d7hvg2hqgf6jchs87g6-process-lib-process-1.6.18.0.drv:{out}
      • The environments do not match:
          + cabalFile=name:          process
version:       1.6.18.0
x-revision: 1
...

Expected behavior

Providing cabalProjectFreeze or index-state shouldn't cause rebuilds even in presence of revisions, this reduces the number of rebuilds we have to make when we want to update a dependency by bumping hackageSrc in the project.

Additional context

it looks like @andreabedini has made recent changes w.r.t revisions and index-state (#1775) , perhaps you know more about the issue ?

• Build system (eg. x86_64-linux or aarch64-darwin): x86_64-linux
• GHC version: 9.8.1
• Haskell.nix version (or commit) used: Available in the provided repo

[andreabedini]

@pranaysashank Thanks for opening this.

I had originally thought that you could have accidentally used a version of hackage.nix earlier than the index-state set in the project file, which would therefore be ineffective. Bumping hackage.nix could then cause the plan to change even without changing the project index-state.

This does not seem to be the case.

❯ nix repl -f default.nix
Welcome to Nix 2.18.1. Type :? for help.

Loading installable ''...
Added 2 variables.
nix-repl> lib = fNew.project.pkgs.lib

nix-repl> fOld.project.index-state-max
"2023-12-15T23:46:08Z"

nix-repl> lib.last (lib.attrNames (import fOld.project.pkgs.haskell-nix.indexStateHashesPath))
"2023-12-16T00:00:00Z"

nix-repl> fNew.project.index-state-max
"2023-12-15T23:46:08Z"

nix-repl> lib.last (lib.attrNames (import fNew.project.pkgs.haskell-nix.indexStateHashesPath))
"2023-12-17T00:00:00Z"

For context, the revision fourmolu-0.14.1.0-r1 has been added on 2023-12-16T06:17:12Z so it should not be available in either plans.

This is a bug, or quirk if you want to be extra diplomatic, in hackage-project and everything that based on it, like hackage-tool. You can see here how it picks the package by itself:

haskell.nix/modules/hackage-project.nix

Lines 28 to 37 in 9cbdd6f

	let
	tarball = config.evalPackages.fetchurl {
	url = "mirror://hackage/${name}-${version}.tar.gz";
	inherit (pkgs.haskell-nix.hackage.${name}.${version}) sha256; };
	rev = pkgs.haskell-nix.hackage.${name}.${version}.revisions.${revision};
	cabalFile = config.evalPackages.fetchurl {
	url = "https://hackage.haskell.org/package/${name}-${version}/revision/${toString rev.revNum}.cabal";
	inherit (rev) sha256;
	};
	revSuffix = lib.optionalString (rev.revNum > 0) "-r${toString rev.revNum}";

So haskell.nix is doing the equivalent of

cabal get fourmolu-0.14.1.0 # Note that this applies revisions
cd fourmolu-0.14.1.0
cabal build --index-state=2023-12-15T23:46:08Z

This indeed produces the same plan (when using ghc-9.8.1).

I think the end result is that the index-state you specify only applies to the dependencies and not to the package itself.

@hamishmack @angerman Is this something we want to fix?

[pranaysashank]

I think the end result is that the index-state you specify only applies to the dependencies and not to the package itself.

In the nix-diff in my comment above, the process package also gets rebuilt and even that has a new revision in the new index. So even dependencies are affected

[andreabedini]

I find that more puzzling. You can see that, a part from fourmolu itself, the inputs to the plan derivations do not change.

nix-diff /nix/store/m468c34q6zf17474j589pvyp56nsnd77-fourmolu-plan-to-nix-pkgs.drv /nix/store/gcqi5fkaid4v2k44bxcp0rkqnr5jzlyn-fourmolu-plan-to-nix-pkgs.drv

and that in both cases we pass --index-state=2023-12-15T23:46:08Z to make-install-plan.
🤔

[pranaysashank]

I find that more puzzling. You can see that, a part from fourmolu itself, the inputs to the plan derivations do not change.

The plans always point to a default revision and what is default keeps changing across hackage.nix commits when there is a new revision.

Here's what I figured out, the packages in hackage.nix always have default revision set to the latest one and the plans pick the default revision. I made a change to hackage-db to extract the timestamp of the revision. This generates hackage packages to add a revTimestamp attribute, see here for an example.

These changes to haskell.nix kronor-io@b057264 use the new revTimestamp attribute and only pick a revision that's present in a given index. These changes fixes the issues observed above.

Earlier I have noticed that updates to hackage.nix pin would sometimes even cause a rebuild of ghc, these changes fix those issues completely. Let me know if I a fix is desired and I should open a PR