Force recompilation of some test modules that use the plugin

[zliu41]

These modules aren't re-compiled if we modify, say, PlutusIR.Transform.Beta. But they should, because the plugin depends on it.

Alternatively we can change the plugin to impurePlugin, but that seems like overkill.

[michaelpj]

I've never quite got my head around this. We use flagRecompile, which I'd kind of hoped would force recompilation if the plugin itself changes, but doesn't seem to. I bet it would do if the plugin was defined in a different package, because then it would come from a different unit after recompilation. Not sure tbh.

Also, if we do this we'd need to do this for all the test modules, since they basically all use the plugin (they just mostly don't use the TH way of adding it). In fact... is that why I did it? Does it work if we do what the other modules do and use the plc function and an explicit OPTIONS_GHC -fplugin flag?

[zliu41]

Yeah these are good questions. I'll have to look into this a bit further.

[zliu41]

So I looked into this, and it has nothing to do with the plugin. What happened is that if we comment out the TyInst case of beta, then rebuild, for whatever reason, in Beta's interface file, beta has the exact same hash as before. This makes GHC think that Beta.hs isn't actually changed, so nothing downstream will be re-compiled. This is very strange. cc @michaelpj

[michaelpj]

Does that mean that it also won't recompile e.g. the tests for beta? Are you sure? That sounds like it would be a serious recompilation avoidance bug in GHC.

[zliu41]

I know for sure that it didn't recompile PlutusIR.Compiler, which depends on Beta.

[michaelpj]

That it might be allowed to do, since it'll be doing separate compilation and it should just be able to re-link Compiler.hs.

[zliu41]

That it might be allowed to do, since it'll be doing separate compilation

Not sure about that since PlutusIR.Transform.Beta and PlutusIR.Compiler are in the same package. Either way, the end result is that Budget.Spec is not re-compiled after modifying Beta.hs, which is not supposed to happen since it uses the plugin, whose behavior depends on Beta.hs.

[zliu41]

If I comment out both the first and the second cases of beta, then beta will have a different fingerprint in the interface file, and PlutusIR.Compiler gets re-compiled.

[zliu41]

Ok ignore the previous two comments. I think it is most likely a problem only when we have a plugin using flagRecompile. Here's what happened:

• PlutusIR.Transform.Beta.beta is modified. In Beta.hi, beta has no unfolding (because of the lack of INLINEABLE beta), and its fingerprint does not change.
• Our plugin uses flagRecompile, and since beta's fingerprint in Beta.hi does not change, the plugin's fingerprint does not change, even though the plugin depends on beta.
• As a result, Budget/Spec.hs is not re-compiled when running plutus-tx-tests, even though the plugin's behavior has changed.

So there are at least 3 options:

1. Add -fexpose-all-unfoldings to all packages the plugin depends on
2. Add -fforce-recomp (this PR)
3. Use impurePlugin rather than flagRecompile.

I feel like 1 is the best option. 3 is probably a non-starter since it affects the users. 2 causes those test modules to be re-compiled every time, even if nothing changes. @michaelpj What do you think?

By the way I opened a GHC issue about this: https://gitlab.haskell.org/ghc/ghc/-/issues/21543

[michaelpj]

Hmm, so trying to think about this. I think maybe the underlying issue is:

• Interface file fingerprints track whether we need to recompile a module that depends on X.
• In the case of changing beta, since it's type signature and unfolding (or lack thereof) stay the same, we wouldn't need to recompile a downstream module.
• But the situation is different for plugins, since their behaviour may change based on the generated code not just the interface file. So in this case the behaviour of the plugin has changed.

[michaelpj]

But Matthew seems to be saying that this shouldn't happen.

[michaelpj]

I commented on the issue.

[kwxm]

This has been open since May 2022. I'm going to close it.