stack and the PVP

We recently had a very large discussion of stack on Reddit, which I thought was great for kicking off some discussion. In that discussion, there was a very active thread about how stack relates to the Package Versioning Policy (aka, the PVP).

The PVP - and in particular its policy on preemptive upper bounds - has been discussed at great length many times in the past, and I have no wish to revisit that discussion here. I also went into some detail on Reddit explaining how stack and Stackage relate to the PVP, and won't repeat those details here (tl;dr: they're orthogonal issues, and upper bounds are neither required nor discouraged by either).

However, the discussion brought up one of my long-held beliefs: "the right way to solve this is with tooling." Specifically: manually keeping track of lots of lower and upper bounds is a hard, tedious process that many people get wrong regularly. One great effort in this direction is the Hackage Matrix Builder project, which helps find overly lax (and, I believe, overly restrictive) bounds on Hackage and report them to authors. I'm announcing an experimental feature I've just added to stack master, and hoping it can help with the initial creation of upper bounds.

The feature itself is quite simple. When you run the sdist or upload commands, there's a new option: --pvp-bounds, which can be set to none (don't modify the cabal file at all), upper (add upper bounds), lower (add lower bounds), and both (add both upper and lower bounds). The default is none (we shouldn't change an author's cabal file without permission), but that default can be overridden in the stack.yaml file (either for your project, or in ~/.stack/stack.yaml). The algorithm is this:

• Find every single dependency in the .cabal file (e.g., bytestring >= 0.9)
• If the user has asked for lower bounds to be added, and that dependency has no lower bound, set the lower bound to the package version specified in the stack.yaml file
• If the user has asked for upper bounds to be added, and that dependency has no upper bound, set the upper bound to be less than the next major version of the package relative to what's specified in the stack.yaml file

That was a bit abstract, so let's give an example. Support you're using LTS 3.0, which includes aeson-0.8.0.2, attoparsec-0.12.1.6, and text-1.2.1.3. Let's further say that in your cabal file you have the following:

build-depends: aeson >= 0.7
             , text < 2
             , attoparsec

If you specify --pvp-bounds both on the command line, you'll end up with the following changes:

• aeson will now be specified as aeson >= 0.7 && < 0.9. Reason: We respect the existing lower bound (>= 0.7), but add in an upper bound based on the version of aeson used in your snapshot. Since we're currently using 0.8.0.2, the next major bump will be 0.9.
• text will be text >= 1.2.1.3 && < 2. We respect the existing upper bound (even though it's no in compliance with PVP rules - this allows you as an author to maintain more control when using this feature), but add in a lower bound to the currently used version of text.
• attoparsec will be attoparsec >= 0.12.1.6 && < 0.13. Since there are no upper or lower bounds, we add both of them in.

Just to round out the feature description:

• if you use --pvp-bounds none, your bounds are unmodified
• with --pvp-bounds lower you get
  • aeson >= 0.7
  • text >= 1.2.1.3 && < 2
  • attoparsec >= 0.12.1.6
• with --pvp-bounds upper you get
  • aeson >= 0.7 && < 0.9
  • text < 2
  • attoparsec < 0.13

The motivation behind this approach is simplicity. For users of stack, your versioning work usually comes down to choosing just one number: the LTS Haskell version (or Stackage Nightly date), which is relatively easy to deal with. Managing version ranges of every single dependency is an arduous process, and hard to get right. (How often have you accidentally started relying on a new feature in a minor version bump of a dependency, but forgotten to bump the lower bound?) Now, stack will handle that drudgery for you.

Of course, there are cases where you'll want to tell stack that you know better than it, e.g. "I'm using a subset of the aeson API that's compatible between both 0.7 and 0.8, so I want to override stack's guess at a lower bound." Or, "even though the PVP says text-1.3 may have a breaking change, I've spoken with the author, and I know that the parts I'm using won't be changed until version 2."

This feature should still be considered experimental, but I'm hopeful that it will be an experiment that works, and can make both upper bounds advocates and detractors happy. As a member of the latter group, I'm actually planning on trying out this functionality on some of my packages for future releases.

There are still downsides to this feature, which are worth mentioning:

• Lower bounds may be too restrictive by default. Solution: consider just using --pvp-bounds upper
• You'll still have to manually relax upper bounds on Hackage. Solution: make sure your project is in Stackage so you get early notifications, and add relaxed upper bounds manually to your .cabal file as necessary.
• This does nothing to add upper bounds to existing uploads to Hackage. Maybe someone can add a tool to do that automatically for you (or, even better, enhance cabal's dependency solver to take date information into account)
• In the past, a strong reason to avoid upper bounds was that it would trigger bugs in the dependency solver that could prevent a valid build plan from being found. For the most part, those bugs have been resolved, but on occasion you may still need to use the --reorder-goals --max-backjumps=-1 flag to cabal. (Note: when using stack's dependency solving capabilities, it passes in these flags for you automatically.)