By default Rebar3 wants to manage it's own dependencies. In the normal non-Nix, this is perfectly acceptable. In the Nix world it is not. To support this we have created two versions of rebar3, rebar3 and rebar3-open. The rebar3 version has been patched to remove the ability to download anything from it. If you are not running it a nix-shell or a nix-build then its probably not going to work for you. rebar3-open is the normal, un-modified rebar3. It should work exactly as would any other version of rebar3. Any Erlang package should rely on rebar3 and thats really what you should be using too.

Both Mix and Erlang.mk work exactly as you would expect. There is a bootstrap process that needs to be run for both of them. However, that is supported by the buildMix and buildErlangMk derivations.

Often, all you want to do is be able to access a valid environment that contains a specific package and its dependencies. we can do that with the env part of a derivation. For example, lets say we want to access an erlang repl with ibrowse loaded up. We could do the following.

      ~/w/nixpkgs ❯❯❯ nix-shell -A beamPackages.ibrowse.env --run "erl"
      Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

      Eshell V7.0  (abort with ^G)
      1> m(ibrowse).
      Module: ibrowse
      MD5: 3b3e0137d0cbb28070146978a3392945
      Compiled: January 10 2016, 23:34
      Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
      Compiler options:  [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
      debug_info,debug_info,nowarn_shadow_vars,
      warn_unused_import,warn_unused_vars,warnings_as_errors,
      {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
      Exports:
      add_config/1                  send_req_direct/7
      all_trace_off/0               set_dest/3
      code_change/3                 set_max_attempts/3
      get_config_value/1            set_max_pipeline_size/3
      get_config_value/2            set_max_sessions/3
      get_metrics/0                 show_dest_status/0
      get_metrics/2                 show_dest_status/1
      handle_call/3                 show_dest_status/2
      handle_cast/2                 spawn_link_worker_process/1
      handle_info/2                 spawn_link_worker_process/2
      init/1                        spawn_worker_process/1
      module_info/0                 spawn_worker_process/2
      module_info/1                 start/0
      rescan_config/0               start_link/0
      rescan_config/1               stop/0
      send_req/3                    stop_worker_process/1
      send_req/4                    stream_close/1
      send_req/5                    stream_next/1
      send_req/6                    terminate/2
      send_req_direct/4             trace_off/0
      send_req_direct/5             trace_off/2
      send_req_direct/6             trace_on/0
      trace_on/2
      ok
      2>
    

Notice the -A beamPackages.ibrowse.env.That is the key to this functionality.

Getting access to an environment often isn't enough to do real development. Many times we need to create a shell.nix file and do our development inside of the environment specified by that file. This file looks a lot like the packaging described above. The main difference is that src points to project root and we call the package directly.

{ pkgs ? import "<nixpkgs"> {} }:

with pkgs;

let

  f = { buildRebar3, ibrowse, jsx, erlware_commons }:
      buildRebar3 {
        name = "hex2nix";
        version = "0.1.0";
        src = ./.;
        erlangDeps = [ ibrowse jsx erlware_commons ];
      };
  drv = beamPackages.callPackage f {};

in
 drv
    

# =============================================================================
# Variables
# =============================================================================

NIX_TEMPLATES := "$(CURDIR)/nix-templates"

TARGET := "$(PREFIX)"

PROJECT_NAME := thorndyke

NIXPKGS=../nixpkgs
NIX_PATH=nixpkgs=$(NIXPKGS)
NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
# =============================================================================
# Rules
# =============================================================================
.PHONY= all test clean repl shell build test analyze configure install \
        test-nix-install publish plt analyze

all: build

guard-%:
        @ if [ "${${*}}" == "" ]; then \
                echo "Environment variable $* not set"; \
                exit 1; \
        fi

clean:
        rm -rf _build
        rm -rf .cache

repl:
        $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"

shell:
        $(NIX_SHELL)

configure:
        $(NIX_SHELL) --command 'eval "$$configurePhase"'

build: configure
        $(NIX_SHELL) --command 'eval "$$buildPhase"'

install:
        $(NIX_SHELL) --command 'eval "$$installPhase"'

test:
        $(NIX_SHELL) --command 'mix test --no-start --no-deps-check'

plt:
        $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"

analyze: build plt
        $(NIX_SHELL) --run "mix dialyzer --no-compile"

    

A simple development environment consists of a Haskell compiler and one or both of the tools cabal-install and stack. We saw in section How to install Haskell packages how you can install those programs into your user profile:

$ nix-env -f "<nixpkgs>" -iA haskellPackages.ghc haskellPackages.cabal-install

Instead of the default package set haskellPackages, you can also use the more precise name haskell.compiler.ghc7102, which has the advantage that it refers to the same GHC version regardless of what Nixpkgs considers “default” at any given time.

Once you’ve made those tools available in $PATH, it’s possible to build Hackage packages the same way people without access to Nix do it all the time:

$ cabal get lens-4.11 && cd lens-4.11
$ cabal install -j --dependencies-only
$ cabal configure
$ cabal build

If you enjoy working with Cabal sandboxes, then that’s entirely possible too: just execute the command

$ cabal sandbox init

before installing the required dependencies.

The nix-shell utility makes it easy to switch to a different compiler version; just enter the Nix shell environment with the command

$ nix-shell -p haskell.compiler.ghc784

to bring GHC 7.8.4 into $PATH. Alternatively, you can use Stack instead of nix-shell directly to select compiler versions and other build tools per-project. It uses nix-shell under the hood when Nix support is turned on. See How to build a Haskell project using Stack.

If you’re using cabal-install, re-running cabal configure inside the spawned shell switches your build to use that compiler instead. If you’re working on a project that doesn’t depend on any additional system libraries outside of GHC, then it’s even sufficient to just run the cabal configure command inside of the shell:

$ nix-shell -p haskell.compiler.ghc784 --command "cabal configure"

Afterwards, all other commands like cabal build work just fine in any shell environment, because the configure phase recorded the absolute paths to all required tools like GHC in its build configuration inside of the dist/ directory. Please note, however, that nix-collect-garbage can break such an environment because the Nix store paths created by nix-shell aren’t “alive” anymore once nix-shell has terminated. If you find that your Haskell builds no longer work after garbage collection, then you’ll have to re-run cabal configure inside of a new nix-shell environment.

GHC expects to find all installed libraries inside of its own lib directory. This approach works fine on traditional Unix systems, but it doesn’t work for Nix, because GHC’s store path is immutable once it’s built. We cannot install additional libraries into that location. As a consequence, our copies of GHC don’t know any packages except their own core libraries, like base, containers, Cabal, etc.

We can register additional libraries to GHC, however, using a special build function called ghcWithPackages. That function expects one argument: a function that maps from an attribute set of Haskell packages to a list of packages, which determines the libraries known to that particular version of GHC. For example, the Nix expression ghcWithPackages (pkgs: [pkgs.mtl]) generates a copy of GHC that has the mtl library registered in addition to its normal core packages:

$ nix-shell -p "haskellPackages.ghcWithPackages (pkgs: [pkgs.mtl])"

[nix-shell:~]$ ghc-pkg list mtl
/nix/store/zy79...-ghc-7.10.2/lib/ghc-7.10.2/package.conf.d:
    mtl-2.2.1

This function allows users to define their own development environment by means of an override. After adding the following snippet to ~/.config/nixpkgs/config.nix,

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskell.packages.ghc7102.ghcWithPackages
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

it’s possible to install that compiler with nix-env -f "<nixpkgs>" -iA myHaskellEnv. If you’d like to switch that development environment to a different version of GHC, just replace the ghc7102 bit in the previous definition with the appropriate name. Of course, it’s also possible to define any number of these development environments! (You can’t install two of them into the same profile at the same time, though, because that would result in file conflicts.)

The generated ghc program is a wrapper script that re-directs the real GHC executable to use a new lib directory — one that we specifically constructed to contain all those packages the user requested:

$ cat $(type -p ghc)
#! /nix/store/xlxj...-bash-4.3-p33/bin/bash -e
export NIX_GHC=/nix/store/19sm...-ghc-7.10.2/bin/ghc
export NIX_GHCPKG=/nix/store/19sm...-ghc-7.10.2/bin/ghc-pkg
export NIX_GHC_DOCDIR=/nix/store/19sm...-ghc-7.10.2/share/doc/ghc/html
export NIX_GHC_LIBDIR=/nix/store/19sm...-ghc-7.10.2/lib/ghc-7.10.2
exec /nix/store/j50p...-ghc-7.10.2/bin/ghc "-B$NIX_GHC_LIBDIR" "$@"

The variables $NIX_GHC, $NIX_GHCPKG, etc. point to the new store path ghcWithPackages constructed specifically for this environment. The last line of the wrapper script then executes the real ghc, but passes the path to the new lib directory using GHC’s -B flag.

The purpose of those environment variables is to work around an impurity in the popular ghc-paths library. That library promises to give its users access to GHC’s installation paths. Only, the library can’t possible know that path when it’s compiled, because the path GHC considers its own is determined only much later, when the user configures it through ghcWithPackages. So we patched ghc-paths to return the paths found in those environment variables at run-time rather than trying to guess them at compile-time.

To make sure that mechanism works properly all the time, we recommend that you set those variables to meaningful values in your shell environment, too, i.e. by adding the following code to your ~/.bashrc:

if type >/dev/null 2>&1 -p ghc; then
  eval "$(egrep ^export "$(type -p ghc)")"
fi

If you are certain that you’ll use only one GHC environment which is located in your user profile, then you can use the following code, too, which has the advantage that it doesn’t contain any paths from the Nix store, i.e. those settings always remain valid even if a nix-env -u operation updates the GHC environment in your profile:

if [ -e ~/.nix-profile/bin/ghc ]; then
  export NIX_GHC="$HOME/.nix-profile/bin/ghc"
  export NIX_GHCPKG="$HOME/.nix-profile/bin/ghc-pkg"
  export NIX_GHC_DOCDIR="$HOME/.nix-profile/share/doc/ghc/html"
  export NIX_GHC_LIBDIR="$HOME/.nix-profile/lib/ghc-$($NIX_GHC --numeric-version)"
fi

If you plan to use your environment for interactive programming, not just compiling random Haskell code, you might want to replace ghcWithPackages in all the listings above with ghcWithHoogle.

This environment generator not only produces an environment with GHC and all the specified libraries, but also generates a hoogle and haddock indexes for all the packages, and provides a wrapper script around hoogle binary that uses all those things. A precise name for this thing would be “ghcWithPackagesAndHoogleAndDocumentationIndexes”, which is, regrettably, too long and scary.

For example, installing the following environment

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskellPackages.ghcWithHoogle
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

allows one to browse module documentation index not too dissimilar to this for all the specified packages and their dependencies by directing a browser of choice to ~/.nix-profiles/share/doc/hoogle/index.html (or /run/current-system/sw/share/doc/hoogle/index.html in case you put it in environment.systemPackages in NixOS).

After you’ve marveled enough at that try adding the following to your ~/.ghc/ghci.conf

:def hoogle \s -> return $ ":! hoogle search -cl --count=15 \"" ++ s ++ "\""
:def doc \s -> return $ ":! hoogle search -cl --info \"" ++ s ++ "\""

and test it by typing into ghci:

:hoogle a -> a
:doc a -> a

Be sure to note the links to haddock files in the output. With any modern and properly configured terminal emulator you can just click those links to navigate there.

Finally, you can run

hoogle server -p 8080

and navigate to http://localhost:8080/ for your own local Hoogle. Note, however, that Firefox and possibly other browsers disallow navigation from http: to file: URIs for security reasons, which might be quite an inconvenience. See this page for workarounds.

Stack is a popular build tool for Haskell projects. It has first-class support for Nix. Stack can optionally use Nix to automatically select the right version of GHC and other build tools to build, test and execute apps in an existing project downloaded from somewhere on the Internet. Pass the --nix flag to any stack command to do so, e.g.

$ git clone --recursive http://github.com/yesodweb/wai
$ cd wai
$ stack --nix build

If you want stack to use Nix by default, you can add a nix section to the stack.yaml file, as explained in the Stack documentation. For example:

nix:
  enable: true
  packages: [pkgconfig zeromq zlib]

The example configuration snippet above tells Stack to create an ad hoc environment for nix-shell as in the below section, in which the pkgconfig, zeromq and zlib packages from Nixpkgs are available. All stack commands will implicitly be executed inside this ad hoc environment.

Some projects have more sophisticated needs. For examples, some ad hoc environments might need to expose Nixpkgs packages compiled in a certain way, or with extra environment variables. In these cases, you’ll need a shell field instead of packages:

nix:
  enable: true
  shell-file: shell.nix

For more on how to write a shell.nix file see the below section. You’ll need to express a derivation. Note that Nixpkgs ships with a convenience wrapper function around mkDerivation called haskell.lib.buildStackProject to help you create this derivation in exactly the way Stack expects. All of the same inputs as mkDerivation can be provided. For example, to build a Stack project that including packages that link against a version of the R library compiled with special options turned on:

with (import <nixpkgs> { });

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
}

You can select a particular GHC version to compile with by setting the ghc attribute as an argument to buildStackProject. Better yet, let Stack choose what GHC version it wants based on the snapshot specified in stack.yaml (only works with Stack >= 1.1.3):

{nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}:

with nixpkgs;

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
  inherit ghc;
}

The easiest way to create an ad hoc development environment is to run nix-shell with the appropriate GHC environment given on the command-line:

nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])"

For more sophisticated use-cases, however, it’s more convenient to save the desired configuration in a file called shell.nix that looks like this:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
let
  inherit (nixpkgs) pkgs;
  ghc = pkgs.haskell.packages.${compiler}.ghcWithPackages (ps: with ps; [
          monad-par mtl
        ]);
in
pkgs.stdenv.mkDerivation {
  name = "my-haskell-env-0";
  buildInputs = [ ghc ];
  shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
}

Now run nix-shell — or even nix-shell --pure — to enter a shell environment that has the appropriate compiler in $PATH. If you use --pure, then add all other packages that your development environment needs into the buildInputs attribute. If you’d like to switch to a different compiler version, then pass an appropriate compiler argument to the expression, i.e. nix-shell --argstr compiler ghc784.

If you need such an environment because you’d like to compile a Hackage package outside of Nix — i.e. because you’re hacking on the latest version from Git —, then the package set provides suitable nix-shell environments for you already! Every Haskell package has an env attribute that provides a shell environment suitable for compiling that particular package. If you’d like to hack the lens library, for example, then you just have to check out the source code and enter the appropriate environment:

  $ cabal get lens-4.11 && cd lens-4.11
  Downloading lens-4.11...
  Unpacking to lens-4.11/

  $ nix-shell "<nixpkgs>" -A haskellPackages.lens.env
  [nix-shell:/tmp/lens-4.11]$

At point, you can run cabal configure, cabal build, and all the other development commands. Note that you need cabal-install installed in your $PATH already to use it here — the nix-shell environment does not provide it.

For example, let’s assume that you’re working on a private project called foo. To generate a Nix build expression for it, change into the project’s top-level directory and run the command:

$ cabal2nix . >foo.nix

Then write the following snippet into a file called default.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./foo.nix { }

Finally, store the following code in a file called shell.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
(import ./default.nix { inherit nixpkgs compiler; }).env

At this point, you can run nix-build to have Nix compile your project and install it into a Nix store path. The local directory will contain a symlink called result after nix-build returns that points into that location. Of course, passing the flag --argstr compiler ghc763 allows switching the build to any version of GHC currently supported.

Furthermore, you can call nix-shell to enter an interactive development environment in which you can use cabal configure and cabal build to develop your code. That environment will automatically contain a proper GHC derivation with all the required libraries registered as well as all the system-level libraries your package might need.

If your package does not depend on any system-level libraries, then it’s sufficient to run

$ nix-shell --command "cabal configure"

once to set up your build. cabal-install determines the absolute paths to all resources required for the build and writes them into a config file in the dist/ directory. Once that’s done, you can run cabal build and any other command for that project even outside of the nix-shell environment. This feature is particularly nice for those of us who like to edit their code with an IDE, like Emacs’ haskell-mode, because it’s not necessary to start Emacs inside of nix-shell just to make it find out the necessary settings for building the project; cabal-install has already done that for us.

If you want to do some quick-and-dirty hacking and don’t want to bother setting up a default.nix and shell.nix file manually, then you can use the --shell flag offered by cabal2nix to have it generate a stand-alone nix-shell environment for you. With that feature, running

$ cabal2nix --shell . >shell.nix
$ nix-shell --command "cabal configure"

is usually enough to set up a build environment for any given Haskell package. You can even use that generated file to run nix-build, too:

$ nix-build shell.nix

If you have multiple private Haskell packages that depend on each other, then you’ll have to register those packages in the Nixpkgs set to make them visible for the dependency resolution performed by callPackage. First of all, change into each of your projects top-level directories and generate a default.nix file with cabal2nix:

$ cd ~/src/foo && cabal2nix . >default.nix
$ cd ~/src/bar && cabal2nix . >default.nix

Then edit your ~/.config/nixpkgs/config.nix file to register those builds in the default Haskell package set:

  {
    packageOverrides = super: let self = super.pkgs; in
    {
      haskellPackages = super.haskellPackages.override {
        overrides = self: super: {
          foo = self.callPackage ../src/foo {};
          bar = self.callPackage ../src/bar {};
        };
      };
    };
  }

Once that’s accomplished, nix-env -f "<nixpkgs>" -qA haskellPackages will show your packages like any other package from Hackage, and you can build them

$ nix-build "<nixpkgs>" -A haskellPackages.foo

or enter an interactive shell environment suitable for building them:

$ nix-shell "<nixpkgs>" -A haskellPackages.bar.env

Every Haskell package set takes a function called overrides that you can use to manipulate the package as much as you please. One useful application of this feature is to replace the default mkDerivation function with one that enables library profiling for all packages. To accomplish that, add configure the following snippet in your ~/.config/nixpkgs/config.nix file:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    profiledHaskellPackages = self.haskellPackages.override {
      overrides = self: super: {
        mkDerivation = args: super.mkDerivation (args // {
          enableLibraryProfiling = true;
        });
      };
    };
  };
}

Then, replace instances of haskellPackages in the cabal2nix-generated default.nix or shell.nix files with profiledHaskellPackages.

Nixpkgs provides the latest version of ghc-events, which is 0.4.4.0 at the time of this writing. This is fine for users of GHC 7.10.x, but GHC 7.8.4 cannot compile that binary. Now, one way to solve that problem is to register an older version of ghc-events in the 7.8.x-specific package set. The first step is to generate Nix build instructions with cabal2nix:

$ cabal2nix cabal://ghc-events-0.4.3.0 >~/.nixpkgs/ghc-events-0.4.3.0.nix

Then add the override in ~/.config/nixpkgs/config.nix:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    haskell = super.haskell // {
      packages = super.haskell.packages // {
        ghc784 = super.haskell.packages.ghc784.override {
          overrides = self: super: {
            ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
          };
        };
      };
    };
  };
}

This code is a little crazy, no doubt, but it’s necessary because the intuitive version

haskell.packages.ghc784 = super.haskell.packages.ghc784.override {
  overrides = self: super: {
    ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
  };
};

doesn’t do what we want it to: that code replaces the haskell package set in Nixpkgs with one that contains only one entry,packages, which contains only one entry ghc784. This override loses the haskell.compiler set, and it loses the haskell.packages.ghcXYZ sets for all compilers but GHC 7.8.4. To avoid that problem, we have to perform the convoluted little dance from above, iterating over each step in hierarchy.

Once it’s accomplished, however, we can install a variant of ghc-events that’s compiled with GHC 7.8.4:

nix-env -f "<nixpkgs>" -iA haskell.packages.ghc784.ghc-events

Unfortunately, it turns out that this build fails again while executing the test suite! Apparently, the release archive on Hackage is missing some data files that the test suite requires, so we cannot run it. We accomplish that by re-generating the Nix expression with the --no-check flag:

$ cabal2nix --no-check cabal://ghc-events-0.4.3.0 >~/.nixpkgs/ghc-events-0.4.3.0.nix

Now the builds succeeds.

Of course, in the concrete example of ghc-events this whole exercise is not an ideal solution, because ghc-events can analyze the output emitted by any version of GHC later than 6.12 regardless of the compiler version that was used to build the ghc-events executable, so strictly speaking there’s no reason to prefer one built with GHC 7.8.x in the first place. However, for users who cannot use GHC 7.10.x at all for some reason, the approach of downgrading to an older version might be useful.

GHC and distributed build farms don’t get along well:

https://ghc.haskell.org/trac/ghc/ticket/4012

When you see an error like this one

package foo-0.7.1.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91

then you have to download and re-install foo and all its dependents from scratch:

# nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \
  | xargs -L 1 nix-store --repair-path

If you’re using additional Hydra servers other than hydra.nixos.org, then it might be necessary to purge the local caches that store data from those machines to disable these binary channels for the duration of the previous command, i.e. by running:

rm /nix/var/nix/binary-cache-v3.sqlite
rm /nix/var/nix/manifests/*
rm /nix/var/nix/channel-cache/*

Open a shell with haste-compiler and haste-cabal-install (you don’t actually need node, but it can be useful to test stuff):

$ nix-shell -p "haskellPackages.ghcWithPackages (self: with self; [haste-cabal-install haste-compiler])" -p nodejs

You may not need the following step but if haste-boot fails to compile all the packages it needs, this might do the trick

$ haste-cabal update

haste-boot builds a set of core libraries so that they can be used from Javascript transpiled programs:

$ haste-boot

Transpile and run a “Hello world” program:

$ echo 'module Main where main = putStrLn "Hello world"' > hello-world.hs
$ hastec --onexec hello-world.hs
$ node hello-world.js
Hello world

Users of GHC on Darwin have occasionally reported that builds fail, because the compiler complains about a missing include file:

fatal error: 'math.h' file not found

The issue has been discussed at length in ticket 6390, and so far no good solution has been proposed. As a work-around, users who run into this problem can configure the environment variables

export NIX_CFLAGS_COMPILE="-idirafter /usr/include"
export NIX_CFLAGS_LINK="-L/usr/lib"

in their ~/.bashrc file to avoid the compiler error.

--  While building package zlib-0.5.4.2 using:
  runhaskell -package=Cabal-1.22.4.0 -clear-package-db [... lots of flags ...]
Process exited with code: ExitFailure 1
Logs have been written to: /home/foo/src/stack-ide/.stack-work/logs/zlib-0.5.4.2.log

Configuring zlib-0.5.4.2...
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

When you run the build inside of the nix-shell environment, the system is configured to find libz.so without any special flags – the compiler and linker “just know” how to find it. Consequently, Cabal won’t record any search paths for libz.so in the package description, which means that the package works fine inside of nix-shell, but once you leave the shell the shared object can no longer be found. That issue is by no means specific to Stack: you’ll have that problem with any other Haskell package that’s built inside of nix-shell but run outside of that environment.

You can remedy this issue in several ways. The easiest is to add a nix section to the stack.yaml like the following:

nix:
  enable: true
  packages: [ zlib ]

Stack’s Nix support knows to add ${zlib.out}/lib and ${zlib.dev}/include as an --extra-lib-dirs and extra-include-dirs, respectively. Alternatively, you can achieve the same effect by hand. First of all, run

$ nix-build --no-out-link "<nixpkgs>" -A zlib
/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8

to find out the store path of the system’s zlib library. Now, you can

Typically, you’ll need –extra-include-dirs as well. It’s possible to add those flag to the project’s “stack.yaml” or your user’s global “~/.stack/global/stack.yaml” file so that you don’t have to specify them manually every time. But again, you’re likely better off using Stack’s Nix support instead.

The same thing applies to cabal configure, of course, if you’re building with cabal-install instead of Stack.

There are two levels of static linking. The first option is to configure the build with the Cabal flag --disable-executable-dynamic. In Nix expressions, this can be achieved by setting the attribute:

enableSharedExecutables = false;

That gives you a binary with statically linked Haskell libraries and dynamically linked system libraries.

To link both Haskell libraries and system libraries statically, the additional flags --ghc-option=-optl=-static --ghc-option=-optl=-pthread need to be used. In Nix, this is accomplished with:

configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];

It’s important to realize, however, that most system libraries in Nix are built as shared libraries only, i.e. there is just no static library available that Cabal could link!

By default GHC implements the Integer type using the GNU Multiple Precision Arithmetic (GMP) library. The implementation can be found in the integer-gmp package.

A potential problem with this is that GMP is licensed under the ​GNU Lesser General Public License (LGPL), a kind of “copyleft” license. According to the terms of the LGPL, paragraph 5, you may distribute a program that is designed to be compiled and dynamically linked with the library under the terms of your choice (i.e., commercially) but if your program incorporates portions of the library, if it is linked statically, then your program is a “derivative”–a “work based on the library”–and according to paragraph 2, section c, you “must cause the whole of the work to be licensed” under the terms of the LGPL (including for free).

The LGPL licensing for GMP is a problem for the overall licensing of binary programs compiled with GHC because most distributions (and builds) of GHC use static libraries. (Dynamic libraries are currently distributed only for OS X.) The LGPL licensing situation may be worse: even though ​The Glasgow Haskell Compiler License is essentially a “free software” license (BSD3), according to paragraph 2 of the LGPL, GHC must be distributed under the terms of the LGPL!

To work around these problems GHC can be build with a slower but LGPL-free alternative implemention for Integer called integer-simple.

To get a GHC compiler build with integer-simple instead of integer-gmp use the attribute: pkgs.haskell.compiler.integer-simple."${ghcVersion}". For example:

$ nix-build -E '(import <nixpkgs> {}).pkgs.haskell.compiler.integer-simple.ghc802'
...
$ result/bin/ghc-pkg list | grep integer
    integer-simple-0.1.1.1

The following command displays the complete list of GHC compilers build with integer-simple:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler.integer-simple
haskell.compiler.integer-simple.ghc7102  ghc-7.10.2
haskell.compiler.integer-simple.ghc7103  ghc-7.10.3
haskell.compiler.integer-simple.ghc722   ghc-7.2.2
haskell.compiler.integer-simple.ghc742   ghc-7.4.2
haskell.compiler.integer-simple.ghc763   ghc-7.6.3
haskell.compiler.integer-simple.ghc783   ghc-7.8.3
haskell.compiler.integer-simple.ghc784   ghc-7.8.4
haskell.compiler.integer-simple.ghc801   ghc-8.0.1
haskell.compiler.integer-simple.ghc802   ghc-8.0.2
haskell.compiler.integer-simple.ghcHEAD  ghc-8.1.20170106

To get a package set supporting integer-simple use the attribute: pkgs.haskell.packages.integer-simple."${ghcVersion}". For example use the following to get the scientific package build with integer-simple:

$ nix-build -A pkgs.haskell.packages.integer-simple.ghc802.scientific

Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python. We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code.

{ # ...

  toolz = buildPythonPackage rec {
    name = "toolz-${version}";
    version = "0.7.4";

    src = pkgs.fetchurl {
      url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
      sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
    };

    meta = {
      homepage = "http://github.com/pytoolz/toolz/";
      description = "List processing tools and functional utilities";
      license = licenses.bsd3;
      maintainers = with maintainers; [ fridh ];
    };
  };
}

with import <nixpkgs> {};

pkgs.python35Packages.buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.8.0";

  src = pkgs.fetchurl {
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
  };

  doCheck = false;

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}

with import <nixpkgs> {};

( let
    toolz = pkgs.python35Packages.buildPythonPackage rec {
      name = "toolz-${version}";
      version = "0.8.0";

      src = pkgs.fetchurl {
        url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
        sha256 = "e8451af61face57b7c5d09e71c0d27b8005f001ead56e9fdf470417e5cc6d479";
      };

      doCheck = false;

      meta = {
        homepage = "http://github.com/pytoolz/toolz/";
        description = "List processing tools and functional utilities";
      };
    };

  in pkgs.python35.withPackages (ps: [ps.numpy toolz])
).env

Our example, toolz, doesn’t have any dependencies on other Python packages or system libraries. According to the manual, buildPythonPackage uses the arguments buildInputs and propagatedBuildInputs to specify dependencies. If something is exclusively a build-time dependency, then the dependency should be included as a buildInput, but if it is (also) a runtime dependency, then it should be added to propagatedBuildInputs. Test dependencies are considered build-time dependencies.

The following example shows which arguments are given to buildPythonPackage in order to build datashape.

{ # ...

  datashape = buildPythonPackage rec {
    name = "datashape-${version}";
    version = "0.4.7";

    src = pkgs.fetchurl {
      url = "mirror://pypi/D/DataShape/${name}.tar.gz";
      sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
    };

    buildInputs = with self; [ pytest ];
    propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];

    meta = {
      homepage = https://github.com/ContinuumIO/datashape;
      description = "A data description language";
      license = licenses.bsd2;
      maintainers = with maintainers; [ fridh ];
    };
  };
}

We can see several runtime dependencies, numpy, multipledispatch, and dateutil. Furthermore, we have one buildInput, i.e. pytest. pytest is a test runner and is only used during the checkPhase and is therefore not added to propagatedBuildInputs.

In the previous case we had only dependencies on other Python packages to consider. Occasionally you have also system libraries to consider. E.g., lxml provides Python bindings to libxml2 and libxslt. These libraries are only required when building the bindings and are therefore added as buildInputs.

{ # ...

  lxml = buildPythonPackage rec {
    name = "lxml-3.4.4";

    src = pkgs.fetchurl {
      url = "mirror://pypi/l/lxml/${name}.tar.gz";
      sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
    };

    buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];

    meta = {
      description = "Pythonic binding for the libxml2 and libxslt libraries";
      homepage = http://lxml.de;
      license = licenses.bsd3;
      maintainers = with maintainers; [ sjourdois ];
    };
  };
}

In this example lxml and Nix are able to work out exactly where the relevant files of the dependencies are. This is not always the case.

The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats ("single", "double", "long-double"). The bindings need all three types, and therefore we add all three as buildInputs. The bindings don’t expect to find each of them in a different folder, and therefore we have to set LDFLAGS and CFLAGS.

{ # ...

  pyfftw = buildPythonPackage rec {
    name = "pyfftw-${version}";
    version = "0.9.2";

    src = pkgs.fetchurl {
      url = "mirror://pypi/p/pyFFTW/pyFFTW-${version}.tar.gz";
      sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
    };

    buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];

    propagatedBuildInputs = with self; [ numpy scipy ];

    # Tests cannot import pyfftw. pyfftw works fine though.
    doCheck = false;

    preConfigure = ''
      export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
      export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
    '';

    meta = {
      description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
      homepage = http://hgomersall.github.com/pyFFTW/;
      license = with licenses; [ bsd2 bsd3 ];
      maintainer = with maintainers; [ fridh ];
    };
  };
}

Note also the line doCheck = false;, we explicitly disabled running the test-suite.

with import <nixpkgs>;
with pkgs.python35Packages;

buildPythonPackage rec {
  name = "mypackage";
  src = ./path/to/package/source;
  propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
}

So far we discussed how you can use Python on Nix, and how you can develop with it. We’ve looked at how you write expressions to package Python packages, and we looked at how you can create environments in which specified packages are available.

At some point you’ll likely have multiple packages which you would like to be able to use in different projects. In order to minimise unnecessary duplication we now look at how you can maintain yourself a repository with your own packages. The important functions here are import and callPackage.

Earlier we created a Python environment using withPackages, and included the toolz package via a let expression. Let’s split the package definition from the environment definition.

We first create a function that builds toolz in ~/path/to/toolz/release.nix

{ pkgs, buildPythonPackage }:

buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl {
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}

It takes two arguments, pkgs and buildPythonPackage. We now call this function using callPackage in the definition of our environment

with import <nixpkgs> {};

( let
    toolz = pkgs.callPackage /path/to/toolz/release.nix {
      pkgs = pkgs;
      buildPythonPackage = pkgs.python35Packages.buildPythonPackage;
    };
  in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
).env

Important to remember is that the Python version for which the package is made depends on the python derivation that is passed to buildPythonPackage. Nix tries to automatically pass arguments when possible, which is why generally you don’t explicitly define which python derivation should be used. In the above example we use buildPythonPackage that is part of the set python35Packages, and in this case the python35 interpreter is automatically used.

Versions 2.7, 3.3, 3.4, 3.5 and 3.6 of the CPython interpreter are available as respectively python27, python33, python34, python35 and python36. The PyPy interpreter is available as pypy. The aliases python2 and python3 correspond to respectively python27 and python35. The default interpreter, python, maps to python2. The Nix expressions for the interpreters can be found in pkgs/development/interpreters/python.

All packages depending on any Python interpreter get appended out/{python.sitePackages} to $PYTHONPATH if such directory exists.

Python libraries and applications that use setuptools or distutils are typically build with respectively the buildPythonPackage and buildPythonApplication functions. These two functions also support installing a wheel.

All Python packages reside in pkgs/top-level/python-packages.nix and all applications elsewhere. In case a package is used as both a library and an application, then the package should be in pkgs/top-level/python-packages.nix since only those packages are made available for all interpreter versions. The preferred location for library expressions is in pkgs/development/python-modules. It is important that these packages are called from pkgs/top-level/python-packages.nix and not elsewhere, to guarantee the right version of the package is built.

Based on the packages defined in pkgs/top-level/python-packages.nix an attribute set is created for each available Python interpreter. The available sets are

and the aliases

{ # ...

  twisted = buildPythonPackage {
    name = "twisted-8.1.0";

    src = pkgs.fetchurl {
      url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
      sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
    };

    propagatedBuildInputs = [ self.ZopeInterface ];

    meta = {
      homepage = http://twistedmatrix.com/;
      description = "Twisted, an event-driven networking engine written in Python";
      license = stdenv.lib.licenses.mit;
    };
  };
}

with import <nixpkgs> {};

python.buildEnv.override {
  extraLibs = [ pkgs.pythonPackages.pyramid ];
  ignoreCollisions = true;
}

/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env

with import <nixpkgs> {};

(python3.buildEnv.override {
  extraLibs = with python3Packages; [ numpy requests2 ];
}).env

with import <nixpkgs> {};

python.withPackages (ps: [ps.pyramid])

with import <nixpkgs> {};

python3.withPackages (ps: [ps.pyramid])

with import <nixpkgs> {};

(python33.withPackages (ps: [ps.numpy ps.requests2])).env

Development or editable mode is supported. To develop Python packages buildPythonPackage has additional logic inside shellPhase to run pip install -e . --prefix $TMPDIR/for the package.

Warning: shellPhase is executed only if setup.py exists.

Given a default.nix:

with import <nixpkgs> {};

buildPythonPackage { name = "myproject";

buildInputs = with pkgs.pythonPackages; [ pyramid ];

src = ./.; }

Running nix-shell with no arguments should give you the environment in which the package would be built with nix-build.

Shortcut to setup environments with C headers/libraries and python packages:

nix-shell -p pythonPackages.pyramid zlib libjpeg git

Note: There is a boolean value lib.inNixShell set to true if nix-shell is invoked.

Packages inside nixpkgs are written by hand. However many tools exist in community to help save time. No tool is preferred at the moment.

Python 2.7, 3.5 and 3.6 are now built deterministically and 3.4 mostly. Minor modifications had to be made to the interpreters in order to generate deterministic bytecode. This has security implications and is relevant for those using Python in a nix-shell.

When the environment variable DETERMINISTIC_BUILD is set, all bytecode will have timestamp 1. The buildPythonPackage function sets DETERMINISTIC_BUILD=1 and PYTHONHASHSEED=0. Both are also exported in nix-shell.

As explained in the user’s guide installing individual Python packages imperatively with nix-env -i or declaratively in environment.systemPackages is not supported. However, it is possible to install a Python environment with packages (python.buildEnv).

In the following examples we create an environment with Python 3.5, numpy and ipython. As you might imagine there is one limitation here, and that’s you can install only one environment at a time. You will notice the complaints about collisions when you try to install a second environment.

with import <nixpkgs> {};

pkgs.python35.withPackages (ps: with ps; [ numpy ipython ])

nix-env -if build.nix

{ # ...

  packageOverrides = pkgs: with pkgs; {
    myEnv = python35.withPackages (ps: with ps; [ numpy ipython ]);
  };
}

nix-env -iA nixpkgs.myEnv

{ # ...

  environment.systemPackages = with pkgs; [
    (python35.withPackages(ps: with ps; [ numpy ipython ]))
  ];
}

Consider the packages A and B that depend on each other. When packaging B, a solution is to override package A not to depend on B as an input. The same should also be done when packaging A.

We can override the interpreter and pass packageOverrides. In the following example we rename the pandas package and build it.

with import <nixpkgs> {};

let
  python = let
    packageOverrides = self: super: {
      pandas = super.pandas.override {name="foo";};
    };
  in pkgs.python35.override {inherit packageOverrides;};

in python.pkgs.pandas

Using nix-build on this expression will build the package pandas but with the new name foo.

All packages in the package set will use the renamed package. A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of django and scipy. In the following example we use a different version of scipy and create an environment that uses it. All packages in the Python package set will now use the updated scipy version.

with import <nixpkgs> {};

( let
    packageOverrides = self: super: {
      scipy = super.scipy_0_17;
    };
  in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
).env

The requested package blaze depends on pandas which itself depends on scipy.

If you want the whole of Nixpkgs to use your modifications, then you can use overlays as explained in this manual. In the following example we build a inkscape using a different version of numpy.

let
  pkgs = import <nixpkgs> {};
  newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
    python27 = let
      packageOverrides = self: super: {
        numpy = super.numpy_1_10;
      };
    in pkgssuper.python27.override {inherit packageOverrides;};
  } ) ]; };
in newpkgs.inkscape

Executing python setup.py bdist_wheel in a nix-shellfails with

ValueError: ZIP does not support timestamps before 1980

This is because files are included that depend on items in the Nix store which have a timestamp of, that is, it corresponds to January the 1st, 1970 at 00:00:00. And as the error informs you, ZIP does not support that. The command bdist_wheel takes into account SOURCE_DATE_EPOCH, and nix-shell sets this to 1. By setting it to a value corresponding to 1980 or later, or by unsetting it, it is possible to build wheels.

Use 1980 as timestamp:

nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"

or the current time:

nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"

or unset:

nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"

If you get the following error:

could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
Permission denied

This is a known bug in setuptools. Setuptools install_data does not respect --prefix. An example of such package using the feature is pkgs/tools/X11/xpra/default.nix. As workaround install it as an extra preInstall step:

${python.interpreter} setup.py install_data --install-dir=$out --root=$out
sed -i '/ = data\_files/d' setup.py

On most operating systems a global site-packages is maintained. This however becomes problematic if you want to run multiple Python versions or have multiple versions of certain libraries for your projects. Generally, you would solve such issues by creating virtual environments using virtualenv.

On Nix each package has an isolated dependency tree which, in the case of Python, guarantees the right versions of the interpreter and libraries or packages are available. There is therefore no need to maintain a global site-packages.

If you want to create a Python environment for development, then the recommended method is to use nix-shell, either with or without the python.buildEnv function.

This is an example of a default.nix for a nix-shell, which allows to consume a virtualenv environment, and install python modules through pip the traditional way.

Create this default.nix file, together with a requirements.txt and simply execute nix-shell.

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    # these packages are required for virtualenv and pip to work:
    #
    python27Full
    python27Packages.virtualenv
    python27Packages.pip
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.tx require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    libxml2
    libxslt
    libzip
    stdenv
    zlib ];
  src = null;
  shellHook = ''
  # set SOURCE_DATE_EPOCH so that we can use python wheels
  SOURCE_DATE_EPOCH=$(date +%s)
  virtualenv --no-setuptools venv
  export PATH=$PWD/venv/bin:$PATH
  pip install -r requirements.txt
  '';
}

Note that the pip install is an imperative action. So every time nix-shell is executed it will attempt to download the python modules listed in requirements.txt. However these will be cached locally within the virtualenv folder and not downloaded again.

If you need to change a package’s attribute(s) from configuration.nix you could do:

  nixpkgs.config.packageOverrides = superP: {
    pythonPackages = superP.pythonPackages.override {
      overrides = self: super: {
        bepasty-server = super.bepasty-server.overrideAttrs ( oldAttrs: {
          src = pkgs.fetchgit {
            url = "https://github.com/bepasty/bepasty-server";
            sha256 = "9ziqshmsf0rjvdhhca55sm0x8jz76fsf2q4rwh4m6lpcf8wr0nps";
            rev = "e2516e8cf4f2afb5185337073607eb9e84a61d2d";
          };
        });
      };
    };
  };

If you are using the bepasty-server package somewhere, for example in systemPackages or indirectly from services.bepasty, then a nixos-rebuild switch will rebuild the system but with the bepasty-server package using a different src attribute. This way one can modify python based software/libraries easily. Using self and super one can also alter dependencies (buildInputs) between the old state (self) and new state (super).

Following rules are desired to be respected: