The basic facts

Over the course of this post, the focus will alternate between Distributive and Representable. In this first section, we will review the basic definitions and laws upon which we will build. Following that, we will work on both ends, aiming at making the classes meet in the middle.

Distributive

Let’s begin by jotting down a few basic facts about Distributive. Here is a minimalistic definition of the class:

class Functor g => Distributive g where
    distribute :: Functor f => f (g a) -> g (f a)

(In what follows, when used as a placeholder name for a functor, g will always stand for a distributive or representable functor, while f will typically stand for the other functor involved in distribute.)

distribute is dual to sequenceA; accordingly, we will adopt the duals of the Traversable laws: ¹

• Identity:

  fmap runIdentity . distribute = runIdentity

• Composition:

  fmap getCompose . distribute
      = distribute . fmap distribute . getCompose

• Naturality (ensured by parametricity):

  -- For any natural transformation t
  -- t :: (Functor f1, Functor f2) => forall x. f1 x -> f2 x
  fmap t . distribute = distribute . t

  This naturality law is stronger than its Traversable counterpart. The Applicative constraint in sequenceA means only natural transformations between applicative functors that preserve pure and (<*>) are preserved by distribute. In contrast, distribute is oblivious to any specifics of f1 and f2 functor, and so any natural transformation will do.

Homogeneous pairs are one example of a distributive functor:

data Duo a = Duo a a
    deriving (Eq, Ord, Show, Functor, Foldable, Traversable)

fstDuo, sndDuo :: Duo a -> a
fstDuo (Duo x _) = x
sndDuo (Duo _ y) = y

instance Distributive Duo where
    distribute m = Duo (fstDuo <$> m) (sndDuo <$> m)

Duo will be used in this post as a running example whenever a concrete illustration of Distributive and adjacent classes is called for. For the moment, here is a simple demonstration of distribute @Duo in action. It illustrates the zip-like flavour of distribute, which is shared by the closely related collect and cotraverse from Data.Distributive:

names :: [Duo String]
names =
    [ Duo "Alex" "Lifeson"
    , Duo "Geddy" "Lee"
    , Duo "Neil" "Peart"
    ]

ghci> distribute names
Duo ["Alex","Geddy","Neil"] ["Lifeson","Lee","Peart"]

The function functor is a very important example of Distributive. Consider the following combinator:

flap :: Functor f => f (r -> a) -> r -> f a
flap m = \r -> (\f -> f r) <$> m

It changes a f (r -> a) functorial value into a r -> f a function, which feeds its argument to all of the available r -> a functions. flap is a lawful implementation of distribute:

instance Distributive ((->) r) where
    distribute = flap

ghci> distribute [(*3), (+7), (^2)] 8
[24,15,64]

flap will be used in this post as a synonym for distribute @((->) _) whenever convenient, or necessary to avoid circularity. ²

Representable

As for Representable, for our immediate purposes it suffices to characterise it as a class for functors isomorphic to functions:

class Functor g => Representable g where
    type Rep g
    tabulate :: (Rep g -> a) -> g a
    index :: g a -> Rep g -> a

Here, Rep g is some concrete type such that tabulate and index witness an isomorphism between Rep g -> a and g a. Accordingly, the laws are:

• Home direction (from g a and back):

  tabulate . index = id

• Away direction (to g a and back):

  index . tabulate = id

Duo can be given a Representable instance: pick Bool (or any other type with two inhabitants) as Rep g, and associate each possible value with a component of the pair:

instance Representable Duo where
    type Rep Duo = Bool
    tabulate f = Duo (f False) (f True)
    index (Duo x y) = \case
        False -> x
        True -> y

In order to treat the two classes in an even-handed way, I have opted to leave out the Distributive g => Representable g relationship that exists in the Data.Functor.Rep version of Representable . In any case, every representable is indeed distributive, with a default definition of distribute which uses the isomorphism to delegate to flap (that is, distribute for functions):

distributeRep :: (Representable g, Functor f) => f (g a) -> g (f a)
distributeRep = tabulate . flap . fmap index

The lawfulness of distributeRep follows from the lawfulness of flap. ³

Our ultimate aim here is to go the other way around, from Distributive to Representable.

No need to choose

If we are to start from Distributive, though, there is a pretty fundamental difficulty: setting up a Representable g instance requires picking a suitable Rep g, and there is nothing in Distributive that could possibly correspond to such a choice. That being so, we will spend some more time contemplating Representable, looking for a way to somehow obviate the need for specifying Rep g.

askRep

Let’s have another look at the type of tabulate:

tabulate :: Representable g => (Rep g -> a) -> g a

tabulate is a natural transformation from the function functor ((->) (Rep g) to g. Now, all natural transformations from a function functor have the form: ⁴

-- For some type R, functor G, and any
t :: forall x. (R -> x) -> G x
-- There is a
w :: G R
-- Such that
t f = f <$> w
w = t id

In words, the natural transformation must amount to mapping the function over some functorial value. In our case, t is tabulate; as for w, we will call it askRep, which is the name it goes by in Data.Functor.Rep. ⁵. That being so, we have:

askRep :: Representable g => g (Rep g)
askRep = tabulate id

tabulate f = f <$> askRep

The Representable laws can be recast in terms of askRep and index. Here is the home direction of the isomorphism:

tabulate . index = id
tabulate (index u) = u
index u <$> askRep = u

That is, we can reconstruct any u :: g a by taking askRep and replacing every Rep g provided by it with the a value that applying index u on it gives us.

It is worth noting that index u <$> askRep = u also tells us that for any u :: g a there is a function (namely, index u) which will change askRep into u through fmap. That largely corresponds to the intuition that a representable functor must have a single shape.

The away direction of the isomorphism becomes:

index . tabulate = id
index (f <$> askRep) = f
-- index is a natural transformation
f <$> index askRep = f
-- fmap @((->) _) = (.)
f . index askRep = f
-- In particular, suppose f = id
-- (note that this step is reversible)
index askRep = id

Intuitively, if we think of Rep g values as corresponding to positions in the g shape that can be queried through index, index askRep = id tells us that each and every Rep g will be found in askRep occupying the position it corresponds to. For example, with the Representable instance from the previous section, askRep @Duo looks like this:

ghci> askRep @Duo
Duo False True

Lastly, we can also express distributeRep in terms of askRep:

distributeRep m
tabulate (flap (index <$> m))
flap (index <$> m) <$> askRep
(\r -> (\f -> f r) <$> (index <$> m)) <$> askRep
(\r -> (\u -> index u r) <$> m) <$> askRep

distributeRep m = (\r -> (\u -> index u r) <$> m) <$> askRep

That is, replace every Rep g in askRep with the result of using it to index every g a in m.

Extracting and revealing

Now let’s direct our attention to index:

index :: Representable g => g a -> Rep g -> a

Flipping index gives us:

fromRep :: Representable g => Rep g -> g a -> a
fromRep r = \u -> index u r

fromRep converts a Rep g into what I will call a polymorphic extractor, of type forall a. g a -> a, which gives us a out of g a. The existence of fromRep is quite suggestive. Since forall a. g a -> a doesn’t use Rep g, finding an inverse to fromRep, and thus showing those two types are isomorphic, might give us a way to work with Representable without relying on Rep g.

How might we go about converting a polymorphic extractor into a Rep g value? To do it in a non-trivial way , we will need a g (Rep g) source of Rep g on which we can use the extractor. Considering the discussion in the previous subsection, askRep looks like a reasonable option:

toRep :: Representable g => (forall x. g x -> x) -> Rep g
toRep p = p askRep

Now let’s check if fromRep and toRep are indeed inverses, beginning with the toRep . fromRep direction:

toRep . fromRep
(\p -> p askRep) . (\r -> \u -> index u r)
\r -> (\u -> index u r) askRep
\r -> index askRep r
-- index askRep = id
id

We can proceed similarly with fromRep . toRep:

fromRep . toRep
(\r -> \u -> index u r) . (\p -> p askRep)
\p -> \u -> index u (p askRep)

To simplify this further, we can note that a polymorphic extractor forall x. g x -> x amounts to natural transformation from g to Identity. That being so, we have, for any extractor p and any f:

f . p = p . fmap f

The above is the usual naturality property, fmap f . p = p . fmap f, except that, to account for the omission of the Identity newtype boilerplate, fmap @Identity has been replaced on the left-hand side by plain function application. We can now carry on:

\p -> \u -> index u (p askRep)
\p -> \u -> p (index u <$> askRep)
-- index u <$> askRep = u
\p -> \u -> p u
id

And there it is: for any Representable, Rep g must be isomorphic to forall x. g x -> x. That being so, we can use forall x. g x -> x as a default Rep g that can be specified in terms of g alone. The change of perspective can be made clearer by setting up an alternative class:

type Pos g = forall x. g x -> x

elide :: g a -> Pos g -> a
elide u = \p -> p u

class Functor g => Revealable g where
    reveal :: (Pos g -> a) -> g a
    chart :: g (Pos g)

    reveal e = e <$> chart
    chart = reveal id
    {-# MINIMAL reveal | chart #-}

Both the arrangement of those definitions and my idiosyncratic choice of names call for some explanation:

• Pos g is a synonym for the type of polymorphic extractors. The name Pos is short for “position”, and is meant to allude to the intuition that an extractor picks a value from some specific position in a g-shaped structure.

• elide corresponds to index, defined in such a way that fromRep = id. Since all it does is applying a Pos g extractor, on its own it doesn’t require any constraints on g. The choice of name is motivated by how elide hides the g shape, in that that the only information about u :: g a that can be recovered from elide u are the a values that a Pos g extractor can reach.

• reveal, in turn, corresponds to tabulate, and is the inverse of elide. If g is Representable, the g shape can be reconstituted with no additional information, and so it is possible to undo the hiding performed by elide.

• chart corresponds to askRep, with it and reveal being interdefinable. In particular, chart can be used to reveal the g a that corresponds to a Pos g -> a function by providing the means to reach every position in the g shape. ⁶

Here is the Duo instance of Revealable. Note how each position in chart holds its own extractor:

instance Revealable Duo where
    reveal e = Duo (e fstDuo) (e sndDuo)
    chart = Duo fstDuo sndDuo

distribute can be implemented for Revealable in a way completely analogous to how it was done for Representable:

distributeRev :: (Revealable g, Functor f) => f (g a) -> g (f a)
distributeRev = reveal . flap . fmap elide

Or, in terms of chart:

distributeRev m = (\p -> p <$> m) <$> chart

That is, distributeRev m amounts to mapping every extractor in chart over m.

As for the laws, just like we were able to choose between expressing the Representable isomorphism directly, via tabulate, or indirectly via askRep, here we can use either reveal or chart:

reveal . elide = id
-- Or, equivalently
elide u <$> chart = u

elide . reveal = id
-- Or, equivalently
p chart = p

With Revealable, though, we can streamline things by showing p chart = p follows from elide u <$> chart = u. The proof relies on the naturality of the polymorphic extractors:

elide u <$> chart = u
-- Apply some p :: Pos g to both sides
p (elide u <$> chart) = p u
-- p is natural
elide u (p chart) = p u
-- elide u p = p u
(p chart) u = p u
-- u is arbitrary
p chart = p

That being so, elide u <$> chart = u is the only law we need to characterise Revealable. Since elide does not depend on the Revealable instance, we might as well inline its definition, which leaves us with:

(\p -> p u) <$> chart = u

I suggest calling it the law of extractors: it tells us that the extractors provided by chart suffice to reconstitute an arbitrary g a value.

Revisiting Distributive

In Revealable, we have a class equivalent to Representable which doesn’t rely on the Rep type family. That makes it feasible to continue our investigation by attempting to show that every Distributive functor is Revealable.

Natural wonders

Naturality laws and parametricity properties not infrequently have interesting consequences that seem to us as hidden in plain sight. Considering the increased strength of Distributive’s naturality law relative to its Traversable counterpart and the important role naturality properties had in setting up Revealable, resuming our work on Distributive from the naturality law sounds like a reasonable bet:

-- For any natural transformation t
-- t :: (Functor f1, Functor f2) => forall x. f1 x -> f2 x
fmap t . distribute = distribute . t

In particular, suppose f1 is a function functor:

-- t :: Functor f => forall x. (r -> x) -> f x
t <$> distribute f = distribute (t f)

Now, by the same argument used back when we defined askRep, t must have the form:

-- m :: f r
t f = f <$> m

Therefore:

(\p -> p <$> m) <$> distribute f = distribute (f <$> m)

In particular, suppose f = id. We then end up with an specification of distribute in terms of distribute id:

(\p -> p <$> m) <$> distribute id = distribute m

distribute id has the following type:

ghci> :t distribute id
distribute id :: Distributive g => g (g a -> a)

This looks a lot like something that holds extractors, and the specification itself mirrors the definition of distributeRev in terms of chart. As a preliminary check, distribute @Duo id holds fstDuo and sndDuo on their respective positions, exactly like chart @Duo:

distribute @Duo id
Duo (id <$> fstDuo) (id <$> sndDuo)
Duo fstDuo sndDuo

Given the clear resemblance, I will optimistically refer to distribute id as chartDist:

chartDist :: Distributive g => g (g a -> a)
chartDist = distribute id

We therefore have:

distribute m = (\p -> p <$> m) <$> chartDist

Now suppose m = Identity u for some u :: g a, and invoke the identity law:

distribute (Identity u) = (\p -> p <$> Identity u) <$> chartDist
distribute (Identity u) = (\p -> Identity (p u)) <$> chartDist
runIdentity <$> distribute (Identity u)
    = runIdentity <$> ((\p ->Identity (p u)) <$> chartDist)
runIdentity <$> distribute (Identity u)
    = (\p -> runIdentity (Identity (p u))) <$> chartDist
runIdentity <$> distribute (Identity u)
    = (\p -> p u) <$> chartDist
-- By the identity law
runIdentity (Identity u) = (\p -> p u) <$> chartDist
u = (\p -> p u) <$> chartDist

We therefore have a Distributive version of the law of extractors, with chartDist playing the role of chart. It is also possible to turn things around and obtain the identity law from this law of extractors:

(\p -> p u) <$> chartDist = u
runIdentity . Identity . (\p -> p u) <$> chartDist = u
runIdentity . (\p -> Identity (p u)) <$> chartDist = u
runIdentity . (\p -> p <$> Identity u) <$> chartDist = u
-- distribute m = (\p -> p <$> m) <$> chartDist
runIdentity <$> distribute (Identity u) = u
runIdentity <$> distribute (Identity u) = runIdentity (Identity u)
fmap runIdentity . distribute = runIdentity

These are auspicious results. Given that the law of extractors is enough to establish an implementation of chart as lawful, and that there can’t be multiple distinct lawful implementations of distribute ⁷, all we need to do is to identify chartDist with chart.

The roadblock, and a detour

Identifying chartDist with chart, however, is not trivial. As similar as chart and chartDist might feel like, their types differ in an insurmountable way:

chart @G     :: G (forall a. G a -> a)  -- G (Pos G)
chartDist @G :: forall a. G (G a -> a)

In particular:

• The a in forall a. G (G a -> a) can be directly specialised to a concrete choice of a, and, as far as the specialised type G (G A -> A) is concerned, it is conceivable that the involved G A -> A functions might not be natural in A.

• Accordingly, a rank-2 function that takes a Pos G, such as the argument to reveal, can be mapped over chart, but not chartDist.

• There is no way to obtain the impredicative type of chart, or the rank-3 type of reveal, through distribute.

To put it in another way, chartDist doesn’t have a type strong enough to, on its own, ensure that it provides natural, polymorphic extractors, and Distributive is not enough to implement a chart which provides such guarantees.

Still, not all is lost. If there is a way to use the laws of Distributive to show that the extractors of chartDist are natural, we should be able to claim chart and chartDist are morally the same, providing the same extractors with subtly different types.

(Meta note: while I believe the following argument suffices for the task at hand, it is not as crystalline as the derivations elsewhere in this post. Upgrading it to a proper proof will probably require some tricky parametricity maneuver which I haven’t managed to fully figure out yet.)

Let’s turn to the composition law, the one we haven’t touched so far:

fmap getCompose . distribute = distribute . fmap distribute . getCompose

That is, given some m :: Compose fo fi (g a) (“o” is for outer, and “i” for inner):

getCompose <$> distribute m = distribute (distribute <$> getCompose m)

Let’s use distribute m = (\p -> p <$> m) <$> chartDist on the left-hand side, and on the outer distribute on the right-hand side:

getCompose <$> ((\p -> p <$> m) <$> chartDist)
    = (\q -> q <$> (distribute <$> getCompose m)) <$> chartDist

Note that the left-hand side chartDist has type g (g a -> a), while the right-hand side one has type g (g (fi a) -> fi a). Since we can’t take for granted that the extractors provided by them (which are bound to p and q, respectively) are natural, it is important to keep track of this difference.

Tidying the equation a little further, we get:

getCompose <$> ((\p -> p <$> m) <$> chartDist)
    = (\q -> q <$> (distribute <$> getCompose m)) <$> chartDist
(\p -> getCompose (p <$> m)) <$> chartDist
    = (\q -> q . distribute <$> getCompose m) <$> chartDist
(\p -> fmap p <$> getCompose m) <$> chartDist
    = (\q -> q . distribute <$> getCompose m) <$> chartDist

On either side of the equation, we have fmap being used to obtain a g (fo (fi a)) result. That being so, any fo (fi a) value that, thanks to fmap, shows up in the left-hand side must also show up in the right-hand side. More precisely, given any p :: g a -> a drawn from chartDist on the left-hand side, there must be some q :: g (fi a) -> fi a drawn from the chartDist on the right hand side such that…

fmap p <$> getCompose m = q . distribute <$> getCompose m

… and vice versa. That allows us to reason about p and q, which amount to the extractors drawn from chartDist we are interested in.

As neither p nor q involve fo, and the equation must hold for all choices of fo, we can freely consider the case in which it is Identity, or anything else that has an injective fmap. If fmap is injective, the equation further simplifies to:

fmap p = q . distribute

Now, fmap p :: fi (g a) -> fi a cannot affect the fi shape; therefore, the same holds for q . distribute :: fi (g a) -> fi a. distribute :: fi (g a) -> g (fi a) is natural in fi, and so it, too, can’t affect the fi shape. It follows that q :: g (fi a) -> fi a is also unable to affect the fi shape.

Zooming back out, we have just established that, if the composition law holds, chartDist :: g (g (fi a) -> fi a) only provides extractors that preserve the fi shape. chartDist, however, is defined as distribute id :: forall b. g (g b -> b), which is fully polymorphic on the element type b. That being so, if there is a way for distribute id to somehow produce non-natural extractors, it cannot possibly rely in any way about the specifics of b. That, in particular, rules out any means of, given b ~ fi a for some functor fi, producing just non-natural extractors that preserve the fi shape: such a distinction cannot be expressed. We must conclude, therefore, that if the composition law holds chartDist can only provide natural extractors, as we hoped to show.

The converse of this conclusion, by the way, also holds: assuming the identity law holds, if all q drawn from chartDist are natural, the composition law must hold. To show that, we can use the fact that, for a natural q :: forall x. g x -> x, q chartDist = q holds, just like it does for chart:

(\p -> p u) <$> chartDist = u
q ((\p -> p u) <$> chartDist)) = q u
-- Since q is natural, q . fmap f = f . q
(\p -> p u) (q chartDist) = q u
(q chartDist) u = q u
q chartDist = q

As a consequence, q . distribute = fmap q:

q (distribute m)
q ((\p -> p <$> m) <$> chartDist)
-- q is natural
(\p -> p <$> m) (q chartDist)
(\p -> p <$> m) q
q <$> m

We can now return to the rearranged version of the composition law we were dealing with in the preceding argument, this time without taking it for granted:

(\p -> fmap p <$> getCompose m) <$> chartDist
    = (\q -> q . distribute <$> getCompose m) <$> chartDist

By the above, however, if q is natural the right-hand side amounts to…

(\q -> fmap q <$> getCompose m) <$> chartDist

… which is the same as the left-hand side.

In summary

After quite a long ride, we have managed to shed some light on the connection between Distributive and Representable:

• Every Distributive is indeed Representable, even though, as expected, Representable cannot be implemented in terms of distribute.

• The connection is mediated by choosing forall x. g x -> x, the type of polymorphic extractors, as a default representation, encoded here as the Revealable class. It can then be shown that this representation is mirrored in Distributive by chartDist = distribute id :: Distributive g => g (g a -> a), which gives a corresponding characterisation of Distributive in terms of extractors.

• The single-shapedness characteristic of both distributive and representable functors follows from the identity law of Distributive.

• The composition law plays an important, if unobvious, role in the connection, as it ensures the naturality of the extractors provided by chartDist, a property that can’t be established on the basis of the involved types.

The Select loophole

There is one aspect of our investigation that is worth a closer look. All the concern with establishing that chartDist can only provide natural extractors, which kept us busy for a good chunk of the previous section, might have felt surprising. chartDist, after all…

chartDist :: forall g a. Distributive g => g (g a -> a)

… is fully polymorphic in a, and therefore its definition cannot rely on anything specific about a. That being so, it may seem outlandish to suppose that specialising chartDist to, say, g (g Integer -> Integer) might somehow bring forth non-natural g Integer -> Integer extractors that perform Integer-specific operations.

To illustrate why the naturality of extractors is, in fact, a relevant issue, let’s consider the curious case of Select:

-- A paraphrased, non-transformer version of Select.
newtype Select r a = Select { runSelect :: (a -> r) -> a }

instance Functor (Select r) where
    fmap f u = Select $ \k -> f (u `runSelect` \a -> k (f a))

(A Select r a value can be thought of as a way to choose an a value based on some user-specified criterion, expressed as an a -> r function.)

Corner cases such as r ~ () aside, Select r cannot be Representable, as that would require it to be isomorphic to a function functor; that being so, it should be similarly ill-suited for Distributive. In spite of that, there is a nontrivial implementation of a Select r combinator with the type chartDist would have: ⁸

chartSelect :: Select r (Select r a -> a)
chartSelect = Select $ \k -> \u -> u `runSelect` \a -> k (const a)

What’s more, chartSelect follows the law of extractors:

-- Goal:
(\p -> p u) <$> chartSelect = u
-- LHS
(\p -> p u) <$> chartSelect
(\p -> p u) <$> Select $ \k -> \u -> u `runSelect` \a -> k (const a)
Select $ \k' ->
    (\p -> p u) (\u -> u `runSelect` \a -> k' ((\p -> p u) (const a)))
Select $ \k' -> u `runSelect` \a -> k' ((\p -> p u) (const a))
Select $ \k' -> u `runSelect` \a -> k' (const a u)
Select $ \k' -> u `runSelect` \a -> k' a
u  -- LHS = RHS

That means the distribute candidate we get out of chartSelect…

nonDistribute :: Functor f => f (Select r a) -> Select r (f a)
nonDistribute m = Select $
    \k -> (\u -> u `runSelect` \a -> k (a <$ m)) <$> m

… follows the identity law. As Select r is not supposed to be Distributive, we expect nonDistribute to break the composition law, and that is indeed what happens. ⁹

Now, by the earlier arguments about the naturality of extractors, if a candidate implementation of chartDist follows the extractors law and only provides natural extractors, the corresponding distribute must follow the composition law. Since chartSelect follows the extractors law but doesn’t give rise to a lawful distribute, we must conclude that it provides non-natural extractors. How does that come to pass?

Every criterion function k :: a -> r gives rise to a non-natural extractor for Select r a, namely \u -> u `runSelect` k :: Select a r -> a. chartSelect indirectly makes all these non-natural extractors available through its own criterion argument, the k that shows up in its definition. (How the encoding works can be seen in the verification above of the law of extractors: note how performing the fmap between the third and fourth lines of the proof requires replacing k :: (Select r a -> a) -> r with k' :: a -> r.)

Non-naturality sneaking into chartSelect has to do with Select r not being a strictly positive functor; that is, it has an occurrence of the element type variable, a, to the left of a function arrow. ¹⁰ The lack of strict positivity creates a loophole, through which things can be incorporated to a Select r a value without being specified. It is a plausible conjecture that the composition law of Distributive is a way of ruling out functors that aren’t strictly positive, with lack of strict positivity being the only possible source of non-naturality in chartDist, and any non-trivial lack of strict positivity leading to non-naturality and the composition law being broken. ¹¹

Further reading

There are other interesting ways of approaching Distributive and Representable that I haven’t covered here to avoid making this post longer than it already is. Here are a few suggestions for further reading:

• Chris Penner’s Adjunctions and Battleship post is a fine introduction to Adjunction, the class for Hask-Hask adjunctions, which provides an alternative encoding of Representable.

• The following Stack Overflow answers by Conor McBride on Naperian functors, “Naperian” here being an alternative name for Representable:

  • Which Haskell Functors are equivalent to the Reader functor, which introduces Naperian functors in a style reminiscent of the askRep-centric formuation of Representable discussed here.

  • Writing cojoin or cobind for n-dimensional grid type, which includes an outline of how Naperian functors are handled by container theory.

On a final note, there is a reworking of Representable being developed as part of a potential future release of the distributive package. It aims at unifying the presentations of distributive into a single class that fits equally well the various use cases. An overview of how this new formulation could be a nice topic for a future, follow-up post.

Besides Gabriella’s post, which is an excellent introduction to Divisible, I recommend as background reading Tom Ellis’ Alternatives convert products to sums, which conveys the core intuition about monoidal functor classes in an accessible manner. There is a second post by Tom, The Mysterious Incomposability of Decidable, that this article will be in constant dialogue with, in particular as a source of examples. From now on I will refer to it as “the Decidable post”. Thanks to Gabriella and Tom for inspiring the writing of this article.

For those of you reading with GHCi on the side, the key general definitions in this post are available from this .hs file.

Applicative

As I hinted at the introduction, this post is not solely about Divisible, but more broadly about monoidal functor classes. To start from familiar ground and set up a reference point, I will first look at the best known of those classes, Applicative. We won’t, however, stick with the usual presentation of Applicative in terms of (<*>), as it doesn’t generalise to the other classes we’re interested in. Instead, we will switch to the monoidal presentation: ¹

zipped :: Applicative f => f a -> f b -> f (a, b)
zipped = liftA2 (,)

-- An operator spelling, for convenience.
(&*&) :: Applicative f => f a -> f b -> f (a, b)
(&*&) = zipped
infixr 5 &*&

unit :: Applicative f => f ()
unit = pure ()

-- Laws:

-- unit &*& v ~ v
-- u &*& unit ~ u
-- (u &*& v) &*& w ~ u &*& (v &*& w)

(Purely for the sake of consistency, I will try to stick to the Data.Functor.Contravariant.Divisible naming conventions for functions like zipped.)

The matter with (<*>) (and also liftA2) that stops it from being generalised for our purposes is that it leans heavily on the fact that Hask is a Cartesian closed category, with pairs as the relevant product. Without that, the currying and the partial application we rely on when writing in applicative style become unfeasible.

While keeping ourselves away from (<*>) and liftA2, we can recover, if not the full flexibility, the power of applicative style with a variant of liftA2 that takes an uncurried function:

lizip :: Applicative f => ((a, b) -> c) -> f a -> f b -> f c
lizip f u v = fmap f (zipped u v)

(That is admittedly a weird name; all the clashing naming conventions around this topic has left me with few good options.)

On a closing note for this section, my choice of operator for zipped is motivated by the similarity with (&&&) from Control.Arrow:

(&&&) :: Arrow p => p a b -> p a c -> p a (b, c)

In particular, (&*&) for the function Applicative coincides with (&&&) for the function Arrow.

Leaning on connections like this one, I will use Control.Arrow combinators liberally, beginning with the definition of the following two convenience functions that will show up shortly:

dup :: a -> (a, a)
dup = id &&& id

forget :: Either a a -> a
forget = id ||| id

Divisible

As summarised at the beginning of the Decidable post, while Applicative converts products to products covariantly, Divisible converts products to products contravariantly. From that point of view, I will take divided, the counterpart to zipped, as the fundamental combinator of the class:

-- This is the divided operator featured on Gabriella's post, soon to
-- become available from Data.Functor.Contravariant.Divisible
(>*<) :: Divisible k => k a -> k b -> k (a, b)
(>*<) = divided
infixr 5 >*<

-- Laws:

-- conquered >*< v ~ v
-- u >*< conquered ~ u
-- (u >*< v) >*< w ~ u >*< (v >*< w)

Recovering divide from divided is straightforward, and entirely analogous to how lizip can be obtained from zipped:

divide :: Divisible k => (a -> (b, c)) -> k b -> k c -> k a
divide f = contramap f (divided u v)

Lessened currying aside, we might say that divide plays the role of liftA2 in Divisible.

It’s about time for an example. For that, I will borrow the one from Gabriella’s post:

data Point = Point { x :: Double, y :: Double, z :: Double }
    deriving Show

nonNegative :: Predicate Double
nonNegative = Predicate (0 <=)

-- (>$<) = contramap
nonNegativeOctant :: Predicate Point
nonNegativeOctant =
    adapt >$< nonNegative >*< nonNegative >*< nonNegative
    where
    adapt = x &&& y &&& z

The slight distortion to Gabriella’s style in using (&&&) to write adapt pointfree is meant to emphasise how that deconstructor can be cleanly assembled out of the component projection functions x, y and z. Importantly, that holds in general: pair-producing functions a -> (b, c) are isomorphic (a -> b, a -> c) pairs of projections. That gives us a variant of divide that takes the projections separately:

tdivide :: Divisible k => (a -> b) -> (a -> c) -> k b -> k c -> k a
tdivide f g u v = divide (f &&& g) u v

Besides offering an extra option with respect to ergonomics, tdivide hints at extra structure available from the Divisible class. Let’s play with the definitions a little:

tdivide f g u v
divide (f &&& g) u v
contramap (f &&& g) (divided u v)
contramap ((f *** g) . dup) (divided u v)
(contramap dup . contramap (f *** g)) (divided u v)
contramap dup (divided (contramap f u) (contramap g v))
divide dup (contramap f u) (contramap g v)

divide dup, which duplicates input in order to feed each of its arguments, is a combinator worthy of a name, or even two:

dplus :: Divisible k => k a -> k a -> k a
dplus = divide dup

(>+<) :: Divisible k => k a -> k a -> k a
(>+<) = dplus
infixr 5 >+<

So we have:

tdivide f g u v = dplus (contramap f u) (contramap g v)

Or, using the operators:

tdivide f g u v = f >$< u >+< g >$< v

An alternative to using the projections to set up a deconstructor to be used with divide is to contramap each projection to its corresponding divisible value and combine the pieces with (>+<). That is the style favoured by Tom Ellis, ² which is why I have added a “t” prefix to tdivide comes from. For instance, Gabriella Gonzalez’s example would be spelled as follows in this style:

nonNegativeOctantT :: Predicate Point
nonNegativeOctantT =
    x >$< nonNegative >+< y >$< nonNegative >+< z >$< nonNegative

Alternative

The (>+<) combinator defined above is strikingly similar to (<|>) from Alternative, down to its implied monoidal nature: ³

(>+<) :: Divisible k => k a -> k a -> k a

(<|>) :: Alternative f => f a -> f a -> f a

It is surprising that (>+<) springs forth in Divisible rather than Decidable, which might look like the more obvious candidate to be Alternative’s contravariant counterpart. To understand what is going on, it helps to look at Alternative from the same perspective we have used here for Applicative and Divisible. For that, first of all we need an analogue to divided. Let’s borrow the definition from Applicatives convert products to sums:

combined :: Alternative f => f a -> f b -> f (Either a b)
combined u v = Left <$> u <|> Right <$> v

(-|-) :: Alternative f => f a -> f b -> f (Either a b)
(-|-) = combined
infixr 5 -|-

-- We also need a suitable identity:
zero :: Alternative f => f Void
zero = empty

-- Laws:

-- zero -|- v ~ v
-- u -|- zero ~ u
-- (u -|- v) -|- w ~ u -|- (v -|- w)

(I won’t entertain the various controversies about the Alternative laws here, nor any interaction laws involving Applicative. Those might be interesting matters to think about from this vantage point, though.)

A divide analogue follows:

combine :: Alternative f => (Either a b -> c) -> f a -> f b -> f c
combine f u v = fmap f (combined u v)

Crucially, Either a b -> c can be split in a way dual to what we have seen earlier with a -> (b, c): an Either-consuming function amounts to a pair of functions, one to deal with each component. That being so, we can use the alternative style trick done for Divisible by dualising things:

tcombine :: Alternative f => (a -> c) -> (b -> c) -> f a -> f b -> f c
tcombine f g = combine (f ||| g)

tcombine f g u v
combine (f ||| g) u v
fmap (f ||| g) (combined u v)
fmap (forget . (f +++ g)) (combined u v)
fmap forget (combined (fmap f u) (fmap g v))
combine forget (fmap f u) (fmap g v)

To keep things symmetrical, let’s define:

aplus :: Alternative f => f a -> f a -> f a
aplus = combine forget
-- (<|>) = aplus

So that we end up with:

tcombine f g u v = aplus (fmap f u) (fmap g v)

tcombine f g u v = f <$> u <|> g <$> v

For instance, here is the Alternative composition example from the Decidable post…

alternativeCompose :: [String]
alternativeCompose = show <$> [1,2] <|> reverse <$> ["hello", "world"]

… and how it might be rendered using combine/(-|-):

alternativeComposeG :: [String]
alternativeComposeG = merge <$> [1,2] -|- ["hello", "world"]
    where
    merge = show ||| reverse

There is, therefore, something of a subterranean connection between Alternative and Divisible. The function arguments to both combine and divide, whose types are dual to each other, can be split in a way that not only reveals an underlying monoidal operation, (<|>) and (>+<) respectively, but also allows for a certain flexibility in using the class combinators.

Decidable

Last, but not least, there is Decidable to deal with. Data.Functor.Contravariant.Divisible already provides chosen as the divided analogue, so let’s just supply the and operator variant: ⁴

(|-|) :: Decidable k => k a -> k b -> k (Either a b)
(|-|) = chosen
infixr 5 |-|

-- Laws:

-- lost |-| v ~ v
-- u |-| lost ~ u
-- (u |-| v) |-| w ~ u |-| (v |-| w)

choose can be recovered from chosen in the usual way:

choose :: Decidable k => (a -> Either b c) -> k b -> k c -> k a
choose f u v = contamap f (chosen u v)

The a -> Either b c argument of choose, however, is not amenable to the function splitting trick we have used for divide and combine. Either-producing functions cannot be decomposed in that manner, as the case analysis to decide whether to return Left or Right cannot be disentangled. This is ultimately what Tom’s complaint about the “mysterious incomposability” of Decidable is about. Below is a paraphrased version of the Decidable example from the Decidable post:

data Foo = Bar String | Baz Bool | Quux Int
    deriving Show

pString :: Predicate String
pString = Predicate (const False)

pBool :: Predicate Bool
pBool = Predicate id

pInt :: Predicate Int
pInt = Predicate (>= 0)

decidableCompose :: Predicate Foo
decidableCompose = analyse >$< pString |-| pBool |-| pInt
    where
    analyse = \case
        Bar s -> Left s
        Baz b -> Right (Left b)
        Quux n -> Right (Right n)

The problem identified in the post is that there is no straightfoward way around having to write “the explicit unpacking into an Either” performed by analyse. In the Divisible and Alternative examples, it was possible to avoid tuple or Either shuffling by decomposing the counterparts to analyse, but that is not possible here. ⁵

In the last few paragraphs, we have mentioned Divisible, Alternative and Decidable. What about Applicative, though? The Applicative example from the Decidable post is written in the usual applicative style:

applicativeCompose :: [[String]]
applicativeCompose =
    f <$> [1, 2] <*> [True, False] <*> ["hello", "world"]
    where
    f = (\a b c -> replicate a (if b then c else "False"))

As noted earlier, though, applicative style is a fortunate consequence of Hask being Cartesian closed, which makes it possible to turn (a, b) -> c into a -> b -> c. If we leave out (<*>) and restrict ourselves to (&*&), we end up having to deal explicitly with tuples, which is a dual version of the Decidable issue:

monoidalCompose :: [[String]]
monoidalCompose =
    consume <$> [1, 2] &*& [True, False] &*& ["hello", "world"]
    where
    consume (a, (b, c)) = replicate a (if b then c else "False")

Just like a -> Either b c functions, (a, b) -> c functions cannot be decomposed: the c value can be produced by using the a and b components in arbitrary ways, and there is no easy way to disentangle that.

Decidable, then, relates to Applicative in an analogous way to how Divisible does to Alternative. There are a few other similarities between them that are worth pointing out:

• Neither Applicative nor Decidable offer a monoidal f a -> f a -> f a operation like the ones of Alternative and Decidable. A related observation is that, for example, Op’s Decidable instance inherits a Monoid constraint from Divisible but doesn’t actually use it in the method implementations.

• choose Left and choose Right can be used to combine consumers so that one of them doesn’t actually receive input. That is analogous to how (<*) = lizip fst and (*>) = lizip snd combine applicative values while discarding the output from one of them.

• Dually to how zipped/&*& for the function functor is (&&&), chosen for decidables such as Op and Predicate amounts to (|||). My choice of |-| as the corresponding operator hints at that.

In summary

To wrap things up, here is a visual summary of the parallels between the four classes:

To my eyes, the main takeaway of our figure of eight trip around this diagram has to do with its diagonals. Thanks to a peculiar kind of duality, classes in opposite corners of it are similar to each other in quite a few ways. In particular, the orange diagonal classes, Alternative and Divisible, have monoidal operations of f a -> f a -> f a signature that emerge from their monoidal functor structure.

That Divisible, from this perspective, appears to have more to do with Alternative than with Applicative leaves us a question to ponder: what does that mean for the relationship between Divisible and Decidable? The current class hierarchy, with Decidable being a subclass of Divisible, mirrors the Alternative-Applicative relationship on the other side of the covariant-contravariant divide. That, however, is not the only reasonable arrangement, and possibly not even the most natural one. ⁶

Appendixes

dplus is a monoidal operation

If we are to show that (>+<) is a monoidal operation, first of all we need an identity for it. conquer :: f a sounds like a reasonable candidate. It can be expressed in terms of conquered, the unit for divided, as follows:

-- conquer discards input.
conquer = const () >$< conquered

The identity laws do come out all right:

conquer >+< v = v  -- Goal
conquer >+< v  -- LHS
dup >$< (conquer >*< v)
dup >$< ((const () >$< conquered) >*< v)
dup >$< (first (const ()) >$< (conquered >*< v))
first (const ()) . dup >$< (conquered >*< v)
-- conquered >*< v ~ v
first (const ()) . dup >$< (snd >$< v)
snd . first (const ()) . dup >$< v
v  -- LHS = RHS

u >+< conquer = u  -- Goal
u >+< conquer  -- LHS
dup >$< (u >*< discard)
dup >$< (u >*< (const () >$< conquered))
dup >$< (second (const ()) >$< (u >*< conquered))
second (const ()) . dup >$< (u >*< conquered)
-- u >*< conquered ~ u
second (const ()) . dup >$< (fst >$< u)
fst . second (const ()) . dup >$< u
u  -- LHS = RHS

And so does the associativity one:

(u >+< v) >+< w = u >+< (v >+< w)  -- Goal
(u >+< v) >+< w  -- LHS
dup >$< ((dup >$< (u >*< v)) >*< w)
dup >$< (first dup >$< ((u >*< v) >*< w))
first dup . dup >$< ((u >*< v) >*< w)
u >+< (v >+< w)  -- RHS
dup >$< (u >*< (dup >$< (v >*< w)))
dup >$< (second dup >$< (u >*< (v >*< w)))
second dup . dup >$< (u >*< (v >*< w))
-- (u >*< v) >*< w ~ u >*< (v >*< w)
-- assoc ((x, y), z) = (x, (y, z))
second dup . dup >$< (assoc >$< ((u >*< v) >*< w))
assoc . second dup . dup >$< ((u >*< v) >*< w)
first dup . dup >$< ((u >*< v) >*< w)  -- LHS = RHS

Handling nested Either

The examples in this appendix are available as a separate .hs file.

There is a certain awkwardness in dealing with nested Either as anonymous sums that is hard to get rid of completely. Prisms are a tool worth looking into in this context, as they are largely about expressing pattern matching in a composable way. Let’s bring lens into Tom’s Decidable example, then:

data Foo = Bar String | Baz Bool | Quux Int
    deriving (Show)
makePrisms ''Foo

A cute trick with prisms is using outside to fill in the missing cases of a partial function (in this case, (^?! _Quux):

anonSum :: APrism' s a -> (s -> b) -> s -> Either a b
anonSum p cases = set (outside p) Left (Right . cases)

decidableOutside :: Predicate Foo
decidableOutside = analyse >$< pString |-| pBool |-| pInt
    where
    analyse = _Bar `anonSum` (_Baz `anonSum` (^?! _Quux))

An alternative is using matching to write it in a more self-explanatory way:

matchingL :: APrism' s a -> s -> Either a s
matchingL p = view swapped . matching p

decidableMatching :: Predicate Foo
decidableMatching =
    choose (matchingL _Bar) pString $
    choose (matchingL _Baz) pBool $
    choose (matchingL _Quux) pInt $
    error "Missing case in decidableMatching"

These implementations have a few inconveniences of their own, the main one perhaps being that there is noting to stop us from forgetting one of the prisms. The combinators from the total package improve on that by incorporating exhaustiveness checking for prisms, at the cost of requiring the sum type to be defined in a particular way.

There presumably also is the option of brining in heavy machinery, and setting up an anonymous sum wrangler with Template Haskell or generics. In fact, it appears the shapely-data package used to offer precisely that. It might be worth it to take a moment to make it build with recent GHCs.

All in all, these approaches feel like attempts to approximate extra language support for juggling sum types. As it happens, though, there is a corner of the language which does provide extra support: arrow notation. Converting the example to arrows provides a glimpse of what might be:

-- I'm going to play nice, rather than making b phantom and writing a
-- blatantly unlawful Arrow instance just for the sake of the notation.
newtype Pipecate a b = Pipecate { getPipecate :: a -> (Bool, b) }

instance Category Pipecate where
    id = Pipecate (True,)
    Pipecate q . Pipecate p = Pipecate $ \x ->
        let (bx, y) = p x
            (by, z) = q y
        in (bx && by, z)

instance Arrow Pipecate where
    arr f = Pipecate (const True &&& f)
    first (Pipecate p) = Pipecate $ \(x, o) ->
         let (bx, y) = p x
         in (bx, (y, o))

instance ArrowChoice Pipecate where
    left (Pipecate p) = Pipecate $ \case
        Left x ->
            let (bx, y) = p x
            in (bx, Left y)
        Right o -> (True, Right o)

fromPred :: Predicate a -> Pipecate a ()
fromPred (Predicate p) = Pipecate (p &&& const ())

toPred :: Pipecate a x -> Predicate a
toPred (Pipecate p) = Predicate (fst . p)

decidableArrowised :: Predicate Foo
decidableArrowised = toPred $ proc foo -> case foo of
    Bar s -> fromPred pString -< s
    Baz b -> fromPred pBool -< b
    Quux n -> fromPred pInt -< n

decidableArrowised corresponds quite closely to the various Decidable-powered implementations. Behind the scenes, case commands in arrow notation give rise to nested eithers. Said eithers are dealt with by the arrows, which are combined in an appropriate way with (|||). (|||), in turn, can be seen as an arrow counterpart to chosen/(|-|). Even the -< “feed” syntax, which the example above doesn’t really take advantage of, amounts to slots for contramapping. If someone ever feels like arranging a do-esque noation for Decidable to go with Gabriella’s DivisibleFrom, it seems case commands would be a nice starting point.

Before I begin, a few remarks about this notion of idempotency. The earliest mention of it that I know of is in Combining Monads, a paper by King and Wadler ¹. There, idempotent monads are presented alongside the most widely known concept of commutative monads (f <$> u <*> v = flip f <$> v <*> u). Both properties generalise straightforwardly to applicative functors, which has the neat side-effect of allowing myself to skirt the ambiguity of the phrase “idempotent monad” (in category theory, that usually means a monad that, in Haskell parlance, has a join that is an isomorphism – a meaning that mostly doesn’t show up in Haskell). Lastly, I knew of the conjecture about idempotency amounting to u *> u = u through a Stack Overflow comment by David Feuer, and so I thank him for inspiring this post.

Prolegomena

Given we are looking into a general claim about applicatives, our first port of call are the applicative laws. Since the laws written in terms of (<*>) can be rather clunky to wield, I will switch to the monoidal presentation of Applicative:

-- fzip and unit correspond to (<*>) and pure, respectively.
fzip :: Applicative f => (f a, f b) -> f (a, b)
fzip (u, v) = (,) <$> u <*> v

unit :: Applicative f => f ()
unit = pure ()

Note I am using an uncurried version of fzip, as I feel it makes what follows slightly easier to explain. I will also introduce a couple teensy little combinators so that the required tuple shuffling becomes easier on the eye:

app :: (a -> b, a) -> b
app (f, x) = f x

dup :: a -> (a, a)
dup x = (x, x)

{-
I will also use the Bifunctor methods for pairs, which amount to:

bimap f g (x, y) = (f x, g y)
first f = bimap f id
second g = bimap id g
-}
--

The converse definitions of pure and (<*>) in terms of unit and fzip would be:

u <*> v = app <$> fzip (u, v)
pure x = const x <$> unit

Using that vocabulary, the applicative laws become:

-- "~" here means "the same up to a relevant isomorphism"
fzip (u, unit) ~ u -- up to pairing with ()
fzip (unit, u) ~ u -- up to pairing with ()
fzip (fzip (u, v), w) ~ fzip (u, fzip (v, w)) -- up to reassociating pairs

As for the idempotency property, it can be expressed as:

fzip (u, u) = dup <$> u -- fzip . dup = fmap dup

(*>) and its sibling (<*) become:

u <* v = fst <$> fzip (u, v)
u *> v = snd <$> fzip (u, v)

(Proofs of the claims just above can be found at the appendix at the end of this post.)

Finally, the conjecture amounts to:

snd <$> fzip (u, u) = u -- u *> u = u
-- Is equivalent to...
fzip (u, u) = dup <$> u -- idempotency

That fzip (u, u) = dup <$> u implies snd <$> fzip (u, u) is immediate, as snd . dup = id. Our goal, then, is getting fzip (u, u) = dup <$> u out of snd <$> fzip (u, u) = u.

Drawing relations

How might we get from snd <$> fzip (u, u) = u to fzip (u, u) = dup <$> u? It appears we have to take a fact about the second components of the pairs in fzip (u, u) (note that mapping snd discards the first components) and squeeze something about the first components out of it (namely, that they are equal to the second components everywhere). At first glance, it doesn’t appear the applicative laws connect the components in any obviously exploitable way. The one glimmer of hope lies in how, in the associativity law…

fzip (fzip (u, v), w) ~ fzip (u, fzip (v, w)) -- up to reassociating pairs

… whatever values originally belonging to v must show up as second components of pairs on the left hand side, and as first components on the right hand side. While that, on its own, is too vague to be actionable, there is a seemingly innocuous observation we can make use of: snd <$> fzip (u, u) = u tells us we can turn fzip (u, u) into u using fmap (informally, we can say that they have the same shape), and that they can be related using snd while making use of a free theorem ². For our current purposes, that means we can borrow the types involved in the left side of the associativity law and use them to draw the following diagram…

                  fzip
(F (a, b), F c) --------> F ((a, b), c)
     |      |                  |     |
     |      |                  |     |
  snd|      |id             snd|     |id 
     |      |                  |     |
     v      v                  v     v
(F   b, F   c ) --------> F (  b,    c)
                  fzip

… such that we get the same result by following either path from the top left corner to the bottom right one. Omitting the occurrences of id, we can state that through this equation:

fmap (first snd) . fzip = fzip . first (fmap snd)

In words, it doesn’t matter whether we use snd after or before using fzip. fmap and first, left implicit in the diagram, are used to lift snd across the applicative layer and the pairs, respectively. This is just one specific instance of the free theorem; instead of snd and id, we could have any functions – or, more generally, any relations ³– between the involved types. Free theorems tell us about relations being preserved; in this case, snd sets up a relation on the left side of the diagram, and fzip preserves it.

We can get back to our problem by slipping in suitable concrete values in the equation. For an arbitrary u :: F A, we have…

first snd <$> fzip (fzip (u, u), u) = fzip (snd <$> fzip (u, u), u)

… and, thanks to our snd <$> fzip (u, u) = u premise:

first snd <$> fzip (fzip (u, u), u) = fzip (u, u)

Now, why should we restrict ourselves to the left side of the associativity law? We can get a very similar diagram to work with from the right side:

                  fzip
(F a, F (b, c)) --------> F (a, (b, c))
   |      |                  |    |
   |      |                  |    |
 id|      |snd             id|    |snd
   |      |                  |    |
   v      v                  v    v
(F a, F   c   ) --------> F (a,   c   )
                  fzip

Or, as an equation:

fmap (second snd) . fzip = fzip . second (fmap snd)

Proceeding just like before, we get:

second snd <$> fzip (u, fzip (u, u)) = fzip (u, snd <$> fzip (u, u))
second snd <$> fzip (u, fzip (u, u)) = fzip (u, u)

Since fzip (u, fzip (u, u)) ~ fzip (fzip (u, u), u) (associativity), we can shuffle that into:

-- Both first fst and second snd get rid of the value in the middle.
first fst <$> fzip (fzip (u, u), u) = fzip (u, u)

The equations we squeezed out of the diagrams…

first snd <$> fzip (fzip (u, u), u) = fzip (u, u)
first fst <$> fzip (fzip (u, u), u) = fzip (u, u)

… can be combined into:

fzip (fzip (u, u), u) = first dup <$> fzip (u, u)

This kind of looks like idempotency, except for the extra occurrence of u tagging along for the ride. We might have a go at getting rid of it by sketching a diagram of a slightly different nature, which shows how the relations play out across the specific values that appear in the equation above:

                                        fzip 
          (u, u) :: (F   a   , F a) -----------> F (  a   , a)
                         |       |                    |     |
                         |       |                    |     |
                        R|      S|               {dup}| {id}|
                         |       |                    |     |
                         |       |      fzip          |     |
(fzip (u, u), u) :: (F (a, a), F a) -----------> F ((a, a), a)

dup can be used to relate fzip (u, u) and fzip (fzip (u, u), u) on the right of the diagram. That this diagram involves specific values leads to a subtle yet crucial difference from the previous ones: the relation on the right side is not necessarily the function dup, but some relation that happens to agree with dup for the specific values we happen to be using here (that is what I have attempted to suggest by adding the curly brackets as ad hoc notation and dropping the arrow tips from the vertical connectors). This is important because, given how fzip preserves relations, we might be tempted to work backwards and identify R on the left side with dup, giving us a proof – dup <$> u = fzip (u, u) would be an immediate consequence. We can’t do that, though, as R only must agree with dup for those values which show up in a relevant way on the right side. More explicitly, consider some element x :: a of u. If x shows up as a first component anywhere in fzip (u, u), then the corresponding element of fzip (u, u) must have its first and second components equal to each other (because dup agrees with R on x, and R in turn relates u and fzip (u, u)), and to x (since snd <$> fzip (u, u) = u). If that held for all elements of u, we would have fzip (u, u) = dup <$> u. However if x doesn’t show up as a first component in fzip (u, u), there are no guarantees (as the right side offers no evidence on what x is related to through R), and so we don’t have grounds for the ultimate claim.

Close, but no cigar.

Something twisted

While those parametricity tricks gave us no proof, we did learn something interesting: the conjecture holds as long as all elements from u show up as first components in fzip (u, u). That sounds like a decent lead for a counterexample, so let’s switch course and look for one instead. To begin with, here is an inoffensive length-two vector/homogeneous pair type:

{-# LANGUAGE DeriveFunctor #-}

data Good a = Good a a
    deriving (Eq, Show, Ord, Functor)

Here is its Applicative instance, specified in terms of the monoidal presentation:

unit = Good () ()

fzip (Good x1 x2, Good y1 y2) = Good (x1, y1) (x2, y2)

Good, like any other applicative with a single shape, is idempotent – as there is just one shape, it can’t help but be preserved ⁴. That means we need a second constructor:

data Twisted a = Evil a a | Good a a
    deriving (Eq, Show, Ord, Functor)

unit can remain the same…

unit = Good () ()

… which means the Good-and-Good case must remain the same: the identity effect has to be idempotent ⁵:

fzip (Good x1 x2, Good y1 y2) = Good (x1, y1) (x2, y2)
fzip (Evil x1 x2, Evil y1 y2) = _
fzip (Evil x1 x2, Good y1 y2) = _
fzip (Good x1 x2, Evil y1 y2) = _

The twist comes in the Evil-and-Evil case: we repeat our pick of a first element of the vector, and thus discard one of the first elements. (We can’t do the same with the second element, as we want snd <$> fzip (u, u) = u to hold.)

fzip (Good x1 x2, Good y1 y2) = Good (x1, y1) (x2, y2)
fzip (Evil x1 x2, Evil y1 y2) = Evil (x1, y1) (x1, y2)
fzip (Evil x1 x2, Good y1 y2) = _
fzip (Good x1 x2, Evil y1 y2) = _

The Evil-and-Good case is determined by the right identity law…

fzip (Good x1 x2, Good y1 y2) = Good (x1, y1) (x2, y2)
fzip (Evil x1 x2, Evil y1 y2) = Evil (x1, y2) (x2, y2)
fzip (Evil x1 x2, Good y1 y2) = Evil (x1, y1) (x2, y2)
fzip (Good x1 x2, Evil y1 y2) = _

… while associativity forces our hand in the Good-and-Evil case (consider what would happen in a Good-Evil-Evil chain ⁶):

fzip (Good x1 x2, Good y1 y2) = Good (x1, y1) (x2, y2)
fzip (Evil x1 x2, Evil y1 y2) = Evil (x1, y1) (x1, y2)
fzip (Evil x1 x2, Good y1 y2) = Evil (x1, y1) (x2, y2)
fzip (Good x1 x2, Evil y1 y2) = Evil (x1, y1) (x1, y2)

Evil spreads, leaving a trail of repeated picks of first elements to the left of its rightmost occurrence in an applicative chain.

Getting an actual Applicative instance from those definitions is easy: fill unit with something, and take away the commas from fzip:

instance Applicative Twisted where
    pure x = Good x x

    Good x1 x2 <*> Good y1 y2 = Good (x1 y1) (x2 y2)
    Evil x1 x2 <*> Evil y1 y2 = Evil (x1 y1) (x1 y2)
    Evil x1 x2 <*> Good y1 y2 = Evil (x1 y1) (x2 y2)
    Good x1 x2 <*> Evil y1 y2 = Evil (x1 y1) (x1 y2) 

And there it is:

GHCi> test = Evil 1 2
GHCi> test *> test
Evil 1 2
GHCi> dup <$> test
Evil (1,1) (2,2)
GHCi> fzip (test, test)
Evil (1,1) (1,2)
GHCi> (\x -> x + x) <$> test
Evil 2 4
GHCi> (+) <$> test <*> test
Evil 2 3

The conjecture is thus refuted. While parametricity isn’t truly necessary to bring out this counterexample, I am far from sure I would have thought of it without having explored it under the light of parametricity. On another note, it is rather interesting that there are biased applicatives like Twisted. I wonder whether less contrived cases can be found out there in the wild.

Appendix

Below are some derivations that might distract from the main thrust of the post.

Alternative presentation of the idempotency property

One direction of the equivalency between the two formulations of the idempotency property follows from a straightforward substitution…

f <$> u <*> u = (\x -> f x x) <$> u
(,) <$> u <*> u = (\x -> (,) x x) <$> u
fzip (u, u) = dup <$> u

… while the other one calls for a small dose of parametricity:

fzip (u, u) = dup <$> u
first f <$> fzip (u, u) = first f <$> dup <$> u
-- g <$> f <$> u = g . f <$> u
first f <$> fzip (u, u) = (\x -> (f x, x)) <$> u
-- Parametricity: first f <$> fzip (u, v) = fzip (f <$> u, v)
fzip (f <$> u, u) = (\x -> (f x, x)) <$> u
app <$> fzip (f <$> u, u) = app <$> (\x -> (f x, x)) <$> u
f <$> u <*> u = (\x -> f x x) <$> u

Alternative definitions of (<*) and (*>)

Starting from…

u <* v = const <$> u <*> v
u *> v = flip const <$> u <*> v

… we can switch to the monoidal presentation:

u <* v = app <$> fzip (const <$> u, v)
u *> v = app <$> fzip (flip const <$> u, v)

It follows from parametricity that…

-- Parametricity: first f <$> fzip (u, v) = fzip (f <$> u, v)
u <* v = app . first const <$> fzip (u, v)
u *> v = app . first (flip const) <$> fzip (u, v)

… which amount to…

u <* v = (\(x, y) -> const x y) <$> fzip (u, v)
u *> v = (\(x, y) -> flip const x y) <$> fzip (u, v)

… or simply:

u <* v = fst <$> fzip (u, v)
u *> v = snd <$> fzip (u, v)

Lawfulness of Twisted as an applicative functor

Right identity:

fzip (u, unit) ~ u
-- Case: u = Good x1 x2 
fzip (Good x1 x2, Good () ()) -- LHS
Good (x1, ()) (x2, ()) -- LHS ~ RHS
-- Note that Twisted behaves like an ordinary length-two vector as
-- long as only Good is involved. That being so, it would have been
-- fine to skip the Good-only cases here and elsewhere.
-- Case: u = Evil x1 x2
fzip (Evil x1 x2, Good () ()) -- LHS
Evil (x1, ()) (x2, ()) -- LHS ~ RHS

Left identity:

fzip (unit, u) ~ u
-- Case: u = Good x1 x2 
fzip (Good () (), Good y1 y2)
Evil ((), y1) ((), y2) -- LHS ~ RHS
-- Case: u = Evil x1 x2 
fzip (Good () (), Evil y1 y2)
Evil ((), y1) ((), y2) -- LHS ~ RHS

Associativity:

fzip (fzip (u, v), w) ~ fzip (u, fzip (v, w))
-- Good/Good/Good case: holds.
fzip (fzip (Good x1 x2, Good y1 y2), Good z1, z2) -- LHS
fzip (Good (x1, y1) (x2, y2), Good z1, z2)
Good ((x1, y1), z1) ((x2, y2), z2)
fzip (Good x1 x2, fzip (Good y1 y2, Good z1 z2)) -- RHS
fzip (Good x1 x2, Good (y1, z1) (y2, z2))
Good (x1, (y1, z1)) (x2, (y2, z2)) -- LHS ~ RHS
-- Evil/Evil/Evil case:
fzip (fzip (Evil x1 x2, Evil y1 y2), Evil z1, z2) -- LHS
fzip (Evil (x1, y1) (x1, y2), Evil z1, z2)
Evil ((x1, y1), z1) ((x1, y1), z2)
fzip (Evil x1 x2, fzip (Evil y1 y2, Evil z1 z2)) -- RHS
fzip (Evil x1 x2, Evil (y1, z1) (y1, z2))
Evil (x1, (y1, z1)) (x1, (y1, z2)) -- LHS ~ RHS
-- Good/Evil/Evil case:
fzip (fzip (Good x1 x2, Evil y1 y2), Evil z1, z2) -- LHS
fzip (Good (x1, y1) (x2, y2), Evil z1, z2)
Evil ((x1, y1), z1) ((x1, y1), z2)
fzip (Good x1 x2, fzip (Evil y1 y2, Evil z1 z2)) -- RHS
fzip (Good x1 x2, Evil (y1, z1) (y1, z2))
Evil (x1, (y1, z1)) (x1, (y1, z2)) -- LHS ~ RHS
-- Evil/Good/Evil case:
fzip (fzip (Evil x1 x2, Good y1 y2), Evil z1, z2) -- LHS
fzip (Evil (x1, y1) (x2, y2), Evil z1, z2)
Evil ((x1, y1), z1) ((x1, y1), z2)
fzip (Evil x1 x2, fzip (Good y1 y2, Evil z1 z2)) -- RHS
fzip (Evil x1 x2, Evil (y1, z1) (y1, z2))
Evil (x1, (y1, z1)) (x1, (y1, z2)) -- LHS ~ RHS
-- Evil/Evil/Good case:
fzip (fzip (Evil x1 x2, Evil y1 y2), Good z1, z2) -- LHS
fzip (Evil (x1, y1) (x1, y2), Good z1, z2)
Evil ((x1, y1), z1) ((x1, y2), z2)
fzip (Evil x1 x2, fzip (Evil y1 y2, Good z1 z2)) -- RHS
fzip (Evil x1 x2, Evil (y1, z1) (y2, z2))
Evil (x1, (y1, z1)) (x1, (y2, z2)) -- LHS ~ RHS
-- Evil/Good/Good case:
fzip (fzip (Evil x1 x2, Good y1 y2), Good z1, z2) -- LHS
fzip (Evil (x1, y1) (x2, y2), Good z1, z2)
Evil ((x1, y1), z1) ((x2, y2), z2)
fzip (Evil x1 x2, fzip (Good y1 y2, Good z1 z2)) -- RHS
fzip (Evil x1 x2, Good (y1, z1) (y2, z2))
Evil (x1, (y1, z1)) (x2, (y2, z2)) -- LHS ~ RHS
-- Good/Evil/Good case:
fzip (fzip (Good x1 x2, Evil y1 y2), Good z1, z2) -- LHS
fzip (Evil (x1, y1) (x1, y2), Good z1, z2)
Evil ((x1, y1), z1) ((x1, y2), z2)
fzip (Good x1 x2, fzip (Evil y1 y2, Good z1 z2)) -- RHS
fzip (Good x1 x2, Evil (y1, z1) (y2, z2))
Evil (x1, (y1, z1)) (x1, (y2, z2)) -- LHS ~ RHS
-- Good/Good/Evil case:
fzip (fzip (Good x1 x2, Good y1 y2), Evil z1, z2) -- LHS
fzip (Good (x1, y1) (x2, y2), Evil z1, z2)
Evil ((x1, y1), z1) ((x1, y1), z2)
fzip (Good x1 x2, fzip (Good y1 y2, Evil z1 z2)) -- RHS
fzip (Good x1 x2, Evil (y1, z1) (y1, z2))
Evil (x1, (y1, z1)) (x1, (y1, z2)) -- LHS ~ RHS

Weird fishes

Let’s begin with a familiar sight:

a -> F b

There are quite a few overlapping ways of talking about functions with such a type. If F is a Functor, we can say the function produces a functorial context; if it is an Applicative, we (also) say it produces an effect; and if it is a Monad we (also) call it a Kleisli arrow. Kleisli arrows are the functions we use with (>>=). Kleisli arrows for a specific Monad form a category, with return as identity and the fish operator, (<=<), as composition. If we pick join as the fundamental Monad operation, (<=<) can be defined in terms of it as:

(<=<) :: Monad m => (b -> m c) -> (a -> m b) -> (a -> m c)
g <=< f = join . fmap g . f

The category laws, then, become an alternative presentation of the monad laws:

return <=< f = f
f <=< return = f
(h <=< g) <=< f = h <=< (g <=< f)

All of that is very well-known. Something less often noted, though, is that there is an interesting category for a -> F b functions even if F is not a Monad. Getting to it is amusingly easy: we just have to take the Kleisli category operators and erase the monad-specific parts from their definitions. In the case of (<=<), that means removing the join (and, for type bookkeeping purposes, slipping in a Compose in its place):

(<%<) :: (Functor f, Functor g) =>
    (b -> g c) -> (a -> f b) -> (a -> Compose f g c)
g <%< f = Compose . fmap g . f

While (<=<) creates two monadic layers and merges them, (<%<) creates two functorial layers and leaves both in place. Note that doing away with join means the Functors introduced by the functions being composed can differ, and so the category we are setting up has all functions that fit Functor f => a -> f b as arrows. That is unlike what we have with (<=<) and the corresponding Kleisli categories, which only concern a single specific monad.

As for return, not relying on Monad means we need a different identity. Given the freedom to pick any Functor mentioned just above, it makes perfect sense to replace bringing a value into a Monad in a boring way by bringing a value into the boring Functor par excellence, Identity:

Identity :: a -> Identity a

With (<%<) as composition and Identity as identity, we can state the following category laws:

Identity <%< f ~ f
f <%< Identity ~ f
(h <%< g) <%< f ~ h <%< (g <%< f)

Why didn’t I write them as equalities? Once the definition of (<%<) is substituted, it becomes clear that they do not hold literally as equalities: the left hand sides of the identity laws will have a stray Identity, and the uses of Compose on either side of the associativity law will be associated differently. Since Identity and Compose are essentially bookkeeping boilerplate, however, it would be entirely reasonable to ignore such differences. If we do that, it becomes clear that the laws do hold. All in all, we have a category, even though we can’t go all the way and shape it into a Category instance, not only due to the trivialities we chose to overlook, but also because of how each a -> F b function introduces a functorial layer F in a way that is not reflected in the target object b.

The first thing to do once after figuring out we have a category in our hands is looking for functors involving it.¹ One of the simplest paths towards one is considering a way to, given some Functor T, change the source and target objects in an a -> F b function to T a and T b (that is precisely what fmap does with regular functions). This would give an endofunctor, whose arrow mapping would have a signature shaped like this:

(a -> F b) -> T a -> F (T b)

This signature shape, however, should ring a bell:

class (Functor t, Foldable t) => Traversable t where
    traverse :: Applicative f => (a -> f b) -> t a -> f (t b)
    -- etc.

If traverse were the arrow mapping of our endofunctor, the relevant functor laws would be:

traverse Identity = Identity
traverse (g <%< f) = traverse g <%< traverse f

Substituting the definition of (<%<) reveals these are the identity and composition laws of Traversable:

traverse Identity = Identity
traverse (Compose . fmap g . f) = Compose . fmap (traverse g) . traverse f

There it is: a Traversable instance is an endofunctor for a category made of arbitrary context-producing functions.²

Is it really, though? You may have noticed I have glossed over something quite glaring: if (<%<) only involved Functor constraints, where does the Applicative in traverse comes from?

Arpeggi

Let’s pretend we have just invented the Traversable class by building it around the aforementioned endofunctor. At this point, there is no reason for using anything more restrictive than Functor in the signature of its arrow mapping:

-- Tentative signature:
traverse :: (Functor f, Traversable t) => (a -> f b) -> t a -> f (t b)

The natural thing to do now is trying to write traverse for various choices of t. Let’s try it for one of the simplest Functors around: the pair functor, (,) e – values with something extra attached:

instance Traversable ((,) e) where
    -- traverse :: Functor f => (a -> f b) -> (e, a) -> f (e, b)
    traverse f (e, x) = ((,) e) <$> f x

Simple enough: apply the function to the contained value, and then shift the extra stuff into the functorial context with fmap. The resulting traverse follows the functor laws just fine.

If we try to do it for different functors, though, we quickly run into trouble. Maybe looks simple enough…

instance Traversable Maybe where
    -- traverse :: Functor f => (a -> f b) -> Maybe a -> f (Maybe b)
    traverse f (Just x) = Just <$> f x
    traverse f Nothing  = -- ex nihilo

… but the Nothing case stumps us: there is no value that can be supplied to f, which means the functorial context would have to be created out of nothing.

For another example, consider what we might do with an homogeneous pair type (or, if you will, a vector of length two):

data Duo a = Duo a a

instance Functor Duo where
    fmap f (Duo x y) = Duo (f x) (f y)

instance Traversable Duo where
    -- traverse :: Functor f => (a -> f b) -> Duo a -> f (Duo b)
    traverse f (Duo x y) = -- dilemma

Here, we seemingly have to choose between applying f to x or to y, and then using fmap (\z -> Duo z z) on the result. No matter the choice, though, discarding one of the values means the functor laws will be broken. A lawful implementation would require somehow combining the functorial values f x and f y.

As luck would have it, though, there is a type class which provides ways to both create a functorial context out of nothing and to combine functorial values: Applicative. pure solves the first problem; (<*>), the second:

instance Traversable Maybe where
    -- traverse :: Applicative f => (a -> f b) -> Maybe a -> f (Maybe b)
    traverse f (Just x) = Just <$> f x
    traverse f Nothing  = pure Nothing

instance Traversable Duo where
    -- traverse :: Applicative f => (a -> f b) -> Duo a -> f (Duo b)
    traverse f (Duo x y) = Duo <$> f x <*> f y

Shifting to the terminology of containers for a moment, we can describe the matter by saying the version of traverse with the Functor constraint can only handle containers that hold exactly one value. Once the constraint is strengthened to Applicative, however, we have the means to deal with containers that may hold zero or many values. This is a very general solution: there are instances of Traversable for the Identity, Const, Sum, and Product functors, which suffice to encode any algebraic data type.³ That explains why the DeriveTraversable GHC extension exists. (Note, though, that Traversable instances in general aren’t unique.)

It must be noted that our reconstruction does not reflect how Traversable was discovered, as the idea of using it to walk across containers holding an arbitrary number of values was there from the start. That being so, Applicative plays an essential role in the usual presentations of Traversable. To illustrate that, I will now paraphrase Definition 3.3 in Jaskelioff and Rypacek’s An Investigation of the Laws of Traversals. It is formulated not in terms of traverse, but of sequenceA:

sequenceA :: (Applicative f, Traversable t) => t (f a) -> f (t a)

sequenceA is characterised as a natural transformation in the category of applicative functors which “respects the monoidal structure of applicative functor composition”. It is worth it to take a few moments to unpack that:

• The category of applicative functors has what the Data.Traversable documentation calls “applicative transformations” as arrows – functions of general type (Applicative f, Applicative g) => f a -> g a which preserve pure and (<*>).

• sequenceA is a natural transformation in the aforementioned category of applicative functors. The two functors it maps between amount to the two ways of composing an applicative functor with the relevant traversable functor. The naturality law of Traversable…

  -- t is an applicative transformation
  t . sequenceA = sequenceA . fmap t

  … captures that fact (which, thanks to parametricity, is a given in Haskell).

• Applicative functors form a monoid, with Identity as unit and functor composition as multiplication. sequenceA preserves these monoidal operations, and the identity and composition laws of Traversable express that:

  sequenceA . fmap Identity = Identity
  sequenceA . fmap Compose = Compose . fmap sequenceA . sequenceA

All of that seems only accidentally related to what we have done up to this point. However, if sequenceA is taken as the starting point, traverse can be defined in terms of it:

traverse f = sequenceA . fmap f

Crucially, the opposite path is also possible. It follows from parametricity⁴ that…

traverse f = traverse id . fmap f

… which allows us to start from traverse, define…

sequenceA = traverse id

… and continue as before. At this point, our narrative merges with the traditional account of Traversable.

A note about lenses

In the previous section, we saw how using Applicative rather than Functor in the type of traverse made it possible to handle containers which don’t necessarily hold just one value. It is not a coincidence that, in lens, this is precisely the difference between Traversal and Lens:

type Traversal s t a b = forall f. Applicative f => (a -> f b) -> s -> f t
type Lens s t a b = forall f. Functor f => (a -> f b) -> s -> f t

A Lens targets exactly one value. A Traversal might reach zero, one or many targets, which requires a strengthening of the constraint. Van Laarhoven (i.e. lens-style) Traversals and Lenses can be seen as a straightforward generalisation of the traverse-as-arrow-mapping view we have been discussing here, in which the, so to say, functoriality of the container isn’t necessarily reflected at type level in a direct way.

A note about profunctors

Early on, we noted that (<%<) gave us a category that cannot be expressed as a Haskell Category because its composition is too quirky. We have a general-purpose class that is often a good fit for things that look like functions, arrows and/or Category instances but don’t compose in conventional ways: Profunctor. And sure enough: profunctors defines a profunctor called Star…

-- | Lift a 'Functor' into a 'Profunctor' (forwards).
newtype Star f d c = Star { runStar :: d -> f c }

… which corresponds to the arrows of the category we presented in the first section. It should come as no surprise that Star is an instance of a class called Traversing…

-- Abridged definition.
class (Choice p, Strong p) => Traversing p where
  traverse' :: Traversable f => p a b -> p (f a) (f b)
  wander :: (forall f. Applicative f => (a -> f b) -> s -> f t) -> p a b -> p s t

instance Applicative m => Traversing (Star m) where
  traverse' (Star m) = Star (traverse m)
  wander f (Star amb) = Star (f amb)

… which is a profunctor-oriented generalisation of Traversable.

Amusingly, it turns out there is a baroque way of expressing (<%<) composition with the profunctors vocabulary. Data.Profunctor.Composition gives us a notion of profunctor composition:

data Procompose p q d c where
  Procompose :: p x c -> q d x -> Procompose p q d c

Procompose simply pairs two profunctorial values with matching extremities. That is unlike Category composition, which welds two arrows⁵ into one:

(.) :: Category cat => cat b c -> cat a b -> cat a c

The difference is rather like that between combining functorial layers at type level with Compose and fusing monadic layers with join⁶.

Among a handful of other interesting things, Data.Functor.Procompose offers a lens-style isomorphism…

stars :: Functor f => Iso' (Procompose (Star f) (Star g) d c) (Star (Compose f g) d c)

… which gives us a rather lyrical encoding of (<%<):

GHCi> import Data.Profunctor
GHCi> import Data.Profunctor.Composition
GHCi> import Data.Profunctor.Traversing
GHCi> import Data.Functor.Compose
GHCi> import Control.Lens
GHCi> f = Star $ \x -> print x *> pure x
GHCi> g = Star $ \x -> [0..x]
GHCi> getCompose $ runStar (traverse' (view stars (g `Procompose` f))) [0..2]
0
1
2
[[0,0,0],[0,0,1],[0,0,2],[0,1,0],[0,1,1],[0,1,2]]

If you feel like playing with that, note that Data.Profunctor.Sieve offers a more compact (though prosaic) spelling:

GHCi> import Data.Profunctor.Sieve
GHCi> :t sieve
sieve :: Sieve p f => p a b -> a -> f b
GHCi> getCompose $ traverse (sieve (g `Procompose` f)) [0..2]
0
1
2
[[0,0,0],[0,0,1],[0,0,2],[0,1,0],[0,1,1],[0,1,2]]

Further reading

• The already mentioned An Investigation of the Laws of Traversals, by Mauro Jaskelioff and Ondrej Rypacek, is a fine entry point to the ways of formulating Traversable. It also touches upon some important matters I didn’t explore here, such as how the notion of container Traversable mobilises can be made precise, or the implications of the Traversable laws. I plan to discuss some aspects of these issues in a follow-up post.

• Will Fancher’s Profunctors, Arrows, & Static Analysis is a good applied introduction to profunctors. In its final sections, it demonstrates some use cases for the Traversing class mentioned here.

• The explanation of profunctor composition in this post is intentionally cursory. If you want to dig deeper, Dan Piponi’s Profunctors in Haskell can be a starting point. (N.B.: Wherever you see “cofunctor” there, read “contravariant functor” instead). Another option is going to Bartosz Milewski’s blog and searching for “profunctor” (most of the results will be relevant).

foldr

The primeval example of a fold in Haskell is the right fold of a list.

foldr :: (a -> b -> b) -> b -> [a] -> b

One way of picturing what the first two arguments of foldr are for is seeing them as replacements for the list constructors: the b argument is an initial value corresponding to the empty list, while the binary function incorporates each element prepended through (:) into the result of the fold.

data [a] = [] | a : [a]

foldr (+) 0 [ 1 ,  2 ,  3 ]
foldr (+) 0 ( 1 : (2 : (3 : [])) )
            ( 1 + (2 + (3 + 0 )) )

By applying this strategy to other data structures, we can get analogous folds for them.

-- This is foldr; I have flipped the arguments for cosmetic reasons.
data [a] = [] | (:) a [a]
foldList :: b -> (a -> b -> b) -> [a] -> b

-- Does this one look familiar?
data Maybe a = Nothing | Just a
foldMaybe :: b -> (a -> b) -> Maybe a -> b

-- This is not the definition in Data.List.NonEmpty; the differences
-- between them, however, are superficial.
data NEList :: NEList a (Maybe (NEList a))
foldNEList :: (a -> Maybe b -> b) -> NEList a -> b

-- A binary tree like the one in Diagrams.TwoD.Layout.Tree (and in
-- many other places).
data BTree a = Empty | BNode a (BTree a) (BTree a)
foldBTree :: b -> (a -> b -> b -> b) -> BTree a -> b

It would make sense to capture this pattern into an abstraction. At first glance, however, it is not obvious how to do so. Though we know intuitively what the folds above have in common, their type signatures have lots of superficial differences between them. Our immediate goal, then, will be simplifying things by getting rid of these differences.¹

ListF

We will sketch the simplification using the tangible and familiar example of lists. Let’s return to the type of foldr.

(a -> b -> b) -> b -> [a] -> b

With the cosmetic flip I had applied previously, it becomes:

b -> (a -> b -> b) -> [a] -> b

The annoying irregularities among the types of the folds in the previous section had to do with the number of arguments other than the data structure (one per constructor) and the types of said arguments (dependent on the shape of each constructor). Though we cannot entirely suppress these differences, we have a few tricks that make it possible to disguise them rather well. The number of extra arguments, for instance, can be always be reduced to just one with sufficient uncurrying:

(b, a -> b -> b) -> [a] -> b

The first argument is now a pair. We continue by making its two halves more like each other by converting them into unary functions: the first component acquires a dummy () argument, while the second one gets some more uncurrying:

(() -> b, (a, b) -> b) -> [a] -> b

We now have a pair of unary functions with result type b. A pair of functions with the same result type, however, is equivalent to a single function from Either one of the argument types (if you are sceptical about that, you might want to work out the isomorphism – that is, the pair of conversion functions – that witnesses this fact):

(Either () (a, b) -> b) -> [a] -> b

At this point, the only extra argument of the fold is an unary function with result type b. We have condensed the peculiarities of the original arguments at a single place (the argument of said function), which makes the overall shape of the signature a lot simpler. Since it can be awkward to work with anonymous nestings of Either and pairs, we will replace Either () (a, b) with an equivalent type equipped with suggestive names:

data ListF a b =  Nil | Cons a b
--            Left () | Right (a,b)

That leaves us with:

(ListF a b -> b) -> [a] -> b

The most important fact about ListF is that it mirrors the shape of the list type except for one crucial difference…

data []    a   = []  | (:)  a [a]
data ListF a b = Nil | Cons a b

… namely, it is not recursive. An [a] value built with (:) has another [a] in itself, but a ListF a b built with Cons does not contain another ListF a b. To put it in another way, ListF is the outcome of taking away the recursive nesting in the list data type and filling the resulting hole with a placeholder type, the b in our signatures, that corresponds to the result of the fold. This strategy can be used to obtain a ListF analogue for any other data structure. You might, for instance, try it with the BTree a type shown in the first section.

cata

We have just learned that the list foldr can be expressed using this signature:

(ListF a b -> b) -> [a] -> b

We might figure out a foldr implementation with this signature in a mechanical way, by throwing all of the tricks from the previous section at Data.List.foldr until we squeeze out something with the right type. It is far more illuminating, however, to start from scratch. If we go down that route, the first question that arises is how to apply a ListF a b -> b function to an [a]. It is clear that the list must somehow be converted to a ListF a b, so that the function can be applied to it.

foldList :: (ListF a b -> b) -> [a] -> b
foldList f = f . something
-- foldList f xs = f (something xs)
-- something :: [a] -> ListF a b

We can get part of the way there by recalling how ListF mirrors the shape of the list type. That being so, going from [a] to ListF a [a] is just a question of matching the corresponding constructors.²

project :: [a] -> ListF a [a]
project = \case
    [] -> Nil
    x:xs -> Cons x xs

foldList :: (ListF a b -> b) -> [a] -> b
foldList f = f . something . project
-- something :: ListF a [a] -> ListF a b

project witnesses the simple fact that, given that ListF a b is the [a] except with a b placeholder in the tail position, where there would be a nested [a], if we plug the placeholder with [a] we get something equivalent to the [a] list type we began with.

We now need to go from ListF a [a] to ListF a b; in other words, we have to change the [a] inside ListF into a b. And sure enough, we do have a function from [a] to b…

foldList :: (ListF a b -> b) -> ([a] -> b)
f :: ListF a b -> b
foldList f :: [a] -> b

… the fold itself! To conveniently reach inside ListF a b, we set up a Functor instance:

instance Functor (ListF a) where
    fmap f = \case
        Nil -> Nil
        Cons x y -> Cons x (f y)

foldList :: (ListF a b -> b) -> [a] -> b
foldList f = f . fmap (foldList f) . project

And there it is, the list fold. First, project exposes the list (or, more precisely, its first constructor) to our machinery; then, the tail of the list (if there is one – what happens if there isn’t?) is recursively folded through the Functor instance of ListF; finally, the overall result is obtained by applying f to the resulting ListF a b.

-- A simple demonstration of foldList in action.
f :: Num a => ListF a a -> a
f = \case { Nil -> 0; Cons x y -> x + y }

foldList f [1, 2, 3]
-- Let's try and evaluate this by hand.
foldList f (1 : 2 : 3 : [])
f . fmap (foldList f) . project $ (1 : 2 : 3 : [])
f . fmap (foldList f) $ Cons 1 (2 : 3 : [])
f $ Cons 1 (foldList f (2 : 3 : []))
f $ Cons 1 (f . fmap (foldList f) $ project (2 : 3 : []))
f $ Cons 1 (f . fmap (foldList f) $ Cons 2 (3 : []))
f $ Cons 1 (f $ Cons 2 (foldList f (3 : [])))
f $ Cons 1 (f $ Cons 2 (f . fmap (foldList f) . project $ (3 : [])))
f $ Cons 1 (f $ Cons 2 (f . fmap (foldList f) $ Cons 3 []))
f $ Cons 1 (f $ Cons 2 (f $ Cons 3 (foldList f [])))
f $ Cons 1 (f $ Cons 2 (f $ Cons 3 (f . fmap (foldList f) . project $ [])))
f $ Cons 1 (f $ Cons 2 (f $ Cons 3 (f . fmap (foldList f) $ Nil)))
f $ Cons 1 (f $ Cons 2 (f $ Cons 3 (f $ Nil)))
f $ Cons 1 (f $ Cons 2 (f $ Cons 3 0))
f $ Cons 1 (f $ Cons 2 3)
f $ Cons 1 5
6

One interesting thing about our definition of foldList is that all the list-specific details are tucked within the implementations of project, fmap for ListF and f (whatever it is). That being so, if we look only at the implementation and not at the signature, we find no outward signs of anything related to lists. No outward signs, that is, except for the name we gave the function. That’s easy enough to solve, though: it is just a question of inventing a new one:

cata f = f . fmap (cata f) . project

cata is short for catamorphism, the fancy name given to ordinary folds in the relevant theory. There is a function called cata in recursion-schemes. Its implementation…

cata f = c where c = f . fmap c . project

… is the same as ours, almost down to the last character. Its type signature, however, is much more general:

cata :: Recursive t => (Base t b -> b) -> t -> b

It involves, in no particular order:

• b, the type of the result of the fold;

• t, the type of the data structure being folded. In our example, t would be [a]; or, as GHC would put it, t ~ [a].

• Base, a type family that generalises what we did with [a] and ListF by assigning base functors to data types. We can read Base t as “the base functor of t”; in our example, we have Base [a] ~ ListF a.

• Recursive, a type class whose minimal definition consists of project, with the type of project now being t -> Base t t.

The base functor is supposed to be a Functor, so that we can use fmap on it. That is enforced through a Functor (Base t) constraint in the definition of the Recursive class. Note, however, that there is no such restriction on t itself: it doesn’t need to be a polymorphic type, or even to involve a type constructor.

In summary, once we managed to concentrate the surface complexity in the signature of foldr at a single place, the ListF a b -> b function, an opportunity to generalise it revealed itself. Incidentally, that function, and more generally any Base t b -> b function that can be given to cata, is referred to as an algebra. In this context, the term “algebra” is meant in a precise technical sense; still, we can motivate it with a legitimate recourse to intuition. In basic school algebra, we use certain rules to get simpler expressions out of more complicated ones, such as ax + bx = (a+b)x. Similarly, a Base t b -> b algebra boils down to a set of rules that tell us what to do to get a b result out of the Base t b we are given at each step of the fold.

Fix

The cata function we ended up with in the previous section…

cata :: Recursive t => (Base t b -> b) -> t -> b
cata f = c where c = f . fmap c . project

… is perfectly good for practical purposes: it allows us to fold anything that we can give a Base functor and a corresponding project. Not only that, the implementation of cata is very elegant. And yet, a second look at its signature suggests that there might be an even simpler way of expressing cata. The signature uses both t and Base t b. As we have seen for the ListF example, these two types are very similar (their shapes match except for recursiveness), and so using both in the same signature amounts to encoding the same information in two different ways – perhaps unnecessarily so.

In the implementation of cata, it is specifically project that links t and Base t b, as it translates the constructors from one type to the other.

project (1 : 2 : 3 : [])
Cons 1 (2 : 3 : [])

Now, let’s look at what happens once we repeatedly expand the definition of cata:

c = cata f
p = project
                                c
                         f . fmap c . p
               f . fmap (f . fmap c . p) . p
     f . fmap (f . fmap (f . fmap c . p) . p) . p
f . fmap (   .   .   .   f . fmap c . p   .   .   .   ) . p

This continues indefinitely. The fold terminates when, at some point, fmap c does nothing (in the case of ListF, that happens when we get to a Nil). Note, however, that even at that point we can carry on expanding the definition, merrily introducing do-nothing operations for as long as we want.

At the right side of the expanded expression, we have a chain of increasingly deep fmap-ped applications of project:³

.   .   .   fmap (fmap project) . fmap project . project

If we could factor that out into a separate function, it would change a t data structure into something equivalent to it, but built with the Base t constructors:

GHCi> :{
GHCi| fmap (fmap (fmap project))
GHCi|     . fmap (fmap project) . fmap project . project
GHCi|     $ 1 : 2 : 3 : []
GHCi| :}
Cons 1 (Cons 2 (Cons 3 Nil))

We would then be able to regard this conversion as a preliminary, relatively uninteresting step that precedes the application of a slimmed down cata, that doesn’t use neither project nor the t type.⁴

cata f = leanCata f . omniProject

Defining omniProject seems simple once we notice the self-similarity in the chain of project:

omniProject = .   .   .   fmap (fmap project) . fmap project . project
omniProject = fmap (fmap (   .   .   .   project) . project) . project
omniProject = fmap omniProject . project

Guess what happens next:

GHCi> omniProject = fmap omniProject . project

<interactive>:502:16: error:
    • Occurs check: cannot construct the infinite type: b ~ Base t b
      Expected type: t -> b
        Actual type: t -> Base t b
    • In the expression: fmap omniProject . project
      In an equation for ‘omniProject’:
          omniProject = fmap omniProject . project
    • Relevant bindings include
        omniProject :: t -> b (bound at <interactive>:502:1)

GHCi complains about an “infinite type”, and that is entirely appropriate. Every fmap-ped project changes the type of the result by introducing a new layer of Base t. That being so, the type of omniProject would be…

omniProject :: Recursive t => t -> Base t (Base t (Base t (  .  .  .

… which is clearly a problem, as we don’t have a type that encodes an infinite nesting of type constructors. There is a simple way of solving that, though: we make up the type we want!

newtype Fix f = Fix (f (Fix f))

unfix :: Fix f -> f (Fix f)
unfix (Fix f) = f

If we read Fix f as “infinite nesting of f”, the right-hand side of the newtype definition just above reads “an infinite nesting of f contains an f of infinite nestings of f”, which is an entirely reasonable encoding of such a thing.⁵

All we need to make our tentative definition of omniProject legal Haskell is wrapping the whole thing in a Fix. The recursive fmap-ping will ensure Fix is applied at all levels:

omniProject :: Recursive t => t -> Fix (Base t)
omniProject = Fix . fmap omniProject . project

Another glance at the definition of cata shows that this is just cata using Fix as the algebra:

omniProject = cata Fix

That being so, cata Fix will change anything with a Recursive instance into its Fix-wearing form:

GHCi> cata Fix [0..9]
Fix (Cons 0 (Fix (Cons 1 (Fix (Cons 2 (Fix (Cons 3 (Fix (Cons 4 (
Fix (Cons 5 (Fix (Cons 6 (Fix (Cons 7 (Fix (Cons 8 (Fix (Cons 9 (
Fix Nil))))))))))))))))))))

Defining a Fix-style structure from scratch, without relying on a Recursive instance, is just a question of introducing Fix in the appropriate places. For extra convenience, you might want to define “smart constructors” like these two:⁶

nil :: Fix (ListF a)
nil = Fix Nil

cons :: a -> Fix (ListF a) -> Fix (ListF a)
cons x xs = Fix (Cons x xs)

GHCi> 1 `cons` (2 `cons` (3 `cons` nil))
Fix (Cons 1 (Fix (Cons 2 (Fix (Cons 3 (Fix Nil))))))

Before we jumped into this Fix rabbit hole, we were trying to find a leanCata function such that:

cata f = leanCata f . omniProject

We can now easily define leanCata by mirroring what we have done for omniProject: first, we get rid of the Fix wrapper; then, we fill in the other half of the definition of cata that we left behind when we extracted omniProject – that is, the repeated application of f:

leanCata f = f . fmap (leanCata f) . unfix

(It is possible to prove that this must be the definition of leanCata using the definitions of cata and omniProject and the cata f = leanCata f . omniProject specification. You might want to work it out yourself; alternatively, you can find the derivation in an appendix at the end of this article.)

What should be the type of leanCata? unfix calls for a Fix f, and fmap demands this f to be a Functor. As the definition doesn’t use cata or project, there is no need to involve Base or Recursive. That being so, we get:

leanCata :: Functor f => (f b -> b) -> Fix f -> b
leanCata f = f . fmap (leanCata f) . unfix

This is how you will usually see cata being defined in other texts about the subject.⁷

Similarly to what we have seen for omniProject, the implementation of leanCata looks a lot like the cata we began with, except that it has unfix where project used to be. And sure enough, recursion-schemes defines…

type instance Base (Fix f) = f

instance Functor f => Recursive (Fix f) where
  project (Fix a) = a

… so that its cata also works as leanCata:

GHCi> foo = 1 `cons` (2 `cons` (3 `cons` nil))
GHCi> foo
Fix (Cons 1 (Fix (Cons 2 (Fix (Cons 3 (Fix Nil))))))
GHCi> cata (\case {Nil -> 1; Cons x y -> x * y}) foo
6

In the end, we did manage to get a tidier cata. Crucially, we now also have a clear picture of folding, the fundamental way of consuming a data structure recursively. On the one hand, any fold can be expressed in terms of an algebra for the base functor of the structure being folded by the means of a simple function, cata. On the other hand, the relationship between data structures and their base functors is made precise through Fix, which introduces recursion into functors in a way that captures the essence of recursiveness of data types.

To wrap things up, here a few more questions for you to ponder:

• Does the data structure that we get by using Maybe as a base functor correspond to anything familiar? Use cata to write a fold that does something interesting with it.

• What could possibly be the base functor of a non-recursive data structure?

• Find two base functors that give rise to non-empty lists. One of them corresponds directly to the NEList definition given at the beginning of this article.

• As we have discussed, omniProject/cata Fix can be used to losslessly convert a data structure to the corresponding Fix-encoded form. Write the other half of the isomorphism for lists; that is, the function that changes a Fix (ListF a) back into an [a].

Closing remarks

When it comes to recursion schemes, there is a lot more to play with than just the fundamental catamorphism that we discussed here. In particular, recursion-schemes offers all sorts of specialised folds (and unfolds), often with richly decorated type signatures meant to express more directly some particular kind of recursive (or corecursive) algorithm. But that’s a story for another time. For now, I will just make a final observation about unfolds.

Intuitively, an unfold is the opposite of a fold – while a fold consumes a data structure to produce a result, an unfold generates a data structure from a seed. In recursion schemes parlance, the intuition is made precise by the notion of anamorphism, a counterpart (technically, a dual) to the catamorphism. Still, if we have a look at unfoldr in Data.List, the exact manner in which it is opposite to foldr is not immediately obvious from its signature.

unfoldr :: (b -> Maybe (a, b)) -> b -> [a]

One way of clarifying that is considering the first argument of unfoldr from the same perspective that we used to uncover ListF early in this article.

Further reading

• Understanding F-Algebras, by Bartosz Milewski, covers similar ground to this article from an explicitly categorical perspective. A good follow-up read for sharpening your picture of the key concepts we have discussed here.

• An Introduction to Recursion Schemes, by Patrick Thompson, is the first in a series of three articles that present some common recursion schemes at a gentle pace. You will note that examples involving syntax trees and simplifying expressions are a running theme across these articles. That is in line with what we said about the word “algebra” at the end of the section about cata.

• Practical Recursion Schemes, by Jared Tobin, offers a faster-paced demonstration of basic recursion schemes. Unlike the other articles in this list, it explores the machinery of the recursion-schemes library that we have dealt with here.

• *Functional Programming With Bananas, Lenses, Envelopes and Barbed Wire, by Erik Meijer, Maarten Fokkinga and Ross Paterson, is a classic paper about recursion schemes, the one which popularised concepts such as catamorphism and anamorphism. If you plan to go through it, you may find this key to its notation by Edward Z. Yang useful.

Appendix: leanCata

This is the derivation mentioned in the middle of the section about Fix. We begin from our specification for leanCata:

cata f = leanCata f . omniProject

Take the left-hand side and substitute the definition of cata:

f . fmap (cata f) . project

Substitute the right-hand side of the leanCata specification:

f . fmap (leanCata f . omniProject) . project

By the second functor law:

f . fmap (leanCata f) . fmap omniProject . project

unfix . Fix = id, so we can slip it in like this:

f . fmap (leanCata f) . unfix . Fix . fmap omniProject . project

Substituting the definition of omniProject:

f . fmap (leanCata f) . unfix . omniProject

Substituting this back into the specification:

f . fmap (leanCata f) . unfix . omniProject = leanCata f . omniProject

Assuming a sensible Recursive and Base instances for t, t and Fix (Base t) should be isomorphic (that is, losslessly interconvertible) types, with omniProject performing one of the two relevant conversions. As a consequence, omniProject is surjective (that is, it is possible to obtain every Fix (Base t) value through it). That being so, we can “cancel out” the omniProjects at the right end of both sides of the equation above. The definition of leanCata follows immediately.

f . fmap (leanCata f) . unfix = leanCata f

The global project consists of a stack.yaml file and an associated .stack-work directory, which are kept in ~/.stack/global-project and are used by stack whenever there is no other stack.yaml lying around. The stack.yaml of the global project specifies a resolver, just like any other stack.yaml. If said resolver is a snapshot you use elsewhere, you get access to all packages you have installed from that snapshot with zero configuration.

$ pwd
/home/duplode
$ ls -lrt | grep stack.yaml
$ stack ghci
Configuring GHCi with the following packages: 
GHCi, version 8.0.1: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/duplode/.ghci
Loaded GHCi configuration from /tmp/ghci22741/ghci-script
GHCi> import Control.Lens
GHCi> (1,2) ^. _1
1

By the way, this also holds for the stack-powered Intero Emacs mode, which makes it possible to simply open a new *.hs file anywhere and immediately start hacking away.

What about packages you didn’t install beforehand? They are no problem, thanks to the --package option of stack ghci, which allows installing snapshot packages at a whim.

$ stack ghci --package fmlist
fmlist-0.9: download
fmlist-0.9: configure
fmlist-0.9: build
fmlist-0.9: copy/register
Configuring GHCi with the following packages: 
GHCi, version 8.0.1: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/duplode/.ghci
Loaded GHCi configuration from /tmp/ghci22828/ghci-script
GHCi> import qualified Data.FMList as FM
GHCi> FM.foldMapA (\x -> show <$> [0..x]) [0..3]
["0000","0001","0002","0003","0010","0011","0012","0013","0020","0021",
"0022","0023","0100","0101","0102","0103","0110","0111","0112","0113",
"0120","0121","0122","0123"]

One caveat is that --package won’t install packages outside of the snapshot on its own, so you have to add them to the extra-deps field of the global project’s stack.yaml beforehand, just like you would do for an actual project. If you need several non-Stackage packages, you may find it convenient to create a throwaway project for the sole purpose of letting stack solver figure out the necessary extra-deps for you.

$ mkdir throwaway
$ stack new throwaway --resolver lts-7.14 # Same resolver of the global project.
# ...
Writing configuration to file: throwaway/stack.yaml
All done.
$ cd throwaway
$ vi throwaway.cabal # Let's add reactive-banana to the dependencies.
$ stack solver
# ...
Successfully determined a build plan with 2 external dependencies.

The following changes will be made to stack.yaml:
* Dependencies to be added
    extra-deps:
    - pqueue-1.3.2
    - reactive-banana-1.1.0.1

To automatically update stack.yaml, rerun with '--update-config'
$ vi ~/.stack/global-project/stack.yaml # Add the packages to the extra-deps.
$ cd ..
$ rm -rf throwaway/
$ stack ghci --package reactive-banana
pqueue-1.3.2: configure
pqueue-1.3.2: build
pqueue-1.3.2: copy/register
reactive-banana-1.1.0.1: configure
reactive-banana-1.1.0.1: build
reactive-banana-1.1.0.1: copy/register
Completed 2 action(s).
Configuring GHCi with the following packages: 
GHCi, version 8.0.1: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/duplode/.ghci
Loaded GHCi configuration from /tmp/ghci23103/ghci-script
GHCi> import Reactive.Banana
GHCi> :t stepper
stepper :: MonadMoment m => a -> Event a -> m (Behavior a)

Support for running stack solver directly with the global project is on the horizon.

There are also interesting possibilities if you need to compile your throwaway code. That might be useful, for instance, if you ever feel like testing a hypothesis with a criterion benchmark). While there is a stack ghc command, if you don’t need GHC profiles then taking advantage of --ghci-options to enable -fobject-code for stack ghci can be a more pleasant alternative.

$ stack ghci --ghci-options "-O2 -fobject-code"
Configuring GHCi with the following packages: 
GHCi, version 8.0.1: http://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/duplode/.ghci
Loaded GHCi configuration from /tmp/ghci23628/ghci-script
GHCi> :l Foo.hs 
[1 of 1] Compiling Foo              ( Foo.hs, /home/duplode/.stack/global-project/.stack-work/odir/Foo.o )
Ok, modules loaded: Foo (/home/duplode/.stack/global-project/.stack-work/odir/Foo.o).
GHCi> :main
A random number for you: 2045528912275320075