Progress report

I remain captivated by the idea of a large-scale software system implemented entirely in purely functional style, and have been noodling away for a while on a much simpler implementation of some of this folly.

The old way was to take pure functions and try to magically incrementalize them. The problem with this – apart from it being really quite difficult – was that I don’t want purely functional style all the way through for its own sake. I want to reduce the number of sources of surprise in day-to-day software development. In practice, a magical incrementalizer can only magically incrementalize some functions, and the rules around which functions get nice big-O behaviour in the incrementalized versions are surprising indeed. So maybe that’s not such a good idea.

Instead of writing pure code and turning that into incremental code, I’m now writing code in terms of a set of primitives (lists, maps, structs) that know about acting incrementally. It works pretty well and I’ve just finished a big reworking so the guts of the thing are much simpler.

The implementation takes the user’s function (written in terms of the below primitives), and turns it into a network. The difference between this and every other FRP or related system I know of is that the change propagation network is completely distinct from the value network, and the primitives and operations are designed so that any change in the input can be pushed through to the system state with nice big-O behaviour. This is accomplished by evaluating the user’s function once – with a funny notion of evaluation – then walking all changes through the resulting network.

Every invocation of a primitive becomes a node in the network. Each node has some inner nodes; changes are propagated from each node to the nodes that have it as an inner node, and a node is only fully compiled once its inner nodes have been compiled. In the current design, change propagation goes like this:

1. A change arrives at a node
2. The node takes the change and produces a (possibly-empty) list of changes that result
3. The new list of changes are propagated from the node.

This is likely to change in a future reworking so that a node can propagate a change differently according to where it arrived from (in the current design, StructElems exist solely to work around this lack).

A change is an impulse applied at a location. A location is a path through the structs and maps that make up the value being changed. Once a change is propagated up to the top-level value, it is ready to be applied.

Primitives and Operations

Currently the following primitives and operations exist:

Lists

With lists, the overall list is a changeable evolving thing, but the individual values are not.

map :: (a -> b) -> [a] -> [b]

If the change is adding an element e, map produces a change that adds the element created by applying e to the function. If the change is removing an element e, map produces a change that removes the element created by applying e to the function.

filter :: (a -> bool) -> [a] -> [a]

For both adding and removing an element e, when applying e to the function returns true filter returns the same change. It produces no changes (effectively swallowing its input) if applying e to the function returns false.

shuffle :: (a -> b) -> [a] -> Map b [a]

Shuffle takes a list and sorts it into buckets. For both adding and removing an element e, shuffle causes that change to happen at a path determined by passing e to the function.

inputList

An inputList is necessary to feed the monster. The idea is that you have one (or more, I guess?) streams of inputs to your application. Constructing impulses that add elements to your input list allows you to have your application’s state evolve over time.

Structs

Structs are intended to hold related values. They are largely a way of translating product types into this space of incrementally-evaluated functions. Their structure and keys are fixed, but their values are changeable evolving things.

Dicts

With a dict, the overall structure is a changeable evolving thing, as are the individual values, but the keys are fixed.

map :: (a -> b) -> Dict k a -> Dict k b

This takes a map and applies the function to all the values in the map. I use this to take a shuffled input list and produce a map of structs of interesting things resulting from that list of values.

Interesting aspects of the implementation

There are two wrinkles that might be interesting if you’re trying to do something like this. The first is not a big deal: if you don’t want to split structs into a struct node plus a node for each element, then you need to be able to handle incoming changes differently depending on which node they came from.

The more interesting one is around mapDict. The arguments passed to mapDict are a function f and a dictionary d. The hopefully unsurprising goal is for every elem (k, v) in d to become (k, f v). One approach might be to generate a change propagation network for f for every element in d, but that’s going to get us a huge change propagation network over time. What can we do? It turns out that changes coming out of d are already changes at a location in d. That means that if we construct a single change propagation network for f and feed changes coming out of d into it, we’ll get suitable changes for the mapDict coming out of it.

The question then is how to do that. We can’t just apply dict to f because f is a function from a -> b, and dict is of type dict k a. Therefore, we introduce the ability for the internals to create an arbitrary node of any type, and we create such a node (which we call a) and pass it to f. At this point we have two choices:

1. Manually introduce the plumbing so that changes coming out of d get propagated to the a, and cause changes coming out of f a to get propagated out of the mapDict call. This turns out to be pretty hairy, but the hairiness is all in MapDict.
2. Declare that any arbitrary node a has an inner node (which is any watchable thing), and changes in that watchable thing appear as changes in a. This turns out to be a whole lot simpler, with the caveat that each type now needs to not just provide a mechanism for producing an arbitrary node, but also to allow that arbitrary node to have an inner node and compile it appropriately.

Spoiler alert: the second approach is better.

Future Work

There’s so much future work! So much. So much.

Right now, the distinctions between what are changeable evolving things and what aren’t is rather ad-hoc. This could stand some more rigor.

It could be quite useful to have operations like

• keys :: Map k a -> [k]
• values :: Map k a -> [a]
• elems :: Map k a -> [(k, a)]

Intriguingly, the elements of the list returned by values could be changeable evolving things (as could the second elements of elems‘ tuples), unlike other lists. This feeds back into the rigor around distinctions mentioned above.

It would probably be useful to have a way of looking up elements in a struct or map.

The syntax for invoking all of this stuff is hairy. I need either a new language or a much prettier embedding into haskell.

I have integers, but the only operation is addition, and the only way to get one is by summing a list. Generalizing that sum to a fold might be useful.

There’s a growing diamond problem under the hood. I’d like to get that under control, but it feels like it needs to get worse before it can get better. It may not turn out to get worse. There’s definitely some work to be done around removing duplication, especially around inners, ids, and the ability to be a tube.

And all of that is without looking at pushing back towards integration with a datastore or pushing forwards into the browser.