(take 5 daniel-spiewak)

Posted by on Apr 19, 2012


Daniel Spiewak is a force of nature. As a highly respected member of the Scala programming language community and an overall thoughtful polyglot he seemed a natural fit as an interesting speaker for the 2011 Clojure/Conj. Daniel's talk entitled *Extreme Cleverness: Functional Data Structures in Scala was highly energetic and astoundingly informative. Daniel's open source contributions are not to be forgotten however. In addition to the important anti-xml Scala library, Daniel is attempting a bit of text-based collaboration magic with his Common Colaborative Coding Protocol project.

In this interview we talk about Clojure and its community, the conj function and Java.next languages learning from each other and their mutual struggle for mindshare.

What are your thoughts on the Clojure community?

Absolutely awesome! I've had the supreme pleasure of spending a fair amount of time amongst Clojurists (Clojurers??), and it's always a blast. Thanks in no small part to Rich Hickey, the Clojure community is smart, creative and absolutely intolerant of intolerance. The very fact that I was asked, as a Scala speaker, to come and present at Clojure/conj is a testament to how welcoming this community really is. Communities of this sort tend to be incredibly fertile soil for new ideas and profound advancement of the state of the art. I eagerly anticipate pillaging future innovations!

Are there any lessons that Clojure could learn from Scala?

I think there are a few lessons, the most important being that rich and uniform collections are extremely valuable. One of the things that drives me up the wall in Clojure is the following:

  
    (conj [1 2 3] 4)     ; => [1 2 3 4]
    (conj '(1 2 3) 4)    ; => (4 1 2 3)
  

Thus, the behavior of conj is a little bit unclear, since it depends on the input type. The object-oriented analogue to this would be if we defined two implementations of an interface, each defining the same method in opposite ways. I can't even begin to imagine what Liskov would say to that.

It should be noted that although Daniel is absolutely correct that conj depends on the input type, it's idea is that it will do the most efficient action given its input type. For lists the most effiecient action is to place an item at the front, for vectors elements are put onto the back.

Even more annoying is the following:

  
  (drop 1 [1 2 3 4])    ; => (2 3 4)
  

So, I drop the first element of a vector and get a list?! That seems, well, weird. Combining this with the conj issue, we can get something really bizarre:

  
    (conj (drop 1 [1 2 3 4]) 1)        ; => (1 2 3 4)
    (conj [2 3 4] 1)                   ; => [2 3 4 1]
  

In other words, Clojure's sequence functions complect behavior with input type.

(my understanding is that a lot of these issues are resolved in ClojureScript, but I haven't had a chance to really look yet)

Clojure's sequence abstraction is a neat idea in theory, but the practice leaves something to be desired. The root of the problem here, incidentally, is that Clojure's sequence abstraction is a little bit of subtyping embedded within an otherwise functional language. This problem could be resolved by playing the same trick that Scala's collections do, putting factory accessors on each collection to allow functions to build a collection of a dynamically-determined type.

For anyone who might not know, Scala's collections solve this in a really nice way, so everything falls out essentially the way you would expect:

  
    List(1, 2, 3) drop 1      // => List(2, 3)
    Vector(1, 2, 3) drop 1    // => Vector(2, 3)

    0 +: List(1, 2, 3)        // => List(0, 1, 2, 3)
    0 +: Vector(1, 2, 3)      // => Vector(0, 1, 2, 3)
  

We can even do fancier things, like working with data structures that can only contain certain data types:

  
    BitSet(1, 2, 3) map { _ * 2 }       // => BitSet(2, 4, 6)
    BitSet(1, 2, 3) map { _.toString }  // => Set("1", "2", "3")
  

It's all very nice and extremely simple (though certainly not simple at all if you want to actually write a new collection, but I digress...). Granted, it has taken Scala four full collections rewrites to get to this point, but it's nice now that we're here! Some of this is using the magic of Scala's type system (like the BitSet thing), but almost all of it could be applied to Clojure.

Aside from collections, I would say that there are a few things that Scala does better than Clojure, but I'm not entirely sure I want Clojure to go down those roads. Polymorphic modules are super-useful, but they bring with them a giant raft of complexity. Virtual dispatch is a gateway to a lot of syntactic power that Clojure really cannot achieve without dodgy macros (for example, see Scala's parser combinator DSL), but again, lots of complexity and not really a good fit for the rest of the Clojure language.

Is there room for both Scala and Clojure?

Absolutely! So, here's the thing: neither Scala nor Clojure are going to displace Java. It's just not going to happen. Both Scala and Clojure are substantially simpler than Java, but vastly harder. Java has achieved what I like to call the "boat anchor" phase of a language lifecycle, where it is so ubiquitous that it has become impossible to unseat without a massive paradigm shift. An example of such a shift would be if quantum computing came around and no one ported the JVM to run on that architecture (or perhaps if it were to run substantially worse than other VMs).

In the meantime, we have a whole bunch of alternative JVM languages who's adoption is likely to remain rounding error for quite some time. Paradoxically, this is very good for the relationship between Scala and Clojure. Our biggest hurdle is overcoming Java's inertia, and that hurdle is so large that petty little power struggles between alternative languages are rendered insignificant.

In other words, it's not a zero sum game between Scala and Clojure. When a new developer picks up Scala or Clojure, they are choosing to bridge a gigantic chasm from the "mainstream" languages to something a little more on the fringe. Yes, this is a win for whatever specific language they choose, but it's an even bigger win for alternative languages in general. People need to realize that they have options, and in particular that they have options on the JVM. Every developer learning or being exposed to an alternative language is a win for everyone, and right now that is the lion's share of the battle.

In the long term, as Clojure and Scala grow in adoption, we may find that the languages are starting to be in competition for the same problem spaces. However, if and when this happens, I suspect the languages will be sufficiently divergent as to have generally disjoint niches. Even today, if you know both Scala and Clojure, there is rarely any ambiguity as to when you should apply one rather than the other. This is likely to continue even as alternative languages steal more and more of Java's developer share.

An idea that has a cult following in Clojure is that of optional (and pluggable) type systems.

What are your thoughts on this perspective?

That's an interesting question. Right off the bat, I think it's important to point out that a "pluggable" type system immediately implies an optional type system. There is also a weaker implication that such a type system could be split out from the normal compilation process and run separately. This is certainly something that I've heard Rich talk about on several occasions, and I think it's an interesting idea. However, there are a couple immediate things to point out.

For starters, I'm not entirely sure that a separate, pluggable static analysis phase is in fact a type system at all, particularly if you aren't using it to definitively reject programs. In a formal context, type systems are an intrinsic and (generally) inseparable part of the language. If you have a type system that is separable from the language, that type system is not a type system at all but merely static analysis. Now, there's absolutely nothing wrong with that! I think static analysis is a very useful tool and one which can answer much deeper and (often) more revealing questions than a baked-in static type checker. However, I think it's a definitional hair that's worth splitting because it has some fairly profound implications (such as the strength of your guarantees and how much you have to worry about composability).

Addressing the idea in general: I think it's a good one. The more questions we can answer about our code before it sits in front of customers, the better we are. We live in the information age, where an incredible amount of research and effort is being put into developing tools that allow us to make sense of truly astronomical amounts of data. Why shouldn't we be applying those tools and techniques to programming? I've seen some code bases which could qualify as astronomical in size, particularly once you consider the information density of most programming languages. Why shouldn't we be using rich, statistical tools to ask semantically deep questions about our code base? This seems like a natural evolution of the modern development process, at least to me. Pluggable static analyses make it possible to apply these sorts of techniques without being tied to the compilation cycle.

There are some very deep traps to beware though. For example, a "pluggable" type system is really not very useful unless you can compose multiple type systems into a coherant whole, and this is where problems arise. Anyone who has studied type theory in a formal context and run a few soundness proofs will understand that seemly innocuous and self-contained type rules will almost always interact in surprising ways. A good example of this is extending the simply-typed lambda calculus with reference values. The moment you do this, your type system explodes in complexity, despite the conceptually tiny nature of the change.

The fact is that you can't just tease type systems apart into composable atoms. Their features are intertwined; they are the very soul of complexity. (note: I'm not saying that they are complex to use or even to understand, but they are certainly complex to design and build) So, while I think that we're going to see an up-tick in the richness and proliferation of static analysis tools, I do not think you're going to see them really replacing type systems. I see these two concepts and orthogonal and complementary.

Is there anything in Clojure that you wish that Scala provided?

Oh, wow... I think the biggest thing is that Clojure has a much stronger focus on functional programming in that it carrots developers (nearly forcibly) to control their state. Anyone reading this who hasn't already watched Rich's talk on concurrency needs to go out and do that right now (then watch it again). The notion of epochal time is quite central to Clojure, and its benefits are manifest. Scala on the other hand is a little bit more laissez-faire with respect to state. Well-written Scala code is going to keep state on a tight leash and will end up looking a great deal like well-written Clojure code, at least in terms of where state is and how it is respected. However, Scala provides comparatively few incentives for developers to do the Right Thing. Unlike Java, it doesn't provide disincentives, but it also doesn't bias the coin in the right direction as Clojure does.

On another note, Clojure is a much simpler language than Scala, and that makes it very nice for a lot of things. Clojure code has an aesthetic which really appeals to me. That's not to say that Scala is overly complex or ugly, I just happen to really like a lot of what I've seen from Clojure. I certainly think there are syntactic corners of Scala that could be substantially smoothed. Maybe not really taking inspiration from Clojure, but certainly improving things in an area where Clojure is quite strong.

By and large though, I think that most of the awesome bullet-points that Clojure hits have already been stolen whole-sale by Scala. :-) A few examples: vector, map/set, agents (in Akka), STM (in Scala STM), SLIME (see ENSIME), etc. The Scala community is very actively watching the Clojure community. Y'all are a very fertile source of inspiration!



blog comments powered by Disqus