The second Clojure Conj started with a bang with a drink-up sponsored by Heroku. Many an old Internet friend was made and reacquainted in the haze of free beer and wine. It was immediately apparent that the excitement for the next day's festivities was palpable.

Day 1

Before I get into discussion of the talks, I would like to recognize the hard work of Chris Redinger for organizing the Conj and Lynn Grogan and Naoko Chamberlain for their efforts in keeping the Conj running smoothly in progress. Who knows what horror might have befallen us without their iron will and focus.

most of the slides for the talks are available in the Relevance Conj repo

Learning Clojure - Next Steps - Stuart Sierra

Stuart Sierra, a Clojure/core member and co-author of Practical Clojure started the Conj off with a bang. The premise of the talk was simply: you've read the books, mastered 4Clojure, and spend 14 hours a day on IRC -- what's next?

Sierra answers that question by enumerating a fairly long list of project ideas that focus on the lesser-known features of Clojure and explore its deeper subtleties. For example:

  • Extending the Clojure reader
  • Using Clojure from Java
  • Utilizing the JVM's base concurrency primitives
  • Using Clojure's rich data-structures instead of JSON
  • Exploring important libraries like core.logic and test.generative

In all, if an enterprising group of programmers were to explore every recommendation provided during Sierra's talk, then the state of the Clojure ecosystem would look very different by the next Conj.

Concurrent Stream Processing - David McNeal

David McNeil as an employee of Revelytix described a system whereby SPARQL queries are split among various storage servers and the results concurrently reassembled via fork-join. The system in play is one analogous to UNIX pipelines with tee support.

Clojail - Anthony Grimes

When the Java programming language was released to the public in 1995 Anthony Grimes was 1 year old. Let that soak in for a moment.

Anthony v1.0 and Java v1.0.

In the context of that juxtaposition it was a blast to watch the resident Clojure prodigy talk about his project Clojail; a sandboxed execution environment for Clojure. One by one Anthony listed the complexities of developing such a system and the ways that he and his friends and collaborators solved them. Anthony's style is an interesting mix of technical chops and deadpan humor and at times he had the room of close to 300 attendees in a laugh riot.

About Clojail itself, the project is used on various Clojure projects including, but not limited to 4Clojure and Try Clojure.

ClojureScript - Chris Houser

A few years ago Chris Houser wrote an implementation of Clojure in JavaScript named ClojureScript that provided interesting insights into how to (and not to) go about targeting JavaScript for Clojure. Earlier this year Chris was approached by Rich Hickey and the Clojure/core team to help with another implementation of Clojure targeting JavaScript, aptly named ClojureScript. Confused yet?

In his talk, Chris discussed some of the details of the latest ClojureScript implementation.

Data-Structures

Chouser started by talking about the implementation of Clojure's immutable data-structures for ClojureScript. The ClojureScript structures provide a subset of those found in Clojure in that although they are the familiar Maps, Vectors, and Sets they are not yet persistent but instead follow copy-on-write semantics.

defprotocols and deftypes

Although Chouser didn't go into exquisite detail about the implementation details of ClojureScript protocols and types I can provide some additional insight there. That is, ClojureScript provides open extension of existing types (including those of existing JavaScript libraries) by emitting code not of the functional inheritance pattern as espoused by Douglas Crockford, but instead in the vein of the pseudoclassical pattern used in Google Closure. For example, a simplified example of the pseudoclassical pattern is as follows:

    Person = function(name) {
      this._name = name;
    };

    Person.prototype.getName = function() {
      return this._name;
    };

    Man = function(name) {
      Person.call(this, name);
    };

    inherits(Man, Person);

    Man.prototype.getName = function() {
      return "Mr. " 
             + Man._super.getName.call(this);
    };

While this type of inheritance pattern is devoid of private data (not terribly surprising given Clojure's likewise view), it allows adding behavior to existing instances. Take the following example:

    Person = function(name) {
      this._name = name;
    };

    Person.prototype.getName = function() {
      return this._name;
    };

    var me  = new Person(‘Fogus’);
    var you = new Person(‘Cercei’);

To add a hasBeard method post hoc to every instance of Person is as simple as Person.prototype.hasBeard = function() { return false };. This type of extension is critical in the implementation of ClojureScript.

Chouser also talked about the Clojure analysis phase, but I will hold off discussing that for a future post.

Striving to Make Things Simple and Fast - Phil Bagwell

Phil Bagwell is an employee of Typesafe, a company built to provide Scala consulting and advocacy. As a member of Typesafe, Phil's official position is in marketing, but his talk was about his hobby -- datastructure design. However, unlike most hobbies I would say that Phil has done more than merely dabbling in a field. Instead, one of his data-structures, the bit partitioned hash trie is the basis of Clojure's hash-map implementation.

However, in this talk the discussion was around a recent datastructure called Relaxed Radix Balanced Trees (RRB-Tree) that provide fast concatenation, split, and indexed update operations. While the tech was interesting, the added anecdotal format of the process by which he and his team arrived at the final form of the RRB-Tree was illuminating. For any project of significant complexity the team was forced to weigh the tradeoffs in each intermediate implementation while at the same time remaining unwavering in their performance goals. As you can imagine he and his team experienced their fair share of eureka moments along the way toward an elegant solution that indeed may prove useful in future versions of Clojure or as a library. The community study of this latest gem will be interesting to participate in and follow.

Introduction to Logic Programming with Clojure - Ambrose Bonnaire-Sergeant

In my experience the progression of a programmer interested in functional programming eventually finds its way into an interest in declarative programming and the Clojure community as a collective is following this path. One of the key contributors to the library is University of Western Australia undergrad Ambrose Bonnaire-Sergeant, who gave a very nice talk about programming in a declarative logic dialect called miniKanren. The miniKanren system at the heart of core.logic is an enhanced version of the system described in the great book The Reasoned Schemer.

There is much more to this story, but I know for a fact that my friend and colleague Stuart Sierra has grand plans for a series of blog posts on this very subject.

Hacking the Human Genome using Clojure and Similarity Search - Arnoldo Jose Muller-Molina

One of my personal favorite talks at the Conj was the description of using Clojure to find binding regions in gene sequences. While at times the technical content seemed quite complex, Arnoldo nicely presented the material in a way that a layman such as myself could comprehend. The work by Arnoldo and his team is nicely illustrative of Clojure's particular strength in processing huge data sets. The enthusiasm of Arnoldo in explaining his work was infectious as it was clear that we were observing a master craftsman using a sharpened tool to make a true difference in our understanding of the human genome.

Ousterhout's Dichotomy Isn't (Part 2 of the Simplicity/Power/Focus Series) - Stuart Halloway

To end the first day of (official) talks, Stuart Halloway gave a talk on power in the same vein as his previous talk along a philosophical bent regarding simplicity. An interesting trend is occurring in Clojure that I have personally never experienced as a member of any other software community. That is, there seems to be a definite effort to take a step back and examine the meaning of some of the words that fly so freely from the mouths and fingertips of programmers and computer scientists. It would seem that in many cases the words "simple", "easy", and "powerful" are taken for granted, but as a result they've almost lost their meaning. Any attempt to regain a common meaning regarding these loose terms is a difficult task, but with his previous talk on simplicity and his latest on power, Stu seems to have made headway in fostering a common lexicon for discussion.

Day 2

(Neal's) Master Plan for Clojure Enterprise Mindshare Domination - Neal Ford

Oddly enough I had never had the privilege to watch Neal Ford give a talk, and after finally having done so with his first talk of the second day I feel somewhat cheated in my ignorance. It's always astonishing to watch a master at any particular craft operate, and in the case of Neal it was clear that he is a formidable speaker. Honestly he could have engaged my attention fully if he'd talked about any topic under the sun, but fortunately for those in attendance he spoke on the matter of gaining a foothold in the enterprise for Clojure. Having previously come from an enterprise shop that freely used Scala, Python, and GWT (with a smattering of Clojure for tooling and one-offs) much of what Neal said was familiar and nostalgic.

One approach that Neal promoted was to use viral tactics to incorporate the clojure.jar into your existing enterprise projects. Once it's in it can be used. Of course this is half tongue in cheek, but the serious side is sound. The inclusion of Clojure in your stack can start modestly and spread into perhaps a test suite and eventually evolve into a one-off tool, perhaps a parser here, a migration tool there. If we spread the word about Clojure within our work place and prove its usefulness, then eventually it can gain a substantial mindshare in the enterprise.

A relevant question was asked during the Q&A that boiled down this this:

Does anyone care if Clojure gains enterprise support?

My personal answer is that I do. Having spent my time in enterprise shops I would have found my life more interesting and my output more voluminous if I had a tool like Clojure at my disposal.

Predicate Dispatch - David Nolen

David Nolen followed Neal Ford's workaday advice with a topic that is near and dear to my heart; predicate dispatch. One of Clojure's core principles is that of openness. For example, Clojure's multimethods are a way to provide function dispatch based on the result of an arbitrary function. For example:

    (defmulti kw->num identity)
    (defmethod kw->num :one [_] 1)
    (defmethod kw->num :two [_] 2)
    ;; ... etc ...
    
    (kw->num :one)
    ;=> 1
    
    (kw->num :three)
    ; No method in multimethod 'kw->num' for 
    ;  dispatch value: :three

Although I never defined a method for the dispatch value :three, Clojure's multimethod system is open for extension at any point. Therefore, I can simply add a new method for my new case:

    (defmethod kw->num :three [_] 3)
    
    (kw->num :three)                    
    ;=> 3

However, this system is only partially open. That is, where new methods can be added (and logical hierarchies extended, which I didn't show) the original dispatch function identity is baked into the multimethod kw->num and cannot be modified[1]. This is where predicate dispatch comes in.

Predicate dispatch would provide a system that is completely open at the level of the dispatch logic and its instantiations. A possible syntax for such a system in Clojure might look like the following:

    (defpred even? even?)
    (defpred odd? odd?)
    
    (defm classify-num [(_ :when even?)] :even)
    (defm classify-num [(_ :when odd?)]  :odd)
    
    (map classify-num [1 2 3 4 5])
    ;=> (:odd :even :odd :even :odd)

What this code does is to define two predicates odd? and even? in terms of the existing functions of the same name. It then defines a function classify-num in terms of the result of the predicates available. The best part about this scheme is that the dispatch logic open to extension on the function by function basis:

    (defpred special-num? #{0 22/7})
    
    (defm classify-num [(n :when special-num? 0)] :zero)
    (defm classify-num [(n :when special-num? 22/7)] :fake-pi)

    (map classify-num (range 5))
    ;=> (:zero :odd :even :odd :even)

Bear in mind that this is all in David's head at the moment, but if his previous projects are any indicator then the future of predciate dispatch in Clojure looks bright indeed.

[1] That's not entirely true since the dispatch function can also be a multimethod, but a chain of such is a recipe for inefficiencies.

Extending JavaScript Libraries from ClojureScript - Kevin Lynagh

As a core developer on ClojureScript I was very excited to see Kevin Lynagh's talk on his team's use of ClojureScript in integrating with the D3 (Data-driven Documents) library. It's the early adopters of a new software system that are invaluable for implementors since they will push said system to its limits and beyond.

Like any good case-study Kevin's presentation sprinkled in a healthy share of gotchas with the heaps of praise. It's this kind of execellent feedback that will help the ClojureScript contributors think about the problems and solve them post-haste.

From Linear to Incremental - Christophe Grand

Every time that I hear Christophe Grand talk I want to run off and spend a year studying in hopes of understanding the depth of his ideas. Last year I was very impressed by his talk (not= DSL macros) -- so much so that my own talk this year was primarily inspired by it. This year, Christophe gave us a window into his thought processes behind the implementation of his incremental parsing library Parsley.

The problem in summary is that the form of the parser is such that it is analogous to the operation of reduce logic. The problem with this is that in terms of the reduction scheme there is no way to perform incremental parsing without fully restarting the parse from the beginning. The solution per Christophe was to view reduction as a composition of function applications of component values in a seq and the incremental accumulator. Because this strategy is built using function composition (comp) the parse nodes are associative and therefore parallelizable. On the change of a source document, the parser can be recalculated in the context of the changed node and its parents. However, this recalculation only happens within the context of the parser composition and not at the level of the parse itself. To alleviate this Christophe memoizes the results of the parse nodes and therefore restarts are fast because they only calculate at the level of the fresh nodes.

Like most things that Christophe devises, the Parsley strategy is extremely elegant and worth exploring side-to-side with the slides from his talk.

Logs as Data - Mark McGranaghan

Mark McGranaghan is an employee of Heroku and the creator of Ring (think Clojure's analogue to Rack). As an employee of Heroku Mark and his team frequently deal with large volumes of execution log data. Like any enormous volume of data, log data can be viewed and handled as a "big data" problem. The problem is that typically log data is devoid of information independent of an active parse step. That is, log files are giant heaps of text with a form of the like:

    info: user foo viewed /home

The talk focused on the relative advantages of instead viewing the same data as an associative data structure, a la:

    {:level :info
     :user "foo"
     :action :view
     :target "/home"}

And using existing abstractions and functions within Clojure to deal with this type of data as ongoing operations within a realtime event stream -- namely Heroku's Pulse dashboard. There are an increasing number of companies making huge bets on Clojure but very few have placed bets higher than Heroku. It's great to see that Heroku is not only providing Clojure services, but also leveraging the language directly in their operations.

Modeling the world probabilistically using Bayesian networks in Clojure - Chas Emerick

Chas Emerick gave an interesting talk on a bayesian classification library that he used in his own work for document analysis tasks. Chas is a co-author of O'Reilly's upcoming book Clojure Programming and as a small business owner has placed a bet on Clojure for his livelihood every bit as enormous as bigger players like Heroku; perhaps even bigger.

Clojure and Android - Daniel Solano Gomez

Daniel Solano Gomez is a hacker's hacker. The general notion before Daniel started his quest to run Clojure on Android was that it couldn't be done. You know how this story ends though... he did it. However, there were definite challenges that forced him to make tradeoffs and maintain a separate fork of the Clojure compiler and runtime. Much of Daniel's talk focused on these challenges, including:

  • Memory size of the runtime
  • Startup times
  • Garbage collection thrashing

Not satisfied with merely enumerating the problems encountered, Daniel also provided advise and actionable recommendations to the Clojure/dev team for alleviating and in some cases solving the limitations encountered in his quest. I for one welcome his thoughts and changes as we move forward into a future where Clojure is a viable choice for Android development.

Exciting times.

Keynote - Rich Hickey

Like Rich's first talk at last year's Conj, the keynote this year was a technical assessment of the state of Clojure with a focus on the types of problems that the Clojure/core team is thinking about, designing, and soliciting help with.

Clojure/ClojureScript unification

With the release of ClojureScript there is an opportunity to leverage existing Clojure libraries. However, the precise mechanics for accomplishing such reuse is not yet in place, but one possible option is conditional compilation along the lines of that available in Common Lisp. Before concluding this portion of his talk, Rich made a call for suggestions in this space should anyone have a better idea than the CL model. This call is still on the table.

Extensible reader

While Clojure's rich data format provides a wealth of types above those provided by JSON (for example), it is limited to those types currently provided. Therefore, a lot of recent thought has gone into ways for safely providing a point of extension of the Clojure reader. With the availability of an extensible reader, the Clojure data formats have a chance to provide arbitrary serialization formats independent of the concrete types represented. That is the formats across the wire would be stable, while the concrete instance types would be (potentially) different on a per-project basis. The extensibility component would therefore be a contextual shim between the reader and the application.

Small footprint and debug compilation targets

Building on the work by Daniel Solano Gomez mentioned above, the Clojure compiler could be modified to provide different compilation targets, each built for a special need. For example, one possible compilation mode might remove the use and storage of metadata at runtime while another could remove assertions from runtime code while another could provide a profile optimized for running on constrained targets like Android. The possibilities are limitless in this space.

Pods

Pods are meant to be useful in coordinating access to other mutable things. The idea has floated around the Clojure zeitgeist for at least a year and a half, but they are attempting to solve a very hard problem and therefore require a lot of thought and design.

Day 3

Cascalog - Nathan Marz

Nathan Marz started day 3 with a bang displaying his amazing Cascalog library that compiles datalog queries into Map-reduce jobs. Datalog in the context of Cascalog is a lightweight query syntax that provides a concise description of the data under query. Cascalog is currently used by Twitter to query great gobs of data and the examples displayed by Nathan were representational of the power of his great system.

Functional Data-Structures - Daniel Spiewak

Daniel Spiewak's reputation preceded him. That is, from observing the Scala community chatter regarding conference speakers, it was well known that a talk from Daniel was an energy filled onslaught of brilliance. This indeed was the case. However, what I was not expecting (sorry Daniel) was that the talk would be incredibly lucid. Daniel is an amazing speaker in that he is able to explain details of complex topic, of which he knows in exquisite detail, with crystalline clarity.

Having said all of that, Daniel's talk was on the details of various functional data-structures, their efficiency concerns, and implementation details. I loved this talk and highly recommend it to anyone wishing to learn more about the modus operandi of Scala and Clojure datastructure models.

Macronomicon - Me

This was my talk and so I hesitate to say too much about it in fear of overinflating my contribution. In the spirit of completeness however, I will say that the talk was focused on the art and science of Clojure macros. The talk was inspired and in fact a direct response to Christophe Grand's talk from the previous Conj entitled "(not= DSL macros)". However, I tried to present further compelling cases for macros above those described by Christophe in the context of my own projects Trammel and Minderbinder. If you want to know more about the talk then you'll have to ask someone else who attended, read my blog "Send More Paramedics", or wait for the release of the video.

Performance in the Wild - Craig Andera

Craig Andera is a co-worker of mine and his talk was a description of the practical matter of performance measurement in the context of a fairly large Clojure project (that I was not involved with). However, although the talk was set in the Clojure context, most of his advice and observations were universally applicable. The interesting anecdotal points in the talk revolved mostly around the various choke points experienced in the course of meeting and measuring performance expectations. Of further note is that the various slow-downs experienced had nothing to do with the use of Clojure, but were instead caused by other factors. Craig is a smart guy who knows how to tell a great story at the same time.

Programming Music with Overtone - Sam Aaron

An interesting tweet circulated during Sam Aaron's Overtone talk:

"All the speakers were just opening for the headliner, Sam Aaron."

And this is an accurate assessment. Watching Sam go over the philosophy of "open source music" was nicely concluded with a jam session within the Clojure REPL using Overtone. It was a fitting end to a wonderful conference. His standing ovation was well-earned.

This concludes my run-down of the mainline talks for the 2011 Clojure Conj. In my next post I'll provide a similar run down of the after-hour talks, BOFs, and hallway tracks.

there is similar analysis of the Conj from Nicholas Watkins Brown, Gary Poster, and Ted Leung

:F