This Friday, sign up for Clojure community office hours to talk about:
- Clojure newbie questions or code review
- Clojure Confluence wiki
- Clojure JIRA tickets and patches
- Clojure web site
- Clojure conferences (Clojure/conj, Clojure/West, or writing abstracts for other conferences)
- Clojure user groups
- Comments or suggestions regarding Clojure
Office hours is a time to chat with me (Alex Miller) about a topic of your choosing. If I don't have an answer to your question, I'll track it down from the appropriate person. With proper notice, I'm happy to review code or do other preparation. I'm particularly interested in talking to people new to Clojure to find out where you encounter difficulties with tools or language.
I'm planning to continue doing this on Fridays while it seems useful. I will post a notice every week on the Clojure mailing list (not here).
I look forward to chatting with you!
core.async is a new contrib library for Clojure that adds support for asynchronous programming using channels.
There comes a time in all good programs when components or subsystems must stop communicating directly with one another. This is often achieved via the introduction of queues between the producers of data and the consumers/processors of that data. This architectural indirection ensures that important decisions can be made with some degree of independence, and leads to systems that are easier to understand, manage, monitor and change, and make better use of computational resources, etc.
On the JVM, the java.util.concurrent package provides some good concurrent blocking queues, and they are a viable and popular choice for Clojure programs. However, in order to use the queues one must dedicate one or more actual threads to their consumption. Per-thread stack allocation and task-switching overheads limit the number of threads that can be used in practice. Another limitation of j.u.c. queues is there is no way to block waiting on a set of alternatives.
Thread overheads or lack of threads often cause people to move to systems based upon events/callbacks, in the pursuit of greater efficiency (often under the misnomer 'scalability', which doesn't apply since you can't scale a single machine). Events complect communication and flow of control. While there are various mechanisms to make events/callbacks cleaner (FRP, Rx/Observables) they don't change their fundamental nature, which is that upon an event an arbitrary amount of other code is run, possibly on the same thread, leading to admonitions such as "don't do too much work in your handler", and phrases like "callback hell".
The objectives of core.async are:
- To provide facilities for independent threads of activity, communicating via queue-like channels
- To support both real threads and shared use of thread pools (in any combination), as well as ClojureScript on JS engines
- To build upon the work done on CSP and its derivatives
It is our hope that async channels will greatly simplify efficient server-side Clojure programs, and offer simpler and more robust techniques for front-end programming in ClojureScript.
In modern incarnations, the notion of a channel becomes first class, and in doing so provides us the indirection and independence we seek.
A key characteristic of channels is that they are blocking. In the most primitive form, an unbuffered channel acts as a rendezvous, any reader will await a writer and vice-versa. Buffering can be introduced, but unbounded buffering is discouraged, as bounded buffering with blocking can be an important tool coordinating pacing and back pressure, ensuring a system doesn't take on more work than it can achieve.
Just a library
core.async is a library. It doesn't modify Clojure. It is designed to support Clojure 1.5+.
You can create a channel with the chan function. This will return a channel that supports multiple writers and readers. By default, the channel is unbuffered, but you can supply a number to indicate a buffer size, or supply a buffer object created via buffer, dropping-buffer or sliding-buffer.
The fundamental operations on channels are putting and taking values. Both of those operations potentially block, but the nature of the blocking depends on the nature of the thread of control in which the operation is performed. core.async supports two kinds of threads of control - ordinary threads and IOC (inversion of control) 'threads'. Ordinary threads can be created in any manner, but IOC threads are created via go blocks. Because JS does not have threads, only go blocks and IOC threads are supported in ClojureScript.
go blocks and IOC 'threads'
go is a macro that takes its body and examines it for any channel operations. It will turn the body into a state machine. Upon reaching any blocking operation, the state machine will be 'parked' and the actual thread of control will be released. This approach is similar to that used in C# async. When the blocking operation completes, the code will be resumed (on a thread-pool thread, or the sole thread in a JS VM). In this way the inversion of control that normally leaks into the program itself with event/callback systems is encapsulated by the mechanism, and you are left with straightforward sequential code. It will also provide the illusion of threads, and more important, separable sequential subsystems, to ClojureScript.
The primary channel operations within go blocks are >! ('put') and <! ('take'). The go block itself immediately returns a channel, on which it will eventually put the value of the last expression of the body (if non-nil), and then close.
Channel on ordinary threads
There are analogous operations for use on ordinary threads - >!! ('put blocking') and <!! ('take blocking'), which will block the thread on which they are called, until complete. While you can use these operations on threads created with e.g. future, there is also a macro, thread , analogous to go, that will launch a first-class thread and similarly return a channel, and should be preferred over future for channel work.
You can put on a channel from either flavor of >!/>!! and similarly take with either of <!/<<! in any combination, i.e. the channel is oblivious to the nature of the threads which use it.
It is often desirable to be able to wait for any one (and only one) of a set of channel operations to complete. This powerful facility is made available through the alts! function (for use in go blocks), and the analogous alts!! ('alts blocking'). If more than one operation is available to complete, one can be chosen at random or by priority (i.e. in the order they are supplied). There are corresponding alt! and alt!! macros that combine the choice with conditional evaluation of expressions.
Timeouts are just channels that automatically close after a period of time. You can create one with the timeout function, then just include the timeout in an alt variant. A nice aspect of this is that timeouts can be shared between threads of control, e.g. in order to have a set of activities share a bound.
The value of values
As with STM, the pervasive use of persistent data structures offers particular benefits for CSP-style channels. In particular, it is always safe and efficient to put a Clojure data structure on a channel, without fear of its subsequent use by either the producer or consumer.
Contrasting Go language channels
core.async has obvious similarities to Go channels. Some differences with Go are:
- All of the operations are expressions (not statements)
- This is a library, not syntax
- alts! is a function (and supports a runtime-variable number of operations)
- Priority is supported in alt
Finally, Clojure is hosted, i.e. we are bringing these facilities to existing platforms, not needing a custom runtime. The flip-side is we don't have the underpinnings we would with a custom runtime. Reaching existing platforms remains a core Clojure value proposition.
I remain unenthusiastic about actors. They still couple the producer with the consumer. Yes, one can emulate or implement certain kinds of queues with actors (and, notably, people often do), but since any actor mechanism already incorporates a queue, it seems evident that queues are more primitive. It should be noted that Clojure's mechanisms for concurrent use of state remain viable, and channels are oriented towards the flow aspects of a system.
Note that, unlike other Clojure concurrency constructs, channels, like all communications, are subject to deadlocks, the simplest being waiting for a message that will never arrive, which must be dealt with manually via timeouts etc. CSP proper is amenable to certain kinds of automated correctness analysis. No work has been done on that front for core.async as yet.
Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein.
Networks channels and distribution are interesting areas for attention. We will also being doing performance tuning and refining the APIs.
I'd like to thank the team that helped bring core.async to life:
- Timothy Baldridge
- Ghadi Shayban
- Alex Miller
- Alex Redington
- Sam Umbach
And once again, Tom Faulhaber for his work on autodoc.
While the library is still in an early state , we are ready for people to start trying it out and giving us feedback. The CLJS port is still work in progress. Please have a look at the examples, which we will expand over time.
It should be noted that the protocols behind the implementation should still be considered an implementation detail for the time being, until we finish our exploratory work around network channels, which might impact their design.
I hope that these async channels will help you build simpler and more robust programs.
It seems appropriate in this, my last installment in the
(take...) interview series, to go back to the beginning and catch up with one of my early interviewees. Close to two years ago I interviewed a high-school Clojurian by the name of Anthony Grimes -- my favorite of the series. I wanted to catch up with Anthony again to see what life held for him since graduation, and possibly to get a glimpse into what the future holds.
What have you been working on since I last interviewed you?
Oh wow. Lots of stuff!
One significant project is https://www.refheap.com. It's a pastebin site written in Clojure. It uses the wonderful Pygments syntax highlighter for generating syntax highlighted code and aims to be on par with the feature offering of Gist while not using git for storing pastes and sporting a (hopefully) slightly more pleasant interface. There is still a ways to go, of course. We want to support Markdown (among other markup languages) rendering, revisions, diffs of revisions, etc. There is plenty to do and issues for almost all of it, so if anybody reading this is feeling frisky and opensourcy, take a look at them.
To go with refheap, I wrote https://github.com/Raynes/refheap.el for pasting from Emacs, https://github.com/Raynes/refheap.vim for pasting from Vim (uses Ruby), and a little Ruby gem for pasting from the command line: https://github.com/Raynes/refh. And, of course, API client libraries for refheap in Ruby and Clojure with more to come.
Some other stuff...
There is also Tentacles, my Github API client lib. Lots of people seem to like it. At least one person (hi Phil!) uses it so he doesn't have to use Github's upload file interface.
Also, Conch, a little library for shelling it. It is useful because, while clojure.java.shell is a very useful and well-written library, it (purposely) covers up all of the Java process API. Sometimes you don't want that though, and Conch exists for those times.
Finally, I've been trying to help out with Noir as much as possible. Mostly just reviewing and pulling pull requests and keeping a steady stream of beta releases going. Chris is really busy these days, so he needs all the help he can get with non-Light Table projects.
I think that's enough horn tooting. Check out my Github page if I'm just so irresistible that you must see all my projects.
How is the book coming?
Slowly. I've had a lot on my plate at work and haven't had a lot of time to work on it, unfortunately. I hope to be able to pick things up soon.
Can you explain Flatland for the uninitiated?
It's pretty simple. I work at Geni and a lot of our code is open source. We just adore modularity and we very often recognize patterns and split things out into standalone libraries. Flatland is our open source home. Things like Jiraph, our graph database, Useful, our utility library, and protobuf, our Google Protobuf library and accompanying lein plugin, among other things, are put there. You might note that lazybot, clojail, and irclj are all there as well. We decided to put those there because 1) we use them in our company IRC channels and 2) Alan Malloy and I both work on them and maintain so it makes sense for us both to have administrator access to the projects.
tl;dr: It's Geni's open source Github organization.
What was going through your mind when presenting to ~300 people at the Conj?
It's hard to recall. At first, it was something between "Just do it. Breath. Now talk. Clear your throat. Breath. Talk." and absolutely nothing at all. I actually anticipated being unable to talk and put lots of notes in my slides so that when I got nervous during the talk I could mostly just read them and avoid having to think.
About 5 minutes in it was all fine though. Once everybody had laughed or chuckled at something I said, my hair slicked back and I grew sunglasses and knew I would be okay.
My favorite two moments of the entire conference (not taking away from the talks at all, this is entirely personal) was how, after my talk, Rich told me I was great in person and said I was awesome on twitter. Easily comparable to teenage girls getting a Justin Bieber autograph. Would totally do it again.
Will we see you at the next Conj?
With bells on. I might even submit a talk again. I'd do it under one condition: at some point in my talk, Michael Fogus must come to the stage with a top hat and cane and dance for precisely 45 seconds.
It's been a wild ride trying to track down interesting Clojure programmers over the past two years, but all good things must come to and end. From the beginning of the
(take...) series I've tried to highlight all types of programmers. While it's definitely been wonderful talking to luminaries such as Rich Hickey, William Byrd, Daniel Spiewak and Martin Odersky, I've always tried to highlight the people using Clojure at work or during their free time to create, teach and inform. Undoubtably the computing industry as a whole would be much the poorer without the likes of Hickey and Odersky, but it's community that has allowed them to make a sustained impact. One of Clojure's greatest strengths is its community, and I feel an overwhelming joy to know that its members include amazing people like David Edgar Liebke, Phil Hagelberg, David Nolen, Brenton Ashworth, George Jahad, Justin Balthrop, Carin Meier, Ambrose Bonnaire-Sergeant, Sam Aaron, Arnoldo Jose Muller-Molina, Kevin Lynagh, Baishampayan Ghose, Colin Jones and Jim Crossley. Thanks to them, and all of the other brilliant Clojurians whom I was not fortunate enough to interview, for making the Clojure community one of the language's most awesome innovations.
I'm off to practice my dance steps.
Jim Crossley and his team at Red Hat are attempting to create the programming language polyglot's dream. Their latest creation Immutant is billed as a Clojure application server built on JBoss, but as you'll discover below is aiming to integrate numerous Java.next languages. Immutant is an exciting project and looks destined to become something special.
What is the "elevator pitch" for Immutant?
Immutant is an application server for Clojure built atop JBoss AS7.
Our primary goal is a robust, scalable deployment platform for Clojure applications, with built-in services like messaging, caching, distributed transactions and clustering. We aim to encapsulate the accidental complexity associated with most non-trivial applications and allow the developer to focus on the essential complexity of his problem domain.
Immutant is an integrated stack of services. Instead of provisioning your servers with Jetty for web, RabbitMQ for messaging, and Memcached for caching, for example, you could simply install Immutant.
Were there any barriers to using Clojure in the implementation of Immutant?
Quite the contrary, Clojure has enabled us to remove much of the boilerplate typically associated with JEE services, so we've been able to encapsulate the complex JEE API's and configuration within a small set of Clojure functions.
Clojure also inspired us to support more dynamic applications. It's not necessary to declare all the services required by your app in some sort of "deployment descriptor" for Immutant; those decisions can be deferred until runtime. We also let you expose the application running within Immutant via either Swank or nREPL, enabling a more interactive development experience than what folks typically expect from a JEE container.
So far, the biggest challenge has been working around Clojure's assumption that there is only one Clojure runtime per JVM - we wanted each application deployed within an Immutant to have its own runtime, and even its own Clojure version. To work around this, each application gets its own isolated ClassLoader and we do some classloader munging to give each application its own runtime.
We're aided in this by JBoss AS7's modular classloading system - it does a great job of isolating an application's dependencies from those of other applications or Immutant itself.
Why do you think that there are some who are mistrustful of "Clojure in the enterprise"?
Answering this question requires making broad assumptions about the fears and motivations of others, not to mention an agreed-upon definition of a pretty overloaded term... but what the hell! :)
Seriously, I don't think good developers are mistrustful of Clojure, though some may be justifiably leery of bloated application servers that are traditionally associated with enterprises. JBoss AS7 is not your father's app server: though obviously bigger than any single component of a home-grown stack, it's fast and light enough to not make you "stabby" while coding on your laptop.
But developers are usually only one minority amongst the decision-making stakeholders in an enterprise...
How can Immutant alleviate those doubts?
Oddly, the integrated, java-app-server-ness aspect of Immutant that gives some developers pause is the very thing that IT managers and sys-admins love. Immutant offers a great opportunity to "get Clojure in the enterprise" because it can deploy those Java legacy apps right alongside the newer Clojure/JRuby/etc ones, with minimal impact to the devops guys.
But truthfully, we don't really care about that. We're looking for ways to make development and deployment simple. And by simple, I mean no more complicated than what's required to address the problems within a specific domain.
And though it's true that forces acting on an enterprise often complect applications in ways that don't affect individuals or small teams, we think simpler development and deployment benefits everyone, whether you develop apps for/in an enterprise or not.
What is the state of Clojure in the broader Red Hat corporation?
Well, when we started Immutant almost a year ago, it was pretty much just me and Toby. ;)
We're part of a research team at Red Hat led by our "Director of Polyglot", Bob McWhirter. We're fortunate to have recently added Charlie Nutter and Tom Enebo (the JRuby guys) to our team, and both are very interested in alternative JVM languages (including Clojure) and eager to apply their skills to our JVM-related projects, including TorqueBox, Immutant and OpenJDK, among others.
Red Hat is a very large company of very smart engineers on very long leashes, so I have little doubt others are using Clojure internally. And we have a solid mandate from our leadership to expose the excellent JBoss services through as many JVM-based languages as makes sense.
Do you think that there is a synergy between JRuby and Clojure in the enterprise?
Absolutely! And not just in the enterprise. The proliferation of web frameworks and tools for Ruby make it a great language for developing web UI's and/or services, whereas to this point Clojure has mostly been used to create some pretty incredible data analytics applications.
It's rare to find individual developers with enough expertise in both JRuby and Clojure to effectively combine frameworks and libraries from each. But we believe they're out there, and as the benefits of polyglot programming become more publicized, their numbers will increase. But in the meantime, we expect larger organizations to form teams of experts from both camps, each producing loosely-coupled applications that coordinate to solve domain-specific problems.
We hope Immutant will appeal to both individuals and teams by allowing those seemingly disparate applications to be easily developed and deployed to a single platform.
 Best job title ever!
Colin Jones is a software craftsman at 8th Light and Clojure veteran. His contributions to the Clojure ecosystem are far and wide, but tend toward helping newcomers to discover the language and provide tools for getting started as quickly as possible. In this installment of the
(take ...) series Colin discusses koans, TDD, REPL-ing, and the Clojure Conj.
What are koans in the context of programming?
The basic idea is that they're exercises to help you learn a particular language or other technology. There's a Zen metaphor that ties things together, but the main thing is to have small, bite-sized chunks to guide folks in their learning without being overwhelming. I first heard about the idea in the EdgeCase Ruby Koans, and they in turn credit other sources (including The Little Lisper! with inspiration. So when Aaron Bedra started the Functional Koans project, I was excited to pitch in, and before I knew it, we had a bunch of people having fun working through the Clojure Koans, as their first exposure to Clojure.
Is there a place for TDD in the Clojure ecosystem?
Of course! I think Brian Marick has shown that most publicly with his library Midje, and related screencasts and conference talks, but there are a ton of other great testing tools out there as well: Speclj by my boss, Micah Martin, LazyTest, Expectations, clojure.test, etc.
I have heard TDD and deep thinking put into stark contrast with each other at times, but the people I respect in the TDD world certainly do a lot of deep thinking about simple design, so I think we sometimes get a bit of a false dichotomy.
I personally find a lot of benefits to unit testing - it's a way of making my thinking about problems concrete, and when I'm done, there's an automated system looking for assumptions that have become untrue.
What itch are you attempting to scratch with your latest project REPL-y?
OK, the 5-second version is that I want it to be easier to use a Clojure REPL, unix-style, at the shell. Projects like Russ Olsen's dejour were working towards similar goals, which I found exciting, so I got in touch with Russ to pitch in. As it happened, a lot of my ideas for improvement could be implemented orthogonally to dejour: handling Ctrl-C interruptions without bailing out of the process, basic code completion, in-REPL examples (via clojuredocs), and a truer readline-like interface (because let's face it: JLine 0.9.94 is broken). Hence REPL-y was born.
Down the line a bit, with some encouragement and help from Phil Hagelberg and the Leiningen team, it's now the default REPL frontend for the upcoming Leiningen 2, which is now in preview (nREPL is the backend). So at this point, REPL-y is wiring up of a bunch of great projects: Clojure, JLine2, clojure-complete, nREPL, and clojuredocs-client. There's a shell script to run REPL-y alone, but of course I expect the version in Leiningen to see a lot more use.
The project lives at https://github.com/trptcolin/reply (varying pronunciations are both acceptable and hilarious).
How can Clojure improve, in general?
Language-wise, I don't have too much to say. I know a lot of folks struggle with the various ways of pulling code in from other namespaces, so maybe there are ways to make those easier to use. My main interests have mostly been in helping people get over hurdles - which is of course a cause shared by a lot of people on the IRC channel and beyond. So, needless to say, I was very impressed by the fact that the planning for Clojure 1.4 included error messages, documentation, even potentially a command-line launcher app.
Tools-wise, I think things are moving in the right direction. Of course I mentioned the REPL stuff and Leiningen, and there are dozens of great new things coming out all the time.
I'd love to see clojure.org become a bit friendlier as an introduction
to Clojure, from the very first steps. I wonder if it'd be possible to have something like the Java Tutorials that's maintained
in an "official" way? There's so much incredibly good information on clojure.org - I refer to it often - but it can be intimidating, and things get out of date from time to time. I also think function findability can be hard in the clojure.core namespace from a REPL if you don't know what you're looking for, like if you want an operation on a particular data structure but don't remember the function name, so maybe a similar function to
find-doc could do some nice grouping, like what's laid out on http://clojure.org/data_structures.
First impressions matter, and if we can hook people right at the start without compromising our principles, I think that's a really big win for the language and the community.
You've been to both Clojure conferences; was this years' offering different than the first?
I was pretty bummed about the lack of Clojure temporary tattoos this year, but you can't win them all.
Seriously, though, this conference is amazing. Certainly there were differences: last year's seemed a bit more language-focused, and this year's seemed more product- and library- focused. That's a pretty broad and unscientific impression, but suffice to say, both were awesome.
Some of the bonuses this year: an additional day of talks, bunches of informal sessions that sprung up in the conference space, and lots of great restaurants within walking distance. The talks, like last year, were fantastic. I suspect we'll see even more of these great stories about the things people are building as Clojure gets used in production more and more.
Plus, Overtone? Bad ass.