Fortran, Clojure, Haskell And Julia Are Not At War

Ars Technica featured an article about a year ago which asked whether Clojure (or a couple of other languages) could overthrow Fortran as the language of scientific computing. It was kind of an odd article, and now Peter Fraenkel has written an excellent rebuttal. As he points out:

  • Ars Technica notwithstanding, Clojure, Haskell and Julia are not locked in competition to replace Fortran.
  • Ars Technica notwithstanding, Fibonacci sequences may be something we learned about once, but it is difficult to disguise their irrelevance to scientific computing.

Incanter scientific package for Clojure

Incanter is a Clojure-based, R-like platform for statistical computing and graphics. Incanter can be used as a standalone, interactive data analysis environment or embedded within other analytics systems as a modular suite of libraries.

Incanter deserves a much longer post on this blog, and that should eventually happen, but not yet — unless someone else who’s knowledgable about it would like to volunteer one 🙂

Functional Reactive Programming for Robots

Arrows, Robots, and Functional Reactive Programming: “Functional reactive programming, or FRP, is a paradigm for programming hybrid systems – i.e., systems containing a combination of both continuous and discrete components – in a high-level, declarative way. The key ideas in FRP are its notions of continuous, time-varying values, and time-ordered sequences of discrete events. Yampa is an instantiation of FRP as a domain-specific language embedded in Haskell. This paper describes Yampa in detail, and shows how it can be used to program a particular kind of hybrid system: a mobile robot. Because performance is critical in robotic programming, Yampa uses arrows (a generalization of monads) to create a disciplined style of programming with time-varying values that helps ensure that common kinds of time- and space-leaks do not occur.”

 

Open Source Clojure project to Monitor Deforestation from Satellite Imagery

Hi, this is George Kierstein!  I was checking out the recent talks from Clojure West and ran across an amazing talk by Dan Hammer about his recent experiences building an Open Source Clojure project to Monitor Deforestation from Satellite Imagery.

After the great talk I had a few burning questions which he was awesome enough to answer:

 

General background? Start in programming?

This is me, Dan Hammer.  I am a PhD student at Berkeley and a Presidential Innovation Fellow at NASA. I work with the NASA CTO, writing open APIs for the Agency’s public data.  I was formerly the Chief Data Scientist at the World Resources Institute, where I led the Data Lab — the technical team behind Global Forest Watch.

My background is not in computer science.  Rather, I am a lost economist who codes to compensate.  I started writing Stata scripts to assess the independent impact of primary care clinics in rural India.  Then, while working with David Wheeler and Robin Kraft, I started to work on deforestation, trying to understand the economic drivers of forest clearing activity.  At the time, there was no open and available information on deforestation at the micro-level — so we generated the data from NASA satellite imagery.  This is a lot of information.  Prepackaged tools to move the data around didn’t cut it.  We moved to Python and R, splitting our tasks on the emergent Amazon Web Services.  I have developed a deep appreciation for engineers who can elegantly move bytes around.

 

In your talk you stated that functional programming, and Clojure in particular, are great for scientific computing. How were you exposed to Clojure in a scientific context and what lead you to that conclusion?

I was introduced to Clojure by my friend and paddling teammate, Sam Ritchie.  Sam found Clojure while looking for a way to develop low-maintenance MapReduce applications.  I rewrote the core algorithms in Clojure, which simplified the code and workflow.  It was … oddly fun.  It felt … strangely good.  This was my first exposure to functional programming.  The process of converting long, tangled scripts into a composition of functions was totally consistent with writing proofs in upper-level math courses, like real analysis or topology.  Mapping concepts in functional programming to axioms in functional composition continues to guide development in my projects – and the mapping helps me reason out some of the more esoteric elements of Clojure.

 

Scientific computing has become much more aware of engineering concerns like maintainability, reliability and correctness over the years.  What advantages for science-based projects do you feel Clojure has over traditional approaches?

So this is the big issue.  Robin, Sam, and I learned to write Clojure at around the same time, with Sam forging ahead into Scala and various abstractions over MapReduce jobs.  We maintained our Clojure project for almost a year until it became clear that the project couldn’t be actively maintained by the deforestation monitoring (read: remote sensing) community.  We are rewriting the code once again in JavaScript for general use and maintenance — which is sort of heartbreaking. I’m no good at writing poetry.  And I’m not even that good at writing code.  But my time with Clojure was about as close to poetry through code that I can imagine.  The sheer volume of the codebase dropped by an order of magnitude.  It’s strange, then, that the project should be harder to maintain with a smaller, self-documented codebase — but them’s the breaks.  There just aren’t many Clojure developers working on satellite imagery; and we open sourced the project for widespread usage.  I can imagine that a tech shop with a critical mass of Clojure developers feels a lot like the micro-cult around Oakley Hall’s Warlock.  Some of the best software literature ever written.

 

Any recommendations to communicate effectively with a culture steeped in FORTRAN?

There is a slightly higher learning curve associated with Clojure.  That I’ll admit.  There is, in economic-speak, a relatively high fixed cost associated with Clojure.  But the marginal costs are much lower, with algorithms easily expressed in code.  When showing NASA scientists code written in Clojure, I’ll show them an example of mapping a function across an array — and then the for-loop equivalent in Python or R (and for good measure, the equivalent lambda functions).  The basics of functional programming and — where computation is the evaluation of elemental functions — are highly appreciated by scientists, and especially researchers that deal with images or time-series where mapping across partitions of a sequence is common.

 

There appears to be heightened interest within Government agencies to work more closely with industry and to make data more accessible to the public.  What is your take on this and what role, if any, do you see specific technologies like Clojure playing?

Access to government data is a public good on par with road building or national defense.  Gray Brooks, Nick Mueltder, and others at 18F and USDS to build api.data.gov are laying the first bricks for this infrastructure.  I am doing my small part in building out the franchised api.nasa.gov alongside the data.nasa.gov team.  One such API is easy access to relevant earth imagery, where the metric for ease is defined by the hassle required by developers to retrieve the precise imagery they want.  Cloud free for the White House?  Done.  The most recent images for the NOAA’s facilities in Alaska?  Done, distributed.  Some of the image screening on the server-side (for clouds, for example) will need to be blazing fast.  As this functionality gets built out, I’m sure that the power of Clojure will be more apparent — and used more often (by me, if no one else, at least).

 

Your deforestation tool and Global Forest Watch are great examples of this kind of cross-over. What would you say are some of the most exciting up-and-coming projects in this area ?

Water.  Monitoring water from commercial satellite imagery.  The Global Forest Watch (GFW) frameworks are well-suited for distilling high-resolution imagery into tractable information, actionable insight on surface water.  And water is becoming a clear and present issue in earth observation.  Plus, Congress just proposed cuts to NASA’s earth observation budget in some bout of legislative idiocy.  Who will step in to mine the value of both existing NASA imagery and new commercial imagery, like that of Planet Labs?  It’s still unclear.  Previous successes in this field (including GFW) will certainly guide the new, distributed development — possibly within the government and possibly in the private sector.

 

Closing thoughts?

Thanks for reading this, George!!  It’s exciting that this effort is coming from within the government.  There’s something brewing.  And it smells like civic tech, well-done.



Thanks for the insight and taking the time!  I’m excited to see what else is out there so if you are a scientific programming nerd with a Clojure bent and hear of anything intriguing, or just want to nerd out about it with me, hit me up!

 

 

 

Learning to think idiomatically

This wasn’t intended for public consumption — it was just a conversation on Slack between Sean, who’s an experienced programmer learning ClojureScript for the first time, and myself, an experienced programmer with a few months of Clojure experience (and no real ClojureScript, not that it’s very different, but I haven’t really gotten a ClojureScript environment up and running yet). Sean was trying to figure out how to transform an array of HTML form field inputs (which are somewhat like maps, and look approximately like {:name :firstname, :value “bob”}) into an ordinary ClojureScript map that looks like {:firstname “bob”}. He and I were both looking at it off and on over the course of the day.

Since so much of our thought process is captured in the chat, it seemed like might be helpful as a demonstration of how people with limited ClojureScript experience can work their way toward increasingly idiomatic code.

It’s been lightly edited, mostly to drop out some unrelated side chatter, and occasionally for clarity.

sean:
i rewrote a little plain js script in clojurescript last night.
was fun.
i ended up just writing a few little helper functions

(defn attr [el key]
  (.getAttribute el key))

(defn by-id [id]
  (.getElementById js/document (name id)))

(defn console-log [message]
  (.log js/console message))

(defn data [el key]
  (.getAttribute el (str "data-" key)))

(defn domready [handler]
  (.addEventListener js/window "DOMContentLoaded" handler))

(defn target [event] (.-target event))

sean:
but i imagine there’s a better way

fyi: https://github.com/somlor/formal.cljs
GitHub: formal.cljs – first experiment with clojurescript

feel free to critique my clojurescipt
or just outright ridicule will also be understood
i’m especially curious how i could refactor this:

; given array of inputs, return map with name/value pairs
(defn input-data [inputs]
  (let [names (map input-name inputs)
        values (map input-value inputs)]
    (zipmap names values)))

sean:
not very familiar with clojure collection functions yet

eggsyntax:
What does ‘inputs’ look like when it comes into input-data? Can you provide a minimal example? (At work, don’t have time to try to get a cljs setup running here, but I’m curious to think a bit about how I’d write that function)

sean:
@eggsyntax: thanks! inputs is an array of input dom elements, (i.e., <input name=“x_firstname” value=“egg">) then input-name and input-value are functions that take an input and return name and value, respectively.

sean:
e.g.,

(defn attr [el key]
  (.getAttribute el key))

(defn input-name [input]
  (attr input "name”))

eggsyntax:
[Several dumb comments in which I’m essentially begging the question 😉 ]

sean:
i have a collection of inputs
that have names and values
and want to just get a simple map out of that

{:x-firstname “egg”}

anyway @eggsyntax i suspect there is a way to build that map in one swoop
instead of what i’m doing which is iterating twice

eggsyntax:
That’s my thought too. I’d throw away the intermediate functions, for one. Seems like they’re just making things more roundabout.

sean:
@eggsyntax: just seemed like it was getting a bit verbose with the map over an anonymous function to get those names, values but yeah it adds a layer of indirection

(let [names (map #(.getAttribute % “name") inputs)])

something like that

eggsyntax:
See, now that looks really clear to me 🙂

sean:
maybe got carried away with the “tiny composable functions that do one thing” thing
;P

sean:

; given array of inputs, return map with name/value pairs
(defn input-data [inputs]
  (let [names (map input-name inputs)
        values (map input-value inputs)]
    (zipmap names values)))

sean:
that would be more verbose way to write out that input-name function there

sean:
values would be a little different

eggsyntax:
Yeah, the one-liner seems way more clear and idiomatic to me.

sean:
gotcha

sean:
you prefer this:

sean:

; given array of inputs, return map with name/value pairs
(defn input-data [inputs]
  (let [names (map #(.getAttribute % "name") inputs)
        values (map #(.-value %) inputs)]
    (zipmap names values)))

eggsyntax:
There’s still a cleaner way to do it, though, I feel so sure. I don’t know what it is yet, though 😉

andrewswerlick:
is there something like ruby’s Hash[kv-array]?

andrewswerlick:
where you can pass an array of key value arrays to create a map

andrewswerlick:
eg [[k1,v1],[k2,v2]] = {k1=> v1, k2=>v2}

sean:
i hope so

andrewswerlick:
https://clojuredocs.org/clojure.core/array-map

sean:
i’m looking at into now
oh nice
this might be it andrew
as long as i return two items at a time
it looks like
i’ll map over inputs and return pairs of name/value

andrewswerlick:
you could also do a loop recur with assoc I think
more like an inject pattern in ruby
actually that would be more like reduce in that case
yeah that would be another route, reduec over inputs starting with an empty map
in the reduce body call assoc to create a new map with the key-value for the current input
return the new map so it used in the next reduce statement

williamblasko:
(.toArray js/_ htmlCollection)
Or something to that effect.

eggsyntax:
I’m definitely seeing it as a reduce too.

eggsyntax:
Groping for it though. I still write Clojure waaaay slower than other languages, and I’m not sure it’ll ever be much faster — it’s just when it’s done, it’s a 2-line perfect diamond that’s trivial to understand later, instead of a 25-line piece of cruft.

What about something like this?

(let [inputs [{:name :first-name :value "bob"} {:name :last-name :value "jones"}]]
   (apply hash-map 
          (for [input inputs 
                getter [:name :value]]
            (getter input))))

Obv that’s not exactly right – since I don’t have access to cljs at the moment, I’m just faking the values with the keywords.
But just something of that general sort…

sean:
yeah that looks great

eggsyntax:
There’s still some annoying incidental complexity, though, since we’re making it a list first and then a map. Gaaah, I gotta get some work done. May post another try later when it’ll probably be too late to be helpful 😉
I’m still so new at this too. I think the only good habit I’ve really internalized yet is just to have zero tolerance left for any bit of code that isn’t an essential expression of the solution.

sean:
sounds like a solid habit
thanks for your help!

(defn input-data [inputs]
  (apply conj
    (for [input inputs]
      {(dom/input-name input) (dom/input-value input)})))

sean:
this worked
not quite ready to cuddle with it yet tho

eggsyntax:
OK, not feeling cuddly yet, but this at least feels like the cleanest and most idiomatically functional thing I’ve been able to come up with so far (feedback appreciated):

(def test-inputs [{:name :first-name :value "bob"} {:name :last-name :value "jones"}])

(defn field-to-map [ob]
  (let [getters [#(:name %)
                 #(:value %)]]
    (apply hash-map
           (map #(% ob) getters))))

(defn form-to-map [form]
  (let [to-map (fn [results cur] (into results cur))]
    (reduce to-map {} (map field-to-map form))))

(form-to-map test-inputs)

eggsyntax:
Again, I’m using ordinary maps as test input, so I’m using simplified getter functions.
But they could be swapped out for yours and still feel pretty clean, I think.
Actually don’t need the empty map in the reduce step, come to think of it.

OK, this may be as cuddly as I can get:

(defn field-to-map [ob]
  (let [getters [#(:name %)
                 #(:value %)]]
    (map #(% ob) getters)))

(defn form-to-map [form]
  (apply hash-map
         (flatten (map field-to-map form))))

(form-to-map test-inputs)

eggsyntax:
And this version is cuddly but sleazy. It works for the test input, but a) I don’t know whether it’ll work on your form object, and b) it relies on the internal ordering of the fields, so I would have zero faith in it without doing more research and testing. Nice and short, though 😉

(def test-inputs [{:name :first-name :value "bob"} {:name :last-name :value "jones"}])

(defn form-to-map [form]
  (apply hash-map
         (flatten (map vals form))))

(form-to-map test-inputs)

Aha! mapcat pays off for the first time:

(def test-inputs [{:name :first-name :value "bob"} {:name :last-name :value "jones"}])

(defn field-to-map [ob]
  (let [getters [#(:name %)
                 #(:value %)]]
    (map #(% ob) getters)))

(defn form-to-map [form]
  (apply hash-map
         (mapcat field-to-map form)))

(form-to-map test-inputs)

sean:
@eggsyntax: great stuff man
re: this version:

(defn input-data [inputs]
  (apply conj
    (for [input inputs]
      {(dom/input-name input) (dom/input-value input)})))

is the non-cuddliness from the use of for?
i do like how succinct it is

eggsyntax:
No, that’s pretty sweet, I totally missed that earlier. Maybe

(apply hash-map (mapcat))

feels more intuitive to me than

(apply conj (for))

but I like yours too.
and of course it doesn’t need the extra field-to-map function, which makes it pretty awesome. Be interesting to see whether there’s a good combination of the two.

sean:
yeah me too :neckbeard:

eggsyntax:
This one’s inspired by yours:

(defn form-to-map [inputs]
  (letfn [(to-map [field]
            {(:name field) (:value field)})]
    (apply merge (map to-map inputs))))

sean:
oooh letfn, not familiar with that
very cool

eggsyntax:
I like that merge finally made it in there, it seems like it expresses our intention the best — create a bunch of separate maps and then make them one map.

sean:
yeah i dig this

eggsyntax:
Without the letfn:

(defn form-to-map [inputs]
  (let [to-map (fn [field] {(:name field) (:value field)})]
    (apply merge (map to-map inputs))))

sean:
ahhh i see, it’s sugar for specifically creating named, scoped functions more or less?

eggsyntax:
Yeah, just shorthand for (let (fn))

sean:
this clojure koolaid is just getting tastier and tastier

eggsyntax:
See, this is why I’m so head over heels about Clojure :). We’ve been thinking about this off and on for hours, and it keeps getting clearer and clearer, and shorter and shorter.

That NEVER seems to happen in, say, Java ;P

sean:
i think the opposite happens in java

sean:
maybe we should make a factory for the factories?

sean:
yeah this is a fun language

eggsyntax:
Ha!

eggsyntax:
Probably @brucehauman or someone will come along soon and be like, “oh, duh, you can replace (apply merge (map)) with this one keyword.”
And it’ll be something that makes it so transparently clear and beautiful that I’ll have to go weep in a corner somewhere for a while 😉

sean:
hehe. yep. eventually our entire application is reduced to a resplendent haiku.
we renounce technology and wander the forests foraging for berries and writing clojure poetry that doesn’t even need to be run.

sPINSYFl

eggsyntax:
TOTALLY

postscript: eventually Sean asked a much more knowledgable Clojure developer, Bruce Hauman, who came up with the following:

(defn input-data [inputs]
  (into {}
    (map (juxt dom/input-name dom/input-value) inputs)))```

sean [10:28 PM]
also here is a less clever reduce version of the same function:

sean [10:29 PM]

(defn input-data [inputs]
      (reduce (fn [result input]
        (assoc result (dom/input-name input) (dom/input-value input))) {} inputs))

 

Welcome to Clojure: for Science!

As software developers who work on scientific software, we believe that functional programming is the optimal approach. In particular, we believe that Clojure is nearly an ideal language for developing scientific software. On this site, our goal is to provide resources for other scientific software developers considering Clojure. The software carpentry movement has done a fantastic job promoting Python as a language for scientists. We agree that Python is an excellent choice for scientists who write code, but for professional developers working on a larger scale, we believe that Clojure is ultimately an even better tool. The functional programming approach, applying a set of transformations to immutable data, is ideal in a scientific software context, where the goal is to maintain a chain of clear reasoning and provenance all the way from raw data to conclusions.

FLf5J6f