Embedded DSLs in Clojure

Finished presentation at Clojure NYC meetup.

• Overall good reviews, and I felt that the discussion was good.
• Some of my friends missed it, next time promise to broadcast more loudly before the event.

Thank you for Ubuntu 9.10

Just upgraded to Ubuntu 9.10 and everything works. I mean everything that bothered me, just works now. Among huge amount of improvements there is one that I am really happy about. I have two 24" monitors and one on the right I like to keep vertical, so I can read the page w/o scrolling. I accomplished it in Ubuntu 9.04 but had to do all kind of hacks in my xorg.conf, etc. The result was also not perfect, since there were all kind of flickering on the screen, which drove me up the walls. I am happy to say that with Ubuntu 9.10, it just works using standard display configuration panel. Also I was able to align the displays so when half of the window is on the left monitor and half is on the right, the window does not look crooked. Thank you Ubuntu Team!

I/O write speed benchmark

At work we had some performance issues with one of our production boxes. I thought it was related to the I/O on the veritas file system which we have to use for high availability. In order to prove it I needed a tool to write 1K block and flush it, fast. I've searched around on the web and could not find anything that measures only writes, so I wrote one in Clojure. I had to make sure that it is fast and does not generate to much garbage in the main tight loop. New transient was a perfect choice for it. Transients only available in Clojure 1.1.0, so you will need to get the latest alpha.

The main loop is pretty straight forward.

(defn write-to-file [#^String file]
  (with-open [s (FileOutputStream. file)]
    (loop [i 0 r (transient [])]
      (if (< i 1e3)
        (recur (inc i) (conj! r (with-time
                                    (doto s
                                      (.write test-row)
                                      (.flush)))))
        (persistent! r)))))

Here is the gist of the rest.

I've used incanter to plot the data.
And this is one of the plots that I've got, when I ran it on my MacBook Pro.
As you can see there is quite a bit of jitter and that will affect latency. On our production Linux servers after tuning the file system, those jitters almost disappeared. The next step is to use Real-Time OS and RTSJ.

What animal letters add up to 100

My friend's daughter had an assignment to find three animals which letters will add up to a hundred, "A" being one, "B" two, etc. After hours of adding they found two of the names, that is when I've heard of this problem. Well, I've cheated and used Clojure and fifteen minutes later I've got this


It was fun to write. Maybe I'll use it as an interview question :)
By the way the answer is:
([Chimpanzee 100] [Starfish 100] [Turkey 100])

[Update: The new assignment was to find as many words in the dictionary as possible with the same requirement. I hacked the code to find all the 2296 words in the words file on my machine. The gist is in the same place.]

International Lisp Conference 2009

ILC2009-46
Originally uploaded by rpgpoet

Looks like Lispers like Macs. I can see at least 10 of them on this picture.

P.S. Can you find me?

Clojure upload to Flickr script.

As promised I've checked in the upload script into the svn repository at gashinsky code.

/home/dig/src/gashinsky/src/clj/com/gashinsky $ ./upload.clj /home/dig/Data/Album/1997/Spock/
Skipping: /home/dig/Data/Album/1997/Spock/1997_12_12-17_11_40.jpg
Skipping: /home/dig/Data/Album/1997/Spock/1997_11_21-20_15_08.jpg
Skipping: /home/dig/Data/Album/1997/Spock/1997_12_27-09_38_58.jpg
...

The script is specific to my needs, but it is a script and pretty simple so you can modify it for your own needs. I've uploaded around nine thousand of my pictures with it. It is quite robust and can retry on failure. My router Actiontec MI424WR is the weakest link and it was keep on getting stuck. Now that all of my pictures uploaded, I can start working on syncopy script to do the bidirectional sync to and from Flickr.

With Retry

I've been using Clojure for work and play for some time now. It is awesome language. It revived my interest and believe in JVM as a platform.

I am writing a script to upload all my pictures to Flickr. I have a pretty big collection of photos starting from 1997 when I've got my first digital camera. I've tried many Flickr uploaders to do the task, but it was too much manual work. I am a programmer and hate manual steps, that is what programs for.

I've tried repeatedly over the years in my little spare time to write the script. Every year I would try in different language because I would loose the script or the interest in it. That was a major cause of my failure to finish that script. And my photo album stayed in Gallery and never was migrated to Flickr.

The title of this blog is with retry and that is exactly what I am doing. This is were Clojure comes to the rescue. There are plenty of Java APIs for Flickr. They are well documented, stable and easy to use. Making Clojure to drive one of these APIs flickrj is a pleasure. I will publish the script, which have a little GUI component, just because it so easy to do in Clojure :)

For now to keep this post a little bit more technical here is a little macro that I wrote to satisfy script need to handle restart on error in the script.

(defmacro with-retry [times & body]
  `(let [done# (ref false)]
     (loop [time# 0 result# nil]
       (if (and (not @done#) (< time# ~times)) 
         (recur (+ 1 time#)
                (try (let [result# (do ~@body)]
                          (dosync (ref-set done# true))
                          result#)
                     (catch Exception ex#
                       (println "Retrying:" time# "time" ex#))))
         result#))))

There is got to be a better way to write it, any suggestions are welcome.
Of cause the point of this post is that it was easy for me to write and to use this macro in Clojure.
Clojure is an excellent language for scripting!