Embedded DSLs in Clojure

Finished presentation at Clojure NYC meetup.

• Overall good reviews, and I felt that the discussion was good.
• Some of my friends missed it, next time promise to broadcast more loudly before the event.

Thank you for Ubuntu 9.10

Just upgraded to Ubuntu 9.10 and everything works. I mean everything that bothered me, just works now. Among huge amount of improvements there is one that I am really happy about. I have two 24" monitors and one on the right I like to keep vertical, so I can read the page w/o scrolling. I accomplished it in Ubuntu 9.04 but had to do all kind of hacks in my xorg.conf, etc. The result was also not perfect, since there were all kind of flickering on the screen, which drove me up the walls. I am happy to say that with Ubuntu 9.10, it just works using standard display configuration panel. Also I was able to align the displays so when half of the window is on the left monitor and half is on the right, the window does not look crooked. Thank you Ubuntu Team!

I/O write speed benchmark

At work we had some performance issues with one of our production boxes. I thought it was related to the I/O on the veritas file system which we have to use for high availability. In order to prove it I needed a tool to write 1K block and flush it, fast. I've searched around on the web and could not find anything that measures only writes, so I wrote one in Clojure. I had to make sure that it is fast and does not generate to much garbage in the main tight loop. New transient was a perfect choice for it. Transients only available in Clojure 1.1.0, so you will need to get the latest alpha.

The main loop is pretty straight forward.

(defn write-to-file [#^String file]
  (with-open [s (FileOutputStream. file)]
    (loop [i 0 r (transient [])]
      (if (< i 1e3)
        (recur (inc i) (conj! r (with-time
                                    (doto s
                                      (.write test-row)
                                      (.flush)))))
        (persistent! r)))))

Here is the gist of the rest.

I've used incanter to plot the data.
And this is one of the plots that I've got, when I ran it on my MacBook Pro.
As you can see there is quite a bit of jitter and that will affect latency. On our production Linux servers after tuning the file system, those jitters almost disappeared. The next step is to use Real-Time OS and RTSJ.

What animal letters add up to 100

My friend's daughter had an assignment to find three animals which letters will add up to a hundred, "A" being one, "B" two, etc. After hours of adding they found two of the names, that is when I've heard of this problem. Well, I've cheated and used Clojure and fifteen minutes later I've got this


It was fun to write. Maybe I'll use it as an interview question :)
By the way the answer is:
([Chimpanzee 100] [Starfish 100] [Turkey 100])

[Update: The new assignment was to find as many words in the dictionary as possible with the same requirement. I hacked the code to find all the 2296 words in the words file on my machine. The gist is in the same place.]

International Lisp Conference 2009

ILC2009-46
Originally uploaded by rpgpoet

Looks like Lispers like Macs. I can see at least 10 of them on this picture.

P.S. Can you find me?

Clojure upload to Flickr script.

As promised I've checked in the upload script into the svn repository at gashinsky code.

/home/dig/src/gashinsky/src/clj/com/gashinsky $ ./upload.clj /home/dig/Data/Album/1997/Spock/
Skipping: /home/dig/Data/Album/1997/Spock/1997_12_12-17_11_40.jpg
Skipping: /home/dig/Data/Album/1997/Spock/1997_11_21-20_15_08.jpg
Skipping: /home/dig/Data/Album/1997/Spock/1997_12_27-09_38_58.jpg
...

The script is specific to my needs, but it is a script and pretty simple so you can modify it for your own needs. I've uploaded around nine thousand of my pictures with it. It is quite robust and can retry on failure. My router Actiontec MI424WR is the weakest link and it was keep on getting stuck. Now that all of my pictures uploaded, I can start working on syncopy script to do the bidirectional sync to and from Flickr.

With Retry

I've been using Clojure for work and play for some time now. It is awesome language. It revived my interest and believe in JVM as a platform.

I am writing a script to upload all my pictures to Flickr. I have a pretty big collection of photos starting from 1997 when I've got my first digital camera. I've tried many Flickr uploaders to do the task, but it was too much manual work. I am a programmer and hate manual steps, that is what programs for.

I've tried repeatedly over the years in my little spare time to write the script. Every year I would try in different language because I would loose the script or the interest in it. That was a major cause of my failure to finish that script. And my photo album stayed in Gallery and never was migrated to Flickr.

The title of this blog is with retry and that is exactly what I am doing. This is were Clojure comes to the rescue. There are plenty of Java APIs for Flickr. They are well documented, stable and easy to use. Making Clojure to drive one of these APIs flickrj is a pleasure. I will publish the script, which have a little GUI component, just because it so easy to do in Clojure :)

For now to keep this post a little bit more technical here is a little macro that I wrote to satisfy script need to handle restart on error in the script.

(defmacro with-retry [times & body]
  `(let [done# (ref false)]
     (loop [time# 0 result# nil]
       (if (and (not @done#) (< time# ~times)) 
         (recur (+ 1 time#)
                (try (let [result# (do ~@body)]
                          (dosync (ref-set done# true))
                          result#)
                     (catch Exception ex#
                       (println "Retrying:" time# "time" ex#))))
         result#))))

There is got to be a better way to write it, any suggestions are welcome.
Of cause the point of this post is that it was easy for me to write and to use this macro in Clojure.
Clojure is an excellent language for scripting!

Scrum

I've started studying Agile Development and in particular Scrum. Reading Scrum in five minutes, so far it is all common sense to me.
Here is the philosophy of Agile Development summarized with the help from screedbot.

Read the whole Manifesto for Agile Software Development if you would like to learn more.

Vecto rules!

Another instant hit from Xach is Vecto. All pure lisp all the time. Here is some fun that I've had with it.

(defpackage #:smiley (:use #:cl #:vecto)) 

  (in-package #:smiley) 

  (with-canvas (:width 200 :height 200) (scale 2 2) (set-rgb-stroke 0 0 0) (set-line-width .5) (set-rgb-fill 1 1 0) (centered-circle-path 50 50 50) (fill-and-stroke) (set-rgb-fill 0 0 0) (centered-ellipse-path 35 70 10 15) (centered-ellipse-path 65 70 10 15) (fill-path) (move-to 30 35) (quadratic-to 50 10 70 35) (quadratic-to 50 15 30 35) (fill-and-stroke) (move-to 25 34) (quadratic-to 30 35 30 40) (move-to 75 34) (quadratic-to 70 35 70 40) 

  (stroke)

  (save-png (merge-pathnames "tmp/smiley.png" (user-homedir-pathname))))

New logo

It is so much fun to change logos for ones blog. Thanks to Xach it is easy too. He just released Easystreet a very cool street sign generator and I could not control myself. I generated couple of dozens and behold my new logo :) It is written in Lisp.

Common Lisp for all of us

Reading Peter Seibel blog I've noticed that he needed some google bombing. So here is my little part in it. Practical Common Lisp is a really good tutorial. I own the dead tree version and the book is really good.

VPS

Moved this blog to TekTonic VPS. So far so good. No problems installing SBCL and Hunchentoot. Let the experiments begin! UPDATE: Moved the blog back to blogger, since the VPS is not good enough for multiple instances of SBCL with only 256MB. I need that instance for other experiments.

Nerd test

My nerd test score says that I am Kinda Dorky High Nerd. I just wonder what does it mean exactly.

nuclblog

The nuclblog is up and running on my test machine. Looks like a pretty good platform for experimentation. This entry is done using w3m-emacs mode. UPDATE: The experiment is over. The cheap virtual host is not good enough to run SBCL with hunchentoot and other experiments. It is good enough for one instance of SBCL but multiple instances are not able to run in 256MB of RAM.

NKS old and new

While reviewing a paper for the complex systems journal I've started looking around the web for my NKS summer school article and found this web page. It is so interesting to read what one wrote four years ago. Which is why I need to write more so I can read it later and write about it so I can read it later and write about it, etc. P.S. Recursion is fun.

New Kind of Blog

I wanted to try this blogging technology for some time. I don't even know what is this blog will be about. I hope I will figure it out pretty soon. So you my gentle reader will not have to suffer through this agony of me becoming a better writer.