clojure.pprint/cl-format is slow

Written by on June 5, 2016, 12:51 pm

Had a serious performance problem last week. After some digging we narrowed it down to a call to cl-format.

Some test code to demonstrate the discrepancy:

;; Format a string, an integer, a conditional and fancy plural thing
(defn cl-format-test [] (clojure.pprint/cl-format nil "
    ~@[(Id: ~D)~] ~:[foo~;bar~] [~D second~:P ago]>"
                                      (rand 1)
                                      (> (rand 2))
                                      (rand 100)))
;; Same inputs, just concat them and return the string
(defn str-test [] (str (java.util.UUID/randomUUID) " "
                            (rand 1) " "
                            (> (rand 2)) " "
                            (rand 100)))
;; Just format a string
(defn cl-format-simple-test [] (clojure.pprint/cl-format nil "~S" (java.util.UUID/randomUUID)))
;; Just return the string
(defn str-simple-test [] (str (java.util.UUID/randomUUID)))

(defn test-func [f tests] (time (doseq [_ (range 0 tests)] (f))))

(test-func cl-format-test 1000)
(test-func str-test 1000)
(test-func cl-format-simple-test 1000)
(test-func str-simple-test 1000)

With the following results. For the simple case, cl-format is 10 times slower than a simple string return. For more complicated format strings, closer to 100 times slower:

(test-func cl-format-test 1000)
"Elapsed time: 328.033183 msecs"
(test-func str-test 1000)
"Elapsed time: 5.939625 msecs"
(test-func cl-format-simple-test 1000)
"Elapsed time: 34.2998 msecs"
(test-func str-simple-test 1000)
"Elapsed time: 3.515013 msecs"

Having a quick look at the implementation of cl-format I immediately noticed a compile-format function which seems like something that might help out with performance for many calls to a cl-format with the same format string:

Compiles format-str into a compiled format which can be used as an argument to cl-format just like a plain format string. Use this function for improved performance when you're using the same format string repeatedly

It also seems like cl-format will check if it's format string is already compiled and skip compilation if that is the case:

(if (string? format-in) (compile-format format-in) format-in)

Oddly however, compile-format is not public, so I can't use it. So I am left a little confused. I am going to do some more digging tomorrow to solve this mystery as I am sure I am missing something here.

Not sure if pre-compilation will help solve the horrible performance you get from cl-format, but I am guessing it might help.


After some discussion with my colleagues, it seems I haven't missed anything and we think this is a bug.

Permalink - Comments - Tags: Development,Clojure

Decoding Polylines from Google Maps Direction API with Clojure

Written by on January 17, 2016, 12:56 pm

My first crack at porting some imperative code to Clojure goodness.

I needed some code to turn the polyline points encoding you get back from Google Directions API. This stuff:

"overview_polyline" : {
            "points" : "[email protected][email protected][email protected][email protected]@@[email protected]}@[email protected]?UDYJi@]^w@[email protected]@[email protected][email protected][email protected][email protected]@[email protected]@[email protected]@[email protected]~@[email protected][email protected]@[email protected]@h@[email protected]@[email protected]@[email protected]@[email protected]][email protected]@[email protected]@[email protected]][email protected]@@eAT[L[[email protected]@[email protected]`XoAbEmB~GqAvDkDzIsFvNcAvCcFrRkBnHcE`Pq@[email protected]@[email protected][email protected]@[email protected][email protected]@[email protected]@[email protected][email protected]@[email protected][email protected][email protected]@[email protected]@[email protected]_BEiBUaEkAcA][email protected]@][email protected]@[email protected][email protected]@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]^Et@@[email protected]@[email protected]@Q"

I found Jeffrey Sambells' Java code here that does the job. I am just getting my teeth into writing Clojure for my day job at GoCatch, so I need a Clojure version. This is my first attempt. It's midnight and I haven't had a chance to check the line ends up on a map correctly, but it looks pretty good to me:

;; Port of the Java code to decode google polylines that I found here ->
(defn decode-next-result [encoded]
  ;; keep scanning through encoded till b>=0x20
  ;; returns the next latitude/longitude increment
  (loop [[current & rest] encoded shift 0 result 0]
    (let [b       (- (int current) 63)
          result  (bit-or result (bit-shift-left (bit-and b 0x1f) shift))
          shift   (+ shift 5)] 
      (if (>= b 0x20) 
          ;; if we are encoding the next result then we 
          ;; must have more characters to scan
          (assert rest)
          ;; keep looking for our next result
          (recur rest shift result))
        ;; we found our next result 
        (let [return-value (if (not= (bit-and result 1) 0)
                             (bit-not (bit-shift-right result 1))
                             (bit-shift-right result 1))]
          [return-value rest])))))

(defn lat-lng-double [lat-lng-int]
  (/ lat-lng-int 1E5))

(defn path-for-encoded-polyline [encoded] 
  (loop [rest encoded lat 0 lng 0 results []]
    (if rest
      ;; if there is anthing in the encoded array
      ;; we should have two more results at least 
      (let [next-result       (decode-next-result rest)
            new-rest          (second next-result)
            next-lat-result   (+ lat (first next-result))
            next-result       (decode-next-result new-rest )
            new-rest          (second next-result)
            next-lng-result   (+ lng (first next-result))]
          ;; add our lat lng result to the results
          (recur new-rest next-lat-result next-lng-result (conj 
                                                            {:latitude (lat-lng-double next-lat-result) 
                                                             :longitude (lat-lng-double next-lng-result)})))
      ;; we are done, return our results

(def example-polyline "pdymEssfy[[email protected][email protected]@[email protected]@[email protected]@uA_Di@[email protected]@[email protected]@[email protected][email protected]@[email protected][email protected]@[email protected]@[[email protected]@[email protected]@][email protected][email protected]@[email protected]@wDK}@@[g@[email protected]@gAk@[email protected]@[email protected]@[email protected]@[email protected][email protected]^iAx@[email protected]@[email protected]@[email protected][email protected]@[email protected]@[email protected]@[email protected][email protected][email protected]@eCVeDRiGF}FGiB[sBc@[email protected]@[email protected]@[email protected]@uByA}BaAwAMUw@[email protected]@[email protected]@[email protected]@[email protected]@[email protected][email protected][email protected]@[email protected]`CiHPcABYB[Dq@@][email protected]@[email protected]@[email protected]@~CuBbC_Bn@]d@[email protected]][email protected]@Hs@@{@Cm@Ii@Ug@UcA_@kAo@uCqAeHg@kCSi@GK]uAUs@OWEGSYQQk@e@}@a@c@Kc@AiBQiAA[CiC?GAwBAUAu@USI[UQ[GMMk@EqBG_DKsCCcBJcDcCIuBK")
(path-for-encoded-polyline (seq (char-array example-polyline)))

Permalink - Comments - Tags: Development,Clojure

Hundreds Chart Dev Diary

Written by on June 28, 2015, 12:32 am

I've had this idea for a game to teach kids times tables since I finished Wordflight two years go. I got bogged down in my initial implementation using OpenGL ES and a few weeks ago I decided to re-boot the project with flat UI, vivid colours and UIDynamics to make things interesting.

This time the project took two solid weekends (and a few late nights), but I am pretty happy with how it turned out

If you are reading this and you live in Australia, make sure you check out goCatch next time you need to catch a taxi.

Permalink - Comments - Tags: Development,iPhone,App

Interactive Hundreds Charts

Written by on June 11, 2015, 12:16 pm

Four engaging math games to help children learn their multiplication tables.

Permalink - Comments - Tags: Development,iPhone,App

I love everything about Daredevil except the suit

Written by on May 2, 2015, 7:46 am

I think I would be ok with keeping the black mask for the entire show. Ok. Maybe not. But I wish I liked the suit.

All these were awesome

I don't know what it is, but this just doesn't do it for me. Is the pointy eyebrows, the weird glowing eyes?

I really want to love it (I love this show) but, sadly, I can't do it.

Then again they could do a lot worse and still be better than this S&M monstrosity.

I can't think of that movie without picturing ol Matt on his hands and knees carefully drawing the DD logo on the subway with a can of Zippo fuel so he can dramatically light it up later. What the hell was that about?

Yep ... that was a really bad movie.

Permalink - Comments - Tags: Misc