clojure.pprint/cl-format is slow

June 5, 2016, 12:51 pm

Had a serious performance problem last week. After some digging we narrowed it down to a call to cl-format.

Some test code to demonstrate the discrepancy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
;; Format a string, an integer, a conditional and fancy plural thing
(defn cl-format-test [] (clojure.pprint/cl-format nil "
    ~@[(Id: ~D)~] ~:[foo~;bar~] [~D second~:P ago]>"
                                      (java.util.UUID/randomUUID)
                                      (rand 1)
                                      (> (rand 2))
                                      (rand 100)))
;; Same inputs, just concat them and return the string
(defn str-test [] (str (java.util.UUID/randomUUID) " "
                            (rand 1) " "
                            (> (rand 2)) " "
                            (rand 100)))
;; Just format a string
(defn cl-format-simple-test [] (clojure.pprint/cl-format nil "~S" (java.util.UUID/randomUUID)))
;; Just return the string
(defn str-simple-test [] (str (java.util.UUID/randomUUID)))

(defn test-func [f tests] (time (doseq [_ (range 0 tests)] (f))))

(test-func cl-format-test 1000)
(test-func str-test 1000)
(test-func cl-format-simple-test 1000)
(test-func str-simple-test 1000)

With the following results. For the simple case, cl-format is 10 times slower than a simple string return. For more complicated format strings, closer to 100 times slower:

(test-func cl-format-test 1000)
"Elapsed time: 328.033183 msecs"
(test-func str-test 1000)
"Elapsed time: 5.939625 msecs"
(test-func cl-format-simple-test 1000)
"Elapsed time: 34.2998 msecs"
(test-func str-simple-test 1000)
"Elapsed time: 3.515013 msecs"

Having a quick look at the implementation of cl-format I immediately noticed a compile-format function which seems like something that might help out with performance for many calls to a cl-format with the same format string:

Compiles format-str into a compiled format which can be used as an argument to cl-format just like a plain format string. Use this function for improved performance when you're using the same format string repeatedly

It also seems like cl-format will check if it's format string is already compiled and skip compilation if that is the case:

(if (string? format-in) (compile-format format-in) format-in)

Oddly however, compile-format is not public, so I can't use it. So I am left a little confused. I am going to do some more digging tomorrow to solve this mystery as I am sure I am missing something here.

Not sure if pre-compilation will help solve the horrible performance you get from cl-format, but I am guessing it might help.

Update

After some discussion with my colleagues, it seems I haven't missed anything and we think this is a bug.

Permalink - Comments - Tags: Development,Clojure

Decoding Polylines from Google Maps Direction API with Clojure

January 17, 2016, 12:56 pm

My first crack at porting some imperative code to Clojure goodness.

I needed some code to turn the polyline points encoding you get back from Google Directions API. This stuff:

"overview_polyline" : {
            "points" : "_rtwFpgubMtBuGVNFPfAr@TNIVkB|FsBpG_C_BqA}@mGeEwH_FsDcCaGwDkNiJsJmG_SoMyByAzA}EPi@nA{Dp@qBtDdCLh@@VCT_@nAKLMBOAIG}@{Aq@eAWWOMIEWEY?UDYJi@]^w@|AcC|Ew@bBy@hCg@dBQ`AU~A_ChH_D|J_AzCc@rAwXl|@yLt_@k@hCSrBAtABp@LlALp@FPZ~@Tf@Z^X^jAx@PXRTrCxB~@`@lA^`@HVLv@j@n@h@`@Rb@Dl@Gl@WVQTU^w@PaA@gAKcAMa@Q]c@i@}@g@iDoAeDaAeDq@qC]i@Cg@@eAT[L[Vk@h@SZQ^_@bAeJ`XoAbEmB~GqAvDkDzIsFvNcAvCcFrRkBnHcE`Pq@~Bi@rAk@hA}@hA_@f@cAfAoDjDiAv@iB|@a@XoAf@iBj@yA`@aB{Dz@yCl@yAmE`A}Aj@qCjAuC~AkBnA_@ZcBxAoApA{BhCoBtCuJrOkAxAkBzCeH~KkAlBeMrR{I|M}CtEyBjD}EpHmLbQ{BdDiBvCUBMJy@l@g@VeAZs@HiABuKT_BEiBUaEkAcA]oCeAmAi@iCsAyCkBgEaDcCuBa@]kHgGkDyCQQOOG_@Fk@h@YfI_DxAi@vAg@^`@h@l@h@dL@v@Ef@Od@m@fAy@nBmAxCc@dAALAHQf@Qv@G^Et@@n@Fr@Z`BHlABnAp@Cj@Q"

I found Jeffrey Sambells' Java code here that does the job. I am just getting my teeth into writing Clojure for my day job at GoCatch, so I need a Clojure version. This is my first attempt. It's midnight and I haven't had a chance to check the line ends up on a map correctly, but it looks pretty good to me:

;; Port of the Java code to decode google polylines that I found here -> http://jeffreysambells.com/2010/05/27/decoding-polylines-from-google-maps-direction-api-with-java
(defn decode-next-result [encoded]
  ;; keep scanning through encoded till b>=0x20
  ;; returns the next latitude/longitude increment
  (loop [[current & rest] encoded shift 0 result 0]
    (let [b       (- (int current) 63)
          result  (bit-or result (bit-shift-left (bit-and b 0x1f) shift))
          shift   (+ shift 5)] 
      (if (>= b 0x20) 
        (do 
          ;; if we are encoding the next result then we 
          ;; must have more characters to scan
          (assert rest)
          ;; keep looking for our next result
          (recur rest shift result))
        ;; we found our next result 
        (let [return-value (if (not= (bit-and result 1) 0)
                             (bit-not (bit-shift-right result 1))
                             (bit-shift-right result 1))]
          [return-value rest])))))

(defn lat-lng-double [lat-lng-int]
  (/ lat-lng-int 1E5))

(defn path-for-encoded-polyline [encoded] 
  (loop [rest encoded lat 0 lng 0 results []]
    (if rest
      ;; if there is anthing in the encoded array
      ;; we should have two more results at least 
      (let [next-result       (decode-next-result rest)
            new-rest          (second next-result)
            next-lat-result   (+ lat (first next-result))
            next-result       (decode-next-result new-rest )
            new-rest          (second next-result)
            next-lng-result   (+ lng (first next-result))]
          ;; add our lat lng result to the results
          (recur new-rest next-lat-result next-lng-result (conj 
                                                            results 
                                                            {:latitude (lat-lng-double next-lat-result) 
                                                             :longitude (lat-lng-double next-lng-result)})))
      ;; we are done, return our results
      results)))

#_
(def example-polyline "pdymEssfy[rJhAlANNeALyBD{@XqFBW^_AZi@NMp@sAPu@Bk@Ce@Qs@uA_Di@{@u@oAi@kAYm@_@mASgBOgBAcC?sBKyBYkFCyD@q@XyDLu@pAsDzDyJ`AaCJa@He@Bq@FoCw@[aBcAMQm@a@eAw@k@]yBaAwBaAqAaA_@S}DmBwD{BuLeHQKu@y@}BiCIOoEaLaJmUSQKSa@sAe@wDK}@@[g@}Am@uAkBkEkCmGg@gAk@{AQk@e@wBo@yAcAyAs@aAgDqEYWSUYa@o@{AoDeKcAwCkBqGg@eCQyAGcBC{AHsCmCf@iD^iAx@}AdAcA|@w@rAmAlBaBt@a@t@Y|Bo@xAs@z@y@n@iAf@wANq@J}@FyAEgAIkBImB?s@DkAHu@eCVeDRiGF}FGiB[sBc@{Am@qBMu@MiBe@gNIwAUmAk@uCMcAU_FKcAe@aCw@uByA}BaAwAMUw@{AoAoCk@mAYi@eAuBmBuD_AuBMICCiB_EeCsFYu@Uu@c@yCo@}CAe@i@sBIc@K_ACaADyAr@iFvBgMbB_Kz@eFj@kD^}A`@uA`CiHPcABYB[Dq@@]Dg@eBZeAr@oAl@w@VUdBsAr@c@~CuBbC_Bn@]d@_@V]NUPa@Pq@Hs@@{@Cm@Ii@Ug@UcA_@kAo@uCqAeHg@kCSi@GK]uAUs@OWEGSYQQk@e@}@a@c@Kc@AiBQiAA[CiC?GAwBAUAu@USI[UQ[GMMk@EqBG_DKsCCcBJcDcCIuBK")
#_
(path-for-encoded-polyline (seq (char-array example-polyline)))


Permalink - Comments - Tags: Development,Clojure