Manipulate source code like the dom
by Chris Zheng,
I've always believed that most programming problems are expression problems. Being able to say what it is that we want usually gets us 95% of the way to solving the problem. The rest usually takes care of itself. Libraries and tools should help us express ourselves through higher paradigms of thinking, where the most powerful features are declarative paradigms that allow us to just say what we want and for the library to figure out how to get us there.
Source code manipulation has always been a great source of difficulty. Most source code manipulation programs are really just built upon regular expressions with some parsing and then lots of string manipulation. Lisp code, having a more regular shape tends to be easier to manipulate, but I haven't really seen any nice tools for directly dealing with source code.
Having said that, I'm extremely excited to show off a new library for source code manipulation based upon principles that lisp code is in essence a huge tree. The library is called jai and is inspired by css/xpath/jquery. I've been working on and off on this concept for about a year but it came together in the past month, having had some time off to polish off the fine grain control and the placement of the zipper at the exact location that I want it to be. The traversal code alone took 3 tries to get right. Tree-walking is super hard and I now have a real appreciation for rewrite-clj, without which, this library would have been impossible.
As lisp code follows a tree-like structure, it is very useful to have a simple language to be able to query as well as update elements of that tree. The best tool for source code manipulation is rewrite-clj. However, it is hard to reason about the higher level concepts of code when using just a zipper for traversal. jai is essentially a query/manipulation tool inspired by jquery and css selectors that make for easy dom manipulation and query. Instead of writing the following code with rewrite-clj:
(use 'rewrite-clj.zip :as z)
(if (and (-> zloc z/prev z/prev z/sexpr (= "defn"))
(-> zloc z/prev z/sexpr vector?)
(do-something zloc)
zloc)
jai allows the same logic to be written in a much more expressive manner:
(use 'jai.query)
($ zloc [(defn ^:% vector? | _)] do-something)
More examples can be seen in the documentation
There have been many forerunners for of thinking about source code as data; tangible data that we can control, reason about and manipulate as we see fit. The first for me was codeq, the second being rewrite-clj. The fact that we can take source code and change it structurally with so much ease is the reason why I find the lisp paradigm so incredible. There is no way to do this in any other language bar html/xml (hence the css/xpath inspired syntax).
Another paradigm that I've been playing with is that the query should take the same shape as the actual thing that we are querying for. This is nothing revolutionary; in fact, it's kind of common-sensical. We are seeing this with mongodb, graphql but to be honest, we should be using it everywhere. We saw a huge uptake in mongodb because people saw how easy it was to create applications quickly due to the fact that there was no mental overhead of using sql. So with jai, I wanted it to feel as intuitive as possible. It makes heavy use of core.match to do some cool pattern matching behind the scenes. In the past, the trade-off between speed and expressiveness meant that being declarative can lead to tremendous losses in performance but now, it matters less and less. Of course there may be exceptions to this rule but in general we as programmers/toolmakers should act as enablers, not gatekeepers. Programming should be easy, intuitive and above all else, fun.
So please have a play =)
