Wed 20 November 2024
I started playing with Clojure in 2013 and was immediately impressed.
It had the killer combination of immutable data structures, functional programming,
an interactive REPL, and structural editing.
I dabbled a bit, created a few demos, fell in love with the language, evangelized everywhere I could,
then went back to writing OtherLang because that's what pays the bills.
This is a common sentiment, as I came to learn.
Clojure is definitely a niche language, but the uncompromising focus on both quality and fun seems to compel programmers to use it.
Ten years later, for no reason in particular, I opened up my old Clojure projects and dove in again.
The ecosystem had changed a bit: We now had deps.edn as an alternative to Leiningen (which I did not enjoy), an LSP, a linter, and shadow-cljs to name a few.
But my decade-old code still "just worked" for the most part; the changes were limited to the outer layer, the developer experience.
The core language is stable, a breath of fresh air!
So I decided to attend the 2024 Clojure Conj on my own dime. And I'm glad I did!
I found a friendly community of thoughtful folks eager to talk about Clojure
and how they're applying it in the wild.
The videos for all 2024 talks are now available on youtube!
Here's a minimally-edited summary of my notes of the experience, better late than never...
Venue
On day 1, I was convinced I'd shown up at the wrong place. The George Washington Masonic Memorial
is an impressive building and not your typical conference venue! Very cool.
The amazing day of datomic
One of my primary quests was to learn everything I could about Datomic, and evaluate it as an alternative to Postgresql for certain use cases.
Datomic Pro is now Apache licensed, free to use by anyone. However, that's only the compiled jar file; the source code isn't available. And I'm fine with that, open-source purists can make that decision for themselves.
The core idea of Datomic can be summarized as an unbundled, immutable database with identity and time as first class citizens.
Let's break that down:
- Datomic is unbundled: the writer, the readers, and the storage layer are separate components.
Rather than a centralized database server that serves and coordinates all client queries, Datomic has a peer architecture.
You run queries directly in your application process, allowing your app to hit storage directly without going through a central server.
Combined with a storage engine like DynamoDB, this gives you reads that can be horizontally scaled.
The write load continues to be served by a single server in charge of building efficient covering indexes and notifying peers, known as the "transactor".
- It is immutable in that it acts as a ledger. You don't delete data, you assert new facts and retract old facts. Those facts never change.
- It uses an EAVT data model where every fact is a tuple of entity, attribute, value and time. The notion of time and identity allows datomic to describe a full history instead of just the current mutable state. You can time travel to any point and compare values across time.
We learned how to set up a schema and transact new entities,
how to write queries with Clojure's flavor of datalog,
discussed some more complex topics around architecture and performance.
Day 2
Day two started with more datomic, this time a deap dive into the architecture and some new
observabilty features: io-stats for disk IO and tx-stats for transaction timing.
It also introduced to me the idea of squiids, sequential UUIDs, which sort by time but still looks like an arbitrary uuid4.
This allows for serial id-like performance (index locality) without exposing an integer that reveals the size of your database.
This sounds like a perfect compromise to the bigint vs uuid tradeoff that most postgres users face.
There was a surprising amount of discussion of Python, mainly in the context of machine learning.
ML/AI is fully dominated by Python and even shops that are all-in on Clojure still work with
models trained in Python. I learned about ONNX, a standard interchange format which allows
to package up a Python model (sklearn or pytorch) and run the inference engine in another language.
I learned of test.contract an impressive library for mocking external systems.
This is ideal for deterministic simulation testing of distributed systems
Though I'll still strongly advocate for more integration and e2e tests,
this talk gave me a glimpse of more advanced unit testing techniques that might come in handy some day.
I was introduced to the standard-clj formatter which aims to be the canonical formatter. I'm not sure how it compares to cljfmt which I current use.
Arne gave a wonderful lightning demo of Overtone, a music composition software library for Clojure.
If you've never seen anyone create a live music experience with an emacs REPL, it's hard to describe. As I'm
currently studying music theory, I plan on diving into overtone with gusto when I return. Inspirational.
Luke gave a wonderful overview of resource description framework (RDF)
and the current state of LLMs. Yes, the same RDF that underpinned the
"Semantic Web" ideas (remeber that?).
The thesis was compelling: LLMs could use RDF to gain accuracy and logical consistency.
He pointed out an AI-from-scratch tutorial that seems comprehensive
and mentioned the seminal paper that started this whole AI gold rush: Attention is all you need
Probably my favorite talk of the conference, Paula gave us an experience report
of porting the clojure.java.math library to ClojureScript.
This meant navigating licenses and IP concerns, the Clojure patch process,
and reimplementing many of the core math functions.
Most impressive was the test harness for thoroughly testing cljs-math against its
Clojure sibling. Very impressive work, and really embodied the community focus on quality, stability,
and keeping parity between the two main hosts platforms (Java and Javascript).
Day 3
Rich Hickey started the day with a brief but inspriring message.
He's on his way to retirement and seems in many ways to be "passing the torch" to others in the community.
And his message was simple: Clojure is for optimists,
expert programmers who take pride in the quality of their work and
face down gnarly problems with intellectual curiosity and a good attitude.
It's a concise and accurate way to describe the vibe of the community.
I learned about Kevel's architecture of distributed system on AWS.
Definitely worth a re-watch, this is an experience report that carries lots of valuable insigh.
There was a strong emphasis on causality, which I appreciated.
More background reading on causal consistency in the cloud.
Scientific clojure has a real potential. It's got interoperability with Python, Wolfram, etc. pluys a
native clojure tool ecosystem that covers much of modern "data science" stack.
All the pieces are there, a la carte in true Clojure fashion.
We got a great Metabase experience report. I learned all about pet birds, astrology signs, and which crystals to put on your desk to enhance your energy flows.
Seriously. The opening 5 minutes of the talk might have been the hardest I've ever laughed at a conference.
Standup commedy gold. Eventually talk turned back to Clojure and I got to hear an experience report that
gave me a taste of some of the challenges faced by teams on large clojure codebases.
The Clojure Camp folks and their presentation was inspiring. Despite Clojure being an unabashedly
expert-oriented tool, there is an unmistakable community commitment to welcoming newbies and
taking learning seriously.
Colin Fleming, the author of Cursive, demonstrated how LLMs could be more tightly integrated into our IDEs, as well as some of their drawbacks.
Once again, the tone was pleasantly objective: no over-hyping nor hating on AI, just solid observations.
The Caveman web framework/tutorial was released a few days earlier;
he showed how Claude sonnet could follow the tutorial and build the software from English instructions. Impressive!
To end the conference, Alex Miller gave an update to last year's keynote, Design in practice.
It's a strategy for a software design process with six steps:
Describe, Diagnose, Delimit, Direction, Design, then Develop
Why front-load the thinking, as opposed to simply sprinting away?
"The solution should feel obvious once you've written the right problem statement" - Rich Hickey (via Alex)
The design process and the presentation itself was in Google Sheets! Definitely the first time I've seen a slideshow embedded in a spreadsheet.
It was a fascinating glimpse into the inner workings of the Clojure language design process leading up to the 1.12 release,
which emphasizes clear thinking and real-world feedback loops.
Notice that writing production code is the final step and you can backtrack anytime;
I can't see many Agile/Scrum shops engaging
with such an extended planning process but it evidently yields high quality software.
Fin
I came away from the conference with renewed enthusiasm, not just for Clojure but for the field of
software engineering as a whole.
The 2024 zeitgeist would say the software industry is in crisis: "the job market is terrible" and "AI is going to take our jobs".
Seems as though software projects are increasingly drowning in unbounded cloud complexity on one hand, and the weight of legacy technical decisions on the other.
But people have been saying that about software for decades!
Is it incompetent programmers or bad managers or greedy executives or the technology they picked? Who knows.
The majority of software projects still fail and we don't really know why.
But functional programming, and Clojure in particular,
offer a viable alternative to the madness, a way out of the tarpit.
It's not a silver bullet of course, no technology is. But in the hands of a talented and motivated group of developers,
you can build challenging software systems to very high standards with Clojure. And have fun doing it.
See you at Clojure/Conj 2025!