New Blog! Evaluate Your Agents!
One of the key concepts that is visited again and again as people seek meaning is the notion of scale. At either end of the spectrum, humans tend to become reflective and we instinctively measure ourselves against this notion.
There are romantic and philosophical schools of thought that tend toward focusing on the small things around us. Blake captures this in “The Auguries of Innocence” as:
To see a world in a grain of sand
And a heaven in a wild flower,
Hold infinity in the palm of your hand
And eternity in an hour.
Then there are also those that encourage us to look to the stars. Mythology, Science Fiction and Fantasy authors have deeply embedded these notions in our culture. Many science fiction authors tease us with the concept of “other” knowledge possessed by extra terrestrials - and this appeals to us, apparently profoundly.
Similarly, consider the worlds that are brought alive in mythologies or fantasies. What would you do if you were living alongside beings that did not die of old age, such as the elves in Tolkien’s books? Is the tragedy of death for a being that could live forever worse than for a mere mortal?
Concrete, pragmatic thinkers might say that this is all nonsense and the stuff of romantics and fantasists, not reality. They may say the middle path is the right one (not too small, not too large) and perhaps point to the pragmatism of Marie Kondo as the path to happiness; “just enough” is perfect. You may also get a reminder that we are all, obviously, finite and perhaps we should consider our personal Johari Window:
“Work on your blind spot”, I might hear, along with an admonition that even small increments will help me become a better person. (For the record, not bad advice.)
Why am I raising these points and why should you care? I am, of course, leading into a discussion on how to think about the scale and experience of working with Large Language Models.
Let us return to the large end of the spectrum and the numinous flashes of insight it can spark.
Douglas Adams, in “The Hitchhiker’s Guide to the Galaxy” introduces us to the “Total Perspective Vortex”. Gemini summarizes this device as follows:
[The Total Perspective Vortex] is reputed to be the most psychologically devastating torture implement in the galaxy. It operates by giving the victim a momentary, unfiltered glimpse of the entire, unimaginable infinity of creation. Simultaneously, it shows them a microscopic dot on a microscopic dot, indicating "You are here." The sheer scale of their own insignificance is meant to annihilate the victim's psyche.
As the passage suggests, toying with the numinous and seeking to touch infinity carries its risks. Classically, the responses were summarized as:
Mysterium tremendum: Awe and reverence tending towards dread or terror in the face of something incomprehensible.
Mysterium fascinans: A sense of fascination and attraction towards a mysterious presence.
As it happens, the “unknown” box of our Johari windows is actually really big and very interesting. LLMs can give us the chance to poke around in it. This may terrify you, fascinate you, or some combination of the two.
So, we are on this journey into a huge space. Fellow travelers, how do we navigate this and how do we do it safely? These are conceptual questions, but also tremendously practical ones that our clients are facing every day as these technologies explode in importance.
The classic management approach of minimizing risk and incrementalism is largely ineffective in this pursuit. To gain a sense of scale of current large models, let’s take a look at Llama 4 Behemoth, with 288 Billion parameters. For simplicity, let’s consider a parameter as a scale on which something is assessed, like temperature or height. Now, let’s consider an additional parameter every second, of every minute, of every hour, of every day since the dawn of the Agricultural revolution. That gets us up into the right sphere of magnitude.
In contrast to incrementalism are the combined concepts of exploration, openness and making conceptual leaps. When examining a problem with the assistance of an LLM, the limit to how far you can go is not typically the knowledge encoded in the system, it’s your enthusiasm for pursuing a thread, your openness to new ideas and your ability to navigate from one pattern to the next as you do so without becoming exhausted.
There are analogies between this divergent exploration experience and the initial “Discover” phase of the classic “double diamond” methodology. But with a difference because of this new and massive scale.
Bringing us back to more practical discussion points, what types of individuals do well in this divergent phase of the new paradigm? This is an area the author has been thinking about for decades and, in large part, the reason why the domain “fluidmind” was reserved over 20 years ago.
Building on this exploration, we arrived at the following framework as a way to define the skillsets for this divergent phase, with some reasonable definition around the categories borrowed from developmental psychology:
The lower levels of this “stack” are becoming reasonably well recognized in AI-forward organizations, but the higher levels are exceptionally rare. Not only are they difficult to find, but in many ways these are roles that do not do well in classically organized companies. Despite heroic case study reports, these are often the individuals that get fired, rather than hired.
There is much more to share on this analysis, possibly in a future blog post. The full analysis comes with a series of self-assessment questions for evaluation on this scale. A couple of the more intriguing ones include:
Are you familiar with and able to achieve “flow state” in your pursuit of problem solutions?
Have you developed a discipline of cognitive perspective-shifting as you examine novel solutions to challenges?
And a challenge for you, reader: Where do you think you fit?
Science fiction fans may see an analogy with finding themselves in the Tesseract from Interstellar and wondering how we re-converge or find a way out of this phase. What will be your book-pushing, binary encoding, moment?
That, readers, will be the subject of the next blog post. LLMs are not infinite, but given the scale of their ingested knowledge and our inability to match it, they may as well be in some respects. We will continue to “touch infinity” as we do so.
Postscript: Credit to Claude for being my assistant in navigating the analyses in this post!