Hadley, I don't know why, but I understand 'the world' better when the explanations come from R community! I was trying hard, very hard to understand, use, build agentic solutions due to workplace adoption. But deep down just felt that I'm acquiring only shallow knowledge (pointed out in the last month's Tidy design principles), and not confident enough to use it intelligently to solve business problems. I'm an R person but works in a Python only team. For me, the best use of Claude (or other similar tools) so far was to translate my R thinking & code into Python. From today, I will have little more confidence in designing, building, govern and cost-benefit analysis of agentic solutions. Thank you!
Good article Hadley - though I do admit I was asking 'Who or what is ellmer?' I've been out of the R game for a while so didn't realise it was an actual package :)
Nice intro and definition of agent. Agent defined as LLM in harness is my takeaway.
Harness seems as important as the LLM from what I see and perhaps a lot of intelligence of an LLM with agentic AI can be attributed to harnesses. If a tool call is likened to a function, I am mentally equating harness to a library. Better the libraries, better (shall we say tidier) the intelligence.
Is it possible Claude has better harnesses and uses them well? Or are there more things that I dont know that I dont know
I liked it. I was wondering if I could create functions and then get and LLM to execute them. for instance, I'm more comfortable with data I gather using my functions than letting an LLM do the work. Is that even possible?
You can! That's one of the strengths of {ellmer}. With `chat$register_tool()` you can provide an LLM with any function. I've made many functions that load and filter data, so that an LLM can use them to answer stuff about my data without having to give it the whole database. You can even make the function return the filtered dataframe along with other summary dataframes inside a list to that the LLM gets the filtered data plus more context (like grouped counts, grouped means, etc.)
Do I understand correctly Bastián, the LLM still gets to see the data? i.e. not just run a pre-defined function natively on your own machine. I deal with health data and am currently avoiding using LLMs for coding because I am responsible for ensuring all the data stays in house.
It does not see the data; it sees the result of calling the function. That implies you need to be careful with what tools you supply, since if the tool just prints the data then the llm would see it.
The tool boundary is the point where an agent stops being only a conversation. Once it can call tools, the design question becomes scope, reversibility and observability: what can it touch, what can it change, and what record does it leave? I wrote about that action layer here: https://petermccannstrain.substack.com/p/tools-give-models-hands
The important thing I learned today is that, because of that stateless thing, "Thanking your LLM" in a long conversation burns through your tokens unnecessarily, because it sends over the entire conversation. Super informative post, Hadley.
Hey Hadley, really intriguing post, it helped a ton with my understanding of these "agents". I have a curious question. How does the LLM know that to delete a file, it first needs to list the files in the directory?
This is presumably just general knowledge baked into the model — if to delete a file you need to know its name, then you need to first determine the possible names.
one afterthought..the word harness sometimes conjures up a horse restrained and running in a controlled fashion...whereas we know that LLMs running amok in a harness is the issue..can someone comment on this word
I think it's more along the lines of this definition: "a set of straps and fittings by which a horse or other draught animal is fastened to a cart, plough, etc. and is controlled by its driver."
i.e. the harness is how we attach the LLM to something that carries the work and how we steer it in the right direction.
Thanks for the reply Hadley. I see harness term widely accepted as well. And genuinely elated reg Ellmer.
I reconciled with the word harness over a chat with claude as follows. (It is entirely possible with a different prompt different type of replies/interpretations are possible, but for me this works)
me to claude -- my issue is that by putting in a word harness, which indicates total control as well, the llm doing wrong things / hallucinations is not explainable
is the driver who controls the horse, missing? is the whole appliance a driverless horse cart with passengers and hence terrifying
claude reply -- You've identified a real tension that the field hasn't fully resolved.
Your analogy is accurate:
The "LLM in a harness" framing implies the harness controls the LLM the way a harness controls a horse. But it doesn't. The harness controls the scaffolding around the LLM — the loop, the tool calls, the memory. It does not control what the LLM decides to do at each step.
So yes — the horse is not under the harness's control. The harness is just strapped onto the horse.
The driver is missing.
The driver's seat exists. It's just mostly empty — or occupied by another horse.
This is exactly why agentic AI is genuinely risky — not because the tools are dangerous, but because the architecture has no true real-time authority figure. The field is working on this
I don't think that's much different from controlling a horse — you point it in the direction you want it to go, but it can still do things that you don't want it to do. (There's maybe also some room in this metaphor for the LLM- equivalent to horseshit 🤣)
Good to see you posting Hadley, it always helps crystallize concepts and ideas.
I have been thinking about AI coding from a trust perspective - coding agents like ClaudeCode get their power by the permissions implicit in letting them be able to delete or modify files. We give them the “agency” and slowly move from approving every function call to giving them broader and broader permissions. OpenClaw went even further giving personal access and unmonitored proactive loops on your behalf.
From playing with other coding harnesses like OpenClaw and Pi, I see performance similar or better than ClaudeCode, so I don’t think the harness itself is a particularly big moat. Am working on one myself with my new AI programming language, AILANG, inspired by functional programming like R and Haskell. Where the trust and permission system lies when using AI in coding I think is where software engineers will be focusing a lot on in the future, and I look forward to what you and Posit think up in that area.
Hadley, I don't know why, but I understand 'the world' better when the explanations come from R community! I was trying hard, very hard to understand, use, build agentic solutions due to workplace adoption. But deep down just felt that I'm acquiring only shallow knowledge (pointed out in the last month's Tidy design principles), and not confident enough to use it intelligently to solve business problems. I'm an R person but works in a Python only team. For me, the best use of Claude (or other similar tools) so far was to translate my R thinking & code into Python. From today, I will have little more confidence in designing, building, govern and cost-benefit analysis of agentic solutions. Thank you!
Because R people are used to abstracting better. And not getting unnecessarily technical. Maybe due to the functional language.
Good article Hadley - though I do admit I was asking 'Who or what is ellmer?' I've been out of the R game for a while so didn't realise it was an actual package :)
Nice intro and definition of agent. Agent defined as LLM in harness is my takeaway.
Harness seems as important as the LLM from what I see and perhaps a lot of intelligence of an LLM with agentic AI can be attributed to harnesses. If a tool call is likened to a function, I am mentally equating harness to a library. Better the libraries, better (shall we say tidier) the intelligence.
Is it possible Claude has better harnesses and uses them well? Or are there more things that I dont know that I dont know
I liked it. I was wondering if I could create functions and then get and LLM to execute them. for instance, I'm more comfortable with data I gather using my functions than letting an LLM do the work. Is that even possible?
You can! That's one of the strengths of {ellmer}. With `chat$register_tool()` you can provide an LLM with any function. I've made many functions that load and filter data, so that an LLM can use them to answer stuff about my data without having to give it the whole database. You can even make the function return the filtered dataframe along with other summary dataframes inside a list to that the LLM gets the filtered data plus more context (like grouped counts, grouped means, etc.)
Do I understand correctly Bastián, the LLM still gets to see the data? i.e. not just run a pre-defined function natively on your own machine. I deal with health data and am currently avoiding using LLMs for coding because I am responsible for ensuring all the data stays in house.
It does not see the data; it sees the result of calling the function. That implies you need to be careful with what tools you supply, since if the tool just prints the data then the llm would see it.
Lots will have same requirement to not let data out but some answers from data. Another demystification that would be good is the buzzword RAG.
Thank you Hadley. It helps to know that, though I am still concerned (now also with my own stupidity of using the wrong tool!).
The tool boundary is the point where an agent stops being only a conversation. Once it can call tools, the design question becomes scope, reversibility and observability: what can it touch, what can it change, and what record does it leave? I wrote about that action layer here: https://petermccannstrain.substack.com/p/tools-give-models-hands
The important thing I learned today is that, because of that stateless thing, "Thanking your LLM" in a long conversation burns through your tokens unnecessarily, because it sends over the entire conversation. Super informative post, Hadley.
Hey Hadley, really intriguing post, it helped a ton with my understanding of these "agents". I have a curious question. How does the LLM know that to delete a file, it first needs to list the files in the directory?
This is presumably just general knowledge baked into the model — if to delete a file you need to know its name, then you need to first determine the possible names.
Excellent post, thanks
one afterthought..the word harness sometimes conjures up a horse restrained and running in a controlled fashion...whereas we know that LLMs running amok in a harness is the issue..can someone comment on this word
I think it's more along the lines of this definition: "a set of straps and fittings by which a horse or other draught animal is fastened to a cart, plough, etc. and is controlled by its driver."
i.e. the harness is how we attach the LLM to something that carries the work and how we steer it in the right direction.
Thanks for the reply Hadley. I see harness term widely accepted as well. And genuinely elated reg Ellmer.
I reconciled with the word harness over a chat with claude as follows. (It is entirely possible with a different prompt different type of replies/interpretations are possible, but for me this works)
me to claude -- my issue is that by putting in a word harness, which indicates total control as well, the llm doing wrong things / hallucinations is not explainable
is the driver who controls the horse, missing? is the whole appliance a driverless horse cart with passengers and hence terrifying
claude reply -- You've identified a real tension that the field hasn't fully resolved.
Your analogy is accurate:
The "LLM in a harness" framing implies the harness controls the LLM the way a harness controls a horse. But it doesn't. The harness controls the scaffolding around the LLM — the loop, the tool calls, the memory. It does not control what the LLM decides to do at each step.
So yes — the horse is not under the harness's control. The harness is just strapped onto the horse.
The driver is missing.
The driver's seat exists. It's just mostly empty — or occupied by another horse.
This is exactly why agentic AI is genuinely risky — not because the tools are dangerous, but because the architecture has no true real-time authority figure. The field is working on this
I don't think that's much different from controlling a horse — you point it in the direction you want it to go, but it can still do things that you don't want it to do. (There's maybe also some room in this metaphor for the LLM- equivalent to horseshit 🤣)
Yes, good one
Good to see you posting Hadley, it always helps crystallize concepts and ideas.
I have been thinking about AI coding from a trust perspective - coding agents like ClaudeCode get their power by the permissions implicit in letting them be able to delete or modify files. We give them the “agency” and slowly move from approving every function call to giving them broader and broader permissions. OpenClaw went even further giving personal access and unmonitored proactive loops on your behalf.
From playing with other coding harnesses like OpenClaw and Pi, I see performance similar or better than ClaudeCode, so I don’t think the harness itself is a particularly big moat. Am working on one myself with my new AI programming language, AILANG, inspired by functional programming like R and Haskell. Where the trust and permission system lies when using AI in coding I think is where software engineers will be focusing a lot on in the future, and I look forward to what you and Posit think up in that area.
Thank you Bastian. Sorry I missed the tilde.