I just stumbled upon your page and I can't wait to read more! I would be curious about how roles as we know them might transition. For example, I work as a data scientist in higher ed. and I think that - because everybody around me is vibe coding now - my job will shift to building the frameworks that let others build well, rather than building tools or doing analyses myself.
When we look at AI's highest potential for disruption, it's in the rent seeking sector: finance, law, education, medicine, and administration. It's obviously not the case that these sectors are 100% rent seeking but they do contain a disproportionately large amount of it. Exploiting information asymmetry for private gains made from intentionally convoluted rules and gatekeeping is far more destructive to equality than tech billionaires. Unnecessary credentialism and the pursuit of it harms innovation. The pie only shrinks from rent seeking. Think of the legions of tax lawyers, accountants, and other specialists and all that intellectual horsepower dedicated to squaring someone's bill with the government. If AI shatters those types of enterprises, intelligent people will have to dedicate their lives to more productive endeavors and in doing so may lead to less inequality, not more.
Thank you Hadley for all you do. In my public service data science team in the Canadian government, we do our best using R to try to make the world a little bit better of a place.
I think the other potential hope for AI is that it becomes way easier for everyone to make cool and useful apps, so more people can make a living from it.
I use Claude Code to write Quarto files directly in terminal. If a good prompt is provided, or a reference is available, it could write a bunch of quarto files and render them.
Agree, we must feel excited and open, but also wary and be careful about the broader implications of these kind of technologies. That being said, I think Posit and the R community in general are well positioned for the AI era, if not maybe "too" well positioned. AI burnout is real and every day more people are getting fed up of everything being about AI. Yet it's still interesting and useful! It's hard to find balance!
Regarding the power consumption issue with AI, there is a simple idea that can make AI much greener: instead of a company building one $500M data center, they can build ten $50M data centers at widely distributed locations, AND tell the local governments "in exchange for rapid permitting, we'll agree to AN INTERRUPTIBLE POWER CONTRACT."
Why would this make things better? Because it would address the main issue with solar and wind power: their availability depends on the weather. In many places, sometimes renewable power exceeds local demand and sometimes there is not enough renewable power. One solution to this issue is upgrading the grid so there's more capacity to load balance by moving power to other places. But turning data centers into dispatchable loads would be a lot cheaper.
A company with a very large number of small data centers could optimize its power mix by shifting compute around to follow power availability.
Thank you for holding this conversation in the open. I look forward to reading your reflections. Two things come to mind I’d love to hear from you: 1) what are your impressions so far from you “6 meetings a week”? Common themes? Patterns? 2) what are emerging standards of practice for organizing the semantic layer to support AI-supported project development in reproducible research?
Glad to see you show back up in my inbox! Something I have noticed is not so much a "learning curve" as a "context curve" with AI, where my use and the tool's context and memory improve overtime. Some of this is deliberate on my part (getting better and refining .md files), some of it is the settings, some of it happens in the background during and across code sessions. That said, I am curious how you think about this and how it gets operationalized at Posit, or in your own work. This is something I have seen new-to-AI folks misunderstand but it seems pretty critical to deploying these tools well.
Interesting that you find this so helpful as I try to do as it as little as possible because I'm sceptical that skills/memories are going to be really transferable from model to model.
I am teaching data science with R at the DHBW in Mannheim Germany. My question is, how much basics of the tidyverse are useful. How can we combine basics and AI for students.
Hey Hadley , thanks for putting this in writing and being interested on how the R community feels about this !
For me, it would be interesting to understand how you (and the Posit team) actually get coding agents to write idiomatic (tidyverse styleguide say) R code. Are you adding this just to a local skill/AGENTS.md , and it works ?
For R specifically, I encounter more problems than say for Python. It may also be because over the years I developed a particular way of writing R , and am not really satisfied with the code the clanker generates, naming, the way it structures classes and functions and so on. Also I agree with the questions @Wizards Points wrote below
I don't think I do anything special in order to get claude to write idiomatic R code. Maybe you're using a different model/provider? Or maybe it's because I'm usually working inside of packages where there's a lot of existing code for it to look at?
I just stumbled upon your page and I can't wait to read more! I would be curious about how roles as we know them might transition. For example, I work as a data scientist in higher ed. and I think that - because everybody around me is vibe coding now - my job will shift to building the frameworks that let others build well, rather than building tools or doing analyses myself.
When we look at AI's highest potential for disruption, it's in the rent seeking sector: finance, law, education, medicine, and administration. It's obviously not the case that these sectors are 100% rent seeking but they do contain a disproportionately large amount of it. Exploiting information asymmetry for private gains made from intentionally convoluted rules and gatekeeping is far more destructive to equality than tech billionaires. Unnecessary credentialism and the pursuit of it harms innovation. The pie only shrinks from rent seeking. Think of the legions of tax lawyers, accountants, and other specialists and all that intellectual horsepower dedicated to squaring someone's bill with the government. If AI shatters those types of enterprises, intelligent people will have to dedicate their lives to more productive endeavors and in doing so may lead to less inequality, not more.
Thank you Hadley for all you do. In my public service data science team in the Canadian government, we do our best using R to try to make the world a little bit better of a place.
I think the other potential hope for AI is that it becomes way easier for everyone to make cool and useful apps, so more people can make a living from it.
I use Claude Code to write Quarto files directly in terminal. If a good prompt is provided, or a reference is available, it could write a bunch of quarto files and render them.
Agree, we must feel excited and open, but also wary and be careful about the broader implications of these kind of technologies. That being said, I think Posit and the R community in general are well positioned for the AI era, if not maybe "too" well positioned. AI burnout is real and every day more people are getting fed up of everything being about AI. Yet it's still interesting and useful! It's hard to find balance!
I’d definitely be curious to hear more about “effective use of git” being more within reach to a broader audience.
looking forward to reading more of your thoughts on all of this. Thanks for rebooting the Substack.
Regarding the power consumption issue with AI, there is a simple idea that can make AI much greener: instead of a company building one $500M data center, they can build ten $50M data centers at widely distributed locations, AND tell the local governments "in exchange for rapid permitting, we'll agree to AN INTERRUPTIBLE POWER CONTRACT."
Why would this make things better? Because it would address the main issue with solar and wind power: their availability depends on the weather. In many places, sometimes renewable power exceeds local demand and sometimes there is not enough renewable power. One solution to this issue is upgrading the grid so there's more capacity to load balance by moving power to other places. But turning data centers into dispatchable loads would be a lot cheaper.
A company with a very large number of small data centers could optimize its power mix by shifting compute around to follow power availability.
Thank you for holding this conversation in the open. I look forward to reading your reflections. Two things come to mind I’d love to hear from you: 1) what are your impressions so far from you “6 meetings a week”? Common themes? Patterns? 2) what are emerging standards of practice for organizing the semantic layer to support AI-supported project development in reproducible research?
Both of those are already on my list 😀
Glad to see you show back up in my inbox! Something I have noticed is not so much a "learning curve" as a "context curve" with AI, where my use and the tool's context and memory improve overtime. Some of this is deliberate on my part (getting better and refining .md files), some of it is the settings, some of it happens in the background during and across code sessions. That said, I am curious how you think about this and how it gets operationalized at Posit, or in your own work. This is something I have seen new-to-AI folks misunderstand but it seems pretty critical to deploying these tools well.
Interesting that you find this so helpful as I try to do as it as little as possible because I'm sceptical that skills/memories are going to be really transferable from model to model.
That said, I do maintain https://github.com/r-lib/usethis/blob/main/inst/claude/CLAUDE.md, which I use across all R packages that I work on.
I am teaching data science with R at the DHBW in Mannheim Germany. My question is, how much basics of the tidyverse are useful. How can we combine basics and AI for students.
Good question! I have a few thoughts which I'll write up for a future post.
Hey Hadley , thanks for putting this in writing and being interested on how the R community feels about this !
For me, it would be interesting to understand how you (and the Posit team) actually get coding agents to write idiomatic (tidyverse styleguide say) R code. Are you adding this just to a local skill/AGENTS.md , and it works ?
For R specifically, I encounter more problems than say for Python. It may also be because over the years I developed a particular way of writing R , and am not really satisfied with the code the clanker generates, naming, the way it structures classes and functions and so on. Also I agree with the questions @Wizards Points wrote below
I don't think I do anything special in order to get claude to write idiomatic R code. Maybe you're using a different model/provider? Or maybe it's because I'm usually working inside of packages where there's a lot of existing code for it to look at?