This is really well explained. I develop projection models that incorporate systems thinking in many of the ways you describe, and you've identified very specifically the places where the human (me, in this case) is still necessary. Beyond that, my models could get orders of magnitude better and still need exactly the same things from me. Spot on, and great article.
Who's running the show? As long as humans lead and take responsibility, AI is nothing more than a tool. Only when the last human leaves the scene — and AI runs everything — can we afford to stop caring about the systems we've built.
It might be ok now, while there are staff around who have done literally everything in designing, testing, maintaining and extending systems. But if AI starts doing some parts of the job, it means human based knowledge and experience has declined. But what happens when modifications are required and new testing shows there are problems? It may well be no human knows the reasons or even the questions and tests to understand what the system is doing. Humans still carry the can. They'll still need to be able to write/approve new requirements and specifications, with the test plans. Sometime there'll be a disaster and possibly no way back. So it's baby steps and if you can, have a way back. Demand more from the AI in a human readable form as some day you may need it.
I think this is a very important caution. AI may be able to assist with designing, testing, maintaining, and extending systems, but if humans gradually lose the ability to understand why a system behaves the way it does, we create a serious fragility. The goal should not be to let AI become an opaque replacement for systems knowledge, but to require AI to produce human-readable reasoning, traceable assumptions, testable specifications, and clear fallback paths.
At the same time, I don’t think technological change only eliminates work. When certain jobs fade or disappear, new roles often emerge because new industries, tools, risks, and coordination needs are created. The challenge is making sure people can transition into those roles instead of being left behind. In systems work especially, we may see less emphasis on manual execution and more demand for validation, governance, human judgment, architecture, ethics, risk management, and translating complex system behavior into decisions people can trust.
So I agree: baby steps, reversibility, and human-readable outputs matter. But I’d also add that the future of work may not be fewer humans in systems thinking, but different kinds of humans doing systems thinking at a higher level of abstraction.
This is really well explained. I develop projection models that incorporate systems thinking in many of the ways you describe, and you've identified very specifically the places where the human (me, in this case) is still necessary. Beyond that, my models could get orders of magnitude better and still need exactly the same things from me. Spot on, and great article.
Thank you!
Who's running the show? As long as humans lead and take responsibility, AI is nothing more than a tool. Only when the last human leaves the scene — and AI runs everything — can we afford to stop caring about the systems we've built.
Well said.
Great work on the distinction. AI accelerates the mechanics and the workflow, but judgment and purpose remain fundamentally human.
Yeah, I think so. I definitely understand the anxiety around displacement.
It might be ok now, while there are staff around who have done literally everything in designing, testing, maintaining and extending systems. But if AI starts doing some parts of the job, it means human based knowledge and experience has declined. But what happens when modifications are required and new testing shows there are problems? It may well be no human knows the reasons or even the questions and tests to understand what the system is doing. Humans still carry the can. They'll still need to be able to write/approve new requirements and specifications, with the test plans. Sometime there'll be a disaster and possibly no way back. So it's baby steps and if you can, have a way back. Demand more from the AI in a human readable form as some day you may need it.
I think this is a very important caution. AI may be able to assist with designing, testing, maintaining, and extending systems, but if humans gradually lose the ability to understand why a system behaves the way it does, we create a serious fragility. The goal should not be to let AI become an opaque replacement for systems knowledge, but to require AI to produce human-readable reasoning, traceable assumptions, testable specifications, and clear fallback paths.
At the same time, I don’t think technological change only eliminates work. When certain jobs fade or disappear, new roles often emerge because new industries, tools, risks, and coordination needs are created. The challenge is making sure people can transition into those roles instead of being left behind. In systems work especially, we may see less emphasis on manual execution and more demand for validation, governance, human judgment, architecture, ethics, risk management, and translating complex system behavior into decisions people can trust.
So I agree: baby steps, reversibility, and human-readable outputs matter. But I’d also add that the future of work may not be fewer humans in systems thinking, but different kinds of humans doing systems thinking at a higher level of abstraction.