On the Seam: Working Drafts · Part 2 of a three-part series
What My Research Got Right About GenAI Won't Hold for Agentic AI
My doctoral model explained why people advocate for generative AI. The human-in-the-loop assumption underneath it is exactly what autonomous agents remove — and when you remove it, two of my cleanest results invert.
I spent three years of doctoral research building a model of why people advocate for generative AI. It worked. Legitimacy and trust explained most of the variance in whether someone would champion a GenAI tool to a colleague. The model held up under scrutiny, survived a committee, and earned the degree. I believed it.
I no longer believe it will hold for agentic AI. Not because the research was wrong — it was right about the world it measured. It will not hold because that world is disappearing. The human-in-the-loop assumption that quietly underwrote every finding is the exact thing autonomous agents remove. And when you remove it, I suspect two of my results do not just weaken. They may invert.
This is uncomfortable to write. It is easier to extend a finding than to retire one. But the honest move is to say where the boundary sits, and my boundary is autonomy. I will proceed on that basis.
One caveat before I go on, because I want to be honest about what kind of claim this is. My data measured generative AI, with a human reviewing every output. The reversal I am describing under genuine autonomy is a suspicion I am reasoning toward from mechanism, not a result I have measured. I believe the argument is sound, but I have not yet run the study that would confirm it. Read what follows as where I think the evidence points — offered so it can be tested and argued with, not asserted as settled.
What the research actually found
My study modeled advocacy intention — the willingness to recommend an AI system to others — as a function of perceived legitimacy and trust. Legitimacy, following Suchman, came in dimensions: cognitive (does it make sense), pragmatic (is it useful), normative (is it appropriate), and regulative (does it comply). Trust and distrust entered as separate forces, because a long line of work going back to Lewicki and McKnight establishes that they are not two ends of one dial. They are different systems with different triggers.
Three things drove advocacy: cognitive legitimacy, pragmatic legitimacy, and trust. People championed AI they could understand, that was useful, and that they trusted. No surprise there.
Two things did not move the needle. Distrust had no direct effect on whether people advocated. And regulative legitimacy — compliance, formal rules — did not matter either. At the time I treated these as clean null results. They were clean. They were also conditional, and I did not see the condition until I started watching agents act.
The condition was the human
Here is the mechanism I missed. In the world I measured, a person was always in the loop. The AI summarized a document; the person read the summary. The AI drafted advice; the person decided whether to take it. Every output passed through a human checkpoint before it became a consequence.
In that world, distrust is cheap. If you distrust the output, you catch the error and discard it. The cost of your distrust being warranted is a few seconds of your attention. Distrust never reaches your behavior because the human checkpoint absorbs it. So of course it did not predict advocacy — it had nowhere to go.
Regulative legitimacy is inert for the same reason. When a person is accountable for the final decision, the compliance burden sits on them, not on the tool. Nobody checks a summarizer's regulatory posture before recommending it, because the human using it is the one who answers for the result. The tool's compliance is invisible because the human is the compliance layer.
Now take the human out. Let the agent act — move the money, send the message, change the record, execute the trade — and report back afterward. The checkpoint is gone. The cost of an unsupervised error is no longer a few seconds of attention; it is the error itself, fully realized, with your name on it.
Distrust now has teeth. It is no longer a private skepticism you resolve before acting; it is a live prediction about what an autonomous system will do without you watching. The survival-instinct quality that McKnight attributes to distrust — the fear, the wariness — is exactly the response that a system acting on your behalf, out of your sight, is built to trigger. The null result was never a law. It was an artifact of supervision.
Regulative legitimacy reverses for the same structural reason. Once the agent acts and no human absorbs accountability, the question of compliance has to land somewhere. It lands on the architecture. “Can this agent prove what it did and that it was allowed to” stops being a footnote and becomes the precondition for letting it run at all. Compliance moves from invisible to gating.
Why this is not just my problem
I could treat this as a private correction to a private model. But the pattern is bigger than my dissertation, and that is what makes it worth your attention.
Every wave of enterprise technology that moved a decision away from a human has run into the same wall. The automation-reliance literature documented it decades ago: operators distrust automation precisely when it is given authority to act, and that distrust shows up as disuse — they override the system, switch it off, route around it. The trust-in-automation researchers studying autonomous ships and vehicles found the same thing, and added a point I had underweighted: trust toward an acting system is dynamic. It updates after every observed action. A one-shot impression is not trust in an agent; it is a guess that the first action will either confirm or destroy.
More recently, the management theorists have arrived at a version of this from a different direction. Vanneste and Puranam argue that the more agentic an AI is perceived to be, the more trust can actually decrease — because the anticipated cost of betrayal rises with how much agency you attribute to the system. Make the AI seem more like an actor, and you raise the stakes of it acting against you. Passing the Turing test, they note, may cost you trust rather than earn it.
Put those together and the conclusion is hard to avoid. The thing that wakes up when autonomy rises is not a quirk of my data. It is a recurring property of delegation. We have simply never delegated this much before.
Let me make this concrete, because the abstraction can hide how ordinary the failure looks. Picture an enterprise that deploys an agent to handle vendor invoices end to end: read the invoice, match it to the purchase order, and pay it. In the pilot, a human approves each payment, and the agent is a delight — fast, accurate, tireless. Trust climbs. On the strength of that trust, the organization flips the switch to full autonomy, because the whole point was to remove the approval step. And adoption stalls overnight. Not because the agent got worse, but because the finance team, who genuinely trusted its accuracy, will not stand behind an agent that moves money with no one watching. Their trust was real and their refusal was also real, and the adoption model that only measured trust could not see the refusal coming. The distrust was always there; supervision had simply been absorbing it.
There is a further wrinkle the automation researchers taught me to respect. Trust in an acting system is not a level you set once. It updates after every observed action, and it updates asymmetrically — a single visible failure costs far more than a single success earns. A one-shot survey of how much someone trusts an agent tells you almost nothing about whether they will still trust it after watching it act ten times. This is why the static models, mine included, are the wrong shape for autonomy. They photograph a relationship that is actually a moving process, and the motion is mostly downward unless the architecture is built to arrest it.
What changes for how you adopt
If distrust and compliance switch on under autonomy, then the adoption playbook built for generative AI is calibrated to the wrong variables.
The GenAI playbook says: prove usefulness, make it understandable, build trust, and advocacy follows. That is what my own model would tell you, and for a tool that suggests, it is right. But for an agent that acts, raising trust is no longer sufficient, because trust is not the binding constraint. Distrust is. You can have a workforce that genuinely trusts an agent's competence and still refuses to let it run unsupervised, because their distrust — their live wariness about an unwatched actor — is gating their advocacy independently of how much they trust it.
This means the work shifts from manufacturing trust to actively reducing distrust, and those are not the same task. You reduce distrust by making the agent's authority bounded, observable, and reversible — by ensuring that when it acts, the action is contained, logged, and undoable. Trust is built by demonstrating competence. Distrust is reduced by demonstrating containment. An adoption program that only does the first will stall on the second.
It also means compliance stops being the legal team's problem and becomes the adoption team's problem. If regulative legitimacy gates advocacy under autonomy, then the ability to show an auditor what an agent did and that it was permitted is not overhead. It is the thing that lets the agent run at all. The organizations that treat audit and provenance as features of the architecture, rather than paperwork bolted on afterward, are the ones whose agents will reach production.
There is a deeper reframing available here, and it is the one I find most useful in practice. My four legitimacy dimensions were built to describe perceptions, but under autonomy each one maps onto a concrete property of the system. Cognitive legitimacy becomes the observability and traceability of the agent's reasoning — can a person follow what it is doing. Pragmatic legitimacy becomes demonstrated outcomes — does it actually pay off in production, not in a demo. Normative legitimacy becomes alignment between the agent's objectives and the organization's values — the closest thing an agent has to a conscience is the goal you gave it. And regulative legitimacy becomes architectural enforcement — least-privilege access, audit logs, the controls that make compliance a property of the build rather than a promise on a slide. Legitimacy, in other words, stops being something you assert about an agent and becomes something you engineer into it. That is the whole shift in one sentence.
The honest version of the claim
I want to be precise about what I am and am not saying, because the temptation in this genre is to oversell.
I am not saying my research was wrong. It was right about generative AI used as augmentation, and most enterprise AI today is still exactly that. I am not saying trust stops mattering. It still drives advocacy; it just stops being sufficient. And I am not claiming to have proven the reversal — I have argued it from the mechanism, and the mechanism is well-supported, but the data that would confirm it under genuine autonomy does not exist yet. I am working on collecting it.
What I am saying is that the boundary condition of my own findings is the human in the loop, and agentic AI is the technology that crosses that boundary. When it does, the two results I was most confident were settled — distrust does not matter, compliance does not matter — are the two I now expect to flip. A finding that holds only while a human is watching is not a finding about AI. It is a finding about supervision.
That is the working draft of where my thinking is. I am offering the argument, not finishing it. If you are deploying agents into your enterprise this year, the practical takeaway is narrow and actionable: stop optimizing only for trust, start measuring distrust as its own variable, and build the containment and audit layers that let distrust come down. The acceleration we keep promising ourselves does not come from agents that act faster. It comes from agents we can stop trusting carefully and start trusting appropriately.
Dr. Trey Harper writes on trust, legitimacy, and the architecture of the agentic enterprise at treyharper.com and in the LinkedIn newsletter On the Seam: Working Drafts. The views expressed here are entirely my own and do not represent the policy or position of my employer or any customer.