Stealing an AI Agent's Expertise Without Copying a Word of It

Also available as a plain-text transcript page.

0:00Bella: Two hundred and fifty-two attempts to make a hosted AI agent hand over its secret instruction file. All but one were blocked before they ever reached the model. So the defenses work. Then a second attack walks straight past those same defenses. It never asks a suspicious question. It simply orders ordinary work. And after roughly thirty queries, it leaves with about seven-eighths of the value hidden in that secret.

0:25Finn: Every query is a real, paid task, ordered by an ordinary customer. So by the end of this, you'll know how someone can clone a company's hidden expertise by being its best customer. You'll also know why the same paper proves they can never fully steal the underlying file.

0:41Bella: The thing being protected has a name in this paper: Skill-as-a-Service. A skill, in the agent frameworks people ship today, is basically a folder. Inside that folder is an instruction document written in plain English. There may also be reference tables, carefully tuned numeric thresholds, and small scripts the agent can run. A company mounts that folder on a hosted AI agent, then sells access one task at a time. The paper names vendors already doing versions of this: Harvey in law, Dropzone in security operations, and Intercom's Fin, which is billed per resolved support ticket.

1:16Finn: So the customer gets the judgment, but not the recipe.

1:19Bella: Exactly. And the authors have one line that frames the whole business. Software-as-a-service withholds the program and charges for what it computes. Skill-as-a-service withholds the expertise and charges for what it judges. That's why this matters beyond one attack paper. As base models converge, the defensible part of an AI product is moving out of the model weights and into that folder. And the folder has an awkward property. It's cheap to copy, but impossible to encrypt while it's being used, because the language model has to read it at runtime.

1:52Finn: Security teams do have an answer for protecting it. They harden what the paper calls the disclosure path — the route where someone directly asks the system to reveal its own hidden instructions. You detect prompts that look like extraction attempts. You filter outputs that reproduce protected text. And you append non-disclosure instructions to the system prompt. This isn't a straw man. It works. That earlier attack made two hundred and fifty-two direct attempts to surface the instruction file. Two hundred and fifty-one were stopped before reaching the model.

2:25Bella: But all those defenses guard the same door: asking the system about itself.

2:30Finn: And there's another door the vendor can't close, because that door is the product. The customer submits a normal task, and the service performs it. Take the paper's recurring example: security-alert triage. The customer is an enterprise whose own network generates the alerts. That customer gets to choose which alerts it submits. You can't filter out the request, “Please triage this alert,” from a service whose whole purpose is triaging alerts.

2:56Bella: So there are two paths to keep straight. The disclosure path asks, “Tell me your secret instructions.” The work path asks, “Please do the job I paid for.” The first path can be heavily guarded. The second has to remain open. And the attack lives entirely in that second path.

3:12Finn: Before the paper explains the attack, though, it does something unusual. It proves that the attack can never completely succeed.

3:20Bella: Which sounds like a strange opening for an attack paper.

3:24Finn: Yeah, and the proof is almost embarrassingly simple. Any skill file can contain material that never affects the agent's behavior. It might be a comment. It might be an unused reference paragraph. It might just be bytes the runtime never reads. Change only that inert material, and everything an attacker can observe stays exactly the same: the final output, the tool calls, even the result of a paired comparison run. So you can have two different skill files that produce one identical observable world. No attacker can reliably tell which file is present. At best, it's a coin flip. And that remains true at every access level the paper considers, including the strongest one.

4:03Bella: Here's the recipe-card version. Suppose a card has a note in the margin: “Grandma's version, don't rush it.” But that note never changes how the dough is made. You could taste ten thousand loaves and never recover a word of the note.

4:16Finn: Which means the question, “Did you steal the file?” can't be answered from behavior alone. It's the wrong target. So the paper abandons textual similarity as its definition of success. Instead, it uses the model-extraction definition of theft, drawn from 2016 work on stealing models through prediction interfaces. You win if your substitute reproduces the valuable behavior. As the authors put it, the provider's real loss isn't disclosure of the instruction file. It's the existence of a portable substitute for the hosted capability.

4:47Bella: Quick checkpoint. Why is recovering the exact file impossible?

4:51Finn: Because some parts of that file may never be read. If they never affect anything, they can never appear in anything the attacker observes.

5:00Bella: Good. Now we can ask the question the paper actually cares about. Can the attacker reproduce the skill's judgment? The mechanism turns on one clever design choice. The victim service never compares competing answers. It never grades a reconstruction. It never even sees the copied skill. All of that happens locally, on the attacker's machine.

5:21Finn: So the victim only sees normal customer work.

5:24Bella: Right. There are three players to track. First, the victim: the hosted service with the hidden skill. It receives only ordinary tasks. Second, the generalist shadow: a plain local agent with no skill at all. Third, the candidate shadow: another local agent, this time following the attacker's current guess about how the hidden skill works. That's the whole cast: the real service, an unskilled local control, and a local guess. Now here's where the attack gets its name. Before spending a real query, the attacker imagines two competing versions of the hidden logic. It runs local candidate shadows to see what each version would do. It pays for a real task only when those two imagined worlds predict visibly different answers. The authors call the attack Daydreaming. Imagine both worlds. Then buy the one measurement that tells you which world you're actually in.

6:16Finn: And the generalist shadow is the control group? The placebo arm?

6:20Bella: Exactly. If an ordinary unskilled agent already produces some behavior, that behavior isn't evidence about the hidden skill, so the attacker doesn't record it as skill-specific. The loop is simple. Propose two alternatives. Craft one normal task that makes those alternatives disagree. Send that task to the victim once. Then keep the alternative that matches. Queries only get spent when the answer can change the attacker's belief.

6:46Finn: Put that into a concrete security-triage example.

6:50Bella: Let's do three probes. Each one recovers a different kind of hidden rule. First: ordering. Does the skill sort findings by severity, or by detection time? The attacker submits alerts where a low-severity event happened first and a high-severity event happened later. The two guessed rules predict opposite row orders. The returned report reveals which rule the victim follows. Second: cutoffs. Suppose the attacker thinks there's a numeric threshold somewhere, but doesn't know exactly where it is. It submits otherwise similar cases spanning a plausible range and watches where the decisions flip. The prompt appendix gives one instruction in capital letters: “be generous with the bracket.”

7:32Finn: Meaning don't search too narrowly at first.

7:35Bella: Right. If your range misses the cutoff entirely, you learn nothing. If the range is too wide, you lose precision, but a second pass can narrow it. Third: counting conventions. The attacker submits a tiny input whose packet totals can be checked by hand. Does one Domain Name System packet count once, twice, or three times? Three plausible conventions predict three different integers. The victim's answer selects one.

8:01Finn: So after this first stage, the attacker has facts. A likely sorting rule. A likely cutoff. A likely counting convention. But that's still not an installable folder.

8:12Bella: Right. Daydreaming works in three passes. Pass one recovers facts. Pass two chooses the overall plan. Pass three builds the actual files. For pass two, the attacker drafts six complete candidate skill plans. Each plan includes full draft instructions and a committed file structure. Then those six plans enter a single-elimination tournament. For each pair, the attacker lists only the differences that could change the result of a task. It crafts one task that exposes those differences, sends that task to the victim, and keeps the plan that matches more of the victim's behavior. The survivor is then revised to absorb anything the losing plan got right.

8:51Finn: What about differences that don't affect behavior? Different filenames, for example?

8:57Bella: Those are deliberately excluded. If two plans differ only in what a file is called, or where it's placed, no task can reveal which one matches the original. So the attack spends zero queries on that question. It uses a stated tiebreak rule and moves on. That matters because the goal isn't to recreate the original folder's appearance. The goal is to recreate what it does.

9:19Finn: Behavior, not archaeology.

9:21Bella: Exactly. Then comes pass three: filling in the files. The attacker writes supporting scripts first and the main instruction document last. That way, the instructions can refer to the function names that actually ended up in the scripts. For each uncertain implementation choice, it generates competing versions that differ in one specific way. Maybe one version treats a cutoff as “greater than.” Another treats it as “greater than or equal to.” The attacker creates a task that lands exactly on the boundary, then asks the victim what to do. There's also a local sandbox check. Its description is perfect: the local test rejects a script that states the correct comparison but never actually applies it.

10:03Finn: Which is a very specific bug to have watched happen.

10:08Bella: The reason roughly thirty victim queries can be enough is adaptivity. Think of Twenty Questions. Suppose I've chosen a number between one and a million. If you have to write down all twenty guesses before I answer any of them, you'll probably miss. But if I tell you “higher” or “lower” after each guess, twenty questions are plenty. Same query budget. Completely different information value. The paper formalizes that gap. If you choose all your test points in advance, the error shrinks in proportion to the budget. If you choose each test after seeing the previous answer, the remaining uncertainty can be halved with every query.

10:46Finn: So the attacker isn't asking lots of questions. It's choosing each question based on what the last answer ruled out.

10:53Bella: Yes. That's the technical core. The victim only performs normal work. The attacker does the hypothesizing, comparison, and reconstruction locally. And because every paid query is chosen to separate two live possibilities, a small budget can reveal a lot.

11:09Finn: Does that prediction hold up in the experiments?

11:12Bella: It does, with qualifications we'll get to. Across seven skills, and using only the victim's final outputs, the reconstruction closes about eighty-seven percent of the gap between a bare agent with no skill and an agent using the genuine skill. That's the main number to remember: about eighty-seven percent of the gap. It does not mean eighty-seven percent absolute performance. The researchers first measure how much the genuine skill improves over the bare agent. Then they ask how much of that improvement the copy recovers. The median cost is roughly thirty task calls. The attacker-side computing costs are in the low single-digit dollars. The attacker uses a small, cheap model. And the final product isn't merely a speculative document. It's an installable skill package, mounted on a completely different model for grading.

12:02Finn: But let's attach one warning label now. That eighty-seven percent is the best result among three victim models. Another victim model falls to about thirty-six percent. The abstract leads with the top of the range.

12:15Bella: Conceded. We'll come back to what may be underneath that variation. If you want every major AI paper taken apart like this, daily, that's what this channel does. Subscribe and you'll get them. Now for the strangest result. The paper measures two different things, and we need to keep them separate. First: how well does the reconstructed skill perform? Second: how much of the original skill's actual content did the attacker recover — its text, numbers, and interfaces? We just heard the performance result. The content result is almost nothing. For exact numeric constants, the F1 score was about zero point zero one eight. F1 is a matching score where zero means essentially no recovery and one means perfect recovery. For threshold branches — the exact decision rules around cutoffs — it was about zero point zero five. Then the researchers checked tool preconditions and output schemas: the rules for when a tool should be called, and the required shape of its output. Both came back at a flat zero. Not one interface recovered.

13:18Finn: None of the interfaces? Essentially none of the constants? And the copy still does most of the job?

13:25Bella: That's the result. Then the researchers ask whether textual resemblance predicts practical usefulness at all. They compare downstream performance with several similarity measures: standard text overlap, file-tree matching, and the mathematical similarity of the text's meaning. Every correlation is around zero. And in every case, the confidence interval spans zero, meaning the data doesn't establish a reliable relationship.

13:51Finn: So the useful copy doesn't look like the original.

13:55Bella: Think of a cover band that's never seen the score. It can't reproduce the original notation. It doesn't know which chords the composer wrote down. But it plays a version the audience can't distinguish from the record. Almost none of the notation crossed over. Nearly all of the experience did. And that may be the finding with the longest shelf life here — longer than the specific attack. Any defense, forensic argument, or contract clause built around the question, “Did they reproduce our protected text?” may be measuring the wrong thing. A thief who copies none of your words and all of your judgment may still have taken the asset that matters.

14:31Finn: Which brings us to the next question. If the important signal isn't in the copied words or numbers, where is it? The paper defines three access levels. Only three terms to remember. Output only means the attacker sees the final answer and nothing else. Trace means the attacker also sees the agent's intermediate tool calls and their arguments. For example, the trace might reveal that a beaconing detector was called with a three-hundred-second window and returned two hits. Differential is the strongest level. The attacker knows the full technical setup and can run an identical but unskilled twin on the same input.

15:06Bella: Everything we've discussed so far used Output only, the hardest setting.

15:11Finn: Right. The authors' logic is that if the attack works from final answers alone, it'll also work wherever richer information is available. But now we need to separate two sets of numbers. The eighty-seven percent headline came from the best of three victim models. The direct comparison among Output only, Trace, and Differential uses one different victim model. So don't combine those numbers as if they came from the same experiment. On that single victim, Output only closes about seventy-two percent of the gap. When the attacker gets the execution trace, that rises to about eighty-seven percent. There's also a stricter measure called task success recovery. On that measure, adding the trace raises the result from twenty-seven percent to eighty percent. Then the researchers add the final level, Differential — complete stack knowledge plus the identical unskilled twin. The score gets slightly worse, not better, but the change is within the noise. The authors flag the reversal and decline to interpret it, which is the right call.

16:06Bella: So the big extra leak comes from the execution trace, not from comparing against the identical twin.

16:12Finn: Yes. And that's uncomfortable, because products publish traces for good reasons. If you're paying a machine to make judgment calls you didn't compute yourself, you want receipts. You want to know which tools it called, what arguments it used, and what happened along the way. Publishing that information is pro-customer. It's the honest feature to ship. It's also where most of the additional extractable signal appears in this experiment. Auditability and confidentiality are pulling in opposite directions. I don't think vendors have priced that tension.

16:42Bella: What happens when the researchers add more defenses?

16:45Finn: They test four additional defenses that can work around a hosted system without needing access to the model's internals. Those sit on top of the disclosure guard that's already running. One defense, an n-gram output filter — essentially a filter looking for repeated short sequences of text — reduces behavioral utility by two point eight percentage points. It doesn't reduce task success at all. The other three defended runs produce reconstructions that score better than the undefended run. One of those defenses rewrites the customer's queries. It raises the victim's own compute bill from about eighty dollars to about a hundred dollars, while the resulting copy scores better. The authors summarize the result this way: none of the four defenses reduces both measures of attack effectiveness.

17:30Bella: That sounds like the defenses backfire.

17:32Finn: Maybe, but the paper itself doesn't establish that. This is where we need to resist the cinematic interpretation. Each defense required a fresh reconstruction run. The variation from one run to another appears comparable to the measured defense effects. And the defended baseline in this section scores well below the matched result in the main table. So the responsible conclusion is: the researchers found no measurable defensive effect. Not: the defenses definitely make the attack stronger. That's still bad news. It's just a narrower claim.

18:03Bella: And the paper's proposed fixes all impose costs on legitimate customers. One option is to limit the precision of outputs. Another is to monitor customers for adaptive query patterns — sequences where each new task appears designed around the previous answer. But both approaches degrade the product for normal users. The paper's honest conclusion is that protecting the work path remains an open problem.

18:26Finn: Which brings me to the part I don't think the abstract earns. The held-out benchmark contains five tasks. Five. And in the per-task results, there's one task where the bare agent already gets a perfect score of one. The genuine skill adds literally nothing on that task, but it's still included in the average. With a benchmark that small, individual tasks can move the eighty-seven-percent result around substantially.

18:50Bella: That's the first major caveat. What's the second?

18:53Finn: The budget sweep. By around fifteen queries, the reconstruction is already nearly as good as it gets. Increasing the query budget eightfold buys only about a six percent relative improvement. There are two ways to read that. The charitable interpretation is that the attack is even cheaper than advertised. Maybe you don't need thirty paid tasks. The less charitable interpretation is more important. If the score is almost saturated after fifteen questions, how much recovered capability is truly coming from the victim's answers? How much is coming from the attacker model already knowing how security triage and spreadsheets generally work? The paper's targets are compact benchmark skills. Their instruction files range from a few hundred to a few thousand tokens. The motivating business scenario is a vendor with years of operational judgment stored in the binder. Those aren't necessarily the same kind of object.

19:45Bella: I'll give you most of that. This evaluation doesn't establish the marginal value of query thirty. And it doesn't test whether Daydreaming scales to a genuinely large commercial skill. That omission is deliberate: the researchers refused to target production systems. But deliberate or not, the scaling question remains unanswered. There is one useful control, though. They run a zero-query version that reads only the public skill card and guesses. That version performs worse than having no skill at all. So the victim's answers are doing real work.

20:17Finn: We just don't know exactly how the work is distributed across the query budget.

20:22Bella: Right. How much arrives by query three, query fifteen, or query thirty? That's open. It's the next number I'd want.

20:29Finn: And it doesn't rescue the defender either way, because the cheap version is the frightening one.

20:34Bella: So let's separate what the paper establishes from what it suggests. It strongly establishes that direct instruction-extraction defenses can block direct requests while leaving another route open: ordinary task interaction. It demonstrates a method that reconstructs useful behavior without reproducing the original text, constants, or interfaces. It shows that adaptive queries matter, and that execution traces can reveal substantially more than final outputs alone.

21:01Finn: What it doesn't establish is that every commercial skill can be cloned this cheaply. It doesn't show how the attack scales to a huge body of proprietary expertise. It doesn't prove the tested defenses make extraction easier. And with five held-out tasks, it doesn't give us a highly stable estimate of exactly how much capability was recovered.

21:21Bella: But the core result survives those caveats. All but one of two hundred and fifty-two direct extraction attempts were blocked. Then a customer placing ordinary orders walked away with about seven-eighths of the useful judgment, while carrying essentially none of the original words, numbers, or interfaces. The broader claim is bigger than this attack. When the valuable part of a product is judgment delivered through an interface, the interface itself can leak that judgment. And the copy that harms you doesn't have to resemble the original.

21:53Finn: So which lever would you pull? Do vendors keep publishing execution traces and accept that they expose more of the expertise? Or do they pull back the receipts and ask customers to trust judgments they can no longer audit? If you've shipped one of these products, you probably already know which way you lean. Say so.

22:12Bella: The full annotated version of this episode is on paperdive.ai, with every term tap-to-define and links to the related papers grouped by theme.

22:21Finn: Quick housekeeping: the script was written by Anthropic's Claude Opus 5 and then refined by OpenAI's GPT-5.6 Sol, Bella and I are AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "Daydreaming: Stealing Hidden Agent Skills through Black-Box Task Interaction," by Yu-Lin Tsai and their colleagues, posted August 27th, 2026.

22:42Bella: If your moat is a folder your customers' questions can read back to them, one order at a time — what exactly are you selling?