Train the Agent, then Replace It!

Or, "Stop paying engineer rates for a secretary!"

There’s a seductive lie that arrives the moment you deploy your first useful AI agent. It goes: I don’t have to worry about hygiene and chaos anymore. AI will just deal with it.

Messy inbox? The agent will sort it. Fragile process? The agent will absorb the chaos. Undefined rules? The agent will figure it out, every hour, forever. FOREVER.

And the thing is, the agent will. That’s what makes the lie so easy to believe. The agent really does deal with your growing chaos. It just does so at a price, and it keeps charging you that price on every single run, whether the work is worth the price or not.

This post is about the step that comes after “my agent works!!” It’s a step nobody talks about, because everyone stops at the triumphant blog about an agent doing the thing. But if you’re like me, running agents on a recurring schedule, that first successful run is where the cost problem starts, not where our heroic story ends.

Now, for the step nobody talks about

Most agentic-workflow content follows the same story arc. The hero identifies a task, deploys an agent, relishes the way it handles ambiguity…posts about it! (Guilty).

Done. That’s the end of the story. The hero retires.

But not so fast! There’s a second phase: the phase that separates a fun prototype from an operational system: once the agent has learned the rules and codified them, you ask the agent to help you rip those rules back out, and hardcode the predictable parts into infrastructure that runs for free.

The title of this post is the thesis. You train the agent. Then you replace it . Or, more precisely, you replace the predictable and codified 75% of it, and only keep the agent focused on the 25% that actually needs LLM-ish reasoning.

If you’ve ever used a compiler, you already understand the mechanism. When you write code in a flexible language, the computer can’t run it directly; something has to translate it into fast, rigid machine instructions. A good compiler doesn’t just translate blindly. It optimizes: if you write 2 + 3, it notices the answer is always 5 and just hardcodes 5, rather than making the machine add two numbers every time the program runs. It pulls predictable stuff out of the expensive live path and resolves it once, ahead of time.

That’s the metaphor here. Imagine creating an “Inbox Triage Agent” for the sake of this post. Your agent, on every run, is “interpreting” some rules that aren’t really judgment calls.

If the sender is no-reply@tableau.com, label it “@Reference”

That’s not reasoning. That’s a lookup table or filter logic, but you’re paying LLM prices to run it. Step 2, the optimization step, is spotting those and compiling them down into free, deterministic infrastructure.

So, Step 1 is train the agent to discover the rules. Then, replace the parts that were never worth an LLM in the first place.

Let me show you what that looked like in practice.

The case study: my beloved inbox

I use Glean (an AI-powered enterprise search and workplace assistant) to build and deploy custom agents that run workflows for me. On my personal AI maturity ladder, this sits squarely in the productivity tier: core efficiency tasks and automations, not role-based deep thinking. (Glean doesn’t have a project-based system yet, so I don’t reach for it when I need sustained, stateful reasoning. Gotta use the right tool for the right tier.)

The problem I pointed it at was the oldest one in knowledge work: my inbox. Hundreds of emails every day, a list of manual Gmail filters I’d built and left for dead over years that suffered from the 3 “F”s: fragile, fragmented, and failing. Most of the volume broke through them and landed in my inbox anyway. And I do not have the attention span to self-categorize hundreds of emails. I never will. That’s not a moral failing; it’s just a principle I live by, and any system that depends on me changing it is a system that will fail.

So I built an agent, as one does in 2026. Here’s the framework that emerged from my journey: a three-phase process I’m officially calling agentic refactoring. Not to be confused with agents who refactor code, by the way.

Phase 1 — Prototyping & Rule Discovery (le sandbox)

The instinct, if you’re obsessive (which I’m absolutely not), is to sit down and try to anticipate every kind of email you receive and write filters for all of them up front. Don’t. You’ll spend hours writing complex rules for a reality you don’t actually understand yet, and you’ll be wrong about the edge cases because you haven’t seen them.

Instead, I deployed a flexible, high-reasoning Glean agent and let it loose. In order to deploy the Glean agent, I’ll be honest with you: I gave Claude access to my gmail and asked it to find and categorize my most common senders, and then I asked Claude to build me a Glean agent. That’s just laziness on my part. Claude and I have an understanding that we keep from Glean. That’s a different post altogether.

Anyway, with my instructions, I deployed a Glean agent and let it loose . This is the part LLMs are really good at: handling ambiguity when you don’t know what you don’t know. By letting the agent run every hour, and then checking the results, I turned it into a data-discovery tool. It observed my real-world inbox patterns. It hit edge cases I would never have predicted. And helpfully, it let me iterate on my business logic in plain English instead of in wild regex. I could go back to edit the agent and adjust instructions conversationally as I watched what ended up in my @Junk label.

Allie wisdom #1: An agent prototype’s job isn’t to be cheap. Its job is to reveal the structure in a mess that had no structure before.

Phase 2 — Auditing & Rule Hardening (le calibration)

I let the agent run for three weeks, every hour of every day (yes, weekends. Especially weekends. Get rid of the dross so I’m not dreading a Monday inbox). Every day, I made corrections. I tightened the instructions. And bit by bit, a reliable form solidified — seven triage buckets that actually match how I work.

I borrowed the backbone concept from David Allen’s Getting Things Done, which I will use forever:

• @Action

• @Reference

• @Waiting

I added the ones specific to efficiency in my role that are really subsets of “Reference”:

• @Performance for reporting,

• @Digest for industry news and research,

• @Competitor_Watch for anything about our competitors.

And a mandatory catch-all, @Junk, evaluated only after every other label had been considered.

The agent’s instructions settled into something like:

1. Fetch unlabeled inbox threads.

2. Classify each thread into one of the seven labels (@Junk last, as the catch-all).

3. Label and archive. @Action stays in the inbox, unread. Everything else gets labeled and archived. I can open any label folder to see what’s there, and delete anything older than three days so it doesn’t pile up.

4. Report a summary back to the Glean platform.

Here’s what I was actually doing during those three weeks, though: I wasn’t just triaging email. I was training an operational playbook for my specific situation, building a historical track record of reality the agent could use. When it finally hit 100% accuracy against my human feedback, I knew the logic was mature. And a mature agent can show you which of its rules don’t need to use an agent at all.

After three weeks, each category had split into two kinds of logic:

• Deterministic rules. If it’s from my manager, label @Action. If it’s from no-reply@tableau.com, label @Reference. Constant in, constant out.

• Inference rules. Is this a newsletter I care about, or a webinar pitch dressed as one? Requires actually reading and weighing intent.

Those deterministic rules become the foundation of a much cheaper operating system. Unless you want to pay more, of course.

Phase 3 — The Replacement (le refactor)

This is it! Eureka!

Using an LLM to check whether an email came from a static address, or contains a static phrase like “Booking.com,” is an expensive waste of compute. So I asked Glean to analyze Glean’s Inbox Triage Agent instructions, to identify every deterministic element and translate it into zero-cost, native Gmail infrastructure. (I simply said, pull out anything I can turn into a Gmail filter, that’s all)

And with such simplicity, the agent helped compile itself. It knew which rules were pure lookups. It gave me a list of exactly which filters to create and which checkboxes to check. I did it in less than 2 minutes.

So I offloaded all the low-level deterministic logic to Gmail’s native filters, which run on Google’s infrastructure, for free, the millisecond an email arrives. Then I updated the Inbox Triage Agent’s instructions to remove those deterministic rules so that the prompt is much shorter. And now I’m keeping the Glean agent exclusively for high-cognitive work.

The result: roughly 75% reduction in AI compute cost, with no loss in quality, because the 75% I removed wasn’t doing anything particularly “intelligent” in the first place.

What compiles, and what doesn’t (le reality)

Allie wisdom #2: the compilable layer is anything a Gmail filter could express as a boolean. The agentic layer is anything that requires reading intent.

Here’s where each rule actually landed.

Compiled down to Gmail (free, instant, deterministic):

• Sender-based labels (from my manager → @Action; from a known no-reply address → @Reference or @Performance, depending on the domain).

• Static keyword matches (a vendor name, a known sender domain).

• The pre-existing Google Alerts filter that already routed those into a folder before the agent ever ran (which is how I knew the agent was operating downstream of Gmail, on a pre-filtered stream).

Stayed in Glean (trully requires reasoning):

• The @Junk catch-all. Gmail filters can’t execute steps like “if nothing else matched, apply X.” There’s no fallback logic filteres. A multi-step agent has to handle the final sweep.

• Context and relevance gates. Gmail cannot tell the difference between a research newsletter from Skift (@Digest) and a webinar sales pitch from Skift (@Junk —sorry, Rafat Ali :/). Same sender, opposite intent. That’s an LLM’s job.

• Self-sent agentic briefings. Gmail can match from:me to:me, but it can’t tell whether there’s a hidden human in the CC block, or whether the body is an agentic summary I sent to myself versus a real email where I happen to be on the To: line. Glean gets it and routes it to the right place for me.

• Vague sales pitches and cold outreach. Gmail can’t reason about whether a stranger acting like they know you is a real contact or a cold pitch. Glean can.

On that note, to anyone on LinkedIn who’s been cold-emailing me and wondering why you’re not getting a reply: it’s because an LLM is moving you to @Junk before I ever see you. Sincerely sorry about that.

This is really about le tokens

The savings aren’t abstract, and they’re not mostly created in the final categorization step. When an agent triages an inbox, its heaviest cost is the input context tonnage. To evaluate my inbox, Glean has to ingest the metadata (and often the snippet or body) of every unlabeled email, every run. That is where the money goes.

• Before: 50 unlabeled emails means Glean parses all 50.

• After: Gmail’s native filters run for free the instant each email arrives, wiping out the high-volume, predictable categories before the agent ever spins up. By the time Glean triggers its batch run, the obvious stuff is already gone.

• Result: Glean finds a handful of residual emails that actually require reasoning, with ambiguous text, cold pitches, etc. It thinks hard about a few ambiguous things instead of parsing through everything.

There’s a second-order effect too. Glean, like most agent platforms, scales credit consumption with logic complexity. So a giant prompt stuffed with hardcoded rules forces the model to chew through a huge instruction set on every run. Moving those rules to Gmail leaves a prompt that’s small and simple, which keeps agent runs in the low-cost bracket. I’m not just processing fewer emails; I’m also running a leaner agent on each run.

The real deliverable

It would be easy to read this as a cost-optimization story. It is (yay!). But the cost savings are almost a byproduct of the principle here.

The ultimate goal of AI automation should never be 100% AI. It’s a hybrid architecture: some hard-coded or native automation as the broad net that catches the bulk volume, and an LLM agent as the precision targeter that handles the complex remainder. The ROI of AI has as much to do with choosing which cases deserve it as with the intelligence itself.

Allie wisdom #3: An agent pointed at everything is an agent wasted on most things.

So here’s the things we should talk about more: the act of separating the codifiable logic from the agentic logic is itself the work. When you force yourself to ask “which of these rules is a lookup, and which requires judgment?”, you’re not just saving money, you’re auditing your own process and discovering what actually requires intelligence. That’s the human-in-the-loop moment. It’s how you catch the agent making decisions you can’t articulate, and it’s how you make sure the reasoning you’re paying for is reasoning you actually endorse.

Hygiene, it turns out, is much more than an aesthetic preference for a tidy system (who cares about that anyway?). Hygiene in the age of agentic AI is a cost control. A validation mechanism. And a diagnostic that tells you what your workflow is really made of.

So stop and clap when you get that agent working, yes of course do that. But don’t stop there. That’s just phase one. Phase two is figuring out how little of it needs to be an agent at all.

Train the agent, then replace it.

And let me know what you’re compiling.

Note: I’m traveling to Montreal in a few weeks, so I’m starting to add “le” to things where it absolutely doesn’t belong. It’s nice.