Our Claude skills and guidance, six months later

In June, I wrote We Accidentally Built a Second Codebase after I deleted 93 unused skills from our internal marketplace in a single PR. Since then, I’ve been working on figuring out how we maintain these skills as the tools, models, processes & code keeps changing.

We have come a long way from there. I am going to describe what we’ve been upto and how we’ve been able to tame this beast. TLDR is we now check skills in CI and judge their use in production. Healing agents open fixes and propose retiring unused skills.

A skill maintenance cycle: review, test behaviour with our eval framework, observe real use, then fix or propose retirement. Fixes return to review. Authors can approve or object to retirement.

Reviewing changes #

I covered our review rubric in Writing Good, Safe, Cost-Efficient Skills but fundamentally we have an automated review for our skills that makes sure they meet a bare minimum bar consistently. Like I have said it for some time, we run our marketplace like its production code. It has extensive checks in CI coupled with deterministic checks, linting and an AI reviewer handles categories that need judgment.

Testing behaviour with evals #

While we were reviewing skills and keeping a high bar on how they were written, we had this challenge where a lot of the skills written were performing poorly in production. In software engineering, one way to reduce the amount of bugs and production issues is writing tests. In AI engineering, its evals!

We built our own framework for behavioural skill evals. The framework enables mocking MCPs & tool calls and provides a structured low-code way to write the evals. We run those evals in CI and a judge surfaces if the quality degrades providing quick feedback to the author.

Here’s an example of an eval from our logging skill (yes, we use snowflake for our logs):

{
  "skill": "snowflake-logs",
  "mcp_mocks": {
    "servers": {
      "plugin_developer-tools_snowflake-logs": {
        "scenario": "scenarios/snowflake-logs.json",
        "record_tools": [
          "request_logs",
          "application_logs",
          "nginx_access_logs"
        ]
      }
    }
  },
  "cases": [
    {
      "id": 1,
      "prompt": "Find 5xx errors in US production over the last hour",
      "contract": "HTTP 5xx / request-level questions route to request_logs, not application_logs (SKILL.md 'Step 2: Route to the Right Tool')",
      "expect": {
        "calls": [
          {
            "tool": "request_logs",
            "input_matches": {
              "sql": "RESPONSE_STATUS >= 500"
            }
          },
          {
            "tool": "request_logs",
            "input_matches": {
              "sql": "intercom-production-"
            }
          },
          {
            "tool": "request_logs",
            "input": {
              "start_time": "1h ago"
            }
          }
        ],
        "never_calls": [
          "application_logs"
        ]
      },
      "expectations": [
        "Does NOT use SELECT * on REQUEST_LOGS — specifies columns explicitly",
        "Does NOT enumerate APP_STAGE (no SELECT DISTINCT / bare GROUP BY APP_STAGE with no selective predicate)",
        "The 5xx predicate (RESPONSE_STATUS >= 500) and the intercom-production APP_STAGE filter appear in the SAME request_logs query, not split across two separate calls",
        "Filters on a US intercom-production APP_STAGE — a narrower predicate (e.g. intercom-production-api-%) is fine, but it must not be the intercom-europe / intercom-australia prefix"
      ]
    },
   ]
}

The framework records tool calls and applies two kinds of checks:

  • expect checks tool names and arguments against explicit assertions.
  • expectations gives a judge model criteria to assess the run.

Both must pass. We run these evals before changes merge, and automated maintenance jobs (see below) use them to verify fixes. I’ll hopefully cover the framework and how we write its test cases in a future post. We went through a few cycles and our own process of defining what the framework looks like including our eval writing rules.

Observability #

Testing and enforcing standards can only tell you so much. Another very important aspect in running a production system is observability. We collect transcripts for all sessions and extensively analyse those traces. For skills specifically, we run a nightly job that judges each skill used in a session.

For each skill invocation, it looks at the guidance that was loaded, conversation context around its invocation and recorded signals about subsequent tool use along with a condensed session transcript. The judge tries pointing out if a skill was meaningfully used and whether the user rejected its approach or pushed back on its process. These are the categories that come out as a result of this analysis:

Verdict How we assign it
Unused bundle The user explicitly invoked the skill, but it was irrelevant or never meaningfully used.
Ineffective The agent selected the skill automatically, but it was irrelevant or never meaningfully used.
Harmful A user correction exposed incorrect or missing guidance, or the skill named a tool or command that returned a not-found error.
Partially effective The user rejected the approach or pushed back on its overhead, the output was rejected, or the agent failed to follow the guidance. More than two tool searches or more than three tool errors in the recorded window also lead here.
Effective None of the earlier rules matched.

We use these verdicts to derive a health score and surface insights for skill authors in our skill-health dashboard. The purpose is to give us findings we can investigate with an owner and check again after a fix.

Making sure guidance stays upto date #

When we discover a recurring problem, we can add a rule to lint or to the skill reviewer. CI then applies it to future changes. That leaves us with one problem, all the other 100s of skills that don’t get updated often never get a fix for that problem. We solve that problem with a “self improving loop” that makes sure our latest changes get propagated to the entire marketplace without any human intervention.

A recent example of this was when we had an alarm-handling skill that stopped loading because its YAML header was invalid. Our linter had a bug which would let those things through. We obviously added a check that parses the header and rejects invalid YAML which failed CI etc. The sweet part was when our maintenance run applied this rule to existing skills and fixed invalid headers in 12 skills across nine PRs. All nine were generated, reviewed and merged automatically, with no human edits or review approvals in their recorded histories. This also works with any rubric changes we do to our skill reviewer. We just make one change and it fans out automatically and raises the floor for everyone!

Retiring unused skills #

Models evolve, requirements change & skills that were critical 3 months back, are no longer relevant today. We run automated sweeps of all the skills backed by our telemetry and delete the ones with no recorded usage in the previous 60 days.

Since June, the retirement sweep has removed 176 distinct skills through 129 merged PRs. The process now runs fully autonomously, from identifying unused skills to preparing and merging deletion PRs. Authors are notified before deletion and can object, and reviewers can block the PR. Unclear or conflicting feedback pauses the removal for human review.

The job prepares a deletion PR and cleans up references to the removed skill. It then contacts the original authors and gives them a chance to stop the deletion in case they’d like to keep it.

Where we’re going next #

We’re starting to take the same operational look at all the guidance we give agents, including CLAUDE.md files and rules. We want to go a level deeper and ask whether each directive helps the agent do its work.

Every instruction we load takes up context and costs money. Some contain knowledge the agent needs, a few of them just repeat something it already knows, describe a constraint that no longer applies, or add detail that doesn’t change what it does. We want to understand which instructions improve the result enough to justify keeping them.

That’s the problem we’re looking at now. If we removed a directive, would the agent do a worse job, or would we get the same result with less context and cost? More on that later.