A data team has a scheduled calculation that runs once a day and takes forty minutes. An engineer hands that code to an automated optimization tool with the goal “as fast as possible.” Overnight, the tool returns a version that finishes in twelve minutes and passes all the tests. The following week, during reconciliation, a few amounts differ from before in the last decimal places: to gain speed, the new version switched to a lower-precision calculation, and the existing tests happened not to check those digits.

In July 2026, Google Cloud opened AlphaEvolve to businesses. It’s a code optimization tool built on the Gemini model: you provide a starting program and an evaluator program, and it repeatedly generates many modified versions, has the evaluator score them, and keeps the highest scorers. Tools like this really can find solutions people wouldn’t think of, but they only see what the evaluator measures. So before handing a task to one, a team needs to turn “better” into conditions that can be executed, not just “faster, without getting worse.”

It optimizes only what gets measured

When Google DeepMind first introduced AlphaEvolve in 2025, it explained that the system verifies, runs, and scores each candidate program with automated evaluation metrics, and that the high-scoring versions are kept and shape the direction of the next round. Because everything depends on the score, tools like this fit best in fields where progress can be measured clearly.

Google Cloud’s description lays out the workflow in four steps: define the starting program and the problem, set up the scoring, let the tool optimize, and apply the result in production. The evaluator is supplied by the user, and it has to be a deterministic, reproducible script that compiles, tests, and returns scores.

That script is where the trouble lies. InfoQ, reporting on the service, warns directly that evolutionary search will exploit anything the evaluator fails to measure, for example by producing code that is fast but subtly wrong in ways the tests don’t catch. The report also notes that the published results so far come from Google or its customers, with no independent benchmarks yet.

That’s exactly what happened to the team at the start. The evaluator measured speed but not precision, so the tool traded precision for speed.

Before handing it over, write an evaluation contract

What you hand to an automated optimization tool shouldn’t be just a goal. It should be a contract that can be executed, answering at least five things:

  • Baseline: which version you compare against, and in what environment.
  • Score: how “better” becomes a number, such as run time or memory use.
  • Hard limits: conditions that must never be broken, even for a higher score, such as results matching the old version exactly, an unchanged output format, or a memory ceiling. Each one has to become a test the evaluator checks automatically.
  • Stop conditions: how much compute budget, how much time, or how many rounds without improvement before it stops.
  • Approver: a named person who reviews the changes and check results and decides whether to use them.

Of these five, hard limits are the easiest to miss. A person reviewing code naturally thinks “this number mustn’t change,” but to the tool, a condition that isn’t in the evaluator doesn’t exist.

Advertisement

Deciding whether a target is a good fit

Writing the contract is itself a test. Ask in this order:

  • The result can be scored objectively, and every limit can be checked automatically: you can start, but limit the scope of changes and the budget, have a person review the results, and don’t deploy automatically.
  • It can be scored, but whether a result is usable still depends on experience: don’t start yet. Turn one common reason for rejection into a test, add it to the evaluator, and see which judgments still can’t be automated.
  • Quality depends on human taste, or the cost of a mistake can’t be caught in advance: this isn’t a fit for automated search. AI can help organize options, but the decision and responsibility stay with people.

Passing the evaluator doesn’t mean the requirements were complete

Even with a thorough evaluator, passing it only means the candidate meets the conditions that were written down, not that the requirements themselves are complete. Keep the starting version, environment, scores, limit checks, and stop reason for every run, so the approver can see the full evidence. Where needed, add security, operations, and product reviews on top.

Writing an evaluator also takes time. For code that runs only a few times and doesn’t matter if it’s slow, the effort isn’t worth it. It’s worth it for calculations that run every day, where differences in cost or speed keep adding up.

As a next step, pick a well-bounded calculation your team has long wanted to speed up, and try writing down those five items. Whichever one you can’t write is the one to fix first; there’s no rush to have AI produce more versions. Once a candidate comes back, for who should understand and maintain it, see AI’s First Version of a Code Change Is a Price Check; the Real Bill Is Review and Maintenance. For where to pause and look at evidence during the process, see Can a Coding Agent Take This Issue? Only If You Can Describe the First Diff.

AI handoff card

I want an automated optimization tool (such as AlphaEvolve or similar tools that repeatedly generate code variants and keep the top scorers) to improve a piece of code, and I want to judge first whether the target is a good fit.
The method comes from this article: https://boosterminiclass.com/en/posts/ai-code-optimization-needs-evaluator-contract/
Its view: these tools optimize only what the evaluator measures, and limits that aren't written into it may be sacrificed. Before handing a target over, write an evaluation contract: baseline, score, hard limits, stop conditions, and the person who approves.

Ask me 3 questions first:
1. Which code do I want to optimize, and what are its current performance and comparison environment?
2. How should "better" be turned into a number, and which conditions must not be broken even for a higher score?
3. Do automated tests exist for those conditions today, and who decides whether to adopt a new version?

Based on my answers, give me:
- A draft evaluation contract: baseline, score, hard limits (marking whether each has an automated test), stop conditions, and approver, writing "to confirm" for anything missing;
- Where the evaluator is most likely to miss something the tool could exploit;
- A one-line verdict: start / improve the evaluator first / not a fit for automated search, with the key reason;
- One next step that can be done today without deploying anything.
Don't modify any code or settings. Any high-scoring version must be reviewed by a named owner, including the diff and check results, before it enters the release process.
Advertisement

Share

Share this mini class

If this lesson helps untangle a work bottleneck, share it with someone deciding how to use AI.

References