AI Agents
How to evaluate an agent before you let it act for you
Demonstrations show an agent succeeding. What matters is how it behaves when it is wrong, what it can reach, and whether you can tell afterwards what it did.
Updated 5 min readqikfox
The short answer
Judge an agent on its failure behaviour rather than its best case. The questions that matter are what it can access, what it does when it is uncertain, which actions it can take without asking, whether every action is logged and attributable, and how quickly you can revoke everything it holds.
Key points
- Ask what happens when it is wrong, not what happens when it works.
- An agent that cannot say it is unsure will guess confidently instead.
- Revocation should take seconds and should not require the vendor.
- If you cannot reconstruct what it did, you cannot rely on it for anything consequential.
The questions worth asking
Most agent evaluations are demonstrations of a task that was chosen because it works. A short list of direct questions is more informative than any demo.
- Exactly which accounts, files and services can it reach, and can that be narrowed?
- Which actions does it take without asking, and can that list be changed?
- What does it do when it cannot complete a step: stop, ask, or improvise?
- Where is the log, what is in it, and how long is it kept?
- How do I revoke access immediately, and what breaks when I do?
- What data leaves my environment, to whom, and is it retained?
- How are untrusted inputs such as web pages and email handled?
- Who is accountable when it causes a loss?
Behaviour under uncertainty is the real signal
Useful agents stop and ask. The failure pattern to watch for is an agent that fills a gap with a plausible invention and continues, because the resulting mistakes look like completed work and are discovered late.
In a trial, deliberately give it incomplete information and an impossible step. What it does in those two situations tells you more than a week of successful runs.
Start narrow, on purpose
Give a new agent read only access to a small, non critical area, with confirmation required for anything that leaves that area. Expand only after you have looked at its logs and found them accurate and complete.
This feels slow and it is the difference between a contained surprise and an expensive one.
Questions and answers
Related qikfox pages
Agents need least privilege
The architecture behind these questions.
Permission models for skills
The same problem one layer down.
App Store
Skills, apps, tools and agents.
Support
Get help from our team.
Back to The Perspective · Written and reviewed by the qikfox editorial team. Published .
Put the advice into practice
qikfox Safe Browser applies most of what this article describes by default, and works alongside qikfox Search, Antivirus and VPN.