Job 05 · Not yet tested
Which framework should I build my agent on?
Libraries for building an agent. Every framework gets the same research bot: two tools, memory across a turn, and a retry when a tool fails once.
Builders choosing a framework, not a hosted chat product.
The standard test
Pack 0.1.0 · specified · Research bot
Same research bot with 2 tools, memory, retry
Setup
A short spec: answer a research question, call two specified tools, remember a fact from turn one on turn two, and retry a tool that fails once then succeeds. Where the framework lets us pick the model, every framework gets the same model, and we publish that model name.
Inputs
The spec, the two tool definitions, and the scripted failure on the first call of one tool.
Time limit
One wall-clock cap for every tool in the job. The number is published with the pack before the first run and does not move between tools.
What we record
- Spec points met
- Human fixes
- Time
- Cost
- Model
- Framework version
How the FitScore is built
FitScore = weighted average × 20, rounded to the nearest integer. The weighted average is the sum of (criterion score × weight) divided by 100.
- Output quality30
- Reliability15
- Cost and free tier15
- Speed10
- Setup friction10
- Agent, API, and MCP support10
- Privacy and data10
5 excellent · 4 good · 3 usable · 2 poor · 1 fails. Full method.
Candidates
Candidates for Agent frameworks. This order is the launch list, not a ranking. No FitScores are published.
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Version and plan recorded when a test is published
Questions this page answers
- Which framework should I build my agent on?
- FitForTool has not published a FitScore for this job. The launch candidates are LangGraph, OpenAI Agents SDK, Claude Agent SDK, CrewAI, Mastra, Pydantic AI, and Google ADK. Each one is listed as not yet tested. A ranking is published only after every tool in the comparison has had the same test pack run, and a person has reviewed the result. Until that page shows a FitScore, there is no winner.
- How will FitForTool test agent frameworks?
- Same research bot with 2 tools, memory, retry. A short spec: answer a research question, call two specified tools, remember a fact from turn one on turn two, and retry a tool that fails once then succeeds. Where the framework lets us pick the model, every framework gets the same model, and we publish that model name. We record spec points met, human fixes, time, cost, model, and framework version. Those notes are then scored from 1 to 5 on seven shared criteria. The pack has been specified and has not been run.
- Can a vendor pay to rank higher on this job?
- No. FitForTool does not sell position or score. A sponsorship, if we run one later, is labelled and sits outside the ranked list. An affiliate link is disclosed and does not change order. A vendor may ask for a retest, and we rerun the same pack. Sponsors do not see the result before it is public.
- What does "Not yet tested" mean?
- The tool is a candidate for this job. There is no published hands-on run, so there is no FitScore, no criterion score, and no rank. Research-only is a different label, used when we have read the docs and still have not run the pack. Neither label is a score.