AI & agents
Evalon
An AI hackathon judge that shows its work — tools measure, agents explain, every score links to evidence.
AGENTS
3+1
MODELS
2
CODE EXECUTED
none
The problem
I have been judged by people with forty submissions and four hours, and I have watched what that does. Late repos get less attention than early ones. A polished README beats a better system. And when you disagree with a score, there is nothing to disagree with — a number arrived, and the reasoning behind it left the building with the judge.
The obvious fix is to point an LLM at it. Almost every "AI judge" does the same thing: paste the repo into a model, ask for a number out of ten. That number is worse than a tired human's, because it is unfalsifiable. The model can't show its work either, and now nobody can even ask.
What I built
An evaluation pipeline that inverts the order. Tools measure first. The AI only explains what the tools measured.
Submit a GitHub URL. Evalon clones it, runs a static analysis pass — cyclomatic complexity, security findings, documentation coverage, test presence, dependency shape — and only then runs the agents, each grounded in those measurements. What comes out is a scorecard where every criterion carries a "why this score?" control that opens onto the specific evidence that produced it.
Click any criterion and the evidence behind it opens. A score you can argue with is the entire point.
How it works
Four agents, one at a time
Three LLM agents — Repository Understanding, Code Quality, Innovation — plus a Comparative agent that is pure analytics and touches no model at all. They run strictly sequentially, never in parallel.
That is not a simplification. It is the load-bearing constraint of the whole design, and everything else follows from it.
The constraint
Judging runs on one laptop. A MacBook Pro M4 with 24GB, sitting on the organiser's desk, evaluating thirty-plus submissions.
Parallel agents look obviously faster on a whiteboard. On that hardware they mean two models resident at once, memory pressure, swap, and an evaluation run that dies somewhere around submission nineteen. Sequential execution through a Redis-locked queue manager is slower per submission and finishes every submission, which is the only property that matters when the results have to exist before the closing ceremony.
Two models total, for the same reason: qwen2.5-coder:7b for inference,
nomic-embed-text for embeddings. Every additional model is another thing competing
for the same GPU.
Degradation instead of failure
If the model is unavailable — queue timeout, resource contention, Ollama restarting — the evaluation still completes from static analysis alone, clearly marked as degraded. Never a 500 a participant has to interpret. Never a submission stuck at 40%.
Every failure state has specific copy explaining what is missing and what the score still rests on. A partial result a participant can read beats a complete result that arrives after judging closed.
The rest of it
A live admin dashboard streamed over SSE — submission counts, score distribution, stack frequency, model queue depth — updating without a refresh. Side-by-side comparison of up to three submissions with shared weaknesses and unique strengths called out. A RAG mentor chatbot grounded in your evaluation, queue-aware so it yields to an active run rather than fighting it for the GPU. Server-side PDF export.
Decisions
Strictly sequential agent execution
Parallel agents behind a worker pool
Judging runs on one consumer GPU. Parallel is faster on paper and out-of-memories on the actual hardware. Sequential through a Redis-locked queue finishes all thirty submissions instead of finishing half of them quickly — and 'all of them, eventually' is the only outcome that helps an organiser at 11pm.
Static analysis first, LLM grounded in its output
Handing the raw repository to the model and asking for a score
A model asked to grade a repo produces a number with no provenance. Measuring first means every score points at a finding a participant can open and contest. The AI's job becomes explanation, which is what it is good at, rather than measurement, which it is not.
Never execute submitted code
Sandboxed execution for test running and dynamic checks
Running arbitrary code from thirty unknown submissions on the organiser's laptop is a genuinely bad idea, and no sandbox is worth trusting under hackathon time pressure. Static analysis loses some signal — I can't verify tests pass, only that they exist — and that trade is worth making explicit rather than quietly accepting the risk.
Degrade to static-analysis-only on model failure
Retrying, or failing the evaluation
An organiser at 11pm needs every submission to have a result. A degraded scorecard that says exactly what it is missing is usable; a spinner is not.
What I'd do differently
The scoring weights are mine, and I picked them in the small hours of a hackathon. They are defensible but they aren't calibrated against anything — I never ran the pipeline over a set of repos with known human scores to see whether Evalon agrees with good judges. Until that exists, the honest claim is that it is consistent and explainable, not that it is right, and those are very different things to promise an organiser.
The Innovation agent is also the weakest of the three. Novelty is exactly the judgement an LLM is worst at, because it has no view of what the other thirty-nine submissions did. Folding it into the Comparative agent — which does see the pool — would be a better shape than asking a model to have taste in isolation.