Every number on this page has a source; click a mark to see the paper, table or record behind it.
Click or tap the portrait for another one.
AI portraits of me, made with ChatGPT Pro's Images 2.6 model from my photos, a new seed each time. The noise is an animation; no model runs here.
~/mahan · zsh
$ evaluate mahan --honestly✓ 5 peer reviewed papers ICSE · ISSTA · MSR · TOSEM✓ ACM SIGSOFT Distinguished Paper ICSE 2025✓ 6 years in industry cofounded a B2B booking platform✓ M.S. Software Engineering UC Irvine, Dec 2026✓ JobAgent LLM agents, calibrated against hand labels5 passed, 1 open issue! needs a research engineering role, from early 2027$
Research · five papers, first author first
Experiments you can run
Each bench is a small version of one study: you do what the study did, and the readout gives the real result from the paper. Each one opens with a click.
An invented system prompt that grows commit by commit, the way the 1,262 real prompt changes we studied did.
Press ▶ Play its history, or drag the History slider.
Watch new lines turn green, and lines that clash turn red.
Check the commit log: does each message say what changed?
Watch: the red notes under the prompt name each contradiction, and the line under the slider says what each commit did.
MSR 2025 · first author
Prompting in the Wild: the prompt that argues with itself
In real apps, the prompt is code. Nobody reviews it like code.
I built a static analyzer on Tree-sitter that pulls prompts out of Python projects, then followed 1,262 prompt changes across 243 repositories. Scrub through one prompt's history below: every commit adds, and some commits quietly contradict what came before.
system_prompt.txtversion 8 of 8
Press ▶ Play its history to watch this prompt grow, one commit at a time.
Commit logsays what changed?
This prompt, now
8 lines
3 contradictions
Commits that explain the change
3 of 8
in this illustration
What the paper found
21.9%
of prompt changes were documented in their commit messages. The study followed 1,262 prompt changes in 243 repositories. Prompts mostly grow, and changes can bring in logical contradictions no test catches.
Seven of Ma11y’s real mutation operators, applied to a tiny hotel booking page.
Press a defect under Plant a defect, starting with the first one.
Look at the page: most defects change nothing you can see.
Read Who notices and What a screen reader says to find out who it shuts out.
Click into the page and press Tab to walk it the way a keyboard user does.
How it’s built: the diagram above the experiment plays one planted defect through Ma11y. Point at a box to read what it does, or switch One tool and All six together to see the paper’s numbers.
Watch:Page source shows the exact edit as a diff, the way the thesis figures draw it.
ISSTA 2024 · first author
Ma11y: break the page. It still looks fine.
Accessibility checkers promise to find what's broken. Ma11y tests the checkers by breaking pages on purpose.
Each of its 25 mutation operators reproduces a failure technique from WCAG 2.1: one small edit to a live page that shuts someone out. Seven of the real operators are below. Plant them, then watch three things change: the code, the screen, and the people using it.
How it's builtredrawn from the paper's Figure 1
pagemutantcaughtmissed, or a planted defect
Redrawn from the paper's Figure 1. The story is its Table 4 example, and the background dots are not individual results.What each part does
Click into the page and press Tab to walk it with a keyboard.
Who notices
Nothing planted yet
Every visitor can book a room.
On screenThe page as its designer meant it.
Page sourceoriginal
Ma11y makes edits like these to the live page in a real browser, through Puppeteer, then runs every checker on the original and on the mutant.
What a screen reader says
Plant a defect, then look at the page: did anything change?
Defects planted
0 of 7
Ma11y makes at most one mutant of a page per operator. Here you can stack them.
Visible on screen
0 of 0
Plant a defect. Most of them change nothing you can see.
What the paper found
3 in 4
of the 366 defects Ma11y planted in real websites slipped past the average checker: 274 of them. The six checkers were A11yLite, Access Continuum, axe, IBM Equal Access, QualWeb and WAVE.
This bench shows an LLM pipeline turning what a tester said during a session into a structured bug report.
▶ Replay the session: plays the transcript from the start, one line a second.
Step: moves forward one line at a time.
Issue 1 (or any card): highlights the transcript lines that card came from.
How it’s built: the diagram above the experiment plays one session through the real pipeline. Point at a box to read what it does, or switch All and Common to see the paper’s precision and recall.
Watch: the Issue report on the right. A card drops in each time the tester runs into a problem. The real pipeline reads the whole transcript at once; the replay slows it down.
Reca11: from a test session to a bug report
A tester talks through an app while using it. Somebody then has to turn that recording into a bug report. Reca11 is the somebody.
It is an LLM pipeline (GPT-4) that reads the auto generated transcript of a recorded user testing session and writes a structured report: each issue with a title, a description and its timestamps. Play the session below and watch the report fill in.
How it's builtredrawn from the paper's Figures 3 and 4
inputsmain pathreportby hand, or left out
Redrawn from the paper's Figures 3 and 4. The issue in the story comes from its Figure 2, and the cluster plot is a sketch, not the paper's data.What each part does
Session transcriptwritten for this page, not a real participant
Issue reportselect a card to see its source lines
This session
5 cards
Four are real issues. One is a dog. That is what precision measures.
Against the human annotators
313 found, 6 missed. One square per issue. The six are parked at the end; the paper does not say where they sit.
What the paper found
6 of 319
issues missed (under 2%), at 85% precision, on 36 real user testing sessions across three applications.
This bench shows how a detector reads code as a tree of grammar pieces, then races the detectors on their real scores.
▶ Walk the tree: visits every piece of the example function, top to bottom.
Rename everything: changes every name in the code and leaves the tree's shape alone.
▶ Run the race: runs a coin flip, the existing detectors and the paper's model to their real F1 scores.
How it’s built: the diagram above the experiment plays one piece of code through the best detector. Point at a box to read what it does.
Watch: the highlighted code and tree rows move together, and in the race the amber bar stops far past the others.
Who wrote this code? The detectors don't know either.
Six detectors promise to spot AI writing. On code, the best of them average an F1 of about 45.
We tested six existing detectors, then tried our own: fine tuned GPT-3.5, few shot prompting with retrieved examples, and classifiers on CodeT5+ embeddings. The best one embedded only the syntax tree. I built the detection experiments and the embedding detector, including the tree walk. Walk one below, then run the race.
How it's builtredrawn from the paper's Figure 2
codetree alone, the winnerprediction
Redrawn from the paper's Figure 2. The tree alone reading reached 82.55 F1, against about 45 for the best existing detectors.What each part does
A functionexample code, not from the datasets
code→syntax tree→node sequence→CodeT5+ embedding→classifier→human or AI?
Its syntax tree
The race · F1 score, drawn to scale
0255075100
The coin is a reference lane, not a result from the paper: flipping a coin scores 50% F1 when half the code is AI written.
Tree walk
22 of 22
nodes visited in this example. Five lines of code, and already a small forest.
Best existing detector
≈45%
F1 for GPTSniffer and GLTR, the best of the six existing detectors the paper tested. The other four scored 36 to 42.
What the paper found
82.6% F1
for the best model (82.55): a classifier on CodeT5+ embeddings of the syntax tree. Nearly double the best existing detector, on code from HumanEval, MBPP and CodeSearchNet.
This bench shows an accessibility checker handing its findings back to an LLM, round after round, so the page it wrote gets better.
▶ Run round 1: sends the checker's problems back to the model and shows which lines it rewrites. Press it again for rounds 2 and 3.
↺ Start over: returns to the model's first draft.
Block by block: switches the second experiment from rewriting whole files to rewriting small blocks.
How it’s built: the diagram above the experiment plays one code block around the real loop. Point at a box to read what it does.
Watch: the Checker report, where fixed problems get crossed out, and the count under "This loop".
FeedA11y: let the checker talk back
Ask an LLM for a web page and it gets the easy accessibility right. Then it reaches for ARIA.
We compared web code from GPT-4o, Qwen2.5-Coder-32B-Instruct-AWQ and Gemini 3 Flash with code people wrote. Asking nicely (zero shot, few shot, self criticism) helps a little. FeedA11y runs an accessibility checker on the output and hands the violations back, for up to three rounds. Run the loop.
Redrawn from the paper's Figure 4. Average inaccessibility rate under AChecker: FeedA11y 0.235, plain generation 0.334, human code 0.443.What each part does
1generate→2check→3feed back→4regenerate↺ back to 2
product-card.htmlround 0 of 3
Checker reportexample, not a study result
Second experiment · how much to rewrite at oncereal rates from the paper
To compare with human code, the study had models regenerate real web files. Press Block by block to see what changes when they rewrite one small block at a time.
opens in a browserdoes not render
This loop
4 issues
Before any feedback. The basics already pass.
Files that still render
15% → 93%
of regenerated UI files rendered, once a custom parser split HTML, JavaScript and CSS into blocks instead of regenerating whole files.
What the paper found
Loop wins
LLM code often beat human code on basics like color contrast and alternative text, and struggled with ARIA. The feedback loop, built on ReAct, outperformed zero shot, few shot and self criticism prompting.
I cofounded Lamasoo, a B2B booking platform for travel agencies and hotels, built its core engine as the sole developer, then led the team that scaled it. Hotels ran their own systems, many on servers in the building that failed for hours. This is the shape of what we built around them. Flip it over for the stack underneath.
The shape of Lamasoo’s platform: travel agencies on the left, the core engine in the middle, six hotel systems on the right.
Press a Hotel button to take that hotel down, and again to bring it back.
Watch searches keep flowing from the poller’s copy while new bookings wait in the retry queue.
Drag Traffic up to 5×, the load after the scaling work.
Press ⟲ Flip it: the stack, then walk the layers with Next layer ▸.
Watch:bookings lost stays at 0.
Lamasoo's booking platform
Under the hood · 2019 to 2022top to bottom, the way a booking travels
1 of 6
Clients
Edge
Services
Data
Watch
Ship
merge to master→GitLab CI tests→Docker image→deploy to Kubernetes→Sentry gate→live
v41 live · downtime 0 s
Take a hotel down with its button: searches keep flowing, and new bookings wait in line. Then flip it over.
searchbookingconfirmedpaymentavailability syncto support
0bookings confirmed
0waiting to retry
0handed to support
0bookings lost
0payments settled
6 of 6hotels with fresh data
5× search throughput, to 500 requests a second
Monthly bookings from 5K to 20K+
Searches 15× faster, answered from the poller's copy instead of the hotels
Stale availability reads from about 60% to under 5%
40+ travel agencies and 300+ hotels by Sept 2022
Deploys went from about a minute of downtime to zero
Projects
Projects with trust issues
An agent that doesn't trust the other agent, and a tool that doesn't trust its analyzer.
2026 · personal project · sole engineer
This bench shows how JobAgent decides whether a job posting is worth my time, and how much of that needs no AI at all.
▶ Run it again: replays the selected posting through every gate.
ML Engineer (or any posting on the left): sends that posting through. Two of the three are stopped by plain code.
⟲ Flip it: how it works: turns the bench over to a comic of the whole system, in six scenes.
Next ▸: plays the next scene. Show the whole cast lists every agent, model and piece of code.
How it’s built: the diagram above the experiment plays one posting through the real system. Point at a box to read what it does, or press Plant a fake quote to watch the Python checks catch it.
Watch: each gate turn green or red, then the evidence table, where the checking agent strikes one of the screening agent's claims. On the flip side, the amber line under each scene is the real fact.
JobAgent: an agent that reads job posts, and a second one that doubts it
Most job postings don't need an LLM to say no. The ones that do get two, and neither is trusted.
JobAgent is a human in the loop job search system. It polls company job boards and screens each posting against my verified career facts. Rules in code go first. Then a screening agent matches each requirement to evidence, a checking agent rules on every match without seeing the screener's reasoning, and Python computes the score. Drop a posting in.
How it's builtdrawn from the system's design, built parts only
postingmain pathverifiedflaggedthe pagesettled
Most background dots stop at the rules, as most real postings do. The story posting and the fake quote are illustrations.What each part does
front: try the pipeline
Pick a postingall three invented for this page
The pipeline
Evidence, one requirement at a timedemo rows; the cited facts are real
How it works · a comic in six scenesthe characters are invented, the system is real
Stage crew: a worker drives the agents through a SQLite job queue, with a FastAPI service and a web console that shows each posting's requirements, evidence and score breakdown
Budget: prompt caching cuts the cost of screening, and daily and per call cost ceilings hold the queue when reached
Rehearsals: 5,000+ tests replay recorded model replies, so no test calls the API
Script: built with Claude Code from a 6,000+ line design spec, each change reviewed by a fresh session that never saw the build
Reviews: I label postings by hand, and check the model's judgments against my own labels
Size: 26,000+ lines of Python, and 41,000+ more in its tests
This posting
…
How I check it
By hand
I label postings by hand and check the model's judgments against my own labels. The project is new and those results still move, so this page prints none of them.
Tested offline
5,000+ tests
They replay recorded model replies, so no test calls a live model.
Python checks every model reply: quotes must appear in the posting, cited facts must exist, and the reply must pass a strict Pydantic schema
The score is computed in code, never by a model
Built with Claude Code against a 6,000+ line design specification, with independent review of every change
26,000+ lines of Python, and 41,000+ more in its tests
M.Sc. thesis · University of Tehran
This bench shows how checking each warning against what the code can really do removes a static analyzer's false alarms.
▶ Check the next warning: walks backward from the flagged line and asks the Z3 prover whether any input can reach it.
Check all: does the same for every warning that is left.
W1 (or any warning): shows that warning's path and verdict.
Reset: brings all ten warnings back.
Watch: the Z3 box on the right. "unsat" means no input can reach the line, so the warning is a false alarm and gets crossed out.
Feature interaction detection: the analyzer that cried wolf
A car's features are built separately. Cruise control wants to speed up; automatic braking has other ideas.
A static analyzer flagged every line of autonomous driving code where two features might collide. Most of those lines can't be reached by any real input. My tool walks backward from each flagged line, collects the conditions needed to get there, and asks the Z3 theorem prover whether they can all be true at once. Try it on these ten.
Static analyzer warningsexample code, not the real system
Symbolic execution
Warnings still standing
10 of 10
Ten alarms. Somebody has to read all of them.
False alarms pruned
0
Lines no real input can reach.
What the thesis found
90%
fewer false positives from an existing static analyzer for autonomous driving code, using targeted symbolic execution built on Clang's C++ ASTs and the Z3 theorem prover.
Playground · three small games
Playground
The work above, as games you can lose. Your scores stay in this tab: nothing is stored or sent.
What the checkers missed
Bugs written for this page, one per Ma11y mutation operator that no tool in the study detected: Change Image Alt Text, Change Page Title and Change Tab Index Order (ISSTA 2024 ↗).
In the study, the six tools caught 35 of the 130 semantic mutants (26%).
Human or AI?
Code from the data of the ICSE 2025 study ↗. Human solutions: HumanEval-X ↗ (Apache 2.0) and MBPP ↗ (CC BY 4.0, Austin et al. 2021). AI solutions: ChatGPT, GPT-4 and Gemini Pro, generated for the study. Comments, docstrings and blank lines are removed, and indentation is set to four spaces, on both sides.
For scale: a coin flip gets 50% on average. The best existing detectors in the paper averaged an F1 of about 45 on the study's datasets. F1 is a different measure from a hit rate, and five snippets are not a test set.
Which commit message was real?
Real prompt changes and commit messages from public MIT, Apache 2.0 and BSD repositories in the Prompting in the Wild ↗ data (MSR 2025). The other three messages in each question were written for this page. {…} marks a value the program fills in. The set is picked to be playable, so it has more documented changes than the study found.
In the study, 21.9% of prompt changes were documented in their commit messages, and 69.6% of those in general terms only, like “Improve prompts”.
Open issue
Looking for a team that asks for proof.
I'm finishing my M.S. in Software Engineering at UC Irvine in December 2026, and looking for research engineering roles in AI evaluation, LLM agents and developer tools, from early 2027.