mahan.tafreshipour
Software engineer · Researcher · UC Irvine

It works on my machine. Prove it.

  • “99% accurate, says the demo.” I find the 1%.

I'm Mahan, a software engineer and researcher. I build systems with large language models, and the evaluations that show where they fail.

I finish my M.S. in Software Engineering at UC Irvine in December 2026, and I'm looking for research engineering roles from early 2027.

Every number on this page has a source; click a mark to see the paper, table or record behind it.

Click or tap the portrait for another one.

AI portraits of me, made with ChatGPT Pro's Images 2.6 model from my photos, a new seed each time. The noise is an animation; no model runs here.
Research · five papers, first author first

Experiments you can run

Each bench is a small version of one study: you do what the study did, and the readout gives the real result from the paper. Each one opens with a click.

MSR 2025 · first author

Prompting in the Wild: the prompt that argues with itself

In real apps, the prompt is code. Nobody reviews it like code.

I built a static analyzer on Tree-sitter that pulls prompts out of Python projects, then followed 1,262 prompt changes across 243 repositories. Scrub through one prompt's history below: every commit adds, and some commits quietly contradict what came before.

system_prompt.txtversion 8 of 8

Press ▶ Play its history to watch this prompt grow, one commit at a time.

Commit logsays what changed?
    This prompt, now
    8 lines

    3 contradictions

    Commits that explain the change
    3 of 8

    in this illustration

    What the paper found
    21.9%

    of prompt changes were documented in their commit messages. The study followed 1,262 prompt changes in 243 repositories. Prompts mostly grow, and changes can bring in logical contradictions no test catches.

    ISSTA 2024 · first author

    Ma11y: break the page. It still looks fine.

    Accessibility checkers promise to find what's broken. Ma11y tests the checkers by breaking pages on purpose.

    Each of its 25 mutation operators reproduces a failure technique from WCAG 2.1: one small edit to a live page that shuts someone out. Seven of the real operators are below. Plant them, then watch three things change: the code, the screen, and the people using it.

    How it's builtredrawn from the paper's Figure 1

    Redrawn from the paper's Figure 1. The story is its Table 4 example, and the background dots are not individual results.
    What each part does
    Plant a defect
    Sunset Hotel · Book a room
    sunsethotel.example/book

    Sunset Hotel

    Rooms from $89 a night

    Click into the page and press Tab to walk it with a keyboard.

    Who notices

    Nothing planted yet

    Every visitor can book a room.

    On screenThe page as its designer meant it.

      Page sourceoriginal

      Ma11y makes edits like these to the live page in a real browser, through Puppeteer, then runs every checker on the original and on the mutant.

      What a screen reader says

        Plant a defect, then look at the page: did anything change?

        Defects planted
        0 of 7

        Ma11y makes at most one mutant of a page per operator. Here you can stack them.

        Visible on screen
        0 of 0

        Plant a defect. Most of them change nothing you can see.

        What the paper found
        3 in 4

        of the 366 defects Ma11y planted in real websites slipped past the average checker: 274 of them. The six checkers were A11yLite, Access Continuum, axe, IBM Equal Access, QualWeb and WAVE.

        ICSE 2025 · second author
        ACM SIGSOFT Distinguished Paper Award · ICSE 2025

        Reca11: from a test session to a bug report

        A tester talks through an app while using it. Somebody then has to turn that recording into a bug report. Reca11 is the somebody.

        It is an LLM pipeline (GPT-4) that reads the auto generated transcript of a recorded user testing session and writes a structured report: each issue with a title, a description and its timestamps. Play the session below and watch the report fill in.

        How it's builtredrawn from the paper's Figures 3 and 4

        Redrawn from the paper's Figures 3 and 4. The issue in the story comes from its Figure 2, and the cluster plot is a sketch, not the paper's data.
        What each part does
        Session transcriptwritten for this page, not a real participant

          Issue reportselect a card to see its source lines
          This session
          5 cards

          Four are real issues. One is a dog. That is what precision measures.

          Against the human annotators

          313 found, 6 missed. One square per issue. The six are parked at the end; the paper does not say where they sit.

          What the paper found
          6 of 319

          issues missed (under 2%), at 85% precision, on 36 real user testing sessions across three applications.

          ICSE 2025 · second author

          Who wrote this code? The detectors don't know either.

          Six detectors promise to spot AI writing. On code, the best of them average an F1 of about 45.

          We tested six existing detectors, then tried our own: fine tuned GPT-3.5, few shot prompting with retrieved examples, and classifiers on CodeT5+ embeddings. The best one embedded only the syntax tree. I built the detection experiments and the embedding detector, including the tree walk. Walk one below, then run the race.

          How it's builtredrawn from the paper's Figure 2

          Redrawn from the paper's Figure 2. The tree alone reading reached 82.55 F1, against about 45 for the best existing detectors.
          What each part does
          A functionexample code, not from the datasets
          
                

          code→syntax tree→node sequence→CodeT5+ embedding→classifier→human or AI?
          Its syntax tree
            The race · F1 score, drawn to scale

            The coin is a reference lane, not a result from the paper: flipping a coin scores 50% F1 when half the code is AI written.

            Tree walk
            22 of 22

            nodes visited in this example. Five lines of code, and already a small forest.

            Best existing detector
            ≈45%

            F1 for GPTSniffer and GLTR, the best of the six existing detectors the paper tested. The other four scored 36 to 42.

            What the paper found
            82.6% F1

            for the best model (82.55): a classifier on CodeT5+ embeddings of the syntax tree. Nearly double the best existing detector, on code from HumanEval, MBPP and CodeSearchNet.

            TOSEM 2026 · second author

            FeedA11y: let the checker talk back

            Ask an LLM for a web page and it gets the easy accessibility right. Then it reaches for ARIA.

            We compared web code from GPT-4o, Qwen2.5-Coder-32B-Instruct-AWQ and Gemini 3 Flash with code people wrote. Asking nicely (zero shot, few shot, self criticism) helps a little. FeedA11y runs an accessibility checker on the output and hands the violations back, for up to three rounds. Run the loop.

            How it's builtredrawn from the paper's Figure 4

            Redrawn from the paper's Figure 4. Average inaccessibility rate under AChecker: FeedA11y 0.235, plain generation 0.334, human code 0.443.
            What each part does

            product-card.htmlround 0 of 3
              Checker reportexample, not a study result
                Second experiment · how much to rewrite at oncereal rates from the paper

                To compare with human code, the study had models regenerate real web files. Press Block by block to see what changes when they rewrite one small block at a time.

                This loop
                4 issues

                Before any feedback. The basics already pass.

                Files that still render
                15% → 93%

                of regenerated UI files rendered, once a custom parser split HTML, JavaScript and CSS into blocks instead of regenerating whole files.

                What the paper found
                Loop wins

                LLM code often beat human code on basics like color contrast and alternative text, and struggled with ARIA. The feedback loop, built on ReAct, outperformed zero shot, few shot and self criticism prompting.

                Industry · 2017 to 2022

                Lamasoo

                Unplug a hotel. The bookings wait in line.

                I cofounded Lamasoo, a B2B booking platform for travel agencies and hotels, built its core engine as the sole developer, then led the team that scaled it. Hotels ran their own systems, many on servers in the building that failed for hours. This is the shape of what we built around them. Flip it over for the stack underneath.

                Lamasoo's booking platform

                Take a hotel down with its button: searches keep flowing, and new bookings wait in line. Then flip it over.

                0bookings confirmed
                0waiting to retry
                0handed to support
                0bookings lost
                0payments settled
                6 of 6hotels with fresh data
                1×: before the scaling work
                • 5× search throughput, to 500 requests a second
                • Monthly bookings from 5K to 20K+
                • Searches 15× faster, answered from the poller's copy instead of the hotels
                • Stale availability reads from about 60% to under 5%
                • 40+ travel agencies and 300+ hotels by Sept 2022
                • Deploys went from about a minute of downtime to zero
                Projects

                Projects with trust issues

                An agent that doesn't trust the other agent, and a tool that doesn't trust its analyzer.

                2026 · personal project · sole engineer

                JobAgent: an agent that reads job posts, and a second one that doubts it

                Most job postings don't need an LLM to say no. The ones that do get two, and neither is trusted.

                JobAgent is a human in the loop job search system. It polls company job boards and screens each posting against my verified career facts. Rules in code go first. Then a screening agent matches each requirement to evidence, a checking agent rules on every match without seeing the screener's reasoning, and Python computes the score. Drop a posting in.

                How it's builtdrawn from the system's design, built parts only

                Most background dots stop at the rules, as most real postings do. The story posting and the fake quote are illustrations.
                What each part does
                front: try the pipeline
                Pick a postingall three invented for this page
                The pipeline

                  Evidence, one requirement at a timedemo rows; the cited facts are real
                  This posting
                  …

                  How I check it
                  By hand

                  I label postings by hand and check the model's judgments against my own labels. The project is new and those results still move, so this page prints none of them.

                  Tested offline
                  5,000+ tests

                  They replay recorded model replies, so no test calls a live model.

                  • Python checks every model reply: quotes must appear in the posting, cited facts must exist, and the reply must pass a strict Pydantic schema
                  • The score is computed in code, never by a model
                  • Built with Claude Code against a 6,000+ line design specification, with independent review of every change
                  • 26,000+ lines of Python, and 41,000+ more in its tests
                  M.Sc. thesis · University of Tehran

                  Feature interaction detection: the analyzer that cried wolf

                  A car's features are built separately. Cruise control wants to speed up; automatic braking has other ideas.

                  A static analyzer flagged every line of autonomous driving code where two features might collide. Most of those lines can't be reached by any real input. My tool walks backward from each flagged line, collects the conditions needed to get there, and asks the Z3 theorem prover whether they can all be true at once. Try it on these ten.

                  Static analyzer warningsexample code, not the real system

                    Symbolic execution
                    
                          
                      Warnings still standing
                      10 of 10

                      Ten alarms. Somebody has to read all of them.

                      False alarms pruned
                      0

                      Lines no real input can reach.

                      What the thesis found
                      90%

                      fewer false positives from an existing static analyzer for autonomous driving code, using targeted symbolic execution built on Clang's C++ ASTs and the Z3 theorem prover.

                      Playground · three small games

                      Playground

                      The work above, as games you can lose. Your scores stay in this tab: nothing is stored or sent.

                      What the checkers missed

                      Bugs written for this page, one per Ma11y mutation operator that no tool in the study detected: Change Image Alt Text, Change Page Title and Change Tab Index Order (ISSTA 2024 ↗).

                      In the study, the six tools caught 35 of the 130 semantic mutants (26%).

                      Human or AI?

                      Code from the data of the ICSE 2025 study ↗. Human solutions: HumanEval-X ↗ (Apache 2.0) and MBPP ↗ (CC BY 4.0, Austin et al. 2021). AI solutions: ChatGPT, GPT-4 and Gemini Pro, generated for the study. Comments, docstrings and blank lines are removed, and indentation is set to four spaces, on both sides.

                      For scale: a coin flip gets 50% on average. The best existing detectors in the paper averaged an F1 of about 45 on the study's datasets. F1 is a different measure from a hit rate, and five snippets are not a test set.

                      Which commit message was real?

                      Real prompt changes and commit messages from public MIT, Apache 2.0 and BSD repositories in the Prompting in the Wild ↗ data (MSR 2025). The other three messages in each question were written for this page. {…} marks a value the program fills in. The set is picked to be playable, so it has more documented changes than the study found.

                      In the study, 21.9% of prompt changes were documented in their commit messages, and 69.6% of those in general terms only, like “Improve prompts”.

                      Open issue

                      Looking for a team that asks for proof.

                      I'm finishing my M.S. in Software Engineering at UC Irvine in December 2026, and looking for research engineering roles in AI evaluation, LLM agents and developer tools, from early 2027.

                      Or write to