I build AI that
tests AI.

A/B harness pitting stable vs proposed versions of two production voice bots for a global card issuer, scoring each call on a 5-item rubric plus two audio judges, with adversarial scenarios seeded from a real failed demo.

An AI persona places side-by-side nightly calls against two prompt revisions of a collections bot, scores on a rubric, then feeds failures into a Claude prompt-improvement loop. The template every later client harness was built from.

The bespoke harness productized into a cross-platform CLI (ruby-qa, 14 subcommands). Its autopilot closes an autonomous evaluate → improve → redeploy → retest → accept/revert loop with a margin gate and byte-for-byte auto-revert.

Packaged CLI + nightly runner driving a voice-eval platform via a hand-rolled MCP client across four bot environments — producing the documented catalog of recurring defects that anchored all later A/B work.

Reverse-engineered the only working audio-aware LLM-judge path on a third-party platform — after proving the documented flag was silently ignored. A dual-judge pattern that hears what speech-to-text destroys.

Turned that hard-won insight into a YAML-driven CLI and a zero-backend Chrome extension, cutting audio-metric setup from a manual API session to a ~30-second edit — so non-developers can deploy judges themselves.

Streams redacted live voice-bot conversations from New Relic into an AI eval platform for continuous LLM scoring — advancing QA from pre-deploy simulation to real-traffic monitoring. In-process PII redactor; per-run cost guard.

A capacity load test ramping real carrier calls (1→250 concurrent) against a multinational bank's collections dialer, with a deterministic self-terminating flow that isolates load without flooding a live-agent queue.

Drives Azure AD B2C OAuth via Playwright to measure time-to-first-token and full-response latency across up to 14 GPT/Claude models on an internal platform, isolating platform overhead against direct-API baselines.

Aggregates live telemetry from New Relic, Grafana Cloud, and Azure App Insights plus Jira across six products into one RAG slide — with a reverse-engineered Grafana-SSO workaround and defensible per-source outage rules.

A Node CLI wrapping the Claude Agent SDK so QA testers write Playwright tests conversationally — using corporate Microsoft Foundry Claude instead of personal keys. Three-tier credential resolver, headless tool-gating, ixqa doctor live-probe.

A one-click MV3 extension letting marketing run voice-AI evaluators against bots — no backend, direct API, fully white-labeled (zero vendor strings, test-enforced). Includes a reCAPTCHA-bypass setup flow.

A zero-dependency Python generator that cross-references Jira items with Xray test executions into a self-contained HTML release-readiness report — reused via a Claude Code slash command.

Root-caused a two-week LLM auth outage (per-route header mismatch, six hypotheses eliminated), then deployed and validated a 1M-context model — proving accurate retrieval at 900k tokens and pricing out the build-vs-buy case.