Human-in-the-Loop AI Operations

What HITL Means in Production

Human-in-the-loop (HITL) is not a fallback for when AI fails. It is the architecture that makes AI deployable at enterprise scale. When designed correctly, a HITL system provides three things simultaneously: a quality gate, a feedback loop for model improvement, and a trust signal for enterprise clients.

At IV.AI, I built this architecture from zero — no existing process, no playbook, no team. I designed the workflows, hired and onboarded the team, and built the quality systems that kept the pipeline reliable through a Fortune 500 global rollout.

What I Built

• Designed end-to-end HITL workflows: task routing, labeler roles and permissions, escalation paths, review queues, and quality checkpoints.
• Defined quality metrics and built the evaluation framework used to measure pipeline accuracy at scale.
• Built and managed a 30+ person global operation across HITL workers, QA analysts, and change management leads in multiple time zones.
• Designed the feedback loop architecture: how human corrections fed back into model retraining cycles and how quality signals were surfaced to the engineering and ML teams.
• Developed and maintained change management processes that kept operational reliability stable through product pivots, market expansions, and client requirement changes.
• Partnered with ML and engineering to translate quality signals into model improvement priorities — acting as the bridge between the human operation and the technical stack.

The Operational Challenge

Scaling a HITL operation globally means different things in different markets: different language requirements, different data formats, different local regulations, and different workforce availability. The system had to be robust enough to handle all of this while maintaining consistent quality standards for a Fortune 500 enterprise client with zero tolerance for pipeline failures.

I treated the operation like a product — with a roadmap, feature iterations, and quality metrics. Not an ops afterthought. That framing is what made it scalable.

Outcomes

• 30+ person HITL operation built, staffed, and managed globally.
• Sustained pipeline quality through global production rollout for a Fortune 500 client.
• Zero major pipeline failures during client expansion across multiple international markets.
• HITL operation became a core enterprise selling point — included in $4M+ of deal scoping.

Skills & Domains

HITL operations · Human-in-the-loop design · Annotation workflow engineering · Data quality at scale · AI quality assurance · Feedback loop architecture · Global team management · Change management · AI operations · MLOps adjacent · Enterprise AI · Fortune 500 client management · Cross-functional leadership · Remote team management · Process design

About Tae Hyun Koh

Tae Hyun Koh is an AI Product Lead and systems builder based in New York City. He specializes in HITL operations, enterprise AI products, and zero-to-one product development. He is currently open to Senior PM and AI Product Lead roles.

kohtae.com · taehyunkoh0607@gmail.com · LinkedIn