Label Studio

At eBay's Payment & Risk team, the data labeling workflow was hindering the timely updates and effectiveness of our detection AI models. Recognizing this critical bottleneck, our team took ownership and spearheaded the development of Label Studio from the ground up, a strategic initiative to revolutionize our data labeling workflows.

Problem

Slow & inefficient labeling delays AI updates

Data labeling significantly impacts our AI model training cycles. Frequent feedback from data scientists indicated that lengthy labeling processes delayed critical model updates, posing potential risks to eBay’s financial security.

Why this problem?

Auditing our current process

To better know the reasons behind this problem, my first steps is to carefully review & evaluate our team’s current solution.

#1 Collaboration among three distinct roles

I first consulted our PM, who closely interacts with the labeling team, to clearly define our target users. This foundational understanding shaped the direction for subsequent observation studies and interviews.

#2 Excel, the primary tool, was unsuitable for labeling tasks

Reviewing existing Excel-based workflows, I identified several usability and efficiency issues:

Observation Study & Interviews

Understanding each role’s pain points

To dive deeper, we conducted remote observation sessions and interviews to pinpoint specific inefficiencies and user needs.

I distilled the core challenge for each role into a clear, concise statement, enabling our team members to quickly understand user needs and effectively guide our subsequent design decisions.

Brainstorming & Scoping

Selecting the MVP solution

After rapidly brainstorming possible solutions, I presented our options to the PM and dev team. We prioritized essential, low-effort solutions for Phase I, deferring additional enhancements to Phase II.

User flow Map

Mapping a unified user experience

To ensure Label Studio delivered a seamless experience, I mapped detailed user flows across all three roles—Data Scientist, Admin, and Annotator. Collaborating closely with our lead engineer during this process allowed us to refine and simplify the architecture, ensuring significantly reducing user steps.

Design

Our final design for launch

Before revealing our design iterations, here's our final design. For more detailed insights into design decisions, feedback integration, and design thinking, feel free to reach out to me.

Create a project

The first step for launching a data labeling job is to create a data labeling project.

Import raw data

Previously a manual download from the transaction database & local sampling is now streamlined through direct database sampling or easy file/url uploads.

Question & instruction setup

Replacing cumbersome Excel processes, our Question and Inline Instruction Editors facilitate easy question creation and content organization.

Preview & launch a job

After uploading data and setting up questions & instructions, Data Scientists can preview labeling jobs exactly as annotators see them, then launch jobs with confidence and easily track progress.

Assign pending Jobs

Admins immediately receive pending job notifications and gain real-time visibility into annotator availability, job statuses, and jobs histories, enabling informed decisions and smoother communication.

Perform the assigned job

Assigned annotators access clear instructions in an optimized workspace, improving focus, efficiency, and overall labeling accuracy compared to previous Excel-based methods.

Iteration

Refining design through feedback

I conducted usability testing with cross-functional team members to evaluate the tool’s clarity, task completion, and overall comprehension.

Result

Achieving user satisfaction and business goals

With Label Studio, Payment & Risk team transitioned to a fully integrated data labeling platform. Collaborating closely with PM and engineers, we set success metrics and validated our solution’s impact. The tool significantly improved process efficiency and positively impacted employees' daily experiences.

I could sense that my efforts not only improved the overall process efficiency but also had a positive impact on employee's daily work.