I’m helping
Outlier AI
find a top candidate to join their team
adaptable
for the role of
AI Agent Evaluation Specialist
.

You will shape the future of autonomous AI agents by refining their reasoning and reliability.

Compensation:

USD 2.4K - 5K/month

Location:

Remote (anywhere)

Mission of Outlier AI:

"Help organizations make better use of their existing data and tools to improve business decision-making."

What makes you a strong candidate:
- You are proficient in Workflows, Technical analysis, Quality assurance (QA), Feedback, Automation.
- You have the potential to develop in Software engineering, Data analysis, Artificial intelligence (AI), AI-focused software engineering, AI evaluation.
- English - Conversational

Responsibilities and more:

As an AI Agent Training Contributor, you may work on tasks involving:
- Evaluating AI-generated responses and agent behavior.
- Reviewing complex, multi-step AI workflows.
- Identifying logical, factual, technical, or execution errors.
- Testing AI agents against edge cases and failure scenarios.




- Analyzing agent traces, tool usage, and reasoning paths.
- Creating and applying evaluation criteria and rubrics.
- Providing structured feedback to improve AI accuracy and reliability.
- Reasoning through software, data, QA, and AI-related problems.
- Helping improve the performance of next-generation autonomous AI agents.

Key Responsibilities:
- Review AI-agent outputs for correctness, relevance, consistency, and instruction-following.
- Detect hallucinations, reasoning failures, workflow errors, and incorrect tool usage.
- Evaluate whether an AI agent successfully completes multi-step technical tasks.
- Compare alternative AI responses and determine which performs better.
- Test models using realistic scenarios, adversarial prompts, and edge cases.
- Document issues clearly and explain why a response or workflow succeeds or fails.
- Follow detailed project guidelines, rubrics, and evaluation standards.
- Provide high-qua

📌 AI Agent Evaluation Specialist (Argentina)
🏢 Emma of Torre.ai
📍 Argentina