About Us:
Paytm is India's leading mobile payments and financial services distribution company. Pioneer of the mobile QR payments revolution in India, Paytm builds technologies that help small businesses with payments and commerce. Paytm’s mission is to serve half a billion Indians and bring them to the mainstream economy with the help of technology.
About the Team:
The AI Operations team is responsible for building, launching, and scaling AI-powered voice and Agentic AI solutions across Paytm’s merchant ecosystem.
The team works closely with Business, Product, Technology, Data, QA, and Operations teams to drive AI-led automation across merchant engagement, retention, activation, support, and service journeys.
The team owns the complete operational lifecycle of AI campaigns — from use-case identification and conversational journey design to voice-bot creation, prompt development, testing, deployment, performance monitoring, and continuous optimization.
About the Role:
We are looking for a Quality Analyst – AI Voice Call QC who will own the quality of AI voice agent conversations across merchant-facing campaigns.
The role will involve listening to AI call recordings, auditing transcripts against the prompt and business rules, identifying conversation and compliance gaps, writing and refining prompts, and validating that fixes improve call quality in production.
The candidate should have a strong combination of call quality auditing, prompt engineering, conversational AI understanding, attention to detail, and analytical thinking.
Role Expectations / Responsibilities:
AI Call Audits: Listen to AI voice call recordings and review transcripts daily to evaluate agent performance against the prompt, conversation flow, business rules, and expected outcome.
Issue Identification: Identify issues such as incorrect or off-script responses, hallucination, repetition, long monologues, missed objections, incorrect intent identification, wrong language/script handling, interruptions, unnecessary probing, and poor call closure.
Prompt Writing & Refinement: Write, test, and refine production prompts based on audit findings — fixing conversation flow, objection handling, edge cases, language handling, tool/variable usage, and closure logic.
Root Cause Analysis: Classify each issue by root cause — prompt gap, business logic, STT/TTS error, latency, tool/API failure, or data issue — and route it to the right owner.
Pre-launch Testing: Test new and updated AI agents before go-live using test calls and scenario-based test cases covering happy paths, objections, edge cases, and negative scenarios.
Post-change Validation: Validate every prompt change on live calls to confirm the fix works and has not introduced regressions elsewhere in the flow.
Quality Framework: Define and maintain QC parameters, audit checklists, scoring rubrics, and critical/non-critical error categories for AI voice calls.
Compliance Monitoring: Flag critical compliance gaps such as mis-selling, incorrect financial information, unauthorized promises, disclosure misses, and data privacy lapses, and ensure immediate corrective action.
Disposition & Outcome Accuracy: Verify that post-call dispositions, feedback tags, and outcome capture accurately reflect what happened on the call.
Quality Reporting: Publish regular quality reports and dashboards covering quality scores, top issue categories, trends, fixes shipped, and their impact.
Feedback Loop: Share call-level insights with Campaign Managers, Product, and Business teams, and convert recurring issues into prompt, flow, or platform improvements.
Multilingual QC: Audit calls across Hindi, English, Hinglish, and regional languages for accuracy, tone, pronunciation, and language-switching behaviour.
Documentation & Governance: Maintain audit logs, prompt change history, test cases, known issues, and quality benchmarks for every campaign.
Key Success Metrics:
The role will be measured across AI call quality, prompt effectiveness, compliance, and audit efficiency, including:
AI conversation quality score
Prompt adherence rate
Reduction in hallucinations and off-script responses
Critical compliance error rateIssue detection-to-fix turnaround time
Post-fix regression rate
Disposition / outcome tagging accuracy
Audit coverage and volume
Pre-launch defect catch rate
Improvement in campaign successful outcome rate driven by quality fixes
Required Qualifications:
Experience: 2–5 years of experience in Quality Assurance, Call Quality Audits, Contact Centre Quality, Conversational AI, or a related domain.
Call QC Experience: Hands-on experience auditing voice calls or chat conversations against defined quality parameters and scorecards.
Prompt Engineering: Hands-on experience writing and refining prompts for AI/LLM-based applications, preferably voice agents.
AI Quality: Ability to analyze AI conversations and pinpoint issues related to intent understanding, prompt adherence, hallucination, repetition, tone, compliance, and outcome acheivement.
Attention to Detail: Strong listening skills and the ability to catch subtle errors in AI responses, language, and flow.
Analytical Skills: Strong analytical and problem-solving skills with proficiency in Excel/Google sheets.
Languages: Fluency in Hindi and English. Proficiency in one or more regional languages will be an advantage.
Communication: Excellent written and verbal communication skills with strong documentation capabilities.
Execution: Strong ownership and ability to audit multiple AI campaigns in parallel with fast turnaround.
Preferred Skills:
Experience in FinTech, Payments, Lending, Merchant Operations, or Contact Centre Operations.
Hands-on experience with AI voice agents / OBD / IVR quality audits.
Understanding of LLMs, speech-to-text, text-to-speech, intent classification, tool calling, and agent workflows.
Familiarity with templating in prompts (e.g., Jinja2) and structured prompt formats.
Experience using LLMs to automate transcript analysis and QC at scale.
Exposure to SQL, dashboards, and CRM/ticketing systems.