AI Scoring evaluates interactions only from the conversation transcript and the evaluation form that you provide.

AI Scoring does not use metadata such as:

• Queue or routing path
• Time stamps or duration
• Platform or system metrics

AI Scoring focuses purely on what was said or written, not on tone, attitude, or inferred meaning.

For more information, see Automated scoring options in evaluation forms.

Do

• Write objective, transcript-driven questions.
• Provide clear help text with measurable criteria.
• Use consistent terminology; for example, “agent” and “customer.”
• Write in complete sentences.
• Refine questions continuously based on AI scoring results.

Don’t

• Ask subjective questions; for example, “Was the agent polite?”
• Use vague shorthand; for example, “Greeting protocol.”
• Assume context not visible in the transcript.
• Ask about tone or emotion that AI cannot infer.

The AI model can’t infer intent. It must find direct, measurable evidence in the transcript.

Avoid multi-condition questions – Do not combine multiple requirements into a single question. Questions that include more than one condition are difficult to evaluate consistently, particularly when the agent satisfies some requirements but not others.

Use help text to reduce ambiguity

Help text should explain when each answer option should be selected. Avoid help text that simply repeats the question without providing additional guidance.

Bad example

Question: Did the agent provide a complete resolution?

Help text: Select the appropriate answer based on whether the agent provided a complete resolution.

Answer options: Yes, No, N/A

This help text is unclear because it does not explain what complete resolution means or when each answer should be selected.

Good example

Question: Did the agent provide a complete resolution?

Help text:

• Select Yes if the agent addressed the customer’s stated issue, provided the appropriate solution or next steps, and confirmed that no unresolved concerns remained.
• Select No if the agent did not address the issue, provided incomplete or incorrect information, or left the customer without clear next steps.
• Select N/A if the interaction ended before the agent had a reasonable opportunity to provide a resolution, such as when the customer disconnected before troubleshooting began or the interaction was transferred before the agent could address the issue.

The help text should define each question’s goal and intended use case, and state clearly when a given answer should be selected, so the right answer is chosen for the right situation.

Example: well-written question and help text

Intent: Verify the customer’s identity before providing account-specific details.

Question: Did the agent verify the customer’s identity before providing account-specific details?

Help text: The agent must confirm at least one credential; for example, account number, date of birth, phone number, or order ID.

• Yes: The agent verified a credential before proceeding.
• No: The agent skipped or asked unrelated questions.

Transcript examples

Establishing alignment between human evaluators and Virtual Supervisor is essential for producing reliable, scalable quality evaluations. One effective approach is to have an expert human evaluator manually score a subset of interactions to establish an agreement score—a measure of how closely human evaluations align with the AI-generated scores produced by Virtual Supervisor.

A high agreement score indicates that both human reviewers and the AI are consistently interpreting and applying the evaluation criteria. A lower agreement score can point to differences in interpretation, unclear scoring instructions, or evaluation questions that require refinement. By analyzing where discrepancies occur, teams can improve question wording and scoring guidance, ensuring that both human and AI evaluations reflect the intended standards and can be applied consistently at scale.

For Agent Auto-Complete evaluation forms, review the agreement score for each question in the latest published version of the form. Questions with low agreement scores may indicate unclear wording, incomplete scoring guidance, or answer options that are difficult to automate. Improving these questions can increase the reliability of automated scoring and reduce the likelihood that questions are excluded or flagged because of low historical accuracy.  Questions with low agreement scores may also be more likely to be affected by the historical accuracy filter and, as a result, may not be answered automatically. For more information about the historical accuracy filter, see Automated scoring options in evaluation forms.

How to establish and improve agreement scores

1. Create a new evaluation form
   • Click Menu > Conversation Intelligence > Quality Management > Evaluation Forms.
   • Click Create and then select either Evaluation Form or the Agent Auto-Evaluation as the form type.
   • Build the evaluation form using the best practices outlined in this article.
   • Publish the form.
2. Evaluate interactions using the form
   • Click Menu > Analytics > Analytics Workspace > Interactions.
   • Open an individual interaction.
   • In the Quality Summary tab, click Create Evaluation.
   • Select the specific form you want to review.
   • Choose one of the following options:
     • For automated submission, select Virtual Supervisor as the evaluator.
     • To review automated scores before submission, select a human evaluator.
   • Click Create. If you select a human evaluator, review the completed evaluation and update any answers that were incorrectly selected based on transcript evidence.
3. Test the form across multiple interactions:
   1. Repeat the evaluation process for at least 20 different interactions.
   2. Click Conversation Intelligence > Quality Management > Evaluation Forms and open the form reviewed in step 2.
   3. Review the overall agreement score and the agreement scores for individual questions.
4. Refine the form as needed:
   • Use the agreement data to identify questions or scoring instructions that need adjustment.
   • Update the evaluation form and repeat the testing process until acceptable agreement levels are achieved.
5. Continuously monitor and retest the form. Review form performance regularly and make adjustments as needed. Once the first 20 evaluations have been completed, subsequent evaluations continue to refine the agreement score as long as they occur within the rolling 30-day window.

• Review AI scoring reports monthly to identify low-confidence questions.
• Add examples reflecting your industry’s phrasing; for example, healthcare vs. retail.
• Align human evaluators with the same definitions used by AI.
• Refine questions and help text when AI misclassifies results.

Question:
Did the agent greet the customer at the beginning of the conversation using a standard greeting such as “hi,” “hello,” or “good morning”?

Answer Options:
Yes / No

Help Text:
The agent must begin with a polite greeting. Acceptable greetings include “hi,” “hello,” “good morning,” or equivalent.

Transcript Example:
Customer: “I need help with my account.”
Agent: “Good morning! Thank you for calling ABC Support.”
→ AI marks Yes

Question:
Did the agent verify the customer’s identity before addressing account-specific concerns?

Answer Options:
Yes / No

Help Text:
The agent must verify at least one customer credential, such as date of birth, account ID, or phone number, before proceeding.

Transcript Example:
Customer: “I want to update my billing address.”
Agent: “Can I confirm the last four digits of your account number?”
→ AI marks Yes

Question:
Did the agent allow the customer to finish speaking without interruptions?

Answer Options:
Yes / No

Help Text:
An interruption occurs when the agent cuts the customer off mid-sentence. Clarifications after the customer finishes are acceptable.

Transcript Example:
Customer: “The issue started when—”
Agent: “Let me stop you there…”
→ AI marks No.

Question:
Did the agent use the customer’s name at least once during the conversation?

Answer Options:
Yes / No

Help Text:
Using the customer’s name personalizes the interaction. The AI checks whether the name, as provided in the record or conversation, was mentioned.

Transcript Example:
Customer: “I’m Maria, and I need help with my order.”
Agent: “Thank you, Maria. Let’s look at your order.”
→ AI marks Yes

Question:
Did the agent acknowledge the customer’s issue before moving to resolution?

Answer Options:
Yes / No

Help Text:
Empathy is demonstrated when the agent acknowledges the issue with phrases such as “I understand,” “I’m sorry that happened,” or “I know this must be frustrating.” The AI looks for acknowledgment before resolution.

Transcript Example:
Customer: “I’ve been overcharged.”
Agent: “I’m sorry to hear that. Let’s review your bill together.”
→ AI marks Yes

Question:
Did the agent provide a clear resolution step, such as a refund, replacement, or troubleshooting instruction?

Answer Options:
Yes / No

Help Text:
A resolution must include a specific action (refund, replacement, or technical steps). General apologies without clear next steps do not count.

Transcript Example:
Agent: “I’ve processed a replacement order that will arrive within two days.”
→ AI marks Yes

Question:
Did the agent explain the escalation process clearly, including who to contact, what information is needed, and expected response times?

Answer Options:
Yes / No

Help Text:
A complete escalation includes:

1. Next contact point.
2. Required information.
3. Expected timeframe for response.

Transcript Example:
Agent: “If this issue persists, I will escalate to Level 2. They will need your case ID and will respond within 24 hours.”
→ AI marks Yes

Question:
Did the agent comply with mandatory disclosure or compliance statements (for example, terms, disclaimers, or legal requirements)?

Answer Options:
Yes / No

Help Text:
Compliance statements vary by industry and may include required disclosures, disclaimers, or legal notices. AI checks for these specific phrases in the transcript.

Transcript Example:
Agent: “For security purposes, I cannot provide account details without verifying your identity first.”
→ AI marks Yes

Question:
Did the agent avoid unnecessary dead air or long silences without explanation?

Answer Options:
Yes / No

Help Text:
The agent must clearly explain to the customer that they will be placed on hold by using phrasing like “I will place you on a brief hold”.

Transcript Example:
Agent: “Give me a moment while I pull up your account details.” [10 seconds of silence]
→ AI marks Yes

Question:
Did the agent confirm customer satisfaction or summarize the next steps before closing the conversation?

Answer Options:
Yes / No

Help Text:
Before closing, the agent should confirm the resolution or clearly summarize the next steps. This path demonstrates completion and helps ensure customer understanding.

Transcript Example:
Agent: “I’ve reset your password. Can you confirm you’re able to log in now?”
→ AI marks Yes