When There’s No Single Right Answer: Evaluating AI Analytics Without a Reference

Conversational AI is changing how people work with data. Instead of manually constructing a visualization, analysts can increasingly ask a question in natural language and have an agent generate a chart, explain what it found, and refine the analysis through conversation. But this creates a difficult evaluation problem: how do we know whether the answer is actually good?

Earlier this year, we introduced Lexara, a toolkit for evaluating conversational visual analytics across dimensions such as data fidelity, visualization design, factual grounding, and conversational coherence. LEXARA-RF extends that work by asking a harder question: how do we evaluate an answer when there is no single expected answer to compare against? 

Most evaluation approaches, including Lexara, rely on a curated reference: a visualization, specification, or response that someone has authored as an acceptable answer. References work well when a task has a narrow set of expected outputs. But analytical questions are often open-ended, with multiple answers that may be equally valid. A request like “Show me revenue trends,” for example, could reasonably produce a monthly line chart, a quarterly aggregation, or an explanation emphasizing a particular pattern in the data. A response can therefore be analytically valid without matching the particular answer chosen for the benchmark. 

This creates three problems. First, references are expensive to author and maintain. As datasets, schemas, prompts, models, and analytical tasks change, expected outputs must be continually created or updated. Second, a single reference can penalize valid alternatives. A response may be analytically correct but still score poorly because it differs from the reference in chart type, aggregation, encoding, or phrasing. Third, references are often unavailable in production. Real users ask new questions against evolving data, so there is often no pre-authored “correct answer” for every interaction. Ironically, the settings where evaluation becomes most valuable—including rapid prompt iteration, model changes, regression testing, and production monitoring—are also where authoring a reference for every interaction becomes least practical. 

LEXARA-RF takes a different approach. Rather than asking whether an agent produced the answer we expected, it asks whether we can verify the answer it actually produced.

From matching answers to verifying them

So how does LEXARA-RF verify an answer without knowing the “right” answer in advance? It starts with evidence already available when an analytical agent runs: the user’s question, underlying data and schema, generated visualization specification, natural-language explanation, and conversation history. It then applies thirteen metrics, each scored on a common 0–100 scale.

The metrics fall into three groups:

• Expressiveness: Did the system answer the question correctly? This checks the data values, fields, filters, and sorting against the user’s request and underlying data.
• Effectiveness: Is the visualization appropriate and interpretable? This evaluates chart type, axes, visual encodings, and interactive details against established visualization-design principles.
• Conversational alignment: Is the explanation truthful and context-aware? This checks whether claims are grounded in the data, important assumptions are surfaced, useful insights are communicated, and context is preserved across conversational turns.

LEXARA-RF does not rely solely on subjective LLM judgments. Wherever possible, it uses deterministic or execution-grounded evidence. For example, Data Fidelity independently re-executes the visualization’s transformation pipeline against the underlying data and compares the result with what the agent actually displayed. Conversational dimensions that are harder to verify directly combine deterministic preprocessing with rubric-based LLM assessment.

Finding not just that something went wrong, but what went wrong

A central goal of LEXARA-RF is to provide diagnostic signals, rather than collapsing quality into a single score.

For example, in one case a user asks for the “top 10 customers by profit in 2023.” The response looks reasonable at first glance: it returns a bar chart of ten customers and correctly applies the 2023 filter. But the agent has made one consequential mistake: it used Sales instead of Profit.

LEXARA-RF isolates the effects of that error. Field Similarity falls to 50 because the visualization uses Sales where the request called for Profit. Sort Accuracy falls to 70 because the customers are not actually being ranked by the requested measure. At the same time, Filter Accuracy, Chart Type, Axis Accuracy, Visual Encoding, and Interactivity remain at 100 because those aspects of the response are not the source of the failure.

The mistake then propagates into the explanation. The agent says, “These are the most profitable customers in 2023,” even though its chart is based on Sales. Factual Grounding drops to 30, while Assumptions Disclosure and Insightfulness also decline. The metric decomposition reveals a chain of failure: selecting the wrong field leads to the wrong ranking, which in turn produces an unsupported analytical claim. 

A second example illustrates a different kind of failure. When asked to “show profit by state,” the agent selects the correct field and computes the underlying values correctly. But it uses a sequential single-hue palette even though Profit includes both negative and positive values, making losses difficult to distinguish from low positive profits. Here, the correctness metrics remain high while Visual Encoding falls to 40. The analysis is correct; the problem is how it is communicated.

Does reference-free evaluation agree with people?

We evaluated LEXARA-RF on 120 human-rated conversational visual analytics responses spanning eight language models, six system prompts, multiple analytical tasks, and datasets from finance, education, and healthcare.

Overall, LEXARA-RF aligned well with expert judgment, particularly for metrics that could be grounded directly in the data or visualization structure. Across the 13 metrics, the median Spearman correlation with expert ratings was 0.60. Execution- and structure-grounded metrics such as Data Fidelity, Filter Accuracy, Sort Accuracy, and Factual Grounding aligned most strongly with human judgment (ρ = 0.68–0.74), while more interpretive dimensions such as Insightfulness and Coherence showed lower agreement.

Importantly, removing the reference did not substantially reduce agreement with human ratings in our sample. The mean per-metric correlation with human ratings was 0.61 for LEXARA-RF and 0.60 for reference-based Lexara, compared with 0.46 for BERTScore, 0.24 for ROUGE-L, and 0.18 for BLEU.

Where reference-free and reference-based evaluation disagreed, the differences helped reveal a limitation of relying on a single expected answer. Among cases where their scores diverged substantially, only 24% were true metric errors; 44% were valid alternative answers, while another 32% expressed equivalent information differently. In other words, many disagreements occurred not because the response was wrong, but because it provided a valid answer the reference did not anticipate. LEXARA-RF was also strongly diagnostic for structural errors: among the 54 structurally grounded cases in our diagnostic analysis, the expected metric localized the failure in 98% of cases. More subjective conversational failures tended to produce signals across several related metrics rather than a single isolated metric.

Why this matters for AI-powered analytics

Reference-based benchmarks remain useful. When we know what a system should produce, carefully authored test cases give teams controlled, reproducible ways to measure regressions.

But they cannot cover everything that happens once an analytical agent encounters real users and real data.

Reference-free evaluation opens up several complementary possibilities. A team changing a prompt could run structural checks across a much larger set of interactions without first authoring a gold response for each one. A model upgrade could be evaluated for changes in filter preservation, field selection, chart design, or factual grounding. Production systems could surface anomalous drops in specific dimensions for investigation. This creates opportunities to use reference-free evaluation across continuous integration, prompt and model iteration, regression triage, and production monitoring, complementing reference-based benchmarks rather than replacing them. 

Importantly, reference-free does not mean human-free. Our results point toward a hybrid approach: automate checks where there is strong, verifiable evidence, while keeping people in the loop where analytical quality requires more subjective judgment. Evaluation results can then help distinguish issues suitable for automated release gates from those that deserve human review. 

More broadly, this work suggests a shift in how we think about quality for analytical agents. As generative systems become more capable, evaluation cannot always mean checking whether AI reproduced a predetermined answer. For many analytical questions, there may be multiple valid answers. Instead, we may increasingly need evaluation systems that can ask: 

• What evidence supports this answer? 
• Does the analysis respect the user’s intent? 
• Can its claims be traced back to the data? 
• Does the visualization communicate those claims faithfully? And where should a person still make the call? 

That is the larger idea behind LEXARA-RF: moving from evaluating AI by matching what we expected toward evaluating it by verifying what it actually did.

Interested in learning more?

Read the full paper: LEXARA-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents, by Srishti Palani and Vidya Setlur, accepted to IEEE VIS 2026.