Executive Summary

Traditional surveys are time-consuming and frequently produce low-quality data. Simulated populations can deliver actionable insights in minutes rather than months.

• The problem: Traditional surveys are slow, costly, and often deliver uncertain data quality—limiting timely, evidence-based decision-making.
• The approach: Synthetic populations use large language models to simulate survey responses based on realistic population structures, complementing—not replacing—real-world surveys.
• The validation: Across 19 diverse survey items, simulated responses closely matched real-world benchmarks, with an average agreement of ~89%.
• Key insight: Highest alignment was observed for demographic and behavioral items; greater variation for attitudinal questions mirrors known survey sensitivities.
• The value: Synthetic surveys enable rapid hypothesis testing, better question design, and earlier insight generation—reducing cost and friction before fieldwork.
• The takeaway: Used alongside traditional surveys, synthetic populations support faster learning, better-designed research, and more informed decisions.

Why Traditional Surveys Fail Modern Decision-Making?

Thoughts, attitudes, and opinions shape human behavior across domains ranging from everyday consumer choices to consequential life decisions and political processes. Understanding the patterns underlying these mental states enables us to design products, services, and public policies that are not only effective, but genuinely desirable—and capable of driving meaningful change.

Because we cannot directly observe people's thoughts or cognitive processes, surveys remain the primary method for accessing them at scale. Yet designing high-quality surveys is both time-consuming and methodologically demanding. Poorly worded questions, response biases, low engagement, and sampling limitations can all compromise data quality. Moreover, traditional data collection methods often struggle to capture nuance, context, and change over time.

As a result, despite the central role surveys play in decision-making across industries and societies, there are strong reasons to question the reliability and validity of much of the data we routinely collect. Organizations today have to make decisions faster than traditional surveys can deliver actionable insights.

Despite decades of methodological development, core challenges in survey research remain largely unresolved by existing tools. Together, these three structural limitations create substantial hidden costs—through inefficiency, delayed insights, and compromised data quality—for organizations that depend on surveys for decision-making (Table 1):

1. The complexity and expertise required to design high-quality surveys
2. The slow and resource-intensive nature of data collection
3. The inconsistent and often low quality of data produced

Table 1: The three-fold problem of conventional survey practices and their impact on of decision-making quality and costs.

Total estimated cost per traditional survey is €13,000–€65,000, excluding downstream costs of incorrect or delayed decisions. Beyond these direct and indirect costs, low-quality survey data can lead to strategic misalignment, ineffective product decisions, or poorly targeted policies—costs that often exceed the survey budget itself but remain largely invisible and unaccounted for.

Synthetic populations based on large language models offer a practical response to the limitations of traditional survey research. By modeling population structures and simulating how different individuals might respond, they provide decision-makers, researchers, and product teams with faster, lower-cost ways to generate high-quality insights.

Rather than replacing real-world data collection, synthetic populations enable early, directional insights in minutes instead of weeks. This allows organizations to test ideas, iterate rapidly, and enter fieldwork with sharper hypotheses—improving both efficiency and data quality.

Solution: kansa.io and synthetic populations

How kansa.io works?

Kansa.io generates synthetic survey data through a multi-stage process that combines population-level modeling with large language model–based simulation and empirical validation.

• First, a population model is constructed using a combination of open and proprietary datasets describing the demographic and structural characteristics of the target population (e.g., the Finnish population). The model captures key population-level distributions and relationships relevant for survey research and can be adapted to different geographic or demographic contexts.
• Second, personas are generated by sampling from this population model. Each persona represents an individual with a coherent set of demographic and structural attributes drawn to reflect realistic population heterogeneity.
• Third, during survey prompting, a large language model is instructed to respond to survey items from the perspective of a specific persona. The model is constrained to answer consistently with the persona's attributes and the survey's response formats.
• Fourth, individual responses are aggregated to form a synthetic dataset that mirrors the structure of a conventional survey dataset, enabling standard analytical workflows.
• Finally, the synthetic data undergo validation by comparing response distributions and key relationships against established real-world survey benchmarks. This step is used to assess systematic deviations

The potential benefits for different user groups are presented in Table 2

Table 2: Potential benefits of synthetic populations and surveys across user groups

Does it work? Validation of kansa.io

To provide a preliminary understanding of our model's capabilities, we present comparative information from carefully selected benchmark surveys across various topics and industries. The process of survey selection and comparison, as well as the findings and their interpretation, is presented below.

4.1 Methdos

Selecting benchmark surveys

The validation survey comprised a deliberately heterogeneous set of items covering attitudes, beliefs, behaviors, and sociodemographic characteristics. Items were selected to maximize conceptual diversity across substantive domains—including institutional trust, political attitudes, perceived safety and wellbeing, economic conditions, media use, technology attitudes, and consumer behavior—rather than to measure a single latent construct.

Wherever possible, items were adapted from well-established national and international survey instruments (e.g., large-scale social, political, and wellbeing surveys) or closely aligned with question formats that have demonstrated validity and widespread use in population-based research. This approach was chosen to ensure that the item set reflected realistic survey content and response structures commonly encountered in applied research and policy contexts.

The final item pool included a mix of binary, ordinal, and continuous response formats, as well as both attitudinal and behavioral questions. Several items were intentionally included in parallel versions with and without explicit "don't know" response options to evaluate robustness across common survey design choices. Collectively, the items were designed to span varying levels of abstraction, sensitivity, and cognitive demand, thereby providing a stringent test bed for validation analyses.

A complete list of survey items and response scales is provided in Table 2.

Table 2: Survey items included in the validation study

Comparison between real and simulated distributions - Composite Distribution Distance (CDD)

To evaluate how closely AI-simulated survey response distributions reproduce empirically observed distributions, we use a Composite Distribution Distance (CDD). The CDD provides a single, interpretable summary of distributional discrepancy while preserving sensitivity to complementary aspects of mismatch that are relevant for ordinal survey data. The metric is used throughout this white paper to assess the quality of the validation results reported herein.

Distribution representation

For each item, both the empirical ("real") and AI-simulated responses are represented as discrete probability distributions over the response scale:

$$p = (p_1, p_2, \dots, p_k), \quad q = (q_1, q_2, \dots, q_k)$$

where: k denotes the number of response categories (e.g., a 5-point Likert scale), p_i and q_i represent the probability of response category i, distributions are normalized such that

$$\sum_{i=1}^{k} p_i = \sum_{i=1}^{k} q_i = 1$$

Component metrics

The Composite Distribution Distance combines three established discrepancy measures, each capturing a distinct aspect of distributional difference.

1. Normalized Earth Mover's Distance (EMD)

Earth Mover's Distance quantifies the minimum amount of probability mass that must be shifted along the ordered response scale to transform the simulated distribution into the empirical distribution. Because response categories are ordinal, EMD directly reflects the magnitude of misplacement in scale units.

To ensure comparability across different scale lengths, EMD is normalized by the maximum possible distance on the scale:

$$EMD_{norm} = \frac{EMD}{k - 1}$$

This normalization bounds the metric to the interval [0,1], where 0 indicates identical distributions and 1 indicates maximal ordinal displacement.

1. Kolmogorov-Smirnov Distance (KS)

The Kolmogorov–Smirnov distance captures the largest absolute difference between the cumulative distribution functions (CDFs) of the empirical and simulated distributions:

$$KS = \max_i \left| CDF_p(i) - CDF_q(i) \right|$$

KS is particularly sensitive to pronounced local discrepancies, identifying the single point on the response scale at which the mismatch between distributions is greatest.

1. Jensen-Shannon Distance (JS)

The Jensen–Shannon distance measures the overall difference in distributional shape and overlap. It is derived from the Jensen–Shannon divergence, which symmetrically compares each distribution to their average:

$$JS = \sqrt{\frac{1}{2} KL(p,m) + \frac{1}{2} KL(q,m)}$$

Using logarithms with base 2, the Jensen–Shannon distance is bounded in [0,1]. Unlike EMD, JS does not depend on category order; instead, it captures global differences in probability allocation across categories.

---

Composite Distribution Distance (CDD)

The final Composite Distribution Distance is defined as the equally weighted mean of the three component metrics:

$$CDD = \frac{EMD_{norm} + KS + JS}{3}$$

The resulting value lies in the interval [0,1], with:

• 0 indicating perfect correspondence between simulated and empirical distributions, and
• larger values indicating increasing distributional discrepancy.

For interpretive convenience, results may also be expressed as a fit score:

$$Fit = 1 - CDD$$

where higher values indicate closer alignment with empirical data.

Rationale for the Composite Approach

Each component metric captures a distinct failure mode that may arise in distributional validation:

• EMD reflects ordinal misplacement along the response scale,
• KS identifies the most severe localized deviation,
• JS captures global shape differences and loss of distributional overlap.

No single metric adequately characterizes all of these aspects in isolation. By combining them, the Composite Distribution Distance provides a balanced and robust summary measure suitable for evaluating the fidelity of AI-generated survey distributions.

Agreement between real and simulated response distributions was assessed using a normalized Composite Distribution Distance, expressed as percentage agreement for interpretability. Grades were assigned using predefined thresholds to facilitate qualitative interpretation: A+ (≥95%), A (90–94.9%), B (85–89.9%), and C (<85%). These grades are intended as descriptive summaries rather than strict accept–reject criteria, reflecting the expected variation across item types. Higher agreement was typically observed for demographic and behavioral items, while attitudinal and normative items showed greater dispersion, consistent with known properties of survey response behavior.

4.2 Results

Across 19 survey items spanning attitudes, behaviors, and demographic characteristics, the synthetic population produced response distributions that closely aligned with established real-world survey benchmarks. Overall agreement between simulated and observed distributions was high, with an average CDD agreement of 88.8%. Most items achieved A or B grades, indicating strong correspondence at the distributional level (Table 3).

Importantly, deviations between real and simulated distributions were rarely extreme and tended to preserve overall shape and ordering, even where agreement was lower. This suggests that synthetic populations capture meaningful population-level patterns rather than producing arbitrary or noisy responses.

Taken together, these results indicate that synthetic populations can generate credible, population-level survey signals across a wide range of domains. While not a substitute for real-world data collection, they provide a robust foundation for early-stage insight generation, question testing, and scenario exploration, enabling organizations to enter fieldwork with clearer hypotheses and better-designed instruments.

Table 3. Comparison of real-world and synthetic survey response distributions with agreement scores

Conclusions

This study demonstrates that synthetic populations grounded in large language models can reproduce real-world survey response distributions with a high degree of fidelity across a diverse set of topics. Across demographic, behavioral, and attitudinal items, simulated responses showed strong alignment with established survey benchmarks, indicating that synthetic populations are capable of capturing meaningful population-level patterns rather than producing arbitrary or generic outputs.

The results also highlight important boundaries. Agreement was highest for structurally stable variables, such as demographics and consumption behaviors, while greater variation was observed for attitudinal and normative questions—an expected pattern that mirrors the inherent sensitivity of such items in traditional survey research. This reinforces the view that synthetic surveys should not be treated as replacements for real-world data collection, but as complementary tools that are particularly valuable in the early stages of research, strategy, and product development.

Taken together, these findings suggest that synthetic populations can meaningfully reduce the cost, time, and friction associated with survey-based insight generation. By enabling rapid hypothesis testing, iterative question design, and early scenario exploration, they help organizations enter real-world fieldwork with sharper questions and clearer priorities. In this way, synthetic populations support a shift from episodic surveying toward more continuous, informed, and adaptive decision-making.

About the team behind kansa.io and call to action

We are a Helsinki-based company of psychologists and IT professionals exploring new methods for understanding populations through data and simulation. Our work bridges behavioral science and AI for more agile, evidence-based decision-making.

We're looking for research partners and organizations interested in testing the next version of our model. Reach out if you'd like to collaborate.