Test Data

Test Data and AI: What Makes Good Test Data?

In this series of articles we’re going to be talking about how to use LLMs to generate synthetic data for QA testing, starting with the basics of test data, then moving on to generation methods, and finally looking at examples for generating test data for the purpose of validating LLM products.

But let’s start at the beginning - in this article we’re going to talk about how to use synthetic test data more generally, what makes good or bad test data, and we’ll also look at some traditional QA methodologies and how test data can inform them.

What is Synthetic Test Data and Why Use it?

Synthetic data refers to any machine-generated data that can be used to execute test cases or mock a production environment scenario. This includes data produced by LLMs, procedural data, and human curated or created data generated outside of production. Of course, production data is incredibly valuable for testing, and when it’s possible to use it, it should be used - but often this is not possible, legal or scalable. Generating production data can also be an expensive process for a new feature or product since you need to hire beta testers. Synthetic data also has some other advantages other than cost.

First and foremost, your test data should be accurate and representative of real world scenarios.

, and test data (synthetic or otherwise) often carries an implicit bias from the time of creation - this also applies to training data for ML tools, so using LLMs to generate test data is also very prone to this.

This famous blog article by Patrick McKenzie “

Falsehoods Programmers Believe About Names

”  may have been published over a decade ago, but it remains relevant today – I often point junior QA Engineers to it if they’re working with forms or databases that hold personal information.

This screenshot shows the first 18 entries (out of 40) in his list of falsehoods, but I recommend reading the 

It sounds quite simple, but you would be surprised how often I’ve seen web platforms that had issues with - for example - diacritics. This isn’t exclusive to small websites - I have seen this in world-leading B2B service platforms too!

This is what I mean when I say 1) Real world data is messy, and 2) Data almost invariably carries an implicit bias.

Second, your test data should by easy to access, update, and integrate into automated testing systems. Depending on the type of data you use, CSV might be sufficient, but for more complex testing you might need something heavy duty. It’s important that as product features and requirements change (including legal compliance shifting with regulations) your data is kept up-to-date.

Last, test data should be diverse and exist in suitable volume to cover critical test surfaces - this includes edge cases and invalid input tests. Though data for negative testing should be clearly marked and isolated as such.

Let’s look at two traditional QA testing methods and how they might inform the selection of test data.

Equivalence Class Partitioning (ECP) is a method of defining test vectors or partitions of input data which can in turn be crafted into distinct test cases. In other words, grouping test data into equivalent sets for the purpose of testing. Thus, using test data from two members of the same equivalence class in a given context amounts to running the same test case.

Each partition of test data covers one area of functionality or path, and many partitions are needed to cover the full test surface of the product - including negative tests and edge cases.

For deterministic programs, this helps you eliminate waste by not running equivalent test cases. For non-deterministic applications like LLM, it helps you group test cases together to assess performance in a particular domain. In both scenarios, defining equivalence classes can help you understand your test surface and ensure that all scenarios are covered sufficiently.

Let’s say we’re testing a registration form that takes a name, phone number, address, email, and password, with the phone number being an optional field. We can define a really basic test set of two example users for our testing:

Do you see the problem here? The names, emails, address, phone, and password all share the same format. Testing your form with these two pieces of input data is effectively an equivalent test as far as your test surface is confirmed.

To have distinct test cases, the data needs to be varied - and to know 

 to vary your test data you need to understand the product you’re trying to test. It’s important to know what qualities make your test data elements different from one another.

 This example is pretty simple, so it should be quite easy to define some different classes of input

Understanding which partitions are important is key in determining what to prioritise what you test. Here we’re not just asking what happens if a user has an invalid phone number or an invalid password, but what happens if 

For more complex applications, especially for something like an LLM, having an understanding of these different classes is crucial. You can define your classes based on multiple complex criteria, for example by user profile (”these are coding questions a junior developer would ask”) or by goal (”these test prompts are designed to evaluate the model’s ability to summarize longer texts”).

Since this series of articles (eventually) focuses on using AI to generate test sets, I want you to think about the kind of data you need and how to partition it - otherwise you risk generating a lot of equivalent test inputs.

Boundary Value Analysis (BVA) is another software testing technique that complements ECP. ECP defines a topology based on test vectors and partitions different components of the product, BVA focuses on testing what happens at the ‘boundaries’ of those partitions. 

Think of the way we mixed different equivalence classes together in the example dataset above - we have equivalence classes for test inputs where the user has an invalid phone number, and another where they have an invalid password. What does the boundary of these inputs look like? A user that has both an invalid phone number and an invalid password. This scenario is more likely to produce a bug than either of those equivalence classes tested in isolation.

The idea is that we are more likely to have issues at the boundaries of our test surface. Let’s take another example - a clock app where a user can enter a time and set the clock to “tick” in real-time.

ECP would give us groupings that focus on hours, minutes, dates etc - BVA encourages us to look at the boundaries between them. In this example, we would want to test the relationship between seconds and minutes, or the relationship between minutes and hours etc.

This would include test cases like: setting the seconds and minutes to 0, 59, 60, 61, -1 etc. – values that exist at the boundaries.

In the context of LLMs, these boundaries are harder to define. But take a minute to think:

I’m curious what you come up with - and I’m sure you came up with something - so feel free to mention your ideas in the comments.

The right answer depends on what you want your LLM product to do, of course, but obvious contenders for BVA test cases include context switching in multi-turn, layered requests that touch multiple domains.

Today we discussed what synthetic data is, why it’s used, and the issues that one can encounter when using it. If there’s one thing I want you to take from this article is that data - synthetic or otherwise - is messy and riddled with bias. Keeping this in mind when defining test sets if going to make you a better tester and improve your product quality.

Then we looked at ECP and BVA, and - I hope - emphasised the importance of understanding your test surface and partitioning your test data, this helps you avoid waste, aggregate tests, and ensure you have the best coverage possible.

In our next article, we’re going to talk about how to generate test data using LLMs - I know you probably wanted to skip to the good stuff, but I assure you that understanding what makes good test data is going to be absolutely crucial when doing this, and it’s going to help you avoid a lot of issues going forwards.

<h1 data-node-type="titlePlaceholder" class="titlePlaceholder">Test Data and AI: What Makes Good Test Data?</h1>In this series of articles we’re going to be talking about how to use LLMs to generate synthetic data for QA testing, starting with the basics of test data, then moving on to generation methods, and finally looking at examples for generating test data for the purpose of validating LLM products.But let’s start at the beginning - in this article we’re going to talk about how to use synthetic test data more generally, what makes good or bad test data, and we’ll also look at some traditional QA methodologies and how test data can inform them.<h2 class="heading-anchor-text" id="what-is-synthetic-test-data-and-why-use-it">What is Synthetic Test Data and Why Use it?<a class="d-inline heading-anchor" title="Direct link to heading" href="#what-is-synthetic-test-data-and-why-use-it">#</a></h2>Synthetic data refers to any machine-generated data that can be used to execute test cases or mock a production environment scenario. This includes data produced by LLMs, procedural data, and human curated or created data generated outside of production. Of course, production data is incredibly valuable for testing, and when it’s possible to use it, it should be used - but often this is not possible, legal or scalable. Generating production data can also be an expensive process for a new feature or product since you need to hire beta testers. Synthetic data also has some other advantages other than cost.<h3 class="heading-anchor-text" id="why-use-synthetic-data">Why Use Synthetic Data?<a class="d-inline heading-anchor" title="Direct link to heading" href="#why-use-synthetic-data">#</a></h3><ol><li>While real production data is more reflexive of usage, human data may have privacy restrictions and/or identifiable data which makes it illegal to use in testing. That said, your synthetic data should reflect production data as much as possible. Note that production data is often not suitable for testing new features.</li><li>Synthetic data can be much more structured (i.e. you can choose its structure when generating) and thus simpler to integrate into testing - but beware of being too beholden to structure, real world data is messy.</li><li>In domains where data is sparse or hard to access, e.g. some domains in science or medicine, synthetic data can augment an existing dataset, or it can be used to cover negative test cases</li><li>Real world data and synthetic data both have a tendency to carry bias - i.e. be more tailored to certain user profiles and misrepresent certain classes of customer. Being mindful with synthetic data generation can help reduce this bias.</li></ol><h2 class="heading-anchor-text" id="features-of-good-test-data">Features of Good Test Data<a class="d-inline heading-anchor" title="Direct link to heading" href="#features-of-good-test-data">#</a></h2>First and foremost, your test data should be accurate and representative of real world scenarios.Real world data is messy, and test data (synthetic or otherwise) often carries an implicit bias from the time of creation - this also applies to training data for ML tools, so using LLMs to generate test data is also very prone to this.Let me give you an example:This famous blog article by Patrick McKenzie “<a href="https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/" target="_blank" rel="noopener noreferrer nofollow">Falsehoods Programmers Believe About Names</a>” may have been published over a decade ago, but it remains relevant today – I often point junior QA Engineers to it if they’re working with forms or databases that hold personal information.This screenshot shows the first 18 entries (out of 40) in his list of falsehoods, but I recommend reading the <a href="https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/" target="_blank" rel="noopener noreferrer nofollow">full post</a>.<figure class="photo figure-position-inline"><img src="https://s3.amazonaws.com/assets.fullstack.io/n/20250505140751645_image.png" attachmentid="5f2c3107-ef27-467d-bdaa-dc90350acce9" contenteditable="false" data-width="674" data-height="578"><div></div></figure>It sounds quite simple, but you would be surprised how often I’ve seen web platforms that had issues with - for example - diacritics. This isn’t exclusive to small websites - I have seen this in world-leading B2B service platforms too!This is what I mean when I say 1) Real world data is messy, and 2) Data almost invariably carries an implicit bias.Second, your test data should by easy to access, update, and integrate into automated testing systems. Depending on the type of data you use, CSV might be sufficient, but for more complex testing you might need something heavy duty. It’s important that as product features and requirements change (including legal compliance shifting with regulations) your data is kept up-to-date.Last, test data should be diverse and exist in suitable volume to cover critical test surfaces - this includes edge cases and invalid input tests. Though data for negative testing should be clearly marked and isolated as such.Let’s look at two traditional QA testing methods and how they might inform the selection of test data.<h2 class="heading-anchor-text" id="what-is-equivalence-class-partitioning">What is Equivalence Class Partitioning?<a class="d-inline heading-anchor" title="Direct link to heading" href="#what-is-equivalence-class-partitioning">#</a></h2>Equivalence Class Partitioning (ECP) is a method of defining test vectors or partitions of input data which can in turn be crafted into distinct test cases. In other words, grouping test data into equivalent sets for the purpose of testing. Thus, using test data from two members of the same equivalence class in a given context amounts to running the same test case.Each partition of test data covers one area of functionality or path, and many partitions are needed to cover the full test surface of the product - including negative tests and edge cases.For deterministic programs, this helps you eliminate waste by not running equivalent test cases. For non-deterministic applications like LLM, it helps you group test cases together to assess performance in a particular domain. In both scenarios, defining equivalence classes can help you understand your test surface and ensure that all scenarios are covered sufficiently.Let’s say we’re testing a registration form that takes a name, phone number, address, email, and password, with the phone number being an optional field. We can define a really basic test set of two example users for our testing:<figure class="photo figure-position-inline"><img src="https://s3.amazonaws.com/assets.fullstack.io/n/20250505140838637_image%201.png" attachmentid="1e653b16-3869-4d5e-aa22-f053a89e70f3" contenteditable="false" data-width="681" data-height="78"><div></div></figure>Do you see the problem here? The names, emails, address, phone, and password all share the same format. Testing your form with these two pieces of input data is effectively an equivalent test as far as your test surface is confirmed.To have distinct test cases, the data needs to be varied - and to know how to vary your test data you need to understand the product you’re trying to test. It’s important to know what qualities make your test data elements different from one another. This example is pretty simple, so it should be quite easy to define some different classes of input<ul><li>Names that follow different conventions or that have diacritics/non-standard characters</li><li>Addresses outside the US or with country/postcode info (in a real form you would probably have these as separate fields, but we’re keeping it simple here for demonstrative purposes)</li><li>Optional fields left empty</li><li>Phone numbers with extensions and non-numeric values</li><li>Passwords that are invalid - too long or too short, lacking special characters etc. (arguably all the different ways a password could be invalid could be separate classes)</li></ul>Now let’s expand the dataset:<figure class="photo figure-position-inline"><img src="https://s3.amazonaws.com/assets.fullstack.io/n/20250505140856475_image%202.png" attachmentid="80ee066f-ddce-4822-8bc6-55eb1a2994b0" contenteditable="false" data-width="956" data-height="219"><div></div></figure>Understanding which partitions are important is key in determining what to prioritise what you test. Here we’re not just asking what happens if a user has an invalid phone number or an invalid password, but what happens if both are invalid?For more complex applications, especially for something like an LLM, having an understanding of these different classes is crucial. You can define your classes based on multiple complex criteria, for example by user profile (”these are coding questions a junior developer would ask”) or by goal (”these test prompts are designed to evaluate the model’s ability to summarize longer texts”).Since this series of articles (eventually) focuses on using AI to generate test sets, I want you to think about the kind of data you need and how to partition it - otherwise you risk generating a lot of equivalent test inputs.<h2 class="heading-anchor-text" id="what-is-boundary-value-analysis">What is Boundary Value Analysis?<a class="d-inline heading-anchor" title="Direct link to heading" href="#what-is-boundary-value-analysis">#</a></h2>Boundary Value Analysis (BVA) is another software testing technique that complements ECP. ECP defines a topology based on test vectors and partitions different components of the product, BVA focuses on testing what happens at the ‘boundaries’ of those partitions. Think of the way we mixed different equivalence classes together in the example dataset above - we have equivalence classes for test inputs where the user has an invalid phone number, and another where they have an invalid password. What does the boundary of these inputs look like? A user that has both an invalid phone number and an invalid password. This scenario is more likely to produce a bug than either of those equivalence classes tested in isolation.The idea is that we are more likely to have issues at the boundaries of our test surface. Let’s take another example - a clock app where a user can enter a time and set the clock to “tick” in real-time.ECP would give us groupings that focus on hours, minutes, dates etc - BVA encourages us to look at the boundaries between them. In this example, we would want to test the relationship between seconds and minutes, or the relationship between minutes and hours etc.This would include test cases like: setting the seconds and minutes to 0, 59, 60, 61, -1 etc. – values that exist at the boundaries.In the context of LLMs, these boundaries are harder to define. But take a minute to think:<ul><li>What equivalence classes apply to LLM testing?</li><li>What do the boundaries between those classes look like?</li></ul>I’m curious what you come up with - and I’m sure you came up with something - so feel free to mention your ideas in the comments.The right answer depends on what you want your LLM product to do, of course, but obvious contenders for BVA test cases include context switching in multi-turn, layered requests that touch multiple domains.<h2 class="heading-anchor-text" id="conclusion">Conclusion<a class="d-inline heading-anchor" title="Direct link to heading" href="#conclusion">#</a></h2>Today we discussed what synthetic data is, why it’s used, and the issues that one can encounter when using it. If there’s one thing I want you to take from this article is that data - synthetic or otherwise - is messy and riddled with bias. Keeping this in mind when defining test sets if going to make you a better tester and improve your product quality.Then we looked at ECP and BVA, and - I hope - emphasised the importance of understanding your test surface and partitioning your test data, this helps you avoid waste, aggregate tests, and ensure you have the best coverage possible.In our next article, we’re going to talk about how to generate test data using LLMs - I know you probably wanted to skip to the good stuff, but I assure you that understanding what makes good test data is going to be absolutely crucial when doing this, and it’s going to help you avoid a lot of issues going forwards.Until next time!

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

Test Data and AI: What Makes Good Test Data?

Tags

Tools

Responses (0)

Masterclasses

Tutorials

Fullstack React with TypeScript