            On the Evolution of Python Test Cases into
                      Property-based Tests
                   Cindy Wauters                           Ruben Opdebeeck                                  Coen De Roover
             Software Languages Lab                    Software Languages Lab                          Software Languages Lab
             Vrije Universiteit Brussel                Vrije Universiteit Brussel                      Vrije Universiteit Brussel
                Brussels, Belgium                         Brussels, Belgium                               Brussels, Belgium
            cindy.suzy.wauters@vub.be               ruben.denzel.opdebeeck@vub.be                       coen.de.roover@vub.be



   Abstract—Conventionally, unit tests exercise their unit under       Scala2 , Python3 and JavaScript4 . Proposing domain-specific
test on predetermined input to validate its behavior. The advent       input generation strategies and invariant properties, research
of property-based testing frameworks has led to unit tests that        initiatives have brought property-based testing to challenging
exercise the unit under test on randomly-generated inputs, for
which the unit’s behavior has to satisfy developer-specified invari-   application domains such as telecom software [2], distributed
ant properties. The promise of increased coverage and stronger         synchronization services [19], neural networks [33], and cyber-
validation may lead developers to evolve existing conventional         physical systems [13].
unit tests into property-based ones. This paper reports on the            Despite these successes, property-based tests often consti-
results of an empirical study of 257 property-based unit tests         tute only a fraction of the test suite. According to a 2023
(PBT) from 64 open-source repositories that have evolved from a
conventional unit test. The study examines each PBT-introducing        JetBrains survey [22], the most common PBT framework
commit for change patterns, and for the type of features of            for Python, Hypothesis, was only used by 5% of Python
property-based testing frameworks that are adopted. Next, it           developers. To bring the benefits of property-based testing
investigates the impact of PBT adoption by running test cases to       to more projects, researchers have proposed tool support for
compute changes in code coverage and test failures. Finally, the       creating property-based tests [16], [38]. However, there is a
study investigates the evolution of the newly-introduced PBT by
comparing its first to its most recent version. The study’s findings   gap in the research when it comes to the evolution of existing
include that 37 out of 257 evolutions required no changes to the       test cases into property-based tests. Therefore, in this paper, we
body of the test. When changes are required, these are most            investigate these evolutions, their impact, and the maintenance
likely to target setup code followed by oracle code. Over half of      of the evolved test cases. By doing so, we aim to provide
the studied PBTs use off-the-shelf input generators, yet see an        valuable insights for testers, tool builders, and researchers
improvement in code coverage by an average of 4 statements.
Additionally, for 5 test cases, the PBT found a counterexample         alike. With this paper, we make the following contributions:
where the original unit test passed. Finally, 80% of the evolved          • We collect a dataset of 257 test case evolutions across
test cases are still present in the current version of the project.          64 open-source GitHub repositories written in Python.
   Index Terms—Empirical study, software testing, property-
based testing, code coverage                                                 Our data collection targets Python projects as, at the time
                                                                             of writing, it is one of the most popular programming
                       I. I NTRODUCTION                                      languages [21]. We provide a replication package [40]
   Conventionally, unit tests take an example-based form in                  containing the resulting dataset and the implementation
which the unit under test is exercised on predetermined input.               of all subsequent analyses.
For a unit test that should be run with several inputs, param-            • An empirical analysis of the changes required to rewrite

eterized unit testing [35] frameworks support substituting the               a conventional unit test into a property-based test, as well
example input by a parameter for which the developer provides                as whether test cases evolved afterwards.
a predetermined collection of values. When enumerating these              • An empirical analysis of the impact of these evolutions

input values becomes intractable, property-based testing [6]                 on the system under test in terms of changes in code
frameworks support generating them randomly according to                     coverage and test failures.
a developer-specified input generation strategy. For each of              The remainder of this paper is structured as follows. Sec-
the generated inputs, the property-based test (PBT) will check         tion II provides a motivating example and introduces the
that the behavior of the unit under test satisfies a developer-        research questions. Section III details the dataset of real-world
specified invariant property.                                          test evolutions we will be working with. Then, Section IV,
   The QuickCheck framework [6] for the Haskell program-               Section V, and Section VI present our research questions. For
ming language was the first to support property-based testing.         each of the three research questions, we detail the research
Since then, more than 35 property-based testing tools [18]
have been introduced for various languages, such as Java1 ,              2 https://github.com/typelevel/scalacheck
                                                                         3 https://github.com/HypothesisWorks/hypothesis
  1 https://github.com/jqwik-team/jqwik                                  4 https://github.com/dubzzz/fast-check
def test_encode_decode():                                              However, it can be difficult for developers to come up
    txt = "hello world"                                             with invariant properties to test [14]. The Hypothesis Quick-
    assert decode(encode(txt)) == txt                               start [43] hints at existing parameterized unit tests (PUTs) and
                                                                    existing sources of randomness in a test suite for inspiration.
Listing 1: An example-based test for encode and decode              However, to our knowledge, no research has looked into real-
                                                                    world test case evolutions such as the one described above. In
import pytest                                                       this paper, we answer the following research questions:
                                                                       • RQ1: How do conventional unit tests evolve into
testdata = ["hello world", "test -", "(/test)"]
@pytest.mark.parametrize("txt", testdata)                                 property-based tests? In this research question, we
def test_encode_decode(txt):                                              investigate real-world commits for the changes required
    assert decode(encode(txt)) == txt                                     to rewrite a conventional example-based unit test or
                                                                          parameterized unit test into a property-based one.
Listing 2: A parameterized unit test for encode and decode             • RQ2: What is the impact of adopting property-based
                                                                          testing? To investigate the impact of these test case
                                                                          evolutions, we compute the difference in code coverage
method, the findings, and present an objective analysis of                before and after the rewrites as well as the test failures.
these findings. Section VII discusses the actionable insights          • RQ3: How are property-based tests maintained?
gained from the findings across the research questions. Here              To assess the effort required to maintain the resulting
we also describe the threats to validity and their mitigation. In         property-based tests, we investigate whether there are
Section VIII we position our work and compare it to current               changes made to the test cases after their rewriting into
research into property-based testing.                                     a property-based test, or if they are removed.

 II. M OTIVATING E XAMPLE AND R ESEARCH Q UESTIONS                                           III. DATASET
                                                                       In order to study real-world unit test evolutions, we first
   Test cases may evolve over time. To illustrate such an evo-
                                                                    need a dataset of open-source repositories that contain at least
lution, consider an encode function and its inverse decode.
                                                                    one property-based unit test (PBT) that started as either an
Listing 1 depicts an example-based unit test (EBT) checking
                                                                    example-based unit test (EBT) or as a parameterized unit test
that decoding the encoding of "hello world" results in
                                                                    (PUT). Our data collection will start from Python repositories
the same string. This unit test validates the behavior of the
                                                                    that have a dependency on the Hypothesis [27] framework
unit under test for a single developer-provided example string.
                                                                    for property-based testing. While other empirical studies have
   Using Pytest [23], the most common Python testing frame-         collected such repositories before (e.g., [32], [39]), we collect
work, the example-based unit test from Listing 1 can be gener-      our own to ensure the dataset contains ample instances of test
alized into a parameterized unit test [36]. Listing 2 depicts an    case evolutions.
example generalization in which a parameter txt substitutes            1) Data collection: We first use the GitHub API to find
for the earlier example input. Three possible values have been      open-source Python repositories that have a dependency on
provided for the parameter, thereby validating the behavior on      Hypothesis. In order to exclude toy projects, we only include
predetermined strings that contain special characters.              projects that have at least three stars. We also filter out larger
   Using Hypothesis [26], [27], the most popular property-          projects with more than 10 stars. While these projects can
based testing framework for Python, the test case can be            be valuable to study, they are also difficult for an outsider
generalized further. Listing 3 depicts the resulting property-      to understand and reason about. We do not consider those
based test, specifying that the composition of decoding after       projects suitable for a deep manual investigation, as required
encoding is the identity operation for all strings generated        for this empirical study section. In total, we considered 1388
using st.text. Hypothesis will automatically generate mul-          repositories that have a dependency on Hypothesis and satisfy
tiple strings, each of which will be encoded and decoded and        our inclusion criteria.
compared to the original string. This way, the property-based          Next, we use PyDriller [34] to find all PBTs in a repository
unit test might uncover defects for edge cases the developer did    that were once either an EBT or a PUT, and are now a PBT.
not enumerate in the example-based nor in the parameterized         To this end, we rely on the presence of @given decorators in
version of the unit test.                                           a file with a PBT. The GitHub API alone does not suffice, as
                                                                    noted by prior empirical studies into real-world Hypothesis
from hypothesis import given, strategies as st                      usage (e.g., [8], [32], [39]), due to the existence of other
                                                                    libraries named Hypothesis. The @given decorator in a test
@given(st.text())                                                   file, in contrast, denotes the entry point of a Hypothesis-based
def test_encode_decode(txt):                                        PBT. If none exists, the repository does not contain a PBT, and
    assert decode(encode(txt)) == txt                               is excluded. If we find at least one PBT in a file, we investigate
                                                                    all previous versions of the file. In case the PBT exists in the
 Listing 3: A property-based test for encode and decode             initial commit of the file, PBTs have always existed in that
                # LOC Python   # evolved test cases   # commits     1      def test_tolerance_sign_EBT():
    Mean            7575              4.02              923         2          r = R.from_min_max(3, 2)
    Std Dev.       17147              6.66              1996        3          assert r.tolerance == 0.2
                                                                    4
    median          2607                2               294
                                                                    5
    1st Quar.       1178                1               121.5       6      @given( unit=st.sampled_from((U, I, P, R)),
    3rd Quar.       5346                4               680.5       7              a=st.floats(allow_nan=False),
    Min             164                 1                15         8              b=st.floats(allow_nan=False))
                                                                    9      def test_tolerance_sign_PBT(unit, a, b):
    Max            109516              39              10274
                                                                    10         eu = unit.from_min_max(a, b)
                                                                               assert eu.tolerance >= 0
TABLE I: Characteristics of the 64 repositories investigated 11
in this paper. Lines of Code computed by Cloc [10].
                                                                         Listing 4: A real test case before and after becoming a PBT.
                                                                         Some simplifications applied for readability. Case taken from
                                                                         https://github.com/wese3112/eecalpy
file, and we do not consider this file relevant for investigating
test case evolution. In case there is a previous version of the
file in which the PBT did not exist, while it does in a newer
                                                                         end, we categorize all changes to the body of the test case into
version, we manually investigate the last version before the
                                                                         the following categories (four per type):
introduction with the first version after the introduction. We
                                                                            • Changes to the oracle code: non-semantic changes, se-
perform this step manually to ensure that we also retrieve
evolutions of tests that have been renamed. Additionally, we                  mantic changes, additional oracles, removal of oracles.
                                                                            • Changes to the setup code: non-semantic changes, se-
take note of whether the transformation in question is from an
EBT to a PBT, or from a PUT to a PBT. We recognize PUTs as                    mantic changes, additional setup code, removal of setup
unit tests adorned with the @pytest.mark.parametrize                          code.
decorator. This renders their recognition purely syntactic.                 To delimit test oracle code, we follow the definition of Barr
   2) Characteristics of the dataset: Out of the 1388                    et al. [3]. It is the part of the code that decides whether
Hypothesis-dependent GitHub repositories initially discovered,           the system under test behaves as expected. This does not
575 have at least one PBT present. Upon filtering for test case          necessarily mean that only (nor all) the code in an assert
evolutions, we identified 64 repositories that feature at least          statement is part of the oracle code. Code that defines the
one evolution from an EBT or PUT into a PBT —amounting                   oracle is included too.
to 257 PBT before-after mappings. For each PBT, we have                     As setup code, we consider those parts of the body of
a mapping from the test before the PBT introduction to the               the test case that create or retrieve values through which the
test after the PBT introduction. For a minority of PBTs, the             behavior of the system under test will be validated. In other
test before its introduction was split into multiple PBTs. On            words, it sets up the system under test. This is similar to the
the other hand, some tests were merged into a single PBT. In             definition of a test fixture by Meszaros [28]. This is sometimes
these cases, we consider the same PBT multiple times (each               also known as test context. However, we only consider the
time with a different mapping). Our 257 before-after mappings            code inside the body of the test case.
therefore consist of 252 unique tests before PBT introduction               As a non-semantic change to the body of a test case,
and 250 unique PBTs. Of the total 257 test case evolutions,              we consider straightforward refactorings such as changing a
182 test cases follow the EBT to PBT evolution, whereas 75               constant to a value generated by the input generation strategy.
follow the PUT to PBT evolution. Table I presents summary                For semantic changes, in contrast, we consider anything that
statistics of the distribution of the data across the most recent        changes the semantics of a test case (e.g., extra calculations
version of the repositories. Included are the lines of code in           within existing code). An addition or removal of setup code
Python (blank lines and comments not included), the number               means any lines added removed, that can be considered setup
of investigated test case evolutions, and the total number of            code. An addition or removal of oracle code is the addition
commits per repository.                                                  of, for example, a new assert statement, or the removal of an
                                                                         old one.
 IV. RQ1: H OW DO CONVENTIONAL UNIT TESTS EVOLVE                            A test case can evolve into a PBT through several of
            INTO PROPERTY- BASED TESTS ?                                 these types of changes. To illustrate, consider Listing 4 which
                                                                         depicts one of the test cases in our dataset before and after
   RQ1 investigates how PBTs can evolve out of conventional              evolving into a PBT. Line 2, as well as the r.tolerance
test cases. To this end, we manually investigate the 257 afore-          in line 3 is considered the setup code: these lines define
mentioned real-world before-after mappings in the dataset.               what will be tested later. The assert statement, excluding
                                                                         r.tolerance, is oracle code. This test case changes in two
A. Research Method
                                                                         different ways. First, a non-semantic change to the setup code:
   1) Changes to the test case body: First, we consider how              on line 10 and 11, we see a renaming of r to eu. Additionally,
test cases are updated when they evolve into a PBT. To this              hardcoded values R, 3, and 2 are replaced with the parameters
that will now be generated data. Second, a semantic change           •   Control: assume, @note, @event, target, used in
in the oracle code: as the values are not hardcoded anymore,             the test case to modify how it is treated by the test runner,
an exact expected value is not known anymore, and the oracle             or to further document it.
needs to be updated. The change from == to >= makes this a            3) Input generation strategies: As the input generation
semantic change.                                                   strategy is an important part of every PBT, we also categorize
   Furthermore, it is also possible that the test case changed     the strategies chosen at migration time as follows:
beyond recognition when it was rewritten into a PBT. During           • Simple: strategies that generate standard values (e.g.,
our manual investigation, we mark such test cases as replaced,           integers, floats, strings), and collections thereof (e.g., lists,
rather than updated, if we cannot recognize it within the PBT            tuples). Strategies that compose standard values into a
anymore. Some large-scale studies into test evolution have               user-defined data type are classified in the next category
used thresholds on automatically-computed similarity metrics             instead.
instead (e.g., 66% used in [5] and [11]), but these make less         • Custom, with reuse: strategies that are custom to the
sense when applied to small test cases instead of large files.           project, but reused across multiple test cases.
We therefore rely on human judgement of the labelers instead.         • Custom, no reuse: strategies that are custom to the
If the labelers cannot identify similar code in the test case            project, but not reused across multiple test cases
before and after the evolution, and especially in their assert        • Other: usage of third-party libraries for input generation.
statements, the test is tagged as different test cases.                  For example, to generate NumPy values.
   The labeling was performed individually by the first and           We assign each test case one of these four categories.
second authors of this paper. After a first round of labeling,     Custom strategies encompass cases where developers define
we assessed inter-rater agreement using Cohen’s Kappa [7]          their own, complex, input generators for the test cases. We also
calculated for each category, obtaining a mean Kappa of 0.475,     consider any usage of builds and from_type in this cate-
signifying “moderate agreement” [25]. To avoid systematic          gory, as developers use them to generate custom user-defined
disagreements, the two labelers discussed and resolved dis-        data types. When a test generates both values of standard and
agreements on a random sample of 20% of the entire dataset to      user-defined data types, we consider its generation strategy
establish a shared understanding. For instance, a recurring dis-   custom overall.
agreement was the presence of setup code in assert statements.
After resolving these differences, the labelers individually       B. Results
performed a second round of labeling on the remaining 80%,            1) Changes to the test case body: We first discuss the
resulting in a mean Kappa of 0.727, signifying “substantial        changes that conventional unit tests undergo when rewritten
agreement” [25]. The remaining disagreements were discussed        into a property-based test. Figure 1 shows the results for both
among the two labelers to obtain consensus.                        EBTs and PUTs that evolved into a PBT.
   2) Features of the property-based testing framework that           For test cases that evolved from EBTs in our dataset, it is
are adopted: Property-based testing frameworks often offer         especially common (117/182) for the setup code to be changed
features to customize a test case. The @example decorator of       in a non-semantic way (for example, replacing a hardcoded
Hypothesis, for example, enables specifying that a value needs     value to instead refer to the input generation strategy, as
to be tested on every run of the test case. This can be used       shown in Listing 4). The second most common change is for
with known edge cases that the developer is aware of. The          the oracle code to be updated with a non-semantic change.
@settings decorator lets developers specify test budgets,          Other common changes are additions of oracle code, semantic
the maximum number of examples to be generated, etc. It is         changes to oracle code, and removal of oracle code or setup
also possible to reproduce test failures by re-using the random    code. Additionally, 12% of cases changed beyond recognition.
input generation seed that found the counterexample (e.g., the        Tests that evolve from PUTs are a bit different. Here,
input for which the property did not hold).                        we see fewer changes in the test cases overall. The most
   We track the use of these features at the time the PBT was      common change is the addition of oracle code, followed by
introduced. By doing so, we aim to discover whether devel-         the addition of setup code, and semantic changes to the oracle
opers are using them from the onset. We use the following          code. Notably, 35/75 test case evolutions from PUTs in our
categorization according to the Hypothesis API [42]:               dataset required no change at all in the body of the test case
   • Strategies: @given to generate inputs. The next para-         (in comparison to 2 stemming from EBTs). Many PUTs might
     graph categorizes the input strategies specified through      already be testing a property, especially if they contain no
     this decorator further.                                       hardcoded expected values in the parameter. This means that
   • Explicit inputs: @example, which enable the tester to         evolving them into a property-based test can be less labor-
     provide inputs to test each run.                              intensive for developers.
   • Reproducing inputs: @reproduce_failure and                       2) Features of the property-based testing framework that
     @seed, to reproduce the data from previous test               are adopted: The results are shown in Table II. All tests
     failures.                                                     specified at least one input generation strategy, as expected
   • Settings: @settings, to impose test budgets, a maxi-          from a PBT. Hypothesis itself sees this decorator as the
     mum number of examples to generate, etc.                      entry point of a PBT. Control (specifically assume), was
                                                                                                                                                             64.3%                                   PBT origin
                                                                                                                                                                                                  from example-based test
                     60                                                                                                                                                                           from parameterized test



                     50
  Percent of tests




                     40



                     30                                            29.1%


                                                                                                                         22.5%                                                      22.5%
                                    20.0%    20.3%
                     20
                          17.6%                                                                    17.0%
                                                          16.0%                                                                  16.0%
                                                                                                                                                     14.7%
                                                                                                                                                                      12.0%                              12.1%
                     10                                                         8.0%
                                                                                                             6.7%                        6.6%                                                   6.7%                6.7%


                     0
                                                                                        tic




                                                                                                                                                                              tic
                                                             tic




                                                                                                                                                                                                                          e
                                                                                                               e




                                                                                                                                                       tic




                                                                                                                                                                                                  e
                                    dd




                                                                                                                                   d
                                                                                                             ov




                                                                                                                                                                                                                         as
                                                                                                                                                                                                ov
                                                                                                                                 ad
                                                                                     an




                                                                                                                                                                           an
                                                          an




                                                                                                                                                     an
                                  :a




                                                                                                            em




                                                                                                                                                                                                                    tc
                                                                                                                                                                                            em
                                                                                   m




                                                                                                                                                                      em
                                                       em




                                                                                                                                                 em
                                                                                                                               p:
                               le




                                                                                                                                                                                                                     s
                                                                                se




                                                                                                        :r




                                                                                                                            t-u




                                                                                                                                                                                            r
                               c




                                                                                                                                                                                                                  te
                                                                                                                                                                       s
                                                   :s




                                                                                                                                                 s




                                                                                                                                                                                         p:
                            ra




                                                                               -




                                                                                                       le




                                                                                                                                                                    n-
                                                                                                                          Se
                                                                            on




                                                                                                                                                                                                                t
                                                                                                                                              p:




                                                                                                                                                                                      t-u
                           O




                                                  le




                                                                                                                                                                                                             en
                                                                                                       c




                                                                                                                                                                  no
                                                                                                                                           t-u
                                                                                                    ra
                                                                           :n
                                                  c




                                                                                                                                                                                    Se




                                                                                                                                                                                                             er
                                               ra




                                                                                                   O




                                                                                                                                                                 p:
                                                                                                                                         Se
                                                                       le




                                                                                                                                                                                                          iff
                                              O




                                                                                                                                                              t-u
                                                                      c




                                                                                                                                                                                                         D
                                                                   ra




                                                                                                                                                             Se
                                                                   O




Fig. 1: The changes example-based (blue) and parameterized (orange) unit tests undergo when evolving into a property-based
test. Each test case can undergo multiple changes from different categories.


                                                  From EBT                                    From PUT                      novice users of the framework, or because the test cases did
 Feature                                    # of test cases            %           # of test cases                 %        not yet have the time to evolve and be tailored to the project.
 Strategies                                       182              100                        75                 100        Therefore, we also investigate how the evolved test cases are
 Explicit inputs                                      4            2.20                       2                  2.67       maintained in Section VI-B2.
 Reproducing inputs                                   0                0                      0                    0           3) Input generation strategies: Table III depicts the results
 Settings                                          11              6.04                       8                  10.67      for our analysis of the input generation strategies used. The
 Control                                              2            1.10                       14                 18.67      majority of PBTs that evolve from an EBT have a simple input
                                                                                                                            generation strategy for standard data types. However, for 76 of
TABLE II: Features initially adopted from PBT framework.                                                                    the 120 simple cases (a majority), more refined configurations
                                                                                                                            are used of these simple input generators (such as min/max
                                                                                                                            bounds for integers). 60 of the EBTs that evolved into PBTs
the second most common feature, followed by @settings                                                                       now have a custom input generation, of which 24 are reused
(mostly imposing deadlines on the test cases) and explicit                                                                  multiple times across different test cases.
inputs (@example). The fact that we see no PBT using                                                                           When considering PBTs that have evolved from a PUT, just
failure-reproducing features is not surprising, as it is the first                                                          over half of the PBTs have a simple input generation strategy.
PBT-version of the test cases. They might not yet have run                                                                  Of those 43 PBTs, 35 refined configurations. All custom input
enough to want to reproduce a specific failure-inducing input.                                                              generation strategies are reused across multiple test cases. It is
   Compared to test cases that evolved from EBTs, test cases                                                                common for evolved PUTs from the same project to already
that evolved from PUTs are more likely to adopt these features                                                              share the same input parameter before their evolution into a
of the PBT framework. One reason for this could be that                                                                     PBT. In our dataset, the use of libraries for input generation is
the developers are already more familiar with more advanced                                                                 not common (only two instances). The only used library was
testing frameworks.                                                                                                         the Hypothesis NumPy extension5 .
   Corgozinho et al. [8] also performed a preliminary study                                                                    Several test cases already contained some form of random-
into commonly used features across 86 PBTs. Their dataset                                                                   ization before being rewritten into a PBT. Afterwards, this
does not consider the initial PBT introduction, and contains                                                                randomization is still reflected in the input generation strategy.
more well-established, big projects that make use of property-                                                              For example, if a test case previously contained a statement a
based testing. In their dataset, they observe a higher feature
adoption rate compared to ours. This could be because devel-                                                                  5 https://hypothesis.readthedocs.io/en/latest/reference/strategies.html#
opers evolving a PBT from an existing test might be more                                                                    hypothesis-numpy
                                From EBT      From PUT                 that define behavior in their test files, these should indeed
                                #      %      #      %                 be considered. For others, including test files can lead to a
           Custom, with reuse   36    19.78   32   42.67               wrong conclusion when the test cases themselves have shrunk
           Custom, no reuse     24    13.19   0      0                 or increased in size. We therefore report both on the absolute
           Simple               120   65.94   43   57.33               and relative coverage twice, once with the test files included
           Other                 2    1.10    0      0                 and once with the test files excluded.
                                                                          We were able to run 219 of the test cases before PBT-
TABLE III: Input generation strategies used within PBTs                introduction, and 219 test cases after PBT-introduction. How-
evolved from an EBT or a PUT.                                          ever, not all were able to obtain code coverage. In the end,
                                                                       we were able to run and obtain code coverage for 211 test
                                                                       case evolutions before and after the introduction of the PBT
= np.random.randint(1, 10), a strategy of the test                     (meaning a total of 422 test cases). To ensure the test cases ran,
case would become @given(st.int(1, 10)).                               some minimal changes to the code were needed. For example,
                                                                       one test case contained a broken import. Upon correcting this
  RQ1 Non-semantic changes to the setup code are es-                   import, the test case did run. However, we made sure to not
  pecially common in evolutions stemming from EBTs                     change the semantics of the system under test. In some cases,
  (117/182). For evolutions from PUTs, it is more common               tests failed but still resulted in some code coverage. Their
  to require no changes in the body of the test case at all            results are also included.
  (35/75). The addition of settings is the most common
                                                                          3) Test failures: Code coverage is not the only way to deter-
  feature (19/257) early on in the adoption of the PBT
                                                                       mine whether tests are effective. During the data collection of
  framework. Most test cases use simple input generation
                                                                       Section V-A2, not all test cases passed, and some test oracles
  strategies (163/257).
                                                                       threw an assertion error. For some test cases, this happened
                                                                       before their evolution into PBT, whereas for others, the PBT
       V. RQ2: W HAT IS THE IMPACT OF ADOPTING
                                                                       did not pass. We investigate how often this occurs, and the
              PROPERTY- BASED TESTING ?
                                                                       failed assertions in each.
  With research question two, we aim to find the impact
of introducing PBTs. We consider commit messages, code                 B. Results
coverage, and test failures of the 257 test evolutions.                   1) PBT-introducing commits: First, we report on our anal-
                                                                       ysis of the commit messages of the PBT-introducing commits.
A. Research Method                                                        Only eight commit messages mention anything positive or
   1) PBT-introducing commits: We investigate and report on            negative about Hypothesis or property-based testing (beyond
73 commit messages across 64 repositories that evolve at least         simply stating that they have started using it). One commit
one conventional test case into a PBT. We aim to uncover               message mentions improving code coverage to find more
reasons as to why the developer decided to evolve their test           errors across the code. Two commit messages mention the new
case, and their potential opinions on this change.                     PBTs failing (a bug was found, but not yet fixed). Finally, five
   2) Code coverage: We first explore whether code coverage            commit messages use the words better, trickier, or improved.
improves once a conventional test case has evolved into a                 Four commit messages also mentioned bug fixes of some
PBT. Improving coverage can be a goal for some PBT-                    sort, meaning the PBT was introduced at the same time as a
introducing developers, even though it does not necessarily            bug fix. This can indicate that the PBT at introduction might
improve test suite effectiveness [20]. Moreover, research has          have found a bug that the initial conventional test case did
proposed variants of PBT with that explicit aim (i.e., coverage-       not, after which the developer fixed the bug. Alternatively, the
guided PBT [24], [29], [30]).                                          developer may also be introducing bug fixes and PBTs at the
   We run each test case with statement coverage before                same time to ensure their most recent bug fix is tested more
and after the introduction of Hypothesis while measuring               in the future. We did not find any evidence in the commit
coverage using Coverage.py [4], a well-established Python              message supporting either hypothesis, but Section V-B3 looks
library for test coverage. We only compute the coverage for            into test case failures before and after the commit.
each individual test case, instead of for the entire test suite as a      2) Code coverage: To verify whether there is a difference
whole. For relative coverage (i.e., in percentage), we normalize       in coverage before and after the evolution into PBT, we use a
the executed statements by those in the files executed by the          Wilcoxon signed-rank test [41] as our data is non-parametric
test case rather than by those in the entire project. We opted         and paired (code coverage of a test case before and after the
for this design as coverage across all statements of the entire        PBT-introducing commit). We consider a p-value of <0.05
project would be very low for individual test cases. We also           statistically significant. The test results are listed in Table IV.
report on coverage in absolute numbers. This can be useful             For all 4 cases, we observe a statistically significant difference.
for test cases that lead to the execution of several files. Note          First, we look at the change in number of statements
that, by default, coverage.py considers all executed Python            covered. Figure 2 shows box plots for the distribution of the
files including the files that define the test case. For projects      absolute statement coverage before and after the test case
             Stmt (all)            % (all)                Stmt (no test)         % (no test)      coverage ∆   # Stmt (all)   % (all)   # Stmt (no test)   % (no test)
 p-value       0.006           2.497 × 10−20             1.477 × 10−13               0.045        Mean           +13.09       +2.31          +4.01            -0.82
                                                                                                  Std Dev.        27.79        6.06          16.81             5.5
TABLE IV: p-values of change in coverage, in terms of
                                                                                                  median           +11        +0.57           +0               +0
absolute and relative statement coverage.
                                                                                                  1st Quar.        +2         -0.06           +0              -1.26
                  total # ran tests        # passing          # failing      with coverage        3rd Quar.        +17        +2.46           +3             +0.28
 Before PBT                  219                  199             20                211           Min              -78        -10.09          -75            -21.90
 After PBT                   219                  207             12                211           Max             +223        +32.61          +77            +32.52

TABLE V: Number of passing and failing test cases. We could                                      TABLE VI: Changes in code coverage, both in terms of
not obtain coverage for all tests (as shown by the last column).                                 number of statements covered and change in the coverage
                                                                                                 percentage, for both all files and all files without test files
                                                                                                 included.

    2500
                                                                                                 evolution, taking into account all executed files (denoted “all”
                                                                                                 in the plot) and all executed files without test files (denoted
    2000                                                                                         “no test”). While we see a slight improvement in median
                                                                                                 and quartiles when all executed files are taken into account,
    1500
                                                                                                 the change becomes much more subtle when excluding the
                                                                                                 executed test files. One reason could be that property-based
                                                                                                 test cases execute more lines of test code. Even if the tests
    1000
                                                                                                 themselves are not necessarily longer, the inclusion of input
                                                                                                 generators can add more test code to cover —especially for
     500                                                                                         projects that have defined custom ones in test files.
                                                                                                    Figure 3 on the other hand shows the relative statement
                                                                                                 coverage in percentage. As mentioned before, by default,
       0
                                                                                                 coverage.py only takes into consideration files that have at
              Before (all)          After (all)          Before (no test)     After (no test)
                                                                                                 least one executed line of code, meaning files that are never
                                                                                                 covered by the test case are not considered. Here we also see
                                                                                                 an improvement when taking into account test files, but once
Fig. 2: Statements covered by test cases before and after PBT-
                                                                                                 they are removed from the data, we notice a slight decrease
introduction. The first two plots show coverage across all files,
                                                                                                 in code coverage on average.
the second two without test files included.
                                                                                                    Table VI shows the deltas for each of the four measure-
                                                                                                 ments. While the mean absolute number of covered statements
                                                                                                 without test files included does improve slightly (4.01), the
                                                                                                 relative coverage percentage goes down slightly. This could
     100                                                                                         be due to extra files being discovered, or more code being
                                                                                                 introduced during the PBT-introducing commit. Most notice-
      80                                                                                         ably, one project’s covered statements decreases by 75. Upon
                                                                                                 further investigation, the test case in question contained many
                                                                                                 assert statements. While the conventional test case before
      60
                                                                                                 PBT-introduction passed, Hypothesis found a counterexample
                                                                                                 on the first assert, thereby stopping the test case and not
      40                                                                                         executing the subsequent assert statements (leading to a drastic
                                                                                                 decrease in coverage). Additionally, 26 test cases from the
                                                                                                 same repository all had one additional change when evolved
      20
                                                                                                 into PBT: the removal of a call to the logging function (which
                                                                                                 was three statements), leading to a decrease of coverage of
       0                                                                                         indeed 3 statements for these 26 test cases.
             Before % (all)        After % (all)        Before % (no test)   After % (no test)      3) Test failures: Table V shows the number of tests failed
                                                                                                 before and after PBT-introduction. Two test cases did not pass
                                                                                                 as a conventional unit test, nor after having been rewritten into
Fig. 3: Coverage percentage by test cases before and after                                       a PBT. For 18 test cases, the initial conventional test did not
PBT-introduction. The first two plots show coverage across                                       pass, but the updated PBT did pass. In another ten cases, the
all files, the second two without test files included.                                           initial test passed but the resulting PBT did not.
   In terms of the before, one project with one evolution              •   Updates to the test case encompassing both semantic and
had a test case that resulted in an assertion error. The PBT-              non-semantic changes.
introducing commit also contained a bug fix. Another project            We make one exception for 39 test cases originating from
had three failing test cases that returned a type error. After the   one repository. In the most recent version of that repository,
introduction of the PBT, all three passed. However, the PBT-         most Python code has been removed, and is scheduled to be
introducing commit does not introduce a bug fix (meaning             added at a later stage. Because of this drastic rewrite affecting
the bug was in the test code itself). Another project had two        the latest version in the repository, we instead consider its most
failing test cases, with an attribute error and a type error. For    recent version with the test cases still present.
both, the PBT versions did pass. In one of the bigger projects          Additionally, we report on the survival rate of the test cases
in our dataset, with 16 evolved test cases, ten tests did not        by looking at the number of commits since introduction, or
pass before the evolution. In three cases this was because of        the number of commits between introduction and removal.
an assertion error (with the other seven not passing due to a           2) Are more parts of the PBT framework adopted?: As de-
value not being defined). All ten of these test cases passed after   velopers become more acquainted with property-based testing
being updated to a PBT. These results suggest that people are        in their projects, they might discover features unbeknownst
introducing PBTs when fixing known errors, perhaps to ensure         to them before. Therefore, similar to Section IV-A2, we
more thorough testing of those parts of the system.                  investigate whether the most recent version of their PBT
   When looking at the failing tests cases after the introduction    uses more features of the PBT framework. We use the same
of PBTs, we found five test cases for which Hypothesis               categories of features from the Hypothesis API [42] to classify
reported a counterexample for which the original test case           the additions. As all PBTs already had an input generation
passed. Further, as mentioned in Section V-B1, some commits          strategy when they were introduced, we investigate whether
did contain additional bug fixes. This indicates that PBTs can       this strategy has been updated in the meantime.
uncover bugs not found by conventional test cases validating
the same behavior, thus improving bug detection. For the             B. Results
other 5 non-passing PBTs, we found other errors (such as                1) Do Property-based Test Cases Change?: To answer this
ValueErrors), even though the original test case passed before.      question, we compare the most recent commit in the GitHub
                                                                     repository to the PBT-introducing commit. Table VII shows
  RQ2 8 PBT-introducing commit messages convey an                    the number of commits the test cases have survived (for those
  intent to improve test cases to improve coverage or find           that are still present in the current version of their project,
  bugs. We observed a statistically significant change in            i.e., the number of commits between introduction and the
  statement coverage, with on average an increase of 4               most current commit), or how many repository commits they
  statements. We also found evidence of PBTs being intro-            survived before their deletion. Most test cases in our dataset
  duced along with a fix for a bug in the code, and of PBTs          (110) are unchanged. Their corresponding repositories have
  finding counterexamples where the original corresponding           also enjoyed the fewest commits since the PBT-introducing
  conventional test passed.                                          one (with 99 on average, with a standard deviation of 205).
                                                                     Interestingly, one test case survived 1728 repository commits
       VI. RQ3: H OW ARE PROPERTY- BASED TESTS                       without undergoing any changes itself. A total of 90 test cases
                    MAINTAINED ?
                                                                     did undergo changes since their introduction, both semantics-
   We consider the most recent version of the 250 unique PBTs        preserving and semantics-affecting ones. We observe that these
from the in total 257 test case evolutions, and investigate the      test cases also survived more commits, with on average 642
changes (or removals) they have been subject to since.               commits and a higher standard deviation. This is expected as
                                                                     PBTs need to co-evolve with their system under test.
A. Research Method
                                                                        Finally, 50 PBTs were removed from the projects since
   For each test case, we compare the version resulting from         their initial introduction. These test cases also survived fewer
the PBT-introducing commit to the version in the most recent         commits; 352 on average. However, one test case survived
commit in its repository. We consider the following:                 3573 commits to the repository before being removed. Reasons
   1) Do Property-based Test Cases Change?: We investigate           cited for the deletion of these test cases were removal of the
whether the 250 PBTs in our dataset change over time, for            functions they tested (5), unexpected behavior (3), and the tests
instance in response to changes in the system under test. For        being slow (1).
each PBT, we manually compare against the same test case in             2) Are more parts of the PBT framework adopted?: Finally,
the most recent version of the GitHub repository. For each of        we consider the changes made to PBTs in terms of the parts of
our test cases, we consider the following:                           the PBT framework that are adopted. Table VIII describes the
   • No changes to the test case. The rest of the project and        results. In 65 test cases, the input generation strategies were
     other test cases might have changed, but the investigated       updated after their introduction. One repository in particular
     PBT has not.                                                    mentions doing so to improve the running time of the test
   • Removal of the test case. In this case, we also investigate     cases. Only one repository added an explicit input to test for on
     possible reasons why.                                           each run. Additionally, one test case added a seed decorator
                        No changes       Changes       Removal       easier to evolve unit tests that exercise the unit under test on
           Mean             99            642              352       inputs of standard data types (e.g., integers) into a PBT with
           Std Dev.       205.31         1450.79        967.80       such a strategy. Our findings should motivate practitioners to
           median           51            146               34       prioritize their generalization efforts on those unit tests first.
           1st Quar.        48              64              17          Finally, a study by Goldstein et al. [14] reports that PBT
           3rd Quar.        63            146               71       adopters have difficulty identifying the properties to test for.
           Min               1              17              7        Using existing test cases, rather than creating new ones, might
           Max             1728           5886           3573
                                                                     help. This is especially true for parameterized unit tests (PUT),
                                                                     and especially those that do not list expected outputs for every
TABLE VII: Number of commits since the introduction of the           given input, as those might already be testing a property (but
PBT. Data split for the test cases that did not change at all, for   not yet randomly generating input). Indeed, for 35/75 PBTs
those with changes, and for those removed in the meantime.           that evolved from a PUT (Section IV-B), no changes to the
                                                                     body of the test case were required at all. This suggests that
           Feature                               # of test cases     PUTs can be considered a stepping stone towards, or even an
           Change in strategies                        65            intermediate step in a planned introduction of PBTs.
           Addition of explicit inputs                 1                For tool developers: Recent years have seen the introduc-
           Addition of reproducing inputs              1             tion of tool support for creating PBTs. From Hypothesis’
           Addition of settings                        19            own ghostwriter6 , over using LLMs to write PBTs [13], [38],
           Addition of control                         0             to tools that aim to aggregate similar test cases into one
           Addition of exceptions                      0             PBT [31]. Tools that take EBTs and transform them into PUTs
                                                                     also exist [37]. However, there remains a need for tools that
TABLE VIII: Changes in used PBT features in the most recent          support developers in evolving existing, dissimilar, test cases
versus the initial PBT-introducing commit of the test case.          into PBTs. Our findings for RQ1 (Section IV-B), and our
                                                                     supporting dataset, can inspire tool builders with examples of
to reproduce inputs. 19 of the test cases eventually got some        real-world transformations that can be partially automated.
form of settings added, which doubles their total number.               As shown in Section IV-B2, not all features of the property-
                                                                     based testing framework are immediately adopted in newly-
  RQ3 Most PBTs (200/250) in our dataset have survived               evolved PBTs. In Section VI-B2 we found that features such
  since their introduction, with 90 having gone through              as PBT settings are often included only later in the life of
  changes. Removed test cases survived on average 352                a PBT. We also saw that inside some PBTs, developers were
  commits to the repository. 65 test cases saw a change              using if-statements, where an assume might have been more
  in their input generation strategy since their introduction.       appropriate. Therefore, future tools could not only help with
  Finally, 19 test cases received additional settings.               the evolution of test cases into PBTs, but also aid with the
                                                                     introduction of features to optimize existing PBTs.
                        VII. D ISCUSSION                                For researchers: There are still few empirical work that
   We now distil the actionable insights from our findings, for      compares conventional unit tests to PBTs. Ours is, to our
software testers, tool builders, and researchers alike. We also      knowledge, the first to do so in a one-to-one comparison of test
discuss the threats to the validity of our findings.                 cases before and after their evolution into property-based ones.
                                                                     Our comparison on test failures and code coverage has been
A. Actionable insights                                               insightful, and could inspire other one-to-one comparisons on
   For software testers: As evidenced by our findings for            other metrics. For instance, mutation score (recently used in
RQ2 (Section V-B), evolving well-chosen unit tests into              prior work [32], see Section VIII) has not yet been explored
property-based ones can increase their code coverage. We             in a one-to-one comparative setting.
moreover found evidence of such real-world evolutions finding
                                                                     B. Threats to validity
counterexamples where the original test passed without any
failures, leading to a deeper validation of the system under            Following standard practice in empirical software engineer-
test. We even found instances of these evolutions being con-         ing, we discuss potential threats to the validity of our results.
ducted alongside bug fixes, most likely to test for regressions      For each threat, we describe the applied mitigation strategies.
more thoroughly. Our findings should motivate practitioners             Construct validity: We considered test case evolutions that
to consider unit tests that have missed edge cases in the past       are found in open-source GitHub repositories, which can
as candidates to be evolved into property-based ones.                raise quality concerns. We mitigated this threat by applying
   Prior studies have found that implementing input generation       strict inclusion criteria, resulting in 575 high-quality candidate
strategies for a PBT can be tedious and effort-intensive [14].       repositories with at least one PBT. As most of those PBTs did
Nonetheless, our findings for RQ1 (Section IV-B) indicate            not result from an evolution, we were left with fewer test case
that many evolved PBTs in our dataset use but a simple,                6 https://hypothesis.readthedocs.io/en/latest/reference/integrations.html#
built-in input generation strategy. Practitioners may find it        ghostwriter
evolutions to study, but the study subjects still stem from 64             Goldstein et al. [14] interviewed 30 developers from a
different repositories, ensuring diversity.                             fintech company, to find insights about how people experience
   Further, while false negatives cannot be excluded, all test          PBT in practice. Another paper by Goldstein et al. [15] reports
case evolutions were manually inspected by both the first               that developers experience difficulties in finding properties to
and the second author, ensuring the dataset is free of false            test for. Our research looks at open-source projects rather than
positives. The same goes for the before-after mappings for              the industry, and investigates existing test cases.
each test case evolution, which were created by the first author           A study by Wauters and De Roover [39] looked into the use
and inspected and confirmed by the second author.                       of PBTs by 28 open-source machine learning projects. The
   Manual labeling of test case evolutions was also used to             study investigated common positive and negative sentiments
answer RQ1. We mitigated the threat of subjective bias by               expressed in GitHub commits and code comments, which
having multiple labelers and multiple labeling rounds until             parts of a project were tested, as well as the complexity of
a substantial inter-rater agreement was reached. To answer              data generation strategies. Our study also investigates data
RQ2, we used tools to execute test cases and compute their              generation strategies but does not focus on ML-intensive
code coverage. To mitigate the threat of measurement bias               projects that often require more complex data.
introduced by the tooling, we only used mature open-source                 Ravi and Coblenz [32] recently performed a large-scale
tools that are well-established within the community.                   empirical study on PBTs in 426 Python programs. Their study
   Internal validity: For RQ2, we measure changes in code               uses a data flow analysis to automatically categorize real-world
coverage induced by the evolution into a PBT. However, the              PBTs into 12 categories. Additionally, they used mutation
introduction of a PBT is not always the only change in a                testing to investigate how many mutations a project’s existing
commit. If additional changes are present, the total number             PBTs can find compared to the project’s existing conventional
of statements may change, which can confound the observed               tests. Our research looks at the impact of evolving test cases,
coverage differences and lead to incorrect conclusions. To              and compares a test case before its evolution into a PBT to
mitigate this threat, we manually inspected commits exhibiting          the resulting PBT. This means we compare two test cases that
large coverage changes. In addition, we report both absolute            are equivalent in terms of the behavior they are supposed to
and relative coverage, computed with and without the test files         validate, and we do so one-to-one.
included, as test files are more likely to change due to the               Unlike our study, the aforementioned papers do not inves-
introduction of PBTs.                                                   tigate the evolution of PBTs, but rather consider snapshots of
   External validity: Finally, our study focuses on Python              PBTs at one point in time.
test cases that use the Hypothesis framework. Hypothesis                            IX. C ONCLUSION AND F UTURE W ORK
is the most widely recognized PBT framework for Python,                    This paper studied the evolution of property-based unit tests
and Python is ranked among the most popular programming                 from conventional ones, the impact of such PBT-introducing
languages. Nevertheless, our findings may not generalize to             evolutions, and how the resulting PBTs are maintained after-
other programming languages nor to other PBT frameworks.                wards. We found that the evolution of parameterized unit tests
                                                                        into a PBT commonly requires no updates to their test body
                       VIII. R ELATED W ORK
                                                                        at all. For example-driven unit tests, in contrast, the evolution
   Several studies have investigated the introduction and main-         commonly required non-semantic changes to the test body.
tenance of various types of test cases, from tests in general [1],      These account for the introduction and usage of a randomly
[9], [44] over GUI tests [5] to performance tests [11]. A               generated input value. The input generation strategy of PBTs
few studies have also looked into changes in code coverage              that evolved from existing unit tests is often simple. Impact-
induced by software patches, e.g., [12] and [17]. However,              wise, we found that code coverage can improve when unit tests
our work is the first to study how PBTs can evolve from                 are replaced by PBTs. More importantly, we found evidence
conventional unit tests, and to study the impact on code                of bugs found through or alongside the introduction of these
coverage —rather than changes in code coverage over time                PBTs. Most of the PBTs in our dataset are still present in the
or induced by changes to the system under test.                         most recent version, with almost half having gone through
   Zooming in on other studies targeting property-based test-           some maintenance during their life time. The adoption of
ing, Corgozinho et al. [8] sampled 86 PBTs from GitHub, and             additional PBT features is not uncommon either.
categorized each of them manually according to the testing                 One avenue for future work is investigating the long-term
patterns they adhere to, which stem from an influential blog            impact of PBT-introducing test case evolutions. Another is
post7 . They also study the use of Hypothesis features in               developer support in the form of a tool that identifies candidate
those PBTs, similar to us. However, while they investigate              unit tests for such an evolution, and for partially automating
bigger, established projects, we investigate the adoption of            the necessary test changes.
these features in test cases that were previously not a PBT.
                                                                                          ACKNOWLEDGEMENTS
We also study whether those features are adopted later on,
while the PBT matures.                                                    This research was partially funded by the Research Foun-
                                                                        dation Flanders (FWO) Grant No. 1SHFI24N and by the
  7 https://fsharpforfunandprofit.com/posts/property-based-testing-2/   Cybersecurity Research Program Flanders (CRPF).
                              R EFERENCES                                        [18] John Hughes. Experiences with QuickCheck: Testing the Hard Stuff and
                                                                                      Staying Sane, pages 169–186. Springer International Publishing, Cham,
 [1] Maurı́cio Aniche, Christoph Treude, and Andy Zaidman. How develop-               2016.
     ers engineer test cases: An observational study. IEEE Transactions on       [19] John Hughes, Benjamin C. Pierce, Thomas Arts, and Ulf Norell. Myster-
     Software Engineering, 48(12):4925–4946, 2022.                                    ies of dropbox: Property-based testing of a distributed synchronization
 [2] Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger. Testing               service. In 2016 IEEE International Conference on Software Testing,
     telecoms software with quviq quickcheck. In Proceedings of the 2006              Verification and Validation (ICST), pages 135–145, 2016.
     ACM SIGPLAN Workshop on Erlang, ERLANG ’06, page 2–10, New                  [20] Laura Inozemtseva and Reid Holmes. Coverage is not strongly correlated
     York, NY, USA, 2006. Association for Computing Machinery.                        with test suite effectiveness. In Proceedings of the 36th International
 [3] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and                    Conference on Software Engineering, ICSE 2014, page 435–445, New
     Shin Yoo. The oracle problem in software testing: A survey. IEEE                 York, NY, USA, 2014. Association for Computing Machinery.
     Transactions on Software Engineering, 41(5):507–525, 2015.                  [21] Paul Jansen. Tiobe index for december 2025. https://web.archive.org/
                                                                                      web/20251218235222/https://www.tiobe.com/tiobe-index/. Accessed on
 [4] Ned Batchelder and Contributors to Coverage.py. Coverage.py: The code
                                                                                      20 december 2025.
     coverage tool for Python. https://web.archive.org/web/20251223093400/
                                                                                 [22] JetBrains.      Python developers survey 2023 results.             https:
     https://github.com/coveragepy/coveragepy. Accessed on 20 december
                                                                                      //web.archive.org/web/20250920050020/https://lp.jetbrains.com/
     2025.
                                                                                      python-developers-survey-2023/. Accessed on 10 december 2025.
 [5] Laurent Christophe, Reinout Stevens, Coen De Roover, and Wolfgang
                                                                                 [23] Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris
     De Meuter. Prevalence and maintenance of automated functional tests for
                                                                                      Bruynooghe, Brianna Laugher, and Florian Bruhin.                  pytest
     web applications. In 2014 IEEE International Conference on Software
                                                                                      9.0.0.               https://web.archive.org/web/20250906051102/https:
     Maintenance and Evolution, pages 141–150, 2014.
                                                                                      //github.com/pytest-dev/pytest/, 2004.      Version 9.0.0 Contributors
 [6] Koen Claessen and John Hughes. Quickcheck: a lightweight tool
                                                                                      include Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris
     for random testing of haskell programs. In Proceedings of the Fifth
                                                                                      Bruynooghe, Brianna Laugher, Florian Bruhin, and others.
     ACM SIGPLAN International Conference on Functional Programming,
                                                                                 [24] Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce. Cov-
     ICFP ’00, page 268–279, New York, NY, USA, 2000. Association for
                                                                                      erage guided, property based testing. Proc. ACM Program. Lang.,
     Computing Machinery.
                                                                                      3(OOPSLA), October 2019.
 [7] Jacob Cohen. A coefficient of agreement for nominal scales. Educational     [25] J. Richard Landis and Gary G. Koch. The measurement of observer
     and psychological measurement, 20(1):37–46, 1960.                                agreement for categorical data. Biometrics, 33(1):159–174, 1977.
 [8] Arthur Lisboa Corgozinho, Marco Tulio Valente, and Henrique Rocha.          [26] David R. MacIver and Alastair F. Donaldson. Test-Case Reduction via
     How Developers Implement Property-Based Tests . In 2023 IEEE Inter-              Test-Case Generation: Insights from the Hypothesis Reducer. In Robert
     national Conference on Software Maintenance and Evolution (ICSME),               Hirschfeld and Tobias Pape, editors, 34th European Conference on
     pages 380–384, Los Alamitos, CA, USA, October 2023. IEEE Computer                Object-Oriented Programming (ECOOP 2020), volume 166 of Leibniz
     Society.                                                                         International Proceedings in Informatics (LIPIcs), pages 13:1–13:27,
 [9] Ermira Daka and Gordon Fraser. A survey on unit testing practices and            Dagstuhl, Germany, 2020. Schloss Dagstuhl – Leibniz-Zentrum für
     problems. In 2014 IEEE 25th International Symposium on Software                  Informatik.
     Reliability Engineering, pages 201–211, 2014.                               [27] David R MacIver, Zac Hatfield-Dodds, et al. Hypothesis: A new
[10] Albert Danial. cloc: v1.92. https://doi.org/10.5281/zenodo.5760077,              approach to property-based testing. Journal of Open Source Software,
     December 2021.                                                                   4(43):1891, 2019.
[11] Sergio Di Meglio, Luigi Libero Lucio Starace, Valeria Pontillo, Ruben       [28] Gerard Meszaros. xUnit test patterns: Refactoring test code. Pearson
     Opdebeeck, Coen De Roover, and Sergio Di Martino. Performance                    Education, 2007.
     testing in open-source web projects: Adoption, maintenance, and a           [29] Agustı́n Mista and Alejandro Russo. Mutagen: Reliable coverage-
     change taxonomy. In Proceedings of the 41st IEEE International                   guided, property-based testing using exhaustive mutations. In 2023 IEEE
     Conference on Software Maintenance and Evolution (ICSME 2025).                   Conference on Software Testing, Verification and Validation (ICST),
     IEEE, sep 2025. 41st IEEE International Conference on Software                   pages 176–187, 2023.
     Maintenance and Evolution (ICSME 2025), ICMSE ; Conference date:            [30] Rohan Padhye, Caroline Lemieux, and Koushik Sen. Jqf: coverage-
     07-09-2025 Through 12-09-2025.                                                   guided property-based testing in java. In Proceedings of the 28th ACM
[12] S. Elbaum, D. Gable, and G. Rothermel. The impact of software                    SIGSOFT International Symposium on Software Testing and Analysis,
     evolution on code coverage information. In Proceedings IEEE Inter-               ISSTA 2019, page 398–401, New York, NY, USA, 2019. Association
     national Conference on Software Maintenance. ICSM 2001, pages 170–               for Computing Machinery.
     179, 2001.                                                                  [31] Hila Peleg, Dan Rasin, and Eran Yahav. Generating tests by example.
[13] Khashayar Etemadi, Marjan Sirjani, Mahshid Helali Moghadam, Per                  In Isil Dillig and Jens Palsberg, editors, Verification, Model Checking,
     Strandberg, and Paul Pettersson. Llm-based property-based test genera-           and Abstract Interpretation, pages 406–429, Cham, 2018. Springer
     tion for guardrailing cyber-physical systems. In International Conference        International Publishing.
     on Bridging the Gap between AI and Reality, pages 18–46. Springer           [32] Savitha Ravi and Michael Coblenz. An empirical evaluation of property-
     Nature Switzerland Cham, 2025.                                                   based testing in python. Proc. ACM Program. Lang., 9(OOPSLA2),
[14] Harrison Goldstein, Joseph W. Cutler, Daniel Dickstein, Benjamin C.              October 2025.
     Pierce, and Andrew Head. Property-based testing in practice. In Pro-        [33] Mohammad Rezaalipour and Carlo A. Furia. An annotation-based
     ceedings of the IEEE/ACM 46th International Conference on Software               approach for finding bugs in neural network programs. Journal of
     Engineering, ICSE ’24, New York, NY, USA, 2024. Association for                  Systems and Software, 201:111669, 2023.
     Computing Machinery.                                                        [34] Davide Spadini, Maurı́cio Aniche, and Alberto Bacchelli. Pydriller:
[15] Harrison Goldstein, Joseph W Cutler, Adam Stein, Benjamin C Pierce,              Python framework for mining software repositories. In Proceedings of
     and Andrew Head. Some problems with properties. In Proc. Workshop                the 2018 26th ACM Joint Meeting on European Software Engineering
     on the Human Aspects of Types and Reasoning Assistants (HATRA),                  Conference and Symposium on the Foundations of Software Engineering,
     2022.                                                                            ESEC/FSE 2018, page 908–911, New York, NY, USA, 2018. Associa-
[16] Harrison Goldstein, Jeffrey Tao, Zac Hatfield-Dodds, Benjamin C.                 tion for Computing Machinery.
     Pierce, and Andrew Head. Tyche: Making sense of pbt effectiveness. In       [35] Nikolai Tillmann and Wolfram Schulte. Parameterized unit tests.
     Proceedings of the 37th Annual ACM Symposium on User Interface                   SIGSOFT Softw. Eng. Notes, 30(5):253–262, September 2005.
     Software and Technology, UIST ’24, New York, NY, USA, 2024.                 [36] Nikolai Tillmann and Wolfram Schulte. Parameterized unit tests. In
     Association for Computing Machinery.                                             Proceedings of the 10th European Software Engineering Conference
[17] Michael Hilton, Jonathan Bell, and Darko Marinov. A large-scale study            Held Jointly with 13th ACM SIGSOFT International Symposium on
     of test coverage evolution. In Proceedings of the 33rd ACM/IEEE                  Foundations of Software Engineering, ESEC/FSE-13, page 253–262,
     International Conference on Automated Software Engineering, ASE ’18,             New York, NY, USA, 2005. Association for Computing Machinery.
     page 53–63, New York, NY, USA, 2018. Association for Computing              [37] Deepika Tiwari, Yogya Gamage, Martin Monperrus, and Benoit Baudry.
     Machinery.                                                                       Proze: Generating parameterized unit tests informed by runtime data.
     In 2024 IEEE International Conference on Source Code Analysis and               tation. https://web.archive.org/web/20251223092313/https://hypothesis.
     Manipulation (SCAM), pages 166–176, 2024.                                       readthedocs.io/en/latest/reference/api.html. Accessed on 10 december
[38] Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Pad-               2025.
     hye. Can large language models write good property-based tests?, 2024.     [43] Hypothesis Works. When to use hypothesis and property-based testing
[39] Cindy Wauters and Coen De Roover. Property-based Testing within ML              - hypothesis 6.148.3 documentation.          https://web.archive.org/web/
     Projects: an Empirical Study . In 2024 IEEE International Conference            20251223092910/https://hypothesis.readthedocs.io/en/latest/tutorial/
     on Software Maintenance and Evolution (ICSME), pages 648–653, Los               introduction.html#when-to-use-hypothesis-and-property-based-testing.
     Alamitos, CA, USA, October 2024. IEEE Computer Society.                         Accessed on 11 december 2025.
[40] Cindy Wauters, Ruben Opdebeeck, and Coen De Roover. Replication            [44] Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie van
     package on the evolution of python test cases into property-based tests.        Deursen. Mining software repositories to study co-evolution of pro-
     https://doi.org/10.6084/m9.figshare.31424447, Feb 2026.                         duction & test code. In 2008 1st International Conference on Software
[41] Frank Wilcoxon. Individual comparisons by ranking methods. Biomet-              Testing, Verification, and Validation, pages 220–229, 2008.
     rics Bulletin, 1(6):80–83, 1945.
[42] Hypothesis Works. Api reference - hypothesis 6.148.3 documen-
