                                         From Natural Language to Executable Properties for
                                         Property-based Testing of Mobile Apps
                                         YIHENG XIONG, East China Normal University, China
                                         TING SU, East China Normal University, China
                                         JINGLING SUN, University of Electronic Science and Technology of China, China
                                         JUE WANG, Nanjing University, China
                                         QIN LI, East China Normal University, China
arXiv:2603.21263v1 [cs.SE] 22 Mar 2026




                                         GEGUANG PU, East China Normal University, China
                                         ZHENDONG SU, ETH Zurich, Switzerland
                                         Property-based testing (PBT) is a popular software testing methodology and is effective in validating the
                                         functionality of mobile applications (apps for short). However, its adoption in practice remains limited, largely
                                         due to the manual effort and technical expertise required to specify executable properties. In this paper, we
                                         propose a novel structured property synthesis approach that automatically translates property descriptions in
                                         natural language into executable properties, and implement it in a tool named iPBT. Our approach decomposes
                                         the problem into UI semantic grounding and executable property synthesis. It first builds an enriched widget
                                         context via multimodal LLMs to align visual elements with their functional semantics, and then uses an LLM
                                         with in-context learning to generate framework-specific executable properties. We evaluate iPBT with a
                                         closed-source LLM (GPT-4o) and an open-source LLM (DeepSeek-V3) on 124 diverse property descriptions
                                         derived from an existing benchmark dataset. iPBT achieves 95.2% (118/124) accuracy on both LLMs. Notably, an
                                         ablation study reveals that the enriched widget context contributes to an absolute improvement of up to 20.2%
                                         (from 75.0% to 95.2%). A user study with 10 participants demonstrates that iPBT reduces the time required to
                                         write executable properties by 56%, suggesting substantially lower manual effort. Furthermore, evaluations on
                                         1,180 linguistically diverse variations demonstrate iPBT’s robustness (87.6% accuracy), indicating its capability
                                         to handle varied expressions.

                                         1   Introduction
                                         Property-based testing (PBT) has emerged as a powerful testing methodology that validates software
                                         program correctness by checking properties. Unlike example-based testing [11] which relies on
                                         specific input-output pairs to determine test outcomes, PBT systematically generates a large number
                                         of inputs to verify whether the system under test satisfies the defined properties. The pioneering PBT
                                         framework QuickCheck [10] has inspired many other PBT frameworks that successfully uncover
                                         bugs difficult to detect with traditional testing techniques across various software domains. [5, 24,
                                         25, 27, 43, 55, 75].
                                            Recently, several research efforts have applied PBT in testing mobile apps [29, 61, 62, 75] to address
                                         the oracle problem, determining whether an app’s behavior aligns with expected outcomes. In
                                         these work, users specify expected behaviors as executable properties, and then the PBT framework
                                         automatically generates GUI event sequences to validate them. For example, consider Amaze [1],
                                         a popular file management app. A typical property is that when a user clicks on a directory (e.g.,
                                         "Download"), the app should open that directory and display its contents, rather than triggering a
                                         file-opening dialog (which is only expected for files). Fig. 1(a) illustrates the expected app behavior.
                                         Fig. 1(b) shows the corresponding executable property written in Kea [75], a recent effective PBT
                                         framework for finding functional bugs in mobile apps. The property consists of a precondition
                                         Authors’ Contact Information: Yiheng Xiong, East China Normal University, , China, xyh@stu.ecnu.edu.cn; Ting Su, East
                                         China Normal University, , China, tsu@sei.ecnu.edu.cn; Jingling Sun, University of Electronic Science and Technology
                                         of China, , China, jingling.sun910@gmail.com; Jue Wang, Nanjing University, , China, juewang591@gmail.com; Qin
                                         Li, East China Normal University, , China, qli@sei.ecnu.edu.cn; Geguang Pu, East China Normal University , , China,
                                         ggpu@sei.ecnu.edu.cn; Zhendong Su, ETH Zurich, , Switzerland, zhendong.su@inf.ethz.ch.


                                                                                                          , Vol. 1, No. 1, Article . Publication date: March 2026.
2                                                             SMU Classification:
                                    Yiheng Xiong, Ting Su, Jingling   Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su
                                                             Restricted



                                      Expected behavior      1 @precondition(
                                          Amaze              2 findWidget(id="line").exists() and
                                         /storage/Download   3 findWidget(id="Search").exists()
                                          Files              4 )
                                           picture.png       5 @property
                                           xx.xx
    Precondition                           video.mp4
                                                             6 def test_open_directory():
                                           xx.xx             7 item_count = findWidget(id="line").count
    Amaze                                                    8 for index in range(item_count):
    /storage
                                                             9    directory_name = findWidget(id="line")[index].get("text")
    Files                                                    10   if "." not in file_name:
    Download
    xx.xx                                                    11     findWidget(id="line")[index].click()
    Data                                                     12      break
    xx.xx                              Property violation    13 assert directory_name in findWidget(id="path").get("text")
    todo.txt
    xx.xx
                                          Amaze                 (b) An example of executable properties in Kea
                                          /storage

                                          Files                  Precondition: The list item and search button exist
                                           Open As               Function body:
                                          Download
                                           Text
                                          xx.xx
                                           Image
                                                                 1. Get the names of all items
                                          Data
                                           Video                 2. Select an item name that does not contain "."
                                          xx.xx
                                           Other
                                          todo.txt               3. Click it
                                          xx.xx                  4. Assert the path contains the item name
               (a) The app’s expected behavior                 (c) The property description in natural language

                                  Fig. 1. An example of the property in app Amaze.



(lines 1-4) to check the presence of the file and search button; an interaction scenario (lines 7-12)
simulates a click action to open the directory; and a postcondition (line 13) checks whether the
path contains the directory name. Then, the PBT framework can generate a large number of GUI
events to verify this property and report bugs when the property is violated.
   However, these PBT frameworks see limited adoption because testers must translate high-level
intents into framework-constrained, executable properties, which demands substantial manual
effort and expertise. Testers must learn framework-specific DSLs, including specialized syntax,
APIs, and conventions. This imposes a steep learning curve, particularly for those without a strong
programming background. Moreover, specifying executable properties requires the tedious manual
inspection of the app’s view hierarchy to locate low-level UI widget identifiers (e.g., id="Search").
It also requires correctly implementing complete executable properties under the constraints of
the testing framework, both of which are non-trivial and error-prone in practice. Together, these
barriers create a significant gap that restricts the widespread application of PBT.
   To bridge this gap, we enable testers to specify properties in natural language, which is more
intuitive and lightweight. To reduce ambiguity while preserving accessibility, we structure property
descriptions in a widely-used Hoare logic format, like Given-When-Then used in Gherkin [20] for
Behavior-Driven Development [59]. Fig. 1(c) illustrates such a structured description. While translat-
ing natural language into executable code has been studied, classic rule-based approaches [12, 48, 63]
are insufficient for mobile PBT due to two main limitations. First, rule-based approaches lack flex-
ibility. Enumerating rules to cover diverse paraphrases of user intent (e.g., “open the folder” vs.
“navigate into the directory”) does not scale and itself incurs substantial manual effort. Second,
these approaches lack semantic grounding. They struggle to accurately map high-level widget
descriptions (e.g., “search bar”) to the low-level widget identifiers.
   More fundamentally, these limitations expose a deeper challenge: grounding the informal test
intent into concrete executable properties. To address this challenge, we propose a structured
property synthesis approach that automatically translates natural-language property descriptions
into executable properties. We decompose this problem into two distinct phases: UI semantic
grounding and executable property synthesis. We first tackle UI semantic grounding by extracting

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                               3


comprehensive GUI information (e.g., view hierarchies and screenshots) from the app and con-
structing enriched widget contexts for each UI widget. We employ Multimodal Large Language
Models (MLLMs) to generate semantic annotations for each widget by jointly reasoning over visual
appearance and structural information. These annotations serve as grounding signals, enabling ac-
curate mapping between user-described widgets (e.g., "item name") and concrete widget identifiers
(e.g., "line"), even when the identifiers themselves are not descriptive. Building on this grounded
widget context, we then perform executable property synthesis. We leverage LLMs as inference
engines to synthesize executable properties that integrate (i) the property description, (ii) enriched
widget context, (iii) framework APIs, and (iv) few-shot demonstrations [51]. This design eliminates
the need for handcrafted rules while enabling the model to adapt to the specific constraints of
property generation.
   We implemented our approach as a tool named iPBT. To evaluate its effectiveness, we conduct
experiments with a closed-source LLM (GPT-4o [45]) and an open-source LLM (DeepSeek-V3 [13])
on 124 diverse properties from the Kea benchmark [75]. iPBT correctly synthesizes executable
properties for 118 out of 124 cases (95.2%) with both models. Notably, our ablation study reveals
that the enriched widget context contributed to a 20.2% absolute increase in accuracy, highlighting
its critical role in grounding UI semantics. Beyond accuracy, a user study demonstrated the practi-
cal utility of iPBT, reducing property authoring time by 56% compared to manual composition.
We further evaluate robustness by using another LLM, Llama-3.1 [39], to generate 1,180 diverse
paraphrased variants of the original property descriptions. Under these variations, iPBT achieves
87.6% accuracy (1,034/1,180) with GPT-4o and 87.5% (1,032/1,180) with DeepSeek-V3. These results
indicate that iPBT is robust to the variability of natural language property descriptions.
   In summary, this paper has made the following contributions:

• At the conceptual level, we introduce a novel approach that reduces the manual effort and lowers
  the technical barrier for property-based testing of mobile apps.
• At the technical level, we have implemented our idea as a tool named iPBT that (i) constructs
  enriched widget contexts via UI semantic grounding to align user-described widgets with concrete
  UI elements, and (ii) leverages LLMs with in-context learning to synthesize framework-specific
  executable properties.
• At the empirical level, we construct a new evaluation dataset with 124 human-written natural
  language property descriptions derived from real-world bugs. We further generate 1,180 linguistically
  diverse LLM-based variants for robustness evaluation. Based on this dataset, we conduct compre-
  hensive evaluations and summarize practical lessons learned on applying LLMs to property-based
  testing.

2     Background
2.1    Large Language Models
Large language models (LLMs) have been shown to perform well on a wide range of tasks in natural
language processing [41] and software engineering [22, 83]. LLMs, such as GPT-4o, DeepSeek-V3,
are deep neural networks trained on massive amounts of text data, enabling them to generate human-
like text, understand complex queries, and perform different tasks. These models are typically
based on the Transformer architecture [66], which relies on self-attention mechanisms to process
and generate text efficiently at scale. Recently, Multimodal Large Language Models (MLLMs) have
emerged, which extend the capabilities of traditional LLMs by incorporating additional modalities
such as images, audio, or structured data alongside text [78]. This multimodal integration allows
MLLMs not only to process natural language but also to understand and reason about visual or

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
4                                        Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


     Preparation                 Phase I: UI Semantic Grounding                                Phase II: Executable Property Synthesis

                          Page Information                  Widget Information                                    Key Components

                                         12:00                                                                Property Description
                             Preview                                  Test

                                  Test                                                                        Enriched Widget Context
                                                                                                  User

    Data Collection          SHOW ANSWER
                                                           Raw Widget Attributes                              Framework APIs
                                                          "text": "Test",
                                                          "resource_id": "qa",
                                                          "content_description": "",                          Few-shot Demonstrations
                       "App name": "AnkiDroid",           "class": "android.view.View"
                       "Activity name": "Previewer"




                                                                                                Synthesizer
                                                 Generated Semantic Information

    View hierarchies                             "semantic label": "Question text display",                           Executable Property
                                                                                                   LLM
    and screenshots Annotator    MLLM            "functionality": "Displays the question..."



                                                        Fig. 2. Overview of iPBT.



other non-textual inputs. In the context of GUI testing, MLLMs are particularly useful as they can
jointly leverage textual descriptions and GUI screenshots to better interpret UI semantics.

2.2      In-Context Learning
LLMs are typically pre-trained on large corpora of text and code data. To adapt LLMs on customized
tasks, fine-tuning [15], which requires training on a pre-trained model with additional massive data,
and prompt engineering [51] (e.g., chain-of-thought [69], in-context learning [51], and multi-step
reasoning [85] ) are two common approaches. In-context learning refers to the ability of LLMs to
perform customized tasks with a few examples provided in the input prompt, without requiring
model training. In our approach, we adopt in-context learning, which enables LLMs to perform
custom tasks using just a few examples included directly in the input prompt.

3      Approach
At a high level, iPBT operates as a structured synthesis approach designed to translate natural
language property descriptions into executable properties. Fig. 2 presents the overall workflow
of iPBT, which consists of two main phases: (i) the UI Semantic Grounding phase (§3.1), which
extracts the GUI information of the app under test, leverages MLLMs to generate functionality
annotations for widgets, and constructs the enriched widget context for each UI element; and (ii)
the Executable Property Synthesis phase (§3.2) that generates executable properties by encoding user
APIs, UI widget identifiers, property description and examples into a carefully designed prompt.
Collectively, these two phases enable iPBT to effectively bridge the gap between informal user
intent and concrete implementation details for property-based testing. We next describe each phase
in detail.

3.1      Phase I: UI Semantic Grounding
In Android app testing, UI widget identifiers (e.g., resource_id, text, and content_description) are
commonly used to locate and interact with target widgets displayed on the screen. Consequently,

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                               5


a critical prerequisite for synthesizing executable properties is grounding high-level natural lan-
guage descriptions onto these concrete identifiers. For instance, an instruction like “click the
Settings button” must be accurately mapped to a specific widget, such as text="Settings" or
id="action_settings". However, relying solely on raw identifiers is often insufficient. In practice,
these identifiers are frequently opaque, poorly maintained, or generic (e.g., id="button1"), failing
to reflect the widget’s true functionality. To bridge this semantic gap, we extract comprehensive
GUI information and leverage Multimodal LLMs (MLLMs) to generate semantic functionality anno-
tations for each widget. The resulting enriched widget context consists of raw attributes (e.g., text,
resource_id) with MLLM-inferred semantics. This context serves as a robust bridge, enabling the
subsequent executable property synthesis phase to accurately match user-described widgets to the
correct identifiers.
3.1.1 Widget Context Extraction. In Android apps, the layout defines the structure of each page,
while widgets (e.g., Button) handle user-triggered events (e.g., onClick) [3]. To extract the contextual
information of these widgets, the first step is to collect the necessary GUI data (view hierarchies
and screenshots). Notably, our approach supports GUI data acquisition via various methods, in-
cluding: (1) manual interaction by testers; (2) automated exploration using testing tools; and (3)
direct provision from app vendors. From the collected GUI data, we extract two complementary
components that facilitate MLLM understanding and the generation of functionality annotations
for widgets:
• Page information provides high-level context about the app page under test, including the app
  name, activity name, and page screenshot. Specifically, the app name typically reflects the domain
  or type of the app (e.g., SimpleNote implies a note-taking app), offering prior knowledge about its
  overall functionality. The app name is obtained by statically analyzing the AndroidManifest.xml
  file. The activity name, parsed from the runtime layout file, specifies the functionality of the
  current page within the app (e.g., LoginActivity for user authentication, SettingsActivity for
  configuration). The page screenshot, captured via the Android Debug Bridge (ADB), preserves the
  visual layout and arrangement of the UI widgets. It serves as a direct reference that complements
  textual and structural information. The target widget is highlighted with a red bounding box in
  the screenshot to help the MLLM accurately locate it.
• Widget information provides fine-grained details about the interactive widgets on each page.
  It consists of two parts: the cropped widget image and the widget attributes. The widget im-
  age is obtained by cropping the page screenshot according to the widget bounds. The widget
  attributes are extracted from the view hierarchy file. We focus on four commonly used fields:
  "text", "resource_id", "content_description", and"class", as they provide valuable signals about
  the intended behavior of the widget. Specifically, the "text" field corresponds to the string dis-
  played on the widget (e.g., "Login"), often directly indicating its functionality. The "resource_id"
  is a developer-assigned identifier that may encode semantic hints about the widget’s role or
  logical grouping (e.g., id=btn_submit). The "content_description" field, typically designed for
  accessibility [2]. Finally, the "class" specifies the Android class type of the widget (e.g., Button,
  EditText, ImageView), which reflects the type of interaction it supports.
   These two components together provide complementary structural and visual information,
enabling the MLLM to better reason about widget semantics. Crucially, the extraction of these
artifacts is fully automated. The extracted widget context serves as the foundation for the subsequent
phase, where MLLMs are leveraged to generate accurate functionality annotations for each widget.
3.1.2 Widget Functionality Annotations Generation. Leveraging the extracted widget context, we
employ an MLLM to synthesize functionality annotations. We construct a structured prompt to

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
6                                   Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


                Table 1. The prompt design for generating functionality annotation of the widget.

    ID Prompt Component                                                 Instantiation
    ①   Role Assignment               You are a professional mobile app UI semantic annotation assistant.
                                      Please annotate the provided UI widget with the semantic label and functionality
                                      description based on the given context.
    ②   Task                          - The full page screenshot, where the target widget is highlighted with a red box.
                                      - The cropped widget image and its attributes.
                                      - The provided app name and foreground activity name.
    ③   Few-shot Demonstrations       [example input] + [example output]
    ④   Input                         [page information]+[widget information]
    ⑤   Constraints                   [Strict rules]




guide the model, as illustrated in Table 1. The prompt is organized into five components: Component
① defines the role of the model to establish a professional persona. Component ② specifies the
Task, directing the MLLM to infer semantics based on the provided visual and structural context.
Component ③ provides two demonstrations that serve as the concrete reference selected from the
SimpleNote app [58]. They illustrate the expected mapping from the raw widget context to the
target semantic label and functionality description. Empirically, we found that two representative
demonstrations are sufficient for the MLLM to grasp the task requirements and generate high-
quality annotations. Component ④ gives the input of context, including page information and
widget information extracted in the previous step. Component ⑤ outlines the constraints to ensure
output consistency.
   Given these instructions, the MLLM generates two key fields for each widget: semantic label
and functionality. The semantic label is a concise identification of the widget, e.g., login button, or
settings option. The functionality offers a description of its behavior, e.g., allows the user to log
into their account, or navigates to settings screen.
   For example, consider the target widget shown in the green box in Fig. 2. Sole reliance on these
raw attributes is insufficient to deduce its functionality. To resolve this ambiguity, we construct an
enriched widget context by aggregating page information, such as the app name AnkiDroid and
activity Previewer, with the widget information. This combined information enables the MLLM to
effectively anchor the widget’s semantics. Consequently, iPBT produces functionality annotations,
augmenting the original attributes with a semantic label ("Question text display") and a precise
functionality description ("Displays the question text to the user for review"). These enriched
widget contexts serve as the foundation for generating executable properties in the next phase.

3.2     Phase II: Executable Property Synthesis
3.2.1 Writing Property Descriptions. To ensure that property descriptions are both intuitive for
testers and structured for synthesis, we adopt a Hoare logic-style representation. This style aligns
with industry-standard practices, such as Gherkin [20] widely employed in Behavior-Driven Devel-
opment (BDD) [59]. In the context of PBT in mobile apps, an executable property is formalized as a
triple ⟨𝑃, 𝐼, 𝑄⟩, where (1) 𝑃 is the precondition, which defines the when could check the property,
(2) 𝐼 is the interaction scenario, which defines the sequence of user actions to execute the target
functionality, and (3) 𝑄 is the postcondition, which specifies the expected UI state after the interac-
tion. To facilitate this formalization, we structure the natural language property description into
two segments:

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                                       7


                       Table 2. The prompt design for generating executable properties.

 ID Prompt Component                                                   Instantiation
                                  You are an expert in Python programming and Android app testing, and your role is
 ①   Role Assignment
                                  to write test snippets for Android apps.
                                  The following APIs are available for writing property:
 ②   Framework APIs               widget: findWidget(identifier); click: widget.click() long click: widget.long_click(); get
                                  text: widget.get("text"); exists: widget.exists(); back: press("back"); ...
                                  The app’s UI widget identifiers are detailed below for reference, ensuring
                                  accurate element selection in tests:
 ③   Enriched Widget Context      { "text": "Download", "resource_id":"line", "description": "null", "class": "an-
                                  droid.widget.TextView", "semantic label": "File name text", "functionality": "Display
                                  the name of the file" }, ...
                                  Here are the two example test snippets that you might write, based on the given
 ④   Few-shot Demonstrations
                                  property descriptions: [Example property description and executable properties]
                                  Your task: Using the available APIs, UI widget identifiers and following the
                                  example format, please write a test snippet for the following property:
                                  Precondition: The list item and search button exist
                                  Function body:
 ⑤   Property Description
                                  1. Get the names of all items
                                  2. Select an item name that does not contain "."
                                  3. Click it
                                  4. Assert the path contains the item name
                                  Respond only with the Python code, strictly adhering to the given property description.
 ⑥   Constraints
                                  Do not include any explanations, comments, or text outside the code block.


• Precondition: This segment maps directly to 𝑃. It describes the initial visible state required to
  trigger the property (e.g., the file name exists”).
• Function Body: This segment encapsulates both 𝐼 and 𝑄. It contains the execution steps of the
  target functionality (e.g., select a file name that does not contain ‘.’ and click it”) and explicitly
  states the expected effects (e.g., “verify the path contains the file name”).

3.2.2 Constructing Prompt and Generating Executable Properties. Table 2 illustrates the structured
prompt employed by iPBT to synthesize executable properties. The prompt comprises six distinct
components, labeled ① through ⑥, designed to guide the LLM’s synthesis process:
• Role Assignment (①): This component defines a specialized persona for the LLM, priming it to
  focus on the domain of mobile app testing and code generation.
• Framework APIs (②): To ensure the synthesized code is syntactically valid, we provide the
  complete list of user-facing APIs supported by the target framework (Kea in our implementation).
  We curated the set of APIs like click() and exists() by analyzing the framework’s official docu-
  mentation and source code. Crucially, these APIs are **app-agnostic**, allowing this component
  to be reused across different apps. Note that this component is modular and can be substituted
  with APIs from other PBT frameworks if desired.
• Enriched Widget Context (③): This component supplies the Enriched Widget Context con-
  structed in Phase I. By integrating raw identifiers with MLLM-generated semantic annotations,
  this section enables the LLM to ground the natural language descriptions to the correct UI widget
  identifiers.
• Few-shot Demonstrations (④): To facilitate in-context learning, we crafted two concrete
  examples derived from the SimpleNote app [58]. Our selection follows two principles to ensure
  robustness: (1) To avoid bias, the examples originate from an app distinct from the subject apps

                                                                    , Vol. 1, No. 1, Article . Publication date: March 2026.
8                                   Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


  used in our evaluation. (2) The examples representatively demonstrate the usage of key APIs (e.g.,
  findWidget, exists) and the mapping to the target Hoare logic structure (⟨𝑃, 𝐼, 𝑄⟩). Empirically,
  we found that two diverse examples are sufficient for the model to generalize the generation
  pattern while maintaining token efficiency.
• Property Description (⑤): The component describes the task with the property description and
  highlights that the output should follow the format of the example executable properties.
• Constraints (⑥): To ensure the output is machine-readable, we enforce strict constraints. We
  explicitly instruct the LLM to generate only the code snippet without verbose explanations or
  markdown formatting, which facilitates the automated extraction of the generated properties.
  Upon construction, the complete prompt is fed into the LLM, which then synthesizes the final
executable properties.

4   Evaluation
We aim to answer the following research questions:
• RQ1: What is the correctness of executable properties generated by iPBT based on property
  descriptions? How important are the widget functionality annotations in affecting the correctness?
• RQ2: To what extent can iPBT reduce manual effort? What are the differences in the complexity
  of the natural language property descriptions and executable properties generated by iPBT?
• RQ3: How robust is iPBT in generating executable properties based on diverse property descrip-
  tions?
Large Language Models. We evaluate our approach using two representative large language
models for executable property generation: one closed-source model (GPT-4o) and one open-source
model (DeepSeek-V3). This selection allows us to assess the effectiveness of our approach across
different model ecosystems. In addition, we employ a multimodal large language model (GPT-4o
mini) to generate functionality annotations for UI widgets, as it supports both textual and visual
inputs.
Writing property descriptions. To evaluate our approach, we construct property descriptions
based on the dataset from the prior work Kea [75], which provides 124 diverse executable properties
across eight popular open-source Android apps covering diverse app categories, e.g., tools, editor,
education, and audio player. Importantly, each property is derived from a distinct real-world
historical bug, ensuring that the dataset reflects practical issues encountered in diverse app contexts.
Since these bugs span different functional modules of the apps (e.g., navigation, data management,
and configuration), the resulting properties capture a broad spectrum of app behaviors. Moreover,
as a benchmark originally curated for evaluating property-based testing of Android apps, this
dataset offers both diversity and practical relevance, making it well-suited for our evaluation. The
dataset also includes the corresponding bug reports to facilitate understanding and reproduction of
each bug.
   To construct the natural language property descriptions, we followed a three-step process:
(1) Property collection. We collected 124 properties from Kea’s dataset, including the executable
properties, associated bug reports, and APK files. (2) Property understanding. Each line of an
executable property typically represents a UI event, making it difficult to infer the corresponding
widget based solely on its identifier. To gain a comprehensive understanding, we manually installed
the associated APK files on mobile devices and interacted with the apps to reach the states satisfying
the precondition. Then, we executed the executable properties on the app to observe each step.
This process can help us understand each component of the executable properties, including the
precondition, interaction scenario, and postcondition. (3) Property description construction. One
author wrote each property description in natural language based on the executable properties

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                               9


and observed app behavior. To ensure clarity and correctness, the remaining co-authors discussed
together to resolve any inconsistencies or ambiguities through iterative revision.
Constructing enriched widget context. The first step of iPBT aims to construct the enriched
context for the widgets in each app. While our approach supports diverse data collection methods
(as detailed in Section 3.1.1), we employed an automated strategy in our evaluation to minimize
manual intervention. We utilized DroidBot [71], a popular open-source automated GUI testing
framework, extended with a random exploration strategy, to perform a three-hour exploration on
each app [60]. We acknowledge that achieving full coverage via automated exploration remains a
long-standing challenge in both research and practice. However, advancing exploration algorithms
is orthogonal to our primary contribution of property generation. To decouple the effectiveness of
our generation approach from the limitations of the exploration tool, we additionally executed the
"main path" provided in the dataset, comprising events from the app entry to the state satisfying the
property’s precondition. This ensures that all relevant UI widgets involved in the target properties
are captured, providing a fair basis for evaluating iPBT’s generation capabilities. Finally, iPBT
extracts the page and widget information and leverages GPT-4o mini to construct the enriched
context.
Correctness of the generated executable properties. The 124 executable properties in the Kea
dataset are derived from 124 historical bugs. Thus, verifying whether the generated executable
properties by iPBT can reproduce the associated bug serves as a primary indicator of its correctness.
Specifically, for each executable property, we follow these steps: (1) Start from the app’s entry
point and navigate along the bug-triggering path. (2) Interact with the app until reaching the state
that satisfies the preconditions described in the executable property. (3) Execute the generated
executable properties to determine whether it successfully triggers the historical bug. The whole
process can be automatically performed by running the scripts (from app entry to the state satisfying
the precondition) and executable properties in Kea.
   For a few cases, even the generated executable properties are able to trigger the historical
bug, they may not faithfully capture the intended logic of the original property. For example, the
generated executable property may omit some conditions in the precondition. To address this, we
additionally evaluate whether the generated properties preserve original intent from the following
two dimensions:
• Precondition and postcondition. We verify whether the clauses and logical operators in
  the generated executable properties match those in the ground-truth. Missing or incorrect
  preconditions or postconditions can lead to inconsistencies in specific scenarios.
• Interaction scenario. First, we check whether the event sequence in the interaction scenario
  matches the ground-truth, where each event contains the event action and the target UI widget.
  Second, for executable properties containing conditional branches, we check whether these
  branches are consistent with the intended logic. Extra or missing branches may still allow bug
  reproduction but fail to reflect the precise execution flow of the original executable property.
   To ensure reliability, two co-authors of this paper independently evaluated each executable prop-
erty. We measured inter-rater agreement using Cohen’s Kappa, achieving a substantial agreement
(𝜅 = 0.96). Any disagreements were resolved through discussion.
   In Android app testing, UI widgets are located using identifiers (e.g., resource_id, text). Therefore,
the same UI widget can be matched through different identifiers. For example, text="Settings"
and resourceId="app.settings" both refer to the same button, which navigates to the system
configuration interface when clicked. When evaluating the widget matching in the generated
executable properties, we treat such cases as correct matches as long as the different attributes
refer to the same underlying widget.

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
10                                    Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su




(a) Page of Antennapod (b) Page of Antennapod   (c) Page of Markor   (d) Page of Markor   (e) Page of AnkiDroid (f) Page of AnkiDroid
       (Correct)             (Incorrect)             (Correct)            (Incorrect)            (Correct)            (Incorrect)


Fig. 3. Examples of failure cases. The green boxes annotate the correct widgets and red boxes annotate the
incorrect widgets.




4.1    RQ1: Correctness
Evaluation setup. To evaluate the correctness of iPBT in generating executable properties, we
constructed 124 prompts with 124 property descriptions based on the designed prompt template
(Table 2). To ensure deterministic and stable results, we set the temperature parameter of the LLM
to 0, a setting commonly used in prior work [18, 46, 77]. For each prompt, we sequentially invoked
the LLM and recorded the generated executable properties as the output. Moreover, we conduct
an ablation study to evaluate the contribution of the widget functionality annotation module.
Specifically, we removed the functionality annotation of each widget and then repeated the same
experimental procedure to assess its impact. We also conduct the ablation study in RQ2 and RQ3 in
the same process.
Evaluation results. Based on our experimental statistics, for all 124 property descriptions, both
GPT-4o and DeepSeek-V3 successfully generated 118 correct executable properties, achieving an
accuracy rate of 95.2%. These results demonstrate the effectiveness of iPBT and highlight its ability
to reliably translate natural language descriptions into executable properties.
   For the six failure cases, all errors manifested as incorrect widget identifiers generated by iPBT.
Upon further analysis, we found that in five cases, the failures were caused by the presence of other
widgets within the same app that shared similar functionality with the target widget, which misled
iPBT during the matching process. In the remaining case, the error originated from an ambiguous
functionality annotation generated by iPBT, which subsequently caused the executable property to
be matched to the wrong widget. Fig. 3 presents three examples from different apps, where the
correct widgets are highlighted with green boxes and incorrect ones with red boxes. The first two
cases (Fig. 3(a)–(d)) failed because different widgets in the same app shared similar functionality.
The third case (Fig. 3(e)–(f)) failed due to ambiguity in the functionality annotation generated by
iPBT. Specifically, iPBT produced the semantic label "Current card number" and the functionality
description "Indicates the number of the current card being viewed". In reality, the widget represents
the number of the currently selected card, which differs subtly from the generated annotation and
led to an incorrect mapping.
   Ablation study. The result shows that after removing the functionality annotation of widgets,
iPBT successfully generated 93 and 96 correct executable properties on GPT-4o and DeekSeek-V3,
corresponding to accuracy rates of 75.0% and 77.4%, respectively. Compared with the full setting
(118 correct executable properties, 95.2% accuracy), this represents a performance drop of 20.2
and 17.8 percentage points, respectively. This substantial decrease highlights the importance of

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                              11


                                         Table 3. Groups in user study


 Group (Participant ID)          Writing property descriptions              Writing executable properties
 Group A (P1-P5)                            Property 1, 3, 5                           Property 2, 4, 6
 Group B (P6-P10)                           Property 2, 4, 6                           Property 1, 3, 5



functionality annotations in helping the LLM accurately understand widgets and generate correct
executable properties.

4.2   RQ2: User study
Evaluation setup. In RQ2, we conduct a controlled human-subject experiment to assess how
iPBT actually supports users in practice. Specifically, participants were asked to write executable
properties manually, as well as to provide property descriptions for iPBT to generate executable
properties. We then analyzed their time cost and correctness across various tasks to evaluate how
iPBT can save manual effort. Our experimental design follows prior work [32, 61, 76].
   Dataset of the user study. We selected 124 executable properties from RQ1 and categorized them
into three groups (low, medium, and high complexity) using the metric defined in Kea. This metric
accounts for the number of logical clauses and operators in the pre- and postconditions, as well as
the number of events in the interaction scenario. From each complexity group, we randomly selected
2 executable properties from different apps, resulting in a final dataset of 6 executable properties:
Property 1 and Property 2 (low complexity), Property 3 and Property 4 (medium complexity), and
Property 5 and Property 6 (high complexity). By selecting executable properties from different
complexity levels, we ensure that the evaluation captures diverse levels of difficulty while keeping
the number of tasks manageable for participants. For each property, we installed the corresponding
app on an Android emulator and recorded a video walkthrough. Each video demonstrated how to:
(1) reach the state that satisfies the precondition, (2) perform the functionality in the interaction
scenario, and (3) verify the expected behavior in the postcondition.
   Participants. We recruited 10 participants for our user study, a sample size similar to previous
related work [9, 61, 84] (which recruited 10, 8, and 12 participants, respectively). All participants
are graduate students majoring in software engineering, with at least four years of programming
experience and familiarity with Python programming. This ensured that they had the necessary
technical background to understand and write the executable properties without using iPBT. A
prior study has shown that graduate students can serve as professionals in software engineering
tasks [54]. In addition, none of the participants was from the authors of this paper.
   Procedure. We designed the study as a conventional within-subject controlled experiment, in
which each participant was required to write both property descriptions and executable properties.
To avoid learning bias caused by increased familiarity, no participant wrote both the description and
executable version of the same property. To achieve this, the 10 participants were evenly divided
into two groups (Group A and Group B) with comparable programming expertise, and each group
was assigned different tasks for the same property (as shown in Table 3). In total, this resulted in 60
property-writing tasks.
   At the beginning, participants attended a dedicated tutorial introducing the study’s background,
the concept of properties in Android apps, and the procedures for writing them. Following the
tutorial, each participant was given an example property video and asked to write both the property
description and the corresponding executable properties. This warm-up exercise, which lasted ap-
proximately 45 minutes, familiarized participants with the task and provided step-by-step guidance.

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
12                                  Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


                               1750            Writing property descriptions
                               1500            Writing executable properties
                               1250
                               1000
                                750
                                500
                                250
                                  0
                                       L1         L2        M1         M2      H1   H2
Fig. 4. Time consumption of participants in writing property descriptions and executable properties. The
x-axis represents different properties, and the y-axis represents the time spent (in seconds).


   After the tutorial, participants independently completed six distinct tasks. For each task, they
were provided with recorded videos containing the necessary information. The study was conducted
in a preconfigured desktop environment, where participants used Visual Studio Code to write their
property descriptions and executable properties. To ensure fairness, code auto-completion features
were disabled.
   We recorded the time each participant spent on the tasks (including the time spent on writing
property descriptions or executable properties, reading the documentation when needed, and self-
checking the written properties). We also collected all written property descriptions and executable
properties. For each collected property description, we leverage iPBT to generate the corresponding
executable properties. The correctness of the generated code was then assessed using the evaluation
metrics described in our experimental setup.
   Complexity of property description and code. We measured the complexity of the 124 property
descriptions and their corresponding executable properties using character count. For instance, the
natural language step "click the undo button" has a complexity of 21 characters. This measurement
allows us to quantitatively compare the writing effort required for natural language descriptions
versus executable properties.
Evaluation results. We conducted a detailed analysis of the time efficiency and correctness of
executable property generation under two settings: (1) writing property descriptions in natural
language and generating executable via iPBT, and (2) manually writing executable properties.
• Time efficiency: Fig. 4 shows the time spent per task in both approaches. We can see that
  writing natural language property descriptions significantly reduces the time required to produce
  executable properties. On average, our approach achieved a 56% reduction in time, compared to
  manual writing property descriptions (272.7s vs 625.7s). For low-complexity executable properties
  (e.g., L2), time savings reached up to 72%, highlighting the efficiency gains of leveraging LLMs
  for code generation.
• Correctness: Table 4 presents the number of correctly generated executable properties in different
  approaches. Writing property descriptions with iPBT produced 29 correct executable properties,
  outperforming the manual approach, which yielded 26 correct executable properties. One incorrect
  executable property generated by iPBT approach stemmed from a user typo (a long-click was
  mistakenly written as a click). In contrast, most errors in the manually written executable
  properties were caused by participants’ unfamiliarity with the Kea framework, such as incorrect
  API usage in postconditions.
  To better understand these differences, we identified two main contributing factors. First, when
manually writing executable properties, participants needed additional time to inspect UI attributes

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                                 13


              Table 4. The correctness of the executable properties generated by different ways.

 Task                                 LLM                Setting            L1    L2    M1     M2    H1     H2     Total
                                                   with annotations          5     5     5      4      5     5      29
                                     GPT-4o
                                                  without annotations        4     5     5      1      4     5      24
 Writing property descriptions
                                                   with annotations          5     5     5      4      5     5      29
                                  DeepSeek-V3
                                                  without annotations        4     5     0      1      4     2      16
 Writing executable properties                                               5     5     4      5      3     4      26


                                                                                       Property Descriptions
      1200                                                                             Executable Properties
      1000
        800
        600
        400
        200
          0
          OmniNotes    Markor     SimpleTask     Amaze ActivitydiaryAntennapod AnkiDroid Transistor

Fig. 5. Comparison of the complexity (measured in character count) between natural language property
descriptions and their corresponding executable properties across eight Android apps.


for widget identification, while iPBT automated this process based on matching widgets from
property descriptions. Second, even though all participants received training on Kea, we observed
frequent consultation of the documentation during manual writing, especially for framework-
specific APIs. By contrast, iPBT allows users to focus on specifying intended behaviors in natural
language, while delegating low-level implementation details to the LLM. This not only improves
efficiency but also reduces errors caused by limited familiarity with the framework.
   Ablation study. In Table 4, we can see that without functionality annotations, iPBT only generated
24 and 16 correct executable properties, respectively. This result indicates the necessity of the
designed functionality annotations in our approach.
   Complexity comparison. Fig. 5 presents the complexity comparison between the 124 natural
language property descriptions and their corresponding executable properties, across eight popu-
lar Android apps. The x-axis denotes the app names, while the y-axis represents the complexity,
quantified by the number of characters. On average, the property descriptions contain 211.3 char-
acters, whereas the corresponding executable properties contain 555.0 characters. This significant
difference in complexity highlights the efficiency of the natural language-based approach. Property
descriptions are inherently easier to write than their corresponding executable properties, further
emphasizing the practicality of iPBT in executable properties generation.

4.3   RQ3: Robustness
Evaluation setup. Building on the user study in RQ2, which evaluates how iPBT supports users
in practice, RQ3 serves as a complementary experiment that examines robustness. While RQ2
considers the property descriptions actually written by participants, RQ3 evaluates whether iPBT

                                                                   , Vol. 1, No. 1, Article . Publication date: March 2026.
14                                  Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


can generate correct executable properties when faced with semantically equivalent but differently
worded property descriptions, simulating the natural variation in how different users might express
the same intent.
   Prior work has shown that LLMs can generate high-quality paraphrases with greater lexical and
syntactic diversity than those produced by crowd workers [7, 8]. Following this line of work, we
use an LLM to automatically generate paraphrased versions of each original property description.
   To avoid bias from using the same model for both paraphrasing and code generation, we employ
a different LLM, Llama-3.1-405B, for paraphrase generation. For each property description, we
invoke Llama-3.1 multiple times to generate a pool of candidate paraphrases. Specifically, for each
property description, we invoke the Llama-3.1 10 times using the designed prompt (the prompt is
provided in the artifact due to space limitations), generating 10 paraphrased descriptions per call,
resulting in a total of 100 paraphrased variations per property description.
   However, the LLM-generated variations may not be mutually diverse between them. To select a
diverse subset of 10 paraphrases per property description, we employed the BLEU score [50] to
quantify lexical similarity and guide the selection process. The BLEU score was originally developed
for assessing machine translation quality by measuring the similarity between machine-generated
translations and human-written reference translations. In the context of paraphrasing, a lower BLEU
score indicates higher diversity compared to the original text. Specifically, we adopted a greedy
selection strategy to identify a set of paraphrases that are mutually diverse. First, we identified the
two most mutually diverse paraphrases (i.e., the pair with the lowest BLEU score between them)
to initialize the selection set 𝑆. Then, in each subsequent iteration, we computed the Self-BLEU
score [86] between each remaining candidate and the current set 𝑆, which quantifies similarity
to the existing set. The candidate with the lowest Self-BLEU score was added to 𝑆. This process
continued until 𝑆 contained 10 paraphrased descriptions. Formally, the objective of our selection
process is to identify a subset 𝑆 ⊆ 𝐶 of size 𝑘, where 𝐶 is the set of all candidate paraphrases and 𝑘
= 10) that minimizes the Self-Bleu score:

                                                           1        ∑︁
                                       arg min                            BLEU(𝑥, 𝑦)
                                       𝑆 ⊆𝐶, |𝑆 |=𝑘 |𝑆 |(|𝑆 | − 1) 𝑥,𝑦 ∈𝑆
                                                                 𝑥≠𝑦


This selection strategy ensures that the chosen paraphrases are not only diverse relative to the
original text but also mutually diverse within the set, resulting in a robust dataset for evaluating
our approach under varied natural language expressions.
  In RQ3, we focus on 118 property descriptions from RQ1 that were successfully translated into
correct executable properties. Using the paraphrasing procedure described above, we generate
10 paraphrases for each property description, resulting in a total of 1,180 paraphrased property
descriptions. We then evaluate whether iPBT can still generate executable properties from these
paraphrases.
Evaluation results. As shown in Fig. 6, GPT-4o and DeepSeek-V3 generated 1,034 and 1,032
correct executable properties, respectively, achieving accuracy rates of 87.6% and 87.5%. Specifically,
both models successfully generated correct executable properties for 81.0% (956/1,180) of the
descriptions, demonstrating that iPBT is robust to variations in natural language expressions.
Among the remaining cases, 6.6% (78/1,180) of the descriptions were handled correctly by GPT-4o
but not by DeepSeek-V3, while 6.4% (76/1,180) were correctly processed by DeepSeek-V3 but not by
GPT-4o. In 5.9% (70/1,180) of the cases, both models failed to generate correct executable properties.
  To further understand the effectiveness of iPBT in executable property generation, we analyzed
the symptoms of the failure cases. Finally, we identified four main categories of failure symptoms.

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                                   15


                                       70                                                                   69
                                                    Widget Mismatch                                  70               84
                                      (5.9%)
                                                                                            39     (5.9%)
                                                Logic Incompleteness                   30
     78            956                76                       78              956                 76
                                                                           11
    (6.6%)        (81.0%)          (6.4%)          Logic Redundancy
                                                             (6.6%)        9 (81.0%)             (6.4%)
                                                                                                            GPT-4o
          G                       D                              G
                                                  Semantic Deviation                  27
                                                                                     25          D          DeepSeek-V3
                                                                       0        20          40       60          80

Fig. 6. Venn diagrams of executable prop-                     Fig. 7. Failure symptom distribution.
erty generation. G: GPT-4o; D: DeepSeek-
V3.



   Fig. 7 presents the distribution of failure symptoms in the incorrect executable properties gener-
ated by iPBT with two LLMs. The results reveal that UI widget mismatch is the most frequent error
type, accounting for 47.3%(69/146) and 56.8%(84/148) of the total failures in executable properties
generated by GPT-4o and DeepSeek-V3, respectively. This indicates that while LLMs demonstrate
a general understanding of UI semantics, they still struggle to precisely align UI descriptions
with their corresponding identifiers in some cases. Incomplete code is the second most common
symptom, responsible for 26.7%(39/146) and 20.3%(30/148) of the errors. This is followed by other
logic errors (18.5%(27/146) and 16.9%(25/148)), which typically involve incorrect API usage or faulty
postcondition assertions. Redundant code ranks fourth, contributing 7.6%(11/146) and 6.1%(9/148)
of the failures. The similar distribution of errors across GPT-4o and DeepSeek-V3 demonstrates that
iPBT performs robustly across different LLMs, while also revealing typical challenges in executable
property generation.
• Widget Mismatch. Widget mismatch occurs when the generated executable properties fail
  to match the intended UI widget. As shown in Fig. 8(a), an example of incorrect executable
  properties generated by GPT-4o for the app AntennaPod [4]. The expected UI event is to click
  the option button, but the UI widget identifier highlighted in red fails to match the target widget.
  In contrast, Fig. 8(b) shows the expected correct executable properties, where the UI widget
  identifier accurately matches the target UI widget.
• Logic Incompleteness. Logic incompleteness refers to generated executable properties that
  omit statements needed. This manifests primarily as (1) incomplete preconditions/postconditions,
  (2) missing conditional branches in interaction logic. Such incompleteness may lead to two
  consequences during execution: (1) execution failures during the property checking, (2) false
  positives in test results. Fig. 8(c) shows an incomplete code example, which was generated by
  DeepSeek-V3 for the app OmniNotes [44]. The precondition fails to verify the existence of the
  "note title" UI widget, causing the testing framework to raise an exception while executing the
  interaction scenario when the widget is absent. Fig. 8(d) shows correct implementation, which
  includes the necessary existence check for the UI widget ( highlighted in green).
• Logic Redundancy. Logic Redundancy refers to the generated code containing unnecessary
  statements, primarily manifested as redundant UI events (e.g., click) in interaction logic. Such
  errors typically lead to execution failures during property checking. Fig. 8(e) shows an example,
  which was generated by GPT-4o for the app AntennaPod. The last line (highlighted in red)
  introduces an unnecessary event to open the notification page. Then, since the GUI state changes
  to the notification page, the assertion statement will fail during the property checking. Fig. 8(f)
  shows the expected executable properties.

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
16                                         SMU Classification:
                                         Yiheng   Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su
                                            Restricted




       Failure Symptom: UI widget mismatch                                 Failure Symptom: Redundant code
     # Tap the settings icon                                         # Ensure a notification appears, indicating the queue has no
     findWidget(id="butShowSettings").click()                        episodes.
                                                                     open_notification()
                   (a) Incorrect property code
                                                                     assert findWidget(id="emptyViewTitle").click()
     # Click the option button
     findWidget(description="More options").click()                                   (e) Incorrect property code
                                                                     # Assert there is a text exists that indicates there is no episode
                   (b) Expected property code                        in the queue
                                                                     assert findWidget(id="emptyViewTitle").click()
         Failure Symptom: Incomplete code                                             (f) Expected property code
     # A search query is available and a note title is known, with
     no "SETTINGS" text found                                               Failure Symptom: Other logic error
     @precondition(
                                                                     # Identify the file with the ".zip" suffix
       findWidget(id="search_query").exists() and
                                                                     zip_file = self.device(textContains=".zip")
       not findWidget(text="SETTINGS").exists()
                                                                     zip_file_name = zip_file.get_text()
     )
                                                                     # Select the identified file
                   (c) Incorrect property code                       zip_file.long_click()
     # Search query exists and note title exists and text                             (g) Incorrect property code
     "SETTINGS" not exists
     @precondition(                                                  # Get the zip file whose name contains ".zip"
       findWidget(id="search_query").exists() and                    zip_file = self.device(textContains=".zip")
       findWidget(id="note_title").exists() and not                  zip_file_name = zip_file.get_text()
       findWidget(text="SETTINGS").exists()                          # Click it
     )                                                               zip_file.click()

                   (d) Expected property code                                         (h) Expected property code


Fig. 8. Illustrative examples of failure symptoms. Incorrect and correct implementations are marked in red
and green, respectively, with property descriptions in gray (code snippets are simplified for clarity).


• Semantic Deviation. This refers to other semantic errors, such as incorrect API usage or
  assertions in postconditions. These types of errors often result in property execution failures or
  inaccurate test results, including false positives and false negatives. Fig. 8(g) shows an example,
  which was generated by DeepSeek-V3 for the app Amaze [1]. The implementation (highlighted
  in red) tries to long-click a file. However, the property description indicates that it should click
  the file. Fig. 8(h) shows the expected executable properties.
   Ablation study. We remove the widget functionality annotations from the prompt. The results
show that iPBT generated 833 and 822 correct executable properties on GPT-4o and DeekSeek-V3,
corresponding to accuracy rates of 70.6% and 69.7%, respectively. In comparison, the full setting
achieved 87.6% and 87.5% accuracy. This represents a performance drop of 17.0 and 17.8 percentage
points, respectively, highlighting the critical role of functionality annotations in guiding the LLM
to generate correct executable properties.

5    Discussion and Lessons
   Generality of our work. First, the evaluation results demonstrate that our approach is effective in
translating natural language property descriptions into executable properties for 124 properties.
These 124 properties are collected from the existing dataset [75] that contains 124 real functional
bugs across 8 popular apps. It is interesting to further understand the generality of the approach
on a larger set of apps. Second, our work’s core methodology is translating informal specifications
into formal properties in mobile apps, and it can be extended naturally to other similar applications,
e.g., web applications (which also have UI widgets and user interactions) and formal specification
generation in program verification. For example, many verification tools (e.g., Dafny [53]) require
formal pre/post-conditions or invariants, which share structural similarities with PBT properties.

, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                              17


   Applying LLMs to PBT of mobile apps. Although existing PBT frameworks are effective, specifying
properties requires significant manual effort and deep familiarity with UI structures and framework-
specific APIs. Allowing testers to write properties in structured natural language shifts this burden.
In practice, this change enables testers to focus on what the app should do, rather than how to
encode it, substantially lowering the barrier to using PBT.
   Importance of UI semantic grounding. A central lesson is that UI semantic grounding is essential.
Without widget functionality annotations, LLMs frequently select incorrect UI widgets, even
when the generated code logic is otherwise correct. Our ablation results confirm that raw widget
identifiers alone are insufficient in real-world apps, where identifiers are often ambiguous or poorly
named.
   Designing natural language as an effective specification interface. We found that natural language
property descriptions should not be treated as completely free-form input. In practice, adopting a
lightweight structure (precondition–interaction–postcondition) significantly reduces ambiguity
and improves generation quality. This indicates that natural language specifications function as an
interface between humans and LLMs, and even minimal structural constraints can greatly enhance
reliability without harming usability.
   What the failures reveal. Most failures arise from UI widget mismatch, not from incorrect control
flow or API usage. This indicates that current LLMs can generally synthesize reasonable test logic,
but still struggle with fine-grained UI disambiguation when multiple widgets have similar semantics.
In addition, paraphrased descriptions sometimes omit implicit constraints, leading to incomplete
preconditions or redundant actions. This suggests that LLM-based property synthesis is robust to
linguistic variation, but sensitive to semantic underspecification.
   Complementing rather than replacing existing PBT frameworks. Rather than replacing existing
property-based testing frameworks, iPBT complements them by addressing one of their most
labor-intensive stages: executable property authoring. Since iPBT does not modify the execution or
input generation mechanisms of PBT, it can be integrated into existing workflows with minimal
disruption. This design choice proved important for maintaining practicality.
   Threats to Validity. Our work may suffer from some threats to validity. First, the properties in our
experiment may not fully represent those in real-world apps. To mitigate this threat, we selected
properties from the Kea dataset [75], where each property is derived from a real historical bug and
covers important app functionalities. In the future, we will include a broader range of properties
from industrial apps. Also, as we utilize dynamic exploration to collect the UI widget identifier
list, it may not capture all the UI widgets in the app. This insufficient exploration is also identified
as a common challenge of input generation in testing apps [6, 60]. Second, the natural language
property descriptions may differ from how practitioners describe properties in practice. To mitigate
this, the initial descriptions were authored by an experienced property-based testing expert and
carefully reviewed by all co-authors. In addition, RQ2 evaluates descriptions written by 10 real users,
and RQ3 further assesses robustness under diverse paraphrased descriptions. Third, our evaluation
considers only two LLMs (GPT-4o and DeepSeek-V3), which may limit generalizability. We selected
them as representative closed-source and open-source state-of-the-art models, and future advances
in LLMs are expected to further improve performance. Finally, there is a potential concern that
the models may have seen the evaluated properties during training. This threat is unlikely, as
both models were trained before the release of the Kea dataset, and all property descriptions were
manually created and have not been publicly available. To further avoid bias, we used a different
LLM (Llama-3.1) for paraphrasing in the robustness evaluation.

                                                                 , Vol. 1, No. 1, Article . Publication date: March 2026.
18                                  Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


6    Related Work
LLM for SE tasks. The rapid advancement of LLMs has spurred significant research into their
application across various software engineering domains. In code completion [16, 42, 81], LLMs have
demonstrated strong capabilities in providing context-aware suggestions and recommendations.
The domain of program repair [73, 82] has also benefited from LLMs’ ability to understand and
fix bugs in code. Moreover, comprehensive studies [22, 83] have systematically evaluated LLMs’
capabilities across multiple software engineering tasks, understanding the applications, effects, and
limitations. Our work specifically addresses the area of executable property generation for PBT.
Automated test generation. The research on automated test generation can be categorized
into two types: traditional and deep learning-based approaches. Traditional approaches include
techniques such as fuzzing [40], symbolic execution [21, 56, 64], search-based [19, 47]. These
approaches primarily aim to achieve high test coverage. However, these approaches often struggle
to generate assertions [49, 57].
   Deep learning-based approaches leverage the capabilities of pre-trained language models to
generate tests from code snippets [28, 65, 77, 79]. For example, AthenaTest [65] leverages a
transformer model, BART [30], to generate unit test cases based on the given method input. In
recent years, LLM-based approaches have shown promising results in test generation [14, 26, 28, 79].
TiCoder [28] leverages LLMs to formalize user intent into tests, and ChatTester [79] can generate
unit tests through interactive conversations with LLMs. While these approaches focus on generating
unit test cases, some recent work has moved closer to property-based testing. Endres et al. [17]
conduct a study to evaluate the LLM’s capability of generating postconditions for individual
functions based on the function comments. Vikram et al. [67] propose an approach to leverage
LLM for generating property-based tests from specifications for Python libraries. Liu et al. [34]
target properties for smart contracts. In contrast, our approach focuses on mobile apps, generating
executable properties (including preconditions, interaction scenarios, and postconditions) from
natural language descriptions to capture real app behavior while reducing manual effort.
   In mobile app testing, recent work like Kea [75], PBFDroid [61], and PDTDroid [62] has demon-
strated the effectiveness of PBT in detecting functional and privacy bugs. However, these frame-
works primarily focus on the execution and input generation phases, assuming the existence of
high-quality executable properties. Consequently, they still require significant manual effort and
domain expertise to write executable properties. Our work complements these approaches by
automating the executable property generation.
   Some work leverages LLM to analyze GUI pages during the dynamic exploration to find data
inconsistency bugs [23], functional bugs [36], or inconsistencies between app design and implemen-
tation [33]. Our work differs from these approaches in its focus: rather than detecting bugs directly,
we generate executable properties that can be used by existing PBT frameworks. Recently, different
agents have been proposed to automatically perform tasks on mobile apps [52, 68, 70, 71, 80]. These
works focus on executing user-specified tasks, whereas our work centers on generating executable
properties to guide property-based testing.
UI widget understanding. Various approaches try to understand the widget from different
perspective [31, 35, 37, 38, 72, 74]. For example, IconIntent [74] leverages program analysis and
computer vision techniques to identify sensitive UI widgets in Android apps. DroidGem [38]
leverages deep neural networks to predict the permissions behind the UI widgets. HintDroid [35]
aims to generate hint-text of the UI widget to improve the accessibilty for low-vision users. In
contrast, our work focuses on constructing UI widget context for matching the property description
with widget identifiers in PBT.



, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                                  19


7   Conclusion
In this work, we presented a novel approach to automatically generate executable properties
from natural language property descriptions. Our approach lowers the manual effort and technical
expertise required for property-based testing of mobile apps. In detail, we first construct the enriched
widget context by extracting GUI information and leveraging MLLMs to generate functionality
annotations of widgets. Then, we employ in-context learning to guide LLMs in generating executable
properties with the carefully designed prompt. Evaluation results show that our approach achieves
95.2% accuracy on original property properties, maintains over 87% accuracy on paraphrased
variations, and substantial reductions in manual effort according to our user study. These results
demonstrate that iPBT makes PBT more practical and accessible for mobile app testing.

References
 [1] AmazeFileManager Team. 2024. AmazeFileManager. https://github.com/TeamAmaze/AmazeFileManager
 [2] Android. 2025. Describe each UI element. https://developer.android.com/guide/topics/ui/accessibility/apps#describe-
     ui-element
 [3] Android. 2025. Layouts in views. https://developer.android.com/develop/ui/views/layout/declaring-layout
 [4] AntennaPod Team. 2024. AntennaPod. https://github.com/AntennaPod/AntennaPod
 [5] Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger. 2006. Testing telecoms software with Quviq QuickCheck.
     In Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang. 2–10.
 [6] Farnaz Behrang and Alessandro Orso. 2020. Seven reasons why: an in-depth study of the limitations of random test
     input generation for Android. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software
     Engineering. 1066–1077.
 [7] Auday Berro, Vitor Gaboardi dos Santos, Boualem Benatallah, and Khalid Benabdeslem. 2025. LLMs to Replace
     Crowdsourcing in Generating Syntactically Diverse Paraphrases for Task-Oriented Chatbots. In International Conference
     on Advanced Information Systems Engineering. Springer, 145–162.
 [8] Jan Cegin, Jakub Simko, and Peter Brusilovsky. 2023. ChatGPT to Replace Crowdsourcing of Paraphrases for Intent
     Classification: Higher Diversity and Comparable Model Robustness. In Proceedings of the 2023 Conference on Empirical
     Methods in Natural Language Processing. 1889–1905.
 [9] Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From ui design image to gui skeleton:
     a neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference
     on Software Engineering. 665–676.
[10] Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In
     Proceedings of the fifth ACM SIGPLAN international conference on Functional programming (ICFP). 268–279.
[11] Ermira Daka and Gordon Fraser. 2014. A survey on unit testing practices and problems. In 2014 IEEE 25th International
     Symposium on Software Reliability Engineering. IEEE, 201–211.
[12] Alaka Das and Rakesh Chandra Balabantaray. 2019. MyNLIDB: a natural language interface to database. In 2019
     International conference on information technology (ICIT). IEEE, 234–238.
[13] DeekSeek. 2024. Introducing DeepSeek-V3. https://api-docs.deepseek.com/news/news1226
[14] Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models
     are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM
     SIGSOFT international symposium on software testing and analysis. 423–435.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional
     Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2019).
[16] Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh
     Nallapati, Parminder Bhatia, Dan Roth, et al. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file
     code completion. Advances in Neural Information Processing Systems 36 (2023), 46701–46723.
[17] Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Can large language models
     transform natural language intent into formal method postconditions? Proceedings of the ACM on Software Engineering
     1, FSE (2024), 1889–1912.
[18] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large
     language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on
     Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53.
[19] Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software.
     In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software


                                                                     , Vol. 1, No. 1, Article . Publication date: March 2026.
20                                  Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


     engineering. 416–419.
[20] Gherkin Team. 2025. Gherkin Reference. https://cucumber.io/docs/gherkin/reference/
[21] Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In Proceedings of
     the 2005 ACM SIGPLAN conference on Programming language design and implementation. 213–223.
[22] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu
     Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on
     Software Engineering and Methodology 33, 8 (2024), 1–79.
[23] Yongxiang Hu, Hailiang Jin, Xuan Wang, Jiazhen Gu, Shiyu Guo, Chaoyi Chen, Xin Wang, and Yangfan Zhou. 2024.
     Autoconsis: Automatic gui-driven data inconsistency detection of mobile apps. In Proceedings of the 46th International
     Conference on Software Engineering: Software Engineering in Practice. 137–146.
[24] John Hughes. 2016. Experiences with QuickCheck: testing the hard stuff and staying sane. In A List of Successes That
     Can Change the World: Essays Dedicated to Philip Wadler on the Occasion of His 60th Birthday. Springer, 169–186.
[25] John Hughes, Benjamin C Pierce, Thomas Arts, and Ulf Norell. 2016. Mysteries of dropbox: property-based testing
     of a distributed synchronization service. In 2016 IEEE International Conference on Software Testing, Verification and
     Validation (ICST). IEEE, 135–145.
[26] Zongze Jiang, Ming Wen, Jialun Cao, Xuanhua Shi, and Hai Jin. 2024. Towards Understanding the Effectiveness of Large
     Language Models on Directed Test Input Generation. In Proceedings of the 39th IEEE/ACM International Conference on
     Automated Software Engineering. 1408–1420.
[27] Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI-
     described RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification
     (ICST). IEEE, 131–141.
[28] Shuvendu K Lahiri, Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, Madanlal Musuvathi, Piali
     Choudhury, Curtis von Veh, Jeevana Priya Inala, Chenglong Wang, et al. 2022. Interactive code generation via
     test-driven user-intent formalization. arXiv preprint arXiv:2208.05950 (2022).
[29] Edmund SL Lam, Peilun Zhang, and Bor-Yuh Evan Chang. 2017. ChimpCheck: property-based randomized test
     generation for interactive apps. In Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New
     Paradigms, and Reflections on Programming and Software. 58–77.
[30] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov,
     and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation,
     translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019).
[31] Linlin Li, Ruifeng Wang, Xian Zhan, Ying Wang, Cuiyun Gao, Sinan Wang, and Yepang Liu. 2023. What you see is
     what you get? it is not the case! detecting misleading icons for mobile applications. In Proceedings of the 32nd ACM
     SIGSOFT International Symposium on Software Testing and Analysis. 538–550.
[32] Jingjing Liang, Ruyi Ji, Jiajun Jiang, Shurui Zhou, Yiling Lou, Yingfei Xiong, and Gang Huang. 2021. Interactive patch
     filtering as debugging aid. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE,
     239–250.
[33] Ruofan Liu, Xiwen Teoh, Yun Lin, Guanjie Chen, Ruofei Ren, Denys Poshyvanyk, and Jin Song Dong. 2025. GUIPilot:
     A Consistency-Based Mobile GUI Testing Approach for Detecting Application-Specific Bugs. Proceedings of the ACM
     on Software Engineering 2, ISSTA (2025), 753–776.
[34] Ye Liu, Yue Xue, Daoyuan Wu, Yuqiang Sun, Yi Li, Miaolei Shi, and Yang Liu. 2024. Propertygpt: Llm-driven formal
     verification of smart contracts through retrieval-augmented property generation. arXiv preprint arXiv:2405.02580
     (2024).
[35] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yuekai Huang, Jun Hu, and Qing Wang. 2024.
     Unblind text inputs: predicting hint-text of text input in mobile apps via LLM. In Proceedings of the 2024 CHI Conference
     on Human Factors in Computing Systems. 1–20.
[36] Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, and Qing
     Wang. 2024. Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps. arXiv preprint
     arXiv:2407.03037 (2024).
[37] Junayed Mahmud, Antu Saha, Oscar Chaparro, Kevin Moran, and Andrian Marcus. 2025. Combining language and
     app ui analysis for the automated assessment of bug reproduction steps. arXiv preprint arXiv:2502.04251 (2025).
[38] Vikas K Malviya, Yan Naing Tun, Chee Wei Leow, Ailys Tee Xynyn, Lwin Khin Shar, and Lingxiao Jiang. 2023. Fine-
     Grained In-Context Permission Classification for Android Apps Using Control-Flow Graph Embedding. In 2023 38th
     IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1225–1237.
[39] Meta. 2024. Meet Llama 3.1 . https://www.llama.com/
[40] Michał Zalewski. 2016. American Fuzzy Lop - Whitepaper. https://lcamtuf.coredump.cx/afl/technical_details.txt
[41] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana
     Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A


, Vol. 1, No. 1, Article . Publication date: March 2026.
From Natural Language to Executable Properties for Property-based Testing of Mobile Apps                                       21


     survey. Comput. Surveys 56, 2 (2023), 1–40.
[42] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.
     2022. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474 30 (2022).
[43] Liam O’Connor and Oskar Wickström. 2022. Quickstrom: property-based acceptance testing with LTL specifications.
     In Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation
     (PLDI). 1025–1038. doi:10.1145/3519939.3523728
[44] Omni-Notes Team. 2024. Omni-Notes. https://github.com/federicoiosue/Omni-Notes
[45] OpenAI. 2024. Introducing GPT-4o and more tools to ChatGPT free users. https://openai.com/index/gpt-4o-and-more-
     tools-to-chatgpt-free/
[46] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non-
     determinism of ChatGPT in Code Generation. arXiv e-prints (2023), arXiv–2308.
[47] Carlos Pacheco and Michael D Ernst. 2007. Randoop: feedback-directed random testing for Java. In Companion to the
     22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion. 815–816.
[48] Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit Paradkar. 2012. Inferring method
     specifications from natural language API descriptions. In 2012 34th international conference on software engineering
     (ICSE). IEEE, 815–825.
[49] Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J Hellendoorn. 2020.
     Revisiting test smells in automatically generated tests: limitations, pitfalls, and opportunities. In 2020 IEEE international
     conference on software maintenance and evolution (ICSME). IEEE, 523–533.
[50] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of
     machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318.
[51] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv
     preprint arXiv:2304.03277 (2023).
[52] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue
     Huang, et al. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326
     (2025).
[53] Rustan Leino. 2009. The Dafny Programming and Verification Language. https://dafny.org/
[54] Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015. Are students representatives of professionals in software
     engineering experiments?. In 2015 IEEE/ACM 37th IEEE international conference on software engineering, Vol. 1. IEEE,
     666–676.
[55] André Santos, Alcino Cunha, and Nuno Macedo. 2018. Property-based testing for the robot operating system. In
     Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation.
     56–62.
[56] Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A concolic unit testing engine for C. ACM SIGSOFT software
     engineering notes 30, 5 (2005), 263–272.
[57] Sina Shamshiri. 2015. Automated unit test generation for evolving software. In Proceedings of the 2015 10th Joint
     Meeting on Foundations of Software Engineering. 1038–1041.
[58] Simplenote Team. 2022. Simplenote. Retrieved 2025-5 from https://github.com/Automattic/simplenote-android
[59] John Ferguson Smart and Jan Molak. 2023. BDD in Action: Behavior-driven development for the whole software lifecycle.
     Simon and Schuster.
[60] Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su.
     2017. Guided, stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th Joint Meeting on
     Foundations of Software Engineering (FSE). 245–256.
[61] Jingling Sun, Ting Su, Jiayi Jiang, Jue Wang, Geguang Pu, and Zhendong Su. 2023. Property-Based Fuzzing for Finding
     Data Manipulation Errors in Android Apps. In Proceedings of the 31st ACM Joint European Software Engineering
     Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 1088–1100.
[62] Jingling Sun, Ting Su, Jun Sun, Jianwen Li, Mengfei Wang, and Geguang Pu. 2024. Property-Based Testing for Validating
     User Privacy-Related Functionalities in Social Media Apps. In Companion Proceedings of the 32nd ACM International
     Conference on the Foundations of Software Engineering. 440–451.
[63] Suresh Thummalapenta, Saurabh Sinha, Nimit Singhania, and Satish Chandra. 2012. Automating test automation. In
     2012 34th international conference on software engineering (ICSE). IEEE, 881–891.
[64] Nikolai Tillmann, Jonathan De Halleux, and Tao Xie. 2014. Transferring an automated test generation tool to practice:
     From Pex to Fakes and Code Digger. In Proceedings of the 29th ACM/IEEE International Conference on Automated
     Software Engineering. 385–396.
[65] Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2020. Unit test case
     generation with transformers and focal context. arXiv preprint arXiv:2009.05617 (2020).



                                                                         , Vol. 1, No. 1, Article . Publication date: March 2026.
22                                  Yiheng Xiong, Ting Su, Jingling Sun, Jue Wang, Qin Li, Geguang Pu, and Zhendong Su


[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
     Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[67] Vasudev Vikram, Caroline Lemieux, Joshua Sunshine, and Rohan Padhye. 2023. Can large language models write good
     property-based tests? arXiv preprint arXiv:2307.04346 (2023).
[68] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024.
     Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv
     preprint arXiv:2406.01014 (2024).
[69] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.
     Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing
     systems 35 (2022), 24824–24837.
[70] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and
     Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International
     Conference on Mobile Computing and Networking. 543–557.
[71] Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. 2023. Droidbot-gpt: Gpt-powered ui automation for android.
     arXiv preprint arXiv:2304.07061 (2023).
[72] Shengqu Xi, Shao Yang, Xusheng Xiao, Yuan Yao, Yayuan Xiong, Fengyuan Xu, Haoyu Wang, Peng Gao, Zhuotao Liu,
     Feng Xu, et al. 2019. Deepintent: Deep icon-behavior learning for detecting intention-behavior discrepancy in mobile
     apps. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 2421–2436.
[73] Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42
     each using ChatGPT. arXiv preprint arXiv:2304.00385 (2023).
[74] Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019. Iconintent: automatic identification of
     sensitive ui widgets based on icon classification for android apps. In 2019 IEEE/ACM 41st International Conference on
     Software Engineering (ICSE). IEEE, 257–268.
[75] Yiheng Xiong, Ting Su, Jue Wang, Jingling Sun, Geguang Pu, and Zhendong Su. 2024. General and Practical Property-
     based Testing for Android Apps. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software
     Engineering. 53–64.
[76] Chenyang Yang, Shurui Zhou, Jin LC Guo, and Christian Kästner. 2021. Subtle bugs everywhere: Generating documen-
     tation for data wrangling code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering
     (ASE). IEEE, 304–316.
[77] Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang,
     Qianxiang Wang, et al. 2024. On the evaluation of large language models in unit test generation. In Proceedings of the
     39th IEEE/ACM International Conference on Automated Software Engineering. 1607–1619.
[78] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal
     large language models. National Science Review 11, 12 (2024), nwae403.
[79] Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and
     improving chatgpt for unit test generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726.
[80] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025.
     Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in
     Computing Systems. 1–20.
[81] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu
     Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint
     arXiv:2303.12570 (2023).
[82] Quanjun Zhang, Chunrong Fang, Yang Xie, Yuxiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A Systematic
     Literature Review on Large Language Models for Automated Program Repair. arXiv preprint arXiv:2405.01466 (2024).
[83] Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen.
     2023. A Survey on Large Language Models for Software Engineering. arXiv preprint arXiv:2312.15223 (2023).
[84] Yu Zhao, Tingting Yu, Ting Su, Yang Liu, Wei Zheng, Jingzhi Zhang, and William GJ Halfond. 2019. Recdroid:
     automatically reproducing android application crashes from bug reports. In 2019 IEEE/ACM 41st International Conference
     on Software Engineering (ICSE). IEEE, 128–139.
[85] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier
     Bousquet, Quoc Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv
     preprint arXiv:2205.10625 (2022).
[86] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking
     platform for text generation models. In The 41st international ACM SIGIR conference on research & development in
     information retrieval. 1097–1100.




, Vol. 1, No. 1, Article . Publication date: March 2026.
