servo: Merge #19417 - Mutation Test: with more mutation strategies (from dsandeephegde:master); r=asajeffrey,jdm

<!-- Please describe your changes on the following line: -->

1. Added following mutation strategies:
 - If True (make if always true)
 - If False(make if always false)
 - Modify Comparision (<= to <, >= to >)
 - Plus To Minus
 - Minus To Plus
 - Changing Atomic Strings (make string constant empty)
 - Duplicate Line
 - Delete If Block
2. Randomized the test order.
3. Introduced logging instead of print.
4. Added retry mechanism when mutation cannot be performed on a file by a strategy.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes #18529 (github issue number if applicable).

<!-- Either: -->
- [X] These changes do not require tests because it is a python script to run mutation test and does not change any behavior.

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

Source-Repo: https://github.com/servo/servo
Source-Revision: 6aae59e7e50531f45d78495cf07970001ef05f86

--HG--
extra : subtree_source : https%3A//hg.mozilla.org/projects/converted-servo-linear
extra : subtree_revision : 5b9134e3b3c9bb21f2a14e759967f8c171c0a6f2

servo/python/servo/mutation/README.md

@@ -7,8 +7,17 @@ For more info refer [Wiki page](https://en.wikipedia.org/wiki/Mutation_testing).
 
 Here Mutation testing is used to test the coverage of WPT for Servo's browser engine.
 
-### Mutation Strategy
-This version of mutation testing consists of a Python script that finds random uses of && in Servo's code base and replaces them by ||. The expectation from the WPT tests is to catch this mutation and result in failures when executed on the corresponding code base.
+### Mutation Strategies
+The mutation test consists of a Python script that mutates random lines in Servo's code base. The expectation from the WPT tests is to catch this bug caused by mutation and result in test failures.
+
+There are few strategies to mutate source code in order to create bugs. The strategies are randomly picked for each file. Some of the strategies are:
+
+* Change Conditional flow
+* Delete if block
+* Change Arithmetic operations
+
+#### How To Add a New Strategy?
+Write new class inheriting the Strategy class in mutator.py and include it in get_strategies method. Override mutate method or provide replace strategy regex if it works with mutate method of Strategy class.
 
 ### Test Run Strategy
 The mutation test aims to run only tests which are concerned with the mutant. Therefore part of WPT test is related to the source code under mutation is invoked. For this it requires a test mapping in source folders.
@@ -49,21 +58,18 @@ The mutation tests can be run by running the below command from the servo direct
 Eg. `python python/servo/mutation/init.py components/script/dom`
 
 ### Running Mutation Test from CI
-
 The CI script for running mutation testing is present in /etc/ci folder. It can be called by executing the below command from the CLI:
 
-`python /etc/ci/mutation_test.py`
+`./etc/ci/mutation_test.sh`
 
 ### Execution Flow
 1. The script is called from the command line, it searches for test_mapping.json in the path entered by user.
-2. If found, it reads the json file and parses it, gets source file to tests mapping.
-3. If the source file does not have any local changes then it is mutated at a random line.
-4. The corresponding WPT tests are run for this mutant and the test results are logged.
-5. Once all WPT are run for the first source file, the mutation continues for other source files mentioned in the json file and runs their corresponding WPT tests.
-6. Once it has completed executing mutation testing for the entered path, it repeats the above procedure for sub-paths present inside the entered path.
+2. If found, it reads the json file and parses it, gets source file to tests mapping. For all source files in the mapping file, it does the following.
+3. If the source file does not have any local changes then it mutates at a random line using a random strategy. It retries with other strategies if that strategy could not produce any mutation.
+4. The code is built and the corresponding WPT tests are run for this mutant and the test results are logged.
+5. Once it has completed executing mutation testing for the entered path, it repeats the above procedure for sub-paths present inside the entered path.
 
 ### Test Summary
-
 At the end of the test run the test summary displayed which looks like this:
 ```
 Test Summary:

servo/python/servo/mutation/init.py

@@ -12,6 +12,9 @@ from os.path import isfile, isdir, join
 import json
 import sys
 import test
+import logging
+import random
+
 test_summary = {
     test.Status.KILLED: 0,
     test.Status.SURVIVED: 0,
@@ -34,22 +37,25 @@ def mutation_test_for(mutation_path):
     if isfile(test_mapping_file):
         json_data = open(test_mapping_file).read()
         test_mapping = json.loads(json_data)
-        # Run mutation test for all source files in mapping file.
-        for src_file in test_mapping.keys():
+        # Run mutation test for all source files in mapping file in a random order.
+        source_files = list(test_mapping.keys())
+        random.shuffle(source_files)
+        for src_file in source_files:
             status = test.mutation_test(join(mutation_path, src_file.encode('utf-8')), test_mapping[src_file])
             test_summary[status] += 1
         # Run mutation test in all folder in the path.
         for folder in get_folders_list(mutation_path):
             mutation_test_for(folder)
     else:
-        print("This folder {0} has no test mapping file.".format(mutation_path))
+        logging.warning("This folder {0} has no test mapping file.".format(mutation_path))
 
 
 mutation_test_for(sys.argv[1])
-print "\nTest Summary:"
-print "Mutant Killed (Success) \t{0}".format(test_summary[test.Status.KILLED])
-print "Mutant Survived (Failure) \t{0}".format(test_summary[test.Status.SURVIVED])
-print "Mutation Skipped \t\t{0}".format(test_summary[test.Status.SKIPPED])
-print "Unexpected error in mutation \t{0}".format(test_summary[test.Status.UNEXPECTED])
+logging.basicConfig(format="%(asctime)s %(levelname)s: %(message)s", level=logging.DEBUG)
+logging.info("Test Summary:")
+logging.info("Mutant Killed (Success) \t\t{0}".format(test_summary[test.Status.KILLED]))
+logging.info("Mutant Survived (Failure) \t{0}".format(test_summary[test.Status.SURVIVED]))
+logging.info("Mutation Skipped \t\t\t{0}".format(test_summary[test.Status.SKIPPED]))
+logging.info("Unexpected error in mutation \t{0}".format(test_summary[test.Status.UNEXPECTED]))
 if test_summary[test.Status.SURVIVED]:
     sys.exit(1)

servo/python/servo/mutation/mutator.py

@@ -0,0 +1,195 @@
+# Copyright 2013 The Servo Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+import fileinput
+import re
+import random
+
+
+def is_comment(line):
+    return re.search(r"\/\/.*", line)
+
+
+def init_variables(if_blocks):
+    random_index = random.randint(0, len(if_blocks) - 1)
+    start_counter = 0
+    end_counter = 0
+    lines_to_delete = []
+    line_to_mutate = if_blocks[random_index]
+    return random_index, start_counter, end_counter, lines_to_delete, line_to_mutate
+
+
+def deleteStatements(file_name, line_numbers):
+    for line in fileinput.input(file_name, inplace=True):
+        if fileinput.lineno() not in line_numbers:
+            print line.rstrip()
+
+
+class Strategy:
+    def __init__(self):
+        self._strategy_name = ""
+        self._replace_strategy = {}
+
+    def mutate(self, file_name):
+        line_numbers = []
+        for line in fileinput.input(file_name):
+            if not is_comment(line) and re.search(self._replace_strategy['regex'], line):
+                line_numbers.append(fileinput.lineno())
+        if len(line_numbers) == 0:
+            return -1
+        else:
+            mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
+            for line in fileinput.input(file_name, inplace=True):
+                if fileinput.lineno() == mutation_line_number:
+                    line = re.sub(self._replace_strategy['regex'], self._replace_strategy['replaceString'], line)
+                print line.rstrip()
+            return mutation_line_number
+
+
+class AndOr(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        logical_and = r"(?<=\s)&&(?=\s)"
+        self._replace_strategy = {
+            'regex': logical_and,
+            'replaceString': '||'
+        }
+
+
+class IfTrue(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        if_condition = r"(?<=if\s)(.*)(?=\s\{)"
+        self._replace_strategy = {
+            'regex': if_condition,
+            'replaceString': 'true'
+        }
+
+
+class IfFalse(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        if_condition = r"(?<=if\s)(.*)(?=\s\{)"
+        self._replace_strategy = {
+            'regex': if_condition,
+            'replaceString': 'false'
+        }
+
+
+class ModifyComparision(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        less_than_equals = r"(?<=\s)(\<)\=(?=\s)"
+        greater_than_equals = r"(?<=\s)(\<)\=(?=\s)"
+        self._replace_strategy = {
+            'regex': (less_than_equals + '|' + greater_than_equals),
+            'replaceString': r"\1"
+        }
+
+
+class MinusToPlus(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        arithmetic_minus = r"(?<=\s)\-(?=\s.+)"
+        minus_in_shorthand = r"(?<=\s)\-(?=\=)"
+        self._replace_strategy = {
+            'regex': (arithmetic_minus + '|' + minus_in_shorthand),
+            'replaceString': '+'
+        }
+
+
+class PlusToMinus(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        arithmetic_plus = r"(?<=[^\"]\s)\+(?=\s[^A-Z\'?\":\{]+)"
+        plus_in_shorthand = r"(?<=\s)\+(?=\=)"
+        self._replace_strategy = {
+            'regex': (arithmetic_plus + '|' + plus_in_shorthand),
+            'replaceString': '-'
+        }
+
+
+class AtomicString(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        string_literal = r"(?<=\").+(?=\")"
+        self._replace_strategy = {
+            'regex': string_literal,
+            'replaceString': ' '
+        }
+
+
+class DuplicateLine(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        self._strategy_name = "duplicate"
+        append_statement = r".+?append\(.+?\).*?;"
+        remove_statement = r".+?remove\(.*?\).*?;"
+        push_statement = r".+?push\(.+?\).*?;"
+        pop_statement = r".+?pop\(.+?\).*?;"
+        plus_equals_statement = r".+?\s\+\=\s.*"
+        minus_equals_statement = r".+?\s\-\=\s.*"
+        self._replace_strategy = {
+            'regex': (append_statement + '|' + remove_statement + '|' + push_statement +
+                      '|' + pop_statement + '|' + plus_equals_statement + '|' + minus_equals_statement),
+            'replaceString': r"\g<0>\n\g<0>",
+        }
+
+
+class DeleteIfBlock(Strategy):
+    def __init__(self):
+        Strategy.__init__(self)
+        self.if_block = r"^\s+if\s(.+)\s\{"
+        self.else_block = r"\selse(.+)\{"
+
+    def mutate(self, file_name):
+        code_lines = []
+        if_blocks = []
+        for line in fileinput.input(file_name):
+            code_lines.append(line)
+            if re.search(self.if_block, line):
+                if_blocks.append(fileinput.lineno())
+        if len(if_blocks) == 0:
+            return -1
+        random_index, start_counter, end_counter, lines_to_delete, line_to_mutate = init_variables(if_blocks)
+        while line_to_mutate <= len(code_lines):
+            current_line = code_lines[line_to_mutate - 1]
+            next_line = code_lines[line_to_mutate]
+            if re.search(self.else_block, current_line) is not None \
+                    or re.search(self.else_block, next_line) is not None:
+                if_blocks.pop(random_index)
+                if len(if_blocks) == 0:
+                    return -1
+                else:
+                    random_index, start_counter, end_counter, lines_to_delete, line_to_mutate = \
+                        init_variables(if_blocks)
+                    continue
+            lines_to_delete.append(line_to_mutate)
+            for ch in current_line:
+                if ch == "{":
+                    start_counter += 1
+                elif ch == "}":
+                    end_counter += 1
+                if start_counter and start_counter == end_counter:
+                    deleteStatements(file_name, lines_to_delete)
+                    return lines_to_delete[0]
+            line_to_mutate += 1
+
+
+def get_strategies():
+    return AndOr, IfTrue, IfFalse, ModifyComparision, PlusToMinus, MinusToPlus, \
+        AtomicString, DuplicateLine, DeleteIfBlock
+
+
+class Mutator:
+    def __init__(self, strategy):
+        self._strategy = strategy
+
+    def mutate(self, file_name):
+        return self._strategy.mutate(file_name)

servo/python/servo/mutation/test.py

@@ -7,15 +7,17 @@
 # option. This file may not be copied, modified, or distributed
 # except according to those terms.
 
-import fileinput
-import re
 import subprocess
-import sys
 import os
 import random
+import logging
+
+from mutator import Mutator, get_strategies
 from enum import Enum
 DEVNULL = open(os.devnull, 'wb')
 
+logging.basicConfig(format="%(asctime)s %(levelname)s: %(message)s", level=logging.DEBUG)
+
 
 class Status(Enum):
     KILLED = 0
@@ -24,55 +26,43 @@ class Status(Enum):
     UNEXPECTED = 3
 
 
-def mutate_random_line(file_name, strategy):
-    line_numbers = []
-    for line in fileinput.input(file_name):
-        if re.search(strategy['regex'], line):
-            line_numbers.append(fileinput.lineno())
-    if len(line_numbers) == 0:
-        return -1
-    else:
-        mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
-        for line in fileinput.input(file_name, inplace=True):
-            if fileinput.lineno() == mutation_line_number:
-                line = re.sub(strategy['regex'], strategy['replaceString'], line)
-            print line.rstrip()
-        return mutation_line_number
-
-
 def mutation_test(file_name, tests):
     status = Status.UNEXPECTED
     local_changes_present = subprocess.call('git diff --quiet {0}'.format(file_name), shell=True)
     if local_changes_present == 1:
         status = Status.SKIPPED
-        print "{0} has local changes, please commit/remove changes before running the test".format(file_name)
+        logging.warning("{0} has local changes, please commit/remove changes before running the test".format(file_name))
     else:
-        strategy = {'regex': r'\s&&\s', 'replaceString': ' || '}
-        mutated_line = mutate_random_line(file_name, strategy)
-        if mutated_line != -1:
-            print "Mutating {0} at line {1}".format(file_name, mutated_line)
-            print "compiling mutant {0}:{1}".format(file_name, mutated_line)
-            sys.stdout.flush()
-            subprocess.call('python mach build --release', shell=True, stdout=DEVNULL)
-            for test in tests:
-                test_command = "python mach test-wpt {0} --release".format(test.encode('utf-8'))
-                print "running `{0}` test for mutant {1}:{2}".format(test, file_name, mutated_line)
-                sys.stdout.flush()
-                test_status = subprocess.call(test_command, shell=True, stdout=DEVNULL)
-                if test_status != 0:
-                    print("Failed: while running `{0}`".format(test_command))
-                    print "mutated file {0} diff".format(file_name)
-                    sys.stdout.flush()
-                    subprocess.call('git --no-pager diff {0}'.format(file_name), shell=True)
-                    status = Status.SURVIVED
+        strategies = list(get_strategies())
+        while len(strategies):
+            strategy = random.choice(strategies)
+            strategies.remove(strategy)
+            mutator = Mutator(strategy())
+            mutated_line = mutator.mutate(file_name)
+            if mutated_line != -1:
+                logging.info("Mutated {0} at line {1}".format(file_name, mutated_line))
+                logging.info("compiling mutant {0}:{1}".format(file_name, mutated_line))
+                if subprocess.call('python mach build --release', shell=True, stdout=DEVNULL):
+                    logging.error("Compilation Failed: Unexpected error")
+                    status = Status.UNEXPECTED
                 else:
-                    print("Success: Mutation killed by {0}".format(test.encode('utf-8')))
-                    status = Status.KILLED
-                    break
-            print "reverting mutant {0}:{1}".format(file_name, mutated_line)
-            sys.stdout.flush()
-            subprocess.call('git checkout {0}'.format(file_name), shell=True)
-        else:
-            print "Cannot mutate {0}".format(file_name)
-        print "-" * 80 + "\n"
+                    for test in tests:
+                        test_command = "python mach test-wpt {0} --release".format(test.encode('utf-8'))
+                        logging.info("running `{0}` test for mutant {1}:{2}".format(test, file_name, mutated_line))
+                        test_status = subprocess.call(test_command, shell=True, stdout=DEVNULL)
+                        if test_status != 0:
+                            logging.error("Failed: while running `{0}`".format(test_command))
+                            logging.error("mutated file {0} diff".format(file_name))
+                            subprocess.call('git --no-pager diff {0}'.format(file_name), shell=True)
+                            status = Status.SURVIVED
+                        else:
+                            logging.info("Success: Mutation killed by {0}".format(test.encode('utf-8')))
+                            status = Status.KILLED
+                            break
+                logging.info("reverting mutant {0}:{1}\n".format(file_name, mutated_line))
+                subprocess.call('git checkout {0}'.format(file_name), shell=True)
+                break
+            elif not len(strategies):
+                # All strategies are tried
+                logging.info("\nCannot mutate {0}\n".format(file_name))
     return status