Project Soon

Testing

Testing your program is crucial, not only to see if certain parts of your program works as intended, both for correct and incorrect data, but also to see if certain found bugs have been fixed. Testing is also used to ensure that the program can run normally. The problem is that doing all this manually is bothersome and time consuming. Therefore, Continuous Integration 1 was introduced, where automatic tests are done on the program. Wube for instance on top of regular playtesting, have integrated tens of thousands of tests 2, to the point of having to invest in not one, but several servers in order to make sure that they can test their code base 3. While for PixelMap this might seem too much, it is still something that have been considered for years, but not put in action.

When PixelMap became a bit too complex and hard to handle, I felt it was time to add some sort of testing. Specifically, unit testing 4. This means that the code was compiled and tested individually, in particular smaller parts of the code. This was sadly not perfectly true, as some parts of the code were completely integrated in itself, one needed o procure some complicated tests to ensure that they work. The system that was used was Catch2 5, which is a common unit testing tool for C++. For a while I did run write some unit tests and ran them manually, but this was only for my own sanity to make sure that certain new features I added was easier to test if they worked as expected. It is hard to write tests, but the more you write, the more they will become like documentation.

But of course, unit testing is not enough, and integration testing 6 became a must, to hand it some specific input and verify its output. For this project, it is a matter of a tuple of worlds and images. The biggest question would be what type of test suit to use. Shell scripting is pretty common, but rather bothersome to work with in large scale. As I also could look into more test suits, I found that CMake already had a test suit integrated, which is CTest 7. The CTest system is documented, but poorly. At least if you are still learning CMake. Still, I was determined to make use of it.

First off, one needs to enable the testing system, which is required to be done in the root CMakeFile.txt. Second, each test added could be added anywhere, and they consist of a name and command. In addition, one could add dependencies to ensure tests are run in a certain order, but it does not skip tests if the previous one fails even if it is dependent on it, so to tackle such an issue, fixtures can be used 8. They are used to have a set of tests set up before running other tests requiring the setup to succeed, and finally cleaning up the group of tests. This is a great way to ensure that the programs being tested actually exist, specifically the test executable with unit tests, and pixelmapcli. As these setups are tests themselves, and therefore commands, they can essentially build the binaries before testing them, making it foolproof to run the tests if one forgets to compile.

Third, while unit tests are self contained, integration tests requires external data. As the current repository is just above 2MB in size, we do not want to fill it with several tens of MB with test data. It is also good to make test data optional, so as CTest has a specific feature for external data 9, I decided to make use of that and move all test files to another repository. To do this, two things are required to be done to secure that test data, and having it in a repository added an optional third thing: Test data in the test repository is stored in a specific folder structure of the hash algorithm used 10 and the hash of the content is the file name; Files representing the actual data contains the hash of the data as well as a suffix extension of the hash algorithm; One can refer to a commit, like a git submodule 11, to guarantee that the test files referring to is the correct version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

# Hashes source files and puts them in out
# Uses hashes as names and copies src to gen
# Gen files keep the actual data, and out
# files is put in other repository

MKDIR := mkdir -p
HASH_BIN := sha256sum
HASH := sha256

SRC_DIR := src
GEN_DIR := gen
OUT_DIR := out
GEN_HASH := $(GEN_DIR)/$(shell echo $(HASH) | tr '[:lower:]' '[:upper:]')

SRC := $(wildcard $(SRC_DIR)/*)
OUT := $(patsubst $(SRC_DIR)/%,$(OUT_DIR)/%.$(HASH),$(SRC))

all: $(OUT_DIR) $(GEN_HASH) $(OUT)

$(OUT_DIR)/%.$(HASH): $(SRC_DIR)/%
  $(HASH_BIN) $< | awk '{print $$1}' > $@
  cp $< $(GEN_HASH)/$$(cat $@)

$(OUT_DIR):
  $(MKDIR) $@
$(GEN_HASH):
  $(MKDIR) $@

clean:
  -rm -rf $(OUT_DIR)/* $(GEN_DIR)/*

Sadly, ExternalData module only downloaded and verified the files. It did not help with unpacking archives or running the tests, even if it added them. To solve this issue, I had to create a bash script which unpacks, executes and compares. It would be better if I had built a binary which could be used instead, but that would add more things to test, so this should do fine for automatic testing at least.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/bash
# Unpack MC world, run it through cli and compare output

set -e

WORLD="$1"
IMAGE="$2"

# Temporary folder with trap
tmpdir=$(mktemp -d)
trap 'rm -rf -- "$tmpdir"' EXIT

# Needs to unpack them each time (cmake issues)
tar -xf "$WORLD" -C "$tmpdir" --strip-components=1

echo $PIXELMAPCLI
# Run default and without output
$PIXELMAPCLI "$tmpdir" "$tmpdir/testimage.png" -q

# Compare output image with prepared image
$COMPARE -metric AE "$IMAGE" "$tmpdir/testimage.png" "$tmpdir/out.png"

As previously mentioned with CMake sometimes have poor documentation, this was the case, because they made one assumption for the user which was not mentioned at all. When an ExternalData test was added, each test also included a target, which means it needs to be built. This was actually a good thing, because then the test data would not be downloaded when CMake populates. Instead, I could add a custom target and add each test target as a dependency, and then create a test with setup fixture together with pixelmapcli and build the custom target. Due to this, whenever ctest is run, it will download all test data and link them to each test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

# Testing area

set(COMMIT "b1be0b39dcc356d74d217c5a18674a1e30d9fcec")

cmake_host_system_information(RESULT nproc QUERY NUMBER_OF_LOGICAL_CORES)

# Ensure availability of tests
add_test(NAME build-tests COMMAND "${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target tests --parallel ${nproc})

# Unit tests
add_test(NAME unit-tests COMMAND $<TARGET_FILE:tests>)

# Dependencies
set_tests_properties(build-tests PROPERTIES FIXTURES_SETUP unit)
set_tests_properties(unit-tests PROPERTIES FIXTURES_REQUIRED unit)

add_custom_target(integration)

# Ensure availability of cli
add_test(NAME build-pixelmapcli COMMAND "${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target pixelmapcli --parallel ${nproc})
add_test(NAME integration-download COMMAND "${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target integration)

set_tests_properties(build-pixelmapcli PROPERTIES DEPENDS unit)
set_tests_properties(integration-download PROPERTIES DEPENDS build-pixelmapcli)

set_tests_properties(build-pixelmapcli PROPERTIES FIXTURES_SETUP integration)
set_tests_properties(integration-download PROPERTIES FIXTURES_SETUP integration)

# Find all programs used
find_program(BASH_EXECUTABLE NAMES bash REQUIRED)
set(PIXELMAPCLI $<TARGET_FILE:pixelmapcli>)
find_program(COMPARE NAMES compare REQUIRED)
find_file(INTEGRATION_SH NAMES integration.sh PATHS tests)

# Integration tests
include(ExternalData)
set(ExternalData_URL_TEMPLATES "https://git.aposoc.net/McTwist/PixelMap_data/raw/commit/${COMMIT}/gen/%(algo)/%(hash)")

file(GLOB test_worlds "test_data/*.tar.sha256")
foreach(path ${test_worlds})
  cmake_path(GET path FILENAME file)
  cmake_path(GET file STEM name)
  ExternalData_Add_Test("e${name}" NAME ${name}
    COMMAND
      ${CMAKE_COMMAND} -E env PIXELMAPCLI=${PIXELMAPCLI} COMPARE=${COMPARE}
      "${BASH_EXECUTABLE}" "${INTEGRATION_SH}" DATA{test_data/${name}.tar} DATA{test_data/${name}.png})
  ExternalData_Add_Target("e${name}")
  add_dependencies(integration "e${name}")
  set_tests_properties(${name} PROPERTIES FIXTURES_REQUIRED integration)
  unset(name)
  unset(file)
endforeach()

Test data is stored in the data repository. It will be updated with more tests, and probably compressed further to reduce the test time. It is great that it works, and while it took some time to figure out how, it will tell me earlier if something fails, and I do not have to look manually at images from different versions to make it do so. It also makes sure that unless I specifically want it to change the output, it will tell me prematurely if something has gone wrong.

Test project /home/mctwist/dev/pixelmap/build
      Start  1: build-tests
 1/21 Test  #1: build-tests ......................   Passed   16.20 sec
      Start  2: unit-tests
 2/21 Test  #2: unit-tests .......................   Passed    0.01 sec
      Start  3: build-pixelmapcli
 3/21 Test  #3: build-pixelmapcli ................   Passed    0.28 sec
      Start  4: integration-download
 4/21 Test  #4: integration-download .............   Passed    0.46 sec
      Start  5: be_1_21_40
 5/21 Test  #5: be_1_21_40 .......................   Passed    0.32 sec
      Start  6: je_1_0
 6/21 Test  #6: je_1_0 ...........................   Passed    0.55 sec
      Start  7: je_1_1
 7/21 Test  #7: je_1_1 ...........................   Passed    0.36 sec
      Start  8: je_1_12_2
 8/21 Test  #8: je_1_12_2 ........................   Passed    0.27 sec
      Start  9: je_1_13_2
 9/21 Test  #9: je_1_13_2 ........................   Passed    0.35 sec
      Start 10: je_1_14_4
10/21 Test #10: je_1_14_4 ........................   Passed    0.32 sec
      Start 11: je_1_15_2
11/21 Test #11: je_1_15_2 ........................   Passed    0.34 sec
      Start 12: je_1_16_5
12/21 Test #12: je_1_16_5 ........................   Passed    0.54 sec
      Start 13: je_1_17_1
13/21 Test #13: je_1_17_1 ........................   Passed    0.47 sec
      Start 14: je_1_18_2
14/21 Test #14: je_1_18_2 ........................   Passed    0.92 sec
      Start 15: je_1_19_0
15/21 Test #15: je_1_19_0 ........................   Passed    0.78 sec
      Start 16: je_1_20_2
16/21 Test #16: je_1_20_2 ........................   Passed    1.25 sec
      Start 17: je_1_21_3
17/21 Test #17: je_1_21_3 ........................   Passed    0.46 sec
      Start 18: je_1_7_10
18/21 Test #18: je_1_7_10 ........................   Passed    0.43 sec
      Start 19: je_1_8_9
19/21 Test #19: je_1_8_9 .........................   Passed    0.34 sec
      Start 20: je_alpha_1_2_6
20/21 Test #20: je_alpha_1_2_6 ...................   Passed    0.28 sec
      Start 21: je_beta_1_8_1
21/21 Test #21: je_beta_1_8_1 ....................   Passed    0.39 sec

100% tests passed, 0 tests failed out of 21

Total Test time (real) =  25.33 sec

An optional feature I am not currently sure if I want to use, is the integration between CMake and Catch2 12. It could make testing easier, but I am not sure how useful it might become.

This turned out to be a pretty exhausting addition, with loads of twists and turns, plenty of experiments to get to where it is now. But is sure was worth it.

  1. https://en.wikipedia.org/wiki/Continuous_integration ↩︎

  2. https://www.factorio.com/blog/post/fff-60 ↩︎

  3. https://www.factorio.com/blog/post/fff-315 ↩︎

  4. https://en.wikipedia.org/wiki/Unit_testing ↩︎

  5. https://github.com/catchorg/Catch2 ↩︎

  6. https://en.wikipedia.org/wiki/Integration_testing ↩︎

  7. https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html ↩︎

  8. https://cmake.org/cmake/help/latest/prop_test/FIXTURES_REQUIRED.html ↩︎

  9. https://cmake.org/cmake/help/latest/module/ExternalData.html ↩︎

  10. Required to be upper case, not specified in the manual. ↩︎

  11. https://git-scm.com/book/en/v2/Git-Tools-Submodules ↩︎

  12. https://github.com/catchorg/Catch2/blob/devel/docs/cmake-integration.md ↩︎