AST tests do not correctly report errors

[Mai-Lapyst]

It seems that the ast checks all are counted as passed, even if one tries to write an failing check on purpose. It seems that when xmllint doesnt find what one tries to search via --xpath, it simply prints out XPath set is empty to stderr, but exits with a non-zero status.

Example:

$ ./tester --ast ast_checks/while_condition.d | xmllint --xpath "//functionDeclaration[name = 'z']" -
XPath set is empty

xmllint version:

$ xmllint --version
xmllint: using libxml version 21105-GITv2.11.5
   compiled with: Threads Tree Output Push Reader Patterns Writer SAXv1 HTTP DTDValid HTML C14N Catalog XPath XPointer XInclude Iconv ICU ISO8859X Unicode Regexps Automata Schemas Schematron Modules Debug Zlib Lzma

[WebFreak001]

Exiting with a non-zero status seems to be correct when it's not found? Anyway xmllint is not a utility we make, so for issues with that you would need to check there.

The ci script also works properly, if you add paths that aren't found to the test txt files, the ci script properly aborts, optionally also printing the XML if you run it with environment variable VERBOSE=1.

[Mai-Lapyst]

It was a long time, atleast until like 11 months ago where this was removed: https://gitlab.gnome.org/GNOME/libxml2/-/commit/e85f9b98a5389c69167176ae6600091e719ec38f

Which aparently was a long standing decision, since --xpath isnt intended to use for scripting: https://gitlab.gnome.org/GNOME/libxml2/-/issues/180

For me the ci script dosnt works at all and also dosnt aborts; atleast when using a newer version of xmllint.

[WebFreak001]

for now you can work around this through this patch:

diff --git a/test/run_tests.sh b/test/run_tests.sh
index 1e267d2..c2d6609 100755
--- a/test/run_tests.sh
+++ b/test/run_tests.sh
@@ -88,12 +88,12 @@ if [[ ${BUILDKITE:-} != "true" ]]; then
                                expectParseFailure=1
                        elif [[ "$line" =~ ^# ]]; then
                                true # comment line
-                       elif echo "$AST" | xmllint --xpath "${line}" - 2>/dev/null > /dev/null; then
-                               ((currentPasses=currentPasses+1))
-                       else
+                       elif echo "$AST" | xmllint --xpath "${line}" - 2>&1 | grep 'XPath set is empty' >/dev/null; then
                                echo
                                echo -e "    ${RED}Check on line $lineCount of $queryFile failed.${NORMAL}"
                                ((currentFailures=currentFailures+1))
+                       else
+                               ((currentPasses=currentPasses+1))
                        fi
                        ((lineCount=lineCount+1))
                done < "$queryFile"

however we will need to find a permanent solution, although it really sucks they removed this feature