Issue 1102: Fixup test suite, fix broken tests

[baylesj]

A recent PR broken the JsonChecker tests by adding support for trailing
commas. This didn't end up breaking the build, because those tests
aren't run, except locally and only using CMake.

This patch fixes the tests by adding exclusions for trailing comma
tests, as well as updates Meson to run these tests as part of ninja test.

See issue #1102.

[coveralls]

Coverage remained the same at 93.858% when pulling 387045d on fixup-tests into 8b20b7a on master.

[cdunn2001]

Hmmm. Using this branch locally, I have this:

[7/8] Running all tests.
1/3 unittest_jsoncpp_test                   OK       0.34 s
2/3 unittest_jsontestrunner                 FAIL     2.24 s (exit status 1)
3/3 jsonchecker_jsontestrunner              FAIL     2.60 s (exit status 1)

... Oh, if I delete the build-dir and start over, then it works. That's odd. Meson/Ninja usually do a good job of auto-updating for changes. Very suspicious, but it does work.

1/3 unittest_jsoncpp_test                   OK       0.15 s
2/3 unittest_jsontestrunner                 OK       3.17 s
3/3 jsonchecker_jsontestrunner              OK       4.04 s

Thanks for fixing this!

... Oh, hold on. It fails if you run a second time:

$ ninja -C build-static/ test
$ ninja -C build-static/ test
1/3 unittest_jsoncpp_test                   OK       0.03 s
2/3 unittest_jsontestrunner                 FAIL     1.17 s (exit status 1)
3/3 jsonchecker_jsontestrunner              OK       3.74 s

So we still have something to fix ...

[cdunn2001]

I've verified that ninja test is repeatable onmaster, so something fishy is happening on this branch.

[baylesj]

I'm digging back into this... looks like a fun problem. Basically both the unittest_jsontestrunner and jsonchecker_jsontestrunner are both intermittently flaky. Maybe a timing issue? I'll update once I have it sorted.

[baylesj]

I figured it out! It's a race condition--for some reason the runjsontests.py script is trying to read the test files before they are done. I'm working on improving the hygeine of the test harness, and hopefully will have a 100% reliable configuration soon.

[baylesj]

I think this is ready for another look!

[dota17]

I think this is ready for another look!

Is there any progress？

[cdunn2001]

This worked:

cd build-static; meson test --no-rebuild --print-errorlogs

But this failed (from top-dir):

ninja -C build-static/ test

ninja: Entering directory `build-static/'
[0/1] Running all tests.
1/3 unittest_jsoncpp_test                   OK       0.06 s
2/3 unittest_jsontestrunner                 FAIL     2.05 s (exit status 1)
3/3 jsonchecker_jsontestrunner              OK       2.67 s
...
FAILED: meson-test
/usr/local/Cellar/meson/0.52.0/libexec/bin/meson test --no-rebuild --print-errorlogs

I'll update meson, but I don't think that's the issue...

[cdunn2001]

After upgrading python/meson/ninja on my Mac, I don't see a failure anymore. Weird. But the tests seem slow... Ok. I see the problem. Meson runs 3 tests: C++ unit-tests, a python round-trip checker on test/data/*.json, and the same python round-trip checker on those same *.json in addition to any expected-failure tests plus (some of) the tests under jsonchecker. The problem is that the non-fail tests under test/data/*.json are run twice. That's easy to prevent.

This whole thing should be done differently, but I try to solve just one specific problem at a time. For now I will revert your last 2 commits and skip the repetition of those tests. Tomorrow....

[cdunn2001]

I pushed some commits, and it seems fine now, though we clearly still need some reorganization of the test-suite(s).

[cdunn2001]

There must be a way to prevent the race. We might merge just to make progress, but I'd like to figure out a way to prevent this problem altogether.

I'm about to try these changes locally....