Re: C Unit testing and mocking
On Dec 26, 2024, at 06:19, Jakub Zelenka <[email protected]> wrote:On Tue, Dec 24, 2024 at
8:22 PM Ben Ramsey <[email protected]> wrote:On Dec 16, 2024, at 07:21, Jakub Zelenka
<[email protected]> wrote:Hi,
I have been looking into how to test some cases where integration tests are very difficult or even
impossible to create for. Those are often found in networking related and system specific code code
(network.c, streams, FPM and more). I was recently fixing one such bug and decided to give a try
which resulted in this PR: https://github.com/php/php-src/pull/16987 .
There was a suggestion of RFC but that might be a bit too much as it's just an internal change
/ addition. But certainly some overview on internals should be done so writing this instead.
I decided to use cmocka in that PR because I had some experience with that. It's quite small
and still very powerful and allow vast mocking options. It's a bit manual but it gives a bigger
control over the mock. It relies on --wrap linking option that replaces original functions with
wraps. This is however available only on Linux (or maybe some other Unix variants) but doesn't
work on MacOS or Windows. The developers that want to use it on those platforms would need to use
some Linux Virtualisation option (e.g. Docker). It also requires static library which is supported
by embed SAPI that can be compiled statically. That limits number of extensions to use but the main
use cases don't really have deps so it should be fine.
I did also some research into the other mocking libraries in C. There is a Unity with CMock, FFF and
some C++ libs like GUnit, Criterion and Trompeloeil that I looked into. I quickly discarded GUnit
and Trompeloeil as they relay on C++ virtual methods and require wrapping C code to C++ which is
very inconvenient. FFF seems too simple and maybe quite inflexible for our needs as well. Criterion
also optionally uses wrap so I didn't see much advantages compare to cmocka. So it left Unity
with CMock that allows generating custom mocks using a Ruby script. That seemed initially quite nice
but after spending around two hours with trying to make it works for PHP codebase, I just gave up.
It gets quite messy for complex scenarios and I just didn't figure out how to nicely mock libc
functions without any modification to php-src.
In terms of CI. It has got its own build which is very simple and it tests just specific parts so we
could just limit it to run only for changed files which might be quite convenient.
So the proposed PR is probably the only reasonable unit testing that I can come up with. I think it
should be completely optional initially for people to use - more like an experiment. If it becomes
used, then good of course. And if it becomes pain, we can just get rid of it. Has anyone got any
objections to get this merged? If not I plan to merge it early in January.
Cheers
Jakub
FWIW, as someone still very new to C, I found Criterion quite easy to use, and I was able to quickly
grasp its concepts and start using it right away. I can’t say the same for other C testing
libraries I tried.
I just checked Criterion a bit more and it actually does not have built-in mocking. It could be used
with --wrap but it doesn't have any expectations like cmocka. There's Mimick [1] from the
same author which is indendent but its docs still say that it's experimental so probably not a
good idea to use an experimental tool as its API can change. So all in all it's not probably an
option for us.
[1] https://github.com/Snaipe/Mimick
I’ve not done any mocking in C yet, so I can’t offer much there, but I see Mimick hasn’t had a
tagged release since 2017, even though development still appears active, but I agree it might not be
mature enough to standardize on.
It’s interesting to note David Carlier (a.k.a. devnexen) has made some contributions to Mimick.
:-)
Cheers,Ben
Thread (13 messages)