After switching to phpunit 9.5 a 2x slowness was witnessed (4.5 min => 9min). Previous reports/issues about new AST parser library caused some false assumptions. Testing with static cache did not improve speed much nor did cache warming improve things to the original levels.

After digging through the coverage logic and adding all sorts of time measurements to the coverage collecting steps, the main slownesses seems to come from ProcessedCodeCoverageData->markCodeAsExecutedByTestCase() and ProcessedCodeCoverageData->initializeUnseenData()
The functions themselves are not big and don't do anything obviously slow.
Adding more debug info showed that the list of classes being processed was 500 files with 20 000 lines and this happened for each test execution, even if test actually called only one short function that has 1 line in it.

After more checks it seems that PCOV extension is returning list of all seen files during the whole run, even if 0 code was actually executed since last call to clear(), causing lots of overhead to the functions that then try to analyze the results.

Adding following code to PcovDriver->stop() after \pcov\clear(); to ignore file entries where ALL lines are marked as not executed restores the speed to original levels:

      $notExecutedFilter = static function ($val): bool
      {
        return $val === self::LINE_NOT_EXECUTED;
      };
      //PCOV driver keeps returning files that have no lines executed in them, clear them away here to avoid overhead later
      foreach ($collect as $fileName => $lines)
      {
        if (count(array_filter($lines, $notExecutedFilter)) === count($lines))
        {
          unset($collect[$fileName]);
        }
      }