t/harness: Add option for faster test suite execution

This commit adds an environment variable, PERL_TEST_HARNESS_ASAP, which
if set to non-zero increases the parallelism in the execution of the
test suite, speeding it up on systems with multiple cores.

Normally, there are two main test sections, one for core and the second
for non-core tests, and the testing of the non-core one doesn't begin
until the core tests are complete.  Within each section, there are a
number of test categories, like 're' for regular expressions, and
'JSON::PP' for the pure perl implementation of JSON.

Within each category, there are various single .t test files.  Some
categories can have those be tested in parallel; some require them to be
done in a particular order, say because an earlier .t does setup for
subsequent ones.  We already have this capability.

Completion of all the tests in a category is not needed before those of
another category can be started.  This is how it already works.

However, the core section categories are ordered so that they begin in a
logical order for someone trying to get perl to work.  First to start
are the basic sanity tests, then by roughly decreasing order of
widespread use in perl programs in the wild, with the final two
categories, porting and perf, being mainly of use to perl5 porters.
These two categories aren't started until all the tests in the earlier
categories are started.  We have some long running tests in those two
categories, and generally they delay the start of the entire second
section.

If those long running tests could be started sooner, shorter tests in
the first section could be run in parallel with them, increasing the
average CPU utilization, and the second section could begin (and hence
end) earlier, shortening the total elapsed execution time of the entire
suite.

The second section has some very long running tests.  JSON-PP is one of
them.  If it could run in parallel with tests from the first section,
that would also speed up the completion of the suite.

The environment variable added by this commit does both things.  The
basic sanity test categories in the first section continue to be started
before anything else.  But then all other tests are run in decreasing
order of elapsed time they take to run, removing the boundaries between
some categories, and between the two sections.

The gain from this increases as the number of jobs run in parallel does;
slower high core platforms have the highest increase.  On the old
dromedary with 24 cores, the gain is 20%, almost 2 minutes.  On my more
modern box with 12 cores, it is 8%.

diff --git a/t/harness b/t/harness
index 58a212f..c3cf906 100644 (file)
--- a/t/harness
+++ b/t/harness
@@ -117,11 +117,20 @@ if (@ARGV) {
     unless (@tests) {
        my @seq = <base/*.t>;
 
-       my @next = qw(comp run cmd io re opbasic op uni mro lib porting perf);
-       push @next, 'japh' if $torture;
-       push @next, 'win32' if $^O eq 'MSWin32';
-       push @next, 'benchmark' if $ENV{PERL_BENCHMARK};
-       push @next, 'bigmem' if $ENV{PERL_TEST_MEMORY};
+        my @last;
+       my @next = qw(comp run cmd);
+
+        # The remaining core tests are either intermixed with the non-core for
+        # more parallelism (if PERL_TEST_HARNESS_ASAP is set non-zero) or done
+        # after the above basic sanity tests, before any non-core ones.
+        my $which = $ENV{PERL_TEST_HARNESS_ASAP} ? \@last : \@next;
+
+        push @$which, qw(io re opbasic op uni mro lib porting perf);
+       push @$which, 'japh' if $torture;
+       push @$which, 'win32' if $^O eq 'MSWin32';
+       push @$which, 'benchmark' if $ENV{PERL_BENCHMARK};
+       push @$which, 'bigmem' if $ENV{PERL_TEST_MEMORY};
+
        # Hopefully TAP::Parser::Scheduler will support this syntax soon.
        # my $next = { par => '{' . join (',', @next) . '}/*.t' };
        my $next = { par => [
@@ -129,6 +138,9 @@ if (@ARGV) {
                            ] };
        @tests = _extract_tests ($next);
 
+       my $last = { par => '{' . join (',', @last) . '}/*.t' };
+       @last = _extract_tests ($last);
+
        # This is a bit of a game, because we only want to sort these tests in
        # speed order. base/*.t wants to run first, and ext,lib etc last and in
        # MANIFEST order
@@ -147,7 +159,6 @@ if (@ARGV) {
        @tests = (@seq, @tests);
        push @seq, $next;
 
-       my @last;
        push @last,
            _tests_from_manifest($Config{extensions}, $Config{known_extensions});
        my %times;
@@ -171,7 +182,7 @@ if (@ARGV) {
 
             # Keep a list of the distinct directory names, and another list of
             # those which contain a file whose name begins with a 0
-            if ( m! \A \.\. /
+            if ( m! \A (?: \.\. / )?
                                 ( .*? )         # $1 is the directory path name
                             /
                                 ( [^/]* \.t )   # $2 is the .t name
@@ -210,7 +221,7 @@ if (@ARGV) {
        for (@last) {
             # Treat every file in each non-serial directory as its own
             # "directory", so that it can be executed in parallel
-            m! \A ( \.\. / (?: $non_serials )
+            m! \A ( (?: \.\. / )? (?: $non_serials )
                          / [^/]+ \.t \z | .* [/] ) !x
                 or die "'$_'";
            push @{$dir{$1}}, $_;