new perldelta

[perl5.git] / dist / Devel-PPPort / HACKERS

=head1 NAME

HACKERS - Devel::PPPort internals for hackers

=head1 SYNOPSIS

So you probably want to hack C<Devel::PPPort>?

Well, here's some information to get you started with what's
lying around in this distribution.

=head1 DESCRIPTION

=head2 How to backport something

First, make sure that what you want to backport is documented.  If it's worth
backporting, it surely is worth documenting.  Submit a documentation patch to
L<https://github.com/Perl/perl5/issues> if necessary.  Also, C<Devel::PPPort>
cannot automatically generate proper information about the item without at
least knowing its API prototype.  It can get this from F<embed.fnc> if the item
is a function, but if it is a macro, there needs to be at least a S<C<=for
apidoc>> line for C<Devel::PPPort> to be able to figure things out on its own.

Next, figure out where to place your implementation.  Look at all the files in
F<parts/inc/> for one that fits what you're planning.  If there isn't one,
just start a new one and remember to include it from within F<PPPort_pm.PL>.
If you do create a new file, it's usually the best approach to just copy an
existing file and use it as a template.

Each file holds all relevant data for implementing a certain part
of the API:

=over 2

=item *

A list of the provided API in the C<=provides> section.

=item *

The optional C<=dontwarn> section is used to suppress warnings about particular
API elements.  Don't use this unless you get such a warning, and be sure to
think about using other other alternatives before resorting to adding something
in this section.

=item *

The implementation to add to F<ppport.h> in the C<=implementation>
section.  See L</Implementation Section Details>.

=item *

The code required to add to PPPort.xs for testing the implementation.
This code goes into the C<=xshead>, C<=xsinit>, C<=xsmisc>, C<=xsboot>
and C<=xsubs> section. Have a look at the template at the bottom
of F<RealPPPort_xs.PL> to see where the code ends up.

=item *

The tests in the C<=tests> section. Remember not to use any fancy
modules or syntax elements, as the test code needs to be able to run
with Perl 5.003.  (This is because Devel::PPPort itself will run all test files
in the process of generating the information about when a feature came into
existence.)  This means, for example

=over

=item C<my> isn't supported in C<for>-loops

    for my $x (1, 2, 3) { }    # won't work with 5.003

Instead declare C<$x> just before the statement

=item The postfix C<for> statement modifier isn't supported

    foo for 1..2

won't compile.  Instead enclose C<foo> in a loop.

=item You can't use plain C<qr//>

Instead, wrap it in a string eval C<eval "qr//">, and be sure it's skipped at
execution time on perls earlier than 5.005

=back

As of version 3.56 of Devel::PPPort, the old Test style tests have been
replaced with the more modern Test::More style, with some limitations.  This
means, for example, that C<is> is finally available, as well as
C<done_testing>.  You can pass the number of tests to C<skip>, instead of
having to have your own C<for> loop.

There is no C<like> nor C<unlike> (as those require C<qr> which didn't exist in
the earliest perls that Devel::PPPort runs on).

C<skip> doesn't do a S<C<last SKIP>>.  (Perhaps it could, but that would mean
converting all the skips in the existing tests.)

The existing tests have been changed only as much as necessary so as to get
things to work.  But feel free to use the full functionality for any new tests
you write.

Here's a list of the supported functions:

 cmp_ok
 curr_test
 diag
 display
 done_testing
 eq_array
 eq_hash
 fail
 is
 isnt
 next_test
 note
 ok
 pass
 plan
 skip
 skip_all
 within

These are copied from F<t/test.pl> in the perl distribution.  Not all of them
have been tested back as far as Devel::PPPort supports.  Bug reports welcome.

It's fine to backport an element only as far as convenient and necessary.  But
remember that your test file will end up being called on all perl versions
available, likely including ones earlier than your backport.  That may mean
that elements in the C<=xs> sections will have to be C<#idef>'d out so that the
object will not get missing symbols when loaded.

It also means you have to check for and skip tests that aren't relevant to this
version.  The recommended way to do this is like:

 if (ivers($]) < ivers(5.6)) {           # No quotes needed
    skip "reason", $count;
 }
 elsif (if (ivers($]) > ivers("5.5.4") {
     # Quotes needed for double-dotted versions prior to 5.6.0
    skip "other reason", $count;
 }
 else {
    do_the_test
 }

C<ivers()> is a function automatically made available to all F<.t> files.  It
converts any reasonble expression of a version number into an integer, which
can reliably be compared using numeric comparison operators, with the output of
a second C<ivers()> call on a different version number, like in the result above.

It's worth emphasizing that, due to bugs in early perl parsing, if you use a
version number containing two dots on a version befor 5.6.0, it has to be
quoted.

=back

In all sections, lines that begin with C<##> are completely ignored.

=head2 Implementation Section Details

You can implement API elements via C functions or macros, or simple variables.
It is preferable to use a macro if feasible.  Otherwise, the user must
explicitly request that it get loaded, by defining a C<NEED_I<function>> (or
I<variable>) as described in F<ppport.h>.  If a function, I<foo> is required,
place its body in this C<=implementation> section, like so:

 #if { NEED foo }

 char *
 foo(pTHX_ const U8 *arg1, const U32 arg2, ...)
 {
    ...
 }

 #endif

Similarly for a variable.

It's obviously best to use a macro if at all feasible.  Sometimes what once
was implemented with a macro now requires a function; perhaps an edge case was
overlooked.  Doing so will cause the new F<ppport.h> to not be drop-in
compatible with the older version, and can hence cause breakage.  This
incompatiblity (while easily solved) really needs to be stressed in
documentation.

=over

=item __UNDEFINED__

If you add the line C<__UNDEFINED__> to the C<=provides> section, you can use
lines like this in the C<=implementation> section:

  __UNDEFINED__ macro    some definition

to both define C<macro> and indicate that it is provided by F<ppport.h>.  This
replaces these C<=implementation> section lines:

  #ifndef macro
  #  define macro    some definition
  #endif

besides automagically making it be considered to be provided.  C<macro> can
have optional arguments and the definition can even span multiple lines, like
in

  __UNDEFINED__ SvMAGIC_set(sv, val) \
                STMT_START { assert(SvTYPE(sv) >= SVt_PVMG); \
                (((XPVMG*) SvANY(sv))->xmg_magic = (val)); } \
                STMT_END

This usually makes the code more compact and readable.

But you should only use this on things that you plan to publicly provide.  If
something, such as a mnemonic for a constant needs to be defined but isn't
really needed for the public at large to know about, you should use

 __UNDEF_NOT_PROVIDED__ macro   some definition

instead.  To avoid name space conflicts, follow what's in L</Helper macros>,
below.

=item __REDEFINE__

If you add the line C<__REDEFINE__> to the C<=provides> section, you can use
lines like this in the C<=implementation> section:

  __REDEFINE__ macro    some definition

to both redefine C<macro> and indicate that it is provided by F<ppport.h>.  This
replaces these C<=implementation> section lines:

  #undef macro
  #ifndef macro
  #  define macro    some definition
  #endif


=item Helper macros

If you need to define a helper macro which is not part of C<Devel::PPPort> API
and its usage is only for the definition of other C<Devel::PPPort> macros, then
use the C<D_PPP_> prefix for this macro name (e.g. C<D_PPP_SVPV_NOLEN_LP_ARG>).
This suppresses any warnings when a macro is defined which is not part of the
Perl public API.

=item Version numbers

Version checking used to be tricky to get correct (besides being buggy in some
perl versions).
C<ivers()> is used in the C<=tests> section to overcome this. and constructs
like the following in the C language sections.

  #if { VERSION < 5.9.3 }

You SHOULD be using this construct or the alternatives listed below for ALL
version checks, and not come up with something on your own.

In this form, the version number can be either of the new form C<5.x.y> or the
older form C<5.00x_yy>. Both are translated into the correct preprocessor
statements. It is also possible to combine this with other statements:

  #if { VERSION >= 5.004 } && !defined(sv_vcatpvf)
    /* a */
  #elif { VERSION < 5.004_63 } && { VERSION != 5.004_05 }
    /* b */
  #endif

This not only works in the C<=implementation> section, but also in
the C<=xsubs>, C<=xsinit>, C<=xsmisc>, C<=xshead> and C<=xsboot> sections.

Alternatively, you can use the forms now available in regular Perl:

  #if PERL_VERSION_EQ(5,9,3)
  #if PERL_VERSION_NE(5,9,3)
  #if PERL_VERSION_LT(5,9,3)
  #if PERL_VERSION_GT(5,9,3)
  #if PERL_VERSION_LE(5,9,3)
  #if PERL_VERSION_GE(5,9,3)

These forms have the advantage over the '{ VERSION ... }' form in that you may
use the special value '*' for the final number to mean ALL possible values for
it.  Thus,

 #if PERL_VERSION_EQ(5,31,'*')

means all perls in the 5.31 series.  And

 #if PERL_VERSION_NE(5,24,'*')

means all perls EXCEPT 5.24 ones.  And

 #if PERL_VERSION_LE(5,9,'*')

is effectively

 #if PERL_VERSION_LT(5,10,0)

=item Hints

If you add a comment like so:

 /* Hint: PL_expect, PL_copline, PL_rsfp
    paragraphs of stuff you want to have shown when ppport.h outputs
    something about any one of PL_expect, PL_copline, or PL_rsfp
 */

Earlier versions of F<ppport.h> required an asterisk at the beginning of every
continuation line, or else the content would be silently dropped.

=item Warnings

A more serious caution can be displayed by instead saying

 /* Warning: PL_expect, PL_copline, PL_rsfp
    paragraphs of stuff you want to have shown when ppport.h outputs
    something about any one of PL_expect, PL_copline, or PL_rsfp
 */

Earlier versions of F<ppport.h> required an asterisk at the beginning of every
continuation line, or else the content would be silently dropped.

=item Replace

When F<ppport.h> is run on a file(s), you can cause it to automatically flag
occurrences of the constructs you specify, encouraging the author to replace
them with different (presumably better) ones.  These also are used in any
suggested edits and generated patches.

There are three ways to do this

=over 4

=item in-line comment

You can add a trailing comment like so:

 #define bar foo    /* Replace */
 __UNDEFINED__ bar foo  /* Replace */

These say that C<foo> should be replaced by C<bar>.  NOT the other way around.

=item separate comment

For situations not amenable to the above, you can say

 /* Replace foo with bar */

=item define a replacement region

It you have several replacements, you can group them together like so:

 /* Replace: 1 */
 #define foo bar
 #define bat baz
 /* Replace: 0 */

These replace C<bar> with C<foo>; C<baz> with C<bat>.  NOT the other way
around.

=back

=item Dependencies

F<ppport.h> automatically gathers information as to what functions are
dependent on what other things from inspecting the source, but if this is
insufficient for you, you can add lines like the following:

 /* foo, bar depends on baz, bat */

Each of C<foo>, C<bar> depends on each of C<baz>, C<bat>.

=back

=head2 Testing

After you have furnished your implementation, you need to test it.

=head2 Special Makefile targets

You can use

    make regen

to regenerate all of the autogenerated files. To get rid of all
generated files (except for F<parts/todo/*> and F<parts/base/*>),
use

    make purge_all

That's it.

To automatically test C<Devel::PPPort> with lots of different Perl
versions, you can use the F<soak> script. Just pass it a list of
all Perl binaries you want to test.

=head2 Regenerating F<ppport.h> and F<PPPort.pm>

C<Devel::PPPort> keeps two directories of generated files, in F<parts/base> and
F<parts/todo>.  The files in each are named after Perl version numbers.  When a
function or macro came into existence is indicated by placing its name in the
corresponding file in F<parts/base>.  The files in F<parts/todo> are the same,
except they indicate the earliest release that F<ppport.h> supports the
element.  The delta is effectively what F<ppport.h> buys you.

The generation process described in this section creates these files.  It does
so by examining as many perl versions as are available to it.  It tries to make
sure each element actually compiles, and it runs the test scripts you have
furnished on every version.

Ideally, this should be done before every release that includes new backporting
and/or when blead has added new public API.  At a minimum, it should be done as
the next major Perl release comes out.

The process isn't platform independent. It has currently been tested only under
Linux, and it definitely requires at least C<gcc> and the C<nm> utility.
The process used to be problematic, with random failures.  But it has now been
fixed to be reliable.

Before starting the regeneration, you need to have gathered certain data.
(Options listed below apply to the tools that eventually will use the data, and
which are described further below).

=over 4

=item *

You will first need a whole bunch of different Perls, the more, the better, but
only one per version tag (which one is random) will actually get used.
dromedary has a sufficient set.  They should all have the same Configure
options with respect to what functions and macros are enabled.  For example,
they should all be threaded, or all non-threaded.  A mixture will screw up the
results.  Similarly, they should all or none have quad math (at least as far
back as that became available).  You can use F<devel/buildperl.pl> to build
them.

Previous maintainers of this module kept those perls in
F</tmp/perl/install/default>, so most of the tools use this as a default, but
you'll likely simply use the C<--install=> option to specify where.  This
should be a path where a S<C<make install>> has been done, so has immediate
subdirectories of C</bin> and C</lib>.  C</bin> should contain the binaries.
It will use all files in this directory whose names begin with C<perl5>.

Actually, not all the binaries need be in this directory.  You can specify
additional places to look since C<--install=> takes a comma separated list of
directories.

=item *

You also need a freshly built bleadperl.  The C<--blead=I<path>> option should
be used to specify it.  (Some of the tools have a default of C<bleadperl-debug>
if this option is omitted.)  Again, it needs the same Configure options as the
earlier versions had.  Using C<-DNO_MATHOMS> will change the results, and
probably should be avoided.  True, these functions are allegedly on their way
out, so it could be argued that they shouldn't be encouraged in any way; but
some of these have been in limbo for many years, so should be documented.

=item *

And you will need updated API information. Copy the latest F<embed.fnc> file
from bleadperl to the F<parts> directory and run F<devel/mkapidoc.pl> to
collect the remaining information in F<parts/apidoc.fnc>.  This needs to be
done after the perl has been compiled, as there are generated files that feed
it.

=item *

The final step before regenerating everything is to run
F<devel/mkppport_fnc.pl> to update the F</parts/ppport.fnc> file.

=back

Having done this, run F<devel/regenerate> which wraps the following steps
(which you could instead do by hand, but it's easy to forget things):

=over

=item *

It first does some sanity checking

=item *

Then it asks you if it's ok to remove all existing todo files in the
F<parts/base> and F<parts/todo> directories.  If you answer no, the process
aborts.

This is crtical to getting accurate results.

=item *

It builds the new baseline by running

    perl devel/mktodo --base

in the root directory of the distribution.

If there are warnings in blead, it will ask you to examine them, and to ok if
it's all right to go ahead.  If there are issues with blead, everything
following could be wrong.

=item *

It builds the new todo files by running

    perl devel/mktodo

in the root directory of the distribution.

=item *

Finally, it adds the remaining information by running

    perl Makefile.PL && make
    perl devel/scanprov --mode=write

=back

=head2 How to build gobs of versions of Perl

C<Devel::PPPort> supports Perl versions between 5.003 and bleadperl.
To guarantee this support, its good to have as many versions as possible to
test on.  dromedary currently has many such versions.

There is a tool to build all the different
versions and configurations. You can find it in F<devel/buildperl.pl>.
It can currently build the following Perl releases:

    5.003
    5.004 - 5.004_05
    5.005 - 5.005_04
    5.6.x
    5.7.x
    5.8.x
    5.9.x
    5.1x.x
    5.2x.x
    5.3x.x

=head2 Implementation

Knowing which parts of the API are not backwards compatible and
probably need C<Devel::PPPort> support is another problem that's
not easy to deal with manually. If you run

    perl Makefile.PL --with-apicheck

a C file is generated by F<parts/apicheck.pl> that is compiled
and linked with C<Devel::PPPort>. This C file has the purpose of
using each of the public API functions/macros once.

The required information is derived from F<parts/embed.fnc> (just
a copy of bleadperl's F<embed.fnc>), F<parts/apidoc.fnc> (which
is generated by F<devel/mkapidoc.pl> and simply collects the rest
of the apidoc entries spread over the Perl source code) and
F<parts/ppport.fnc> (which lists the API provided purely by
Devel::PPPort, along with other elements that are tested only using
F<ppport.h>).

The generated C file (usually, F<apicheck.c>) won't compile as-is
with older perls. And even if it compiles, there's still a good chance of the
dynamic linker failing at C<make test> time. But that's on purpose!

We can use these failures to find changes in the API automatically.
The Perl script F<devel/mktodo> calls another script F<devel/mktodo.pl>
repeatedly to run C<Devel::PPPort> on version after version of perl, in
decreasing version order, so we start with blead and work backwards.  The
latter script generates an F<apicheck.c>.  It starts with the code that
successfully worked in the previously tested Perl version, which should be the
version one higher than the current one.  Call the current one I<n>, and the
previous one I<n+1>.  The items that fail to compile in I<n>, but did compile
in I<n+1> must have become available in I<n+1>.  We run the Linux command C<nm>
to find those undefined symbols in I<n>.  We change F<apicheck.c> to ignore
(through C<#ifdef>'s) those and recompile, repeating until F<apicheck.c>
successfully compiles, the dynamic linker is happy, and C<make test> runs on
this version.  Then we repeat the process for I<n-1>, and so on.  (Actually,
this process may generate false positives, so by default each failing API call
is checked again.  If possible, this is done by generating an F<apicheck.c> for
just the one failing API.)  Note that the make test is run using F<ppport.h>
during both passes.

Running F<devel/mktodo> currently takes a couple hours on dromedary.

If you run it with the C<--nocheck> option, it won't recheck the API calls
that failed in the compilation stage and it'll take significantly less time.
No one currently associated with maintaining this module understands under what
circumstances it is safe to run with C<--nocheck>.

By repeating the process over and over, we build up information on when every
element first became supported.  This information is stored in files in the
F<parts/base> directory, one file per version.  The file for version I<n+1> is
generated by running version I<n> of perl.

We actually want a second piece of information, which is how much F<ppport.h>
buys you.  What happens when regenerating is actually two entire runs through
all the perls.  The first is accomplished by calling F<devel/mktodo> with the
C<--base> option.  It automically will call F<devel/mktodo.pl> with each
version of perl, NOT using anything in F<ppport.h>.  When done the results
indicate  when each API element became available in stock perl, without using
F<ppport.h>.

And then the whole process is repeated, but this time F<ppport.h> is included.
The files are placed in F<parts/todo>.  Thus, at the end, we know when each
element became available in modified perl, using F<ppport.h>.

However, only the public API that is implemented as functions (and must appear
in F<embed.fnc>) plus macros whose calling sequence is documented can be
checked this way.  The final step in the process is calling F<devel/scanprov>.
It looks through the header files for when all the symbols provided by
C<Devel::PPPort> first became defined.  It doesn't test the symbols or try to
compile them, as it doesn't generally know the API, but it can tell that
something exists in release I<n+1> but not I<n> (by scanning the include files
in the F<CORE> directory of various Perl versions).  (It does know if a macro
has zero arguments or non-zero arguments, so it does get extra information from
the zero argument ones.)

=head2 Files

Residing in F<parts/inc/> is the "heart" of C<Devel::PPPort>. Each
of the files implements a part of the supported API, along with
hints, dependency information, XS code and tests.
The files are in a POD-like format that is parsed using the
functions in F<parts/ppptools.pl>.

The scripts F<PPPort_pm.PL>, F<RealPPPort_xs.PL> and F<mktests.PL> all
use the information in F<parts/inc/> to generate the main module
F<PPPort.pm>, the XS code in F<RealPPPort.xs> and various test files
in F<t/>.

You can get extra information from F<PPPort_pm.PL> by setting the environment
variable C<DPPP_CHECK_LEVEL> to 1 or 2.

All of these files could be generated on the fly while building
C<Devel::PPPort>, but not having the tests in F<t/> will confuse
TEST/harness in the core. Not having F<PPPort.pm> will be bad for
viewing the docs on C<search.cpan.org>. So unfortunately, it's
unavoidable to put some redundancy into the package.

=head2 Submitting Patches

If you've added some functionality to C<Devel::PPPort>, please
consider submitting a patch with your work to P5P by sending a pull request to

L<https://github.com/Dual-Life/Devel-PPPort/pulls>.

When submitting patches, please only add the relevant changes
and don't include the differences of the generated files. You
can use the C<purge_all> target to delete all autogenerated
files.

=head2 Integrating into the Perl core

When integrating this module into the Perl core, be sure to
remove the following files from the distribution. They are
either not needed or generated on the fly when building this
module in the core:

  MANIFEST
  META.yml
  PPPort.pm

=head1 BUGS

No known bugs.

=head1 COPYRIGHT

Version 3.x, Copyright (C) 2004-2020, Marcus Holland-Moritz
and Perl 5 porters

Version 2.x, Copyright (C) 2001, Paul Marquess.

Version 1.x, Copyright (C) 1999, Kenneth Albanowski.

This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=head1 SEE ALSO

See F<ppport.h> and F<devel/regenerate>.

=cut