HACKERS: Update

[perl5.git] / dist / Devel-PPPort / HACKERS

=head1 NAME

HACKERS - Devel::PPPort internals for hackers

=head1 SYNOPSIS

So you probably want to hack C<Devel::PPPort>?

Well, here's some information to get you started with what's
lying around in this distribution.

=head1 DESCRIPTION

=head2 How to build 366 versions of Perl

C<Devel::PPPort> supports Perl versions between 5.003 and bleadperl.
To guarantee this support, I need some of these versions on my
machine. I currently have 366 different Perl version/configuration
combinations installed on my laptop.

As many of the old Perl distributions need patching to compile
cleanly on newer systems (and because building 366 Perls by hand
just isn't fun), I wrote a tool to build all the different
versions and configurations. You can find it in F<devel/buildperl.pl>.
It can currently build the following Perl releases:

    5.003
    5.004 - 5.004_05
    5.005 - 5.005_04
    5.6.x
    5.7.x
    5.8.x
    5.9.x
    5.1x.x
    5.2x.x
    5.3x.x

=head2 Fully automatic API checks

Knowing which parts of the API are not backwards compatible and
probably need C<Devel::PPPort> support is another problem that's
not easy to deal with manually. If you run

    perl Makefile.PL --with-apicheck

a C file is generated by F<parts/apicheck.pl> that is compiled
and linked with C<Devel::PPPort>. This C file has the purpose of
using each of the public API functions/macros once.

The required information is derived from F<parts/embed.fnc> (just
a copy of bleadperl's F<embed.fnc>), F<parts/apidoc.fnc> (which
is generated by F<devel/mkapidoc.sh> and simply collects the rest
of the apidoc entries spread over the Perl source code) and
F<parts/ppport.fnc> (which lists all API provided purely by
Devel::PPPort).
The generated C file F<apicheck.c> is currently about 500k in size
and takes quite a while to compile.

Usually, F<apicheck.c> won't compile with older perls. And even if
it compiles, there's still a good chance of the dynamic linker
failing at C<make test> time. But that's on purpose!

We can use these failures to find changes in the API automatically.
The two Perl scripts F<devel/mktodo> and F<devel/mktodo.pl>
repeatedly run C<Devel::PPPort> with the apicheck code through
all different versions of perl. Scanning the output of the compiler
and the dynamic linker for errors, the files in F<parts/todo/> are
generated. These files list all parts of the public API that don't
work with less than a certain version of Perl.

This information is in turn used by F<parts/apicheck.pl> to mask
API calls in the generated C file for these versions, so the
process can be stopped by the time F<apicheck.c> compiles cleanly
and the dynamic linker is happy. (Actually, this process may generate
false positives, so by default each API call is checked once more
afterwards.)

Running F<devel/mktodo> takes about an hour, depending of course
on the machine you're running it on. If you run it with
the C<--nocheck> option, it won't recheck the API calls that failed
in the compilation stage and it'll take significantly less time.
Running with C<--nocheck> should usually be safe.

When running F<devel/mktodo> with the C<--base> option, it will
generate the I<baseline> todo files by disabling all functionality
provided by C<Devel::PPPort>. These are required for implementing
the C<--compat-version> option of the F<ppport.h> script. The
baseline todo files hold the information about which version of
Perl lacks a certain part of the API.

However, only the documented public API can be checked this way.
And since C<Devel::PPPort> provides more macros, these would not be
affected by C<--compat-version>. It's the job of F<devel/scanprov>
to figure out the baseline information for all remaining provided
macros by scanning the include files in the F<CORE> directory of
various Perl versions.

The whole process isn't platform independent. It has currently been
tested only under Linux, and it definitely requires at least C<gcc> and
the C<nm> utility.

It's not very often that one has to regenerate the baseline and todo
files. If you have to, you can either run F<devel/regenerate> or just
execute the following steps by hand:

=over 4

=item *

You need a whole bunch of different Perls. The more, the better.
You can use F<devel/buildperl.pl> to build them. I keep my perls
in F</tmp/perl>, so most of the tools take this as a default.

=item *

You also need a freshly built bleadperl that is in the path under
exactly this name. (The name of the executable is currently hardcoded
in F<devel/mktodo> and F<devel/scanprov>.)

=item *

Remove all existing todo files in the F<parts/base> and
F<parts/todo> directories.

=item *

Update the API information. Copy the latest F<embed.fnc> file from
bleadperl to the F<parts> directory and run F<devel/mkapidoc.sh> to
collect the remaining information in F<parts/apidoc.fnc>.

=item *

Build the new baseline by running

    perl devel/mktodo --base

in the root directory of the distribution. When it's finished,
move all files from the F<parts/todo> directory to F<parts/base>.

=item *

Build the new todo files by running

    perl devel/mktodo

in the root directory of the distribution.

=item *

Finally, add the remaining baseline information by running

    perl Makefile.PL && make
    perl devel/scanprov --mode=write

=back

=head2 Implementation

Residing in F<parts/inc/> is the "heart" of C<Devel::PPPort>. Each
of the files implements a part of the supported API, along with
hints, dependency information, XS code and tests.
The files are in a POD-like format that is parsed using the
functions in F<parts/ppptools.pl>.

The scripts F<PPPort_pm.PL>, F<RealPPPort_xs.PL> and F<mktests.PL> all
use the information in F<parts/inc/> to generate the main module
F<PPPort.pm>, the XS code in F<RealPPPort.xs> and various test files
in F<t/>.

You can get extra information from F<PPPort_pm.PL> by setting the environment
variable C<DPPP_CHECK_LEVEL> to 1 or 2.

All of these files could be generated on the fly while building
C<Devel::PPPort>, but not having the tests in F<t/> will confuse
TEST/harness in the core. Not having F<PPPort.pm> will be bad for
viewing the docs on C<search.cpan.org>. So unfortunately, it's
unavoidable to put some redundancy into the package.

=head2 Adding stuff to Devel::PPPort

First, check if the code you plan to add fits into one of the
existing files in F<parts/inc/>. If not, just start a new one and
remember to include it from within F<PPPort_pm.PL>.

Each file holds all relevant data for implementing a certain part
of the API:

=over 2

=item *

A list of the provided API in the C<=provides> section.

=item *

The implementation to add to F<ppport.h> in the C<=implementation>
section.

Here you can add additional information for a given item that will be displayed
when F<ppport.h> is run.  If your item is named C<foo>, you add a
comment like so:

 /* Hint: foo
    paragraphs of stuff about foo you want to have
    shown when ppport.h outputs something about foo
 */

This will cause S<C<perl ppport.h>> to display this hint when it outputs
something about C<foo>.

A more serious caution about C<foo> can be displayed by instead saying

 /* Warning: foo
    paragraphs of stuff about foo you want to have
    shown when ppport.h outputs something about foo
 */

=item *

The code required to add to PPPort.xs for testing the implementation.
This code goes into the C<=xshead>, C<=xsinit>, C<=xsmisc>, C<=xsboot>
and C<=xsubs> section. Have a look at the template at the bottom
of F<RealPPPort_xs.PL> to see where the code ends up.

=item *

The tests in the C<=tests> section. Remember not to use any fancy
modules or syntax elements, as the test code needs to be able to run
with Perl 5.003.  (This is because Devel::PPPort itself will run all test files
in the process of generating the information about when a feature came into
existence.)  This means, for example

=over

=item C<my> isn't supported in C<for>-loops

    for my $x (1, 2, 3) { }    # won't work with 5.003

Instead declare C<$x> just before the statement

=item The postfix for statement modifier isn't supported

    foo for 1..2

won't compile.  Instead enclose C<foo> in a loop.

=item You can't use plain C<qr//>

Instead, wrap it in a string eval C<eval "qr//">, and be sure it's skipped at
execution time on perls earlier than 5.005

=back

You can use C<ok()> to report success or failure:

 ok($got, $expected, 'name');
 ok($got == 42);     # Doesn't give good runtime diagnostics

 ok($got, eval "qr/foo/", 'name') # But don't execute this statement
                                  # on perls earlier than 5.005

=back

It's usually the best approach to just copy an existing file and
use it as a template.

=head2 Implementation Hints

In the C<=implementation> section, you can use

  __UNDEFINED__ macro    some definition

instead of

  #ifndef macro
  #  define macro    some definition
  #endif

The macro can have optional arguments and the definition can even
span multiple lines, like in

  __UNDEFINED__ SvMAGIC_set(sv, val) \
                STMT_START { assert(SvTYPE(sv) >= SVt_PVMG); \
                (((XPVMG*) SvANY(sv))->xmg_magic = (val)); } STMT_END

This usually makes the code more compact and readable. And you only have to add
the line C<__UNDEFINED__> to the C<=provides> section to get all macros
implemented this way to be imported into this section, so they all get
documented as being provided.

If you need to define a helper macro which is not part of C<Devel::PPPort> API
and its usage is only for definition of other C<Devel::PPPort> macros then use
the C<D_PPP_> prefix for this macro name (e.g. C<D_PPP_SVPV_NOLEN_LP_ARG>).
Some C<Devel::PPPort> scripts would report a warning when there is defined a
macro which is not part of Perl public API, except those macros which start
with the C<D_PPP_> prefix.

Version checking can be tricky to get correct.
You can use

  #if { VERSION < 5.9.3 }

instead of

  #if ((PERL_VERSION < 9) || (PERL_VERSION == 9 && PERL_SUBVERSION < 3))

The version number can be either of the new form C<5.x.x> or of the older
form C<5.00x_yy>. Both are translated into the correct preprocessor
statements. It is also possible to combine this with other statements:

  #if { VERSION >= 5.004 } && !defined(sv_vcatpvf)
    /* a */
  #elif { VERSION < 5.004_63 } && { VERSION != 5.004_05 }
    /* b */
  #endif

This not only works in the C<=implementation> section, but also in
the C<=xsubs>, C<=xsinit>, C<=xsmisc>, C<=xshead> and C<=xsboot> sections.

=head2 Testing

To automatically test C<Devel::PPPort> with lots of different Perl
versions, you can use the F<soak> script. Just pass it a list of
all Perl binaries you want to test.

=head2 Special Makefile targets

You can use

    make regen

to regenerate all of the autogenerated files. To get rid of all
generated files (except for F<parts/todo/*> and F<parts/base/*>),
use

    make purge_all

That's it.

=head2 Submitting Patches

If you've added some functionality to C<Devel::PPPort>, please
consider submitting a patch with your work to P5P by sending a mail
L<perlbug@perl.org|mailto:perlbug@perl.org>.

When submitting patches, please only add the relevant changes
and don't include the differences of the generated files. You
can use the C<purge_all> target to delete all autogenerated
files.

=head2 Integrating into the Perl core

When integrating this module into the Perl core, be sure to
remove the following files from the distribution. They are
either not needed or generated on the fly when building this
module in the core:

  MANIFEST
  META.yml
  PPPort.pm

=head1 COPYRIGHT

Version 3.x, Copyright (C) 2004-2013, Marcus Holland-Moritz.

Version 2.x, Copyright (C) 2001, Paul Marquess.

Version 1.x, Copyright (C) 1999, Kenneth Albanowski.

This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=head1 SEE ALSO

See F<ppport.h> and F<devel/regenerate>.

=cut