This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Fix typos in HACKERS; add clarification
[perl5.git] / dist / Devel-PPPort / HACKERS
CommitLineData
adfe19db
MHM
1=head1 NAME
2
3HACKERS - Devel::PPPort internals for hackers
4
5=head1 SYNOPSIS
6
7So you probably want to hack C<Devel::PPPort>?
8
9Well, here's some information to get you started with what's
10lying around in this distribution.
11
12=head1 DESCRIPTION
13
8913c038
KW
14=head2 How to backport something
15
16First, make sure that what you want to backport is documented. If it's worth
17backporting, it surely is worth documenting. Send a documentation patch to
18L<perlbug@perl.org|mailto:perlbug@perl.org> if necessary. Also,
19C<Devel::PPPort> cannot automatically generate proper information about the
20item without at least knowing its API prototype. It can get this from
21F<embed.fnc> if the item is a function, but if it is a macro, there needs to be
22at least a S<C<=for apidoc>> line for C<Devel::PPPort> to be able to figure
23things out on its own.
24
25Next, figure out where to place your implementation. Look at all the files in
26F<parts/inc/> for one that fits what you're planning. If there isn't one,
27just start a new one and remember to include it from within F<PPPort_pm.PL>.
28If you do create a new file, it's usually the best approach to just copy an
29existing file and use it as a template.
adfe19db
MHM
30
31Each file holds all relevant data for implementing a certain part
32of the API:
33
34=over 2
35
36=item *
37
38A list of the provided API in the C<=provides> section.
39
40=item *
41
8913c038
KW
42The optional C<=dontwarn> section is used to suppress warnings about particular
43API elements. Don't use this unless you get such a warning, and be sure to
44think about using other other alternatives before resorting to adding something
45in this section.
9712ab2b 46
8913c038 47=item *
9712ab2b 48
8913c038
KW
49The implementation to add to F<ppport.h> in the C<=implementation>
50section. See L</Implementation Section Details>.
9712ab2b 51
adfe19db
MHM
52=item *
53
54The code required to add to PPPort.xs for testing the implementation.
55This code goes into the C<=xshead>, C<=xsinit>, C<=xsmisc>, C<=xsboot>
0d0f8426 56and C<=xsubs> section. Have a look at the template at the bottom
3bfe1c68 57of F<RealPPPort_xs.PL> to see where the code ends up.
adfe19db
MHM
58
59=item *
60
61The tests in the C<=tests> section. Remember not to use any fancy
f6db508c
KW
62modules or syntax elements, as the test code needs to be able to run
63with Perl 5.003. (This is because Devel::PPPort itself will run all test files
64in the process of generating the information about when a feature came into
65existence.) This means, for example
66
67=over
68
69=item C<my> isn't supported in C<for>-loops
adfe19db 70
0d0f8426 71 for my $x (1, 2, 3) { } # won't work with 5.003
adfe19db 72
f6db508c
KW
73Instead declare C<$x> just before the statement
74
8913c038 75=item The postfix C<for> statement modifier isn't supported
f6db508c
KW
76
77 foo for 1..2
78
79won't compile. Instead enclose C<foo> in a loop.
80
81=item You can't use plain C<qr//>
82
83Instead, wrap it in a string eval C<eval "qr//">, and be sure it's skipped at
84execution time on perls earlier than 5.005
85
86=back
87
0d0f8426
MHM
88You can use C<ok()> to report success or failure:
89
f6db508c
KW
90 ok($got, $expected, 'name');
91 ok($got == 42); # Doesn't give good runtime diagnostics
0d0f8426 92
f6db508c
KW
93 ok($got, eval "qr/foo/", 'name') # But don't execute this statement
94 # on perls earlier than 5.005
adfe19db
MHM
95
96=back
97
8913c038 98In all sections, lines that begin with C<##> are completely ignored.
adfe19db 99
8913c038
KW
100=head2 Implementation Section Details
101
d4b3485f
KW
102You can implement API elements via C functions or macros, or simple variables.
103For a function, just place its body in this C<=implementation> section. But it
104is preferable to use a macro if feasible. Otherwise, the user must explicitly
105request that it get loaded, by defining a C<NEED_I<function>> (or I<variable>)
106as described in F<ppport.h>.
107
8913c038 108=over
0d0f8426 109
8913c038
KW
110=item __UNDEFINED__
111
112If you add the line C<__UNDEFINED__> to the C<=provides> section, you can use
113lines like this in the C<=implementation> section:
0d0f8426
MHM
114
115 __UNDEFINED__ macro some definition
116
8913c038
KW
117to both define C<macro> and indicate that it is provided by F<ppport.h>. This
118replaces these C<=implementation> section lines:
0d0f8426
MHM
119
120 #ifndef macro
121 # define macro some definition
122 #endif
123
8913c038
KW
124besides automagically making it be considered to be provided. C<macro> can
125have optional arguments and the definition can even span multiple lines, like
126in
0d0f8426
MHM
127
128 __UNDEFINED__ SvMAGIC_set(sv, val) \
129 STMT_START { assert(SvTYPE(sv) >= SVt_PVMG); \
1470b692
N
130 (((XPVMG*) SvANY(sv))->xmg_magic = (val)); } \
131 STMT_END
0d0f8426 132
8913c038
KW
133This usually makes the code more compact and readable.
134
135But you should only use this on things that you plan to publicly provide. If
136something, such as a mnemonic for a constant needs to be defined but isn't
d4b3485f 137really needed for the public at large to know about, you should use
8913c038
KW
138
139 __UNDEF_NOT_PROVIDED__ macro some definition
140
141instead. To avoid name space conflicts, follow what's in L</Helper macros>,
142below.
143
144=item Helper macros
0d0f8426 145
3451ddaf 146If you need to define a helper macro which is not part of C<Devel::PPPort> API
8913c038
KW
147and its usage is only for the definition of other C<Devel::PPPort> macros, then
148use the C<D_PPP_> prefix for this macro name (e.g. C<D_PPP_SVPV_NOLEN_LP_ARG>).
149This suppresses any warnings when a macro is defined which is not part of the
150Perl public API.
151
152=item Version numbers
3451ddaf 153
f6db508c 154Version checking can be tricky to get correct.
0d0f8426
MHM
155You can use
156
157 #if { VERSION < 5.9.3 }
158
159instead of
160
1470b692
N
161 #if ((PERL_VERSION < 9) \
162 || (PERL_VERSION == 9 && PERL_SUBVERSION < 3))
0d0f8426 163
8913c038 164The version number can be either of the new form C<5.x.x> or the older
0d0f8426
MHM
165form C<5.00x_yy>. Both are translated into the correct preprocessor
166statements. It is also possible to combine this with other statements:
167
168 #if { VERSION >= 5.004 } && !defined(sv_vcatpvf)
b2049988 169 /* a */
0d0f8426
MHM
170 #elif { VERSION < 5.004_63 } && { VERSION != 5.004_05 }
171 /* b */
172 #endif
173
174This not only works in the C<=implementation> section, but also in
175the C<=xsubs>, C<=xsinit>, C<=xsmisc>, C<=xshead> and C<=xsboot> sections.
176
8913c038
KW
177=item Hints
178
179If you add a comment like so:
180
181 /* Hint: PL_expect, PL_copline, PL_rsfp
182 paragraphs of stuff about foo you want to have
183 shown when ppport.h outputs something about PL_expect or
184 PL_copline or PL_rsfp
185 */
186
187Earlier versions of F<ppport.h> required an asterisk at the beginning of every
188continuation line, or else the content would be silently dropped.
189
190=item Warnings
191
192A more serious caution about C<foo> can be displayed by instead saying
193
194 /* Warning: PL_expect, PL_copline, PL_rsfp
195 paragraphs of stuff about foo you want to have
196 shown when ppport.h outputs something about PL_expect or
197 PL_copline or PL_rsfp
198 */
199
200Earlier versions of F<ppport.h> required an asterisk at the beginning of every
201continuation line, or else the content would be silently dropped.
202
203=item Replace
204
205When F<ppport.h> is run on a file(s), you can cause it to automatically flag
206occurrences of the constructs you specify, encouraging the author to replace
207them with different (presumably better) ones. These also are used in any
208suggested edits and generated patches.
209
210There are three ways to do this
211
212=over 4
213
214=item in-line comment
215
216You can add a trailing comment like so:
217
218 #define bar foo /* Replace */
219 __UNDEFINED__ bar foo /* Replace */
220
221These say that C<foo> should be replaced by C<bar>. NOT the other way around.
222
223=item separate comment
224
225For situations not amenable to the above, you can say
226
227 /* Replace foo with bar */
228
229=item define a replacement region
230
231It you have several replacements, you can group them together like so:
232
233 /* Replace: 1 */
234 #define foo bar
235 #define bat baz
236 /* Replace: 0 */
237
238These replace C<foo> with C<bar>; C<bat> with C<baz>.
239
240=back
241
242=item Dependencies
243
244F<ppport.h> automatically gathers information as to what functions are
245dependent on what other things from inspecting the source, but if this is
246insufficient for you, you can add lines like the following:
247
248 /* foo, bar depends on baz, bat */
249
250Each of C<foo>, C<bar> depends on each of C<baz>, C<bat>.
251
252=back
253
adfe19db
MHM
254=head2 Testing
255
8913c038 256After you have furnished your implementation, you need to test it.
adfe19db
MHM
257
258=head2 Special Makefile targets
259
4a582685 260You can use
adfe19db
MHM
261
262 make regen
263
4a582685
NC
264to regenerate all of the autogenerated files. To get rid of all
265generated files (except for F<parts/todo/*> and F<parts/base/*>),
266use
adfe19db
MHM
267
268 make purge_all
269
270That's it.
271
8913c038
KW
272To automatically test C<Devel::PPPort> with lots of different Perl
273versions, you can use the F<soak> script. Just pass it a list of
274all Perl binaries you want to test.
275
276=head2 Regenerating F<ppport.h> and F<PPPort.pm>
277
278C<Devel::PPPort> keeps two directories of generated files, in F<parts/base> and
279F<parts/todo>. The files in each are named after Perl version numbers. When a
280function or macro came into existence is indicated by placing its name in the
281corresponding file in F<parts/base>. The files in F<parts/todo> are the same,
282except they indicate the earliest release that F<ppport.h> supports the
283element. The delta is effectively what F<ppport.h> buys you.
284
285The generation process described in this section creates these files. It does
286so by examining as many perl versions as are available to it. It tries to make
287sure each element actually compiles, and it runs the test scripts you have
288furnished on every version.
289
290Ideally, this should be done before every release that includes new backporting
291and/or when blead has added new public API. At a minimum, it should be done as
292the next major Perl release comes out.
293
294The process isn't platform independent. It has currently been tested only under
295Linux, and it definitely requires at least C<gcc> and the C<nm> utility.
296The process used to be problematic, with random failures. But it has now been
297fixed to be reliable.
298
299Before starting the regeneration, you need to have gathered certain data.
300(Options listed below apply to the tools that eventually will use the data, and
301which are described further below).
302
303=over 4
304
305=item *
306
307You will first need a whole bunch of different Perls, the more, the better, but
308only one per version tag (which one is random) will actually get used.
309dromedary has a sufficient set. They should all have the same Configure
310options with respect to what functions and macros are enabled. For example,
311they should all be threaded, or all non-threaded. A mixture will screw up the
312results. Similarly, they should all or none have quad math (at least as far
313back as that became available). You can use F<devel/buildperl.pl> to build
314them.
315
316Previous maintainers of this module kept those perls in
317F</tmp/perl/install/default>, so most of the tools use this as a default, but
318you'll likely simply use the C<--install=> option to specify where. This
319should be a path where a S<C<make install>> has been done, so has immediate
320subdirectories of C</bin> and C</lib>. C</bin> should contain the binaries.
321It will use all files in this directory whose names begin with C<perl5>.
322
323Actually, not all the binaries need be in this directory. You can specify
324additional places to look since C<--install=> takes a comma separated list of
325directories.
326
327=item *
328
329You also need a freshly built bleadperl. The C<--blead=I<path>> option should
330be used to specify it. (Some of the tools have a default of C<bleadperl-debug>
331if this option is omitted.) Again, it needs the same Configure options as the
332earlier versions had.
333
334=item *
335
336And you will need updated API information. Copy the latest F<embed.fnc> file
337from bleadperl to the F<parts> directory and run F<devel/mkapidoc.pl> to
338collect the remaining information in F<parts/apidoc.fnc>.
339
340=item *
341
d4b3485f
KW
342The final step before regenerating everything is to run
343F</devel/mkppport_fnc.pl> to update the F</parts/ppport.fnc> file.
8913c038
KW
344
345=back
346
347Having done this, run F<devel/regenerate> which wraps the following steps
348(which you could instead do by hand, but it's easy to forget things):
349
350=over
351
352=item *
353
354It first does some sanity checking
355
356=item *
357
358Then it asks you if it's ok to remove all existing todo files in the
359F<parts/base> and F<parts/todo> directories. If you answer no, the process
360aborts.
361
362This is crtical to getting accurate results.
363
364=item *
365
366It builds the new baseline by running
367
368 perl devel/mktodo --base
369
370in the root directory of the distribution.
371
372If there are warnings in blead, it will ask you to examine them, and to ok if
373it's all right to go ahead. If there are issues with blead, everything
374following could be wrong.
375
376=item *
377
378It builds the new todo files by running
379
380 perl devel/mktodo
381
382in the root directory of the distribution.
383
384=item *
385
386Finally, it adds the remaining information by running
387
388 perl Makefile.PL && make
389 perl devel/scanprov --mode=write
390
391=back
392
393=head2 How to build gobs of versions of Perl
394
395C<Devel::PPPort> supports Perl versions between 5.003 and bleadperl.
396To guarantee this support, its good to have as many versions as possible to
397test on. dromedary currently has many such versions.
398
399There is a tool to build all the different
400versions and configurations. You can find it in F<devel/buildperl.pl>.
401It can currently build the following Perl releases:
402
403 5.003
404 5.004 - 5.004_05
405 5.005 - 5.005_04
406 5.6.x
407 5.7.x
408 5.8.x
409 5.9.x
410 5.1x.x
411 5.2x.x
412 5.3x.x
413
414=head2 Implementation
415
416Knowing which parts of the API are not backwards compatible and
417probably need C<Devel::PPPort> support is another problem that's
418not easy to deal with manually. If you run
419
420 perl Makefile.PL --with-apicheck
421
422a C file is generated by F<parts/apicheck.pl> that is compiled
423and linked with C<Devel::PPPort>. This C file has the purpose of
424using each of the public API functions/macros once.
425
426The required information is derived from F<parts/embed.fnc> (just
427a copy of bleadperl's F<embed.fnc>), F<parts/apidoc.fnc> (which
428is generated by F<devel/mkapidoc.sh> and simply collects the rest
429of the apidoc entries spread over the Perl source code) and
430F<parts/ppport.fnc> (which lists the API provided purely by
431Devel::PPPort, along with other elements that are tested only using
432F<ppport.h>).
433
434The generated C file (usually, F<apicheck.c>) won't compile as-is
435with older perls. And even if it compiles, there's still a good chance of the
436dynamic linker failing at C<make test> time. But that's on purpose!
437
438We can use these failures to find changes in the API automatically.
439The Perl script F<devel/mktodo> calls another script F<devel/mktodo.pl>
440repeatedly to run C<Devel::PPPort> on version after version of perl, in
441decreasing version order, so we start with blead and work backwards. The
442latter script generates an F<apicheck.c>. It starts with the code that
443successfully worked in the previously tested Perl version, which should be the
444version one higher than the current one. Call the current one I<n>, and the
445previous one I<n+1>. The items that fail to compile in I<n>, but did compile
446in I<n+1> must have become available in I<n+1>. We run the Linux command C<nm>
447to find those undefined symbols in I<n>. We change F<apicheck.c> to ignore
448(through C<#ifdef>'s) those and recompile, repeating until F<apicheck.c>
449successfully compiles, the dynamic linker is happy, and C<make test> runs on
450this version. Then we repeat the process for I<n-1>, and so on. (Actually,
451this process may generate false positives, so by default each failing API call
452is checked again. If possible, this is done by generating an F<apicheck.c> for
d4b3485f
KW
453just the one failing API.) Note that the make test is run using F<ppport.h>
454during both passes.
8913c038
KW
455
456Running F<devel/mktodo> currently takes a couple hours on dromedary.
457
458If you run it with the C<--nocheck> option, it won't recheck the API calls
459that failed in the compilation stage and it'll take significantly less time.
460No one currently associated with maintaining this module understands under what
461circumstances it is safe to run with C<--nocheck>.
462
463By repeating the process over and over, we build up information on when every
464element first became supported. This information is stored in files in the
465F<parts/base> directory, one file per version. The file for version I<n+1> is
466generated by running version I<n> of perl.
467
468We actually want a second piece of information, which is how much F<ppport.h>
469buys you. What happens when regenerating is actually two entire runs through
470all the perls. The first is accomplished by calling F<devel/mktodo> with the
471C<--base> option. It automically will call F<devel/mktodo.pl> with each
472version of perl, NOT using anything in F<ppport.h>. When done the results
473indicate when each API element became available in stock perl, without using
474F<ppport.h>.
475
476And then the whole process is repeated, but this time F<ppport.h> is included.
477The files are placed in F<parts/todo>. Thus, at the end, we know when each
478element became available in modified perl, using F<ppport.h>.
479
480However, only the public API that is implemented as functions (and must appear
481in F<embed.fnc>) plus macros whose calling sequence is documented can be
482checked this way. The final step in the process is calling F<devel/scanprov>.
483It looks through the header files for when all the symbols provided by
484C<Devel::PPPort> first became defined. It doesn't test the symbols or try to
485compile them, as it doesn't generally know the API, but it can tell that
486something exists in release I<n+1> but not I<n> (by scanning the include files
487in the F<CORE> directory of various Perl versions). (It does know if a macro
488has zero arguments or non-zero arguments, so it does get extra information from
489the zero argument ones.)
490
491=head2 Files
492
493Residing in F<parts/inc/> is the "heart" of C<Devel::PPPort>. Each
494of the files implements a part of the supported API, along with
495hints, dependency information, XS code and tests.
496The files are in a POD-like format that is parsed using the
497functions in F<parts/ppptools.pl>.
498
499The scripts F<PPPort_pm.PL>, F<RealPPPort_xs.PL> and F<mktests.PL> all
500use the information in F<parts/inc/> to generate the main module
501F<PPPort.pm>, the XS code in F<RealPPPort.xs> and various test files
502in F<t/>.
503
504You can get extra information from F<PPPort_pm.PL> by setting the environment
505variable C<DPPP_CHECK_LEVEL> to 1 or 2.
506
507All of these files could be generated on the fly while building
508C<Devel::PPPort>, but not having the tests in F<t/> will confuse
509TEST/harness in the core. Not having F<PPPort.pm> will be bad for
510viewing the docs on C<search.cpan.org>. So unfortunately, it's
511unavoidable to put some redundancy into the package.
512
0d0f8426
MHM
513=head2 Submitting Patches
514
515If you've added some functionality to C<Devel::PPPort>, please
8913c038
KW
516consider submitting a patch with your work to P5P by sending a pull request to
517
518L<https://github.com/Dual-Life/Devel-PPPort/pulls>.
0d0f8426
MHM
519
520When submitting patches, please only add the relevant changes
521and don't include the differences of the generated files. You
522can use the C<purge_all> target to delete all autogenerated
523files.
524
56093a11
MHM
525=head2 Integrating into the Perl core
526
527When integrating this module into the Perl core, be sure to
528remove the following files from the distribution. They are
529either not needed or generated on the fly when building this
530module in the core:
531
532 MANIFEST
533 META.yml
534 PPPort.pm
535
8913c038
KW
536=head1 BUGS
537
1470b692
N
538No known bugs.
539
adfe19db
MHM
540=head1 COPYRIGHT
541
8913c038
KW
542Version 3.x, Copyright (C) 2004-2019, Marcus Holland-Moritz
543and Perl 5 porters
adfe19db
MHM
544
545Version 2.x, Copyright (C) 2001, Paul Marquess.
546
547Version 1.x, Copyright (C) 1999, Kenneth Albanowski.
548
549This program is free software; you can redistribute it and/or
550modify it under the same terms as Perl itself.
551
552=head1 SEE ALSO
553
5cde1e48 554See F<ppport.h> and F<devel/regenerate>.
adfe19db
MHM
555
556=cut