This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
HACKERS: Add info about Hints and Warnings
[perl5.git] / dist / Devel-PPPort / HACKERS
CommitLineData
adfe19db
MHM
1=head1 NAME
2
3HACKERS - Devel::PPPort internals for hackers
4
5=head1 SYNOPSIS
6
7So you probably want to hack C<Devel::PPPort>?
8
9Well, here's some information to get you started with what's
10lying around in this distribution.
11
12=head1 DESCRIPTION
13
8913c038
KW
14=head2 How to backport something
15
16First, make sure that what you want to backport is documented. If it's worth
17backporting, it surely is worth documenting. Send a documentation patch to
18L<perlbug@perl.org|mailto:perlbug@perl.org> if necessary. Also,
19C<Devel::PPPort> cannot automatically generate proper information about the
20item without at least knowing its API prototype. It can get this from
21F<embed.fnc> if the item is a function, but if it is a macro, there needs to be
22at least a S<C<=for apidoc>> line for C<Devel::PPPort> to be able to figure
23things out on its own.
24
25Next, figure out where to place your implementation. Look at all the files in
26F<parts/inc/> for one that fits what you're planning. If there isn't one,
27just start a new one and remember to include it from within F<PPPort_pm.PL>.
28If you do create a new file, it's usually the best approach to just copy an
29existing file and use it as a template.
adfe19db
MHM
30
31Each file holds all relevant data for implementing a certain part
32of the API:
33
34=over 2
35
36=item *
37
38A list of the provided API in the C<=provides> section.
39
40=item *
41
8913c038
KW
42The optional C<=dontwarn> section is used to suppress warnings about particular
43API elements. Don't use this unless you get such a warning, and be sure to
44think about using other other alternatives before resorting to adding something
45in this section.
9712ab2b 46
8913c038 47=item *
9712ab2b 48
8913c038
KW
49The implementation to add to F<ppport.h> in the C<=implementation>
50section. See L</Implementation Section Details>.
9712ab2b 51
adfe19db
MHM
52=item *
53
54The code required to add to PPPort.xs for testing the implementation.
55This code goes into the C<=xshead>, C<=xsinit>, C<=xsmisc>, C<=xsboot>
0d0f8426 56and C<=xsubs> section. Have a look at the template at the bottom
3bfe1c68 57of F<RealPPPort_xs.PL> to see where the code ends up.
adfe19db
MHM
58
59=item *
60
61The tests in the C<=tests> section. Remember not to use any fancy
f6db508c
KW
62modules or syntax elements, as the test code needs to be able to run
63with Perl 5.003. (This is because Devel::PPPort itself will run all test files
64in the process of generating the information about when a feature came into
65existence.) This means, for example
66
67=over
68
69=item C<my> isn't supported in C<for>-loops
adfe19db 70
0d0f8426 71 for my $x (1, 2, 3) { } # won't work with 5.003
adfe19db 72
f6db508c
KW
73Instead declare C<$x> just before the statement
74
8913c038 75=item The postfix C<for> statement modifier isn't supported
f6db508c
KW
76
77 foo for 1..2
78
79won't compile. Instead enclose C<foo> in a loop.
80
81=item You can't use plain C<qr//>
82
83Instead, wrap it in a string eval C<eval "qr//">, and be sure it's skipped at
84execution time on perls earlier than 5.005
85
86=back
87
0d0f8426
MHM
88You can use C<ok()> to report success or failure:
89
f6db508c
KW
90 ok($got, $expected, 'name');
91 ok($got == 42); # Doesn't give good runtime diagnostics
0d0f8426 92
f6db508c
KW
93 ok($got, eval "qr/foo/", 'name') # But don't execute this statement
94 # on perls earlier than 5.005
adfe19db
MHM
95
96=back
97
8913c038 98In all sections, lines that begin with C<##> are completely ignored.
adfe19db 99
8913c038
KW
100=head2 Implementation Section Details
101
102=over
0d0f8426 103
8913c038
KW
104=item __UNDEFINED__
105
106If you add the line C<__UNDEFINED__> to the C<=provides> section, you can use
107lines like this in the C<=implementation> section:
0d0f8426
MHM
108
109 __UNDEFINED__ macro some definition
110
8913c038
KW
111to both define C<macro> and indicate that it is provided by F<ppport.h>. This
112replaces these C<=implementation> section lines:
0d0f8426
MHM
113
114 #ifndef macro
115 # define macro some definition
116 #endif
117
8913c038
KW
118besides automagically making it be considered to be provided. C<macro> can
119have optional arguments and the definition can even span multiple lines, like
120in
0d0f8426
MHM
121
122 __UNDEFINED__ SvMAGIC_set(sv, val) \
123 STMT_START { assert(SvTYPE(sv) >= SVt_PVMG); \
124 (((XPVMG*) SvANY(sv))->xmg_magic = (val)); } STMT_END
125
8913c038
KW
126This usually makes the code more compact and readable.
127
128But you should only use this on things that you plan to publicly provide. If
129something, such as a mnemonic for a constant needs to be defined but isn't
130really needed for the public at larget to know about, you should use
131
132 __UNDEF_NOT_PROVIDED__ macro some definition
133
134instead. To avoid name space conflicts, follow what's in L</Helper macros>,
135below.
136
137=item Helper macros
0d0f8426 138
3451ddaf 139If you need to define a helper macro which is not part of C<Devel::PPPort> API
8913c038
KW
140and its usage is only for the definition of other C<Devel::PPPort> macros, then
141use the C<D_PPP_> prefix for this macro name (e.g. C<D_PPP_SVPV_NOLEN_LP_ARG>).
142This suppresses any warnings when a macro is defined which is not part of the
143Perl public API.
144
145=item Version numbers
3451ddaf 146
f6db508c 147Version checking can be tricky to get correct.
0d0f8426
MHM
148You can use
149
150 #if { VERSION < 5.9.3 }
151
152instead of
153
154 #if ((PERL_VERSION < 9) || (PERL_VERSION == 9 && PERL_SUBVERSION < 3))
155
8913c038 156The version number can be either of the new form C<5.x.x> or the older
0d0f8426
MHM
157form C<5.00x_yy>. Both are translated into the correct preprocessor
158statements. It is also possible to combine this with other statements:
159
160 #if { VERSION >= 5.004 } && !defined(sv_vcatpvf)
b2049988 161 /* a */
0d0f8426
MHM
162 #elif { VERSION < 5.004_63 } && { VERSION != 5.004_05 }
163 /* b */
164 #endif
165
166This not only works in the C<=implementation> section, but also in
167the C<=xsubs>, C<=xsinit>, C<=xsmisc>, C<=xshead> and C<=xsboot> sections.
168
8913c038
KW
169=item Hints
170
171If you add a comment like so:
172
173 /* Hint: PL_expect, PL_copline, PL_rsfp
174 paragraphs of stuff about foo you want to have
175 shown when ppport.h outputs something about PL_expect or
176 PL_copline or PL_rsfp
177 */
178
179Earlier versions of F<ppport.h> required an asterisk at the beginning of every
180continuation line, or else the content would be silently dropped.
181
182=item Warnings
183
184A more serious caution about C<foo> can be displayed by instead saying
185
186 /* Warning: PL_expect, PL_copline, PL_rsfp
187 paragraphs of stuff about foo you want to have
188 shown when ppport.h outputs something about PL_expect or
189 PL_copline or PL_rsfp
190 */
191
192Earlier versions of F<ppport.h> required an asterisk at the beginning of every
193continuation line, or else the content would be silently dropped.
194
195=item Replace
196
197When F<ppport.h> is run on a file(s), you can cause it to automatically flag
198occurrences of the constructs you specify, encouraging the author to replace
199them with different (presumably better) ones. These also are used in any
200suggested edits and generated patches.
201
202There are three ways to do this
203
204=over 4
205
206=item in-line comment
207
208You can add a trailing comment like so:
209
210 #define bar foo /* Replace */
211 __UNDEFINED__ bar foo /* Replace */
212
213These say that C<foo> should be replaced by C<bar>. NOT the other way around.
214
215=item separate comment
216
217For situations not amenable to the above, you can say
218
219 /* Replace foo with bar */
220
221=item define a replacement region
222
223It you have several replacements, you can group them together like so:
224
225 /* Replace: 1 */
226 #define foo bar
227 #define bat baz
228 /* Replace: 0 */
229
230These replace C<foo> with C<bar>; C<bat> with C<baz>.
231
232=back
233
234=item Dependencies
235
236F<ppport.h> automatically gathers information as to what functions are
237dependent on what other things from inspecting the source, but if this is
238insufficient for you, you can add lines like the following:
239
240 /* foo, bar depends on baz, bat */
241
242Each of C<foo>, C<bar> depends on each of C<baz>, C<bat>.
243
244=back
245
adfe19db
MHM
246=head2 Testing
247
8913c038 248After you have furnished your implementation, you need to test it.
adfe19db
MHM
249
250=head2 Special Makefile targets
251
4a582685 252You can use
adfe19db
MHM
253
254 make regen
255
4a582685
NC
256to regenerate all of the autogenerated files. To get rid of all
257generated files (except for F<parts/todo/*> and F<parts/base/*>),
258use
adfe19db
MHM
259
260 make purge_all
261
262That's it.
263
8913c038
KW
264To automatically test C<Devel::PPPort> with lots of different Perl
265versions, you can use the F<soak> script. Just pass it a list of
266all Perl binaries you want to test.
267
268=head2 Regenerating F<ppport.h> and F<PPPort.pm>
269
270C<Devel::PPPort> keeps two directories of generated files, in F<parts/base> and
271F<parts/todo>. The files in each are named after Perl version numbers. When a
272function or macro came into existence is indicated by placing its name in the
273corresponding file in F<parts/base>. The files in F<parts/todo> are the same,
274except they indicate the earliest release that F<ppport.h> supports the
275element. The delta is effectively what F<ppport.h> buys you.
276
277The generation process described in this section creates these files. It does
278so by examining as many perl versions as are available to it. It tries to make
279sure each element actually compiles, and it runs the test scripts you have
280furnished on every version.
281
282Ideally, this should be done before every release that includes new backporting
283and/or when blead has added new public API. At a minimum, it should be done as
284the next major Perl release comes out.
285
286The process isn't platform independent. It has currently been tested only under
287Linux, and it definitely requires at least C<gcc> and the C<nm> utility.
288The process used to be problematic, with random failures. But it has now been
289fixed to be reliable.
290
291Before starting the regeneration, you need to have gathered certain data.
292(Options listed below apply to the tools that eventually will use the data, and
293which are described further below).
294
295=over 4
296
1728d8c7
KW
297Here you can add additional information for a given item that will be displayed
298when F<ppport.h> is run. If your item is named C<foo>, you add a
299comment like so:
300
301 /* Hint: foo
302 paragraphs of stuff about foo you want to have
303 shown when ppport.h outputs something about foo
304 */
305
306This will cause S<C<perl ppport.h>> to display this hint when it outputs
307something about C<foo>.
308
309A more serious caution about C<foo> can be displayed by instead saying
310
311 /* Warning: foo
312 paragraphs of stuff about foo you want to have
313 shown when ppport.h outputs something about foo
314 */
315
8913c038
KW
316=item *
317
318You will first need a whole bunch of different Perls, the more, the better, but
319only one per version tag (which one is random) will actually get used.
320dromedary has a sufficient set. They should all have the same Configure
321options with respect to what functions and macros are enabled. For example,
322they should all be threaded, or all non-threaded. A mixture will screw up the
323results. Similarly, they should all or none have quad math (at least as far
324back as that became available). You can use F<devel/buildperl.pl> to build
325them.
326
327Previous maintainers of this module kept those perls in
328F</tmp/perl/install/default>, so most of the tools use this as a default, but
329you'll likely simply use the C<--install=> option to specify where. This
330should be a path where a S<C<make install>> has been done, so has immediate
331subdirectories of C</bin> and C</lib>. C</bin> should contain the binaries.
332It will use all files in this directory whose names begin with C<perl5>.
333
334Actually, not all the binaries need be in this directory. You can specify
335additional places to look since C<--install=> takes a comma separated list of
336directories.
337
338=item *
339
340You also need a freshly built bleadperl. The C<--blead=I<path>> option should
341be used to specify it. (Some of the tools have a default of C<bleadperl-debug>
342if this option is omitted.) Again, it needs the same Configure options as the
343earlier versions had.
344
345=item *
346
347And you will need updated API information. Copy the latest F<embed.fnc> file
348from bleadperl to the F<parts> directory and run F<devel/mkapidoc.pl> to
349collect the remaining information in F<parts/apidoc.fnc>.
350
351=item *
352
353The final step before regenerating everything is to run F</devel/mkppport.fnc>
354to update the F</parts/ppport.fnc> file.
355
356=back
357
358Having done this, run F<devel/regenerate> which wraps the following steps
359(which you could instead do by hand, but it's easy to forget things):
360
361=over
362
363=item *
364
365It first does some sanity checking
366
367=item *
368
369Then it asks you if it's ok to remove all existing todo files in the
370F<parts/base> and F<parts/todo> directories. If you answer no, the process
371aborts.
372
373This is crtical to getting accurate results.
374
375=item *
376
377It builds the new baseline by running
378
379 perl devel/mktodo --base
380
381in the root directory of the distribution.
382
383If there are warnings in blead, it will ask you to examine them, and to ok if
384it's all right to go ahead. If there are issues with blead, everything
385following could be wrong.
386
387=item *
388
389It builds the new todo files by running
390
391 perl devel/mktodo
392
393in the root directory of the distribution.
394
395=item *
396
397Finally, it adds the remaining information by running
398
399 perl Makefile.PL && make
400 perl devel/scanprov --mode=write
401
402=back
403
404=head2 How to build gobs of versions of Perl
405
406C<Devel::PPPort> supports Perl versions between 5.003 and bleadperl.
407To guarantee this support, its good to have as many versions as possible to
408test on. dromedary currently has many such versions.
409
410There is a tool to build all the different
411versions and configurations. You can find it in F<devel/buildperl.pl>.
412It can currently build the following Perl releases:
413
414 5.003
415 5.004 - 5.004_05
416 5.005 - 5.005_04
417 5.6.x
418 5.7.x
419 5.8.x
420 5.9.x
421 5.1x.x
422 5.2x.x
423 5.3x.x
424
425=head2 Implementation
426
427Knowing which parts of the API are not backwards compatible and
428probably need C<Devel::PPPort> support is another problem that's
429not easy to deal with manually. If you run
430
431 perl Makefile.PL --with-apicheck
432
433a C file is generated by F<parts/apicheck.pl> that is compiled
434and linked with C<Devel::PPPort>. This C file has the purpose of
435using each of the public API functions/macros once.
436
437The required information is derived from F<parts/embed.fnc> (just
438a copy of bleadperl's F<embed.fnc>), F<parts/apidoc.fnc> (which
439is generated by F<devel/mkapidoc.sh> and simply collects the rest
440of the apidoc entries spread over the Perl source code) and
441F<parts/ppport.fnc> (which lists the API provided purely by
442Devel::PPPort, along with other elements that are tested only using
443F<ppport.h>).
444
445The generated C file (usually, F<apicheck.c>) won't compile as-is
446with older perls. And even if it compiles, there's still a good chance of the
447dynamic linker failing at C<make test> time. But that's on purpose!
448
449We can use these failures to find changes in the API automatically.
450The Perl script F<devel/mktodo> calls another script F<devel/mktodo.pl>
451repeatedly to run C<Devel::PPPort> on version after version of perl, in
452decreasing version order, so we start with blead and work backwards. The
453latter script generates an F<apicheck.c>. It starts with the code that
454successfully worked in the previously tested Perl version, which should be the
455version one higher than the current one. Call the current one I<n>, and the
456previous one I<n+1>. The items that fail to compile in I<n>, but did compile
457in I<n+1> must have become available in I<n+1>. We run the Linux command C<nm>
458to find those undefined symbols in I<n>. We change F<apicheck.c> to ignore
459(through C<#ifdef>'s) those and recompile, repeating until F<apicheck.c>
460successfully compiles, the dynamic linker is happy, and C<make test> runs on
461this version. Then we repeat the process for I<n-1>, and so on. (Actually,
462this process may generate false positives, so by default each failing API call
463is checked again. If possible, this is done by generating an F<apicheck.c> for
464just the one failing API.)
465
466Running F<devel/mktodo> currently takes a couple hours on dromedary.
467
468If you run it with the C<--nocheck> option, it won't recheck the API calls
469that failed in the compilation stage and it'll take significantly less time.
470No one currently associated with maintaining this module understands under what
471circumstances it is safe to run with C<--nocheck>.
472
473By repeating the process over and over, we build up information on when every
474element first became supported. This information is stored in files in the
475F<parts/base> directory, one file per version. The file for version I<n+1> is
476generated by running version I<n> of perl.
477
478We actually want a second piece of information, which is how much F<ppport.h>
479buys you. What happens when regenerating is actually two entire runs through
480all the perls. The first is accomplished by calling F<devel/mktodo> with the
481C<--base> option. It automically will call F<devel/mktodo.pl> with each
482version of perl, NOT using anything in F<ppport.h>. When done the results
483indicate when each API element became available in stock perl, without using
484F<ppport.h>.
485
486And then the whole process is repeated, but this time F<ppport.h> is included.
487The files are placed in F<parts/todo>. Thus, at the end, we know when each
488element became available in modified perl, using F<ppport.h>.
489
490However, only the public API that is implemented as functions (and must appear
491in F<embed.fnc>) plus macros whose calling sequence is documented can be
492checked this way. The final step in the process is calling F<devel/scanprov>.
493It looks through the header files for when all the symbols provided by
494C<Devel::PPPort> first became defined. It doesn't test the symbols or try to
495compile them, as it doesn't generally know the API, but it can tell that
496something exists in release I<n+1> but not I<n> (by scanning the include files
497in the F<CORE> directory of various Perl versions). (It does know if a macro
498has zero arguments or non-zero arguments, so it does get extra information from
499the zero argument ones.)
500
501=head2 Files
502
503Residing in F<parts/inc/> is the "heart" of C<Devel::PPPort>. Each
504of the files implements a part of the supported API, along with
505hints, dependency information, XS code and tests.
506The files are in a POD-like format that is parsed using the
507functions in F<parts/ppptools.pl>.
508
509The scripts F<PPPort_pm.PL>, F<RealPPPort_xs.PL> and F<mktests.PL> all
510use the information in F<parts/inc/> to generate the main module
511F<PPPort.pm>, the XS code in F<RealPPPort.xs> and various test files
512in F<t/>.
513
514You can get extra information from F<PPPort_pm.PL> by setting the environment
515variable C<DPPP_CHECK_LEVEL> to 1 or 2.
516
517All of these files could be generated on the fly while building
518C<Devel::PPPort>, but not having the tests in F<t/> will confuse
519TEST/harness in the core. Not having F<PPPort.pm> will be bad for
520viewing the docs on C<search.cpan.org>. So unfortunately, it's
521unavoidable to put some redundancy into the package.
522
0d0f8426
MHM
523=head2 Submitting Patches
524
525If you've added some functionality to C<Devel::PPPort>, please
8913c038
KW
526consider submitting a patch with your work to P5P by sending a pull request to
527
528L<https://github.com/Dual-Life/Devel-PPPort/pulls>.
0d0f8426
MHM
529
530When submitting patches, please only add the relevant changes
531and don't include the differences of the generated files. You
532can use the C<purge_all> target to delete all autogenerated
533files.
534
56093a11
MHM
535=head2 Integrating into the Perl core
536
537When integrating this module into the Perl core, be sure to
538remove the following files from the distribution. They are
539either not needed or generated on the fly when building this
540module in the core:
541
542 MANIFEST
543 META.yml
544 PPPort.pm
545
8913c038
KW
546=head1 BUGS
547
adfe19db
MHM
548=head1 COPYRIGHT
549
8913c038
KW
550Version 3.x, Copyright (C) 2004-2019, Marcus Holland-Moritz
551and Perl 5 porters
adfe19db
MHM
552
553Version 2.x, Copyright (C) 2001, Paul Marquess.
554
555Version 1.x, Copyright (C) 1999, Kenneth Albanowski.
556
557This program is free software; you can redistribute it and/or
558modify it under the same terms as Perl itself.
559
560=head1 SEE ALSO
561
5cde1e48 562See F<ppport.h> and F<devel/regenerate>.
adfe19db
MHM
563
564=cut