This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
[perl #56526] m/a{1,0}/ compiles but doesn't match a literal string
[perl5.git] / pod / perlhack.pod
CommitLineData
e8cd7eae
GS
1=head1 NAME
2
3perlhack - How to hack at the Perl internals
4
5=head1 DESCRIPTION
6
7This document attempts to explain how Perl development takes place,
8and ends with some suggestions for people wanting to become bona fide
9porters.
10
11The perl5-porters mailing list is where the Perl standard distribution
12is maintained and developed. The list can get anywhere from 10 to 150
13messages a day, depending on the heatedness of the debate. Most days
14there are two or three patches, extensions, features, or bugs being
15discussed at a time.
16
f8e3975a 17A searchable archive of the list is at either:
e8cd7eae
GS
18
19 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
20
f8e3975a
IP
21or
22
23 http://archive.develooper.com/perl5-porters@perl.org/
24
e8cd7eae
GS
25List subscribers (the porters themselves) come in several flavours.
26Some are quiet curious lurkers, who rarely pitch in and instead watch
27the ongoing development to ensure they're forewarned of new changes or
28features in Perl. Some are representatives of vendors, who are there
29to make sure that Perl continues to compile and work on their
30platforms. Some patch any reported bug that they know how to fix,
31some are actively patching their pet area (threads, Win32, the regexp
32engine), while others seem to do nothing but complain. In other
33words, it's your usual mix of technical people.
34
35Over this group of porters presides Larry Wall. He has the final word
f6c51b38 36in what does and does not change in the Perl language. Various
b432a672
AL
37releases of Perl are shepherded by a "pumpking", a porter
38responsible for gathering patches, deciding on a patch-by-patch,
f6c51b38 39feature-by-feature basis what will and will not go into the release.
caf100c0 40For instance, Gurusamy Sarathy was the pumpking for the 5.6 release of
961f29c6 41Perl, and Jarkko Hietaniemi was the pumpking for the 5.8 release, and
1a88dbf8 42Rafael Garcia-Suarez holds the pumpking crown for the 5.10 release.
e8cd7eae
GS
43
44In addition, various people are pumpkings for different things. For
961f29c6
MB
45instance, Andy Dougherty and Jarkko Hietaniemi did a grand job as the
46I<Configure> pumpkin up till the 5.8 release. For the 5.10 release
47H.Merijn Brand took over.
e8cd7eae
GS
48
49Larry sees Perl development along the lines of the US government:
50there's the Legislature (the porters), the Executive branch (the
51pumpkings), and the Supreme Court (Larry). The legislature can
52discuss and submit patches to the executive branch all they like, but
53the executive branch is free to veto them. Rarely, the Supreme Court
54will side with the executive branch over the legislature, or the
55legislature over the executive branch. Mostly, however, the
56legislature and the executive branch are supposed to get along and
57work out their differences without impeachment or court cases.
58
59You might sometimes see reference to Rule 1 and Rule 2. Larry's power
60as Supreme Court is expressed in The Rules:
61
62=over 4
63
64=item 1
65
66Larry is always by definition right about how Perl should behave.
67This means he has final veto power on the core functionality.
68
69=item 2
70
71Larry is allowed to change his mind about any matter at a later date,
72regardless of whether he previously invoked Rule 1.
73
74=back
75
76Got that? Larry is always right, even when he was wrong. It's rare
77to see either Rule exercised, but they are often alluded to.
78
79New features and extensions to the language are contentious, because
80the criteria used by the pumpkings, Larry, and other porters to decide
81which features should be implemented and incorporated are not codified
82in a few small design goals as with some other languages. Instead,
83the heuristics are flexible and often difficult to fathom. Here is
84one person's list, roughly in decreasing order of importance, of
85heuristics that new features have to be weighed against:
86
87=over 4
88
89=item Does concept match the general goals of Perl?
90
91These haven't been written anywhere in stone, but one approximation
92is:
93
94 1. Keep it fast, simple, and useful.
95 2. Keep features/concepts as orthogonal as possible.
96 3. No arbitrary limits (platforms, data sizes, cultures).
97 4. Keep it open and exciting to use/patch/advocate Perl everywhere.
98 5. Either assimilate new technologies, or build bridges to them.
99
100=item Where is the implementation?
101
102All the talk in the world is useless without an implementation. In
103almost every case, the person or people who argue for a new feature
104will be expected to be the ones who implement it. Porters capable
105of coding new features have their own agendas, and are not available
106to implement your (possibly good) idea.
107
108=item Backwards compatibility
109
110It's a cardinal sin to break existing Perl programs. New warnings are
111contentious--some say that a program that emits warnings is not
112broken, while others say it is. Adding keywords has the potential to
113break programs, changing the meaning of existing token sequences or
114functions might break programs.
115
116=item Could it be a module instead?
117
118Perl 5 has extension mechanisms, modules and XS, specifically to avoid
119the need to keep changing the Perl interpreter. You can write modules
120that export functions, you can give those functions prototypes so they
121can be called like built-in functions, you can even write XS code to
122mess with the runtime data structures of the Perl interpreter if you
123want to implement really complicated things. If it can be done in a
124module instead of in the core, it's highly unlikely to be added.
125
126=item Is the feature generic enough?
127
128Is this something that only the submitter wants added to the language,
129or would it be broadly useful? Sometimes, instead of adding a feature
130with a tight focus, the porters might decide to wait until someone
131implements the more generalized feature. For instance, instead of
b432a672 132implementing a "delayed evaluation" feature, the porters are waiting
e8cd7eae
GS
133for a macro system that would permit delayed evaluation and much more.
134
135=item Does it potentially introduce new bugs?
136
137Radical rewrites of large chunks of the Perl interpreter have the
138potential to introduce new bugs. The smaller and more localized the
139change, the better.
140
141=item Does it preclude other desirable features?
142
143A patch is likely to be rejected if it closes off future avenues of
144development. For instance, a patch that placed a true and final
145interpretation on prototypes is likely to be rejected because there
146are still options for the future of prototypes that haven't been
147addressed.
148
149=item Is the implementation robust?
150
151Good patches (tight code, complete, correct) stand more chance of
152going in. Sloppy or incorrect patches might be placed on the back
153burner until the pumpking has time to fix, or might be discarded
154altogether without further notice.
155
156=item Is the implementation generic enough to be portable?
157
158The worst patches make use of a system-specific features. It's highly
353c6505 159unlikely that non-portable additions to the Perl language will be
e8cd7eae
GS
160accepted.
161
a936dd3c
NC
162=item Is the implementation tested?
163
164Patches which change behaviour (fixing bugs or introducing new features)
165must include regression tests to verify that everything works as expected.
166Without tests provided by the original author, how can anyone else changing
167perl in the future be sure that they haven't unwittingly broken the behaviour
168the patch implements? And without tests, how can the patch's author be
9d077eaa 169confident that his/her hard work put into the patch won't be accidentally
a936dd3c
NC
170thrown away by someone in the future?
171
e8cd7eae
GS
172=item Is there enough documentation?
173
174Patches without documentation are probably ill-thought out or
175incomplete. Nothing can be added without documentation, so submitting
176a patch for the appropriate manpages as well as the source code is
a936dd3c 177always a good idea.
e8cd7eae
GS
178
179=item Is there another way to do it?
180
b432a672
AL
181Larry said "Although the Perl Slogan is I<There's More Than One Way
182to Do It>, I hesitate to make 10 ways to do something". This is a
e8cd7eae
GS
183tricky heuristic to navigate, though--one man's essential addition is
184another man's pointless cruft.
185
186=item Does it create too much work?
187
188Work for the pumpking, work for Perl programmers, work for module
189authors, ... Perl is supposed to be easy.
190
f6c51b38
GS
191=item Patches speak louder than words
192
193Working code is always preferred to pie-in-the-sky ideas. A patch to
194add a feature stands a much higher chance of making it to the language
195than does a random feature request, no matter how fervently argued the
b432a672 196request might be. This ties into "Will it be useful?", as the fact
f6c51b38
GS
197that someone took the time to make the patch demonstrates a strong
198desire for the feature.
199
e8cd7eae
GS
200=back
201
b432a672
AL
202If you're on the list, you might hear the word "core" bandied
203around. It refers to the standard distribution. "Hacking on the
204core" means you're changing the C source code to the Perl
205interpreter. "A core module" is one that ships with Perl.
e8cd7eae 206
a1f349fd
MB
207=head2 Keeping in sync
208
e8cd7eae 209The source code to the Perl interpreter, in its different versions, is
f224927c
JH
210kept in a repository managed by a revision control system ( which is
211currently the Perforce program, see http://perforce.com/ ). The
e8cd7eae
GS
212pumpkings and a few others have access to the repository to check in
213changes. Periodically the pumpking for the development version of Perl
214will release a new version, so the rest of the porters can see what's
2be4c08b
GS
215changed. The current state of the main trunk of repository, and patches
216that describe the individual changes that have happened since the last
217public release are available at this location:
218
eb8afeef
SP
219 http://public.activestate.com/pub/apc/
220 ftp://public.activestate.com/pub/apc/
2be4c08b 221
0cfb3454
GS
222If you're looking for a particular change, or a change that affected
223a particular set of files, you may find the B<Perl Repository Browser>
224useful:
225
226 http://public.activestate.com/cgi-bin/perlbrowse
227
228You may also want to subscribe to the perl5-changes mailing list to
229receive a copy of each patch that gets submitted to the maintenance
230and development "branches" of the perl repository. See
231http://lists.perl.org/ for subscription information.
232
a1f349fd
MB
233If you are a member of the perl5-porters mailing list, it is a good
234thing to keep in touch with the most recent changes. If not only to
235verify if what you would have posted as a bug report isn't already
236solved in the most recent available perl development branch, also
237known as perl-current, bleading edge perl, bleedperl or bleadperl.
2be4c08b
GS
238
239Needless to say, the source code in perl-current is usually in a perpetual
240state of evolution. You should expect it to be very buggy. Do B<not> use
241it for any purpose other than testing and development.
e8cd7eae 242
3e148164
JH
243Keeping in sync with the most recent branch can be done in several ways,
244but the most convenient and reliable way is using B<rsync>, available at
245ftp://rsync.samba.org/pub/rsync/ . (You can also get the most recent
246branch by FTP.)
a1f349fd
MB
247
248If you choose to keep in sync using rsync, there are two approaches
3e148164 249to doing so:
a1f349fd
MB
250
251=over 4
252
253=item rsync'ing the source tree
254
3e148164 255Presuming you are in the directory where your perl source resides
b432a672 256and you have rsync installed and available, you can "upgrade" to
a1f349fd
MB
257the bleadperl using:
258
eb8afeef 259 # rsync -avz rsync://public.activestate.com/perl-current/ .
a1f349fd
MB
260
261This takes care of updating every single item in the source tree to
262the latest applied patch level, creating files that are new (to your
263distribution) and setting date/time stamps of existing files to
264reflect the bleadperl status.
265
c6d0653e
LC
266Note that this will not delete any files that were in '.' before
267the rsync. Once you are sure that the rsync is running correctly,
268run it with the --delete and the --dry-run options like this:
269
eb8afeef 270 # rsync -avz --delete --dry-run rsync://public.activestate.com/perl-current/ .
c6d0653e
LC
271
272This will I<simulate> an rsync run that also deletes files not
273present in the bleadperl master copy. Observe the results from
274this run closely. If you are sure that the actual run would delete
275no files precious to you, you could remove the '--dry-run' option.
276
a1f349fd
MB
277You can than check what patch was the latest that was applied by
278looking in the file B<.patch>, which will show the number of the
279latest patch.
280
281If you have more than one machine to keep in sync, and not all of
282them have access to the WAN (so you are not able to rsync all the
283source trees to the real source), there are some ways to get around
284this problem.
285
286=over 4
287
288=item Using rsync over the LAN
289
290Set up a local rsync server which makes the rsynced source tree
3e148164 291available to the LAN and sync the other machines against this
a1f349fd
MB
292directory.
293
1577cd80 294From http://rsync.samba.org/README.html :
a1f349fd
MB
295
296 "Rsync uses rsh or ssh for communication. It does not need to be
297 setuid and requires no special privileges for installation. It
3958b146 298 does not require an inetd entry or a daemon. You must, however,
a1f349fd
MB
299 have a working rsh or ssh system. Using ssh is recommended for
300 its security features."
301
302=item Using pushing over the NFS
303
304Having the other systems mounted over the NFS, you can take an
3e148164
JH
305active pushing approach by checking the just updated tree against
306the other not-yet synced trees. An example would be
307
308 #!/usr/bin/perl -w
309
310 use strict;
311 use File::Copy;
312
313 my %MF = map {
314 m/(\S+)/;
315 $1 => [ (stat $1)[2, 7, 9] ]; # mode, size, mtime
316 } `cat MANIFEST`;
317
318 my %remote = map { $_ => "/$_/pro/3gl/CPAN/perl-5.7.1" } qw(host1 host2);
319
320 foreach my $host (keys %remote) {
321 unless (-d $remote{$host}) {
322 print STDERR "Cannot Xsync for host $host\n";
323 next;
324 }
325 foreach my $file (keys %MF) {
326 my $rfile = "$remote{$host}/$file";
327 my ($mode, $size, $mtime) = (stat $rfile)[2, 7, 9];
328 defined $size or ($mode, $size, $mtime) = (0, 0, 0);
329 $size == $MF{$file}[1] && $mtime == $MF{$file}[2] and next;
330 printf "%4s %-34s %8d %9d %8d %9d\n",
331 $host, $file, $MF{$file}[1], $MF{$file}[2], $size, $mtime;
332 unlink $rfile;
333 copy ($file, $rfile);
334 utime time, $MF{$file}[2], $rfile;
335 chmod $MF{$file}[0], $rfile;
336 }
337 }
338
339though this is not perfect. It could be improved with checking
a1f349fd
MB
340file checksums before updating. Not all NFS systems support
341reliable utime support (when used over the NFS).
342
343=back
344
345=item rsync'ing the patches
346
347The source tree is maintained by the pumpking who applies patches to
348the files in the tree. These patches are either created by the
349pumpking himself using C<diff -c> after updating the file manually or
350by applying patches sent in by posters on the perl5-porters list.
351These patches are also saved and rsync'able, so you can apply them
352yourself to the source files.
353
354Presuming you are in a directory where your patches reside, you can
3e148164 355get them in sync with
a1f349fd 356
eb8afeef 357 # rsync -avz rsync://public.activestate.com/perl-current-diffs/ .
a1f349fd
MB
358
359This makes sure the latest available patch is downloaded to your
360patch directory.
361
3e148164 362It's then up to you to apply these patches, using something like
a1f349fd 363
7d3c2c28 364 # last="`cat ../perl-current/.patch`.gz"
eb8afeef 365 # rsync -avz rsync://public.activestate.com/perl-current-diffs/ .
a1f349fd
MB
366 # find . -name '*.gz' -newer $last -exec gzcat {} \; >blead.patch
367 # cd ../perl-current
368 # patch -p1 -N <../perl-current-diffs/blead.patch
369
370or, since this is only a hint towards how it works, use CPAN-patchaperl
371from Andreas K├Ânig to have better control over the patching process.
372
373=back
374
f7e1e956 375=head2 Why rsync the source tree
a1f349fd
MB
376
377=over 4
378
10f58044 379=item It's easier to rsync the source tree
a1f349fd
MB
380
381Since you don't have to apply the patches yourself, you are sure all
382files in the source tree are in the right state.
383
a1f349fd
MB
384=item It's more reliable
385
0cfb3454
GS
386While both the rsync-able source and patch areas are automatically
387updated every few minutes, keep in mind that applying patches may
388sometimes mean careful hand-holding, especially if your version of
389the C<patch> program does not understand how to deal with new files,
390files with 8-bit characters, or files without trailing newlines.
a1f349fd
MB
391
392=back
393
f7e1e956 394=head2 Why rsync the patches
a1f349fd
MB
395
396=over 4
397
10f58044 398=item It's easier to rsync the patches
a1f349fd
MB
399
400If you have more than one machine that you want to keep in track with
3e148164 401bleadperl, it's easier to rsync the patches only once and then apply
a1f349fd
MB
402them to all the source trees on the different machines.
403
404In case you try to keep in pace on 5 different machines, for which
405only one of them has access to the WAN, rsync'ing all the source
3e148164 406trees should than be done 5 times over the NFS. Having
a1f349fd 407rsync'ed the patches only once, I can apply them to all the source
3e148164 408trees automatically. Need you say more ;-)
a1f349fd
MB
409
410=item It's a good reference
411
412If you do not only like to have the most recent development branch,
413but also like to B<fix> bugs, or extend features, you want to dive
414into the sources. If you are a seasoned perl core diver, you don't
415need no manuals, tips, roadmaps, perlguts.pod or other aids to find
416your way around. But if you are a starter, the patches may help you
417in finding where you should start and how to change the bits that
418bug you.
419
420The file B<Changes> is updated on occasions the pumpking sees as his
421own little sync points. On those occasions, he releases a tar-ball of
422the current source tree (i.e. perl@7582.tar.gz), which will be an
423excellent point to start with when choosing to use the 'rsync the
424patches' scheme. Starting with perl@7582, which means a set of source
425files on which the latest applied patch is number 7582, you apply all
f18956b7 426succeeding patches available from then on (7583, 7584, ...).
a1f349fd
MB
427
428You can use the patches later as a kind of search archive.
429
430=over 4
431
432=item Finding a start point
433
434If you want to fix/change the behaviour of function/feature Foo, just
435scan the patches for patches that mention Foo either in the subject,
3e148164 436the comments, or the body of the fix. A good chance the patch shows
a1f349fd
MB
437you the files that are affected by that patch which are very likely
438to be the starting point of your journey into the guts of perl.
439
440=item Finding how to fix a bug
441
442If you've found I<where> the function/feature Foo misbehaves, but you
443don't know how to fix it (but you do know the change you want to
444make), you can, again, peruse the patches for similar changes and
445look how others apply the fix.
446
447=item Finding the source of misbehaviour
448
449When you keep in sync with bleadperl, the pumpking would love to
3958b146 450I<see> that the community efforts really work. So after each of his
a1f349fd
MB
451sync points, you are to 'make test' to check if everything is still
452in working order. If it is, you do 'make ok', which will send an OK
902821cc 453report to I<perlbug@perl.org>. (If you do not have access to a mailer
3e148164 454from the system you just finished successfully 'make test', you can
a1f349fd
MB
455do 'make okfile', which creates the file C<perl.ok>, which you can
456than take to your favourite mailer and mail yourself).
457
3958b146 458But of course, as always, things will not always lead to a success
a1f349fd
MB
459path, and one or more test do not pass the 'make test'. Before
460sending in a bug report (using 'make nok' or 'make nokfile'), check
461the mailing list if someone else has reported the bug already and if
462so, confirm it by replying to that message. If not, you might want to
463trace the source of that misbehaviour B<before> sending in the bug,
464which will help all the other porters in finding the solution.
465
3e148164
JH
466Here the saved patches come in very handy. You can check the list of
467patches to see which patch changed what file and what change caused
468the misbehaviour. If you note that in the bug report, it saves the
469one trying to solve it, looking for that point.
a1f349fd
MB
470
471=back
472
473If searching the patches is too bothersome, you might consider using
474perl's bugtron to find more information about discussions and
475ramblings on posted bugs.
476
3e148164
JH
477If you want to get the best of both worlds, rsync both the source
478tree for convenience, reliability and ease and rsync the patches
479for reference.
480
52315700
RF
481=back
482
fcc89a64
DR
483=head2 Working with the source
484
485Because you cannot use the Perforce client, you cannot easily generate
486diffs against the repository, nor will merges occur when you update
487via rsync. If you edit a file locally and then rsync against the
488latest source, changes made in the remote copy will I<overwrite> your
489local versions!
490
491The best way to deal with this is to maintain a tree of symlinks to
492the rsync'd source. Then, when you want to edit a file, you remove
493the symlink, copy the real file into the other tree, and edit it. You
494can then diff your edited file against the original to generate a
495patch, and you can safely update the original tree.
496
497Perl's F<Configure> script can generate this tree of symlinks for you.
498The following example assumes that you have used rsync to pull a copy
499of the Perl source into the F<perl-rsync> directory. In the directory
500above that one, you can execute the following commands:
501
502 mkdir perl-dev
503 cd perl-dev
504 ../perl-rsync/Configure -Dmksymlinks -Dusedevel -D"optimize=-g"
505
506This will start the Perl configuration process. After a few prompts,
507you should see something like this:
508
509 Symbolic links are supported.
510
511 Checking how to test for symbolic links...
512 Your builtin 'test -h' may be broken.
513 Trying external '/usr/bin/test -h'.
514 You can test for symbolic links with '/usr/bin/test -h'.
515
516 Creating the symbolic links...
517 (First creating the subdirectories...)
518 (Then creating the symlinks...)
519
520The specifics may vary based on your operating system, of course.
521After you see this, you can abort the F<Configure> script, and you
522will see that the directory you are in has a tree of symlinks to the
523F<perl-rsync> directories and files.
524
525If you plan to do a lot of work with the Perl source, here are some
526Bourne shell script functions that can make your life easier:
527
528 function edit {
529 if [ -L $1 ]; then
530 mv $1 $1.orig
62aa7ed0
RGS
531 cp $1.orig $1
532 vi $1
fcc89a64 533 else
62aa7ed0
RGS
534 vi $1
535 fi
fcc89a64
DR
536 }
537
538 function unedit {
539 if [ -L $1.orig ]; then
540 rm $1
62aa7ed0
RGS
541 mv $1.orig $1
542 fi
fcc89a64
DR
543 }
544
545Replace "vi" with your favorite flavor of editor.
546
547Here is another function which will quickly generate a patch for the
548files which have been edited in your symlink tree:
549
550 mkpatchorig() {
551 local diffopts
62aa7ed0
RGS
552 for f in `find . -name '*.orig' | sed s,^\./,,`
553 do
554 case `echo $f | sed 's,.orig$,,;s,.*\.,,'` in
555 c) diffopts=-p ;;
fcc89a64
DR
556 pod) diffopts='-F^=' ;;
557 *) diffopts= ;;
62aa7ed0
RGS
558 esac
559 diff -du $diffopts $f `echo $f | sed 's,.orig$,,'`
560 done
fcc89a64
DR
561 }
562
563This function produces patches which include enough context to make
564your changes obvious. This makes it easier for the Perl pumpking(s)
565to review them when you send them to the perl5-porters list, and that
566means they're more likely to get applied.
567
568This function assumed a GNU diff, and may require some tweaking for
569other diff variants.
52315700 570
3fd28c4e 571=head2 Perlbug administration
52315700 572
902821cc
RGS
573There is a single remote administrative interface for modifying bug status,
574category, open issues etc. using the B<RT> bugtracker system, maintained
575by Robert Spier. Become an administrator, and close any bugs you can get
3fd28c4e 576your sticky mitts on:
52315700 577
39417508 578 http://bugs.perl.org/
52315700 579
3fd28c4e 580To email the bug system administrators:
52315700 581
3fd28c4e 582 "perlbug-admin" <perlbug-admin@perl.org>
52315700 583
a1f349fd
MB
584=head2 Submitting patches
585
f7e1e956
MS
586Always submit patches to I<perl5-porters@perl.org>. If you're
587patching a core module and there's an author listed, send the author a
588copy (see L<Patching a core module>). This lets other porters review
589your patch, which catches a surprising number of errors in patches.
590Either use the diff program (available in source code form from
f224927c 591ftp://ftp.gnu.org/pub/gnu/ , or use Johan Vromans' I<makepatch>
f7e1e956
MS
592(available from I<CPAN/authors/id/JV/>). Unified diffs are preferred,
593but context diffs are accepted. Do not send RCS-style diffs or diffs
594without context lines. More information is given in the
595I<Porting/patching.pod> file in the Perl source distribution. Please
824d470b
RGS
596patch against the latest B<development> version. (e.g., even if you're
597fixing a bug in the 5.8 track, patch against the latest B<development>
598version rsynced from rsync://public.activestate.com/perl-current/ )
599
600If changes are accepted, they are applied to the development branch. Then
601the 5.8 pumpking decides which of those patches is to be backported to the
602maint branch. Only patches that survive the heat of the development
f7e1e956
MS
603branch get applied to maintenance versions.
604
2c8694a7
JH
605Your patch should update the documentation and test suite. See
606L<Writing a test>. If you have added or removed files in the distribution,
607edit the MANIFEST file accordingly, sort the MANIFEST file using
608C<make manisort>, and include those changes as part of your patch.
e8cd7eae 609
824d470b
RGS
610Patching documentation also follows the same order: if accepted, a patch
611is first applied to B<development>, and if relevant then it's backported
612to B<maintenance>. (With an exception for some patches that document
613behaviour that only appears in the maintenance branch, but which has
614changed in the development version.)
615
e8cd7eae
GS
616To report a bug in Perl, use the program I<perlbug> which comes with
617Perl (if you can't get Perl to work, send mail to the address
f18956b7 618I<perlbug@perl.org> or I<perlbug@perl.com>). Reporting bugs through
e8cd7eae 619I<perlbug> feeds into the automated bug-tracking system, access to
902821cc 620which is provided through the web at http://rt.perl.org/rt3/ . It
e8cd7eae
GS
621often pays to check the archives of the perl5-porters mailing list to
622see whether the bug you're reporting has been reported before, and if
623so whether it was considered a bug. See above for the location of
624the searchable archives.
625
f224927c 626The CPAN testers ( http://testers.cpan.org/ ) are a group of
ba139f7d 627volunteers who test CPAN modules on a variety of platforms. Perl
d3e8af89
GS
628Smokers ( http://www.nntp.perl.org/group/perl.daily-build and
629http://www.nntp.perl.org/group/perl.daily-build.reports/ )
902821cc 630automatically test Perl source releases on platforms with various
d3e8af89
GS
631configurations. Both efforts welcome volunteers. In order to get
632involved in smoke testing of the perl itself visit
633L<http://search.cpan.org/dist/Test-Smoke>. In order to start smoke
634testing CPAN modules visit L<http://search.cpan.org/dist/CPAN-YACSmoke/>
635or L<http://search.cpan.org/dist/POE-Component-CPAN-YACSmoke/> or
636L<http://search.cpan.org/dist/CPAN-Reporter/>.
e8cd7eae 637
e8cd7eae
GS
638It's a good idea to read and lurk for a while before chipping in.
639That way you'll get to see the dynamic of the conversations, learn the
640personalities of the players, and hopefully be better prepared to make
641a useful contribution when do you speak up.
642
643If after all this you still think you want to join the perl5-porters
f6c51b38
GS
644mailing list, send mail to I<perl5-porters-subscribe@perl.org>. To
645unsubscribe, send mail to I<perl5-porters-unsubscribe@perl.org>.
e8cd7eae 646
a422fd2d
SC
647To hack on the Perl guts, you'll need to read the following things:
648
649=over 3
650
651=item L<perlguts>
652
653This is of paramount importance, since it's the documentation of what
654goes where in the Perl source. Read it over a couple of times and it
655might start to make sense - don't worry if it doesn't yet, because the
656best way to study it is to read it in conjunction with poking at Perl
657source, and we'll do that later on.
658
659You might also want to look at Gisle Aas's illustrated perlguts -
660there's no guarantee that this will be absolutely up-to-date with the
661latest documentation in the Perl core, but the fundamentals will be
1577cd80 662right. ( http://gisle.aas.no/perl/illguts/ )
a422fd2d
SC
663
664=item L<perlxstut> and L<perlxs>
665
666A working knowledge of XSUB programming is incredibly useful for core
667hacking; XSUBs use techniques drawn from the PP code, the portion of the
668guts that actually executes a Perl program. It's a lot gentler to learn
669those techniques from simple examples and explanation than from the core
670itself.
671
672=item L<perlapi>
673
674The documentation for the Perl API explains what some of the internal
675functions do, as well as the many macros used in the source.
676
677=item F<Porting/pumpkin.pod>
678
679This is a collection of words of wisdom for a Perl porter; some of it is
680only useful to the pumpkin holder, but most of it applies to anyone
681wanting to go about Perl development.
682
683=item The perl5-porters FAQ
684
902821cc
RGS
685This should be available from http://dev.perl.org/perl5/docs/p5p-faq.html .
686It contains hints on reading perl5-porters, information on how
687perl5-porters works and how Perl development in general works.
a422fd2d
SC
688
689=back
690
691=head2 Finding Your Way Around
692
693Perl maintenance can be split into a number of areas, and certain people
694(pumpkins) will have responsibility for each area. These areas sometimes
695correspond to files or directories in the source kit. Among the areas are:
696
697=over 3
698
699=item Core modules
700
701Modules shipped as part of the Perl core live in the F<lib/> and F<ext/>
702subdirectories: F<lib/> is for the pure-Perl modules, and F<ext/>
703contains the core XS modules.
704
f7e1e956
MS
705=item Tests
706
707There are tests for nearly all the modules, built-ins and major bits
708of functionality. Test files all have a .t suffix. Module tests live
709in the F<lib/> and F<ext/> directories next to the module being
710tested. Others live in F<t/>. See L<Writing a test>
711
a422fd2d
SC
712=item Documentation
713
714Documentation maintenance includes looking after everything in the
715F<pod/> directory, (as well as contributing new documentation) and
716the documentation to the modules in core.
717
718=item Configure
719
720The configure process is the way we make Perl portable across the
721myriad of operating systems it supports. Responsibility for the
722configure, build and installation process, as well as the overall
723portability of the core code rests with the configure pumpkin - others
724help out with individual operating systems.
725
726The files involved are the operating system directories, (F<win32/>,
727F<os2/>, F<vms/> and so on) the shell scripts which generate F<config.h>
728and F<Makefile>, as well as the metaconfig files which generate
729F<Configure>. (metaconfig isn't included in the core distribution.)
730
731=item Interpreter
732
733And of course, there's the core of the Perl interpreter itself. Let's
734have a look at that in a little more detail.
735
736=back
737
738Before we leave looking at the layout, though, don't forget that
739F<MANIFEST> contains not only the file names in the Perl distribution,
740but short descriptions of what's in them, too. For an overview of the
741important files, try this:
742
743 perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST
744
745=head2 Elements of the interpreter
746
747The work of the interpreter has two main stages: compiling the code
748into the internal representation, or bytecode, and then executing it.
749L<perlguts/Compiled code> explains exactly how the compilation stage
750happens.
751
752Here is a short breakdown of perl's operation:
753
754=over 3
755
756=item Startup
757
758The action begins in F<perlmain.c>. (or F<miniperlmain.c> for miniperl)
759This is very high-level code, enough to fit on a single screen, and it
760resembles the code found in L<perlembed>; most of the real action takes
761place in F<perl.c>
762
9df8f87f
LB
763F<perlmain.c> is generated by L<writemain> from F<miniperlmain.c> at
764make time, so you should make perl to follow this along.
765
a422fd2d 766First, F<perlmain.c> allocates some memory and constructs a Perl
9df8f87f 767interpreter, along these lines:
a422fd2d
SC
768
769 1 PERL_SYS_INIT3(&argc,&argv,&env);
770 2
771 3 if (!PL_do_undump) {
772 4 my_perl = perl_alloc();
773 5 if (!my_perl)
774 6 exit(1);
775 7 perl_construct(my_perl);
776 8 PL_perl_destruct_level = 0;
777 9 }
778
779Line 1 is a macro, and its definition is dependent on your operating
780system. Line 3 references C<PL_do_undump>, a global variable - all
781global variables in Perl start with C<PL_>. This tells you whether the
782current running program was created with the C<-u> flag to perl and then
783F<undump>, which means it's going to be false in any sane context.
784
785Line 4 calls a function in F<perl.c> to allocate memory for a Perl
786interpreter. It's quite a simple function, and the guts of it looks like
787this:
788
789 my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter));
790
791Here you see an example of Perl's system abstraction, which we'll see
792later: C<PerlMem_malloc> is either your system's C<malloc>, or Perl's
793own C<malloc> as defined in F<malloc.c> if you selected that option at
794configure time.
795
9df8f87f
LB
796Next, in line 7, we construct the interpreter using perl_construct,
797also in F<perl.c>; this sets up all the special variables that Perl
798needs, the stacks, and so on.
a422fd2d
SC
799
800Now we pass Perl the command line options, and tell it to go:
801
802 exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL);
9df8f87f
LB
803 if (!exitstatus)
804 perl_run(my_perl);
805
806 exitstatus = perl_destruct(my_perl);
a422fd2d 807
9df8f87f 808 perl_free(my_perl);
a422fd2d
SC
809
810C<perl_parse> is actually a wrapper around C<S_parse_body>, as defined
811in F<perl.c>, which processes the command line options, sets up any
812statically linked XS modules, opens the program and calls C<yyparse> to
813parse it.
814
815=item Parsing
816
817The aim of this stage is to take the Perl source, and turn it into an op
818tree. We'll see what one of those looks like later. Strictly speaking,
819there's three things going on here.
820
821C<yyparse>, the parser, lives in F<perly.c>, although you're better off
822reading the original YACC input in F<perly.y>. (Yes, Virginia, there
823B<is> a YACC grammar for Perl!) The job of the parser is to take your
b432a672 824code and "understand" it, splitting it into sentences, deciding which
a422fd2d
SC
825operands go with which operators and so on.
826
827The parser is nobly assisted by the lexer, which chunks up your input
828into tokens, and decides what type of thing each token is: a variable
829name, an operator, a bareword, a subroutine, a core function, and so on.
830The main point of entry to the lexer is C<yylex>, and that and its
831associated routines can be found in F<toke.c>. Perl isn't much like
832other computer languages; it's highly context sensitive at times, it can
833be tricky to work out what sort of token something is, or where a token
834ends. As such, there's a lot of interplay between the tokeniser and the
835parser, which can get pretty frightening if you're not used to it.
836
837As the parser understands a Perl program, it builds up a tree of
838operations for the interpreter to perform during execution. The routines
839which construct and link together the various operations are to be found
840in F<op.c>, and will be examined later.
841
842=item Optimization
843
844Now the parsing stage is complete, and the finished tree represents
845the operations that the Perl interpreter needs to perform to execute our
846program. Next, Perl does a dry run over the tree looking for
847optimisations: constant expressions such as C<3 + 4> will be computed
848now, and the optimizer will also see if any multiple operations can be
849replaced with a single one. For instance, to fetch the variable C<$foo>,
850instead of grabbing the glob C<*foo> and looking at the scalar
851component, the optimizer fiddles the op tree to use a function which
852directly looks up the scalar in question. The main optimizer is C<peep>
853in F<op.c>, and many ops have their own optimizing functions.
854
855=item Running
856
857Now we're finally ready to go: we have compiled Perl byte code, and all
858that's left to do is run it. The actual execution is done by the
859C<runops_standard> function in F<run.c>; more specifically, it's done by
860these three innocent looking lines:
861
862 while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
863 PERL_ASYNC_CHECK();
864 }
865
866You may be more comfortable with the Perl version of that:
867
868 PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}};
869
870Well, maybe not. Anyway, each op contains a function pointer, which
871stipulates the function which will actually carry out the operation.
872This function will return the next op in the sequence - this allows for
873things like C<if> which choose the next op dynamically at run time.
874The C<PERL_ASYNC_CHECK> makes sure that things like signals interrupt
875execution if required.
876
877The actual functions called are known as PP code, and they're spread
b432a672 878between four files: F<pp_hot.c> contains the "hot" code, which is most
a422fd2d
SC
879often used and highly optimized, F<pp_sys.c> contains all the
880system-specific functions, F<pp_ctl.c> contains the functions which
881implement control structures (C<if>, C<while> and the like) and F<pp.c>
882contains everything else. These are, if you like, the C code for Perl's
883built-in functions and operators.
884
dfc98234
DM
885Note that each C<pp_> function is expected to return a pointer to the next
886op. Calls to perl subs (and eval blocks) are handled within the same
887runops loop, and do not consume extra space on the C stack. For example,
888C<pp_entersub> and C<pp_entertry> just push a C<CxSUB> or C<CxEVAL> block
889struct onto the context stack which contain the address of the op
890following the sub call or eval. They then return the first op of that sub
891or eval block, and so execution continues of that sub or block. Later, a
892C<pp_leavesub> or C<pp_leavetry> op pops the C<CxSUB> or C<CxEVAL>,
893retrieves the return op from it, and returns it.
894
895=item Exception handing
896
0503309d 897Perl's exception handing (i.e. C<die> etc.) is built on top of the low-level
dfc98234 898C<setjmp()>/C<longjmp()> C-library functions. These basically provide a
28a5cf3b 899way to capture the current PC and SP registers and later restore them; i.e.
dfc98234
DM
900a C<longjmp()> continues at the point in code where a previous C<setjmp()>
901was done, with anything further up on the C stack being lost. This is why
902code should always save values using C<SAVE_FOO> rather than in auto
903variables.
904
905The perl core wraps C<setjmp()> etc in the macros C<JMPENV_PUSH> and
906C<JMPENV_JUMP>. The basic rule of perl exceptions is that C<exit>, and
907C<die> (in the absence of C<eval>) perform a C<JMPENV_JUMP(2)>, while
908C<die> within C<eval> does a C<JMPENV_JUMP(3)>.
909
910At entry points to perl, such as C<perl_parse()>, C<perl_run()> and
911C<call_sv(cv, G_EVAL)> each does a C<JMPENV_PUSH>, then enter a runops
912loop or whatever, and handle possible exception returns. For a 2 return,
913final cleanup is performed, such as popping stacks and calling C<CHECK> or
914C<END> blocks. Amongst other things, this is how scope cleanup still
915occurs during an C<exit>.
916
917If a C<die> can find a C<CxEVAL> block on the context stack, then the
918stack is popped to that level and the return op in that block is assigned
919to C<PL_restartop>; then a C<JMPENV_JUMP(3)> is performed. This normally
920passes control back to the guard. In the case of C<perl_run> and
921C<call_sv>, a non-null C<PL_restartop> triggers re-entry to the runops
922loop. The is the normal way that C<die> or C<croak> is handled within an
923C<eval>.
924
925Sometimes ops are executed within an inner runops loop, such as tie, sort
926or overload code. In this case, something like
927
928 sub FETCH { eval { die } }
929
930would cause a longjmp right back to the guard in C<perl_run>, popping both
931runops loops, which is clearly incorrect. One way to avoid this is for the
932tie code to do a C<JMPENV_PUSH> before executing C<FETCH> in the inner
933runops loop, but for efficiency reasons, perl in fact just sets a flag,
934using C<CATCH_SET(TRUE)>. The C<pp_require>, C<pp_entereval> and
935C<pp_entertry> ops check this flag, and if true, they call C<docatch>,
936which does a C<JMPENV_PUSH> and starts a new runops level to execute the
937code, rather than doing it on the current loop.
938
939As a further optimisation, on exit from the eval block in the C<FETCH>,
940execution of the code following the block is still carried on in the inner
941loop. When an exception is raised, C<docatch> compares the C<JMPENV>
942level of the C<CxEVAL> with C<PL_top_env> and if they differ, just
943re-throws the exception. In this way any inner loops get popped.
944
945Here's an example.
946
947 1: eval { tie @a, 'A' };
948 2: sub A::TIEARRAY {
949 3: eval { die };
950 4: die;
951 5: }
952
953To run this code, C<perl_run> is called, which does a C<JMPENV_PUSH> then
954enters a runops loop. This loop executes the eval and tie ops on line 1,
955with the eval pushing a C<CxEVAL> onto the context stack.
956
957The C<pp_tie> does a C<CATCH_SET(TRUE)>, then starts a second runops loop
958to execute the body of C<TIEARRAY>. When it executes the entertry op on
959line 3, C<CATCH_GET> is true, so C<pp_entertry> calls C<docatch> which
960does a C<JMPENV_PUSH> and starts a third runops loop, which then executes
961the die op. At this point the C call stack looks like this:
962
963 Perl_pp_die
964 Perl_runops # third loop
965 S_docatch_body
966 S_docatch
967 Perl_pp_entertry
968 Perl_runops # second loop
969 S_call_body
970 Perl_call_sv
971 Perl_pp_tie
972 Perl_runops # first loop
973 S_run_body
974 perl_run
975 main
976
977and the context and data stacks, as shown by C<-Dstv>, look like:
978
979 STACK 0: MAIN
980 CX 0: BLOCK =>
981 CX 1: EVAL => AV() PV("A"\0)
982 retop=leave
983 STACK 1: MAGIC
984 CX 0: SUB =>
985 retop=(null)
986 CX 1: EVAL => *
987 retop=nextstate
988
989The die pops the first C<CxEVAL> off the context stack, sets
990C<PL_restartop> from it, does a C<JMPENV_JUMP(3)>, and control returns to
991the top C<docatch>. This then starts another third-level runops level,
992which executes the nextstate, pushmark and die ops on line 4. At the point
993that the second C<pp_die> is called, the C call stack looks exactly like
994that above, even though we are no longer within an inner eval; this is
995because of the optimization mentioned earlier. However, the context stack
996now looks like this, ie with the top CxEVAL popped:
997
998 STACK 0: MAIN
999 CX 0: BLOCK =>
1000 CX 1: EVAL => AV() PV("A"\0)
1001 retop=leave
1002 STACK 1: MAGIC
1003 CX 0: SUB =>
1004 retop=(null)
1005
1006The die on line 4 pops the context stack back down to the CxEVAL, leaving
1007it as:
1008
1009 STACK 0: MAIN
1010 CX 0: BLOCK =>
1011
1012As usual, C<PL_restartop> is extracted from the C<CxEVAL>, and a
1013C<JMPENV_JUMP(3)> done, which pops the C stack back to the docatch:
1014
1015 S_docatch
1016 Perl_pp_entertry
1017 Perl_runops # second loop
1018 S_call_body
1019 Perl_call_sv
1020 Perl_pp_tie
1021 Perl_runops # first loop
1022 S_run_body
1023 perl_run
1024 main
1025
1026In this case, because the C<JMPENV> level recorded in the C<CxEVAL>
1027differs from the current one, C<docatch> just does a C<JMPENV_JUMP(3)>
1028and the C stack unwinds to:
1029
1030 perl_run
1031 main
1032
1033Because C<PL_restartop> is non-null, C<run_body> starts a new runops loop
1034and execution continues.
1035
a422fd2d
SC
1036=back
1037
1038=head2 Internal Variable Types
1039
1040You should by now have had a look at L<perlguts>, which tells you about
1041Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do
1042that now.
1043
1044These variables are used not only to represent Perl-space variables, but
1045also any constants in the code, as well as some structures completely
1046internal to Perl. The symbol table, for instance, is an ordinary Perl
1047hash. Your code is represented by an SV as it's read into the parser;
1048any program files you call are opened via ordinary Perl filehandles, and
1049so on.
1050
1051The core L<Devel::Peek|Devel::Peek> module lets us examine SVs from a
1052Perl program. Let's see, for instance, how Perl treats the constant
1053C<"hello">.
1054
1055 % perl -MDevel::Peek -e 'Dump("hello")'
1056 1 SV = PV(0xa041450) at 0xa04ecbc
1057 2 REFCNT = 1
1058 3 FLAGS = (POK,READONLY,pPOK)
1059 4 PV = 0xa0484e0 "hello"\0
1060 5 CUR = 5
1061 6 LEN = 6
1062
1063Reading C<Devel::Peek> output takes a bit of practise, so let's go
1064through it line by line.
1065
1066Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in
1067memory. SVs themselves are very simple structures, but they contain a
1068pointer to a more complex structure. In this case, it's a PV, a
1069structure which holds a string value, at location C<0xa041450>. Line 2
1070is the reference count; there are no other references to this data, so
1071it's 1.
1072
1073Line 3 are the flags for this SV - it's OK to use it as a PV, it's a
1074read-only SV (because it's a constant) and the data is a PV internally.
1075Next we've got the contents of the string, starting at location
1076C<0xa0484e0>.
1077
1078Line 5 gives us the current length of the string - note that this does
1079B<not> include the null terminator. Line 6 is not the length of the
1080string, but the length of the currently allocated buffer; as the string
1081grows, Perl automatically extends the available storage via a routine
1082called C<SvGROW>.
1083
1084You can get at any of these quantities from C very easily; just add
1085C<Sv> to the name of the field shown in the snippet, and you've got a
1086macro which will return the value: C<SvCUR(sv)> returns the current
1087length of the string, C<SvREFCOUNT(sv)> returns the reference count,
1088C<SvPV(sv, len)> returns the string itself with its length, and so on.
1089More macros to manipulate these properties can be found in L<perlguts>.
1090
1091Let's take an example of manipulating a PV, from C<sv_catpvn>, in F<sv.c>
1092
1093 1 void
1094 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len)
1095 3 {
1096 4 STRLEN tlen;
1097 5 char *junk;
1098
1099 6 junk = SvPV_force(sv, tlen);
1100 7 SvGROW(sv, tlen + len + 1);
1101 8 if (ptr == junk)
1102 9 ptr = SvPVX(sv);
1103 10 Move(ptr,SvPVX(sv)+tlen,len,char);
1104 11 SvCUR(sv) += len;
1105 12 *SvEND(sv) = '\0';
1106 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */
1107 14 SvTAINT(sv);
1108 15 }
1109
1110This is a function which adds a string, C<ptr>, of length C<len> onto
1111the end of the PV stored in C<sv>. The first thing we do in line 6 is
1112make sure that the SV B<has> a valid PV, by calling the C<SvPV_force>
1113macro to force a PV. As a side effect, C<tlen> gets set to the current
1114value of the PV, and the PV itself is returned to C<junk>.
1115
b1866b2d 1116In line 7, we make sure that the SV will have enough room to accommodate
a422fd2d
SC
1117the old string, the new string and the null terminator. If C<LEN> isn't
1118big enough, C<SvGROW> will reallocate space for us.
1119
1120Now, if C<junk> is the same as the string we're trying to add, we can
1121grab the string directly from the SV; C<SvPVX> is the address of the PV
1122in the SV.
1123
1124Line 10 does the actual catenation: the C<Move> macro moves a chunk of
1125memory around: we move the string C<ptr> to the end of the PV - that's
1126the start of the PV plus its current length. We're moving C<len> bytes
1127of type C<char>. After doing so, we need to tell Perl we've extended the
1128string, by altering C<CUR> to reflect the new length. C<SvEND> is a
1129macro which gives us the end of the string, so that needs to be a
1130C<"\0">.
1131
1132Line 13 manipulates the flags; since we've changed the PV, any IV or NV
1133values will no longer be valid: if we have C<$a=10; $a.="6";> we don't
1e54db1a 1134want to use the old IV of 10. C<SvPOK_only_utf8> is a special UTF-8-aware
a422fd2d
SC
1135version of C<SvPOK_only>, a macro which turns off the IOK and NOK flags
1136and turns on POK. The final C<SvTAINT> is a macro which launders tainted
1137data if taint mode is turned on.
1138
1139AVs and HVs are more complicated, but SVs are by far the most common
1140variable type being thrown around. Having seen something of how we
1141manipulate these, let's go on and look at how the op tree is
1142constructed.
1143
1144=head2 Op Trees
1145
1146First, what is the op tree, anyway? The op tree is the parsed
1147representation of your program, as we saw in our section on parsing, and
1148it's the sequence of operations that Perl goes through to execute your
1149program, as we saw in L</Running>.
1150
1151An op is a fundamental operation that Perl can perform: all the built-in
1152functions and operators are ops, and there are a series of ops which
1153deal with concepts the interpreter needs internally - entering and
1154leaving a block, ending a statement, fetching a variable, and so on.
1155
1156The op tree is connected in two ways: you can imagine that there are two
1157"routes" through it, two orders in which you can traverse the tree.
1158First, parse order reflects how the parser understood the code, and
1159secondly, execution order tells perl what order to perform the
1160operations in.
1161
1162The easiest way to examine the op tree is to stop Perl after it has
1163finished parsing, and get it to dump out the tree. This is exactly what
7d7d5695
RGS
1164the compiler backends L<B::Terse|B::Terse>, L<B::Concise|B::Concise>
1165and L<B::Debug|B::Debug> do.
a422fd2d
SC
1166
1167Let's have a look at how Perl sees C<$a = $b + $c>:
1168
1169 % perl -MO=Terse -e '$a=$b+$c'
1170 1 LISTOP (0x8179888) leave
1171 2 OP (0x81798b0) enter
1172 3 COP (0x8179850) nextstate
1173 4 BINOP (0x8179828) sassign
1174 5 BINOP (0x8179800) add [1]
1175 6 UNOP (0x81796e0) null [15]
1176 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b
1177 8 UNOP (0x81797e0) null [15]
1178 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c
1179 10 UNOP (0x816b4f0) null [15]
1180 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a
1181
1182Let's start in the middle, at line 4. This is a BINOP, a binary
1183operator, which is at location C<0x8179828>. The specific operator in
1184question is C<sassign> - scalar assignment - and you can find the code
1185which implements it in the function C<pp_sassign> in F<pp_hot.c>. As a
1186binary operator, it has two children: the add operator, providing the
1187result of C<$b+$c>, is uppermost on line 5, and the left hand side is on
1188line 10.
1189
1190Line 10 is the null op: this does exactly nothing. What is that doing
1191there? If you see the null op, it's a sign that something has been
1192optimized away after parsing. As we mentioned in L</Optimization>,
1193the optimization stage sometimes converts two operations into one, for
1194example when fetching a scalar variable. When this happens, instead of
1195rewriting the op tree and cleaning up the dangling pointers, it's easier
1196just to replace the redundant operation with the null op. Originally,
1197the tree would have looked like this:
1198
1199 10 SVOP (0x816b4f0) rv2sv [15]
1200 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a
1201
1202That is, fetch the C<a> entry from the main symbol table, and then look
1203at the scalar component of it: C<gvsv> (C<pp_gvsv> into F<pp_hot.c>)
1204happens to do both these things.
1205
1206The right hand side, starting at line 5 is similar to what we've just
1207seen: we have the C<add> op (C<pp_add> also in F<pp_hot.c>) add together
1208two C<gvsv>s.
1209
1210Now, what's this about?
1211
1212 1 LISTOP (0x8179888) leave
1213 2 OP (0x81798b0) enter
1214 3 COP (0x8179850) nextstate
1215
1216C<enter> and C<leave> are scoping ops, and their job is to perform any
1217housekeeping every time you enter and leave a block: lexical variables
1218are tidied up, unreferenced variables are destroyed, and so on. Every
1219program will have those first three lines: C<leave> is a list, and its
1220children are all the statements in the block. Statements are delimited
1221by C<nextstate>, so a block is a collection of C<nextstate> ops, with
1222the ops to be performed for each statement being the children of
1223C<nextstate>. C<enter> is a single op which functions as a marker.
1224
1225That's how Perl parsed the program, from top to bottom:
1226
1227 Program
1228 |
1229 Statement
1230 |
1231 =
1232 / \
1233 / \
1234 $a +
1235 / \
1236 $b $c
1237
1238However, it's impossible to B<perform> the operations in this order:
1239you have to find the values of C<$b> and C<$c> before you add them
1240together, for instance. So, the other thread that runs through the op
1241tree is the execution order: each op has a field C<op_next> which points
1242to the next op to be run, so following these pointers tells us how perl
1243executes the code. We can traverse the tree in this order using
1244the C<exec> option to C<B::Terse>:
1245
1246 % perl -MO=Terse,exec -e '$a=$b+$c'
1247 1 OP (0x8179928) enter
1248 2 COP (0x81798c8) nextstate
1249 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b
1250 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c
1251 5 BINOP (0x8179878) add [1]
1252 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a
1253 7 BINOP (0x81798a0) sassign
1254 8 LISTOP (0x8179900) leave
1255
1256This probably makes more sense for a human: enter a block, start a
1257statement. Get the values of C<$b> and C<$c>, and add them together.
1258Find C<$a>, and assign one to the other. Then leave.
1259
1260The way Perl builds up these op trees in the parsing process can be
1261unravelled by examining F<perly.y>, the YACC grammar. Let's take the
1262piece we need to construct the tree for C<$a = $b + $c>
1263
1264 1 term : term ASSIGNOP term
1265 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); }
1266 3 | term ADDOP term
1267 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
1268
1269If you're not used to reading BNF grammars, this is how it works: You're
1270fed certain things by the tokeniser, which generally end up in upper
1271case. Here, C<ADDOP>, is provided when the tokeniser sees C<+> in your
1272code. C<ASSIGNOP> is provided when C<=> is used for assigning. These are
b432a672 1273"terminal symbols", because you can't get any simpler than them.
a422fd2d
SC
1274
1275The grammar, lines one and three of the snippet above, tells you how to
b432a672 1276build up more complex forms. These complex forms, "non-terminal symbols"
a422fd2d
SC
1277are generally placed in lower case. C<term> here is a non-terminal
1278symbol, representing a single expression.
1279
1280The grammar gives you the following rule: you can make the thing on the
1281left of the colon if you see all the things on the right in sequence.
1282This is called a "reduction", and the aim of parsing is to completely
1283reduce the input. There are several different ways you can perform a
1284reduction, separated by vertical bars: so, C<term> followed by C<=>
1285followed by C<term> makes a C<term>, and C<term> followed by C<+>
1286followed by C<term> can also make a C<term>.
1287
1288So, if you see two terms with an C<=> or C<+>, between them, you can
1289turn them into a single expression. When you do this, you execute the
1290code in the block on the next line: if you see C<=>, you'll do the code
1291in line 2. If you see C<+>, you'll do the code in line 4. It's this code
1292which contributes to the op tree.
1293
1294 | term ADDOP term
1295 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
1296
1297What this does is creates a new binary op, and feeds it a number of
1298variables. The variables refer to the tokens: C<$1> is the first token in
1299the input, C<$2> the second, and so on - think regular expression
1300backreferences. C<$$> is the op returned from this reduction. So, we
1301call C<newBINOP> to create a new binary operator. The first parameter to
1302C<newBINOP>, a function in F<op.c>, is the op type. It's an addition
1303operator, so we want the type to be C<ADDOP>. We could specify this
1304directly, but it's right there as the second token in the input, so we
b432a672
AL
1305use C<$2>. The second parameter is the op's flags: 0 means "nothing
1306special". Then the things to add: the left and right hand side of our
a422fd2d
SC
1307expression, in scalar context.
1308
1309=head2 Stacks
1310
1311When perl executes something like C<addop>, how does it pass on its
1312results to the next op? The answer is, through the use of stacks. Perl
1313has a number of stacks to store things it's currently working on, and
1314we'll look at the three most important ones here.
1315
1316=over 3
1317
1318=item Argument stack
1319
1320Arguments are passed to PP code and returned from PP code using the
1321argument stack, C<ST>. The typical way to handle arguments is to pop
1322them off the stack, deal with them how you wish, and then push the result
1323back onto the stack. This is how, for instance, the cosine operator
1324works:
1325
1326 NV value;
1327 value = POPn;
1328 value = Perl_cos(value);
1329 XPUSHn(value);
1330
1331We'll see a more tricky example of this when we consider Perl's macros
1332below. C<POPn> gives you the NV (floating point value) of the top SV on
1333the stack: the C<$x> in C<cos($x)>. Then we compute the cosine, and push
1334the result back as an NV. The C<X> in C<XPUSHn> means that the stack
1335should be extended if necessary - it can't be necessary here, because we
1336know there's room for one more item on the stack, since we've just
1337removed one! The C<XPUSH*> macros at least guarantee safety.
1338
1339Alternatively, you can fiddle with the stack directly: C<SP> gives you
1340the first element in your portion of the stack, and C<TOP*> gives you
1341the top SV/IV/NV/etc. on the stack. So, for instance, to do unary
1342negation of an integer:
1343
1344 SETi(-TOPi);
1345
1346Just set the integer value of the top stack entry to its negation.
1347
1348Argument stack manipulation in the core is exactly the same as it is in
1349XSUBs - see L<perlxstut>, L<perlxs> and L<perlguts> for a longer
1350description of the macros used in stack manipulation.
1351
1352=item Mark stack
1353
b432a672 1354I say "your portion of the stack" above because PP code doesn't
a422fd2d
SC
1355necessarily get the whole stack to itself: if your function calls
1356another function, you'll only want to expose the arguments aimed for the
1357called function, and not (necessarily) let it get at your own data. The
b432a672 1358way we do this is to have a "virtual" bottom-of-stack, exposed to each
a422fd2d
SC
1359function. The mark stack keeps bookmarks to locations in the argument
1360stack usable by each function. For instance, when dealing with a tied
b432a672 1361variable, (internally, something with "P" magic) Perl has to call
a422fd2d
SC
1362methods for accesses to the tied variables. However, we need to separate
1363the arguments exposed to the method to the argument exposed to the
ed233832
DM
1364original function - the store or fetch or whatever it may be. Here's
1365roughly how the tied C<push> is implemented; see C<av_push> in F<av.c>:
a422fd2d
SC
1366
1367 1 PUSHMARK(SP);
1368 2 EXTEND(SP,2);
1369 3 PUSHs(SvTIED_obj((SV*)av, mg));
1370 4 PUSHs(val);
1371 5 PUTBACK;
1372 6 ENTER;
1373 7 call_method("PUSH", G_SCALAR|G_DISCARD);
1374 8 LEAVE;
13a2d996 1375
a422fd2d
SC
1376Let's examine the whole implementation, for practice:
1377
1378 1 PUSHMARK(SP);
1379
1380Push the current state of the stack pointer onto the mark stack. This is
1381so that when we've finished adding items to the argument stack, Perl
1382knows how many things we've added recently.
1383
1384 2 EXTEND(SP,2);
1385 3 PUSHs(SvTIED_obj((SV*)av, mg));
1386 4 PUSHs(val);
1387
1388We're going to add two more items onto the argument stack: when you have
1389a tied array, the C<PUSH> subroutine receives the object and the value
1390to be pushed, and that's exactly what we have here - the tied object,
1391retrieved with C<SvTIED_obj>, and the value, the SV C<val>.
1392
1393 5 PUTBACK;
1394
e89a6d4e
JD
1395Next we tell Perl to update the global stack pointer from our internal
1396variable: C<dSP> only gave us a local copy, not a reference to the global.
a422fd2d
SC
1397
1398 6 ENTER;
1399 7 call_method("PUSH", G_SCALAR|G_DISCARD);
1400 8 LEAVE;
1401
1402C<ENTER> and C<LEAVE> localise a block of code - they make sure that all
1403variables are tidied up, everything that has been localised gets
1404its previous value returned, and so on. Think of them as the C<{> and
1405C<}> of a Perl block.
1406
1407To actually do the magic method call, we have to call a subroutine in
1408Perl space: C<call_method> takes care of that, and it's described in
1409L<perlcall>. We call the C<PUSH> method in scalar context, and we're
e89a6d4e
JD
1410going to discard its return value. The call_method() function
1411removes the top element of the mark stack, so there is nothing for
1412the caller to clean up.
a422fd2d 1413
a422fd2d
SC
1414=item Save stack
1415
1416C doesn't have a concept of local scope, so perl provides one. We've
1417seen that C<ENTER> and C<LEAVE> are used as scoping braces; the save
1418stack implements the C equivalent of, for example:
1419
1420 {
1421 local $foo = 42;
1422 ...
1423 }
1424
1425See L<perlguts/Localising Changes> for how to use the save stack.
1426
1427=back
1428
1429=head2 Millions of Macros
1430
1431One thing you'll notice about the Perl source is that it's full of
1432macros. Some have called the pervasive use of macros the hardest thing
1433to understand, others find it adds to clarity. Let's take an example,
1434the code which implements the addition operator:
1435
1436 1 PP(pp_add)
1437 2 {
39644a26 1438 3 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
a422fd2d
SC
1439 4 {
1440 5 dPOPTOPnnrl_ul;
1441 6 SETn( left + right );
1442 7 RETURN;
1443 8 }
1444 9 }
1445
1446Every line here (apart from the braces, of course) contains a macro. The
1447first line sets up the function declaration as Perl expects for PP code;
1448line 3 sets up variable declarations for the argument stack and the
1449target, the return value of the operation. Finally, it tries to see if
1450the addition operation is overloaded; if so, the appropriate subroutine
1451is called.
1452
1453Line 5 is another variable declaration - all variable declarations start
1454with C<d> - which pops from the top of the argument stack two NVs (hence
1455C<nn>) and puts them into the variables C<right> and C<left>, hence the
1456C<rl>. These are the two operands to the addition operator. Next, we
1457call C<SETn> to set the NV of the return value to the result of adding
1458the two values. This done, we return - the C<RETURN> macro makes sure
1459that our return value is properly handled, and we pass the next operator
1460to run back to the main run loop.
1461
1462Most of these macros are explained in L<perlapi>, and some of the more
1463important ones are explained in L<perlxs> as well. Pay special attention
1464to L<perlguts/Background and PERL_IMPLICIT_CONTEXT> for information on
1465the C<[pad]THX_?> macros.
1466
52d59bef
JH
1467=head2 The .i Targets
1468
1469You can expand the macros in a F<foo.c> file by saying
1470
1471 make foo.i
1472
1473which will expand the macros using cpp. Don't be scared by the results.
1474
955fec6b
JH
1475=head1 SOURCE CODE STATIC ANALYSIS
1476
1477Various tools exist for analysing C source code B<statically>, as
1478opposed to B<dynamically>, that is, without executing the code.
1479It is possible to detect resource leaks, undefined behaviour, type
1480mismatches, portability problems, code paths that would cause illegal
1481memory accesses, and other similar problems by just parsing the C code
1482and looking at the resulting graph, what does it tell about the
1483execution and data flows. As a matter of fact, this is exactly
1484how C compilers know to give warnings about dubious code.
1485
1486=head2 lint, splint
1487
1488The good old C code quality inspector, C<lint>, is available in
1489several platforms, but please be aware that there are several
1490different implementations of it by different vendors, which means that
1491the flags are not identical across different platforms.
1492
1493There is a lint variant called C<splint> (Secure Programming Lint)
1494available from http://www.splint.org/ that should compile on any
1495Unix-like platform.
1496
1497There are C<lint> and <splint> targets in Makefile, but you may have
1498to diddle with the flags (see above).
1499
1500=head2 Coverity
1501
1502Coverity (http://www.coverity.com/) is a product similar to lint and
1503as a testbed for their product they periodically check several open
1504source projects, and they give out accounts to open source developers
1505to the defect databases.
1506
1507=head2 cpd (cut-and-paste detector)
1508
1509The cpd tool detects cut-and-paste coding. If one instance of the
1510cut-and-pasted code changes, all the other spots should probably be
1511changed, too. Therefore such code should probably be turned into a
1512subroutine or a macro.
1513
1514cpd (http://pmd.sourceforge.net/cpd.html) is part of the pmd project
1515(http://pmd.sourceforge.net/). pmd was originally written for static
1516analysis of Java code, but later the cpd part of it was extended to
1517parse also C and C++.
1518
a52aaefa
RGS
1519Download the pmd-bin-X.Y.zip () from the SourceForge site, extract the
1520pmd-X.Y.jar from it, and then run that on source code thusly:
955fec6b
JH
1521
1522 java -cp pmd-X.Y.jar net.sourceforge.pmd.cpd.CPD --minimum-tokens 100 --files /some/where/src --language c > cpd.txt
1523
1524You may run into memory limits, in which case you should use the -Xmx option:
1525
1526 java -Xmx512M ...
1527
1528=head2 gcc warnings
1529
1530Though much can be written about the inconsistency and coverage
1531problems of gcc warnings (like C<-Wall> not meaning "all the
1532warnings", or some common portability problems not being covered by
1533C<-Wall>, or C<-ansi> and C<-pedantic> both being a poorly defined
1534collection of warnings, and so forth), gcc is still a useful tool in
1535keeping our coding nose clean.
1536
1537The C<-Wall> is by default on.
1538
a8e98a71
JH
1539The C<-ansi> (and its sidekick, C<-pedantic>) would be nice to be on
1540always, but unfortunately they are not safe on all platforms, they can
1541for example cause fatal conflicts with the system headers (Solaris
1542being a prime example). If Configure C<-Dgccansipedantic> is used,
1543the C<cflags> frontend selects C<-ansi -pedantic> for the platforms
1544where they are known to be safe.
955fec6b
JH
1545
1546Starting from Perl 5.9.4 the following extra flags are added:
1547
1548=over 4
1549
1550=item *
1551
1552C<-Wendif-labels>
1553
1554=item *
1555
1556C<-Wextra>
1557
1558=item *
1559
1560C<-Wdeclaration-after-statement>
1561
1562=back
1563
1564The following flags would be nice to have but they would first need
0503309d 1565their own Augean stablemaster:
955fec6b
JH
1566
1567=over 4
1568
1569=item *
1570
1571C<-Wpointer-arith>
1572
1573=item *
1574
1575C<-Wshadow>
1576
1577=item *
1578
1579C<-Wstrict-prototypes>
1580
955fec6b
JH
1581=back
1582
1583The C<-Wtraditional> is another example of the annoying tendency of
1584gcc to bundle a lot of warnings under one switch -- it would be
1585impossible to deploy in practice because it would complain a lot -- but
1586it does contain some warnings that would be beneficial to have available
1587on their own, such as the warning about string constants inside macros
1588containing the macro arguments: this behaved differently pre-ANSI
1589than it does in ANSI, and some C compilers are still in transition,
1590AIX being an example.
1591
1592=head2 Warnings of other C compilers
1593
1594Other C compilers (yes, there B<are> other C compilers than gcc) often
1595have their "strict ANSI" or "strict ANSI with some portability extensions"
1596modes on, like for example the Sun Workshop has its C<-Xa> mode on
1597(though implicitly), or the DEC (these days, HP...) has its C<-std1>
1598mode on.
1599
1600=head2 DEBUGGING
1601
1602You can compile a special debugging version of Perl, which allows you
1603to use the C<-D> option of Perl to tell more about what Perl is doing.
1604But sometimes there is no alternative than to dive in with a debugger,
1605either to see the stack trace of a core dump (very useful in a bug
1606report), or trying to figure out what went wrong before the core dump
1607happened, or how did we end up having wrong or unexpected results.
1608
a422fd2d
SC
1609=head2 Poking at Perl
1610
1611To really poke around with Perl, you'll probably want to build Perl for
1612debugging, like this:
1613
1614 ./Configure -d -D optimize=-g
1615 make
1616
1617C<-g> is a flag to the C compiler to have it produce debugging
955fec6b
JH
1618information which will allow us to step through a running program,
1619and to see in which C function we are at (without the debugging
1620information we might see only the numerical addresses of the functions,
1621which is not very helpful).
1622
a422fd2d
SC
1623F<Configure> will also turn on the C<DEBUGGING> compilation symbol which
1624enables all the internal debugging code in Perl. There are a whole bunch
1625of things you can debug with this: L<perlrun> lists them all, and the
1626best way to find out about them is to play about with them. The most
1627useful options are probably
1628
1629 l Context (loop) stack processing
1630 t Trace execution
1631 o Method and overloading resolution
1632 c String/numeric conversions
1633
1634Some of the functionality of the debugging code can be achieved using XS
1635modules.
13a2d996 1636
a422fd2d
SC
1637 -Dr => use re 'debug'
1638 -Dx => use O 'Debug'
1639
1640=head2 Using a source-level debugger
1641
1642If the debugging output of C<-D> doesn't help you, it's time to step
1643through perl's execution with a source-level debugger.
1644
1645=over 3
1646
1647=item *
1648
955fec6b
JH
1649We'll use C<gdb> for our examples here; the principles will apply to
1650any debugger (many vendors call their debugger C<dbx>), but check the
1651manual of the one you're using.
a422fd2d
SC
1652
1653=back
1654
1655To fire up the debugger, type
1656
1657 gdb ./perl
1658
955fec6b
JH
1659Or if you have a core dump:
1660
1661 gdb ./perl core
1662
a422fd2d
SC
1663You'll want to do that in your Perl source tree so the debugger can read
1664the source code. You should see the copyright message, followed by the
1665prompt.
1666
1667 (gdb)
1668
1669C<help> will get you into the documentation, but here are the most
1670useful commands:
1671
1672=over 3
1673
1674=item run [args]
1675
1676Run the program with the given arguments.
1677
1678=item break function_name
1679
1680=item break source.c:xxx
1681
1682Tells the debugger that we'll want to pause execution when we reach
cea6626f 1683either the named function (but see L<perlguts/Internal Functions>!) or the given
a422fd2d
SC
1684line in the named source file.
1685
1686=item step
1687
1688Steps through the program a line at a time.
1689
1690=item next
1691
1692Steps through the program a line at a time, without descending into
1693functions.
1694
1695=item continue
1696
1697Run until the next breakpoint.
1698
1699=item finish
1700
1701Run until the end of the current function, then stop again.
1702
13a2d996 1703=item 'enter'
a422fd2d
SC
1704
1705Just pressing Enter will do the most recent operation again - it's a
1706blessing when stepping through miles of source code.
1707
1708=item print
1709
1710Execute the given C code and print its results. B<WARNING>: Perl makes
52d59bef
JH
1711heavy use of macros, and F<gdb> does not necessarily support macros
1712(see later L</"gdb macro support">). You'll have to substitute them
1713yourself, or to invoke cpp on the source code files
1714(see L</"The .i Targets">)
1715So, for instance, you can't say
a422fd2d
SC
1716
1717 print SvPV_nolen(sv)
1718
1719but you have to say
1720
1721 print Perl_sv_2pv_nolen(sv)
1722
ffc145e8
RK
1723=back
1724
a422fd2d
SC
1725You may find it helpful to have a "macro dictionary", which you can
1726produce by saying C<cpp -dM perl.c | sort>. Even then, F<cpp> won't
07aa3531 1727recursively apply those macros for you.
52d59bef
JH
1728
1729=head2 gdb macro support
a422fd2d 1730
52d59bef 1731Recent versions of F<gdb> have fairly good macro support, but
ea031e66
RGS
1732in order to use it you'll need to compile perl with macro definitions
1733included in the debugging information. Using F<gcc> version 3.1, this
1734means configuring with C<-Doptimize=-g3>. Other compilers might use a
1735different switch (if they support debugging macros at all).
1736
a422fd2d
SC
1737=head2 Dumping Perl Data Structures
1738
1739One way to get around this macro hell is to use the dumping functions in
1740F<dump.c>; these work a little like an internal
1741L<Devel::Peek|Devel::Peek>, but they also cover OPs and other structures
1742that you can't get at from Perl. Let's take an example. We'll use the
07aa3531 1743C<$a = $b + $c> we used before, but give it a bit of context:
a422fd2d
SC
1744C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around?
1745
1746What about C<pp_add>, the function we examined earlier to implement the
1747C<+> operator:
1748
1749 (gdb) break Perl_pp_add
1750 Breakpoint 1 at 0x46249f: file pp_hot.c, line 309.
1751
cea6626f 1752Notice we use C<Perl_pp_add> and not C<pp_add> - see L<perlguts/Internal Functions>.
a422fd2d
SC
1753With the breakpoint in place, we can run our program:
1754
1755 (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c'
1756
1757Lots of junk will go past as gdb reads in the relevant source files and
1758libraries, and then:
1759
1760 Breakpoint 1, Perl_pp_add () at pp_hot.c:309
39644a26 1761 309 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
a422fd2d
SC
1762 (gdb) step
1763 311 dPOPTOPnnrl_ul;
1764 (gdb)
1765
1766We looked at this bit of code before, and we said that C<dPOPTOPnnrl_ul>
1767arranges for two C<NV>s to be placed into C<left> and C<right> - let's
1768slightly expand it:
1769
1770 #define dPOPTOPnnrl_ul NV right = POPn; \
1771 SV *leftsv = TOPs; \
1772 NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0
1773
1774C<POPn> takes the SV from the top of the stack and obtains its NV either
1775directly (if C<SvNOK> is set) or by calling the C<sv_2nv> function.
1776C<TOPs> takes the next SV from the top of the stack - yes, C<POPn> uses
1777C<TOPs> - but doesn't remove it. We then use C<SvNV> to get the NV from
07aa3531 1778C<leftsv> in the same way as before - yes, C<POPn> uses C<SvNV>.
a422fd2d
SC
1779
1780Since we don't have an NV for C<$b>, we'll have to use C<sv_2nv> to
1781convert it. If we step again, we'll find ourselves there:
1782
1783 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669
1784 1669 if (!sv)
1785 (gdb)
1786
1787We can now use C<Perl_sv_dump> to investigate the SV:
1788
1789 SV = PV(0xa057cc0) at 0xa0675d0
1790 REFCNT = 1
1791 FLAGS = (POK,pPOK)
1792 PV = 0xa06a510 "6XXXX"\0
1793 CUR = 5
1794 LEN = 6
1795 $1 = void
1796
1797We know we're going to get C<6> from this, so let's finish the
1798subroutine:
1799
1800 (gdb) finish
1801 Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671
1802 0x462669 in Perl_pp_add () at pp_hot.c:311
1803 311 dPOPTOPnnrl_ul;
1804
1805We can also dump out this op: the current op is always stored in
1806C<PL_op>, and we can dump it with C<Perl_op_dump>. This'll give us
1807similar output to L<B::Debug|B::Debug>.
1808
1809 {
1810 13 TYPE = add ===> 14
1811 TARG = 1
1812 FLAGS = (SCALAR,KIDS)
1813 {
1814 TYPE = null ===> (12)
1815 (was rv2sv)
1816 FLAGS = (SCALAR,KIDS)
1817 {
1818 11 TYPE = gvsv ===> 12
1819 FLAGS = (SCALAR)
1820 GV = main::b
1821 }
1822 }
1823
10f58044 1824# finish this later #
a422fd2d
SC
1825
1826=head2 Patching
1827
1828All right, we've now had a look at how to navigate the Perl sources and
1829some things you'll need to know when fiddling with them. Let's now get
1830on and create a simple patch. Here's something Larry suggested: if a
07aa3531 1831C<U> is the first active format during a C<pack>, (for example,
a422fd2d 1832C<pack "U3C8", @stuff>) then the resulting string should be treated as
1e54db1a 1833UTF-8 encoded.
a422fd2d
SC
1834
1835How do we prepare to fix this up? First we locate the code in question -
1836the C<pack> happens at runtime, so it's going to be in one of the F<pp>
1837files. Sure enough, C<pp_pack> is in F<pp.c>. Since we're going to be
1838altering this file, let's copy it to F<pp.c~>.
1839
a6ec74c1
JH
1840[Well, it was in F<pp.c> when this tutorial was written. It has now been
1841split off with C<pp_unpack> to its own file, F<pp_pack.c>]
1842
a422fd2d
SC
1843Now let's look over C<pp_pack>: we take a pattern into C<pat>, and then
1844loop over the pattern, taking each format character in turn into
1845C<datum_type>. Then for each possible format character, we swallow up
1846the other arguments in the pattern (a field width, an asterisk, and so
1847on) and convert the next chunk input into the specified format, adding
1848it onto the output SV C<cat>.
1849
1850How do we know if the C<U> is the first format in the C<pat>? Well, if
1851we have a pointer to the start of C<pat> then, if we see a C<U> we can
1852test whether we're still at the start of the string. So, here's where
1853C<pat> is set up:
1854
1855 STRLEN fromlen;
1856 register char *pat = SvPVx(*++MARK, fromlen);
1857 register char *patend = pat + fromlen;
1858 register I32 len;
1859 I32 datumtype;
1860 SV *fromstr;
1861
1862We'll have another string pointer in there:
1863
1864 STRLEN fromlen;
1865 register char *pat = SvPVx(*++MARK, fromlen);
1866 register char *patend = pat + fromlen;
1867 + char *patcopy;
1868 register I32 len;
1869 I32 datumtype;
1870 SV *fromstr;
1871
1872And just before we start the loop, we'll set C<patcopy> to be the start
1873of C<pat>:
1874
1875 items = SP - MARK;
1876 MARK++;
1877 sv_setpvn(cat, "", 0);
1878 + patcopy = pat;
1879 while (pat < patend) {
1880
1881Now if we see a C<U> which was at the start of the string, we turn on
1e54db1a 1882the C<UTF8> flag for the output SV, C<cat>:
a422fd2d
SC
1883
1884 + if (datumtype == 'U' && pat==patcopy+1)
1885 + SvUTF8_on(cat);
1886 if (datumtype == '#') {
1887 while (pat < patend && *pat != '\n')
1888 pat++;
1889
1890Remember that it has to be C<patcopy+1> because the first character of
1891the string is the C<U> which has been swallowed into C<datumtype!>
1892
1893Oops, we forgot one thing: what if there are spaces at the start of the
1894pattern? C<pack(" U*", @stuff)> will have C<U> as the first active
1895character, even though it's not the first thing in the pattern. In this
1896case, we have to advance C<patcopy> along with C<pat> when we see spaces:
1897
1898 if (isSPACE(datumtype))
1899 continue;
1900
1901needs to become
1902
1903 if (isSPACE(datumtype)) {
1904 patcopy++;
1905 continue;
1906 }
1907
1908OK. That's the C part done. Now we must do two additional things before
1909this patch is ready to go: we've changed the behaviour of Perl, and so
1910we must document that change. We must also provide some more regression
1911tests to make sure our patch works and doesn't create a bug somewhere
1912else along the line.
1913
b23b8711
MS
1914The regression tests for each operator live in F<t/op/>, and so we
1915make a copy of F<t/op/pack.t> to F<t/op/pack.t~>. Now we can add our
1916tests to the end. First, we'll test that the C<U> does indeed create
07aa3531 1917Unicode strings.
b23b8711
MS
1918
1919t/op/pack.t has a sensible ok() function, but if it didn't we could
35c336e6 1920use the one from t/test.pl.
b23b8711 1921
35c336e6
MS
1922 require './test.pl';
1923 plan( tests => 159 );
b23b8711
MS
1924
1925so instead of this:
a422fd2d
SC
1926
1927 print 'not ' unless "1.20.300.4000" eq sprintf "%vd", pack("U*",1,20,300,4000);
1928 print "ok $test\n"; $test++;
1929
35c336e6
MS
1930we can write the more sensible (see L<Test::More> for a full
1931explanation of is() and other testing functions).
b23b8711 1932
07aa3531 1933 is( "1.20.300.4000", sprintf "%vd", pack("U*",1,20,300,4000),
38a44b82 1934 "U* produces Unicode" );
b23b8711 1935
a422fd2d
SC
1936Now we'll test that we got that space-at-the-beginning business right:
1937
35c336e6 1938 is( "1.20.300.4000", sprintf "%vd", pack(" U*",1,20,300,4000),
812f5127 1939 " with spaces at the beginning" );
a422fd2d
SC
1940
1941And finally we'll test that we don't make Unicode strings if C<U> is B<not>
1942the first active format:
1943
35c336e6 1944 isnt( v1.20.300.4000, sprintf "%vd", pack("C0U*",1,20,300,4000),
38a44b82 1945 "U* not first isn't Unicode" );
a422fd2d 1946
35c336e6
MS
1947Mustn't forget to change the number of tests which appears at the top,
1948or else the automated tester will get confused. This will either look
1949like this:
a422fd2d 1950
35c336e6
MS
1951 print "1..156\n";
1952
1953or this:
1954
1955 plan( tests => 156 );
a422fd2d
SC
1956
1957We now compile up Perl, and run it through the test suite. Our new
1958tests pass, hooray!
1959
1960Finally, the documentation. The job is never done until the paperwork is
1961over, so let's describe the change we've just made. The relevant place
1962is F<pod/perlfunc.pod>; again, we make a copy, and then we'll insert
1963this text in the description of C<pack>:
1964
1965 =item *
1966
1967 If the pattern begins with a C<U>, the resulting string will be treated
1e54db1a
JH
1968 as UTF-8-encoded Unicode. You can force UTF-8 encoding on in a string
1969 with an initial C<U0>, and the bytes that follow will be interpreted as
1970 Unicode characters. If you don't want this to happen, you can begin your
1971 pattern with C<C0> (or anything else) to force Perl not to UTF-8 encode your
a422fd2d
SC
1972 string, and then follow this with a C<U*> somewhere in your pattern.
1973
1974All done. Now let's create the patch. F<Porting/patching.pod> tells us
1975that if we're making major changes, we should copy the entire directory
1976to somewhere safe before we begin fiddling, and then do
13a2d996 1977
a422fd2d
SC
1978 diff -ruN old new > patch
1979
1980However, we know which files we've changed, and we can simply do this:
1981
1982 diff -u pp.c~ pp.c > patch
1983 diff -u t/op/pack.t~ t/op/pack.t >> patch
1984 diff -u pod/perlfunc.pod~ pod/perlfunc.pod >> patch
1985
1986We end up with a patch looking a little like this:
1987
1988 --- pp.c~ Fri Jun 02 04:34:10 2000
1989 +++ pp.c Fri Jun 16 11:37:25 2000
1990 @@ -4375,6 +4375,7 @@
1991 register I32 items;
1992 STRLEN fromlen;
1993 register char *pat = SvPVx(*++MARK, fromlen);
1994 + char *patcopy;
1995 register char *patend = pat + fromlen;
1996 register I32 len;
1997 I32 datumtype;
1998 @@ -4405,6 +4406,7 @@
1999 ...
2000
2001And finally, we submit it, with our rationale, to perl5-porters. Job
2002done!
2003
f7e1e956
MS
2004=head2 Patching a core module
2005
2006This works just like patching anything else, with an extra
2007consideration. Many core modules also live on CPAN. If this is so,
2008patch the CPAN version instead of the core and send the patch off to
2009the module maintainer (with a copy to p5p). This will help the module
2010maintainer keep the CPAN version in sync with the core version without
2011constantly scanning p5p.
2012
db300100
RGS
2013The list of maintainers of core modules is usefully documented in
2014F<Porting/Maintainers.pl>.
2015
acbe17fc
JP
2016=head2 Adding a new function to the core
2017
2018If, as part of a patch to fix a bug, or just because you have an
2019especially good idea, you decide to add a new function to the core,
2020discuss your ideas on p5p well before you start work. It may be that
2021someone else has already attempted to do what you are considering and
2022can give lots of good advice or even provide you with bits of code
2023that they already started (but never finished).
2024
2025You have to follow all of the advice given above for patching. It is
2026extremely important to test any addition thoroughly and add new tests
2027to explore all boundary conditions that your new function is expected
2028to handle. If your new function is used only by one module (e.g. toke),
2029then it should probably be named S_your_function (for static); on the
210b36aa 2030other hand, if you expect it to accessible from other functions in
acbe17fc
JP
2031Perl, you should name it Perl_your_function. See L<perlguts/Internal Functions>
2032for more details.
2033
2034The location of any new code is also an important consideration. Don't
2035just create a new top level .c file and put your code there; you would
2036have to make changes to Configure (so the Makefile is created properly),
2037as well as possibly lots of include files. This is strictly pumpking
2038business.
2039
2040It is better to add your function to one of the existing top level
2041source code files, but your choice is complicated by the nature of
2042the Perl distribution. Only the files that are marked as compiled
2043static are located in the perl executable. Everything else is located
2044in the shared library (or DLL if you are running under WIN32). So,
2045for example, if a function was only used by functions located in
2046toke.c, then your code can go in toke.c. If, however, you want to call
2047the function from universal.c, then you should put your code in another
2048location, for example util.c.
2049
2050In addition to writing your c-code, you will need to create an
2051appropriate entry in embed.pl describing your function, then run
2052'make regen_headers' to create the entries in the numerous header
2053files that perl needs to compile correctly. See L<perlguts/Internal Functions>
2054for information on the various options that you can set in embed.pl.
2055You will forget to do this a few (or many) times and you will get
2056warnings during the compilation phase. Make sure that you mention
2057this when you post your patch to P5P; the pumpking needs to know this.
2058
2059When you write your new code, please be conscious of existing code
884bad00 2060conventions used in the perl source files. See L<perlstyle> for
acbe17fc
JP
2061details. Although most of the guidelines discussed seem to focus on
2062Perl code, rather than c, they all apply (except when they don't ;).
2063See also I<Porting/patching.pod> file in the Perl source distribution
2064for lots of details about both formatting and submitting patches of
2065your changes.
2066
2067Lastly, TEST TEST TEST TEST TEST any code before posting to p5p.
2068Test on as many platforms as you can find. Test as many perl
2069Configure options as you can (e.g. MULTIPLICITY). If you have
2070profiling or memory tools, see L<EXTERNAL TOOLS FOR DEBUGGING PERL>
210b36aa 2071below for how to use them to further test your code. Remember that
acbe17fc
JP
2072most of the people on P5P are doing this on their own time and
2073don't have the time to debug your code.
f7e1e956
MS
2074
2075=head2 Writing a test
2076
2077Every module and built-in function has an associated test file (or
2078should...). If you add or change functionality, you have to write a
2079test. If you fix a bug, you have to write a test so that bug never
2080comes back. If you alter the docs, it would be nice to test what the
2081new documentation says.
2082
2083In short, if you submit a patch you probably also have to patch the
2084tests.
2085
2086For modules, the test file is right next to the module itself.
2087F<lib/strict.t> tests F<lib/strict.pm>. This is a recent innovation,
2088so there are some snags (and it would be wonderful for you to brush
2089them out), but it basically works that way. Everything else lives in
2090F<t/>.
2091
2092=over 3
2093
2094=item F<t/base/>
2095
2096Testing of the absolute basic functionality of Perl. Things like
2097C<if>, basic file reads and writes, simple regexes, etc. These are
2098run first in the test suite and if any of them fail, something is
2099I<really> broken.
2100
2101=item F<t/cmd/>
2102
2103These test the basic control structures, C<if/else>, C<while>,
35c336e6 2104subroutines, etc.
f7e1e956
MS
2105
2106=item F<t/comp/>
2107
2108Tests basic issues of how Perl parses and compiles itself.
2109
2110=item F<t/io/>
2111
2112Tests for built-in IO functions, including command line arguments.
2113
2114=item F<t/lib/>
2115
2116The old home for the module tests, you shouldn't put anything new in
2117here. There are still some bits and pieces hanging around in here
2118that need to be moved. Perhaps you could move them? Thanks!
2119
3c295041
RGS
2120=item F<t/mro/>
2121
0503309d 2122Tests for perl's method resolution order implementations
3c295041
RGS
2123(see L<mro>).
2124
f7e1e956
MS
2125=item F<t/op/>
2126
2127Tests for perl's built in functions that don't fit into any of the
2128other directories.
2129
2130=item F<t/pod/>
2131
2132Tests for POD directives. There are still some tests for the Pod
2133modules hanging around in here that need to be moved out into F<lib/>.
2134
2135=item F<t/run/>
2136
2137Testing features of how perl actually runs, including exit codes and
2138handling of PERL* environment variables.
2139
244d9cb7
RGS
2140=item F<t/uni/>
2141
2142Tests for the core support of Unicode.
2143
2144=item F<t/win32/>
2145
2146Windows-specific tests.
2147
2148=item F<t/x2p>
2149
2150A test suite for the s2p converter.
2151
f7e1e956
MS
2152=back
2153
2154The core uses the same testing style as the rest of Perl, a simple
2155"ok/not ok" run through Test::Harness, but there are a few special
2156considerations.
2157
35c336e6
MS
2158There are three ways to write a test in the core. Test::More,
2159t/test.pl and ad hoc C<print $test ? "ok 42\n" : "not ok 42\n">. The
2160decision of which to use depends on what part of the test suite you're
2161working on. This is a measure to prevent a high-level failure (such
2162as Config.pm breaking) from causing basic functionality tests to fail.
2163
07aa3531 2164=over 4
35c336e6
MS
2165
2166=item t/base t/comp
2167
2168Since we don't know if require works, or even subroutines, use ad hoc
2169tests for these two. Step carefully to avoid using the feature being
2170tested.
2171
2172=item t/cmd t/run t/io t/op
2173
2174Now that basic require() and subroutines are tested, you can use the
2175t/test.pl library which emulates the important features of Test::More
2176while using a minimum of core features.
2177
2178You can also conditionally use certain libraries like Config, but be
2179sure to skip the test gracefully if it's not there.
2180
2181=item t/lib ext lib
2182
2183Now that the core of Perl is tested, Test::More can be used. You can
2184also use the full suite of core modules in the tests.
2185
2186=back
f7e1e956
MS
2187
2188When you say "make test" Perl uses the F<t/TEST> program to run the
07aa3531
JC
2189test suite (except under Win32 where it uses F<t/harness> instead.)
2190All tests are run from the F<t/> directory, B<not> the directory
2191which contains the test. This causes some problems with the tests
7205a85d 2192in F<lib/>, so here's some opportunity for some patching.
f7e1e956
MS
2193
2194You must be triply conscious of cross-platform concerns. This usually
2195boils down to using File::Spec and avoiding things like C<fork()> and
2196C<system()> unless absolutely necessary.
2197
e018f8be
JH
2198=head2 Special Make Test Targets
2199
2200There are various special make targets that can be used to test Perl
2201slightly differently than the standard "test" target. Not all them
2202are expected to give a 100% success rate. Many of them have several
7205a85d
YO
2203aliases, and many of them are not available on certain operating
2204systems.
e018f8be
JH
2205
2206=over 4
2207
2208=item coretest
2209
7d7d5695 2210Run F<perl> on all core tests (F<t/*> and F<lib/[a-z]*> pragma tests).
e018f8be 2211
7205a85d
YO
2212(Not available on Win32)
2213
e018f8be
JH
2214=item test.deparse
2215
b26492ee
RGS
2216Run all the tests through B::Deparse. Not all tests will succeed.
2217
7205a85d
YO
2218(Not available on Win32)
2219
b26492ee
RGS
2220=item test.taintwarn
2221
2222Run all tests with the B<-t> command-line switch. Not all tests
2223are expected to succeed (until they're specifically fixed, of course).
e018f8be 2224
7205a85d
YO
2225(Not available on Win32)
2226
e018f8be
JH
2227=item minitest
2228
2229Run F<miniperl> on F<t/base>, F<t/comp>, F<t/cmd>, F<t/run>, F<t/io>,
8cebccf4 2230F<t/op>, F<t/uni> and F<t/mro> tests.
e018f8be 2231
7a834142
JH
2232=item test.valgrind check.valgrind utest.valgrind ucheck.valgrind
2233
2234(Only in Linux) Run all the tests using the memory leak + naughty
2235memory access tool "valgrind". The log files will be named
2236F<testname.valgrind>.
2237
e018f8be
JH
2238=item test.third check.third utest.third ucheck.third
2239
2240(Only in Tru64) Run all the tests using the memory leak + naughty
2241memory access tool "Third Degree". The log files will be named
60a57c1c 2242F<perl.3log.testname>.
e018f8be
JH
2243
2244=item test.torture torturetest
2245
2246Run all the usual tests and some extra tests. As of Perl 5.8.0 the
244d9cb7 2247only extra tests are Abigail's JAPHs, F<t/japh/abigail.t>.
e018f8be
JH
2248
2249You can also run the torture test with F<t/harness> by giving
2250C<-torture> argument to F<t/harness>.
2251
2252=item utest ucheck test.utf8 check.utf8
2253
2254Run all the tests with -Mutf8. Not all tests will succeed.
2255
7205a85d
YO
2256(Not available on Win32)
2257
cc0710ff
RGS
2258=item minitest.utf16 test.utf16
2259
2260Runs the tests with UTF-16 encoded scripts, encoded with different
2261versions of this encoding.
2262
2263C<make utest.utf16> runs the test suite with a combination of C<-utf8> and
2264C<-utf16> arguments to F<t/TEST>.
2265
7205a85d
YO
2266(Not available on Win32)
2267
244d9cb7
RGS
2268=item test_harness
2269
2270Run the test suite with the F<t/harness> controlling program, instead of
2271F<t/TEST>. F<t/harness> is more sophisticated, and uses the
2272L<Test::Harness> module, thus using this test target supposes that perl
2273mostly works. The main advantage for our purposes is that it prints a
00bf5cd9
RGS
2274detailed summary of failed tests at the end. Also, unlike F<t/TEST>, it
2275doesn't redirect stderr to stdout.
244d9cb7 2276
7205a85d
YO
2277Note that under Win32 F<t/harness> is always used instead of F<t/TEST>, so
2278there is no special "test_harness" target.
2279
2280Under Win32's "test" target you may use the TEST_SWITCHES and TEST_FILES
2281environment variables to control the behaviour of F<t/harness>. This means
2282you can say
2283
2284 nmake test TEST_FILES="op/*.t"
2285 nmake test TEST_SWITCHES="-torture" TEST_FILES="op/*.t"
2286
2287=item test-notty test_notty
2288
2289Sets PERL_SKIP_TTY_TEST to true before running normal test.
2290
244d9cb7
RGS
2291=back
2292
2293=head2 Running tests by hand
2294
2295You can run part of the test suite by hand by using one the following
2296commands from the F<t/> directory :
2297
2298 ./perl -I../lib TEST list-of-.t-files
2299
2300or
2301
2302 ./perl -I../lib harness list-of-.t-files
2303
2304(if you don't specify test scripts, the whole test suite will be run.)
2305
7205a85d
YO
2306=head3 Using t/harness for testing
2307
2308If you use C<harness> for testing you have several command line options
2309available to you. The arguments are as follows, and are in the order
2310that they must appear if used together.
2311
2312 harness -v -torture -re=pattern LIST OF FILES TO TEST
2313 harness -v -torture -re LIST OF PATTERNS TO MATCH
2314
2315If C<LIST OF FILES TO TEST> is omitted the file list is obtained from
07aa3531 2316the manifest. The file list may include shell wildcards which will be
7205a85d
YO
2317expanded out.
2318
2319=over 4
2320
2321=item -v
2322
07aa3531 2323Run the tests under verbose mode so you can see what tests were run,
7205a85d
YO
2324and debug outbut.
2325
2326=item -torture
2327
2328Run the torture tests as well as the normal set.
2329
2330=item -re=PATTERN
2331
2332Filter the file list so that all the test files run match PATTERN.
2333Note that this form is distinct from the B<-re LIST OF PATTERNS> form below
2334in that it allows the file list to be provided as well.
2335
2336=item -re LIST OF PATTERNS
2337
07aa3531 2338Filter the file list so that all the test files run match
7205a85d
YO
2339/(LIST|OF|PATTERNS)/. Note that with this form the patterns
2340are joined by '|' and you cannot supply a list of files, instead
2341the test files are obtained from the MANIFEST.
2342
2343=back
2344
244d9cb7
RGS
2345You can run an individual test by a command similar to
2346
2347 ./perl -I../lib patho/to/foo.t
2348
2349except that the harnesses set up some environment variables that may
2350affect the execution of the test :
2351
07aa3531 2352=over 4
244d9cb7
RGS
2353
2354=item PERL_CORE=1
2355
2356indicates that we're running this test part of the perl core test suite.
2357This is useful for modules that have a dual life on CPAN.
2358
2359=item PERL_DESTRUCT_LEVEL=2
2360
2361is set to 2 if it isn't set already (see L</PERL_DESTRUCT_LEVEL>)
2362
2363=item PERL
2364
2365(used only by F<t/TEST>) if set, overrides the path to the perl executable
2366that should be used to run the tests (the default being F<./perl>).
2367
2368=item PERL_SKIP_TTY_TEST
2369
2370if set, tells to skip the tests that need a terminal. It's actually set
2371automatically by the Makefile, but can also be forced artificially by
2372running 'make test_notty'.
2373
e018f8be 2374=back
f7e1e956 2375
7cd58830
RGS
2376=head3 Other environment variables that may influence tests
2377
2378=over 4
2379
2380=item PERL_TEST_Net_Ping
2381
2382Setting this variable runs all the Net::Ping modules tests,
2383otherwise some tests that interact with the outside world are skipped.
2384See L<perl58delta>.
2385
2386=item PERL_TEST_NOVREXX
2387
2388Setting this variable skips the vrexx.t tests for OS2::REXX.
2389
2390=item PERL_TEST_NUMCONVERTS
2391
2392This sets a variable in op/numconvert.t.
2393
2394=back
2395
2396See also the documentation for the Test and Test::Harness modules,
2397for more environment variables that affect testing.
2398
d7889f52
JH
2399=head2 Common problems when patching Perl source code
2400
2401Perl source plays by ANSI C89 rules: no C99 (or C++) extensions. In
2402some cases we have to take pre-ANSI requirements into consideration.
2403You don't care about some particular platform having broken Perl?
2404I hear there is still a strong demand for J2EE programmers.
2405
2406=head2 Perl environment problems
2407
2408=over 4
2409
2410=item *
2411
2412Not compiling with threading
2413
2414Compiling with threading (-Duseithreads) completely rewrites
2415the function prototypes of Perl. You better try your changes
0bec6c03 2416with that. Related to this is the difference between "Perl_-less"
d7889f52
JH
2417and "Perl_-ly" APIs, for example:
2418
2419 Perl_sv_setiv(aTHX_ ...);
2420 sv_setiv(...);
2421
ee9468a2
RGS
2422The first one explicitly passes in the context, which is needed for e.g.
2423threaded builds. The second one does that implicitly; do not get them
def4ed7d
JH
2424mixed. If you are not passing in a aTHX_, you will need to do a dTHX
2425(or a dVAR) as the first thing in the function.
d7889f52
JH
2426
2427See L<perlguts/"How multiple interpreters and concurrency are supported">
2428for further discussion about context.
2429
2430=item *
2431
2432Not compiling with -DDEBUGGING
2433
2434The DEBUGGING define exposes more code to the compiler,
0bec6c03 2435therefore more ways for things to go wrong. You should try it.
d7889f52
JH
2436
2437=item *
2438
ee9468a2
RGS
2439Introducing (non-read-only) globals
2440
2441Do not introduce any modifiable globals, truly global or file static.
bc028b6b
JH
2442They are bad form and complicate multithreading and other forms of
2443concurrency. The right way is to introduce them as new interpreter
2444variables, see F<intrpvar.h> (at the very end for binary compatibility).
ee9468a2
RGS
2445
2446Introducing read-only (const) globals is okay, as long as you verify
2447with e.g. C<nm libperl.a|egrep -v ' [TURtr] '> (if your C<nm> has
2448BSD-style output) that the data you added really is read-only.
2449(If it is, it shouldn't show up in the output of that command.)
2450
2451If you want to have static strings, make them constant:
2452
2453 static const char etc[] = "...";
2454
bc028b6b 2455If you want to have arrays of constant strings, note carefully
ee9468a2
RGS
2456the right combination of C<const>s:
2457
2458 static const char * const yippee[] =
2459 {"hi", "ho", "silver"};
2460
bc028b6b
JH
2461There is a way to completely hide any modifiable globals (they are all
2462moved to heap), the compilation setting C<-DPERL_GLOBAL_STRUCT_PRIVATE>.
2463It is not normally used, but can be used for testing, read more
def4ed7d 2464about it in L<perlguts/"Background and PERL_IMPLICIT_CONTEXT">.
bc028b6b 2465
ee9468a2
RGS
2466=item *
2467
d7889f52
JH
2468Not exporting your new function
2469
2470Some platforms (Win32, AIX, VMS, OS/2, to name a few) require any
2471function that is part of the public API (the shared Perl library)
2472to be explicitly marked as exported. See the discussion about
2473F<embed.pl> in L<perlguts>.
2474
2475=item *
2476
2477Exporting your new function
2478
2479The new shiny result of either genuine new functionality or your
2480arduous refactoring is now ready and correctly exported. So what
def4ed7d 2481could possibly go wrong?
d7889f52
JH
2482
2483Maybe simply that your function did not need to be exported in the
2484first place. Perl has a long and not so glorious history of exporting
2485functions that it should not have.
2486
2487If the function is used only inside one source code file, make it
2488static. See the discussion about F<embed.pl> in L<perlguts>.
2489
2490If the function is used across several files, but intended only for
2491Perl's internal use (and this should be the common case), do not
2492export it to the public API. See the discussion about F<embed.pl>
2493in L<perlguts>.
2494
2495=back
2496
5b38b9cd 2497=head2 Portability problems
d7889f52
JH
2498
2499The following are common causes of compilation and/or execution
2500failures, not common to Perl as such. The C FAQ is good bedtime
0bec6c03
JH
2501reading. Please test your changes with as many C compilers and
2502platforms as possible -- we will, anyway, and it's nice to save
2503oneself from public embarrassment.
2504
9aaf14db
RGS
2505If using gcc, you can add the C<-std=c89> option which will hopefully
2506catch most of these unportabilities. (However it might also catch
2507incompatibilities in your system's header files.)
d1307786 2508
a8e98a71
JH
2509Use the Configure C<-Dgccansipedantic> flag to enable the gcc
2510C<-ansi -pedantic> flags which enforce stricter ANSI rules.
2511
def4ed7d
JH
2512If using the C<gcc -Wall> note that not all the possible warnings
2513(like C<-Wunitialized>) are given unless you also compile with C<-O>.
2514
2515Note that if using gcc, starting from Perl 5.9.5 the Perl core source
2516code files (the ones at the top level of the source code distribution,
2517but not e.g. the extensions under ext/) are automatically compiled
2518with as many as possible of the C<-std=c89>, C<-ansi>, C<-pedantic>,
2519and a selection of C<-W> flags (see cflags.SH).
27565cb6 2520
0bec6c03 2521Also study L<perlport> carefully to avoid any bad assumptions
def4ed7d 2522about the operating system, filesystems, and so forth.
0bec6c03 2523
606fd33d 2524You may once in a while try a "make microperl" to see whether we
63796a85 2525can still compile Perl with just the bare minimum of interfaces.
606fd33d 2526(See README.micro.)
ee9468a2 2527
0bec6c03 2528Do not assume an operating system indicates a certain compiler.
d7889f52
JH
2529
2530=over 4
2531
2532=item *
2533
2534Casting pointers to integers or casting integers to pointers
2535
2536 void castaway(U8* p)
2537 {
2538 IV i = p;
2539
2540or
2541
2542 void castaway(U8* p)
2543 {
2544 IV i = (IV)p;
2545
ee9468a2 2546Both are bad, and broken, and unportable. Use the PTR2IV()
d7889f52
JH
2547macro that does it right. (Likewise, there are PTR2UV(), PTR2NV(),
2548INT2PTR(), and NUM2PTR().)
2549
2550=item *
2551
0bec6c03
JH
2552Casting between data function pointers and data pointers
2553
d7889f52
JH
2554Technically speaking casting between function pointers and data
2555pointers is unportable and undefined, but practically speaking
2556it seems to work, but you should use the FPTR2DPTR() and DPTR2FPTR()
0bec6c03 2557macros. Sometimes you can also play games with unions.
d7889f52
JH
2558
2559=item *
2560
2561Assuming sizeof(int) == sizeof(long)
2562
2563There are platforms where longs are 64 bits, and platforms where ints
2564are 64 bits, and while we are out to shock you, even platforms where
2565shorts are 64 bits. This is all legal according to the C standard.
2566(In other words, "long long" is not a portable way to specify 64 bits,
2567and "long long" is not even guaranteed to be any wider than "long".)
63796a85
JH
2568
2569Instead, use the definitions IV, UV, IVSIZE, I32SIZE, and so forth.
2570Avoid things like I32 because they are B<not> guaranteed to be
2571I<exactly> 32 bits, they are I<at least> 32 bits, nor are they
2572guaranteed to be B<int> or B<long>. If you really explicitly need
257364-bit variables, use I64 and U64, but only if guarded by HAS_QUAD.
d7889f52
JH
2574
2575=item *
2576
2577Assuming one can dereference any type of pointer for any type of data
2578
2579 char *p = ...;
def4ed7d 2580 long pony = *p; /* BAD */
d7889f52
JH
2581
2582Many platforms, quite rightly so, will give you a core dump instead
2583of a pony if the p happens not be correctly aligned.
2584
2585=item *
2586
2587Lvalue casts
2588
def4ed7d 2589 (int)*p = ...; /* BAD */
d7889f52
JH
2590
2591Simply not portable. Get your lvalue to be of the right type,
27565cb6
JH
2592or maybe use temporary variables, or dirty tricks with unions.
2593
2594=item *
2595
606fd33d
JH
2596Assume B<anything> about structs (especially the ones you
2597don't control, like the ones coming from the system headers)
27565cb6
JH
2598
2599=over 8
2600
2601=item *
2602
2603That a certain field exists in a struct
2604
2605=item *
2606
902821cc 2607That no other fields exist besides the ones you know of
27565cb6
JH
2608
2609=item *
2610
606fd33d 2611That a field is of certain signedness, sizeof, or type
27565cb6
JH
2612
2613=item *
2614
2615That the fields are in a certain order
2616
606fd33d
JH
2617=over 8
2618
27565cb6
JH
2619=item *
2620
606fd33d
JH
2621While C guarantees the ordering specified in the struct definition,
2622between different platforms the definitions might differ
2623
2624=back
27565cb6
JH
2625
2626=item *
2627
606fd33d
JH
2628That the sizeof(struct) or the alignments are the same everywhere
2629
2630=over 8
27565cb6
JH
2631
2632=item *
2633
606fd33d
JH
2634There might be padding bytes between the fields to align the fields -
2635the bytes can be anything
2636
2637=item *
2638
2639Structs are required to be aligned to the maximum alignment required
2640by the fields - which for native types is for usually equivalent to
2641sizeof() of the field
2642
2643=back
27565cb6
JH
2644
2645=back
d7889f52
JH
2646
2647=item *
2648
0bec6c03
JH
2649Mixing #define and #ifdef
2650
2651 #define BURGLE(x) ... \
def4ed7d 2652 #ifdef BURGLE_OLD_STYLE /* BAD */
0bec6c03
JH
2653 ... do it the old way ... \
2654 #else
2655 ... do it the new way ... \
2656 #endif
2657
ee9468a2
RGS
2658You cannot portably "stack" cpp directives. For example in the above
2659you need two separate BURGLE() #defines, one for each #ifdef branch.
2660
2661=item *
2662
2663Adding stuff after #endif or #else
2664
2665 #ifdef SNOSH
2666 ...
def4ed7d 2667 #else !SNOSH /* BAD */
ee9468a2 2668 ...
def4ed7d 2669 #endif SNOSH /* BAD */
ee9468a2 2670
def4ed7d
JH
2671The #endif and #else cannot portably have anything non-comment after
2672them. If you want to document what is going (which is a good idea
2673especially if the branches are long), use (C) comments:
ee9468a2
RGS
2674
2675 #ifdef SNOSH
2676 ...
2677 #else /* !SNOSH */
2678 ...
2679 #endif /* SNOSH */
2680
2681The gcc option C<-Wendif-labels> warns about the bad variant
2682(by default on starting from Perl 5.9.4).
0bec6c03
JH
2683
2684=item *
2685
27565cb6
JH
2686Having a comma after the last element of an enum list
2687
2688 enum color {
2689 CERULEAN,
2690 CHARTREUSE,
def4ed7d 2691 CINNABAR, /* BAD */
27565cb6
JH
2692 };
2693
2694is not portable. Leave out the last comma.
2695
2696Also note that whether enums are implicitly morphable to ints
2697varies between compilers, you might need to (int).
2698
2699=item *
2700
d7889f52
JH
2701Using //-comments
2702
def4ed7d 2703 // This function bamfoodles the zorklator. /* BAD */
d7889f52
JH
2704
2705That is C99 or C++. Perl is C89. Using the //-comments is silently
0bec6c03
JH
2706allowed by many C compilers but cranking up the ANSI C89 strictness
2707(which we like to do) causes the compilation to fail.
d7889f52
JH
2708
2709=item *
2710
2711Mixing declarations and code
2712
2713 void zorklator()
2714 {
2715 int n = 3;
def4ed7d 2716 set_zorkmids(n); /* BAD */
d7889f52
JH
2717 int q = 4;
2718
0bec6c03
JH
2719That is C99 or C++. Some C compilers allow that, but you shouldn't.
2720
63796a85
JH
2721The gcc option C<-Wdeclaration-after-statements> scans for such problems
2722(by default on starting from Perl 5.9.4).
2723
0bec6c03
JH
2724=item *
2725
2726Introducing variables inside for()
2727
def4ed7d 2728 for(int i = ...; ...; ...) { /* BAD */
0bec6c03
JH
2729
2730That is C99 or C++. While it would indeed be awfully nice to have that
2731also in C89, to limit the scope of the loop variable, alas, we cannot.
d7889f52
JH
2732
2733=item *
2734
2735Mixing signed char pointers with unsigned char pointers
2736
2737 int foo(char *s) { ... }
2738 ...
2739 unsigned char *t = ...; /* Or U8* t = ... */
def4ed7d 2740 foo(t); /* BAD */
d7889f52
JH
2741
2742While this is legal practice, it is certainly dubious, and downright
2743fatal in at least one platform: for example VMS cc considers this a
def4ed7d
JH
2744fatal error. One cause for people often making this mistake is that a
2745"naked char" and therefore dereferencing a "naked char pointer" have
2746an undefined signedness: it depends on the compiler and the flags of
2747the compiler and the underlying platform whether the result is signed
2748or unsigned. For this very same reason using a 'char' as an array
2749index is bad.
d7889f52
JH
2750
2751=item *
2752
2753Macros that have string constants and their arguments as substrings of
2754the string constants
2755
def4ed7d 2756 #define FOO(n) printf("number = %d\n", n) /* BAD */
d7889f52
JH
2757 FOO(10);
2758
2759Pre-ANSI semantics for that was equivalent to
2760
2761 printf("10umber = %d\10");
2762
0bec6c03
JH
2763which is probably not what you were expecting. Unfortunately at least
2764one reasonably common and modern C compiler does "real backward
63796a85 2765compatibility" here, in AIX that is what still happens even though the
0bec6c03
JH
2766rest of the AIX compiler is very happily C89.
2767
2768=item *
2769
ee9468a2
RGS
2770Using printf formats for non-basic C types
2771
2772 IV i = ...;
def4ed7d 2773 printf("i = %d\n", i); /* BAD */
ee9468a2
RGS
2774
2775While this might by accident work in some platform (where IV happens
2776to be an C<int>), in general it cannot. IV might be something larger.
2777Even worse the situation is with more specific types (defined by Perl's
2778configuration step in F<config.h>):
2779
2780 Uid_t who = ...;
def4ed7d 2781 printf("who = %d\n", who); /* BAD */
ee9468a2
RGS
2782
2783The problem here is that Uid_t might be not only not C<int>-wide
2784but it might also be unsigned, in which case large uids would be
2785printed as negative values.
2786
2787There is no simple solution to this because of printf()'s limited
2788intelligence, but for many types the right format is available as
2789with either 'f' or '_f' suffix, for example:
2790
2791 IVdf /* IV in decimal */
2792 UVxf /* UV is hexadecimal */
2793
2794 printf("i = %"IVdf"\n", i); /* The IVdf is a string constant. */
2795
2796 Uid_t_f /* Uid_t in decimal */
2797
2798 printf("who = %"Uid_t_f"\n", who);
2799
63796a85
JH
2800Or you can try casting to a "wide enough" type:
2801
2802 printf("i = %"IVdf"\n", (IV)something_very_small_and_signed);
2803
2804Also remember that the C<%p> format really does require a void pointer:
2805
2806 U8* p = ...;
2807 printf("p = %p\n", (void*)p);
2808
ee9468a2
RGS
2809The gcc option C<-Wformat> scans for such problems.
2810
2811=item *
2812
0bec6c03
JH
2813Blindly using variadic macros
2814
63796a85
JH
2815gcc has had them for a while with its own syntax, and C99 brought
2816them with a standardized syntax. Don't use the former, and use
2817the latter only if the HAS_C99_VARIADIC_MACROS is defined.
0bec6c03
JH
2818
2819=item *
2820
2821Blindly passing va_list
2822
2823Not all platforms support passing va_list to further varargs (stdarg)
2824functions. The right thing to do is to copy the va_list using the
2825Perl_va_copy() if the NEED_VA_COPY is defined.
d7889f52 2826
ee9468a2
RGS
2827=item *
2828
e5afc1ae 2829Using gcc statement expressions
63796a85 2830
def4ed7d 2831 val = ({...;...;...}); /* BAD */
63796a85 2832
def4ed7d
JH
2833While a nice extension, it's not portable. The Perl code does
2834admittedly use them if available to gain some extra speed
2835(essentially as a funky form of inlining), but you shouldn't.
63796a85
JH
2836
2837=item *
2838
2839Binding together several statements
2840
2841Use the macros STMT_START and STMT_END.
2842
2843 STMT_START {
2844 ...
2845 } STMT_END
2846
2847=item *
2848
ee9468a2
RGS
2849Testing for operating systems or versions when should be testing for features
2850
def4ed7d 2851 #ifdef __FOONIX__ /* BAD */
ee9468a2
RGS
2852 foo = quux();
2853 #endif
2854
2855Unless you know with 100% certainty that quux() is only ever available
2856for the "Foonix" operating system B<and> that is available B<and>
2857correctly working for B<all> past, present, B<and> future versions of
2858"Foonix", the above is very wrong. This is more correct (though still
2859not perfect, because the below is a compile-time check):
2860
2861 #ifdef HAS_QUUX
2862 foo = quux();
2863 #endif
2864
def4ed7d 2865How does the HAS_QUUX become defined where it needs to be? Well, if
353c6505 2866Foonix happens to be UNIXy enough to be able to run the Configure
ee9468a2
RGS
2867script, and Configure has been taught about detecting and testing
2868quux(), the HAS_QUUX will be correctly defined. In other platforms,
2869the corresponding configuration step will hopefully do the same.
2870
2871In a pinch, if you cannot wait for Configure to be educated,
2872or if you have a good hunch of where quux() might be available,
2873you can temporarily try the following:
2874
2875 #if (defined(__FOONIX__) || defined(__BARNIX__))
2876 # define HAS_QUUX
2877 #endif
2878
2879 ...
2880
2881 #ifdef HAS_QUUX
2882 foo = quux();
2883 #endif
2884
2885But in any case, try to keep the features and operating systems separate.
2886
d7889f52
JH
2887=back
2888
ad7244db
JH
2889=head2 Problematic System Interfaces
2890
2891=over 4
2892
2893=item *
2894
353c6505 2895malloc(0), realloc(0), calloc(0, 0) are non-portable. To be portable
ad7244db
JH
2896allocate at least one byte. (In general you should rarely need to
2897work at this low level, but instead use the various malloc wrappers.)
2898
2899=item *
2900
2901snprintf() - the return type is unportable. Use my_snprintf() instead.
2902
2903=back
2904
d7889f52
JH
2905=head2 Security problems
2906
2907Last but not least, here are various tips for safer coding.
2908
2909=over 4
2910
2911=item *
2912
2913Do not use gets()
2914
2915Or we will publicly ridicule you. Seriously.
2916
2917=item *
2918
d1307786 2919Do not use strcpy() or strcat() or strncpy() or strncat()
d7889f52 2920
d1307786
JH
2921Use my_strlcpy() and my_strlcat() instead: they either use the native
2922implementation, or Perl's own implementation (borrowed from the public
2923domain implementation of INN).
d7889f52
JH
2924
2925=item *
2926
2927Do not use sprintf() or vsprintf()
2928
0bec6c03 2929If you really want just plain byte strings, use my_snprintf()
64d9b66b 2930and my_vsnprintf() instead, which will try to use snprintf() and
0bec6c03
JH
2931vsnprintf() if those safer APIs are available. If you want something
2932fancier than a plain byte string, use SVs and Perl_sv_catpvf().
d7889f52
JH
2933
2934=back
2935
902b9dbf
MF
2936=head1 EXTERNAL TOOLS FOR DEBUGGING PERL
2937
2938Sometimes it helps to use external tools while debugging and
2939testing Perl. This section tries to guide you through using
2940some common testing and debugging tools with Perl. This is
2941meant as a guide to interfacing these tools with Perl, not
2942as any kind of guide to the use of the tools themselves.
2943
a958818a
JH
2944B<NOTE 1>: Running under memory debuggers such as Purify, valgrind, or
2945Third Degree greatly slows down the execution: seconds become minutes,
2946minutes become hours. For example as of Perl 5.8.1, the
2947ext/Encode/t/Unicode.t takes extraordinarily long to complete under
2948e.g. Purify, Third Degree, and valgrind. Under valgrind it takes more
2949than six hours, even on a snappy computer-- the said test must be
2950doing something that is quite unfriendly for memory debuggers. If you
2951don't feel like waiting, that you can simply kill away the perl
2952process.
2953
2954B<NOTE 2>: To minimize the number of memory leak false alarms (see
2955L</PERL_DESTRUCT_LEVEL> for more information), you have to have
2956environment variable PERL_DESTRUCT_LEVEL set to 2. The F<TEST>
2957and harness scripts do that automatically. But if you are running
2958some of the tests manually-- for csh-like shells:
2959
2960 setenv PERL_DESTRUCT_LEVEL 2
2961
2962and for Bourne-type shells:
2963
2964 PERL_DESTRUCT_LEVEL=2
2965 export PERL_DESTRUCT_LEVEL
2966
2967or in UNIXy environments you can also use the C<env> command:
2968
2969 env PERL_DESTRUCT_LEVEL=2 valgrind ./perl -Ilib ...
a1b65709 2970
37c0adeb
JH
2971B<NOTE 3>: There are known memory leaks when there are compile-time
2972errors within eval or require, seeing C<S_doeval> in the call stack
2973is a good sign of these. Fixing these leaks is non-trivial,
2974unfortunately, but they must be fixed eventually.
2975
f50e5b73
MH
2976B<NOTE 4>: L<DynaLoader> will not clean up after itself completely
2977unless Perl is built with the Configure option
2978C<-Accflags=-DDL_UNLOAD_ALL_AT_EXIT>.
2979
902b9dbf
MF
2980=head2 Rational Software's Purify
2981
2982Purify is a commercial tool that is helpful in identifying
2983memory overruns, wild pointers, memory leaks and other such
2984badness. Perl must be compiled in a specific way for
2985optimal testing with Purify. Purify is available under
2986Windows NT, Solaris, HP-UX, SGI, and Siemens Unix.
2987
902b9dbf
MF
2988=head2 Purify on Unix
2989
2990On Unix, Purify creates a new Perl binary. To get the most
2991benefit out of Purify, you should create the perl to Purify
2992using:
2993
2994 sh Configure -Accflags=-DPURIFY -Doptimize='-g' \
2995 -Uusemymalloc -Dusemultiplicity
2996
2997where these arguments mean:
2998
2999=over 4
3000
3001=item -Accflags=-DPURIFY
3002
3003Disables Perl's arena memory allocation functions, as well as
3004forcing use of memory allocation functions derived from the
3005system malloc.
3006
3007=item -Doptimize='-g'
3008
3009Adds debugging information so that you see the exact source
3010statements where the problem occurs. Without this flag, all
3011you will see is the source filename of where the error occurred.
3012
3013=item -Uusemymalloc
3014
3015Disable Perl's malloc so that Purify can more closely monitor
3016allocations and leaks. Using Perl's malloc will make Purify
3017report most leaks in the "potential" leaks category.
3018
3019=item -Dusemultiplicity
3020
3021Enabling the multiplicity option allows perl to clean up
3022thoroughly when the interpreter shuts down, which reduces the
3023number of bogus leak reports from Purify.
3024
3025=back
3026
3027Once you've compiled a perl suitable for Purify'ing, then you
3028can just:
3029
07aa3531 3030 make pureperl
902b9dbf
MF
3031
3032which creates a binary named 'pureperl' that has been Purify'ed.
3033This binary is used in place of the standard 'perl' binary
3034when you want to debug Perl memory problems.
3035
3036As an example, to show any memory leaks produced during the
3037standard Perl testset you would create and run the Purify'ed
3038perl as:
3039
3040 make pureperl
3041 cd t
07aa3531 3042 ../pureperl -I../lib harness
902b9dbf
MF
3043
3044which would run Perl on test.pl and report any memory problems.
3045
3046Purify outputs messages in "Viewer" windows by default. If
3047you don't have a windowing environment or if you simply
3048want the Purify output to unobtrusively go to a log file
3049instead of to the interactive window, use these following
3050options to output to the log file "perl.log":
3051
3052 setenv PURIFYOPTIONS "-chain-length=25 -windows=no \
3053 -log-file=perl.log -append-logfile=yes"
3054
3055If you plan to use the "Viewer" windows, then you only need this option:
3056
3057 setenv PURIFYOPTIONS "-chain-length=25"
3058
c406981e
JH
3059In Bourne-type shells:
3060
98631ff8
JL
3061 PURIFYOPTIONS="..."
3062 export PURIFYOPTIONS
c406981e
JH
3063
3064or if you have the "env" utility:
3065
98631ff8 3066 env PURIFYOPTIONS="..." ../pureperl ...
c406981e 3067
902b9dbf
MF
3068=head2 Purify on NT
3069
3070Purify on Windows NT instruments the Perl binary 'perl.exe'
3071on the fly. There are several options in the makefile you
3072should change to get the most use out of Purify:
3073
3074=over 4
3075
3076=item DEFINES
3077
3078You should add -DPURIFY to the DEFINES line so the DEFINES
3079line looks something like:
3080
07aa3531 3081 DEFINES = -DWIN32 -D_CONSOLE -DNO_STRICT $(CRYPT_FLAG) -DPURIFY=1
902b9dbf
MF
3082
3083to disable Perl's arena memory allocation functions, as
3084well as to force use of memory allocation functions derived
3085from the system malloc.
3086
3087=item USE_MULTI = define
3088
3089Enabling the multiplicity option allows perl to clean up
3090thoroughly when the interpreter shuts down, which reduces the
3091number of bogus leak reports from Purify.
3092
3093=item #PERL_MALLOC = define
3094
3095Disable Perl's malloc so that Purify can more closely monitor
3096allocations and leaks. Using Perl's malloc will make Purify
3097report most leaks in the "potential" leaks category.
3098
3099=item CFG = Debug
3100
3101Adds debugging information so that you see the exact source
3102statements where the problem occurs. Without this flag, all
3103you will see is the source filename of where the error occurred.
3104
3105=back
3106
3107As an example, to show any memory leaks produced during the
3108standard Perl testset you would create and run Purify as:
3109
3110 cd win32
3111 make
3112 cd ../t
07aa3531 3113 purify ../perl -I../lib harness
902b9dbf
MF
3114
3115which would instrument Perl in memory, run Perl on test.pl,
3116then finally report any memory problems.
3117
7a834142
JH
3118=head2 valgrind
3119
3120The excellent valgrind tool can be used to find out both memory leaks
9df8f87f
LB
3121and illegal memory accesses. As of version 3.3.0, Valgrind only
3122supports Linux on x86, x86-64 and PowerPC. The special "test.valgrind"
3123target can be used to run the tests under valgrind. Found errors
3124and memory leaks are logged in files named F<testfile.valgrind>.
07aa3531
JC
3125
3126Valgrind also provides a cachegrind tool, invoked on perl as:
3127
038c294a 3128 VG_OPTS=--tool=cachegrind make test.valgrind
d44161bf
MHM
3129
3130As system libraries (most notably glibc) are also triggering errors,
3131valgrind allows to suppress such errors using suppression files. The
3132default suppression file that comes with valgrind already catches a lot
3133of them. Some additional suppressions are defined in F<t/perl.supp>.
7a834142
JH
3134
3135To get valgrind and for more information see
3136
3137 http://developer.kde.org/~sewardj/
3138
f134cc4e 3139=head2 Compaq's/Digital's/HP's Third Degree
09187cb1
JH
3140
3141Third Degree is a tool for memory leak detection and memory access checks.
3142It is one of the many tools in the ATOM toolkit. The toolkit is only
3143available on Tru64 (formerly known as Digital UNIX formerly known as
3144DEC OSF/1).
3145
3146When building Perl, you must first run Configure with -Doptimize=-g
3147and -Uusemymalloc flags, after that you can use the make targets
51a35ef1
JH
3148"perl.third" and "test.third". (What is required is that Perl must be
3149compiled using the C<-g> flag, you may need to re-Configure.)
09187cb1 3150
64cea5fd 3151The short story is that with "atom" you can instrument the Perl
83f0ef60 3152executable to create a new executable called F<perl.third>. When the
4ae3d70a 3153instrumented executable is run, it creates a log of dubious memory
83f0ef60 3154traffic in file called F<perl.3log>. See the manual pages of atom and
4ae3d70a
JH
3155third for more information. The most extensive Third Degree
3156documentation is available in the Compaq "Tru64 UNIX Programmer's
3157Guide", chapter "Debugging Programs with Third Degree".
64cea5fd 3158
9c54ecba 3159The "test.third" leaves a lot of files named F<foo_bar.3log> in the t/
64cea5fd
JH
3160subdirectory. There is a problem with these files: Third Degree is so
3161effective that it finds problems also in the system libraries.
9c54ecba
JH
3162Therefore you should used the Porting/thirdclean script to cleanup
3163the F<*.3log> files.
64cea5fd
JH
3164
3165There are also leaks that for given certain definition of a leak,
3166aren't. See L</PERL_DESTRUCT_LEVEL> for more information.
3167
3168=head2 PERL_DESTRUCT_LEVEL
3169
a958818a
JH
3170If you want to run any of the tests yourself manually using e.g.
3171valgrind, or the pureperl or perl.third executables, please note that
3172by default perl B<does not> explicitly cleanup all the memory it has
3173allocated (such as global memory arenas) but instead lets the exit()
3174of the whole program "take care" of such allocations, also known as
3175"global destruction of objects".
64cea5fd
JH
3176
3177There is a way to tell perl to do complete cleanup: set the
3178environment variable PERL_DESTRUCT_LEVEL to a non-zero value.
3179The t/TEST wrapper does set this to 2, and this is what you
3180need to do too, if you don't want to see the "global leaks":
1f56d61a 3181For example, for "third-degreed" Perl:
64cea5fd 3182
1f56d61a 3183 env PERL_DESTRUCT_LEVEL=2 ./perl.third -Ilib t/foo/bar.t
09187cb1 3184
414f2397
RGS
3185(Note: the mod_perl apache module uses also this environment variable
3186for its own purposes and extended its semantics. Refer to the mod_perl
287a822c
RGS
3187documentation for more information. Also, spawned threads do the
3188equivalent of setting this variable to the value 1.)
5a6c59ef
DM
3189
3190If, at the end of a run you get the message I<N scalars leaked>, you can
fd0854ff
DM
3191recompile with C<-DDEBUG_LEAKING_SCALARS>, which will cause the addresses
3192of all those leaked SVs to be dumped along with details as to where each
3193SV was originally allocated. This information is also displayed by
3194Devel::Peek. Note that the extra details recorded with each SV increases
3195memory usage, so it shouldn't be used in production environments. It also
3196converts C<new_SV()> from a macro into a real function, so you can use
3197your favourite debugger to discover where those pesky SVs were allocated.
414f2397 3198
d7a2c63c
MHM
3199If you see that you're leaking memory at runtime, but neither valgrind
3200nor C<-DDEBUG_LEAKING_SCALARS> will find anything, you're probably
3201leaking SVs that are still reachable and will be properly cleaned up
3202during destruction of the interpreter. In such cases, using the C<-Dm>
3203switch can point you to the source of the leak. If the executable was
3204built with C<-DDEBUG_LEAKING_SCALARS>, C<-Dm> will output SV allocations
3205in addition to memory allocations. Each SV allocation has a distinct
3206serial number that will be written on creation and destruction of the SV.
3207So if you're executing the leaking code in a loop, you need to look for
3208SVs that are created, but never destroyed between each cycle. If such an
3209SV is found, set a conditional breakpoint within C<new_SV()> and make it
3210break only when C<PL_sv_serial> is equal to the serial number of the
3211leaking SV. Then you will catch the interpreter in exactly the state
3212where the leaking SV is allocated, which is sufficient in many cases to
3213find the source of the leak.
3214
3215As C<-Dm> is using the PerlIO layer for output, it will by itself
3216allocate quite a bunch of SVs, which are hidden to avoid recursion.
3217You can bypass the PerlIO layer if you use the SV logging provided
3218by C<-DPERL_MEM_LOG> instead.
3219
46c6c7e2
JH
3220=head2 PERL_MEM_LOG
3221
9f653bb5 3222If compiled with C<-DPERL_MEM_LOG>, all Newx() and Renew() allocations
46c6c7e2
JH
3223and Safefree() in the Perl core go through logging functions, which is
3224handy for breakpoint setting. If also compiled with C<-DPERL_MEM_LOG_STDERR>,
e352bcff
JH
3225the allocations and frees are logged to STDERR (or more precisely, to the
3226file descriptor 2) in these logging functions, with the calling source code
3227file and line number (and C function name, if supported by the C compiler).
3228
3229This logging is somewhat similar to C<-Dm> but independent of C<-DDEBUGGING>,
3230and at a higher level (the C<-Dm> is directly at the point of C<malloc()>,
a015a77e 3231while the C<PERL_MEM_LOG> is at the level of C<New()>).
46c6c7e2 3232
d7a2c63c
MHM
3233In addition to memory allocations, SV allocations will be logged, just as
3234with C<-Dm>. However, since the logging doesn't use PerlIO, all SV allocations
3235are logged and no extra SV allocations are introduced by enabling the logging.
3236If compiled with C<-DDEBUG_LEAKING_SCALARS>, the serial number for each SV
3237allocation is also logged.
3238
3239You can control the logging from your environment if you compile with
3240C<-DPERL_MEM_LOG_ENV>. Then you need to explicitly set C<PERL_MEM_LOG> and/or
3241C<PERL_SV_LOG> to a non-zero value to enable logging of memory and/or SV
3242allocations.
3243
51a35ef1
JH
3244=head2 Profiling
3245
3246Depending on your platform there are various of profiling Perl.
3247
3248There are two commonly used techniques of profiling executables:
10f58044 3249I<statistical time-sampling> and I<basic-block counting>.
51a35ef1
JH
3250
3251The first method takes periodically samples of the CPU program
3252counter, and since the program counter can be correlated with the code
3253generated for functions, we get a statistical view of in which
3254functions the program is spending its time. The caveats are that very
3255small/fast functions have lower probability of showing up in the
3256profile, and that periodically interrupting the program (this is
3257usually done rather frequently, in the scale of milliseconds) imposes
3258an additional overhead that may skew the results. The first problem
3259can be alleviated by running the code for longer (in general this is a
3260good idea for profiling), the second problem is usually kept in guard
3261by the profiling tools themselves.
3262
10f58044 3263The second method divides up the generated code into I<basic blocks>.
51a35ef1
JH
3264Basic blocks are sections of code that are entered only in the
3265beginning and exited only at the end. For example, a conditional jump
3266starts a basic block. Basic block profiling usually works by
10f58044 3267I<instrumenting> the code by adding I<enter basic block #nnnn>
51a35ef1
JH
3268book-keeping code to the generated code. During the execution of the
3269code the basic block counters are then updated appropriately. The
3270caveat is that the added extra code can skew the results: again, the
3271profiling tools usually try to factor their own effects out of the
3272results.
3273
83f0ef60
JH
3274=head2 Gprof Profiling
3275
51a35ef1
JH
3276gprof is a profiling tool available in many UNIX platforms,
3277it uses F<statistical time-sampling>.
83f0ef60
JH
3278
3279You can build a profiled version of perl called "perl.gprof" by
51a35ef1
JH
3280invoking the make target "perl.gprof" (What is required is that Perl
3281must be compiled using the C<-pg> flag, you may need to re-Configure).
3282Running the profiled version of Perl will create an output file called
3283F<gmon.out> is created which contains the profiling data collected
3284during the execution.
83f0ef60
JH
3285
3286The gprof tool can then display the collected data in various ways.
3287Usually gprof understands the following options:
3288
3289=over 4
3290
3291=item -a
3292
3293Suppress statically defined functions from the profile.
3294
3295=item -b
3296
3297Suppress the verbose descriptions in the profile.
3298
3299=item -e routine
3300
3301Exclude the given routine and its descendants from the profile.
3302
3303=item -f routine
3304
3305Display only the given routine and its descendants in the profile.
3306
3307=item -s
3308
3309Generate a summary file called F<gmon.sum> which then may be given
3310to subsequent gprof runs to accumulate data over several runs.
3311
3312=item -z
3313
3314Display routines that have zero usage.
3315
3316=back
3317
3318For more detailed explanation of the available commands and output
3319formats, see your own local documentation of gprof.
3320
038c294a 3321quick hint:
07aa3531
JC
3322
3323 $ sh Configure -des -Dusedevel -Doptimize='-g' -Accflags='-pg' -Aldflags='-pg' && make
3324 $ ./perl someprog # creates gmon.out in current directory
3325 $ gprof perl > out
3326 $ view out
3327
51a35ef1
JH
3328=head2 GCC gcov Profiling
3329
10f58044 3330Starting from GCC 3.0 I<basic block profiling> is officially available
51a35ef1
JH
3331for the GNU CC.
3332
3333You can build a profiled version of perl called F<perl.gcov> by
3334invoking the make target "perl.gcov" (what is required that Perl must
3335be compiled using gcc with the flags C<-fprofile-arcs
3336-ftest-coverage>, you may need to re-Configure).
3337
3338Running the profiled version of Perl will cause profile output to be
3339generated. For each source file an accompanying ".da" file will be
3340created.
3341
3342To display the results you use the "gcov" utility (which should
3343be installed if you have gcc 3.0 or newer installed). F<gcov> is
3344run on source code files, like this
3345
3346 gcov sv.c
3347
3348which will cause F<sv.c.gcov> to be created. The F<.gcov> files
3349contain the source code annotated with relative frequencies of
3350execution indicated by "#" markers.
3351
3352Useful options of F<gcov> include C<-b> which will summarise the
3353basic block, branch, and function call coverage, and C<-c> which
3354instead of relative frequencies will use the actual counts. For
3355more information on the use of F<gcov> and basic block profiling
3356with gcc, see the latest GNU CC manual, as of GCC 3.0 see
3357
3358 http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc.html
3359
3360and its section titled "8. gcov: a Test Coverage Program"
3361
3362 http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html#SEC132
3363
07aa3531
JC
3364quick hint:
3365
3366 $ sh Configure -des -Doptimize='-g' -Accflags='-fprofile-arcs -ftest-coverage' \
3367 -Aldflags='-fprofile-arcs -ftest-coverage' && make perl.gcov
3368 $ rm -f regexec.c.gcov regexec.gcda
3369 $ ./perl.gcov
3370 $ gcov regexec.c
3371 $ view regexec.c.gcov
3372
4ae3d70a
JH
3373=head2 Pixie Profiling
3374
51a35ef1
JH
3375Pixie is a profiling tool available on IRIX and Tru64 (aka Digital
3376UNIX aka DEC OSF/1) platforms. Pixie does its profiling using
10f58044 3377I<basic-block counting>.
4ae3d70a 3378
83f0ef60 3379You can build a profiled version of perl called F<perl.pixie> by
51a35ef1
JH
3380invoking the make target "perl.pixie" (what is required is that Perl
3381must be compiled using the C<-g> flag, you may need to re-Configure).
3382
3383In Tru64 a file called F<perl.Addrs> will also be silently created,
3384this file contains the addresses of the basic blocks. Running the
3385profiled version of Perl will create a new file called "perl.Counts"
3386which contains the counts for the basic block for that particular
3387program execution.
4ae3d70a 3388
51a35ef1 3389To display the results you use the F<prof> utility. The exact
4ae3d70a
JH
3390incantation depends on your operating system, "prof perl.Counts" in
3391IRIX, and "prof -pixie -all -L. perl" in Tru64.
3392
6c41479b
JH
3393In IRIX the following prof options are available:
3394
3395=over 4
3396
3397=item -h
3398
3399Reports the most heavily used lines in descending order of use.
6e36760b 3400Useful for finding the hotspot lines.
6c41479b
JH
3401
3402=item -l
3403
3404Groups lines by procedure, with procedures sorted in descending order of use.
3405Within a procedure, lines are listed in source order.
6e36760b 3406Useful for finding the hotspots of procedures.
6c41479b
JH
3407
3408=back
3409
3410In Tru64 the following options are available:
3411
3412=over 4
3413
3958b146 3414=item -p[rocedures]
6c41479b 3415
3958b146 3416Procedures sorted in descending order by the number of cycles executed
6e36760b 3417in each procedure. Useful for finding the hotspot procedures.
6c41479b
JH
3418(This is the default option.)
3419
24000d2f 3420=item -h[eavy]
6c41479b 3421
6e36760b
JH
3422Lines sorted in descending order by the number of cycles executed in
3423each line. Useful for finding the hotspot lines.
6c41479b 3424
24000d2f 3425=item -i[nvocations]
6c41479b 3426
6e36760b
JH
3427The called procedures are sorted in descending order by number of calls
3428made to the procedures. Useful for finding the most used procedures.
6c41479b 3429
24000d2f 3430=item -l[ines]
6c41479b
JH
3431
3432Grouped by procedure, sorted by cycles executed per procedure.
6e36760b 3433Useful for finding the hotspots of procedures.
6c41479b
JH
3434
3435=item -testcoverage
3436
3437The compiler emitted code for these lines, but the code was unexecuted.
3438
24000d2f 3439=item -z[ero]
6c41479b
JH
3440
3441Unexecuted procedures.
3442
aa500c9e 3443=back
6c41479b
JH
3444
3445For further information, see your system's manual pages for pixie and prof.
4ae3d70a 3446
b8ddf6b3
SB
3447=head2 Miscellaneous tricks
3448
3449=over 4
3450
3451=item *
3452
cc177e1a 3453Those debugging perl with the DDD frontend over gdb may find the
b8ddf6b3
SB
3454following useful:
3455
3456You can extend the data conversion shortcuts menu, so for example you
3457can display an SV's IV value with one click, without doing any typing.
3458To do that simply edit ~/.ddd/init file and add after:
3459
3460 ! Display shortcuts.
3461 Ddd*gdbDisplayShortcuts: \
3462 /t () // Convert to Bin\n\
3463 /d () // Convert to Dec\n\
3464 /x () // Convert to Hex\n\
3465 /o () // Convert to Oct(\n\
3466
3467the following two lines:
3468
3469 ((XPV*) (())->sv_any )->xpv_pv // 2pvx\n\
3470 ((XPVIV*) (())->sv_any )->xiv_iv // 2ivx
3471
3472so now you can do ivx and pvx lookups or you can plug there the
3473sv_peek "conversion":
3474
3475 Perl_sv_peek(my_perl, (SV*)()) // sv_peek
3476
3477(The my_perl is for threaded builds.)
3478Just remember that every line, but the last one, should end with \n\
3479
3480Alternatively edit the init file interactively via:
34813rd mouse button -> New Display -> Edit Menu
3482
3483Note: you can define up to 20 conversion shortcuts in the gdb
3484section.
3485
9965345d
JH
3486=item *
3487
7e337ee0
JH
3488If you see in a debugger a memory area mysteriously full of 0xABABABAB
3489or 0xEFEFEFEF, you may be seeing the effect of the Poison() macros,
3490see L<perlclib>.
9965345d 3491
f1fac472
NC
3492=item *
3493
3494Under ithreads the optree is read only. If you want to enforce this, to check
3495for write accesses from buggy code, compile with C<-DPL_OP_SLAB_ALLOC> to
3496enable the OP slab allocator and C<-DPERL_DEBUG_READONLY_OPS> to enable code
3497that allocates op memory via C<mmap>, and sets it read-only at run time.
3498Any write access to an op results in a C<SIGBUS> and abort.
3499
3500This code is intended for development only, and may not be portable even to
3501all Unix variants. Also, it is an 80% solution, in that it isn't able to make
3502all ops read only. Specifically it
3503
3504=over
3505
3506=item 1
3507
3508Only sets read-only on all slabs of ops at C<CHECK> time, hence ops allocated
3509later via C<require> or C<eval> will be re-write
3510
3511=item 2
3512
3513Turns an entire slab of ops read-write if the refcount of any op in the slab
3514needs to be decreased.
3515
3516=item 3
3517
3518Turns an entire slab of ops read-write if any op from the slab is freed.
3519
b8ddf6b3
SB
3520=back
3521
f1fac472
NC
3522It's not possible to turn the slabs to read-only after an action requiring
3523read-write access, as either can happen during op tree building time, so
3524there may still be legitimate write access.
3525
3526However, as an 80% solution it is still effective, as currently it catches
3527a write access during the generation of F<Config.pm>, which means that we
3528can't yet build F<perl> with this enabled.
3529
3530=back
3531
3532
955fec6b 3533=head1 CONCLUSION
a422fd2d 3534
955fec6b
JH
3535We've had a brief look around the Perl source, how to maintain quality
3536of the source code, an overview of the stages F<perl> goes through
3537when it's running your code, how to use debuggers to poke at the Perl
3538guts, and finally how to analyse the execution of Perl. We took a very
3539simple problem and demonstrated how to solve it fully - with
3540documentation, regression tests, and finally a patch for submission to
3541p5p. Finally, we talked about how to use external tools to debug and
3542test Perl.
a422fd2d
SC
3543
3544I'd now suggest you read over those references again, and then, as soon
3545as possible, get your hands dirty. The best way to learn is by doing,
07aa3531 3546so:
a422fd2d
SC
3547
3548=over 3
3549
3550=item *
3551
3552Subscribe to perl5-porters, follow the patches and try and understand
3553them; don't be afraid to ask if there's a portion you're not clear on -
3554who knows, you may unearth a bug in the patch...
3555
3556=item *
3557
3558Keep up to date with the bleeding edge Perl distributions and get
3559familiar with the changes. Try and get an idea of what areas people are
3560working on and the changes they're making.
3561
3562=item *
3563
3e148164 3564Do read the README associated with your operating system, e.g. README.aix
a1f349fd
MB
3565on the IBM AIX OS. Don't hesitate to supply patches to that README if
3566you find anything missing or changed over a new OS release.
3567
3568=item *
3569
a422fd2d
SC
3570Find an area of Perl that seems interesting to you, and see if you can
3571work out how it works. Scan through the source, and step over it in the
3572debugger. Play, poke, investigate, fiddle! You'll probably get to
3573understand not just your chosen area but a much wider range of F<perl>'s
3574activity as well, and probably sooner than you'd think.
3575
3576=back
3577
3578=over 3
3579
3580=item I<The Road goes ever on and on, down from the door where it began.>
3581
3582=back
3583
64d9b66b 3584If you can do these things, you've started on the long road to Perl porting.
a422fd2d
SC
3585Thanks for wanting to help make Perl better - and happy hacking!
3586
e8cd7eae
GS
3587=head1 AUTHOR
3588
3589This document was written by Nathan Torkington, and is maintained by
3590the perl5-porters mailing list.