This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Rewrite the tests section of Makefile to be less redundant
[perl5.git] / pod / perlhack.pod
CommitLineData
e8cd7eae
GS
1=head1 NAME
2
3perlhack - How to hack at the Perl internals
4
5=head1 DESCRIPTION
6
7This document attempts to explain how Perl development takes place,
8and ends with some suggestions for people wanting to become bona fide
9porters.
10
11The perl5-porters mailing list is where the Perl standard distribution
12is maintained and developed. The list can get anywhere from 10 to 150
13messages a day, depending on the heatedness of the debate. Most days
14there are two or three patches, extensions, features, or bugs being
15discussed at a time.
16
17A searchable archive of the list is at:
18
19 http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
20
21The list is also archived under the usenet group name
22C<perl.porters-gw> at:
23
24 http://www.deja.com/
25
26List subscribers (the porters themselves) come in several flavours.
27Some are quiet curious lurkers, who rarely pitch in and instead watch
28the ongoing development to ensure they're forewarned of new changes or
29features in Perl. Some are representatives of vendors, who are there
30to make sure that Perl continues to compile and work on their
31platforms. Some patch any reported bug that they know how to fix,
32some are actively patching their pet area (threads, Win32, the regexp
33engine), while others seem to do nothing but complain. In other
34words, it's your usual mix of technical people.
35
36Over this group of porters presides Larry Wall. He has the final word
f6c51b38
GS
37in what does and does not change in the Perl language. Various
38releases of Perl are shepherded by a ``pumpking'', a porter
39responsible for gathering patches, deciding on a patch-by-patch
40feature-by-feature basis what will and will not go into the release.
41For instance, Gurusamy Sarathy is the pumpking for the 5.6 release of
42Perl.
e8cd7eae
GS
43
44In addition, various people are pumpkings for different things. For
45instance, Andy Dougherty and Jarkko Hietaniemi share the I<Configure>
46pumpkin, and Tom Christiansen is the documentation pumpking.
47
48Larry sees Perl development along the lines of the US government:
49there's the Legislature (the porters), the Executive branch (the
50pumpkings), and the Supreme Court (Larry). The legislature can
51discuss and submit patches to the executive branch all they like, but
52the executive branch is free to veto them. Rarely, the Supreme Court
53will side with the executive branch over the legislature, or the
54legislature over the executive branch. Mostly, however, the
55legislature and the executive branch are supposed to get along and
56work out their differences without impeachment or court cases.
57
58You might sometimes see reference to Rule 1 and Rule 2. Larry's power
59as Supreme Court is expressed in The Rules:
60
61=over 4
62
63=item 1
64
65Larry is always by definition right about how Perl should behave.
66This means he has final veto power on the core functionality.
67
68=item 2
69
70Larry is allowed to change his mind about any matter at a later date,
71regardless of whether he previously invoked Rule 1.
72
73=back
74
75Got that? Larry is always right, even when he was wrong. It's rare
76to see either Rule exercised, but they are often alluded to.
77
78New features and extensions to the language are contentious, because
79the criteria used by the pumpkings, Larry, and other porters to decide
80which features should be implemented and incorporated are not codified
81in a few small design goals as with some other languages. Instead,
82the heuristics are flexible and often difficult to fathom. Here is
83one person's list, roughly in decreasing order of importance, of
84heuristics that new features have to be weighed against:
85
86=over 4
87
88=item Does concept match the general goals of Perl?
89
90These haven't been written anywhere in stone, but one approximation
91is:
92
93 1. Keep it fast, simple, and useful.
94 2. Keep features/concepts as orthogonal as possible.
95 3. No arbitrary limits (platforms, data sizes, cultures).
96 4. Keep it open and exciting to use/patch/advocate Perl everywhere.
97 5. Either assimilate new technologies, or build bridges to them.
98
99=item Where is the implementation?
100
101All the talk in the world is useless without an implementation. In
102almost every case, the person or people who argue for a new feature
103will be expected to be the ones who implement it. Porters capable
104of coding new features have their own agendas, and are not available
105to implement your (possibly good) idea.
106
107=item Backwards compatibility
108
109It's a cardinal sin to break existing Perl programs. New warnings are
110contentious--some say that a program that emits warnings is not
111broken, while others say it is. Adding keywords has the potential to
112break programs, changing the meaning of existing token sequences or
113functions might break programs.
114
115=item Could it be a module instead?
116
117Perl 5 has extension mechanisms, modules and XS, specifically to avoid
118the need to keep changing the Perl interpreter. You can write modules
119that export functions, you can give those functions prototypes so they
120can be called like built-in functions, you can even write XS code to
121mess with the runtime data structures of the Perl interpreter if you
122want to implement really complicated things. If it can be done in a
123module instead of in the core, it's highly unlikely to be added.
124
125=item Is the feature generic enough?
126
127Is this something that only the submitter wants added to the language,
128or would it be broadly useful? Sometimes, instead of adding a feature
129with a tight focus, the porters might decide to wait until someone
130implements the more generalized feature. For instance, instead of
131implementing a ``delayed evaluation'' feature, the porters are waiting
132for a macro system that would permit delayed evaluation and much more.
133
134=item Does it potentially introduce new bugs?
135
136Radical rewrites of large chunks of the Perl interpreter have the
137potential to introduce new bugs. The smaller and more localized the
138change, the better.
139
140=item Does it preclude other desirable features?
141
142A patch is likely to be rejected if it closes off future avenues of
143development. For instance, a patch that placed a true and final
144interpretation on prototypes is likely to be rejected because there
145are still options for the future of prototypes that haven't been
146addressed.
147
148=item Is the implementation robust?
149
150Good patches (tight code, complete, correct) stand more chance of
151going in. Sloppy or incorrect patches might be placed on the back
152burner until the pumpking has time to fix, or might be discarded
153altogether without further notice.
154
155=item Is the implementation generic enough to be portable?
156
157The worst patches make use of a system-specific features. It's highly
158unlikely that nonportable additions to the Perl language will be
159accepted.
160
161=item Is there enough documentation?
162
163Patches without documentation are probably ill-thought out or
164incomplete. Nothing can be added without documentation, so submitting
165a patch for the appropriate manpages as well as the source code is
166always a good idea. If appropriate, patches should add to the test
167suite as well.
168
169=item Is there another way to do it?
170
171Larry said ``Although the Perl Slogan is I<There's More Than One Way
172to Do It>, I hesitate to make 10 ways to do something''. This is a
173tricky heuristic to navigate, though--one man's essential addition is
174another man's pointless cruft.
175
176=item Does it create too much work?
177
178Work for the pumpking, work for Perl programmers, work for module
179authors, ... Perl is supposed to be easy.
180
f6c51b38
GS
181=item Patches speak louder than words
182
183Working code is always preferred to pie-in-the-sky ideas. A patch to
184add a feature stands a much higher chance of making it to the language
185than does a random feature request, no matter how fervently argued the
186request might be. This ties into ``Will it be useful?'', as the fact
187that someone took the time to make the patch demonstrates a strong
188desire for the feature.
189
e8cd7eae
GS
190=back
191
192If you're on the list, you might hear the word ``core'' bandied
193around. It refers to the standard distribution. ``Hacking on the
194core'' means you're changing the C source code to the Perl
195interpreter. ``A core module'' is one that ships with Perl.
196
a1f349fd
MB
197=head2 Keeping in sync
198
e8cd7eae
GS
199The source code to the Perl interpreter, in its different versions, is
200kept in a repository managed by a revision control system (which is
201currently the Perforce program, see http://perforce.com/). The
202pumpkings and a few others have access to the repository to check in
203changes. Periodically the pumpking for the development version of Perl
204will release a new version, so the rest of the porters can see what's
2be4c08b
GS
205changed. The current state of the main trunk of repository, and patches
206that describe the individual changes that have happened since the last
207public release are available at this location:
208
209 ftp://ftp.linux.activestate.com/pub/staff/gsar/APC/
210
a1f349fd
MB
211If you are a member of the perl5-porters mailing list, it is a good
212thing to keep in touch with the most recent changes. If not only to
213verify if what you would have posted as a bug report isn't already
214solved in the most recent available perl development branch, also
215known as perl-current, bleading edge perl, bleedperl or bleadperl.
2be4c08b
GS
216
217Needless to say, the source code in perl-current is usually in a perpetual
218state of evolution. You should expect it to be very buggy. Do B<not> use
219it for any purpose other than testing and development.
e8cd7eae 220
3e148164
JH
221Keeping in sync with the most recent branch can be done in several ways,
222but the most convenient and reliable way is using B<rsync>, available at
223ftp://rsync.samba.org/pub/rsync/ . (You can also get the most recent
224branch by FTP.)
a1f349fd
MB
225
226If you choose to keep in sync using rsync, there are two approaches
3e148164 227to doing so:
a1f349fd
MB
228
229=over 4
230
231=item rsync'ing the source tree
232
3e148164 233Presuming you are in the directory where your perl source resides
a1f349fd
MB
234and you have rsync installed and available, you can `upgrade' to
235the bleadperl using:
236
237 # rsync -avz rsync://ftp.linux.activestate.com/perl-current/ .
238
239This takes care of updating every single item in the source tree to
240the latest applied patch level, creating files that are new (to your
241distribution) and setting date/time stamps of existing files to
242reflect the bleadperl status.
243
c6d0653e
LC
244Note that this will not delete any files that were in '.' before
245the rsync. Once you are sure that the rsync is running correctly,
246run it with the --delete and the --dry-run options like this:
247
248 # rsync -avz --delete --dry-run rsync://ftp.linux.activestate.com/perl-current/ .
249
250This will I<simulate> an rsync run that also deletes files not
251present in the bleadperl master copy. Observe the results from
252this run closely. If you are sure that the actual run would delete
253no files precious to you, you could remove the '--dry-run' option.
254
a1f349fd
MB
255You can than check what patch was the latest that was applied by
256looking in the file B<.patch>, which will show the number of the
257latest patch.
258
259If you have more than one machine to keep in sync, and not all of
260them have access to the WAN (so you are not able to rsync all the
261source trees to the real source), there are some ways to get around
262this problem.
263
264=over 4
265
266=item Using rsync over the LAN
267
268Set up a local rsync server which makes the rsynced source tree
3e148164 269available to the LAN and sync the other machines against this
a1f349fd
MB
270directory.
271
272From http://rsync.samba.org/README.html:
273
274 "Rsync uses rsh or ssh for communication. It does not need to be
275 setuid and requires no special privileges for installation. It
276 does not require a inetd entry or a deamon. You must, however,
277 have a working rsh or ssh system. Using ssh is recommended for
278 its security features."
279
280=item Using pushing over the NFS
281
282Having the other systems mounted over the NFS, you can take an
3e148164
JH
283active pushing approach by checking the just updated tree against
284the other not-yet synced trees. An example would be
285
286 #!/usr/bin/perl -w
287
288 use strict;
289 use File::Copy;
290
291 my %MF = map {
292 m/(\S+)/;
293 $1 => [ (stat $1)[2, 7, 9] ]; # mode, size, mtime
294 } `cat MANIFEST`;
295
296 my %remote = map { $_ => "/$_/pro/3gl/CPAN/perl-5.7.1" } qw(host1 host2);
297
298 foreach my $host (keys %remote) {
299 unless (-d $remote{$host}) {
300 print STDERR "Cannot Xsync for host $host\n";
301 next;
302 }
303 foreach my $file (keys %MF) {
304 my $rfile = "$remote{$host}/$file";
305 my ($mode, $size, $mtime) = (stat $rfile)[2, 7, 9];
306 defined $size or ($mode, $size, $mtime) = (0, 0, 0);
307 $size == $MF{$file}[1] && $mtime == $MF{$file}[2] and next;
308 printf "%4s %-34s %8d %9d %8d %9d\n",
309 $host, $file, $MF{$file}[1], $MF{$file}[2], $size, $mtime;
310 unlink $rfile;
311 copy ($file, $rfile);
312 utime time, $MF{$file}[2], $rfile;
313 chmod $MF{$file}[0], $rfile;
314 }
315 }
316
317though this is not perfect. It could be improved with checking
a1f349fd
MB
318file checksums before updating. Not all NFS systems support
319reliable utime support (when used over the NFS).
320
321=back
322
323=item rsync'ing the patches
324
325The source tree is maintained by the pumpking who applies patches to
326the files in the tree. These patches are either created by the
327pumpking himself using C<diff -c> after updating the file manually or
328by applying patches sent in by posters on the perl5-porters list.
329These patches are also saved and rsync'able, so you can apply them
330yourself to the source files.
331
332Presuming you are in a directory where your patches reside, you can
3e148164 333get them in sync with
a1f349fd
MB
334
335 # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ .
336
337This makes sure the latest available patch is downloaded to your
338patch directory.
339
3e148164 340It's then up to you to apply these patches, using something like
a1f349fd
MB
341
342 # last=`ls -rt1 *.gz | tail -1`
343 # rsync -avz rsync://ftp.linux.activestate.com/perl-current-diffs/ .
344 # find . -name '*.gz' -newer $last -exec gzcat {} \; >blead.patch
345 # cd ../perl-current
346 # patch -p1 -N <../perl-current-diffs/blead.patch
347
348or, since this is only a hint towards how it works, use CPAN-patchaperl
349from Andreas K├Ânig to have better control over the patching process.
350
351=back
352
dcff5598 353=head3 Why rsync the source tree
a1f349fd
MB
354
355=over 4
356
357=item It's easier
358
359Since you don't have to apply the patches yourself, you are sure all
360files in the source tree are in the right state.
361
362=item It's more recent
363
364According to Gurusamy Sarathy:
365
366 "... The rsync mirror is automatic and syncs with the repository
367 every five minutes.
368
3e148164 369 "Updating the patch area still requires manual intervention
a1f349fd
MB
370 (with all the goofiness that implies, which you've noted) and
371 is typically on a daily cycle. Making this process automatic
372 is on my tuit list, but don't ask me when."
373
374=item It's more reliable
375
3e148164 376Well, since the patches are updated by hand, I don't have to say any
a1f349fd
MB
377more ... (see Sarathy's remark).
378
379=back
380
dcff5598 381=head3 Why rsync the patches
a1f349fd
MB
382
383=over 4
384
385=item It's easier
386
387If you have more than one machine that you want to keep in track with
3e148164 388bleadperl, it's easier to rsync the patches only once and then apply
a1f349fd
MB
389them to all the source trees on the different machines.
390
391In case you try to keep in pace on 5 different machines, for which
392only one of them has access to the WAN, rsync'ing all the source
3e148164 393trees should than be done 5 times over the NFS. Having
a1f349fd 394rsync'ed the patches only once, I can apply them to all the source
3e148164 395trees automatically. Need you say more ;-)
a1f349fd
MB
396
397=item It's a good reference
398
399If you do not only like to have the most recent development branch,
400but also like to B<fix> bugs, or extend features, you want to dive
401into the sources. If you are a seasoned perl core diver, you don't
402need no manuals, tips, roadmaps, perlguts.pod or other aids to find
403your way around. But if you are a starter, the patches may help you
404in finding where you should start and how to change the bits that
405bug you.
406
407The file B<Changes> is updated on occasions the pumpking sees as his
408own little sync points. On those occasions, he releases a tar-ball of
409the current source tree (i.e. perl@7582.tar.gz), which will be an
410excellent point to start with when choosing to use the 'rsync the
411patches' scheme. Starting with perl@7582, which means a set of source
412files on which the latest applied patch is number 7582, you apply all
413succeeding patches available from than on (7583, 7584, ...).
414
415You can use the patches later as a kind of search archive.
416
417=over 4
418
419=item Finding a start point
420
421If you want to fix/change the behaviour of function/feature Foo, just
422scan the patches for patches that mention Foo either in the subject,
3e148164 423the comments, or the body of the fix. A good chance the patch shows
a1f349fd
MB
424you the files that are affected by that patch which are very likely
425to be the starting point of your journey into the guts of perl.
426
427=item Finding how to fix a bug
428
429If you've found I<where> the function/feature Foo misbehaves, but you
430don't know how to fix it (but you do know the change you want to
431make), you can, again, peruse the patches for similar changes and
432look how others apply the fix.
433
434=item Finding the source of misbehaviour
435
436When you keep in sync with bleadperl, the pumpking would love to
437I<see> that the community efforts realy work. So after each of his
438sync points, you are to 'make test' to check if everything is still
439in working order. If it is, you do 'make ok', which will send an OK
440report to perlbug@perl.org. (If you do not have access to a mailer
3e148164 441from the system you just finished successfully 'make test', you can
a1f349fd
MB
442do 'make okfile', which creates the file C<perl.ok>, which you can
443than take to your favourite mailer and mail yourself).
444
3e148164 445But of course, as always, things will not allways lead to a success
a1f349fd
MB
446path, and one or more test do not pass the 'make test'. Before
447sending in a bug report (using 'make nok' or 'make nokfile'), check
448the mailing list if someone else has reported the bug already and if
449so, confirm it by replying to that message. If not, you might want to
450trace the source of that misbehaviour B<before> sending in the bug,
451which will help all the other porters in finding the solution.
452
3e148164
JH
453Here the saved patches come in very handy. You can check the list of
454patches to see which patch changed what file and what change caused
455the misbehaviour. If you note that in the bug report, it saves the
456one trying to solve it, looking for that point.
a1f349fd
MB
457
458=back
459
460If searching the patches is too bothersome, you might consider using
461perl's bugtron to find more information about discussions and
462ramblings on posted bugs.
463
464=back
465
3e148164
JH
466If you want to get the best of both worlds, rsync both the source
467tree for convenience, reliability and ease and rsync the patches
468for reference.
469
a1f349fd
MB
470=head2 Submitting patches
471
e8cd7eae
GS
472Always submit patches to I<perl5-porters@perl.org>. This lets other
473porters review your patch, which catches a surprising number of errors
f6c51b38
GS
474in patches. Either use the diff program (available in source code
475form from I<ftp://ftp.gnu.org/pub/gnu/>), or use Johan Vromans'
476I<makepatch> (available from I<CPAN/authors/id/JV/>). Unified diffs
477are preferred, but context diffs are accepted. Do not send RCS-style
478diffs or diffs without context lines. More information is given in
479the I<Porting/patching.pod> file in the Perl source distribution.
480Please patch against the latest B<development> version (e.g., if
481you're fixing a bug in the 5.005 track, patch against the latest
4825.005_5x version). Only patches that survive the heat of the
483development branch get applied to maintenance versions.
e8cd7eae
GS
484
485Your patch should update the documentation and test suite.
486
487To report a bug in Perl, use the program I<perlbug> which comes with
488Perl (if you can't get Perl to work, send mail to the address
489I<perlbug@perl.com> or I<perlbug@perl.org>). Reporting bugs through
490I<perlbug> feeds into the automated bug-tracking system, access to
491which is provided through the web at I<http://bugs.perl.org/>. It
492often pays to check the archives of the perl5-porters mailing list to
493see whether the bug you're reporting has been reported before, and if
494so whether it was considered a bug. See above for the location of
495the searchable archives.
496
497The CPAN testers (I<http://testers.cpan.org/>) are a group of
498volunteers who test CPAN modules on a variety of platforms. Perl Labs
f6c51b38
GS
499(I<http://labs.perl.org/>) automatically tests Perl source releases on
500platforms and gives feedback to the CPAN testers mailing list. Both
501efforts welcome volunteers.
e8cd7eae 502
e8cd7eae
GS
503It's a good idea to read and lurk for a while before chipping in.
504That way you'll get to see the dynamic of the conversations, learn the
505personalities of the players, and hopefully be better prepared to make
506a useful contribution when do you speak up.
507
508If after all this you still think you want to join the perl5-porters
f6c51b38
GS
509mailing list, send mail to I<perl5-porters-subscribe@perl.org>. To
510unsubscribe, send mail to I<perl5-porters-unsubscribe@perl.org>.
e8cd7eae 511
a422fd2d
SC
512To hack on the Perl guts, you'll need to read the following things:
513
514=over 3
515
516=item L<perlguts>
517
518This is of paramount importance, since it's the documentation of what
519goes where in the Perl source. Read it over a couple of times and it
520might start to make sense - don't worry if it doesn't yet, because the
521best way to study it is to read it in conjunction with poking at Perl
522source, and we'll do that later on.
523
524You might also want to look at Gisle Aas's illustrated perlguts -
525there's no guarantee that this will be absolutely up-to-date with the
526latest documentation in the Perl core, but the fundamentals will be
527right. (http://gisle.aas.no/perl/illguts/)
528
529=item L<perlxstut> and L<perlxs>
530
531A working knowledge of XSUB programming is incredibly useful for core
532hacking; XSUBs use techniques drawn from the PP code, the portion of the
533guts that actually executes a Perl program. It's a lot gentler to learn
534those techniques from simple examples and explanation than from the core
535itself.
536
537=item L<perlapi>
538
539The documentation for the Perl API explains what some of the internal
540functions do, as well as the many macros used in the source.
541
542=item F<Porting/pumpkin.pod>
543
544This is a collection of words of wisdom for a Perl porter; some of it is
545only useful to the pumpkin holder, but most of it applies to anyone
546wanting to go about Perl development.
547
548=item The perl5-porters FAQ
549
550This is posted to perl5-porters at the beginning on every month, and
551should be available from http://perlhacker.org/p5p-faq; alternatively,
552you can get the FAQ emailed to you by sending mail to
553C<perl5-porters-faq@perl.org>. It contains hints on reading
554perl5-porters, information on how perl5-porters works and how Perl
555development in general works.
556
557=back
558
559=head2 Finding Your Way Around
560
561Perl maintenance can be split into a number of areas, and certain people
562(pumpkins) will have responsibility for each area. These areas sometimes
563correspond to files or directories in the source kit. Among the areas are:
564
565=over 3
566
567=item Core modules
568
569Modules shipped as part of the Perl core live in the F<lib/> and F<ext/>
570subdirectories: F<lib/> is for the pure-Perl modules, and F<ext/>
571contains the core XS modules.
572
573=item Documentation
574
575Documentation maintenance includes looking after everything in the
576F<pod/> directory, (as well as contributing new documentation) and
577the documentation to the modules in core.
578
579=item Configure
580
581The configure process is the way we make Perl portable across the
582myriad of operating systems it supports. Responsibility for the
583configure, build and installation process, as well as the overall
584portability of the core code rests with the configure pumpkin - others
585help out with individual operating systems.
586
587The files involved are the operating system directories, (F<win32/>,
588F<os2/>, F<vms/> and so on) the shell scripts which generate F<config.h>
589and F<Makefile>, as well as the metaconfig files which generate
590F<Configure>. (metaconfig isn't included in the core distribution.)
591
592=item Interpreter
593
594And of course, there's the core of the Perl interpreter itself. Let's
595have a look at that in a little more detail.
596
597=back
598
599Before we leave looking at the layout, though, don't forget that
600F<MANIFEST> contains not only the file names in the Perl distribution,
601but short descriptions of what's in them, too. For an overview of the
602important files, try this:
603
604 perl -lne 'print if /^[^\/]+\.[ch]\s+/' MANIFEST
605
606=head2 Elements of the interpreter
607
608The work of the interpreter has two main stages: compiling the code
609into the internal representation, or bytecode, and then executing it.
610L<perlguts/Compiled code> explains exactly how the compilation stage
611happens.
612
613Here is a short breakdown of perl's operation:
614
615=over 3
616
617=item Startup
618
619The action begins in F<perlmain.c>. (or F<miniperlmain.c> for miniperl)
620This is very high-level code, enough to fit on a single screen, and it
621resembles the code found in L<perlembed>; most of the real action takes
622place in F<perl.c>
623
624First, F<perlmain.c> allocates some memory and constructs a Perl
625interpreter:
626
627 1 PERL_SYS_INIT3(&argc,&argv,&env);
628 2
629 3 if (!PL_do_undump) {
630 4 my_perl = perl_alloc();
631 5 if (!my_perl)
632 6 exit(1);
633 7 perl_construct(my_perl);
634 8 PL_perl_destruct_level = 0;
635 9 }
636
637Line 1 is a macro, and its definition is dependent on your operating
638system. Line 3 references C<PL_do_undump>, a global variable - all
639global variables in Perl start with C<PL_>. This tells you whether the
640current running program was created with the C<-u> flag to perl and then
641F<undump>, which means it's going to be false in any sane context.
642
643Line 4 calls a function in F<perl.c> to allocate memory for a Perl
644interpreter. It's quite a simple function, and the guts of it looks like
645this:
646
647 my_perl = (PerlInterpreter*)PerlMem_malloc(sizeof(PerlInterpreter));
648
649Here you see an example of Perl's system abstraction, which we'll see
650later: C<PerlMem_malloc> is either your system's C<malloc>, or Perl's
651own C<malloc> as defined in F<malloc.c> if you selected that option at
652configure time.
653
654Next, in line 7, we construct the interpreter; this sets up all the
655special variables that Perl needs, the stacks, and so on.
656
657Now we pass Perl the command line options, and tell it to go:
658
659 exitstatus = perl_parse(my_perl, xs_init, argc, argv, (char **)NULL);
660 if (!exitstatus) {
661 exitstatus = perl_run(my_perl);
662 }
663
664
665C<perl_parse> is actually a wrapper around C<S_parse_body>, as defined
666in F<perl.c>, which processes the command line options, sets up any
667statically linked XS modules, opens the program and calls C<yyparse> to
668parse it.
669
670=item Parsing
671
672The aim of this stage is to take the Perl source, and turn it into an op
673tree. We'll see what one of those looks like later. Strictly speaking,
674there's three things going on here.
675
676C<yyparse>, the parser, lives in F<perly.c>, although you're better off
677reading the original YACC input in F<perly.y>. (Yes, Virginia, there
678B<is> a YACC grammar for Perl!) The job of the parser is to take your
679code and `understand' it, splitting it into sentences, deciding which
680operands go with which operators and so on.
681
682The parser is nobly assisted by the lexer, which chunks up your input
683into tokens, and decides what type of thing each token is: a variable
684name, an operator, a bareword, a subroutine, a core function, and so on.
685The main point of entry to the lexer is C<yylex>, and that and its
686associated routines can be found in F<toke.c>. Perl isn't much like
687other computer languages; it's highly context sensitive at times, it can
688be tricky to work out what sort of token something is, or where a token
689ends. As such, there's a lot of interplay between the tokeniser and the
690parser, which can get pretty frightening if you're not used to it.
691
692As the parser understands a Perl program, it builds up a tree of
693operations for the interpreter to perform during execution. The routines
694which construct and link together the various operations are to be found
695in F<op.c>, and will be examined later.
696
697=item Optimization
698
699Now the parsing stage is complete, and the finished tree represents
700the operations that the Perl interpreter needs to perform to execute our
701program. Next, Perl does a dry run over the tree looking for
702optimisations: constant expressions such as C<3 + 4> will be computed
703now, and the optimizer will also see if any multiple operations can be
704replaced with a single one. For instance, to fetch the variable C<$foo>,
705instead of grabbing the glob C<*foo> and looking at the scalar
706component, the optimizer fiddles the op tree to use a function which
707directly looks up the scalar in question. The main optimizer is C<peep>
708in F<op.c>, and many ops have their own optimizing functions.
709
710=item Running
711
712Now we're finally ready to go: we have compiled Perl byte code, and all
713that's left to do is run it. The actual execution is done by the
714C<runops_standard> function in F<run.c>; more specifically, it's done by
715these three innocent looking lines:
716
717 while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
718 PERL_ASYNC_CHECK();
719 }
720
721You may be more comfortable with the Perl version of that:
722
723 PERL_ASYNC_CHECK() while $Perl::op = &{$Perl::op->{function}};
724
725Well, maybe not. Anyway, each op contains a function pointer, which
726stipulates the function which will actually carry out the operation.
727This function will return the next op in the sequence - this allows for
728things like C<if> which choose the next op dynamically at run time.
729The C<PERL_ASYNC_CHECK> makes sure that things like signals interrupt
730execution if required.
731
732The actual functions called are known as PP code, and they're spread
733between four files: F<pp_hot.c> contains the `hot' code, which is most
734often used and highly optimized, F<pp_sys.c> contains all the
735system-specific functions, F<pp_ctl.c> contains the functions which
736implement control structures (C<if>, C<while> and the like) and F<pp.c>
737contains everything else. These are, if you like, the C code for Perl's
738built-in functions and operators.
739
740=back
741
742=head2 Internal Variable Types
743
744You should by now have had a look at L<perlguts>, which tells you about
745Perl's internal variable types: SVs, HVs, AVs and the rest. If not, do
746that now.
747
748These variables are used not only to represent Perl-space variables, but
749also any constants in the code, as well as some structures completely
750internal to Perl. The symbol table, for instance, is an ordinary Perl
751hash. Your code is represented by an SV as it's read into the parser;
752any program files you call are opened via ordinary Perl filehandles, and
753so on.
754
755The core L<Devel::Peek|Devel::Peek> module lets us examine SVs from a
756Perl program. Let's see, for instance, how Perl treats the constant
757C<"hello">.
758
759 % perl -MDevel::Peek -e 'Dump("hello")'
760 1 SV = PV(0xa041450) at 0xa04ecbc
761 2 REFCNT = 1
762 3 FLAGS = (POK,READONLY,pPOK)
763 4 PV = 0xa0484e0 "hello"\0
764 5 CUR = 5
765 6 LEN = 6
766
767Reading C<Devel::Peek> output takes a bit of practise, so let's go
768through it line by line.
769
770Line 1 tells us we're looking at an SV which lives at C<0xa04ecbc> in
771memory. SVs themselves are very simple structures, but they contain a
772pointer to a more complex structure. In this case, it's a PV, a
773structure which holds a string value, at location C<0xa041450>. Line 2
774is the reference count; there are no other references to this data, so
775it's 1.
776
777Line 3 are the flags for this SV - it's OK to use it as a PV, it's a
778read-only SV (because it's a constant) and the data is a PV internally.
779Next we've got the contents of the string, starting at location
780C<0xa0484e0>.
781
782Line 5 gives us the current length of the string - note that this does
783B<not> include the null terminator. Line 6 is not the length of the
784string, but the length of the currently allocated buffer; as the string
785grows, Perl automatically extends the available storage via a routine
786called C<SvGROW>.
787
788You can get at any of these quantities from C very easily; just add
789C<Sv> to the name of the field shown in the snippet, and you've got a
790macro which will return the value: C<SvCUR(sv)> returns the current
791length of the string, C<SvREFCOUNT(sv)> returns the reference count,
792C<SvPV(sv, len)> returns the string itself with its length, and so on.
793More macros to manipulate these properties can be found in L<perlguts>.
794
795Let's take an example of manipulating a PV, from C<sv_catpvn>, in F<sv.c>
796
797 1 void
798 2 Perl_sv_catpvn(pTHX_ register SV *sv, register const char *ptr, register STRLEN len)
799 3 {
800 4 STRLEN tlen;
801 5 char *junk;
802
803 6 junk = SvPV_force(sv, tlen);
804 7 SvGROW(sv, tlen + len + 1);
805 8 if (ptr == junk)
806 9 ptr = SvPVX(sv);
807 10 Move(ptr,SvPVX(sv)+tlen,len,char);
808 11 SvCUR(sv) += len;
809 12 *SvEND(sv) = '\0';
810 13 (void)SvPOK_only_UTF8(sv); /* validate pointer */
811 14 SvTAINT(sv);
812 15 }
813
814This is a function which adds a string, C<ptr>, of length C<len> onto
815the end of the PV stored in C<sv>. The first thing we do in line 6 is
816make sure that the SV B<has> a valid PV, by calling the C<SvPV_force>
817macro to force a PV. As a side effect, C<tlen> gets set to the current
818value of the PV, and the PV itself is returned to C<junk>.
819
b1866b2d 820In line 7, we make sure that the SV will have enough room to accommodate
a422fd2d
SC
821the old string, the new string and the null terminator. If C<LEN> isn't
822big enough, C<SvGROW> will reallocate space for us.
823
824Now, if C<junk> is the same as the string we're trying to add, we can
825grab the string directly from the SV; C<SvPVX> is the address of the PV
826in the SV.
827
828Line 10 does the actual catenation: the C<Move> macro moves a chunk of
829memory around: we move the string C<ptr> to the end of the PV - that's
830the start of the PV plus its current length. We're moving C<len> bytes
831of type C<char>. After doing so, we need to tell Perl we've extended the
832string, by altering C<CUR> to reflect the new length. C<SvEND> is a
833macro which gives us the end of the string, so that needs to be a
834C<"\0">.
835
836Line 13 manipulates the flags; since we've changed the PV, any IV or NV
837values will no longer be valid: if we have C<$a=10; $a.="6";> we don't
838want to use the old IV of 10. C<SvPOK_only_utf8> is a special UTF8-aware
839version of C<SvPOK_only>, a macro which turns off the IOK and NOK flags
840and turns on POK. The final C<SvTAINT> is a macro which launders tainted
841data if taint mode is turned on.
842
843AVs and HVs are more complicated, but SVs are by far the most common
844variable type being thrown around. Having seen something of how we
845manipulate these, let's go on and look at how the op tree is
846constructed.
847
848=head2 Op Trees
849
850First, what is the op tree, anyway? The op tree is the parsed
851representation of your program, as we saw in our section on parsing, and
852it's the sequence of operations that Perl goes through to execute your
853program, as we saw in L</Running>.
854
855An op is a fundamental operation that Perl can perform: all the built-in
856functions and operators are ops, and there are a series of ops which
857deal with concepts the interpreter needs internally - entering and
858leaving a block, ending a statement, fetching a variable, and so on.
859
860The op tree is connected in two ways: you can imagine that there are two
861"routes" through it, two orders in which you can traverse the tree.
862First, parse order reflects how the parser understood the code, and
863secondly, execution order tells perl what order to perform the
864operations in.
865
866The easiest way to examine the op tree is to stop Perl after it has
867finished parsing, and get it to dump out the tree. This is exactly what
868the compiler backends L<B::Terse|B::Terse> and L<B::Debug|B::Debug> do.
869
870Let's have a look at how Perl sees C<$a = $b + $c>:
871
872 % perl -MO=Terse -e '$a=$b+$c'
873 1 LISTOP (0x8179888) leave
874 2 OP (0x81798b0) enter
875 3 COP (0x8179850) nextstate
876 4 BINOP (0x8179828) sassign
877 5 BINOP (0x8179800) add [1]
878 6 UNOP (0x81796e0) null [15]
879 7 SVOP (0x80fafe0) gvsv GV (0x80fa4cc) *b
880 8 UNOP (0x81797e0) null [15]
881 9 SVOP (0x8179700) gvsv GV (0x80efeb0) *c
882 10 UNOP (0x816b4f0) null [15]
883 11 SVOP (0x816dcf0) gvsv GV (0x80fa460) *a
884
885Let's start in the middle, at line 4. This is a BINOP, a binary
886operator, which is at location C<0x8179828>. The specific operator in
887question is C<sassign> - scalar assignment - and you can find the code
888which implements it in the function C<pp_sassign> in F<pp_hot.c>. As a
889binary operator, it has two children: the add operator, providing the
890result of C<$b+$c>, is uppermost on line 5, and the left hand side is on
891line 10.
892
893Line 10 is the null op: this does exactly nothing. What is that doing
894there? If you see the null op, it's a sign that something has been
895optimized away after parsing. As we mentioned in L</Optimization>,
896the optimization stage sometimes converts two operations into one, for
897example when fetching a scalar variable. When this happens, instead of
898rewriting the op tree and cleaning up the dangling pointers, it's easier
899just to replace the redundant operation with the null op. Originally,
900the tree would have looked like this:
901
902 10 SVOP (0x816b4f0) rv2sv [15]
903 11 SVOP (0x816dcf0) gv GV (0x80fa460) *a
904
905That is, fetch the C<a> entry from the main symbol table, and then look
906at the scalar component of it: C<gvsv> (C<pp_gvsv> into F<pp_hot.c>)
907happens to do both these things.
908
909The right hand side, starting at line 5 is similar to what we've just
910seen: we have the C<add> op (C<pp_add> also in F<pp_hot.c>) add together
911two C<gvsv>s.
912
913Now, what's this about?
914
915 1 LISTOP (0x8179888) leave
916 2 OP (0x81798b0) enter
917 3 COP (0x8179850) nextstate
918
919C<enter> and C<leave> are scoping ops, and their job is to perform any
920housekeeping every time you enter and leave a block: lexical variables
921are tidied up, unreferenced variables are destroyed, and so on. Every
922program will have those first three lines: C<leave> is a list, and its
923children are all the statements in the block. Statements are delimited
924by C<nextstate>, so a block is a collection of C<nextstate> ops, with
925the ops to be performed for each statement being the children of
926C<nextstate>. C<enter> is a single op which functions as a marker.
927
928That's how Perl parsed the program, from top to bottom:
929
930 Program
931 |
932 Statement
933 |
934 =
935 / \
936 / \
937 $a +
938 / \
939 $b $c
940
941However, it's impossible to B<perform> the operations in this order:
942you have to find the values of C<$b> and C<$c> before you add them
943together, for instance. So, the other thread that runs through the op
944tree is the execution order: each op has a field C<op_next> which points
945to the next op to be run, so following these pointers tells us how perl
946executes the code. We can traverse the tree in this order using
947the C<exec> option to C<B::Terse>:
948
949 % perl -MO=Terse,exec -e '$a=$b+$c'
950 1 OP (0x8179928) enter
951 2 COP (0x81798c8) nextstate
952 3 SVOP (0x81796c8) gvsv GV (0x80fa4d4) *b
953 4 SVOP (0x8179798) gvsv GV (0x80efeb0) *c
954 5 BINOP (0x8179878) add [1]
955 6 SVOP (0x816dd38) gvsv GV (0x80fa468) *a
956 7 BINOP (0x81798a0) sassign
957 8 LISTOP (0x8179900) leave
958
959This probably makes more sense for a human: enter a block, start a
960statement. Get the values of C<$b> and C<$c>, and add them together.
961Find C<$a>, and assign one to the other. Then leave.
962
963The way Perl builds up these op trees in the parsing process can be
964unravelled by examining F<perly.y>, the YACC grammar. Let's take the
965piece we need to construct the tree for C<$a = $b + $c>
966
967 1 term : term ASSIGNOP term
968 2 { $$ = newASSIGNOP(OPf_STACKED, $1, $2, $3); }
969 3 | term ADDOP term
970 4 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
971
972If you're not used to reading BNF grammars, this is how it works: You're
973fed certain things by the tokeniser, which generally end up in upper
974case. Here, C<ADDOP>, is provided when the tokeniser sees C<+> in your
975code. C<ASSIGNOP> is provided when C<=> is used for assigning. These are
976`terminal symbols', because you can't get any simpler than them.
977
978The grammar, lines one and three of the snippet above, tells you how to
979build up more complex forms. These complex forms, `non-terminal symbols'
980are generally placed in lower case. C<term> here is a non-terminal
981symbol, representing a single expression.
982
983The grammar gives you the following rule: you can make the thing on the
984left of the colon if you see all the things on the right in sequence.
985This is called a "reduction", and the aim of parsing is to completely
986reduce the input. There are several different ways you can perform a
987reduction, separated by vertical bars: so, C<term> followed by C<=>
988followed by C<term> makes a C<term>, and C<term> followed by C<+>
989followed by C<term> can also make a C<term>.
990
991So, if you see two terms with an C<=> or C<+>, between them, you can
992turn them into a single expression. When you do this, you execute the
993code in the block on the next line: if you see C<=>, you'll do the code
994in line 2. If you see C<+>, you'll do the code in line 4. It's this code
995which contributes to the op tree.
996
997 | term ADDOP term
998 { $$ = newBINOP($2, 0, scalar($1), scalar($3)); }
999
1000What this does is creates a new binary op, and feeds it a number of
1001variables. The variables refer to the tokens: C<$1> is the first token in
1002the input, C<$2> the second, and so on - think regular expression
1003backreferences. C<$$> is the op returned from this reduction. So, we
1004call C<newBINOP> to create a new binary operator. The first parameter to
1005C<newBINOP>, a function in F<op.c>, is the op type. It's an addition
1006operator, so we want the type to be C<ADDOP>. We could specify this
1007directly, but it's right there as the second token in the input, so we
1008use C<$2>. The second parameter is the op's flags: 0 means `nothing
1009special'. Then the things to add: the left and right hand side of our
1010expression, in scalar context.
1011
1012=head2 Stacks
1013
1014When perl executes something like C<addop>, how does it pass on its
1015results to the next op? The answer is, through the use of stacks. Perl
1016has a number of stacks to store things it's currently working on, and
1017we'll look at the three most important ones here.
1018
1019=over 3
1020
1021=item Argument stack
1022
1023Arguments are passed to PP code and returned from PP code using the
1024argument stack, C<ST>. The typical way to handle arguments is to pop
1025them off the stack, deal with them how you wish, and then push the result
1026back onto the stack. This is how, for instance, the cosine operator
1027works:
1028
1029 NV value;
1030 value = POPn;
1031 value = Perl_cos(value);
1032 XPUSHn(value);
1033
1034We'll see a more tricky example of this when we consider Perl's macros
1035below. C<POPn> gives you the NV (floating point value) of the top SV on
1036the stack: the C<$x> in C<cos($x)>. Then we compute the cosine, and push
1037the result back as an NV. The C<X> in C<XPUSHn> means that the stack
1038should be extended if necessary - it can't be necessary here, because we
1039know there's room for one more item on the stack, since we've just
1040removed one! The C<XPUSH*> macros at least guarantee safety.
1041
1042Alternatively, you can fiddle with the stack directly: C<SP> gives you
1043the first element in your portion of the stack, and C<TOP*> gives you
1044the top SV/IV/NV/etc. on the stack. So, for instance, to do unary
1045negation of an integer:
1046
1047 SETi(-TOPi);
1048
1049Just set the integer value of the top stack entry to its negation.
1050
1051Argument stack manipulation in the core is exactly the same as it is in
1052XSUBs - see L<perlxstut>, L<perlxs> and L<perlguts> for a longer
1053description of the macros used in stack manipulation.
1054
1055=item Mark stack
1056
1057I say `your portion of the stack' above because PP code doesn't
1058necessarily get the whole stack to itself: if your function calls
1059another function, you'll only want to expose the arguments aimed for the
1060called function, and not (necessarily) let it get at your own data. The
1061way we do this is to have a `virtual' bottom-of-stack, exposed to each
1062function. The mark stack keeps bookmarks to locations in the argument
1063stack usable by each function. For instance, when dealing with a tied
1064variable, (internally, something with `P' magic) Perl has to call
1065methods for accesses to the tied variables. However, we need to separate
1066the arguments exposed to the method to the argument exposed to the
1067original function - the store or fetch or whatever it may be. Here's how
1068the tied C<push> is implemented; see C<av_push> in F<av.c>:
1069
1070 1 PUSHMARK(SP);
1071 2 EXTEND(SP,2);
1072 3 PUSHs(SvTIED_obj((SV*)av, mg));
1073 4 PUSHs(val);
1074 5 PUTBACK;
1075 6 ENTER;
1076 7 call_method("PUSH", G_SCALAR|G_DISCARD);
1077 8 LEAVE;
1078 9 POPSTACK;
13a2d996 1079
a422fd2d
SC
1080The lines which concern the mark stack are the first, fifth and last
1081lines: they save away, restore and remove the current position of the
1082argument stack.
1083
1084Let's examine the whole implementation, for practice:
1085
1086 1 PUSHMARK(SP);
1087
1088Push the current state of the stack pointer onto the mark stack. This is
1089so that when we've finished adding items to the argument stack, Perl
1090knows how many things we've added recently.
1091
1092 2 EXTEND(SP,2);
1093 3 PUSHs(SvTIED_obj((SV*)av, mg));
1094 4 PUSHs(val);
1095
1096We're going to add two more items onto the argument stack: when you have
1097a tied array, the C<PUSH> subroutine receives the object and the value
1098to be pushed, and that's exactly what we have here - the tied object,
1099retrieved with C<SvTIED_obj>, and the value, the SV C<val>.
1100
1101 5 PUTBACK;
1102
1103Next we tell Perl to make the change to the global stack pointer: C<dSP>
1104only gave us a local copy, not a reference to the global.
1105
1106 6 ENTER;
1107 7 call_method("PUSH", G_SCALAR|G_DISCARD);
1108 8 LEAVE;
1109
1110C<ENTER> and C<LEAVE> localise a block of code - they make sure that all
1111variables are tidied up, everything that has been localised gets
1112its previous value returned, and so on. Think of them as the C<{> and
1113C<}> of a Perl block.
1114
1115To actually do the magic method call, we have to call a subroutine in
1116Perl space: C<call_method> takes care of that, and it's described in
1117L<perlcall>. We call the C<PUSH> method in scalar context, and we're
1118going to discard its return value.
1119
1120 9 POPSTACK;
1121
1122Finally, we remove the value we placed on the mark stack, since we
1123don't need it any more.
1124
1125=item Save stack
1126
1127C doesn't have a concept of local scope, so perl provides one. We've
1128seen that C<ENTER> and C<LEAVE> are used as scoping braces; the save
1129stack implements the C equivalent of, for example:
1130
1131 {
1132 local $foo = 42;
1133 ...
1134 }
1135
1136See L<perlguts/Localising Changes> for how to use the save stack.
1137
1138=back
1139
1140=head2 Millions of Macros
1141
1142One thing you'll notice about the Perl source is that it's full of
1143macros. Some have called the pervasive use of macros the hardest thing
1144to understand, others find it adds to clarity. Let's take an example,
1145the code which implements the addition operator:
1146
1147 1 PP(pp_add)
1148 2 {
39644a26 1149 3 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
a422fd2d
SC
1150 4 {
1151 5 dPOPTOPnnrl_ul;
1152 6 SETn( left + right );
1153 7 RETURN;
1154 8 }
1155 9 }
1156
1157Every line here (apart from the braces, of course) contains a macro. The
1158first line sets up the function declaration as Perl expects for PP code;
1159line 3 sets up variable declarations for the argument stack and the
1160target, the return value of the operation. Finally, it tries to see if
1161the addition operation is overloaded; if so, the appropriate subroutine
1162is called.
1163
1164Line 5 is another variable declaration - all variable declarations start
1165with C<d> - which pops from the top of the argument stack two NVs (hence
1166C<nn>) and puts them into the variables C<right> and C<left>, hence the
1167C<rl>. These are the two operands to the addition operator. Next, we
1168call C<SETn> to set the NV of the return value to the result of adding
1169the two values. This done, we return - the C<RETURN> macro makes sure
1170that our return value is properly handled, and we pass the next operator
1171to run back to the main run loop.
1172
1173Most of these macros are explained in L<perlapi>, and some of the more
1174important ones are explained in L<perlxs> as well. Pay special attention
1175to L<perlguts/Background and PERL_IMPLICIT_CONTEXT> for information on
1176the C<[pad]THX_?> macros.
1177
1178
1179=head2 Poking at Perl
1180
1181To really poke around with Perl, you'll probably want to build Perl for
1182debugging, like this:
1183
1184 ./Configure -d -D optimize=-g
1185 make
1186
1187C<-g> is a flag to the C compiler to have it produce debugging
1188information which will allow us to step through a running program.
1189F<Configure> will also turn on the C<DEBUGGING> compilation symbol which
1190enables all the internal debugging code in Perl. There are a whole bunch
1191of things you can debug with this: L<perlrun> lists them all, and the
1192best way to find out about them is to play about with them. The most
1193useful options are probably
1194
1195 l Context (loop) stack processing
1196 t Trace execution
1197 o Method and overloading resolution
1198 c String/numeric conversions
1199
1200Some of the functionality of the debugging code can be achieved using XS
1201modules.
13a2d996 1202
a422fd2d
SC
1203 -Dr => use re 'debug'
1204 -Dx => use O 'Debug'
1205
1206=head2 Using a source-level debugger
1207
1208If the debugging output of C<-D> doesn't help you, it's time to step
1209through perl's execution with a source-level debugger.
1210
1211=over 3
1212
1213=item *
1214
1215We'll use C<gdb> for our examples here; the principles will apply to any
1216debugger, but check the manual of the one you're using.
1217
1218=back
1219
1220To fire up the debugger, type
1221
1222 gdb ./perl
1223
1224You'll want to do that in your Perl source tree so the debugger can read
1225the source code. You should see the copyright message, followed by the
1226prompt.
1227
1228 (gdb)
1229
1230C<help> will get you into the documentation, but here are the most
1231useful commands:
1232
1233=over 3
1234
1235=item run [args]
1236
1237Run the program with the given arguments.
1238
1239=item break function_name
1240
1241=item break source.c:xxx
1242
1243Tells the debugger that we'll want to pause execution when we reach
cea6626f 1244either the named function (but see L<perlguts/Internal Functions>!) or the given
a422fd2d
SC
1245line in the named source file.
1246
1247=item step
1248
1249Steps through the program a line at a time.
1250
1251=item next
1252
1253Steps through the program a line at a time, without descending into
1254functions.
1255
1256=item continue
1257
1258Run until the next breakpoint.
1259
1260=item finish
1261
1262Run until the end of the current function, then stop again.
1263
13a2d996 1264=item 'enter'
a422fd2d
SC
1265
1266Just pressing Enter will do the most recent operation again - it's a
1267blessing when stepping through miles of source code.
1268
1269=item print
1270
1271Execute the given C code and print its results. B<WARNING>: Perl makes
1272heavy use of macros, and F<gdb> is not aware of macros. You'll have to
1273substitute them yourself. So, for instance, you can't say
1274
1275 print SvPV_nolen(sv)
1276
1277but you have to say
1278
1279 print Perl_sv_2pv_nolen(sv)
1280
1281You may find it helpful to have a "macro dictionary", which you can
1282produce by saying C<cpp -dM perl.c | sort>. Even then, F<cpp> won't
1283recursively apply the macros for you.
1284
1285=back
1286
1287=head2 Dumping Perl Data Structures
1288
1289One way to get around this macro hell is to use the dumping functions in
1290F<dump.c>; these work a little like an internal
1291L<Devel::Peek|Devel::Peek>, but they also cover OPs and other structures
1292that you can't get at from Perl. Let's take an example. We'll use the
1293C<$a = $b + $c> we used before, but give it a bit of context:
1294C<$b = "6XXXX"; $c = 2.3;>. Where's a good place to stop and poke around?
1295
1296What about C<pp_add>, the function we examined earlier to implement the
1297C<+> operator:
1298
1299 (gdb) break Perl_pp_add
1300 Breakpoint 1 at 0x46249f: file pp_hot.c, line 309.
1301
cea6626f 1302Notice we use C<Perl_pp_add> and not C<pp_add> - see L<perlguts/Internal Functions>.
a422fd2d
SC
1303With the breakpoint in place, we can run our program:
1304
1305 (gdb) run -e '$b = "6XXXX"; $c = 2.3; $a = $b + $c'
1306
1307Lots of junk will go past as gdb reads in the relevant source files and
1308libraries, and then:
1309
1310 Breakpoint 1, Perl_pp_add () at pp_hot.c:309
39644a26 1311 309 dSP; dATARGET; tryAMAGICbin(add,opASSIGN);
a422fd2d
SC
1312 (gdb) step
1313 311 dPOPTOPnnrl_ul;
1314 (gdb)
1315
1316We looked at this bit of code before, and we said that C<dPOPTOPnnrl_ul>
1317arranges for two C<NV>s to be placed into C<left> and C<right> - let's
1318slightly expand it:
1319
1320 #define dPOPTOPnnrl_ul NV right = POPn; \
1321 SV *leftsv = TOPs; \
1322 NV left = USE_LEFT(leftsv) ? SvNV(leftsv) : 0.0
1323
1324C<POPn> takes the SV from the top of the stack and obtains its NV either
1325directly (if C<SvNOK> is set) or by calling the C<sv_2nv> function.
1326C<TOPs> takes the next SV from the top of the stack - yes, C<POPn> uses
1327C<TOPs> - but doesn't remove it. We then use C<SvNV> to get the NV from
1328C<leftsv> in the same way as before - yes, C<POPn> uses C<SvNV>.
1329
1330Since we don't have an NV for C<$b>, we'll have to use C<sv_2nv> to
1331convert it. If we step again, we'll find ourselves there:
1332
1333 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1669
1334 1669 if (!sv)
1335 (gdb)
1336
1337We can now use C<Perl_sv_dump> to investigate the SV:
1338
1339 SV = PV(0xa057cc0) at 0xa0675d0
1340 REFCNT = 1
1341 FLAGS = (POK,pPOK)
1342 PV = 0xa06a510 "6XXXX"\0
1343 CUR = 5
1344 LEN = 6
1345 $1 = void
1346
1347We know we're going to get C<6> from this, so let's finish the
1348subroutine:
1349
1350 (gdb) finish
1351 Run till exit from #0 Perl_sv_2nv (sv=0xa0675d0) at sv.c:1671
1352 0x462669 in Perl_pp_add () at pp_hot.c:311
1353 311 dPOPTOPnnrl_ul;
1354
1355We can also dump out this op: the current op is always stored in
1356C<PL_op>, and we can dump it with C<Perl_op_dump>. This'll give us
1357similar output to L<B::Debug|B::Debug>.
1358
1359 {
1360 13 TYPE = add ===> 14
1361 TARG = 1
1362 FLAGS = (SCALAR,KIDS)
1363 {
1364 TYPE = null ===> (12)
1365 (was rv2sv)
1366 FLAGS = (SCALAR,KIDS)
1367 {
1368 11 TYPE = gvsv ===> 12
1369 FLAGS = (SCALAR)
1370 GV = main::b
1371 }
1372 }
1373
1374< finish this later >
1375
1376=head2 Patching
1377
1378All right, we've now had a look at how to navigate the Perl sources and
1379some things you'll need to know when fiddling with them. Let's now get
1380on and create a simple patch. Here's something Larry suggested: if a
1381C<U> is the first active format during a C<pack>, (for example,
1382C<pack "U3C8", @stuff>) then the resulting string should be treated as
1383UTF8 encoded.
1384
1385How do we prepare to fix this up? First we locate the code in question -
1386the C<pack> happens at runtime, so it's going to be in one of the F<pp>
1387files. Sure enough, C<pp_pack> is in F<pp.c>. Since we're going to be
1388altering this file, let's copy it to F<pp.c~>.
1389
1390Now let's look over C<pp_pack>: we take a pattern into C<pat>, and then
1391loop over the pattern, taking each format character in turn into
1392C<datum_type>. Then for each possible format character, we swallow up
1393the other arguments in the pattern (a field width, an asterisk, and so
1394on) and convert the next chunk input into the specified format, adding
1395it onto the output SV C<cat>.
1396
1397How do we know if the C<U> is the first format in the C<pat>? Well, if
1398we have a pointer to the start of C<pat> then, if we see a C<U> we can
1399test whether we're still at the start of the string. So, here's where
1400C<pat> is set up:
1401
1402 STRLEN fromlen;
1403 register char *pat = SvPVx(*++MARK, fromlen);
1404 register char *patend = pat + fromlen;
1405 register I32 len;
1406 I32 datumtype;
1407 SV *fromstr;
1408
1409We'll have another string pointer in there:
1410
1411 STRLEN fromlen;
1412 register char *pat = SvPVx(*++MARK, fromlen);
1413 register char *patend = pat + fromlen;
1414 + char *patcopy;
1415 register I32 len;
1416 I32 datumtype;
1417 SV *fromstr;
1418
1419And just before we start the loop, we'll set C<patcopy> to be the start
1420of C<pat>:
1421
1422 items = SP - MARK;
1423 MARK++;
1424 sv_setpvn(cat, "", 0);
1425 + patcopy = pat;
1426 while (pat < patend) {
1427
1428Now if we see a C<U> which was at the start of the string, we turn on
1429the UTF8 flag for the output SV, C<cat>:
1430
1431 + if (datumtype == 'U' && pat==patcopy+1)
1432 + SvUTF8_on(cat);
1433 if (datumtype == '#') {
1434 while (pat < patend && *pat != '\n')
1435 pat++;
1436
1437Remember that it has to be C<patcopy+1> because the first character of
1438the string is the C<U> which has been swallowed into C<datumtype!>
1439
1440Oops, we forgot one thing: what if there are spaces at the start of the
1441pattern? C<pack(" U*", @stuff)> will have C<U> as the first active
1442character, even though it's not the first thing in the pattern. In this
1443case, we have to advance C<patcopy> along with C<pat> when we see spaces:
1444
1445 if (isSPACE(datumtype))
1446 continue;
1447
1448needs to become
1449
1450 if (isSPACE(datumtype)) {
1451 patcopy++;
1452 continue;
1453 }
1454
1455OK. That's the C part done. Now we must do two additional things before
1456this patch is ready to go: we've changed the behaviour of Perl, and so
1457we must document that change. We must also provide some more regression
1458tests to make sure our patch works and doesn't create a bug somewhere
1459else along the line.
1460
1461The regression tests for each operator live in F<t/op/>, and so we make
1462a copy of F<t/op/pack.t> to F<t/op/pack.t~>. Now we can add our tests
1463to the end. First, we'll test that the C<U> does indeed create Unicode
1464strings:
1465
1466 print 'not ' unless "1.20.300.4000" eq sprintf "%vd", pack("U*",1,20,300,4000);
1467 print "ok $test\n"; $test++;
1468
1469Now we'll test that we got that space-at-the-beginning business right:
1470
1471 print 'not ' unless "1.20.300.4000" eq
1472 sprintf "%vd", pack(" U*",1,20,300,4000);
1473 print "ok $test\n"; $test++;
1474
1475And finally we'll test that we don't make Unicode strings if C<U> is B<not>
1476the first active format:
1477
1478 print 'not ' unless v1.20.300.4000 ne
1479 sprintf "%vd", pack("C0U*",1,20,300,4000);
1480 print "ok $test\n"; $test++;
1481
b1866b2d 1482Mustn't forget to change the number of tests which appears at the top, or
a422fd2d
SC
1483else the automated tester will get confused:
1484
1485 -print "1..156\n";
1486 +print "1..159\n";
1487
1488We now compile up Perl, and run it through the test suite. Our new
1489tests pass, hooray!
1490
1491Finally, the documentation. The job is never done until the paperwork is
1492over, so let's describe the change we've just made. The relevant place
1493is F<pod/perlfunc.pod>; again, we make a copy, and then we'll insert
1494this text in the description of C<pack>:
1495
1496 =item *
1497
1498 If the pattern begins with a C<U>, the resulting string will be treated
1499 as Unicode-encoded. You can force UTF8 encoding on in a string with an
1500 initial C<U0>, and the bytes that follow will be interpreted as Unicode
1501 characters. If you don't want this to happen, you can begin your pattern
1502 with C<C0> (or anything else) to force Perl not to UTF8 encode your
1503 string, and then follow this with a C<U*> somewhere in your pattern.
1504
1505All done. Now let's create the patch. F<Porting/patching.pod> tells us
1506that if we're making major changes, we should copy the entire directory
1507to somewhere safe before we begin fiddling, and then do
13a2d996 1508
a422fd2d
SC
1509 diff -ruN old new > patch
1510
1511However, we know which files we've changed, and we can simply do this:
1512
1513 diff -u pp.c~ pp.c > patch
1514 diff -u t/op/pack.t~ t/op/pack.t >> patch
1515 diff -u pod/perlfunc.pod~ pod/perlfunc.pod >> patch
1516
1517We end up with a patch looking a little like this:
1518
1519 --- pp.c~ Fri Jun 02 04:34:10 2000
1520 +++ pp.c Fri Jun 16 11:37:25 2000
1521 @@ -4375,6 +4375,7 @@
1522 register I32 items;
1523 STRLEN fromlen;
1524 register char *pat = SvPVx(*++MARK, fromlen);
1525 + char *patcopy;
1526 register char *patend = pat + fromlen;
1527 register I32 len;
1528 I32 datumtype;
1529 @@ -4405,6 +4406,7 @@
1530 ...
1531
1532And finally, we submit it, with our rationale, to perl5-porters. Job
1533done!
1534
902b9dbf
MF
1535=head1 EXTERNAL TOOLS FOR DEBUGGING PERL
1536
1537Sometimes it helps to use external tools while debugging and
1538testing Perl. This section tries to guide you through using
1539some common testing and debugging tools with Perl. This is
1540meant as a guide to interfacing these tools with Perl, not
1541as any kind of guide to the use of the tools themselves.
1542
1543=head2 Rational Software's Purify
1544
1545Purify is a commercial tool that is helpful in identifying
1546memory overruns, wild pointers, memory leaks and other such
1547badness. Perl must be compiled in a specific way for
1548optimal testing with Purify. Purify is available under
1549Windows NT, Solaris, HP-UX, SGI, and Siemens Unix.
1550
1551The only currently known leaks happen when there are
1552compile-time errors within eval or require. (Fixing these
1553is non-trivial, unfortunately, but they must be fixed
1554eventually.)
1555
1556=head2 Purify on Unix
1557
1558On Unix, Purify creates a new Perl binary. To get the most
1559benefit out of Purify, you should create the perl to Purify
1560using:
1561
1562 sh Configure -Accflags=-DPURIFY -Doptimize='-g' \
1563 -Uusemymalloc -Dusemultiplicity
1564
1565where these arguments mean:
1566
1567=over 4
1568
1569=item -Accflags=-DPURIFY
1570
1571Disables Perl's arena memory allocation functions, as well as
1572forcing use of memory allocation functions derived from the
1573system malloc.
1574
1575=item -Doptimize='-g'
1576
1577Adds debugging information so that you see the exact source
1578statements where the problem occurs. Without this flag, all
1579you will see is the source filename of where the error occurred.
1580
1581=item -Uusemymalloc
1582
1583Disable Perl's malloc so that Purify can more closely monitor
1584allocations and leaks. Using Perl's malloc will make Purify
1585report most leaks in the "potential" leaks category.
1586
1587=item -Dusemultiplicity
1588
1589Enabling the multiplicity option allows perl to clean up
1590thoroughly when the interpreter shuts down, which reduces the
1591number of bogus leak reports from Purify.
1592
1593=back
1594
1595Once you've compiled a perl suitable for Purify'ing, then you
1596can just:
1597
1598 make pureperl
1599
1600which creates a binary named 'pureperl' that has been Purify'ed.
1601This binary is used in place of the standard 'perl' binary
1602when you want to debug Perl memory problems.
1603
1604As an example, to show any memory leaks produced during the
1605standard Perl testset you would create and run the Purify'ed
1606perl as:
1607
1608 make pureperl
1609 cd t
1610 ../pureperl -I../lib harness
1611
1612which would run Perl on test.pl and report any memory problems.
1613
1614Purify outputs messages in "Viewer" windows by default. If
1615you don't have a windowing environment or if you simply
1616want the Purify output to unobtrusively go to a log file
1617instead of to the interactive window, use these following
1618options to output to the log file "perl.log":
1619
1620 setenv PURIFYOPTIONS "-chain-length=25 -windows=no \
1621 -log-file=perl.log -append-logfile=yes"
1622
1623If you plan to use the "Viewer" windows, then you only need this option:
1624
1625 setenv PURIFYOPTIONS "-chain-length=25"
1626
1627=head2 Purify on NT
1628
1629Purify on Windows NT instruments the Perl binary 'perl.exe'
1630on the fly. There are several options in the makefile you
1631should change to get the most use out of Purify:
1632
1633=over 4
1634
1635=item DEFINES
1636
1637You should add -DPURIFY to the DEFINES line so the DEFINES
1638line looks something like:
1639
1640 DEFINES = -DWIN32 -D_CONSOLE -DNO_STRICT $(CRYPT_FLAG) -DPURIFY=1
1641
1642to disable Perl's arena memory allocation functions, as
1643well as to force use of memory allocation functions derived
1644from the system malloc.
1645
1646=item USE_MULTI = define
1647
1648Enabling the multiplicity option allows perl to clean up
1649thoroughly when the interpreter shuts down, which reduces the
1650number of bogus leak reports from Purify.
1651
1652=item #PERL_MALLOC = define
1653
1654Disable Perl's malloc so that Purify can more closely monitor
1655allocations and leaks. Using Perl's malloc will make Purify
1656report most leaks in the "potential" leaks category.
1657
1658=item CFG = Debug
1659
1660Adds debugging information so that you see the exact source
1661statements where the problem occurs. Without this flag, all
1662you will see is the source filename of where the error occurred.
1663
1664=back
1665
1666As an example, to show any memory leaks produced during the
1667standard Perl testset you would create and run Purify as:
1668
1669 cd win32
1670 make
1671 cd ../t
1672 purify ../perl -I../lib harness
1673
1674which would instrument Perl in memory, run Perl on test.pl,
1675then finally report any memory problems.
1676
09187cb1
JH
1677=head2 Compaq's/Digital's Third Degree
1678
1679Third Degree is a tool for memory leak detection and memory access checks.
1680It is one of the many tools in the ATOM toolkit. The toolkit is only
1681available on Tru64 (formerly known as Digital UNIX formerly known as
1682DEC OSF/1).
1683
1684When building Perl, you must first run Configure with -Doptimize=-g
1685and -Uusemymalloc flags, after that you can use the make targets
1686
1687"perl.third" and "test.third". The short story is that with "atom"
1688you can instrument the Perl executable to create a new executable
1689called "perl.third." When the instrumented executable is run, it
1690creates a log of dubious memory traffic in file called "perl.3log".
1691See man atom and man third for more information. The most extensive
1692Third Degree documentation is available in the Compaq "Tru64 UNIX
1693Programmer's Guide", chapter "Debugging Programs with Third Degree".
1694
a422fd2d
SC
1695=head2 CONCLUSION
1696
1697We've had a brief look around the Perl source, an overview of the stages
1698F<perl> goes through when it's running your code, and how to use a
902b9dbf
MF
1699debugger to poke at the Perl guts. We took a very simple problem and
1700demonstrated how to solve it fully - with documentation, regression
1701tests, and finally a patch for submission to p5p. Finally, we talked
1702about how to use external tools to debug and test Perl.
a422fd2d
SC
1703
1704I'd now suggest you read over those references again, and then, as soon
1705as possible, get your hands dirty. The best way to learn is by doing,
1706so:
1707
1708=over 3
1709
1710=item *
1711
1712Subscribe to perl5-porters, follow the patches and try and understand
1713them; don't be afraid to ask if there's a portion you're not clear on -
1714who knows, you may unearth a bug in the patch...
1715
1716=item *
1717
1718Keep up to date with the bleeding edge Perl distributions and get
1719familiar with the changes. Try and get an idea of what areas people are
1720working on and the changes they're making.
1721
1722=item *
1723
3e148164 1724Do read the README associated with your operating system, e.g. README.aix
a1f349fd
MB
1725on the IBM AIX OS. Don't hesitate to supply patches to that README if
1726you find anything missing or changed over a new OS release.
1727
1728=item *
1729
a422fd2d
SC
1730Find an area of Perl that seems interesting to you, and see if you can
1731work out how it works. Scan through the source, and step over it in the
1732debugger. Play, poke, investigate, fiddle! You'll probably get to
1733understand not just your chosen area but a much wider range of F<perl>'s
1734activity as well, and probably sooner than you'd think.
1735
1736=back
1737
1738=over 3
1739
1740=item I<The Road goes ever on and on, down from the door where it began.>
1741
1742=back
1743
1744If you can do these things, you've started on the long road to Perl porting.
1745Thanks for wanting to help make Perl better - and happy hacking!
1746
e8cd7eae
GS
1747=head1 AUTHOR
1748
1749This document was written by Nathan Torkington, and is maintained by
1750the perl5-porters mailing list.
1751