This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
regcomp.sym: Add nodes for script runs
[perl5.git] / pod / perldebguts.pod
CommitLineData
055fd3a9
GS
1=head1 NAME
2
3perldebguts - Guts of Perl debugging
4
5=head1 DESCRIPTION
6
ba555bf5 7This is not L<perldebug>, which tells you how to use
74410c12
JM
8the debugger. This manpage describes low-level details concerning
9the debugger's internals, which range from difficult to impossible
10to understand for anyone who isn't incredibly intimate with Perl's guts.
11Caveat lector.
055fd3a9
GS
12
13=head1 Debugger Internals
14
15Perl has special debugging hooks at compile-time and run-time used
16to create debugging environments. These hooks are not to be confused
4375e838
GS
17with the I<perl -Dxxx> command described in L<perlrun>, which is
18usable only if a special Perl is built per the instructions in the
055fd3a9
GS
19F<INSTALL> podpage in the Perl source tree.
20
21For example, whenever you call Perl's built-in C<caller> function
74410c12
JM
22from the package C<DB>, the arguments that the corresponding stack
23frame was called with are copied to the C<@DB::args> array. These
24mechanisms are enabled by calling Perl with the B<-d> switch.
25Specifically, the following additional features are enabled
26(cf. L<perlvar/$^P>):
055fd3a9 27
13a2d996 28=over 4
055fd3a9
GS
29
30=item *
31
32Perl inserts the contents of C<$ENV{PERL5DB}> (or C<BEGIN {require
33'perl5db.pl'}> if not present) before the first line of your program.
34
35=item *
36
aa0b556f 37Each array C<@{"_<$filename"}> holds the lines of $filename for a
74410c12
JM
38file compiled by Perl. The same is also true for C<eval>ed strings
39that contain subroutines, or which are currently being executed.
40The $filename for C<eval>ed strings looks like C<(eval 34)>.
8894c26d
MJD
41
42Values in this array are magical in numeric context: they compare
43equal to zero only if the line is not breakable.
055fd3a9
GS
44
45=item *
46
aa0b556f 47Each hash C<%{"_<$filename"}> contains breakpoints and actions keyed
055fd3a9
GS
48by line number. Individual entries (as opposed to the whole hash)
49are settable. Perl only cares about Boolean true here, although
50the values used by F<perl5db.pl> have the form
8894c26d 51C<"$break_condition\0$action">.
055fd3a9
GS
52
53The same holds for evaluated strings that contain subroutines, or
54which are currently being executed. The $filename for C<eval>ed strings
d24ca0c5 55looks like C<(eval 34)>.
055fd3a9
GS
56
57=item *
58
aa0b556f 59Each scalar C<${"_<$filename"}> contains C<"_<$filename">. This is
055fd3a9
GS
60also the case for evaluated strings that contain subroutines, or
61which are currently being executed. The $filename for C<eval>ed
d24ca0c5 62strings looks like C<(eval 34)>.
055fd3a9
GS
63
64=item *
65
66After each C<require>d file is compiled, but before it is executed,
67C<DB::postponed(*{"_<$filename"})> is called if the subroutine
68C<DB::postponed> exists. Here, the $filename is the expanded name of
69the C<require>d file, as found in the values of %INC.
70
71=item *
72
73After each subroutine C<subname> is compiled, the existence of
74C<$DB::postponed{subname}> is checked. If this key exists,
75C<DB::postponed(subname)> is called if the C<DB::postponed> subroutine
76also exists.
77
78=item *
79
80A hash C<%DB::sub> is maintained, whose keys are subroutine names
81and whose values have the form C<filename:startline-endline>.
82C<filename> has the form C<(eval 34)> for subroutines defined inside
d24ca0c5 83C<eval>s.
055fd3a9
GS
84
85=item *
86
87When the execution of your program reaches a point that can hold a
74410c12
JM
88breakpoint, the C<DB::DB()> subroutine is called if any of the variables
89C<$DB::trace>, C<$DB::single>, or C<$DB::signal> is true. These variables
055fd3a9
GS
90are not C<local>izable. This feature is disabled when executing
91inside C<DB::DB()>, including functions called from it
92unless C<< $^D & (1<<30) >> is true.
93
94=item *
95
96When execution of the program reaches a subroutine call, a call to
97C<&DB::sub>(I<args>) is made instead, with C<$DB::sub> holding the
7270698f
Z
98name of the called subroutine. (This doesn't happen if the calling
99subroutine was compiled in the C<DB> package.)
055fd3a9 100
77e42cd2
TC
101X<&DB::lsub>If the call is to an lvalue subroutine, and C<&DB::lsub>
102is defined C<&DB::lsub>(I<args>) is called instead, otherwise falling
103back to C<&DB::sub>(I<args>).
104
261cbad1
TC
105=item *
106
107When execution of the program uses C<goto> to enter a non-XS
108subroutine and the 0x80 bit is set in C<$^P>, a call to C<&DB::goto>
109is made, with C<$DB::sub> holding the name of the subroutine being
110entered.
111
055fd3a9
GS
112=back
113
114Note that if C<&DB::sub> needs external data for it to work, no
74410c12
JM
115subroutine call is possible without it. As an example, the standard
116debugger's C<&DB::sub> depends on the C<$DB::deep> variable
117(it defines how many levels of recursion deep into the debugger you can go
118before a mandatory break). If C<$DB::deep> is not defined, subroutine
119calls are not possible, even though C<&DB::sub> exists.
055fd3a9
GS
120
121=head2 Writing Your Own Debugger
122
74410c12 123=head3 Environment Variables
666f95b9 124
74410c12
JM
125The C<PERL5DB> environment variable can be used to define a debugger.
126For example, the minimal "working" debugger (it actually doesn't do anything)
127consists of one line:
666f95b9 128
055fd3a9
GS
129 sub DB::DB {}
130
74410c12 131It can easily be defined like this:
666f95b9 132
055fd3a9
GS
133 $ PERL5DB="sub DB::DB {}" perl -d your-script
134
74410c12 135Another brief debugger, slightly more useful, can be created
055fd3a9
GS
136with only the line:
137
138 sub DB::DB {print ++$i; scalar <STDIN>}
139
74410c12
JM
140This debugger prints a number which increments for each statement
141encountered and waits for you to hit a newline before continuing
142to the next statement.
666f95b9 143
74410c12 144The following debugger is actually useful:
666f95b9 145
055fd3a9
GS
146 {
147 package DB;
148 sub DB {}
149 sub sub {print ++$i, " $sub\n"; &$sub}
150 }
151
74410c12
JM
152It prints the sequence number of each subroutine call and the name of the
153called subroutine. Note that C<&DB::sub> is being compiled into the
154package C<DB> through the use of the C<package> directive.
055fd3a9 155
74410c12
JM
156When it starts, the debugger reads your rc file (F<./.perldb> or
157F<~/.perldb> under Unix), which can set important options.
158(A subroutine (C<&afterinit>) can be defined here as well; it is executed
159after the debugger completes its own initialization.)
055fd3a9
GS
160
161After the rc file is read, the debugger reads the PERLDB_OPTS
74410c12
JM
162environment variable and uses it to set debugger options. The
163contents of this variable are treated as if they were the argument
96090e4f 164of an C<o ...> debugger command (q.v. in L<perldebug/"Configurable Options">).
74410c12 165
7b406369 166=head3 Debugger Internal Variables
25cf7dea 167
74410c12
JM
168In addition to the file and subroutine-related variables mentioned above,
169the debugger also maintains various magical internal variables.
170
171=over 4
172
173=item *
055fd3a9 174
74410c12
JM
175C<@DB::dbline> is an alias for C<@{"::_<current_file"}>, which
176holds the lines of the currently-selected file (compiled by Perl), either
177explicitly chosen with the debugger's C<f> command, or implicitly by flow
178of execution.
179
180Values in this array are magical in numeric context: they compare
181equal to zero only if the line is not breakable.
182
183=item *
184
7b406369 185C<%DB::dbline> is an alias for C<%{"::_<current_file"}>, which
74410c12
JM
186contains breakpoints and actions keyed by line number in
187the currently-selected file, either explicitly chosen with the
055fd3a9
GS
188debugger's C<f> command, or implicitly by flow of execution.
189
74410c12
JM
190As previously noted, individual entries (as opposed to the whole hash)
191are settable. Perl only cares about Boolean true here, although
192the values used by F<perl5db.pl> have the form
193C<"$break_condition\0$action">.
194
195=back
196
7b406369 197=head3 Debugger Customization Functions
74410c12
JM
198
199Some functions are provided to simplify customization.
200
201=over 4
202
203=item *
204
71110851
RGS
205See L<perldebug/"Configurable Options"> for a description of options parsed by
206C<DB::parse_options(string)>.
74410c12
JM
207
208=item *
209
210C<DB::dump_trace(skip[,count])> skips the specified number of frames
211and returns a list containing information about the calling frames (all
212of them, if C<count> is missing). Each entry is reference to a hash
213with keys C<context> (either C<.>, C<$>, or C<@>), C<sub> (subroutine
055fd3a9
GS
214name, or info about C<eval>), C<args> (C<undef> or a reference to
215an array), C<file>, and C<line>.
216
74410c12
JM
217=item *
218
219C<DB::print_trace(FH, skip[, count[, short]])> prints
055fd3a9
GS
220formatted info about caller frames. The last two functions may be
221convenient as arguments to C<< < >>, C<< << >> commands.
222
74410c12
JM
223=back
224
055fd3a9
GS
225Note that any variables and functions that are not documented in
226this manpages (or in L<perldebug>) are considered for internal
227use only, and as such are subject to change without notice.
228
229=head1 Frame Listing Output Examples
230
231The C<frame> option can be used to control the output of frame
232information. For example, contrast this expression trace:
233
234 $ perl -de 42
235 Stack dump during die enabled outside of evals.
236
237 Loading DB routines from perl5db.pl patch level 0.94
238 Emacs support available.
239
ccf3535a 240 Enter h or 'h h' for help.
055fd3a9
GS
241
242 main::(-e:1): 0
243 DB<1> sub foo { 14 }
244
245 DB<2> sub bar { 3 }
246
247 DB<3> t print foo() * bar()
248 main::((eval 172):3): print foo() + bar();
249 main::foo((eval 168):2):
250 main::bar((eval 170):2):
251 42
252
492652be 253with this one, once the C<o>ption C<frame=2> has been set:
055fd3a9 254
492652be 255 DB<4> o f=2
055fd3a9
GS
256 frame = '2'
257 DB<5> t print foo() * bar()
258 3: foo() * bar()
259 entering main::foo
260 2: sub foo { 14 };
261 exited main::foo
262 entering main::bar
263 2: sub bar { 3 };
264 exited main::bar
265 42
266
267By way of demonstration, we present below a laborious listing
268resulting from setting your C<PERLDB_OPTS> environment variable to
269the value C<f=n N>, and running I<perl -d -V> from the command line.
7b406369
FC
270Examples using various values of C<n> are shown to give you a feel
271for the difference between settings. Long though it may be, this
055fd3a9
GS
272is not a complete listing, but only excerpts.
273
274=over 4
275
276=item 1
277
f185f654
KW
278 entering main::BEGIN
279 entering Config::BEGIN
280 Package lib/Exporter.pm.
281 Package lib/Carp.pm.
282 Package lib/Config.pm.
283 entering Config::TIEHASH
284 entering Exporter::import
285 entering Exporter::export
286 entering Config::myconfig
287 entering Config::FETCH
288 entering Config::FETCH
289 entering Config::FETCH
290 entering Config::FETCH
055fd3a9
GS
291
292=item 2
293
f185f654
KW
294 entering main::BEGIN
295 entering Config::BEGIN
296 Package lib/Exporter.pm.
297 Package lib/Carp.pm.
298 exited Config::BEGIN
299 Package lib/Config.pm.
300 entering Config::TIEHASH
301 exited Config::TIEHASH
302 entering Exporter::import
303 entering Exporter::export
304 exited Exporter::export
305 exited Exporter::import
306 exited main::BEGIN
307 entering Config::myconfig
308 entering Config::FETCH
309 exited Config::FETCH
310 entering Config::FETCH
311 exited Config::FETCH
312 entering Config::FETCH
055fd3a9 313
d5e42f17 314=item 3
055fd3a9 315
f185f654
KW
316 in $=main::BEGIN() from /dev/null:0
317 in $=Config::BEGIN() from lib/Config.pm:2
318 Package lib/Exporter.pm.
319 Package lib/Carp.pm.
320 Package lib/Config.pm.
321 in $=Config::TIEHASH('Config') from lib/Config.pm:644
322 in $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
323 in $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from li
324 in @=Config::myconfig() from /dev/null:0
325 in $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
326 in $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
327 in $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
328 in $=Config::FETCH(ref(Config), 'PERL_SUBVERSION') from lib/Config.pm:574
329 in $=Config::FETCH(ref(Config), 'osname') from lib/Config.pm:574
330 in $=Config::FETCH(ref(Config), 'osvers') from lib/Config.pm:574
055fd3a9 331
d5e42f17 332=item 4
055fd3a9 333
f185f654
KW
334 in $=main::BEGIN() from /dev/null:0
335 in $=Config::BEGIN() from lib/Config.pm:2
336 Package lib/Exporter.pm.
337 Package lib/Carp.pm.
338 out $=Config::BEGIN() from lib/Config.pm:0
339 Package lib/Config.pm.
340 in $=Config::TIEHASH('Config') from lib/Config.pm:644
341 out $=Config::TIEHASH('Config') from lib/Config.pm:644
342 in $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
343 in $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/
344 out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/
345 out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
346 out $=main::BEGIN() from /dev/null:0
347 in @=Config::myconfig() from /dev/null:0
348 in $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
349 out $=Config::FETCH(ref(Config), 'package') from lib/Config.pm:574
350 in $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
351 out $=Config::FETCH(ref(Config), 'baserev') from lib/Config.pm:574
352 in $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
353 out $=Config::FETCH(ref(Config), 'PERL_VERSION') from lib/Config.pm:574
354 in $=Config::FETCH(ref(Config), 'PERL_SUBVERSION') from lib/Config.pm:574
055fd3a9 355
d5e42f17 356=item 5
055fd3a9 357
f185f654
KW
358 in $=main::BEGIN() from /dev/null:0
359 in $=Config::BEGIN() from lib/Config.pm:2
360 Package lib/Exporter.pm.
361 Package lib/Carp.pm.
362 out $=Config::BEGIN() from lib/Config.pm:0
363 Package lib/Config.pm.
364 in $=Config::TIEHASH('Config') from lib/Config.pm:644
365 out $=Config::TIEHASH('Config') from lib/Config.pm:644
366 in $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
367 in $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/E
368 out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/E
369 out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
370 out $=main::BEGIN() from /dev/null:0
371 in @=Config::myconfig() from /dev/null:0
372 in $=Config::FETCH('Config=HASH(0x1aa444)', 'package') from lib/Config.pm:574
373 out $=Config::FETCH('Config=HASH(0x1aa444)', 'package') from lib/Config.pm:574
374 in $=Config::FETCH('Config=HASH(0x1aa444)', 'baserev') from lib/Config.pm:574
375 out $=Config::FETCH('Config=HASH(0x1aa444)', 'baserev') from lib/Config.pm:574
055fd3a9 376
d5e42f17 377=item 6
055fd3a9 378
f185f654
KW
379 in $=CODE(0x15eca4)() from /dev/null:0
380 in $=CODE(0x182528)() from lib/Config.pm:2
381 Package lib/Exporter.pm.
382 out $=CODE(0x182528)() from lib/Config.pm:0
383 scalar context return from CODE(0x182528): undef
384 Package lib/Config.pm.
385 in $=Config::TIEHASH('Config') from lib/Config.pm:628
386 out $=Config::TIEHASH('Config') from lib/Config.pm:628
387 scalar context return from Config::TIEHASH: empty hash
388 in $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
389 in $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/Exporter.pm:171
390 out $=Exporter::export('Config', 'main', 'myconfig', 'config_vars') from lib/Exporter.pm:171
391 scalar context return from Exporter::export: ''
392 out $=Exporter::import('Config', 'myconfig', 'config_vars') from /dev/null:0
393 scalar context return from Exporter::import: ''
055fd3a9
GS
394
395=back
396
397In all cases shown above, the line indentation shows the call tree.
398If bit 2 of C<frame> is set, a line is printed on exit from a
399subroutine as well. If bit 4 is set, the arguments are printed
400along with the caller info. If bit 8 is set, the arguments are
401printed even if they are tied or references. If bit 16 is set, the
402return value is printed, too.
403
404When a package is compiled, a line like this
405
406 Package lib/Carp.pm.
407
408is printed with proper indentation.
409
7b406369 410=head1 Debugging Regular Expressions
055fd3a9
GS
411
412There are two ways to enable debugging output for regular expressions.
413
414If your perl is compiled with C<-DDEBUGGING>, you may use the
415B<-Dr> flag on the command line.
416
417Otherwise, one can C<use re 'debug'>, which has effects at
3d71525d
NJ
418compile time and run time. Since Perl 5.9.5, this pragma is lexically
419scoped.
055fd3a9 420
7b406369 421=head2 Compile-time Output
055fd3a9
GS
422
423The debugging output at compile time looks like this:
424
ccf3535a 425 Compiling REx '[bc]d(ef*g)+h[ij]k$'
1c102323
MJD
426 size 45 Got 364 bytes for offset annotations.
427 first at 1
428 rarest char g at 0
429 rarest char d at 0
430 1: ANYOF[bc](12)
431 12: EXACT <d>(14)
432 14: CURLYX[0] {1,32767}(28)
433 16: OPEN1(18)
434 18: EXACT <e>(20)
435 20: STAR(23)
436 21: EXACT <f>(0)
437 23: EXACT <g>(25)
438 25: CLOSE1(27)
439 27: WHILEM[1/1](0)
440 28: NOTHING(29)
441 29: EXACT <h>(31)
442 31: ANYOF[ij](42)
443 42: EXACT <k>(44)
444 44: EOL(45)
445 45: END(0)
ccf3535a
JK
446 anchored 'de' at 1 floating 'gh' at 3..2147483647 (checking floating)
447 stclass 'ANYOF[bc]' minlen 7
1c102323
MJD
448 Offsets: [45]
449 1[4] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 5[1]
450 0[0] 12[1] 0[0] 6[1] 0[0] 7[1] 0[0] 9[1] 8[1] 0[0] 10[1] 0[0]
451 11[1] 0[0] 12[0] 12[0] 13[1] 0[0] 14[4] 0[0] 0[0] 0[0] 0[0]
452 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 18[1] 0[0] 19[1] 20[0]
453 Omitting $` $& $' support.
055fd3a9
GS
454
455The first line shows the pre-compiled form of the regex. The second
456shows the size of the compiled form (in arbitrary units, usually
1c102323
MJD
4574-byte words) and the total number of bytes allocated for the
458offset/length table, usually 4+C<size>*8. The next line shows the
459label I<id> of the first node that does a match.
055fd3a9 460
1c102323
MJD
461The
462
ccf3535a
JK
463 anchored 'de' at 1 floating 'gh' at 3..2147483647 (checking floating)
464 stclass 'ANYOF[bc]' minlen 7
1c102323
MJD
465
466line (split into two lines above) contains optimizer
055fd3a9
GS
467information. In the example shown, the optimizer found that the match
468should contain a substring C<de> at offset 1, plus substring C<gh>
469at some offset between 3 and infinity. Moreover, when checking for
470these substrings (to abandon impossible matches quickly), Perl will check
471for the substring C<gh> before checking for the substring C<de>. The
472optimizer may also use the knowledge that the match starts (at the
1c102323
MJD
473C<first> I<id>) with a character class, and no string
474shorter than 7 characters can possibly match.
055fd3a9 475
1c102323 476The fields of interest which may appear in this line are
055fd3a9 477
13a2d996 478=over 4
055fd3a9
GS
479
480=item C<anchored> I<STRING> C<at> I<POS>
481
482=item C<floating> I<STRING> C<at> I<POS1..POS2>
483
484See above.
485
486=item C<matching floating/anchored>
487
488Which substring to check first.
489
490=item C<minlen>
491
492The minimal length of the match.
493
494=item C<stclass> I<TYPE>
495
496Type of first matching node.
497
498=item C<noscan>
499
500Don't scan for the found substrings.
501
502=item C<isall>
503
1c102323 504Means that the optimizer information is all that the regular
055fd3a9
GS
505expression contains, and thus one does not need to enter the regex engine at
506all.
507
508=item C<GPOS>
509
510Set if the pattern contains C<\G>.
511
512=item C<plus>
513
514Set if the pattern starts with a repeated char (as in C<x+y>).
515
516=item C<implicit>
517
518Set if the pattern starts with C<.*>.
519
520=item C<with eval>
521
522Set if the pattern contain eval-groups, such as C<(?{ code })> and
523C<(??{ code })>.
524
525=item C<anchored(TYPE)>
526
7b406369 527If the pattern may match only at a handful of places, with C<TYPE>
d3d47aac 528being C<SBOL>, C<MBOL>, or C<GPOS>. See the table below.
055fd3a9
GS
529
530=back
531
532If a substring is known to match at end-of-line only, it may be
ccf3535a 533followed by C<$>, as in C<floating 'k'$>.
055fd3a9 534
1c102323
MJD
535The optimizer-specific information is used to avoid entering (a slow) regex
536engine on strings that will not definitely match. If the C<isall> flag
055fd3a9
GS
537is set, a call to the regex engine may be avoided even when the optimizer
538found an appropriate place for the match.
539
1c102323 540Above the optimizer section is the list of I<nodes> of the compiled
055fd3a9
GS
541form of the regex. Each line has format
542
543C< >I<id>: I<TYPE> I<OPTIONAL-INFO> (I<next-id>)
544
7b406369 545=head2 Types of Nodes
055fd3a9 546
78465a4b 547Here are the current possible types, with short descriptions:
055fd3a9 548
65aa4ca7
FC
549=for comment
550This table is generated by regen/regcomp.pl. Any changes made here
551will be lost.
552
553=for regcomp.pl begin
554
5da6b59a
KW
555 # TYPE arg-description [num-args] [longjump-len] DESCRIPTION
556
557 # Exit points
65aa4ca7 558
7bc66b18
KW
559 END no End of program.
560 SUCCEED no Return from a subroutine, basically.
5da6b59a 561
d3d47aac
YO
562 # Line Start Anchors:
563 SBOL no Match "" at beginning of line: /^/, /\A/
564 MBOL no Same, assuming multiline: /^/m
5da6b59a 565
d3d47aac
YO
566 # Line End Anchors:
567 SEOL no Match "" at end of line: /$/
568 MEOL no Same, assuming multiline: /$/m
569 EOS no Match "" at end of string: /\z/
570
571 # Match Start Anchors:
572 GPOS no Matches where last m//g left off.
573
574 # Word Boundary Opcodes:
c440a570
KW
575 BOUND no Like BOUNDA for non-utf8, otherwise match ""
576 between any Unicode \w\W or \W\w
577 BOUNDL no Like BOUND/BOUNDU, but \w and \W are defined
578 by current locale
64935bc6
KW
579 BOUNDU no Match "" at any boundary of a given type
580 using Unicode rules
c440a570
KW
581 BOUNDA no Match "" at any boundary between \w\W or
582 \W\w, where \w is [_a-zA-Z0-9]
583 NBOUND no Like NBOUNDA for non-utf8, otherwise match
584 "" between any Unicode \w\w or \W\W
585 NBOUNDL no Like NBOUND/NBOUNDU, but \w and \W are
586 defined by current locale
587 NBOUNDU no Match "" at any non-boundary of a given type
64935bc6 588 using using Unicode rules
c440a570
KW
589 NBOUNDA no Match "" betweeen any \w\w or \W\W, where \w
590 is [_a-zA-Z0-9]
5da6b59a
KW
591
592 # [Special] alternatives:
7bc66b18
KW
593 REG_ANY no Match any one character (except newline).
594 SANY no Match any one character.
975a06f7 595 ANYOF sv 1 Match character in (or not in) this class,
7bc66b18 596 single char match only
ac44c12e 597 ANYOFD sv 1 Like ANYOF, but /d is in effect
a4525e78 598 ANYOFL sv 1 Like ANYOF, but /l is in effect
7bc66b18 599
d3d47aac 600 # POSIX Character Classes:
7bc66b18
KW
601 POSIXD none Some [[:class:]] under /d; the FLAGS field
602 gives which one
603 POSIXL none Some [[:class:]] under /l; the FLAGS field
604 gives which one
605 POSIXU none Some [[:class:]] under /u; the FLAGS field
606 gives which one
607 POSIXA none Some [[:class:]] under /a; the FLAGS field
608 gives which one
609 NPOSIXD none complement of POSIXD, [[:^class:]]
610 NPOSIXL none complement of POSIXL, [[:^class:]]
611 NPOSIXU none complement of POSIXU, [[:^class:]]
612 NPOSIXA none complement of POSIXA, [[:^class:]]
613
614 CLUMP no Match any extended grapheme cluster sequence
5da6b59a
KW
615
616 # Alternation
617
65aa4ca7
FC
618 # BRANCH The set of branches constituting a single choice are
619 # hooked together with their "next" pointers, since
620 # precedence prevents anything being concatenated to
621 # any individual branch. The "next" pointer of the last
622 # BRANCH in a choice points to the thing following the
623 # whole choice. This is also where the final "next"
624 # pointer of each individual branch points; each branch
625 # starts with the operand node of a BRANCH node.
5da6b59a 626 #
7bc66b18 627 BRANCH node Match this alternative, or the next...
5da6b59a 628
5da6b59a
KW
629 # Literals
630
7bc66b18 631 EXACT str Match this string (preceded by length).
b24abbc8
KW
632 EXACTL str Like EXACT, but /l is in effect (used so
633 locale-related warnings can be checked for).
7bc66b18
KW
634 EXACTF str Match this non-UTF-8 string (not guaranteed
635 to be folded) using /id rules (w/len).
636 EXACTFL str Match this string (not guaranteed to be
637 folded) using /il rules (w/len).
638 EXACTFU str Match this string (folded iff in UTF-8,
639 length in folding doesn't change if not in
640 UTF-8) using /iu rules (w/len).
641 EXACTFA str Match this string (not guaranteed to be
642 folded) using /iaa rules (w/len).
a4525e78 643
7bc66b18
KW
644 EXACTFU_SS str Match this string (folded iff in UTF-8,
645 length in folding may change even if not in
646 UTF-8) using /iu rules (w/len).
a4525e78
KW
647 EXACTFLU8 str Rare cirucmstances: like EXACTFU, but is
648 under /l, UTF-8, folded, and everything in
649 it is above 255.
7bc66b18
KW
650 EXACTFA_NO_TRIE str Match this string (which is not trie-able;
651 not guaranteed to be folded) using /iaa
652 rules (w/len).
5da6b59a
KW
653
654 # Do nothing types
655
7bc66b18 656 NOTHING no Match empty string.
5da6b59a 657 # A variant of above which delimits a group, thus stops optimizations
7bc66b18
KW
658 TAIL no Match empty string. Can jump here from
659 outside.
5da6b59a
KW
660
661 # Loops
662
65aa4ca7 663 # STAR,PLUS '?', and complex '*' and '+', are implemented as
62e6ef33 664 # circular BRANCH structures. Simple cases
65aa4ca7
FC
665 # (one character per match) are implemented with STAR
666 # and PLUS for speed and to minimize recursive plunges.
5da6b59a 667 #
7bc66b18
KW
668 STAR node Match this (simple) thing 0 or more times.
669 PLUS node Match this (simple) thing 1 or more times.
670
671 CURLY sv 2 Match this simple thing {n,m} times.
672 CURLYN no 2 Capture next-after-this simple thing
673 CURLYM no 2 Capture this medium-complex thing {n,m}
674 times.
675 CURLYX sv 2 Match this complex thing {n,m} times.
5da6b59a
KW
676
677 # This terminator creates a loop structure for CURLYX
7bc66b18 678 WHILEM no Do curly processing and see if rest matches.
5da6b59a
KW
679
680 # Buffer related
681
682 # OPEN,CLOSE,GROUPP ...are numbered at compile time.
7bc66b18 683 OPEN num 1 Mark this point in input as start of #n.
48e88d42 684 CLOSE num 1 Close corresponding OPEN of #n.
07093db4
KW
685 SROPEN none Same as OPEN, but for script run
686 SRCLOSE none Close preceding SROPEN
7bc66b18
KW
687
688 REF num 1 Match some already matched string
689 REFF num 1 Match already matched string, folded using
850b7ec9 690 native charset rules for non-utf8
7bc66b18
KW
691 REFFL num 1 Match already matched string, folded in loc.
692 REFFU num 1 Match already matched string, folded using
850b7ec9 693 unicode rules for non-utf8
7bc66b18 694 REFFA num 1 Match already matched string, folded using
850b7ec9
KW
695 unicode rules for non-utf8, no mixing ASCII,
696 non-ASCII
65aa4ca7
FC
697
698 # Named references. Code in regcomp.c assumes that these all are after
699 # the numbered references
7bc66b18
KW
700 NREF no-sv 1 Match some already matched string
701 NREFF no-sv 1 Match already matched string, folded using
850b7ec9 702 native charset rules for non-utf8
7bc66b18
KW
703 NREFFL no-sv 1 Match already matched string, folded in loc.
704 NREFFU num 1 Match already matched string, folded using
850b7ec9 705 unicode rules for non-utf8
7bc66b18 706 NREFFA num 1 Match already matched string, folded using
850b7ec9
KW
707 unicode rules for non-utf8, no mixing ASCII,
708 non-ASCII
7bc66b18 709
d3d47aac
YO
710 # Support for long RE
711 LONGJMP off 1 1 Jump far away.
712 BRANCHJ off 1 1 BRANCH with long offset.
713
714 # Special Case Regops
7bc66b18
KW
715 IFMATCH off 1 2 Succeeds if the following matches.
716 UNLESSM off 1 2 Fails if the following matches.
717 SUSPEND off 1 1 "Independent" sub-RE.
718 IFTHEN off 1 1 Switch, should be preceded by switcher.
719 GROUPP num 1 Whether the group matched.
5da6b59a 720
5da6b59a
KW
721 # The heavy worker
722
13f27704
KW
723 EVAL evl/flags Execute some Perl code.
724 2L
5da6b59a
KW
725
726 # Modifiers
727
7bc66b18
KW
728 MINMOD no Next operator is not greedy.
729 LOGICAL no Next opcode should set the flag only.
5da6b59a
KW
730
731 # This is not used yet
7bc66b18 732 RENUM off 1 1 Group with independently numbered parens.
5da6b59a
KW
733
734 # Trie Related
735
65aa4ca7
FC
736 # Behave the same as A|LIST|OF|WORDS would. The '..C' variants
737 # have inline charclass data (ascii only), the 'C' store it in the
738 # structure.
5da6b59a 739
7bc66b18
KW
740 TRIE trie 1 Match many EXACT(F[ALU]?)? at once.
741 flags==type
742 TRIEC trie Same as TRIE, but with embedded charclass
743 charclass data
5da6b59a 744
7bc66b18
KW
745 AHOCORASICK trie 1 Aho Corasick stclass. flags==type
746 AHOCORASICKC trie Same as AHOCORASICK, but with embedded
747 charclass charclass data
5da6b59a
KW
748
749 # Regex Subroutines
7bc66b18 750 GOSUB num/ofs 2L recurse to paren arg1 at (signed) ofs arg2
5da6b59a
KW
751
752 # Special conditionals
7bc66b18
KW
753 NGROUPP no-sv 1 Whether the group matched.
754 INSUBP num 1 Whether we are in a specific recurse.
755 DEFINEP none 1 Never execute directly.
5da6b59a
KW
756
757 # Backtracking Verbs
7bc66b18 758 ENDLIKE none Used only for the type field of verbs
fee50582
YO
759 OPFAIL no-sv 1 Same as (?!), but with verb arg
760 ACCEPT no-sv/num Accepts the current matched string, with
761 2L verbar
5da6b59a
KW
762
763 # Verbs With Arguments
7bc66b18
KW
764 VERB no-sv 1 Used only for the type field of verbs
765 PRUNE no-sv 1 Pattern fails at this startpoint if no-
766 backtracking through this
767 MARKPOINT no-sv 1 Push the current location for rollback by
768 cut.
769 SKIP no-sv 1 On failure skip forward (to the mark) before
770 retrying
771 COMMIT no-sv 1 Pattern fails outright if backtracking
772 through this
773 CUTGROUP no-sv 1 On failure go to the next alternation in the
774 group
5da6b59a
KW
775
776 # Control what to keep in $&.
7bc66b18 777 KEEPS no $& begins here.
5da6b59a
KW
778
779 # New charclass like patterns
7bc66b18 780 LNBREAK none generic newline pattern
5da6b59a
KW
781
782 # SPECIAL REGOPS
783
65aa4ca7
FC
784 # This is not really a node, but an optimized away piece of a "long"
785 # node. To simplify debugging output, we mark it as if it were a node
7bc66b18 786 OPTIMIZED off Placeholder for dump.
5da6b59a
KW
787
788 # Special opcode with the property that no opcode in a compiled program
789 # will ever be of this type. Thus it can be used as a flag value that
790 # no other opcode has been seen. END is used similarly, in that an END
65aa4ca7
FC
791 # node cant be optimized. So END implies "unoptimizable" and PSEUDO
792 # mean "not seen anything to optimize yet".
7bc66b18 793 PSEUDO off Pseudo opcode for internal use.
65aa4ca7
FC
794
795=for regcomp.pl end
055fd3a9 796
1c102323
MJD
797=for unprinted-credits
798Next section M-J. Dominus (mjd-perl-patch+@plover.com) 20010421
799
800Following the optimizer information is a dump of the offset/length
801table, here split across several lines:
802
803 Offsets: [45]
804 1[4] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 5[1]
805 0[0] 12[1] 0[0] 6[1] 0[0] 7[1] 0[0] 9[1] 8[1] 0[0] 10[1] 0[0]
806 11[1] 0[0] 12[0] 12[0] 13[1] 0[0] 14[4] 0[0] 0[0] 0[0] 0[0]
807 0[0] 0[0] 0[0] 0[0] 0[0] 0[0] 18[1] 0[0] 19[1] 20[0]
808
809The first line here indicates that the offset/length table contains 45
810entries. Each entry is a pair of integers, denoted by C<offset[length]>.
17c338f3 811Entries are numbered starting with 1, so entry #1 here is C<1[4]> and
1c102323
MJD
812entry #12 is C<5[1]>. C<1[4]> indicates that the node labeled C<1:>
813(the C<1: ANYOF[bc]>) begins at character position 1 in the
814pre-compiled form of the regex, and has a length of 4 characters.
815C<5[1]> in position 12
816indicates that the node labeled C<12:>
817(the C<< 12: EXACT <d> >>) begins at character position 5 in the
818pre-compiled form of the regex, and has a length of 1 character.
819C<12[1]> in position 14
820indicates that the node labeled C<14:>
821(the C<< 14: CURLYX[0] {1,32767} >>) begins at character position 12 in the
822pre-compiled form of the regex, and has a length of 1 character---that
823is, it corresponds to the C<+> symbol in the precompiled regex.
824
825C<0[0]> items indicate that there is no corresponding node.
826
7b406369 827=head2 Run-time Output
055fd3a9
GS
828
829First of all, when doing a match, one may get no run-time output even
830if debugging is enabled. This means that the regex engine was never
831entered and that all of the job was therefore done by the optimizer.
832
833If the regex engine was entered, the output may look like this:
834
ccf3535a 835 Matching '[bc]d(ef*g)+h[ij]k$' against 'abcdefg__gh__'
055fd3a9
GS
836 Setting an EVAL scope, savestack=3
837 2 <ab> <cdefg__gh_> | 1: ANYOF
838 3 <abc> <defg__gh_> | 11: EXACT <d>
839 4 <abcd> <efg__gh_> | 13: CURLYX {1,32767}
840 4 <abcd> <efg__gh_> | 26: WHILEM
841 0 out of 1..32767 cc=effff31c
842 4 <abcd> <efg__gh_> | 15: OPEN1
843 4 <abcd> <efg__gh_> | 17: EXACT <e>
844 5 <abcde> <fg__gh_> | 19: STAR
845 EXACT <f> can match 1 times out of 32767...
846 Setting an EVAL scope, savestack=3
847 6 <bcdef> <g__gh__> | 22: EXACT <g>
848 7 <bcdefg> <__gh__> | 24: CLOSE1
849 7 <bcdefg> <__gh__> | 26: WHILEM
850 1 out of 1..32767 cc=effff31c
851 Setting an EVAL scope, savestack=12
852 7 <bcdefg> <__gh__> | 15: OPEN1
853 7 <bcdefg> <__gh__> | 17: EXACT <e>
854 restoring \1 to 4(4)..7
855 failed, try continuation...
856 7 <bcdefg> <__gh__> | 27: NOTHING
857 7 <bcdefg> <__gh__> | 28: EXACT <h>
858 failed...
859 failed...
860
861The most significant information in the output is about the particular I<node>
862of the compiled regex that is currently being tested against the target string.
863The format of these lines is
864
865C< >I<STRING-OFFSET> <I<PRE-STRING>> <I<POST-STRING>> |I<ID>: I<TYPE>
866
867The I<TYPE> info is indented with respect to the backtracking level.
868Other incidental information appears interspersed within.
869
7b406369 870=head1 Debugging Perl Memory Usage
055fd3a9
GS
871
872Perl is a profligate wastrel when it comes to memory use. There
873is a saying that to estimate memory usage of Perl, assume a reasonable
874algorithm for memory allocation, multiply that estimate by 10, and
875while you still may miss the mark, at least you won't be quite so
4375e838 876astonished. This is not absolutely true, but may provide a good
055fd3a9
GS
877grasp of what happens.
878
879Assume that an integer cannot take less than 20 bytes of memory, a
880float cannot take less than 24 bytes, a string cannot take less
881than 32 bytes (all these examples assume 32-bit architectures, the
882result are quite a bit worse on 64-bit architectures). If a variable
883is accessed in two of three different ways (which require an integer,
884a float, or a string), the memory footprint may increase yet another
b9449ee0 88520 bytes. A sloppy malloc(3) implementation can inflate these
055fd3a9
GS
886numbers dramatically.
887
888On the opposite end of the scale, a declaration like
889
890 sub foo;
891
892may take up to 500 bytes of memory, depending on which release of Perl
893you're running.
894
895Anecdotal estimates of source-to-compiled code bloat suggest an
896eightfold increase. This means that the compiled form of reasonable
897(normally commented, properly indented etc.) code will take
898about eight times more space in memory than the code took
899on disk.
900
b30f304a
JH
901The B<-DL> command-line switch is obsolete since circa Perl 5.6.0
902(it was available only if Perl was built with C<-DDEBUGGING>).
903The switch was used to track Perl's memory allocations and possible
904memory leaks. These days the use of malloc debugging tools like
5b6a3331 905F<Purify> or F<valgrind> is suggested instead. See also
7b406369 906L<perlhacktips/PERL_MEM_LOG>.
b30f304a
JH
907
908One way to find out how much memory is being used by Perl data
909structures is to install the Devel::Size module from CPAN: it gives
910you the minimum number of bytes required to store a particular data
911structure. Please be mindful of the difference between the size()
912and total_size().
913
914If Perl has been compiled using Perl's malloc you can analyze Perl
7b406369 915memory usage by setting $ENV{PERL_DEBUG_MSTATS}.
055fd3a9
GS
916
917=head2 Using C<$ENV{PERL_DEBUG_MSTATS}>
918
919If your perl is using Perl's malloc() and was compiled with the
920necessary switches (this is the default), then it will print memory
4375e838 921usage statistics after compiling your code when C<< $ENV{PERL_DEBUG_MSTATS}
055fd3a9
GS
922> 1 >>, and before termination of the program when C<<
923$ENV{PERL_DEBUG_MSTATS} >= 1 >>. The report format is similar to
924the following example:
925
f185f654
KW
926 $ PERL_DEBUG_MSTATS=2 perl -e "require Carp"
927 Memory allocation statistics after compilation: (buckets 4(4)..8188(8192)
928 14216 free: 130 117 28 7 9 0 2 2 1 0 0
055fd3a9 929 437 61 36 0 5
f185f654 930 60924 used: 125 137 161 55 7 8 6 16 2 0 1
055fd3a9 931 74 109 304 84 20
f185f654
KW
932 Total sbrk(): 77824/21:119. Odd ends: pad+heads+chain+tail: 0+636+0+2048.
933 Memory allocation statistics after execution: (buckets 4(4)..8188(8192)
934 30888 free: 245 78 85 13 6 2 1 3 2 0 1
055fd3a9 935 315 162 39 42 11
f185f654 936 175816 used: 265 176 1112 111 26 22 11 27 2 1 1
055fd3a9 937 196 178 1066 798 39
f185f654 938 Total sbrk(): 215040/47:145. Odd ends: pad+heads+chain+tail: 0+2192+0+6144.
055fd3a9
GS
939
940It is possible to ask for such a statistic at arbitrary points in
b9449ee0 941your execution using the mstat() function out of the standard
055fd3a9
GS
942Devel::Peek module.
943
944Here is some explanation of that format:
945
13a2d996 946=over 4
055fd3a9
GS
947
948=item C<buckets SMALLEST(APPROX)..GREATEST(APPROX)>
949
950Perl's malloc() uses bucketed allocations. Every request is rounded
951up to the closest bucket size available, and a bucket is taken from
952the pool of buckets of that size.
953
954The line above describes the limits of buckets currently in use.
955Each bucket has two sizes: memory footprint and the maximal size
956of user data that can fit into this bucket. Suppose in the above
957example that the smallest bucket were size 4. The biggest bucket
958would have usable size 8188, and the memory footprint would be 8192.
959
960In a Perl built for debugging, some buckets may have negative usable
961size. This means that these buckets cannot (and will not) be used.
962For larger buckets, the memory footprint may be one page greater
7b406369 963than a power of 2. If so, the corresponding power of two is
055fd3a9
GS
964printed in the C<APPROX> field above.
965
966=item Free/Used
967
968The 1 or 2 rows of numbers following that correspond to the number
969of buckets of each size between C<SMALLEST> and C<GREATEST>. In
970the first row, the sizes (memory footprints) of buckets are powers
971of two--or possibly one page greater. In the second row, if present,
972the memory footprints of the buckets are between the memory footprints
973of two buckets "above".
974
4375e838 975For example, suppose under the previous example, the memory footprints
055fd3a9
GS
976were
977
f185f654 978 free: 8 16 32 64 128 256 512 1024 2048 4096 8192
055fd3a9
GS
979 4 12 24 48 80
980
7b406369 981With a non-C<DEBUGGING> perl, the buckets starting from C<128> have
d1be9408 982a 4-byte overhead, and thus an 8192-long bucket may take up to
055fd3a9
GS
9838188-byte allocations.
984
985=item C<Total sbrk(): SBRKed/SBRKs:CONTINUOUS>
986
987The first two fields give the total amount of memory perl sbrk(2)ed
988(ess-broken? :-) and number of sbrk(2)s used. The third number is
989what perl thinks about continuity of returned chunks. So long as
990this number is positive, malloc() will assume that it is probable
991that sbrk(2) will provide continuous memory.
992
993Memory allocated by external libraries is not counted.
994
995=item C<pad: 0>
996
997The amount of sbrk(2)ed memory needed to keep buckets aligned.
998
999=item C<heads: 2192>
1000
1001Although memory overhead of bigger buckets is kept inside the bucket, for
1002smaller buckets, it is kept in separate areas. This field gives the
1003total size of these areas.
1004
1005=item C<chain: 0>
1006
1007malloc() may want to subdivide a bigger bucket into smaller buckets.
1008If only a part of the deceased bucket is left unsubdivided, the rest
1009is kept as an element of a linked list. This field gives the total
1010size of these chunks.
1011
1012=item C<tail: 6144>
1013
1014To minimize the number of sbrk(2)s, malloc() asks for more memory. This
1015field gives the size of the yet unused part, which is sbrk(2)ed, but
1016never touched.
1017
1018=back
1019
055fd3a9
GS
1020=head1 SEE ALSO
1021
1022L<perldebug>,
1023L<perlguts>,
1024L<perlrun>
1025L<re>,
1026and
fe854a6f 1027L<Devel::DProf>.