This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
typo fix for reapi pod
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
5a2b28ce 3perlreapi - Perl regular expression plugin interface
108003db
RGS
4
5=head1 DESCRIPTION
6
5a2b28ce
KW
7As of Perl 5.9.5 there is a new interface for plugging and using
8regular expression engines other than the default one.
a0e97681
RGS
9
10Each engine is supposed to provide access to a constant structure of the
11following format:
108003db
RGS
12
13 typedef struct regexp_engine {
02c01adb
KW
14 REGEXP* (*comp) (pTHX_
15 const SV * const pattern, const U32 flags);
16 I32 (*exec) (pTHX_
17 REGEXP * const rx,
18 char* stringarg,
19 char* strend, char* strbeg,
20 I32 minend, SV* screamer,
2fdbfb4d 21 void* data, U32 flags);
02c01adb
KW
22 char* (*intuit) (pTHX_
23 REGEXP * const rx, SV *sv,
24 char *strpos, char *strend, U32 flags,
2fdbfb4d 25 struct re_scream_pos_data_s *data);
49d7dfbc
AB
26 SV* (*checkstr) (pTHX_ REGEXP * const rx);
27 void (*free) (pTHX_ REGEXP * const rx);
02c01adb
KW
28 void (*numbered_buff_FETCH) (pTHX_
29 REGEXP * const rx,
30 const I32 paren,
31 SV * const sv);
32 void (*numbered_buff_STORE) (pTHX_
33 REGEXP * const rx,
34 const I32 paren,
5a2b28ce 35 SV const * const value);
02c01adb
KW
36 I32 (*numbered_buff_LENGTH) (pTHX_
37 REGEXP * const rx,
38 const SV * const sv,
39 const I32 paren);
40 SV* (*named_buff) (pTHX_
41 REGEXP * const rx,
42 SV * const key,
43 SV * const value,
44 U32 flags);
45 SV* (*named_buff_iter) (pTHX_
46 REGEXP * const rx,
47 const SV * const lastkey,
192b9cd1 48 const U32 flags);
49d7dfbc 49 SV* (*qr_package)(pTHX_ REGEXP * const rx);
108003db 50 #ifdef USE_ITHREADS
49d7dfbc 51 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db 52 #endif
3c13cae6
DM
53 REGEXP* (*op_comp) (...);
54
108003db
RGS
55
56When a regexp is compiled, its C<engine> field is then set to point at
a0e97681 57the appropriate structure, so that when it needs to be used Perl can find
108003db
RGS
58the right routines to do so.
59
60In order to install a new regexp handler, C<$^H{regcomp}> is set
61to an integer which (when casted appropriately) resolves to one of these
2edc787c 62structures. When compiling, the C<comp> method is executed, and the
5a2b28ce 63resulting C<regexp> structure's engine field is expected to point back at
108003db
RGS
64the same structure.
65
5a2b28ce 66The pTHX_ symbol in the definition is a macro used by Perl under threading
108003db
RGS
67to provide an extra argument to the routine holding a pointer back to
68the interpreter that is executing the regexp. So under threading all
69routines get an extra argument.
70
882227b7 71=head1 Callbacks
108003db
RGS
72
73=head2 comp
74
3ab4a224 75 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 76
3ab4a224
AB
77Compile the pattern stored in C<pattern> using the given C<flags> and
78return a pointer to a prepared C<REGEXP> structure that can perform
2edc787c 79the match. See L</The REGEXP structure> below for an explanation of
3ab4a224
AB
80the individual fields in the REGEXP struct.
81
82The C<pattern> parameter is the scalar that was used as the
5a2b28ce
KW
83pattern. Previous versions of Perl would pass two C<char*> indicating
84the start and end of the stringified pattern; the following snippet can
3ab4a224
AB
85be used to get the old parameters:
86
87 STRLEN plen;
88 char* exp = SvPV(pattern, plen);
89 char* xend = exp + plen;
90
5a2b28ce 91Since any scalar can be passed as a pattern, it's possible to implement
3ab4a224
AB
92an engine that does something with an array (C<< "ook" =~ [ qw/ eek
93hlagh / ] >>) or with the non-stringified form of a compiled regular
5a2b28ce
KW
94expression (C<< "ook" =~ qr/eek/ >>). Perl's own engine will always
95stringify everything using the snippet above, but that doesn't mean
3ab4a224 96other engines have to.
108003db 97
a0e97681 98The C<flags> parameter is a bitfield which indicates which of the
2edc787c 99C<msixp> flags the regex was compiled with. It also contains
5a2b28ce 100additional info, such as if C<use locale> is in effect.
108003db
RGS
101
102The C<eogc> flags are stripped out before being passed to the comp
2edc787c 103routine. The regex engine does not need to know if any of these
5a2b28ce 104are set, as those flags should only affect what Perl does with the
c998b245
AB
105pattern and its match variables, not how it gets compiled and
106executed.
108003db 107
c998b245 108By the time the comp callback is called, some of these flags have
2edc787c 109already had effect (noted below where applicable). However most of
5a2b28ce 110their effect occurs after the comp callback has run, in routines that
c998b245 111read the C<< rx->extflags >> field which it populates.
108003db 112
c998b245
AB
113In general the flags should be preserved in C<< rx->extflags >> after
114compilation, although the regex engine might want to add or delete
2edc787c 115some of them to invoke or disable some special behavior in Perl. The
c998b245 116flags along with any special behavior they cause are documented below:
108003db 117
c998b245 118The pattern modifiers:
108003db 119
c998b245 120=over 4
108003db 121
c998b245 122=item C</m> - RXf_PMf_MULTILINE
108003db 123
c998b245
AB
124If this is in C<< rx->extflags >> it will be passed to
125C<Perl_fbm_instr> by C<pp_split> which will treat the subject string
126as a multi-line string.
108003db 127
c998b245 128=item C</s> - RXf_PMf_SINGLELINE
108003db 129
c998b245 130=item C</i> - RXf_PMf_FOLD
108003db 131
c998b245 132=item C</x> - RXf_PMf_EXTENDED
108003db 133
5a2b28ce 134If present on a regex, C<"#"> comments will be handled differently by the
c998b245 135tokenizer in some cases.
108003db 136
c998b245 137TODO: Document those cases.
108003db 138
c998b245 139=item C</p> - RXf_PMf_KEEPCOPY
108003db 140
e72ec78c
KW
141TODO: Document this
142
f0f9b3b8
KW
143=item Character set
144
145The character set semantics are determined by an enum that is contained
146in this field. This is still experimental and subject to change, but
147the current interface returns the rules by use of the in-line function
148C<get_regex_charset(const U32 flags)>. The only currently documented
149value returned from it is REGEX_LOCALE_CHARSET, which is set if
e72ec78c
KW
150C<use locale> is in effect. If present in C<< rx->extflags >>,
151C<split> will use the locale dependent definition of whitespace
2edc787c 152when RXf_SKIPWHITE or RXf_WHITE is in effect. ASCII whitespace
96090e4f 153is defined as per L<isSPACE|perlapi/isSPACE>, and by the internal
e72ec78c 154macros C<is_utf8_space> under UTF-8, and C<isSPACE_LC> under C<use
c998b245 155locale>.
108003db 156
f0f9b3b8
KW
157=back
158
159Additional flags:
160
161=over 4
162
0ac6acae
AB
163=item RXf_SPLIT
164
9f6ecbda
FC
165This flag was removed in perl 5.18.0. C<split ' '> is now special-cased
166solely in the parser. RXf_SPLIT is still #defined, so you can test for it.
167This is how it used to work:
168
0ac6acae 169If C<split> is invoked as C<split ' '> or with no arguments (which
5a2b28ce 170really means C<split(' ', $_)>, see L<split|perlfunc/split>), Perl will
2edc787c
FC
171set this flag. The regex engine can then check for it and set the
172SKIPWHITE and WHITE extflags. To do this, the Perl engine does:
0ac6acae
AB
173
174 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
175 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
176
108003db
RGS
177=back
178
c998b245
AB
179These flags can be set during compilation to enable optimizations in
180the C<split> operator.
181
182=over 4
183
0ac6acae
AB
184=item RXf_SKIPWHITE
185
51b1fee8
FC
186This flag was removed in perl 5.18.0. It is still #defined, so you can
187set it, but doing so will have no effect. This is how it used to work:
188
0ac6acae
AB
189If the flag is present in C<< rx->extflags >> C<split> will delete
190whitespace from the start of the subject string before it's operated
2edc787c 191on. What is considered whitespace depends on if the subject is a
5a2b28ce 192UTF-8 string and if the C<RXf_PMf_LOCALE> flag is set.
0ac6acae 193
5a2b28ce
KW
194If RXf_WHITE is set in addition to this flag, C<split> will behave like
195C<split " "> under the Perl engine.
0ac6acae 196
c998b245
AB
197=item RXf_START_ONLY
198
199Tells the split operator to split the target string on newlines
200(C<\n>) without invoking the regex engine.
201
202Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp
2edc787c 203== '^'>), even under C</^/s>; see L<split|perlfunc>. Of course a
c998b245
AB
204different regex engine might want to use the same optimizations
205with a different syntax.
206
207=item RXf_WHITE
208
209Tells the split operator to split the target string on whitespace
2edc787c 210without invoking the regex engine. The definition of whitespace varies
5a2b28ce
KW
211depending on if the target string is a UTF-8 string and on
212if RXf_PMf_LOCALE is set.
c998b245 213
0ac6acae 214Perl's engine sets this flag if the pattern is C<\s+>.
c998b245 215
640f820d
AB
216=item RXf_NULL
217
a0e97681 218Tells the split operator to split the target string on
2edc787c 219characters. The definition of character varies depending on if
640f820d
AB
220the target string is a UTF-8 string.
221
222Perl's engine sets this flag on empty patterns, this optimization
2edc787c 223makes C<split //> much faster than it would otherwise be. It's even
640f820d
AB
224faster than C<unpack>.
225
dbc200c5 226=item RXf_NO_INPLACE_SUBST
94bbc698
FC
227
228Added in perl 5.18.0, this flag indicates that a regular expression might
579794ec 229perform an operation that would interfere with inplace substitution. For
dbc200c5
YO
230instance it might contain lookbehind, or assign to non-magical variables
231(such as $REGMARK and $REGERROR) during matching. C<s///> will skip
232certain optimisations when this is set.
94bbc698 233
c998b245 234=back
108003db
RGS
235
236=head2 exec
237
49d7dfbc 238 I32 exec(pTHX_ REGEXP * const rx,
108003db
RGS
239 char *stringarg, char* strend, char* strbeg,
240 I32 minend, SV* screamer,
241 void* data, U32 flags);
242
8fd1a950
DM
243Execute a regexp. The arguments are
244
245=over 4
246
247=item rx
248
249The regular expression to execute.
250
251=item screamer
252
2edc787c 253This strangely-named arg is the SV to be matched against. Note that the
8fd1a950
DM
254actual char array to be matched against is supplied by the arguments
255described below; the SV is just used to determine UTF8ness, C<pos()> etc.
256
257=item strbeg
258
259Pointer to the physical start of the string.
260
261=item strend
262
263Pointer to the character following the physical end of the string (i.e.
5a2b28ce 264the C<\0>).
8fd1a950
DM
265
266=item stringarg
267
268Pointer to the position in the string where matching should start; it might
269not be equal to C<strbeg> (for example in a later iteration of C</.../g>).
270
271=item minend
272
273Minimum length of string (measured in bytes from C<stringarg>) that must
274match; if the engine reaches the end of the match but hasn't reached this
275position in the string, it should fail.
276
277=item data
278
279Optimisation data; subject to change.
280
281=item flags
282
283Optimisation flags; subject to change.
284
285=back
108003db
RGS
286
287=head2 intuit
288
49d7dfbc 289 char* intuit(pTHX_ REGEXP * const rx,
108003db 290 SV *sv, char *strpos, char *strend,
49d7dfbc 291 const U32 flags, struct re_scream_pos_data_s *data);
108003db
RGS
292
293Find the start position where a regex match should be attempted,
5a2b28ce 294or possibly if the regex engine should not be run because the
2edc787c 295pattern can't match. This is called, as appropriate, by the core,
5a2b28ce 296depending on the values of the C<extflags> member of the C<regexp>
108003db
RGS
297structure.
298
299=head2 checkstr
300
49d7dfbc 301 SV* checkstr(pTHX_ REGEXP * const rx);
108003db
RGS
302
303Return a SV containing a string that must appear in the pattern. Used
304by C<split> for optimising matches.
305
306=head2 free
307
49d7dfbc 308 void free(pTHX_ REGEXP * const rx);
108003db 309
5a2b28ce 310Called by Perl when it is freeing a regexp pattern so that the engine
108003db 311can release any resources pointed to by the C<pprivate> member of the
2edc787c 312C<regexp> structure. This is only responsible for freeing private data;
5a2b28ce 313Perl will handle releasing anything else contained in the C<regexp> structure.
108003db 314
192b9cd1 315=head2 Numbered capture callbacks
108003db 316
192b9cd1
AB
317Called to get/set the value of C<$`>, C<$'>, C<$&> and their named
318equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
c27a5cfe 319numbered capture groups (C<$1>, C<$2>, ...).
49d7dfbc 320
c149d39e
DM
321The C<paren> parameter will be C<1> for C<$1>, C<2> for C<$2> and so
322forth, and have these symbolic values for the special variables:
323
324 ${^PREMATCH} RX_BUFF_IDX_CARET_PREMATCH
325 ${^POSTMATCH} RX_BUFF_IDX_CARET_POSTMATCH
326 ${^MATCH} RX_BUFF_IDX_CARET_FULLMATCH
327 $` RX_BUFF_IDX_PREMATCH
328 $' RX_BUFF_IDX_POSTMATCH
329 $& RX_BUFF_IDX_FULLMATCH
330
5a2b28ce 331Note that in Perl 5.17.3 and earlier, the last three constants were also
c149d39e
DM
332used for the caret variants of the variables.
333
49d7dfbc 334
192b9cd1 335The names have been chosen by analogy with L<Tie::Scalar> methods
2edc787c 336names with an additional B<LENGTH> callback for efficiency. However
192b9cd1
AB
337named capture variables are currently not tied internally but
338implemented via magic.
339
340=head3 numbered_buff_FETCH
341
342 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
343 SV * const sv);
344
2edc787c 345Fetch a specified numbered capture. C<sv> should be set to the scalar
192b9cd1 346to return, the scalar is passed as an argument rather than being
5a2b28ce 347returned from the function because when it's called Perl already has a
192b9cd1 348scalar to store the value, creating another one would be
2edc787c 349redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and
192b9cd1 350friends, see L<perlapi>.
49d7dfbc 351
5a2b28ce 352This callback is where Perl untaints its own capture variables under
2edc787c 353taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch>
49d7dfbc
AB
354function in F<regcomp.c> for how to untaint capture variables if
355that's something you'd like your engine to do as well.
108003db 356
192b9cd1 357=head3 numbered_buff_STORE
108003db 358
02c01adb
KW
359 void (*numbered_buff_STORE) (pTHX_
360 REGEXP * const rx,
361 const I32 paren,
2fdbfb4d 362 SV const * const value);
108003db 363
2edc787c
FC
364Set the value of a numbered capture variable. C<value> is the scalar
365that is to be used as the new value. It's up to the engine to make
192b9cd1 366sure this is used as the new value (or reject it).
2fdbfb4d
AB
367
368Example:
369
370 if ("ook" =~ /(o*)/) {
ccf3535a 371 # 'paren' will be '1' and 'value' will be 'ee'
2fdbfb4d
AB
372 $1 =~ tr/o/e/;
373 }
374
375Perl's own engine will croak on any attempt to modify the capture
a0e97681 376variables, to do this in another engine use the following callback
2fdbfb4d
AB
377(copied from C<Perl_reg_numbered_buff_store>):
378
379 void
02c01adb
KW
380 Example_reg_numbered_buff_store(pTHX_
381 REGEXP * const rx,
382 const I32 paren,
383 SV const * const value)
2fdbfb4d
AB
384 {
385 PERL_UNUSED_ARG(rx);
386 PERL_UNUSED_ARG(paren);
387 PERL_UNUSED_ARG(value);
388
389 if (!PL_localizing)
390 Perl_croak(aTHX_ PL_no_modify);
391 }
392
5a2b28ce 393Actually Perl will not I<always> croak in a statement that looks
2edc787c 394like it would modify a numbered capture variable. This is because the
5a2b28ce 395STORE callback will not be called if Perl can determine that it
2edc787c 396doesn't have to modify the value. This is exactly how tied variables
2fdbfb4d
AB
397behave in the same situation:
398
399 package CaptureVar;
400 use base 'Tie::Scalar';
401
402 sub TIESCALAR { bless [] }
403 sub FETCH { undef }
404 sub STORE { die "This doesn't get called" }
405
406 package main;
407
c69ca1d4 408 tie my $sv => "CaptureVar";
2fdbfb4d
AB
409 $sv =~ y/a/b/;
410
5a2b28ce 411Because C<$sv> is C<undef> when the C<y///> operator is applied to it,
2fdbfb4d 412the transliteration won't actually execute and the program won't
2edc787c 413C<die>. This is different to how 5.8 and earlier versions behaved
5a2b28ce 414since the capture variables were READONLY variables then; now they'll
192b9cd1 415just die when assigned to in the default engine.
2fdbfb4d 416
192b9cd1 417=head3 numbered_buff_LENGTH
2fdbfb4d 418
02c01adb
KW
419 I32 numbered_buff_LENGTH (pTHX_
420 REGEXP * const rx,
421 const SV * const sv,
2fdbfb4d
AB
422 const I32 paren);
423
2edc787c 424Get the C<length> of a capture variable. There's a special callback
5a2b28ce
KW
425for this so that Perl doesn't have to do a FETCH and run C<length> on
426the result, since the length is (in Perl's case) known from an offset
427stored in C<< rx->offs >>, this is much more efficient:
2fdbfb4d
AB
428
429 I32 s1 = rx->offs[paren].start;
430 I32 s2 = rx->offs[paren].end;
431 I32 len = t1 - s1;
432
433This is a little bit more complex in the case of UTF-8, see what
434C<Perl_reg_numbered_buff_length> does with
435L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
436
192b9cd1
AB
437=head2 Named capture callbacks
438
5a2b28ce 439Called to get/set the value of C<%+> and C<%->, as well as by some
192b9cd1
AB
440utility functions in L<re>.
441
442There are two callbacks, C<named_buff> is called in all the cases the
443FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks
444would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the
445same cases as FIRSTKEY and NEXTKEY.
446
447The C<flags> parameter can be used to determine which of these
5a2b28ce 448operations the callbacks should respond to. The following flags are
192b9cd1
AB
449currently defined:
450
451Which L<Tie::Hash> operation is being performed from the Perl level on
452C<%+> or C<%+>, if any:
453
f1b875a0
YO
454 RXapif_FETCH
455 RXapif_STORE
456 RXapif_DELETE
457 RXapif_CLEAR
458 RXapif_EXISTS
459 RXapif_SCALAR
460 RXapif_FIRSTKEY
461 RXapif_NEXTKEY
192b9cd1 462
5a2b28ce 463If C<%+> or C<%-> is being operated on, if any.
2fdbfb4d 464
f1b875a0
YO
465 RXapif_ONE /* %+ */
466 RXapif_ALL /* %- */
2fdbfb4d 467
5a2b28ce 468If this is being called as C<re::regname>, C<re::regnames> or
2edc787c 469C<re::regnames_count>, if any. The first two will be combined with
f1b875a0 470C<RXapif_ONE> or C<RXapif_ALL>.
192b9cd1 471
f1b875a0
YO
472 RXapif_REGNAME
473 RXapif_REGNAMES
474 RXapif_REGNAMES_COUNT
192b9cd1
AB
475
476Internally C<%+> and C<%-> are implemented with a real tied interface
2edc787c
FC
477via L<Tie::Hash::NamedCapture>. The methods in that package will call
478back into these functions. However the usage of
192b9cd1 479L<Tie::Hash::NamedCapture> for this purpose might change in future
2edc787c 480releases. For instance this might be implemented by magic instead
192b9cd1
AB
481(would need an extension to mgvtbl).
482
483=head3 named_buff
484
485 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
486 SV * const value, U32 flags);
487
488=head3 named_buff_iter
489
02c01adb
KW
490 SV* (*named_buff_iter) (pTHX_
491 REGEXP * const rx,
492 const SV * const lastkey,
192b9cd1 493 const U32 flags);
108003db 494
49d7dfbc 495=head2 qr_package
108003db 496
49d7dfbc 497 SV* qr_package(pTHX_ REGEXP * const rx);
108003db
RGS
498
499The package the qr// magic object is blessed into (as seen by C<ref
2edc787c 500qr//>). It is recommended that engines change this to their package
5a2b28ce 501name for identification regardless of if they implement methods
49d7dfbc
AB
502on the object.
503
192b9cd1 504The package this method returns should also have the internal
2edc787c 505C<Regexp> package in its C<@ISA>. C<< qr//->isa("Regexp") >> should always
192b9cd1
AB
506be true regardless of what engine is being used.
507
508Example implementation might be:
108003db
RGS
509
510 SV*
192b9cd1 511 Example_qr_package(pTHX_ REGEXP * const rx)
108003db
RGS
512 {
513 PERL_UNUSED_ARG(rx);
514 return newSVpvs("re::engine::Example");
515 }
516
517Any method calls on an object created with C<qr//> will be dispatched to the
518package as a normal object.
519
520 use re::engine::Example;
521 my $re = qr//;
522 $re->meth; # dispatched to re::engine::Example::meth()
523
f7e71195
AB
524To retrieve the C<REGEXP> object from the scalar in an XS function use
525the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP
526Functions>.
108003db
RGS
527
528 void meth(SV * rv)
529 PPCODE:
f7e71195 530 REGEXP * re = SvRX(sv);
108003db 531
108003db
RGS
532=head2 dupe
533
49d7dfbc 534 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
535
536On threaded builds a regexp may need to be duplicated so that the pattern
2edc787c 537can be used by multiple threads. This routine is expected to handle the
108003db 538duplication of any private data pointed to by the C<pprivate> member of
5a2b28ce
KW
539the C<regexp> structure. It will be called with the preconstructed new
540C<regexp> structure as an argument, the C<pprivate> member will point at
a0e97681 541the B<old> private structure, and it is this routine's responsibility to
5a2b28ce 542construct a copy and return a pointer to it (which Perl will then use to
108003db
RGS
543overwrite the field as passed to this routine.)
544
545This allows the engine to dupe its private data but also if necessary
546modify the final structure if it really must.
547
548On unthreaded builds this field doesn't exist.
549
3c13cae6
DM
550=head2 op_comp
551
5a2b28ce 552This is private to the Perl core and subject to change. Should be left
3c13cae6
DM
553null.
554
108003db
RGS
555=head1 The REGEXP structure
556
2edc787c
FC
557The REGEXP struct is defined in F<regexp.h>.
558All regex engines must be able to
108003db
RGS
559correctly build such a structure in their L</comp> routine.
560
5a2b28ce 561The REGEXP structure contains all the data that Perl needs to be aware of
2edc787c 562to properly work with the regular expression. It includes data about
5a2b28ce 563optimisations that Perl can use to determine if the regex engine should
108003db 564really be used, and various other control info that is needed to properly
5a2b28ce
KW
565execute patterns in various contexts, such as if the pattern anchored in
566some way, or what flags were used during the compile, or if the
567program contains special constructs that Perl needs to be aware of.
108003db 568
882227b7 569In addition it contains two fields that are intended for the private
2edc787c
FC
570use of the regex engine that compiled the pattern. These are the
571C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
5a2b28ce 572an arbitrary structure, whose use and management is the responsibility
2edc787c 573of the compiling engine. Perl will never modify either of these
882227b7 574values.
108003db
RGS
575
576 typedef struct regexp {
577 /* what engine created this regexp? */
578 const struct regexp_engine* engine;
579
580 /* what re is this a lightweight copy of? */
581 struct regexp* mother_re;
582
5a2b28ce 583 /* Information about the match that the Perl core uses to manage
02c01adb 584 * things */
108003db 585 U32 extflags; /* Flags used both externally and internally */
2d608413
KW
586 I32 minlen; /* mininum possible number of chars in */
587 string to match */
588 I32 minlenret; /* mininum possible number of chars in $& */
108003db
RGS
589 U32 gofs; /* chars left of pos that we search from */
590
591 /* substring data about strings that must appear
592 in the final match, used for optimisations */
593 struct reg_substr_data *substrs;
594
c27a5cfe 595 U32 nparens; /* number of capture groups */
108003db
RGS
596
597 /* private engine specific data */
598 U32 intflags; /* Engine Specific Internal flags */
599 void *pprivate; /* Data private to the regex engine which
600 created this object. */
601
02c01adb
KW
602 /* Data about the last/current match. These are modified during
603 * matching*/
82f14494
DM
604 U32 lastparen; /* highest close paren matched ($+) */
605 U32 lastcloseparen; /* last close paren matched ($^N) */
108003db 606 regexp_paren_pair *swap; /* Swap copy of *offs */
02c01adb
KW
607 regexp_paren_pair *offs; /* Array of offsets for (@-) and
608 (@+) */
108003db 609
02c01adb
KW
610 char *subbeg; /* saved or original string so \digit works
611 forever. */
108003db
RGS
612 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
613 I32 sublen; /* Length of string pointed by subbeg */
02c01adb
KW
614 I32 suboffset; /* byte offset of subbeg from logical start of
615 str */
6502e081 616 I32 subcoffset; /* suboffset equiv, but in chars (for @-/@+) */
108003db
RGS
617
618 /* Information about the match that isn't often used */
619 I32 prelen; /* length of precomp */
620 const char *precomp; /* pre-compilation regular expression */
621
108003db
RGS
622 char *wrapped; /* wrapped version of the pattern */
623 I32 wraplen; /* length of wrapped */
624
02c01adb
KW
625 I32 seen_evals; /* number of eval groups in the pattern - for
626 security checks */
108003db
RGS
627 HV *paren_names; /* Optional hash of paren names */
628
629 /* Refcount of this regexp */
630 I32 refcnt; /* Refcount of this regexp */
631 } regexp;
632
633The fields are discussed in more detail below:
634
882227b7 635=head2 C<engine>
108003db 636
5a2b28ce 637This field points at a C<regexp_engine> structure which contains pointers
2edc787c 638to the subroutines that are to be used for performing a match. It
108003db
RGS
639is the compiling routine's responsibility to populate this field before
640returning the regexp object.
641
642Internally this is set to C<NULL> unless a custom engine is specified in
5a2b28ce 643C<$^H{regcomp}>, Perl's own set of callbacks can be accessed in the struct
108003db
RGS
644pointed to by C<RE_ENGINE_PTR>.
645
882227b7 646=head2 C<mother_re>
108003db
RGS
647
648TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
649
882227b7 650=head2 C<extflags>
108003db 651
5a2b28ce 652This will be used by Perl to see what flags the regexp was compiled
192b9cd1 653with, this will normally be set to the value of the flags parameter by
2edc787c 654the L<comp|/comp> callback. See the L<comp|/comp> documentation for
c998b245 655valid flags.
108003db 656
882227b7 657=head2 C<minlen> C<minlenret>
108003db 658
2d608413
KW
659The minimum string length (in characters) required for the pattern to match.
660This is used to
108003db 661prune the search space by not bothering to match any closer to the end of a
2edc787c 662string than would allow a match. For instance there is no point in even
108003db 663starting the regex engine if the minlen is 10 but the string is only 5
2edc787c 664characters long. There is no way that the pattern can match.
108003db 665
2d608413
KW
666C<minlenret> is the minimum length (in characters) of the string that would
667be found in $& after a match.
108003db
RGS
668
669The difference between C<minlen> and C<minlenret> can be seen in the
670following pattern:
671
672 /ns(?=\d)/
673
674where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
2edc787c
FC
675required to match but is not actually
676included in the matched content. This
108003db 677distinction is particularly important as the substitution logic uses the
5a2b28ce
KW
678C<minlenret> to tell if it can do in-place substitutions (these can
679result in considerable speed-up).
108003db 680
882227b7 681=head2 C<gofs>
108003db
RGS
682
683Left offset from pos() to start match at.
684
882227b7 685=head2 C<substrs>
108003db 686
2edc787c 687Substring data about strings that must appear in the final match. This
5a2b28ce 688is currently only used internally by Perl's engine, but might be
c998b245 689used in the future for all engines for optimisations.
108003db 690
1cecf2c0 691=head2 C<nparens>, C<lastparen>, and C<lastcloseparen>
108003db
RGS
692
693These fields are used to keep track of how many paren groups could be matched
694in the pattern, which was the last open paren to be entered, and which was
695the last close paren to be entered.
696
882227b7 697=head2 C<intflags>
108003db
RGS
698
699The engine's private copy of the flags the pattern was compiled with. Usually
192b9cd1 700this is the same as C<extflags> unless the engine chose to modify one of them.
108003db 701
882227b7 702=head2 C<pprivate>
108003db 703
2edc787c
FC
704A void* pointing to an engine-defined
705data structure. The Perl engine uses the
108003db
RGS
706C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
707engine should use something else.
708
882227b7 709=head2 C<swap>
108003db 710
2edc787c 711Unused. Left in for compatibility with Perl 5.10.0.
108003db 712
882227b7 713=head2 C<offs>
108003db
RGS
714
715A C<regexp_paren_pair> structure which defines offsets into the string being
716matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
717C<regexp_paren_pair> struct is defined as follows:
718
719 typedef struct regexp_paren_pair {
720 I32 start;
721 I32 end;
722 } regexp_paren_pair;
723
724If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
2edc787c
FC
725capture group did not match.
726C<< ->offs[0].start/end >> represents C<$&> (or
c149d39e 727C<${^MATCH}> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
108003db
RGS
728C<$paren >= 1>.
729
882227b7 730=head2 C<precomp> C<prelen>
108003db 731
2edc787c
FC
732Used for optimisations. C<precomp> holds a copy of the pattern that
733was compiled and C<prelen> its length. When a new pattern is to be
192b9cd1 734compiled (such as inside a loop) the internal C<regcomp> operator
5a2b28ce 735checks if the last compiled C<REGEXP>'s C<precomp> and C<prelen>
192b9cd1
AB
736are equivalent to the new one, and if so uses the old pattern instead
737of compiling a new one.
738
739The relevant snippet from C<Perl_pp_regcomp>:
740
741 if (!re || !re->precomp || re->prelen != (I32)len ||
742 memNE(re->precomp, t, len))
743 /* Compile a new pattern */
108003db 744
882227b7 745=head2 C<paren_names>
108003db 746
c27a5cfe 747This is a hash used internally to track named capture groups and their
2edc787c 748offsets. The keys are the names of the buffers the values are dualvars,
108003db
RGS
749with the IV slot holding the number of buffers with the given name and the
750pv being an embedded array of I32. The values may also be contained
751independently in the data array in cases where named backreferences are
752used.
753
c998b245 754=head2 C<substrs>
108003db
RGS
755
756Holds information on the longest string that must occur at a fixed
757offset from the start of the pattern, and the longest string that must
2edc787c 758occur at a floating offset from the start of the pattern. Used to do
108003db
RGS
759Fast-Boyer-Moore searches on the string to find out if its worth using
760the regex engine at all, and if so where in the string to search.
761
6502e081
DM
762=head2 C<subbeg> C<sublen> C<saved_copy> C<suboffset> C<subcoffset>
763
764Used during the execution phase for managing search and replace patterns,
765and for providing the text for C<$&>, C<$1> etc. C<subbeg> points to a
766buffer (either the original string, or a copy in the case of
767C<RX_MATCH_COPIED(rx)>), and C<sublen> is the length of the buffer. The
768C<RX_OFFS> start and end indices index into this buffer.
769
770In the presence of the C<REXEC_COPY_STR> flag, but with the addition of
771the C<REXEC_COPY_SKIP_PRE> or C<REXEC_COPY_SKIP_POST> flags, an engine
772can choose not to copy the full buffer (although it must still do so in
773the presence of C<RXf_PMf_KEEPCOPY> or the relevant bits being set in
2edc787c 774C<PL_sawampersand>). In this case, it may set C<suboffset> to indicate the
6502e081 775number of bytes from the logical start of the buffer to the physical start
2edc787c 776(i.e. C<subbeg>). It should also set C<subcoffset>, the number of
6502e081
DM
777characters in the offset. The latter is needed to support C<@-> and C<@+>
778which work in characters, not bytes.
108003db 779
882227b7 780=head2 C<wrapped> C<wraplen>
108003db 781
5a2b28ce 782Stores the string C<qr//> stringifies to. The Perl engine for example
ed215d3c 783stores C<(?^:eek)> in the case of C<qr/eek/>.
108003db 784
c998b245
AB
785When using a custom engine that doesn't support the C<(?:)> construct
786for inline modifiers, it's probably best to have C<qr//> stringify to
787the supplied pattern, note that this will create undesired patterns in
788cases such as:
108003db
RGS
789
790 my $x = qr/a|b/; # "a|b"
192b9cd1 791 my $y = qr/c/i; # "c"
108003db
RGS
792 my $z = qr/$x$y/; # "a|bc"
793
192b9cd1
AB
794There's no solution for this problem other than making the custom
795engine understand a construct like C<(?:)>.
108003db 796
882227b7 797=head2 C<seen_evals>
108003db 798
2edc787c
FC
799This stores the number of eval groups in
800the pattern. This is used for security
108003db
RGS
801purposes when embedding compiled regexes into larger patterns with C<qr//>.
802
882227b7 803=head2 C<refcnt>
108003db 804
2edc787c
FC
805The number of times the structure is referenced. When
806this falls to 0, the regexp is automatically freed
807by a call to pregfree. This should be set to 1 in
108003db
RGS
808each engine's L</comp> routine.
809
108003db
RGS
810=head1 HISTORY
811
812Originally part of L<perlreguts>.
813
814=head1 AUTHORS
815
816Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
817Bjarmason.
818
819=head1 LICENSE
820
821Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
822
823This program is free software; you can redistribute it and/or modify it under
824the same terms as Perl itself.
825
826=cut