This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perlreapi.pod: Document RXf_MODIFIES_VARS
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
5a2b28ce 3perlreapi - Perl regular expression plugin interface
108003db
RGS
4
5=head1 DESCRIPTION
6
5a2b28ce
KW
7As of Perl 5.9.5 there is a new interface for plugging and using
8regular expression engines other than the default one.
a0e97681
RGS
9
10Each engine is supposed to provide access to a constant structure of the
11following format:
108003db
RGS
12
13 typedef struct regexp_engine {
02c01adb
KW
14 REGEXP* (*comp) (pTHX_
15 const SV * const pattern, const U32 flags);
16 I32 (*exec) (pTHX_
17 REGEXP * const rx,
18 char* stringarg,
19 char* strend, char* strbeg,
20 I32 minend, SV* screamer,
2fdbfb4d 21 void* data, U32 flags);
02c01adb
KW
22 char* (*intuit) (pTHX_
23 REGEXP * const rx, SV *sv,
24 char *strpos, char *strend, U32 flags,
2fdbfb4d 25 struct re_scream_pos_data_s *data);
49d7dfbc
AB
26 SV* (*checkstr) (pTHX_ REGEXP * const rx);
27 void (*free) (pTHX_ REGEXP * const rx);
02c01adb
KW
28 void (*numbered_buff_FETCH) (pTHX_
29 REGEXP * const rx,
30 const I32 paren,
31 SV * const sv);
32 void (*numbered_buff_STORE) (pTHX_
33 REGEXP * const rx,
34 const I32 paren,
5a2b28ce 35 SV const * const value);
02c01adb
KW
36 I32 (*numbered_buff_LENGTH) (pTHX_
37 REGEXP * const rx,
38 const SV * const sv,
39 const I32 paren);
40 SV* (*named_buff) (pTHX_
41 REGEXP * const rx,
42 SV * const key,
43 SV * const value,
44 U32 flags);
45 SV* (*named_buff_iter) (pTHX_
46 REGEXP * const rx,
47 const SV * const lastkey,
192b9cd1 48 const U32 flags);
49d7dfbc 49 SV* (*qr_package)(pTHX_ REGEXP * const rx);
108003db 50 #ifdef USE_ITHREADS
49d7dfbc 51 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db 52 #endif
3c13cae6
DM
53 REGEXP* (*op_comp) (...);
54
108003db
RGS
55
56When a regexp is compiled, its C<engine> field is then set to point at
a0e97681 57the appropriate structure, so that when it needs to be used Perl can find
108003db
RGS
58the right routines to do so.
59
60In order to install a new regexp handler, C<$^H{regcomp}> is set
61to an integer which (when casted appropriately) resolves to one of these
62structures. When compiling, the C<comp> method is executed, and the
5a2b28ce 63resulting C<regexp> structure's engine field is expected to point back at
108003db
RGS
64the same structure.
65
5a2b28ce 66The pTHX_ symbol in the definition is a macro used by Perl under threading
108003db
RGS
67to provide an extra argument to the routine holding a pointer back to
68the interpreter that is executing the regexp. So under threading all
69routines get an extra argument.
70
882227b7 71=head1 Callbacks
108003db
RGS
72
73=head2 comp
74
3ab4a224 75 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 76
3ab4a224
AB
77Compile the pattern stored in C<pattern> using the given C<flags> and
78return a pointer to a prepared C<REGEXP> structure that can perform
79the match. See L</The REGEXP structure> below for an explanation of
80the individual fields in the REGEXP struct.
81
82The C<pattern> parameter is the scalar that was used as the
5a2b28ce
KW
83pattern. Previous versions of Perl would pass two C<char*> indicating
84the start and end of the stringified pattern; the following snippet can
3ab4a224
AB
85be used to get the old parameters:
86
87 STRLEN plen;
88 char* exp = SvPV(pattern, plen);
89 char* xend = exp + plen;
90
5a2b28ce 91Since any scalar can be passed as a pattern, it's possible to implement
3ab4a224
AB
92an engine that does something with an array (C<< "ook" =~ [ qw/ eek
93hlagh / ] >>) or with the non-stringified form of a compiled regular
5a2b28ce
KW
94expression (C<< "ook" =~ qr/eek/ >>). Perl's own engine will always
95stringify everything using the snippet above, but that doesn't mean
3ab4a224 96other engines have to.
108003db 97
a0e97681 98The C<flags> parameter is a bitfield which indicates which of the
c998b245 99C<msixp> flags the regex was compiled with. It also contains
5a2b28ce 100additional info, such as if C<use locale> is in effect.
108003db
RGS
101
102The C<eogc> flags are stripped out before being passed to the comp
5a2b28ce
KW
103routine. The regex engine does not need to know if any of these
104are set, as those flags should only affect what Perl does with the
c998b245
AB
105pattern and its match variables, not how it gets compiled and
106executed.
108003db 107
c998b245
AB
108By the time the comp callback is called, some of these flags have
109already had effect (noted below where applicable). However most of
5a2b28ce 110their effect occurs after the comp callback has run, in routines that
c998b245 111read the C<< rx->extflags >> field which it populates.
108003db 112
c998b245
AB
113In general the flags should be preserved in C<< rx->extflags >> after
114compilation, although the regex engine might want to add or delete
5a2b28ce 115some of them to invoke or disable some special behavior in Perl. The
c998b245 116flags along with any special behavior they cause are documented below:
108003db 117
c998b245 118The pattern modifiers:
108003db 119
c998b245 120=over 4
108003db 121
c998b245 122=item C</m> - RXf_PMf_MULTILINE
108003db 123
c998b245
AB
124If this is in C<< rx->extflags >> it will be passed to
125C<Perl_fbm_instr> by C<pp_split> which will treat the subject string
126as a multi-line string.
108003db 127
c998b245 128=item C</s> - RXf_PMf_SINGLELINE
108003db 129
c998b245 130=item C</i> - RXf_PMf_FOLD
108003db 131
c998b245 132=item C</x> - RXf_PMf_EXTENDED
108003db 133
5a2b28ce 134If present on a regex, C<"#"> comments will be handled differently by the
c998b245 135tokenizer in some cases.
108003db 136
c998b245 137TODO: Document those cases.
108003db 138
c998b245 139=item C</p> - RXf_PMf_KEEPCOPY
108003db 140
e72ec78c
KW
141TODO: Document this
142
f0f9b3b8
KW
143=item Character set
144
145The character set semantics are determined by an enum that is contained
146in this field. This is still experimental and subject to change, but
147the current interface returns the rules by use of the in-line function
148C<get_regex_charset(const U32 flags)>. The only currently documented
149value returned from it is REGEX_LOCALE_CHARSET, which is set if
e72ec78c
KW
150C<use locale> is in effect. If present in C<< rx->extflags >>,
151C<split> will use the locale dependent definition of whitespace
152when RXf_SKIPWHITE or RXf_WHITE is in effect. ASCII whitespace
96090e4f 153is defined as per L<isSPACE|perlapi/isSPACE>, and by the internal
e72ec78c 154macros C<is_utf8_space> under UTF-8, and C<isSPACE_LC> under C<use
c998b245 155locale>.
108003db 156
f0f9b3b8
KW
157=back
158
159Additional flags:
160
161=over 4
162
0ac6acae
AB
163=item RXf_SPLIT
164
9f6ecbda
FC
165This flag was removed in perl 5.18.0. C<split ' '> is now special-cased
166solely in the parser. RXf_SPLIT is still #defined, so you can test for it.
167This is how it used to work:
168
0ac6acae 169If C<split> is invoked as C<split ' '> or with no arguments (which
5a2b28ce 170really means C<split(' ', $_)>, see L<split|perlfunc/split>), Perl will
0ac6acae 171set this flag. The regex engine can then check for it and set the
5a2b28ce 172SKIPWHITE and WHITE extflags. To do this, the Perl engine does:
0ac6acae
AB
173
174 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
175 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
176
108003db
RGS
177=back
178
c998b245
AB
179These flags can be set during compilation to enable optimizations in
180the C<split> operator.
181
182=over 4
183
0ac6acae
AB
184=item RXf_SKIPWHITE
185
51b1fee8
FC
186This flag was removed in perl 5.18.0. It is still #defined, so you can
187set it, but doing so will have no effect. This is how it used to work:
188
0ac6acae
AB
189If the flag is present in C<< rx->extflags >> C<split> will delete
190whitespace from the start of the subject string before it's operated
5a2b28ce
KW
191on. What is considered whitespace depends on if the subject is a
192UTF-8 string and if the C<RXf_PMf_LOCALE> flag is set.
0ac6acae 193
5a2b28ce
KW
194If RXf_WHITE is set in addition to this flag, C<split> will behave like
195C<split " "> under the Perl engine.
0ac6acae 196
c998b245
AB
197=item RXf_START_ONLY
198
199Tells the split operator to split the target string on newlines
200(C<\n>) without invoking the regex engine.
201
202Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp
5a2b28ce 203== '^'>), even under C</^/s>; see L<split|perlfunc>. Of course a
c998b245
AB
204different regex engine might want to use the same optimizations
205with a different syntax.
206
207=item RXf_WHITE
208
209Tells the split operator to split the target string on whitespace
210without invoking the regex engine. The definition of whitespace varies
5a2b28ce
KW
211depending on if the target string is a UTF-8 string and on
212if RXf_PMf_LOCALE is set.
c998b245 213
0ac6acae 214Perl's engine sets this flag if the pattern is C<\s+>.
c998b245 215
640f820d
AB
216=item RXf_NULL
217
a0e97681 218Tells the split operator to split the target string on
5a2b28ce 219characters. The definition of character varies depending on if
640f820d
AB
220the target string is a UTF-8 string.
221
222Perl's engine sets this flag on empty patterns, this optimization
a0e97681 223makes C<split //> much faster than it would otherwise be. It's even
640f820d
AB
224faster than C<unpack>.
225
94bbc698
FC
226=item RXf_MODIFIES_VARS
227
228Added in perl 5.18.0, this flag indicates that a regular expression might
229assign to non-magical variables (such as $REGMARK and $REGERROR) during
230matching. C<s///> will skip certain optimisations when this is set.
231
c998b245 232=back
108003db
RGS
233
234=head2 exec
235
49d7dfbc 236 I32 exec(pTHX_ REGEXP * const rx,
108003db
RGS
237 char *stringarg, char* strend, char* strbeg,
238 I32 minend, SV* screamer,
239 void* data, U32 flags);
240
8fd1a950
DM
241Execute a regexp. The arguments are
242
243=over 4
244
245=item rx
246
247The regular expression to execute.
248
249=item screamer
250
251This strangely-named arg is the SV to be matched against. Note that the
252actual char array to be matched against is supplied by the arguments
253described below; the SV is just used to determine UTF8ness, C<pos()> etc.
254
255=item strbeg
256
257Pointer to the physical start of the string.
258
259=item strend
260
261Pointer to the character following the physical end of the string (i.e.
5a2b28ce 262the C<\0>).
8fd1a950
DM
263
264=item stringarg
265
266Pointer to the position in the string where matching should start; it might
267not be equal to C<strbeg> (for example in a later iteration of C</.../g>).
268
269=item minend
270
271Minimum length of string (measured in bytes from C<stringarg>) that must
272match; if the engine reaches the end of the match but hasn't reached this
273position in the string, it should fail.
274
275=item data
276
277Optimisation data; subject to change.
278
279=item flags
280
281Optimisation flags; subject to change.
282
283=back
108003db
RGS
284
285=head2 intuit
286
49d7dfbc 287 char* intuit(pTHX_ REGEXP * const rx,
108003db 288 SV *sv, char *strpos, char *strend,
49d7dfbc 289 const U32 flags, struct re_scream_pos_data_s *data);
108003db
RGS
290
291Find the start position where a regex match should be attempted,
5a2b28ce
KW
292or possibly if the regex engine should not be run because the
293pattern can't match. This is called, as appropriate, by the core,
294depending on the values of the C<extflags> member of the C<regexp>
108003db
RGS
295structure.
296
297=head2 checkstr
298
49d7dfbc 299 SV* checkstr(pTHX_ REGEXP * const rx);
108003db
RGS
300
301Return a SV containing a string that must appear in the pattern. Used
302by C<split> for optimising matches.
303
304=head2 free
305
49d7dfbc 306 void free(pTHX_ REGEXP * const rx);
108003db 307
5a2b28ce 308Called by Perl when it is freeing a regexp pattern so that the engine
108003db 309can release any resources pointed to by the C<pprivate> member of the
5a2b28ce
KW
310C<regexp> structure. This is only responsible for freeing private data;
311Perl will handle releasing anything else contained in the C<regexp> structure.
108003db 312
192b9cd1 313=head2 Numbered capture callbacks
108003db 314
192b9cd1
AB
315Called to get/set the value of C<$`>, C<$'>, C<$&> and their named
316equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
c27a5cfe 317numbered capture groups (C<$1>, C<$2>, ...).
49d7dfbc 318
c149d39e
DM
319The C<paren> parameter will be C<1> for C<$1>, C<2> for C<$2> and so
320forth, and have these symbolic values for the special variables:
321
322 ${^PREMATCH} RX_BUFF_IDX_CARET_PREMATCH
323 ${^POSTMATCH} RX_BUFF_IDX_CARET_POSTMATCH
324 ${^MATCH} RX_BUFF_IDX_CARET_FULLMATCH
325 $` RX_BUFF_IDX_PREMATCH
326 $' RX_BUFF_IDX_POSTMATCH
327 $& RX_BUFF_IDX_FULLMATCH
328
5a2b28ce 329Note that in Perl 5.17.3 and earlier, the last three constants were also
c149d39e
DM
330used for the caret variants of the variables.
331
49d7dfbc 332
192b9cd1
AB
333The names have been chosen by analogy with L<Tie::Scalar> methods
334names with an additional B<LENGTH> callback for efficiency. However
335named capture variables are currently not tied internally but
336implemented via magic.
337
338=head3 numbered_buff_FETCH
339
340 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
341 SV * const sv);
342
343Fetch a specified numbered capture. C<sv> should be set to the scalar
344to return, the scalar is passed as an argument rather than being
5a2b28ce 345returned from the function because when it's called Perl already has a
192b9cd1
AB
346scalar to store the value, creating another one would be
347redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and
348friends, see L<perlapi>.
49d7dfbc 349
5a2b28ce 350This callback is where Perl untaints its own capture variables under
c998b245 351taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch>
49d7dfbc
AB
352function in F<regcomp.c> for how to untaint capture variables if
353that's something you'd like your engine to do as well.
108003db 354
192b9cd1 355=head3 numbered_buff_STORE
108003db 356
02c01adb
KW
357 void (*numbered_buff_STORE) (pTHX_
358 REGEXP * const rx,
359 const I32 paren,
2fdbfb4d 360 SV const * const value);
108003db 361
192b9cd1
AB
362Set the value of a numbered capture variable. C<value> is the scalar
363that is to be used as the new value. It's up to the engine to make
364sure this is used as the new value (or reject it).
2fdbfb4d
AB
365
366Example:
367
368 if ("ook" =~ /(o*)/) {
ccf3535a 369 # 'paren' will be '1' and 'value' will be 'ee'
2fdbfb4d
AB
370 $1 =~ tr/o/e/;
371 }
372
373Perl's own engine will croak on any attempt to modify the capture
a0e97681 374variables, to do this in another engine use the following callback
2fdbfb4d
AB
375(copied from C<Perl_reg_numbered_buff_store>):
376
377 void
02c01adb
KW
378 Example_reg_numbered_buff_store(pTHX_
379 REGEXP * const rx,
380 const I32 paren,
381 SV const * const value)
2fdbfb4d
AB
382 {
383 PERL_UNUSED_ARG(rx);
384 PERL_UNUSED_ARG(paren);
385 PERL_UNUSED_ARG(value);
386
387 if (!PL_localizing)
388 Perl_croak(aTHX_ PL_no_modify);
389 }
390
5a2b28ce 391Actually Perl will not I<always> croak in a statement that looks
2fdbfb4d 392like it would modify a numbered capture variable. This is because the
5a2b28ce 393STORE callback will not be called if Perl can determine that it
2fdbfb4d
AB
394doesn't have to modify the value. This is exactly how tied variables
395behave in the same situation:
396
397 package CaptureVar;
398 use base 'Tie::Scalar';
399
400 sub TIESCALAR { bless [] }
401 sub FETCH { undef }
402 sub STORE { die "This doesn't get called" }
403
404 package main;
405
c69ca1d4 406 tie my $sv => "CaptureVar";
2fdbfb4d
AB
407 $sv =~ y/a/b/;
408
5a2b28ce 409Because C<$sv> is C<undef> when the C<y///> operator is applied to it,
2fdbfb4d 410the transliteration won't actually execute and the program won't
192b9cd1 411C<die>. This is different to how 5.8 and earlier versions behaved
5a2b28ce 412since the capture variables were READONLY variables then; now they'll
192b9cd1 413just die when assigned to in the default engine.
2fdbfb4d 414
192b9cd1 415=head3 numbered_buff_LENGTH
2fdbfb4d 416
02c01adb
KW
417 I32 numbered_buff_LENGTH (pTHX_
418 REGEXP * const rx,
419 const SV * const sv,
2fdbfb4d
AB
420 const I32 paren);
421
422Get the C<length> of a capture variable. There's a special callback
5a2b28ce
KW
423for this so that Perl doesn't have to do a FETCH and run C<length> on
424the result, since the length is (in Perl's case) known from an offset
425stored in C<< rx->offs >>, this is much more efficient:
2fdbfb4d
AB
426
427 I32 s1 = rx->offs[paren].start;
428 I32 s2 = rx->offs[paren].end;
429 I32 len = t1 - s1;
430
431This is a little bit more complex in the case of UTF-8, see what
432C<Perl_reg_numbered_buff_length> does with
433L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
434
192b9cd1
AB
435=head2 Named capture callbacks
436
5a2b28ce 437Called to get/set the value of C<%+> and C<%->, as well as by some
192b9cd1
AB
438utility functions in L<re>.
439
440There are two callbacks, C<named_buff> is called in all the cases the
441FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks
442would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the
443same cases as FIRSTKEY and NEXTKEY.
444
445The C<flags> parameter can be used to determine which of these
5a2b28ce 446operations the callbacks should respond to. The following flags are
192b9cd1
AB
447currently defined:
448
449Which L<Tie::Hash> operation is being performed from the Perl level on
450C<%+> or C<%+>, if any:
451
f1b875a0
YO
452 RXapif_FETCH
453 RXapif_STORE
454 RXapif_DELETE
455 RXapif_CLEAR
456 RXapif_EXISTS
457 RXapif_SCALAR
458 RXapif_FIRSTKEY
459 RXapif_NEXTKEY
192b9cd1 460
5a2b28ce 461If C<%+> or C<%-> is being operated on, if any.
2fdbfb4d 462
f1b875a0
YO
463 RXapif_ONE /* %+ */
464 RXapif_ALL /* %- */
2fdbfb4d 465
5a2b28ce 466If this is being called as C<re::regname>, C<re::regnames> or
c998b245 467C<re::regnames_count>, if any. The first two will be combined with
f1b875a0 468C<RXapif_ONE> or C<RXapif_ALL>.
192b9cd1 469
f1b875a0
YO
470 RXapif_REGNAME
471 RXapif_REGNAMES
472 RXapif_REGNAMES_COUNT
192b9cd1
AB
473
474Internally C<%+> and C<%-> are implemented with a real tied interface
475via L<Tie::Hash::NamedCapture>. The methods in that package will call
476back into these functions. However the usage of
477L<Tie::Hash::NamedCapture> for this purpose might change in future
478releases. For instance this might be implemented by magic instead
479(would need an extension to mgvtbl).
480
481=head3 named_buff
482
483 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
484 SV * const value, U32 flags);
485
486=head3 named_buff_iter
487
02c01adb
KW
488 SV* (*named_buff_iter) (pTHX_
489 REGEXP * const rx,
490 const SV * const lastkey,
192b9cd1 491 const U32 flags);
108003db 492
49d7dfbc 493=head2 qr_package
108003db 494
49d7dfbc 495 SV* qr_package(pTHX_ REGEXP * const rx);
108003db
RGS
496
497The package the qr// magic object is blessed into (as seen by C<ref
49d7dfbc 498qr//>). It is recommended that engines change this to their package
5a2b28ce 499name for identification regardless of if they implement methods
49d7dfbc
AB
500on the object.
501
192b9cd1 502The package this method returns should also have the internal
d5213412 503C<Regexp> package in its C<@ISA>. C<< qr//->isa("Regexp") >> should always
192b9cd1
AB
504be true regardless of what engine is being used.
505
506Example implementation might be:
108003db
RGS
507
508 SV*
192b9cd1 509 Example_qr_package(pTHX_ REGEXP * const rx)
108003db
RGS
510 {
511 PERL_UNUSED_ARG(rx);
512 return newSVpvs("re::engine::Example");
513 }
514
515Any method calls on an object created with C<qr//> will be dispatched to the
516package as a normal object.
517
518 use re::engine::Example;
519 my $re = qr//;
520 $re->meth; # dispatched to re::engine::Example::meth()
521
f7e71195
AB
522To retrieve the C<REGEXP> object from the scalar in an XS function use
523the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP
524Functions>.
108003db
RGS
525
526 void meth(SV * rv)
527 PPCODE:
f7e71195 528 REGEXP * re = SvRX(sv);
108003db 529
108003db
RGS
530=head2 dupe
531
49d7dfbc 532 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
533
534On threaded builds a regexp may need to be duplicated so that the pattern
a0e97681 535can be used by multiple threads. This routine is expected to handle the
108003db 536duplication of any private data pointed to by the C<pprivate> member of
5a2b28ce
KW
537the C<regexp> structure. It will be called with the preconstructed new
538C<regexp> structure as an argument, the C<pprivate> member will point at
a0e97681 539the B<old> private structure, and it is this routine's responsibility to
5a2b28ce 540construct a copy and return a pointer to it (which Perl will then use to
108003db
RGS
541overwrite the field as passed to this routine.)
542
543This allows the engine to dupe its private data but also if necessary
544modify the final structure if it really must.
545
546On unthreaded builds this field doesn't exist.
547
3c13cae6
DM
548=head2 op_comp
549
5a2b28ce 550This is private to the Perl core and subject to change. Should be left
3c13cae6
DM
551null.
552
108003db
RGS
553=head1 The REGEXP structure
554
555The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
556correctly build such a structure in their L</comp> routine.
557
5a2b28ce 558The REGEXP structure contains all the data that Perl needs to be aware of
108003db 559to properly work with the regular expression. It includes data about
5a2b28ce 560optimisations that Perl can use to determine if the regex engine should
108003db 561really be used, and various other control info that is needed to properly
5a2b28ce
KW
562execute patterns in various contexts, such as if the pattern anchored in
563some way, or what flags were used during the compile, or if the
564program contains special constructs that Perl needs to be aware of.
108003db 565
882227b7
AB
566In addition it contains two fields that are intended for the private
567use of the regex engine that compiled the pattern. These are the
568C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
5a2b28ce
KW
569an arbitrary structure, whose use and management is the responsibility
570of the compiling engine. Perl will never modify either of these
882227b7 571values.
108003db
RGS
572
573 typedef struct regexp {
574 /* what engine created this regexp? */
575 const struct regexp_engine* engine;
576
577 /* what re is this a lightweight copy of? */
578 struct regexp* mother_re;
579
5a2b28ce 580 /* Information about the match that the Perl core uses to manage
02c01adb 581 * things */
108003db 582 U32 extflags; /* Flags used both externally and internally */
2d608413
KW
583 I32 minlen; /* mininum possible number of chars in */
584 string to match */
585 I32 minlenret; /* mininum possible number of chars in $& */
108003db
RGS
586 U32 gofs; /* chars left of pos that we search from */
587
588 /* substring data about strings that must appear
589 in the final match, used for optimisations */
590 struct reg_substr_data *substrs;
591
c27a5cfe 592 U32 nparens; /* number of capture groups */
108003db
RGS
593
594 /* private engine specific data */
595 U32 intflags; /* Engine Specific Internal flags */
596 void *pprivate; /* Data private to the regex engine which
597 created this object. */
598
02c01adb
KW
599 /* Data about the last/current match. These are modified during
600 * matching*/
82f14494
DM
601 U32 lastparen; /* highest close paren matched ($+) */
602 U32 lastcloseparen; /* last close paren matched ($^N) */
108003db 603 regexp_paren_pair *swap; /* Swap copy of *offs */
02c01adb
KW
604 regexp_paren_pair *offs; /* Array of offsets for (@-) and
605 (@+) */
108003db 606
02c01adb
KW
607 char *subbeg; /* saved or original string so \digit works
608 forever. */
108003db
RGS
609 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
610 I32 sublen; /* Length of string pointed by subbeg */
02c01adb
KW
611 I32 suboffset; /* byte offset of subbeg from logical start of
612 str */
6502e081 613 I32 subcoffset; /* suboffset equiv, but in chars (for @-/@+) */
108003db
RGS
614
615 /* Information about the match that isn't often used */
616 I32 prelen; /* length of precomp */
617 const char *precomp; /* pre-compilation regular expression */
618
108003db
RGS
619 char *wrapped; /* wrapped version of the pattern */
620 I32 wraplen; /* length of wrapped */
621
02c01adb
KW
622 I32 seen_evals; /* number of eval groups in the pattern - for
623 security checks */
108003db
RGS
624 HV *paren_names; /* Optional hash of paren names */
625
626 /* Refcount of this regexp */
627 I32 refcnt; /* Refcount of this regexp */
628 } regexp;
629
630The fields are discussed in more detail below:
631
882227b7 632=head2 C<engine>
108003db 633
5a2b28ce 634This field points at a C<regexp_engine> structure which contains pointers
108003db
RGS
635to the subroutines that are to be used for performing a match. It
636is the compiling routine's responsibility to populate this field before
637returning the regexp object.
638
639Internally this is set to C<NULL> unless a custom engine is specified in
5a2b28ce 640C<$^H{regcomp}>, Perl's own set of callbacks can be accessed in the struct
108003db
RGS
641pointed to by C<RE_ENGINE_PTR>.
642
882227b7 643=head2 C<mother_re>
108003db
RGS
644
645TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
646
882227b7 647=head2 C<extflags>
108003db 648
5a2b28ce 649This will be used by Perl to see what flags the regexp was compiled
192b9cd1 650with, this will normally be set to the value of the flags parameter by
c998b245
AB
651the L<comp|/comp> callback. See the L<comp|/comp> documentation for
652valid flags.
108003db 653
882227b7 654=head2 C<minlen> C<minlenret>
108003db 655
2d608413
KW
656The minimum string length (in characters) required for the pattern to match.
657This is used to
108003db
RGS
658prune the search space by not bothering to match any closer to the end of a
659string than would allow a match. For instance there is no point in even
660starting the regex engine if the minlen is 10 but the string is only 5
661characters long. There is no way that the pattern can match.
662
2d608413
KW
663C<minlenret> is the minimum length (in characters) of the string that would
664be found in $& after a match.
108003db
RGS
665
666The difference between C<minlen> and C<minlenret> can be seen in the
667following pattern:
668
669 /ns(?=\d)/
670
671where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
672required to match but is not actually included in the matched content. This
673distinction is particularly important as the substitution logic uses the
5a2b28ce
KW
674C<minlenret> to tell if it can do in-place substitutions (these can
675result in considerable speed-up).
108003db 676
882227b7 677=head2 C<gofs>
108003db
RGS
678
679Left offset from pos() to start match at.
680
882227b7 681=head2 C<substrs>
108003db 682
192b9cd1 683Substring data about strings that must appear in the final match. This
5a2b28ce 684is currently only used internally by Perl's engine, but might be
c998b245 685used in the future for all engines for optimisations.
108003db 686
1cecf2c0 687=head2 C<nparens>, C<lastparen>, and C<lastcloseparen>
108003db
RGS
688
689These fields are used to keep track of how many paren groups could be matched
690in the pattern, which was the last open paren to be entered, and which was
691the last close paren to be entered.
692
882227b7 693=head2 C<intflags>
108003db
RGS
694
695The engine's private copy of the flags the pattern was compiled with. Usually
192b9cd1 696this is the same as C<extflags> unless the engine chose to modify one of them.
108003db 697
882227b7 698=head2 C<pprivate>
108003db 699
5a2b28ce 700A void* pointing to an engine-defined data structure. The Perl engine uses the
108003db
RGS
701C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
702engine should use something else.
703
882227b7 704=head2 C<swap>
108003db 705
5a2b28ce 706Unused. Left in for compatibility with Perl 5.10.0.
108003db 707
882227b7 708=head2 C<offs>
108003db
RGS
709
710A C<regexp_paren_pair> structure which defines offsets into the string being
711matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
712C<regexp_paren_pair> struct is defined as follows:
713
714 typedef struct regexp_paren_pair {
715 I32 start;
716 I32 end;
717 } regexp_paren_pair;
718
719If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
c27a5cfe 720capture group did not match. C<< ->offs[0].start/end >> represents C<$&> (or
c149d39e 721C<${^MATCH}> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
108003db
RGS
722C<$paren >= 1>.
723
882227b7 724=head2 C<precomp> C<prelen>
108003db 725
192b9cd1
AB
726Used for optimisations. C<precomp> holds a copy of the pattern that
727was compiled and C<prelen> its length. When a new pattern is to be
728compiled (such as inside a loop) the internal C<regcomp> operator
5a2b28ce 729checks if the last compiled C<REGEXP>'s C<precomp> and C<prelen>
192b9cd1
AB
730are equivalent to the new one, and if so uses the old pattern instead
731of compiling a new one.
732
733The relevant snippet from C<Perl_pp_regcomp>:
734
735 if (!re || !re->precomp || re->prelen != (I32)len ||
736 memNE(re->precomp, t, len))
737 /* Compile a new pattern */
108003db 738
882227b7 739=head2 C<paren_names>
108003db 740
c27a5cfe 741This is a hash used internally to track named capture groups and their
108003db
RGS
742offsets. The keys are the names of the buffers the values are dualvars,
743with the IV slot holding the number of buffers with the given name and the
744pv being an embedded array of I32. The values may also be contained
745independently in the data array in cases where named backreferences are
746used.
747
c998b245 748=head2 C<substrs>
108003db
RGS
749
750Holds information on the longest string that must occur at a fixed
751offset from the start of the pattern, and the longest string that must
752occur at a floating offset from the start of the pattern. Used to do
753Fast-Boyer-Moore searches on the string to find out if its worth using
754the regex engine at all, and if so where in the string to search.
755
6502e081
DM
756=head2 C<subbeg> C<sublen> C<saved_copy> C<suboffset> C<subcoffset>
757
758Used during the execution phase for managing search and replace patterns,
759and for providing the text for C<$&>, C<$1> etc. C<subbeg> points to a
760buffer (either the original string, or a copy in the case of
761C<RX_MATCH_COPIED(rx)>), and C<sublen> is the length of the buffer. The
762C<RX_OFFS> start and end indices index into this buffer.
763
764In the presence of the C<REXEC_COPY_STR> flag, but with the addition of
765the C<REXEC_COPY_SKIP_PRE> or C<REXEC_COPY_SKIP_POST> flags, an engine
766can choose not to copy the full buffer (although it must still do so in
767the presence of C<RXf_PMf_KEEPCOPY> or the relevant bits being set in
768C<PL_sawampersand>). In this case, it may set C<suboffset> to indicate the
769number of bytes from the logical start of the buffer to the physical start
770(i.e. C<subbeg>). It should also set C<subcoffset>, the number of
771characters in the offset. The latter is needed to support C<@-> and C<@+>
772which work in characters, not bytes.
108003db 773
882227b7 774=head2 C<wrapped> C<wraplen>
108003db 775
5a2b28ce 776Stores the string C<qr//> stringifies to. The Perl engine for example
ed215d3c 777stores C<(?^:eek)> in the case of C<qr/eek/>.
108003db 778
c998b245
AB
779When using a custom engine that doesn't support the C<(?:)> construct
780for inline modifiers, it's probably best to have C<qr//> stringify to
781the supplied pattern, note that this will create undesired patterns in
782cases such as:
108003db
RGS
783
784 my $x = qr/a|b/; # "a|b"
192b9cd1 785 my $y = qr/c/i; # "c"
108003db
RGS
786 my $z = qr/$x$y/; # "a|bc"
787
192b9cd1
AB
788There's no solution for this problem other than making the custom
789engine understand a construct like C<(?:)>.
108003db 790
882227b7 791=head2 C<seen_evals>
108003db
RGS
792
793This stores the number of eval groups in the pattern. This is used for security
794purposes when embedding compiled regexes into larger patterns with C<qr//>.
795
882227b7 796=head2 C<refcnt>
108003db 797
5a2b28ce 798The number of times the structure is referenced. When this falls to 0, the
108003db
RGS
799regexp is automatically freed by a call to pregfree. This should be set to 1 in
800each engine's L</comp> routine.
801
108003db
RGS
802=head1 HISTORY
803
804Originally part of L<perlreguts>.
805
806=head1 AUTHORS
807
808Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
809Bjarmason.
810
811=head1 LICENSE
812
813Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
814
815This program is free software; you can redistribute it and/or modify it under
816the same terms as Perl itself.
817
818=cut