This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Refactor to use test.pl instead of making TAP by hand. Add test names.
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
3perlreapi - perl regular expression plugin interface
4
5=head1 DESCRIPTION
6
a0e97681
RGS
7As of Perl 5.9.5 there is a new interface for plugging and using other
8regular expression engines than the default one.
9
10Each engine is supposed to provide access to a constant structure of the
11following format:
108003db
RGS
12
13 typedef struct regexp_engine {
3ab4a224 14 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
49d7dfbc 15 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
2fdbfb4d
AB
16 char* strbeg, I32 minend, SV* screamer,
17 void* data, U32 flags);
49d7dfbc 18 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
2fdbfb4d
AB
19 char *strend, U32 flags,
20 struct re_scream_pos_data_s *data);
49d7dfbc
AB
21 SV* (*checkstr) (pTHX_ REGEXP * const rx);
22 void (*free) (pTHX_ REGEXP * const rx);
2fdbfb4d
AB
23 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
24 SV * const sv);
25 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
26 SV const * const value);
27 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
28 const I32 paren);
192b9cd1
AB
29 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
30 SV * const value, U32 flags);
31 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
32 const U32 flags);
49d7dfbc 33 SV* (*qr_package)(pTHX_ REGEXP * const rx);
108003db 34 #ifdef USE_ITHREADS
49d7dfbc 35 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db 36 #endif
3c13cae6
DM
37 REGEXP* (*op_comp) (...);
38
108003db
RGS
39
40When a regexp is compiled, its C<engine> field is then set to point at
a0e97681 41the appropriate structure, so that when it needs to be used Perl can find
108003db
RGS
42the right routines to do so.
43
44In order to install a new regexp handler, C<$^H{regcomp}> is set
45to an integer which (when casted appropriately) resolves to one of these
46structures. When compiling, the C<comp> method is executed, and the
47resulting regexp structure's engine field is expected to point back at
48the same structure.
49
50The pTHX_ symbol in the definition is a macro used by perl under threading
51to provide an extra argument to the routine holding a pointer back to
52the interpreter that is executing the regexp. So under threading all
53routines get an extra argument.
54
882227b7 55=head1 Callbacks
108003db
RGS
56
57=head2 comp
58
3ab4a224 59 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 60
3ab4a224
AB
61Compile the pattern stored in C<pattern> using the given C<flags> and
62return a pointer to a prepared C<REGEXP> structure that can perform
63the match. See L</The REGEXP structure> below for an explanation of
64the individual fields in the REGEXP struct.
65
66The C<pattern> parameter is the scalar that was used as the
67pattern. previous versions of perl would pass two C<char*> indicating
a0e97681 68the start and end of the stringified pattern, the following snippet can
3ab4a224
AB
69be used to get the old parameters:
70
71 STRLEN plen;
72 char* exp = SvPV(pattern, plen);
73 char* xend = exp + plen;
74
75Since any scalar can be passed as a pattern it's possible to implement
76an engine that does something with an array (C<< "ook" =~ [ qw/ eek
77hlagh / ] >>) or with the non-stringified form of a compiled regular
78expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
79stringify everything using the snippet above but that doesn't mean
80other engines have to.
108003db 81
a0e97681 82The C<flags> parameter is a bitfield which indicates which of the
c998b245
AB
83C<msixp> flags the regex was compiled with. It also contains
84additional info such as whether C<use locale> is in effect.
108003db
RGS
85
86The C<eogc> flags are stripped out before being passed to the comp
87routine. The regex engine does not need to know whether any of these
3ab4a224 88are set as those flags should only affect what perl does with the
c998b245
AB
89pattern and its match variables, not how it gets compiled and
90executed.
108003db 91
c998b245
AB
92By the time the comp callback is called, some of these flags have
93already had effect (noted below where applicable). However most of
94their effect occurs after the comp callback has run in routines that
95read the C<< rx->extflags >> field which it populates.
108003db 96
c998b245
AB
97In general the flags should be preserved in C<< rx->extflags >> after
98compilation, although the regex engine might want to add or delete
99some of them to invoke or disable some special behavior in perl. The
100flags along with any special behavior they cause are documented below:
108003db 101
c998b245 102The pattern modifiers:
108003db 103
c998b245 104=over 4
108003db 105
c998b245 106=item C</m> - RXf_PMf_MULTILINE
108003db 107
c998b245
AB
108If this is in C<< rx->extflags >> it will be passed to
109C<Perl_fbm_instr> by C<pp_split> which will treat the subject string
110as a multi-line string.
108003db 111
c998b245 112=item C</s> - RXf_PMf_SINGLELINE
108003db 113
c998b245 114=item C</i> - RXf_PMf_FOLD
108003db 115
c998b245 116=item C</x> - RXf_PMf_EXTENDED
108003db 117
c998b245
AB
118If present on a regex C<#> comments will be handled differently by the
119tokenizer in some cases.
108003db 120
c998b245 121TODO: Document those cases.
108003db 122
c998b245 123=item C</p> - RXf_PMf_KEEPCOPY
108003db 124
e72ec78c
KW
125TODO: Document this
126
f0f9b3b8
KW
127=item Character set
128
129The character set semantics are determined by an enum that is contained
130in this field. This is still experimental and subject to change, but
131the current interface returns the rules by use of the in-line function
132C<get_regex_charset(const U32 flags)>. The only currently documented
133value returned from it is REGEX_LOCALE_CHARSET, which is set if
e72ec78c
KW
134C<use locale> is in effect. If present in C<< rx->extflags >>,
135C<split> will use the locale dependent definition of whitespace
136when RXf_SKIPWHITE or RXf_WHITE is in effect. ASCII whitespace
96090e4f 137is defined as per L<isSPACE|perlapi/isSPACE>, and by the internal
e72ec78c 138macros C<is_utf8_space> under UTF-8, and C<isSPACE_LC> under C<use
c998b245 139locale>.
108003db 140
f0f9b3b8
KW
141=back
142
143Additional flags:
144
145=over 4
146
0ac6acae
AB
147=item RXf_SPLIT
148
149If C<split> is invoked as C<split ' '> or with no arguments (which
5137fa37 150really means C<split(' ', $_)>, see L<split|perlfunc/split>), perl will
0ac6acae
AB
151set this flag. The regex engine can then check for it and set the
152SKIPWHITE and WHITE extflags. To do this the perl engine does:
153
154 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
155 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
156
108003db
RGS
157=back
158
c998b245
AB
159These flags can be set during compilation to enable optimizations in
160the C<split> operator.
161
162=over 4
163
0ac6acae
AB
164=item RXf_SKIPWHITE
165
166If the flag is present in C<< rx->extflags >> C<split> will delete
167whitespace from the start of the subject string before it's operated
168on. What is considered whitespace depends on whether the subject is a
169UTF-8 string and whether the C<RXf_PMf_LOCALE> flag is set.
170
171If RXf_WHITE is set in addition to this flag C<split> will behave like
172C<split " "> under the perl engine.
173
c998b245
AB
174=item RXf_START_ONLY
175
176Tells the split operator to split the target string on newlines
177(C<\n>) without invoking the regex engine.
178
179Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp
180== '^'>), even under C</^/s>, see L<split|perlfunc>. Of course a
181different regex engine might want to use the same optimizations
182with a different syntax.
183
184=item RXf_WHITE
185
186Tells the split operator to split the target string on whitespace
187without invoking the regex engine. The definition of whitespace varies
188depending on whether the target string is a UTF-8 string and on
189whether RXf_PMf_LOCALE is set.
190
0ac6acae 191Perl's engine sets this flag if the pattern is C<\s+>.
c998b245 192
640f820d
AB
193=item RXf_NULL
194
a0e97681 195Tells the split operator to split the target string on
640f820d
AB
196characters. The definition of character varies depending on whether
197the target string is a UTF-8 string.
198
199Perl's engine sets this flag on empty patterns, this optimization
a0e97681 200makes C<split //> much faster than it would otherwise be. It's even
640f820d
AB
201faster than C<unpack>.
202
c998b245 203=back
108003db
RGS
204
205=head2 exec
206
49d7dfbc 207 I32 exec(pTHX_ REGEXP * const rx,
108003db
RGS
208 char *stringarg, char* strend, char* strbeg,
209 I32 minend, SV* screamer,
210 void* data, U32 flags);
211
8fd1a950
DM
212Execute a regexp. The arguments are
213
214=over 4
215
216=item rx
217
218The regular expression to execute.
219
220=item screamer
221
222This strangely-named arg is the SV to be matched against. Note that the
223actual char array to be matched against is supplied by the arguments
224described below; the SV is just used to determine UTF8ness, C<pos()> etc.
225
226=item strbeg
227
228Pointer to the physical start of the string.
229
230=item strend
231
232Pointer to the character following the physical end of the string (i.e.
233the \0).
234
235=item stringarg
236
237Pointer to the position in the string where matching should start; it might
238not be equal to C<strbeg> (for example in a later iteration of C</.../g>).
239
240=item minend
241
242Minimum length of string (measured in bytes from C<stringarg>) that must
243match; if the engine reaches the end of the match but hasn't reached this
244position in the string, it should fail.
245
246=item data
247
248Optimisation data; subject to change.
249
250=item flags
251
252Optimisation flags; subject to change.
253
254=back
108003db
RGS
255
256=head2 intuit
257
49d7dfbc 258 char* intuit(pTHX_ REGEXP * const rx,
108003db 259 SV *sv, char *strpos, char *strend,
49d7dfbc 260 const U32 flags, struct re_scream_pos_data_s *data);
108003db
RGS
261
262Find the start position where a regex match should be attempted,
263or possibly whether the regex engine should not be run because the
264pattern can't match. This is called as appropriate by the core
265depending on the values of the extflags member of the regexp
266structure.
267
268=head2 checkstr
269
49d7dfbc 270 SV* checkstr(pTHX_ REGEXP * const rx);
108003db
RGS
271
272Return a SV containing a string that must appear in the pattern. Used
273by C<split> for optimising matches.
274
275=head2 free
276
49d7dfbc 277 void free(pTHX_ REGEXP * const rx);
108003db
RGS
278
279Called by perl when it is freeing a regexp pattern so that the engine
280can release any resources pointed to by the C<pprivate> member of the
281regexp structure. This is only responsible for freeing private data;
282perl will handle releasing anything else contained in the regexp structure.
283
192b9cd1 284=head2 Numbered capture callbacks
108003db 285
192b9cd1
AB
286Called to get/set the value of C<$`>, C<$'>, C<$&> and their named
287equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
c27a5cfe 288numbered capture groups (C<$1>, C<$2>, ...).
49d7dfbc 289
a0e97681 290The C<paren> parameter will be C<-2> for C<$`>, C<-1> for C<$'>, C<0>
49d7dfbc
AB
291for C<$&>, C<1> for C<$1> and so forth.
292
192b9cd1
AB
293The names have been chosen by analogy with L<Tie::Scalar> methods
294names with an additional B<LENGTH> callback for efficiency. However
295named capture variables are currently not tied internally but
296implemented via magic.
297
298=head3 numbered_buff_FETCH
299
300 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
301 SV * const sv);
302
303Fetch a specified numbered capture. C<sv> should be set to the scalar
304to return, the scalar is passed as an argument rather than being
305returned from the function because when it's called perl already has a
306scalar to store the value, creating another one would be
307redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and
308friends, see L<perlapi>.
49d7dfbc
AB
309
310This callback is where perl untaints its own capture variables under
c998b245 311taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch>
49d7dfbc
AB
312function in F<regcomp.c> for how to untaint capture variables if
313that's something you'd like your engine to do as well.
108003db 314
192b9cd1 315=head3 numbered_buff_STORE
108003db 316
2fdbfb4d
AB
317 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
318 SV const * const value);
108003db 319
192b9cd1
AB
320Set the value of a numbered capture variable. C<value> is the scalar
321that is to be used as the new value. It's up to the engine to make
322sure this is used as the new value (or reject it).
2fdbfb4d
AB
323
324Example:
325
326 if ("ook" =~ /(o*)/) {
ccf3535a 327 # 'paren' will be '1' and 'value' will be 'ee'
2fdbfb4d
AB
328 $1 =~ tr/o/e/;
329 }
330
331Perl's own engine will croak on any attempt to modify the capture
a0e97681 332variables, to do this in another engine use the following callback
2fdbfb4d
AB
333(copied from C<Perl_reg_numbered_buff_store>):
334
335 void
336 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
337 SV const * const value)
338 {
339 PERL_UNUSED_ARG(rx);
340 PERL_UNUSED_ARG(paren);
341 PERL_UNUSED_ARG(value);
342
343 if (!PL_localizing)
344 Perl_croak(aTHX_ PL_no_modify);
345 }
346
99d59c4d 347Actually perl will not I<always> croak in a statement that looks
2fdbfb4d
AB
348like it would modify a numbered capture variable. This is because the
349STORE callback will not be called if perl can determine that it
350doesn't have to modify the value. This is exactly how tied variables
351behave in the same situation:
352
353 package CaptureVar;
354 use base 'Tie::Scalar';
355
356 sub TIESCALAR { bless [] }
357 sub FETCH { undef }
358 sub STORE { die "This doesn't get called" }
359
360 package main;
361
c69ca1d4 362 tie my $sv => "CaptureVar";
2fdbfb4d
AB
363 $sv =~ y/a/b/;
364
365Because C<$sv> is C<undef> when the C<y///> operator is applied to it
366the transliteration won't actually execute and the program won't
192b9cd1
AB
367C<die>. This is different to how 5.8 and earlier versions behaved
368since the capture variables were READONLY variables then, now they'll
369just die when assigned to in the default engine.
2fdbfb4d 370
192b9cd1 371=head3 numbered_buff_LENGTH
2fdbfb4d
AB
372
373 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
374 const I32 paren);
375
376Get the C<length> of a capture variable. There's a special callback
377for this so that perl doesn't have to do a FETCH and run C<length> on
192b9cd1 378the result, since the length is (in perl's case) known from an offset
0a3a8dc0 379stored in C<< rx->offs >> this is much more efficient:
2fdbfb4d
AB
380
381 I32 s1 = rx->offs[paren].start;
382 I32 s2 = rx->offs[paren].end;
383 I32 len = t1 - s1;
384
385This is a little bit more complex in the case of UTF-8, see what
386C<Perl_reg_numbered_buff_length> does with
387L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
388
192b9cd1
AB
389=head2 Named capture callbacks
390
391Called to get/set the value of C<%+> and C<%-> as well as by some
392utility functions in L<re>.
393
394There are two callbacks, C<named_buff> is called in all the cases the
395FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks
396would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the
397same cases as FIRSTKEY and NEXTKEY.
398
399The C<flags> parameter can be used to determine which of these
400operations the callbacks should respond to, the following flags are
401currently defined:
402
403Which L<Tie::Hash> operation is being performed from the Perl level on
404C<%+> or C<%+>, if any:
405
f1b875a0
YO
406 RXapif_FETCH
407 RXapif_STORE
408 RXapif_DELETE
409 RXapif_CLEAR
410 RXapif_EXISTS
411 RXapif_SCALAR
412 RXapif_FIRSTKEY
413 RXapif_NEXTKEY
192b9cd1
AB
414
415Whether C<%+> or C<%-> is being operated on, if any.
2fdbfb4d 416
f1b875a0
YO
417 RXapif_ONE /* %+ */
418 RXapif_ALL /* %- */
2fdbfb4d 419
192b9cd1 420Whether this is being called as C<re::regname>, C<re::regnames> or
c998b245 421C<re::regnames_count>, if any. The first two will be combined with
f1b875a0 422C<RXapif_ONE> or C<RXapif_ALL>.
192b9cd1 423
f1b875a0
YO
424 RXapif_REGNAME
425 RXapif_REGNAMES
426 RXapif_REGNAMES_COUNT
192b9cd1
AB
427
428Internally C<%+> and C<%-> are implemented with a real tied interface
429via L<Tie::Hash::NamedCapture>. The methods in that package will call
430back into these functions. However the usage of
431L<Tie::Hash::NamedCapture> for this purpose might change in future
432releases. For instance this might be implemented by magic instead
433(would need an extension to mgvtbl).
434
435=head3 named_buff
436
437 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
438 SV * const value, U32 flags);
439
440=head3 named_buff_iter
441
442 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
443 const U32 flags);
108003db 444
49d7dfbc 445=head2 qr_package
108003db 446
49d7dfbc 447 SV* qr_package(pTHX_ REGEXP * const rx);
108003db
RGS
448
449The package the qr// magic object is blessed into (as seen by C<ref
49d7dfbc
AB
450qr//>). It is recommended that engines change this to their package
451name for identification regardless of whether they implement methods
452on the object.
453
192b9cd1 454The package this method returns should also have the internal
d5213412 455C<Regexp> package in its C<@ISA>. C<< qr//->isa("Regexp") >> should always
192b9cd1
AB
456be true regardless of what engine is being used.
457
458Example implementation might be:
108003db
RGS
459
460 SV*
192b9cd1 461 Example_qr_package(pTHX_ REGEXP * const rx)
108003db
RGS
462 {
463 PERL_UNUSED_ARG(rx);
464 return newSVpvs("re::engine::Example");
465 }
466
467Any method calls on an object created with C<qr//> will be dispatched to the
468package as a normal object.
469
470 use re::engine::Example;
471 my $re = qr//;
472 $re->meth; # dispatched to re::engine::Example::meth()
473
f7e71195
AB
474To retrieve the C<REGEXP> object from the scalar in an XS function use
475the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP
476Functions>.
108003db
RGS
477
478 void meth(SV * rv)
479 PPCODE:
f7e71195 480 REGEXP * re = SvRX(sv);
108003db 481
108003db
RGS
482=head2 dupe
483
49d7dfbc 484 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
485
486On threaded builds a regexp may need to be duplicated so that the pattern
a0e97681 487can be used by multiple threads. This routine is expected to handle the
108003db
RGS
488duplication of any private data pointed to by the C<pprivate> member of
489the regexp structure. It will be called with the preconstructed new
490regexp structure as an argument, the C<pprivate> member will point at
a0e97681 491the B<old> private structure, and it is this routine's responsibility to
108003db
RGS
492construct a copy and return a pointer to it (which perl will then use to
493overwrite the field as passed to this routine.)
494
495This allows the engine to dupe its private data but also if necessary
496modify the final structure if it really must.
497
498On unthreaded builds this field doesn't exist.
499
3c13cae6
DM
500=head2 op_comp
501
502This is private to the perl core and subject to change. Should be left
503null.
504
108003db
RGS
505=head1 The REGEXP structure
506
507The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
508correctly build such a structure in their L</comp> routine.
509
510The REGEXP structure contains all the data that perl needs to be aware of
511to properly work with the regular expression. It includes data about
512optimisations that perl can use to determine if the regex engine should
513really be used, and various other control info that is needed to properly
514execute patterns in various contexts such as is the pattern anchored in
515some way, or what flags were used during the compile, or whether the
516program contains special constructs that perl needs to be aware of.
517
882227b7
AB
518In addition it contains two fields that are intended for the private
519use of the regex engine that compiled the pattern. These are the
520C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
521an arbitrary structure whose use and management is the responsibility
522of the compiling engine. perl will never modify either of these
523values.
108003db
RGS
524
525 typedef struct regexp {
526 /* what engine created this regexp? */
527 const struct regexp_engine* engine;
528
529 /* what re is this a lightweight copy of? */
530 struct regexp* mother_re;
531
532 /* Information about the match that the perl core uses to manage things */
533 U32 extflags; /* Flags used both externally and internally */
534 I32 minlen; /* mininum possible length of string to match */
535 I32 minlenret; /* mininum possible length of $& */
536 U32 gofs; /* chars left of pos that we search from */
537
538 /* substring data about strings that must appear
539 in the final match, used for optimisations */
540 struct reg_substr_data *substrs;
541
c27a5cfe 542 U32 nparens; /* number of capture groups */
108003db
RGS
543
544 /* private engine specific data */
545 U32 intflags; /* Engine Specific Internal flags */
546 void *pprivate; /* Data private to the regex engine which
547 created this object. */
548
549 /* Data about the last/current match. These are modified during matching*/
82f14494
DM
550 U32 lastparen; /* highest close paren matched ($+) */
551 U32 lastcloseparen; /* last close paren matched ($^N) */
108003db
RGS
552 regexp_paren_pair *swap; /* Swap copy of *offs */
553 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
554
555 char *subbeg; /* saved or original string so \digit works forever. */
556 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
557 I32 sublen; /* Length of string pointed by subbeg */
6502e081
DM
558 I32 suboffset; /* byte offset of subbeg from logical start of str */
559 I32 subcoffset; /* suboffset equiv, but in chars (for @-/@+) */
108003db
RGS
560
561 /* Information about the match that isn't often used */
562 I32 prelen; /* length of precomp */
563 const char *precomp; /* pre-compilation regular expression */
564
108003db
RGS
565 char *wrapped; /* wrapped version of the pattern */
566 I32 wraplen; /* length of wrapped */
567
568 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
569 HV *paren_names; /* Optional hash of paren names */
570
571 /* Refcount of this regexp */
572 I32 refcnt; /* Refcount of this regexp */
573 } regexp;
574
575The fields are discussed in more detail below:
576
882227b7 577=head2 C<engine>
108003db
RGS
578
579This field points at a regexp_engine structure which contains pointers
580to the subroutines that are to be used for performing a match. It
581is the compiling routine's responsibility to populate this field before
582returning the regexp object.
583
584Internally this is set to C<NULL> unless a custom engine is specified in
585C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
586pointed to by C<RE_ENGINE_PTR>.
587
882227b7 588=head2 C<mother_re>
108003db
RGS
589
590TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
591
882227b7 592=head2 C<extflags>
108003db 593
192b9cd1
AB
594This will be used by perl to see what flags the regexp was compiled
595with, this will normally be set to the value of the flags parameter by
c998b245
AB
596the L<comp|/comp> callback. See the L<comp|/comp> documentation for
597valid flags.
108003db 598
882227b7 599=head2 C<minlen> C<minlenret>
108003db
RGS
600
601The minimum string length required for the pattern to match. This is used to
602prune the search space by not bothering to match any closer to the end of a
603string than would allow a match. For instance there is no point in even
604starting the regex engine if the minlen is 10 but the string is only 5
605characters long. There is no way that the pattern can match.
606
607C<minlenret> is the minimum length of the string that would be found
608in $& after a match.
609
610The difference between C<minlen> and C<minlenret> can be seen in the
611following pattern:
612
613 /ns(?=\d)/
614
615where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
616required to match but is not actually included in the matched content. This
617distinction is particularly important as the substitution logic uses the
a0e97681 618C<minlenret> to tell whether it can do in-place substitution which can result in
108003db
RGS
619considerable speedup.
620
882227b7 621=head2 C<gofs>
108003db
RGS
622
623Left offset from pos() to start match at.
624
882227b7 625=head2 C<substrs>
108003db 626
192b9cd1
AB
627Substring data about strings that must appear in the final match. This
628is currently only used internally by perl's engine for but might be
c998b245 629used in the future for all engines for optimisations.
108003db 630
1cecf2c0 631=head2 C<nparens>, C<lastparen>, and C<lastcloseparen>
108003db
RGS
632
633These fields are used to keep track of how many paren groups could be matched
634in the pattern, which was the last open paren to be entered, and which was
635the last close paren to be entered.
636
882227b7 637=head2 C<intflags>
108003db
RGS
638
639The engine's private copy of the flags the pattern was compiled with. Usually
192b9cd1 640this is the same as C<extflags> unless the engine chose to modify one of them.
108003db 641
882227b7 642=head2 C<pprivate>
108003db
RGS
643
644A void* pointing to an engine-defined data structure. The perl engine uses the
645C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
646engine should use something else.
647
882227b7 648=head2 C<swap>
108003db 649
e9105d30 650Unused. Left in for compatibility with perl 5.10.0.
108003db 651
882227b7 652=head2 C<offs>
108003db
RGS
653
654A C<regexp_paren_pair> structure which defines offsets into the string being
655matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
656C<regexp_paren_pair> struct is defined as follows:
657
658 typedef struct regexp_paren_pair {
659 I32 start;
660 I32 end;
661 } regexp_paren_pair;
662
663If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
c27a5cfe 664capture group did not match. C<< ->offs[0].start/end >> represents C<$&> (or
108003db
RGS
665C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
666C<$paren >= 1>.
667
882227b7 668=head2 C<precomp> C<prelen>
108003db 669
192b9cd1
AB
670Used for optimisations. C<precomp> holds a copy of the pattern that
671was compiled and C<prelen> its length. When a new pattern is to be
672compiled (such as inside a loop) the internal C<regcomp> operator
673checks whether the last compiled C<REGEXP>'s C<precomp> and C<prelen>
674are equivalent to the new one, and if so uses the old pattern instead
675of compiling a new one.
676
677The relevant snippet from C<Perl_pp_regcomp>:
678
679 if (!re || !re->precomp || re->prelen != (I32)len ||
680 memNE(re->precomp, t, len))
681 /* Compile a new pattern */
108003db 682
882227b7 683=head2 C<paren_names>
108003db 684
c27a5cfe 685This is a hash used internally to track named capture groups and their
108003db
RGS
686offsets. The keys are the names of the buffers the values are dualvars,
687with the IV slot holding the number of buffers with the given name and the
688pv being an embedded array of I32. The values may also be contained
689independently in the data array in cases where named backreferences are
690used.
691
c998b245 692=head2 C<substrs>
108003db
RGS
693
694Holds information on the longest string that must occur at a fixed
695offset from the start of the pattern, and the longest string that must
696occur at a floating offset from the start of the pattern. Used to do
697Fast-Boyer-Moore searches on the string to find out if its worth using
698the regex engine at all, and if so where in the string to search.
699
6502e081
DM
700=head2 C<subbeg> C<sublen> C<saved_copy> C<suboffset> C<subcoffset>
701
702Used during the execution phase for managing search and replace patterns,
703and for providing the text for C<$&>, C<$1> etc. C<subbeg> points to a
704buffer (either the original string, or a copy in the case of
705C<RX_MATCH_COPIED(rx)>), and C<sublen> is the length of the buffer. The
706C<RX_OFFS> start and end indices index into this buffer.
707
708In the presence of the C<REXEC_COPY_STR> flag, but with the addition of
709the C<REXEC_COPY_SKIP_PRE> or C<REXEC_COPY_SKIP_POST> flags, an engine
710can choose not to copy the full buffer (although it must still do so in
711the presence of C<RXf_PMf_KEEPCOPY> or the relevant bits being set in
712C<PL_sawampersand>). In this case, it may set C<suboffset> to indicate the
713number of bytes from the logical start of the buffer to the physical start
714(i.e. C<subbeg>). It should also set C<subcoffset>, the number of
715characters in the offset. The latter is needed to support C<@-> and C<@+>
716which work in characters, not bytes.
108003db 717
882227b7 718=head2 C<wrapped> C<wraplen>
108003db 719
c998b245 720Stores the string C<qr//> stringifies to. The perl engine for example
ed215d3c 721stores C<(?^:eek)> in the case of C<qr/eek/>.
108003db 722
c998b245
AB
723When using a custom engine that doesn't support the C<(?:)> construct
724for inline modifiers, it's probably best to have C<qr//> stringify to
725the supplied pattern, note that this will create undesired patterns in
726cases such as:
108003db
RGS
727
728 my $x = qr/a|b/; # "a|b"
192b9cd1 729 my $y = qr/c/i; # "c"
108003db
RGS
730 my $z = qr/$x$y/; # "a|bc"
731
192b9cd1
AB
732There's no solution for this problem other than making the custom
733engine understand a construct like C<(?:)>.
108003db 734
882227b7 735=head2 C<seen_evals>
108003db
RGS
736
737This stores the number of eval groups in the pattern. This is used for security
738purposes when embedding compiled regexes into larger patterns with C<qr//>.
739
882227b7 740=head2 C<refcnt>
108003db
RGS
741
742The number of times the structure is referenced. When this falls to 0 the
743regexp is automatically freed by a call to pregfree. This should be set to 1 in
744each engine's L</comp> routine.
745
108003db
RGS
746=head1 HISTORY
747
748Originally part of L<perlreguts>.
749
750=head1 AUTHORS
751
752Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
753Bjarmason.
754
755=head1 LICENSE
756
757Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
758
759This program is free software; you can redistribute it and/or modify it under
760the same terms as Perl itself.
761
762=cut