3 perlreapi - perl regular expression plugin interface
7 As of Perl 5.9.5 there is a new interface for plugging and using other
8 regular expression engines than the default one.
10 Each engine is supposed to provide access to a constant structure of the
13 typedef struct regexp_engine {
14 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
15 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
16 char* strbeg, I32 minend, SV* screamer,
17 void* data, U32 flags);
18 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
19 char *strend, U32 flags,
20 struct re_scream_pos_data_s *data);
21 SV* (*checkstr) (pTHX_ REGEXP * const rx);
22 void (*free) (pTHX_ REGEXP * const rx);
23 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
25 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
26 SV const * const value);
27 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
29 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
30 SV * const value, U32 flags);
31 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
33 SV* (*qr_package)(pTHX_ REGEXP * const rx);
35 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
38 When a regexp is compiled, its C<engine> field is then set to point at
39 the appropriate structure, so that when it needs to be used Perl can find
40 the right routines to do so.
42 In order to install a new regexp handler, C<$^H{regcomp}> is set
43 to an integer which (when casted appropriately) resolves to one of these
44 structures. When compiling, the C<comp> method is executed, and the
45 resulting regexp structure's engine field is expected to point back at
48 The pTHX_ symbol in the definition is a macro used by perl under threading
49 to provide an extra argument to the routine holding a pointer back to
50 the interpreter that is executing the regexp. So under threading all
51 routines get an extra argument.
57 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
59 Compile the pattern stored in C<pattern> using the given C<flags> and
60 return a pointer to a prepared C<REGEXP> structure that can perform
61 the match. See L</The REGEXP structure> below for an explanation of
62 the individual fields in the REGEXP struct.
64 The C<pattern> parameter is the scalar that was used as the
65 pattern. previous versions of perl would pass two C<char*> indicating
66 the start and end of the stringified pattern, the following snippet can
67 be used to get the old parameters:
70 char* exp = SvPV(pattern, plen);
71 char* xend = exp + plen;
73 Since any scalar can be passed as a pattern it's possible to implement
74 an engine that does something with an array (C<< "ook" =~ [ qw/ eek
75 hlagh / ] >>) or with the non-stringified form of a compiled regular
76 expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
77 stringify everything using the snippet above but that doesn't mean
78 other engines have to.
80 The C<flags> parameter is a bitfield which indicates which of the
81 C<msixp> flags the regex was compiled with. It also contains
82 additional info such as whether C<use locale> is in effect.
84 The C<eogc> flags are stripped out before being passed to the comp
85 routine. The regex engine does not need to know whether any of these
86 are set as those flags should only affect what perl does with the
87 pattern and its match variables, not how it gets compiled and
90 By the time the comp callback is called, some of these flags have
91 already had effect (noted below where applicable). However most of
92 their effect occurs after the comp callback has run in routines that
93 read the C<< rx->extflags >> field which it populates.
95 In general the flags should be preserved in C<< rx->extflags >> after
96 compilation, although the regex engine might want to add or delete
97 some of them to invoke or disable some special behavior in perl. The
98 flags along with any special behavior they cause are documented below:
100 The pattern modifiers:
104 =item C</m> - RXf_PMf_MULTILINE
106 If this is in C<< rx->extflags >> it will be passed to
107 C<Perl_fbm_instr> by C<pp_split> which will treat the subject string
108 as a multi-line string.
110 =item C</s> - RXf_PMf_SINGLELINE
112 =item C</i> - RXf_PMf_FOLD
114 =item C</x> - RXf_PMf_EXTENDED
116 If present on a regex C<#> comments will be handled differently by the
117 tokenizer in some cases.
119 TODO: Document those cases.
121 =item C</p> - RXf_PMf_KEEPCOPY
125 The character set semantics are determined by an enum that is contained
126 in this field. This is still experimental and subject to change, but
127 the current interface returns the rules by use of the in-line function
128 C<get_regex_charset(const U32 flags)>. The only currently documented
129 value returned from it is REGEX_LOCALE_CHARSET, which is set if
130 C<use locale> is in effect. If present in C<< rx->extflags >>
131 C<split> will use the locale dependent definition of whitespace under
132 when RXf_SKIPWHITE or RXf_WHITE are in effect. Under ASCII whitespace
133 is defined as per L<isSPACE|perlapi/isSPACE>, and by the internal
134 macros C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use
145 Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime.
147 A regex engine may want to set or disable this flag during
148 compilation. The perl engine for instance may upgrade non-UTF-8
149 strings to UTF-8 if the pattern includes constructs such as C<\x{...}>
150 that can only match Unicode values.
154 If C<split> is invoked as C<split ' '> or with no arguments (which
155 really means C<split(' ', $_)>, see L<split|perlfunc/split>), perl will
156 set this flag. The regex engine can then check for it and set the
157 SKIPWHITE and WHITE extflags. To do this the perl engine does:
159 if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ')
160 r->extflags |= (RXf_SKIPWHITE|RXf_WHITE);
164 These flags can be set during compilation to enable optimizations in
165 the C<split> operator.
171 If the flag is present in C<< rx->extflags >> C<split> will delete
172 whitespace from the start of the subject string before it's operated
173 on. What is considered whitespace depends on whether the subject is a
174 UTF-8 string and whether the C<RXf_PMf_LOCALE> flag is set.
176 If RXf_WHITE is set in addition to this flag C<split> will behave like
177 C<split " "> under the perl engine.
181 Tells the split operator to split the target string on newlines
182 (C<\n>) without invoking the regex engine.
184 Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp
185 == '^'>), even under C</^/s>, see L<split|perlfunc>. Of course a
186 different regex engine might want to use the same optimizations
187 with a different syntax.
191 Tells the split operator to split the target string on whitespace
192 without invoking the regex engine. The definition of whitespace varies
193 depending on whether the target string is a UTF-8 string and on
194 whether RXf_PMf_LOCALE is set.
196 Perl's engine sets this flag if the pattern is C<\s+>.
200 Tells the split operator to split the target string on
201 characters. The definition of character varies depending on whether
202 the target string is a UTF-8 string.
204 Perl's engine sets this flag on empty patterns, this optimization
205 makes C<split //> much faster than it would otherwise be. It's even
206 faster than C<unpack>.
212 I32 exec(pTHX_ REGEXP * const rx,
213 char *stringarg, char* strend, char* strbeg,
214 I32 minend, SV* screamer,
215 void* data, U32 flags);
221 char* intuit(pTHX_ REGEXP * const rx,
222 SV *sv, char *strpos, char *strend,
223 const U32 flags, struct re_scream_pos_data_s *data);
225 Find the start position where a regex match should be attempted,
226 or possibly whether the regex engine should not be run because the
227 pattern can't match. This is called as appropriate by the core
228 depending on the values of the extflags member of the regexp
233 SV* checkstr(pTHX_ REGEXP * const rx);
235 Return a SV containing a string that must appear in the pattern. Used
236 by C<split> for optimising matches.
240 void free(pTHX_ REGEXP * const rx);
242 Called by perl when it is freeing a regexp pattern so that the engine
243 can release any resources pointed to by the C<pprivate> member of the
244 regexp structure. This is only responsible for freeing private data;
245 perl will handle releasing anything else contained in the regexp structure.
247 =head2 Numbered capture callbacks
249 Called to get/set the value of C<$`>, C<$'>, C<$&> and their named
250 equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the
251 numbered capture groups (C<$1>, C<$2>, ...).
253 The C<paren> parameter will be C<-2> for C<$`>, C<-1> for C<$'>, C<0>
254 for C<$&>, C<1> for C<$1> and so forth.
256 The names have been chosen by analogy with L<Tie::Scalar> methods
257 names with an additional B<LENGTH> callback for efficiency. However
258 named capture variables are currently not tied internally but
259 implemented via magic.
261 =head3 numbered_buff_FETCH
263 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
266 Fetch a specified numbered capture. C<sv> should be set to the scalar
267 to return, the scalar is passed as an argument rather than being
268 returned from the function because when it's called perl already has a
269 scalar to store the value, creating another one would be
270 redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and
271 friends, see L<perlapi>.
273 This callback is where perl untaints its own capture variables under
274 taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch>
275 function in F<regcomp.c> for how to untaint capture variables if
276 that's something you'd like your engine to do as well.
278 =head3 numbered_buff_STORE
280 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
281 SV const * const value);
283 Set the value of a numbered capture variable. C<value> is the scalar
284 that is to be used as the new value. It's up to the engine to make
285 sure this is used as the new value (or reject it).
289 if ("ook" =~ /(o*)/) {
290 # `paren' will be `1' and `value' will be `ee'
294 Perl's own engine will croak on any attempt to modify the capture
295 variables, to do this in another engine use the following callback
296 (copied from C<Perl_reg_numbered_buff_store>):
299 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
300 SV const * const value)
303 PERL_UNUSED_ARG(paren);
304 PERL_UNUSED_ARG(value);
307 Perl_croak(aTHX_ PL_no_modify);
310 Actually perl will not I<always> croak in a statement that looks
311 like it would modify a numbered capture variable. This is because the
312 STORE callback will not be called if perl can determine that it
313 doesn't have to modify the value. This is exactly how tied variables
314 behave in the same situation:
317 use base 'Tie::Scalar';
319 sub TIESCALAR { bless [] }
321 sub STORE { die "This doesn't get called" }
325 tie my $sv => "CaptureVar";
328 Because C<$sv> is C<undef> when the C<y///> operator is applied to it
329 the transliteration won't actually execute and the program won't
330 C<die>. This is different to how 5.8 and earlier versions behaved
331 since the capture variables were READONLY variables then, now they'll
332 just die when assigned to in the default engine.
334 =head3 numbered_buff_LENGTH
336 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
339 Get the C<length> of a capture variable. There's a special callback
340 for this so that perl doesn't have to do a FETCH and run C<length> on
341 the result, since the length is (in perl's case) known from an offset
342 stored in C<< rx->offs >> this is much more efficient:
344 I32 s1 = rx->offs[paren].start;
345 I32 s2 = rx->offs[paren].end;
348 This is a little bit more complex in the case of UTF-8, see what
349 C<Perl_reg_numbered_buff_length> does with
350 L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
352 =head2 Named capture callbacks
354 Called to get/set the value of C<%+> and C<%-> as well as by some
355 utility functions in L<re>.
357 There are two callbacks, C<named_buff> is called in all the cases the
358 FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks
359 would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the
360 same cases as FIRSTKEY and NEXTKEY.
362 The C<flags> parameter can be used to determine which of these
363 operations the callbacks should respond to, the following flags are
366 Which L<Tie::Hash> operation is being performed from the Perl level on
367 C<%+> or C<%+>, if any:
378 Whether C<%+> or C<%-> is being operated on, if any.
383 Whether this is being called as C<re::regname>, C<re::regnames> or
384 C<re::regnames_count>, if any. The first two will be combined with
385 C<RXapif_ONE> or C<RXapif_ALL>.
389 RXapif_REGNAMES_COUNT
391 Internally C<%+> and C<%-> are implemented with a real tied interface
392 via L<Tie::Hash::NamedCapture>. The methods in that package will call
393 back into these functions. However the usage of
394 L<Tie::Hash::NamedCapture> for this purpose might change in future
395 releases. For instance this might be implemented by magic instead
396 (would need an extension to mgvtbl).
400 SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key,
401 SV * const value, U32 flags);
403 =head3 named_buff_iter
405 SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey,
410 SV* qr_package(pTHX_ REGEXP * const rx);
412 The package the qr// magic object is blessed into (as seen by C<ref
413 qr//>). It is recommended that engines change this to their package
414 name for identification regardless of whether they implement methods
417 The package this method returns should also have the internal
418 C<Regexp> package in its C<@ISA>. C<< qr//->isa("Regexp") >> should always
419 be true regardless of what engine is being used.
421 Example implementation might be:
424 Example_qr_package(pTHX_ REGEXP * const rx)
427 return newSVpvs("re::engine::Example");
430 Any method calls on an object created with C<qr//> will be dispatched to the
431 package as a normal object.
433 use re::engine::Example;
435 $re->meth; # dispatched to re::engine::Example::meth()
437 To retrieve the C<REGEXP> object from the scalar in an XS function use
438 the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP
443 REGEXP * re = SvRX(sv);
447 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
449 On threaded builds a regexp may need to be duplicated so that the pattern
450 can be used by multiple threads. This routine is expected to handle the
451 duplication of any private data pointed to by the C<pprivate> member of
452 the regexp structure. It will be called with the preconstructed new
453 regexp structure as an argument, the C<pprivate> member will point at
454 the B<old> private structure, and it is this routine's responsibility to
455 construct a copy and return a pointer to it (which perl will then use to
456 overwrite the field as passed to this routine.)
458 This allows the engine to dupe its private data but also if necessary
459 modify the final structure if it really must.
461 On unthreaded builds this field doesn't exist.
463 =head1 The REGEXP structure
465 The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
466 correctly build such a structure in their L</comp> routine.
468 The REGEXP structure contains all the data that perl needs to be aware of
469 to properly work with the regular expression. It includes data about
470 optimisations that perl can use to determine if the regex engine should
471 really be used, and various other control info that is needed to properly
472 execute patterns in various contexts such as is the pattern anchored in
473 some way, or what flags were used during the compile, or whether the
474 program contains special constructs that perl needs to be aware of.
476 In addition it contains two fields that are intended for the private
477 use of the regex engine that compiled the pattern. These are the
478 C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
479 an arbitrary structure whose use and management is the responsibility
480 of the compiling engine. perl will never modify either of these
483 typedef struct regexp {
484 /* what engine created this regexp? */
485 const struct regexp_engine* engine;
487 /* what re is this a lightweight copy of? */
488 struct regexp* mother_re;
490 /* Information about the match that the perl core uses to manage things */
491 U32 extflags; /* Flags used both externally and internally */
492 I32 minlen; /* mininum possible length of string to match */
493 I32 minlenret; /* mininum possible length of $& */
494 U32 gofs; /* chars left of pos that we search from */
496 /* substring data about strings that must appear
497 in the final match, used for optimisations */
498 struct reg_substr_data *substrs;
500 U32 nparens; /* number of capture groups */
502 /* private engine specific data */
503 U32 intflags; /* Engine Specific Internal flags */
504 void *pprivate; /* Data private to the regex engine which
505 created this object. */
507 /* Data about the last/current match. These are modified during matching*/
508 U32 lastparen; /* last open paren matched */
509 U32 lastcloseparen; /* last close paren matched */
510 regexp_paren_pair *swap; /* Swap copy of *offs */
511 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
513 char *subbeg; /* saved or original string so \digit works forever. */
514 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
515 I32 sublen; /* Length of string pointed by subbeg */
517 /* Information about the match that isn't often used */
518 I32 prelen; /* length of precomp */
519 const char *precomp; /* pre-compilation regular expression */
521 char *wrapped; /* wrapped version of the pattern */
522 I32 wraplen; /* length of wrapped */
524 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
525 HV *paren_names; /* Optional hash of paren names */
527 /* Refcount of this regexp */
528 I32 refcnt; /* Refcount of this regexp */
531 The fields are discussed in more detail below:
535 This field points at a regexp_engine structure which contains pointers
536 to the subroutines that are to be used for performing a match. It
537 is the compiling routine's responsibility to populate this field before
538 returning the regexp object.
540 Internally this is set to C<NULL> unless a custom engine is specified in
541 C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
542 pointed to by C<RE_ENGINE_PTR>.
546 TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
550 This will be used by perl to see what flags the regexp was compiled
551 with, this will normally be set to the value of the flags parameter by
552 the L<comp|/comp> callback. See the L<comp|/comp> documentation for
555 =head2 C<minlen> C<minlenret>
557 The minimum string length required for the pattern to match. This is used to
558 prune the search space by not bothering to match any closer to the end of a
559 string than would allow a match. For instance there is no point in even
560 starting the regex engine if the minlen is 10 but the string is only 5
561 characters long. There is no way that the pattern can match.
563 C<minlenret> is the minimum length of the string that would be found
566 The difference between C<minlen> and C<minlenret> can be seen in the
571 where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
572 required to match but is not actually included in the matched content. This
573 distinction is particularly important as the substitution logic uses the
574 C<minlenret> to tell whether it can do in-place substitution which can result in
575 considerable speedup.
579 Left offset from pos() to start match at.
583 Substring data about strings that must appear in the final match. This
584 is currently only used internally by perl's engine for but might be
585 used in the future for all engines for optimisations.
587 =head2 C<nparens>, C<lasparen>, and C<lastcloseparen>
589 These fields are used to keep track of how many paren groups could be matched
590 in the pattern, which was the last open paren to be entered, and which was
591 the last close paren to be entered.
595 The engine's private copy of the flags the pattern was compiled with. Usually
596 this is the same as C<extflags> unless the engine chose to modify one of them.
600 A void* pointing to an engine-defined data structure. The perl engine uses the
601 C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
602 engine should use something else.
606 Unused. Left in for compatibility with perl 5.10.0.
610 A C<regexp_paren_pair> structure which defines offsets into the string being
611 matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
612 C<regexp_paren_pair> struct is defined as follows:
614 typedef struct regexp_paren_pair {
619 If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
620 capture group did not match. C<< ->offs[0].start/end >> represents C<$&> (or
621 C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
624 =head2 C<precomp> C<prelen>
626 Used for optimisations. C<precomp> holds a copy of the pattern that
627 was compiled and C<prelen> its length. When a new pattern is to be
628 compiled (such as inside a loop) the internal C<regcomp> operator
629 checks whether the last compiled C<REGEXP>'s C<precomp> and C<prelen>
630 are equivalent to the new one, and if so uses the old pattern instead
631 of compiling a new one.
633 The relevant snippet from C<Perl_pp_regcomp>:
635 if (!re || !re->precomp || re->prelen != (I32)len ||
636 memNE(re->precomp, t, len))
637 /* Compile a new pattern */
639 =head2 C<paren_names>
641 This is a hash used internally to track named capture groups and their
642 offsets. The keys are the names of the buffers the values are dualvars,
643 with the IV slot holding the number of buffers with the given name and the
644 pv being an embedded array of I32. The values may also be contained
645 independently in the data array in cases where named backreferences are
650 Holds information on the longest string that must occur at a fixed
651 offset from the start of the pattern, and the longest string that must
652 occur at a floating offset from the start of the pattern. Used to do
653 Fast-Boyer-Moore searches on the string to find out if its worth using
654 the regex engine at all, and if so where in the string to search.
656 =head2 C<subbeg> C<sublen> C<saved_copy>
658 Used during execution phase for managing search and replace patterns.
660 =head2 C<wrapped> C<wraplen>
662 Stores the string C<qr//> stringifies to. The perl engine for example
663 stores C<(?^:eek)> in the case of C<qr/eek/>.
665 When using a custom engine that doesn't support the C<(?:)> construct
666 for inline modifiers, it's probably best to have C<qr//> stringify to
667 the supplied pattern, note that this will create undesired patterns in
670 my $x = qr/a|b/; # "a|b"
671 my $y = qr/c/i; # "c"
672 my $z = qr/$x$y/; # "a|bc"
674 There's no solution for this problem other than making the custom
675 engine understand a construct like C<(?:)>.
679 This stores the number of eval groups in the pattern. This is used for security
680 purposes when embedding compiled regexes into larger patterns with C<qr//>.
684 The number of times the structure is referenced. When this falls to 0 the
685 regexp is automatically freed by a call to pregfree. This should be set to 1 in
686 each engine's L</comp> routine.
690 Originally part of L<perlreguts>.
694 Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
699 Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
701 This program is free software; you can redistribute it and/or modify it under
702 the same terms as Perl itself.