This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Upgrade to encoding::warnings 0.11
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
3perlreapi - perl regular expression plugin interface
4
5=head1 DESCRIPTION
6
7As of Perl 5.9.5 there is a new interface for using other regexp engines than
8the default one. Each engine is supposed to provide access to a constant
9structure of the following format:
10
11 typedef struct regexp_engine {
3ab4a224 12 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
49d7dfbc 13 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
2fdbfb4d
AB
14 char* strbeg, I32 minend, SV* screamer,
15 void* data, U32 flags);
49d7dfbc 16 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
2fdbfb4d
AB
17 char *strend, U32 flags,
18 struct re_scream_pos_data_s *data);
49d7dfbc
AB
19 SV* (*checkstr) (pTHX_ REGEXP * const rx);
20 void (*free) (pTHX_ REGEXP * const rx);
2fdbfb4d
AB
21 void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren,
22 SV * const sv);
23 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
24 SV const * const value);
25 I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv,
26 const I32 paren);
27 SV* (*named_buff_FETCH) (pTHX_ REGEXP * const rx, SV * const sv,
28 const U32 flags);
49d7dfbc 29 SV* (*qr_package)(pTHX_ REGEXP * const rx);
108003db 30 #ifdef USE_ITHREADS
49d7dfbc 31 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db 32 #endif
108003db
RGS
33
34When a regexp is compiled, its C<engine> field is then set to point at
35the appropriate structure so that when it needs to be used Perl can find
36the right routines to do so.
37
38In order to install a new regexp handler, C<$^H{regcomp}> is set
39to an integer which (when casted appropriately) resolves to one of these
40structures. When compiling, the C<comp> method is executed, and the
41resulting regexp structure's engine field is expected to point back at
42the same structure.
43
44The pTHX_ symbol in the definition is a macro used by perl under threading
45to provide an extra argument to the routine holding a pointer back to
46the interpreter that is executing the regexp. So under threading all
47routines get an extra argument.
48
882227b7 49=head1 Callbacks
108003db
RGS
50
51=head2 comp
52
3ab4a224 53 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 54
3ab4a224
AB
55Compile the pattern stored in C<pattern> using the given C<flags> and
56return a pointer to a prepared C<REGEXP> structure that can perform
57the match. See L</The REGEXP structure> below for an explanation of
58the individual fields in the REGEXP struct.
59
60The C<pattern> parameter is the scalar that was used as the
61pattern. previous versions of perl would pass two C<char*> indicating
62the start and end of the stringifed pattern, the following snippet can
63be used to get the old parameters:
64
65 STRLEN plen;
66 char* exp = SvPV(pattern, plen);
67 char* xend = exp + plen;
68
69Since any scalar can be passed as a pattern it's possible to implement
70an engine that does something with an array (C<< "ook" =~ [ qw/ eek
71hlagh / ] >>) or with the non-stringified form of a compiled regular
72expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
73stringify everything using the snippet above but that doesn't mean
74other engines have to.
108003db
RGS
75
76The C<flags> paramater is a bitfield which indicates which of the
77C<msixk> flags the regex was compiled with. In addition it contains
78info about whether C<use locale> is in effect and optimization info
79for C<split>. A regex engine might want to use the same split
80optimizations with a different syntax, for instance a Perl6 engine
81would treat C<split /^^/> equivalently to perl's C<split /^/>, see
82L<split documentation|perlfunc> and the relevant code in C<pp_split>
83in F<pp.c> to find out whether your engine should be setting these.
84
85The C<eogc> flags are stripped out before being passed to the comp
86routine. The regex engine does not need to know whether any of these
3ab4a224
AB
87are set as those flags should only affect what perl does with the
88pattern and its match variables, not how it gets compiled & executed.
108003db
RGS
89
90=over 4
91
92=item RXf_SKIPWHITE
93
94C<split ' '> or C<split> with no arguments (which really means
95C<split(' ', $_> see L<split|perlfunc>).
96
97=item RXf_START_ONLY
98
99Set if the pattern is C</^/> (C<<r->prelen == 1 && r->precomp[0] ==
100'^'>>). Will be used by the C<split> operator to split the given
101string on C<\n> (even under C</^/s>, see L<split|perlfunc>).
102
103=item RXf_WHITE
104
105Set if the pattern is exactly C</\s+/> and used by C<split>, the
106definition of whitespace varies depending on whether RXf_UTF8 or
107RXf_PMf_LOCALE is set.
108
109=item RXf_PMf_LOCALE
110
111Makes C<split> use the locale dependant definition of whitespace under C<use
112locale> when RXf_SKIPWHITE or RXf_WHITE is in effect. Under ASCII whitespace is
113defined as per L<isSPACE|perlapi/ISSPACE>, and by the internal macros
114C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use locale>.
115
116=item RXf_PMf_MULTILINE
117
118The C</m> flag, this ends up being passed to C<Perl_fbm_instr> by
119C<pp_split> regardless of the engine.
120
121=item RXf_PMf_SINGLELINE
122
123The C</s> flag. Guaranteed not to be used outside the regex engine.
124
125=item RXf_PMf_FOLD
126
127The C</i> flag. Guaranteed not to be used outside the regex engine.
128
129=item RXf_PMf_EXTENDED
130
131The C</x> flag. Guaranteed not to be used outside the regex
132engine. However if present on a regex C<#> comments will be stripped
133by the tokenizer regardless of the engine currently in use.
134
135=item RXf_PMf_KEEPCOPY
136
49d7dfbc 137The C</p> flag.
108003db
RGS
138
139=item RXf_UTF8
140
141Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime.
142
143=back
144
882227b7
AB
145In general these flags should be preserved in C<< rx->extflags >>
146after compilation, although it is possible the regex includes
147constructs that changes them. The perl engine for instance may upgrade
148non-utf8 strings to utf8 if the pattern includes constructs such as
149C<\x{...}> that can only match unicode values. RXf_SKIPWHITE should
150always be preserved verbatim in C<< regex->extflags >>.
108003db
RGS
151
152=head2 exec
153
49d7dfbc 154 I32 exec(pTHX_ REGEXP * const rx,
108003db
RGS
155 char *stringarg, char* strend, char* strbeg,
156 I32 minend, SV* screamer,
157 void* data, U32 flags);
158
159Execute a regexp.
160
161=head2 intuit
162
49d7dfbc 163 char* intuit(pTHX_ REGEXP * const rx,
108003db 164 SV *sv, char *strpos, char *strend,
49d7dfbc 165 const U32 flags, struct re_scream_pos_data_s *data);
108003db
RGS
166
167Find the start position where a regex match should be attempted,
168or possibly whether the regex engine should not be run because the
169pattern can't match. This is called as appropriate by the core
170depending on the values of the extflags member of the regexp
171structure.
172
173=head2 checkstr
174
49d7dfbc 175 SV* checkstr(pTHX_ REGEXP * const rx);
108003db
RGS
176
177Return a SV containing a string that must appear in the pattern. Used
178by C<split> for optimising matches.
179
180=head2 free
181
49d7dfbc 182 void free(pTHX_ REGEXP * const rx);
108003db
RGS
183
184Called by perl when it is freeing a regexp pattern so that the engine
185can release any resources pointed to by the C<pprivate> member of the
186regexp structure. This is only responsible for freeing private data;
187perl will handle releasing anything else contained in the regexp structure.
188
2fdbfb4d 189=head2 numbered_buff_FETCH
108003db 190
2fdbfb4d
AB
191 void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren,
192 SV * const sv);
108003db 193
49d7dfbc
AB
194Called to get the value of C<$`>, C<$'>, C<$&> (and their named
195equivalents, see L<perlvar>) and the numbered capture buffers (C<$1>,
196C<$2>, ...).
197
198The C<paren> paramater will be C<-2> for C<$`>, C<-1> for C<$'>, C<0>
199for C<$&>, C<1> for C<$1> and so forth.
200
2fdbfb4d
AB
201C<sv> should be set to the scalar to return, the scalar is passed as
202an argument rather than being returned from the function because when
203it's called perl already has a scalar to store the value, creating
204another one would be redundant. The scalar can be set with
49d7dfbc
AB
205C<sv_setsv>, C<sv_setpvn> and friends, see L<perlapi>.
206
207This callback is where perl untaints its own capture variables under
208taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_get>
209function in F<regcomp.c> for how to untaint capture variables if
210that's something you'd like your engine to do as well.
108003db 211
2fdbfb4d 212=head2 numbered_buff_STORE
108003db 213
2fdbfb4d
AB
214 void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren,
215 SV const * const value);
108003db 216
2fdbfb4d
AB
217Called to set the value of a numbered capture variable. C<paren> is
218the paren number (see the L<mapping|/numbered_buff_FETCH> above) and
219C<value> is the scalar that is to be used as the new value. It's up to
220the engine to make sure this is used as the new value (or reject it).
221
222Example:
223
224 if ("ook" =~ /(o*)/) {
225 # `paren' will be `1' and `value' will be `ee'
226 $1 =~ tr/o/e/;
227 }
228
229Perl's own engine will croak on any attempt to modify the capture
230variables, to do this in another engine use the following callack
231(copied from C<Perl_reg_numbered_buff_store>):
232
233 void
234 Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren,
235 SV const * const value)
236 {
237 PERL_UNUSED_ARG(rx);
238 PERL_UNUSED_ARG(paren);
239 PERL_UNUSED_ARG(value);
240
241 if (!PL_localizing)
242 Perl_croak(aTHX_ PL_no_modify);
243 }
244
245Actually perl 5.10 will not I<always> croak in a statement that looks
246like it would modify a numbered capture variable. This is because the
247STORE callback will not be called if perl can determine that it
248doesn't have to modify the value. This is exactly how tied variables
249behave in the same situation:
250
251 package CaptureVar;
252 use base 'Tie::Scalar';
253
254 sub TIESCALAR { bless [] }
255 sub FETCH { undef }
256 sub STORE { die "This doesn't get called" }
257
258 package main;
259
260 tie my $sv => "CatptureVar";
261 $sv =~ y/a/b/;
262
263Because C<$sv> is C<undef> when the C<y///> operator is applied to it
264the transliteration won't actually execute and the program won't
265C<die>. This is different to how 5.8 behaved since the capture
266variables were READONLY variables then, now they'll just die on
267assignment in the default engine.
268
269=head2 numbered_buff_LENGTH
270
271 I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv,
272 const I32 paren);
273
274Get the C<length> of a capture variable. There's a special callback
275for this so that perl doesn't have to do a FETCH and run C<length> on
276the result, since the length is (in perl's case) known from a memory
277offset this is much more efficient:
278
279 I32 s1 = rx->offs[paren].start;
280 I32 s2 = rx->offs[paren].end;
281 I32 len = t1 - s1;
282
283This is a little bit more complex in the case of UTF-8, see what
284C<Perl_reg_numbered_buff_length> does with
285L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>.
286
287=head2 named_buff_FETCH
288
289 SV* named_buff_FETCH(pTHX_ REGEXP * const rx, SV * const key,
290 const U32 flags);
291
292Called to get the value of key in the C<%+> and C<%-> hashes, C<key>
293is the hash key being requested and if C<flags & 1> is true C<%-> is
294being requested (and C<%+> if it's not).
108003db 295
49d7dfbc 296=head2 qr_package
108003db 297
49d7dfbc 298 SV* qr_package(pTHX_ REGEXP * const rx);
108003db
RGS
299
300The package the qr// magic object is blessed into (as seen by C<ref
49d7dfbc
AB
301qr//>). It is recommended that engines change this to their package
302name for identification regardless of whether they implement methods
303on the object.
304
305A callback implementation might be:
108003db
RGS
306
307 SV*
49d7dfbc 308 Example_reg_qr_package(pTHX_ REGEXP * const rx)
108003db
RGS
309 {
310 PERL_UNUSED_ARG(rx);
311 return newSVpvs("re::engine::Example");
312 }
313
314Any method calls on an object created with C<qr//> will be dispatched to the
315package as a normal object.
316
317 use re::engine::Example;
318 my $re = qr//;
319 $re->meth; # dispatched to re::engine::Example::meth()
320
321To retrieve the C<REGEXP> object from the scalar in an XS function use the
322following snippet:
323
324 void meth(SV * rv)
325 PPCODE:
326 MAGIC * mg;
327 REGEXP * re;
328
329 if (SvMAGICAL(sv))
330 mg_get(sv);
331 if (SvROK(sv) &&
332 (sv = (SV*)SvRV(sv)) && /* assignment deliberate */
333 SvTYPE(sv) == SVt_PVMG &&
334 (mg = mg_find(sv, PERL_MAGIC_qr))) /* assignment deliberate */
335 {
336 re = (REGEXP *)mg->mg_obj;
337 }
338
339Or use the (CURRENTLY UNDOCUMENETED!) C<Perl_get_re_arg> function:
340
341 void meth(SV * rv)
342 PPCODE:
343 const REGEXP * const re = (REGEXP *)Perl_get_re_arg( aTHX_ rv, 0, NULL );
344
345=head2 dupe
346
49d7dfbc 347 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
348
349On threaded builds a regexp may need to be duplicated so that the pattern
350can be used by mutiple threads. This routine is expected to handle the
351duplication of any private data pointed to by the C<pprivate> member of
352the regexp structure. It will be called with the preconstructed new
353regexp structure as an argument, the C<pprivate> member will point at
354the B<old> private structue, and it is this routine's responsibility to
355construct a copy and return a pointer to it (which perl will then use to
356overwrite the field as passed to this routine.)
357
358This allows the engine to dupe its private data but also if necessary
359modify the final structure if it really must.
360
361On unthreaded builds this field doesn't exist.
362
363=head1 The REGEXP structure
364
365The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
366correctly build such a structure in their L</comp> routine.
367
368The REGEXP structure contains all the data that perl needs to be aware of
369to properly work with the regular expression. It includes data about
370optimisations that perl can use to determine if the regex engine should
371really be used, and various other control info that is needed to properly
372execute patterns in various contexts such as is the pattern anchored in
373some way, or what flags were used during the compile, or whether the
374program contains special constructs that perl needs to be aware of.
375
882227b7
AB
376In addition it contains two fields that are intended for the private
377use of the regex engine that compiled the pattern. These are the
378C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to
379an arbitrary structure whose use and management is the responsibility
380of the compiling engine. perl will never modify either of these
381values.
108003db
RGS
382
383 typedef struct regexp {
384 /* what engine created this regexp? */
385 const struct regexp_engine* engine;
386
387 /* what re is this a lightweight copy of? */
388 struct regexp* mother_re;
389
390 /* Information about the match that the perl core uses to manage things */
391 U32 extflags; /* Flags used both externally and internally */
392 I32 minlen; /* mininum possible length of string to match */
393 I32 minlenret; /* mininum possible length of $& */
394 U32 gofs; /* chars left of pos that we search from */
395
396 /* substring data about strings that must appear
397 in the final match, used for optimisations */
398 struct reg_substr_data *substrs;
399
400 U32 nparens; /* number of capture buffers */
401
402 /* private engine specific data */
403 U32 intflags; /* Engine Specific Internal flags */
404 void *pprivate; /* Data private to the regex engine which
405 created this object. */
406
407 /* Data about the last/current match. These are modified during matching*/
408 U32 lastparen; /* last open paren matched */
409 U32 lastcloseparen; /* last close paren matched */
410 regexp_paren_pair *swap; /* Swap copy of *offs */
411 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
412
413 char *subbeg; /* saved or original string so \digit works forever. */
414 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
415 I32 sublen; /* Length of string pointed by subbeg */
416
417 /* Information about the match that isn't often used */
418 I32 prelen; /* length of precomp */
419 const char *precomp; /* pre-compilation regular expression */
420
421 /* wrapped can't be const char*, as it is returned by sv_2pv_flags */
422 char *wrapped; /* wrapped version of the pattern */
423 I32 wraplen; /* length of wrapped */
424
425 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
426 HV *paren_names; /* Optional hash of paren names */
427
428 /* Refcount of this regexp */
429 I32 refcnt; /* Refcount of this regexp */
430 } regexp;
431
432The fields are discussed in more detail below:
433
882227b7 434=head2 C<engine>
108003db
RGS
435
436This field points at a regexp_engine structure which contains pointers
437to the subroutines that are to be used for performing a match. It
438is the compiling routine's responsibility to populate this field before
439returning the regexp object.
440
441Internally this is set to C<NULL> unless a custom engine is specified in
442C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
443pointed to by C<RE_ENGINE_PTR>.
444
882227b7 445=head2 C<mother_re>
108003db
RGS
446
447TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
448
882227b7 449=head2 C<extflags>
108003db
RGS
450
451This will be used by perl to see what flags the regexp was compiled with, this
452will normally be set to the value of the flags parameter on L</comp>.
453
882227b7 454=head2 C<minlen> C<minlenret>
108003db
RGS
455
456The minimum string length required for the pattern to match. This is used to
457prune the search space by not bothering to match any closer to the end of a
458string than would allow a match. For instance there is no point in even
459starting the regex engine if the minlen is 10 but the string is only 5
460characters long. There is no way that the pattern can match.
461
462C<minlenret> is the minimum length of the string that would be found
463in $& after a match.
464
465The difference between C<minlen> and C<minlenret> can be seen in the
466following pattern:
467
468 /ns(?=\d)/
469
470where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
471required to match but is not actually included in the matched content. This
472distinction is particularly important as the substitution logic uses the
473C<minlenret> to tell whether it can do in-place substition which can result in
474considerable speedup.
475
882227b7 476=head2 C<gofs>
108003db
RGS
477
478Left offset from pos() to start match at.
479
882227b7 480=head2 C<substrs>
108003db
RGS
481
482TODO: document
483
882227b7 484=head2 C<nparens>, C<lasparen>, and C<lastcloseparen>
108003db
RGS
485
486These fields are used to keep track of how many paren groups could be matched
487in the pattern, which was the last open paren to be entered, and which was
488the last close paren to be entered.
489
882227b7 490=head2 C<intflags>
108003db
RGS
491
492The engine's private copy of the flags the pattern was compiled with. Usually
493this is the same as C<extflags> unless the engine chose to modify one of them
494
882227b7 495=head2 C<pprivate>
108003db
RGS
496
497A void* pointing to an engine-defined data structure. The perl engine uses the
498C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
499engine should use something else.
500
882227b7 501=head2 C<swap>
108003db
RGS
502
503TODO: document
504
882227b7 505=head2 C<offs>
108003db
RGS
506
507A C<regexp_paren_pair> structure which defines offsets into the string being
508matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
509C<regexp_paren_pair> struct is defined as follows:
510
511 typedef struct regexp_paren_pair {
512 I32 start;
513 I32 end;
514 } regexp_paren_pair;
515
516If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
517capture buffer did not match. C<< ->offs[0].start/end >> represents C<$&> (or
518C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
519C<$paren >= 1>.
520
882227b7 521=head2 C<precomp> C<prelen>
108003db
RGS
522
523Used for debugging purposes. C<precomp> holds a copy of the pattern
524that was compiled and C<prelen> its length.
525
882227b7 526=head2 C<paren_names>
108003db
RGS
527
528This is a hash used internally to track named capture buffers and their
529offsets. The keys are the names of the buffers the values are dualvars,
530with the IV slot holding the number of buffers with the given name and the
531pv being an embedded array of I32. The values may also be contained
532independently in the data array in cases where named backreferences are
533used.
534
882227b7 535=head2 C<reg_substr_data>
108003db
RGS
536
537Holds information on the longest string that must occur at a fixed
538offset from the start of the pattern, and the longest string that must
539occur at a floating offset from the start of the pattern. Used to do
540Fast-Boyer-Moore searches on the string to find out if its worth using
541the regex engine at all, and if so where in the string to search.
542
882227b7 543=head2 C<subbeg> C<sublen> C<saved_copy>
108003db
RGS
544
545 #define SAVEPVN(p,n) ((p) ? savepvn(p,n) : NULL)
546 if (RX_MATCH_COPIED(ret))
547 ret->subbeg = SAVEPVN(ret->subbeg, ret->sublen);
548 else
549 ret->subbeg = NULL;
550
551C<PL_sawampersand || rx->extflags & RXf_PMf_KEEPCOPY>
552
553These are used during execution phase for managing search and replace
554patterns.
555
882227b7 556=head2 C<wrapped> C<wraplen>
108003db
RGS
557
558Stores the string C<qr//> stringifies to, for example C<(?-xism:eek)>
559in the case of C<qr/eek/>.
560
561When using a custom engine that doesn't support the C<(?:)> construct for
562inline modifiers it's best to have C<qr//> stringify to the supplied pattern,
563note that this will create invalid patterns in cases such as:
564
565 my $x = qr/a|b/; # "a|b"
566 my $y = qr/c/; # "c"
567 my $z = qr/$x$y/; # "a|bc"
568
569There's no solution for such problems other than making the custom engine
570understand some for of inline modifiers.
571
572The C<Perl_reg_stringify> in F<regcomp.c> does the stringification work.
573
882227b7 574=head2 C<seen_evals>
108003db
RGS
575
576This stores the number of eval groups in the pattern. This is used for security
577purposes when embedding compiled regexes into larger patterns with C<qr//>.
578
882227b7 579=head2 C<refcnt>
108003db
RGS
580
581The number of times the structure is referenced. When this falls to 0 the
582regexp is automatically freed by a call to pregfree. This should be set to 1 in
583each engine's L</comp> routine.
584
108003db
RGS
585=head1 HISTORY
586
587Originally part of L<perlreguts>.
588
589=head1 AUTHORS
590
591Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
592Bjarmason.
593
594=head1 LICENSE
595
596Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
597
598This program is free software; you can redistribute it and/or modify it under
599the same terms as Perl itself.
600
601=cut