This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Move PL_delayedisa into a better spot in the interpreter structure.
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
3perlreapi - perl regular expression plugin interface
4
5=head1 DESCRIPTION
6
7As of Perl 5.9.5 there is a new interface for using other regexp engines than
8the default one. Each engine is supposed to provide access to a constant
9structure of the following format:
10
11 typedef struct regexp_engine {
3ab4a224 12 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
49d7dfbc
AB
13 I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend,
14 char* strbeg, I32 minend, SV* screamer,
15 void* data, U32 flags);
16 char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos,
17 char *strend, U32 flags,
18 struct re_scream_pos_data_s *data);
19 SV* (*checkstr) (pTHX_ REGEXP * const rx);
20 void (*free) (pTHX_ REGEXP * const rx);
21 void (*numbered_buff_get) (pTHX_ REGEXP * const rx,
22 const I32 paren, SV * const usesv);
23 SV* (*named_buff_get)(pTHX_ REGEXP * const rx, SV * const namesv,
24 const U32 flags);
25 SV* (*qr_package)(pTHX_ REGEXP * const rx);
108003db 26 #ifdef USE_ITHREADS
49d7dfbc 27 void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
28 #endif
29 } regexp_engine;
30
31When a regexp is compiled, its C<engine> field is then set to point at
32the appropriate structure so that when it needs to be used Perl can find
33the right routines to do so.
34
35In order to install a new regexp handler, C<$^H{regcomp}> is set
36to an integer which (when casted appropriately) resolves to one of these
37structures. When compiling, the C<comp> method is executed, and the
38resulting regexp structure's engine field is expected to point back at
39the same structure.
40
41The pTHX_ symbol in the definition is a macro used by perl under threading
42to provide an extra argument to the routine holding a pointer back to
43the interpreter that is executing the regexp. So under threading all
44routines get an extra argument.
45
46The routines are as follows:
47
48=head2 comp
49
3ab4a224 50 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 51
3ab4a224
AB
52Compile the pattern stored in C<pattern> using the given C<flags> and
53return a pointer to a prepared C<REGEXP> structure that can perform
54the match. See L</The REGEXP structure> below for an explanation of
55the individual fields in the REGEXP struct.
56
57The C<pattern> parameter is the scalar that was used as the
58pattern. previous versions of perl would pass two C<char*> indicating
59the start and end of the stringifed pattern, the following snippet can
60be used to get the old parameters:
61
62 STRLEN plen;
63 char* exp = SvPV(pattern, plen);
64 char* xend = exp + plen;
65
66Since any scalar can be passed as a pattern it's possible to implement
67an engine that does something with an array (C<< "ook" =~ [ qw/ eek
68hlagh / ] >>) or with the non-stringified form of a compiled regular
69expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
70stringify everything using the snippet above but that doesn't mean
71other engines have to.
108003db
RGS
72
73The C<flags> paramater is a bitfield which indicates which of the
74C<msixk> flags the regex was compiled with. In addition it contains
75info about whether C<use locale> is in effect and optimization info
76for C<split>. A regex engine might want to use the same split
77optimizations with a different syntax, for instance a Perl6 engine
78would treat C<split /^^/> equivalently to perl's C<split /^/>, see
79L<split documentation|perlfunc> and the relevant code in C<pp_split>
80in F<pp.c> to find out whether your engine should be setting these.
81
82The C<eogc> flags are stripped out before being passed to the comp
83routine. The regex engine does not need to know whether any of these
3ab4a224
AB
84are set as those flags should only affect what perl does with the
85pattern and its match variables, not how it gets compiled & executed.
108003db
RGS
86
87=over 4
88
89=item RXf_SKIPWHITE
90
91C<split ' '> or C<split> with no arguments (which really means
92C<split(' ', $_> see L<split|perlfunc>).
93
94=item RXf_START_ONLY
95
96Set if the pattern is C</^/> (C<<r->prelen == 1 && r->precomp[0] ==
97'^'>>). Will be used by the C<split> operator to split the given
98string on C<\n> (even under C</^/s>, see L<split|perlfunc>).
99
100=item RXf_WHITE
101
102Set if the pattern is exactly C</\s+/> and used by C<split>, the
103definition of whitespace varies depending on whether RXf_UTF8 or
104RXf_PMf_LOCALE is set.
105
106=item RXf_PMf_LOCALE
107
108Makes C<split> use the locale dependant definition of whitespace under C<use
109locale> when RXf_SKIPWHITE or RXf_WHITE is in effect. Under ASCII whitespace is
110defined as per L<isSPACE|perlapi/ISSPACE>, and by the internal macros
111C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use locale>.
112
113=item RXf_PMf_MULTILINE
114
115The C</m> flag, this ends up being passed to C<Perl_fbm_instr> by
116C<pp_split> regardless of the engine.
117
118=item RXf_PMf_SINGLELINE
119
120The C</s> flag. Guaranteed not to be used outside the regex engine.
121
122=item RXf_PMf_FOLD
123
124The C</i> flag. Guaranteed not to be used outside the regex engine.
125
126=item RXf_PMf_EXTENDED
127
128The C</x> flag. Guaranteed not to be used outside the regex
129engine. However if present on a regex C<#> comments will be stripped
130by the tokenizer regardless of the engine currently in use.
131
132=item RXf_PMf_KEEPCOPY
133
49d7dfbc 134The C</p> flag.
108003db
RGS
135
136=item RXf_UTF8
137
138Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime.
139
140=back
141
142In general these flags should be preserved in regex->extflags after
143compilation, although it is possible the regex includes constructs
144that changes them. The perl engine for instance may upgrade non-utf8
145strings to utf8 if the pattern includes constructs such as C<\x{...}>
146that can only match unicode values. RXf_SKIPWHITE should always be
147preserved verbatim in regex->extflags.
148
149=head2 exec
150
49d7dfbc 151 I32 exec(pTHX_ REGEXP * const rx,
108003db
RGS
152 char *stringarg, char* strend, char* strbeg,
153 I32 minend, SV* screamer,
154 void* data, U32 flags);
155
156Execute a regexp.
157
158=head2 intuit
159
49d7dfbc 160 char* intuit(pTHX_ REGEXP * const rx,
108003db 161 SV *sv, char *strpos, char *strend,
49d7dfbc 162 const U32 flags, struct re_scream_pos_data_s *data);
108003db
RGS
163
164Find the start position where a regex match should be attempted,
165or possibly whether the regex engine should not be run because the
166pattern can't match. This is called as appropriate by the core
167depending on the values of the extflags member of the regexp
168structure.
169
170=head2 checkstr
171
49d7dfbc 172 SV* checkstr(pTHX_ REGEXP * const rx);
108003db
RGS
173
174Return a SV containing a string that must appear in the pattern. Used
175by C<split> for optimising matches.
176
177=head2 free
178
49d7dfbc 179 void free(pTHX_ REGEXP * const rx);
108003db
RGS
180
181Called by perl when it is freeing a regexp pattern so that the engine
182can release any resources pointed to by the C<pprivate> member of the
183regexp structure. This is only responsible for freeing private data;
184perl will handle releasing anything else contained in the regexp structure.
185
186=head2 numbered_buff_get
187
49d7dfbc
AB
188 void numbered_buff_get(pTHX_ REGEXP * const rx, const I32 paren,
189 SV * const usesv);
108003db 190
49d7dfbc
AB
191Called to get the value of C<$`>, C<$'>, C<$&> (and their named
192equivalents, see L<perlvar>) and the numbered capture buffers (C<$1>,
193C<$2>, ...).
194
195The C<paren> paramater will be C<-2> for C<$`>, C<-1> for C<$'>, C<0>
196for C<$&>, C<1> for C<$1> and so forth.
197
198C<usesv> should be set to the scalar to return, the scalar is passed
199as an argument rather than being returned from the function because
200when it's called perl already has a scalar to store the value,
201creating another one would be redundant. The scalar can be set with
202C<sv_setsv>, C<sv_setpvn> and friends, see L<perlapi>.
203
204This callback is where perl untaints its own capture variables under
205taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_get>
206function in F<regcomp.c> for how to untaint capture variables if
207that's something you'd like your engine to do as well.
108003db
RGS
208
209=head2 named_buff_get
210
49d7dfbc
AB
211 SV* named_buff_get(pTHX_ REGEXP * const rx, SV * const namesv,
212 const U32 flags);
108003db 213
49d7dfbc
AB
214Called to get the value of key in the C<%+> and C<%-> hashes,
215C<namesv> is the hash key being requested and if C<flags & 1> is true
216C<%-> is being requested (and C<%+> if it's not).
108003db 217
49d7dfbc 218=head2 qr_package
108003db 219
49d7dfbc 220 SV* qr_package(pTHX_ REGEXP * const rx);
108003db
RGS
221
222The package the qr// magic object is blessed into (as seen by C<ref
49d7dfbc
AB
223qr//>). It is recommended that engines change this to their package
224name for identification regardless of whether they implement methods
225on the object.
226
227A callback implementation might be:
108003db
RGS
228
229 SV*
49d7dfbc 230 Example_reg_qr_package(pTHX_ REGEXP * const rx)
108003db
RGS
231 {
232 PERL_UNUSED_ARG(rx);
233 return newSVpvs("re::engine::Example");
234 }
235
236Any method calls on an object created with C<qr//> will be dispatched to the
237package as a normal object.
238
239 use re::engine::Example;
240 my $re = qr//;
241 $re->meth; # dispatched to re::engine::Example::meth()
242
243To retrieve the C<REGEXP> object from the scalar in an XS function use the
244following snippet:
245
246 void meth(SV * rv)
247 PPCODE:
248 MAGIC * mg;
249 REGEXP * re;
250
251 if (SvMAGICAL(sv))
252 mg_get(sv);
253 if (SvROK(sv) &&
254 (sv = (SV*)SvRV(sv)) && /* assignment deliberate */
255 SvTYPE(sv) == SVt_PVMG &&
256 (mg = mg_find(sv, PERL_MAGIC_qr))) /* assignment deliberate */
257 {
258 re = (REGEXP *)mg->mg_obj;
259 }
260
261Or use the (CURRENTLY UNDOCUMENETED!) C<Perl_get_re_arg> function:
262
263 void meth(SV * rv)
264 PPCODE:
265 const REGEXP * const re = (REGEXP *)Perl_get_re_arg( aTHX_ rv, 0, NULL );
266
267=head2 dupe
268
49d7dfbc 269 void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param);
108003db
RGS
270
271On threaded builds a regexp may need to be duplicated so that the pattern
272can be used by mutiple threads. This routine is expected to handle the
273duplication of any private data pointed to by the C<pprivate> member of
274the regexp structure. It will be called with the preconstructed new
275regexp structure as an argument, the C<pprivate> member will point at
276the B<old> private structue, and it is this routine's responsibility to
277construct a copy and return a pointer to it (which perl will then use to
278overwrite the field as passed to this routine.)
279
280This allows the engine to dupe its private data but also if necessary
281modify the final structure if it really must.
282
283On unthreaded builds this field doesn't exist.
284
285=head1 The REGEXP structure
286
287The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
288correctly build such a structure in their L</comp> routine.
289
290The REGEXP structure contains all the data that perl needs to be aware of
291to properly work with the regular expression. It includes data about
292optimisations that perl can use to determine if the regex engine should
293really be used, and various other control info that is needed to properly
294execute patterns in various contexts such as is the pattern anchored in
295some way, or what flags were used during the compile, or whether the
296program contains special constructs that perl needs to be aware of.
297
298In addition it contains two fields that are intended for the private use
299of the regex engine that compiled the pattern. These are the C<intflags>
300and pprivate members. The C<pprivate> is a void pointer to an arbitrary
301structure whose use and management is the responsibility of the compiling
302engine. perl will never modify either of these values.
303
304 typedef struct regexp {
305 /* what engine created this regexp? */
306 const struct regexp_engine* engine;
307
308 /* what re is this a lightweight copy of? */
309 struct regexp* mother_re;
310
311 /* Information about the match that the perl core uses to manage things */
312 U32 extflags; /* Flags used both externally and internally */
313 I32 minlen; /* mininum possible length of string to match */
314 I32 minlenret; /* mininum possible length of $& */
315 U32 gofs; /* chars left of pos that we search from */
316
317 /* substring data about strings that must appear
318 in the final match, used for optimisations */
319 struct reg_substr_data *substrs;
320
321 U32 nparens; /* number of capture buffers */
322
323 /* private engine specific data */
324 U32 intflags; /* Engine Specific Internal flags */
325 void *pprivate; /* Data private to the regex engine which
326 created this object. */
327
328 /* Data about the last/current match. These are modified during matching*/
329 U32 lastparen; /* last open paren matched */
330 U32 lastcloseparen; /* last close paren matched */
331 regexp_paren_pair *swap; /* Swap copy of *offs */
332 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
333
334 char *subbeg; /* saved or original string so \digit works forever. */
335 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
336 I32 sublen; /* Length of string pointed by subbeg */
337
338 /* Information about the match that isn't often used */
339 I32 prelen; /* length of precomp */
340 const char *precomp; /* pre-compilation regular expression */
341
342 /* wrapped can't be const char*, as it is returned by sv_2pv_flags */
343 char *wrapped; /* wrapped version of the pattern */
344 I32 wraplen; /* length of wrapped */
345
346 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
347 HV *paren_names; /* Optional hash of paren names */
348
349 /* Refcount of this regexp */
350 I32 refcnt; /* Refcount of this regexp */
351 } regexp;
352
353The fields are discussed in more detail below:
354
355=over 4
356
357=item C<engine>
358
359This field points at a regexp_engine structure which contains pointers
360to the subroutines that are to be used for performing a match. It
361is the compiling routine's responsibility to populate this field before
362returning the regexp object.
363
364Internally this is set to C<NULL> unless a custom engine is specified in
365C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
366pointed to by C<RE_ENGINE_PTR>.
367
368=item C<mother_re>
369
370TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
371
372=item C<extflags>
373
374This will be used by perl to see what flags the regexp was compiled with, this
375will normally be set to the value of the flags parameter on L</comp>.
376
377=item C<minlen> C<minlenret>
378
379The minimum string length required for the pattern to match. This is used to
380prune the search space by not bothering to match any closer to the end of a
381string than would allow a match. For instance there is no point in even
382starting the regex engine if the minlen is 10 but the string is only 5
383characters long. There is no way that the pattern can match.
384
385C<minlenret> is the minimum length of the string that would be found
386in $& after a match.
387
388The difference between C<minlen> and C<minlenret> can be seen in the
389following pattern:
390
391 /ns(?=\d)/
392
393where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
394required to match but is not actually included in the matched content. This
395distinction is particularly important as the substitution logic uses the
396C<minlenret> to tell whether it can do in-place substition which can result in
397considerable speedup.
398
399=item C<gofs>
400
401Left offset from pos() to start match at.
402
403=item C<substrs>
404
405TODO: document
406
407=item C<nparens>, C<lasparen>, and C<lastcloseparen>
408
409These fields are used to keep track of how many paren groups could be matched
410in the pattern, which was the last open paren to be entered, and which was
411the last close paren to be entered.
412
413=item C<intflags>
414
415The engine's private copy of the flags the pattern was compiled with. Usually
416this is the same as C<extflags> unless the engine chose to modify one of them
417
418=item C<pprivate>
419
420A void* pointing to an engine-defined data structure. The perl engine uses the
421C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
422engine should use something else.
423
424=item C<swap>
425
426TODO: document
427
428=item C<offs>
429
430A C<regexp_paren_pair> structure which defines offsets into the string being
431matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
432C<regexp_paren_pair> struct is defined as follows:
433
434 typedef struct regexp_paren_pair {
435 I32 start;
436 I32 end;
437 } regexp_paren_pair;
438
439If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
440capture buffer did not match. C<< ->offs[0].start/end >> represents C<$&> (or
441C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
442C<$paren >= 1>.
443
444=item C<precomp> C<prelen>
445
446Used for debugging purposes. C<precomp> holds a copy of the pattern
447that was compiled and C<prelen> its length.
448
449=item C<paren_names>
450
451This is a hash used internally to track named capture buffers and their
452offsets. The keys are the names of the buffers the values are dualvars,
453with the IV slot holding the number of buffers with the given name and the
454pv being an embedded array of I32. The values may also be contained
455independently in the data array in cases where named backreferences are
456used.
457
458=item C<reg_substr_data>
459
460Holds information on the longest string that must occur at a fixed
461offset from the start of the pattern, and the longest string that must
462occur at a floating offset from the start of the pattern. Used to do
463Fast-Boyer-Moore searches on the string to find out if its worth using
464the regex engine at all, and if so where in the string to search.
465
108003db
RGS
466=item C<subbeg> C<sublen> C<saved_copy>
467
468 #define SAVEPVN(p,n) ((p) ? savepvn(p,n) : NULL)
469 if (RX_MATCH_COPIED(ret))
470 ret->subbeg = SAVEPVN(ret->subbeg, ret->sublen);
471 else
472 ret->subbeg = NULL;
473
474C<PL_sawampersand || rx->extflags & RXf_PMf_KEEPCOPY>
475
476These are used during execution phase for managing search and replace
477patterns.
478
479=item C<wrapped> C<wraplen>
480
481Stores the string C<qr//> stringifies to, for example C<(?-xism:eek)>
482in the case of C<qr/eek/>.
483
484When using a custom engine that doesn't support the C<(?:)> construct for
485inline modifiers it's best to have C<qr//> stringify to the supplied pattern,
486note that this will create invalid patterns in cases such as:
487
488 my $x = qr/a|b/; # "a|b"
489 my $y = qr/c/; # "c"
490 my $z = qr/$x$y/; # "a|bc"
491
492There's no solution for such problems other than making the custom engine
493understand some for of inline modifiers.
494
495The C<Perl_reg_stringify> in F<regcomp.c> does the stringification work.
496
497=item C<seen_evals>
498
499This stores the number of eval groups in the pattern. This is used for security
500purposes when embedding compiled regexes into larger patterns with C<qr//>.
501
502=item C<refcnt>
503
504The number of times the structure is referenced. When this falls to 0 the
505regexp is automatically freed by a call to pregfree. This should be set to 1 in
506each engine's L</comp> routine.
507
508=back
509
510=head2 De-allocation and Cloning
511
512Any patch that adds data items to the REGEXP struct will need to include
513changes to F<sv.c> (C<Perl_re_dup()>) and F<regcomp.c> (C<pregfree()>). This
514involves freeing or cloning items in the regexp's data array based on the data
515item's type.
516
517=head1 HISTORY
518
519Originally part of L<perlreguts>.
520
521=head1 AUTHORS
522
523Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
524Bjarmason.
525
526=head1 LICENSE
527
528Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
529
530This program is free software; you can redistribute it and/or modify it under
531the same terms as Perl itself.
532
533=cut