This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
VMS-specific tweak to FindBin for when $0 returns a unix-style filespec.
[perl5.git] / pod / perlreapi.pod
CommitLineData
108003db
RGS
1=head1 NAME
2
3perlreapi - perl regular expression plugin interface
4
5=head1 DESCRIPTION
6
7As of Perl 5.9.5 there is a new interface for using other regexp engines than
8the default one. Each engine is supposed to provide access to a constant
9structure of the following format:
10
11 typedef struct regexp_engine {
3ab4a224 12 REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags);
108003db
RGS
13 I32 (*exec) (pTHX_ regexp* prog, char* stringarg, char* strend,
14 char* strbeg, I32 minend, SV* screamer,
15 void* data, U32 flags);
16 char* (*intuit) (pTHX_ regexp *prog, SV *sv, char *strpos,
17 char *strend, U32 flags,
18 struct re_scream_pos_data_s *data);
19 SV* (*checkstr) (pTHX_ regexp *prog);
20 void (*free) (pTHX_ struct regexp* r);
21 SV* (*numbered_buff_get) (pTHX_ const REGEXP * const rx, I32 paren, SV* usesv);
22 SV* (*named_buff_get)(pTHX_ const REGEXP * const rx, SV* namesv, U32 flags);
23 SV* (*qr_pkg)(pTHX_ const REGEXP * const rx);
24 #ifdef USE_ITHREADS
25 void* (*dupe) (pTHX_ const regexp *r, CLONE_PARAMS *param);
26 #endif
27 } regexp_engine;
28
29When a regexp is compiled, its C<engine> field is then set to point at
30the appropriate structure so that when it needs to be used Perl can find
31the right routines to do so.
32
33In order to install a new regexp handler, C<$^H{regcomp}> is set
34to an integer which (when casted appropriately) resolves to one of these
35structures. When compiling, the C<comp> method is executed, and the
36resulting regexp structure's engine field is expected to point back at
37the same structure.
38
39The pTHX_ symbol in the definition is a macro used by perl under threading
40to provide an extra argument to the routine holding a pointer back to
41the interpreter that is executing the regexp. So under threading all
42routines get an extra argument.
43
44The routines are as follows:
45
46=head2 comp
47
3ab4a224 48 REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags);
108003db 49
3ab4a224
AB
50Compile the pattern stored in C<pattern> using the given C<flags> and
51return a pointer to a prepared C<REGEXP> structure that can perform
52the match. See L</The REGEXP structure> below for an explanation of
53the individual fields in the REGEXP struct.
54
55The C<pattern> parameter is the scalar that was used as the
56pattern. previous versions of perl would pass two C<char*> indicating
57the start and end of the stringifed pattern, the following snippet can
58be used to get the old parameters:
59
60 STRLEN plen;
61 char* exp = SvPV(pattern, plen);
62 char* xend = exp + plen;
63
64Since any scalar can be passed as a pattern it's possible to implement
65an engine that does something with an array (C<< "ook" =~ [ qw/ eek
66hlagh / ] >>) or with the non-stringified form of a compiled regular
67expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always
68stringify everything using the snippet above but that doesn't mean
69other engines have to.
108003db
RGS
70
71The C<flags> paramater is a bitfield which indicates which of the
72C<msixk> flags the regex was compiled with. In addition it contains
73info about whether C<use locale> is in effect and optimization info
74for C<split>. A regex engine might want to use the same split
75optimizations with a different syntax, for instance a Perl6 engine
76would treat C<split /^^/> equivalently to perl's C<split /^/>, see
77L<split documentation|perlfunc> and the relevant code in C<pp_split>
78in F<pp.c> to find out whether your engine should be setting these.
79
80The C<eogc> flags are stripped out before being passed to the comp
81routine. The regex engine does not need to know whether any of these
3ab4a224
AB
82are set as those flags should only affect what perl does with the
83pattern and its match variables, not how it gets compiled & executed.
108003db
RGS
84
85=over 4
86
87=item RXf_SKIPWHITE
88
89C<split ' '> or C<split> with no arguments (which really means
90C<split(' ', $_> see L<split|perlfunc>).
91
92=item RXf_START_ONLY
93
94Set if the pattern is C</^/> (C<<r->prelen == 1 && r->precomp[0] ==
95'^'>>). Will be used by the C<split> operator to split the given
96string on C<\n> (even under C</^/s>, see L<split|perlfunc>).
97
98=item RXf_WHITE
99
100Set if the pattern is exactly C</\s+/> and used by C<split>, the
101definition of whitespace varies depending on whether RXf_UTF8 or
102RXf_PMf_LOCALE is set.
103
104=item RXf_PMf_LOCALE
105
106Makes C<split> use the locale dependant definition of whitespace under C<use
107locale> when RXf_SKIPWHITE or RXf_WHITE is in effect. Under ASCII whitespace is
108defined as per L<isSPACE|perlapi/ISSPACE>, and by the internal macros
109C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use locale>.
110
111=item RXf_PMf_MULTILINE
112
113The C</m> flag, this ends up being passed to C<Perl_fbm_instr> by
114C<pp_split> regardless of the engine.
115
116=item RXf_PMf_SINGLELINE
117
118The C</s> flag. Guaranteed not to be used outside the regex engine.
119
120=item RXf_PMf_FOLD
121
122The C</i> flag. Guaranteed not to be used outside the regex engine.
123
124=item RXf_PMf_EXTENDED
125
126The C</x> flag. Guaranteed not to be used outside the regex
127engine. However if present on a regex C<#> comments will be stripped
128by the tokenizer regardless of the engine currently in use.
129
130=item RXf_PMf_KEEPCOPY
131
132The C</k> flag.
133
134=item RXf_UTF8
135
136Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime.
137
138=back
139
140In general these flags should be preserved in regex->extflags after
141compilation, although it is possible the regex includes constructs
142that changes them. The perl engine for instance may upgrade non-utf8
143strings to utf8 if the pattern includes constructs such as C<\x{...}>
144that can only match unicode values. RXf_SKIPWHITE should always be
145preserved verbatim in regex->extflags.
146
147=head2 exec
148
149 I32 exec(regexp* prog,
150 char *stringarg, char* strend, char* strbeg,
151 I32 minend, SV* screamer,
152 void* data, U32 flags);
153
154Execute a regexp.
155
156=head2 intuit
157
158 char* intuit( regexp *prog,
159 SV *sv, char *strpos, char *strend,
160 U32 flags, struct re_scream_pos_data_s *data);
161
162Find the start position where a regex match should be attempted,
163or possibly whether the regex engine should not be run because the
164pattern can't match. This is called as appropriate by the core
165depending on the values of the extflags member of the regexp
166structure.
167
168=head2 checkstr
169
170 SV* checkstr(regexp *prog);
171
172Return a SV containing a string that must appear in the pattern. Used
173by C<split> for optimising matches.
174
175=head2 free
176
177 void free(regexp *prog);
178
179Called by perl when it is freeing a regexp pattern so that the engine
180can release any resources pointed to by the C<pprivate> member of the
181regexp structure. This is only responsible for freeing private data;
182perl will handle releasing anything else contained in the regexp structure.
183
184=head2 numbered_buff_get
185
186 SV* numbered_buff_get(pTHX_ const REGEXP * const rx, I32 paren, SV* usesv);
187
188TODO: document
189
190=head2 named_buff_get
191
192 SV* named_buff_get(pTHX_ const REGEXP * const rx, SV* namesv, U32 flags);
193
194TODO: document
195
196=head2 qr_pkg
197
198 SV* qr_pkg(pTHX_ const REGEXP * const rx);
199
200The package the qr// magic object is blessed into (as seen by C<ref
201qr//>). It is recommended that engines change this to its package
202name, for instance:
203
204 SV*
205 Example_reg_qr_pkg(pTHX_ const REGEXP * const rx)
206 {
207 PERL_UNUSED_ARG(rx);
208 return newSVpvs("re::engine::Example");
209 }
210
211Any method calls on an object created with C<qr//> will be dispatched to the
212package as a normal object.
213
214 use re::engine::Example;
215 my $re = qr//;
216 $re->meth; # dispatched to re::engine::Example::meth()
217
218To retrieve the C<REGEXP> object from the scalar in an XS function use the
219following snippet:
220
221 void meth(SV * rv)
222 PPCODE:
223 MAGIC * mg;
224 REGEXP * re;
225
226 if (SvMAGICAL(sv))
227 mg_get(sv);
228 if (SvROK(sv) &&
229 (sv = (SV*)SvRV(sv)) && /* assignment deliberate */
230 SvTYPE(sv) == SVt_PVMG &&
231 (mg = mg_find(sv, PERL_MAGIC_qr))) /* assignment deliberate */
232 {
233 re = (REGEXP *)mg->mg_obj;
234 }
235
236Or use the (CURRENTLY UNDOCUMENETED!) C<Perl_get_re_arg> function:
237
238 void meth(SV * rv)
239 PPCODE:
240 const REGEXP * const re = (REGEXP *)Perl_get_re_arg( aTHX_ rv, 0, NULL );
241
242=head2 dupe
243
244 void* dupe(const regexp *r, CLONE_PARAMS *param);
245
246On threaded builds a regexp may need to be duplicated so that the pattern
247can be used by mutiple threads. This routine is expected to handle the
248duplication of any private data pointed to by the C<pprivate> member of
249the regexp structure. It will be called with the preconstructed new
250regexp structure as an argument, the C<pprivate> member will point at
251the B<old> private structue, and it is this routine's responsibility to
252construct a copy and return a pointer to it (which perl will then use to
253overwrite the field as passed to this routine.)
254
255This allows the engine to dupe its private data but also if necessary
256modify the final structure if it really must.
257
258On unthreaded builds this field doesn't exist.
259
260=head1 The REGEXP structure
261
262The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to
263correctly build such a structure in their L</comp> routine.
264
265The REGEXP structure contains all the data that perl needs to be aware of
266to properly work with the regular expression. It includes data about
267optimisations that perl can use to determine if the regex engine should
268really be used, and various other control info that is needed to properly
269execute patterns in various contexts such as is the pattern anchored in
270some way, or what flags were used during the compile, or whether the
271program contains special constructs that perl needs to be aware of.
272
273In addition it contains two fields that are intended for the private use
274of the regex engine that compiled the pattern. These are the C<intflags>
275and pprivate members. The C<pprivate> is a void pointer to an arbitrary
276structure whose use and management is the responsibility of the compiling
277engine. perl will never modify either of these values.
278
279 typedef struct regexp {
280 /* what engine created this regexp? */
281 const struct regexp_engine* engine;
282
283 /* what re is this a lightweight copy of? */
284 struct regexp* mother_re;
285
286 /* Information about the match that the perl core uses to manage things */
287 U32 extflags; /* Flags used both externally and internally */
288 I32 minlen; /* mininum possible length of string to match */
289 I32 minlenret; /* mininum possible length of $& */
290 U32 gofs; /* chars left of pos that we search from */
291
292 /* substring data about strings that must appear
293 in the final match, used for optimisations */
294 struct reg_substr_data *substrs;
295
296 U32 nparens; /* number of capture buffers */
297
298 /* private engine specific data */
299 U32 intflags; /* Engine Specific Internal flags */
300 void *pprivate; /* Data private to the regex engine which
301 created this object. */
302
303 /* Data about the last/current match. These are modified during matching*/
304 U32 lastparen; /* last open paren matched */
305 U32 lastcloseparen; /* last close paren matched */
306 regexp_paren_pair *swap; /* Swap copy of *offs */
307 regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */
308
309 char *subbeg; /* saved or original string so \digit works forever. */
310 SV_SAVED_COPY /* If non-NULL, SV which is COW from original */
311 I32 sublen; /* Length of string pointed by subbeg */
312
313 /* Information about the match that isn't often used */
314 I32 prelen; /* length of precomp */
315 const char *precomp; /* pre-compilation regular expression */
316
317 /* wrapped can't be const char*, as it is returned by sv_2pv_flags */
318 char *wrapped; /* wrapped version of the pattern */
319 I32 wraplen; /* length of wrapped */
320
321 I32 seen_evals; /* number of eval groups in the pattern - for security checks */
322 HV *paren_names; /* Optional hash of paren names */
323
324 /* Refcount of this regexp */
325 I32 refcnt; /* Refcount of this regexp */
326 } regexp;
327
328The fields are discussed in more detail below:
329
330=over 4
331
332=item C<engine>
333
334This field points at a regexp_engine structure which contains pointers
335to the subroutines that are to be used for performing a match. It
336is the compiling routine's responsibility to populate this field before
337returning the regexp object.
338
339Internally this is set to C<NULL> unless a custom engine is specified in
340C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct
341pointed to by C<RE_ENGINE_PTR>.
342
343=item C<mother_re>
344
345TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html>
346
347=item C<extflags>
348
349This will be used by perl to see what flags the regexp was compiled with, this
350will normally be set to the value of the flags parameter on L</comp>.
351
352=item C<minlen> C<minlenret>
353
354The minimum string length required for the pattern to match. This is used to
355prune the search space by not bothering to match any closer to the end of a
356string than would allow a match. For instance there is no point in even
357starting the regex engine if the minlen is 10 but the string is only 5
358characters long. There is no way that the pattern can match.
359
360C<minlenret> is the minimum length of the string that would be found
361in $& after a match.
362
363The difference between C<minlen> and C<minlenret> can be seen in the
364following pattern:
365
366 /ns(?=\d)/
367
368where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is
369required to match but is not actually included in the matched content. This
370distinction is particularly important as the substitution logic uses the
371C<minlenret> to tell whether it can do in-place substition which can result in
372considerable speedup.
373
374=item C<gofs>
375
376Left offset from pos() to start match at.
377
378=item C<substrs>
379
380TODO: document
381
382=item C<nparens>, C<lasparen>, and C<lastcloseparen>
383
384These fields are used to keep track of how many paren groups could be matched
385in the pattern, which was the last open paren to be entered, and which was
386the last close paren to be entered.
387
388=item C<intflags>
389
390The engine's private copy of the flags the pattern was compiled with. Usually
391this is the same as C<extflags> unless the engine chose to modify one of them
392
393=item C<pprivate>
394
395A void* pointing to an engine-defined data structure. The perl engine uses the
396C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom
397engine should use something else.
398
399=item C<swap>
400
401TODO: document
402
403=item C<offs>
404
405A C<regexp_paren_pair> structure which defines offsets into the string being
406matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the
407C<regexp_paren_pair> struct is defined as follows:
408
409 typedef struct regexp_paren_pair {
410 I32 start;
411 I32 end;
412 } regexp_paren_pair;
413
414If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that
415capture buffer did not match. C<< ->offs[0].start/end >> represents C<$&> (or
416C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where
417C<$paren >= 1>.
418
419=item C<precomp> C<prelen>
420
421Used for debugging purposes. C<precomp> holds a copy of the pattern
422that was compiled and C<prelen> its length.
423
424=item C<paren_names>
425
426This is a hash used internally to track named capture buffers and their
427offsets. The keys are the names of the buffers the values are dualvars,
428with the IV slot holding the number of buffers with the given name and the
429pv being an embedded array of I32. The values may also be contained
430independently in the data array in cases where named backreferences are
431used.
432
433=item C<reg_substr_data>
434
435Holds information on the longest string that must occur at a fixed
436offset from the start of the pattern, and the longest string that must
437occur at a floating offset from the start of the pattern. Used to do
438Fast-Boyer-Moore searches on the string to find out if its worth using
439the regex engine at all, and if so where in the string to search.
440
441=item C<startp>, C<endp>
442
443These fields store arrays that are used to hold the offsets of the begining
444and end of each capture group that has matched. -1 is used to indicate no match.
445
446These are the source for @- and @+.
447
448=item C<subbeg> C<sublen> C<saved_copy>
449
450 #define SAVEPVN(p,n) ((p) ? savepvn(p,n) : NULL)
451 if (RX_MATCH_COPIED(ret))
452 ret->subbeg = SAVEPVN(ret->subbeg, ret->sublen);
453 else
454 ret->subbeg = NULL;
455
456C<PL_sawampersand || rx->extflags & RXf_PMf_KEEPCOPY>
457
458These are used during execution phase for managing search and replace
459patterns.
460
461=item C<wrapped> C<wraplen>
462
463Stores the string C<qr//> stringifies to, for example C<(?-xism:eek)>
464in the case of C<qr/eek/>.
465
466When using a custom engine that doesn't support the C<(?:)> construct for
467inline modifiers it's best to have C<qr//> stringify to the supplied pattern,
468note that this will create invalid patterns in cases such as:
469
470 my $x = qr/a|b/; # "a|b"
471 my $y = qr/c/; # "c"
472 my $z = qr/$x$y/; # "a|bc"
473
474There's no solution for such problems other than making the custom engine
475understand some for of inline modifiers.
476
477The C<Perl_reg_stringify> in F<regcomp.c> does the stringification work.
478
479=item C<seen_evals>
480
481This stores the number of eval groups in the pattern. This is used for security
482purposes when embedding compiled regexes into larger patterns with C<qr//>.
483
484=item C<refcnt>
485
486The number of times the structure is referenced. When this falls to 0 the
487regexp is automatically freed by a call to pregfree. This should be set to 1 in
488each engine's L</comp> routine.
489
490=back
491
492=head2 De-allocation and Cloning
493
494Any patch that adds data items to the REGEXP struct will need to include
495changes to F<sv.c> (C<Perl_re_dup()>) and F<regcomp.c> (C<pregfree()>). This
496involves freeing or cloning items in the regexp's data array based on the data
497item's type.
498
499=head1 HISTORY
500
501Originally part of L<perlreguts>.
502
503=head1 AUTHORS
504
505Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth>
506Bjarmason.
507
508=head1 LICENSE
509
510Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason.
511
512This program is free software; you can redistribute it and/or modify it under
513the same terms as Perl itself.
514
515=cut