Commit | Line | Data |
---|---|---|
108003db RGS |
1 | =head1 NAME |
2 | ||
3 | perlreapi - perl regular expression plugin interface | |
4 | ||
5 | =head1 DESCRIPTION | |
6 | ||
a0e97681 RGS |
7 | As of Perl 5.9.5 there is a new interface for plugging and using other |
8 | regular expression engines than the default one. | |
9 | ||
10 | Each engine is supposed to provide access to a constant structure of the | |
11 | following format: | |
108003db RGS |
12 | |
13 | typedef struct regexp_engine { | |
3ab4a224 | 14 | REGEXP* (*comp) (pTHX_ const SV * const pattern, const U32 flags); |
49d7dfbc | 15 | I32 (*exec) (pTHX_ REGEXP * const rx, char* stringarg, char* strend, |
2fdbfb4d AB |
16 | char* strbeg, I32 minend, SV* screamer, |
17 | void* data, U32 flags); | |
49d7dfbc | 18 | char* (*intuit) (pTHX_ REGEXP * const rx, SV *sv, char *strpos, |
2fdbfb4d AB |
19 | char *strend, U32 flags, |
20 | struct re_scream_pos_data_s *data); | |
49d7dfbc AB |
21 | SV* (*checkstr) (pTHX_ REGEXP * const rx); |
22 | void (*free) (pTHX_ REGEXP * const rx); | |
2fdbfb4d AB |
23 | void (*numbered_buff_FETCH) (pTHX_ REGEXP * const rx, const I32 paren, |
24 | SV * const sv); | |
25 | void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren, | |
26 | SV const * const value); | |
27 | I32 (*numbered_buff_LENGTH) (pTHX_ REGEXP * const rx, const SV * const sv, | |
28 | const I32 paren); | |
192b9cd1 AB |
29 | SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key, |
30 | SV * const value, U32 flags); | |
31 | SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey, | |
32 | const U32 flags); | |
49d7dfbc | 33 | SV* (*qr_package)(pTHX_ REGEXP * const rx); |
108003db | 34 | #ifdef USE_ITHREADS |
49d7dfbc | 35 | void* (*dupe) (pTHX_ REGEXP * const rx, CLONE_PARAMS *param); |
108003db | 36 | #endif |
108003db RGS |
37 | |
38 | When a regexp is compiled, its C<engine> field is then set to point at | |
a0e97681 | 39 | the appropriate structure, so that when it needs to be used Perl can find |
108003db RGS |
40 | the right routines to do so. |
41 | ||
42 | In order to install a new regexp handler, C<$^H{regcomp}> is set | |
43 | to an integer which (when casted appropriately) resolves to one of these | |
44 | structures. When compiling, the C<comp> method is executed, and the | |
45 | resulting regexp structure's engine field is expected to point back at | |
46 | the same structure. | |
47 | ||
48 | The pTHX_ symbol in the definition is a macro used by perl under threading | |
49 | to provide an extra argument to the routine holding a pointer back to | |
50 | the interpreter that is executing the regexp. So under threading all | |
51 | routines get an extra argument. | |
52 | ||
882227b7 | 53 | =head1 Callbacks |
108003db RGS |
54 | |
55 | =head2 comp | |
56 | ||
3ab4a224 | 57 | REGEXP* comp(pTHX_ const SV * const pattern, const U32 flags); |
108003db | 58 | |
3ab4a224 AB |
59 | Compile the pattern stored in C<pattern> using the given C<flags> and |
60 | return a pointer to a prepared C<REGEXP> structure that can perform | |
61 | the match. See L</The REGEXP structure> below for an explanation of | |
62 | the individual fields in the REGEXP struct. | |
63 | ||
64 | The C<pattern> parameter is the scalar that was used as the | |
65 | pattern. previous versions of perl would pass two C<char*> indicating | |
a0e97681 | 66 | the start and end of the stringified pattern, the following snippet can |
3ab4a224 AB |
67 | be used to get the old parameters: |
68 | ||
69 | STRLEN plen; | |
70 | char* exp = SvPV(pattern, plen); | |
71 | char* xend = exp + plen; | |
72 | ||
73 | Since any scalar can be passed as a pattern it's possible to implement | |
74 | an engine that does something with an array (C<< "ook" =~ [ qw/ eek | |
75 | hlagh / ] >>) or with the non-stringified form of a compiled regular | |
76 | expression (C<< "ook" =~ qr/eek/ >>). perl's own engine will always | |
77 | stringify everything using the snippet above but that doesn't mean | |
78 | other engines have to. | |
108003db | 79 | |
a0e97681 | 80 | The C<flags> parameter is a bitfield which indicates which of the |
c998b245 AB |
81 | C<msixp> flags the regex was compiled with. It also contains |
82 | additional info such as whether C<use locale> is in effect. | |
108003db RGS |
83 | |
84 | The C<eogc> flags are stripped out before being passed to the comp | |
85 | routine. The regex engine does not need to know whether any of these | |
3ab4a224 | 86 | are set as those flags should only affect what perl does with the |
c998b245 AB |
87 | pattern and its match variables, not how it gets compiled and |
88 | executed. | |
108003db | 89 | |
c998b245 AB |
90 | By the time the comp callback is called, some of these flags have |
91 | already had effect (noted below where applicable). However most of | |
92 | their effect occurs after the comp callback has run in routines that | |
93 | read the C<< rx->extflags >> field which it populates. | |
108003db | 94 | |
c998b245 AB |
95 | In general the flags should be preserved in C<< rx->extflags >> after |
96 | compilation, although the regex engine might want to add or delete | |
97 | some of them to invoke or disable some special behavior in perl. The | |
98 | flags along with any special behavior they cause are documented below: | |
108003db | 99 | |
c998b245 | 100 | The pattern modifiers: |
108003db | 101 | |
c998b245 | 102 | =over 4 |
108003db | 103 | |
c998b245 | 104 | =item C</m> - RXf_PMf_MULTILINE |
108003db | 105 | |
c998b245 AB |
106 | If this is in C<< rx->extflags >> it will be passed to |
107 | C<Perl_fbm_instr> by C<pp_split> which will treat the subject string | |
108 | as a multi-line string. | |
108003db | 109 | |
c998b245 | 110 | =item C</s> - RXf_PMf_SINGLELINE |
108003db | 111 | |
c998b245 | 112 | =item C</i> - RXf_PMf_FOLD |
108003db | 113 | |
c998b245 | 114 | =item C</x> - RXf_PMf_EXTENDED |
108003db | 115 | |
c998b245 AB |
116 | If present on a regex C<#> comments will be handled differently by the |
117 | tokenizer in some cases. | |
108003db | 118 | |
c998b245 | 119 | TODO: Document those cases. |
108003db | 120 | |
c998b245 | 121 | =item C</p> - RXf_PMf_KEEPCOPY |
108003db | 122 | |
c998b245 | 123 | =back |
108003db | 124 | |
c998b245 | 125 | Additional flags: |
108003db | 126 | |
c998b245 | 127 | =over 4 |
108003db | 128 | |
c998b245 | 129 | =item RXf_PMf_LOCALE |
108003db | 130 | |
c998b245 | 131 | Set if C<use locale> is in effect. If present in C<< rx->extflags >> |
a0e97681 | 132 | C<split> will use the locale dependent definition of whitespace under |
c998b245 AB |
133 | when RXf_SKIPWHITE or RXf_WHITE are in effect. Under ASCII whitespace |
134 | is defined as per L<isSPACE|perlapi/ISSPACE>, and by the internal | |
135 | macros C<is_utf8_space> under UTF-8 and C<isSPACE_LC> under C<use | |
136 | locale>. | |
108003db RGS |
137 | |
138 | =item RXf_UTF8 | |
139 | ||
140 | Set if the pattern is L<SvUTF8()|perlapi/SvUTF8>, set by Perl_pmruntime. | |
141 | ||
c998b245 AB |
142 | A regex engine may want to set or disable this flag during |
143 | compilation. The perl engine for instance may upgrade non-UTF-8 | |
144 | strings to UTF-8 if the pattern includes constructs such as C<\x{...}> | |
145 | that can only match Unicode values. | |
146 | ||
0ac6acae AB |
147 | =item RXf_SPLIT |
148 | ||
149 | If C<split> is invoked as C<split ' '> or with no arguments (which | |
5137fa37 | 150 | really means C<split(' ', $_)>, see L<split|perlfunc/split>), perl will |
0ac6acae AB |
151 | set this flag. The regex engine can then check for it and set the |
152 | SKIPWHITE and WHITE extflags. To do this the perl engine does: | |
153 | ||
154 | if (flags & RXf_SPLIT && r->prelen == 1 && r->precomp[0] == ' ') | |
155 | r->extflags |= (RXf_SKIPWHITE|RXf_WHITE); | |
156 | ||
108003db RGS |
157 | =back |
158 | ||
c998b245 AB |
159 | These flags can be set during compilation to enable optimizations in |
160 | the C<split> operator. | |
161 | ||
162 | =over 4 | |
163 | ||
0ac6acae AB |
164 | =item RXf_SKIPWHITE |
165 | ||
166 | If the flag is present in C<< rx->extflags >> C<split> will delete | |
167 | whitespace from the start of the subject string before it's operated | |
168 | on. What is considered whitespace depends on whether the subject is a | |
169 | UTF-8 string and whether the C<RXf_PMf_LOCALE> flag is set. | |
170 | ||
171 | If RXf_WHITE is set in addition to this flag C<split> will behave like | |
172 | C<split " "> under the perl engine. | |
173 | ||
c998b245 AB |
174 | =item RXf_START_ONLY |
175 | ||
176 | Tells the split operator to split the target string on newlines | |
177 | (C<\n>) without invoking the regex engine. | |
178 | ||
179 | Perl's engine sets this if the pattern is C</^/> (C<plen == 1 && *exp | |
180 | == '^'>), even under C</^/s>, see L<split|perlfunc>. Of course a | |
181 | different regex engine might want to use the same optimizations | |
182 | with a different syntax. | |
183 | ||
184 | =item RXf_WHITE | |
185 | ||
186 | Tells the split operator to split the target string on whitespace | |
187 | without invoking the regex engine. The definition of whitespace varies | |
188 | depending on whether the target string is a UTF-8 string and on | |
189 | whether RXf_PMf_LOCALE is set. | |
190 | ||
0ac6acae | 191 | Perl's engine sets this flag if the pattern is C<\s+>. |
c998b245 | 192 | |
640f820d AB |
193 | =item RXf_NULL |
194 | ||
a0e97681 | 195 | Tells the split operator to split the target string on |
640f820d AB |
196 | characters. The definition of character varies depending on whether |
197 | the target string is a UTF-8 string. | |
198 | ||
199 | Perl's engine sets this flag on empty patterns, this optimization | |
a0e97681 | 200 | makes C<split //> much faster than it would otherwise be. It's even |
640f820d AB |
201 | faster than C<unpack>. |
202 | ||
c998b245 | 203 | =back |
108003db RGS |
204 | |
205 | =head2 exec | |
206 | ||
49d7dfbc | 207 | I32 exec(pTHX_ REGEXP * const rx, |
108003db RGS |
208 | char *stringarg, char* strend, char* strbeg, |
209 | I32 minend, SV* screamer, | |
210 | void* data, U32 flags); | |
211 | ||
212 | Execute a regexp. | |
213 | ||
214 | =head2 intuit | |
215 | ||
49d7dfbc | 216 | char* intuit(pTHX_ REGEXP * const rx, |
108003db | 217 | SV *sv, char *strpos, char *strend, |
49d7dfbc | 218 | const U32 flags, struct re_scream_pos_data_s *data); |
108003db RGS |
219 | |
220 | Find the start position where a regex match should be attempted, | |
221 | or possibly whether the regex engine should not be run because the | |
222 | pattern can't match. This is called as appropriate by the core | |
223 | depending on the values of the extflags member of the regexp | |
224 | structure. | |
225 | ||
226 | =head2 checkstr | |
227 | ||
49d7dfbc | 228 | SV* checkstr(pTHX_ REGEXP * const rx); |
108003db RGS |
229 | |
230 | Return a SV containing a string that must appear in the pattern. Used | |
231 | by C<split> for optimising matches. | |
232 | ||
233 | =head2 free | |
234 | ||
49d7dfbc | 235 | void free(pTHX_ REGEXP * const rx); |
108003db RGS |
236 | |
237 | Called by perl when it is freeing a regexp pattern so that the engine | |
238 | can release any resources pointed to by the C<pprivate> member of the | |
239 | regexp structure. This is only responsible for freeing private data; | |
240 | perl will handle releasing anything else contained in the regexp structure. | |
241 | ||
192b9cd1 | 242 | =head2 Numbered capture callbacks |
108003db | 243 | |
192b9cd1 AB |
244 | Called to get/set the value of C<$`>, C<$'>, C<$&> and their named |
245 | equivalents, ${^PREMATCH}, ${^POSTMATCH} and $^{MATCH}, as well as the | |
246 | numbered capture buffers (C<$1>, C<$2>, ...). | |
49d7dfbc | 247 | |
a0e97681 | 248 | The C<paren> parameter will be C<-2> for C<$`>, C<-1> for C<$'>, C<0> |
49d7dfbc AB |
249 | for C<$&>, C<1> for C<$1> and so forth. |
250 | ||
192b9cd1 AB |
251 | The names have been chosen by analogy with L<Tie::Scalar> methods |
252 | names with an additional B<LENGTH> callback for efficiency. However | |
253 | named capture variables are currently not tied internally but | |
254 | implemented via magic. | |
255 | ||
256 | =head3 numbered_buff_FETCH | |
257 | ||
258 | void numbered_buff_FETCH(pTHX_ REGEXP * const rx, const I32 paren, | |
259 | SV * const sv); | |
260 | ||
261 | Fetch a specified numbered capture. C<sv> should be set to the scalar | |
262 | to return, the scalar is passed as an argument rather than being | |
263 | returned from the function because when it's called perl already has a | |
264 | scalar to store the value, creating another one would be | |
265 | redundant. The scalar can be set with C<sv_setsv>, C<sv_setpvn> and | |
266 | friends, see L<perlapi>. | |
49d7dfbc AB |
267 | |
268 | This callback is where perl untaints its own capture variables under | |
c998b245 | 269 | taint mode (see L<perlsec>). See the C<Perl_reg_numbered_buff_fetch> |
49d7dfbc AB |
270 | function in F<regcomp.c> for how to untaint capture variables if |
271 | that's something you'd like your engine to do as well. | |
108003db | 272 | |
192b9cd1 | 273 | =head3 numbered_buff_STORE |
108003db | 274 | |
2fdbfb4d AB |
275 | void (*numbered_buff_STORE) (pTHX_ REGEXP * const rx, const I32 paren, |
276 | SV const * const value); | |
108003db | 277 | |
192b9cd1 AB |
278 | Set the value of a numbered capture variable. C<value> is the scalar |
279 | that is to be used as the new value. It's up to the engine to make | |
280 | sure this is used as the new value (or reject it). | |
2fdbfb4d AB |
281 | |
282 | Example: | |
283 | ||
284 | if ("ook" =~ /(o*)/) { | |
285 | # `paren' will be `1' and `value' will be `ee' | |
286 | $1 =~ tr/o/e/; | |
287 | } | |
288 | ||
289 | Perl's own engine will croak on any attempt to modify the capture | |
a0e97681 | 290 | variables, to do this in another engine use the following callback |
2fdbfb4d AB |
291 | (copied from C<Perl_reg_numbered_buff_store>): |
292 | ||
293 | void | |
294 | Example_reg_numbered_buff_store(pTHX_ REGEXP * const rx, const I32 paren, | |
295 | SV const * const value) | |
296 | { | |
297 | PERL_UNUSED_ARG(rx); | |
298 | PERL_UNUSED_ARG(paren); | |
299 | PERL_UNUSED_ARG(value); | |
300 | ||
301 | if (!PL_localizing) | |
302 | Perl_croak(aTHX_ PL_no_modify); | |
303 | } | |
304 | ||
99d59c4d | 305 | Actually perl will not I<always> croak in a statement that looks |
2fdbfb4d AB |
306 | like it would modify a numbered capture variable. This is because the |
307 | STORE callback will not be called if perl can determine that it | |
308 | doesn't have to modify the value. This is exactly how tied variables | |
309 | behave in the same situation: | |
310 | ||
311 | package CaptureVar; | |
312 | use base 'Tie::Scalar'; | |
313 | ||
314 | sub TIESCALAR { bless [] } | |
315 | sub FETCH { undef } | |
316 | sub STORE { die "This doesn't get called" } | |
317 | ||
318 | package main; | |
319 | ||
320 | tie my $sv => "CatptureVar"; | |
321 | $sv =~ y/a/b/; | |
322 | ||
323 | Because C<$sv> is C<undef> when the C<y///> operator is applied to it | |
324 | the transliteration won't actually execute and the program won't | |
192b9cd1 AB |
325 | C<die>. This is different to how 5.8 and earlier versions behaved |
326 | since the capture variables were READONLY variables then, now they'll | |
327 | just die when assigned to in the default engine. | |
2fdbfb4d | 328 | |
192b9cd1 | 329 | =head3 numbered_buff_LENGTH |
2fdbfb4d AB |
330 | |
331 | I32 numbered_buff_LENGTH (pTHX_ REGEXP * const rx, const SV * const sv, | |
332 | const I32 paren); | |
333 | ||
334 | Get the C<length> of a capture variable. There's a special callback | |
335 | for this so that perl doesn't have to do a FETCH and run C<length> on | |
192b9cd1 | 336 | the result, since the length is (in perl's case) known from an offset |
0a3a8dc0 | 337 | stored in C<< rx->offs >> this is much more efficient: |
2fdbfb4d AB |
338 | |
339 | I32 s1 = rx->offs[paren].start; | |
340 | I32 s2 = rx->offs[paren].end; | |
341 | I32 len = t1 - s1; | |
342 | ||
343 | This is a little bit more complex in the case of UTF-8, see what | |
344 | C<Perl_reg_numbered_buff_length> does with | |
345 | L<is_utf8_string_loclen|perlapi/is_utf8_string_loclen>. | |
346 | ||
192b9cd1 AB |
347 | =head2 Named capture callbacks |
348 | ||
349 | Called to get/set the value of C<%+> and C<%-> as well as by some | |
350 | utility functions in L<re>. | |
351 | ||
352 | There are two callbacks, C<named_buff> is called in all the cases the | |
353 | FETCH, STORE, DELETE, CLEAR, EXISTS and SCALAR L<Tie::Hash> callbacks | |
354 | would be on changes to C<%+> and C<%-> and C<named_buff_iter> in the | |
355 | same cases as FIRSTKEY and NEXTKEY. | |
356 | ||
357 | The C<flags> parameter can be used to determine which of these | |
358 | operations the callbacks should respond to, the following flags are | |
359 | currently defined: | |
360 | ||
361 | Which L<Tie::Hash> operation is being performed from the Perl level on | |
362 | C<%+> or C<%+>, if any: | |
363 | ||
f1b875a0 YO |
364 | RXapif_FETCH |
365 | RXapif_STORE | |
366 | RXapif_DELETE | |
367 | RXapif_CLEAR | |
368 | RXapif_EXISTS | |
369 | RXapif_SCALAR | |
370 | RXapif_FIRSTKEY | |
371 | RXapif_NEXTKEY | |
192b9cd1 AB |
372 | |
373 | Whether C<%+> or C<%-> is being operated on, if any. | |
2fdbfb4d | 374 | |
f1b875a0 YO |
375 | RXapif_ONE /* %+ */ |
376 | RXapif_ALL /* %- */ | |
2fdbfb4d | 377 | |
192b9cd1 | 378 | Whether this is being called as C<re::regname>, C<re::regnames> or |
c998b245 | 379 | C<re::regnames_count>, if any. The first two will be combined with |
f1b875a0 | 380 | C<RXapif_ONE> or C<RXapif_ALL>. |
192b9cd1 | 381 | |
f1b875a0 YO |
382 | RXapif_REGNAME |
383 | RXapif_REGNAMES | |
384 | RXapif_REGNAMES_COUNT | |
192b9cd1 AB |
385 | |
386 | Internally C<%+> and C<%-> are implemented with a real tied interface | |
387 | via L<Tie::Hash::NamedCapture>. The methods in that package will call | |
388 | back into these functions. However the usage of | |
389 | L<Tie::Hash::NamedCapture> for this purpose might change in future | |
390 | releases. For instance this might be implemented by magic instead | |
391 | (would need an extension to mgvtbl). | |
392 | ||
393 | =head3 named_buff | |
394 | ||
395 | SV* (*named_buff) (pTHX_ REGEXP * const rx, SV * const key, | |
396 | SV * const value, U32 flags); | |
397 | ||
398 | =head3 named_buff_iter | |
399 | ||
400 | SV* (*named_buff_iter) (pTHX_ REGEXP * const rx, const SV * const lastkey, | |
401 | const U32 flags); | |
108003db | 402 | |
49d7dfbc | 403 | =head2 qr_package |
108003db | 404 | |
49d7dfbc | 405 | SV* qr_package(pTHX_ REGEXP * const rx); |
108003db RGS |
406 | |
407 | The package the qr// magic object is blessed into (as seen by C<ref | |
49d7dfbc AB |
408 | qr//>). It is recommended that engines change this to their package |
409 | name for identification regardless of whether they implement methods | |
410 | on the object. | |
411 | ||
192b9cd1 AB |
412 | The package this method returns should also have the internal |
413 | C<Regexp> package in its C<@ISA>. C<qr//->isa("Regexp")> should always | |
414 | be true regardless of what engine is being used. | |
415 | ||
416 | Example implementation might be: | |
108003db RGS |
417 | |
418 | SV* | |
192b9cd1 | 419 | Example_qr_package(pTHX_ REGEXP * const rx) |
108003db RGS |
420 | { |
421 | PERL_UNUSED_ARG(rx); | |
422 | return newSVpvs("re::engine::Example"); | |
423 | } | |
424 | ||
425 | Any method calls on an object created with C<qr//> will be dispatched to the | |
426 | package as a normal object. | |
427 | ||
428 | use re::engine::Example; | |
429 | my $re = qr//; | |
430 | $re->meth; # dispatched to re::engine::Example::meth() | |
431 | ||
f7e71195 AB |
432 | To retrieve the C<REGEXP> object from the scalar in an XS function use |
433 | the C<SvRX> macro, see L<"REGEXP Functions" in perlapi|perlapi/REGEXP | |
434 | Functions>. | |
108003db RGS |
435 | |
436 | void meth(SV * rv) | |
437 | PPCODE: | |
f7e71195 | 438 | REGEXP * re = SvRX(sv); |
108003db | 439 | |
108003db RGS |
440 | =head2 dupe |
441 | ||
49d7dfbc | 442 | void* dupe(pTHX_ REGEXP * const rx, CLONE_PARAMS *param); |
108003db RGS |
443 | |
444 | On threaded builds a regexp may need to be duplicated so that the pattern | |
a0e97681 | 445 | can be used by multiple threads. This routine is expected to handle the |
108003db RGS |
446 | duplication of any private data pointed to by the C<pprivate> member of |
447 | the regexp structure. It will be called with the preconstructed new | |
448 | regexp structure as an argument, the C<pprivate> member will point at | |
a0e97681 | 449 | the B<old> private structure, and it is this routine's responsibility to |
108003db RGS |
450 | construct a copy and return a pointer to it (which perl will then use to |
451 | overwrite the field as passed to this routine.) | |
452 | ||
453 | This allows the engine to dupe its private data but also if necessary | |
454 | modify the final structure if it really must. | |
455 | ||
456 | On unthreaded builds this field doesn't exist. | |
457 | ||
458 | =head1 The REGEXP structure | |
459 | ||
460 | The REGEXP struct is defined in F<regexp.h>. All regex engines must be able to | |
461 | correctly build such a structure in their L</comp> routine. | |
462 | ||
463 | The REGEXP structure contains all the data that perl needs to be aware of | |
464 | to properly work with the regular expression. It includes data about | |
465 | optimisations that perl can use to determine if the regex engine should | |
466 | really be used, and various other control info that is needed to properly | |
467 | execute patterns in various contexts such as is the pattern anchored in | |
468 | some way, or what flags were used during the compile, or whether the | |
469 | program contains special constructs that perl needs to be aware of. | |
470 | ||
882227b7 AB |
471 | In addition it contains two fields that are intended for the private |
472 | use of the regex engine that compiled the pattern. These are the | |
473 | C<intflags> and C<pprivate> members. C<pprivate> is a void pointer to | |
474 | an arbitrary structure whose use and management is the responsibility | |
475 | of the compiling engine. perl will never modify either of these | |
476 | values. | |
108003db RGS |
477 | |
478 | typedef struct regexp { | |
479 | /* what engine created this regexp? */ | |
480 | const struct regexp_engine* engine; | |
481 | ||
482 | /* what re is this a lightweight copy of? */ | |
483 | struct regexp* mother_re; | |
484 | ||
485 | /* Information about the match that the perl core uses to manage things */ | |
486 | U32 extflags; /* Flags used both externally and internally */ | |
487 | I32 minlen; /* mininum possible length of string to match */ | |
488 | I32 minlenret; /* mininum possible length of $& */ | |
489 | U32 gofs; /* chars left of pos that we search from */ | |
490 | ||
491 | /* substring data about strings that must appear | |
492 | in the final match, used for optimisations */ | |
493 | struct reg_substr_data *substrs; | |
494 | ||
495 | U32 nparens; /* number of capture buffers */ | |
496 | ||
497 | /* private engine specific data */ | |
498 | U32 intflags; /* Engine Specific Internal flags */ | |
499 | void *pprivate; /* Data private to the regex engine which | |
500 | created this object. */ | |
501 | ||
502 | /* Data about the last/current match. These are modified during matching*/ | |
503 | U32 lastparen; /* last open paren matched */ | |
504 | U32 lastcloseparen; /* last close paren matched */ | |
505 | regexp_paren_pair *swap; /* Swap copy of *offs */ | |
506 | regexp_paren_pair *offs; /* Array of offsets for (@-) and (@+) */ | |
507 | ||
508 | char *subbeg; /* saved or original string so \digit works forever. */ | |
509 | SV_SAVED_COPY /* If non-NULL, SV which is COW from original */ | |
510 | I32 sublen; /* Length of string pointed by subbeg */ | |
511 | ||
512 | /* Information about the match that isn't often used */ | |
513 | I32 prelen; /* length of precomp */ | |
514 | const char *precomp; /* pre-compilation regular expression */ | |
515 | ||
108003db RGS |
516 | char *wrapped; /* wrapped version of the pattern */ |
517 | I32 wraplen; /* length of wrapped */ | |
518 | ||
519 | I32 seen_evals; /* number of eval groups in the pattern - for security checks */ | |
520 | HV *paren_names; /* Optional hash of paren names */ | |
521 | ||
522 | /* Refcount of this regexp */ | |
523 | I32 refcnt; /* Refcount of this regexp */ | |
524 | } regexp; | |
525 | ||
526 | The fields are discussed in more detail below: | |
527 | ||
882227b7 | 528 | =head2 C<engine> |
108003db RGS |
529 | |
530 | This field points at a regexp_engine structure which contains pointers | |
531 | to the subroutines that are to be used for performing a match. It | |
532 | is the compiling routine's responsibility to populate this field before | |
533 | returning the regexp object. | |
534 | ||
535 | Internally this is set to C<NULL> unless a custom engine is specified in | |
536 | C<$^H{regcomp}>, perl's own set of callbacks can be accessed in the struct | |
537 | pointed to by C<RE_ENGINE_PTR>. | |
538 | ||
882227b7 | 539 | =head2 C<mother_re> |
108003db RGS |
540 | |
541 | TODO, see L<http://www.mail-archive.com/perl5-changes@perl.org/msg17328.html> | |
542 | ||
882227b7 | 543 | =head2 C<extflags> |
108003db | 544 | |
192b9cd1 AB |
545 | This will be used by perl to see what flags the regexp was compiled |
546 | with, this will normally be set to the value of the flags parameter by | |
c998b245 AB |
547 | the L<comp|/comp> callback. See the L<comp|/comp> documentation for |
548 | valid flags. | |
108003db | 549 | |
882227b7 | 550 | =head2 C<minlen> C<minlenret> |
108003db RGS |
551 | |
552 | The minimum string length required for the pattern to match. This is used to | |
553 | prune the search space by not bothering to match any closer to the end of a | |
554 | string than would allow a match. For instance there is no point in even | |
555 | starting the regex engine if the minlen is 10 but the string is only 5 | |
556 | characters long. There is no way that the pattern can match. | |
557 | ||
558 | C<minlenret> is the minimum length of the string that would be found | |
559 | in $& after a match. | |
560 | ||
561 | The difference between C<minlen> and C<minlenret> can be seen in the | |
562 | following pattern: | |
563 | ||
564 | /ns(?=\d)/ | |
565 | ||
566 | where the C<minlen> would be 3 but C<minlenret> would only be 2 as the \d is | |
567 | required to match but is not actually included in the matched content. This | |
568 | distinction is particularly important as the substitution logic uses the | |
a0e97681 | 569 | C<minlenret> to tell whether it can do in-place substitution which can result in |
108003db RGS |
570 | considerable speedup. |
571 | ||
882227b7 | 572 | =head2 C<gofs> |
108003db RGS |
573 | |
574 | Left offset from pos() to start match at. | |
575 | ||
882227b7 | 576 | =head2 C<substrs> |
108003db | 577 | |
192b9cd1 AB |
578 | Substring data about strings that must appear in the final match. This |
579 | is currently only used internally by perl's engine for but might be | |
c998b245 | 580 | used in the future for all engines for optimisations. |
108003db | 581 | |
882227b7 | 582 | =head2 C<nparens>, C<lasparen>, and C<lastcloseparen> |
108003db RGS |
583 | |
584 | These fields are used to keep track of how many paren groups could be matched | |
585 | in the pattern, which was the last open paren to be entered, and which was | |
586 | the last close paren to be entered. | |
587 | ||
882227b7 | 588 | =head2 C<intflags> |
108003db RGS |
589 | |
590 | The engine's private copy of the flags the pattern was compiled with. Usually | |
192b9cd1 | 591 | this is the same as C<extflags> unless the engine chose to modify one of them. |
108003db | 592 | |
882227b7 | 593 | =head2 C<pprivate> |
108003db RGS |
594 | |
595 | A void* pointing to an engine-defined data structure. The perl engine uses the | |
596 | C<regexp_internal> structure (see L<perlreguts/Base Structures>) but a custom | |
597 | engine should use something else. | |
598 | ||
882227b7 | 599 | =head2 C<swap> |
108003db RGS |
600 | |
601 | TODO: document | |
602 | ||
882227b7 | 603 | =head2 C<offs> |
108003db RGS |
604 | |
605 | A C<regexp_paren_pair> structure which defines offsets into the string being | |
606 | matched which correspond to the C<$&> and C<$1>, C<$2> etc. captures, the | |
607 | C<regexp_paren_pair> struct is defined as follows: | |
608 | ||
609 | typedef struct regexp_paren_pair { | |
610 | I32 start; | |
611 | I32 end; | |
612 | } regexp_paren_pair; | |
613 | ||
614 | If C<< ->offs[num].start >> or C<< ->offs[num].end >> is C<-1> then that | |
615 | capture buffer did not match. C<< ->offs[0].start/end >> represents C<$&> (or | |
616 | C<${^MATCH> under C<//p>) and C<< ->offs[paren].end >> matches C<$$paren> where | |
617 | C<$paren >= 1>. | |
618 | ||
882227b7 | 619 | =head2 C<precomp> C<prelen> |
108003db | 620 | |
192b9cd1 AB |
621 | Used for optimisations. C<precomp> holds a copy of the pattern that |
622 | was compiled and C<prelen> its length. When a new pattern is to be | |
623 | compiled (such as inside a loop) the internal C<regcomp> operator | |
624 | checks whether the last compiled C<REGEXP>'s C<precomp> and C<prelen> | |
625 | are equivalent to the new one, and if so uses the old pattern instead | |
626 | of compiling a new one. | |
627 | ||
628 | The relevant snippet from C<Perl_pp_regcomp>: | |
629 | ||
630 | if (!re || !re->precomp || re->prelen != (I32)len || | |
631 | memNE(re->precomp, t, len)) | |
632 | /* Compile a new pattern */ | |
108003db | 633 | |
882227b7 | 634 | =head2 C<paren_names> |
108003db RGS |
635 | |
636 | This is a hash used internally to track named capture buffers and their | |
637 | offsets. The keys are the names of the buffers the values are dualvars, | |
638 | with the IV slot holding the number of buffers with the given name and the | |
639 | pv being an embedded array of I32. The values may also be contained | |
640 | independently in the data array in cases where named backreferences are | |
641 | used. | |
642 | ||
c998b245 | 643 | =head2 C<substrs> |
108003db RGS |
644 | |
645 | Holds information on the longest string that must occur at a fixed | |
646 | offset from the start of the pattern, and the longest string that must | |
647 | occur at a floating offset from the start of the pattern. Used to do | |
648 | Fast-Boyer-Moore searches on the string to find out if its worth using | |
649 | the regex engine at all, and if so where in the string to search. | |
650 | ||
882227b7 | 651 | =head2 C<subbeg> C<sublen> C<saved_copy> |
108003db | 652 | |
c998b245 | 653 | Used during execution phase for managing search and replace patterns. |
108003db | 654 | |
882227b7 | 655 | =head2 C<wrapped> C<wraplen> |
108003db | 656 | |
c998b245 AB |
657 | Stores the string C<qr//> stringifies to. The perl engine for example |
658 | stores C<(?-xism:eek)> in the case of C<qr/eek/>. | |
108003db | 659 | |
c998b245 AB |
660 | When using a custom engine that doesn't support the C<(?:)> construct |
661 | for inline modifiers, it's probably best to have C<qr//> stringify to | |
662 | the supplied pattern, note that this will create undesired patterns in | |
663 | cases such as: | |
108003db RGS |
664 | |
665 | my $x = qr/a|b/; # "a|b" | |
192b9cd1 | 666 | my $y = qr/c/i; # "c" |
108003db RGS |
667 | my $z = qr/$x$y/; # "a|bc" |
668 | ||
192b9cd1 AB |
669 | There's no solution for this problem other than making the custom |
670 | engine understand a construct like C<(?:)>. | |
108003db | 671 | |
882227b7 | 672 | =head2 C<seen_evals> |
108003db RGS |
673 | |
674 | This stores the number of eval groups in the pattern. This is used for security | |
675 | purposes when embedding compiled regexes into larger patterns with C<qr//>. | |
676 | ||
882227b7 | 677 | =head2 C<refcnt> |
108003db RGS |
678 | |
679 | The number of times the structure is referenced. When this falls to 0 the | |
680 | regexp is automatically freed by a call to pregfree. This should be set to 1 in | |
681 | each engine's L</comp> routine. | |
682 | ||
108003db RGS |
683 | =head1 HISTORY |
684 | ||
685 | Originally part of L<perlreguts>. | |
686 | ||
687 | =head1 AUTHORS | |
688 | ||
689 | Originally written by Yves Orton, expanded by E<AElig>var ArnfjE<ouml>rE<eth> | |
690 | Bjarmason. | |
691 | ||
692 | =head1 LICENSE | |
693 | ||
694 | Copyright 2006 Yves Orton and 2007 E<AElig>var ArnfjE<ouml>rE<eth> Bjarmason. | |
695 | ||
696 | This program is free software; you can redistribute it and/or modify it under | |
697 | the same terms as Perl itself. | |
698 | ||
699 | =cut |