This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
[perl #82708] Fix unescaped chars in L<>.
[perl5.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
d74e8afc 2X<regular expression> X<regex> X<regexp>
a0d0e21e
LW
3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
5d458dd8 8This page describes the syntax of regular expressions in Perl.
91e0c79e 9
cc46d5f2 10If you haven't used regular expressions before, a quick-start
91e0c79e
MJD
11introduction is available in L<perlrequick>, and a longer tutorial
12introduction is available in L<perlretut>.
13
14For reference on how regular expressions are used in matching
15operations, plus various examples of the same, see discussions of
16C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
17Operators">.
cb1a09d0 18
0d017f4d
WL
19
20=head2 Modifiers
21
19799a22 22Matching operations can have various modifiers. Modifiers
5a964f20 23that relate to the interpretation of the regular expression inside
19799a22 24are listed below. Modifiers that alter the way a regular expression
5d458dd8 25is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 26L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 27
55497cff
PP
28=over 4
29
54310121 30=item m
d74e8afc 31X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
55497cff
PP
32
33Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 34the start or end of the string to matching the start or end of any
7f761169 35line anywhere within the string.
55497cff 36
54310121 37=item s
d74e8afc
ITB
38X</s> X<regex, single-line> X<regexp, single-line>
39X<regular expression, single-line>
55497cff
PP
40
41Treat string as single line. That is, change "." to match any character
19799a22 42whatsoever, even a newline, which normally it would not match.
55497cff 43
34d67d80 44Used together, as C</ms>, they let the "." match any character whatsoever,
fb55449c 45while still allowing "^" and "$" to match, respectively, just after
19799a22 46and just before newlines within the string.
7b8d334a 47
87e95b7f
YO
48=item i
49X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
50X<regular expression, case-insensitive>
51
52Do case-insensitive pattern matching.
53
54If C<use locale> is in effect, the case map is taken from the current
55locale. See L<perllocale>.
56
54310121 57=item x
d74e8afc 58X</x>
55497cff
PP
59
60Extend your pattern's legibility by permitting whitespace and comments.
61
87e95b7f
YO
62=item p
63X</p> X<regex, preserve> X<regexp, preserve>
64
632a1772 65Preserve the string matched such that ${^PREMATCH}, ${^MATCH}, and
87e95b7f
YO
66${^POSTMATCH} are available for use after matching.
67
e2e6bec7
DN
68=item g and c
69X</g> X</c>
70
71Global matching, and keep the Current position after failed matching.
72Unlike i, m, s and x, these two flags affect the way the regex is used
73rather than the regex itself. See
74L<perlretut/"Using regular expressions in Perl"> for further explanation
75of the g and c modifiers.
76
55497cff 77=back
a0d0e21e
LW
78
79These are usually written as "the C</x> modifier", even though the delimiter
14218588 80in question might not really be a slash. Any of these
a0d0e21e 81modifiers may also be embedded within the regular expression itself using
da392a17
KW
82the C<(?...)> construct. Also are new (in 5.14) character set semantics
83modifiers B<C<<"a">>, B<C<"d">>, B<C<"l">> and B<C<"u">>, which, in 5.14
84only, must be used embedded in the regular expression, and not after the
85trailing delimiter. All this is discussed below in
86L</Extended Patterns>.
87X</a> X</d> X</l> X</u>
a0d0e21e 88
4633a7c4 89The C</x> modifier itself needs a little more explanation. It tells
7b059540 90the regular expression parser to ignore most whitespace that is neither
55497cff 91backslashed nor within a character class. You can use this to break up
4633a7c4 92your regular expression into (slightly) more readable parts. The C<#>
54310121 93character is also treated as a metacharacter introducing a comment,
55497cff 94just as in ordinary Perl code. This also means that if you want real
14218588 95whitespace or C<#> characters in the pattern (outside a character
f9a3ff1a 96class, where they are unaffected by C</x>), then you'll either have to
7b059540
KW
97escape them (using backslashes or C<\Q...\E>) or encode them using octal,
98hex, or C<\N{}> escapes. Taken together, these features go a long way towards
8933a740
RGS
99making Perl's regular expressions more readable. Note that you have to
100be careful not to include the pattern delimiter in the comment--perl has
101no way of knowing you did not intend to close the pattern early. See
102the C-comment deletion code in L<perlop>. Also note that anything inside
7651b971
KW
103a C<\Q...\E> stays unaffected by C</x>. And note that C</x> doesn't affect
104whether space interpretation within a single multi-character construct. For
105example in C<\x{...}>, regardless of the C</x> modifier, there can be no
9bb1f947 106spaces. Same for a L<quantifier|/Quantifiers> such as C<{3}> or
f9e949fd
KW
107C<{5,}>. Similarly, C<(?:...)> can't have a space between the C<?> and C<:>,
108but can between the C<(> and C<?>. Within any delimiters for such a
109construct, allowed spaces are not affected by C</x>, and depend on the
110construct. For example, C<\x{...}> can't have spaces because hexadecimal
111numbers don't have spaces in them. But, Unicode properties can have spaces, so
112in C<\p{...}> there can be spaces that follow the Unicode rules, for which see
9bb1f947 113L<perluniprops/Properties accessible through \p{} and \P{}>.
d74e8afc 114X</x>
a0d0e21e
LW
115
116=head2 Regular Expressions
117
04838cea
RGS
118=head3 Metacharacters
119
384f06ae 120The patterns used in Perl pattern matching evolved from those supplied in
14218588 121the Version 8 regex routines. (The routines are derived
19799a22
GS
122(distantly) from Henry Spencer's freely redistributable reimplementation
123of the V8 routines.) See L<Version 8 Regular Expressions> for
124details.
a0d0e21e
LW
125
126In particular the following metacharacters have their standard I<egrep>-ish
127meanings:
d74e8afc
ITB
128X<metacharacter>
129X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
130
a0d0e21e 131
f793d64a
KW
132 \ Quote the next metacharacter
133 ^ Match the beginning of the line
134 . Match any character (except newline)
135 $ Match the end of the line (or before newline at the end)
136 | Alternation
137 () Grouping
138 [] Bracketed Character class
a0d0e21e 139
14218588
GS
140By default, the "^" character is guaranteed to match only the
141beginning of the string, the "$" character only the end (or before the
142newline at the end), and Perl does certain optimizations with the
a0d0e21e
LW
143assumption that the string contains only one line. Embedded newlines
144will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 145string as a multi-line buffer, such that the "^" will match after any
0d520e8e
YO
146newline within the string (except if the newline is the last character in
147the string), and "$" will match before any newline. At the
a0d0e21e
LW
148cost of a little more overhead, you can do this by using the /m modifier
149on the pattern match operator. (Older programs did this by setting C<$*>,
f02c194e 150but this practice has been removed in perl 5.9.)
d74e8afc 151X<^> X<$> X</m>
a0d0e21e 152
14218588 153To simplify multi-line substitutions, the "." character never matches a
55497cff 154newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 155the string is a single line--even if it isn't.
d74e8afc 156X<.> X</s>
a0d0e21e 157
04838cea
RGS
158=head3 Quantifiers
159
a0d0e21e 160The following standard quantifiers are recognized:
d74e8afc 161X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
a0d0e21e 162
f793d64a
KW
163 * Match 0 or more times
164 + Match 1 or more times
165 ? Match 1 or 0 times
166 {n} Match exactly n times
167 {n,} Match at least n times
168 {n,m} Match at least n but not more than m times
a0d0e21e
LW
169
170(If a curly bracket occurs in any other context, it is treated
b975c076 171as a regular character. In particular, the lower bound
527e91da
BB
172is not optional.) The "*" quantifier is equivalent to C<{0,}>, the "+"
173quantifier to C<{1,}>, and the "?" quantifier to C<{0,1}>. n and m are limited
d0b16107 174to non-negative integral values less than a preset limit defined when perl is built.
9c79236d
GS
175This is usually 32766 on the most common platforms. The actual limit can
176be seen in the error message generated by code such as this:
177
820475bd 178 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 179
54310121
PP
180By default, a quantified subpattern is "greedy", that is, it will match as
181many times as possible (given a particular starting location) while still
182allowing the rest of the pattern to match. If you want it to match the
183minimum number of times possible, follow the quantifier with a "?". Note
184that the meanings don't change, just the "greediness":
0d017f4d 185X<metacharacter> X<greedy> X<greediness>
d74e8afc 186X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{n,m}?>
a0d0e21e 187
f793d64a
KW
188 *? Match 0 or more times, not greedily
189 +? Match 1 or more times, not greedily
190 ?? Match 0 or 1 time, not greedily
191 {n}? Match exactly n times, not greedily
192 {n,}? Match at least n times, not greedily
193 {n,m}? Match at least n but not more than m times, not greedily
a0d0e21e 194
b9b4dddf
YO
195By default, when a quantified subpattern does not allow the rest of the
196overall pattern to match, Perl will backtrack. However, this behaviour is
0d017f4d 197sometimes undesirable. Thus Perl provides the "possessive" quantifier form
b9b4dddf
YO
198as well.
199
f793d64a
KW
200 *+ Match 0 or more times and give nothing back
201 ++ Match 1 or more times and give nothing back
202 ?+ Match 0 or 1 time and give nothing back
203 {n}+ Match exactly n times and give nothing back (redundant)
204 {n,}+ Match at least n times and give nothing back
205 {n,m}+ Match at least n but not more than m times and give nothing back
b9b4dddf
YO
206
207For instance,
208
209 'aaaa' =~ /a++a/
210
211will never match, as the C<a++> will gobble up all the C<a>'s in the
212string and won't leave any for the remaining part of the pattern. This
213feature can be extremely useful to give perl hints about where it
214shouldn't backtrack. For instance, the typical "match a double-quoted
215string" problem can be most efficiently performed when written as:
216
217 /"(?:[^"\\]++|\\.)*+"/
218
0d017f4d 219as we know that if the final quote does not match, backtracking will not
b9b4dddf
YO
220help. See the independent subexpression C<< (?>...) >> for more details;
221possessive quantifiers are just syntactic sugar for that construct. For
222instance the above example could also be written as follows:
223
224 /"(?>(?:(?>[^"\\]+)|\\.)*)"/
225
04838cea
RGS
226=head3 Escape sequences
227
5f05dabc 228Because patterns are processed as double quoted strings, the following
a0d0e21e
LW
229also work:
230
f793d64a
KW
231 \t tab (HT, TAB)
232 \n newline (LF, NL)
233 \r return (CR)
234 \f form feed (FF)
235 \a alarm (bell) (BEL)
236 \e escape (think troff) (ESC)
f793d64a 237 \cK control char (example: VT)
dc0d9c48 238 \x{}, \x00 character whose ordinal is the given hexadecimal number
fb121860 239 \N{name} named Unicode character or character sequence
f793d64a 240 \N{U+263D} Unicode character (example: FIRST QUARTER MOON)
f0a2b745 241 \o{}, \000 character whose ordinal is the given octal number
f793d64a
KW
242 \l lowercase next char (think vi)
243 \u uppercase next char (think vi)
244 \L lowercase till \E (think vi)
245 \U uppercase till \E (think vi)
246 \Q quote (disable) pattern metacharacters till \E
247 \E end either case modification or quoted section, think vi
a0d0e21e 248
9bb1f947 249Details are in L<perlop/Quote and Quote-like Operators>.
1d2dff63 250
e1d1eefb 251=head3 Character Classes and other Special Escapes
04838cea 252
a0d0e21e 253In addition, Perl defines the following:
d0b16107 254X<\g> X<\k> X<\K> X<backreference>
a0d0e21e 255
f793d64a
KW
256 Sequence Note Description
257 [...] [1] Match a character according to the rules of the
258 bracketed character class defined by the "...".
259 Example: [a-z] matches "a" or "b" or "c" ... or "z"
260 [[:...:]] [2] Match a character according to the rules of the POSIX
261 character class "..." within the outer bracketed
262 character class. Example: [[:upper:]] matches any
263 uppercase character.
d35dd6c6
KW
264 \w [3] Match a "word" character (alphanumeric plus "_", plus
265 other connector punctuation chars plus Unicode
266 marks
f793d64a
KW
267 \W [3] Match a non-"word" character
268 \s [3] Match a whitespace character
269 \S [3] Match a non-whitespace character
270 \d [3] Match a decimal digit character
271 \D [3] Match a non-digit character
272 \pP [3] Match P, named property. Use \p{Prop} for longer names
273 \PP [3] Match non-P
274 \X [4] Match Unicode "eXtended grapheme cluster"
275 \C Match a single C-language char (octet) even if that is
276 part of a larger UTF-8 character. Thus it breaks up
277 characters into their UTF-8 bytes, so you may end up
278 with malformed pieces of UTF-8. Unsupported in
279 lookbehind.
c27a5cfe 280 \1 [5] Backreference to a specific capture group or buffer.
f793d64a
KW
281 '1' may actually be any positive integer.
282 \g1 [5] Backreference to a specific or previous group,
283 \g{-1} [5] The number may be negative indicating a relative
c27a5cfe 284 previous group and may optionally be wrapped in
f793d64a
KW
285 curly brackets for safer parsing.
286 \g{name} [5] Named backreference
287 \k<name> [5] Named backreference
288 \K [6] Keep the stuff left of the \K, don't include it in $&
289 \N [7] Any character but \n (experimental). Not affected by
290 /s modifier
291 \v [3] Vertical whitespace
292 \V [3] Not vertical whitespace
293 \h [3] Horizontal whitespace
294 \H [3] Not horizontal whitespace
295 \R [4] Linebreak
e1d1eefb 296
9bb1f947
KW
297=over 4
298
299=item [1]
300
301See L<perlrecharclass/Bracketed Character Classes> for details.
df225385 302
9bb1f947 303=item [2]
b8c5462f 304
9bb1f947 305See L<perlrecharclass/POSIX Character Classes> for details.
b8c5462f 306
9bb1f947 307=item [3]
5496314a 308
9bb1f947 309See L<perlrecharclass/Backslash sequences> for details.
5496314a 310
9bb1f947 311=item [4]
5496314a 312
9bb1f947 313See L<perlrebackslash/Misc> for details.
d0b16107 314
9bb1f947 315=item [5]
b8c5462f 316
c27a5cfe 317See L</Capture groups> below for details.
93733859 318
9bb1f947 319=item [6]
b8c5462f 320
9bb1f947
KW
321See L</Extended Patterns> below for details.
322
323=item [7]
324
325Note that C<\N> has two meanings. When of the form C<\N{NAME}>, it matches the
fb121860
KW
326character or character sequence whose name is C<NAME>; and similarly
327when of the form C<\N{U+I<hex>}>, it matches the character whose Unicode
328code point is I<hex>. Otherwise it matches any character but C<\n>.
9bb1f947
KW
329
330=back
d0b16107 331
04838cea
RGS
332=head3 Assertions
333
a0d0e21e 334Perl defines the following zero-width assertions:
d74e8afc
ITB
335X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
336X<regexp, zero-width assertion>
337X<regular expression, zero-width assertion>
338X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
a0d0e21e 339
9bb1f947
KW
340 \b Match a word boundary
341 \B Match except at a word boundary
342 \A Match only at beginning of string
343 \Z Match only at end of string, or before newline at the end
344 \z Match only at end of string
345 \G Match only at pos() (e.g. at the end-of-match position
9da458fc 346 of prior m//g)
a0d0e21e 347
14218588 348A word boundary (C<\b>) is a spot between two characters
19799a22
GS
349that has a C<\w> on one side of it and a C<\W> on the other side
350of it (in either order), counting the imaginary characters off the
351beginning and end of the string as matching a C<\W>. (Within
352character classes C<\b> represents backspace rather than a word
353boundary, just as it normally does in any double-quoted string.)
354The C<\A> and C<\Z> are just like "^" and "$", except that they
355won't match multiple times when the C</m> modifier is used, while
356"^" and "$" will match at every internal line boundary. To match
357the actual end of the string and not ignore an optional trailing
358newline, use C<\z>.
d74e8afc 359X<\b> X<\A> X<\Z> X<\z> X</m>
19799a22
GS
360
361The C<\G> assertion can be used to chain global matches (using
362C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
363It is also useful when writing C<lex>-like scanners, when you have
364several patterns that you want to match against consequent substrings
365of your string, see the previous reference. The actual location
366where C<\G> will match can also be influenced by using C<pos()> as
58e23c8d
YO
367an lvalue: see L<perlfunc/pos>. Note that the rule for zero-length
368matches is modified somewhat, in that contents to the left of C<\G> is
369not counted when determining the length of the match. Thus the following
370will not match forever:
d74e8afc 371X<\G>
c47ff5f1 372
e761bb84
CO
373 my $string = 'ABC';
374 pos($string) = 1;
375 while ($string =~ /(.\G)/g) {
376 print $1;
377 }
58e23c8d
YO
378
379It will print 'A' and then terminate, as it considers the match to
380be zero-width, and thus will not match at the same position twice in a
381row.
382
383It is worth noting that C<\G> improperly used can result in an infinite
384loop. Take care when using patterns that include C<\G> in an alternation.
385
c27a5cfe 386=head3 Capture groups
04838cea 387
c27a5cfe
KW
388The bracketing construct C<( ... )> creates capture groups (also referred to as
389capture buffers). To refer to the current contents of a group later on, within
d8b950dc
KW
390the same pattern, use C<\g1> (or C<\g{1}>) for the first, C<\g2> (or C<\g{2}>)
391for the second, and so on.
392This is called a I<backreference>.
d74e8afc 393X<regex, capture buffer> X<regexp, capture buffer>
c27a5cfe 394X<regex, capture group> X<regexp, capture group>
d74e8afc 395X<regular expression, capture buffer> X<backreference>
c27a5cfe 396X<regular expression, capture group> X<backreference>
1f1031fe 397X<\g{1}> X<\g{-1}> X<\g{name}> X<relative backreference> X<named backreference>
d8b950dc
KW
398X<named capture buffer> X<regular expression, named capture buffer>
399X<named capture group> X<regular expression, named capture group>
400X<%+> X<$+{name}> X<< \k<name> >>
401There is no limit to the number of captured substrings that you may use.
402Groups are numbered with the leftmost open parenthesis being number 1, etc. If
403a group did not match, the associated backreference won't match either. (This
404can happen if the group is optional, or in a different branch of an
405alternation.)
406You can omit the C<"g">, and write C<"\1">, etc, but there are some issues with
407this form, described below.
408
409You can also refer to capture groups relatively, by using a negative number, so
410that C<\g-1> and C<\g{-1}> both refer to the immediately preceding capture
411group, and C<\g-2> and C<\g{-2}> both refer to the group before it. For
412example:
5624f11d
YO
413
414 /
c27a5cfe
KW
415 (Y) # group 1
416 ( # group 2
417 (X) # group 3
418 \g{-1} # backref to group 3
419 \g{-3} # backref to group 1
5624f11d
YO
420 )
421 /x
422
d8b950dc
KW
423would match the same as C</(Y) ( (X) \g3 \g1 )/x>. This allows you to
424interpolate regexes into larger regexes and not have to worry about the
425capture groups being renumbered.
426
427You can dispense with numbers altogether and create named capture groups.
428The notation is C<(?E<lt>I<name>E<gt>...)> to declare and C<\g{I<name>}> to
429reference. (To be compatible with .Net regular expressions, C<\g{I<name>}> may
430also be written as C<\k{I<name>}>, C<\kE<lt>I<name>E<gt>> or C<\k'I<name>'>.)
431I<name> must not begin with a number, nor contain hyphens.
432When different groups within the same pattern have the same name, any reference
433to that name assumes the leftmost defined group. Named groups count in
434absolute and relative numbering, and so can also be referred to by those
435numbers.
436(It's possible to do things with named capture groups that would otherwise
437require C<(??{})>.)
438
439Capture group contents are dynamically scoped and available to you outside the
440pattern until the end of the enclosing block or until the next successful
441match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
442You can refer to them by absolute number (using C<"$1"> instead of C<"\g1">,
443etc); or by name via the C<%+> hash, using C<"$+{I<name>}">.
444
445Braces are required in referring to named capture groups, but are optional for
446absolute or relative numbered ones. Braces are safer when creating a regex by
447concatenating smaller strings. For example if you have C<qr/$a$b/>, and C<$a>
448contained C<"\g1">, and C<$b> contained C<"37">, you would get C</\g137/> which
449is probably not what you intended.
450
451The C<\g> and C<\k> notations were introduced in Perl 5.10.0. Prior to that
452there were no named nor relative numbered capture groups. Absolute numbered
453groups were referred to using C<\1>, C<\2>, etc, and this notation is still
454accepted (and likely always will be). But it leads to some ambiguities if
455there are more than 9 capture groups, as C<\10> could mean either the tenth
456capture group, or the character whose ordinal in octal is 010 (a backspace in
457ASCII). Perl resolves this ambiguity by interpreting C<\10> as a backreference
458only if at least 10 left parentheses have opened before it. Likewise C<\11> is
459a backreference only if at least 11 left parentheses have opened before it.
e1f120a9
KW
460And so on. C<\1> through C<\9> are always interpreted as backreferences.
461There are several examples below that illustrate these perils. You can avoid
462the ambiguity by always using C<\g{}> or C<\g> if you mean capturing groups;
463and for octal constants always using C<\o{}>, or for C<\077> and below, using 3
464digits padded with leading zeros, since a leading zero implies an octal
465constant.
d8b950dc
KW
466
467The C<\I<digit>> notation also works in certain circumstances outside
468the pattern. See L</Warning on \1 Instead of $1> below for details.)
81714fb9 469
14218588 470Examples:
a0d0e21e
LW
471
472 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
473
d8b950dc 474 /(.)\g1/ # find first doubled char
81714fb9
YO
475 and print "'$1' is the first doubled character\n";
476
477 /(?<char>.)\k<char>/ # ... a different way
478 and print "'$+{char}' is the first doubled character\n";
479
d8b950dc 480 /(?'char'.)\g1/ # ... mix and match
81714fb9 481 and print "'$1' is the first doubled character\n";
c47ff5f1 482
14218588 483 if (/Time: (..):(..):(..)/) { # parse out values
f793d64a
KW
484 $hours = $1;
485 $minutes = $2;
486 $seconds = $3;
a0d0e21e 487 }
c47ff5f1 488
9d860678
KW
489 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\g10/ # \g10 is a backreference
490 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\10/ # \10 is octal
491 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\10/ # \10 is a backreference
492 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\010/ # \010 is octal
493
494 $a = '(.)\1'; # Creates problems when concatenated.
495 $b = '(.)\g{1}'; # Avoids the problems.
496 "aa" =~ /${a}/; # True
497 "aa" =~ /${b}/; # True
498 "aa0" =~ /${a}0/; # False!
499 "aa0" =~ /${b}0/; # True
dc0d9c48
KW
500 "aa\x08" =~ /${a}0/; # True!
501 "aa\x08" =~ /${b}0/; # False
9d860678 502
14218588
GS
503Several special variables also refer back to portions of the previous
504match. C<$+> returns whatever the last bracket match matched.
505C<$&> returns the entire matched string. (At one point C<$0> did
506also, but now it returns the name of the program.) C<$`> returns
77ea4f6d
JV
507everything before the matched string. C<$'> returns everything
508after the matched string. And C<$^N> contains whatever was matched by
509the most-recently closed group (submatch). C<$^N> can be used in
510extended patterns (see below), for example to assign a submatch to a
81714fb9 511variable.
d74e8afc 512X<$+> X<$^N> X<$&> X<$`> X<$'>
14218588 513
d8b950dc
KW
514These special variables, like the C<%+> hash and the numbered match variables
515(C<$1>, C<$2>, C<$3>, etc.) are dynamically scoped
14218588
GS
516until the end of the enclosing block or until the next successful
517match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
d74e8afc
ITB
518X<$+> X<$^N> X<$&> X<$`> X<$'>
519X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
520
0d017f4d 521B<NOTE>: Failed matches in Perl do not reset the match variables,
5146ce24 522which makes it easier to write code that tests for a series of more
665e98b9
JH
523specific cases and remembers the best match.
524
14218588
GS
525B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
526C<$'> anywhere in the program, it has to provide them for every
527pattern match. This may substantially slow your program. Perl
d8b950dc 528uses the same mechanism to produce C<$1>, C<$2>, etc, so you also pay a
14218588
GS
529price for each pattern that contains capturing parentheses. (To
530avoid this cost while retaining the grouping behaviour, use the
531extended regular expression C<(?: ... )> instead.) But if you never
532use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
533parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
534if you can, but if you can't (and some algorithms really appreciate
535them), once you've used them once, use them at will, because you've
536already paid the price. As of 5.005, C<$&> is not so costly as the
537other two.
d74e8afc 538X<$&> X<$`> X<$'>
68dc0745 539
99d59c4d 540As a workaround for this problem, Perl 5.10.0 introduces C<${^PREMATCH}>,
cde0cee5
YO
541C<${^MATCH}> and C<${^POSTMATCH}>, which are equivalent to C<$`>, C<$&>
542and C<$'>, B<except> that they are only guaranteed to be defined after a
87e95b7f 543successful match that was executed with the C</p> (preserve) modifier.
cde0cee5
YO
544The use of these variables incurs no global performance penalty, unlike
545their punctuation char equivalents, however at the trade-off that you
546have to tell perl when you want to use them.
87e95b7f 547X</p> X<p modifier>
cde0cee5 548
9d727203
KW
549=head2 Quoting metacharacters
550
19799a22
GS
551Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
552C<\w>, C<\n>. Unlike some other regular expression languages, there
553are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 554that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22
GS
555interpreted as a literal character, not a metacharacter. This was
556once used in a common idiom to disable or quote the special meanings
557of regular expression metacharacters in a string that you want to
36bbe248 558use for a pattern. Simply quote all non-"word" characters:
a0d0e21e
LW
559
560 $pattern =~ s/(\W)/\\$1/g;
561
f1cbbd6e 562(If C<use locale> is set, then this depends on the current locale.)
14218588
GS
563Today it is more common to use the quotemeta() function or the C<\Q>
564metaquoting escape sequence to disable all metacharacters' special
565meanings like this:
a0d0e21e
LW
566
567 /$unquoted\Q$quoted\E$unquoted/
568
9da458fc
IZ
569Beware that if you put literal backslashes (those not inside
570interpolated variables) between C<\Q> and C<\E>, double-quotish
571backslash interpolation may lead to confusing results. If you
572I<need> to use literal backslashes within C<\Q...\E>,
573consult L<perlop/"Gory details of parsing quoted constructs">.
574
19799a22
GS
575=head2 Extended Patterns
576
14218588
GS
577Perl also defines a consistent extension syntax for features not
578found in standard tools like B<awk> and B<lex>. The syntax is a
579pair of parentheses with a question mark as the first thing within
580the parentheses. The character after the question mark indicates
581the extension.
19799a22 582
14218588
GS
583The stability of these extensions varies widely. Some have been
584part of the core language for many years. Others are experimental
585and may change without warning or be completely removed. Check
586the documentation on an individual feature to verify its current
587status.
19799a22 588
14218588
GS
589A question mark was chosen for this and for the minimal-matching
590construct because 1) question marks are rare in older regular
591expressions, and 2) whenever you see one, you should stop and
592"question" exactly what is going on. That's psychology...
a0d0e21e
LW
593
594=over 10
595
cc6b7395 596=item C<(?#text)>
d74e8afc 597X<(?#)>
a0d0e21e 598
14218588 599A comment. The text is ignored. If the C</x> modifier enables
19799a22 600whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3
GS
601the comment as soon as it sees a C<)>, so there is no way to put a literal
602C<)> in the comment.
a0d0e21e 603
cfaf538b 604=item C<(?adlupimsx-imsx)>
fb85c044 605
cfaf538b 606=item C<(?^alupimsx)>
fb85c044 607X<(?)> X<(?^)>
19799a22 608
0b6d1084
JH
609One or more embedded pattern-match modifiers, to be turned on (or
610turned off, if preceded by C<->) for the remainder of the pattern or
fb85c044
KW
611the remainder of the enclosing pattern group (if any).
612
fb85c044 613This is particularly useful for dynamic patterns, such as those read in from a
0d017f4d
WL
614configuration file, taken from an argument, or specified in a table
615somewhere. Consider the case where some patterns want to be case
616sensitive and some do not: The case insensitive ones merely need to
617include C<(?i)> at the front of the pattern. For example:
19799a22
GS
618
619 $pattern = "foobar";
5d458dd8 620 if ( /$pattern/i ) { }
19799a22
GS
621
622 # more flexible:
623
624 $pattern = "(?i)foobar";
5d458dd8 625 if ( /$pattern/ ) { }
19799a22 626
0b6d1084 627These modifiers are restored at the end of the enclosing group. For example,
19799a22 628
d8b950dc 629 ( (?i) blah ) \s+ \g1
19799a22 630
0d017f4d
WL
631will match C<blah> in any case, some spaces, and an exact (I<including the case>!)
632repetition of the previous word, assuming the C</x> modifier, and no C</i>
633modifier outside this group.
19799a22 634
8eb5594e
DR
635These modifiers do not carry over into named subpatterns called in the
636enclosing group. In other words, a pattern such as C<((?i)(&NAME))> does not
637change the case-sensitivity of the "NAME" pattern.
638
dc925305
KW
639Any of these modifiers can be set to apply globally to all regular
640expressions compiled within the scope of a C<use re>. See
a0bbd6ff 641L<re/"'/flags' mode">.
dc925305 642
9de15fec
KW
643Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
644after the C<"?"> is a shorthand equivalent to C<d-imsx>. Flags (except
645C<"d">) may follow the caret to override it.
646But a minus sign is not legal with it.
647
cfaf538b
KW
648Also, starting in Perl 5.14, are modifiers C<"a">, C<"d">, C<"l">, and
649C<"u">, which for 5.14 may not be used as suffix modifiers.
9de15fec
KW
650
651C<"l"> means to use a locale (see L<perllocale>) when pattern matching.
652The locale used will be the one in effect at the time of execution of
653the pattern match. This may not be the same as the compilation-time
654locale, and can differ from one match to another if there is an
655intervening call of the
656L<setlocale() function|perllocale/The setlocale function>.
657This modifier is automatically set if the regular expression is compiled
20db7501
KW
658within the scope of a C<"use locale"> pragma. Results are not
659well-defined when using this and matching against a utf8-encoded string.
9de15fec 660
a12cf05f 661C<"u"> means to use Unicode semantics when pattern matching. It is
20db7501
KW
662automatically set if the regular expression is encoded in utf8, or is
663compiled within the scope of a
664L<C<"use feature 'unicode_strings">|feature> pragma (and isn't also in
665the scope of L<C<"use locale">|locale> nor L<C<"use bytes">|bytes>
666pragmas. On ASCII platforms, the code points between 128 and 255 take on their
667Latin-1 (ISO-8859-1) meanings (which are the same as Unicode's), whereas
668in strict ASCII their meanings are undefined. Thus the platform
669effectively becomes a Unicode platform. The ASCII characters remain as
670ASCII characters (since ASCII is a subset of Latin-1 and Unicode). For
dc925305 671example, when this option is not on, on a non-utf8 string, C<"\w">
20db7501
KW
672matches precisely C<[A-Za-z0-9_]>. When the option is on, it matches
673not just those, but all the Latin-1 word characters (such as an "n" with
674a tilde). On EBCDIC platforms, which already are equivalent to Latin-1,
675this modifier changes behavior only when the C<"/i"> modifier is also
676specified, and affects only two characters, giving them full Unicode
677semantics: the C<MICRO SIGN> will match the Greek capital and
678small letters C<MU>; otherwise not; and the C<LATIN CAPITAL LETTER SHARP
679S> will match any of C<SS>, C<Ss>, C<sS>, and C<ss>, otherwise not.
680(This last case is buggy, however.)
9de15fec 681
dc925305
KW
682C<"a"> is the same as C<"u">, except that C<\d>, C<\s>, C<\w>, and the
683Posix character classes are restricted to matching in the ASCII range
684only. That is, with this modifier, C<\d> always means precisely the
685digits C<"0"> to C<"9">; C<\s> means the five characters C<[ \f\n\r\t]>;
686C<\w> means the 63 characters C<[A-Za-z0-9_]>; and likewise, all the
cfaf538b
KW
687Posix classes such as C<[[:print:]]> match only the appropriate
688ASCII-range characters. As you would expect, this modifier causes, for
dc925305
KW
689example, C<\D> to mean the same thing as C<[^0-9]>; in fact, all
690non-ASCII characters match C<\D>, C<\S>, and C<\W>. C<\b> still means
691to match at the boundary between C<\w> and C<\W>, using the C<"a">
692definitions of them (similarly for C<\B>). Otherwise, C<"a"> behaves
693like the C<"u"> modifier, in that case-insensitive matching uses Unicode
694semantics; for example, "k" will match the Unicode C<\N{KELVIN SIGN}>
695under C</i> matching, and code points in the Latin1 range, above ASCII
696will have Unicode semantics when it comes to case-insensitive matching.
cfaf538b 697
9de15fec
KW
698C<"d"> means to use the traditional Perl pattern matching behavior.
699This is dualistic (hence the name C<"d">, which also could stand for
20db7501 700"depends"). When this is in effect, Perl matches utf8-encoded strings
9de15fec 701using Unicode rules, and matches non-utf8-encoded strings using the
20db7501
KW
702platform's native character set rules. (If the regular expression
703itself is encoded in utf8, Unicode rules are used regardless of the
704target string's encoding.)
9de15fec
KW
705See L<perlunicode/The "Unicode Bug">. It is automatically selected by
706default if the regular expression is compiled neither within the scope
707of a C<"use locale"> pragma nor a <C<"use feature 'unicode_strings">
708pragma.
709
dc925305
KW
710Note that the C<a>, C<d>, C<l>, C<p>, and C<u> modifiers are special in
711that they can only be enabled, not disabled, and the C<d>, C<l>, and
712C<u> modifiers are mutually exclusive: specifying one de-specifies the
bdc22dd5 713others, and a maximum of one may appear in the construct. Thus, for
315f11aa 714example, C<(?-p)>, C<(?-d:...)>, and C<(?dl:...)> will warn when
bdc22dd5 715compiled under C<use warnings>.
9de15fec
KW
716
717Note also that the C<p> modifier is special in that its presence
718anywhere in a pattern has a global effect.
cde0cee5 719
5a964f20 720=item C<(?:pattern)>
d74e8afc 721X<(?:)>
a0d0e21e 722
cfaf538b 723=item C<(?adluimsx-imsx:pattern)>
ca9dfc88 724
cfaf538b 725=item C<(?^aluimsx:pattern)>
fb85c044
KW
726X<(?^:)>
727
5a964f20
TC
728This is for clustering, not capturing; it groups subexpressions like
729"()", but doesn't make backreferences as "()" does. So
a0d0e21e 730
5a964f20 731 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e
LW
732
733is like
734
5a964f20 735 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 736
19799a22
GS
737but doesn't spit out extra fields. It's also cheaper not to capture
738characters if you don't need to.
a0d0e21e 739
19799a22 740Any letters between C<?> and C<:> act as flags modifiers as with
cfaf538b 741C<(?adluimsx-imsx)>. For example,
ca9dfc88
IZ
742
743 /(?s-i:more.*than).*million/i
744
14218588 745is equivalent to the more verbose
ca9dfc88
IZ
746
747 /(?:(?s-i)more.*than).*million/i
748
fb85c044 749Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
9de15fec
KW
750after the C<"?"> is a shorthand equivalent to C<d-imsx>. Any positive
751flags (except C<"d">) may follow the caret, so
fb85c044
KW
752
753 (?^x:foo)
754
755is equivalent to
756
757 (?x-ims:foo)
758
759The caret tells Perl that this cluster doesn't inherit the flags of any
9de15fec 760surrounding pattern, but to go back to the system defaults (C<d-imsx>),
fb85c044
KW
761modified by any flags specified.
762
763The caret allows for simpler stringification of compiled regular
764expressions. These look like
765
766 (?^:pattern)
767
768with any non-default flags appearing between the caret and the colon.
769A test that looks at such stringification thus doesn't need to have the
770system default flags hard-coded in it, just the caret. If new flags are
771added to Perl, the meaning of the caret's expansion will change to include
772the default for those flags, so the test will still work, unchanged.
773
774Specifying a negative flag after the caret is an error, as the flag is
775redundant.
776
777Mnemonic for C<(?^...)>: A fresh beginning since the usual use of a caret is
778to match at the beginning.
779
594d7033
YO
780=item C<(?|pattern)>
781X<(?|)> X<Branch reset>
782
783This is the "branch reset" pattern, which has the special property
c27a5cfe 784that the capture groups are numbered from the same starting point
99d59c4d 785in each alternation branch. It is available starting from perl 5.10.0.
4deaaa80 786
c27a5cfe 787Capture groups are numbered from left to right, but inside this
693596a8 788construct the numbering is restarted for each branch.
4deaaa80 789
c27a5cfe 790The numbering within each branch will be as normal, and any groups
4deaaa80
PJ
791following this construct will be numbered as though the construct
792contained only one branch, that being the one with the most capture
c27a5cfe 793groups in it.
4deaaa80
PJ
794
795This construct will be useful when you want to capture one of a
796number of alternative matches.
797
798Consider the following pattern. The numbers underneath show in
c27a5cfe 799which group the captured content will be stored.
594d7033
YO
800
801
802 # before ---------------branch-reset----------- after
803 / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
804 # 1 2 2 3 2 3 4
805
ab106183
A
806Be careful when using the branch reset pattern in combination with
807named captures. Named captures are implemented as being aliases to
c27a5cfe 808numbered groups holding the captures, and that interferes with the
ab106183
A
809implementation of the branch reset pattern. If you are using named
810captures in a branch reset pattern, it's best to use the same names,
811in the same order, in each of the alternations:
812
813 /(?| (?<a> x ) (?<b> y )
814 | (?<a> z ) (?<b> w )) /x
815
816Not doing so may lead to surprises:
817
818 "12" =~ /(?| (?<a> \d+ ) | (?<b> \D+))/x;
819 say $+ {a}; # Prints '12'
820 say $+ {b}; # *Also* prints '12'.
821
c27a5cfe
KW
822The problem here is that both the group named C<< a >> and the group
823named C<< b >> are aliases for the group belonging to C<< $1 >>.
90a18110 824
ee9b8eae
YO
825=item Look-Around Assertions
826X<look-around assertion> X<lookaround assertion> X<look-around> X<lookaround>
827
828Look-around assertions are zero width patterns which match a specific
829pattern without including it in C<$&>. Positive assertions match when
830their subpattern matches, negative assertions match when their subpattern
831fails. Look-behind matches text up to the current match position,
832look-ahead matches text following the current match position.
833
834=over 4
835
5a964f20 836=item C<(?=pattern)>
d74e8afc 837X<(?=)> X<look-ahead, positive> X<lookahead, positive>
a0d0e21e 838
19799a22 839A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e
LW
840matches a word followed by a tab, without including the tab in C<$&>.
841
5a964f20 842=item C<(?!pattern)>
d74e8afc 843X<(?!)> X<look-ahead, negative> X<lookahead, negative>
a0d0e21e 844
19799a22 845A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 846matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22
GS
847however that look-ahead and look-behind are NOT the same thing. You cannot
848use this for look-behind.
7b8d334a 849
5a964f20 850If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a
GS
851will not do what you want. That's because the C<(?!foo)> is just saying that
852the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
853match. You would have to do something like C</(?!foo)...bar/> for that. We
854say "like" because there's the case of your "bar" not having three characters
855before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
856Sometimes it's still easier just to say:
a0d0e21e 857
a3cb178b 858 if (/bar/ && $` !~ /foo$/)
a0d0e21e 859
19799a22 860For look-behind see below.
c277df42 861
ee9b8eae
YO
862=item C<(?<=pattern)> C<\K>
863X<(?<=)> X<look-behind, positive> X<lookbehind, positive> X<\K>
c277df42 864
c47ff5f1 865A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22
GS
866matches a word that follows a tab, without including the tab in C<$&>.
867Works only for fixed-width look-behind.
c277df42 868
ee9b8eae
YO
869There is a special form of this construct, called C<\K>, which causes the
870regex engine to "keep" everything it had matched prior to the C<\K> and
871not include it in C<$&>. This effectively provides variable length
872look-behind. The use of C<\K> inside of another look-around assertion
873is allowed, but the behaviour is currently not well defined.
874
c62285ac 875For various reasons C<\K> may be significantly more efficient than the
ee9b8eae
YO
876equivalent C<< (?<=...) >> construct, and it is especially useful in
877situations where you want to efficiently remove something following
878something else in a string. For instance
879
880 s/(foo)bar/$1/g;
881
882can be rewritten as the much more efficient
883
884 s/foo\Kbar//g;
885
5a964f20 886=item C<(?<!pattern)>
d74e8afc 887X<(?<!)> X<look-behind, negative> X<lookbehind, negative>
c277df42 888
19799a22
GS
889A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
890matches any occurrence of "foo" that does not follow "bar". Works
891only for fixed-width look-behind.
c277df42 892
ee9b8eae
YO
893=back
894
81714fb9
YO
895=item C<(?'NAME'pattern)>
896
897=item C<< (?<NAME>pattern) >>
898X<< (?<NAME>) >> X<(?'NAME')> X<named capture> X<capture>
899
c27a5cfe 900A named capture group. Identical in every respect to normal capturing
90a18110 901parentheses C<()> but for the additional fact that C<%+> or C<%-> may be
c27a5cfe 902used after a successful match to refer to a named group. See C<perlvar>
90a18110 903for more details on the C<%+> and C<%-> hashes.
81714fb9 904
c27a5cfe
KW
905If multiple distinct capture groups have the same name then the
906$+{NAME} will refer to the leftmost defined group in the match.
81714fb9 907
0d017f4d 908The forms C<(?'NAME'pattern)> and C<< (?<NAME>pattern) >> are equivalent.
81714fb9
YO
909
910B<NOTE:> While the notation of this construct is the same as the similar
c27a5cfe 911function in .NET regexes, the behavior is not. In Perl the groups are
81714fb9
YO
912numbered sequentially regardless of being named or not. Thus in the
913pattern
914
915 /(x)(?<foo>y)(z)/
916
917$+{foo} will be the same as $2, and $3 will contain 'z' instead of
918the opposite which is what a .NET regex hacker might expect.
919
1f1031fe
YO
920Currently NAME is restricted to simple identifiers only.
921In other words, it must match C</^[_A-Za-z][_A-Za-z0-9]*\z/> or
922its Unicode extension (see L<utf8>),
923though it isn't extended by the locale (see L<perllocale>).
81714fb9 924
1f1031fe 925B<NOTE:> In order to make things easier for programmers with experience
ae5648b3 926with the Python or PCRE regex engines, the pattern C<< (?PE<lt>NAMEE<gt>pattern) >>
0d017f4d 927may be used instead of C<< (?<NAME>pattern) >>; however this form does not
64c5a566 928support the use of single quotes as a delimiter for the name.
81714fb9 929
1f1031fe
YO
930=item C<< \k<NAME> >>
931
932=item C<< \k'NAME' >>
81714fb9
YO
933
934Named backreference. Similar to numeric backreferences, except that
935the group is designated by name and not number. If multiple groups
936have the same name then it refers to the leftmost defined group in
937the current match.
938
0d017f4d 939It is an error to refer to a name not defined by a C<< (?<NAME>) >>
81714fb9
YO
940earlier in the pattern.
941
942Both forms are equivalent.
943
1f1031fe 944B<NOTE:> In order to make things easier for programmers with experience
0d017f4d 945with the Python or PCRE regex engines, the pattern C<< (?P=NAME) >>
64c5a566 946may be used instead of C<< \k<NAME> >>.
1f1031fe 947
cc6b7395 948=item C<(?{ code })>
d74e8afc 949X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
c277df42 950
19799a22 951B<WARNING>: This extended regular expression feature is considered
b9b4dddf
YO
952experimental, and may be changed without notice. Code executed that
953has side effects may not perform identically from version to version
954due to the effect of future optimisations in the regex engine.
c277df42 955
cc46d5f2 956This zero-width assertion evaluates any embedded Perl code. It
19799a22
GS
957always succeeds, and its C<code> is not interpolated. Currently,
958the rules to determine where the C<code> ends are somewhat convoluted.
959
77ea4f6d
JV
960This feature can be used together with the special variable C<$^N> to
961capture the results of submatches in variables without having to keep
962track of the number of nested parentheses. For example:
963
964 $_ = "The brown fox jumps over the lazy dog";
965 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
966 print "color = $color, animal = $animal\n";
967
754091cb
RGS
968Inside the C<(?{...})> block, C<$_> refers to the string the regular
969expression is matching against. You can also use C<pos()> to know what is
fa11829f 970the current position of matching within this string.
754091cb 971
19799a22
GS
972The C<code> is properly scoped in the following sense: If the assertion
973is backtracked (compare L<"Backtracking">), all changes introduced after
974C<local>ization are undone, so that
b9ac3b5b
GS
975
976 $_ = 'a' x 8;
5d458dd8 977 m<
f793d64a 978 (?{ $cnt = 0 }) # Initialize $cnt.
b9ac3b5b 979 (
5d458dd8 980 a
b9ac3b5b 981 (?{
f793d64a 982 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
b9ac3b5b 983 })
5d458dd8 984 )*
b9ac3b5b 985 aaaa
f793d64a
KW
986 (?{ $res = $cnt }) # On success copy to
987 # non-localized location.
b9ac3b5b
GS
988 >x;
989
0d017f4d 990will set C<$res = 4>. Note that after the match, C<$cnt> returns to the globally
14218588 991introduced value, because the scopes that restrict C<local> operators
b9ac3b5b
GS
992are unwound.
993
19799a22
GS
994This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
995switch. If I<not> used in this way, the result of evaluation of
996C<code> is put into the special variable C<$^R>. This happens
997immediately, so C<$^R> can be used from other C<(?{ code })> assertions
998inside the same regular expression.
b9ac3b5b 999
19799a22
GS
1000The assignment to C<$^R> above is properly localized, so the old
1001value of C<$^R> is restored if the assertion is backtracked; compare
1002L<"Backtracking">.
b9ac3b5b 1003
19799a22
GS
1004For reasons of security, this construct is forbidden if the regular
1005expression involves run-time interpolation of variables, unless the
1006perilous C<use re 'eval'> pragma has been used (see L<re>), or the
1007variables contain results of C<qr//> operator (see
9bb1f947 1008L<perlop/"qr/STRINGE<sol>msixpo">).
871b0233 1009
0d017f4d 1010This restriction is due to the wide-spread and remarkably convenient
19799a22 1011custom of using run-time determined strings as patterns. For example:
871b0233
IZ
1012
1013 $re = <>;
1014 chomp $re;
1015 $string =~ /$re/;
1016
14218588
GS
1017Before Perl knew how to execute interpolated code within a pattern,
1018this operation was completely safe from a security point of view,
1019although it could raise an exception from an illegal pattern. If
1020you turn on the C<use re 'eval'>, though, it is no longer secure,
1021so you should only do so if you are also using taint checking.
1022Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 1023compartment. See L<perlsec> for details about both these mechanisms.
871b0233 1024
e95d7314
GG
1025B<WARNING>: Use of lexical (C<my>) variables in these blocks is
1026broken. The result is unpredictable and will make perl unstable. The
1027workaround is to use global (C<our>) variables.
1028
8525cfae
FC
1029B<WARNING>: In perl 5.12.x and earlier, the regex engine
1030was not re-entrant, so interpolated code could not
1031safely invoke the regex engine either directly with
e95d7314 1032C<m//> or C<s///>), or indirectly with functions such as
8525cfae 1033C<split>. Invoking the regex engine in these blocks would make perl
e95d7314 1034unstable.
8988a1bb 1035
14455d6c 1036=item C<(??{ code })>
d74e8afc
ITB
1037X<(??{})>
1038X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
0f5d15d6 1039
19799a22 1040B<WARNING>: This extended regular expression feature is considered
b9b4dddf
YO
1041experimental, and may be changed without notice. Code executed that
1042has side effects may not perform identically from version to version
1043due to the effect of future optimisations in the regex engine.
0f5d15d6 1044
19799a22
GS
1045This is a "postponed" regular subexpression. The C<code> is evaluated
1046at run time, at the moment this subexpression may match. The result
1047of evaluation is considered as a regular expression and matched as
61528107 1048if it were inserted instead of this construct. Note that this means
c27a5cfe 1049that the contents of capture groups defined inside an eval'ed pattern
6bda09f9 1050are not available outside of the pattern, and vice versa, there is no
c27a5cfe 1051way for the inner pattern to refer to a capture group defined outside.
6bda09f9
YO
1052Thus,
1053
1054 ('a' x 100)=~/(??{'(.)' x 100})/
1055
81714fb9 1056B<will> match, it will B<not> set $1.
0f5d15d6 1057
428594d9 1058The C<code> is not interpolated. As before, the rules to determine
19799a22
GS
1059where the C<code> ends are currently somewhat convoluted.
1060
1061The following pattern matches a parenthesized group:
0f5d15d6
IZ
1062
1063 $re = qr{
f793d64a
KW
1064 \(
1065 (?:
1066 (?> [^()]+ ) # Non-parens without backtracking
1067 |
1068 (??{ $re }) # Group with matching parens
1069 )*
1070 \)
1071 }x;
0f5d15d6 1072
6bda09f9
YO
1073See also C<(?PARNO)> for a different, more efficient way to accomplish
1074the same task.
1075
0b370c0a
A
1076For reasons of security, this construct is forbidden if the regular
1077expression involves run-time interpolation of variables, unless the
1078perilous C<use re 'eval'> pragma has been used (see L<re>), or the
1079variables contain results of C<qr//> operator (see
9bb1f947 1080L<perlop/"qrE<sol>STRINGE<sol>msixpo">).
0b370c0a 1081
8525cfae
FC
1082In perl 5.12.x and earlier, because the regex engine was not re-entrant,
1083delayed code could not safely invoke the regex engine either directly with
1084C<m//> or C<s///>), or indirectly with functions such as C<split>.
8988a1bb 1085
5d458dd8
YO
1086Recursing deeper than 50 times without consuming any input string will
1087result in a fatal error. The maximum depth is compiled into perl, so
6bda09f9
YO
1088changing it requires a custom build.
1089
542fa716
YO
1090=item C<(?PARNO)> C<(?-PARNO)> C<(?+PARNO)> C<(?R)> C<(?0)>
1091X<(?PARNO)> X<(?1)> X<(?R)> X<(?0)> X<(?-1)> X<(?+1)> X<(?-PARNO)> X<(?+PARNO)>
6bda09f9 1092X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
542fa716 1093X<regex, relative recursion>
6bda09f9 1094
81714fb9 1095Similar to C<(??{ code })> except it does not involve compiling any code,
c27a5cfe
KW
1096instead it treats the contents of a capture group as an independent
1097pattern that must match at the current position. Capture groups
81714fb9 1098contained by the pattern will have the value as determined by the
6bda09f9
YO
1099outermost recursion.
1100
894be9b7 1101PARNO is a sequence of digits (not starting with 0) whose value reflects
c27a5cfe 1102the paren-number of the capture group to recurse to. C<(?R)> recurses to
894be9b7 1103the beginning of the whole pattern. C<(?0)> is an alternate syntax for
542fa716 1104C<(?R)>. If PARNO is preceded by a plus or minus sign then it is assumed
c27a5cfe 1105to be relative, with negative numbers indicating preceding capture groups
542fa716 1106and positive ones following. Thus C<(?-1)> refers to the most recently
c27a5cfe 1107declared group, and C<(?+1)> indicates the next group to be declared.
c74340f9 1108Note that the counting for relative recursion differs from that of
c27a5cfe 1109relative backreferences, in that with recursion unclosed groups B<are>
c74340f9 1110included.
6bda09f9 1111
81714fb9 1112The following pattern matches a function foo() which may contain
f145b7e9 1113balanced parentheses as the argument.
6bda09f9
YO
1114
1115 $re = qr{ ( # paren group 1 (full function)
81714fb9 1116 foo
6bda09f9
YO
1117 ( # paren group 2 (parens)
1118 \(
1119 ( # paren group 3 (contents of parens)
1120 (?:
1121 (?> [^()]+ ) # Non-parens without backtracking
1122 |
1123 (?2) # Recurse to start of paren group 2
1124 )*
1125 )
1126 \)
1127 )
1128 )
1129 }x;
1130
1131If the pattern was used as follows
1132
1133 'foo(bar(baz)+baz(bop))'=~/$re/
1134 and print "\$1 = $1\n",
1135 "\$2 = $2\n",
1136 "\$3 = $3\n";
1137
1138the output produced should be the following:
1139
1140 $1 = foo(bar(baz)+baz(bop))
1141 $2 = (bar(baz)+baz(bop))
81714fb9 1142 $3 = bar(baz)+baz(bop)
6bda09f9 1143
c27a5cfe 1144If there is no corresponding capture group defined, then it is a
61528107 1145fatal error. Recursing deeper than 50 times without consuming any input
81714fb9 1146string will also result in a fatal error. The maximum depth is compiled
6bda09f9
YO
1147into perl, so changing it requires a custom build.
1148
542fa716
YO
1149The following shows how using negative indexing can make it
1150easier to embed recursive patterns inside of a C<qr//> construct
1151for later use:
1152
1153 my $parens = qr/(\((?:[^()]++|(?-1))*+\))/;
1154 if (/foo $parens \s+ + \s+ bar $parens/x) {
1155 # do something here...
1156 }
1157
81714fb9 1158B<Note> that this pattern does not behave the same way as the equivalent
0d017f4d 1159PCRE or Python construct of the same form. In Perl you can backtrack into
6bda09f9 1160a recursed group, in PCRE and Python the recursed into group is treated
542fa716
YO
1161as atomic. Also, modifiers are resolved at compile time, so constructs
1162like (?i:(?1)) or (?:(?i)(?1)) do not affect how the sub-pattern will
1163be processed.
6bda09f9 1164
894be9b7
YO
1165=item C<(?&NAME)>
1166X<(?&NAME)>
1167
0d017f4d
WL
1168Recurse to a named subpattern. Identical to C<(?PARNO)> except that the
1169parenthesis to recurse to is determined by name. If multiple parentheses have
894be9b7
YO
1170the same name, then it recurses to the leftmost.
1171
1172It is an error to refer to a name that is not declared somewhere in the
1173pattern.
1174
1f1031fe
YO
1175B<NOTE:> In order to make things easier for programmers with experience
1176with the Python or PCRE regex engines the pattern C<< (?P>NAME) >>
64c5a566 1177may be used instead of C<< (?&NAME) >>.
1f1031fe 1178
e2e6a0f1
YO
1179=item C<(?(condition)yes-pattern|no-pattern)>
1180X<(?()>
286f584a 1181
e2e6a0f1 1182=item C<(?(condition)yes-pattern)>
286f584a 1183
e2e6a0f1
YO
1184Conditional expression. C<(condition)> should be either an integer in
1185parentheses (which is valid if the corresponding pair of parentheses
1186matched), a look-ahead/look-behind/evaluate zero-width assertion, a
c27a5cfe 1187name in angle brackets or single quotes (which is valid if a group
e2e6a0f1
YO
1188with the given name matched), or the special symbol (R) (true when
1189evaluated inside of recursion or eval). Additionally the R may be
1190followed by a number, (which will be true when evaluated when recursing
1191inside of the appropriate group), or by C<&NAME>, in which case it will
1192be true only when evaluated during recursion in the named group.
1193
1194Here's a summary of the possible predicates:
1195
1196=over 4
1197
1198=item (1) (2) ...
1199
c27a5cfe 1200Checks if the numbered capturing group has matched something.
e2e6a0f1
YO
1201
1202=item (<NAME>) ('NAME')
1203
c27a5cfe 1204Checks if a group with the given name has matched something.
e2e6a0f1
YO
1205
1206=item (?{ CODE })
1207
1208Treats the code block as the condition.
1209
1210=item (R)
1211
1212Checks if the expression has been evaluated inside of recursion.
1213
1214=item (R1) (R2) ...
1215
1216Checks if the expression has been evaluated while executing directly
1217inside of the n-th capture group. This check is the regex equivalent of
1218
1219 if ((caller(0))[3] eq 'subname') { ... }
1220
1221In other words, it does not check the full recursion stack.
1222
1223=item (R&NAME)
1224
1225Similar to C<(R1)>, this predicate checks to see if we're executing
1226directly inside of the leftmost group with a given name (this is the same
1227logic used by C<(?&NAME)> to disambiguate). It does not check the full
1228stack, but only the name of the innermost active recursion.
1229
1230=item (DEFINE)
1231
1232In this case, the yes-pattern is never directly executed, and no
1233no-pattern is allowed. Similar in spirit to C<(?{0})> but more efficient.
1234See below for details.
1235
1236=back
1237
1238For example:
1239
1240 m{ ( \( )?
1241 [^()]+
1242 (?(1) \) )
1243 }x
1244
1245matches a chunk of non-parentheses, possibly included in parentheses
1246themselves.
1247
1248A special form is the C<(DEFINE)> predicate, which never executes directly
1249its yes-pattern, and does not allow a no-pattern. This allows to define
1250subpatterns which will be executed only by using the recursion mechanism.
1251This way, you can define a set of regular expression rules that can be
1252bundled into any pattern you choose.
1253
1254It is recommended that for this usage you put the DEFINE block at the
1255end of the pattern, and that you name any subpatterns defined within it.
1256
1257Also, it's worth noting that patterns defined this way probably will
1258not be as efficient, as the optimiser is not very clever about
1259handling them.
1260
1261An example of how this might be used is as follows:
1262
2bf803e2 1263 /(?<NAME>(?&NAME_PAT))(?<ADDR>(?&ADDRESS_PAT))
e2e6a0f1 1264 (?(DEFINE)
2bf803e2
YO
1265 (?<NAME_PAT>....)
1266 (?<ADRESS_PAT>....)
e2e6a0f1
YO
1267 )/x
1268
c27a5cfe
KW
1269Note that capture groups matched inside of recursion are not accessible
1270after the recursion returns, so the extra layer of capturing groups is
e2e6a0f1
YO
1271necessary. Thus C<$+{NAME_PAT}> would not be defined even though
1272C<$+{NAME}> would be.
286f584a 1273
c47ff5f1 1274=item C<< (?>pattern) >>
6bda09f9 1275X<backtrack> X<backtracking> X<atomic> X<possessive>
5a964f20 1276
19799a22
GS
1277An "independent" subexpression, one which matches the substring
1278that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 1279position, and it matches I<nothing other than this substring>. This
19799a22
GS
1280construct is useful for optimizations of what would otherwise be
1281"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc
IZ
1282It may also be useful in places where the "grab all you can, and do not
1283give anything back" semantic is desirable.
19799a22 1284
c47ff5f1 1285For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22
GS
1286(anchored at the beginning of string, as above) will match I<all>
1287characters C<a> at the beginning of string, leaving no C<a> for
1288C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
1289since the match of the subgroup C<a*> is influenced by the following
1290group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
1291C<a*ab> will match fewer characters than a standalone C<a*>, since
1292this makes the tail match.
1293
c47ff5f1 1294An effect similar to C<< (?>pattern) >> may be achieved by writing
d8b950dc
KW
1295C<(?=(pattern))\g1>. This matches the same substring as a standalone
1296C<a+>, and the following C<\g1> eats the matched string; it therefore
c47ff5f1 1297makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22
GS
1298(The difference between these two constructs is that the second one
1299uses a capturing group, thus shifting ordinals of backreferences
1300in the rest of a regular expression.)
1301
1302Consider this pattern:
c277df42 1303
871b0233 1304 m{ \(
e2e6a0f1 1305 (
f793d64a 1306 [^()]+ # x+
e2e6a0f1 1307 |
871b0233
IZ
1308 \( [^()]* \)
1309 )+
e2e6a0f1 1310 \)
871b0233 1311 }x
5a964f20 1312
19799a22
GS
1313That will efficiently match a nonempty group with matching parentheses
1314two levels deep or less. However, if there is no such group, it
1315will take virtually forever on a long string. That's because there
1316are so many different ways to split a long string into several
1317substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
1318to a subpattern of the above pattern. Consider how the pattern
1319above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
1320seconds, but that each extra letter doubles this time. This
1321exponential performance will make it appear that your program has
14218588 1322hung. However, a tiny change to this pattern
5a964f20 1323
e2e6a0f1
YO
1324 m{ \(
1325 (
f793d64a 1326 (?> [^()]+ ) # change x+ above to (?> x+ )
e2e6a0f1 1327 |
871b0233
IZ
1328 \( [^()]* \)
1329 )+
e2e6a0f1 1330 \)
871b0233 1331 }x
c277df42 1332
c47ff5f1 1333which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20
TC
1334this yourself would be a productive exercise), but finishes in a fourth
1335the time when used on a similar string with 1000000 C<a>s. Be aware,
1336however, that this pattern currently triggers a warning message under
9f1b1f2d 1337the C<use warnings> pragma or B<-w> switch saying it
6bab786b 1338C<"matches null string many times in regex">.
c277df42 1339
c47ff5f1 1340On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 1341effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42
IZ
1342This was only 4 times slower on a string with 1000000 C<a>s.
1343
9da458fc
IZ
1344The "grab all you can, and do not give anything back" semantic is desirable
1345in many situations where on the first sight a simple C<()*> looks like
1346the correct solution. Suppose we parse text with comments being delimited
1347by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 1348its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc
IZ
1349the comment delimiter, because it may "give up" some whitespace if
1350the remainder of the pattern can be made to match that way. The correct
1351answer is either one of these:
1352
1353 (?>#[ \t]*)
1354 #[ \t]*(?![ \t])
1355
1356For example, to grab non-empty comments into $1, one should use either
1357one of these:
1358
1359 / (?> \# [ \t]* ) ( .+ ) /x;
1360 / \# [ \t]* ( [^ \t] .* ) /x;
1361
1362Which one you pick depends on which of these expressions better reflects
1363the above specification of comments.
1364
6bda09f9
YO
1365In some literature this construct is called "atomic matching" or
1366"possessive matching".
1367
b9b4dddf
YO
1368Possessive quantifiers are equivalent to putting the item they are applied
1369to inside of one of these constructs. The following equivalences apply:
1370
1371 Quantifier Form Bracketing Form
1372 --------------- ---------------
1373 PAT*+ (?>PAT*)
1374 PAT++ (?>PAT+)
1375 PAT?+ (?>PAT?)
1376 PAT{min,max}+ (?>PAT{min,max})
1377
e2e6a0f1
YO
1378=back
1379
1380=head2 Special Backtracking Control Verbs
1381
1382B<WARNING:> These patterns are experimental and subject to change or
0d017f4d 1383removal in a future version of Perl. Their usage in production code should
e2e6a0f1
YO
1384be noted to avoid problems during upgrades.
1385
1386These special patterns are generally of the form C<(*VERB:ARG)>. Unless
1387otherwise stated the ARG argument is optional; in some cases, it is
1388forbidden.
1389
1390Any pattern containing a special backtracking verb that allows an argument
e1020413 1391has the special behaviour that when executed it sets the current package's
5d458dd8
YO
1392C<$REGERROR> and C<$REGMARK> variables. When doing so the following
1393rules apply:
e2e6a0f1 1394
5d458dd8
YO
1395On failure, the C<$REGERROR> variable will be set to the ARG value of the
1396verb pattern, if the verb was involved in the failure of the match. If the
1397ARG part of the pattern was omitted, then C<$REGERROR> will be set to the
1398name of the last C<(*MARK:NAME)> pattern executed, or to TRUE if there was
1399none. Also, the C<$REGMARK> variable will be set to FALSE.
e2e6a0f1 1400
5d458dd8
YO
1401On a successful match, the C<$REGERROR> variable will be set to FALSE, and
1402the C<$REGMARK> variable will be set to the name of the last
1403C<(*MARK:NAME)> pattern executed. See the explanation for the
1404C<(*MARK:NAME)> verb below for more details.
e2e6a0f1 1405
5d458dd8
YO
1406B<NOTE:> C<$REGERROR> and C<$REGMARK> are not magic variables like C<$1>
1407and most other regex related variables. They are not local to a scope, nor
1408readonly, but instead are volatile package variables similar to C<$AUTOLOAD>.
1409Use C<local> to localize changes to them to a specific scope if necessary.
e2e6a0f1
YO
1410
1411If a pattern does not contain a special backtracking verb that allows an
5d458dd8 1412argument, then C<$REGERROR> and C<$REGMARK> are not touched at all.
e2e6a0f1
YO
1413
1414=over 4
1415
1416=item Verbs that take an argument
1417
1418=over 4
1419
5d458dd8 1420=item C<(*PRUNE)> C<(*PRUNE:NAME)>
f7819f85 1421X<(*PRUNE)> X<(*PRUNE:NAME)>
54612592 1422
5d458dd8
YO
1423This zero-width pattern prunes the backtracking tree at the current point
1424when backtracked into on failure. Consider the pattern C<A (*PRUNE) B>,
1425where A and B are complex patterns. Until the C<(*PRUNE)> verb is reached,
1426A may backtrack as necessary to match. Once it is reached, matching
1427continues in B, which may also backtrack as necessary; however, should B
1428not match, then no further backtracking will take place, and the pattern
1429will fail outright at the current starting position.
54612592
YO
1430
1431The following example counts all the possible matching strings in a
1432pattern (without actually matching any of them).
1433
e2e6a0f1 1434 'aaab' =~ /a+b?(?{print "$&\n"; $count++})(*FAIL)/;
54612592
YO
1435 print "Count=$count\n";
1436
1437which produces:
1438
1439 aaab
1440 aaa
1441 aa
1442 a
1443 aab
1444 aa
1445 a
1446 ab
1447 a
1448 Count=9
1449
5d458dd8 1450If we add a C<(*PRUNE)> before the count like the following
54612592 1451
5d458dd8 1452 'aaab' =~ /a+b?(*PRUNE)(?{print "$&\n"; $count++})(*FAIL)/;
54612592
YO
1453 print "Count=$count\n";
1454
1455we prevent backtracking and find the count of the longest matching
353c6505 1456at each matching starting point like so:
54612592
YO
1457
1458 aaab
1459 aab
1460 ab
1461 Count=3
1462
5d458dd8 1463Any number of C<(*PRUNE)> assertions may be used in a pattern.
54612592 1464
5d458dd8
YO
1465See also C<< (?>pattern) >> and possessive quantifiers for other ways to
1466control backtracking. In some cases, the use of C<(*PRUNE)> can be
1467replaced with a C<< (?>pattern) >> with no functional difference; however,
1468C<(*PRUNE)> can be used to handle cases that cannot be expressed using a
1469C<< (?>pattern) >> alone.
54612592 1470
e2e6a0f1 1471
5d458dd8
YO
1472=item C<(*SKIP)> C<(*SKIP:NAME)>
1473X<(*SKIP)>
e2e6a0f1 1474
5d458dd8 1475This zero-width pattern is similar to C<(*PRUNE)>, except that on
e2e6a0f1 1476failure it also signifies that whatever text that was matched leading up
5d458dd8
YO
1477to the C<(*SKIP)> pattern being executed cannot be part of I<any> match
1478of this pattern. This effectively means that the regex engine "skips" forward
1479to this position on failure and tries to match again, (assuming that
1480there is sufficient room to match).
1481
1482The name of the C<(*SKIP:NAME)> pattern has special significance. If a
1483C<(*MARK:NAME)> was encountered while matching, then it is that position
1484which is used as the "skip point". If no C<(*MARK)> of that name was
1485encountered, then the C<(*SKIP)> operator has no effect. When used
1486without a name the "skip point" is where the match point was when
1487executing the (*SKIP) pattern.
1488
1489Compare the following to the examples in C<(*PRUNE)>, note the string
24b23f37
YO
1490is twice as long:
1491
5d458dd8 1492 'aaabaaab' =~ /a+b?(*SKIP)(?{print "$&\n"; $count++})(*FAIL)/;
24b23f37
YO
1493 print "Count=$count\n";
1494
1495outputs
1496
1497 aaab
1498 aaab
1499 Count=2
1500
5d458dd8 1501Once the 'aaab' at the start of the string has matched, and the C<(*SKIP)>
353c6505 1502executed, the next starting point will be where the cursor was when the
5d458dd8
YO
1503C<(*SKIP)> was executed.
1504
5d458dd8
YO
1505=item C<(*MARK:NAME)> C<(*:NAME)>
1506X<(*MARK)> C<(*MARK:NAME)> C<(*:NAME)>
1507
1508This zero-width pattern can be used to mark the point reached in a string
1509when a certain part of the pattern has been successfully matched. This
1510mark may be given a name. A later C<(*SKIP)> pattern will then skip
1511forward to that point if backtracked into on failure. Any number of
b4222fa9 1512C<(*MARK)> patterns are allowed, and the NAME portion may be duplicated.
5d458dd8
YO
1513
1514In addition to interacting with the C<(*SKIP)> pattern, C<(*MARK:NAME)>
1515can be used to "label" a pattern branch, so that after matching, the
1516program can determine which branches of the pattern were involved in the
1517match.
1518
1519When a match is successful, the C<$REGMARK> variable will be set to the
1520name of the most recently executed C<(*MARK:NAME)> that was involved
1521in the match.
1522
1523This can be used to determine which branch of a pattern was matched
c27a5cfe 1524without using a separate capture group for each branch, which in turn
5d458dd8
YO
1525can result in a performance improvement, as perl cannot optimize
1526C</(?:(x)|(y)|(z))/> as efficiently as something like
1527C</(?:x(*MARK:x)|y(*MARK:y)|z(*MARK:z))/>.
1528
1529When a match has failed, and unless another verb has been involved in
1530failing the match and has provided its own name to use, the C<$REGERROR>
1531variable will be set to the name of the most recently executed
1532C<(*MARK:NAME)>.
1533
1534See C<(*SKIP)> for more details.
1535
b62d2d15
YO
1536As a shortcut C<(*MARK:NAME)> can be written C<(*:NAME)>.
1537
5d458dd8
YO
1538=item C<(*THEN)> C<(*THEN:NAME)>
1539
241e7389 1540This is similar to the "cut group" operator C<::> from Perl 6. Like
5d458dd8
YO
1541C<(*PRUNE)>, this verb always matches, and when backtracked into on
1542failure, it causes the regex engine to try the next alternation in the
1543innermost enclosing group (capturing or otherwise).
1544
1545Its name comes from the observation that this operation combined with the
1546alternation operator (C<|>) can be used to create what is essentially a
1547pattern-based if/then/else block:
1548
1549 ( COND (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ )
1550
1551Note that if this operator is used and NOT inside of an alternation then
1552it acts exactly like the C<(*PRUNE)> operator.
1553
1554 / A (*PRUNE) B /
1555
1556is the same as
1557
1558 / A (*THEN) B /
1559
1560but
1561
1562 / ( A (*THEN) B | C (*THEN) D ) /
1563
1564is not the same as
1565
1566 / ( A (*PRUNE) B | C (*PRUNE) D ) /
1567
1568as after matching the A but failing on the B the C<(*THEN)> verb will
1569backtrack and try C; but the C<(*PRUNE)> verb will simply fail.
24b23f37 1570
e2e6a0f1
YO
1571=item C<(*COMMIT)>
1572X<(*COMMIT)>
24b23f37 1573
241e7389 1574This is the Perl 6 "commit pattern" C<< <commit> >> or C<:::>. It's a
5d458dd8
YO
1575zero-width pattern similar to C<(*SKIP)>, except that when backtracked
1576into on failure it causes the match to fail outright. No further attempts
1577to find a valid match by advancing the start pointer will occur again.
1578For example,
24b23f37 1579
e2e6a0f1 1580 'aaabaaab' =~ /a+b?(*COMMIT)(?{print "$&\n"; $count++})(*FAIL)/;
24b23f37
YO
1581 print "Count=$count\n";
1582
1583outputs
1584
1585 aaab
1586 Count=1
1587
e2e6a0f1
YO
1588In other words, once the C<(*COMMIT)> has been entered, and if the pattern
1589does not match, the regex engine will not try any further matching on the
1590rest of the string.
c277df42 1591
e2e6a0f1 1592=back
9af228c6 1593
e2e6a0f1 1594=item Verbs without an argument
9af228c6
YO
1595
1596=over 4
1597
e2e6a0f1
YO
1598=item C<(*FAIL)> C<(*F)>
1599X<(*FAIL)> X<(*F)>
9af228c6 1600
e2e6a0f1
YO
1601This pattern matches nothing and always fails. It can be used to force the
1602engine to backtrack. It is equivalent to C<(?!)>, but easier to read. In
1603fact, C<(?!)> gets optimised into C<(*FAIL)> internally.
9af228c6 1604
e2e6a0f1 1605It is probably useful only when combined with C<(?{})> or C<(??{})>.
9af228c6 1606
e2e6a0f1
YO
1607=item C<(*ACCEPT)>
1608X<(*ACCEPT)>
9af228c6 1609
e2e6a0f1
YO
1610B<WARNING:> This feature is highly experimental. It is not recommended
1611for production code.
9af228c6 1612
e2e6a0f1
YO
1613This pattern matches nothing and causes the end of successful matching at
1614the point at which the C<(*ACCEPT)> pattern was encountered, regardless of
1615whether there is actually more to match in the string. When inside of a
0d017f4d 1616nested pattern, such as recursion, or in a subpattern dynamically generated
e2e6a0f1 1617via C<(??{})>, only the innermost pattern is ended immediately.
9af228c6 1618
c27a5cfe 1619If the C<(*ACCEPT)> is inside of capturing groups then the groups are
e2e6a0f1
YO
1620marked as ended at the point at which the C<(*ACCEPT)> was encountered.
1621For instance:
9af228c6 1622
e2e6a0f1 1623 'AB' =~ /(A (A|B(*ACCEPT)|C) D)(E)/x;
9af228c6 1624
e2e6a0f1 1625will match, and C<$1> will be C<AB> and C<$2> will be C<B>, C<$3> will not
0d017f4d 1626be set. If another branch in the inner parentheses were matched, such as in the
e2e6a0f1 1627string 'ACDE', then the C<D> and C<E> would have to be matched as well.
9af228c6
YO
1628
1629=back
c277df42 1630
a0d0e21e
LW
1631=back
1632
c07a80fd 1633=head2 Backtracking
d74e8afc 1634X<backtrack> X<backtracking>
c07a80fd 1635
35a734be
IZ
1636NOTE: This section presents an abstract approximation of regular
1637expression behavior. For a more rigorous (and complicated) view of
1638the rules involved in selecting a match among possible alternatives,
0d017f4d 1639see L<Combining RE Pieces>.
35a734be 1640
c277df42 1641A fundamental feature of regular expression matching involves the
5a964f20 1642notion called I<backtracking>, which is currently used (when needed)
0d017f4d 1643by all regular non-possessive expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc
IZ
1644C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
1645internally, but the general principle outlined here is valid.
c07a80fd
PP
1646
1647For a regular expression to match, the I<entire> regular expression must
1648match, not just part of it. So if the beginning of a pattern containing a
1649quantifier succeeds in a way that causes later parts in the pattern to
1650fail, the matching engine backs up and recalculates the beginning
1651part--that's why it's called backtracking.
1652
1653Here is an example of backtracking: Let's say you want to find the
1654word following "foo" in the string "Food is on the foo table.":
1655
1656 $_ = "Food is on the foo table.";
1657 if ( /\b(foo)\s+(\w+)/i ) {
f793d64a 1658 print "$2 follows $1.\n";
c07a80fd
PP
1659 }
1660
1661When the match runs, the first part of the regular expression (C<\b(foo)>)
1662finds a possible match right at the beginning of the string, and loads up
1663$1 with "Foo". However, as soon as the matching engine sees that there's
1664no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 1665mistake and starts over again one character after where it had the
c07a80fd
PP
1666tentative match. This time it goes all the way until the next occurrence
1667of "foo". The complete regular expression matches this time, and you get
1668the expected output of "table follows foo."
1669
1670Sometimes minimal matching can help a lot. Imagine you'd like to match
1671everything between "foo" and "bar". Initially, you write something
1672like this:
1673
1674 $_ = "The food is under the bar in the barn.";
1675 if ( /foo(.*)bar/ ) {
f793d64a 1676 print "got <$1>\n";
c07a80fd
PP
1677 }
1678
1679Which perhaps unexpectedly yields:
1680
1681 got <d is under the bar in the >
1682
1683That's because C<.*> was greedy, so you get everything between the
14218588 1684I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd
PP
1685to use minimal matching to make sure you get the text between a "foo"
1686and the first "bar" thereafter.
1687
1688 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
1689 got <d is under the >
1690
0d017f4d 1691Here's another example. Let's say you'd like to match a number at the end
b6e13d97 1692of a string, and you also want to keep the preceding part of the match.
c07a80fd
PP
1693So you write this:
1694
1695 $_ = "I have 2 numbers: 53147";
f793d64a
KW
1696 if ( /(.*)(\d*)/ ) { # Wrong!
1697 print "Beginning is <$1>, number is <$2>.\n";
c07a80fd
PP
1698 }
1699
1700That won't work at all, because C<.*> was greedy and gobbled up the
1701whole string. As C<\d*> can match on an empty string the complete
1702regular expression matched successfully.
1703
8e1088bc 1704 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd
PP
1705
1706Here are some variants, most of which don't work:
1707
1708 $_ = "I have 2 numbers: 53147";
1709 @pats = qw{
f793d64a
KW
1710 (.*)(\d*)
1711 (.*)(\d+)
1712 (.*?)(\d*)
1713 (.*?)(\d+)
1714 (.*)(\d+)$
1715 (.*?)(\d+)$
1716 (.*)\b(\d+)$
1717 (.*\D)(\d+)$
c07a80fd
PP
1718 };
1719
1720 for $pat (@pats) {
f793d64a
KW
1721 printf "%-12s ", $pat;
1722 if ( /$pat/ ) {
1723 print "<$1> <$2>\n";
1724 } else {
1725 print "FAIL\n";
1726 }
c07a80fd
PP
1727 }
1728
1729That will print out:
1730
1731 (.*)(\d*) <I have 2 numbers: 53147> <>
1732 (.*)(\d+) <I have 2 numbers: 5314> <7>
1733 (.*?)(\d*) <> <>
1734 (.*?)(\d+) <I have > <2>
1735 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
1736 (.*?)(\d+)$ <I have 2 numbers: > <53147>
1737 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
1738 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
1739
1740As you see, this can be a bit tricky. It's important to realize that a
1741regular expression is merely a set of assertions that gives a definition
1742of success. There may be 0, 1, or several different ways that the
1743definition might succeed against a particular string. And if there are
5a964f20
TC
1744multiple ways it might succeed, you need to understand backtracking to
1745know which variety of success you will achieve.
c07a80fd 1746
19799a22 1747When using look-ahead assertions and negations, this can all get even
8b19b778 1748trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd
PP
1749followed by "123". You might try to write that as
1750
871b0233 1751 $_ = "ABC123";
f793d64a
KW
1752 if ( /^\D*(?!123)/ ) { # Wrong!
1753 print "Yup, no 123 in $_\n";
871b0233 1754 }
c07a80fd
PP
1755
1756But that isn't going to match; at least, not the way you're hoping. It
1757claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 1758why that pattern matches, contrary to popular expectations:
c07a80fd 1759
4358a253
SS
1760 $x = 'ABC123';
1761 $y = 'ABC445';
c07a80fd 1762
4358a253
SS
1763 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/;
1764 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/;
c07a80fd 1765
4358a253
SS
1766 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/;
1767 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/;
c07a80fd
PP
1768
1769This prints
1770
1771 2: got ABC
1772 3: got AB
1773 4: got ABC
1774
5f05dabc 1775You might have expected test 3 to fail because it seems to a more
c07a80fd
PP
1776general purpose version of test 1. The important difference between
1777them is that test 3 contains a quantifier (C<\D*>) and so can use
1778backtracking, whereas test 1 will not. What's happening is
1779that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 1780non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 1781let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 1782fail.
14218588 1783
c07a80fd 1784The search engine will initially match C<\D*> with "ABC". Then it will
14218588 1785try to match C<(?!123> with "123", which fails. But because
c07a80fd
PP
1786a quantifier (C<\D*>) has been used in the regular expression, the
1787search engine can backtrack and retry the match differently
54310121 1788in the hope of matching the complete regular expression.
c07a80fd 1789
5a964f20
TC
1790The pattern really, I<really> wants to succeed, so it uses the
1791standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 1792time. Now there's indeed something following "AB" that is not
14218588 1793"123". It's "C123", which suffices.
c07a80fd 1794
14218588
GS
1795We can deal with this by using both an assertion and a negation.
1796We'll say that the first part in $1 must be followed both by a digit
1797and by something that's not "123". Remember that the look-aheads
1798are zero-width expressions--they only look, but don't consume any
1799of the string in their match. So rewriting this way produces what
c07a80fd
PP
1800you'd expect; that is, case 5 will fail, but case 6 succeeds:
1801
4358a253
SS
1802 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/;
1803 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/;
c07a80fd
PP
1804
1805 6: got ABC
1806
5a964f20 1807In other words, the two zero-width assertions next to each other work as though
19799a22 1808they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd
PP
1809matches only if you're at the beginning of the line AND the end of the
1810line simultaneously. The deeper underlying truth is that juxtaposition in
1811regular expressions always means AND, except when you write an explicit OR
1812using the vertical bar. C</ab/> means match "a" AND (then) match "b",
1813although the attempted matches are made at different positions because "a"
1814is not a zero-width assertion, but a one-width assertion.
1815
0d017f4d 1816B<WARNING>: Particularly complicated regular expressions can take
14218588 1817exponential time to solve because of the immense number of possible
0d017f4d 1818ways they can use backtracking to try for a match. For example, without
9da458fc
IZ
1819internal optimizations done by the regular expression engine, this will
1820take a painfully long time to run:
c07a80fd 1821
e1901655
IZ
1822 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
1823
1824And if you used C<*>'s in the internal groups instead of limiting them
1825to 0 through 5 matches, then it would take forever--or until you ran
1826out of stack space. Moreover, these internal optimizations are not
1827always applicable. For example, if you put C<{0,5}> instead of C<*>
1828on the external group, no current optimization is applicable, and the
1829match takes a long time to finish.
c07a80fd 1830
9da458fc
IZ
1831A powerful tool for optimizing such beasts is what is known as an
1832"independent group",
c47ff5f1 1833which does not backtrack (see L<C<< (?>pattern) >>>). Note also that
9da458fc 1834zero-length look-ahead/look-behind assertions will not backtrack to make
5d458dd8 1835the tail match, since they are in "logical" context: only
14218588 1836whether they match is considered relevant. For an example
9da458fc 1837where side-effects of look-ahead I<might> have influenced the
c47ff5f1 1838following match, see L<C<< (?>pattern) >>>.
c277df42 1839
a0d0e21e 1840=head2 Version 8 Regular Expressions
d74e8afc 1841X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
a0d0e21e 1842
5a964f20 1843In case you're not familiar with the "regular" Version 8 regex
a0d0e21e
LW
1844routines, here are the pattern-matching rules not described above.
1845
54310121 1846Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 1847with a special meaning described here or above. You can cause
5a964f20 1848characters that normally function as metacharacters to be interpreted
5f05dabc 1849literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
0d017f4d
WL
1850character; "\\" matches a "\"). This escape mechanism is also required
1851for the character used as the pattern delimiter.
1852
1853A series of characters matches that series of characters in the target
1854string, so the pattern C<blurfl> would match "blurfl" in the target
1855string.
a0d0e21e
LW
1856
1857You can specify a character class, by enclosing a list of characters
5d458dd8 1858in C<[]>, which will match any character from the list. If the
a0d0e21e 1859first character after the "[" is "^", the class matches any character not
14218588 1860in the list. Within a list, the "-" character specifies a
5a964f20 1861range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2
GS
1862inclusive. If you want either "-" or "]" itself to be a member of a
1863class, put it at the start of the list (possibly after a "^"), or
1864escape it with a backslash. "-" is also taken literally when it is
1865at the end of the list, just before the closing "]". (The
84850974
DD
1866following all specify the same class of three characters: C<[-az]>,
1867C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
5d458dd8
YO
1868specifies a class containing twenty-six characters, even on EBCDIC-based
1869character sets.) Also, if you try to use the character
1870classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1871a range, the "-" is understood literally.
a0d0e21e 1872
8ada0baa
JH
1873Note also that the whole range idea is rather unportable between
1874character sets--and even within character sets they may cause results
1875you probably didn't expect. A sound principle is to use only ranges
0d017f4d 1876that begin from and end at either alphabetics of equal case ([a-e],
8ada0baa
JH
1877[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1878spell out the character sets in full.
1879
54310121 1880Characters may be specified using a metacharacter syntax much like that
a0d0e21e
LW
1881used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1882"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
dc0d9c48 1883of three octal digits, matches the character whose coded character set value
5d458dd8 1884is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
dc0d9c48 1885matches the character whose ordinal is I<nn>. The expression \cI<x>
5d458dd8 1886matches the character control-I<x>. Finally, the "." metacharacter
fb55449c 1887matches any character except "\n" (unless you use C</s>).
a0d0e21e
LW
1888
1889You can specify a series of alternatives for a pattern using "|" to
1890separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1891or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e
LW
1892first alternative includes everything from the last pattern delimiter
1893("(", "[", or the beginning of the pattern) up to the first "|", and
1894the last alternative contains everything from the last "|" to the next
14218588
GS
1895pattern delimiter. That's why it's common practice to include
1896alternatives in parentheses: to minimize confusion about where they
a3cb178b
GS
1897start and end.
1898
5a964f20 1899Alternatives are tried from left to right, so the first
a3cb178b
GS
1900alternative found for which the entire expression matches, is the one that
1901is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1902example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b
GS
1903part will match, as that is the first alternative tried, and it successfully
1904matches the target string. (This might not seem important, but it is
1905important when you are capturing matched text using parentheses.)
1906
5a964f20 1907Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1908so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1909
14218588
GS
1910Within a pattern, you may designate subpatterns for later reference
1911by enclosing them in parentheses, and you may refer back to the
1912I<n>th subpattern later in the pattern using the metacharacter
1913\I<n>. Subpatterns are numbered based on the left to right order
1914of their opening parenthesis. A backreference matches whatever
1915actually matched the subpattern in the string being examined, not
d8b950dc 1916the rules for that subpattern. Therefore, C<(0|0x)\d*\s\g1\d*> will
14218588
GS
1917match "0x1234 0x4321", but not "0x1234 01234", because subpattern
19181 matched "0x", even though the rule C<0|0x> could potentially match
1919the leading 0 in the second number.
cb1a09d0 1920
0d017f4d 1921=head2 Warning on \1 Instead of $1
cb1a09d0 1922
5a964f20 1923Some people get too used to writing things like:
cb1a09d0
AD
1924
1925 $pattern =~ s/(\W)/\\\1/g;
1926
3ff1c45a
KW
1927This is grandfathered (for \1 to \9) for the RHS of a substitute to avoid
1928shocking the
cb1a09d0 1929B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1930PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0
AD
1931the usual double-quoted string means a control-A. The customary Unix
1932meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1933of doing that, you get yourself into trouble if you then add an C</e>
1934modifier.
1935
f793d64a 1936 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0
AD
1937
1938Or if you try to do
1939
1940 s/(\d+)/\1000/;
1941
1942You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1943C<${1}000>. The operation of interpolation should not be confused
cb1a09d0
AD
1944with the operation of matching a backreference. Certainly they mean two
1945different things on the I<left> side of the C<s///>.
9fa51da4 1946
0d017f4d 1947=head2 Repeated Patterns Matching a Zero-length Substring
c84d73f1 1948
19799a22 1949B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1
IZ
1950
1951Regular expressions provide a terse and powerful programming language. As
1952with most other power tools, power comes together with the ability
1953to wreak havoc.
1954
1955A common abuse of this power stems from the ability to make infinite
628afcb5 1956loops using regular expressions, with something as innocuous as:
c84d73f1
IZ
1957
1958 'foo' =~ m{ ( o? )* }x;
1959
0d017f4d 1960The C<o?> matches at the beginning of C<'foo'>, and since the position
c84d73f1 1961in the string is not moved by the match, C<o?> would match again and again
527e91da 1962because of the C<*> quantifier. Another common way to create a similar cycle
c84d73f1
IZ
1963is with the looping modifier C<//g>:
1964
1965 @matches = ( 'foo' =~ m{ o? }xg );
1966
1967or
1968
1969 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
1970
1971or the loop implied by split().
1972
1973However, long experience has shown that many programming tasks may
14218588
GS
1974be significantly simplified by using repeated subexpressions that
1975may match zero-length substrings. Here's a simple example being:
c84d73f1 1976
f793d64a 1977 @chars = split //, $string; # // is not magic in split
c84d73f1
IZ
1978 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
1979
9da458fc 1980Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1 1981the infinite loop>. The rules for this are different for lower-level
527e91da 1982loops given by the greedy quantifiers C<*+{}>, and for higher-level
c84d73f1
IZ
1983ones like the C</g> modifier or split() operator.
1984
19799a22
GS
1985The lower-level loops are I<interrupted> (that is, the loop is
1986broken) when Perl detects that a repeated expression matched a
1987zero-length substring. Thus
c84d73f1
IZ
1988
1989 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
1990
5d458dd8 1991is made equivalent to
c84d73f1 1992
5d458dd8
YO
1993 m{ (?: NON_ZERO_LENGTH )*
1994 |
1995 (?: ZERO_LENGTH )?
c84d73f1
IZ
1996 }x;
1997
1998The higher level-loops preserve an additional state between iterations:
5d458dd8 1999whether the last match was zero-length. To break the loop, the following
c84d73f1 2000match after a zero-length match is prohibited to have a length of zero.
5d458dd8 2001This prohibition interacts with backtracking (see L<"Backtracking">),
c84d73f1
IZ
2002and so the I<second best> match is chosen if the I<best> match is of
2003zero length.
2004
19799a22 2005For example:
c84d73f1
IZ
2006
2007 $_ = 'bar';
2008 s/\w??/<$&>/g;
2009
20fb949f 2010results in C<< <><b><><a><><r><> >>. At each position of the string the best
5d458dd8 2011match given by non-greedy C<??> is the zero-length match, and the I<second
c84d73f1
IZ
2012best> match is what is matched by C<\w>. Thus zero-length matches
2013alternate with one-character-long matches.
2014
5d458dd8 2015Similarly, for repeated C<m/()/g> the second-best match is the match at the
c84d73f1
IZ
2016position one notch further in the string.
2017
19799a22 2018The additional state of being I<matched with zero-length> is associated with
c84d73f1 2019the matched string, and is reset by each assignment to pos().
9da458fc
IZ
2020Zero-length matches at the end of the previous match are ignored
2021during C<split>.
c84d73f1 2022
0d017f4d 2023=head2 Combining RE Pieces
35a734be
IZ
2024
2025Each of the elementary pieces of regular expressions which were described
2026before (such as C<ab> or C<\Z>) could match at most one substring
2027at the given position of the input string. However, in a typical regular
2028expression these elementary pieces are combined into more complicated
2029patterns using combining operators C<ST>, C<S|T>, C<S*> etc
2030(in these examples C<S> and C<T> are regular subexpressions).
2031
2032Such combinations can include alternatives, leading to a problem of choice:
2033if we match a regular expression C<a|ab> against C<"abc">, will it match
2034substring C<"a"> or C<"ab">? One way to describe which substring is
2035actually matched is the concept of backtracking (see L<"Backtracking">).
2036However, this description is too low-level and makes you think
2037in terms of a particular implementation.
2038
2039Another description starts with notions of "better"/"worse". All the
2040substrings which may be matched by the given regular expression can be
2041sorted from the "best" match to the "worst" match, and it is the "best"
2042match which is chosen. This substitutes the question of "what is chosen?"
2043by the question of "which matches are better, and which are worse?".
2044
2045Again, for elementary pieces there is no such question, since at most
2046one match at a given position is possible. This section describes the
2047notion of better/worse for combining operators. In the description
2048below C<S> and C<T> are regular subexpressions.
2049
13a2d996 2050=over 4
35a734be
IZ
2051
2052=item C<ST>
2053
2054Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
2055substrings which can be matched by C<S>, C<B> and C<B'> are substrings
5d458dd8 2056which can be matched by C<T>.
35a734be
IZ
2057
2058If C<A> is better match for C<S> than C<A'>, C<AB> is a better
2059match than C<A'B'>.
2060
2061If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
2062C<B> is better match for C<T> than C<B'>.
2063
2064=item C<S|T>
2065
2066When C<S> can match, it is a better match than when only C<T> can match.
2067
2068Ordering of two matches for C<S> is the same as for C<S>. Similar for
2069two matches for C<T>.
2070
2071=item C<S{REPEAT_COUNT}>
2072
2073Matches as C<SSS...S> (repeated as many times as necessary).
2074
2075=item C<S{min,max}>
2076
2077Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
2078
2079=item C<S{min,max}?>
2080
2081Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
2082
2083=item C<S?>, C<S*>, C<S+>
2084
2085Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
2086
2087=item C<S??>, C<S*?>, C<S+?>
2088
2089Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
2090
c47ff5f1 2091=item C<< (?>S) >>
35a734be
IZ
2092
2093Matches the best match for C<S> and only that.
2094
2095=item C<(?=S)>, C<(?<=S)>
2096
2097Only the best match for C<S> is considered. (This is important only if
2098C<S> has capturing parentheses, and backreferences are used somewhere
2099else in the whole regular expression.)
2100
2101=item C<(?!S)>, C<(?<!S)>
2102
2103For this grouping operator there is no need to describe the ordering, since
2104only whether or not C<S> can match is important.
2105
6bda09f9 2106=item C<(??{ EXPR })>, C<(?PARNO)>
35a734be
IZ
2107
2108The ordering is the same as for the regular expression which is
c27a5cfe 2109the result of EXPR, or the pattern contained by capture group PARNO.
35a734be
IZ
2110
2111=item C<(?(condition)yes-pattern|no-pattern)>
2112
2113Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
2114already determined. The ordering of the matches is the same as for the
2115chosen subexpression.
2116
2117=back
2118
2119The above recipes describe the ordering of matches I<at a given position>.
2120One more rule is needed to understand how a match is determined for the
2121whole regular expression: a match at an earlier position is always better
2122than a match at a later position.
2123
0d017f4d 2124=head2 Creating Custom RE Engines
c84d73f1
IZ
2125
2126Overloaded constants (see L<overload>) provide a simple way to extend
2127the functionality of the RE engine.
2128
2129Suppose that we want to enable a new RE escape-sequence C<\Y|> which
0d017f4d 2130matches at a boundary between whitespace characters and non-whitespace
c84d73f1
IZ
2131characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
2132at these positions, so we want to have each C<\Y|> in the place of the
2133more complicated version. We can create a module C<customre> to do
2134this:
2135
2136 package customre;
2137 use overload;
2138
2139 sub import {
2140 shift;
2141 die "No argument to customre::import allowed" if @_;
2142 overload::constant 'qr' => \&convert;
2143 }
2144
2145 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
2146
580a9fe1
RGS
2147 # We must also take care of not escaping the legitimate \\Y|
2148 # sequence, hence the presence of '\\' in the conversion rules.
5d458dd8 2149 my %rules = ( '\\' => '\\\\',
f793d64a 2150 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
c84d73f1
IZ
2151 sub convert {
2152 my $re = shift;
5d458dd8 2153 $re =~ s{
c84d73f1
IZ
2154 \\ ( \\ | Y . )
2155 }
5d458dd8 2156 { $rules{$1} or invalid($re,$1) }sgex;
c84d73f1
IZ
2157 return $re;
2158 }
2159
2160Now C<use customre> enables the new escape in constant regular
2161expressions, i.e., those without any runtime variable interpolations.
2162As documented in L<overload>, this conversion will work only over
2163literal parts of regular expressions. For C<\Y|$re\Y|> the variable
2164part of this regular expression needs to be converted explicitly
2165(but only if the special meaning of C<\Y|> should be enabled inside $re):
2166
2167 use customre;
2168 $re = <>;
2169 chomp $re;
2170 $re = customre::convert $re;
2171 /\Y|$re\Y|/;
2172
1f1031fe
YO
2173=head1 PCRE/Python Support
2174
99d59c4d 2175As of Perl 5.10.0, Perl supports several Python/PCRE specific extensions
1f1031fe 2176to the regex syntax. While Perl programmers are encouraged to use the
99d59c4d 2177Perl specific syntax, the following are also accepted:
1f1031fe
YO
2178
2179=over 4
2180
ae5648b3 2181=item C<< (?PE<lt>NAMEE<gt>pattern) >>
1f1031fe 2182
c27a5cfe 2183Define a named capture group. Equivalent to C<< (?<NAME>pattern) >>.
1f1031fe
YO
2184
2185=item C<< (?P=NAME) >>
2186
c27a5cfe 2187Backreference to a named capture group. Equivalent to C<< \g{NAME} >>.
1f1031fe
YO
2188
2189=item C<< (?P>NAME) >>
2190
c27a5cfe 2191Subroutine call to a named capture group. Equivalent to C<< (?&NAME) >>.
1f1031fe 2192
ee9b8eae 2193=back
1f1031fe 2194
19799a22
GS
2195=head1 BUGS
2196
78288b8e
KW
2197There are numerous problems with case insensitive matching of characters
2198outside the ASCII range, especially with those whose folds are multiple
2199characters, such as ligatures like C<LATIN SMALL LIGATURE FF>.
2200
f253210b
KW
2201In a bracketed character class with case insensitive matching, ranges only work
2202for ASCII characters. For example,
2203C<m/[\N{CYRILLIC CAPITAL LETTER A}-\N{CYRILLIC CAPITAL LETTER YA}]/i>
2204doesn't match all the Russian upper and lower case letters.
2205
88c9975e
KW
2206Many regular expression constructs don't work on EBCDIC platforms.
2207
9da458fc
IZ
2208This document varies from difficult to understand to completely
2209and utterly opaque. The wandering prose riddled with jargon is
2210hard to fathom in several places.
2211
2212This document needs a rewrite that separates the tutorial content
2213from the reference content.
19799a22
GS
2214
2215=head1 SEE ALSO
9fa51da4 2216
91e0c79e
MJD
2217L<perlrequick>.
2218
2219L<perlretut>.
2220
9b599b2a
GS
2221L<perlop/"Regexp Quote-Like Operators">.
2222
1e66bd83
PP
2223L<perlop/"Gory details of parsing quoted constructs">.
2224
14218588
GS
2225L<perlfaq6>.
2226
9b599b2a
GS
2227L<perlfunc/pos>.
2228
2229L<perllocale>.
2230
fb55449c
JH
2231L<perlebcdic>.
2232
14218588
GS
2233I<Mastering Regular Expressions> by Jeffrey Friedl, published
2234by O'Reilly and Associates.