This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
reg_eval_scope.t: skip coring tests on all arches
[perl5.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
d74e8afc 2X<regular expression> X<regex> X<regexp>
a0d0e21e
LW
3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
5d458dd8 8This page describes the syntax of regular expressions in Perl.
91e0c79e 9
cc46d5f2 10If you haven't used regular expressions before, a quick-start
91e0c79e
MJD
11introduction is available in L<perlrequick>, and a longer tutorial
12introduction is available in L<perlretut>.
13
14For reference on how regular expressions are used in matching
15operations, plus various examples of the same, see discussions of
16C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
17Operators">.
cb1a09d0 18
0d017f4d
WL
19
20=head2 Modifiers
21
19799a22 22Matching operations can have various modifiers. Modifiers
5a964f20 23that relate to the interpretation of the regular expression inside
19799a22 24are listed below. Modifiers that alter the way a regular expression
5d458dd8 25is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 26L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 27
55497cff
PP
28=over 4
29
54310121 30=item m
d74e8afc 31X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
55497cff
PP
32
33Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 34the start or end of the string to matching the start or end of any
7f761169 35line anywhere within the string.
55497cff 36
54310121 37=item s
d74e8afc
ITB
38X</s> X<regex, single-line> X<regexp, single-line>
39X<regular expression, single-line>
55497cff
PP
40
41Treat string as single line. That is, change "." to match any character
19799a22 42whatsoever, even a newline, which normally it would not match.
55497cff 43
34d67d80 44Used together, as C</ms>, they let the "." match any character whatsoever,
fb55449c 45while still allowing "^" and "$" to match, respectively, just after
19799a22 46and just before newlines within the string.
7b8d334a 47
87e95b7f
YO
48=item i
49X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
50X<regular expression, case-insensitive>
51
52Do case-insensitive pattern matching.
53
54If C<use locale> is in effect, the case map is taken from the current
17580e7a
KW
55locale for code points less than 255, and from Unicode rules for larger
56code points. See L<perllocale>.
87e95b7f 57
54310121 58=item x
d74e8afc 59X</x>
55497cff
PP
60
61Extend your pattern's legibility by permitting whitespace and comments.
62
87e95b7f
YO
63=item p
64X</p> X<regex, preserve> X<regexp, preserve>
65
632a1772 66Preserve the string matched such that ${^PREMATCH}, ${^MATCH}, and
87e95b7f
YO
67${^POSTMATCH} are available for use after matching.
68
e2e6bec7
DN
69=item g and c
70X</g> X</c>
71
72Global matching, and keep the Current position after failed matching.
73Unlike i, m, s and x, these two flags affect the way the regex is used
74rather than the regex itself. See
75L<perlretut/"Using regular expressions in Perl"> for further explanation
76of the g and c modifiers.
77
b6fa137b
FC
78=item a, d, l and u
79X</a> X</d> X</l> X</u>
80
81These modifiers, new in 5.14, affect which character-set semantics
82(Unicode, ASCII, etc.) are used, as described below.
83
55497cff 84=back
a0d0e21e
LW
85
86These are usually written as "the C</x> modifier", even though the delimiter
b6fa137b 87in question might not really be a slash. The modifiers C</imsxadlup>
ab7bb42d 88may also be embedded within the regular expression itself using
b6fa137b
FC
89the C<(?...)> construct.
90
91The C</x>, C</l>, C</u>, C</a> and C</d> modifiers need a little more
92explanation.
a0d0e21e 93
b6fa137b 94C</x> tells
7b059540 95the regular expression parser to ignore most whitespace that is neither
55497cff 96backslashed nor within a character class. You can use this to break up
4633a7c4 97your regular expression into (slightly) more readable parts. The C<#>
54310121 98character is also treated as a metacharacter introducing a comment,
55497cff 99just as in ordinary Perl code. This also means that if you want real
14218588 100whitespace or C<#> characters in the pattern (outside a character
f9a3ff1a 101class, where they are unaffected by C</x>), then you'll either have to
7b059540
KW
102escape them (using backslashes or C<\Q...\E>) or encode them using octal,
103hex, or C<\N{}> escapes. Taken together, these features go a long way towards
8933a740
RGS
104making Perl's regular expressions more readable. Note that you have to
105be careful not to include the pattern delimiter in the comment--perl has
106no way of knowing you did not intend to close the pattern early. See
107the C-comment deletion code in L<perlop>. Also note that anything inside
7651b971 108a C<\Q...\E> stays unaffected by C</x>. And note that C</x> doesn't affect
0b928c2f 109space interpretation within a single multi-character construct. For
7651b971 110example in C<\x{...}>, regardless of the C</x> modifier, there can be no
9bb1f947 111spaces. Same for a L<quantifier|/Quantifiers> such as C<{3}> or
f9e949fd
KW
112C<{5,}>. Similarly, C<(?:...)> can't have a space between the C<?> and C<:>,
113but can between the C<(> and C<?>. Within any delimiters for such a
114construct, allowed spaces are not affected by C</x>, and depend on the
115construct. For example, C<\x{...}> can't have spaces because hexadecimal
116numbers don't have spaces in them. But, Unicode properties can have spaces, so
0b928c2f 117in C<\p{...}> there can be spaces that follow the Unicode rules, for which see
9bb1f947 118L<perluniprops/Properties accessible through \p{} and \P{}>.
d74e8afc 119X</x>
a0d0e21e 120
b6fa137b
FC
121C</l> means to use a locale (see L<perllocale>) when pattern matching.
122The locale used will be the one in effect at the time of execution of
123the pattern match. This may not be the same as the compilation-time
124locale, and can differ from one match to another if there is an
125intervening call of the
126L<setlocale() function|perllocale/The setlocale function>.
127This modifier is automatically set if the regular expression is compiled
128within the scope of a C<"use locale"> pragma.
129Perl only allows single-byte locales. This means that code points above
130255 are treated as Unicode no matter what locale is in effect.
131Under Unicode rules, there are a few case-insensitive matches that cross the
0b928c2f 132255/256 boundary. These are disallowed. For example,
b6fa137b
FC
1330xFF does not caselessly match the character at 0x178, LATIN CAPITAL
134LETTER Y WITH DIAERESIS, because 0xFF may not be LATIN SMALL LETTER Y
135in the current locale, and Perl has no way of knowing if that character
136even exists in the locale, much less what code point it is.
137X</l>
138
139C</u> means to use Unicode semantics when pattern matching. It is
0b928c2f
FC
140automatically set if the regular expression is encoded in utf8 internally,
141or is compiled within the scope of a
b6fa137b 142L<C<"use feature 'unicode_strings">|feature> pragma (and isn't also in
0b928c2f
FC
143the scope of the L<C<"use locale">|locale> or the L<C<"use bytes">|bytes>
144pragma). On ASCII platforms, the code points between 128 and 255 take on their
b6fa137b
FC
145Latin-1 (ISO-8859-1) meanings (which are the same as Unicode's), whereas
146in strict ASCII their meanings are undefined. Thus the platform
147effectively becomes a Unicode platform. The ASCII characters remain as
148ASCII characters (since ASCII is a subset of Latin-1 and Unicode). For
149example, when this option is not on, on a non-utf8 string, C<"\w">
150matches precisely C<[A-Za-z0-9_]>. When the option is on, it matches
151not just those, but all the Latin-1 word characters (such as an "n" with
152a tilde). On EBCDIC platforms, which already are equivalent to Latin-1,
153this modifier changes behavior only when the C<"/i"> modifier is also
154specified, and affects only two characters, giving them full Unicode
155semantics: the C<MICRO SIGN> will match the Greek capital and
156small letters C<MU>; otherwise not; and the C<LATIN CAPITAL LETTER SHARP
157S> will match any of C<SS>, C<Ss>, C<sS>, and C<ss>, otherwise not.
158(This last case is buggy, however.)
159X</u>
160
161C</a> is the same as C</u>, except that C<\d>, C<\s>, C<\w>, and the
162Posix character classes are restricted to matching in the ASCII range
163only. That is, with this modifier, C<\d> always means precisely the
164digits C<"0"> to C<"9">; C<\s> means the five characters C<[ \f\n\r\t]>;
165C<\w> means the 63 characters C<[A-Za-z0-9_]>; and likewise, all the
166Posix classes such as C<[[:print:]]> match only the appropriate
167ASCII-range characters. As you would expect, this modifier causes, for
168example, C<\D> to mean the same thing as C<[^0-9]>; in fact, all
169non-ASCII characters match C<\D>, C<\S>, and C<\W>. C<\b> still means
170to match at the boundary between C<\w> and C<\W>, using the C<"a">
171definitions of them (similarly for C<\B>). Otherwise, C<"a"> behaves
172like the C<"u"> modifier, in that case-insensitive matching uses Unicode
173semantics; for example, "k" will match the Unicode C<\N{KELVIN SIGN}>
174under C</i> matching, and code points in the Latin1 range, above ASCII
175will have Unicode semantics when it comes to case-insensitive matching.
176But writing two in "a"'s in a row will increase its effect, causing the
0b928c2f 177Kelvin sign and all other non-ASCII characters not to match any ASCII
b6fa137b
FC
178character under C</i> matching.
179X</a>
180
181C</d> means to use the traditional Perl pattern-matching behavior.
182This is dualistic (hence the name C</d>, which also could stand for
183"depends"). When this is in effect, Perl matches according to the
184platform's native character set rules unless there is something that
185indicates to use Unicode rules. If either the target string or the
186pattern itself is encoded in UTF-8, Unicode rules are used. Also, if
187the pattern contains Unicode-only features, such as code points above
188255, C<\p()> Unicode properties or C<\N{}> Unicode names, Unicode rules
189will be used. It is automatically selected by default if the regular
190expression is compiled neither within the scope of a C<"use locale">
191pragma nor a <C<"use feature 'unicode_strings"> pragma.
192This behavior causes a number of glitches, see
193L<perlunicode/The "Unicode Bug">.
194X</d>
195
a0d0e21e
LW
196=head2 Regular Expressions
197
04838cea
RGS
198=head3 Metacharacters
199
384f06ae 200The patterns used in Perl pattern matching evolved from those supplied in
14218588 201the Version 8 regex routines. (The routines are derived
19799a22
GS
202(distantly) from Henry Spencer's freely redistributable reimplementation
203of the V8 routines.) See L<Version 8 Regular Expressions> for
204details.
a0d0e21e
LW
205
206In particular the following metacharacters have their standard I<egrep>-ish
207meanings:
d74e8afc
ITB
208X<metacharacter>
209X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
210
a0d0e21e 211
f793d64a
KW
212 \ Quote the next metacharacter
213 ^ Match the beginning of the line
214 . Match any character (except newline)
215 $ Match the end of the line (or before newline at the end)
216 | Alternation
217 () Grouping
218 [] Bracketed Character class
a0d0e21e 219
14218588
GS
220By default, the "^" character is guaranteed to match only the
221beginning of the string, the "$" character only the end (or before the
222newline at the end), and Perl does certain optimizations with the
a0d0e21e
LW
223assumption that the string contains only one line. Embedded newlines
224will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 225string as a multi-line buffer, such that the "^" will match after any
0d520e8e
YO
226newline within the string (except if the newline is the last character in
227the string), and "$" will match before any newline. At the
a0d0e21e
LW
228cost of a little more overhead, you can do this by using the /m modifier
229on the pattern match operator. (Older programs did this by setting C<$*>,
0b928c2f 230but this option was removed in perl 5.9.)
d74e8afc 231X<^> X<$> X</m>
a0d0e21e 232
14218588 233To simplify multi-line substitutions, the "." character never matches a
55497cff 234newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 235the string is a single line--even if it isn't.
d74e8afc 236X<.> X</s>
a0d0e21e 237
04838cea
RGS
238=head3 Quantifiers
239
a0d0e21e 240The following standard quantifiers are recognized:
d74e8afc 241X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
a0d0e21e 242
f793d64a
KW
243 * Match 0 or more times
244 + Match 1 or more times
245 ? Match 1 or 0 times
246 {n} Match exactly n times
247 {n,} Match at least n times
248 {n,m} Match at least n but not more than m times
a0d0e21e 249
0b928c2f
FC
250(If a curly bracket occurs in any other context and does not form part of
251a backslashed sequence like C<\x{...}>, it is treated
b975c076 252as a regular character. In particular, the lower bound
527e91da
BB
253is not optional.) The "*" quantifier is equivalent to C<{0,}>, the "+"
254quantifier to C<{1,}>, and the "?" quantifier to C<{0,1}>. n and m are limited
d0b16107 255to non-negative integral values less than a preset limit defined when perl is built.
9c79236d
GS
256This is usually 32766 on the most common platforms. The actual limit can
257be seen in the error message generated by code such as this:
258
820475bd 259 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 260
54310121
PP
261By default, a quantified subpattern is "greedy", that is, it will match as
262many times as possible (given a particular starting location) while still
263allowing the rest of the pattern to match. If you want it to match the
264minimum number of times possible, follow the quantifier with a "?". Note
265that the meanings don't change, just the "greediness":
0d017f4d 266X<metacharacter> X<greedy> X<greediness>
d74e8afc 267X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{n,m}?>
a0d0e21e 268
f793d64a
KW
269 *? Match 0 or more times, not greedily
270 +? Match 1 or more times, not greedily
271 ?? Match 0 or 1 time, not greedily
0b928c2f 272 {n}? Match exactly n times, not greedily (redundant)
f793d64a
KW
273 {n,}? Match at least n times, not greedily
274 {n,m}? Match at least n but not more than m times, not greedily
a0d0e21e 275
b9b4dddf
YO
276By default, when a quantified subpattern does not allow the rest of the
277overall pattern to match, Perl will backtrack. However, this behaviour is
0d017f4d 278sometimes undesirable. Thus Perl provides the "possessive" quantifier form
b9b4dddf
YO
279as well.
280
f793d64a
KW
281 *+ Match 0 or more times and give nothing back
282 ++ Match 1 or more times and give nothing back
283 ?+ Match 0 or 1 time and give nothing back
284 {n}+ Match exactly n times and give nothing back (redundant)
285 {n,}+ Match at least n times and give nothing back
286 {n,m}+ Match at least n but not more than m times and give nothing back
b9b4dddf
YO
287
288For instance,
289
290 'aaaa' =~ /a++a/
291
292will never match, as the C<a++> will gobble up all the C<a>'s in the
293string and won't leave any for the remaining part of the pattern. This
294feature can be extremely useful to give perl hints about where it
295shouldn't backtrack. For instance, the typical "match a double-quoted
296string" problem can be most efficiently performed when written as:
297
298 /"(?:[^"\\]++|\\.)*+"/
299
0d017f4d 300as we know that if the final quote does not match, backtracking will not
0b928c2f
FC
301help. See the independent subexpression
302L</C<< (?>pattern) >>> for more details;
b9b4dddf
YO
303possessive quantifiers are just syntactic sugar for that construct. For
304instance the above example could also be written as follows:
305
306 /"(?>(?:(?>[^"\\]+)|\\.)*)"/
307
04838cea
RGS
308=head3 Escape sequences
309
0b928c2f 310Because patterns are processed as double-quoted strings, the following
a0d0e21e
LW
311also work:
312
f793d64a
KW
313 \t tab (HT, TAB)
314 \n newline (LF, NL)
315 \r return (CR)
316 \f form feed (FF)
317 \a alarm (bell) (BEL)
318 \e escape (think troff) (ESC)
f793d64a 319 \cK control char (example: VT)
dc0d9c48 320 \x{}, \x00 character whose ordinal is the given hexadecimal number
fb121860 321 \N{name} named Unicode character or character sequence
f793d64a 322 \N{U+263D} Unicode character (example: FIRST QUARTER MOON)
f0a2b745 323 \o{}, \000 character whose ordinal is the given octal number
f793d64a
KW
324 \l lowercase next char (think vi)
325 \u uppercase next char (think vi)
326 \L lowercase till \E (think vi)
327 \U uppercase till \E (think vi)
328 \Q quote (disable) pattern metacharacters till \E
329 \E end either case modification or quoted section, think vi
a0d0e21e 330
9bb1f947 331Details are in L<perlop/Quote and Quote-like Operators>.
1d2dff63 332
e1d1eefb 333=head3 Character Classes and other Special Escapes
04838cea 334
a0d0e21e 335In addition, Perl defines the following:
d0b16107 336X<\g> X<\k> X<\K> X<backreference>
a0d0e21e 337
f793d64a
KW
338 Sequence Note Description
339 [...] [1] Match a character according to the rules of the
340 bracketed character class defined by the "...".
341 Example: [a-z] matches "a" or "b" or "c" ... or "z"
342 [[:...:]] [2] Match a character according to the rules of the POSIX
343 character class "..." within the outer bracketed
344 character class. Example: [[:upper:]] matches any
345 uppercase character.
d35dd6c6
KW
346 \w [3] Match a "word" character (alphanumeric plus "_", plus
347 other connector punctuation chars plus Unicode
0b928c2f 348 marks)
f793d64a
KW
349 \W [3] Match a non-"word" character
350 \s [3] Match a whitespace character
351 \S [3] Match a non-whitespace character
352 \d [3] Match a decimal digit character
353 \D [3] Match a non-digit character
354 \pP [3] Match P, named property. Use \p{Prop} for longer names
355 \PP [3] Match non-P
356 \X [4] Match Unicode "eXtended grapheme cluster"
357 \C Match a single C-language char (octet) even if that is
358 part of a larger UTF-8 character. Thus it breaks up
359 characters into their UTF-8 bytes, so you may end up
360 with malformed pieces of UTF-8. Unsupported in
361 lookbehind.
c27a5cfe 362 \1 [5] Backreference to a specific capture group or buffer.
f793d64a
KW
363 '1' may actually be any positive integer.
364 \g1 [5] Backreference to a specific or previous group,
365 \g{-1} [5] The number may be negative indicating a relative
c27a5cfe 366 previous group and may optionally be wrapped in
f793d64a
KW
367 curly brackets for safer parsing.
368 \g{name} [5] Named backreference
369 \k<name> [5] Named backreference
370 \K [6] Keep the stuff left of the \K, don't include it in $&
371 \N [7] Any character but \n (experimental). Not affected by
372 /s modifier
373 \v [3] Vertical whitespace
374 \V [3] Not vertical whitespace
375 \h [3] Horizontal whitespace
376 \H [3] Not horizontal whitespace
377 \R [4] Linebreak
e1d1eefb 378
9bb1f947
KW
379=over 4
380
381=item [1]
382
383See L<perlrecharclass/Bracketed Character Classes> for details.
df225385 384
9bb1f947 385=item [2]
b8c5462f 386
9bb1f947 387See L<perlrecharclass/POSIX Character Classes> for details.
b8c5462f 388
9bb1f947 389=item [3]
5496314a 390
9bb1f947 391See L<perlrecharclass/Backslash sequences> for details.
5496314a 392
9bb1f947 393=item [4]
5496314a 394
9bb1f947 395See L<perlrebackslash/Misc> for details.
d0b16107 396
9bb1f947 397=item [5]
b8c5462f 398
c27a5cfe 399See L</Capture groups> below for details.
93733859 400
9bb1f947 401=item [6]
b8c5462f 402
9bb1f947
KW
403See L</Extended Patterns> below for details.
404
405=item [7]
406
407Note that C<\N> has two meanings. When of the form C<\N{NAME}>, it matches the
fb121860
KW
408character or character sequence whose name is C<NAME>; and similarly
409when of the form C<\N{U+I<hex>}>, it matches the character whose Unicode
410code point is I<hex>. Otherwise it matches any character but C<\n>.
9bb1f947
KW
411
412=back
d0b16107 413
04838cea
RGS
414=head3 Assertions
415
a0d0e21e 416Perl defines the following zero-width assertions:
d74e8afc
ITB
417X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
418X<regexp, zero-width assertion>
419X<regular expression, zero-width assertion>
420X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
a0d0e21e 421
9bb1f947
KW
422 \b Match a word boundary
423 \B Match except at a word boundary
424 \A Match only at beginning of string
425 \Z Match only at end of string, or before newline at the end
426 \z Match only at end of string
427 \G Match only at pos() (e.g. at the end-of-match position
9da458fc 428 of prior m//g)
a0d0e21e 429
14218588 430A word boundary (C<\b>) is a spot between two characters
19799a22
GS
431that has a C<\w> on one side of it and a C<\W> on the other side
432of it (in either order), counting the imaginary characters off the
433beginning and end of the string as matching a C<\W>. (Within
434character classes C<\b> represents backspace rather than a word
435boundary, just as it normally does in any double-quoted string.)
436The C<\A> and C<\Z> are just like "^" and "$", except that they
437won't match multiple times when the C</m> modifier is used, while
438"^" and "$" will match at every internal line boundary. To match
439the actual end of the string and not ignore an optional trailing
440newline, use C<\z>.
d74e8afc 441X<\b> X<\A> X<\Z> X<\z> X</m>
19799a22
GS
442
443The C<\G> assertion can be used to chain global matches (using
444C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
445It is also useful when writing C<lex>-like scanners, when you have
446several patterns that you want to match against consequent substrings
0b928c2f 447of your string; see the previous reference. The actual location
19799a22 448where C<\G> will match can also be influenced by using C<pos()> as
58e23c8d 449an lvalue: see L<perlfunc/pos>. Note that the rule for zero-length
0b928c2f
FC
450matches (see L</"Repeated Patterns Matching a Zero-length Substring">)
451is modified somewhat, in that contents to the left of C<\G> are
58e23c8d
YO
452not counted when determining the length of the match. Thus the following
453will not match forever:
d74e8afc 454X<\G>
c47ff5f1 455
e761bb84
CO
456 my $string = 'ABC';
457 pos($string) = 1;
458 while ($string =~ /(.\G)/g) {
459 print $1;
460 }
58e23c8d
YO
461
462It will print 'A' and then terminate, as it considers the match to
463be zero-width, and thus will not match at the same position twice in a
464row.
465
466It is worth noting that C<\G> improperly used can result in an infinite
467loop. Take care when using patterns that include C<\G> in an alternation.
468
c27a5cfe 469=head3 Capture groups
04838cea 470
c27a5cfe
KW
471The bracketing construct C<( ... )> creates capture groups (also referred to as
472capture buffers). To refer to the current contents of a group later on, within
d8b950dc
KW
473the same pattern, use C<\g1> (or C<\g{1}>) for the first, C<\g2> (or C<\g{2}>)
474for the second, and so on.
475This is called a I<backreference>.
d74e8afc 476X<regex, capture buffer> X<regexp, capture buffer>
c27a5cfe 477X<regex, capture group> X<regexp, capture group>
d74e8afc 478X<regular expression, capture buffer> X<backreference>
c27a5cfe 479X<regular expression, capture group> X<backreference>
1f1031fe 480X<\g{1}> X<\g{-1}> X<\g{name}> X<relative backreference> X<named backreference>
d8b950dc
KW
481X<named capture buffer> X<regular expression, named capture buffer>
482X<named capture group> X<regular expression, named capture group>
483X<%+> X<$+{name}> X<< \k<name> >>
484There is no limit to the number of captured substrings that you may use.
485Groups are numbered with the leftmost open parenthesis being number 1, etc. If
486a group did not match, the associated backreference won't match either. (This
487can happen if the group is optional, or in a different branch of an
488alternation.)
489You can omit the C<"g">, and write C<"\1">, etc, but there are some issues with
490this form, described below.
491
492You can also refer to capture groups relatively, by using a negative number, so
493that C<\g-1> and C<\g{-1}> both refer to the immediately preceding capture
494group, and C<\g-2> and C<\g{-2}> both refer to the group before it. For
495example:
5624f11d
YO
496
497 /
c27a5cfe
KW
498 (Y) # group 1
499 ( # group 2
500 (X) # group 3
501 \g{-1} # backref to group 3
502 \g{-3} # backref to group 1
5624f11d
YO
503 )
504 /x
505
d8b950dc
KW
506would match the same as C</(Y) ( (X) \g3 \g1 )/x>. This allows you to
507interpolate regexes into larger regexes and not have to worry about the
508capture groups being renumbered.
509
510You can dispense with numbers altogether and create named capture groups.
511The notation is C<(?E<lt>I<name>E<gt>...)> to declare and C<\g{I<name>}> to
512reference. (To be compatible with .Net regular expressions, C<\g{I<name>}> may
513also be written as C<\k{I<name>}>, C<\kE<lt>I<name>E<gt>> or C<\k'I<name>'>.)
514I<name> must not begin with a number, nor contain hyphens.
515When different groups within the same pattern have the same name, any reference
516to that name assumes the leftmost defined group. Named groups count in
517absolute and relative numbering, and so can also be referred to by those
518numbers.
519(It's possible to do things with named capture groups that would otherwise
520require C<(??{})>.)
521
522Capture group contents are dynamically scoped and available to you outside the
523pattern until the end of the enclosing block or until the next successful
524match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
525You can refer to them by absolute number (using C<"$1"> instead of C<"\g1">,
526etc); or by name via the C<%+> hash, using C<"$+{I<name>}">.
527
528Braces are required in referring to named capture groups, but are optional for
529absolute or relative numbered ones. Braces are safer when creating a regex by
530concatenating smaller strings. For example if you have C<qr/$a$b/>, and C<$a>
531contained C<"\g1">, and C<$b> contained C<"37">, you would get C</\g137/> which
532is probably not what you intended.
533
534The C<\g> and C<\k> notations were introduced in Perl 5.10.0. Prior to that
535there were no named nor relative numbered capture groups. Absolute numbered
0b928c2f
FC
536groups were referred to using C<\1>,
537C<\2>, etc., and this notation is still
d8b950dc
KW
538accepted (and likely always will be). But it leads to some ambiguities if
539there are more than 9 capture groups, as C<\10> could mean either the tenth
540capture group, or the character whose ordinal in octal is 010 (a backspace in
541ASCII). Perl resolves this ambiguity by interpreting C<\10> as a backreference
542only if at least 10 left parentheses have opened before it. Likewise C<\11> is
543a backreference only if at least 11 left parentheses have opened before it.
e1f120a9
KW
544And so on. C<\1> through C<\9> are always interpreted as backreferences.
545There are several examples below that illustrate these perils. You can avoid
546the ambiguity by always using C<\g{}> or C<\g> if you mean capturing groups;
547and for octal constants always using C<\o{}>, or for C<\077> and below, using 3
548digits padded with leading zeros, since a leading zero implies an octal
549constant.
d8b950dc
KW
550
551The C<\I<digit>> notation also works in certain circumstances outside
552the pattern. See L</Warning on \1 Instead of $1> below for details.)
81714fb9 553
14218588 554Examples:
a0d0e21e
LW
555
556 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
557
d8b950dc 558 /(.)\g1/ # find first doubled char
81714fb9
YO
559 and print "'$1' is the first doubled character\n";
560
561 /(?<char>.)\k<char>/ # ... a different way
562 and print "'$+{char}' is the first doubled character\n";
563
d8b950dc 564 /(?'char'.)\g1/ # ... mix and match
81714fb9 565 and print "'$1' is the first doubled character\n";
c47ff5f1 566
14218588 567 if (/Time: (..):(..):(..)/) { # parse out values
f793d64a
KW
568 $hours = $1;
569 $minutes = $2;
570 $seconds = $3;
a0d0e21e 571 }
c47ff5f1 572
9d860678
KW
573 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\g10/ # \g10 is a backreference
574 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\10/ # \10 is octal
575 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\10/ # \10 is a backreference
576 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\010/ # \010 is octal
577
578 $a = '(.)\1'; # Creates problems when concatenated.
579 $b = '(.)\g{1}'; # Avoids the problems.
580 "aa" =~ /${a}/; # True
581 "aa" =~ /${b}/; # True
582 "aa0" =~ /${a}0/; # False!
583 "aa0" =~ /${b}0/; # True
dc0d9c48
KW
584 "aa\x08" =~ /${a}0/; # True!
585 "aa\x08" =~ /${b}0/; # False
9d860678 586
14218588
GS
587Several special variables also refer back to portions of the previous
588match. C<$+> returns whatever the last bracket match matched.
589C<$&> returns the entire matched string. (At one point C<$0> did
590also, but now it returns the name of the program.) C<$`> returns
77ea4f6d
JV
591everything before the matched string. C<$'> returns everything
592after the matched string. And C<$^N> contains whatever was matched by
593the most-recently closed group (submatch). C<$^N> can be used in
594extended patterns (see below), for example to assign a submatch to a
81714fb9 595variable.
d74e8afc 596X<$+> X<$^N> X<$&> X<$`> X<$'>
14218588 597
d8b950dc
KW
598These special variables, like the C<%+> hash and the numbered match variables
599(C<$1>, C<$2>, C<$3>, etc.) are dynamically scoped
14218588
GS
600until the end of the enclosing block or until the next successful
601match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
d74e8afc
ITB
602X<$+> X<$^N> X<$&> X<$`> X<$'>
603X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
604
0d017f4d 605B<NOTE>: Failed matches in Perl do not reset the match variables,
5146ce24 606which makes it easier to write code that tests for a series of more
665e98b9
JH
607specific cases and remembers the best match.
608
14218588
GS
609B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
610C<$'> anywhere in the program, it has to provide them for every
611pattern match. This may substantially slow your program. Perl
d8b950dc 612uses the same mechanism to produce C<$1>, C<$2>, etc, so you also pay a
14218588
GS
613price for each pattern that contains capturing parentheses. (To
614avoid this cost while retaining the grouping behaviour, use the
615extended regular expression C<(?: ... )> instead.) But if you never
616use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
617parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
618if you can, but if you can't (and some algorithms really appreciate
619them), once you've used them once, use them at will, because you've
620already paid the price. As of 5.005, C<$&> is not so costly as the
621other two.
d74e8afc 622X<$&> X<$`> X<$'>
68dc0745 623
99d59c4d 624As a workaround for this problem, Perl 5.10.0 introduces C<${^PREMATCH}>,
cde0cee5
YO
625C<${^MATCH}> and C<${^POSTMATCH}>, which are equivalent to C<$`>, C<$&>
626and C<$'>, B<except> that they are only guaranteed to be defined after a
87e95b7f 627successful match that was executed with the C</p> (preserve) modifier.
cde0cee5
YO
628The use of these variables incurs no global performance penalty, unlike
629their punctuation char equivalents, however at the trade-off that you
630have to tell perl when you want to use them.
87e95b7f 631X</p> X<p modifier>
cde0cee5 632
9d727203
KW
633=head2 Quoting metacharacters
634
19799a22
GS
635Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
636C<\w>, C<\n>. Unlike some other regular expression languages, there
637are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 638that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22
GS
639interpreted as a literal character, not a metacharacter. This was
640once used in a common idiom to disable or quote the special meanings
641of regular expression metacharacters in a string that you want to
36bbe248 642use for a pattern. Simply quote all non-"word" characters:
a0d0e21e
LW
643
644 $pattern =~ s/(\W)/\\$1/g;
645
f1cbbd6e 646(If C<use locale> is set, then this depends on the current locale.)
14218588
GS
647Today it is more common to use the quotemeta() function or the C<\Q>
648metaquoting escape sequence to disable all metacharacters' special
649meanings like this:
a0d0e21e
LW
650
651 /$unquoted\Q$quoted\E$unquoted/
652
9da458fc
IZ
653Beware that if you put literal backslashes (those not inside
654interpolated variables) between C<\Q> and C<\E>, double-quotish
655backslash interpolation may lead to confusing results. If you
656I<need> to use literal backslashes within C<\Q...\E>,
657consult L<perlop/"Gory details of parsing quoted constructs">.
658
19799a22
GS
659=head2 Extended Patterns
660
14218588 661Perl also defines a consistent extension syntax for features not
0b928c2f
FC
662found in standard tools like B<awk> and
663B<lex>. The syntax for most of these is a
14218588
GS
664pair of parentheses with a question mark as the first thing within
665the parentheses. The character after the question mark indicates
666the extension.
19799a22 667
14218588
GS
668The stability of these extensions varies widely. Some have been
669part of the core language for many years. Others are experimental
670and may change without warning or be completely removed. Check
671the documentation on an individual feature to verify its current
672status.
19799a22 673
14218588
GS
674A question mark was chosen for this and for the minimal-matching
675construct because 1) question marks are rare in older regular
676expressions, and 2) whenever you see one, you should stop and
0b928c2f 677"question" exactly what is going on. That's psychology....
a0d0e21e
LW
678
679=over 10
680
cc6b7395 681=item C<(?#text)>
d74e8afc 682X<(?#)>
a0d0e21e 683
14218588 684A comment. The text is ignored. If the C</x> modifier enables
19799a22 685whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3
GS
686the comment as soon as it sees a C<)>, so there is no way to put a literal
687C<)> in the comment.
a0d0e21e 688
cfaf538b 689=item C<(?adlupimsx-imsx)>
fb85c044 690
cfaf538b 691=item C<(?^alupimsx)>
fb85c044 692X<(?)> X<(?^)>
19799a22 693
0b6d1084
JH
694One or more embedded pattern-match modifiers, to be turned on (or
695turned off, if preceded by C<->) for the remainder of the pattern or
fb85c044
KW
696the remainder of the enclosing pattern group (if any).
697
fb85c044 698This is particularly useful for dynamic patterns, such as those read in from a
0d017f4d 699configuration file, taken from an argument, or specified in a table
0b928c2f
FC
700somewhere. Consider the case where some patterns want to be
701case-sensitive and some do not: The case-insensitive ones merely need to
0d017f4d 702include C<(?i)> at the front of the pattern. For example:
19799a22
GS
703
704 $pattern = "foobar";
5d458dd8 705 if ( /$pattern/i ) { }
19799a22
GS
706
707 # more flexible:
708
709 $pattern = "(?i)foobar";
5d458dd8 710 if ( /$pattern/ ) { }
19799a22 711
0b6d1084 712These modifiers are restored at the end of the enclosing group. For example,
19799a22 713
d8b950dc 714 ( (?i) blah ) \s+ \g1
19799a22 715
0d017f4d
WL
716will match C<blah> in any case, some spaces, and an exact (I<including the case>!)
717repetition of the previous word, assuming the C</x> modifier, and no C</i>
718modifier outside this group.
19799a22 719
8eb5594e
DR
720These modifiers do not carry over into named subpatterns called in the
721enclosing group. In other words, a pattern such as C<((?i)(&NAME))> does not
722change the case-sensitivity of the "NAME" pattern.
723
dc925305
KW
724Any of these modifiers can be set to apply globally to all regular
725expressions compiled within the scope of a C<use re>. See
a0bbd6ff 726L<re/"'/flags' mode">.
dc925305 727
9de15fec
KW
728Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
729after the C<"?"> is a shorthand equivalent to C<d-imsx>. Flags (except
730C<"d">) may follow the caret to override it.
731But a minus sign is not legal with it.
732
dc925305 733Note that the C<a>, C<d>, C<l>, C<p>, and C<u> modifiers are special in
e1d8d8ac 734that they can only be enabled, not disabled, and the C<a>, C<d>, C<l>, and
dc925305 735C<u> modifiers are mutually exclusive: specifying one de-specifies the
bdc22dd5 736others, and a maximum of one may appear in the construct. Thus, for
0b928c2f 737example, C<(?-p)> will warn when compiled under C<use warnings>;
b6fa137b 738C<(?-d:...)> and C<(?dl:...)> are fatal errors.
9de15fec
KW
739
740Note also that the C<p> modifier is special in that its presence
741anywhere in a pattern has a global effect.
cde0cee5 742
5a964f20 743=item C<(?:pattern)>
d74e8afc 744X<(?:)>
a0d0e21e 745
cfaf538b 746=item C<(?adluimsx-imsx:pattern)>
ca9dfc88 747
cfaf538b 748=item C<(?^aluimsx:pattern)>
fb85c044
KW
749X<(?^:)>
750
5a964f20
TC
751This is for clustering, not capturing; it groups subexpressions like
752"()", but doesn't make backreferences as "()" does. So
a0d0e21e 753
5a964f20 754 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e
LW
755
756is like
757
5a964f20 758 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 759
19799a22
GS
760but doesn't spit out extra fields. It's also cheaper not to capture
761characters if you don't need to.
a0d0e21e 762
19799a22 763Any letters between C<?> and C<:> act as flags modifiers as with
cfaf538b 764C<(?adluimsx-imsx)>. For example,
ca9dfc88
IZ
765
766 /(?s-i:more.*than).*million/i
767
14218588 768is equivalent to the more verbose
ca9dfc88
IZ
769
770 /(?:(?s-i)more.*than).*million/i
771
fb85c044 772Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
9de15fec
KW
773after the C<"?"> is a shorthand equivalent to C<d-imsx>. Any positive
774flags (except C<"d">) may follow the caret, so
fb85c044
KW
775
776 (?^x:foo)
777
778is equivalent to
779
780 (?x-ims:foo)
781
782The caret tells Perl that this cluster doesn't inherit the flags of any
0b928c2f 783surrounding pattern, but uses the system defaults (C<d-imsx>),
fb85c044
KW
784modified by any flags specified.
785
786The caret allows for simpler stringification of compiled regular
787expressions. These look like
788
789 (?^:pattern)
790
791with any non-default flags appearing between the caret and the colon.
792A test that looks at such stringification thus doesn't need to have the
793system default flags hard-coded in it, just the caret. If new flags are
794added to Perl, the meaning of the caret's expansion will change to include
795the default for those flags, so the test will still work, unchanged.
796
797Specifying a negative flag after the caret is an error, as the flag is
798redundant.
799
800Mnemonic for C<(?^...)>: A fresh beginning since the usual use of a caret is
801to match at the beginning.
802
594d7033
YO
803=item C<(?|pattern)>
804X<(?|)> X<Branch reset>
805
806This is the "branch reset" pattern, which has the special property
c27a5cfe 807that the capture groups are numbered from the same starting point
99d59c4d 808in each alternation branch. It is available starting from perl 5.10.0.
4deaaa80 809
c27a5cfe 810Capture groups are numbered from left to right, but inside this
693596a8 811construct the numbering is restarted for each branch.
4deaaa80 812
c27a5cfe 813The numbering within each branch will be as normal, and any groups
4deaaa80
PJ
814following this construct will be numbered as though the construct
815contained only one branch, that being the one with the most capture
c27a5cfe 816groups in it.
4deaaa80 817
0b928c2f 818This construct is useful when you want to capture one of a
4deaaa80
PJ
819number of alternative matches.
820
821Consider the following pattern. The numbers underneath show in
c27a5cfe 822which group the captured content will be stored.
594d7033
YO
823
824
825 # before ---------------branch-reset----------- after
826 / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
827 # 1 2 2 3 2 3 4
828
ab106183
A
829Be careful when using the branch reset pattern in combination with
830named captures. Named captures are implemented as being aliases to
c27a5cfe 831numbered groups holding the captures, and that interferes with the
ab106183
A
832implementation of the branch reset pattern. If you are using named
833captures in a branch reset pattern, it's best to use the same names,
834in the same order, in each of the alternations:
835
836 /(?| (?<a> x ) (?<b> y )
837 | (?<a> z ) (?<b> w )) /x
838
839Not doing so may lead to surprises:
840
841 "12" =~ /(?| (?<a> \d+ ) | (?<b> \D+))/x;
842 say $+ {a}; # Prints '12'
843 say $+ {b}; # *Also* prints '12'.
844
c27a5cfe
KW
845The problem here is that both the group named C<< a >> and the group
846named C<< b >> are aliases for the group belonging to C<< $1 >>.
90a18110 847
ee9b8eae
YO
848=item Look-Around Assertions
849X<look-around assertion> X<lookaround assertion> X<look-around> X<lookaround>
850
0b928c2f 851Look-around assertions are zero-width patterns which match a specific
ee9b8eae
YO
852pattern without including it in C<$&>. Positive assertions match when
853their subpattern matches, negative assertions match when their subpattern
854fails. Look-behind matches text up to the current match position,
855look-ahead matches text following the current match position.
856
857=over 4
858
5a964f20 859=item C<(?=pattern)>
d74e8afc 860X<(?=)> X<look-ahead, positive> X<lookahead, positive>
a0d0e21e 861
19799a22 862A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e
LW
863matches a word followed by a tab, without including the tab in C<$&>.
864
5a964f20 865=item C<(?!pattern)>
d74e8afc 866X<(?!)> X<look-ahead, negative> X<lookahead, negative>
a0d0e21e 867
19799a22 868A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 869matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22
GS
870however that look-ahead and look-behind are NOT the same thing. You cannot
871use this for look-behind.
7b8d334a 872
5a964f20 873If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a
GS
874will not do what you want. That's because the C<(?!foo)> is just saying that
875the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
0b928c2f 876match. Use look-behind instead (see below).
c277df42 877
ee9b8eae
YO
878=item C<(?<=pattern)> C<\K>
879X<(?<=)> X<look-behind, positive> X<lookbehind, positive> X<\K>
c277df42 880
c47ff5f1 881A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22
GS
882matches a word that follows a tab, without including the tab in C<$&>.
883Works only for fixed-width look-behind.
c277df42 884
ee9b8eae
YO
885There is a special form of this construct, called C<\K>, which causes the
886regex engine to "keep" everything it had matched prior to the C<\K> and
0b928c2f 887not include it in C<$&>. This effectively provides variable-length
ee9b8eae
YO
888look-behind. The use of C<\K> inside of another look-around assertion
889is allowed, but the behaviour is currently not well defined.
890
c62285ac 891For various reasons C<\K> may be significantly more efficient than the
ee9b8eae
YO
892equivalent C<< (?<=...) >> construct, and it is especially useful in
893situations where you want to efficiently remove something following
894something else in a string. For instance
895
896 s/(foo)bar/$1/g;
897
898can be rewritten as the much more efficient
899
900 s/foo\Kbar//g;
901
5a964f20 902=item C<(?<!pattern)>
d74e8afc 903X<(?<!)> X<look-behind, negative> X<lookbehind, negative>
c277df42 904
19799a22
GS
905A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
906matches any occurrence of "foo" that does not follow "bar". Works
907only for fixed-width look-behind.
c277df42 908
ee9b8eae
YO
909=back
910
81714fb9
YO
911=item C<(?'NAME'pattern)>
912
913=item C<< (?<NAME>pattern) >>
914X<< (?<NAME>) >> X<(?'NAME')> X<named capture> X<capture>
915
c27a5cfe 916A named capture group. Identical in every respect to normal capturing
0b928c2f
FC
917parentheses C<()> but for the additional fact that the group
918can be referred to by name in various regular expression
919constructs (like C<\g{NAME}>) and can be accessed by name
920after a successful match via C<%+> or C<%->. See L<perlvar>
90a18110 921for more details on the C<%+> and C<%-> hashes.
81714fb9 922
c27a5cfe
KW
923If multiple distinct capture groups have the same name then the
924$+{NAME} will refer to the leftmost defined group in the match.
81714fb9 925
0d017f4d 926The forms C<(?'NAME'pattern)> and C<< (?<NAME>pattern) >> are equivalent.
81714fb9
YO
927
928B<NOTE:> While the notation of this construct is the same as the similar
c27a5cfe 929function in .NET regexes, the behavior is not. In Perl the groups are
81714fb9
YO
930numbered sequentially regardless of being named or not. Thus in the
931pattern
932
933 /(x)(?<foo>y)(z)/
934
935$+{foo} will be the same as $2, and $3 will contain 'z' instead of
936the opposite which is what a .NET regex hacker might expect.
937
1f1031fe
YO
938Currently NAME is restricted to simple identifiers only.
939In other words, it must match C</^[_A-Za-z][_A-Za-z0-9]*\z/> or
940its Unicode extension (see L<utf8>),
941though it isn't extended by the locale (see L<perllocale>).
81714fb9 942
1f1031fe 943B<NOTE:> In order to make things easier for programmers with experience
ae5648b3 944with the Python or PCRE regex engines, the pattern C<< (?PE<lt>NAMEE<gt>pattern) >>
0d017f4d 945may be used instead of C<< (?<NAME>pattern) >>; however this form does not
64c5a566 946support the use of single quotes as a delimiter for the name.
81714fb9 947
1f1031fe
YO
948=item C<< \k<NAME> >>
949
950=item C<< \k'NAME' >>
81714fb9
YO
951
952Named backreference. Similar to numeric backreferences, except that
953the group is designated by name and not number. If multiple groups
954have the same name then it refers to the leftmost defined group in
955the current match.
956
0d017f4d 957It is an error to refer to a name not defined by a C<< (?<NAME>) >>
81714fb9
YO
958earlier in the pattern.
959
960Both forms are equivalent.
961
1f1031fe 962B<NOTE:> In order to make things easier for programmers with experience
0d017f4d 963with the Python or PCRE regex engines, the pattern C<< (?P=NAME) >>
64c5a566 964may be used instead of C<< \k<NAME> >>.
1f1031fe 965
cc6b7395 966=item C<(?{ code })>
d74e8afc 967X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
c277df42 968
19799a22 969B<WARNING>: This extended regular expression feature is considered
b9b4dddf
YO
970experimental, and may be changed without notice. Code executed that
971has side effects may not perform identically from version to version
972due to the effect of future optimisations in the regex engine.
c277df42 973
cc46d5f2 974This zero-width assertion evaluates any embedded Perl code. It
19799a22
GS
975always succeeds, and its C<code> is not interpolated. Currently,
976the rules to determine where the C<code> ends are somewhat convoluted.
977
77ea4f6d
JV
978This feature can be used together with the special variable C<$^N> to
979capture the results of submatches in variables without having to keep
980track of the number of nested parentheses. For example:
981
982 $_ = "The brown fox jumps over the lazy dog";
983 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
984 print "color = $color, animal = $animal\n";
985
754091cb
RGS
986Inside the C<(?{...})> block, C<$_> refers to the string the regular
987expression is matching against. You can also use C<pos()> to know what is
fa11829f 988the current position of matching within this string.
754091cb 989
19799a22
GS
990The C<code> is properly scoped in the following sense: If the assertion
991is backtracked (compare L<"Backtracking">), all changes introduced after
992C<local>ization are undone, so that
b9ac3b5b
GS
993
994 $_ = 'a' x 8;
5d458dd8 995 m<
f793d64a 996 (?{ $cnt = 0 }) # Initialize $cnt.
b9ac3b5b 997 (
5d458dd8 998 a
b9ac3b5b 999 (?{
f793d64a 1000 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
b9ac3b5b 1001 })
5d458dd8 1002 )*
b9ac3b5b 1003 aaaa
f793d64a
KW
1004 (?{ $res = $cnt }) # On success copy to
1005 # non-localized location.
b9ac3b5b
GS
1006 >x;
1007
0d017f4d 1008will set C<$res = 4>. Note that after the match, C<$cnt> returns to the globally
14218588 1009introduced value, because the scopes that restrict C<local> operators
b9ac3b5b
GS
1010are unwound.
1011
19799a22
GS
1012This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
1013switch. If I<not> used in this way, the result of evaluation of
1014C<code> is put into the special variable C<$^R>. This happens
1015immediately, so C<$^R> can be used from other C<(?{ code })> assertions
1016inside the same regular expression.
b9ac3b5b 1017
19799a22
GS
1018The assignment to C<$^R> above is properly localized, so the old
1019value of C<$^R> is restored if the assertion is backtracked; compare
1020L<"Backtracking">.
b9ac3b5b 1021
19799a22
GS
1022For reasons of security, this construct is forbidden if the regular
1023expression involves run-time interpolation of variables, unless the
1024perilous C<use re 'eval'> pragma has been used (see L<re>), or the
0b928c2f 1025variables contain results of the C<qr//> operator (see
b6fa137b 1026L<perlop/"qr/STRINGE<sol>msixpodual">).
871b0233 1027
0d017f4d 1028This restriction is due to the wide-spread and remarkably convenient
19799a22 1029custom of using run-time determined strings as patterns. For example:
871b0233
IZ
1030
1031 $re = <>;
1032 chomp $re;
1033 $string =~ /$re/;
1034
14218588
GS
1035Before Perl knew how to execute interpolated code within a pattern,
1036this operation was completely safe from a security point of view,
1037although it could raise an exception from an illegal pattern. If
1038you turn on the C<use re 'eval'>, though, it is no longer secure,
1039so you should only do so if you are also using taint checking.
1040Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 1041compartment. See L<perlsec> for details about both these mechanisms.
871b0233 1042
e95d7314
GG
1043B<WARNING>: Use of lexical (C<my>) variables in these blocks is
1044broken. The result is unpredictable and will make perl unstable. The
1045workaround is to use global (C<our>) variables.
1046
8525cfae
FC
1047B<WARNING>: In perl 5.12.x and earlier, the regex engine
1048was not re-entrant, so interpolated code could not
1049safely invoke the regex engine either directly with
e95d7314 1050C<m//> or C<s///>), or indirectly with functions such as
8525cfae 1051C<split>. Invoking the regex engine in these blocks would make perl
e95d7314 1052unstable.
8988a1bb 1053
14455d6c 1054=item C<(??{ code })>
d74e8afc
ITB
1055X<(??{})>
1056X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
0f5d15d6 1057
19799a22 1058B<WARNING>: This extended regular expression feature is considered
b9b4dddf
YO
1059experimental, and may be changed without notice. Code executed that
1060has side effects may not perform identically from version to version
1061due to the effect of future optimisations in the regex engine.
0f5d15d6 1062
19799a22
GS
1063This is a "postponed" regular subexpression. The C<code> is evaluated
1064at run time, at the moment this subexpression may match. The result
0b928c2f 1065of evaluation is considered a regular expression and matched as
61528107 1066if it were inserted instead of this construct. Note that this means
c27a5cfe 1067that the contents of capture groups defined inside an eval'ed pattern
6bda09f9 1068are not available outside of the pattern, and vice versa, there is no
c27a5cfe 1069way for the inner pattern to refer to a capture group defined outside.
6bda09f9
YO
1070Thus,
1071
1072 ('a' x 100)=~/(??{'(.)' x 100})/
1073
81714fb9 1074B<will> match, it will B<not> set $1.
0f5d15d6 1075
428594d9 1076The C<code> is not interpolated. As before, the rules to determine
19799a22
GS
1077where the C<code> ends are currently somewhat convoluted.
1078
1079The following pattern matches a parenthesized group:
0f5d15d6
IZ
1080
1081 $re = qr{
f793d64a
KW
1082 \(
1083 (?:
1084 (?> [^()]+ ) # Non-parens without backtracking
1085 |
1086 (??{ $re }) # Group with matching parens
1087 )*
1088 \)
1089 }x;
0f5d15d6 1090
6bda09f9
YO
1091See also C<(?PARNO)> for a different, more efficient way to accomplish
1092the same task.
1093
0b370c0a
A
1094For reasons of security, this construct is forbidden if the regular
1095expression involves run-time interpolation of variables, unless the
1096perilous C<use re 'eval'> pragma has been used (see L<re>), or the
0b928c2f 1097variables contain results of the C<qr//> operator (see
b6fa137b 1098L<perlop/"qrE<sol>STRINGE<sol>msixpodual">).
0b370c0a 1099
8525cfae
FC
1100In perl 5.12.x and earlier, because the regex engine was not re-entrant,
1101delayed code could not safely invoke the regex engine either directly with
1102C<m//> or C<s///>), or indirectly with functions such as C<split>.
8988a1bb 1103
5d458dd8
YO
1104Recursing deeper than 50 times without consuming any input string will
1105result in a fatal error. The maximum depth is compiled into perl, so
6bda09f9
YO
1106changing it requires a custom build.
1107
542fa716
YO
1108=item C<(?PARNO)> C<(?-PARNO)> C<(?+PARNO)> C<(?R)> C<(?0)>
1109X<(?PARNO)> X<(?1)> X<(?R)> X<(?0)> X<(?-1)> X<(?+1)> X<(?-PARNO)> X<(?+PARNO)>
6bda09f9 1110X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
542fa716 1111X<regex, relative recursion>
6bda09f9 1112
81714fb9 1113Similar to C<(??{ code })> except it does not involve compiling any code,
c27a5cfe
KW
1114instead it treats the contents of a capture group as an independent
1115pattern that must match at the current position. Capture groups
81714fb9 1116contained by the pattern will have the value as determined by the
6bda09f9
YO
1117outermost recursion.
1118
894be9b7 1119PARNO is a sequence of digits (not starting with 0) whose value reflects
c27a5cfe 1120the paren-number of the capture group to recurse to. C<(?R)> recurses to
894be9b7 1121the beginning of the whole pattern. C<(?0)> is an alternate syntax for
542fa716 1122C<(?R)>. If PARNO is preceded by a plus or minus sign then it is assumed
c27a5cfe 1123to be relative, with negative numbers indicating preceding capture groups
542fa716 1124and positive ones following. Thus C<(?-1)> refers to the most recently
c27a5cfe 1125declared group, and C<(?+1)> indicates the next group to be declared.
c74340f9 1126Note that the counting for relative recursion differs from that of
c27a5cfe 1127relative backreferences, in that with recursion unclosed groups B<are>
c74340f9 1128included.
6bda09f9 1129
81714fb9 1130The following pattern matches a function foo() which may contain
f145b7e9 1131balanced parentheses as the argument.
6bda09f9
YO
1132
1133 $re = qr{ ( # paren group 1 (full function)
81714fb9 1134 foo
6bda09f9
YO
1135 ( # paren group 2 (parens)
1136 \(
1137 ( # paren group 3 (contents of parens)
1138 (?:
1139 (?> [^()]+ ) # Non-parens without backtracking
1140 |
1141 (?2) # Recurse to start of paren group 2
1142 )*
1143 )
1144 \)
1145 )
1146 )
1147 }x;
1148
1149If the pattern was used as follows
1150
1151 'foo(bar(baz)+baz(bop))'=~/$re/
1152 and print "\$1 = $1\n",
1153 "\$2 = $2\n",
1154 "\$3 = $3\n";
1155
1156the output produced should be the following:
1157
1158 $1 = foo(bar(baz)+baz(bop))
1159 $2 = (bar(baz)+baz(bop))
81714fb9 1160 $3 = bar(baz)+baz(bop)
6bda09f9 1161
c27a5cfe 1162If there is no corresponding capture group defined, then it is a
61528107 1163fatal error. Recursing deeper than 50 times without consuming any input
81714fb9 1164string will also result in a fatal error. The maximum depth is compiled
6bda09f9
YO
1165into perl, so changing it requires a custom build.
1166
542fa716
YO
1167The following shows how using negative indexing can make it
1168easier to embed recursive patterns inside of a C<qr//> construct
1169for later use:
1170
1171 my $parens = qr/(\((?:[^()]++|(?-1))*+\))/;
1172 if (/foo $parens \s+ + \s+ bar $parens/x) {
1173 # do something here...
1174 }
1175
81714fb9 1176B<Note> that this pattern does not behave the same way as the equivalent
0d017f4d 1177PCRE or Python construct of the same form. In Perl you can backtrack into
6bda09f9 1178a recursed group, in PCRE and Python the recursed into group is treated
542fa716
YO
1179as atomic. Also, modifiers are resolved at compile time, so constructs
1180like (?i:(?1)) or (?:(?i)(?1)) do not affect how the sub-pattern will
1181be processed.
6bda09f9 1182
894be9b7
YO
1183=item C<(?&NAME)>
1184X<(?&NAME)>
1185
0d017f4d
WL
1186Recurse to a named subpattern. Identical to C<(?PARNO)> except that the
1187parenthesis to recurse to is determined by name. If multiple parentheses have
894be9b7
YO
1188the same name, then it recurses to the leftmost.
1189
1190It is an error to refer to a name that is not declared somewhere in the
1191pattern.
1192
1f1031fe
YO
1193B<NOTE:> In order to make things easier for programmers with experience
1194with the Python or PCRE regex engines the pattern C<< (?P>NAME) >>
64c5a566 1195may be used instead of C<< (?&NAME) >>.
1f1031fe 1196
e2e6a0f1
YO
1197=item C<(?(condition)yes-pattern|no-pattern)>
1198X<(?()>
286f584a 1199
e2e6a0f1 1200=item C<(?(condition)yes-pattern)>
286f584a 1201
41ef34de
ML
1202Conditional expression. Matches C<yes-pattern> if C<condition> yields
1203a true value, matches C<no-pattern> otherwise. A missing pattern always
1204matches.
1205
1206C<(condition)> should be either an integer in
e2e6a0f1
YO
1207parentheses (which is valid if the corresponding pair of parentheses
1208matched), a look-ahead/look-behind/evaluate zero-width assertion, a
c27a5cfe 1209name in angle brackets or single quotes (which is valid if a group
e2e6a0f1
YO
1210with the given name matched), or the special symbol (R) (true when
1211evaluated inside of recursion or eval). Additionally the R may be
1212followed by a number, (which will be true when evaluated when recursing
1213inside of the appropriate group), or by C<&NAME>, in which case it will
1214be true only when evaluated during recursion in the named group.
1215
1216Here's a summary of the possible predicates:
1217
1218=over 4
1219
1220=item (1) (2) ...
1221
c27a5cfe 1222Checks if the numbered capturing group has matched something.
e2e6a0f1
YO
1223
1224=item (<NAME>) ('NAME')
1225
c27a5cfe 1226Checks if a group with the given name has matched something.
e2e6a0f1 1227
f01cd190
FC
1228=item (?=...) (?!...) (?<=...) (?<!...)
1229
1230Checks whether the pattern matches (or does not match, for the '!'
1231variants).
1232
e2e6a0f1
YO
1233=item (?{ CODE })
1234
f01cd190 1235Treats the return value of the code block as the condition.
e2e6a0f1
YO
1236
1237=item (R)
1238
1239Checks if the expression has been evaluated inside of recursion.
1240
1241=item (R1) (R2) ...
1242
1243Checks if the expression has been evaluated while executing directly
1244inside of the n-th capture group. This check is the regex equivalent of
1245
1246 if ((caller(0))[3] eq 'subname') { ... }
1247
1248In other words, it does not check the full recursion stack.
1249
1250=item (R&NAME)
1251
1252Similar to C<(R1)>, this predicate checks to see if we're executing
1253directly inside of the leftmost group with a given name (this is the same
1254logic used by C<(?&NAME)> to disambiguate). It does not check the full
1255stack, but only the name of the innermost active recursion.
1256
1257=item (DEFINE)
1258
1259In this case, the yes-pattern is never directly executed, and no
1260no-pattern is allowed. Similar in spirit to C<(?{0})> but more efficient.
1261See below for details.
1262
1263=back
1264
1265For example:
1266
1267 m{ ( \( )?
1268 [^()]+
1269 (?(1) \) )
1270 }x
1271
1272matches a chunk of non-parentheses, possibly included in parentheses
1273themselves.
1274
0b928c2f
FC
1275A special form is the C<(DEFINE)> predicate, which never executes its
1276yes-pattern directly, and does not allow a no-pattern. This allows one to
1277define subpatterns which will be executed only by the recursion mechanism.
e2e6a0f1
YO
1278This way, you can define a set of regular expression rules that can be
1279bundled into any pattern you choose.
1280
1281It is recommended that for this usage you put the DEFINE block at the
1282end of the pattern, and that you name any subpatterns defined within it.
1283
1284Also, it's worth noting that patterns defined this way probably will
1285not be as efficient, as the optimiser is not very clever about
1286handling them.
1287
1288An example of how this might be used is as follows:
1289
2bf803e2 1290 /(?<NAME>(?&NAME_PAT))(?<ADDR>(?&ADDRESS_PAT))
e2e6a0f1 1291 (?(DEFINE)
2bf803e2
YO
1292 (?<NAME_PAT>....)
1293 (?<ADRESS_PAT>....)
e2e6a0f1
YO
1294 )/x
1295
c27a5cfe
KW
1296Note that capture groups matched inside of recursion are not accessible
1297after the recursion returns, so the extra layer of capturing groups is
e2e6a0f1
YO
1298necessary. Thus C<$+{NAME_PAT}> would not be defined even though
1299C<$+{NAME}> would be.
286f584a 1300
c47ff5f1 1301=item C<< (?>pattern) >>
6bda09f9 1302X<backtrack> X<backtracking> X<atomic> X<possessive>
5a964f20 1303
19799a22
GS
1304An "independent" subexpression, one which matches the substring
1305that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 1306position, and it matches I<nothing other than this substring>. This
19799a22
GS
1307construct is useful for optimizations of what would otherwise be
1308"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc
IZ
1309It may also be useful in places where the "grab all you can, and do not
1310give anything back" semantic is desirable.
19799a22 1311
c47ff5f1 1312For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22
GS
1313(anchored at the beginning of string, as above) will match I<all>
1314characters C<a> at the beginning of string, leaving no C<a> for
1315C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
1316since the match of the subgroup C<a*> is influenced by the following
1317group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
1318C<a*ab> will match fewer characters than a standalone C<a*>, since
1319this makes the tail match.
1320
0b928c2f
FC
1321C<< (?>pattern) >> does not disable backtracking altogether once it has
1322matched. It is still possible to backtrack past the construct, but not
1323into it. So C<< ((?>a*)|(?>b*))ar >> will still match "bar".
1324
c47ff5f1 1325An effect similar to C<< (?>pattern) >> may be achieved by writing
0b928c2f
FC
1326C<(?=(pattern))\g{-1}>. This matches the same substring as a standalone
1327C<a+>, and the following C<\g{-1}> eats the matched string; it therefore
c47ff5f1 1328makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22
GS
1329(The difference between these two constructs is that the second one
1330uses a capturing group, thus shifting ordinals of backreferences
1331in the rest of a regular expression.)
1332
1333Consider this pattern:
c277df42 1334
871b0233 1335 m{ \(
e2e6a0f1 1336 (
f793d64a 1337 [^()]+ # x+
e2e6a0f1 1338 |
871b0233
IZ
1339 \( [^()]* \)
1340 )+
e2e6a0f1 1341 \)
871b0233 1342 }x
5a964f20 1343
19799a22
GS
1344That will efficiently match a nonempty group with matching parentheses
1345two levels deep or less. However, if there is no such group, it
1346will take virtually forever on a long string. That's because there
1347are so many different ways to split a long string into several
1348substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
1349to a subpattern of the above pattern. Consider how the pattern
1350above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
1351seconds, but that each extra letter doubles this time. This
1352exponential performance will make it appear that your program has
14218588 1353hung. However, a tiny change to this pattern
5a964f20 1354
e2e6a0f1
YO
1355 m{ \(
1356 (
f793d64a 1357 (?> [^()]+ ) # change x+ above to (?> x+ )
e2e6a0f1 1358 |
871b0233
IZ
1359 \( [^()]* \)
1360 )+
e2e6a0f1 1361 \)
871b0233 1362 }x
c277df42 1363
c47ff5f1 1364which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20
TC
1365this yourself would be a productive exercise), but finishes in a fourth
1366the time when used on a similar string with 1000000 C<a>s. Be aware,
0b928c2f
FC
1367however, that, when this construct is followed by a
1368quantifier, it currently triggers a warning message under
9f1b1f2d 1369the C<use warnings> pragma or B<-w> switch saying it
6bab786b 1370C<"matches null string many times in regex">.
c277df42 1371
c47ff5f1 1372On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 1373effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42
IZ
1374This was only 4 times slower on a string with 1000000 C<a>s.
1375
9da458fc
IZ
1376The "grab all you can, and do not give anything back" semantic is desirable
1377in many situations where on the first sight a simple C<()*> looks like
1378the correct solution. Suppose we parse text with comments being delimited
1379by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 1380its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc
IZ
1381the comment delimiter, because it may "give up" some whitespace if
1382the remainder of the pattern can be made to match that way. The correct
1383answer is either one of these:
1384
1385 (?>#[ \t]*)
1386 #[ \t]*(?![ \t])
1387
1388For example, to grab non-empty comments into $1, one should use either
1389one of these:
1390
1391 / (?> \# [ \t]* ) ( .+ ) /x;
1392 / \# [ \t]* ( [^ \t] .* ) /x;
1393
1394Which one you pick depends on which of these expressions better reflects
1395the above specification of comments.
1396
6bda09f9
YO
1397In some literature this construct is called "atomic matching" or
1398"possessive matching".
1399
b9b4dddf
YO
1400Possessive quantifiers are equivalent to putting the item they are applied
1401to inside of one of these constructs. The following equivalences apply:
1402
1403 Quantifier Form Bracketing Form
1404 --------------- ---------------
1405 PAT*+ (?>PAT*)
1406 PAT++ (?>PAT+)
1407 PAT?+ (?>PAT?)
1408 PAT{min,max}+ (?>PAT{min,max})
1409
e2e6a0f1
YO
1410=back
1411
1412=head2 Special Backtracking Control Verbs
1413
1414B<WARNING:> These patterns are experimental and subject to change or
0d017f4d 1415removal in a future version of Perl. Their usage in production code should
e2e6a0f1
YO
1416be noted to avoid problems during upgrades.
1417
1418These special patterns are generally of the form C<(*VERB:ARG)>. Unless
1419otherwise stated the ARG argument is optional; in some cases, it is
1420forbidden.
1421
1422Any pattern containing a special backtracking verb that allows an argument
e1020413 1423has the special behaviour that when executed it sets the current package's
5d458dd8
YO
1424C<$REGERROR> and C<$REGMARK> variables. When doing so the following
1425rules apply:
e2e6a0f1 1426
5d458dd8
YO
1427On failure, the C<$REGERROR> variable will be set to the ARG value of the
1428verb pattern, if the verb was involved in the failure of the match. If the
1429ARG part of the pattern was omitted, then C<$REGERROR> will be set to the
1430name of the last C<(*MARK:NAME)> pattern executed, or to TRUE if there was
1431none. Also, the C<$REGMARK> variable will be set to FALSE.
e2e6a0f1 1432
5d458dd8
YO
1433On a successful match, the C<$REGERROR> variable will be set to FALSE, and
1434the C<$REGMARK> variable will be set to the name of the last
1435C<(*MARK:NAME)> pattern executed. See the explanation for the
1436C<(*MARK:NAME)> verb below for more details.
e2e6a0f1 1437
5d458dd8 1438B<NOTE:> C<$REGERROR> and C<$REGMARK> are not magic variables like C<$1>
0b928c2f 1439and most other regex-related variables. They are not local to a scope, nor
5d458dd8
YO
1440readonly, but instead are volatile package variables similar to C<$AUTOLOAD>.
1441Use C<local> to localize changes to them to a specific scope if necessary.
e2e6a0f1
YO
1442
1443If a pattern does not contain a special backtracking verb that allows an
5d458dd8 1444argument, then C<$REGERROR> and C<$REGMARK> are not touched at all.
e2e6a0f1
YO
1445
1446=over 4
1447
1448=item Verbs that take an argument
1449
1450=over 4
1451
5d458dd8 1452=item C<(*PRUNE)> C<(*PRUNE:NAME)>
f7819f85 1453X<(*PRUNE)> X<(*PRUNE:NAME)>
54612592 1454
5d458dd8
YO
1455This zero-width pattern prunes the backtracking tree at the current point
1456when backtracked into on failure. Consider the pattern C<A (*PRUNE) B>,
1457where A and B are complex patterns. Until the C<(*PRUNE)> verb is reached,
1458A may backtrack as necessary to match. Once it is reached, matching
1459continues in B, which may also backtrack as necessary; however, should B
1460not match, then no further backtracking will take place, and the pattern
1461will fail outright at the current starting position.
54612592
YO
1462
1463The following example counts all the possible matching strings in a
1464pattern (without actually matching any of them).
1465
e2e6a0f1 1466 'aaab' =~ /a+b?(?{print "$&\n"; $count++})(*FAIL)/;
54612592
YO
1467 print "Count=$count\n";
1468
1469which produces:
1470
1471 aaab
1472 aaa
1473 aa
1474 a
1475 aab
1476 aa
1477 a
1478 ab
1479 a
1480 Count=9
1481
5d458dd8 1482If we add a C<(*PRUNE)> before the count like the following
54612592 1483
5d458dd8 1484 'aaab' =~ /a+b?(*PRUNE)(?{print "$&\n"; $count++})(*FAIL)/;
54612592
YO
1485 print "Count=$count\n";
1486
0b928c2f 1487we prevent backtracking and find the count of the longest matching string
353c6505 1488at each matching starting point like so:
54612592
YO
1489
1490 aaab
1491 aab
1492 ab
1493 Count=3
1494
5d458dd8 1495Any number of C<(*PRUNE)> assertions may be used in a pattern.
54612592 1496
5d458dd8
YO
1497See also C<< (?>pattern) >> and possessive quantifiers for other ways to
1498control backtracking. In some cases, the use of C<(*PRUNE)> can be
1499replaced with a C<< (?>pattern) >> with no functional difference; however,
1500C<(*PRUNE)> can be used to handle cases that cannot be expressed using a
1501C<< (?>pattern) >> alone.
54612592 1502
e2e6a0f1 1503
5d458dd8
YO
1504=item C<(*SKIP)> C<(*SKIP:NAME)>
1505X<(*SKIP)>
e2e6a0f1 1506
5d458dd8 1507This zero-width pattern is similar to C<(*PRUNE)>, except that on
e2e6a0f1 1508failure it also signifies that whatever text that was matched leading up
5d458dd8
YO
1509to the C<(*SKIP)> pattern being executed cannot be part of I<any> match
1510of this pattern. This effectively means that the regex engine "skips" forward
1511to this position on failure and tries to match again, (assuming that
1512there is sufficient room to match).
1513
1514The name of the C<(*SKIP:NAME)> pattern has special significance. If a
1515C<(*MARK:NAME)> was encountered while matching, then it is that position
1516which is used as the "skip point". If no C<(*MARK)> of that name was
1517encountered, then the C<(*SKIP)> operator has no effect. When used
1518without a name the "skip point" is where the match point was when
1519executing the (*SKIP) pattern.
1520
0b928c2f 1521Compare the following to the examples in C<(*PRUNE)>; note the string
24b23f37
YO
1522is twice as long:
1523
5d458dd8 1524 'aaabaaab' =~ /a+b?(*SKIP)(?{print "$&\n"; $count++})(*FAIL)/;
24b23f37
YO
1525 print "Count=$count\n";
1526
1527outputs
1528
1529 aaab
1530 aaab
1531 Count=2
1532
5d458dd8 1533Once the 'aaab' at the start of the string has matched, and the C<(*SKIP)>
353c6505 1534executed, the next starting point will be where the cursor was when the
5d458dd8
YO
1535C<(*SKIP)> was executed.
1536
5d458dd8
YO
1537=item C<(*MARK:NAME)> C<(*:NAME)>
1538X<(*MARK)> C<(*MARK:NAME)> C<(*:NAME)>
1539
1540This zero-width pattern can be used to mark the point reached in a string
1541when a certain part of the pattern has been successfully matched. This
1542mark may be given a name. A later C<(*SKIP)> pattern will then skip
1543forward to that point if backtracked into on failure. Any number of
b4222fa9 1544C<(*MARK)> patterns are allowed, and the NAME portion may be duplicated.
5d458dd8
YO
1545
1546In addition to interacting with the C<(*SKIP)> pattern, C<(*MARK:NAME)>
1547can be used to "label" a pattern branch, so that after matching, the
1548program can determine which branches of the pattern were involved in the
1549match.
1550
1551When a match is successful, the C<$REGMARK> variable will be set to the
1552name of the most recently executed C<(*MARK:NAME)> that was involved
1553in the match.
1554
1555This can be used to determine which branch of a pattern was matched
c27a5cfe 1556without using a separate capture group for each branch, which in turn
5d458dd8
YO
1557can result in a performance improvement, as perl cannot optimize
1558C</(?:(x)|(y)|(z))/> as efficiently as something like
1559C</(?:x(*MARK:x)|y(*MARK:y)|z(*MARK:z))/>.
1560
1561When a match has failed, and unless another verb has been involved in
1562failing the match and has provided its own name to use, the C<$REGERROR>
1563variable will be set to the name of the most recently executed
1564C<(*MARK:NAME)>.
1565
1566See C<(*SKIP)> for more details.
1567
b62d2d15
YO
1568As a shortcut C<(*MARK:NAME)> can be written C<(*:NAME)>.
1569
5d458dd8
YO
1570=item C<(*THEN)> C<(*THEN:NAME)>
1571
241e7389 1572This is similar to the "cut group" operator C<::> from Perl 6. Like
5d458dd8
YO
1573C<(*PRUNE)>, this verb always matches, and when backtracked into on
1574failure, it causes the regex engine to try the next alternation in the
1575innermost enclosing group (capturing or otherwise).
1576
1577Its name comes from the observation that this operation combined with the
1578alternation operator (C<|>) can be used to create what is essentially a
1579pattern-based if/then/else block:
1580
1581 ( COND (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ )
1582
1583Note that if this operator is used and NOT inside of an alternation then
1584it acts exactly like the C<(*PRUNE)> operator.
1585
1586 / A (*PRUNE) B /
1587
1588is the same as
1589
1590 / A (*THEN) B /
1591
1592but
1593
1594 / ( A (*THEN) B | C (*THEN) D ) /
1595
1596is not the same as
1597
1598 / ( A (*PRUNE) B | C (*PRUNE) D ) /
1599
1600as after matching the A but failing on the B the C<(*THEN)> verb will
1601backtrack and try C; but the C<(*PRUNE)> verb will simply fail.
24b23f37 1602
e2e6a0f1
YO
1603=item C<(*COMMIT)>
1604X<(*COMMIT)>
24b23f37 1605
241e7389 1606This is the Perl 6 "commit pattern" C<< <commit> >> or C<:::>. It's a
5d458dd8
YO
1607zero-width pattern similar to C<(*SKIP)>, except that when backtracked
1608into on failure it causes the match to fail outright. No further attempts
1609to find a valid match by advancing the start pointer will occur again.
1610For example,
24b23f37 1611
e2e6a0f1 1612 'aaabaaab' =~ /a+b?(*COMMIT)(?{print "$&\n"; $count++})(*FAIL)/;
24b23f37
YO
1613 print "Count=$count\n";
1614
1615outputs
1616
1617 aaab
1618 Count=1
1619
e2e6a0f1
YO
1620In other words, once the C<(*COMMIT)> has been entered, and if the pattern
1621does not match, the regex engine will not try any further matching on the
1622rest of the string.
c277df42 1623
e2e6a0f1 1624=back
9af228c6 1625
e2e6a0f1 1626=item Verbs without an argument
9af228c6
YO
1627
1628=over 4
1629
e2e6a0f1
YO
1630=item C<(*FAIL)> C<(*F)>
1631X<(*FAIL)> X<(*F)>
9af228c6 1632
e2e6a0f1
YO
1633This pattern matches nothing and always fails. It can be used to force the
1634engine to backtrack. It is equivalent to C<(?!)>, but easier to read. In
1635fact, C<(?!)> gets optimised into C<(*FAIL)> internally.
9af228c6 1636
e2e6a0f1 1637It is probably useful only when combined with C<(?{})> or C<(??{})>.
9af228c6 1638
e2e6a0f1
YO
1639=item C<(*ACCEPT)>
1640X<(*ACCEPT)>
9af228c6 1641
e2e6a0f1
YO
1642B<WARNING:> This feature is highly experimental. It is not recommended
1643for production code.
9af228c6 1644
e2e6a0f1
YO
1645This pattern matches nothing and causes the end of successful matching at
1646the point at which the C<(*ACCEPT)> pattern was encountered, regardless of
1647whether there is actually more to match in the string. When inside of a
0d017f4d 1648nested pattern, such as recursion, or in a subpattern dynamically generated
e2e6a0f1 1649via C<(??{})>, only the innermost pattern is ended immediately.
9af228c6 1650
c27a5cfe 1651If the C<(*ACCEPT)> is inside of capturing groups then the groups are
e2e6a0f1
YO
1652marked as ended at the point at which the C<(*ACCEPT)> was encountered.
1653For instance:
9af228c6 1654
e2e6a0f1 1655 'AB' =~ /(A (A|B(*ACCEPT)|C) D)(E)/x;
9af228c6 1656
e2e6a0f1 1657will match, and C<$1> will be C<AB> and C<$2> will be C<B>, C<$3> will not
0b928c2f 1658be set. If another branch in the inner parentheses was matched, such as in the
e2e6a0f1 1659string 'ACDE', then the C<D> and C<E> would have to be matched as well.
9af228c6
YO
1660
1661=back
c277df42 1662
a0d0e21e
LW
1663=back
1664
c07a80fd 1665=head2 Backtracking
d74e8afc 1666X<backtrack> X<backtracking>
c07a80fd 1667
35a734be
IZ
1668NOTE: This section presents an abstract approximation of regular
1669expression behavior. For a more rigorous (and complicated) view of
1670the rules involved in selecting a match among possible alternatives,
0d017f4d 1671see L<Combining RE Pieces>.
35a734be 1672
c277df42 1673A fundamental feature of regular expression matching involves the
5a964f20 1674notion called I<backtracking>, which is currently used (when needed)
0d017f4d 1675by all regular non-possessive expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc
IZ
1676C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
1677internally, but the general principle outlined here is valid.
c07a80fd
PP
1678
1679For a regular expression to match, the I<entire> regular expression must
1680match, not just part of it. So if the beginning of a pattern containing a
1681quantifier succeeds in a way that causes later parts in the pattern to
1682fail, the matching engine backs up and recalculates the beginning
1683part--that's why it's called backtracking.
1684
1685Here is an example of backtracking: Let's say you want to find the
1686word following "foo" in the string "Food is on the foo table.":
1687
1688 $_ = "Food is on the foo table.";
1689 if ( /\b(foo)\s+(\w+)/i ) {
f793d64a 1690 print "$2 follows $1.\n";
c07a80fd
PP
1691 }
1692
1693When the match runs, the first part of the regular expression (C<\b(foo)>)
1694finds a possible match right at the beginning of the string, and loads up
1695$1 with "Foo". However, as soon as the matching engine sees that there's
1696no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 1697mistake and starts over again one character after where it had the
c07a80fd
PP
1698tentative match. This time it goes all the way until the next occurrence
1699of "foo". The complete regular expression matches this time, and you get
1700the expected output of "table follows foo."
1701
1702Sometimes minimal matching can help a lot. Imagine you'd like to match
1703everything between "foo" and "bar". Initially, you write something
1704like this:
1705
1706 $_ = "The food is under the bar in the barn.";
1707 if ( /foo(.*)bar/ ) {
f793d64a 1708 print "got <$1>\n";
c07a80fd
PP
1709 }
1710
1711Which perhaps unexpectedly yields:
1712
1713 got <d is under the bar in the >
1714
1715That's because C<.*> was greedy, so you get everything between the
14218588 1716I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd
PP
1717to use minimal matching to make sure you get the text between a "foo"
1718and the first "bar" thereafter.
1719
1720 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
1721 got <d is under the >
1722
0d017f4d 1723Here's another example. Let's say you'd like to match a number at the end
b6e13d97 1724of a string, and you also want to keep the preceding part of the match.
c07a80fd
PP
1725So you write this:
1726
1727 $_ = "I have 2 numbers: 53147";
f793d64a
KW
1728 if ( /(.*)(\d*)/ ) { # Wrong!
1729 print "Beginning is <$1>, number is <$2>.\n";
c07a80fd
PP
1730 }
1731
1732That won't work at all, because C<.*> was greedy and gobbled up the
1733whole string. As C<\d*> can match on an empty string the complete
1734regular expression matched successfully.
1735
8e1088bc 1736 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd
PP
1737
1738Here are some variants, most of which don't work:
1739
1740 $_ = "I have 2 numbers: 53147";
1741 @pats = qw{
f793d64a
KW
1742 (.*)(\d*)
1743 (.*)(\d+)
1744 (.*?)(\d*)
1745 (.*?)(\d+)
1746 (.*)(\d+)$
1747 (.*?)(\d+)$
1748 (.*)\b(\d+)$
1749 (.*\D)(\d+)$
c07a80fd
PP
1750 };
1751
1752 for $pat (@pats) {
f793d64a
KW
1753 printf "%-12s ", $pat;
1754 if ( /$pat/ ) {
1755 print "<$1> <$2>\n";
1756 } else {
1757 print "FAIL\n";
1758 }
c07a80fd
PP
1759 }
1760
1761That will print out:
1762
1763 (.*)(\d*) <I have 2 numbers: 53147> <>
1764 (.*)(\d+) <I have 2 numbers: 5314> <7>
1765 (.*?)(\d*) <> <>
1766 (.*?)(\d+) <I have > <2>
1767 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
1768 (.*?)(\d+)$ <I have 2 numbers: > <53147>
1769 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
1770 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
1771
1772As you see, this can be a bit tricky. It's important to realize that a
1773regular expression is merely a set of assertions that gives a definition
1774of success. There may be 0, 1, or several different ways that the
1775definition might succeed against a particular string. And if there are
5a964f20
TC
1776multiple ways it might succeed, you need to understand backtracking to
1777know which variety of success you will achieve.
c07a80fd 1778
19799a22 1779When using look-ahead assertions and negations, this can all get even
8b19b778 1780trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd
PP
1781followed by "123". You might try to write that as
1782
871b0233 1783 $_ = "ABC123";
f793d64a
KW
1784 if ( /^\D*(?!123)/ ) { # Wrong!
1785 print "Yup, no 123 in $_\n";
871b0233 1786 }
c07a80fd
PP
1787
1788But that isn't going to match; at least, not the way you're hoping. It
1789claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 1790why that pattern matches, contrary to popular expectations:
c07a80fd 1791
4358a253
SS
1792 $x = 'ABC123';
1793 $y = 'ABC445';
c07a80fd 1794
4358a253
SS
1795 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/;
1796 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/;
c07a80fd 1797
4358a253
SS
1798 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/;
1799 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/;
c07a80fd
PP
1800
1801This prints
1802
1803 2: got ABC
1804 3: got AB
1805 4: got ABC
1806
5f05dabc 1807You might have expected test 3 to fail because it seems to a more
c07a80fd
PP
1808general purpose version of test 1. The important difference between
1809them is that test 3 contains a quantifier (C<\D*>) and so can use
1810backtracking, whereas test 1 will not. What's happening is
1811that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 1812non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 1813let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 1814fail.
14218588 1815
c07a80fd 1816The search engine will initially match C<\D*> with "ABC". Then it will
0b928c2f 1817try to match C<(?!123)> with "123", which fails. But because
c07a80fd
PP
1818a quantifier (C<\D*>) has been used in the regular expression, the
1819search engine can backtrack and retry the match differently
54310121 1820in the hope of matching the complete regular expression.
c07a80fd 1821
5a964f20
TC
1822The pattern really, I<really> wants to succeed, so it uses the
1823standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 1824time. Now there's indeed something following "AB" that is not
14218588 1825"123". It's "C123", which suffices.
c07a80fd 1826
14218588
GS
1827We can deal with this by using both an assertion and a negation.
1828We'll say that the first part in $1 must be followed both by a digit
1829and by something that's not "123". Remember that the look-aheads
1830are zero-width expressions--they only look, but don't consume any
1831of the string in their match. So rewriting this way produces what
c07a80fd
PP
1832you'd expect; that is, case 5 will fail, but case 6 succeeds:
1833
4358a253
SS
1834 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/;
1835 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/;
c07a80fd
PP
1836
1837 6: got ABC
1838
5a964f20 1839In other words, the two zero-width assertions next to each other work as though
19799a22 1840they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd
PP
1841matches only if you're at the beginning of the line AND the end of the
1842line simultaneously. The deeper underlying truth is that juxtaposition in
1843regular expressions always means AND, except when you write an explicit OR
1844using the vertical bar. C</ab/> means match "a" AND (then) match "b",
1845although the attempted matches are made at different positions because "a"
1846is not a zero-width assertion, but a one-width assertion.
1847
0d017f4d 1848B<WARNING>: Particularly complicated regular expressions can take
14218588 1849exponential time to solve because of the immense number of possible
0d017f4d 1850ways they can use backtracking to try for a match. For example, without
9da458fc
IZ
1851internal optimizations done by the regular expression engine, this will
1852take a painfully long time to run:
c07a80fd 1853
e1901655
IZ
1854 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
1855
1856And if you used C<*>'s in the internal groups instead of limiting them
1857to 0 through 5 matches, then it would take forever--or until you ran
1858out of stack space. Moreover, these internal optimizations are not
1859always applicable. For example, if you put C<{0,5}> instead of C<*>
1860on the external group, no current optimization is applicable, and the
1861match takes a long time to finish.
c07a80fd 1862
9da458fc
IZ
1863A powerful tool for optimizing such beasts is what is known as an
1864"independent group",
96090e4f 1865which does not backtrack (see L</C<< (?>pattern) >>>). Note also that
9da458fc 1866zero-length look-ahead/look-behind assertions will not backtrack to make
5d458dd8 1867the tail match, since they are in "logical" context: only
14218588 1868whether they match is considered relevant. For an example
9da458fc 1869where side-effects of look-ahead I<might> have influenced the
96090e4f 1870following match, see L</C<< (?>pattern) >>>.
c277df42 1871
a0d0e21e 1872=head2 Version 8 Regular Expressions
d74e8afc 1873X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
a0d0e21e 1874
5a964f20 1875In case you're not familiar with the "regular" Version 8 regex
a0d0e21e
LW
1876routines, here are the pattern-matching rules not described above.
1877
54310121 1878Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 1879with a special meaning described here or above. You can cause
5a964f20 1880characters that normally function as metacharacters to be interpreted
5f05dabc 1881literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
0d017f4d
WL
1882character; "\\" matches a "\"). This escape mechanism is also required
1883for the character used as the pattern delimiter.
1884
1885A series of characters matches that series of characters in the target
0b928c2f 1886string, so the pattern C<blurfl> would match "blurfl" in the target
0d017f4d 1887string.
a0d0e21e
LW
1888
1889You can specify a character class, by enclosing a list of characters
5d458dd8 1890in C<[]>, which will match any character from the list. If the
a0d0e21e 1891first character after the "[" is "^", the class matches any character not
14218588 1892in the list. Within a list, the "-" character specifies a
5a964f20 1893range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2
GS
1894inclusive. If you want either "-" or "]" itself to be a member of a
1895class, put it at the start of the list (possibly after a "^"), or
1896escape it with a backslash. "-" is also taken literally when it is
1897at the end of the list, just before the closing "]". (The
84850974
DD
1898following all specify the same class of three characters: C<[-az]>,
1899C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
5d458dd8
YO
1900specifies a class containing twenty-six characters, even on EBCDIC-based
1901character sets.) Also, if you try to use the character
1902classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1903a range, the "-" is understood literally.
a0d0e21e 1904
8ada0baa
JH
1905Note also that the whole range idea is rather unportable between
1906character sets--and even within character sets they may cause results
1907you probably didn't expect. A sound principle is to use only ranges
0d017f4d 1908that begin from and end at either alphabetics of equal case ([a-e],
8ada0baa
JH
1909[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1910spell out the character sets in full.
1911
54310121 1912Characters may be specified using a metacharacter syntax much like that
a0d0e21e
LW
1913used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1914"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
dc0d9c48 1915of three octal digits, matches the character whose coded character set value
5d458dd8 1916is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
dc0d9c48 1917matches the character whose ordinal is I<nn>. The expression \cI<x>
5d458dd8 1918matches the character control-I<x>. Finally, the "." metacharacter
fb55449c 1919matches any character except "\n" (unless you use C</s>).
a0d0e21e
LW
1920
1921You can specify a series of alternatives for a pattern using "|" to
1922separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1923or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e 1924first alternative includes everything from the last pattern delimiter
0b928c2f 1925("(", "(?:", etc. or the beginning of the pattern) up to the first "|", and
a0d0e21e 1926the last alternative contains everything from the last "|" to the next
0b928c2f 1927closing pattern delimiter. That's why it's common practice to include
14218588 1928alternatives in parentheses: to minimize confusion about where they
a3cb178b
GS
1929start and end.
1930
5a964f20 1931Alternatives are tried from left to right, so the first
a3cb178b
GS
1932alternative found for which the entire expression matches, is the one that
1933is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1934example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b
GS
1935part will match, as that is the first alternative tried, and it successfully
1936matches the target string. (This might not seem important, but it is
1937important when you are capturing matched text using parentheses.)
1938
5a964f20 1939Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1940so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1941
14218588
GS
1942Within a pattern, you may designate subpatterns for later reference
1943by enclosing them in parentheses, and you may refer back to the
1944I<n>th subpattern later in the pattern using the metacharacter
0b928c2f 1945\I<n> or \gI<n>. Subpatterns are numbered based on the left to right order
14218588
GS
1946of their opening parenthesis. A backreference matches whatever
1947actually matched the subpattern in the string being examined, not
d8b950dc 1948the rules for that subpattern. Therefore, C<(0|0x)\d*\s\g1\d*> will
14218588
GS
1949match "0x1234 0x4321", but not "0x1234 01234", because subpattern
19501 matched "0x", even though the rule C<0|0x> could potentially match
1951the leading 0 in the second number.
cb1a09d0 1952
0d017f4d 1953=head2 Warning on \1 Instead of $1
cb1a09d0 1954
5a964f20 1955Some people get too used to writing things like:
cb1a09d0
AD
1956
1957 $pattern =~ s/(\W)/\\\1/g;
1958
3ff1c45a
KW
1959This is grandfathered (for \1 to \9) for the RHS of a substitute to avoid
1960shocking the
cb1a09d0 1961B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1962PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0
AD
1963the usual double-quoted string means a control-A. The customary Unix
1964meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1965of doing that, you get yourself into trouble if you then add an C</e>
1966modifier.
1967
f793d64a 1968 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0
AD
1969
1970Or if you try to do
1971
1972 s/(\d+)/\1000/;
1973
1974You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1975C<${1}000>. The operation of interpolation should not be confused
cb1a09d0
AD
1976with the operation of matching a backreference. Certainly they mean two
1977different things on the I<left> side of the C<s///>.
9fa51da4 1978
0d017f4d 1979=head2 Repeated Patterns Matching a Zero-length Substring
c84d73f1 1980
19799a22 1981B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1
IZ
1982
1983Regular expressions provide a terse and powerful programming language. As
1984with most other power tools, power comes together with the ability
1985to wreak havoc.
1986
1987A common abuse of this power stems from the ability to make infinite
628afcb5 1988loops using regular expressions, with something as innocuous as:
c84d73f1
IZ
1989
1990 'foo' =~ m{ ( o? )* }x;
1991
0d017f4d 1992The C<o?> matches at the beginning of C<'foo'>, and since the position
c84d73f1 1993in the string is not moved by the match, C<o?> would match again and again
527e91da 1994because of the C<*> quantifier. Another common way to create a similar cycle
c84d73f1
IZ
1995is with the looping modifier C<//g>:
1996
1997 @matches = ( 'foo' =~ m{ o? }xg );
1998
1999or
2000
2001 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
2002
2003or the loop implied by split().
2004
2005However, long experience has shown that many programming tasks may
14218588
GS
2006be significantly simplified by using repeated subexpressions that
2007may match zero-length substrings. Here's a simple example being:
c84d73f1 2008
f793d64a 2009 @chars = split //, $string; # // is not magic in split
c84d73f1
IZ
2010 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
2011
9da458fc 2012Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1 2013the infinite loop>. The rules for this are different for lower-level
527e91da 2014loops given by the greedy quantifiers C<*+{}>, and for higher-level
c84d73f1
IZ
2015ones like the C</g> modifier or split() operator.
2016
19799a22
GS
2017The lower-level loops are I<interrupted> (that is, the loop is
2018broken) when Perl detects that a repeated expression matched a
2019zero-length substring. Thus
c84d73f1
IZ
2020
2021 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
2022
5d458dd8 2023is made equivalent to
c84d73f1 2024
0b928c2f
FC
2025 m{ (?: NON_ZERO_LENGTH )* (?: ZERO_LENGTH )? }x;
2026
2027For example, this program
2028
2029 #!perl -l
2030 "aaaaab" =~ /
2031 (?:
2032 a # non-zero
2033 | # or
2034 (?{print "hello"}) # print hello whenever this
2035 # branch is tried
2036 (?=(b)) # zero-width assertion
2037 )* # any number of times
2038 /x;
2039 print $&;
2040 print $1;
c84d73f1 2041
0b928c2f
FC
2042prints
2043
2044 hello
2045 aaaaa
2046 b
2047
2048Notice that "hello" is only printed once, as when Perl sees that the sixth
2049iteration of the outermost C<(?:)*> matches a zero-length string, it stops
2050the C<*>.
2051
2052The higher-level loops preserve an additional state between iterations:
5d458dd8 2053whether the last match was zero-length. To break the loop, the following
c84d73f1 2054match after a zero-length match is prohibited to have a length of zero.
5d458dd8 2055This prohibition interacts with backtracking (see L<"Backtracking">),
c84d73f1
IZ
2056and so the I<second best> match is chosen if the I<best> match is of
2057zero length.
2058
19799a22 2059For example:
c84d73f1
IZ
2060
2061 $_ = 'bar';
2062 s/\w??/<$&>/g;
2063
20fb949f 2064results in C<< <><b><><a><><r><> >>. At each position of the string the best
5d458dd8 2065match given by non-greedy C<??> is the zero-length match, and the I<second
c84d73f1
IZ
2066best> match is what is matched by C<\w>. Thus zero-length matches
2067alternate with one-character-long matches.
2068
5d458dd8 2069Similarly, for repeated C<m/()/g> the second-best match is the match at the
c84d73f1
IZ
2070position one notch further in the string.
2071
19799a22 2072The additional state of being I<matched with zero-length> is associated with
c84d73f1 2073the matched string, and is reset by each assignment to pos().
9da458fc
IZ
2074Zero-length matches at the end of the previous match are ignored
2075during C<split>.
c84d73f1 2076
0d017f4d 2077=head2 Combining RE Pieces
35a734be
IZ
2078
2079Each of the elementary pieces of regular expressions which were described
2080before (such as C<ab> or C<\Z>) could match at most one substring
2081at the given position of the input string. However, in a typical regular
2082expression these elementary pieces are combined into more complicated
0b928c2f 2083patterns using combining operators C<ST>, C<S|T>, C<S*> etc.
35a734be
IZ
2084(in these examples C<S> and C<T> are regular subexpressions).
2085
2086Such combinations can include alternatives, leading to a problem of choice:
2087if we match a regular expression C<a|ab> against C<"abc">, will it match
2088substring C<"a"> or C<"ab">? One way to describe which substring is
2089actually matched is the concept of backtracking (see L<"Backtracking">).
2090However, this description is too low-level and makes you think
2091in terms of a particular implementation.
2092
2093Another description starts with notions of "better"/"worse". All the
2094substrings which may be matched by the given regular expression can be
2095sorted from the "best" match to the "worst" match, and it is the "best"
2096match which is chosen. This substitutes the question of "what is chosen?"
2097by the question of "which matches are better, and which are worse?".
2098
2099Again, for elementary pieces there is no such question, since at most
2100one match at a given position is possible. This section describes the
2101notion of better/worse for combining operators. In the description
2102below C<S> and C<T> are regular subexpressions.
2103
13a2d996 2104=over 4
35a734be
IZ
2105
2106=item C<ST>
2107
2108Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
2109substrings which can be matched by C<S>, C<B> and C<B'> are substrings
5d458dd8 2110which can be matched by C<T>.
35a734be 2111
0b928c2f 2112If C<A> is a better match for C<S> than C<A'>, C<AB> is a better
35a734be
IZ
2113match than C<A'B'>.
2114
2115If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
0b928c2f 2116C<B> is a better match for C<T> than C<B'>.
35a734be
IZ
2117
2118=item C<S|T>
2119
2120When C<S> can match, it is a better match than when only C<T> can match.
2121
2122Ordering of two matches for C<S> is the same as for C<S>. Similar for
2123two matches for C<T>.
2124
2125=item C<S{REPEAT_COUNT}>
2126
2127Matches as C<SSS...S> (repeated as many times as necessary).
2128
2129=item C<S{min,max}>
2130
2131Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
2132
2133=item C<S{min,max}?>
2134
2135Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
2136
2137=item C<S?>, C<S*>, C<S+>
2138
2139Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
2140
2141=item C<S??>, C<S*?>, C<S+?>
2142
2143Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
2144
c47ff5f1 2145=item C<< (?>S) >>
35a734be
IZ
2146
2147Matches the best match for C<S> and only that.
2148
2149=item C<(?=S)>, C<(?<=S)>
2150
2151Only the best match for C<S> is considered. (This is important only if
2152C<S> has capturing parentheses, and backreferences are used somewhere
2153else in the whole regular expression.)
2154
2155=item C<(?!S)>, C<(?<!S)>
2156
2157For this grouping operator there is no need to describe the ordering, since
2158only whether or not C<S> can match is important.
2159
6bda09f9 2160=item C<(??{ EXPR })>, C<(?PARNO)>
35a734be
IZ
2161
2162The ordering is the same as for the regular expression which is
c27a5cfe 2163the result of EXPR, or the pattern contained by capture group PARNO.
35a734be
IZ
2164
2165=item C<(?(condition)yes-pattern|no-pattern)>
2166
2167Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
2168already determined. The ordering of the matches is the same as for the
2169chosen subexpression.
2170
2171=back
2172
2173The above recipes describe the ordering of matches I<at a given position>.
2174One more rule is needed to understand how a match is determined for the
2175whole regular expression: a match at an earlier position is always better
2176than a match at a later position.
2177
0d017f4d 2178=head2 Creating Custom RE Engines
c84d73f1 2179
0b928c2f
FC
2180As of Perl 5.10.0, one can create custom regular expression engines. This
2181is not for the faint of heart, as they have to plug in at the C level. See
2182L<perlreapi> for more details.
2183
2184As an alternative, overloaded constants (see L<overload>) provide a simple
2185way to extend the functionality of the RE engine, by substituting one
2186pattern for another.
c84d73f1
IZ
2187
2188Suppose that we want to enable a new RE escape-sequence C<\Y|> which
0d017f4d 2189matches at a boundary between whitespace characters and non-whitespace
c84d73f1
IZ
2190characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
2191at these positions, so we want to have each C<\Y|> in the place of the
2192more complicated version. We can create a module C<customre> to do
2193this:
2194
2195 package customre;
2196 use overload;
2197
2198 sub import {
2199 shift;
2200 die "No argument to customre::import allowed" if @_;
2201 overload::constant 'qr' => \&convert;
2202 }
2203
2204 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
2205
580a9fe1
RGS
2206 # We must also take care of not escaping the legitimate \\Y|
2207 # sequence, hence the presence of '\\' in the conversion rules.
5d458dd8 2208 my %rules = ( '\\' => '\\\\',
f793d64a 2209 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
c84d73f1
IZ
2210 sub convert {
2211 my $re = shift;
5d458dd8 2212 $re =~ s{
c84d73f1
IZ
2213 \\ ( \\ | Y . )
2214 }
5d458dd8 2215 { $rules{$1} or invalid($re,$1) }sgex;
c84d73f1
IZ
2216 return $re;
2217 }
2218
2219Now C<use customre> enables the new escape in constant regular
2220expressions, i.e., those without any runtime variable interpolations.
2221As documented in L<overload>, this conversion will work only over
2222literal parts of regular expressions. For C<\Y|$re\Y|> the variable
2223part of this regular expression needs to be converted explicitly
2224(but only if the special meaning of C<\Y|> should be enabled inside $re):
2225
2226 use customre;
2227 $re = <>;
2228 chomp $re;
2229 $re = customre::convert $re;
2230 /\Y|$re\Y|/;
2231
0b928c2f 2232=head2 PCRE/Python Support
1f1031fe 2233
0b928c2f 2234As of Perl 5.10.0, Perl supports several Python/PCRE-specific extensions
1f1031fe 2235to the regex syntax. While Perl programmers are encouraged to use the
0b928c2f 2236Perl-specific syntax, the following are also accepted:
1f1031fe
YO
2237
2238=over 4
2239
ae5648b3 2240=item C<< (?PE<lt>NAMEE<gt>pattern) >>
1f1031fe 2241
c27a5cfe 2242Define a named capture group. Equivalent to C<< (?<NAME>pattern) >>.
1f1031fe
YO
2243
2244=item C<< (?P=NAME) >>
2245
c27a5cfe 2246Backreference to a named capture group. Equivalent to C<< \g{NAME} >>.
1f1031fe
YO
2247
2248=item C<< (?P>NAME) >>
2249
c27a5cfe 2250Subroutine call to a named capture group. Equivalent to C<< (?&NAME) >>.
1f1031fe 2251
ee9b8eae 2252=back
1f1031fe 2253
19799a22
GS
2254=head1 BUGS
2255
0b928c2f 2256There are numerous problems with case-insensitive matching of characters
78288b8e
KW
2257outside the ASCII range, especially with those whose folds are multiple
2258characters, such as ligatures like C<LATIN SMALL LIGATURE FF>.
2259
0b928c2f 2260In a bracketed character class with case-insensitive matching, ranges only work
f253210b
KW
2261for ASCII characters. For example,
2262C<m/[\N{CYRILLIC CAPITAL LETTER A}-\N{CYRILLIC CAPITAL LETTER YA}]/i>
2263doesn't match all the Russian upper and lower case letters.
2264
88c9975e
KW
2265Many regular expression constructs don't work on EBCDIC platforms.
2266
9da458fc
IZ
2267This document varies from difficult to understand to completely
2268and utterly opaque. The wandering prose riddled with jargon is
2269hard to fathom in several places.
2270
2271This document needs a rewrite that separates the tutorial content
2272from the reference content.
19799a22
GS
2273
2274=head1 SEE ALSO
9fa51da4 2275
91e0c79e
MJD
2276L<perlrequick>.
2277
2278L<perlretut>.
2279
9b599b2a
GS
2280L<perlop/"Regexp Quote-Like Operators">.
2281
1e66bd83
PP
2282L<perlop/"Gory details of parsing quoted constructs">.
2283
14218588
GS
2284L<perlfaq6>.
2285
9b599b2a
GS
2286L<perlfunc/pos>.
2287
2288L<perllocale>.
2289
fb55449c
JH
2290L<perlebcdic>.
2291
14218588
GS
2292I<Mastering Regular Expressions> by Jeffrey Friedl, published
2293by O'Reilly and Associates.