This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Fix a few pod nits in perlre.
[perl5.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
d74e8afc 2X<regular expression> X<regex> X<regexp>
a0d0e21e
LW
3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
91e0c79e
MJD
8This page describes the syntax of regular expressions in Perl.
9
cc46d5f2 10If you haven't used regular expressions before, a quick-start
91e0c79e
MJD
11introduction is available in L<perlrequick>, and a longer tutorial
12introduction is available in L<perlretut>.
13
14For reference on how regular expressions are used in matching
15operations, plus various examples of the same, see discussions of
16C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
17Operators">.
cb1a09d0 18
19799a22 19Matching operations can have various modifiers. Modifiers
5a964f20 20that relate to the interpretation of the regular expression inside
19799a22
GS
21are listed below. Modifiers that alter the way a regular expression
22is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 23L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 24
55497cff
PP
25=over 4
26
27=item i
d74e8afc
ITB
28X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
29X<regular expression, case-insensitive>
55497cff
PP
30
31Do case-insensitive pattern matching.
32
a034a98d
DD
33If C<use locale> is in effect, the case map is taken from the current
34locale. See L<perllocale>.
35
54310121 36=item m
d74e8afc 37X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
55497cff
PP
38
39Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 40the start or end of the string to matching the start or end of any
7f761169 41line anywhere within the string.
55497cff 42
54310121 43=item s
d74e8afc
ITB
44X</s> X<regex, single-line> X<regexp, single-line>
45X<regular expression, single-line>
55497cff
PP
46
47Treat string as single line. That is, change "." to match any character
19799a22 48whatsoever, even a newline, which normally it would not match.
55497cff 49
f02c194e 50Used together, as /ms, they let the "." match any character whatsoever,
fb55449c 51while still allowing "^" and "$" to match, respectively, just after
19799a22 52and just before newlines within the string.
7b8d334a 53
54310121 54=item x
d74e8afc 55X</x>
55497cff
PP
56
57Extend your pattern's legibility by permitting whitespace and comments.
58
59=back
a0d0e21e
LW
60
61These are usually written as "the C</x> modifier", even though the delimiter
14218588 62in question might not really be a slash. Any of these
a0d0e21e 63modifiers may also be embedded within the regular expression itself using
14218588 64the C<(?...)> construct. See below.
a0d0e21e 65
4633a7c4 66The C</x> modifier itself needs a little more explanation. It tells
55497cff
PP
67the regular expression parser to ignore whitespace that is neither
68backslashed nor within a character class. You can use this to break up
4633a7c4 69your regular expression into (slightly) more readable parts. The C<#>
54310121 70character is also treated as a metacharacter introducing a comment,
55497cff 71just as in ordinary Perl code. This also means that if you want real
14218588 72whitespace or C<#> characters in the pattern (outside a character
f9a3ff1a
RGS
73class, where they are unaffected by C</x>), then you'll either have to
74escape them (using backslashes or C<\Q...\E>) or encode them using octal
8933a740
RGS
75or hex escapes. Taken together, these features go a long way towards
76making Perl's regular expressions more readable. Note that you have to
77be careful not to include the pattern delimiter in the comment--perl has
78no way of knowing you did not intend to close the pattern early. See
79the C-comment deletion code in L<perlop>. Also note that anything inside
1031e5db 80a C<\Q...\E> stays unaffected by C</x>.
d74e8afc 81X</x>
a0d0e21e
LW
82
83=head2 Regular Expressions
84
19799a22 85The patterns used in Perl pattern matching derive from supplied in
14218588 86the Version 8 regex routines. (The routines are derived
19799a22
GS
87(distantly) from Henry Spencer's freely redistributable reimplementation
88of the V8 routines.) See L<Version 8 Regular Expressions> for
89details.
a0d0e21e
LW
90
91In particular the following metacharacters have their standard I<egrep>-ish
92meanings:
d74e8afc
ITB
93X<metacharacter>
94X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
95
a0d0e21e 96
54310121 97 \ Quote the next metacharacter
a0d0e21e
LW
98 ^ Match the beginning of the line
99 . Match any character (except newline)
c07a80fd 100 $ Match the end of the line (or before newline at the end)
a0d0e21e
LW
101 | Alternation
102 () Grouping
103 [] Character class
104
14218588
GS
105By default, the "^" character is guaranteed to match only the
106beginning of the string, the "$" character only the end (or before the
107newline at the end), and Perl does certain optimizations with the
a0d0e21e
LW
108assumption that the string contains only one line. Embedded newlines
109will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 110string as a multi-line buffer, such that the "^" will match after any
a0d0e21e
LW
111newline within the string, and "$" will match before any newline. At the
112cost of a little more overhead, you can do this by using the /m modifier
113on the pattern match operator. (Older programs did this by setting C<$*>,
f02c194e 114but this practice has been removed in perl 5.9.)
d74e8afc 115X<^> X<$> X</m>
a0d0e21e 116
14218588 117To simplify multi-line substitutions, the "." character never matches a
55497cff 118newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 119the string is a single line--even if it isn't.
d74e8afc 120X<.> X</s>
a0d0e21e
LW
121
122The following standard quantifiers are recognized:
d74e8afc 123X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
a0d0e21e
LW
124
125 * Match 0 or more times
126 + Match 1 or more times
127 ? Match 1 or 0 times
128 {n} Match exactly n times
129 {n,} Match at least n times
130 {n,m} Match at least n but not more than m times
131
132(If a curly bracket occurs in any other context, it is treated
b975c076
JL
133as a regular character. In particular, the lower bound
134is not optional.) The "*" modifier is equivalent to C<{0,}>, the "+"
25f94b33 135modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited
9c79236d
GS
136to integral values less than a preset limit defined when perl is built.
137This is usually 32766 on the most common platforms. The actual limit can
138be seen in the error message generated by code such as this:
139
820475bd 140 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 141
54310121
PP
142By default, a quantified subpattern is "greedy", that is, it will match as
143many times as possible (given a particular starting location) while still
144allowing the rest of the pattern to match. If you want it to match the
145minimum number of times possible, follow the quantifier with a "?". Note
146that the meanings don't change, just the "greediness":
d74e8afc
ITB
147X<metacharacter> X<greedy> X<greedyness>
148X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{n,m}?>
a0d0e21e
LW
149
150 *? Match 0 or more times
151 +? Match 1 or more times
152 ?? Match 0 or 1 time
153 {n}? Match exactly n times
154 {n,}? Match at least n times
155 {n,m}? Match at least n but not more than m times
156
5f05dabc 157Because patterns are processed as double quoted strings, the following
a0d0e21e 158also work:
d74e8afc
ITB
159X<\t> X<\n> X<\r> X<\f> X<\a> X<\l> X<\u> X<\L> X<\U> X<\E> X<\Q>
160X<\0> X<\c> X<\N> X<\x>
a0d0e21e 161
0f36ee90
PP
162 \t tab (HT, TAB)
163 \n newline (LF, NL)
164 \r return (CR)
165 \f form feed (FF)
166 \a alarm (bell) (BEL)
167 \e escape (think troff) (ESC)
cb1a09d0
AD
168 \033 octal char (think of a PDP-11)
169 \x1B hex char
a0ed51b3 170 \x{263a} wide hex char (Unicode SMILEY)
a0d0e21e 171 \c[ control char
4a2d328f 172 \N{name} named char
cb1a09d0
AD
173 \l lowercase next char (think vi)
174 \u uppercase next char (think vi)
175 \L lowercase till \E (think vi)
176 \U uppercase till \E (think vi)
177 \E end case modification (think vi)
5a964f20 178 \Q quote (disable) pattern metacharacters till \E
a0d0e21e 179
a034a98d 180If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u>
423cee85 181and C<\U> is taken from the current locale. See L<perllocale>. For
4a2d328f 182documentation of C<\N{name}>, see L<charnames>.
a034a98d 183
1d2dff63
GS
184You cannot include a literal C<$> or C<@> within a C<\Q> sequence.
185An unescaped C<$> or C<@> interpolates the corresponding variable,
186while escaping will cause the literal string C<\$> to be matched.
187You'll need to write something like C<m/\Quser\E\@\Qhost/>.
188
a0d0e21e 189In addition, Perl defines the following:
d74e8afc
ITB
190X<metacharacter>
191X<\w> X<\W> X<\s> X<\S> X<\d> X<\D> X<\X> X<\p> X<\P> X<\C>
192X<word> X<whitespace>
a0d0e21e
LW
193
194 \w Match a "word" character (alphanumeric plus "_")
36bbe248 195 \W Match a non-"word" character
a0d0e21e
LW
196 \s Match a whitespace character
197 \S Match a non-whitespace character
198 \d Match a digit character
199 \D Match a non-digit character
a0ed51b3
LW
200 \pP Match P, named property. Use \p{Prop} for longer names.
201 \PP Match non-P
f244e06d 202 \X Match eXtended Unicode "combining character sequence",
07698885 203 equivalent to (?:\PM\pM*)
72ff2908 204 \C Match a single C char (octet) even under Unicode.
07698885 205 NOTE: breaks up characters into their UTF-8 bytes,
72ff2908 206 so you may end up with malformed pieces of UTF-8.
f14c76ed 207 Unsupported in lookbehind.
a0d0e21e 208
08ce8fc6
JH
209A C<\w> matches a single alphanumeric character (an alphabetic
210character, or a decimal digit) or C<_>, not a whole word. Use C<\w+>
211to match a string of Perl-identifier characters (which isn't the same
212as matching an English word). If C<use locale> is in effect, the list
213of alphabetic characters generated by C<\w> is taken from the current
214locale. See L<perllocale>. You may use C<\w>, C<\W>, C<\s>, C<\S>,
1209ba90 215C<\d>, and C<\D> within character classes, but if you try to use them
08ce8fc6
JH
216as endpoints of a range, that's not a range, the "-" is understood
217literally. If Unicode is in effect, C<\s> matches also "\x{85}",
218"\x{2028}, and "\x{2029}", see L<perlunicode> for more details about
491fd90a 219C<\pP>, C<\PP>, and C<\X>, and L<perluniintro> about Unicode in general.
fa11829f 220You can define your own C<\p> and C<\P> properties, see L<perlunicode>.
d74e8afc 221X<\w> X<\W> X<word>
a0d0e21e 222
b8c5462f 223The POSIX character class syntax
d74e8afc 224X<character class>
b8c5462f 225
820475bd 226 [:class:]
b8c5462f 227
5496314a
SP
228is also available. Note that the C<[> and C<]> braces are I<literal>;
229they must always be used within a character class expression.
230
231 # this is correct:
232 $string =~ /[[:alpha:]]/;
233
234 # this is not, and will generate a warning:
235 $string =~ /[:alpha:]/;
236
237The available classes and their backslash equivalents (if available) are
238as follows:
d74e8afc
ITB
239X<character class>
240X<alpha> X<alnum> X<ascii> X<blank> X<cntrl> X<digit> X<graph>
241X<lower> X<print> X<punct> X<space> X<upper> X<word> X<xdigit>
b8c5462f
JH
242
243 alpha
244 alnum
245 ascii
aaa51d5e 246 blank [1]
b8c5462f
JH
247 cntrl
248 digit \d
249 graph
250 lower
251 print
252 punct
aaa51d5e 253 space \s [2]
b8c5462f 254 upper
aaa51d5e 255 word \w [3]
b8c5462f
JH
256 xdigit
257
07698885
RGS
258=over
259
260=item [1]
261
b432a672 262A GNU extension equivalent to C<[ \t]>, "all horizontal whitespace".
07698885
RGS
263
264=item [2]
265
266Not exactly equivalent to C<\s> since the C<[[:space:]]> includes
b432a672 267also the (very rare) "vertical tabulator", "\ck", chr(11).
07698885
RGS
268
269=item [3]
270
08ce8fc6 271A Perl extension, see above.
07698885
RGS
272
273=back
aaa51d5e 274
26b44a0a 275For example use C<[:upper:]> to match all the uppercase characters.
aaa51d5e
JF
276Note that the C<[]> are part of the C<[::]> construct, not part of the
277whole character class. For example:
b8c5462f 278
820475bd 279 [01[:alpha:]%]
b8c5462f 280
593df60c 281matches zero, one, any alphabetic character, and the percentage sign.
b8c5462f 282
72ff2908
JH
283The following equivalences to Unicode \p{} constructs and equivalent
284backslash character classes (if available), will hold:
d74e8afc 285X<character class> X<\p> X<\p{}>
72ff2908 286
5496314a 287 [[:...:]] \p{...} backslash
b8c5462f
JH
288
289 alpha IsAlpha
290 alnum IsAlnum
291 ascii IsASCII
b432a672 292 blank IsSpace
b8c5462f 293 cntrl IsCntrl
3bec3564 294 digit IsDigit \d
b8c5462f
JH
295 graph IsGraph
296 lower IsLower
297 print IsPrint
298 punct IsPunct
299 space IsSpace
3bec3564 300 IsSpacePerl \s
b8c5462f
JH
301 upper IsUpper
302 word IsWord
303 xdigit IsXDigit
304
5496314a 305For example C<[[:lower:]]> and C<\p{IsLower}> are equivalent.
b8c5462f
JH
306
307If the C<utf8> pragma is not used but the C<locale> pragma is, the
aaa51d5e 308classes correlate with the usual isalpha(3) interface (except for
b432a672 309"word" and "blank").
b8c5462f
JH
310
311The assumedly non-obviously named classes are:
312
313=over 4
314
315=item cntrl
d74e8afc 316X<cntrl>
b8c5462f 317
820475bd
GS
318Any control character. Usually characters that don't produce output as
319such but instead control the terminal somehow: for example newline and
320backspace are control characters. All characters with ord() less than
593df60c 32132 are most often classified as control characters (assuming ASCII,
7be5a6cf
JF
322the ISO Latin character sets, and Unicode), as is the character with
323the ord() value of 127 (C<DEL>).
b8c5462f
JH
324
325=item graph
d74e8afc 326X<graph>
b8c5462f 327
f1cbbd6e 328Any alphanumeric or punctuation (special) character.
b8c5462f
JH
329
330=item print
d74e8afc 331X<print>
b8c5462f 332
f79b3095 333Any alphanumeric or punctuation (special) character or the space character.
b8c5462f
JH
334
335=item punct
d74e8afc 336X<punct>
b8c5462f 337
f1cbbd6e 338Any punctuation (special) character.
b8c5462f
JH
339
340=item xdigit
d74e8afc 341X<xdigit>
b8c5462f 342
593df60c 343Any hexadecimal digit. Though this may feel silly ([0-9A-Fa-f] would
820475bd 344work just fine) it is included for completeness.
b8c5462f 345
b8c5462f
JH
346=back
347
348You can negate the [::] character classes by prefixing the class name
349with a '^'. This is a Perl extension. For example:
d74e8afc 350X<character class, negation>
b8c5462f 351
5496314a 352 POSIX traditional Unicode
93733859 353
5496314a
SP
354 [[:^digit:]] \D \P{IsDigit}
355 [[:^space:]] \S \P{IsSpace}
356 [[:^word:]] \W \P{IsWord}
b8c5462f 357
54c18d04
MK
358Perl respects the POSIX standard in that POSIX character classes are
359only supported within a character class. The POSIX character classes
360[.cc.] and [=cc=] are recognized but B<not> supported and trying to
361use them will cause an error.
b8c5462f 362
a0d0e21e 363Perl defines the following zero-width assertions:
d74e8afc
ITB
364X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
365X<regexp, zero-width assertion>
366X<regular expression, zero-width assertion>
367X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
a0d0e21e
LW
368
369 \b Match a word boundary
370 \B Match a non-(word boundary)
b85d18e9
IZ
371 \A Match only at beginning of string
372 \Z Match only at end of string, or before newline at the end
373 \z Match only at end of string
9da458fc
IZ
374 \G Match only at pos() (e.g. at the end-of-match position
375 of prior m//g)
a0d0e21e 376
14218588 377A word boundary (C<\b>) is a spot between two characters
19799a22
GS
378that has a C<\w> on one side of it and a C<\W> on the other side
379of it (in either order), counting the imaginary characters off the
380beginning and end of the string as matching a C<\W>. (Within
381character classes C<\b> represents backspace rather than a word
382boundary, just as it normally does in any double-quoted string.)
383The C<\A> and C<\Z> are just like "^" and "$", except that they
384won't match multiple times when the C</m> modifier is used, while
385"^" and "$" will match at every internal line boundary. To match
386the actual end of the string and not ignore an optional trailing
387newline, use C<\z>.
d74e8afc 388X<\b> X<\A> X<\Z> X<\z> X</m>
19799a22
GS
389
390The C<\G> assertion can be used to chain global matches (using
391C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
392It is also useful when writing C<lex>-like scanners, when you have
393several patterns that you want to match against consequent substrings
394of your string, see the previous reference. The actual location
395where C<\G> will match can also be influenced by using C<pos()> as
25cf8c22
HS
396an lvalue: see L<perlfunc/pos>. Currently C<\G> is only fully
397supported when anchored to the start of the pattern; while it
398is permitted to use it elsewhere, as in C</(?<=\G..)./g>, some
399such uses (C</.\G/g>, for example) currently cause problems, and
400it is recommended that you avoid such usage for now.
d74e8afc 401X<\G>
c47ff5f1 402
14218588 403The bracketing construct C<( ... )> creates capture buffers. To
c47ff5f1 404refer to the digit'th buffer use \<digit> within the
14218588 405match. Outside the match use "$" instead of "\". (The
c47ff5f1 406\<digit> notation works in certain circumstances outside
14218588
GS
407the match. See the warning below about \1 vs $1 for details.)
408Referring back to another part of the match is called a
409I<backreference>.
d74e8afc
ITB
410X<regex, capture buffer> X<regexp, capture buffer>
411X<regular expression, capture buffer> X<backreference>
14218588
GS
412
413There is no limit to the number of captured substrings that you may
414use. However Perl also uses \10, \11, etc. as aliases for \010,
fb55449c
JH
415\011, etc. (Recall that 0 means octal, so \011 is the character at
416number 9 in your coded character set; which would be the 10th character,
417a horizontal tab under ASCII.) Perl resolves this
418ambiguity by interpreting \10 as a backreference only if at least 10
419left parentheses have opened before it. Likewise \11 is a
420backreference only if at least 11 left parentheses have opened
421before it. And so on. \1 through \9 are always interpreted as
422backreferences.
14218588
GS
423
424Examples:
a0d0e21e
LW
425
426 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
427
14218588
GS
428 if (/(.)\1/) { # find first doubled char
429 print "'$1' is the first doubled character\n";
430 }
c47ff5f1 431
14218588 432 if (/Time: (..):(..):(..)/) { # parse out values
a0d0e21e
LW
433 $hours = $1;
434 $minutes = $2;
435 $seconds = $3;
436 }
c47ff5f1 437
14218588
GS
438Several special variables also refer back to portions of the previous
439match. C<$+> returns whatever the last bracket match matched.
440C<$&> returns the entire matched string. (At one point C<$0> did
441also, but now it returns the name of the program.) C<$`> returns
77ea4f6d
JV
442everything before the matched string. C<$'> returns everything
443after the matched string. And C<$^N> contains whatever was matched by
444the most-recently closed group (submatch). C<$^N> can be used in
445extended patterns (see below), for example to assign a submatch to a
446variable.
d74e8afc 447X<$+> X<$^N> X<$&> X<$`> X<$'>
14218588 448
665e98b9 449The numbered match variables ($1, $2, $3, etc.) and the related punctuation
77ea4f6d 450set (C<$+>, C<$&>, C<$`>, C<$'>, and C<$^N>) are all dynamically scoped
14218588
GS
451until the end of the enclosing block or until the next successful
452match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
d74e8afc
ITB
453X<$+> X<$^N> X<$&> X<$`> X<$'>
454X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
455
14218588 456
665e98b9 457B<NOTE>: failed matches in Perl do not reset the match variables,
5146ce24 458which makes it easier to write code that tests for a series of more
665e98b9
JH
459specific cases and remembers the best match.
460
14218588
GS
461B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
462C<$'> anywhere in the program, it has to provide them for every
463pattern match. This may substantially slow your program. Perl
464uses the same mechanism to produce $1, $2, etc, so you also pay a
465price for each pattern that contains capturing parentheses. (To
466avoid this cost while retaining the grouping behaviour, use the
467extended regular expression C<(?: ... )> instead.) But if you never
468use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
469parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
470if you can, but if you can't (and some algorithms really appreciate
471them), once you've used them once, use them at will, because you've
472already paid the price. As of 5.005, C<$&> is not so costly as the
473other two.
d74e8afc 474X<$&> X<$`> X<$'>
68dc0745 475
19799a22
GS
476Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
477C<\w>, C<\n>. Unlike some other regular expression languages, there
478are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 479that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22
GS
480interpreted as a literal character, not a metacharacter. This was
481once used in a common idiom to disable or quote the special meanings
482of regular expression metacharacters in a string that you want to
36bbe248 483use for a pattern. Simply quote all non-"word" characters:
a0d0e21e
LW
484
485 $pattern =~ s/(\W)/\\$1/g;
486
f1cbbd6e 487(If C<use locale> is set, then this depends on the current locale.)
14218588
GS
488Today it is more common to use the quotemeta() function or the C<\Q>
489metaquoting escape sequence to disable all metacharacters' special
490meanings like this:
a0d0e21e
LW
491
492 /$unquoted\Q$quoted\E$unquoted/
493
9da458fc
IZ
494Beware that if you put literal backslashes (those not inside
495interpolated variables) between C<\Q> and C<\E>, double-quotish
496backslash interpolation may lead to confusing results. If you
497I<need> to use literal backslashes within C<\Q...\E>,
498consult L<perlop/"Gory details of parsing quoted constructs">.
499
19799a22
GS
500=head2 Extended Patterns
501
14218588
GS
502Perl also defines a consistent extension syntax for features not
503found in standard tools like B<awk> and B<lex>. The syntax is a
504pair of parentheses with a question mark as the first thing within
505the parentheses. The character after the question mark indicates
506the extension.
19799a22 507
14218588
GS
508The stability of these extensions varies widely. Some have been
509part of the core language for many years. Others are experimental
510and may change without warning or be completely removed. Check
511the documentation on an individual feature to verify its current
512status.
19799a22 513
14218588
GS
514A question mark was chosen for this and for the minimal-matching
515construct because 1) question marks are rare in older regular
516expressions, and 2) whenever you see one, you should stop and
517"question" exactly what is going on. That's psychology...
a0d0e21e
LW
518
519=over 10
520
cc6b7395 521=item C<(?#text)>
d74e8afc 522X<(?#)>
a0d0e21e 523
14218588 524A comment. The text is ignored. If the C</x> modifier enables
19799a22 525whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3
GS
526the comment as soon as it sees a C<)>, so there is no way to put a literal
527C<)> in the comment.
a0d0e21e 528
19799a22 529=item C<(?imsx-imsx)>
d74e8afc 530X<(?)>
19799a22 531
0b6d1084
JH
532One or more embedded pattern-match modifiers, to be turned on (or
533turned off, if preceded by C<->) for the remainder of the pattern or
534the remainder of the enclosing pattern group (if any). This is
535particularly useful for dynamic patterns, such as those read in from a
536configuration file, read in as an argument, are specified in a table
537somewhere, etc. Consider the case that some of which want to be case
538sensitive and some do not. The case insensitive ones need to include
539merely C<(?i)> at the front of the pattern. For example:
19799a22
GS
540
541 $pattern = "foobar";
542 if ( /$pattern/i ) { }
543
544 # more flexible:
545
546 $pattern = "(?i)foobar";
547 if ( /$pattern/ ) { }
548
0b6d1084 549These modifiers are restored at the end of the enclosing group. For example,
19799a22
GS
550
551 ( (?i) blah ) \s+ \1
552
553will match a repeated (I<including the case>!) word C<blah> in any
14218588 554case, assuming C<x> modifier, and no C<i> modifier outside this
19799a22
GS
555group.
556
5a964f20 557=item C<(?:pattern)>
d74e8afc 558X<(?:)>
a0d0e21e 559
ca9dfc88
IZ
560=item C<(?imsx-imsx:pattern)>
561
5a964f20
TC
562This is for clustering, not capturing; it groups subexpressions like
563"()", but doesn't make backreferences as "()" does. So
a0d0e21e 564
5a964f20 565 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e
LW
566
567is like
568
5a964f20 569 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 570
19799a22
GS
571but doesn't spit out extra fields. It's also cheaper not to capture
572characters if you don't need to.
a0d0e21e 573
19799a22
GS
574Any letters between C<?> and C<:> act as flags modifiers as with
575C<(?imsx-imsx)>. For example,
ca9dfc88
IZ
576
577 /(?s-i:more.*than).*million/i
578
14218588 579is equivalent to the more verbose
ca9dfc88
IZ
580
581 /(?:(?s-i)more.*than).*million/i
582
5a964f20 583=item C<(?=pattern)>
d74e8afc 584X<(?=)> X<look-ahead, positive> X<lookahead, positive>
a0d0e21e 585
19799a22 586A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e
LW
587matches a word followed by a tab, without including the tab in C<$&>.
588
5a964f20 589=item C<(?!pattern)>
d74e8afc 590X<(?!)> X<look-ahead, negative> X<lookahead, negative>
a0d0e21e 591
19799a22 592A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 593matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22
GS
594however that look-ahead and look-behind are NOT the same thing. You cannot
595use this for look-behind.
7b8d334a 596
5a964f20 597If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a
GS
598will not do what you want. That's because the C<(?!foo)> is just saying that
599the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
600match. You would have to do something like C</(?!foo)...bar/> for that. We
601say "like" because there's the case of your "bar" not having three characters
602before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
603Sometimes it's still easier just to say:
a0d0e21e 604
a3cb178b 605 if (/bar/ && $` !~ /foo$/)
a0d0e21e 606
19799a22 607For look-behind see below.
c277df42 608
c47ff5f1 609=item C<(?<=pattern)>
d74e8afc 610X<(?<=)> X<look-behind, positive> X<lookbehind, positive>
c277df42 611
c47ff5f1 612A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22
GS
613matches a word that follows a tab, without including the tab in C<$&>.
614Works only for fixed-width look-behind.
c277df42 615
5a964f20 616=item C<(?<!pattern)>
d74e8afc 617X<(?<!)> X<look-behind, negative> X<lookbehind, negative>
c277df42 618
19799a22
GS
619A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
620matches any occurrence of "foo" that does not follow "bar". Works
621only for fixed-width look-behind.
c277df42 622
cc6b7395 623=item C<(?{ code })>
d74e8afc 624X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
c277df42 625
19799a22
GS
626B<WARNING>: This extended regular expression feature is considered
627highly experimental, and may be changed or deleted without notice.
c277df42 628
cc46d5f2 629This zero-width assertion evaluates any embedded Perl code. It
19799a22
GS
630always succeeds, and its C<code> is not interpolated. Currently,
631the rules to determine where the C<code> ends are somewhat convoluted.
632
77ea4f6d
JV
633This feature can be used together with the special variable C<$^N> to
634capture the results of submatches in variables without having to keep
635track of the number of nested parentheses. For example:
636
637 $_ = "The brown fox jumps over the lazy dog";
638 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
639 print "color = $color, animal = $animal\n";
640
754091cb
RGS
641Inside the C<(?{...})> block, C<$_> refers to the string the regular
642expression is matching against. You can also use C<pos()> to know what is
fa11829f 643the current position of matching within this string.
754091cb 644
19799a22
GS
645The C<code> is properly scoped in the following sense: If the assertion
646is backtracked (compare L<"Backtracking">), all changes introduced after
647C<local>ization are undone, so that
b9ac3b5b
GS
648
649 $_ = 'a' x 8;
650 m<
651 (?{ $cnt = 0 }) # Initialize $cnt.
652 (
653 a
654 (?{
655 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
656 })
657 )*
658 aaaa
659 (?{ $res = $cnt }) # On success copy to non-localized
660 # location.
661 >x;
662
19799a22 663will set C<$res = 4>. Note that after the match, $cnt returns to the globally
14218588 664introduced value, because the scopes that restrict C<local> operators
b9ac3b5b
GS
665are unwound.
666
19799a22
GS
667This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
668switch. If I<not> used in this way, the result of evaluation of
669C<code> is put into the special variable C<$^R>. This happens
670immediately, so C<$^R> can be used from other C<(?{ code })> assertions
671inside the same regular expression.
b9ac3b5b 672
19799a22
GS
673The assignment to C<$^R> above is properly localized, so the old
674value of C<$^R> is restored if the assertion is backtracked; compare
675L<"Backtracking">.
b9ac3b5b 676
61528107
SP
677Due to an unfortunate implementation issue, the Perl code contained in these
678blocks is treated as a compile time closure that can have seemingly bizarre
6bda09f9 679consequences when used with lexically scoped variables inside of subroutines
61528107
SP
680or loops. There are various workarounds for this, including simply using
681global variables instead. If you are using this construct and strange results
6bda09f9
YO
682occur then check for the use of lexically scoped variables.
683
19799a22
GS
684For reasons of security, this construct is forbidden if the regular
685expression involves run-time interpolation of variables, unless the
686perilous C<use re 'eval'> pragma has been used (see L<re>), or the
687variables contain results of C<qr//> operator (see
688L<perlop/"qr/STRING/imosx">).
871b0233 689
14218588 690This restriction is because of the wide-spread and remarkably convenient
19799a22 691custom of using run-time determined strings as patterns. For example:
871b0233
IZ
692
693 $re = <>;
694 chomp $re;
695 $string =~ /$re/;
696
14218588
GS
697Before Perl knew how to execute interpolated code within a pattern,
698this operation was completely safe from a security point of view,
699although it could raise an exception from an illegal pattern. If
700you turn on the C<use re 'eval'>, though, it is no longer secure,
701so you should only do so if you are also using taint checking.
702Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 703compartment. See L<perlsec> for details about both these mechanisms.
871b0233 704
8988a1bb
DD
705Because perl's regex engine is not currently re-entrant, interpolated
706code may not invoke the regex engine either directly with C<m//> or C<s///>),
707or indirectly with functions such as C<split>.
708
14455d6c 709=item C<(??{ code })>
d74e8afc
ITB
710X<(??{})>
711X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
0f5d15d6 712
19799a22
GS
713B<WARNING>: This extended regular expression feature is considered
714highly experimental, and may be changed or deleted without notice.
9da458fc
IZ
715A simplified version of the syntax may be introduced for commonly
716used idioms.
0f5d15d6 717
19799a22
GS
718This is a "postponed" regular subexpression. The C<code> is evaluated
719at run time, at the moment this subexpression may match. The result
720of evaluation is considered as a regular expression and matched as
61528107 721if it were inserted instead of this construct. Note that this means
6bda09f9
YO
722that the contents of capture buffers defined inside an eval'ed pattern
723are not available outside of the pattern, and vice versa, there is no
724way for the inner pattern to refer to a capture buffer defined outside.
725Thus,
726
727 ('a' x 100)=~/(??{'(.)' x 100})/
728
729B<will> match, it will B<not> set $1.
0f5d15d6 730
428594d9 731The C<code> is not interpolated. As before, the rules to determine
19799a22
GS
732where the C<code> ends are currently somewhat convoluted.
733
734The following pattern matches a parenthesized group:
0f5d15d6
IZ
735
736 $re = qr{
737 \(
738 (?:
739 (?> [^()]+ ) # Non-parens without backtracking
740 |
14455d6c 741 (??{ $re }) # Group with matching parens
0f5d15d6
IZ
742 )*
743 \)
744 }x;
745
6bda09f9
YO
746See also C<(?PARNO)> for a different, more efficient way to accomplish
747the same task.
748
8988a1bb
DD
749Because perl's regex engine is not currently re-entrant, delayed
750code may not invoke the regex engine either directly with C<m//> or C<s///>),
751or indirectly with functions such as C<split>.
752
6bda09f9 753Recursing deeper than 50 times without consuming any input string will
61528107 754result in a fatal error. The maximum depth is compiled into perl, so
6bda09f9
YO
755changing it requires a custom build.
756
757=item C<(?PARNO)> C<(?R)>
758
759X<(?PARNO)> X<(?1)>
760X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
761
61528107 762B<WARNING>: This extended regular expression feature is considered
6bda09f9
YO
763highly experimental, and may be changed or deleted without notice.
764
765Similar to C<(??{ code })> except it does not involve compiling any code,
766instead it treats the contents of a capture buffer as an independent
61528107 767pattern that must match at the current position. Capture buffers
6bda09f9
YO
768contained by the pattern will have the value as determined by the
769outermost recursion.
770
771PARNO is a sequence of digits not starting with 0 whose value
772reflects the paren-number of the capture buffer to recurse to.
773C<(?R)> curses to the beginning of the pattern.
774
775The following pattern matches a function foo() which may contain
776balanced parenthesis as the argument.
777
778 $re = qr{ ( # paren group 1 (full function)
779 foo
780 ( # paren group 2 (parens)
781 \(
782 ( # paren group 3 (contents of parens)
783 (?:
784 (?> [^()]+ ) # Non-parens without backtracking
785 |
786 (?2) # Recurse to start of paren group 2
787 )*
788 )
789 \)
790 )
791 )
792 }x;
793
794If the pattern was used as follows
795
796 'foo(bar(baz)+baz(bop))'=~/$re/
797 and print "\$1 = $1\n",
798 "\$2 = $2\n",
799 "\$3 = $3\n";
800
801the output produced should be the following:
802
803 $1 = foo(bar(baz)+baz(bop))
804 $2 = (bar(baz)+baz(bop))
805 $3 = bar(baz)+baz(bop)
806
807If there is no corresponding capture buffer defined, then it is a
61528107
SP
808fatal error. Recursing deeper than 50 times without consuming any input
809string will also result in a fatal error. The maximum depth is compiled
6bda09f9
YO
810into perl, so changing it requires a custom build.
811
812B<Note> that this pattern does not behave the same way as the equivalent
813PCRE or Python construct of the same form. In perl you can backtrack into
814a recursed group, in PCRE and Python the recursed into group is treated
815as atomic. Also, constructs like (?i:(?1)) or (?:(?i)(?1)) do not affect
816the pattern being recursed into.
817
c47ff5f1 818=item C<< (?>pattern) >>
6bda09f9 819X<backtrack> X<backtracking> X<atomic> X<possessive>
5a964f20 820
19799a22
GS
821B<WARNING>: This extended regular expression feature is considered
822highly experimental, and may be changed or deleted without notice.
823
824An "independent" subexpression, one which matches the substring
825that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 826position, and it matches I<nothing other than this substring>. This
19799a22
GS
827construct is useful for optimizations of what would otherwise be
828"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc
IZ
829It may also be useful in places where the "grab all you can, and do not
830give anything back" semantic is desirable.
19799a22 831
c47ff5f1 832For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22
GS
833(anchored at the beginning of string, as above) will match I<all>
834characters C<a> at the beginning of string, leaving no C<a> for
835C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
836since the match of the subgroup C<a*> is influenced by the following
837group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
838C<a*ab> will match fewer characters than a standalone C<a*>, since
839this makes the tail match.
840
c47ff5f1 841An effect similar to C<< (?>pattern) >> may be achieved by writing
19799a22
GS
842C<(?=(pattern))\1>. This matches the same substring as a standalone
843C<a+>, and the following C<\1> eats the matched string; it therefore
c47ff5f1 844makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22
GS
845(The difference between these two constructs is that the second one
846uses a capturing group, thus shifting ordinals of backreferences
847in the rest of a regular expression.)
848
849Consider this pattern:
c277df42 850
871b0233
IZ
851 m{ \(
852 (
9da458fc 853 [^()]+ # x+
871b0233
IZ
854 |
855 \( [^()]* \)
856 )+
857 \)
858 }x
5a964f20 859
19799a22
GS
860That will efficiently match a nonempty group with matching parentheses
861two levels deep or less. However, if there is no such group, it
862will take virtually forever on a long string. That's because there
863are so many different ways to split a long string into several
864substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
865to a subpattern of the above pattern. Consider how the pattern
866above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
867seconds, but that each extra letter doubles this time. This
868exponential performance will make it appear that your program has
14218588 869hung. However, a tiny change to this pattern
5a964f20 870
871b0233
IZ
871 m{ \(
872 (
9da458fc 873 (?> [^()]+ ) # change x+ above to (?> x+ )
871b0233
IZ
874 |
875 \( [^()]* \)
876 )+
877 \)
878 }x
c277df42 879
c47ff5f1 880which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20
TC
881this yourself would be a productive exercise), but finishes in a fourth
882the time when used on a similar string with 1000000 C<a>s. Be aware,
883however, that this pattern currently triggers a warning message under
9f1b1f2d 884the C<use warnings> pragma or B<-w> switch saying it
6bab786b 885C<"matches null string many times in regex">.
c277df42 886
c47ff5f1 887On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 888effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42
IZ
889This was only 4 times slower on a string with 1000000 C<a>s.
890
9da458fc
IZ
891The "grab all you can, and do not give anything back" semantic is desirable
892in many situations where on the first sight a simple C<()*> looks like
893the correct solution. Suppose we parse text with comments being delimited
894by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 895its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc
IZ
896the comment delimiter, because it may "give up" some whitespace if
897the remainder of the pattern can be made to match that way. The correct
898answer is either one of these:
899
900 (?>#[ \t]*)
901 #[ \t]*(?![ \t])
902
903For example, to grab non-empty comments into $1, one should use either
904one of these:
905
906 / (?> \# [ \t]* ) ( .+ ) /x;
907 / \# [ \t]* ( [^ \t] .* ) /x;
908
909Which one you pick depends on which of these expressions better reflects
910the above specification of comments.
911
6bda09f9
YO
912In some literature this construct is called "atomic matching" or
913"possessive matching".
914
5a964f20 915=item C<(?(condition)yes-pattern|no-pattern)>
d74e8afc 916X<(?()>
c277df42 917
5a964f20 918=item C<(?(condition)yes-pattern)>
c277df42 919
19799a22
GS
920B<WARNING>: This extended regular expression feature is considered
921highly experimental, and may be changed or deleted without notice.
922
c277df42
IZ
923Conditional expression. C<(condition)> should be either an integer in
924parentheses (which is valid if the corresponding pair of parentheses
19799a22 925matched), or look-ahead/look-behind/evaluate zero-width assertion.
c277df42 926
19799a22 927For example:
c277df42 928
5a964f20 929 m{ ( \( )?
871b0233 930 [^()]+
5a964f20 931 (?(1) \) )
871b0233 932 }x
c277df42
IZ
933
934matches a chunk of non-parentheses, possibly included in parentheses
935themselves.
a0d0e21e 936
a0d0e21e
LW
937=back
938
c07a80fd 939=head2 Backtracking
d74e8afc 940X<backtrack> X<backtracking>
c07a80fd 941
35a734be
IZ
942NOTE: This section presents an abstract approximation of regular
943expression behavior. For a more rigorous (and complicated) view of
944the rules involved in selecting a match among possible alternatives,
945see L<Combining pieces together>.
946
c277df42 947A fundamental feature of regular expression matching involves the
5a964f20 948notion called I<backtracking>, which is currently used (when needed)
c277df42 949by all regular expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc
IZ
950C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
951internally, but the general principle outlined here is valid.
c07a80fd
PP
952
953For a regular expression to match, the I<entire> regular expression must
954match, not just part of it. So if the beginning of a pattern containing a
955quantifier succeeds in a way that causes later parts in the pattern to
956fail, the matching engine backs up and recalculates the beginning
957part--that's why it's called backtracking.
958
959Here is an example of backtracking: Let's say you want to find the
960word following "foo" in the string "Food is on the foo table.":
961
962 $_ = "Food is on the foo table.";
963 if ( /\b(foo)\s+(\w+)/i ) {
964 print "$2 follows $1.\n";
965 }
966
967When the match runs, the first part of the regular expression (C<\b(foo)>)
968finds a possible match right at the beginning of the string, and loads up
969$1 with "Foo". However, as soon as the matching engine sees that there's
970no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 971mistake and starts over again one character after where it had the
c07a80fd
PP
972tentative match. This time it goes all the way until the next occurrence
973of "foo". The complete regular expression matches this time, and you get
974the expected output of "table follows foo."
975
976Sometimes minimal matching can help a lot. Imagine you'd like to match
977everything between "foo" and "bar". Initially, you write something
978like this:
979
980 $_ = "The food is under the bar in the barn.";
981 if ( /foo(.*)bar/ ) {
982 print "got <$1>\n";
983 }
984
985Which perhaps unexpectedly yields:
986
987 got <d is under the bar in the >
988
989That's because C<.*> was greedy, so you get everything between the
14218588 990I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd
PP
991to use minimal matching to make sure you get the text between a "foo"
992and the first "bar" thereafter.
993
994 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
995 got <d is under the >
996
997Here's another example: let's say you'd like to match a number at the end
b6e13d97 998of a string, and you also want to keep the preceding part of the match.
c07a80fd
PP
999So you write this:
1000
1001 $_ = "I have 2 numbers: 53147";
1002 if ( /(.*)(\d*)/ ) { # Wrong!
1003 print "Beginning is <$1>, number is <$2>.\n";
1004 }
1005
1006That won't work at all, because C<.*> was greedy and gobbled up the
1007whole string. As C<\d*> can match on an empty string the complete
1008regular expression matched successfully.
1009
8e1088bc 1010 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd
PP
1011
1012Here are some variants, most of which don't work:
1013
1014 $_ = "I have 2 numbers: 53147";
1015 @pats = qw{
1016 (.*)(\d*)
1017 (.*)(\d+)
1018 (.*?)(\d*)
1019 (.*?)(\d+)
1020 (.*)(\d+)$
1021 (.*?)(\d+)$
1022 (.*)\b(\d+)$
1023 (.*\D)(\d+)$
1024 };
1025
1026 for $pat (@pats) {
1027 printf "%-12s ", $pat;
1028 if ( /$pat/ ) {
1029 print "<$1> <$2>\n";
1030 } else {
1031 print "FAIL\n";
1032 }
1033 }
1034
1035That will print out:
1036
1037 (.*)(\d*) <I have 2 numbers: 53147> <>
1038 (.*)(\d+) <I have 2 numbers: 5314> <7>
1039 (.*?)(\d*) <> <>
1040 (.*?)(\d+) <I have > <2>
1041 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
1042 (.*?)(\d+)$ <I have 2 numbers: > <53147>
1043 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
1044 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
1045
1046As you see, this can be a bit tricky. It's important to realize that a
1047regular expression is merely a set of assertions that gives a definition
1048of success. There may be 0, 1, or several different ways that the
1049definition might succeed against a particular string. And if there are
5a964f20
TC
1050multiple ways it might succeed, you need to understand backtracking to
1051know which variety of success you will achieve.
c07a80fd 1052
19799a22 1053When using look-ahead assertions and negations, this can all get even
8b19b778 1054trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd
PP
1055followed by "123". You might try to write that as
1056
871b0233
IZ
1057 $_ = "ABC123";
1058 if ( /^\D*(?!123)/ ) { # Wrong!
1059 print "Yup, no 123 in $_\n";
1060 }
c07a80fd
PP
1061
1062But that isn't going to match; at least, not the way you're hoping. It
1063claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 1064why that pattern matches, contrary to popular expectations:
c07a80fd 1065
4358a253
SS
1066 $x = 'ABC123';
1067 $y = 'ABC445';
c07a80fd 1068
4358a253
SS
1069 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/;
1070 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/;
c07a80fd 1071
4358a253
SS
1072 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/;
1073 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/;
c07a80fd
PP
1074
1075This prints
1076
1077 2: got ABC
1078 3: got AB
1079 4: got ABC
1080
5f05dabc 1081You might have expected test 3 to fail because it seems to a more
c07a80fd
PP
1082general purpose version of test 1. The important difference between
1083them is that test 3 contains a quantifier (C<\D*>) and so can use
1084backtracking, whereas test 1 will not. What's happening is
1085that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 1086non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 1087let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 1088fail.
14218588 1089
c07a80fd 1090The search engine will initially match C<\D*> with "ABC". Then it will
14218588 1091try to match C<(?!123> with "123", which fails. But because
c07a80fd
PP
1092a quantifier (C<\D*>) has been used in the regular expression, the
1093search engine can backtrack and retry the match differently
54310121 1094in the hope of matching the complete regular expression.
c07a80fd 1095
5a964f20
TC
1096The pattern really, I<really> wants to succeed, so it uses the
1097standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 1098time. Now there's indeed something following "AB" that is not
14218588 1099"123". It's "C123", which suffices.
c07a80fd 1100
14218588
GS
1101We can deal with this by using both an assertion and a negation.
1102We'll say that the first part in $1 must be followed both by a digit
1103and by something that's not "123". Remember that the look-aheads
1104are zero-width expressions--they only look, but don't consume any
1105of the string in their match. So rewriting this way produces what
c07a80fd
PP
1106you'd expect; that is, case 5 will fail, but case 6 succeeds:
1107
4358a253
SS
1108 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/;
1109 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/;
c07a80fd
PP
1110
1111 6: got ABC
1112
5a964f20 1113In other words, the two zero-width assertions next to each other work as though
19799a22 1114they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd
PP
1115matches only if you're at the beginning of the line AND the end of the
1116line simultaneously. The deeper underlying truth is that juxtaposition in
1117regular expressions always means AND, except when you write an explicit OR
1118using the vertical bar. C</ab/> means match "a" AND (then) match "b",
1119although the attempted matches are made at different positions because "a"
1120is not a zero-width assertion, but a one-width assertion.
1121
19799a22 1122B<WARNING>: particularly complicated regular expressions can take
14218588 1123exponential time to solve because of the immense number of possible
9da458fc
IZ
1124ways they can use backtracking to try match. For example, without
1125internal optimizations done by the regular expression engine, this will
1126take a painfully long time to run:
c07a80fd 1127
e1901655
IZ
1128 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
1129
1130And if you used C<*>'s in the internal groups instead of limiting them
1131to 0 through 5 matches, then it would take forever--or until you ran
1132out of stack space. Moreover, these internal optimizations are not
1133always applicable. For example, if you put C<{0,5}> instead of C<*>
1134on the external group, no current optimization is applicable, and the
1135match takes a long time to finish.
c07a80fd 1136
9da458fc
IZ
1137A powerful tool for optimizing such beasts is what is known as an
1138"independent group",
c47ff5f1 1139which does not backtrack (see L<C<< (?>pattern) >>>). Note also that
9da458fc 1140zero-length look-ahead/look-behind assertions will not backtrack to make
14218588
GS
1141the tail match, since they are in "logical" context: only
1142whether they match is considered relevant. For an example
9da458fc 1143where side-effects of look-ahead I<might> have influenced the
c47ff5f1 1144following match, see L<C<< (?>pattern) >>>.
c277df42 1145
a0d0e21e 1146=head2 Version 8 Regular Expressions
d74e8afc 1147X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
a0d0e21e 1148
5a964f20 1149In case you're not familiar with the "regular" Version 8 regex
a0d0e21e
LW
1150routines, here are the pattern-matching rules not described above.
1151
54310121 1152Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 1153with a special meaning described here or above. You can cause
5a964f20 1154characters that normally function as metacharacters to be interpreted
5f05dabc 1155literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
a0d0e21e
LW
1156character; "\\" matches a "\"). A series of characters matches that
1157series of characters in the target string, so the pattern C<blurfl>
1158would match "blurfl" in the target string.
1159
1160You can specify a character class, by enclosing a list of characters
5a964f20 1161in C<[]>, which will match any one character from the list. If the
a0d0e21e 1162first character after the "[" is "^", the class matches any character not
14218588 1163in the list. Within a list, the "-" character specifies a
5a964f20 1164range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2
GS
1165inclusive. If you want either "-" or "]" itself to be a member of a
1166class, put it at the start of the list (possibly after a "^"), or
1167escape it with a backslash. "-" is also taken literally when it is
1168at the end of the list, just before the closing "]". (The
84850974
DD
1169following all specify the same class of three characters: C<[-az]>,
1170C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
fb55449c
JH
1171specifies a class containing twenty-six characters, even on EBCDIC
1172based coded character sets.) Also, if you try to use the character
1173classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1174a range, that's not a range, the "-" is understood literally.
a0d0e21e 1175
8ada0baa
JH
1176Note also that the whole range idea is rather unportable between
1177character sets--and even within character sets they may cause results
1178you probably didn't expect. A sound principle is to use only ranges
1179that begin from and end at either alphabets of equal case ([a-e],
1180[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1181spell out the character sets in full.
1182
54310121 1183Characters may be specified using a metacharacter syntax much like that
a0d0e21e
LW
1184used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1185"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
fb55449c
JH
1186of octal digits, matches the character whose coded character set value
1187is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
1188matches the character whose numeric value is I<nn>. The expression \cI<x>
1189matches the character control-I<x>. Finally, the "." metacharacter
1190matches any character except "\n" (unless you use C</s>).
a0d0e21e
LW
1191
1192You can specify a series of alternatives for a pattern using "|" to
1193separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1194or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e
LW
1195first alternative includes everything from the last pattern delimiter
1196("(", "[", or the beginning of the pattern) up to the first "|", and
1197the last alternative contains everything from the last "|" to the next
14218588
GS
1198pattern delimiter. That's why it's common practice to include
1199alternatives in parentheses: to minimize confusion about where they
a3cb178b
GS
1200start and end.
1201
5a964f20 1202Alternatives are tried from left to right, so the first
a3cb178b
GS
1203alternative found for which the entire expression matches, is the one that
1204is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1205example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b
GS
1206part will match, as that is the first alternative tried, and it successfully
1207matches the target string. (This might not seem important, but it is
1208important when you are capturing matched text using parentheses.)
1209
5a964f20 1210Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1211so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1212
14218588
GS
1213Within a pattern, you may designate subpatterns for later reference
1214by enclosing them in parentheses, and you may refer back to the
1215I<n>th subpattern later in the pattern using the metacharacter
1216\I<n>. Subpatterns are numbered based on the left to right order
1217of their opening parenthesis. A backreference matches whatever
1218actually matched the subpattern in the string being examined, not
1219the rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will
1220match "0x1234 0x4321", but not "0x1234 01234", because subpattern
12211 matched "0x", even though the rule C<0|0x> could potentially match
1222the leading 0 in the second number.
cb1a09d0 1223
19799a22 1224=head2 Warning on \1 vs $1
cb1a09d0 1225
5a964f20 1226Some people get too used to writing things like:
cb1a09d0
AD
1227
1228 $pattern =~ s/(\W)/\\\1/g;
1229
1230This is grandfathered for the RHS of a substitute to avoid shocking the
1231B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1232PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0
AD
1233the usual double-quoted string means a control-A. The customary Unix
1234meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1235of doing that, you get yourself into trouble if you then add an C</e>
1236modifier.
1237
5a964f20 1238 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0
AD
1239
1240Or if you try to do
1241
1242 s/(\d+)/\1000/;
1243
1244You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1245C<${1}000>. The operation of interpolation should not be confused
cb1a09d0
AD
1246with the operation of matching a backreference. Certainly they mean two
1247different things on the I<left> side of the C<s///>.
9fa51da4 1248
c84d73f1
IZ
1249=head2 Repeated patterns matching zero-length substring
1250
19799a22 1251B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1
IZ
1252
1253Regular expressions provide a terse and powerful programming language. As
1254with most other power tools, power comes together with the ability
1255to wreak havoc.
1256
1257A common abuse of this power stems from the ability to make infinite
628afcb5 1258loops using regular expressions, with something as innocuous as:
c84d73f1
IZ
1259
1260 'foo' =~ m{ ( o? )* }x;
1261
1262The C<o?> can match at the beginning of C<'foo'>, and since the position
1263in the string is not moved by the match, C<o?> would match again and again
14218588 1264because of the C<*> modifier. Another common way to create a similar cycle
c84d73f1
IZ
1265is with the looping modifier C<//g>:
1266
1267 @matches = ( 'foo' =~ m{ o? }xg );
1268
1269or
1270
1271 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
1272
1273or the loop implied by split().
1274
1275However, long experience has shown that many programming tasks may
14218588
GS
1276be significantly simplified by using repeated subexpressions that
1277may match zero-length substrings. Here's a simple example being:
c84d73f1
IZ
1278
1279 @chars = split //, $string; # // is not magic in split
1280 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
1281
9da458fc 1282Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1
IZ
1283the infinite loop>. The rules for this are different for lower-level
1284loops given by the greedy modifiers C<*+{}>, and for higher-level
1285ones like the C</g> modifier or split() operator.
1286
19799a22
GS
1287The lower-level loops are I<interrupted> (that is, the loop is
1288broken) when Perl detects that a repeated expression matched a
1289zero-length substring. Thus
c84d73f1
IZ
1290
1291 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
1292
1293is made equivalent to
1294
1295 m{ (?: NON_ZERO_LENGTH )*
1296 |
1297 (?: ZERO_LENGTH )?
1298 }x;
1299
1300The higher level-loops preserve an additional state between iterations:
1301whether the last match was zero-length. To break the loop, the following
1302match after a zero-length match is prohibited to have a length of zero.
1303This prohibition interacts with backtracking (see L<"Backtracking">),
1304and so the I<second best> match is chosen if the I<best> match is of
1305zero length.
1306
19799a22 1307For example:
c84d73f1
IZ
1308
1309 $_ = 'bar';
1310 s/\w??/<$&>/g;
1311
20fb949f 1312results in C<< <><b><><a><><r><> >>. At each position of the string the best
c84d73f1
IZ
1313match given by non-greedy C<??> is the zero-length match, and the I<second
1314best> match is what is matched by C<\w>. Thus zero-length matches
1315alternate with one-character-long matches.
1316
1317Similarly, for repeated C<m/()/g> the second-best match is the match at the
1318position one notch further in the string.
1319
19799a22 1320The additional state of being I<matched with zero-length> is associated with
c84d73f1 1321the matched string, and is reset by each assignment to pos().
9da458fc
IZ
1322Zero-length matches at the end of the previous match are ignored
1323during C<split>.
c84d73f1 1324
35a734be
IZ
1325=head2 Combining pieces together
1326
1327Each of the elementary pieces of regular expressions which were described
1328before (such as C<ab> or C<\Z>) could match at most one substring
1329at the given position of the input string. However, in a typical regular
1330expression these elementary pieces are combined into more complicated
1331patterns using combining operators C<ST>, C<S|T>, C<S*> etc
1332(in these examples C<S> and C<T> are regular subexpressions).
1333
1334Such combinations can include alternatives, leading to a problem of choice:
1335if we match a regular expression C<a|ab> against C<"abc">, will it match
1336substring C<"a"> or C<"ab">? One way to describe which substring is
1337actually matched is the concept of backtracking (see L<"Backtracking">).
1338However, this description is too low-level and makes you think
1339in terms of a particular implementation.
1340
1341Another description starts with notions of "better"/"worse". All the
1342substrings which may be matched by the given regular expression can be
1343sorted from the "best" match to the "worst" match, and it is the "best"
1344match which is chosen. This substitutes the question of "what is chosen?"
1345by the question of "which matches are better, and which are worse?".
1346
1347Again, for elementary pieces there is no such question, since at most
1348one match at a given position is possible. This section describes the
1349notion of better/worse for combining operators. In the description
1350below C<S> and C<T> are regular subexpressions.
1351
13a2d996 1352=over 4
35a734be
IZ
1353
1354=item C<ST>
1355
1356Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
1357substrings which can be matched by C<S>, C<B> and C<B'> are substrings
1358which can be matched by C<T>.
1359
1360If C<A> is better match for C<S> than C<A'>, C<AB> is a better
1361match than C<A'B'>.
1362
1363If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
1364C<B> is better match for C<T> than C<B'>.
1365
1366=item C<S|T>
1367
1368When C<S> can match, it is a better match than when only C<T> can match.
1369
1370Ordering of two matches for C<S> is the same as for C<S>. Similar for
1371two matches for C<T>.
1372
1373=item C<S{REPEAT_COUNT}>
1374
1375Matches as C<SSS...S> (repeated as many times as necessary).
1376
1377=item C<S{min,max}>
1378
1379Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
1380
1381=item C<S{min,max}?>
1382
1383Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
1384
1385=item C<S?>, C<S*>, C<S+>
1386
1387Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
1388
1389=item C<S??>, C<S*?>, C<S+?>
1390
1391Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
1392
c47ff5f1 1393=item C<< (?>S) >>
35a734be
IZ
1394
1395Matches the best match for C<S> and only that.
1396
1397=item C<(?=S)>, C<(?<=S)>
1398
1399Only the best match for C<S> is considered. (This is important only if
1400C<S> has capturing parentheses, and backreferences are used somewhere
1401else in the whole regular expression.)
1402
1403=item C<(?!S)>, C<(?<!S)>
1404
1405For this grouping operator there is no need to describe the ordering, since
1406only whether or not C<S> can match is important.
1407
6bda09f9 1408=item C<(??{ EXPR })>, C<(?PARNO)>
35a734be
IZ
1409
1410The ordering is the same as for the regular expression which is
6bda09f9 1411the result of EXPR, or the pattern contained by capture buffer PARNO.
35a734be
IZ
1412
1413=item C<(?(condition)yes-pattern|no-pattern)>
1414
1415Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
1416already determined. The ordering of the matches is the same as for the
1417chosen subexpression.
1418
1419=back
1420
1421The above recipes describe the ordering of matches I<at a given position>.
1422One more rule is needed to understand how a match is determined for the
1423whole regular expression: a match at an earlier position is always better
1424than a match at a later position.
1425
c84d73f1
IZ
1426=head2 Creating custom RE engines
1427
1428Overloaded constants (see L<overload>) provide a simple way to extend
1429the functionality of the RE engine.
1430
1431Suppose that we want to enable a new RE escape-sequence C<\Y|> which
6b0ac556 1432matches at boundary between whitespace characters and non-whitespace
c84d73f1
IZ
1433characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
1434at these positions, so we want to have each C<\Y|> in the place of the
1435more complicated version. We can create a module C<customre> to do
1436this:
1437
1438 package customre;
1439 use overload;
1440
1441 sub import {
1442 shift;
1443 die "No argument to customre::import allowed" if @_;
1444 overload::constant 'qr' => \&convert;
1445 }
1446
1447 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
1448
580a9fe1
RGS
1449 # We must also take care of not escaping the legitimate \\Y|
1450 # sequence, hence the presence of '\\' in the conversion rules.
141db969 1451 my %rules = ( '\\' => '\\\\',
c84d73f1
IZ
1452 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
1453 sub convert {
1454 my $re = shift;
1455 $re =~ s{
1456 \\ ( \\ | Y . )
1457 }
1458 { $rules{$1} or invalid($re,$1) }sgex;
1459 return $re;
1460 }
1461
1462Now C<use customre> enables the new escape in constant regular
1463expressions, i.e., those without any runtime variable interpolations.
1464As documented in L<overload>, this conversion will work only over
1465literal parts of regular expressions. For C<\Y|$re\Y|> the variable
1466part of this regular expression needs to be converted explicitly
1467(but only if the special meaning of C<\Y|> should be enabled inside $re):
1468
1469 use customre;
1470 $re = <>;
1471 chomp $re;
1472 $re = customre::convert $re;
1473 /\Y|$re\Y|/;
1474
19799a22
GS
1475=head1 BUGS
1476
9da458fc
IZ
1477This document varies from difficult to understand to completely
1478and utterly opaque. The wandering prose riddled with jargon is
1479hard to fathom in several places.
1480
1481This document needs a rewrite that separates the tutorial content
1482from the reference content.
19799a22
GS
1483
1484=head1 SEE ALSO
9fa51da4 1485
91e0c79e
MJD
1486L<perlrequick>.
1487
1488L<perlretut>.
1489
9b599b2a
GS
1490L<perlop/"Regexp Quote-Like Operators">.
1491
1e66bd83
PP
1492L<perlop/"Gory details of parsing quoted constructs">.
1493
14218588
GS
1494L<perlfaq6>.
1495
9b599b2a
GS
1496L<perlfunc/pos>.
1497
1498L<perllocale>.
1499
fb55449c
JH
1500L<perlebcdic>.
1501
14218588
GS
1502I<Mastering Regular Expressions> by Jeffrey Friedl, published
1503by O'Reilly and Associates.