This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Upgrade to Module::CoreList 2.09:
[perl5.git] / pod / perlre.pod
CommitLineData
a0d0e21e 1=head1 NAME
d74e8afc 2X<regular expression> X<regex> X<regexp>
a0d0e21e
LW
3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
91e0c79e
MJD
8This page describes the syntax of regular expressions in Perl.
9
cc46d5f2 10If you haven't used regular expressions before, a quick-start
91e0c79e
MJD
11introduction is available in L<perlrequick>, and a longer tutorial
12introduction is available in L<perlretut>.
13
14For reference on how regular expressions are used in matching
15operations, plus various examples of the same, see discussions of
16C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like
17Operators">.
cb1a09d0 18
19799a22 19Matching operations can have various modifiers. Modifiers
5a964f20 20that relate to the interpretation of the regular expression inside
19799a22
GS
21are listed below. Modifiers that alter the way a regular expression
22is used by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
1e66bd83 23L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 24
55497cff
PP
25=over 4
26
27=item i
d74e8afc
ITB
28X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
29X<regular expression, case-insensitive>
55497cff
PP
30
31Do case-insensitive pattern matching.
32
a034a98d
DD
33If C<use locale> is in effect, the case map is taken from the current
34locale. See L<perllocale>.
35
54310121 36=item m
d74e8afc 37X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
55497cff
PP
38
39Treat string as multiple lines. That is, change "^" and "$" from matching
14218588 40the start or end of the string to matching the start or end of any
7f761169 41line anywhere within the string.
55497cff 42
54310121 43=item s
d74e8afc
ITB
44X</s> X<regex, single-line> X<regexp, single-line>
45X<regular expression, single-line>
55497cff
PP
46
47Treat string as single line. That is, change "." to match any character
19799a22 48whatsoever, even a newline, which normally it would not match.
55497cff 49
f02c194e 50Used together, as /ms, they let the "." match any character whatsoever,
fb55449c 51while still allowing "^" and "$" to match, respectively, just after
19799a22 52and just before newlines within the string.
7b8d334a 53
54310121 54=item x
d74e8afc 55X</x>
55497cff
PP
56
57Extend your pattern's legibility by permitting whitespace and comments.
58
59=back
a0d0e21e
LW
60
61These are usually written as "the C</x> modifier", even though the delimiter
14218588 62in question might not really be a slash. Any of these
a0d0e21e 63modifiers may also be embedded within the regular expression itself using
14218588 64the C<(?...)> construct. See below.
a0d0e21e 65
4633a7c4 66The C</x> modifier itself needs a little more explanation. It tells
55497cff
PP
67the regular expression parser to ignore whitespace that is neither
68backslashed nor within a character class. You can use this to break up
4633a7c4 69your regular expression into (slightly) more readable parts. The C<#>
54310121 70character is also treated as a metacharacter introducing a comment,
55497cff 71just as in ordinary Perl code. This also means that if you want real
14218588 72whitespace or C<#> characters in the pattern (outside a character
f9a3ff1a
RGS
73class, where they are unaffected by C</x>), then you'll either have to
74escape them (using backslashes or C<\Q...\E>) or encode them using octal
8933a740
RGS
75or hex escapes. Taken together, these features go a long way towards
76making Perl's regular expressions more readable. Note that you have to
77be careful not to include the pattern delimiter in the comment--perl has
78no way of knowing you did not intend to close the pattern early. See
79the C-comment deletion code in L<perlop>. Also note that anything inside
1031e5db 80a C<\Q...\E> stays unaffected by C</x>.
d74e8afc 81X</x>
a0d0e21e
LW
82
83=head2 Regular Expressions
84
19799a22 85The patterns used in Perl pattern matching derive from supplied in
14218588 86the Version 8 regex routines. (The routines are derived
19799a22
GS
87(distantly) from Henry Spencer's freely redistributable reimplementation
88of the V8 routines.) See L<Version 8 Regular Expressions> for
89details.
a0d0e21e
LW
90
91In particular the following metacharacters have their standard I<egrep>-ish
92meanings:
d74e8afc
ITB
93X<metacharacter>
94X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
95
a0d0e21e 96
54310121 97 \ Quote the next metacharacter
a0d0e21e
LW
98 ^ Match the beginning of the line
99 . Match any character (except newline)
c07a80fd 100 $ Match the end of the line (or before newline at the end)
a0d0e21e
LW
101 | Alternation
102 () Grouping
103 [] Character class
104
14218588
GS
105By default, the "^" character is guaranteed to match only the
106beginning of the string, the "$" character only the end (or before the
107newline at the end), and Perl does certain optimizations with the
a0d0e21e
LW
108assumption that the string contains only one line. Embedded newlines
109will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 110string as a multi-line buffer, such that the "^" will match after any
a0d0e21e
LW
111newline within the string, and "$" will match before any newline. At the
112cost of a little more overhead, you can do this by using the /m modifier
113on the pattern match operator. (Older programs did this by setting C<$*>,
f02c194e 114but this practice has been removed in perl 5.9.)
d74e8afc 115X<^> X<$> X</m>
a0d0e21e 116
14218588 117To simplify multi-line substitutions, the "." character never matches a
55497cff 118newline unless you use the C</s> modifier, which in effect tells Perl to pretend
f02c194e 119the string is a single line--even if it isn't.
d74e8afc 120X<.> X</s>
a0d0e21e
LW
121
122The following standard quantifiers are recognized:
d74e8afc 123X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
a0d0e21e
LW
124
125 * Match 0 or more times
126 + Match 1 or more times
127 ? Match 1 or 0 times
128 {n} Match exactly n times
129 {n,} Match at least n times
130 {n,m} Match at least n but not more than m times
131
132(If a curly bracket occurs in any other context, it is treated
b975c076
JL
133as a regular character. In particular, the lower bound
134is not optional.) The "*" modifier is equivalent to C<{0,}>, the "+"
25f94b33 135modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited
9c79236d
GS
136to integral values less than a preset limit defined when perl is built.
137This is usually 32766 on the most common platforms. The actual limit can
138be seen in the error message generated by code such as this:
139
820475bd 140 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 141
54310121
PP
142By default, a quantified subpattern is "greedy", that is, it will match as
143many times as possible (given a particular starting location) while still
144allowing the rest of the pattern to match. If you want it to match the
145minimum number of times possible, follow the quantifier with a "?". Note
146that the meanings don't change, just the "greediness":
d74e8afc
ITB
147X<metacharacter> X<greedy> X<greedyness>
148X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{n,m}?>
a0d0e21e
LW
149
150 *? Match 0 or more times
151 +? Match 1 or more times
152 ?? Match 0 or 1 time
153 {n}? Match exactly n times
154 {n,}? Match at least n times
155 {n,m}? Match at least n but not more than m times
156
5f05dabc 157Because patterns are processed as double quoted strings, the following
a0d0e21e 158also work:
d74e8afc
ITB
159X<\t> X<\n> X<\r> X<\f> X<\a> X<\l> X<\u> X<\L> X<\U> X<\E> X<\Q>
160X<\0> X<\c> X<\N> X<\x>
a0d0e21e 161
0f36ee90
PP
162 \t tab (HT, TAB)
163 \n newline (LF, NL)
164 \r return (CR)
165 \f form feed (FF)
166 \a alarm (bell) (BEL)
167 \e escape (think troff) (ESC)
cb1a09d0
AD
168 \033 octal char (think of a PDP-11)
169 \x1B hex char
a0ed51b3 170 \x{263a} wide hex char (Unicode SMILEY)
a0d0e21e 171 \c[ control char
4a2d328f 172 \N{name} named char
cb1a09d0
AD
173 \l lowercase next char (think vi)
174 \u uppercase next char (think vi)
175 \L lowercase till \E (think vi)
176 \U uppercase till \E (think vi)
177 \E end case modification (think vi)
5a964f20 178 \Q quote (disable) pattern metacharacters till \E
a0d0e21e 179
a034a98d 180If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u>
423cee85 181and C<\U> is taken from the current locale. See L<perllocale>. For
4a2d328f 182documentation of C<\N{name}>, see L<charnames>.
a034a98d 183
1d2dff63
GS
184You cannot include a literal C<$> or C<@> within a C<\Q> sequence.
185An unescaped C<$> or C<@> interpolates the corresponding variable,
186while escaping will cause the literal string C<\$> to be matched.
187You'll need to write something like C<m/\Quser\E\@\Qhost/>.
188
a0d0e21e 189In addition, Perl defines the following:
d74e8afc
ITB
190X<metacharacter>
191X<\w> X<\W> X<\s> X<\S> X<\d> X<\D> X<\X> X<\p> X<\P> X<\C>
192X<word> X<whitespace>
a0d0e21e
LW
193
194 \w Match a "word" character (alphanumeric plus "_")
36bbe248 195 \W Match a non-"word" character
a0d0e21e
LW
196 \s Match a whitespace character
197 \S Match a non-whitespace character
198 \d Match a digit character
199 \D Match a non-digit character
a0ed51b3
LW
200 \pP Match P, named property. Use \p{Prop} for longer names.
201 \PP Match non-P
f244e06d 202 \X Match eXtended Unicode "combining character sequence",
07698885 203 equivalent to (?:\PM\pM*)
72ff2908 204 \C Match a single C char (octet) even under Unicode.
07698885 205 NOTE: breaks up characters into their UTF-8 bytes,
72ff2908 206 so you may end up with malformed pieces of UTF-8.
f14c76ed 207 Unsupported in lookbehind.
a0d0e21e 208
08ce8fc6
JH
209A C<\w> matches a single alphanumeric character (an alphabetic
210character, or a decimal digit) or C<_>, not a whole word. Use C<\w+>
211to match a string of Perl-identifier characters (which isn't the same
212as matching an English word). If C<use locale> is in effect, the list
213of alphabetic characters generated by C<\w> is taken from the current
214locale. See L<perllocale>. You may use C<\w>, C<\W>, C<\s>, C<\S>,
1209ba90 215C<\d>, and C<\D> within character classes, but if you try to use them
08ce8fc6
JH
216as endpoints of a range, that's not a range, the "-" is understood
217literally. If Unicode is in effect, C<\s> matches also "\x{85}",
218"\x{2028}, and "\x{2029}", see L<perlunicode> for more details about
491fd90a 219C<\pP>, C<\PP>, and C<\X>, and L<perluniintro> about Unicode in general.
fa11829f 220You can define your own C<\p> and C<\P> properties, see L<perlunicode>.
d74e8afc 221X<\w> X<\W> X<word>
a0d0e21e 222
b8c5462f 223The POSIX character class syntax
d74e8afc 224X<character class>
b8c5462f 225
820475bd 226 [:class:]
b8c5462f 227
5496314a
SP
228is also available. Note that the C<[> and C<]> braces are I<literal>;
229they must always be used within a character class expression.
230
231 # this is correct:
232 $string =~ /[[:alpha:]]/;
233
234 # this is not, and will generate a warning:
235 $string =~ /[:alpha:]/;
236
237The available classes and their backslash equivalents (if available) are
238as follows:
d74e8afc
ITB
239X<character class>
240X<alpha> X<alnum> X<ascii> X<blank> X<cntrl> X<digit> X<graph>
241X<lower> X<print> X<punct> X<space> X<upper> X<word> X<xdigit>
b8c5462f
JH
242
243 alpha
244 alnum
245 ascii
aaa51d5e 246 blank [1]
b8c5462f
JH
247 cntrl
248 digit \d
249 graph
250 lower
251 print
252 punct
aaa51d5e 253 space \s [2]
b8c5462f 254 upper
aaa51d5e 255 word \w [3]
b8c5462f
JH
256 xdigit
257
07698885
RGS
258=over
259
260=item [1]
261
b432a672 262A GNU extension equivalent to C<[ \t]>, "all horizontal whitespace".
07698885
RGS
263
264=item [2]
265
266Not exactly equivalent to C<\s> since the C<[[:space:]]> includes
b432a672 267also the (very rare) "vertical tabulator", "\ck", chr(11).
07698885
RGS
268
269=item [3]
270
08ce8fc6 271A Perl extension, see above.
07698885
RGS
272
273=back
aaa51d5e 274
26b44a0a 275For example use C<[:upper:]> to match all the uppercase characters.
aaa51d5e
JF
276Note that the C<[]> are part of the C<[::]> construct, not part of the
277whole character class. For example:
b8c5462f 278
820475bd 279 [01[:alpha:]%]
b8c5462f 280
593df60c 281matches zero, one, any alphabetic character, and the percentage sign.
b8c5462f 282
72ff2908
JH
283The following equivalences to Unicode \p{} constructs and equivalent
284backslash character classes (if available), will hold:
d74e8afc 285X<character class> X<\p> X<\p{}>
72ff2908 286
5496314a 287 [[:...:]] \p{...} backslash
b8c5462f
JH
288
289 alpha IsAlpha
290 alnum IsAlnum
291 ascii IsASCII
b432a672 292 blank IsSpace
b8c5462f 293 cntrl IsCntrl
3bec3564 294 digit IsDigit \d
b8c5462f
JH
295 graph IsGraph
296 lower IsLower
297 print IsPrint
298 punct IsPunct
299 space IsSpace
3bec3564 300 IsSpacePerl \s
b8c5462f
JH
301 upper IsUpper
302 word IsWord
303 xdigit IsXDigit
304
5496314a 305For example C<[[:lower:]]> and C<\p{IsLower}> are equivalent.
b8c5462f
JH
306
307If the C<utf8> pragma is not used but the C<locale> pragma is, the
aaa51d5e 308classes correlate with the usual isalpha(3) interface (except for
b432a672 309"word" and "blank").
b8c5462f
JH
310
311The assumedly non-obviously named classes are:
312
313=over 4
314
315=item cntrl
d74e8afc 316X<cntrl>
b8c5462f 317
820475bd
GS
318Any control character. Usually characters that don't produce output as
319such but instead control the terminal somehow: for example newline and
320backspace are control characters. All characters with ord() less than
593df60c 32132 are most often classified as control characters (assuming ASCII,
7be5a6cf
JF
322the ISO Latin character sets, and Unicode), as is the character with
323the ord() value of 127 (C<DEL>).
b8c5462f
JH
324
325=item graph
d74e8afc 326X<graph>
b8c5462f 327
f1cbbd6e 328Any alphanumeric or punctuation (special) character.
b8c5462f
JH
329
330=item print
d74e8afc 331X<print>
b8c5462f 332
f79b3095 333Any alphanumeric or punctuation (special) character or the space character.
b8c5462f
JH
334
335=item punct
d74e8afc 336X<punct>
b8c5462f 337
f1cbbd6e 338Any punctuation (special) character.
b8c5462f
JH
339
340=item xdigit
d74e8afc 341X<xdigit>
b8c5462f 342
593df60c 343Any hexadecimal digit. Though this may feel silly ([0-9A-Fa-f] would
820475bd 344work just fine) it is included for completeness.
b8c5462f 345
b8c5462f
JH
346=back
347
348You can negate the [::] character classes by prefixing the class name
349with a '^'. This is a Perl extension. For example:
d74e8afc 350X<character class, negation>
b8c5462f 351
5496314a 352 POSIX traditional Unicode
93733859 353
5496314a
SP
354 [[:^digit:]] \D \P{IsDigit}
355 [[:^space:]] \S \P{IsSpace}
356 [[:^word:]] \W \P{IsWord}
b8c5462f 357
54c18d04
MK
358Perl respects the POSIX standard in that POSIX character classes are
359only supported within a character class. The POSIX character classes
360[.cc.] and [=cc=] are recognized but B<not> supported and trying to
361use them will cause an error.
b8c5462f 362
a0d0e21e 363Perl defines the following zero-width assertions:
d74e8afc
ITB
364X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
365X<regexp, zero-width assertion>
366X<regular expression, zero-width assertion>
367X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
a0d0e21e
LW
368
369 \b Match a word boundary
370 \B Match a non-(word boundary)
b85d18e9
IZ
371 \A Match only at beginning of string
372 \Z Match only at end of string, or before newline at the end
373 \z Match only at end of string
9da458fc
IZ
374 \G Match only at pos() (e.g. at the end-of-match position
375 of prior m//g)
a0d0e21e 376
14218588 377A word boundary (C<\b>) is a spot between two characters
19799a22
GS
378that has a C<\w> on one side of it and a C<\W> on the other side
379of it (in either order), counting the imaginary characters off the
380beginning and end of the string as matching a C<\W>. (Within
381character classes C<\b> represents backspace rather than a word
382boundary, just as it normally does in any double-quoted string.)
383The C<\A> and C<\Z> are just like "^" and "$", except that they
384won't match multiple times when the C</m> modifier is used, while
385"^" and "$" will match at every internal line boundary. To match
386the actual end of the string and not ignore an optional trailing
387newline, use C<\z>.
d74e8afc 388X<\b> X<\A> X<\Z> X<\z> X</m>
19799a22
GS
389
390The C<\G> assertion can be used to chain global matches (using
391C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
392It is also useful when writing C<lex>-like scanners, when you have
393several patterns that you want to match against consequent substrings
394of your string, see the previous reference. The actual location
395where C<\G> will match can also be influenced by using C<pos()> as
25cf8c22
HS
396an lvalue: see L<perlfunc/pos>. Currently C<\G> is only fully
397supported when anchored to the start of the pattern; while it
398is permitted to use it elsewhere, as in C</(?<=\G..)./g>, some
399such uses (C</.\G/g>, for example) currently cause problems, and
400it is recommended that you avoid such usage for now.
d74e8afc 401X<\G>
c47ff5f1 402
14218588 403The bracketing construct C<( ... )> creates capture buffers. To
c47ff5f1 404refer to the digit'th buffer use \<digit> within the
14218588 405match. Outside the match use "$" instead of "\". (The
c47ff5f1 406\<digit> notation works in certain circumstances outside
14218588
GS
407the match. See the warning below about \1 vs $1 for details.)
408Referring back to another part of the match is called a
409I<backreference>.
d74e8afc
ITB
410X<regex, capture buffer> X<regexp, capture buffer>
411X<regular expression, capture buffer> X<backreference>
14218588
GS
412
413There is no limit to the number of captured substrings that you may
414use. However Perl also uses \10, \11, etc. as aliases for \010,
fb55449c
JH
415\011, etc. (Recall that 0 means octal, so \011 is the character at
416number 9 in your coded character set; which would be the 10th character,
417a horizontal tab under ASCII.) Perl resolves this
418ambiguity by interpreting \10 as a backreference only if at least 10
419left parentheses have opened before it. Likewise \11 is a
420backreference only if at least 11 left parentheses have opened
421before it. And so on. \1 through \9 are always interpreted as
422backreferences.
14218588
GS
423
424Examples:
a0d0e21e
LW
425
426 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
427
14218588
GS
428 if (/(.)\1/) { # find first doubled char
429 print "'$1' is the first doubled character\n";
430 }
c47ff5f1 431
14218588 432 if (/Time: (..):(..):(..)/) { # parse out values
a0d0e21e
LW
433 $hours = $1;
434 $minutes = $2;
435 $seconds = $3;
436 }
c47ff5f1 437
14218588
GS
438Several special variables also refer back to portions of the previous
439match. C<$+> returns whatever the last bracket match matched.
440C<$&> returns the entire matched string. (At one point C<$0> did
441also, but now it returns the name of the program.) C<$`> returns
77ea4f6d
JV
442everything before the matched string. C<$'> returns everything
443after the matched string. And C<$^N> contains whatever was matched by
444the most-recently closed group (submatch). C<$^N> can be used in
445extended patterns (see below), for example to assign a submatch to a
446variable.
d74e8afc 447X<$+> X<$^N> X<$&> X<$`> X<$'>
14218588 448
665e98b9 449The numbered match variables ($1, $2, $3, etc.) and the related punctuation
77ea4f6d 450set (C<$+>, C<$&>, C<$`>, C<$'>, and C<$^N>) are all dynamically scoped
14218588
GS
451until the end of the enclosing block or until the next successful
452match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
d74e8afc
ITB
453X<$+> X<$^N> X<$&> X<$`> X<$'>
454X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
455
14218588 456
665e98b9 457B<NOTE>: failed matches in Perl do not reset the match variables,
5146ce24 458which makes it easier to write code that tests for a series of more
665e98b9
JH
459specific cases and remembers the best match.
460
14218588
GS
461B<WARNING>: Once Perl sees that you need one of C<$&>, C<$`>, or
462C<$'> anywhere in the program, it has to provide them for every
463pattern match. This may substantially slow your program. Perl
464uses the same mechanism to produce $1, $2, etc, so you also pay a
465price for each pattern that contains capturing parentheses. (To
466avoid this cost while retaining the grouping behaviour, use the
467extended regular expression C<(?: ... )> instead.) But if you never
468use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
469parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
470if you can, but if you can't (and some algorithms really appreciate
471them), once you've used them once, use them at will, because you've
472already paid the price. As of 5.005, C<$&> is not so costly as the
473other two.
d74e8afc 474X<$&> X<$`> X<$'>
68dc0745 475
19799a22
GS
476Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
477C<\w>, C<\n>. Unlike some other regular expression languages, there
478are no backslashed symbols that aren't alphanumeric. So anything
c47ff5f1 479that looks like \\, \(, \), \<, \>, \{, or \} is always
19799a22
GS
480interpreted as a literal character, not a metacharacter. This was
481once used in a common idiom to disable or quote the special meanings
482of regular expression metacharacters in a string that you want to
36bbe248 483use for a pattern. Simply quote all non-"word" characters:
a0d0e21e
LW
484
485 $pattern =~ s/(\W)/\\$1/g;
486
f1cbbd6e 487(If C<use locale> is set, then this depends on the current locale.)
14218588
GS
488Today it is more common to use the quotemeta() function or the C<\Q>
489metaquoting escape sequence to disable all metacharacters' special
490meanings like this:
a0d0e21e
LW
491
492 /$unquoted\Q$quoted\E$unquoted/
493
9da458fc
IZ
494Beware that if you put literal backslashes (those not inside
495interpolated variables) between C<\Q> and C<\E>, double-quotish
496backslash interpolation may lead to confusing results. If you
497I<need> to use literal backslashes within C<\Q...\E>,
498consult L<perlop/"Gory details of parsing quoted constructs">.
499
19799a22
GS
500=head2 Extended Patterns
501
14218588
GS
502Perl also defines a consistent extension syntax for features not
503found in standard tools like B<awk> and B<lex>. The syntax is a
504pair of parentheses with a question mark as the first thing within
505the parentheses. The character after the question mark indicates
506the extension.
19799a22 507
14218588
GS
508The stability of these extensions varies widely. Some have been
509part of the core language for many years. Others are experimental
510and may change without warning or be completely removed. Check
511the documentation on an individual feature to verify its current
512status.
19799a22 513
14218588
GS
514A question mark was chosen for this and for the minimal-matching
515construct because 1) question marks are rare in older regular
516expressions, and 2) whenever you see one, you should stop and
517"question" exactly what is going on. That's psychology...
a0d0e21e
LW
518
519=over 10
520
cc6b7395 521=item C<(?#text)>
d74e8afc 522X<(?#)>
a0d0e21e 523
14218588 524A comment. The text is ignored. If the C</x> modifier enables
19799a22 525whitespace formatting, a simple C<#> will suffice. Note that Perl closes
259138e3
GS
526the comment as soon as it sees a C<)>, so there is no way to put a literal
527C<)> in the comment.
a0d0e21e 528
19799a22 529=item C<(?imsx-imsx)>
d74e8afc 530X<(?)>
19799a22 531
0b6d1084
JH
532One or more embedded pattern-match modifiers, to be turned on (or
533turned off, if preceded by C<->) for the remainder of the pattern or
534the remainder of the enclosing pattern group (if any). This is
535particularly useful for dynamic patterns, such as those read in from a
536configuration file, read in as an argument, are specified in a table
537somewhere, etc. Consider the case that some of which want to be case
538sensitive and some do not. The case insensitive ones need to include
539merely C<(?i)> at the front of the pattern. For example:
19799a22
GS
540
541 $pattern = "foobar";
542 if ( /$pattern/i ) { }
543
544 # more flexible:
545
546 $pattern = "(?i)foobar";
547 if ( /$pattern/ ) { }
548
0b6d1084 549These modifiers are restored at the end of the enclosing group. For example,
19799a22
GS
550
551 ( (?i) blah ) \s+ \1
552
553will match a repeated (I<including the case>!) word C<blah> in any
14218588 554case, assuming C<x> modifier, and no C<i> modifier outside this
19799a22
GS
555group.
556
5a964f20 557=item C<(?:pattern)>
d74e8afc 558X<(?:)>
a0d0e21e 559
ca9dfc88
IZ
560=item C<(?imsx-imsx:pattern)>
561
5a964f20
TC
562This is for clustering, not capturing; it groups subexpressions like
563"()", but doesn't make backreferences as "()" does. So
a0d0e21e 564
5a964f20 565 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e
LW
566
567is like
568
5a964f20 569 @fields = split(/\b(a|b|c)\b/)
a0d0e21e 570
19799a22
GS
571but doesn't spit out extra fields. It's also cheaper not to capture
572characters if you don't need to.
a0d0e21e 573
19799a22
GS
574Any letters between C<?> and C<:> act as flags modifiers as with
575C<(?imsx-imsx)>. For example,
ca9dfc88
IZ
576
577 /(?s-i:more.*than).*million/i
578
14218588 579is equivalent to the more verbose
ca9dfc88
IZ
580
581 /(?:(?s-i)more.*than).*million/i
582
5a964f20 583=item C<(?=pattern)>
d74e8afc 584X<(?=)> X<look-ahead, positive> X<lookahead, positive>
a0d0e21e 585
19799a22 586A zero-width positive look-ahead assertion. For example, C</\w+(?=\t)/>
a0d0e21e
LW
587matches a word followed by a tab, without including the tab in C<$&>.
588
5a964f20 589=item C<(?!pattern)>
d74e8afc 590X<(?!)> X<look-ahead, negative> X<lookahead, negative>
a0d0e21e 591
19799a22 592A zero-width negative look-ahead assertion. For example C</foo(?!bar)/>
a0d0e21e 593matches any occurrence of "foo" that isn't followed by "bar". Note
19799a22
GS
594however that look-ahead and look-behind are NOT the same thing. You cannot
595use this for look-behind.
7b8d334a 596
5a964f20 597If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a
GS
598will not do what you want. That's because the C<(?!foo)> is just saying that
599the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
600match. You would have to do something like C</(?!foo)...bar/> for that. We
601say "like" because there's the case of your "bar" not having three characters
602before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
603Sometimes it's still easier just to say:
a0d0e21e 604
a3cb178b 605 if (/bar/ && $` !~ /foo$/)
a0d0e21e 606
19799a22 607For look-behind see below.
c277df42 608
c47ff5f1 609=item C<(?<=pattern)>
d74e8afc 610X<(?<=)> X<look-behind, positive> X<lookbehind, positive>
c277df42 611
c47ff5f1 612A zero-width positive look-behind assertion. For example, C</(?<=\t)\w+/>
19799a22
GS
613matches a word that follows a tab, without including the tab in C<$&>.
614Works only for fixed-width look-behind.
c277df42 615
5a964f20 616=item C<(?<!pattern)>
d74e8afc 617X<(?<!)> X<look-behind, negative> X<lookbehind, negative>
c277df42 618
19799a22
GS
619A zero-width negative look-behind assertion. For example C</(?<!bar)foo/>
620matches any occurrence of "foo" that does not follow "bar". Works
621only for fixed-width look-behind.
c277df42 622
cc6b7395 623=item C<(?{ code })>
d74e8afc 624X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
c277df42 625
19799a22
GS
626B<WARNING>: This extended regular expression feature is considered
627highly experimental, and may be changed or deleted without notice.
c277df42 628
cc46d5f2 629This zero-width assertion evaluates any embedded Perl code. It
19799a22
GS
630always succeeds, and its C<code> is not interpolated. Currently,
631the rules to determine where the C<code> ends are somewhat convoluted.
632
77ea4f6d
JV
633This feature can be used together with the special variable C<$^N> to
634capture the results of submatches in variables without having to keep
635track of the number of nested parentheses. For example:
636
637 $_ = "The brown fox jumps over the lazy dog";
638 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
639 print "color = $color, animal = $animal\n";
640
754091cb
RGS
641Inside the C<(?{...})> block, C<$_> refers to the string the regular
642expression is matching against. You can also use C<pos()> to know what is
fa11829f 643the current position of matching within this string.
754091cb 644
19799a22
GS
645The C<code> is properly scoped in the following sense: If the assertion
646is backtracked (compare L<"Backtracking">), all changes introduced after
647C<local>ization are undone, so that
b9ac3b5b
GS
648
649 $_ = 'a' x 8;
650 m<
651 (?{ $cnt = 0 }) # Initialize $cnt.
652 (
653 a
654 (?{
655 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
656 })
657 )*
658 aaaa
659 (?{ $res = $cnt }) # On success copy to non-localized
660 # location.
661 >x;
662
19799a22 663will set C<$res = 4>. Note that after the match, $cnt returns to the globally
14218588 664introduced value, because the scopes that restrict C<local> operators
b9ac3b5b
GS
665are unwound.
666
19799a22
GS
667This assertion may be used as a C<(?(condition)yes-pattern|no-pattern)>
668switch. If I<not> used in this way, the result of evaluation of
669C<code> is put into the special variable C<$^R>. This happens
670immediately, so C<$^R> can be used from other C<(?{ code })> assertions
671inside the same regular expression.
b9ac3b5b 672
19799a22
GS
673The assignment to C<$^R> above is properly localized, so the old
674value of C<$^R> is restored if the assertion is backtracked; compare
675L<"Backtracking">.
b9ac3b5b 676
19799a22
GS
677For reasons of security, this construct is forbidden if the regular
678expression involves run-time interpolation of variables, unless the
679perilous C<use re 'eval'> pragma has been used (see L<re>), or the
680variables contain results of C<qr//> operator (see
681L<perlop/"qr/STRING/imosx">).
871b0233 682
14218588 683This restriction is because of the wide-spread and remarkably convenient
19799a22 684custom of using run-time determined strings as patterns. For example:
871b0233
IZ
685
686 $re = <>;
687 chomp $re;
688 $string =~ /$re/;
689
14218588
GS
690Before Perl knew how to execute interpolated code within a pattern,
691this operation was completely safe from a security point of view,
692although it could raise an exception from an illegal pattern. If
693you turn on the C<use re 'eval'>, though, it is no longer secure,
694so you should only do so if you are also using taint checking.
695Better yet, use the carefully constrained evaluation within a Safe
cc46d5f2 696compartment. See L<perlsec> for details about both these mechanisms.
871b0233 697
8988a1bb
DD
698Because perl's regex engine is not currently re-entrant, interpolated
699code may not invoke the regex engine either directly with C<m//> or C<s///>),
700or indirectly with functions such as C<split>.
701
14455d6c 702=item C<(??{ code })>
d74e8afc
ITB
703X<(??{})>
704X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
705X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
0f5d15d6 706
19799a22
GS
707B<WARNING>: This extended regular expression feature is considered
708highly experimental, and may be changed or deleted without notice.
9da458fc
IZ
709A simplified version of the syntax may be introduced for commonly
710used idioms.
0f5d15d6 711
19799a22
GS
712This is a "postponed" regular subexpression. The C<code> is evaluated
713at run time, at the moment this subexpression may match. The result
714of evaluation is considered as a regular expression and matched as
715if it were inserted instead of this construct.
0f5d15d6 716
428594d9 717The C<code> is not interpolated. As before, the rules to determine
19799a22
GS
718where the C<code> ends are currently somewhat convoluted.
719
720The following pattern matches a parenthesized group:
0f5d15d6
IZ
721
722 $re = qr{
723 \(
724 (?:
725 (?> [^()]+ ) # Non-parens without backtracking
726 |
14455d6c 727 (??{ $re }) # Group with matching parens
0f5d15d6
IZ
728 )*
729 \)
730 }x;
731
8988a1bb
DD
732Because perl's regex engine is not currently re-entrant, delayed
733code may not invoke the regex engine either directly with C<m//> or C<s///>),
734or indirectly with functions such as C<split>.
735
c47ff5f1 736=item C<< (?>pattern) >>
d74e8afc 737X<backtrack> X<backtracking>
5a964f20 738
19799a22
GS
739B<WARNING>: This extended regular expression feature is considered
740highly experimental, and may be changed or deleted without notice.
741
742An "independent" subexpression, one which matches the substring
743that a I<standalone> C<pattern> would match if anchored at the given
9da458fc 744position, and it matches I<nothing other than this substring>. This
19799a22
GS
745construct is useful for optimizations of what would otherwise be
746"eternal" matches, because it will not backtrack (see L<"Backtracking">).
9da458fc
IZ
747It may also be useful in places where the "grab all you can, and do not
748give anything back" semantic is desirable.
19799a22 749
c47ff5f1 750For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
19799a22
GS
751(anchored at the beginning of string, as above) will match I<all>
752characters C<a> at the beginning of string, leaving no C<a> for
753C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
754since the match of the subgroup C<a*> is influenced by the following
755group C<ab> (see L<"Backtracking">). In particular, C<a*> inside
756C<a*ab> will match fewer characters than a standalone C<a*>, since
757this makes the tail match.
758
c47ff5f1 759An effect similar to C<< (?>pattern) >> may be achieved by writing
19799a22
GS
760C<(?=(pattern))\1>. This matches the same substring as a standalone
761C<a+>, and the following C<\1> eats the matched string; it therefore
c47ff5f1 762makes a zero-length assertion into an analogue of C<< (?>...) >>.
19799a22
GS
763(The difference between these two constructs is that the second one
764uses a capturing group, thus shifting ordinals of backreferences
765in the rest of a regular expression.)
766
767Consider this pattern:
c277df42 768
871b0233
IZ
769 m{ \(
770 (
9da458fc 771 [^()]+ # x+
871b0233
IZ
772 |
773 \( [^()]* \)
774 )+
775 \)
776 }x
5a964f20 777
19799a22
GS
778That will efficiently match a nonempty group with matching parentheses
779two levels deep or less. However, if there is no such group, it
780will take virtually forever on a long string. That's because there
781are so many different ways to split a long string into several
782substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
783to a subpattern of the above pattern. Consider how the pattern
784above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
785seconds, but that each extra letter doubles this time. This
786exponential performance will make it appear that your program has
14218588 787hung. However, a tiny change to this pattern
5a964f20 788
871b0233
IZ
789 m{ \(
790 (
9da458fc 791 (?> [^()]+ ) # change x+ above to (?> x+ )
871b0233
IZ
792 |
793 \( [^()]* \)
794 )+
795 \)
796 }x
c277df42 797
c47ff5f1 798which uses C<< (?>...) >> matches exactly when the one above does (verifying
5a964f20
TC
799this yourself would be a productive exercise), but finishes in a fourth
800the time when used on a similar string with 1000000 C<a>s. Be aware,
801however, that this pattern currently triggers a warning message under
9f1b1f2d 802the C<use warnings> pragma or B<-w> switch saying it
6bab786b 803C<"matches null string many times in regex">.
c277df42 804
c47ff5f1 805On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
19799a22 806effect may be achieved by negative look-ahead, as in C<[^()]+ (?! [^()] )>.
c277df42
IZ
807This was only 4 times slower on a string with 1000000 C<a>s.
808
9da458fc
IZ
809The "grab all you can, and do not give anything back" semantic is desirable
810in many situations where on the first sight a simple C<()*> looks like
811the correct solution. Suppose we parse text with comments being delimited
812by C<#> followed by some optional (horizontal) whitespace. Contrary to
4375e838 813its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
9da458fc
IZ
814the comment delimiter, because it may "give up" some whitespace if
815the remainder of the pattern can be made to match that way. The correct
816answer is either one of these:
817
818 (?>#[ \t]*)
819 #[ \t]*(?![ \t])
820
821For example, to grab non-empty comments into $1, one should use either
822one of these:
823
824 / (?> \# [ \t]* ) ( .+ ) /x;
825 / \# [ \t]* ( [^ \t] .* ) /x;
826
827Which one you pick depends on which of these expressions better reflects
828the above specification of comments.
829
5a964f20 830=item C<(?(condition)yes-pattern|no-pattern)>
d74e8afc 831X<(?()>
c277df42 832
5a964f20 833=item C<(?(condition)yes-pattern)>
c277df42 834
19799a22
GS
835B<WARNING>: This extended regular expression feature is considered
836highly experimental, and may be changed or deleted without notice.
837
c277df42
IZ
838Conditional expression. C<(condition)> should be either an integer in
839parentheses (which is valid if the corresponding pair of parentheses
19799a22 840matched), or look-ahead/look-behind/evaluate zero-width assertion.
c277df42 841
19799a22 842For example:
c277df42 843
5a964f20 844 m{ ( \( )?
871b0233 845 [^()]+
5a964f20 846 (?(1) \) )
871b0233 847 }x
c277df42
IZ
848
849matches a chunk of non-parentheses, possibly included in parentheses
850themselves.
a0d0e21e 851
a0d0e21e
LW
852=back
853
c07a80fd 854=head2 Backtracking
d74e8afc 855X<backtrack> X<backtracking>
c07a80fd 856
35a734be
IZ
857NOTE: This section presents an abstract approximation of regular
858expression behavior. For a more rigorous (and complicated) view of
859the rules involved in selecting a match among possible alternatives,
860see L<Combining pieces together>.
861
c277df42 862A fundamental feature of regular expression matching involves the
5a964f20 863notion called I<backtracking>, which is currently used (when needed)
c277df42 864by all regular expression quantifiers, namely C<*>, C<*?>, C<+>,
9da458fc
IZ
865C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often optimized
866internally, but the general principle outlined here is valid.
c07a80fd
PP
867
868For a regular expression to match, the I<entire> regular expression must
869match, not just part of it. So if the beginning of a pattern containing a
870quantifier succeeds in a way that causes later parts in the pattern to
871fail, the matching engine backs up and recalculates the beginning
872part--that's why it's called backtracking.
873
874Here is an example of backtracking: Let's say you want to find the
875word following "foo" in the string "Food is on the foo table.":
876
877 $_ = "Food is on the foo table.";
878 if ( /\b(foo)\s+(\w+)/i ) {
879 print "$2 follows $1.\n";
880 }
881
882When the match runs, the first part of the regular expression (C<\b(foo)>)
883finds a possible match right at the beginning of the string, and loads up
884$1 with "Foo". However, as soon as the matching engine sees that there's
885no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 886mistake and starts over again one character after where it had the
c07a80fd
PP
887tentative match. This time it goes all the way until the next occurrence
888of "foo". The complete regular expression matches this time, and you get
889the expected output of "table follows foo."
890
891Sometimes minimal matching can help a lot. Imagine you'd like to match
892everything between "foo" and "bar". Initially, you write something
893like this:
894
895 $_ = "The food is under the bar in the barn.";
896 if ( /foo(.*)bar/ ) {
897 print "got <$1>\n";
898 }
899
900Which perhaps unexpectedly yields:
901
902 got <d is under the bar in the >
903
904That's because C<.*> was greedy, so you get everything between the
14218588 905I<first> "foo" and the I<last> "bar". Here it's more effective
c07a80fd
PP
906to use minimal matching to make sure you get the text between a "foo"
907and the first "bar" thereafter.
908
909 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
910 got <d is under the >
911
912Here's another example: let's say you'd like to match a number at the end
b6e13d97 913of a string, and you also want to keep the preceding part of the match.
c07a80fd
PP
914So you write this:
915
916 $_ = "I have 2 numbers: 53147";
917 if ( /(.*)(\d*)/ ) { # Wrong!
918 print "Beginning is <$1>, number is <$2>.\n";
919 }
920
921That won't work at all, because C<.*> was greedy and gobbled up the
922whole string. As C<\d*> can match on an empty string the complete
923regular expression matched successfully.
924
8e1088bc 925 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd
PP
926
927Here are some variants, most of which don't work:
928
929 $_ = "I have 2 numbers: 53147";
930 @pats = qw{
931 (.*)(\d*)
932 (.*)(\d+)
933 (.*?)(\d*)
934 (.*?)(\d+)
935 (.*)(\d+)$
936 (.*?)(\d+)$
937 (.*)\b(\d+)$
938 (.*\D)(\d+)$
939 };
940
941 for $pat (@pats) {
942 printf "%-12s ", $pat;
943 if ( /$pat/ ) {
944 print "<$1> <$2>\n";
945 } else {
946 print "FAIL\n";
947 }
948 }
949
950That will print out:
951
952 (.*)(\d*) <I have 2 numbers: 53147> <>
953 (.*)(\d+) <I have 2 numbers: 5314> <7>
954 (.*?)(\d*) <> <>
955 (.*?)(\d+) <I have > <2>
956 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
957 (.*?)(\d+)$ <I have 2 numbers: > <53147>
958 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
959 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
960
961As you see, this can be a bit tricky. It's important to realize that a
962regular expression is merely a set of assertions that gives a definition
963of success. There may be 0, 1, or several different ways that the
964definition might succeed against a particular string. And if there are
5a964f20
TC
965multiple ways it might succeed, you need to understand backtracking to
966know which variety of success you will achieve.
c07a80fd 967
19799a22 968When using look-ahead assertions and negations, this can all get even
8b19b778 969trickier. Imagine you'd like to find a sequence of non-digits not
c07a80fd
PP
970followed by "123". You might try to write that as
971
871b0233
IZ
972 $_ = "ABC123";
973 if ( /^\D*(?!123)/ ) { # Wrong!
974 print "Yup, no 123 in $_\n";
975 }
c07a80fd
PP
976
977But that isn't going to match; at least, not the way you're hoping. It
978claims that there is no 123 in the string. Here's a clearer picture of
9b9391b2 979why that pattern matches, contrary to popular expectations:
c07a80fd 980
4358a253
SS
981 $x = 'ABC123';
982 $y = 'ABC445';
c07a80fd 983
4358a253
SS
984 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/;
985 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/;
c07a80fd 986
4358a253
SS
987 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/;
988 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/;
c07a80fd
PP
989
990This prints
991
992 2: got ABC
993 3: got AB
994 4: got ABC
995
5f05dabc 996You might have expected test 3 to fail because it seems to a more
c07a80fd
PP
997general purpose version of test 1. The important difference between
998them is that test 3 contains a quantifier (C<\D*>) and so can use
999backtracking, whereas test 1 will not. What's happening is
1000that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 1001non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 1002let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 1003fail.
14218588 1004
c07a80fd 1005The search engine will initially match C<\D*> with "ABC". Then it will
14218588 1006try to match C<(?!123> with "123", which fails. But because
c07a80fd
PP
1007a quantifier (C<\D*>) has been used in the regular expression, the
1008search engine can backtrack and retry the match differently
54310121 1009in the hope of matching the complete regular expression.
c07a80fd 1010
5a964f20
TC
1011The pattern really, I<really> wants to succeed, so it uses the
1012standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd 1013time. Now there's indeed something following "AB" that is not
14218588 1014"123". It's "C123", which suffices.
c07a80fd 1015
14218588
GS
1016We can deal with this by using both an assertion and a negation.
1017We'll say that the first part in $1 must be followed both by a digit
1018and by something that's not "123". Remember that the look-aheads
1019are zero-width expressions--they only look, but don't consume any
1020of the string in their match. So rewriting this way produces what
c07a80fd
PP
1021you'd expect; that is, case 5 will fail, but case 6 succeeds:
1022
4358a253
SS
1023 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/;
1024 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/;
c07a80fd
PP
1025
1026 6: got ABC
1027
5a964f20 1028In other words, the two zero-width assertions next to each other work as though
19799a22 1029they're ANDed together, just as you'd use any built-in assertions: C</^$/>
c07a80fd
PP
1030matches only if you're at the beginning of the line AND the end of the
1031line simultaneously. The deeper underlying truth is that juxtaposition in
1032regular expressions always means AND, except when you write an explicit OR
1033using the vertical bar. C</ab/> means match "a" AND (then) match "b",
1034although the attempted matches are made at different positions because "a"
1035is not a zero-width assertion, but a one-width assertion.
1036
19799a22 1037B<WARNING>: particularly complicated regular expressions can take
14218588 1038exponential time to solve because of the immense number of possible
9da458fc
IZ
1039ways they can use backtracking to try match. For example, without
1040internal optimizations done by the regular expression engine, this will
1041take a painfully long time to run:
c07a80fd 1042
e1901655
IZ
1043 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
1044
1045And if you used C<*>'s in the internal groups instead of limiting them
1046to 0 through 5 matches, then it would take forever--or until you ran
1047out of stack space. Moreover, these internal optimizations are not
1048always applicable. For example, if you put C<{0,5}> instead of C<*>
1049on the external group, no current optimization is applicable, and the
1050match takes a long time to finish.
c07a80fd 1051
9da458fc
IZ
1052A powerful tool for optimizing such beasts is what is known as an
1053"independent group",
c47ff5f1 1054which does not backtrack (see L<C<< (?>pattern) >>>). Note also that
9da458fc 1055zero-length look-ahead/look-behind assertions will not backtrack to make
14218588
GS
1056the tail match, since they are in "logical" context: only
1057whether they match is considered relevant. For an example
9da458fc 1058where side-effects of look-ahead I<might> have influenced the
c47ff5f1 1059following match, see L<C<< (?>pattern) >>>.
c277df42 1060
a0d0e21e 1061=head2 Version 8 Regular Expressions
d74e8afc 1062X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
a0d0e21e 1063
5a964f20 1064In case you're not familiar with the "regular" Version 8 regex
a0d0e21e
LW
1065routines, here are the pattern-matching rules not described above.
1066
54310121 1067Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 1068with a special meaning described here or above. You can cause
5a964f20 1069characters that normally function as metacharacters to be interpreted
5f05dabc 1070literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
a0d0e21e
LW
1071character; "\\" matches a "\"). A series of characters matches that
1072series of characters in the target string, so the pattern C<blurfl>
1073would match "blurfl" in the target string.
1074
1075You can specify a character class, by enclosing a list of characters
5a964f20 1076in C<[]>, which will match any one character from the list. If the
a0d0e21e 1077first character after the "[" is "^", the class matches any character not
14218588 1078in the list. Within a list, the "-" character specifies a
5a964f20 1079range, so that C<a-z> represents all characters between "a" and "z",
8a4f6ac2
GS
1080inclusive. If you want either "-" or "]" itself to be a member of a
1081class, put it at the start of the list (possibly after a "^"), or
1082escape it with a backslash. "-" is also taken literally when it is
1083at the end of the list, just before the closing "]". (The
84850974
DD
1084following all specify the same class of three characters: C<[-az]>,
1085C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
fb55449c
JH
1086specifies a class containing twenty-six characters, even on EBCDIC
1087based coded character sets.) Also, if you try to use the character
1088classes C<\w>, C<\W>, C<\s>, C<\S>, C<\d>, or C<\D> as endpoints of
1089a range, that's not a range, the "-" is understood literally.
a0d0e21e 1090
8ada0baa
JH
1091Note also that the whole range idea is rather unportable between
1092character sets--and even within character sets they may cause results
1093you probably didn't expect. A sound principle is to use only ranges
1094that begin from and end at either alphabets of equal case ([a-e],
1095[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
1096spell out the character sets in full.
1097
54310121 1098Characters may be specified using a metacharacter syntax much like that
a0d0e21e
LW
1099used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
1100"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
fb55449c
JH
1101of octal digits, matches the character whose coded character set value
1102is I<nnn>. Similarly, \xI<nn>, where I<nn> are hexadecimal digits,
1103matches the character whose numeric value is I<nn>. The expression \cI<x>
1104matches the character control-I<x>. Finally, the "." metacharacter
1105matches any character except "\n" (unless you use C</s>).
a0d0e21e
LW
1106
1107You can specify a series of alternatives for a pattern using "|" to
1108separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 1109or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e
LW
1110first alternative includes everything from the last pattern delimiter
1111("(", "[", or the beginning of the pattern) up to the first "|", and
1112the last alternative contains everything from the last "|" to the next
14218588
GS
1113pattern delimiter. That's why it's common practice to include
1114alternatives in parentheses: to minimize confusion about where they
a3cb178b
GS
1115start and end.
1116
5a964f20 1117Alternatives are tried from left to right, so the first
a3cb178b
GS
1118alternative found for which the entire expression matches, is the one that
1119is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 1120example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b
GS
1121part will match, as that is the first alternative tried, and it successfully
1122matches the target string. (This might not seem important, but it is
1123important when you are capturing matched text using parentheses.)
1124
5a964f20 1125Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 1126so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 1127
14218588
GS
1128Within a pattern, you may designate subpatterns for later reference
1129by enclosing them in parentheses, and you may refer back to the
1130I<n>th subpattern later in the pattern using the metacharacter
1131\I<n>. Subpatterns are numbered based on the left to right order
1132of their opening parenthesis. A backreference matches whatever
1133actually matched the subpattern in the string being examined, not
1134the rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will
1135match "0x1234 0x4321", but not "0x1234 01234", because subpattern
11361 matched "0x", even though the rule C<0|0x> could potentially match
1137the leading 0 in the second number.
cb1a09d0 1138
19799a22 1139=head2 Warning on \1 vs $1
cb1a09d0 1140
5a964f20 1141Some people get too used to writing things like:
cb1a09d0
AD
1142
1143 $pattern =~ s/(\W)/\\\1/g;
1144
1145This is grandfathered for the RHS of a substitute to avoid shocking the
1146B<sed> addicts, but it's a dirty habit to get into. That's because in
d1be9408 1147PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
cb1a09d0
AD
1148the usual double-quoted string means a control-A. The customary Unix
1149meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
1150of doing that, you get yourself into trouble if you then add an C</e>
1151modifier.
1152
5a964f20 1153 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0
AD
1154
1155Or if you try to do
1156
1157 s/(\d+)/\1000/;
1158
1159You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
14218588 1160C<${1}000>. The operation of interpolation should not be confused
cb1a09d0
AD
1161with the operation of matching a backreference. Certainly they mean two
1162different things on the I<left> side of the C<s///>.
9fa51da4 1163
c84d73f1
IZ
1164=head2 Repeated patterns matching zero-length substring
1165
19799a22 1166B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
c84d73f1
IZ
1167
1168Regular expressions provide a terse and powerful programming language. As
1169with most other power tools, power comes together with the ability
1170to wreak havoc.
1171
1172A common abuse of this power stems from the ability to make infinite
628afcb5 1173loops using regular expressions, with something as innocuous as:
c84d73f1
IZ
1174
1175 'foo' =~ m{ ( o? )* }x;
1176
1177The C<o?> can match at the beginning of C<'foo'>, and since the position
1178in the string is not moved by the match, C<o?> would match again and again
14218588 1179because of the C<*> modifier. Another common way to create a similar cycle
c84d73f1
IZ
1180is with the looping modifier C<//g>:
1181
1182 @matches = ( 'foo' =~ m{ o? }xg );
1183
1184or
1185
1186 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
1187
1188or the loop implied by split().
1189
1190However, long experience has shown that many programming tasks may
14218588
GS
1191be significantly simplified by using repeated subexpressions that
1192may match zero-length substrings. Here's a simple example being:
c84d73f1
IZ
1193
1194 @chars = split //, $string; # // is not magic in split
1195 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
1196
9da458fc 1197Thus Perl allows such constructs, by I<forcefully breaking
c84d73f1
IZ
1198the infinite loop>. The rules for this are different for lower-level
1199loops given by the greedy modifiers C<*+{}>, and for higher-level
1200ones like the C</g> modifier or split() operator.
1201
19799a22
GS
1202The lower-level loops are I<interrupted> (that is, the loop is
1203broken) when Perl detects that a repeated expression matched a
1204zero-length substring. Thus
c84d73f1
IZ
1205
1206 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
1207
1208is made equivalent to
1209
1210 m{ (?: NON_ZERO_LENGTH )*
1211 |
1212 (?: ZERO_LENGTH )?
1213 }x;
1214
1215The higher level-loops preserve an additional state between iterations:
1216whether the last match was zero-length. To break the loop, the following
1217match after a zero-length match is prohibited to have a length of zero.
1218This prohibition interacts with backtracking (see L<"Backtracking">),
1219and so the I<second best> match is chosen if the I<best> match is of
1220zero length.
1221
19799a22 1222For example:
c84d73f1
IZ
1223
1224 $_ = 'bar';
1225 s/\w??/<$&>/g;
1226
20fb949f 1227results in C<< <><b><><a><><r><> >>. At each position of the string the best
c84d73f1
IZ
1228match given by non-greedy C<??> is the zero-length match, and the I<second
1229best> match is what is matched by C<\w>. Thus zero-length matches
1230alternate with one-character-long matches.
1231
1232Similarly, for repeated C<m/()/g> the second-best match is the match at the
1233position one notch further in the string.
1234
19799a22 1235The additional state of being I<matched with zero-length> is associated with
c84d73f1 1236the matched string, and is reset by each assignment to pos().
9da458fc
IZ
1237Zero-length matches at the end of the previous match are ignored
1238during C<split>.
c84d73f1 1239
35a734be
IZ
1240=head2 Combining pieces together
1241
1242Each of the elementary pieces of regular expressions which were described
1243before (such as C<ab> or C<\Z>) could match at most one substring
1244at the given position of the input string. However, in a typical regular
1245expression these elementary pieces are combined into more complicated
1246patterns using combining operators C<ST>, C<S|T>, C<S*> etc
1247(in these examples C<S> and C<T> are regular subexpressions).
1248
1249Such combinations can include alternatives, leading to a problem of choice:
1250if we match a regular expression C<a|ab> against C<"abc">, will it match
1251substring C<"a"> or C<"ab">? One way to describe which substring is
1252actually matched is the concept of backtracking (see L<"Backtracking">).
1253However, this description is too low-level and makes you think
1254in terms of a particular implementation.
1255
1256Another description starts with notions of "better"/"worse". All the
1257substrings which may be matched by the given regular expression can be
1258sorted from the "best" match to the "worst" match, and it is the "best"
1259match which is chosen. This substitutes the question of "what is chosen?"
1260by the question of "which matches are better, and which are worse?".
1261
1262Again, for elementary pieces there is no such question, since at most
1263one match at a given position is possible. This section describes the
1264notion of better/worse for combining operators. In the description
1265below C<S> and C<T> are regular subexpressions.
1266
13a2d996 1267=over 4
35a734be
IZ
1268
1269=item C<ST>
1270
1271Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
1272substrings which can be matched by C<S>, C<B> and C<B'> are substrings
1273which can be matched by C<T>.
1274
1275If C<A> is better match for C<S> than C<A'>, C<AB> is a better
1276match than C<A'B'>.
1277
1278If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
1279C<B> is better match for C<T> than C<B'>.
1280
1281=item C<S|T>
1282
1283When C<S> can match, it is a better match than when only C<T> can match.
1284
1285Ordering of two matches for C<S> is the same as for C<S>. Similar for
1286two matches for C<T>.
1287
1288=item C<S{REPEAT_COUNT}>
1289
1290Matches as C<SSS...S> (repeated as many times as necessary).
1291
1292=item C<S{min,max}>
1293
1294Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
1295
1296=item C<S{min,max}?>
1297
1298Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
1299
1300=item C<S?>, C<S*>, C<S+>
1301
1302Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
1303
1304=item C<S??>, C<S*?>, C<S+?>
1305
1306Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
1307
c47ff5f1 1308=item C<< (?>S) >>
35a734be
IZ
1309
1310Matches the best match for C<S> and only that.
1311
1312=item C<(?=S)>, C<(?<=S)>
1313
1314Only the best match for C<S> is considered. (This is important only if
1315C<S> has capturing parentheses, and backreferences are used somewhere
1316else in the whole regular expression.)
1317
1318=item C<(?!S)>, C<(?<!S)>
1319
1320For this grouping operator there is no need to describe the ordering, since
1321only whether or not C<S> can match is important.
1322
14455d6c 1323=item C<(??{ EXPR })>
35a734be
IZ
1324
1325The ordering is the same as for the regular expression which is
1326the result of EXPR.
1327
1328=item C<(?(condition)yes-pattern|no-pattern)>
1329
1330Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
1331already determined. The ordering of the matches is the same as for the
1332chosen subexpression.
1333
1334=back
1335
1336The above recipes describe the ordering of matches I<at a given position>.
1337One more rule is needed to understand how a match is determined for the
1338whole regular expression: a match at an earlier position is always better
1339than a match at a later position.
1340
c84d73f1
IZ
1341=head2 Creating custom RE engines
1342
1343Overloaded constants (see L<overload>) provide a simple way to extend
1344the functionality of the RE engine.
1345
1346Suppose that we want to enable a new RE escape-sequence C<\Y|> which
6b0ac556 1347matches at boundary between whitespace characters and non-whitespace
c84d73f1
IZ
1348characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
1349at these positions, so we want to have each C<\Y|> in the place of the
1350more complicated version. We can create a module C<customre> to do
1351this:
1352
1353 package customre;
1354 use overload;
1355
1356 sub import {
1357 shift;
1358 die "No argument to customre::import allowed" if @_;
1359 overload::constant 'qr' => \&convert;
1360 }
1361
1362 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
1363
580a9fe1
RGS
1364 # We must also take care of not escaping the legitimate \\Y|
1365 # sequence, hence the presence of '\\' in the conversion rules.
141db969 1366 my %rules = ( '\\' => '\\\\',
c84d73f1
IZ
1367 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
1368 sub convert {
1369 my $re = shift;
1370 $re =~ s{
1371 \\ ( \\ | Y . )
1372 }
1373 { $rules{$1} or invalid($re,$1) }sgex;
1374 return $re;
1375 }
1376
1377Now C<use customre> enables the new escape in constant regular
1378expressions, i.e., those without any runtime variable interpolations.
1379As documented in L<overload>, this conversion will work only over
1380literal parts of regular expressions. For C<\Y|$re\Y|> the variable
1381part of this regular expression needs to be converted explicitly
1382(but only if the special meaning of C<\Y|> should be enabled inside $re):
1383
1384 use customre;
1385 $re = <>;
1386 chomp $re;
1387 $re = customre::convert $re;
1388 /\Y|$re\Y|/;
1389
19799a22
GS
1390=head1 BUGS
1391
9da458fc
IZ
1392This document varies from difficult to understand to completely
1393and utterly opaque. The wandering prose riddled with jargon is
1394hard to fathom in several places.
1395
1396This document needs a rewrite that separates the tutorial content
1397from the reference content.
19799a22
GS
1398
1399=head1 SEE ALSO
9fa51da4 1400
91e0c79e
MJD
1401L<perlrequick>.
1402
1403L<perlretut>.
1404
9b599b2a
GS
1405L<perlop/"Regexp Quote-Like Operators">.
1406
1e66bd83
PP
1407L<perlop/"Gory details of parsing quoted constructs">.
1408
14218588
GS
1409L<perlfaq6>.
1410
9b599b2a
GS
1411L<perlfunc/pos>.
1412
1413L<perllocale>.
1414
fb55449c
JH
1415L<perlebcdic>.
1416
14218588
GS
1417I<Mastering Regular Expressions> by Jeffrey Friedl, published
1418by O'Reilly and Associates.