This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
perlfaq update from Tom Christiansen
[perl5.git] / pod / perlre.pod
CommitLineData
a0d0e21e
LW
1=head1 NAME
2
3perlre - Perl regular expressions
4
5=head1 DESCRIPTION
6
cb1a09d0 7This page describes the syntax of regular expressions in Perl. For a
5f05dabc 8description of how to I<use> regular expressions in matching
75e14d17 9operations, plus various examples of the same, see discussion
1e66bd83 10of C<m//>, C<s///>, C<qr//> and C<??> in L<perlop/"Regexp Quote-Like Operators">.
cb1a09d0 11
68dc0745 12The matching operations can have various modifiers. The modifiers
5a964f20 13that relate to the interpretation of the regular expression inside
1e66bd83
PP
14are listed below. For the modifiers that alter the way a regular expression
15is used by Perl, see L<perlop/"Regexp Quote-Like Operators"> and
16L<perlop/"Gory details of parsing quoted constructs">.
a0d0e21e 17
55497cff
PP
18=over 4
19
20=item i
21
22Do case-insensitive pattern matching.
23
a034a98d
DD
24If C<use locale> is in effect, the case map is taken from the current
25locale. See L<perllocale>.
26
54310121 27=item m
55497cff
PP
28
29Treat string as multiple lines. That is, change "^" and "$" from matching
5f05dabc 30at only the very start or end of the string to the start or end of any
7f761169 31line anywhere within the string.
55497cff 32
54310121 33=item s
55497cff
PP
34
35Treat string as single line. That is, change "." to match any character
36whatsoever, even a newline, which it normally would not match.
37
5a964f20
TC
38The C</s> and C</m> modifiers both override the C<$*> setting. That is, no matter
39what C<$*> contains, C</s> without C</m> will force "^" to match only at the
7b8d334a
GS
40beginning of the string and "$" to match only at the end (or just before a
41newline at the end) of the string. Together, as /ms, they let the "." match
42any character whatsoever, while yet allowing "^" and "$" to match,
43respectively, just after and just before newlines within the string.
44
54310121 45=item x
55497cff
PP
46
47Extend your pattern's legibility by permitting whitespace and comments.
48
49=back
a0d0e21e
LW
50
51These are usually written as "the C</x> modifier", even though the delimiter
52in question might not actually be a slash. In fact, any of these
53modifiers may also be embedded within the regular expression itself using
54the new C<(?...)> construct. See below.
55
4633a7c4 56The C</x> modifier itself needs a little more explanation. It tells
55497cff
PP
57the regular expression parser to ignore whitespace that is neither
58backslashed nor within a character class. You can use this to break up
4633a7c4 59your regular expression into (slightly) more readable parts. The C<#>
54310121 60character is also treated as a metacharacter introducing a comment,
55497cff 61just as in ordinary Perl code. This also means that if you want real
5a964f20
TC
62whitespace or C<#> characters in the pattern (outside of a character
63class, where they are unaffected by C</x>), that you'll either have to
55497cff
PP
64escape them or encode them using octal or hex escapes. Taken together,
65these features go a long way towards making Perl's regular expressions
0c815be9
HS
66more readable. Note that you have to be careful not to include the
67pattern delimiter in the comment--perl has no way of knowing you did
5a964f20 68not intend to close the pattern early. See the C-comment deletion code
0c815be9 69in L<perlop>.
a0d0e21e
LW
70
71=head2 Regular Expressions
72
73The patterns used in pattern matching are regular expressions such as
5a964f20 74those supplied in the Version 8 regex routines. (In fact, the
a0d0e21e
LW
75routines are derived (distantly) from Henry Spencer's freely
76redistributable reimplementation of the V8 routines.)
77See L<Version 8 Regular Expressions> for details.
78
79In particular the following metacharacters have their standard I<egrep>-ish
80meanings:
81
54310121 82 \ Quote the next metacharacter
a0d0e21e
LW
83 ^ Match the beginning of the line
84 . Match any character (except newline)
c07a80fd 85 $ Match the end of the line (or before newline at the end)
a0d0e21e
LW
86 | Alternation
87 () Grouping
88 [] Character class
89
5f05dabc
PP
90By default, the "^" character is guaranteed to match at only the
91beginning of the string, the "$" character at only the end (or before the
a0d0e21e
LW
92newline at the end) and Perl does certain optimizations with the
93assumption that the string contains only one line. Embedded newlines
94will not be matched by "^" or "$". You may, however, wish to treat a
4a6725af 95string as a multi-line buffer, such that the "^" will match after any
a0d0e21e
LW
96newline within the string, and "$" will match before any newline. At the
97cost of a little more overhead, you can do this by using the /m modifier
98on the pattern match operator. (Older programs did this by setting C<$*>,
5f05dabc 99but this practice is now deprecated.)
a0d0e21e 100
4a6725af 101To facilitate multi-line substitutions, the "." character never matches a
55497cff 102newline unless you use the C</s> modifier, which in effect tells Perl to pretend
a0d0e21e
LW
103the string is a single line--even if it isn't. The C</s> modifier also
104overrides the setting of C<$*>, in case you have some (badly behaved) older
105code that sets it in another module.
106
107The following standard quantifiers are recognized:
108
109 * Match 0 or more times
110 + Match 1 or more times
111 ? Match 1 or 0 times
112 {n} Match exactly n times
113 {n,} Match at least n times
114 {n,m} Match at least n but not more than m times
115
116(If a curly bracket occurs in any other context, it is treated
117as a regular character.) The "*" modifier is equivalent to C<{0,}>, the "+"
25f94b33 118modifier to C<{1,}>, and the "?" modifier to C<{0,1}>. n and m are limited
9c79236d
GS
119to integral values less than a preset limit defined when perl is built.
120This is usually 32766 on the most common platforms. The actual limit can
121be seen in the error message generated by code such as this:
122
123 $_ **= $_ , / {$_} / for 2 .. 42;
a0d0e21e 124
54310121
PP
125By default, a quantified subpattern is "greedy", that is, it will match as
126many times as possible (given a particular starting location) while still
127allowing the rest of the pattern to match. If you want it to match the
128minimum number of times possible, follow the quantifier with a "?". Note
129that the meanings don't change, just the "greediness":
a0d0e21e
LW
130
131 *? Match 0 or more times
132 +? Match 1 or more times
133 ?? Match 0 or 1 time
134 {n}? Match exactly n times
135 {n,}? Match at least n times
136 {n,m}? Match at least n but not more than m times
137
5f05dabc 138Because patterns are processed as double quoted strings, the following
a0d0e21e
LW
139also work:
140
0f36ee90
PP
141 \t tab (HT, TAB)
142 \n newline (LF, NL)
143 \r return (CR)
144 \f form feed (FF)
145 \a alarm (bell) (BEL)
146 \e escape (think troff) (ESC)
cb1a09d0
AD
147 \033 octal char (think of a PDP-11)
148 \x1B hex char
a0ed51b3 149 \x{263a} wide hex char (Unicode SMILEY)
a0d0e21e 150 \c[ control char
cb1a09d0
AD
151 \l lowercase next char (think vi)
152 \u uppercase next char (think vi)
153 \L lowercase till \E (think vi)
154 \U uppercase till \E (think vi)
155 \E end case modification (think vi)
5a964f20 156 \Q quote (disable) pattern metacharacters till \E
a0d0e21e 157
a034a98d 158If C<use locale> is in effect, the case map used by C<\l>, C<\L>, C<\u>
7b8d334a 159and C<\U> is taken from the current locale. See L<perllocale>.
a034a98d 160
1d2dff63
GS
161You cannot include a literal C<$> or C<@> within a C<\Q> sequence.
162An unescaped C<$> or C<@> interpolates the corresponding variable,
163while escaping will cause the literal string C<\$> to be matched.
164You'll need to write something like C<m/\Quser\E\@\Qhost/>.
165
a0d0e21e
LW
166In addition, Perl defines the following:
167
168 \w Match a "word" character (alphanumeric plus "_")
169 \W Match a non-word character
170 \s Match a whitespace character
171 \S Match a non-whitespace character
172 \d Match a digit character
173 \D Match a non-digit character
a0ed51b3
LW
174 \pP Match P, named property. Use \p{Prop} for longer names.
175 \PP Match non-P
f244e06d
GS
176 \X Match eXtended Unicode "combining character sequence",
177 equivalent to C<(?:\PM\pM*)>
a0ed51b3 178 \C Match a single C char (octet) even under utf8.
a0d0e21e 179
5a964f20 180A C<\w> matches a single alphanumeric character, not a whole
a034a98d
DD
181word. To match a word you'd need to say C<\w+>. If C<use locale> is in
182effect, the list of alphabetic characters generated by C<\w> is taken
183from the current locale. See L<perllocale>. You may use C<\w>, C<\W>,
184C<\s>, C<\S>, C<\d>, and C<\D> within character classes (though not as
185either end of a range).
a0d0e21e
LW
186
187Perl defines the following zero-width assertions:
188
189 \b Match a word boundary
190 \B Match a non-(word boundary)
b85d18e9
IZ
191 \A Match only at beginning of string
192 \Z Match only at end of string, or before newline at the end
193 \z Match only at end of string
a99df21c 194 \G Match only where previous m//g left off (works only with /g)
a0d0e21e
LW
195
196A word boundary (C<\b>) is defined as a spot between two characters that
68dc0745 197has a C<\w> on one side of it and a C<\W> on the other side of it (in
a0d0e21e
LW
198either order), counting the imaginary characters off the beginning and
199end of the string as matching a C<\W>. (Within character classes C<\b>
200represents backspace rather than a word boundary.) The C<\A> and C<\Z> are
5a964f20 201just like "^" and "$", except that they won't match multiple times when the
a0d0e21e 202C</m> modifier is used, while "^" and "$" will match at every internal line
c07a80fd 203boundary. To match the actual end of the string, not ignoring newline,
b85d18e9 204you can use C<\z>. The C<\G> assertion can be used to chain global
a99df21c 205matches (using C<m//g>), as described in
e7ea3e70 206L<perlop/"Regexp Quote-Like Operators">.
a99df21c 207
e7ea3e70 208It is also useful when writing C<lex>-like scanners, when you have several
5a964f20 209patterns that you want to match against consequent substrings of your
e7ea3e70 210string, see the previous reference.
44a8e56a
PP
211The actual location where C<\G> will match can also be influenced
212by using C<pos()> as an lvalue. See L<perlfunc/pos>.
a0d0e21e 213
0f36ee90 214When the bracketing construct C<( ... )> is used, \E<lt>digitE<gt> matches the
cb1a09d0 215digit'th substring. Outside of the pattern, always use "$" instead of "\"
0f36ee90 216in front of the digit. (While the \E<lt>digitE<gt> notation can on rare occasion work
cb1a09d0 217outside the current pattern, this should not be relied upon. See the
0f36ee90 218WARNING below.) The scope of $E<lt>digitE<gt> (and C<$`>, C<$&>, and C<$'>)
cb1a09d0
AD
219extends to the end of the enclosing BLOCK or eval string, or to the next
220successful pattern match, whichever comes first. If you want to use
5f05dabc 221parentheses to delimit a subpattern (e.g., a set of alternatives) without
84dc3c4d 222saving it as a subpattern, follow the ( with a ?:.
cb1a09d0
AD
223
224You may have as many parentheses as you wish. If you have more
a0d0e21e
LW
225than 9 substrings, the variables $10, $11, ... refer to the
226corresponding substring. Within the pattern, \10, \11, etc. refer back
5f05dabc 227to substrings if there have been at least that many left parentheses before
c07a80fd 228the backreference. Otherwise (for backward compatibility) \10 is the
a0d0e21e
LW
229same as \010, a backspace, and \11 the same as \011, a tab. And so
230on. (\1 through \9 are always backreferences.)
231
232C<$+> returns whatever the last bracket match matched. C<$&> returns the
0f36ee90 233entire matched string. (C<$0> used to return the same thing, but not any
a0d0e21e
LW
234more.) C<$`> returns everything before the matched string. C<$'> returns
235everything after the matched string. Examples:
236
237 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
238
239 if (/Time: (..):(..):(..)/) {
240 $hours = $1;
241 $minutes = $2;
242 $seconds = $3;
243 }
244
68dc0745
PP
245Once perl sees that you need one of C<$&>, C<$`> or C<$'> anywhere in
246the program, it has to provide them on each and every pattern match.
247This can slow your program down. The same mechanism that handles
248these provides for the use of $1, $2, etc., so you pay the same price
5a964f20
TC
249for each pattern that contains capturing parentheses. But if you never
250use $&, etc., in your script, then patterns I<without> capturing
68dc0745
PP
251parentheses won't be penalized. So avoid $&, $', and $` if you can,
252but if you can't (and some algorithms really appreciate them), once
253you've used them once, use them at will, because you've already paid
5a964f20 254the price. As of 5.005, $& is not so costly as the other two.
68dc0745 255
5a964f20 256Backslashed metacharacters in Perl are
201ecf35
AL
257alphanumeric, such as C<\b>, C<\w>, C<\n>. Unlike some other regular
258expression languages, there are no backslashed symbols that aren't
259alphanumeric. So anything that looks like \\, \(, \), \E<lt>, \E<gt>,
260\{, or \} is always interpreted as a literal character, not a
261metacharacter. This was once used in a common idiom to disable or
262quote the special meanings of regular expression metacharacters in a
5a964f20 263string that you want to use for a pattern. Simply quote all
a0d0e21e
LW
264non-alphanumeric characters:
265
266 $pattern =~ s/(\W)/\\$1/g;
267
201ecf35 268Now it is much more common to see either the quotemeta() function or
7b8d334a 269the C<\Q> escape sequence used to disable all metacharacters' special
201ecf35 270meanings like this:
a0d0e21e
LW
271
272 /$unquoted\Q$quoted\E$unquoted/
273
5f05dabc
PP
274Perl defines a consistent extension syntax for regular expressions.
275The syntax is a pair of parentheses with a question mark as the first
276thing within the parentheses (this was a syntax error in older
277versions of Perl). The character after the question mark gives the
278function of the extension. Several extensions are already supported:
a0d0e21e
LW
279
280=over 10
281
cc6b7395 282=item C<(?#text)>
a0d0e21e 283
cb1a09d0 284A comment. The text is ignored. If the C</x> switch is used to enable
259138e3
GS
285whitespace formatting, a simple C<#> will suffice. Note that perl closes
286the comment as soon as it sees a C<)>, so there is no way to put a literal
287C<)> in the comment.
a0d0e21e 288
5a964f20 289=item C<(?:pattern)>
a0d0e21e 290
ca9dfc88
IZ
291=item C<(?imsx-imsx:pattern)>
292
5a964f20
TC
293This is for clustering, not capturing; it groups subexpressions like
294"()", but doesn't make backreferences as "()" does. So
a0d0e21e 295
5a964f20 296 @fields = split(/\b(?:a|b|c)\b/)
a0d0e21e
LW
297
298is like
299
5a964f20 300 @fields = split(/\b(a|b|c)\b/)
a0d0e21e
LW
301
302but doesn't spit out extra fields.
303
ca9dfc88
IZ
304The letters between C<?> and C<:> act as flags modifiers, see
305L<C<(?imsx-imsx)>>. In particular,
306
307 /(?s-i:more.*than).*million/i
308
309is equivalent to more verbose
310
311 /(?:(?s-i)more.*than).*million/i
312
5a964f20 313=item C<(?=pattern)>
a0d0e21e
LW
314
315A zero-width positive lookahead assertion. For example, C</\w+(?=\t)/>
316matches a word followed by a tab, without including the tab in C<$&>.
317
5a964f20 318=item C<(?!pattern)>
a0d0e21e
LW
319
320A zero-width negative lookahead assertion. For example C</foo(?!bar)/>
321matches any occurrence of "foo" that isn't followed by "bar". Note
322however that lookahead and lookbehind are NOT the same thing. You cannot
7b8d334a
GS
323use this for lookbehind.
324
5a964f20 325If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
7b8d334a
GS
326will not do what you want. That's because the C<(?!foo)> is just saying that
327the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
328match. You would have to do something like C</(?!foo)...bar/> for that. We
329say "like" because there's the case of your "bar" not having three characters
330before it. You could cover that this way: C</(?:(?!foo)...|^.{0,2})bar/>.
331Sometimes it's still easier just to say:
a0d0e21e 332
a3cb178b 333 if (/bar/ && $` !~ /foo$/)
a0d0e21e 334
c277df42
IZ
335For lookbehind see below.
336
5a964f20 337=item C<(?E<lt>=pattern)>
c277df42 338
5a964f20 339A zero-width positive lookbehind assertion. For example, C</(?E<lt>=\t)\w+/>
c277df42
IZ
340matches a word following a tab, without including the tab in C<$&>.
341Works only for fixed-width lookbehind.
342
5a964f20 343=item C<(?<!pattern)>
c277df42
IZ
344
345A zero-width negative lookbehind assertion. For example C</(?<!bar)foo/>
346matches any occurrence of "foo" that isn't following "bar".
347Works only for fixed-width lookbehind.
348
cc6b7395 349=item C<(?{ code })>
c277df42
IZ
350
351Experimental "evaluate any Perl code" zero-width assertion. Always
cc6b7395
IZ
352succeeds. C<code> is not interpolated. Currently the rules to
353determine where the C<code> ends are somewhat convoluted.
c277df42 354
b9ac3b5b
GS
355The C<code> is properly scoped in the following sense: if the assertion
356is backtracked (compare L<"Backtracking">), all the changes introduced after
357C<local>isation are undone, so
358
359 $_ = 'a' x 8;
360 m<
361 (?{ $cnt = 0 }) # Initialize $cnt.
362 (
363 a
364 (?{
365 local $cnt = $cnt + 1; # Update $cnt, backtracking-safe.
366 })
367 )*
368 aaaa
369 (?{ $res = $cnt }) # On success copy to non-localized
370 # location.
371 >x;
372
373will set C<$res = 4>. Note that after the match $cnt returns to the globally
374introduced value 0, since the scopes which restrict C<local> statements
375are unwound.
376
377This assertion may be used as L<C<(?(condition)yes-pattern|no-pattern)>>
378switch. If I<not> used in this way, the result of evaluation of C<code>
379is put into variable $^R. This happens immediately, so $^R can be used from
380other C<(?{ code })> assertions inside the same regular expression.
381
382The above assignment to $^R is properly localized, thus the old value of $^R
383is restored if the assertion is backtracked (compare L<"Backtracking">).
384
871b0233
IZ
385Due to security concerns, this construction is not allowed if the regular
386expression involves run-time interpolation of variables, unless
387C<use re 'eval'> pragma is used (see L<re>), or the variables contain
388results of qr() operator (see L<perlop/"qr/STRING/imosx">).
389
390This restriction is due to the wide-spread (questionable) practice of
391using the construct
392
393 $re = <>;
394 chomp $re;
395 $string =~ /$re/;
396
397without tainting. While this code is frowned upon from security point
398of view, when C<(?{})> was introduced, it was considered bad to add
399I<new> security holes to existing scripts.
400
401B<NOTE:> Use of the above insecure snippet without also enabling taint mode
402is to be severely frowned upon. C<use re 'eval'> does not disable tainting
403checks, thus to allow $re in the above snippet to contain C<(?{})>
404I<with tainting enabled>, one needs both C<use re 'eval'> and untaint
405the $re.
406
0f5d15d6
IZ
407=item C<(?p{ code })>
408
409I<Very experimental> "postponed" regular subexpression. C<code> is evaluated
410at runtime, at the moment this subexpression may match. The result of
411evaluation is considered as a regular expression, and matched as if it
412were inserted instead of this construct.
413
414C<code> is not interpolated. Currently the rules to
415determine where the C<code> ends are somewhat convoluted.
416
417The following regular expression matches matching parenthesized group:
418
419 $re = qr{
420 \(
421 (?:
422 (?> [^()]+ ) # Non-parens without backtracking
423 |
424 (?p{ $re }) # Group with matching parens
425 )*
426 \)
427 }x;
428
5a964f20
TC
429=item C<(?E<gt>pattern)>
430
431An "independent" subexpression. Matches the substring that a
432I<standalone> C<pattern> would match if anchored at the given position,
c277df42
IZ
433B<and only this substring>.
434
435Say, C<^(?E<gt>a*)ab> will never match, since C<(?E<gt>a*)> (anchored
5a964f20 436at the beginning of string, as above) will match I<all> characters
c277df42
IZ
437C<a> at the beginning of string, leaving no C<a> for C<ab> to match.
438In contrast, C<a*ab> will match the same as C<a+b>, since the match of
439the subgroup C<a*> is influenced by the following group C<ab> (see
440L<"Backtracking">). In particular, C<a*> inside C<a*ab> will match
aca73f04 441fewer characters than a standalone C<a*>, since this makes the tail match.
c277df42 442
5a964f20 443An effect similar to C<(?E<gt>pattern)> may be achieved by
c277df42 444
5a964f20 445 (?=(pattern))\1
c277df42
IZ
446
447since the lookahead is in I<"logical"> context, thus matches the same
448substring as a standalone C<a+>. The following C<\1> eats the matched
449string, thus making a zero-length assertion into an analogue of
871b0233 450C<(?E<gt>...)>. (The difference between these two constructs is that the
5a964f20 451second one uses a catching group, thus shifting ordinals of
c277df42
IZ
452backreferences in the rest of a regular expression.)
453
5a964f20
TC
454This construct is useful for optimizations of "eternal"
455matches, because it will not backtrack (see L<"Backtracking">).
c277df42 456
871b0233
IZ
457 m{ \(
458 (
459 [^()]+
460 |
461 \( [^()]* \)
462 )+
463 \)
464 }x
5a964f20
TC
465
466That will efficiently match a nonempty group with matching
467two-or-less-level-deep parentheses. However, if there is no such group,
468it will take virtually forever on a long string. That's because there are
469so many different ways to split a long string into several substrings.
871b0233
IZ
470This is what C<(.+)+> is doing, and C<(.+)+> is similar to a subpattern
471of the above pattern. Consider that the above pattern detects no-match
472on C<((()aaaaaaaaaaaaaaaaaa> in several seconds, but that each extra
5a964f20
TC
473letter doubles this time. This exponential performance will make it
474appear that your program has hung.
475
476However, a tiny modification of this pattern
477
871b0233
IZ
478 m{ \(
479 (
480 (?> [^()]+ )
481 |
482 \( [^()]* \)
483 )+
484 \)
485 }x
c277df42 486
5a964f20
TC
487which uses C<(?E<gt>...)> matches exactly when the one above does (verifying
488this yourself would be a productive exercise), but finishes in a fourth
489the time when used on a similar string with 1000000 C<a>s. Be aware,
490however, that this pattern currently triggers a warning message under
491B<-w> saying it C<"matches the null string many times">):
c277df42 492
8d300b32 493On simple groups, such as the pattern C<(?E<gt> [^()]+ )>, a comparable
c277df42
IZ
494effect may be achieved by negative lookahead, as in C<[^()]+ (?! [^()] )>.
495This was only 4 times slower on a string with 1000000 C<a>s.
496
5a964f20 497=item C<(?(condition)yes-pattern|no-pattern)>
c277df42 498
5a964f20 499=item C<(?(condition)yes-pattern)>
c277df42
IZ
500
501Conditional expression. C<(condition)> should be either an integer in
502parentheses (which is valid if the corresponding pair of parentheses
503matched), or lookahead/lookbehind/evaluate zero-width assertion.
504
505Say,
506
5a964f20 507 m{ ( \( )?
871b0233 508 [^()]+
5a964f20 509 (?(1) \) )
871b0233 510 }x
c277df42
IZ
511
512matches a chunk of non-parentheses, possibly included in parentheses
513themselves.
a0d0e21e 514
ca9dfc88 515=item C<(?imsx-imsx)>
a0d0e21e
LW
516
517One or more embedded pattern-match modifiers. This is particularly
518useful for patterns that are specified in a table somewhere, some of
519which want to be case sensitive, and some of which don't. The case
5f05dabc 520insensitive ones need to include merely C<(?i)> at the front of the
a0d0e21e
LW
521pattern. For example:
522
523 $pattern = "foobar";
5a964f20 524 if ( /$pattern/i ) { }
a0d0e21e
LW
525
526 # more flexible:
527
528 $pattern = "(?i)foobar";
5a964f20 529 if ( /$pattern/ ) { }
a0d0e21e 530
ca9dfc88
IZ
531Letters after C<-> switch modifiers off.
532
5a964f20 533These modifiers are localized inside an enclosing group (if any). Say,
c277df42
IZ
534
535 ( (?i) blah ) \s+ \1
536
537(assuming C<x> modifier, and no C<i> modifier outside of this group)
538will match a repeated (I<including the case>!) word C<blah> in any
539case.
540
a0d0e21e
LW
541=back
542
5a964f20
TC
543A question mark was chosen for this and for the new minimal-matching
544construct because 1) question mark is pretty rare in older regular
545expressions, and 2) whenever you see one, you should stop and "question"
546exactly what is going on. That's psychology...
a0d0e21e 547
c07a80fd
PP
548=head2 Backtracking
549
c277df42 550A fundamental feature of regular expression matching involves the
5a964f20 551notion called I<backtracking>, which is currently used (when needed)
c277df42
IZ
552by all regular expression quantifiers, namely C<*>, C<*?>, C<+>,
553C<+?>, C<{n,m}>, and C<{n,m}?>.
c07a80fd
PP
554
555For a regular expression to match, the I<entire> regular expression must
556match, not just part of it. So if the beginning of a pattern containing a
557quantifier succeeds in a way that causes later parts in the pattern to
558fail, the matching engine backs up and recalculates the beginning
559part--that's why it's called backtracking.
560
561Here is an example of backtracking: Let's say you want to find the
562word following "foo" in the string "Food is on the foo table.":
563
564 $_ = "Food is on the foo table.";
565 if ( /\b(foo)\s+(\w+)/i ) {
566 print "$2 follows $1.\n";
567 }
568
569When the match runs, the first part of the regular expression (C<\b(foo)>)
570finds a possible match right at the beginning of the string, and loads up
571$1 with "Foo". However, as soon as the matching engine sees that there's
572no whitespace following the "Foo" that it had saved in $1, it realizes its
68dc0745 573mistake and starts over again one character after where it had the
c07a80fd
PP
574tentative match. This time it goes all the way until the next occurrence
575of "foo". The complete regular expression matches this time, and you get
576the expected output of "table follows foo."
577
578Sometimes minimal matching can help a lot. Imagine you'd like to match
579everything between "foo" and "bar". Initially, you write something
580like this:
581
582 $_ = "The food is under the bar in the barn.";
583 if ( /foo(.*)bar/ ) {
584 print "got <$1>\n";
585 }
586
587Which perhaps unexpectedly yields:
588
589 got <d is under the bar in the >
590
591That's because C<.*> was greedy, so you get everything between the
592I<first> "foo" and the I<last> "bar". In this case, it's more effective
593to use minimal matching to make sure you get the text between a "foo"
594and the first "bar" thereafter.
595
596 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
597 got <d is under the >
598
599Here's another example: let's say you'd like to match a number at the end
600of a string, and you also want to keep the preceding part the match.
601So you write this:
602
603 $_ = "I have 2 numbers: 53147";
604 if ( /(.*)(\d*)/ ) { # Wrong!
605 print "Beginning is <$1>, number is <$2>.\n";
606 }
607
608That won't work at all, because C<.*> was greedy and gobbled up the
609whole string. As C<\d*> can match on an empty string the complete
610regular expression matched successfully.
611
8e1088bc 612 Beginning is <I have 2 numbers: 53147>, number is <>.
c07a80fd
PP
613
614Here are some variants, most of which don't work:
615
616 $_ = "I have 2 numbers: 53147";
617 @pats = qw{
618 (.*)(\d*)
619 (.*)(\d+)
620 (.*?)(\d*)
621 (.*?)(\d+)
622 (.*)(\d+)$
623 (.*?)(\d+)$
624 (.*)\b(\d+)$
625 (.*\D)(\d+)$
626 };
627
628 for $pat (@pats) {
629 printf "%-12s ", $pat;
630 if ( /$pat/ ) {
631 print "<$1> <$2>\n";
632 } else {
633 print "FAIL\n";
634 }
635 }
636
637That will print out:
638
639 (.*)(\d*) <I have 2 numbers: 53147> <>
640 (.*)(\d+) <I have 2 numbers: 5314> <7>
641 (.*?)(\d*) <> <>
642 (.*?)(\d+) <I have > <2>
643 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
644 (.*?)(\d+)$ <I have 2 numbers: > <53147>
645 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
646 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
647
648As you see, this can be a bit tricky. It's important to realize that a
649regular expression is merely a set of assertions that gives a definition
650of success. There may be 0, 1, or several different ways that the
651definition might succeed against a particular string. And if there are
5a964f20
TC
652multiple ways it might succeed, you need to understand backtracking to
653know which variety of success you will achieve.
c07a80fd
PP
654
655When using lookahead assertions and negations, this can all get even
54310121 656tricker. Imagine you'd like to find a sequence of non-digits not
c07a80fd
PP
657followed by "123". You might try to write that as
658
871b0233
IZ
659 $_ = "ABC123";
660 if ( /^\D*(?!123)/ ) { # Wrong!
661 print "Yup, no 123 in $_\n";
662 }
c07a80fd
PP
663
664But that isn't going to match; at least, not the way you're hoping. It
665claims that there is no 123 in the string. Here's a clearer picture of
666why it that pattern matches, contrary to popular expectations:
667
668 $x = 'ABC123' ;
669 $y = 'ABC445' ;
670
671 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/ ;
672 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/ ;
673
674 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/ ;
675 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/ ;
676
677This prints
678
679 2: got ABC
680 3: got AB
681 4: got ABC
682
5f05dabc 683You might have expected test 3 to fail because it seems to a more
c07a80fd
PP
684general purpose version of test 1. The important difference between
685them is that test 3 contains a quantifier (C<\D*>) and so can use
686backtracking, whereas test 1 will not. What's happening is
687that you've asked "Is it true that at the start of $x, following 0 or more
5f05dabc 688non-digits, you have something that's not 123?" If the pattern matcher had
c07a80fd 689let C<\D*> expand to "ABC", this would have caused the whole pattern to
54310121 690fail.
c07a80fd 691The search engine will initially match C<\D*> with "ABC". Then it will
5a964f20 692try to match C<(?!123> with "123", which of course fails. But because
c07a80fd
PP
693a quantifier (C<\D*>) has been used in the regular expression, the
694search engine can backtrack and retry the match differently
54310121 695in the hope of matching the complete regular expression.
c07a80fd 696
5a964f20
TC
697The pattern really, I<really> wants to succeed, so it uses the
698standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
c07a80fd
PP
699time. Now there's indeed something following "AB" that is not
700"123". It's in fact "C123", which suffices.
701
702We can deal with this by using both an assertion and a negation. We'll
703say that the first part in $1 must be followed by a digit, and in fact, it
704must also be followed by something that's not "123". Remember that the
705lookaheads are zero-width expressions--they only look, but don't consume
706any of the string in their match. So rewriting this way produces what
707you'd expect; that is, case 5 will fail, but case 6 succeeds:
708
709 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/ ;
710 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/ ;
711
712 6: got ABC
713
5a964f20 714In other words, the two zero-width assertions next to each other work as though
c07a80fd
PP
715they're ANDed together, just as you'd use any builtin assertions: C</^$/>
716matches only if you're at the beginning of the line AND the end of the
717line simultaneously. The deeper underlying truth is that juxtaposition in
718regular expressions always means AND, except when you write an explicit OR
719using the vertical bar. C</ab/> means match "a" AND (then) match "b",
720although the attempted matches are made at different positions because "a"
721is not a zero-width assertion, but a one-width assertion.
722
723One warning: particularly complicated regular expressions can take
724exponential time to solve due to the immense number of possible ways they
725can use backtracking to try match. For example this will take a very long
726time to run
727
728 /((a{0,5}){0,5}){0,5}/
729
730And if you used C<*>'s instead of limiting it to 0 through 5 matches, then
731it would take literally forever--or until you ran out of stack space.
732
c277df42 733A powerful tool for optimizing such beasts is "independent" groups,
5a964f20 734which do not backtrace (see L<C<(?E<gt>pattern)>>). Note also that
c277df42
IZ
735zero-length lookahead/lookbehind assertions will not backtrace to make
736the tail match, since they are in "logical" context: only the fact
737whether they match or not is considered relevant. For an example
738where side-effects of a lookahead I<might> have influenced the
5a964f20 739following match, see L<C<(?E<gt>pattern)>>.
c277df42 740
a0d0e21e
LW
741=head2 Version 8 Regular Expressions
742
5a964f20 743In case you're not familiar with the "regular" Version 8 regex
a0d0e21e
LW
744routines, here are the pattern-matching rules not described above.
745
54310121 746Any single character matches itself, unless it is a I<metacharacter>
a0d0e21e 747with a special meaning described here or above. You can cause
5a964f20 748characters that normally function as metacharacters to be interpreted
5f05dabc 749literally by prefixing them with a "\" (e.g., "\." matches a ".", not any
a0d0e21e
LW
750character; "\\" matches a "\"). A series of characters matches that
751series of characters in the target string, so the pattern C<blurfl>
752would match "blurfl" in the target string.
753
754You can specify a character class, by enclosing a list of characters
5a964f20 755in C<[]>, which will match any one character from the list. If the
a0d0e21e
LW
756first character after the "[" is "^", the class matches any character not
757in the list. Within a list, the "-" character is used to specify a
5a964f20 758range, so that C<a-z> represents all characters between "a" and "z",
84850974
DD
759inclusive. If you want "-" itself to be a member of a class, put it
760at the start or end of the list, or escape it with a backslash. (The
761following all specify the same class of three characters: C<[-az]>,
762C<[az-]>, and C<[a\-z]>. All are different from C<[a-z]>, which
763specifies a class containing twenty-six characters.)
a0d0e21e 764
8ada0baa
JH
765Note also that the whole range idea is rather unportable between
766character sets--and even within character sets they may cause results
767you probably didn't expect. A sound principle is to use only ranges
768that begin from and end at either alphabets of equal case ([a-e],
769[A-E]), or digits ([0-9]). Anything else is unsafe. If in doubt,
770spell out the character sets in full.
771
54310121 772Characters may be specified using a metacharacter syntax much like that
a0d0e21e
LW
773used in C: "\n" matches a newline, "\t" a tab, "\r" a carriage return,
774"\f" a form feed, etc. More generally, \I<nnn>, where I<nnn> is a string
775of octal digits, matches the character whose ASCII value is I<nnn>.
0f36ee90 776Similarly, \xI<nn>, where I<nn> are hexadecimal digits, matches the
a0d0e21e 777character whose ASCII value is I<nn>. The expression \cI<x> matches the
54310121 778ASCII character control-I<x>. Finally, the "." metacharacter matches any
a0d0e21e
LW
779character except "\n" (unless you use C</s>).
780
781You can specify a series of alternatives for a pattern using "|" to
782separate them, so that C<fee|fie|foe> will match any of "fee", "fie",
5a964f20 783or "foe" in the target string (as would C<f(e|i|o)e>). The
a0d0e21e
LW
784first alternative includes everything from the last pattern delimiter
785("(", "[", or the beginning of the pattern) up to the first "|", and
786the last alternative contains everything from the last "|" to the next
787pattern delimiter. For this reason, it's common practice to include
788alternatives in parentheses, to minimize confusion about where they
a3cb178b
GS
789start and end.
790
5a964f20 791Alternatives are tried from left to right, so the first
a3cb178b
GS
792alternative found for which the entire expression matches, is the one that
793is chosen. This means that alternatives are not necessarily greedy. For
628afcb5 794example: when matching C<foo|foot> against "barefoot", only the "foo"
a3cb178b
GS
795part will match, as that is the first alternative tried, and it successfully
796matches the target string. (This might not seem important, but it is
797important when you are capturing matched text using parentheses.)
798
5a964f20 799Also remember that "|" is interpreted as a literal within square brackets,
a3cb178b 800so if you write C<[fee|fie|foe]> you're really only matching C<[feio|]>.
a0d0e21e 801
54310121 802Within a pattern, you may designate subpatterns for later reference by
a0d0e21e 803enclosing them in parentheses, and you may refer back to the I<n>th
54310121
PP
804subpattern later in the pattern using the metacharacter \I<n>.
805Subpatterns are numbered based on the left to right order of their
5a964f20 806opening parenthesis. A backreference matches whatever
54310121
PP
807actually matched the subpattern in the string being examined, not the
808rules for that subpattern. Therefore, C<(0|0x)\d*\s\1\d*> will
5a964f20 809match "0x1234 0x4321", but not "0x1234 01234", because subpattern 1
748a9306 810actually matched "0x", even though the rule C<0|0x> could
a0d0e21e 811potentially match the leading 0 in the second number.
cb1a09d0
AD
812
813=head2 WARNING on \1 vs $1
814
5a964f20 815Some people get too used to writing things like:
cb1a09d0
AD
816
817 $pattern =~ s/(\W)/\\\1/g;
818
819This is grandfathered for the RHS of a substitute to avoid shocking the
820B<sed> addicts, but it's a dirty habit to get into. That's because in
5f05dabc 821PerlThink, the righthand side of a C<s///> is a double-quoted string. C<\1> in
cb1a09d0
AD
822the usual double-quoted string means a control-A. The customary Unix
823meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
824of doing that, you get yourself into trouble if you then add an C</e>
825modifier.
826
5a964f20 827 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
cb1a09d0
AD
828
829Or if you try to do
830
831 s/(\d+)/\1000/;
832
833You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
834C<${1}000>. Basically, the operation of interpolation should not be confused
835with the operation of matching a backreference. Certainly they mean two
836different things on the I<left> side of the C<s///>.
9fa51da4 837
c84d73f1
IZ
838=head2 Repeated patterns matching zero-length substring
839
840WARNING: Difficult material (and prose) ahead. This section needs a rewrite.
841
842Regular expressions provide a terse and powerful programming language. As
843with most other power tools, power comes together with the ability
844to wreak havoc.
845
846A common abuse of this power stems from the ability to make infinite
628afcb5 847loops using regular expressions, with something as innocuous as:
c84d73f1
IZ
848
849 'foo' =~ m{ ( o? )* }x;
850
851The C<o?> can match at the beginning of C<'foo'>, and since the position
852in the string is not moved by the match, C<o?> would match again and again
853due to the C<*> modifier. Another common way to create a similar cycle
854is with the looping modifier C<//g>:
855
856 @matches = ( 'foo' =~ m{ o? }xg );
857
858or
859
860 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
861
862or the loop implied by split().
863
864However, long experience has shown that many programming tasks may
865be significantly simplified by using repeated subexpressions which
866may match zero-length substrings, with a simple example being:
867
868 @chars = split //, $string; # // is not magic in split
869 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
870
871Thus Perl allows the C</()/> construct, which I<forcefully breaks
872the infinite loop>. The rules for this are different for lower-level
873loops given by the greedy modifiers C<*+{}>, and for higher-level
874ones like the C</g> modifier or split() operator.
875
876The lower-level loops are I<interrupted> when it is detected that a
877repeated expression did match a zero-length substring, thus
878
879 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
880
881is made equivalent to
882
883 m{ (?: NON_ZERO_LENGTH )*
884 |
885 (?: ZERO_LENGTH )?
886 }x;
887
888The higher level-loops preserve an additional state between iterations:
889whether the last match was zero-length. To break the loop, the following
890match after a zero-length match is prohibited to have a length of zero.
891This prohibition interacts with backtracking (see L<"Backtracking">),
892and so the I<second best> match is chosen if the I<best> match is of
893zero length.
894
895Say,
896
897 $_ = 'bar';
898 s/\w??/<$&>/g;
899
900results in C<"<><b><><a><><r><>">. At each position of the string the best
901match given by non-greedy C<??> is the zero-length match, and the I<second
902best> match is what is matched by C<\w>. Thus zero-length matches
903alternate with one-character-long matches.
904
905Similarly, for repeated C<m/()/g> the second-best match is the match at the
906position one notch further in the string.
907
908The additional state of being I<matched with zero-length> is associated to
909the matched string, and is reset by each assignment to pos().
910
911=head2 Creating custom RE engines
912
913Overloaded constants (see L<overload>) provide a simple way to extend
914the functionality of the RE engine.
915
916Suppose that we want to enable a new RE escape-sequence C<\Y|> which
917matches at boundary between white-space characters and non-whitespace
918characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
919at these positions, so we want to have each C<\Y|> in the place of the
920more complicated version. We can create a module C<customre> to do
921this:
922
923 package customre;
924 use overload;
925
926 sub import {
927 shift;
928 die "No argument to customre::import allowed" if @_;
929 overload::constant 'qr' => \&convert;
930 }
931
932 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
933
934 my %rules = ( '\\' => '\\',
935 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
936 sub convert {
937 my $re = shift;
938 $re =~ s{
939 \\ ( \\ | Y . )
940 }
941 { $rules{$1} or invalid($re,$1) }sgex;
942 return $re;
943 }
944
945Now C<use customre> enables the new escape in constant regular
946expressions, i.e., those without any runtime variable interpolations.
947As documented in L<overload>, this conversion will work only over
948literal parts of regular expressions. For C<\Y|$re\Y|> the variable
949part of this regular expression needs to be converted explicitly
950(but only if the special meaning of C<\Y|> should be enabled inside $re):
951
952 use customre;
953 $re = <>;
954 chomp $re;
955 $re = customre::convert $re;
956 /\Y|$re\Y|/;
957
9fa51da4
CS
958=head2 SEE ALSO
959
9b599b2a
GS
960L<perlop/"Regexp Quote-Like Operators">.
961
1e66bd83
PP
962L<perlop/"Gory details of parsing quoted constructs">.
963
9b599b2a
GS
964L<perlfunc/pos>.
965
966L<perllocale>.
967
5a964f20 968I<Mastering Regular Expressions> (see L<perlbook>) by Jeffrey Friedl.