This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
new perldelta
[perl5.git] / pod / perlre.pod
... / ...
CommitLineData
1=head1 NAME
2X<regular expression> X<regex> X<regexp>
3
4perlre - Perl regular expressions
5
6=head1 DESCRIPTION
7
8This page describes the syntax of regular expressions in Perl.
9
10If you haven't used regular expressions before, a tutorial introduction
11is available in L<perlretut>. If you know just a little about them,
12a quick-start introduction is available in L<perlrequick>.
13
14Except for L</The Basics> section, this page assumes you are familiar
15with regular expression basics, like what is a "pattern", what does it
16look like, and how it is basically used. For a reference on how they
17are used, plus various examples of the same, see discussions of C<m//>,
18C<s///>, C<qr//> and C<"??"> in L<perlop/"Regexp Quote-Like Operators">.
19
20New in v5.22, L<C<use re 'strict'>|re/'strict' mode> applies stricter
21rules than otherwise when compiling regular expression patterns. It can
22find things that, while legal, may not be what you intended.
23
24=head2 The Basics
25X<regular expression, version 8> X<regex, version 8> X<regexp, version 8>
26
27Regular expressions are strings with the very particular syntax and
28meaning described in this document and auxiliary documents referred to
29by this one. The strings are called "patterns". Patterns are used to
30determine if some other string, called the "target", has (or doesn't
31have) the characteristics specified by the pattern. We call this
32"matching" the target string against the pattern. Usually the match is
33done by having the target be the first operand, and the pattern be the
34second operand, of one of the two binary operators C<=~> and C<!~>,
35listed in L<perlop/Binding Operators>; and the pattern will have been
36converted from an ordinary string by one of the operators in
37L<perlop/"Regexp Quote-Like Operators">, like so:
38
39 $foo =~ m/abc/
40
41This evaluates to true if and only if the string in the variable C<$foo>
42contains somewhere in it, the sequence of characters "a", "b", then "c".
43(The C<=~ m>, or match operator, is described in
44L<perlop/m/PATTERN/msixpodualngc>.)
45
46Patterns that aren't already stored in some variable must be delimited,
47at both ends, by delimiter characters. These are often, as in the
48example above, forward slashes, and the typical way a pattern is written
49in documentation is with those slashes. In most cases, the delimiter
50is the same character, fore and aft, but there are a few cases where a
51character looks like it has a mirror-image mate, where the opening
52version is the beginning delimiter, and the closing one is the ending
53delimiter, like
54
55 $foo =~ m<abc>
56
57Most times, the pattern is evaluated in double-quotish context, but it
58is possible to choose delimiters to force single-quotish, like
59
60 $foo =~ m'abc'
61
62If the pattern contains its delimiter within it, that delimiter must be
63escaped. Prefixing it with a backslash (I<e.g.>, C<"/foo\/bar/">)
64serves this purpose.
65
66Any single character in a pattern matches that same character in the
67target string, unless the character is a I<metacharacter> with a special
68meaning described in this document. A sequence of non-metacharacters
69matches the same sequence in the target string, as we saw above with
70C<m/abc/>.
71
72Only a few characters (all of them being ASCII punctuation characters)
73are metacharacters. The most commonly used one is a dot C<".">, which
74normally matches almost any character (including a dot itself).
75
76You can cause characters that normally function as metacharacters to be
77interpreted literally by prefixing them with a C<"\">, just like the
78pattern's delimiter must be escaped if it also occurs within the
79pattern. Thus, C<"\."> matches just a literal dot, C<"."> instead of
80its normal meaning. This means that the backslash is also a
81metacharacter, so C<"\\"> matches a single C<"\">. And a sequence that
82contains an escaped metacharacter matches the same sequence (but without
83the escape) in the target string. So, the pattern C</blur\\fl/> would
84match any target string that contains the sequence C<"blur\fl">.
85
86The metacharacter C<"|"> is used to match one thing or another. Thus
87
88 $foo =~ m/this|that/
89
90is TRUE if and only if C<$foo> contains either the sequence C<"this"> or
91the sequence C<"that">. Like all metacharacters, prefixing the C<"|">
92with a backslash makes it match the plain punctuation character; in its
93case, the VERTICAL LINE.
94
95 $foo =~ m/this\|that/
96
97is TRUE if and only if C<$foo> contains the sequence C<"this|that">.
98
99You aren't limited to just a single C<"|">.
100
101 $foo =~ m/fee|fie|foe|fum/
102
103is TRUE if and only if C<$foo> contains any of those 4 sequences from
104the children's story "Jack and the Beanstalk".
105
106As you can see, the C<"|"> binds less tightly than a sequence of
107ordinary characters. We can override this by using the grouping
108metacharacters, the parentheses C<"("> and C<")">.
109
110 $foo =~ m/th(is|at) thing/
111
112is TRUE if and only if C<$foo> contains either the sequence S<C<"this
113thing">> or the sequence S<C<"that thing">>. The portions of the string
114that match the portions of the pattern enclosed in parentheses are
115normally made available separately for use later in the pattern,
116substitution, or program. This is called "capturing", and it can get
117complicated. See L</Capture groups>.
118
119The first alternative includes everything from the last pattern
120delimiter (C<"(">, C<"(?:"> (described later), I<etc>. or the beginning
121of the pattern) up to the first C<"|">, and the last alternative
122contains everything from the last C<"|"> to the next closing pattern
123delimiter. That's why it's common practice to include alternatives in
124parentheses: to minimize confusion about where they start and end.
125
126Alternatives are tried from left to right, so the first
127alternative found for which the entire expression matches, is the one that
128is chosen. This means that alternatives are not necessarily greedy. For
129example: when matching C<foo|foot> against C<"barefoot">, only the C<"foo">
130part will match, as that is the first alternative tried, and it successfully
131matches the target string. (This might not seem important, but it is
132important when you are capturing matched text using parentheses.)
133
134Besides taking away the special meaning of a metacharacter, a prefixed
135backslash changes some letter and digit characters away from matching
136just themselves to instead have special meaning. These are called
137"escape sequences", and all such are described in L<perlrebackslash>. A
138backslash sequence (of a letter or digit) that doesn't currently have
139special meaning to Perl will raise a warning if warnings are enabled,
140as those are reserved for potential future use.
141
142One such sequence is C<\b>, which matches a boundary of some sort.
143C<\b{wb}> and a few others give specialized types of boundaries.
144(They are all described in detail starting at
145L<perlrebackslash/\b{}, \b, \B{}, \B>.) Note that these don't match
146characters, but the zero-width spaces between characters. They are an
147example of a L<zero-width assertion|/Assertions>. Consider again,
148
149 $foo =~ m/fee|fie|foe|fum/
150
151It evaluates to TRUE if, besides those 4 words, any of the sequences
152"feed", "field", "Defoe", "fume", and many others are in C<$foo>. By
153judicious use of C<\b> (or better (because it is designed to handle
154natural language) C<\b{wb}>), we can make sure that only the Giant's
155words are matched:
156
157 $foo =~ m/\b(fee|fie|foe|fum)\b/
158 $foo =~ m/\b{wb}(fee|fie|foe|fum)\b{wb}/
159
160The final example shows that the characters C<"{"> and C<"}"> are
161metacharacters.
162
163Another use for escape sequences is to specify characters that cannot
164(or which you prefer not to) be written literally. These are described
165in detail in L<perlrebackslash/Character Escapes>, but the next three
166paragraphs briefly describe some of them.
167
168Various control characters can be written in C language style: C<"\n">
169matches a newline, C<"\t"> a tab, C<"\r"> a carriage return, C<"\f"> a
170form feed, I<etc>.
171
172More generally, C<\I<nnn>>, where I<nnn> is a string of three octal
173digits, matches the character whose native code point is I<nnn>. You
174can easily run into trouble if you don't have exactly three digits. So
175always use three, or since Perl 5.14, you can use C<\o{...}> to specify
176any number of octal digits.
177
178Similarly, C<\xI<nn>>, where I<nn> are hexadecimal digits, matches the
179character whose native ordinal is I<nn>. Again, not using exactly two
180digits is a recipe for disaster, but you can use C<\x{...}> to specify
181any number of hex digits.
182
183Besides being a metacharacter, the C<"."> is an example of a "character
184class", something that can match any single character of a given set of
185them. In its case, the set is just about all possible characters. Perl
186predefines several character classes besides the C<".">; there is a
187separate reference page about just these, L<perlrecharclass>.
188
189You can define your own custom character classes, by putting into your
190pattern in the appropriate place(s), a list of all the characters you
191want in the set. You do this by enclosing the list within C<[]> bracket
192characters. These are called "bracketed character classes" when we are
193being precise, but often the word "bracketed" is dropped. (Dropping it
194usually doesn't cause confusion.) This means that the C<"["> character
195is another metacharacter. It doesn't match anything just by itself; it
196is used only to tell Perl that what follows it is a bracketed character
197class. If you want to match a literal left square bracket, you must
198escape it, like C<"\[">. The matching C<"]"> is also a metacharacter;
199again it doesn't match anything by itself, but just marks the end of
200your custom class to Perl. It is an example of a "sometimes
201metacharacter". It isn't a metacharacter if there is no corresponding
202C<"[">, and matches its literal self:
203
204 print "]" =~ /]/; # prints 1
205
206The list of characters within the character class gives the set of
207characters matched by the class. C<"[abc]"> matches a single "a" or "b"
208or "c". But if the first character after the C<"["> is C<"^">, the
209class instead matches any character not in the list. Within a list, the
210C<"-"> character specifies a range of characters, so that C<a-z>
211represents all characters between "a" and "z", inclusive. If you want
212either C<"-"> or C<"]"> itself to be a member of a class, put it at the
213start of the list (possibly after a C<"^">), or escape it with a
214backslash. C<"-"> is also taken literally when it is at the end of the
215list, just before the closing C<"]">. (The following all specify the
216same class of three characters: C<[-az]>, C<[az-]>, and C<[a\-z]>. All
217are different from C<[a-z]>, which specifies a class containing
218twenty-six characters, even on EBCDIC-based character sets.)
219
220There is lots more to bracketed character classes; full details are in
221L<perlrecharclass/Bracketed Character Classes>.
222
223=head3 Metacharacters
224X<metacharacter>
225X<\> X<^> X<.> X<$> X<|> X<(> X<()> X<[> X<[]>
226
227L</The Basics> introduced some of the metacharacters. This section
228gives them all. Most of them have the same meaning as in the I<egrep>
229command.
230
231Only the C<"\"> is always a metacharacter. The others are metacharacters
232just sometimes. The following tables lists all of them, summarizes
233their use, and gives the contexts where they are metacharacters.
234Outside those contexts or if prefixed by a C<"\">, they match their
235corresponding punctuation character. In some cases, their meaning
236varies depending on various pattern modifiers that alter the default
237behaviors. See L</Modifiers>.
238
239
240 PURPOSE WHERE
241 \ Escape the next character Always, except when
242 escaped by another \
243 ^ Match the beginning of the string Not in []
244 (or line, if /m is used)
245 ^ Complement the [] class At the beginning of []
246 . Match any single character except newline Not in []
247 (under /s, includes newline)
248 $ Match the end of the string Not in [], but can
249 (or before newline at the end of the mean interpolate a
250 string; or before any newline if /m is scalar
251 used)
252 | Alternation Not in []
253 () Grouping Not in []
254 [ Start Bracketed Character class Not in []
255 ] End Bracketed Character class Only in [], and
256 not first
257 * Matches the preceding element 0 or more Not in []
258 times
259 + Matches the preceding element 1 or more Not in []
260 times
261 ? Matches the preceding element 0 or 1 Not in []
262 times
263 { Starts a sequence that gives number(s) Not in []
264 of times the preceding element can be
265 matched
266 { when following certain escape sequences
267 starts a modifier to the meaning of the
268 sequence
269 } End sequence started by {
270 - Indicates a range Only in [] interior
271 # Beginning of comment, extends to line end Only with /x modifier
272
273Notice that most of the metacharacters lose their special meaning when
274they occur in a bracketed character class, except C<"^"> has a different
275meaning when it is at the beginning of such a class. And C<"-"> and C<"]">
276are metacharacters only at restricted positions within bracketed
277character classes; while C<"}"> is a metacharacter only when closing a
278special construct started by C<"{">.
279
280In double-quotish context, as is usually the case, you need to be
281careful about C<"$"> and the non-metacharacter C<"@">. Those could
282interpolate variables, which may or may not be what you intended.
283
284These rules were designed for compactness of expression, rather than
285legibility and maintainability. The L</E<sol>x and E<sol>xx> pattern
286modifiers allow you to insert white space to improve readability. And
287use of S<C<L<re 'strict'|re/'strict' mode>>> adds extra checking to
288catch some typos that might silently compile into something unintended.
289
290By default, the C<"^"> character is guaranteed to match only the
291beginning of the string, the C<"$"> character only the end (or before the
292newline at the end), and Perl does certain optimizations with the
293assumption that the string contains only one line. Embedded newlines
294will not be matched by C<"^"> or C<"$">. You may, however, wish to treat a
295string as a multi-line buffer, such that the C<"^"> will match after any
296newline within the string (except if the newline is the last character in
297the string), and C<"$"> will match before any newline. At the
298cost of a little more overhead, you can do this by using the
299C<L</E<sol>m>> modifier on the pattern match operator. (Older programs
300did this by setting C<$*>, but this option was removed in perl 5.10.)
301X<^> X<$> X</m>
302
303To simplify multi-line substitutions, the C<"."> character never matches a
304newline unless you use the L<C<E<sol>s>|/s> modifier, which in effect tells
305Perl to pretend the string is a single line--even if it isn't.
306X<.> X</s>
307
308=head2 Modifiers
309
310=head3 Overview
311
312The default behavior for matching can be changed, using various
313modifiers. Modifiers that relate to the interpretation of the pattern
314are listed just below. Modifiers that alter the way a pattern is used
315by Perl are detailed in L<perlop/"Regexp Quote-Like Operators"> and
316L<perlop/"Gory details of parsing quoted constructs">. Modifiers can be added
317dynamically; see L</Extended Patterns> below.
318
319=over 4
320
321=item B<C<m>>
322X</m> X<regex, multiline> X<regexp, multiline> X<regular expression, multiline>
323
324Treat the string being matched against as multiple lines. That is, change C<"^"> and C<"$"> from matching
325the start of the string's first line and the end of its last line to
326matching the start and end of each line within the string.
327
328=item B<C<s>>
329X</s> X<regex, single-line> X<regexp, single-line>
330X<regular expression, single-line>
331
332Treat the string as single line. That is, change C<"."> to match any character
333whatsoever, even a newline, which normally it would not match.
334
335Used together, as C</ms>, they let the C<"."> match any character whatsoever,
336while still allowing C<"^"> and C<"$"> to match, respectively, just after
337and just before newlines within the string.
338
339=item B<C<i>>
340X</i> X<regex, case-insensitive> X<regexp, case-insensitive>
341X<regular expression, case-insensitive>
342
343Do case-insensitive pattern matching. For example, "A" will match "a"
344under C</i>.
345
346If locale matching rules are in effect, the case map is taken from the
347current
348locale for code points less than 255, and from Unicode rules for larger
349code points. However, matches that would cross the Unicode
350rules/non-Unicode rules boundary (ords 255/256) will not succeed, unless
351the locale is a UTF-8 one. See L<perllocale>.
352
353There are a number of Unicode characters that match a sequence of
354multiple characters under C</i>. For example,
355C<LATIN SMALL LIGATURE FI> should match the sequence C<fi>. Perl is not
356currently able to do this when the multiple characters are in the pattern and
357are split between groupings, or when one or more are quantified. Thus
358
359 "\N{LATIN SMALL LIGATURE FI}" =~ /fi/i; # Matches
360 "\N{LATIN SMALL LIGATURE FI}" =~ /[fi][fi]/i; # Doesn't match!
361 "\N{LATIN SMALL LIGATURE FI}" =~ /fi*/i; # Doesn't match!
362
363 # The below doesn't match, and it isn't clear what $1 and $2 would
364 # be even if it did!!
365 "\N{LATIN SMALL LIGATURE FI}" =~ /(f)(i)/i; # Doesn't match!
366
367Perl doesn't match multiple characters in a bracketed
368character class unless the character that maps to them is explicitly
369mentioned, and it doesn't match them at all if the character class is
370inverted, which otherwise could be highly confusing. See
371L<perlrecharclass/Bracketed Character Classes>, and
372L<perlrecharclass/Negation>.
373
374=item B<C<x>> and B<C<xx>>
375X</x>
376
377Extend your pattern's legibility by permitting whitespace and comments.
378Details in L</E<sol>x and E<sol>xx>
379
380=item B<C<p>>
381X</p> X<regex, preserve> X<regexp, preserve>
382
383Preserve the string matched such that C<${^PREMATCH}>, C<${^MATCH}>, and
384C<${^POSTMATCH}> are available for use after matching.
385
386In Perl 5.20 and higher this is ignored. Due to a new copy-on-write
387mechanism, C<${^PREMATCH}>, C<${^MATCH}>, and C<${^POSTMATCH}> will be available
388after the match regardless of the modifier.
389
390=item B<C<a>>, B<C<d>>, B<C<l>>, and B<C<u>>
391X</a> X</d> X</l> X</u>
392
393These modifiers, all new in 5.14, affect which character-set rules
394(Unicode, I<etc>.) are used, as described below in
395L</Character set modifiers>.
396
397=item B<C<n>>
398X</n> X<regex, non-capture> X<regexp, non-capture>
399X<regular expression, non-capture>
400
401Prevent the grouping metacharacters C<()> from capturing. This modifier,
402new in 5.22, will stop C<$1>, C<$2>, I<etc>... from being filled in.
403
404 "hello" =~ /(hi|hello)/; # $1 is "hello"
405 "hello" =~ /(hi|hello)/n; # $1 is undef
406
407This is equivalent to putting C<?:> at the beginning of every capturing group:
408
409 "hello" =~ /(?:hi|hello)/; # $1 is undef
410
411C</n> can be negated on a per-group basis. Alternatively, named captures
412may still be used.
413
414 "hello" =~ /(?-n:(hi|hello))/n; # $1 is "hello"
415 "hello" =~ /(?<greet>hi|hello)/n; # $1 is "hello", $+{greet} is
416 # "hello"
417
418=item Other Modifiers
419
420There are a number of flags that can be found at the end of regular
421expression constructs that are I<not> generic regular expression flags, but
422apply to the operation being performed, like matching or substitution (C<m//>
423or C<s///> respectively).
424
425Flags described further in
426L<perlretut/"Using regular expressions in Perl"> are:
427
428 c - keep the current position during repeated matching
429 g - globally match the pattern repeatedly in the string
430
431Substitution-specific modifiers described in
432L<perlop/"s/PATTERN/REPLACEMENT/msixpodualngcer"> are:
433
434 e - evaluate the right-hand side as an expression
435 ee - evaluate the right side as a string then eval the result
436 o - pretend to optimize your code, but actually introduce bugs
437 r - perform non-destructive substitution and return the new value
438
439=back
440
441Regular expression modifiers are usually written in documentation
442as I<e.g.>, "the C</x> modifier", even though the delimiter
443in question might not really be a slash. The modifiers C</imnsxadlup>
444may also be embedded within the regular expression itself using
445the C<(?...)> construct, see L</Extended Patterns> below.
446
447=head3 Details on some modifiers
448
449Some of the modifiers require more explanation than given in the
450L</Overview> above.
451
452=head4 C</x> and C</xx>
453
454A single C</x> tells
455the regular expression parser to ignore most whitespace that is neither
456backslashed nor within a bracketed character class, nor within the characters
457of a multi-character metapattern like C<(?i: ... )>. You can use this to
458break up your regular expression into more readable parts.
459Also, the C<"#"> character is treated as a metacharacter introducing a
460comment that runs up to the pattern's closing delimiter, or to the end
461of the current line if the pattern extends onto the next line. Hence,
462this is very much like an ordinary Perl code comment. (You can include
463the closing delimiter within the comment only if you precede it with a
464backslash, so be careful!)
465
466Use of C</x> means that if you want real
467whitespace or C<"#"> characters in the pattern (outside a bracketed character
468class, which is unaffected by C</x>), then you'll either have to
469escape them (using backslashes or C<\Q...\E>) or encode them using octal,
470hex, or C<\N{}> or C<\p{name=...}> escapes.
471It is ineffective to try to continue a comment onto the next line by
472escaping the C<\n> with a backslash or C<\Q>.
473
474You can use L</(?#text)> to create a comment that ends earlier than the
475end of the current line, but C<text> also can't contain the closing
476delimiter unless escaped with a backslash.
477
478A common pitfall is to forget that C<"#"> characters (outside a
479bracketed character class) begin a comment under C</x> and are not
480matched literally. Just keep that in mind when trying to puzzle out why
481a particular C</x> pattern isn't working as expected.
482Inside a bracketed character class, C<"#"> retains its non-special,
483literal meaning.
484
485Starting in Perl v5.26, if the modifier has a second C<"x"> within it,
486the effect of a single C</x> is increased. The only difference is that
487inside bracketed character classes, non-escaped (by a backslash) SPACE
488and TAB characters are not added to the class, and hence can be inserted
489to make the classes more readable:
490
491 / [d-e g-i 3-7]/xx
492 /[ ! @ " # $ % ^ & * () = ? <> ' ]/xx
493
494may be easier to grasp than the squashed equivalents
495
496 /[d-eg-i3-7]/
497 /[!@"#$%^&*()=?<>']/
498
499Note that this unfortunately doesn't mean that your bracketed classes
500can contain comments or extend over multiple lines. A C<#> inside a
501character class is still just a literal C<#>, and doesn't introduce a
502comment. And, unless the closing bracket is on the same line as the
503opening one, the newline character (and everything on the next line(s)
504until terminated by a C<]> will be part of the class, just as if you'd
505written C<\n>.
506
507Taken together, these features go a long way towards
508making Perl's regular expressions more readable. Here's an example:
509
510 # Delete (most) C comments.
511 $program =~ s {
512 /\* # Match the opening delimiter.
513 .*? # Match a minimal number of characters.
514 \*/ # Match the closing delimiter.
515 } []gsx;
516
517Note that anything inside
518a C<\Q...\E> stays unaffected by C</x>. And note that C</x> doesn't affect
519space interpretation within a single multi-character construct. For
520example C<(?:...)> can't have a space between the C<"(">,
521C<"?">, and C<":">. Within any delimiters for such a construct, allowed
522spaces are not affected by C</x>, and depend on the construct. For
523example, all constructs using curly braces as delimiters, such as
524C<\x{...}> can have blanks within but adjacent to the braces, but not
525elsewhere, and no non-blank space characters. An exception are Unicode
526properties which follow Unicode rules, for which see
527L<perluniprops/Properties accessible through \p{} and \P{}>.
528X</x>
529
530The set of characters that are deemed whitespace are those that Unicode
531calls "Pattern White Space", namely:
532
533 U+0009 CHARACTER TABULATION
534 U+000A LINE FEED
535 U+000B LINE TABULATION
536 U+000C FORM FEED
537 U+000D CARRIAGE RETURN
538 U+0020 SPACE
539 U+0085 NEXT LINE
540 U+200E LEFT-TO-RIGHT MARK
541 U+200F RIGHT-TO-LEFT MARK
542 U+2028 LINE SEPARATOR
543 U+2029 PARAGRAPH SEPARATOR
544
545=head4 Character set modifiers
546
547C</d>, C</u>, C</a>, and C</l>, available starting in 5.14, are called
548the character set modifiers; they affect the character set rules
549used for the regular expression.
550
551The C</d>, C</u>, and C</l> modifiers are not likely to be of much use
552to you, and so you need not worry about them very much. They exist for
553Perl's internal use, so that complex regular expression data structures
554can be automatically serialized and later exactly reconstituted,
555including all their nuances. But, since Perl can't keep a secret, and
556there may be rare instances where they are useful, they are documented
557here.
558
559The C</a> modifier, on the other hand, may be useful. Its purpose is to
560allow code that is to work mostly on ASCII data to not have to concern
561itself with Unicode.
562
563Briefly, C</l> sets the character set to that of whatever B<L>ocale is in
564effect at the time of the execution of the pattern match.
565
566C</u> sets the character set to B<U>nicode.
567
568C</a> also sets the character set to Unicode, BUT adds several
569restrictions for B<A>SCII-safe matching.
570
571C</d> is the old, problematic, pre-5.14 B<D>efault character set
572behavior. Its only use is to force that old behavior.
573
574At any given time, exactly one of these modifiers is in effect. Their
575existence allows Perl to keep the originally compiled behavior of a
576regular expression, regardless of what rules are in effect when it is
577actually executed. And if it is interpolated into a larger regex, the
578original's rules continue to apply to it, and don't affect the other
579parts.
580
581The C</l> and C</u> modifiers are automatically selected for
582regular expressions compiled within the scope of various pragmas,
583and we recommend that in general, you use those pragmas instead of
584specifying these modifiers explicitly. For one thing, the modifiers
585affect only pattern matching, and do not extend to even any replacement
586done, whereas using the pragmas gives consistent results for all
587appropriate operations within their scopes. For example,
588
589 s/foo/\Ubar/il
590
591will match "foo" using the locale's rules for case-insensitive matching,
592but the C</l> does not affect how the C<\U> operates. Most likely you
593want both of them to use locale rules. To do this, instead compile the
594regular expression within the scope of C<use locale>. This both
595implicitly adds the C</l>, and applies locale rules to the C<\U>. The
596lesson is to C<use locale>, and not C</l> explicitly.
597
598Similarly, it would be better to use C<use feature 'unicode_strings'>
599instead of,
600
601 s/foo/\Lbar/iu
602
603to get Unicode rules, as the C<\L> in the former (but not necessarily
604the latter) would also use Unicode rules.
605
606More detail on each of the modifiers follows. Most likely you don't
607need to know this detail for C</l>, C</u>, and C</d>, and can skip ahead
608to L<E<sol>a|/E<sol>a (and E<sol>aa)>.
609
610=head4 /l
611
612means to use the current locale's rules (see L<perllocale>) when pattern
613matching. For example, C<\w> will match the "word" characters of that
614locale, and C<"/i"> case-insensitive matching will match according to
615the locale's case folding rules. The locale used will be the one in
616effect at the time of execution of the pattern match. This may not be
617the same as the compilation-time locale, and can differ from one match
618to another if there is an intervening call of the
619L<setlocale() function|perllocale/The setlocale function>.
620
621Prior to v5.20, Perl did not support multi-byte locales. Starting then,
622UTF-8 locales are supported. No other multi byte locales are ever
623likely to be supported. However, in all locales, one can have code
624points above 255 and these will always be treated as Unicode no matter
625what locale is in effect.
626
627Under Unicode rules, there are a few case-insensitive matches that cross
628the 255/256 boundary. Except for UTF-8 locales in Perls v5.20 and
629later, these are disallowed under C</l>. For example, 0xFF (on ASCII
630platforms) does not caselessly match the character at 0x178, C<LATIN
631CAPITAL LETTER Y WITH DIAERESIS>, because 0xFF may not be C<LATIN SMALL
632LETTER Y WITH DIAERESIS> in the current locale, and Perl has no way of
633knowing if that character even exists in the locale, much less what code
634point it is.
635
636In a UTF-8 locale in v5.20 and later, the only visible difference
637between locale and non-locale in regular expressions should be tainting,
638if your perl supports taint checking (see L<perlsec>).
639
640This modifier may be specified to be the default by C<use locale>, but
641see L</Which character set modifier is in effect?>.
642X</l>
643
644=head4 /u
645
646means to use Unicode rules when pattern matching. On ASCII platforms,
647this means that the code points between 128 and 255 take on their
648Latin-1 (ISO-8859-1) meanings (which are the same as Unicode's).
649(Otherwise Perl considers their meanings to be undefined.) Thus,
650under this modifier, the ASCII platform effectively becomes a Unicode
651platform; and hence, for example, C<\w> will match any of the more than
652100_000 word characters in Unicode.
653
654Unlike most locales, which are specific to a language and country pair,
655Unicode classifies all the characters that are letters I<somewhere> in
656the world as
657C<\w>. For example, your locale might not think that C<LATIN SMALL
658LETTER ETH> is a letter (unless you happen to speak Icelandic), but
659Unicode does. Similarly, all the characters that are decimal digits
660somewhere in the world will match C<\d>; this is hundreds, not 10,
661possible matches. And some of those digits look like some of the 10
662ASCII digits, but mean a different number, so a human could easily think
663a number is a different quantity than it really is. For example,
664C<BENGALI DIGIT FOUR> (U+09EA) looks very much like an
665C<ASCII DIGIT EIGHT> (U+0038), and C<LEPCHA DIGIT SIX> (U+1C46) looks
666very much like an C<ASCII DIGIT FIVE> (U+0035). And, C<\d+>, may match
667strings of digits that are a mixture from different writing systems,
668creating a security issue. A fraudulent website, for example, could
669display the price of something using U+1C46, and it would appear to the
670user that something cost 500 units, but it really costs 600. A browser
671that enforced script runs (L</Script Runs>) would prevent that
672fraudulent display. L<Unicode::UCD/num()> can also be used to sort this
673out. Or the C</a> modifier can be used to force C<\d> to match just the
674ASCII 0 through 9.
675
676Also, under this modifier, case-insensitive matching works on the full
677set of Unicode
678characters. The C<KELVIN SIGN>, for example matches the letters "k" and
679"K"; and C<LATIN SMALL LIGATURE FF> matches the sequence "ff", which,
680if you're not prepared, might make it look like a hexadecimal constant,
681presenting another potential security issue. See
682L<https://unicode.org/reports/tr36> for a detailed discussion of Unicode
683security issues.
684
685This modifier may be specified to be the default by C<use feature
686'unicode_strings>, C<use locale ':not_characters'>, or
687C<L<use v5.12|perlfunc/use VERSION>> (or higher),
688but see L</Which character set modifier is in effect?>.
689X</u>
690
691=head4 /d
692
693B<IMPORTANT:> Because of the unpredictable behaviors this
694modifier causes, only use it to maintain weird backward compatibilities.
695Use the
696L<< C<unicode_strings>|feature/"The 'unicode_strings' feature" >>
697feature
698in new code to avoid inadvertently enabling this modifier by default.
699
700What does this modifier do? It "Depends"!
701
702This modifier means to use platform-native matching rules
703except when there is cause to use Unicode rules instead, as follows:
704
705=over 4
706
707=item 1
708
709the target string's L<UTF8 flag|perlunifaq/What is "the UTF8 flag"?>
710(see below) is set; or
711
712=item 2
713
714the pattern's L<UTF8 flag|perlunifaq/What is "the UTF8 flag"?>
715(see below) is set; or
716
717=item 3
718
719the pattern explicitly mentions a code point that is above 255 (say by
720C<\x{100}>); or
721
722=item 4
723
724the pattern uses a Unicode name (C<\N{...}>); or
725
726=item 5
727
728the pattern uses a Unicode property (C<\p{...}> or C<\P{...}>); or
729
730=item 6
731
732the pattern uses a Unicode break (C<\b{...}> or C<\B{...}>); or
733
734=item 7
735
736the pattern uses C<L</(?[ ])>>
737
738=item 8
739
740the pattern uses L<C<(*script_run: ...)>|/Script Runs>
741
742=back
743
744Regarding the "UTF8 flag" references above: normally Perl applications
745shouldn't think about that flag. It's part of Perl's internals,
746so it can change whenever Perl wants. C</d> may thus cause unpredictable
747results. See L<perlunicode/The "Unicode Bug">. This bug
748has become rather infamous, leading to yet other (without swearing) names
749for this modifier like "Dicey" and "Dodgy".
750
751Here are some examples of how that works on an ASCII platform:
752
753 $str = "\xDF"; #
754 utf8::downgrade($str); # $str is not UTF8-flagged.
755 $str =~ /^\w/; # No match, since no UTF8 flag.
756
757 $str .= "\x{0e0b}"; # Now $str is UTF8-flagged.
758 $str =~ /^\w/; # Match! $str is now UTF8-flagged.
759 chop $str;
760 $str =~ /^\w/; # Still a match! $str retains its UTF8 flag.
761
762Under Perl's default configuration this modifier is automatically
763selected by default when none of the others are, so yet another name
764for it (unfortunately) is "Default".
765
766Whenever you can, use the
767L<< C<unicode_strings>|feature/"The 'unicode_strings' feature" >>
768to cause X</u> to be the default instead.
769
770=head4 /a (and /aa)
771
772This modifier stands for ASCII-restrict (or ASCII-safe). This modifier
773may be doubled-up to increase its effect.
774
775When it appears singly, it causes the sequences C<\d>, C<\s>, C<\w>, and
776the Posix character classes to match only in the ASCII range. They thus
777revert to their pre-5.6, pre-Unicode meanings. Under C</a>, C<\d>
778always means precisely the digits C<"0"> to C<"9">; C<\s> means the five
779characters C<[ \f\n\r\t]>, and starting in Perl v5.18, the vertical tab;
780C<\w> means the 63 characters
781C<[A-Za-z0-9_]>; and likewise, all the Posix classes such as
782C<[[:print:]]> match only the appropriate ASCII-range characters.
783
784This modifier is useful for people who only incidentally use Unicode,
785and who do not wish to be burdened with its complexities and security
786concerns.
787
788With C</a>, one can write C<\d> with confidence that it will only match
789ASCII characters, and should the need arise to match beyond ASCII, you
790can instead use C<\p{Digit}> (or C<\p{Word}> for C<\w>). There are
791similar C<\p{...}> constructs that can match beyond ASCII both white
792space (see L<perlrecharclass/Whitespace>), and Posix classes (see
793L<perlrecharclass/POSIX Character Classes>). Thus, this modifier
794doesn't mean you can't use Unicode, it means that to get Unicode
795matching you must explicitly use a construct (C<\p{}>, C<\P{}>) that
796signals Unicode.
797
798As you would expect, this modifier causes, for example, C<\D> to mean
799the same thing as C<[^0-9]>; in fact, all non-ASCII characters match
800C<\D>, C<\S>, and C<\W>. C<\b> still means to match at the boundary
801between C<\w> and C<\W>, using the C</a> definitions of them (similarly
802for C<\B>).
803
804Otherwise, C</a> behaves like the C</u> modifier, in that
805case-insensitive matching uses Unicode rules; for example, "k" will
806match the Unicode C<\N{KELVIN SIGN}> under C</i> matching, and code
807points in the Latin1 range, above ASCII will have Unicode rules when it
808comes to case-insensitive matching.
809
810To forbid ASCII/non-ASCII matches (like "k" with C<\N{KELVIN SIGN}>),
811specify the C<"a"> twice, for example C</aai> or C</aia>. (The first
812occurrence of C<"a"> restricts the C<\d>, I<etc>., and the second occurrence
813adds the C</i> restrictions.) But, note that code points outside the
814ASCII range will use Unicode rules for C</i> matching, so the modifier
815doesn't really restrict things to just ASCII; it just forbids the
816intermixing of ASCII and non-ASCII.
817
818To summarize, this modifier provides protection for applications that
819don't wish to be exposed to all of Unicode. Specifying it twice
820gives added protection.
821
822This modifier may be specified to be the default by C<use re '/a'>
823or C<use re '/aa'>. If you do so, you may actually have occasion to use
824the C</u> modifier explicitly if there are a few regular expressions
825where you do want full Unicode rules (but even here, it's best if
826everything were under feature C<"unicode_strings">, along with the
827C<use re '/aa'>). Also see L</Which character set modifier is in
828effect?>.
829X</a>
830X</aa>
831
832=head4 Which character set modifier is in effect?
833
834Which of these modifiers is in effect at any given point in a regular
835expression depends on a fairly complex set of interactions. These have
836been designed so that in general you don't have to worry about it, but
837this section gives the gory details. As
838explained below in L</Extended Patterns> it is possible to explicitly
839specify modifiers that apply only to portions of a regular expression.
840The innermost always has priority over any outer ones, and one applying
841to the whole expression has priority over any of the default settings that are
842described in the remainder of this section.
843
844The C<L<use re 'E<sol>foo'|re/"'/flags' mode">> pragma can be used to set
845default modifiers (including these) for regular expressions compiled
846within its scope. This pragma has precedence over the other pragmas
847listed below that also change the defaults.
848
849Otherwise, C<L<use locale|perllocale>> sets the default modifier to C</l>;
850and C<L<use feature 'unicode_strings|feature>>, or
851C<L<use v5.12|perlfunc/use VERSION>> (or higher) set the default to
852C</u> when not in the same scope as either C<L<use locale|perllocale>>
853or C<L<use bytes|bytes>>.
854(C<L<use locale ':not_characters'|perllocale/Unicode and UTF-8>> also
855sets the default to C</u>, overriding any plain C<use locale>.)
856Unlike the mechanisms mentioned above, these
857affect operations besides regular expressions pattern matching, and so
858give more consistent results with other operators, including using
859C<\U>, C<\l>, I<etc>. in substitution replacements.
860
861If none of the above apply, for backwards compatibility reasons, the
862C</d> modifier is the one in effect by default. As this can lead to
863unexpected results, it is best to specify which other rule set should be
864used.
865
866=head4 Character set modifier behavior prior to Perl 5.14
867
868Prior to 5.14, there were no explicit modifiers, but C</l> was implied
869for regexes compiled within the scope of C<use locale>, and C</d> was
870implied otherwise. However, interpolating a regex into a larger regex
871would ignore the original compilation in favor of whatever was in effect
872at the time of the second compilation. There were a number of
873inconsistencies (bugs) with the C</d> modifier, where Unicode rules
874would be used when inappropriate, and vice versa. C<\p{}> did not imply
875Unicode rules, and neither did all occurrences of C<\N{}>, until 5.12.
876
877=head2 Regular Expressions
878
879=head3 Quantifiers
880
881Quantifiers are used when a particular portion of a pattern needs to
882match a certain number (or numbers) of times. If there isn't a
883quantifier the number of times to match is exactly one. The following
884standard quantifiers are recognized:
885X<metacharacter> X<quantifier> X<*> X<+> X<?> X<{n}> X<{n,}> X<{n,m}>
886
887 * Match 0 or more times
888 + Match 1 or more times
889 ? Match 1 or 0 times
890 {n} Match exactly n times
891 {n,} Match at least n times
892 {,n} Match at most n times
893 {n,m} Match at least n but not more than m times
894
895(If a non-escaped curly bracket occurs in a context other than one of
896the quantifiers listed above, where it does not form part of a
897backslashed sequence like C<\x{...}>, it is either a fatal syntax error,
898or treated as a regular character, generally with a deprecation warning
899raised. To escape it, you can precede it with a backslash (C<"\{">) or
900enclose it within square brackets (C<"[{]">).
901This change will allow for future syntax extensions (like making the
902lower bound of a quantifier optional), and better error checking of
903quantifiers).
904
905The C<"*"> quantifier is equivalent to C<{0,}>, the C<"+">
906quantifier to C<{1,}>, and the C<"?"> quantifier to C<{0,1}>. I<n> and I<m> are limited
907to non-negative integral values less than a preset limit defined when perl is built.
908This is usually 65534 on the most common platforms. The actual limit can
909be seen in the error message generated by code such as this:
910
911 $_ **= $_ , / {$_} / for 2 .. 42;
912
913By default, a quantified subpattern is "greedy", that is, it will match as
914many times as possible (given a particular starting location) while still
915allowing the rest of the pattern to match. If you want it to match the
916minimum number of times possible, follow the quantifier with a C<"?">. Note
917that the meanings don't change, just the "greediness":
918X<metacharacter> X<greedy> X<greediness>
919X<?> X<*?> X<+?> X<??> X<{n}?> X<{n,}?> X<{,n}?> X<{n,m}?>
920
921 *? Match 0 or more times, not greedily
922 +? Match 1 or more times, not greedily
923 ?? Match 0 or 1 time, not greedily
924 {n}? Match exactly n times, not greedily (redundant)
925 {n,}? Match at least n times, not greedily
926 {,n}? Match at most n times, not greedily
927 {n,m}? Match at least n but not more than m times, not greedily
928
929Normally when a quantified subpattern does not allow the rest of the
930overall pattern to match, Perl will backtrack. However, this behaviour is
931sometimes undesirable. Thus Perl provides the "possessive" quantifier form
932as well.
933
934 *+ Match 0 or more times and give nothing back
935 ++ Match 1 or more times and give nothing back
936 ?+ Match 0 or 1 time and give nothing back
937 {n}+ Match exactly n times and give nothing back (redundant)
938 {n,}+ Match at least n times and give nothing back
939 {,n}+ Match at most n times and give nothing back
940 {n,m}+ Match at least n but not more than m times and give nothing back
941
942For instance,
943
944 'aaaa' =~ /a++a/
945
946will never match, as the C<a++> will gobble up all the C<"a">'s in the
947string and won't leave any for the remaining part of the pattern. This
948feature can be extremely useful to give perl hints about where it
949shouldn't backtrack. For instance, the typical "match a double-quoted
950string" problem can be most efficiently performed when written as:
951
952 /"(?:[^"\\]++|\\.)*+"/
953
954as we know that if the final quote does not match, backtracking will not
955help. See the independent subexpression
956C<L</(?E<gt>I<pattern>)>> for more details;
957possessive quantifiers are just syntactic sugar for that construct. For
958instance the above example could also be written as follows:
959
960 /"(?>(?:(?>[^"\\]+)|\\.)*)"/
961
962Note that the possessive quantifier modifier can not be combined
963with the non-greedy modifier. This is because it would make no sense.
964Consider the follow equivalency table:
965
966 Illegal Legal
967 ------------ ------
968 X??+ X{0}
969 X+?+ X{1}
970 X{min,max}?+ X{min}
971
972=head3 Escape sequences
973
974Because patterns are processed as double-quoted strings, the following
975also work:
976
977 \t tab (HT, TAB)
978 \n newline (LF, NL)
979 \r return (CR)
980 \f form feed (FF)
981 \a alarm (bell) (BEL)
982 \e escape (think troff) (ESC)
983 \cK control char (example: VT)
984 \x{}, \x00 character whose ordinal is the given hexadecimal number
985 \N{name} named Unicode character or character sequence
986 \N{U+263D} Unicode character (example: FIRST QUARTER MOON)
987 \o{}, \000 character whose ordinal is the given octal number
988 \l lowercase next char (think vi)
989 \u uppercase next char (think vi)
990 \L lowercase until \E (think vi)
991 \U uppercase until \E (think vi)
992 \Q quote (disable) pattern metacharacters until \E
993 \E end either case modification or quoted section, think vi
994
995Details are in L<perlop/Quote and Quote-like Operators>.
996
997=head3 Character Classes and other Special Escapes
998
999In addition, Perl defines the following:
1000X<\g> X<\k> X<\K> X<backreference>
1001
1002 Sequence Note Description
1003 [...] [1] Match a character according to the rules of the
1004 bracketed character class defined by the "...".
1005 Example: [a-z] matches "a" or "b" or "c" ... or "z"
1006 [[:...:]] [2] Match a character according to the rules of the POSIX
1007 character class "..." within the outer bracketed
1008 character class. Example: [[:upper:]] matches any
1009 uppercase character.
1010 (?[...]) [8] Extended bracketed character class
1011 \w [3] Match a "word" character (alphanumeric plus "_", plus
1012 other connector punctuation chars plus Unicode
1013 marks)
1014 \W [3] Match a non-"word" character
1015 \s [3] Match a whitespace character
1016 \S [3] Match a non-whitespace character
1017 \d [3] Match a decimal digit character
1018 \D [3] Match a non-digit character
1019 \pP [3] Match P, named property. Use \p{Prop} for longer names
1020 \PP [3] Match non-P
1021 \X [4] Match Unicode "eXtended grapheme cluster"
1022 \1 [5] Backreference to a specific capture group or buffer.
1023 '1' may actually be any positive integer.
1024 \g1 [5] Backreference to a specific or previous group,
1025 \g{-1} [5] The number may be negative indicating a relative
1026 previous group and may optionally be wrapped in
1027 curly brackets for safer parsing.
1028 \g{name} [5] Named backreference
1029 \k<name> [5] Named backreference
1030 \k'name' [5] Named backreference
1031 \k{name} [5] Named backreference
1032 \K [6] Keep the stuff left of the \K, don't include it in $&
1033 \N [7] Any character but \n. Not affected by /s modifier
1034 \v [3] Vertical whitespace
1035 \V [3] Not vertical whitespace
1036 \h [3] Horizontal whitespace
1037 \H [3] Not horizontal whitespace
1038 \R [4] Linebreak
1039
1040=over 4
1041
1042=item [1]
1043
1044See L<perlrecharclass/Bracketed Character Classes> for details.
1045
1046=item [2]
1047
1048See L<perlrecharclass/POSIX Character Classes> for details.
1049
1050=item [3]
1051
1052See L<perlunicode/Unicode Character Properties> for details
1053
1054=item [4]
1055
1056See L<perlrebackslash/Misc> for details.
1057
1058=item [5]
1059
1060See L</Capture groups> below for details.
1061
1062=item [6]
1063
1064See L</Extended Patterns> below for details.
1065
1066=item [7]
1067
1068Note that C<\N> has two meanings. When of the form C<\N{I<NAME>}>, it
1069matches the character or character sequence whose name is I<NAME>; and
1070similarly
1071when of the form C<\N{U+I<hex>}>, it matches the character whose Unicode
1072code point is I<hex>. Otherwise it matches any character but C<\n>.
1073
1074=item [8]
1075
1076See L<perlrecharclass/Extended Bracketed Character Classes> for details.
1077
1078=back
1079
1080=head3 Assertions
1081
1082Besides L<C<"^"> and C<"$">|/Metacharacters>, Perl defines the following
1083zero-width assertions:
1084X<zero-width assertion> X<assertion> X<regex, zero-width assertion>
1085X<regexp, zero-width assertion>
1086X<regular expression, zero-width assertion>
1087X<\b> X<\B> X<\A> X<\Z> X<\z> X<\G>
1088
1089 \b{} Match at Unicode boundary of specified type
1090 \B{} Match where corresponding \b{} doesn't match
1091 \b Match a \w\W or \W\w boundary
1092 \B Match except at a \w\W or \W\w boundary
1093 \A Match only at beginning of string
1094 \Z Match only at end of string, or before newline at the end
1095 \z Match only at end of string
1096 \G Match only at pos() (e.g. at the end-of-match position
1097 of prior m//g)
1098
1099A Unicode boundary (C<\b{}>), available starting in v5.22, is a spot
1100between two characters, or before the first character in the string, or
1101after the final character in the string where certain criteria defined
1102by Unicode are met. See L<perlrebackslash/\b{}, \b, \B{}, \B> for
1103details.
1104
1105A word boundary (C<\b>) is a spot between two characters
1106that has a C<\w> on one side of it and a C<\W> on the other side
1107of it (in either order), counting the imaginary characters off the
1108beginning and end of the string as matching a C<\W>. (Within
1109character classes C<\b> represents backspace rather than a word
1110boundary, just as it normally does in any double-quoted string.)
1111The C<\A> and C<\Z> are just like C<"^"> and C<"$">, except that they
1112won't match multiple times when the C</m> modifier is used, while
1113C<"^"> and C<"$"> will match at every internal line boundary. To match
1114the actual end of the string and not ignore an optional trailing
1115newline, use C<\z>.
1116X<\b> X<\A> X<\Z> X<\z> X</m>
1117
1118The C<\G> assertion can be used to chain global matches (using
1119C<m//g>), as described in L<perlop/"Regexp Quote-Like Operators">.
1120It is also useful when writing C<lex>-like scanners, when you have
1121several patterns that you want to match against consequent substrings
1122of your string; see the previous reference. The actual location
1123where C<\G> will match can also be influenced by using C<pos()> as
1124an lvalue: see L<perlfunc/pos>. Note that the rule for zero-length
1125matches (see L</"Repeated Patterns Matching a Zero-length Substring">)
1126is modified somewhat, in that contents to the left of C<\G> are
1127not counted when determining the length of the match. Thus the following
1128will not match forever:
1129X<\G>
1130
1131 my $string = 'ABC';
1132 pos($string) = 1;
1133 while ($string =~ /(.\G)/g) {
1134 print $1;
1135 }
1136
1137It will print 'A' and then terminate, as it considers the match to
1138be zero-width, and thus will not match at the same position twice in a
1139row.
1140
1141It is worth noting that C<\G> improperly used can result in an infinite
1142loop. Take care when using patterns that include C<\G> in an alternation.
1143
1144Note also that C<s///> will refuse to overwrite part of a substitution
1145that has already been replaced; so for example this will stop after the
1146first iteration, rather than iterating its way backwards through the
1147string:
1148
1149 $_ = "123456789";
1150 pos = 6;
1151 s/.(?=.\G)/X/g;
1152 print; # prints 1234X6789, not XXXXX6789
1153
1154
1155=head3 Capture groups
1156
1157The grouping construct C<( ... )> creates capture groups (also referred to as
1158capture buffers). To refer to the current contents of a group later on, within
1159the same pattern, use C<\g1> (or C<\g{1}>) for the first, C<\g2> (or C<\g{2}>)
1160for the second, and so on.
1161This is called a I<backreference>.
1162X<regex, capture buffer> X<regexp, capture buffer>
1163X<regex, capture group> X<regexp, capture group>
1164X<regular expression, capture buffer> X<backreference>
1165X<regular expression, capture group> X<backreference>
1166X<\g{1}> X<\g{-1}> X<\g{name}> X<relative backreference> X<named backreference>
1167X<named capture buffer> X<regular expression, named capture buffer>
1168X<named capture group> X<regular expression, named capture group>
1169X<%+> X<$+{name}> X<< \k<name> >>
1170There is no limit to the number of captured substrings that you may use.
1171Groups are numbered with the leftmost open parenthesis being number 1, I<etc>. If
1172a group did not match, the associated backreference won't match either. (This
1173can happen if the group is optional, or in a different branch of an
1174alternation.)
1175You can omit the C<"g">, and write C<"\1">, I<etc>, but there are some issues with
1176this form, described below.
1177
1178You can also refer to capture groups relatively, by using a negative number, so
1179that C<\g-1> and C<\g{-1}> both refer to the immediately preceding capture
1180group, and C<\g-2> and C<\g{-2}> both refer to the group before it. For
1181example:
1182
1183 /
1184 (Y) # group 1
1185 ( # group 2
1186 (X) # group 3
1187 \g{-1} # backref to group 3
1188 \g{-3} # backref to group 1
1189 )
1190 /x
1191
1192would match the same as C</(Y) ( (X) \g3 \g1 )/x>. This allows you to
1193interpolate regexes into larger regexes and not have to worry about the
1194capture groups being renumbered.
1195
1196You can dispense with numbers altogether and create named capture groups.
1197The notation is C<(?E<lt>I<name>E<gt>...)> to declare and C<\g{I<name>}> to
1198reference. (To be compatible with .Net regular expressions, C<\g{I<name>}> may
1199also be written as C<\k{I<name>}>, C<\kE<lt>I<name>E<gt>> or C<\k'I<name>'>.)
1200I<name> must not begin with a number, nor contain hyphens.
1201When different groups within the same pattern have the same name, any reference
1202to that name assumes the leftmost defined group. Named groups count in
1203absolute and relative numbering, and so can also be referred to by those
1204numbers.
1205(It's possible to do things with named capture groups that would otherwise
1206require C<(??{})>.)
1207
1208Capture group contents are dynamically scoped and available to you outside the
1209pattern until the end of the enclosing block or until the next successful
1210match in the same scope, whichever comes first.
1211See L<perlsyn/"Compound Statements"> and
1212L<perlvar/"Scoping Rules of Regex Variables"> for more details.
1213
1214You can access the contents of a capture group by absolute number (using
1215C<"$1"> instead of C<"\g1">, I<etc>); or by name via the C<%+> hash,
1216using C<"$+{I<name>}">.
1217
1218Braces are required in referring to named capture groups, but are optional for
1219absolute or relative numbered ones. Braces are safer when creating a regex by
1220concatenating smaller strings. For example if you have C<qr/$x$y/>, and C<$x>
1221contained C<"\g1">, and C<$y> contained C<"37">, you would get C</\g137/> which
1222is probably not what you intended.
1223
1224If you use braces, you may also optionally add any number of blank
1225(space or tab) characters within but adjacent to the braces, like
1226S<C<\g{ -1 }>>, or S<C<\k{ I<name> }>>.
1227
1228The C<\g> and C<\k> notations were introduced in Perl 5.10.0. Prior to that
1229there were no named nor relative numbered capture groups. Absolute numbered
1230groups were referred to using C<\1>,
1231C<\2>, I<etc>., and this notation is still
1232accepted (and likely always will be). But it leads to some ambiguities if
1233there are more than 9 capture groups, as C<\10> could mean either the tenth
1234capture group, or the character whose ordinal in octal is 010 (a backspace in
1235ASCII). Perl resolves this ambiguity by interpreting C<\10> as a backreference
1236only if at least 10 left parentheses have opened before it. Likewise C<\11> is
1237a backreference only if at least 11 left parentheses have opened before it.
1238And so on. C<\1> through C<\9> are always interpreted as backreferences.
1239There are several examples below that illustrate these perils. You can avoid
1240the ambiguity by always using C<\g{}> or C<\g> if you mean capturing groups;
1241and for octal constants always using C<\o{}>, or for C<\077> and below, using 3
1242digits padded with leading zeros, since a leading zero implies an octal
1243constant.
1244
1245The C<\I<digit>> notation also works in certain circumstances outside
1246the pattern. See L</Warning on \1 Instead of $1> below for details.
1247
1248Examples:
1249
1250 s/^([^ ]*) *([^ ]*)/$2 $1/; # swap first two words
1251
1252 /(.)\g1/ # find first doubled char
1253 and print "'$1' is the first doubled character\n";
1254
1255 /(?<char>.)\k<char>/ # ... a different way
1256 and print "'$+{char}' is the first doubled character\n";
1257
1258 /(?'char'.)\g1/ # ... mix and match
1259 and print "'$1' is the first doubled character\n";
1260
1261 if (/Time: (..):(..):(..)/) { # parse out values
1262 $hours = $1;
1263 $minutes = $2;
1264 $seconds = $3;
1265 }
1266
1267 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\g10/ # \g10 is a backreference
1268 /(.)(.)(.)(.)(.)(.)(.)(.)(.)\10/ # \10 is octal
1269 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\10/ # \10 is a backreference
1270 /((.)(.)(.)(.)(.)(.)(.)(.)(.))\010/ # \010 is octal
1271
1272 $x = '(.)\1'; # Creates problems when concatenated.
1273 $y = '(.)\g{1}'; # Avoids the problems.
1274 "aa" =~ /${x}/; # True
1275 "aa" =~ /${y}/; # True
1276 "aa0" =~ /${x}0/; # False!
1277 "aa0" =~ /${y}0/; # True
1278 "aa\x08" =~ /${x}0/; # True!
1279 "aa\x08" =~ /${y}0/; # False
1280
1281Several special variables also refer back to portions of the previous
1282match. C<$+> returns whatever the last bracket match matched.
1283C<$&> returns the entire matched string. (At one point C<$0> did
1284also, but now it returns the name of the program.) C<$`> returns
1285everything before the matched string. C<$'> returns everything
1286after the matched string. And C<$^N> contains whatever was matched by
1287the most-recently closed group (submatch). C<$^N> can be used in
1288extended patterns (see below), for example to assign a submatch to a
1289variable.
1290X<$+> X<$^N> X<$&> X<$`> X<$'>
1291
1292These special variables, like the C<%+> hash and the numbered match variables
1293(C<$1>, C<$2>, C<$3>, I<etc>.) are dynamically scoped
1294until the end of the enclosing block or until the next successful
1295match, whichever comes first. (See L<perlsyn/"Compound Statements">.)
1296X<$+> X<$^N> X<$&> X<$`> X<$'>
1297X<$1> X<$2> X<$3> X<$4> X<$5> X<$6> X<$7> X<$8> X<$9>
1298X<@{^CAPTURE}>
1299
1300The C<@{^CAPTURE}> array may be used to access ALL of the capture buffers
1301as an array without needing to know how many there are. For instance
1302
1303 $string=~/$pattern/ and @captured = @{^CAPTURE};
1304
1305will place a copy of each capture variable, C<$1>, C<$2> etc, into the
1306C<@captured> array.
1307
1308Be aware that when interpolating a subscript of the C<@{^CAPTURE}>
1309array you must use demarcated curly brace notation:
1310
1311 print "${^CAPTURE[0]}";
1312
1313See L<perldata/"Demarcated variable names using braces"> for more on
1314this notation.
1315
1316B<NOTE>: Failed matches in Perl do not reset the match variables,
1317which makes it easier to write code that tests for a series of more
1318specific cases and remembers the best match.
1319
1320B<WARNING>: If your code is to run on Perl 5.16 or earlier,
1321beware that once Perl sees that you need one of C<$&>, C<$`>, or
1322C<$'> anywhere in the program, it has to provide them for every
1323pattern match. This may substantially slow your program.
1324
1325Perl uses the same mechanism to produce C<$1>, C<$2>, I<etc>, so you also
1326pay a price for each pattern that contains capturing parentheses.
1327(To avoid this cost while retaining the grouping behaviour, use the
1328extended regular expression C<(?: ... )> instead.) But if you never
1329use C<$&>, C<$`> or C<$'>, then patterns I<without> capturing
1330parentheses will not be penalized. So avoid C<$&>, C<$'>, and C<$`>
1331if you can, but if you can't (and some algorithms really appreciate
1332them), once you've used them once, use them at will, because you've
1333already paid the price.
1334X<$&> X<$`> X<$'>
1335
1336Perl 5.16 introduced a slightly more efficient mechanism that notes
1337separately whether each of C<$`>, C<$&>, and C<$'> have been seen, and
1338thus may only need to copy part of the string. Perl 5.20 introduced a
1339much more efficient copy-on-write mechanism which eliminates any slowdown.
1340
1341As another workaround for this problem, Perl 5.10.0 introduced C<${^PREMATCH}>,
1342C<${^MATCH}> and C<${^POSTMATCH}>, which are equivalent to C<$`>, C<$&>
1343and C<$'>, B<except> that they are only guaranteed to be defined after a
1344successful match that was executed with the C</p> (preserve) modifier.
1345The use of these variables incurs no global performance penalty, unlike
1346their punctuation character equivalents, however at the trade-off that you
1347have to tell perl when you want to use them.
1348X</p> X<p modifier>
1349
1350=head2 Quoting metacharacters
1351
1352Backslashed metacharacters in Perl are alphanumeric, such as C<\b>,
1353C<\w>, C<\n>. Unlike some other regular expression languages, there
1354are no backslashed symbols that aren't alphanumeric. So anything
1355that looks like C<\\>, C<\(>, C<\)>, C<\[>, C<\]>, C<\{>, or C<\}> is
1356always
1357interpreted as a literal character, not a metacharacter. This was
1358once used in a common idiom to disable or quote the special meanings
1359of regular expression metacharacters in a string that you want to
1360use for a pattern. Simply quote all non-"word" characters:
1361
1362 $pattern =~ s/(\W)/\\$1/g;
1363
1364(If C<use locale> is set, then this depends on the current locale.)
1365Today it is more common to use the C<L<quotemeta()|perlfunc/quotemeta>>
1366function or the C<\Q> metaquoting escape sequence to disable all
1367metacharacters' special meanings like this:
1368
1369 /$unquoted\Q$quoted\E$unquoted/
1370
1371Beware that if you put literal backslashes (those not inside
1372interpolated variables) between C<\Q> and C<\E>, double-quotish
1373backslash interpolation may lead to confusing results. If you
1374I<need> to use literal backslashes within C<\Q...\E>,
1375consult L<perlop/"Gory details of parsing quoted constructs">.
1376
1377C<quotemeta()> and C<\Q> are fully described in L<perlfunc/quotemeta>.
1378
1379=head2 Extended Patterns
1380
1381Perl also defines a consistent extension syntax for features not
1382found in standard tools like B<awk> and
1383B<lex>. The syntax for most of these is a
1384pair of parentheses with a question mark as the first thing within
1385the parentheses. The character after the question mark indicates
1386the extension.
1387
1388A question mark was chosen for this and for the minimal-matching
1389construct because 1) question marks are rare in older regular
1390expressions, and 2) whenever you see one, you should stop and
1391"question" exactly what is going on. That's psychology....
1392
1393=over 4
1394
1395=item C<(?#I<text>)>
1396X<(?#)>
1397
1398A comment. The I<text> is ignored.
1399Note that Perl closes
1400the comment as soon as it sees a C<")">, so there is no way to put a literal
1401C<")"> in the comment. The pattern's closing delimiter must be escaped by
1402a backslash if it appears in the comment.
1403
1404See L</E<sol>x> for another way to have comments in patterns.
1405
1406Note that a comment can go just about anywhere, except in the middle of
1407an escape sequence. Examples:
1408
1409 qr/foo(?#comment)bar/' # Matches 'foobar'
1410
1411 # The pattern below matches 'abcd', 'abccd', or 'abcccd'
1412 qr/abc(?#comment between literal and its quantifier){1,3}d/
1413
1414 # The pattern below generates a syntax error, because the '\p' must
1415 # be followed immediately by a '{'.
1416 qr/\p(?#comment between \p and its property name){Any}/
1417
1418 # The pattern below generates a syntax error, because the initial
1419 # '\(' is a literal opening parenthesis, and so there is nothing
1420 # for the closing ')' to match
1421 qr/\(?#the backslash means this isn't a comment)p{Any}/
1422
1423 # Comments can be used to fold long patterns into multiple lines
1424 qr/First part of a long regex(?#
1425 )remaining part/
1426
1427=item C<(?adlupimnsx-imnsx)>
1428
1429=item C<(?^alupimnsx)>
1430X<(?)> X<(?^)>
1431
1432Zero or more embedded pattern-match modifiers, to be turned on (or
1433turned off if preceded by C<"-">) for the remainder of the pattern or
1434the remainder of the enclosing pattern group (if any).
1435
1436This is particularly useful for dynamically-generated patterns,
1437such as those read in from a
1438configuration file, taken from an argument, or specified in a table
1439somewhere. Consider the case where some patterns want to be
1440case-sensitive and some do not: The case-insensitive ones merely need to
1441include C<(?i)> at the front of the pattern. For example:
1442
1443 $pattern = "foobar";
1444 if ( /$pattern/i ) { }
1445
1446 # more flexible:
1447
1448 $pattern = "(?i)foobar";
1449 if ( /$pattern/ ) { }
1450
1451These modifiers are restored at the end of the enclosing group. For example,
1452
1453 ( (?i) blah ) \s+ \g1
1454
1455will match C<blah> in any case, some spaces, and an exact (I<including the case>!)
1456repetition of the previous word, assuming the C</x> modifier, and no C</i>
1457modifier outside this group.
1458
1459These modifiers do not carry over into named subpatterns called in the
1460enclosing group. In other words, a pattern such as C<((?i)(?&I<NAME>))> does not
1461change the case-sensitivity of the I<NAME> pattern.
1462
1463A modifier is overridden by later occurrences of this construct in the
1464same scope containing the same modifier, so that
1465
1466 /((?im)foo(?-m)bar)/
1467
1468matches all of C<foobar> case insensitively, but uses C</m> rules for
1469only the C<foo> portion. The C<"a"> flag overrides C<aa> as well;
1470likewise C<aa> overrides C<"a">. The same goes for C<"x"> and C<xx>.
1471Hence, in
1472
1473 /(?-x)foo/xx
1474
1475both C</x> and C</xx> are turned off during matching C<foo>. And in
1476
1477 /(?x)foo/x
1478
1479C</x> but NOT C</xx> is turned on for matching C<foo>. (One might
1480mistakenly think that since the inner C<(?x)> is already in the scope of
1481C</x>, that the result would effectively be the sum of them, yielding
1482C</xx>. It doesn't work that way.) Similarly, doing something like
1483C<(?xx-x)foo> turns off all C<"x"> behavior for matching C<foo>, it is not
1484that you subtract 1 C<"x"> from 2 to get 1 C<"x"> remaining.
1485
1486Any of these modifiers can be set to apply globally to all regular
1487expressions compiled within the scope of a C<use re>. See
1488L<re/"'/flags' mode">.
1489
1490Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
1491after the C<"?"> is a shorthand equivalent to C<d-imnsx>. Flags (except
1492C<"d">) may follow the caret to override it.
1493But a minus sign is not legal with it.
1494
1495Note that the C<"a">, C<"d">, C<"l">, C<"p">, and C<"u"> modifiers are special in
1496that they can only be enabled, not disabled, and the C<"a">, C<"d">, C<"l">, and
1497C<"u"> modifiers are mutually exclusive: specifying one de-specifies the
1498others, and a maximum of one (or two C<"a">'s) may appear in the
1499construct. Thus, for
1500example, C<(?-p)> will warn when compiled under C<use warnings>;
1501C<(?-d:...)> and C<(?dl:...)> are fatal errors.
1502
1503Note also that the C<"p"> modifier is special in that its presence
1504anywhere in a pattern has a global effect.
1505
1506Having zero modifiers makes this a no-op (so why did you specify it,
1507unless it's generated code), and starting in v5.30, warns under L<C<use
1508re 'strict'>|re/'strict' mode>.
1509
1510=item C<(?:I<pattern>)>
1511X<(?:)>
1512
1513=item C<(?adluimnsx-imnsx:I<pattern>)>
1514
1515=item C<(?^aluimnsx:I<pattern>)>
1516X<(?^:)>
1517
1518This is for clustering, not capturing; it groups subexpressions like
1519C<"()">, but doesn't make backreferences as C<"()"> does. So
1520
1521 @fields = split(/\b(?:a|b|c)\b/)
1522
1523matches the same field delimiters as
1524
1525 @fields = split(/\b(a|b|c)\b/)
1526
1527but doesn't spit out the delimiters themselves as extra fields (even though
1528that's the behaviour of L<perlfunc/split> when its pattern contains capturing
1529groups). It's also cheaper not to capture
1530characters if you don't need to.
1531
1532Any letters between C<"?"> and C<":"> act as flags modifiers as with
1533C<(?adluimnsx-imnsx)>. For example,
1534
1535 /(?s-i:more.*than).*million/i
1536
1537is equivalent to the more verbose
1538
1539 /(?:(?s-i)more.*than).*million/i
1540
1541Note that any C<()> constructs enclosed within this one will still
1542capture unless the C</n> modifier is in effect.
1543
1544Like the L</(?adlupimnsx-imnsx)> construct, C<aa> and C<"a"> override each
1545other, as do C<xx> and C<"x">. They are not additive. So, doing
1546something like C<(?xx-x:foo)> turns off all C<"x"> behavior for matching
1547C<foo>.
1548
1549Starting in Perl 5.14, a C<"^"> (caret or circumflex accent) immediately
1550after the C<"?"> is a shorthand equivalent to C<d-imnsx>. Any positive
1551flags (except C<"d">) may follow the caret, so
1552
1553 (?^x:foo)
1554
1555is equivalent to
1556
1557 (?x-imns:foo)
1558
1559The caret tells Perl that this cluster doesn't inherit the flags of any
1560surrounding pattern, but uses the system defaults (C<d-imnsx>),
1561modified by any flags specified.
1562
1563The caret allows for simpler stringification of compiled regular
1564expressions. These look like
1565
1566 (?^:pattern)
1567
1568with any non-default flags appearing between the caret and the colon.
1569A test that looks at such stringification thus doesn't need to have the
1570system default flags hard-coded in it, just the caret. If new flags are
1571added to Perl, the meaning of the caret's expansion will change to include
1572the default for those flags, so the test will still work, unchanged.
1573
1574Specifying a negative flag after the caret is an error, as the flag is
1575redundant.
1576
1577Mnemonic for C<(?^...)>: A fresh beginning since the usual use of a caret is
1578to match at the beginning.
1579
1580=item C<(?|I<pattern>)>
1581X<(?|)> X<Branch reset>
1582
1583This is the "branch reset" pattern, which has the special property
1584that the capture groups are numbered from the same starting point
1585in each alternation branch. It is available starting from perl 5.10.0.
1586
1587Capture groups are numbered from left to right, but inside this
1588construct the numbering is restarted for each branch.
1589
1590The numbering within each branch will be as normal, and any groups
1591following this construct will be numbered as though the construct
1592contained only one branch, that being the one with the most capture
1593groups in it.
1594
1595This construct is useful when you want to capture one of a
1596number of alternative matches.
1597
1598Consider the following pattern. The numbers underneath show in
1599which group the captured content will be stored.
1600
1601
1602 # before ---------------branch-reset----------- after
1603 / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
1604 # 1 2 2 3 2 3 4
1605
1606Be careful when using the branch reset pattern in combination with
1607named captures. Named captures are implemented as being aliases to
1608numbered groups holding the captures, and that interferes with the
1609implementation of the branch reset pattern. If you are using named
1610captures in a branch reset pattern, it's best to use the same names,
1611in the same order, in each of the alternations:
1612
1613 /(?| (?<a> x ) (?<b> y )
1614 | (?<a> z ) (?<b> w )) /x
1615
1616Not doing so may lead to surprises:
1617
1618 "12" =~ /(?| (?<a> \d+ ) | (?<b> \D+))/x;
1619 say $+{a}; # Prints '12'
1620 say $+{b}; # *Also* prints '12'.
1621
1622The problem here is that both the group named C<< a >> and the group
1623named C<< b >> are aliases for the group belonging to C<< $1 >>.
1624
1625=item Lookaround Assertions
1626X<look-around assertion> X<lookaround assertion> X<look-around> X<lookaround>
1627
1628Lookaround assertions are zero-width patterns which match a specific
1629pattern without including it in C<$&>. Positive assertions match when
1630their subpattern matches, negative assertions match when their subpattern
1631fails. Lookbehind matches text up to the current match position,
1632lookahead matches text following the current match position.
1633
1634=over 4
1635
1636=item C<(?=I<pattern>)>
1637
1638=item C<(*pla:I<pattern>)>
1639
1640=item C<(*positive_lookahead:I<pattern>)>
1641X<(?=)>
1642X<(*pla>
1643X<(*positive_lookahead>
1644X<look-ahead, positive> X<lookahead, positive>
1645
1646A zero-width positive lookahead assertion. For example, C</\w+(?=\t)/>
1647matches a word followed by a tab, without including the tab in C<$&>.
1648
1649=item C<(?!I<pattern>)>
1650
1651=item C<(*nla:I<pattern>)>
1652
1653=item C<(*negative_lookahead:I<pattern>)>
1654X<(?!)>
1655X<(*nla>
1656X<(*negative_lookahead>
1657X<look-ahead, negative> X<lookahead, negative>
1658
1659A zero-width negative lookahead assertion. For example C</foo(?!bar)/>
1660matches any occurrence of "foo" that isn't followed by "bar". Note
1661however that lookahead and lookbehind are NOT the same thing. You cannot
1662use this for lookbehind.
1663
1664If you are looking for a "bar" that isn't preceded by a "foo", C</(?!foo)bar/>
1665will not do what you want. That's because the C<(?!foo)> is just saying that
1666the next thing cannot be "foo"--and it's not, it's a "bar", so "foobar" will
1667match. Use lookbehind instead (see below).
1668
1669=item C<(?<=I<pattern>)>
1670
1671=item C<\K>
1672
1673=item C<(*plb:I<pattern>)>
1674
1675=item C<(*positive_lookbehind:I<pattern>)>
1676X<(?<=)>
1677X<(*plb>
1678X<(*positive_lookbehind>
1679X<look-behind, positive> X<lookbehind, positive> X<\K>
1680
1681A zero-width positive lookbehind assertion. For example, C</(?<=\t)\w+/>
1682matches a word that follows a tab, without including the tab in C<$&>.
1683
1684Prior to Perl 5.30, it worked only for fixed-width lookbehind, but
1685starting in that release, it can handle variable lengths from 1 to 255
1686characters as an experimental feature. The feature is enabled
1687automatically if you use a variable length positive lookbehind assertion.
1688
1689In Perl 5.35.10 the scope of the experimental nature of this construct
1690has been reduced, and experimental warnings will only be produced when
1691the construct contains capturing parenthesis. The warnings will be
1692raised at pattern compilation time, unless turned off, in the
1693C<experimental::vlb> category. This is to warn you that the exact
1694contents of capturing buffers in a variable length positive lookbehind
1695is not well defined and is subject to change in a future release of perl.
1696
1697Currently if you use capture buffers inside of a positive variable length
1698lookbehind the result will be the longest and thus leftmost match possible.
1699This means that
1700
1701 "aax" =~ /(?=x)(?<=(a|aa))/
1702 "aax" =~ /(?=x)(?<=(aa|a))/
1703 "aax" =~ /(?=x)(?<=(a{1,2}?)/
1704 "aax" =~ /(?=x)(?<=(a{1,2})/
1705
1706will all result in C<$1> containing C<"aa">. It is possible in a future
1707release of perl we will change this behavior.
1708
1709There is a special form of this construct, called C<\K>
1710(available since Perl 5.10.0), which causes the
1711regex engine to "keep" everything it had matched prior to the C<\K> and
1712not include it in C<$&>. This effectively provides non-experimental
1713variable-length lookbehind of any length.
1714
1715And, there is a technique that can be used to handle variable length
1716lookbehinds on earlier releases, and longer than 255 characters. It is
1717described in
1718L<http://www.drregex.com/2019/02/variable-length-lookbehinds-actually.html>.
1719
1720Note that under C</i>, a few single characters match two or three other
1721characters. This makes them variable length, and the 255 length applies
1722to the maximum number of characters in the match. For
1723example C<qr/\N{LATIN SMALL LETTER SHARP S}/i> matches the sequence
1724C<"ss">. Your lookbehind assertion could contain 127 Sharp S
1725characters under C</i>, but adding a 128th would generate a compilation
1726error, as that could match 256 C<"s"> characters in a row.
1727
1728The use of C<\K> inside of another lookaround assertion
1729is allowed, but the behaviour is currently not well defined.
1730
1731For various reasons C<\K> may be significantly more efficient than the
1732equivalent C<< (?<=...) >> construct, and it is especially useful in
1733situations where you want to efficiently remove something following
1734something else in a string. For instance
1735
1736 s/(foo)bar/$1/g;
1737
1738can be rewritten as the much more efficient
1739
1740 s/foo\Kbar//g;
1741
1742Use of the non-greedy modifier C<"?"> may not give you the expected
1743results if it is within a capturing group within the construct.
1744
1745=item C<(?<!I<pattern>)>
1746
1747=item C<(*nlb:I<pattern>)>
1748
1749=item C<(*negative_lookbehind:I<pattern>)>
1750X<(?<!)>
1751X<(*nlb>
1752X<(*negative_lookbehind>
1753X<look-behind, negative> X<lookbehind, negative>
1754
1755A zero-width negative lookbehind assertion. For example C</(?<!bar)foo/>
1756matches any occurrence of "foo" that does not follow "bar".
1757
1758Prior to Perl 5.30, it worked only for fixed-width lookbehind, but
1759starting in that release, it can handle variable lengths from 1 to 255
1760characters as an experimental feature. The feature is enabled
1761automatically if you use a variable length negative lookbehind assertion.
1762
1763In Perl 5.35.10 the scope of the experimental nature of this construct
1764has been reduced, and experimental warnings will only be produced when
1765the construct contains capturing parentheses. The warnings will be
1766raised at pattern compilation time, unless turned off, in the
1767C<experimental::vlb> category. This is to warn you that the exact
1768contents of capturing buffers in a variable length negative lookbehind
1769is not well defined and is subject to change in a future release of perl.
1770
1771Currently if you use capture buffers inside of a negative variable length
1772lookbehind the result may not be what you expect, for instance:
1773
1774 say "axfoo"=~/(?=foo)(?<!(a|ax)(?{ say $1 }))/ ? "y" : "n";
1775
1776will output the following:
1777
1778 a
1779 no
1780
1781which does not make sense as this should print out "ax" as the "a" does
1782not line up at the correct place. Another example would be:
1783
1784 say "yes: '$1-$2'" if "aayfoo"=~/(?=foo)(?<!(a|aa)(a|aa)x)/;
1785
1786will output the following:
1787
1788 yes: 'aa-a'
1789
1790It is possible in a future release of perl we will change this behavior
1791so both of these examples produced more reasonable output.
1792
1793Note that we are confident that the construct will match and reject
1794patterns appropriately, the undefined behavior strictly relates to the
1795value of the capture buffer during or after matching.
1796
1797There is a technique that can be used to handle variable length
1798lookbehind on earlier releases, and longer than 255 characters. It is
1799described in
1800L<http://www.drregex.com/2019/02/variable-length-lookbehinds-actually.html>.
1801
1802Note that under C</i>, a few single characters match two or three other
1803characters. This makes them variable length, and the 255 length applies
1804to the maximum number of characters in the match. For
1805example C<qr/\N{LATIN SMALL LETTER SHARP S}/i> matches the sequence
1806C<"ss">. Your lookbehind assertion could contain 127 Sharp S
1807characters under C</i>, but adding a 128th would generate a compilation
1808error, as that could match 256 C<"s"> characters in a row.
1809
1810Use of the non-greedy modifier C<"?"> may not give you the expected
1811results if it is within a capturing group within the construct.
1812
1813=back
1814
1815=item C<< (?<I<NAME>>I<pattern>) >>
1816
1817=item C<(?'I<NAME>'I<pattern>)>
1818X<< (?<NAME>) >> X<(?'NAME')> X<named capture> X<capture>
1819
1820A named capture group. Identical in every respect to normal capturing
1821parentheses C<()> but for the additional fact that the group
1822can be referred to by name in various regular expression
1823constructs (like C<\g{I<NAME>}>) and can be accessed by name
1824after a successful match via C<%+> or C<%->. See L<perlvar>
1825for more details on the C<%+> and C<%-> hashes.
1826
1827If multiple distinct capture groups have the same name, then
1828C<$+{I<NAME>}> will refer to the leftmost defined group in the match.
1829
1830The forms C<(?'I<NAME>'I<pattern>)> and C<< (?<I<NAME>>I<pattern>) >>
1831are equivalent.
1832
1833B<NOTE:> While the notation of this construct is the same as the similar
1834function in .NET regexes, the behavior is not. In Perl the groups are
1835numbered sequentially regardless of being named or not. Thus in the
1836pattern
1837
1838 /(x)(?<foo>y)(z)/
1839
1840C<$+{foo}> will be the same as C<$2>, and C<$3> will contain 'z' instead of
1841the opposite which is what a .NET regex hacker might expect.
1842
1843Currently I<NAME> is restricted to simple identifiers only.
1844In other words, it must match C</^[_A-Za-z][_A-Za-z0-9]*\z/> or
1845its Unicode extension (see L<utf8>),
1846though it isn't extended by the locale (see L<perllocale>).
1847
1848B<NOTE:> In order to make things easier for programmers with experience
1849with the Python or PCRE regex engines, the pattern C<<
1850(?PE<lt>I<NAME>E<gt>I<pattern>) >>
1851may be used instead of C<< (?<I<NAME>>I<pattern>) >>; however this form does not
1852support the use of single quotes as a delimiter for the name.
1853
1854=item C<< \k<I<NAME>> >>
1855
1856=item C<< \k'I<NAME>' >>
1857
1858=item C<< \k{I<NAME>} >>
1859
1860Named backreference. Similar to numeric backreferences, except that
1861the group is designated by name and not number. If multiple groups
1862have the same name then it refers to the leftmost defined group in
1863the current match.
1864
1865It is an error to refer to a name not defined by a C<< (?<I<NAME>>) >>
1866earlier in the pattern.
1867
1868All three forms are equivalent, although with C<< \k{ I<NAME> } >>,
1869you may optionally have blanks within but adjacent to the braces, as
1870shown.
1871
1872B<NOTE:> In order to make things easier for programmers with experience
1873with the Python or PCRE regex engines, the pattern C<< (?P=I<NAME>) >>
1874may be used instead of C<< \k<I<NAME>> >>.
1875
1876=item C<(?{ I<code> })>
1877X<(?{})> X<regex, code in> X<regexp, code in> X<regular expression, code in>
1878
1879B<WARNING>: Using this feature safely requires that you understand its
1880limitations. Code executed that has side effects may not perform identically
1881from version to version due to the effect of future optimisations in the regex
1882engine. For more information on this, see L</Embedded Code Execution
1883Frequency>.
1884
1885This zero-width assertion executes any embedded Perl code. It always
1886succeeds, and its return value is set as C<$^R>.
1887
1888In literal patterns, the code is parsed at the same time as the
1889surrounding code. While within the pattern, control is passed temporarily
1890back to the perl parser, until the logically-balancing closing brace is
1891encountered. This is similar to the way that an array index expression in
1892a literal string is handled, for example
1893
1894 "abc$array[ 1 + f('[') + g()]def"
1895
1896In particular, braces do not need to be balanced:
1897
1898 s/abc(?{ f('{'); })/def/
1899
1900Even in a pattern that is interpolated and compiled at run-time, literal
1901code blocks will be compiled once, at perl compile time; the following
1902prints "ABCD":
1903
1904 print "D";
1905 my $qr = qr/(?{ BEGIN { print "A" } })/;
1906 my $foo = "foo";
1907 /$foo$qr(?{ BEGIN { print "B" } })/;
1908 BEGIN { print "C" }
1909
1910In patterns where the text of the code is derived from run-time
1911information rather than appearing literally in a source code /pattern/,
1912the code is compiled at the same time that the pattern is compiled, and
1913for reasons of security, C<use re 'eval'> must be in scope. This is to
1914stop user-supplied patterns containing code snippets from being
1915executable.
1916
1917In situations where you need to enable this with C<use re 'eval'>, you should
1918also have taint checking enabled, if your perl supports it.
1919Better yet, use the carefully constrained evaluation within a Safe compartment.
1920See L<perlsec> for details about both these mechanisms.
1921
1922From the viewpoint of parsing, lexical variable scope and closures,
1923
1924 /AAA(?{ BBB })CCC/
1925
1926behaves approximately like
1927
1928 /AAA/ && do { BBB } && /CCC/
1929
1930Similarly,
1931
1932 qr/AAA(?{ BBB })CCC/
1933
1934behaves approximately like
1935
1936 sub { /AAA/ && do { BBB } && /CCC/ }
1937
1938In particular:
1939
1940 { my $i = 1; $r = qr/(?{ print $i })/ }
1941 my $i = 2;
1942 /$r/; # prints "1"
1943
1944Inside a C<(?{...})> block, C<$_> refers to the string the regular
1945expression is matching against. You can also use C<pos()> to know what is
1946the current position of matching within this string.
1947
1948The code block introduces a new scope from the perspective of lexical
1949variable declarations, but B<not> from the perspective of C<local> and
1950similar localizing behaviours. So later code blocks within the same
1951pattern will still see the values which were localized in earlier blocks.
1952These accumulated localizations are undone either at the end of a
1953successful match, or if the assertion is backtracked (compare
1954L</"Backtracking">). For example,
1955
1956 $_ = 'a' x 8;
1957 m<
1958 (?{ $cnt = 0 }) # Initialize $cnt.
1959 (
1960 a
1961 (?{
1962 local $cnt = $cnt + 1; # Update $cnt,
1963 # backtracking-safe.
1964 })
1965 )*
1966 aaaa
1967 (?{ $res = $cnt }) # On success copy to
1968 # non-localized location.
1969 >x;
1970
1971will initially increment C<$cnt> up to 8; then during backtracking, its
1972value will be unwound back to 4, which is the value assigned to C<$res>.
1973At the end of the regex execution, C<$cnt> will be wound back to its initial
1974value of 0.
1975
1976This assertion may be used as the condition in a
1977
1978 (?(condition)yes-pattern|no-pattern)
1979
1980switch. If I<not> used in this way, the result of evaluation of I<code>
1981is put into the special variable C<$^R>. This happens immediately, so
1982C<$^R> can be used from other C<(?{ I<code> })> assertions inside the same
1983regular expression.
1984
1985The assignment to C<$^R> above is properly localized, so the old
1986value of C<$^R> is restored if the assertion is backtracked; compare
1987L</"Backtracking">.
1988
1989Note that the special variable C<$^N> is particularly useful with code
1990blocks to capture the results of submatches in variables without having to
1991keep track of the number of nested parentheses. For example:
1992
1993 $_ = "The brown fox jumps over the lazy dog";
1994 /the (\S+)(?{ $color = $^N }) (\S+)(?{ $animal = $^N })/i;
1995 print "color = $color, animal = $animal\n";
1996
1997The use of this construct disables some optimisations globally in the
1998pattern, and the pattern may execute much slower as a consequence.
1999Use a C<*> instead of the C<?> block to create an optimistic form of
2000this construct. C<(*{ ... })> should not disable any optimisations.
2001
2002=item C<(*{ I<code> })>
2003X<(*{})> X<regex, optimistic code>
2004
2005This is *exactly* the same as C<(?{ I<code> })> with the exception
2006that it does not disable B<any> optimisations at all in the regex engine.
2007How often it is executed may vary from perl release to perl release.
2008In a failing match it may not even be executed at all.
2009
2010=item C<(??{ I<code> })>
2011X<(??{})>
2012X<regex, postponed> X<regexp, postponed> X<regular expression, postponed>
2013
2014B<WARNING>: Using this feature safely requires that you understand its
2015limitations. Code executed that has side effects may not perform
2016identically from version to version due to the effect of future
2017optimisations in the regex engine. For more information on this, see
2018L</Embedded Code Execution Frequency>.
2019
2020This is a "postponed" regular subexpression. It behaves in I<exactly> the
2021same way as a C<(?{ I<code> })> code block as described above, except that
2022its return value, rather than being assigned to C<$^R>, is treated as a
2023pattern, compiled if it's a string (or used as-is if its a qr// object),
2024then matched as if it were inserted instead of this construct.
2025
2026During the matching of this sub-pattern, it has its own set of
2027captures which are valid during the sub-match, but are discarded once
2028control returns to the main pattern. For example, the following matches,
2029with the inner pattern capturing "B" and matching "BB", while the outer
2030pattern captures "A";
2031
2032 my $inner = '(.)\1';
2033 "ABBA" =~ /^(.)(??{ $inner })\1/;
2034 print $1; # prints "A";
2035
2036Note that this means that there is no way for the inner pattern to refer
2037to a capture group defined outside. (The code block itself can use C<$1>,
2038I<etc>., to refer to the enclosing pattern's capture groups.) Thus, although
2039
2040 ('a' x 100)=~/(??{'(.)' x 100})/
2041
2042I<will> match, it will I<not> set C<$1> on exit.
2043
2044The following pattern matches a parenthesized group:
2045
2046 $re = qr{
2047 \(
2048 (?:
2049 (?> [^()]+ ) # Non-parens without backtracking
2050 |
2051 (??{ $re }) # Group with matching parens
2052 )*
2053 \)
2054 }x;
2055
2056See also
2057L<C<(?I<PARNO>)>|/(?I<PARNO>) (?-I<PARNO>) (?+I<PARNO>) (?R) (?0)>
2058for a different, more efficient way to accomplish
2059the same task.
2060
2061Executing a postponed regular expression too many times without
2062consuming any input string will also result in a fatal error. The depth
2063at which that happens is compiled into perl, so it can be changed with a
2064custom build.
2065
2066The use of this construct disables some optimisations globally in the pattern,
2067and the pattern may execute much slower as a consequence.
2068
2069=item C<(?I<PARNO>)> C<(?-I<PARNO>)> C<(?+I<PARNO>)> C<(?R)> C<(?0)>
2070X<(?PARNO)> X<(?1)> X<(?R)> X<(?0)> X<(?-1)> X<(?+1)> X<(?-PARNO)> X<(?+PARNO)>
2071X<regex, recursive> X<regexp, recursive> X<regular expression, recursive>
2072X<regex, relative recursion> X<GOSUB> X<GOSTART>
2073
2074Recursive subpattern. Treat the contents of a given capture buffer in the
2075current pattern as an independent subpattern and attempt to match it at
2076the current position in the string. Information about capture state from
2077the caller for things like backreferences is available to the subpattern,
2078but capture buffers set by the subpattern are not visible to the caller.
2079
2080Similar to C<(??{ I<code> })> except that it does not involve executing any
2081code or potentially compiling a returned pattern string; instead it treats
2082the part of the current pattern contained within a specified capture group
2083as an independent pattern that must match at the current position. Also
2084different is the treatment of capture buffers, unlike C<(??{ I<code> })>
2085recursive patterns have access to their caller's match state, so one can
2086use backreferences safely.
2087
2088I<PARNO> is a sequence of digits (not starting with 0) whose value reflects
2089the paren-number of the capture group to recurse to. C<(?R)> recurses to
2090the beginning of the whole pattern. C<(?0)> is an alternate syntax for
2091C<(?R)>. If I<PARNO> is preceded by a plus or minus sign then it is assumed
2092to be relative, with negative numbers indicating preceding capture groups
2093and positive ones following. Thus C<(?-1)> refers to the most recently
2094declared group, and C<(?+1)> indicates the next group to be declared.
2095Note that the counting for relative recursion differs from that of
2096relative backreferences, in that with recursion unclosed groups B<are>
2097included.
2098
2099The following pattern matches a function C<foo()> which may contain
2100balanced parentheses as the argument.
2101
2102 $re = qr{ ( # paren group 1 (full function)
2103 foo
2104 ( # paren group 2 (parens)
2105 \(
2106 ( # paren group 3 (contents of parens)
2107 (?:
2108 (?> [^()]+ ) # Non-parens without backtracking
2109 |
2110 (?2) # Recurse to start of paren group 2
2111 )*
2112 )
2113 \)
2114 )
2115 )
2116 }x;
2117
2118If the pattern was used as follows
2119
2120 'foo(bar(baz)+baz(bop))'=~/$re/
2121 and print "\$1 = $1\n",
2122 "\$2 = $2\n",
2123 "\$3 = $3\n";
2124
2125the output produced should be the following:
2126
2127 $1 = foo(bar(baz)+baz(bop))
2128 $2 = (bar(baz)+baz(bop))
2129 $3 = bar(baz)+baz(bop)
2130
2131If there is no corresponding capture group defined, then it is a
2132fatal error. Recursing deeply without consuming any input string will
2133also result in a fatal error. The depth at which that happens is
2134compiled into perl, so it can be changed with a custom build.
2135
2136The following shows how using negative indexing can make it
2137easier to embed recursive patterns inside of a C<qr//> construct
2138for later use:
2139
2140 my $parens = qr/(\((?:[^()]++|(?-1))*+\))/;
2141 if (/foo $parens \s+ \+ \s+ bar $parens/x) {
2142 # do something here...
2143 }
2144
2145B<Note> that this pattern does not behave the same way as the equivalent
2146PCRE or Python construct of the same form. In Perl you can backtrack into
2147a recursed group, in PCRE and Python the recursed into group is treated
2148as atomic. Also, modifiers are resolved at compile time, so constructs
2149like C<(?i:(?1))> or C<(?:(?i)(?1))> do not affect how the sub-pattern will
2150be processed.
2151
2152=item C<(?&I<NAME>)>
2153X<(?&NAME)>
2154
2155Recurse to a named subpattern. Identical to C<(?I<PARNO>)> except that the
2156parenthesis to recurse to is determined by name. If multiple parentheses have
2157the same name, then it recurses to the leftmost.
2158
2159It is an error to refer to a name that is not declared somewhere in the
2160pattern.
2161
2162B<NOTE:> In order to make things easier for programmers with experience
2163with the Python or PCRE regex engines the pattern C<< (?P>I<NAME>) >>
2164may be used instead of C<< (?&I<NAME>) >>.
2165
2166=item C<(?(I<condition>)I<yes-pattern>|I<no-pattern>)>
2167X<(?()>
2168
2169=item C<(?(I<condition>)I<yes-pattern>)>
2170
2171Conditional expression. Matches I<yes-pattern> if I<condition> yields
2172a true value, matches I<no-pattern> otherwise. A missing pattern always
2173matches.
2174
2175C<(I<condition>)> should be one of:
2176
2177=over 4
2178
2179=item an integer in parentheses
2180
2181(which is valid if the corresponding pair of parentheses
2182matched);
2183
2184=item a lookahead/lookbehind/evaluate zero-width assertion;
2185
2186=item a name in angle brackets or single quotes
2187
2188(which is valid if a group with the given name matched);
2189
2190=item the special symbol C<(R)>
2191
2192(true when evaluated inside of recursion or eval). Additionally the
2193C<"R"> may be
2194followed by a number, (which will be true when evaluated when recursing
2195inside of the appropriate group), or by C<&I<NAME>>, in which case it will
2196be true only when evaluated during recursion in the named group.
2197
2198=back
2199
2200Here's a summary of the possible predicates:
2201
2202=over 4
2203
2204=item C<(1)> C<(2)> ...
2205
2206Checks if the numbered capturing group has matched something.
2207Full syntax: C<< (?(1)then|else) >>
2208
2209=item C<(E<lt>I<NAME>E<gt>)> C<('I<NAME>')>
2210
2211Checks if a group with the given name has matched something.
2212Full syntax: C<< (?(<name>)then|else) >>
2213
2214=item C<(?=...)> C<(?!...)> C<(?<=...)> C<(?<!...)>
2215
2216Checks whether the pattern matches (or does not match, for the C<"!">
2217variants).
2218Full syntax: C<< (?(?=I<lookahead>)I<then>|I<else>) >>
2219
2220=item C<(?{ I<CODE> })>
2221
2222Treats the return value of the code block as the condition.
2223Full syntax: C<< (?(?{ I<CODE> })I<then>|I<else>) >>
2224
2225Note use of this construct may globally affect the performance
2226of the pattern. Consider using C<(*{ I<CODE> })>
2227
2228=item C<(*{ I<CODE> })>
2229
2230Treats the return value of the code block as the condition.
2231Full syntax: C<< (?(*{ I<CODE> })I<then>|I<else>) >>
2232
2233=item C<(R)>
2234
2235Checks if the expression has been evaluated inside of recursion.
2236Full syntax: C<< (?(R)I<then>|I<else>) >>
2237
2238=item C<(R1)> C<(R2)> ...
2239
2240Checks if the expression has been evaluated while executing directly
2241inside of the n-th capture group. This check is the regex equivalent of
2242
2243 if ((caller(0))[3] eq 'subname') { ... }
2244
2245In other words, it does not check the full recursion stack.
2246
2247Full syntax: C<< (?(R1)I<then>|I<else>) >>
2248
2249=item C<(R&I<NAME>)>
2250
2251Similar to C<(R1)>, this predicate checks to see if we're executing
2252directly inside of the leftmost group with a given name (this is the same
2253logic used by C<(?&I<NAME>)> to disambiguate). It does not check the full
2254stack, but only the name of the innermost active recursion.
2255Full syntax: C<< (?(R&I<name>)I<then>|I<else>) >>
2256
2257=item C<(DEFINE)>
2258
2259In this case, the yes-pattern is never directly executed, and no
2260no-pattern is allowed. Similar in spirit to C<(?{0})> but more efficient.
2261See below for details.
2262Full syntax: C<< (?(DEFINE)I<definitions>...) >>
2263
2264=back
2265
2266For example:
2267
2268 m{ ( \( )?
2269 [^()]+
2270 (?(1) \) )
2271 }x
2272
2273matches a chunk of non-parentheses, possibly included in parentheses
2274themselves.
2275
2276A special form is the C<(DEFINE)> predicate, which never executes its
2277yes-pattern directly, and does not allow a no-pattern. This allows one to
2278define subpatterns which will be executed only by the recursion mechanism.
2279This way, you can define a set of regular expression rules that can be
2280bundled into any pattern you choose.
2281
2282It is recommended that for this usage you put the DEFINE block at the
2283end of the pattern, and that you name any subpatterns defined within it.
2284
2285Also, it's worth noting that patterns defined this way probably will
2286not be as efficient, as the optimizer is not very clever about
2287handling them.
2288
2289An example of how this might be used is as follows:
2290
2291 /(?<NAME>(?&NAME_PAT))(?<ADDR>(?&ADDRESS_PAT))
2292 (?(DEFINE)
2293 (?<NAME_PAT>....)
2294 (?<ADDRESS_PAT>....)
2295 )/x
2296
2297Note that capture groups matched inside of recursion are not accessible
2298after the recursion returns, so the extra layer of capturing groups is
2299necessary. Thus C<$+{NAME_PAT}> would not be defined even though
2300C<$+{NAME}> would be.
2301
2302Finally, keep in mind that subpatterns created inside a DEFINE block
2303count towards the absolute and relative number of captures, so this:
2304
2305 my @captures = "a" =~ /(.) # First capture
2306 (?(DEFINE)
2307 (?<EXAMPLE> 1 ) # Second capture
2308 )/x;
2309 say scalar @captures;
2310
2311Will output 2, not 1. This is particularly important if you intend to
2312compile the definitions with the C<qr//> operator, and later
2313interpolate them in another pattern.
2314
2315=item C<< (?>I<pattern>) >>
2316
2317=item C<< (*atomic:I<pattern>) >>
2318X<(?E<gt>pattern)>
2319X<(*atomic>
2320X<backtrack> X<backtracking> X<atomic> X<possessive>
2321
2322An "independent" subexpression, one which matches the substring
2323that a standalone I<pattern> would match if anchored at the given
2324position, and it matches I<nothing other than this substring>. This
2325construct is useful for optimizations of what would otherwise be
2326"eternal" matches, because it will not backtrack (see L</"Backtracking">).
2327It may also be useful in places where the "grab all you can, and do not
2328give anything back" semantic is desirable.
2329
2330For example: C<< ^(?>a*)ab >> will never match, since C<< (?>a*) >>
2331(anchored at the beginning of string, as above) will match I<all>
2332characters C<"a"> at the beginning of string, leaving no C<"a"> for
2333C<ab> to match. In contrast, C<a*ab> will match the same as C<a+b>,
2334since the match of the subgroup C<a*> is influenced by the following
2335group C<ab> (see L</"Backtracking">). In particular, C<a*> inside
2336C<a*ab> will match fewer characters than a standalone C<a*>, since
2337this makes the tail match.
2338
2339C<< (?>I<pattern>) >> does not disable backtracking altogether once it has
2340matched. It is still possible to backtrack past the construct, but not
2341into it. So C<< ((?>a*)|(?>b*))ar >> will still match "bar".
2342
2343An effect similar to C<< (?>I<pattern>) >> may be achieved by writing
2344C<(?=(I<pattern>))\g{-1}>. This matches the same substring as a standalone
2345C<a+>, and the following C<\g{-1}> eats the matched string; it therefore
2346makes a zero-length assertion into an analogue of C<< (?>...) >>.
2347(The difference between these two constructs is that the second one
2348uses a capturing group, thus shifting ordinals of backreferences
2349in the rest of a regular expression.)
2350
2351Consider this pattern:
2352
2353 m{ \(
2354 (
2355 [^()]+ # x+
2356 |
2357 \( [^()]* \)
2358 )+
2359 \)
2360 }x
2361
2362That will efficiently match a nonempty group with matching parentheses
2363two levels deep or less. However, if there is no such group, it
2364will take virtually forever on a long string. That's because there
2365are so many different ways to split a long string into several
2366substrings. This is what C<(.+)+> is doing, and C<(.+)+> is similar
2367to a subpattern of the above pattern. Consider how the pattern
2368above detects no-match on C<((()aaaaaaaaaaaaaaaaaa> in several
2369seconds, but that each extra letter doubles this time. This
2370exponential performance will make it appear that your program has
2371hung. However, a tiny change to this pattern
2372
2373 m{ \(
2374 (
2375 (?> [^()]+ ) # change x+ above to (?> x+ )
2376 |
2377 \( [^()]* \)
2378 )+
2379 \)
2380 }x
2381
2382which uses C<< (?>...) >> matches exactly when the one above does (verifying
2383this yourself would be a productive exercise), but finishes in a fourth
2384the time when used on a similar string with 1000000 C<"a">s. Be aware,
2385however, that, when this construct is followed by a
2386quantifier, it currently triggers a warning message under
2387the C<use warnings> pragma or B<-w> switch saying it
2388C<"matches null string many times in regex">.
2389
2390On simple groups, such as the pattern C<< (?> [^()]+ ) >>, a comparable
2391effect may be achieved by negative lookahead, as in C<[^()]+ (?! [^()] )>.
2392This was only 4 times slower on a string with 1000000 C<"a">s.
2393
2394The "grab all you can, and do not give anything back" semantic is desirable
2395in many situations where on the first sight a simple C<()*> looks like
2396the correct solution. Suppose we parse text with comments being delimited
2397by C<"#"> followed by some optional (horizontal) whitespace. Contrary to
2398its appearance, C<#[ \t]*> I<is not> the correct subexpression to match
2399the comment delimiter, because it may "give up" some whitespace if
2400the remainder of the pattern can be made to match that way. The correct
2401answer is either one of these:
2402
2403 (?>#[ \t]*)
2404 #[ \t]*(?![ \t])
2405
2406For example, to grab non-empty comments into C<$1>, one should use either
2407one of these:
2408
2409 / (?> \# [ \t]* ) ( .+ ) /x;
2410 / \# [ \t]* ( [^ \t] .* ) /x;
2411
2412Which one you pick depends on which of these expressions better reflects
2413the above specification of comments.
2414
2415In some literature this construct is called "atomic matching" or
2416"possessive matching".
2417
2418Possessive quantifiers are equivalent to putting the item they are applied
2419to inside of one of these constructs. The following equivalences apply:
2420
2421 Quantifier Form Bracketing Form
2422 --------------- ---------------
2423 PAT*+ (?>PAT*)
2424 PAT++ (?>PAT+)
2425 PAT?+ (?>PAT?)
2426 PAT{min,max}+ (?>PAT{min,max})
2427
2428Nested C<(?E<gt>...)> constructs are not no-ops, even if at first glance
2429they might seem to be. This is because the nested C<(?E<gt>...)> can
2430restrict internal backtracking that otherwise might occur. For example,
2431
2432 "abc" =~ /(?>a[bc]*c)/
2433
2434matches, but
2435
2436 "abc" =~ /(?>a(?>[bc]*)c)/
2437
2438does not.
2439
2440=item C<(?[ ])>
2441
2442See L<perlrecharclass/Extended Bracketed Character Classes>.
2443
2444=back
2445
2446=head2 Backtracking
2447X<backtrack> X<backtracking>
2448
2449NOTE: This section presents an abstract approximation of regular
2450expression behavior. For a more rigorous (and complicated) view of
2451the rules involved in selecting a match among possible alternatives,
2452see L</Combining RE Pieces>.
2453
2454A fundamental feature of regular expression matching involves the
2455notion called I<backtracking>, which is currently used (when needed)
2456by all regular non-possessive expression quantifiers, namely C<"*">,
2457C<*?>, C<"+">, C<+?>, C<{n,m}>, and C<{n,m}?>. Backtracking is often
2458optimized internally, but the general principle outlined here is valid.
2459
2460For a regular expression to match, the I<entire> regular expression must
2461match, not just part of it. So if the beginning of a pattern containing a
2462quantifier succeeds in a way that causes later parts in the pattern to
2463fail, the matching engine backs up and recalculates the beginning
2464part--that's why it's called backtracking.
2465
2466Here is an example of backtracking: Let's say you want to find the
2467word following "foo" in the string "Food is on the foo table.":
2468
2469 $_ = "Food is on the foo table.";
2470 if ( /\b(foo)\s+(\w+)/i ) {
2471 print "$2 follows $1.\n";
2472 }
2473
2474When the match runs, the first part of the regular expression (C<\b(foo)>)
2475finds a possible match right at the beginning of the string, and loads up
2476C<$1> with "Foo". However, as soon as the matching engine sees that there's
2477no whitespace following the "Foo" that it had saved in C<$1>, it realizes its
2478mistake and starts over again one character after where it had the
2479tentative match. This time it goes all the way until the next occurrence
2480of "foo". The complete regular expression matches this time, and you get
2481the expected output of "table follows foo."
2482
2483Sometimes minimal matching can help a lot. Imagine you'd like to match
2484everything between "foo" and "bar". Initially, you write something
2485like this:
2486
2487 $_ = "The food is under the bar in the barn.";
2488 if ( /foo(.*)bar/ ) {
2489 print "got <$1>\n";
2490 }
2491
2492Which perhaps unexpectedly yields:
2493
2494 got <d is under the bar in the >
2495
2496That's because C<.*> was greedy, so you get everything between the
2497I<first> "foo" and the I<last> "bar". Here it's more effective
2498to use minimal matching to make sure you get the text between a "foo"
2499and the first "bar" thereafter.
2500
2501 if ( /foo(.*?)bar/ ) { print "got <$1>\n" }
2502 got <d is under the >
2503
2504Here's another example. Let's say you'd like to match a number at the end
2505of a string, and you also want to keep the preceding part of the match.
2506So you write this:
2507
2508 $_ = "I have 2 numbers: 53147";
2509 if ( /(.*)(\d*)/ ) { # Wrong!
2510 print "Beginning is <$1>, number is <$2>.\n";
2511 }
2512
2513That won't work at all, because C<.*> was greedy and gobbled up the
2514whole string. As C<\d*> can match on an empty string the complete
2515regular expression matched successfully.
2516
2517 Beginning is <I have 2 numbers: 53147>, number is <>.
2518
2519Here are some variants, most of which don't work:
2520
2521 $_ = "I have 2 numbers: 53147";
2522 @pats = qw{
2523 (.*)(\d*)
2524 (.*)(\d+)
2525 (.*?)(\d*)
2526 (.*?)(\d+)
2527 (.*)(\d+)$
2528 (.*?)(\d+)$
2529 (.*)\b(\d+)$
2530 (.*\D)(\d+)$
2531 };
2532
2533 for $pat (@pats) {
2534 printf "%-12s ", $pat;
2535 if ( /$pat/ ) {
2536 print "<$1> <$2>\n";
2537 } else {
2538 print "FAIL\n";
2539 }
2540 }
2541
2542That will print out:
2543
2544 (.*)(\d*) <I have 2 numbers: 53147> <>
2545 (.*)(\d+) <I have 2 numbers: 5314> <7>
2546 (.*?)(\d*) <> <>
2547 (.*?)(\d+) <I have > <2>
2548 (.*)(\d+)$ <I have 2 numbers: 5314> <7>
2549 (.*?)(\d+)$ <I have 2 numbers: > <53147>
2550 (.*)\b(\d+)$ <I have 2 numbers: > <53147>
2551 (.*\D)(\d+)$ <I have 2 numbers: > <53147>
2552
2553As you see, this can be a bit tricky. It's important to realize that a
2554regular expression is merely a set of assertions that gives a definition
2555of success. There may be 0, 1, or several different ways that the
2556definition might succeed against a particular string. And if there are
2557multiple ways it might succeed, you need to understand backtracking to
2558know which variety of success you will achieve.
2559
2560When using lookahead assertions and negations, this can all get even
2561trickier. Imagine you'd like to find a sequence of non-digits not
2562followed by "123". You might try to write that as
2563
2564 $_ = "ABC123";
2565 if ( /^\D*(?!123)/ ) { # Wrong!
2566 print "Yup, no 123 in $_\n";
2567 }
2568
2569But that isn't going to match; at least, not the way you're hoping. It
2570claims that there is no 123 in the string. Here's a clearer picture of
2571why that pattern matches, contrary to popular expectations:
2572
2573 $x = 'ABC123';
2574 $y = 'ABC445';
2575
2576 print "1: got $1\n" if $x =~ /^(ABC)(?!123)/;
2577 print "2: got $1\n" if $y =~ /^(ABC)(?!123)/;
2578
2579 print "3: got $1\n" if $x =~ /^(\D*)(?!123)/;
2580 print "4: got $1\n" if $y =~ /^(\D*)(?!123)/;
2581
2582This prints
2583
2584 2: got ABC
2585 3: got AB
2586 4: got ABC
2587
2588You might have expected test 3 to fail because it seems to a more
2589general purpose version of test 1. The important difference between
2590them is that test 3 contains a quantifier (C<\D*>) and so can use
2591backtracking, whereas test 1 will not. What's happening is
2592that you've asked "Is it true that at the start of C<$x>, following 0 or more
2593non-digits, you have something that's not 123?" If the pattern matcher had
2594let C<\D*> expand to "ABC", this would have caused the whole pattern to
2595fail.
2596
2597The search engine will initially match C<\D*> with "ABC". Then it will
2598try to match C<(?!123)> with "123", which fails. But because
2599a quantifier (C<\D*>) has been used in the regular expression, the
2600search engine can backtrack and retry the match differently
2601in the hope of matching the complete regular expression.
2602
2603The pattern really, I<really> wants to succeed, so it uses the
2604standard pattern back-off-and-retry and lets C<\D*> expand to just "AB" this
2605time. Now there's indeed something following "AB" that is not
2606"123". It's "C123", which suffices.
2607
2608We can deal with this by using both an assertion and a negation.
2609We'll say that the first part in C<$1> must be followed both by a digit
2610and by something that's not "123". Remember that the lookaheads
2611are zero-width expressions--they only look, but don't consume any
2612of the string in their match. So rewriting this way produces what
2613you'd expect; that is, case 5 will fail, but case 6 succeeds:
2614
2615 print "5: got $1\n" if $x =~ /^(\D*)(?=\d)(?!123)/;
2616 print "6: got $1\n" if $y =~ /^(\D*)(?=\d)(?!123)/;
2617
2618 6: got ABC
2619
2620In other words, the two zero-width assertions next to each other work as though
2621they're ANDed together, just as you'd use any built-in assertions: C</^$/>
2622matches only if you're at the beginning of the line AND the end of the
2623line simultaneously. The deeper underlying truth is that juxtaposition in
2624regular expressions always means AND, except when you write an explicit OR
2625using the vertical bar. C</ab/> means match "a" AND (then) match "b",
2626although the attempted matches are made at different positions because "a"
2627is not a zero-width assertion, but a one-width assertion.
2628
2629B<WARNING>: Particularly complicated regular expressions can take
2630exponential time to solve because of the immense number of possible
2631ways they can use backtracking to try for a match. For example, without
2632internal optimizations done by the regular expression engine, this will
2633take a painfully long time to run:
2634
2635 'aaaaaaaaaaaa' =~ /((a{0,5}){0,5})*[c]/
2636
2637And if you used C<"*">'s in the internal groups instead of limiting them
2638to 0 through 5 matches, then it would take forever--or until you ran
2639out of stack space. Moreover, these internal optimizations are not
2640always applicable. For example, if you put C<{0,5}> instead of C<"*">
2641on the external group, no current optimization is applicable, and the
2642match takes a long time to finish.
2643
2644A powerful tool for optimizing such beasts is what is known as an
2645"independent group",
2646which does not backtrack (see C<L</(?E<gt>pattern)>>). Note also that
2647zero-length lookahead/lookbehind assertions will not backtrack to make
2648the tail match, since they are in "logical" context: only
2649whether they match is considered relevant. For an example
2650where side-effects of lookahead I<might> have influenced the
2651following match, see C<L</(?E<gt>pattern)>>.
2652
2653=head2 Script Runs
2654X<(*script_run:...)> X<(sr:...)>
2655X<(*atomic_script_run:...)> X<(asr:...)>
2656
2657A script run is basically a sequence of characters, all from the same
2658Unicode script (see L<perlunicode/Scripts>), such as Latin or Greek. In
2659most places a single word would never be written in multiple scripts,
2660unless it is a spoofing attack. An infamous example, is
2661
2662 paypal.com
2663
2664Those letters could all be Latin (as in the example just above), or they
2665could be all Cyrillic (except for the dot), or they could be a mixture
2666of the two. In the case of an internet address the C<.com> would be in
2667Latin, And any Cyrillic ones would cause it to be a mixture, not a
2668script run. Someone clicking on such a link would not be directed to
2669the real Paypal website, but an attacker would craft a look-alike one to
2670attempt to gather sensitive information from the person.
2671
2672Starting in Perl 5.28, it is now easy to detect strings that aren't
2673script runs. Simply enclose just about any pattern like either of
2674these:
2675
2676 (*script_run:pattern)
2677 (*sr:pattern)
2678
2679What happens is that after I<pattern> succeeds in matching, it is
2680subjected to the additional criterion that every character in it must be
2681from the same script (see exceptions below). If this isn't true,
2682backtracking occurs until something all in the same script is found that
2683matches, or all possibilities are exhausted. This can cause a lot of
2684backtracking, but generally, only malicious input will result in this,
2685though the slow down could cause a denial of service attack. If your
2686needs permit, it is best to make the pattern atomic to cut down on the
2687amount of backtracking. This is so likely to be what you want, that
2688instead of writing this:
2689
2690 (*script_run:(?>pattern))
2691
2692you can write either of these:
2693
2694 (*atomic_script_run:pattern)
2695 (*asr:pattern)
2696
2697(See C<L</(?E<gt>I<pattern>)>>.)
2698
2699In Taiwan, Japan, and Korea, it is common for text to have a mixture of
2700characters from their native scripts and base Chinese. Perl follows
2701Unicode's UTS 39 (L<https://unicode.org/reports/tr39/>) Unicode Security
2702Mechanisms in allowing such mixtures. For example, the Japanese scripts
2703Katakana and Hiragana are commonly mixed together in practice, along
2704with some Chinese characters, and hence are treated as being in a single
2705script run by Perl.
2706
2707The rules used for matching decimal digits are slightly stricter. Many
2708scripts have their own sets of digits equivalent to the Western C<0>
2709through C<9> ones. A few, such as Arabic, have more than one set. For
2710a string to be considered a script run, all digits in it must come from
2711the same set of ten, as determined by the first digit encountered.
2712As an example,
2713
2714 qr/(*script_run: \d+ \b )/x
2715
2716guarantees that the digits matched will all be from the same set of 10.
2717You won't get a look-alike digit from a different script that has a
2718different value than what it appears to be.
2719
2720Unicode has three pseudo scripts that are handled specially.
2721
2722"Unknown" is applied to code points whose meaning has yet to be
2723determined. Perl currently will match as a script run, any single
2724character string consisting of one of these code points. But any string
2725longer than one code point containing one of these will not be
2726considered a script run.
2727
2728"Inherited" is applied to characters that modify another, such as an
2729accent of some type. These are considered to be in the script of the
2730master character, and so never cause a script run to not match.
2731
2732The other one is "Common". This consists of mostly punctuation, emoji,
2733characters used in mathematics and music, the ASCII digits C<0>
2734through C<9>, and full-width forms of these digits. These characters
2735can appear intermixed in text in many of the world's scripts. These
2736also don't cause a script run to not match. But like other scripts, all
2737digits in a run must come from the same set of 10.
2738
2739This construct is non-capturing. You can add parentheses to I<pattern>
2740to capture, if desired. You will have to do this if you plan to use
2741L</(*ACCEPT) (*ACCEPT:arg)> and not have it bypass the script run
2742checking.
2743
2744The C<Script_Extensions> property as modified by UTS 39
2745(L<https://unicode.org/reports/tr39/>) is used as the basis for this
2746feature.
2747
2748To summarize,
2749
2750=over 4
2751
2752=item *
2753
2754All length 0 or length 1 sequences are script runs.
2755
2756=item *
2757
2758A longer sequence is a script run if and only if B<all> of the following
2759conditions are met:
2760
2761Z<>
2762
2763=over
2764
2765=item 1
2766
2767No code point in the sequence has the C<Script_Extension> property of
2768C<Unknown>.
2769
2770This currently means that all code points in the sequence have been
2771assigned by Unicode to be characters that aren't private use nor
2772surrogate code points.
2773
2774=item 2
2775
2776All characters in the sequence come from the Common script and/or the
2777Inherited script and/or a single other script.
2778
2779The script of a character is determined by the C<Script_Extensions>
2780property as modified by UTS 39 (L<https://unicode.org/reports/tr39/>), as
2781described above.
2782
2783=item 3
2784
2785All decimal digits in the sequence come from the same block of 10
2786consecutive digits.
2787
2788=back
2789
2790=back
2791
2792=head2 Special Backtracking Control Verbs
2793
2794These special patterns are generally of the form C<(*I<VERB>:I<arg>)>. Unless
2795otherwise stated the I<arg> argument is optional; in some cases, it is
2796mandatory.
2797
2798Any pattern containing a special backtracking verb that allows an argument
2799has the special behaviour that when executed it sets the current package's
2800C<$REGERROR> and C<$REGMARK> variables. When doing so the following
2801rules apply:
2802
2803On failure, the C<$REGERROR> variable will be set to the I<arg> value of the
2804verb pattern, if the verb was involved in the failure of the match. If the
2805I<arg> part of the pattern was omitted, then C<$REGERROR> will be set to the
2806name of the last C<(*MARK:I<NAME>)> pattern executed, or to TRUE if there was
2807none. Also, the C<$REGMARK> variable will be set to FALSE.
2808
2809On a successful match, the C<$REGERROR> variable will be set to FALSE, and
2810the C<$REGMARK> variable will be set to the name of the last
2811C<(*MARK:I<NAME>)> pattern executed. See the explanation for the
2812C<(*MARK:I<NAME>)> verb below for more details.
2813
2814B<NOTE:> C<$REGERROR> and C<$REGMARK> are not magic variables like C<$1>
2815and most other regex-related variables. They are not local to a scope, nor
2816readonly, but instead are volatile package variables similar to C<$AUTOLOAD>.
2817They are set in the package containing the code that I<executed> the regex
2818(rather than the one that compiled it, where those differ). If necessary, you
2819can use C<local> to localize changes to these variables to a specific scope
2820before executing a regex.
2821
2822If a pattern does not contain a special backtracking verb that allows an
2823argument, then C<$REGERROR> and C<$REGMARK> are not touched at all.
2824
2825=over 3
2826
2827=item Verbs
2828
2829=over 4
2830
2831=item C<(*PRUNE)> C<(*PRUNE:I<NAME>)>
2832X<(*PRUNE)> X<(*PRUNE:NAME)>
2833
2834This zero-width pattern prunes the backtracking tree at the current point
2835when backtracked into on failure. Consider the pattern C</I<A> (*PRUNE) I<B>/>,
2836where I<A> and I<B> are complex patterns. Until the C<(*PRUNE)> verb is reached,
2837I<A> may backtrack as necessary to match. Once it is reached, matching
2838continues in I<B>, which may also backtrack as necessary; however, should B
2839not match, then no further backtracking will take place, and the pattern
2840will fail outright at the current starting position.
2841
2842The following example counts all the possible matching strings in a
2843pattern (without actually matching any of them).
2844
2845 'aaab' =~ /a+b?(?{print "$&\n"; $count++})(*FAIL)/;
2846 print "Count=$count\n";
2847
2848which produces:
2849
2850 aaab
2851 aaa
2852 aa
2853 a
2854 aab
2855 aa
2856 a
2857 ab
2858 a
2859 Count=9
2860
2861If we add a C<(*PRUNE)> before the count like the following
2862
2863 'aaab' =~ /a+b?(*PRUNE)(?{print "$&\n"; $count++})(*FAIL)/;
2864 print "Count=$count\n";
2865
2866we prevent backtracking and find the count of the longest matching string
2867at each matching starting point like so:
2868
2869 aaab
2870 aab
2871 ab
2872 Count=3
2873
2874Any number of C<(*PRUNE)> assertions may be used in a pattern.
2875
2876See also C<<< L<< /(?>I<pattern>) >> >>> and possessive quantifiers for
2877other ways to
2878control backtracking. In some cases, the use of C<(*PRUNE)> can be
2879replaced with a C<< (?>pattern) >> with no functional difference; however,
2880C<(*PRUNE)> can be used to handle cases that cannot be expressed using a
2881C<< (?>pattern) >> alone.
2882
2883=item C<(*SKIP)> C<(*SKIP:I<NAME>)>
2884X<(*SKIP)>
2885
2886This zero-width pattern is similar to C<(*PRUNE)>, except that on
2887failure it also signifies that whatever text that was matched leading up
2888to the C<(*SKIP)> pattern being executed cannot be part of I<any> match
2889of this pattern. This effectively means that the regex engine "skips" forward
2890to this position on failure and tries to match again, (assuming that
2891there is sufficient room to match).
2892
2893The name of the C<(*SKIP:I<NAME>)> pattern has special significance. If a
2894C<(*MARK:I<NAME>)> was encountered while matching, then it is that position
2895which is used as the "skip point". If no C<(*MARK)> of that name was
2896encountered, then the C<(*SKIP)> operator has no effect. When used
2897without a name the "skip point" is where the match point was when
2898executing the C<(*SKIP)> pattern.
2899
2900Compare the following to the examples in C<(*PRUNE)>; note the string
2901is twice as long:
2902
2903 'aaabaaab' =~ /a+b?(*SKIP)(?{print "$&\n"; $count++})(*FAIL)/;
2904 print "Count=$count\n";
2905
2906outputs
2907
2908 aaab
2909 aaab
2910 Count=2
2911
2912Once the 'aaab' at the start of the string has matched, and the C<(*SKIP)>
2913executed, the next starting point will be where the cursor was when the
2914C<(*SKIP)> was executed.
2915
2916=item C<(*MARK:I<NAME>)> C<(*:I<NAME>)>
2917X<(*MARK)> X<(*MARK:NAME)> X<(*:NAME)>
2918
2919This zero-width pattern can be used to mark the point reached in a string
2920when a certain part of the pattern has been successfully matched. This
2921mark may be given a name. A later C<(*SKIP)> pattern will then skip
2922forward to that point if backtracked into on failure. Any number of
2923C<(*MARK)> patterns are allowed, and the I<NAME> portion may be duplicated.
2924
2925In addition to interacting with the C<(*SKIP)> pattern, C<(*MARK:I<NAME>)>
2926can be used to "label" a pattern branch, so that after matching, the
2927program can determine which branches of the pattern were involved in the
2928match.
2929
2930When a match is successful, the C<$REGMARK> variable will be set to the
2931name of the most recently executed C<(*MARK:I<NAME>)> that was involved
2932in the match.
2933
2934This can be used to determine which branch of a pattern was matched
2935without using a separate capture group for each branch, which in turn
2936can result in a performance improvement, as perl cannot optimize
2937C</(?:(x)|(y)|(z))/> as efficiently as something like
2938C</(?:x(*MARK:x)|y(*MARK:y)|z(*MARK:z))/>.
2939
2940When a match has failed, and unless another verb has been involved in
2941failing the match and has provided its own name to use, the C<$REGERROR>
2942variable will be set to the name of the most recently executed
2943C<(*MARK:I<NAME>)>.
2944
2945See L</(*SKIP)> for more details.
2946
2947As a shortcut C<(*MARK:I<NAME>)> can be written C<(*:I<NAME>)>.
2948
2949=item C<(*THEN)> C<(*THEN:I<NAME>)>
2950
2951This is similar to the "cut group" operator C<::> from Raku. Like
2952C<(*PRUNE)>, this verb always matches, and when backtracked into on
2953failure, it causes the regex engine to try the next alternation in the
2954innermost enclosing group (capturing or otherwise) that has alternations.
2955The two branches of a C<(?(I<condition>)I<yes-pattern>|I<no-pattern>)> do not
2956count as an alternation, as far as C<(*THEN)> is concerned.
2957
2958Its name comes from the observation that this operation combined with the
2959alternation operator (C<"|">) can be used to create what is essentially a
2960pattern-based if/then/else block:
2961
2962 ( COND (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ )
2963
2964Note that if this operator is used and NOT inside of an alternation then
2965it acts exactly like the C<(*PRUNE)> operator.
2966
2967 / A (*PRUNE) B /
2968
2969is the same as
2970
2971 / A (*THEN) B /
2972
2973but
2974
2975 / ( A (*THEN) B | C ) /
2976
2977is not the same as
2978
2979 / ( A (*PRUNE) B | C ) /
2980
2981as after matching the I<A> but failing on the I<B> the C<(*THEN)> verb will
2982backtrack and try I<C>; but the C<(*PRUNE)> verb will simply fail.
2983
2984=item C<(*COMMIT)> C<(*COMMIT:I<arg>)>
2985X<(*COMMIT)>
2986
2987This is the Raku "commit pattern" C<< <commit> >> or C<:::>. It's a
2988zero-width pattern similar to C<(*SKIP)>, except that when backtracked
2989into on failure it causes the match to fail outright. No further attempts
2990to find a valid match by advancing the start pointer will occur again.
2991For example,
2992
2993 'aaabaaab' =~ /a+b?(*COMMIT)(?{print "$&\n"; $count++})(*FAIL)/;
2994 print "Count=$count\n";
2995
2996outputs
2997
2998 aaab
2999 Count=1
3000
3001In other words, once the C<(*COMMIT)> has been entered, and if the pattern
3002does not match, the regex engine will not try any further matching on the
3003rest of the string.
3004
3005=item C<(*FAIL)> C<(*F)> C<(*FAIL:I<arg>)>
3006X<(*FAIL)> X<(*F)>
3007
3008This pattern matches nothing and always fails. It can be used to force the
3009engine to backtrack. It is equivalent to C<(?!)>, but easier to read. In
3010fact, C<(?!)> gets optimised into C<(*FAIL)> internally. You can provide
3011an argument so that if the match fails because of this C<FAIL> directive
3012the argument can be obtained from C<$REGERROR>.
3013
3014It is probably useful only when combined with C<(?{})> or C<(??{})>.
3015
3016=item C<(*ACCEPT)> C<(*ACCEPT:I<arg>)>
3017X<(*ACCEPT)>
3018
3019This pattern matches nothing and causes the end of successful matching at
3020the point at which the C<(*ACCEPT)> pattern was encountered, regardless of
3021whether there is actually more to match in the string. When inside of a
3022nested pattern, such as recursion, or in a subpattern dynamically generated
3023via C<(??{})>, only the innermost pattern is ended immediately.
3024
3025If the C<(*ACCEPT)> is inside of capturing groups then the groups are
3026marked as ended at the point at which the C<(*ACCEPT)> was encountered.
3027For instance:
3028
3029 'AB' =~ /(A (A|B(*ACCEPT)|C) D)(E)/x;
3030
3031will match, and C<$1> will be C<AB> and C<$2> will be C<"B">, C<$3> will not
3032be set. If another branch in the inner parentheses was matched, such as in the
3033string 'ACDE', then the C<"D"> and C<"E"> would have to be matched as well.
3034
3035You can provide an argument, which will be available in the var
3036C<$REGMARK> after the match completes.
3037
3038=back
3039
3040=back
3041
3042=head2 Warning on C<\1> Instead of C<$1>
3043
3044Some people get too used to writing things like:
3045
3046 $pattern =~ s/(\W)/\\\1/g;
3047
3048This is grandfathered (for \1 to \9) for the RHS of a substitute to avoid
3049shocking the
3050B<sed> addicts, but it's a dirty habit to get into. That's because in
3051PerlThink, the righthand side of an C<s///> is a double-quoted string. C<\1> in
3052the usual double-quoted string means a control-A. The customary Unix
3053meaning of C<\1> is kludged in for C<s///>. However, if you get into the habit
3054of doing that, you get yourself into trouble if you then add an C</e>
3055modifier.
3056
3057 s/(\d+)/ \1 + 1 /eg; # causes warning under -w
3058
3059Or if you try to do
3060
3061 s/(\d+)/\1000/;
3062
3063You can't disambiguate that by saying C<\{1}000>, whereas you can fix it with
3064C<${1}000>. The operation of interpolation should not be confused
3065with the operation of matching a backreference. Certainly they mean two
3066different things on the I<left> side of the C<s///>.
3067
3068=head2 Repeated Patterns Matching a Zero-length Substring
3069
3070B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
3071
3072Regular expressions provide a terse and powerful programming language. As
3073with most other power tools, power comes together with the ability
3074to wreak havoc.
3075
3076A common abuse of this power stems from the ability to make infinite
3077loops using regular expressions, with something as innocuous as:
3078
3079 'foo' =~ m{ ( o? )* }x;
3080
3081The C<o?> matches at the beginning of "C<foo>", and since the position
3082in the string is not moved by the match, C<o?> would match again and again
3083because of the C<"*"> quantifier. Another common way to create a similar cycle
3084is with the looping modifier C</g>:
3085
3086 @matches = ( 'foo' =~ m{ o? }xg );
3087
3088or
3089
3090 print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
3091
3092or the loop implied by C<split()>.
3093
3094However, long experience has shown that many programming tasks may
3095be significantly simplified by using repeated subexpressions that
3096may match zero-length substrings. Here's a simple example being:
3097
3098 @chars = split //, $string; # // is not magic in split
3099 ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
3100
3101Thus Perl allows such constructs, by I<forcefully breaking
3102the infinite loop>. The rules for this are different for lower-level
3103loops given by the greedy quantifiers C<*+{}>, and for higher-level
3104ones like the C</g> modifier or C<split()> operator.
3105
3106The lower-level loops are I<interrupted> (that is, the loop is
3107broken) when Perl detects that a repeated expression matched a
3108zero-length substring. Thus
3109
3110 m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
3111
3112is made equivalent to
3113
3114 m{ (?: NON_ZERO_LENGTH )* (?: ZERO_LENGTH )? }x;
3115
3116For example, this program
3117
3118 #!perl -l
3119 "aaaaab" =~ /
3120 (?:
3121 a # non-zero
3122 | # or
3123 (?{print "hello"}) # print hello whenever this
3124 # branch is tried
3125 (?=(b)) # zero-width assertion
3126 )* # any number of times
3127 /x;
3128 print $&;
3129 print $1;
3130
3131prints
3132
3133 hello
3134 aaaaa
3135 b
3136
3137Notice that "hello" is only printed once, as when Perl sees that the sixth
3138iteration of the outermost C<(?:)*> matches a zero-length string, it stops
3139the C<"*">.
3140
3141The higher-level loops preserve an additional state between iterations:
3142whether the last match was zero-length. To break the loop, the following
3143match after a zero-length match is prohibited to have a length of zero.
3144This prohibition interacts with backtracking (see L</"Backtracking">),
3145and so the I<second best> match is chosen if the I<best> match is of
3146zero length.
3147
3148For example:
3149
3150 $_ = 'bar';
3151 s/\w??/<$&>/g;
3152
3153results in C<< <><b><><a><><r><> >>. At each position of the string the best
3154match given by non-greedy C<??> is the zero-length match, and the I<second
3155best> match is what is matched by C<\w>. Thus zero-length matches
3156alternate with one-character-long matches.
3157
3158Similarly, for repeated C<m/()/g> the second-best match is the match at the
3159position one notch further in the string.
3160
3161The additional state of being I<matched with zero-length> is associated with
3162the matched string, and is reset by each assignment to C<pos()>.
3163Zero-length matches at the end of the previous match are ignored
3164during C<split>.
3165
3166=head2 Combining RE Pieces
3167
3168Each of the elementary pieces of regular expressions which were described
3169before (such as C<ab> or C<\Z>) could match at most one substring
3170at the given position of the input string. However, in a typical regular
3171expression these elementary pieces are combined into more complicated
3172patterns using combining operators C<ST>, C<S|T>, C<S*> I<etc>.
3173(in these examples C<"S"> and C<"T"> are regular subexpressions).
3174
3175Such combinations can include alternatives, leading to a problem of choice:
3176if we match a regular expression C<a|ab> against C<"abc">, will it match
3177substring C<"a"> or C<"ab">? One way to describe which substring is
3178actually matched is the concept of backtracking (see L</"Backtracking">).
3179However, this description is too low-level and makes you think
3180in terms of a particular implementation.
3181
3182Another description starts with notions of "better"/"worse". All the
3183substrings which may be matched by the given regular expression can be
3184sorted from the "best" match to the "worst" match, and it is the "best"
3185match which is chosen. This substitutes the question of "what is chosen?"
3186by the question of "which matches are better, and which are worse?".
3187
3188Again, for elementary pieces there is no such question, since at most
3189one match at a given position is possible. This section describes the
3190notion of better/worse for combining operators. In the description
3191below C<"S"> and C<"T"> are regular subexpressions.
3192
3193=over 4
3194
3195=item C<ST>
3196
3197Consider two possible matches, C<AB> and C<A'B'>, C<"A"> and C<A'> are
3198substrings which can be matched by C<"S">, C<"B"> and C<B'> are substrings
3199which can be matched by C<"T">.
3200
3201If C<"A"> is a better match for C<"S"> than C<A'>, C<AB> is a better
3202match than C<A'B'>.
3203
3204If C<"A"> and C<A'> coincide: C<AB> is a better match than C<AB'> if
3205C<"B"> is a better match for C<"T"> than C<B'>.
3206
3207=item C<S|T>
3208
3209When C<"S"> can match, it is a better match than when only C<"T"> can match.
3210
3211Ordering of two matches for C<"S"> is the same as for C<"S">. Similar for
3212two matches for C<"T">.
3213
3214=item C<S{REPEAT_COUNT}>
3215
3216Matches as C<SSS...S> (repeated as many times as necessary).
3217
3218=item C<S{min,max}>
3219
3220Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
3221
3222=item C<S{min,max}?>
3223
3224Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
3225
3226=item C<S?>, C<S*>, C<S+>
3227
3228Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
3229
3230=item C<S??>, C<S*?>, C<S+?>
3231
3232Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
3233
3234=item C<< (?>S) >>
3235
3236Matches the best match for C<"S"> and only that.
3237
3238=item C<(?=S)>, C<(?<=S)>
3239
3240Only the best match for C<"S"> is considered. (This is important only if
3241C<"S"> has capturing parentheses, and backreferences are used somewhere
3242else in the whole regular expression.)
3243
3244=item C<(?!S)>, C<(?<!S)>
3245
3246For this grouping operator there is no need to describe the ordering, since
3247only whether or not C<"S"> can match is important.
3248
3249=item C<(??{ I<EXPR> })>, C<(?I<PARNO>)>
3250
3251The ordering is the same as for the regular expression which is
3252the result of I<EXPR>, or the pattern contained by capture group I<PARNO>.
3253
3254=item C<(?(I<condition>)I<yes-pattern>|I<no-pattern>)>
3255
3256Recall that which of I<yes-pattern> or I<no-pattern> actually matches is
3257already determined. The ordering of the matches is the same as for the
3258chosen subexpression.
3259
3260=back
3261
3262The above recipes describe the ordering of matches I<at a given position>.
3263One more rule is needed to understand how a match is determined for the
3264whole regular expression: a match at an earlier position is always better
3265than a match at a later position.
3266
3267=head2 Creating Custom RE Engines
3268
3269As of Perl 5.10.0, one can create custom regular expression engines. This
3270is not for the faint of heart, as they have to plug in at the C level. See
3271L<perlreapi> for more details.
3272
3273As an alternative, overloaded constants (see L<overload>) provide a simple
3274way to extend the functionality of the RE engine, by substituting one
3275pattern for another.
3276
3277Suppose that we want to enable a new RE escape-sequence C<\Y|> which
3278matches at a boundary between whitespace characters and non-whitespace
3279characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
3280at these positions, so we want to have each C<\Y|> in the place of the
3281more complicated version. We can create a module C<customre> to do
3282this:
3283
3284 package customre;
3285 use overload;
3286
3287 sub import {
3288 shift;
3289 die "No argument to customre::import allowed" if @_;
3290 overload::constant 'qr' => \&convert;
3291 }
3292
3293 sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
3294
3295 # We must also take care of not escaping the legitimate \\Y|
3296 # sequence, hence the presence of '\\' in the conversion rules.
3297 my %rules = ( '\\' => '\\\\',
3298 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
3299 sub convert {
3300 my $re = shift;
3301 $re =~ s{
3302 \\ ( \\ | Y . )
3303 }
3304 { $rules{$1} or invalid($re,$1) }sgex;
3305 return $re;
3306 }
3307
3308Now C<use customre> enables the new escape in constant regular
3309expressions, I<i.e.>, those without any runtime variable interpolations.
3310As documented in L<overload>, this conversion will work only over
3311literal parts of regular expressions. For C<\Y|$re\Y|> the variable
3312part of this regular expression needs to be converted explicitly
3313(but only if the special meaning of C<\Y|> should be enabled inside C<$re>):
3314
3315 use customre;
3316 $re = <>;
3317 chomp $re;
3318 $re = customre::convert $re;
3319 /\Y|$re\Y|/;
3320
3321=head2 Embedded Code Execution Frequency
3322
3323The exact rules for how often C<(?{})> and C<(??{})> are executed in a pattern
3324are unspecified, and this is even more true of C<(*{})>.
3325In the case of a successful match you can assume that they DWIM and
3326will be executed in left to right order the appropriate number of times in the
3327accepting path of the pattern as would any other meta-pattern. How non-
3328accepting pathways and match failures affect the number of times a pattern is
3329executed is specifically unspecified and may vary depending on what
3330optimizations can be applied to the pattern and is likely to change from
3331version to version.
3332
3333For instance in
3334
3335 "aaabcdeeeee"=~/a(?{print "a"})b(?{print "b"})cde/;
3336
3337the exact number of times "a" or "b" are printed out is unspecified for
3338failure, but you may assume they will be printed at least once during
3339a successful match, additionally you may assume that if "b" is printed,
3340it will be preceded by at least one "a".
3341
3342In the case of branching constructs like the following:
3343
3344 /a(b|(?{ print "a" }))c(?{ print "c" })/;
3345
3346you can assume that the input "ac" will output "ac", and that "abc"
3347will output only "c".
3348
3349When embedded code is quantified, successful matches will call the
3350code once for each matched iteration of the quantifier. For
3351example:
3352
3353 "good" =~ /g(?:o(?{print "o"}))*d/;
3354
3355will output "o" twice.
3356
3357For historical and consistency reasons the use of normal code blocks
3358anywhere in a pattern will disable certain optimisations. As of 5.37.7
3359you can use an "optimistic" codeblock, C<(*{ ... })> as a replacement
3360for C<(?{ ... })>, if you do *not* wish to disable these optimisations.
3361This may result in the code block being called less often than it might
3362have been had they not been optimistic.
3363
3364=head2 PCRE/Python Support
3365
3366As of Perl 5.10.0, Perl supports several Python/PCRE-specific extensions
3367to the regex syntax. While Perl programmers are encouraged to use the
3368Perl-specific syntax, the following are also accepted:
3369
3370=over 4
3371
3372=item C<< (?PE<lt>I<NAME>E<gt>I<pattern>) >>
3373
3374Define a named capture group. Equivalent to C<< (?<I<NAME>>I<pattern>) >>.
3375
3376=item C<< (?P=I<NAME>) >>
3377
3378Backreference to a named capture group. Equivalent to C<< \g{I<NAME>} >>.
3379
3380=item C<< (?P>I<NAME>) >>
3381
3382Subroutine call to a named capture group. Equivalent to C<< (?&I<NAME>) >>.
3383
3384=back
3385
3386=head1 BUGS
3387
3388There are a number of issues with regard to case-insensitive matching
3389in Unicode rules. See C<"i"> under L</Modifiers> above.
3390
3391This document varies from difficult to understand to completely
3392and utterly opaque. The wandering prose riddled with jargon is
3393hard to fathom in several places.
3394
3395This document needs a rewrite that separates the tutorial content
3396from the reference content.
3397
3398=head1 SEE ALSO
3399
3400The syntax of patterns used in Perl pattern matching evolved from those
3401supplied in the Bell Labs Research Unix 8th Edition (Version 8) regex
3402routines. (The code is actually derived (distantly) from Henry
3403Spencer's freely redistributable reimplementation of those V8 routines.)
3404
3405L<perlrequick>.
3406
3407L<perlretut>.
3408
3409L<perlop/"Regexp Quote-Like Operators">.
3410
3411L<perlop/"Gory details of parsing quoted constructs">.
3412
3413L<perlfaq6>.
3414
3415L<perlfunc/pos>.
3416
3417L<perllocale>.
3418
3419L<perlebcdic>.
3420
3421I<Mastering Regular Expressions> by Jeffrey Friedl, published
3422by O'Reilly and Associates.