+=head2 Repeated patterns matching zero-length substring
+
+B<WARNING>: Difficult material (and prose) ahead. This section needs a rewrite.
+
+Regular expressions provide a terse and powerful programming language. As
+with most other power tools, power comes together with the ability
+to wreak havoc.
+
+A common abuse of this power stems from the ability to make infinite
+loops using regular expressions, with something as innocuous as:
+
+ 'foo' =~ m{ ( o? )* }x;
+
+The C<o?> can match at the beginning of C<'foo'>, and since the position
+in the string is not moved by the match, C<o?> would match again and again
+because of the C<*> modifier. Another common way to create a similar cycle
+is with the looping modifier C<//g>:
+
+ @matches = ( 'foo' =~ m{ o? }xg );
+
+or
+
+ print "match: <$&>\n" while 'foo' =~ m{ o? }xg;
+
+or the loop implied by split().
+
+However, long experience has shown that many programming tasks may
+be significantly simplified by using repeated subexpressions that
+may match zero-length substrings. Here's a simple example being:
+
+ @chars = split //, $string; # // is not magic in split
+ ($whitewashed = $string) =~ s/()/ /g; # parens avoid magic s// /
+
+Thus Perl allows such constructs, by I<forcefully breaking
+the infinite loop>. The rules for this are different for lower-level
+loops given by the greedy modifiers C<*+{}>, and for higher-level
+ones like the C</g> modifier or split() operator.
+
+The lower-level loops are I<interrupted> (that is, the loop is
+broken) when Perl detects that a repeated expression matched a
+zero-length substring. Thus
+
+ m{ (?: NON_ZERO_LENGTH | ZERO_LENGTH )* }x;
+
+is made equivalent to
+
+ m{ (?: NON_ZERO_LENGTH )*
+ |
+ (?: ZERO_LENGTH )?
+ }x;
+
+The higher level-loops preserve an additional state between iterations:
+whether the last match was zero-length. To break the loop, the following
+match after a zero-length match is prohibited to have a length of zero.
+This prohibition interacts with backtracking (see L<"Backtracking">),
+and so the I<second best> match is chosen if the I<best> match is of
+zero length.
+
+For example:
+
+ $_ = 'bar';
+ s/\w??/<$&>/g;
+
+results in C<< <><b><><a><><r><> >>. At each position of the string the best
+match given by non-greedy C<??> is the zero-length match, and the I<second
+best> match is what is matched by C<\w>. Thus zero-length matches
+alternate with one-character-long matches.
+
+Similarly, for repeated C<m/()/g> the second-best match is the match at the
+position one notch further in the string.
+
+The additional state of being I<matched with zero-length> is associated with
+the matched string, and is reset by each assignment to pos().
+Zero-length matches at the end of the previous match are ignored
+during C<split>.
+
+=head2 Combining pieces together
+
+Each of the elementary pieces of regular expressions which were described
+before (such as C<ab> or C<\Z>) could match at most one substring
+at the given position of the input string. However, in a typical regular
+expression these elementary pieces are combined into more complicated
+patterns using combining operators C<ST>, C<S|T>, C<S*> etc
+(in these examples C<S> and C<T> are regular subexpressions).
+
+Such combinations can include alternatives, leading to a problem of choice:
+if we match a regular expression C<a|ab> against C<"abc">, will it match
+substring C<"a"> or C<"ab">? One way to describe which substring is
+actually matched is the concept of backtracking (see L<"Backtracking">).
+However, this description is too low-level and makes you think
+in terms of a particular implementation.
+
+Another description starts with notions of "better"/"worse". All the
+substrings which may be matched by the given regular expression can be
+sorted from the "best" match to the "worst" match, and it is the "best"
+match which is chosen. This substitutes the question of "what is chosen?"
+by the question of "which matches are better, and which are worse?".
+
+Again, for elementary pieces there is no such question, since at most
+one match at a given position is possible. This section describes the
+notion of better/worse for combining operators. In the description
+below C<S> and C<T> are regular subexpressions.
+
+=over 4
+
+=item C<ST>
+
+Consider two possible matches, C<AB> and C<A'B'>, C<A> and C<A'> are
+substrings which can be matched by C<S>, C<B> and C<B'> are substrings
+which can be matched by C<T>.
+
+If C<A> is better match for C<S> than C<A'>, C<AB> is a better
+match than C<A'B'>.
+
+If C<A> and C<A'> coincide: C<AB> is a better match than C<AB'> if
+C<B> is better match for C<T> than C<B'>.
+
+=item C<S|T>
+
+When C<S> can match, it is a better match than when only C<T> can match.
+
+Ordering of two matches for C<S> is the same as for C<S>. Similar for
+two matches for C<T>.
+
+=item C<S{REPEAT_COUNT}>
+
+Matches as C<SSS...S> (repeated as many times as necessary).
+
+=item C<S{min,max}>
+
+Matches as C<S{max}|S{max-1}|...|S{min+1}|S{min}>.
+
+=item C<S{min,max}?>
+
+Matches as C<S{min}|S{min+1}|...|S{max-1}|S{max}>.
+
+=item C<S?>, C<S*>, C<S+>
+
+Same as C<S{0,1}>, C<S{0,BIG_NUMBER}>, C<S{1,BIG_NUMBER}> respectively.
+
+=item C<S??>, C<S*?>, C<S+?>
+
+Same as C<S{0,1}?>, C<S{0,BIG_NUMBER}?>, C<S{1,BIG_NUMBER}?> respectively.
+
+=item C<< (?>S) >>
+
+Matches the best match for C<S> and only that.
+
+=item C<(?=S)>, C<(?<=S)>
+
+Only the best match for C<S> is considered. (This is important only if
+C<S> has capturing parentheses, and backreferences are used somewhere
+else in the whole regular expression.)
+
+=item C<(?!S)>, C<(?<!S)>
+
+For this grouping operator there is no need to describe the ordering, since
+only whether or not C<S> can match is important.
+
+=item C<(??{ EXPR })>
+
+The ordering is the same as for the regular expression which is
+the result of EXPR.
+
+=item C<(?(condition)yes-pattern|no-pattern)>
+
+Recall that which of C<yes-pattern> or C<no-pattern> actually matches is
+already determined. The ordering of the matches is the same as for the
+chosen subexpression.
+
+=back
+
+The above recipes describe the ordering of matches I<at a given position>.
+One more rule is needed to understand how a match is determined for the
+whole regular expression: a match at an earlier position is always better
+than a match at a later position.
+
+=head2 Creating custom RE engines
+
+Overloaded constants (see L<overload>) provide a simple way to extend
+the functionality of the RE engine.
+
+Suppose that we want to enable a new RE escape-sequence C<\Y|> which
+matches at boundary between whitespace characters and non-whitespace
+characters. Note that C<(?=\S)(?<!\S)|(?!\S)(?<=\S)> matches exactly
+at these positions, so we want to have each C<\Y|> in the place of the
+more complicated version. We can create a module C<customre> to do
+this:
+
+ package customre;
+ use overload;
+
+ sub import {
+ shift;
+ die "No argument to customre::import allowed" if @_;
+ overload::constant 'qr' => \&convert;
+ }
+
+ sub invalid { die "/$_[0]/: invalid escape '\\$_[1]'"}
+
+ # We must also take care of not escaping the legitimate \\Y|
+ # sequence, hence the presence of '\\' in the conversion rules.
+ my %rules = ( '\\' => '\\\\',
+ 'Y|' => qr/(?=\S)(?<!\S)|(?!\S)(?<=\S)/ );
+ sub convert {
+ my $re = shift;
+ $re =~ s{
+ \\ ( \\ | Y . )
+ }
+ { $rules{$1} or invalid($re,$1) }sgex;
+ return $re;
+ }
+
+Now C<use customre> enables the new escape in constant regular
+expressions, i.e., those without any runtime variable interpolations.
+As documented in L<overload>, this conversion will work only over
+literal parts of regular expressions. For C<\Y|$re\Y|> the variable
+part of this regular expression needs to be converted explicitly
+(but only if the special meaning of C<\Y|> should be enabled inside $re):
+
+ use customre;
+ $re = <>;
+ chomp $re;
+ $re = customre::convert $re;
+ /\Y|$re\Y|/;
+
+=head1 BUGS
+
+This document varies from difficult to understand to completely
+and utterly opaque. The wandering prose riddled with jargon is
+hard to fathom in several places.
+
+This document needs a rewrite that separates the tutorial content
+from the reference content.
+
+=head1 SEE ALSO
+
+L<perlrequick>.
+
+L<perlretut>.
https://perl5.git.perl.org / perl5.git / blobdiff