| 1 | =head1 NAME |
| 2 | |
| 3 | perlretut - Perl regular expressions tutorial |
| 4 | |
| 5 | =head1 DESCRIPTION |
| 6 | |
| 7 | This page provides a basic tutorial on understanding, creating and |
| 8 | using regular expressions in Perl. It serves as a complement to the |
| 9 | reference page on regular expressions L<perlre>. Regular expressions |
| 10 | are an integral part of the C<m//>, C<s///>, C<qr//> and C<split> |
| 11 | operators and so this tutorial also overlaps with |
| 12 | L<perlop/"Regexp Quote-Like Operators"> and L<perlfunc/split>. |
| 13 | |
| 14 | Perl is widely renowned for excellence in text processing, and regular |
| 15 | expressions are one of the big factors behind this fame. Perl regular |
| 16 | expressions display an efficiency and flexibility unknown in most |
| 17 | other computer languages. Mastering even the basics of regular |
| 18 | expressions will allow you to manipulate text with surprising ease. |
| 19 | |
| 20 | What is a regular expression? A regular expression is simply a string |
| 21 | that describes a pattern. Patterns are in common use these days; |
| 22 | examples are the patterns typed into a search engine to find web pages |
| 23 | and the patterns used to list files in a directory, e.g., C<ls *.txt> |
| 24 | or C<dir *.*>. In Perl, the patterns described by regular expressions |
| 25 | are used to search strings, extract desired parts of strings, and to |
| 26 | do search and replace operations. |
| 27 | |
| 28 | Regular expressions have the undeserved reputation of being abstract |
| 29 | and difficult to understand. Regular expressions are constructed using |
| 30 | simple concepts like conditionals and loops and are no more difficult |
| 31 | to understand than the corresponding C<if> conditionals and C<while> |
| 32 | loops in the Perl language itself. In fact, the main challenge in |
| 33 | learning regular expressions is just getting used to the terse |
| 34 | notation used to express these concepts. |
| 35 | |
| 36 | This tutorial flattens the learning curve by discussing regular |
| 37 | expression concepts, along with their notation, one at a time and with |
| 38 | many examples. The first part of the tutorial will progress from the |
| 39 | simplest word searches to the basic regular expression concepts. If |
| 40 | you master the first part, you will have all the tools needed to solve |
| 41 | about 98% of your needs. The second part of the tutorial is for those |
| 42 | comfortable with the basics and hungry for more power tools. It |
| 43 | discusses the more advanced regular expression operators and |
| 44 | introduces the latest cutting-edge innovations. |
| 45 | |
| 46 | A note: to save time, 'regular expression' is often abbreviated as |
| 47 | regexp or regex. Regexp is a more natural abbreviation than regex, but |
| 48 | is harder to pronounce. The Perl pod documentation is evenly split on |
| 49 | regexp vs regex; in Perl, there is more than one way to abbreviate it. |
| 50 | We'll use regexp in this tutorial. |
| 51 | |
| 52 | New in v5.22, L<C<use re 'strict'>|re/'strict' mode> applies stricter |
| 53 | rules than otherwise when compiling regular expression patterns. It can |
| 54 | find things that, while legal, may not be what you intended. |
| 55 | |
| 56 | =head1 Part 1: The basics |
| 57 | |
| 58 | =head2 Simple word matching |
| 59 | |
| 60 | The simplest regexp is simply a word, or more generally, a string of |
| 61 | characters. A regexp consisting of a word matches any string that |
| 62 | contains that word: |
| 63 | |
| 64 | "Hello World" =~ /World/; # matches |
| 65 | |
| 66 | What is this Perl statement all about? C<"Hello World"> is a simple |
| 67 | double-quoted string. C<World> is the regular expression and the |
| 68 | C<//> enclosing C</World/> tells Perl to search a string for a match. |
| 69 | The operator C<=~> associates the string with the regexp match and |
| 70 | produces a true value if the regexp matched, or false if the regexp |
| 71 | did not match. In our case, C<World> matches the second word in |
| 72 | C<"Hello World">, so the expression is true. Expressions like this |
| 73 | are useful in conditionals: |
| 74 | |
| 75 | if ("Hello World" =~ /World/) { |
| 76 | print "It matches\n"; |
| 77 | } |
| 78 | else { |
| 79 | print "It doesn't match\n"; |
| 80 | } |
| 81 | |
| 82 | There are useful variations on this theme. The sense of the match can |
| 83 | be reversed by using the C<!~> operator: |
| 84 | |
| 85 | if ("Hello World" !~ /World/) { |
| 86 | print "It doesn't match\n"; |
| 87 | } |
| 88 | else { |
| 89 | print "It matches\n"; |
| 90 | } |
| 91 | |
| 92 | The literal string in the regexp can be replaced by a variable: |
| 93 | |
| 94 | $greeting = "World"; |
| 95 | if ("Hello World" =~ /$greeting/) { |
| 96 | print "It matches\n"; |
| 97 | } |
| 98 | else { |
| 99 | print "It doesn't match\n"; |
| 100 | } |
| 101 | |
| 102 | If you're matching against the special default variable C<$_>, the |
| 103 | C<$_ =~> part can be omitted: |
| 104 | |
| 105 | $_ = "Hello World"; |
| 106 | if (/World/) { |
| 107 | print "It matches\n"; |
| 108 | } |
| 109 | else { |
| 110 | print "It doesn't match\n"; |
| 111 | } |
| 112 | |
| 113 | And finally, the C<//> default delimiters for a match can be changed |
| 114 | to arbitrary delimiters by putting an C<'m'> out front: |
| 115 | |
| 116 | "Hello World" =~ m!World!; # matches, delimited by '!' |
| 117 | "Hello World" =~ m{World}; # matches, note the matching '{}' |
| 118 | "/usr/bin/perl" =~ m"/perl"; # matches after '/usr/bin', |
| 119 | # '/' becomes an ordinary char |
| 120 | |
| 121 | C</World/>, C<m!World!>, and C<m{World}> all represent the |
| 122 | same thing. When, e.g., the quote (C<">) is used as a delimiter, the forward |
| 123 | slash C<'/'> becomes an ordinary character and can be used in this regexp |
| 124 | without trouble. |
| 125 | |
| 126 | Let's consider how different regexps would match C<"Hello World">: |
| 127 | |
| 128 | "Hello World" =~ /world/; # doesn't match |
| 129 | "Hello World" =~ /o W/; # matches |
| 130 | "Hello World" =~ /oW/; # doesn't match |
| 131 | "Hello World" =~ /World /; # doesn't match |
| 132 | |
| 133 | The first regexp C<world> doesn't match because regexps are |
| 134 | case-sensitive. The second regexp matches because the substring |
| 135 | S<C<'o W'>> occurs in the string S<C<"Hello World">>. The space |
| 136 | character ' ' is treated like any other character in a regexp and is |
| 137 | needed to match in this case. The lack of a space character is the |
| 138 | reason the third regexp C<'oW'> doesn't match. The fourth regexp |
| 139 | C<'World '> doesn't match because there is a space at the end of the |
| 140 | regexp, but not at the end of the string. The lesson here is that |
| 141 | regexps must match a part of the string I<exactly> in order for the |
| 142 | statement to be true. |
| 143 | |
| 144 | If a regexp matches in more than one place in the string, Perl will |
| 145 | always match at the earliest possible point in the string: |
| 146 | |
| 147 | "Hello World" =~ /o/; # matches 'o' in 'Hello' |
| 148 | "That hat is red" =~ /hat/; # matches 'hat' in 'That' |
| 149 | |
| 150 | With respect to character matching, there are a few more points you |
| 151 | need to know about. First of all, not all characters can be used 'as |
| 152 | is' in a match. Some characters, called I<metacharacters>, are reserved |
| 153 | for use in regexp notation. The metacharacters are |
| 154 | |
| 155 | {}[]()^$.|*+?\ |
| 156 | |
| 157 | The significance of each of these will be explained |
| 158 | in the rest of the tutorial, but for now, it is important only to know |
| 159 | that a metacharacter can be matched by putting a backslash before it: |
| 160 | |
| 161 | "2+2=4" =~ /2+2/; # doesn't match, + is a metacharacter |
| 162 | "2+2=4" =~ /2\+2/; # matches, \+ is treated like an ordinary + |
| 163 | "The interval is [0,1)." =~ /[0,1)./ # is a syntax error! |
| 164 | "The interval is [0,1)." =~ /\[0,1\)\./ # matches |
| 165 | "#!/usr/bin/perl" =~ /#!\/usr\/bin\/perl/; # matches |
| 166 | |
| 167 | In the last regexp, the forward slash C<'/'> is also backslashed, |
| 168 | because it is used to delimit the regexp. This can lead to LTS |
| 169 | (leaning toothpick syndrome), however, and it is often more readable |
| 170 | to change delimiters. |
| 171 | |
| 172 | "#!/usr/bin/perl" =~ m!#\!/usr/bin/perl!; # easier to read |
| 173 | |
| 174 | The backslash character C<'\'> is a metacharacter itself and needs to |
| 175 | be backslashed: |
| 176 | |
| 177 | 'C:\WIN32' =~ /C:\\WIN/; # matches |
| 178 | |
| 179 | In addition to the metacharacters, there are some ASCII characters |
| 180 | which don't have printable character equivalents and are instead |
| 181 | represented by I<escape sequences>. Common examples are C<\t> for a |
| 182 | tab, C<\n> for a newline, C<\r> for a carriage return and C<\a> for a |
| 183 | bell (or alert). If your string is better thought of as a sequence of arbitrary |
| 184 | bytes, the octal escape sequence, e.g., C<\033>, or hexadecimal escape |
| 185 | sequence, e.g., C<\x1B> may be a more natural representation for your |
| 186 | bytes. Here are some examples of escapes: |
| 187 | |
| 188 | "1000\t2000" =~ m(0\t2) # matches |
| 189 | "1000\n2000" =~ /0\n20/ # matches |
| 190 | "1000\t2000" =~ /\000\t2/ # doesn't match, "0" ne "\000" |
| 191 | "cat" =~ /\o{143}\x61\x74/ # matches in ASCII, but a weird way |
| 192 | # to spell cat |
| 193 | |
| 194 | If you've been around Perl a while, all this talk of escape sequences |
| 195 | may seem familiar. Similar escape sequences are used in double-quoted |
| 196 | strings and in fact the regexps in Perl are mostly treated as |
| 197 | double-quoted strings. This means that variables can be used in |
| 198 | regexps as well. Just like double-quoted strings, the values of the |
| 199 | variables in the regexp will be substituted in before the regexp is |
| 200 | evaluated for matching purposes. So we have: |
| 201 | |
| 202 | $foo = 'house'; |
| 203 | 'housecat' =~ /$foo/; # matches |
| 204 | 'cathouse' =~ /cat$foo/; # matches |
| 205 | 'housecat' =~ /${foo}cat/; # matches |
| 206 | |
| 207 | So far, so good. With the knowledge above you can already perform |
| 208 | searches with just about any literal string regexp you can dream up. |
| 209 | Here is a I<very simple> emulation of the Unix grep program: |
| 210 | |
| 211 | % cat > simple_grep |
| 212 | #!/usr/bin/perl |
| 213 | $regexp = shift; |
| 214 | while (<>) { |
| 215 | print if /$regexp/; |
| 216 | } |
| 217 | ^D |
| 218 | |
| 219 | % chmod +x simple_grep |
| 220 | |
| 221 | % simple_grep abba /usr/dict/words |
| 222 | Babbage |
| 223 | cabbage |
| 224 | cabbages |
| 225 | sabbath |
| 226 | Sabbathize |
| 227 | Sabbathizes |
| 228 | sabbatical |
| 229 | scabbard |
| 230 | scabbards |
| 231 | |
| 232 | This program is easy to understand. C<#!/usr/bin/perl> is the standard |
| 233 | way to invoke a perl program from the shell. |
| 234 | S<C<$regexp = shift;>> saves the first command line argument as the |
| 235 | regexp to be used, leaving the rest of the command line arguments to |
| 236 | be treated as files. S<C<< while (<>) >>> loops over all the lines in |
| 237 | all the files. For each line, S<C<print if /$regexp/;>> prints the |
| 238 | line if the regexp matches the line. In this line, both C<print> and |
| 239 | C</$regexp/> use the default variable C<$_> implicitly. |
| 240 | |
| 241 | With all of the regexps above, if the regexp matched anywhere in the |
| 242 | string, it was considered a match. Sometimes, however, we'd like to |
| 243 | specify I<where> in the string the regexp should try to match. To do |
| 244 | this, we would use the I<anchor> metacharacters C<^> and C<$>. The |
| 245 | anchor C<^> means match at the beginning of the string and the anchor |
| 246 | C<$> means match at the end of the string, or before a newline at the |
| 247 | end of the string. Here is how they are used: |
| 248 | |
| 249 | "housekeeper" =~ /keeper/; # matches |
| 250 | "housekeeper" =~ /^keeper/; # doesn't match |
| 251 | "housekeeper" =~ /keeper$/; # matches |
| 252 | "housekeeper\n" =~ /keeper$/; # matches |
| 253 | |
| 254 | The second regexp doesn't match because C<^> constrains C<keeper> to |
| 255 | match only at the beginning of the string, but C<"housekeeper"> has |
| 256 | keeper starting in the middle. The third regexp does match, since the |
| 257 | C<$> constrains C<keeper> to match only at the end of the string. |
| 258 | |
| 259 | When both C<^> and C<$> are used at the same time, the regexp has to |
| 260 | match both the beginning and the end of the string, i.e., the regexp |
| 261 | matches the whole string. Consider |
| 262 | |
| 263 | "keeper" =~ /^keep$/; # doesn't match |
| 264 | "keeper" =~ /^keeper$/; # matches |
| 265 | "" =~ /^$/; # ^$ matches an empty string |
| 266 | |
| 267 | The first regexp doesn't match because the string has more to it than |
| 268 | C<keep>. Since the second regexp is exactly the string, it |
| 269 | matches. Using both C<^> and C<$> in a regexp forces the complete |
| 270 | string to match, so it gives you complete control over which strings |
| 271 | match and which don't. Suppose you are looking for a fellow named |
| 272 | bert, off in a string by himself: |
| 273 | |
| 274 | "dogbert" =~ /bert/; # matches, but not what you want |
| 275 | |
| 276 | "dilbert" =~ /^bert/; # doesn't match, but .. |
| 277 | "bertram" =~ /^bert/; # matches, so still not good enough |
| 278 | |
| 279 | "bertram" =~ /^bert$/; # doesn't match, good |
| 280 | "dilbert" =~ /^bert$/; # doesn't match, good |
| 281 | "bert" =~ /^bert$/; # matches, perfect |
| 282 | |
| 283 | Of course, in the case of a literal string, one could just as easily |
| 284 | use the string comparison S<C<$string eq 'bert'>> and it would be |
| 285 | more efficient. The C<^...$> regexp really becomes useful when we |
| 286 | add in the more powerful regexp tools below. |
| 287 | |
| 288 | =head2 Using character classes |
| 289 | |
| 290 | Although one can already do quite a lot with the literal string |
| 291 | regexps above, we've only scratched the surface of regular expression |
| 292 | technology. In this and subsequent sections we will introduce regexp |
| 293 | concepts (and associated metacharacter notations) that will allow a |
| 294 | regexp to represent not just a single character sequence, but a I<whole |
| 295 | class> of them. |
| 296 | |
| 297 | One such concept is that of a I<character class>. A character class |
| 298 | allows a set of possible characters, rather than just a single |
| 299 | character, to match at a particular point in a regexp. You can define |
| 300 | your own custom character classes. These |
| 301 | are denoted by brackets C<[...]>, with the set of characters |
| 302 | to be possibly matched inside. Here are some examples: |
| 303 | |
| 304 | /cat/; # matches 'cat' |
| 305 | /[bcr]at/; # matches 'bat, 'cat', or 'rat' |
| 306 | /item[0123456789]/; # matches 'item0' or ... or 'item9' |
| 307 | "abc" =~ /[cab]/; # matches 'a' |
| 308 | |
| 309 | In the last statement, even though C<'c'> is the first character in |
| 310 | the class, C<'a'> matches because the first character position in the |
| 311 | string is the earliest point at which the regexp can match. |
| 312 | |
| 313 | /[yY][eE][sS]/; # match 'yes' in a case-insensitive way |
| 314 | # 'yes', 'Yes', 'YES', etc. |
| 315 | |
| 316 | This regexp displays a common task: perform a case-insensitive |
| 317 | match. Perl provides a way of avoiding all those brackets by simply |
| 318 | appending an C<'i'> to the end of the match. Then C</[yY][eE][sS]/;> |
| 319 | can be rewritten as C</yes/i;>. The C<'i'> stands for |
| 320 | case-insensitive and is an example of a I<modifier> of the matching |
| 321 | operation. We will meet other modifiers later in the tutorial. |
| 322 | |
| 323 | We saw in the section above that there were ordinary characters, which |
| 324 | represented themselves, and special characters, which needed a |
| 325 | backslash C<\> to represent themselves. The same is true in a |
| 326 | character class, but the sets of ordinary and special characters |
| 327 | inside a character class are different than those outside a character |
| 328 | class. The special characters for a character class are C<-]\^$> (and |
| 329 | the pattern delimiter, whatever it is). |
| 330 | C<]> is special because it denotes the end of a character class. C<$> is |
| 331 | special because it denotes a scalar variable. C<\> is special because |
| 332 | it is used in escape sequences, just like above. Here is how the |
| 333 | special characters C<]$\> are handled: |
| 334 | |
| 335 | /[\]c]def/; # matches ']def' or 'cdef' |
| 336 | $x = 'bcr'; |
| 337 | /[$x]at/; # matches 'bat', 'cat', or 'rat' |
| 338 | /[\$x]at/; # matches '$at' or 'xat' |
| 339 | /[\\$x]at/; # matches '\at', 'bat, 'cat', or 'rat' |
| 340 | |
| 341 | The last two are a little tricky. In C<[\$x]>, the backslash protects |
| 342 | the dollar sign, so the character class has two members C<$> and C<x>. |
| 343 | In C<[\\$x]>, the backslash is protected, so C<$x> is treated as a |
| 344 | variable and substituted in double quote fashion. |
| 345 | |
| 346 | The special character C<'-'> acts as a range operator within character |
| 347 | classes, so that a contiguous set of characters can be written as a |
| 348 | range. With ranges, the unwieldy C<[0123456789]> and C<[abc...xyz]> |
| 349 | become the svelte C<[0-9]> and C<[a-z]>. Some examples are |
| 350 | |
| 351 | /item[0-9]/; # matches 'item0' or ... or 'item9' |
| 352 | /[0-9bx-z]aa/; # matches '0aa', ..., '9aa', |
| 353 | # 'baa', 'xaa', 'yaa', or 'zaa' |
| 354 | /[0-9a-fA-F]/; # matches a hexadecimal digit |
| 355 | /[0-9a-zA-Z_]/; # matches a "word" character, |
| 356 | # like those in a Perl variable name |
| 357 | |
| 358 | If C<'-'> is the first or last character in a character class, it is |
| 359 | treated as an ordinary character; C<[-ab]>, C<[ab-]> and C<[a\-b]> are |
| 360 | all equivalent. |
| 361 | |
| 362 | The special character C<^> in the first position of a character class |
| 363 | denotes a I<negated character class>, which matches any character but |
| 364 | those in the brackets. Both C<[...]> and C<[^...]> must match a |
| 365 | character, or the match fails. Then |
| 366 | |
| 367 | /[^a]at/; # doesn't match 'aat' or 'at', but matches |
| 368 | # all other 'bat', 'cat, '0at', '%at', etc. |
| 369 | /[^0-9]/; # matches a non-numeric character |
| 370 | /[a^]at/; # matches 'aat' or '^at'; here '^' is ordinary |
| 371 | |
| 372 | Now, even C<[0-9]> can be a bother to write multiple times, so in the |
| 373 | interest of saving keystrokes and making regexps more readable, Perl |
| 374 | has several abbreviations for common character classes, as shown below. |
| 375 | Since the introduction of Unicode, unless the C<//a> modifier is in |
| 376 | effect, these character classes match more than just a few characters in |
| 377 | the ASCII range. |
| 378 | |
| 379 | =over 4 |
| 380 | |
| 381 | =item * |
| 382 | |
| 383 | \d matches a digit, not just [0-9] but also digits from non-roman scripts |
| 384 | |
| 385 | =item * |
| 386 | |
| 387 | \s matches a whitespace character, the set [\ \t\r\n\f] and others |
| 388 | |
| 389 | =item * |
| 390 | |
| 391 | \w matches a word character (alphanumeric or _), not just [0-9a-zA-Z_] |
| 392 | but also digits and characters from non-roman scripts |
| 393 | |
| 394 | =item * |
| 395 | |
| 396 | \D is a negated \d; it represents any other character than a digit, or [^\d] |
| 397 | |
| 398 | =item * |
| 399 | |
| 400 | \S is a negated \s; it represents any non-whitespace character [^\s] |
| 401 | |
| 402 | =item * |
| 403 | |
| 404 | \W is a negated \w; it represents any non-word character [^\w] |
| 405 | |
| 406 | =item * |
| 407 | |
| 408 | The period '.' matches any character but "\n" (unless the modifier C<//s> is |
| 409 | in effect, as explained below). |
| 410 | |
| 411 | =item * |
| 412 | |
| 413 | \N, like the period, matches any character but "\n", but it does so |
| 414 | regardless of whether the modifier C<//s> is in effect. |
| 415 | |
| 416 | =back |
| 417 | |
| 418 | The C<//a> modifier, available starting in Perl 5.14, is used to |
| 419 | restrict the matches of \d, \s, and \w to just those in the ASCII range. |
| 420 | It is useful to keep your program from being needlessly exposed to full |
| 421 | Unicode (and its accompanying security considerations) when all you want |
| 422 | is to process English-like text. (The "a" may be doubled, C<//aa>, to |
| 423 | provide even more restrictions, preventing case-insensitive matching of |
| 424 | ASCII with non-ASCII characters; otherwise a Unicode "Kelvin Sign" |
| 425 | would caselessly match a "k" or "K".) |
| 426 | |
| 427 | The C<\d\s\w\D\S\W> abbreviations can be used both inside and outside |
| 428 | of bracketed character classes. Here are some in use: |
| 429 | |
| 430 | /\d\d:\d\d:\d\d/; # matches a hh:mm:ss time format |
| 431 | /[\d\s]/; # matches any digit or whitespace character |
| 432 | /\w\W\w/; # matches a word char, followed by a |
| 433 | # non-word char, followed by a word char |
| 434 | /..rt/; # matches any two chars, followed by 'rt' |
| 435 | /end\./; # matches 'end.' |
| 436 | /end[.]/; # same thing, matches 'end.' |
| 437 | |
| 438 | Because a period is a metacharacter, it needs to be escaped to match |
| 439 | as an ordinary period. Because, for example, C<\d> and C<\w> are sets |
| 440 | of characters, it is incorrect to think of C<[^\d\w]> as C<[\D\W]>; in |
| 441 | fact C<[^\d\w]> is the same as C<[^\w]>, which is the same as |
| 442 | C<[\W]>. Think DeMorgan's laws. |
| 443 | |
| 444 | In actuality, the period and C<\d\s\w\D\S\W> abbreviations are |
| 445 | themselves types of character classes, so the ones surrounded by |
| 446 | brackets are just one type of character class. When we need to make a |
| 447 | distinction, we refer to them as "bracketed character classes." |
| 448 | |
| 449 | An anchor useful in basic regexps is the I<word anchor> |
| 450 | C<\b>. This matches a boundary between a word character and a non-word |
| 451 | character C<\w\W> or C<\W\w>: |
| 452 | |
| 453 | $x = "Housecat catenates house and cat"; |
| 454 | $x =~ /cat/; # matches cat in 'housecat' |
| 455 | $x =~ /\bcat/; # matches cat in 'catenates' |
| 456 | $x =~ /cat\b/; # matches cat in 'housecat' |
| 457 | $x =~ /\bcat\b/; # matches 'cat' at end of string |
| 458 | |
| 459 | Note in the last example, the end of the string is considered a word |
| 460 | boundary. |
| 461 | |
| 462 | You might wonder why C<'.'> matches everything but C<"\n"> - why not |
| 463 | every character? The reason is that often one is matching against |
| 464 | lines and would like to ignore the newline characters. For instance, |
| 465 | while the string C<"\n"> represents one line, we would like to think |
| 466 | of it as empty. Then |
| 467 | |
| 468 | "" =~ /^$/; # matches |
| 469 | "\n" =~ /^$/; # matches, $ anchors before "\n" |
| 470 | |
| 471 | "" =~ /./; # doesn't match; it needs a char |
| 472 | "" =~ /^.$/; # doesn't match; it needs a char |
| 473 | "\n" =~ /^.$/; # doesn't match; it needs a char other than "\n" |
| 474 | "a" =~ /^.$/; # matches |
| 475 | "a\n" =~ /^.$/; # matches, $ anchors before "\n" |
| 476 | |
| 477 | This behavior is convenient, because we usually want to ignore |
| 478 | newlines when we count and match characters in a line. Sometimes, |
| 479 | however, we want to keep track of newlines. We might even want C<^> |
| 480 | and C<$> to anchor at the beginning and end of lines within the |
| 481 | string, rather than just the beginning and end of the string. Perl |
| 482 | allows us to choose between ignoring and paying attention to newlines |
| 483 | by using the C<//s> and C<//m> modifiers. C<//s> and C<//m> stand for |
| 484 | single line and multi-line and they determine whether a string is to |
| 485 | be treated as one continuous string, or as a set of lines. The two |
| 486 | modifiers affect two aspects of how the regexp is interpreted: 1) how |
| 487 | the C<'.'> character class is defined, and 2) where the anchors C<^> |
| 488 | and C<$> are able to match. Here are the four possible combinations: |
| 489 | |
| 490 | =over 4 |
| 491 | |
| 492 | =item * |
| 493 | |
| 494 | no modifiers (//): Default behavior. C<'.'> matches any character |
| 495 | except C<"\n">. C<^> matches only at the beginning of the string and |
| 496 | C<$> matches only at the end or before a newline at the end. |
| 497 | |
| 498 | =item * |
| 499 | |
| 500 | s modifier (//s): Treat string as a single long line. C<'.'> matches |
| 501 | any character, even C<"\n">. C<^> matches only at the beginning of |
| 502 | the string and C<$> matches only at the end or before a newline at the |
| 503 | end. |
| 504 | |
| 505 | =item * |
| 506 | |
| 507 | m modifier (//m): Treat string as a set of multiple lines. C<'.'> |
| 508 | matches any character except C<"\n">. C<^> and C<$> are able to match |
| 509 | at the start or end of I<any> line within the string. |
| 510 | |
| 511 | =item * |
| 512 | |
| 513 | both s and m modifiers (//sm): Treat string as a single long line, but |
| 514 | detect multiple lines. C<'.'> matches any character, even |
| 515 | C<"\n">. C<^> and C<$>, however, are able to match at the start or end |
| 516 | of I<any> line within the string. |
| 517 | |
| 518 | =back |
| 519 | |
| 520 | Here are examples of C<//s> and C<//m> in action: |
| 521 | |
| 522 | $x = "There once was a girl\nWho programmed in Perl\n"; |
| 523 | |
| 524 | $x =~ /^Who/; # doesn't match, "Who" not at start of string |
| 525 | $x =~ /^Who/s; # doesn't match, "Who" not at start of string |
| 526 | $x =~ /^Who/m; # matches, "Who" at start of second line |
| 527 | $x =~ /^Who/sm; # matches, "Who" at start of second line |
| 528 | |
| 529 | $x =~ /girl.Who/; # doesn't match, "." doesn't match "\n" |
| 530 | $x =~ /girl.Who/s; # matches, "." matches "\n" |
| 531 | $x =~ /girl.Who/m; # doesn't match, "." doesn't match "\n" |
| 532 | $x =~ /girl.Who/sm; # matches, "." matches "\n" |
| 533 | |
| 534 | Most of the time, the default behavior is what is wanted, but C<//s> and |
| 535 | C<//m> are occasionally very useful. If C<//m> is being used, the start |
| 536 | of the string can still be matched with C<\A> and the end of the string |
| 537 | can still be matched with the anchors C<\Z> (matches both the end and |
| 538 | the newline before, like C<$>), and C<\z> (matches only the end): |
| 539 | |
| 540 | $x =~ /^Who/m; # matches, "Who" at start of second line |
| 541 | $x =~ /\AWho/m; # doesn't match, "Who" is not at start of string |
| 542 | |
| 543 | $x =~ /girl$/m; # matches, "girl" at end of first line |
| 544 | $x =~ /girl\Z/m; # doesn't match, "girl" is not at end of string |
| 545 | |
| 546 | $x =~ /Perl\Z/m; # matches, "Perl" is at newline before end |
| 547 | $x =~ /Perl\z/m; # doesn't match, "Perl" is not at end of string |
| 548 | |
| 549 | We now know how to create choices among classes of characters in a |
| 550 | regexp. What about choices among words or character strings? Such |
| 551 | choices are described in the next section. |
| 552 | |
| 553 | =head2 Matching this or that |
| 554 | |
| 555 | Sometimes we would like our regexp to be able to match different |
| 556 | possible words or character strings. This is accomplished by using |
| 557 | the I<alternation> metacharacter C<|>. To match C<dog> or C<cat>, we |
| 558 | form the regexp C<dog|cat>. As before, Perl will try to match the |
| 559 | regexp at the earliest possible point in the string. At each |
| 560 | character position, Perl will first try to match the first |
| 561 | alternative, C<dog>. If C<dog> doesn't match, Perl will then try the |
| 562 | next alternative, C<cat>. If C<cat> doesn't match either, then the |
| 563 | match fails and Perl moves to the next position in the string. Some |
| 564 | examples: |
| 565 | |
| 566 | "cats and dogs" =~ /cat|dog|bird/; # matches "cat" |
| 567 | "cats and dogs" =~ /dog|cat|bird/; # matches "cat" |
| 568 | |
| 569 | Even though C<dog> is the first alternative in the second regexp, |
| 570 | C<cat> is able to match earlier in the string. |
| 571 | |
| 572 | "cats" =~ /c|ca|cat|cats/; # matches "c" |
| 573 | "cats" =~ /cats|cat|ca|c/; # matches "cats" |
| 574 | |
| 575 | Here, all the alternatives match at the first string position, so the |
| 576 | first alternative is the one that matches. If some of the |
| 577 | alternatives are truncations of the others, put the longest ones first |
| 578 | to give them a chance to match. |
| 579 | |
| 580 | "cab" =~ /a|b|c/ # matches "c" |
| 581 | # /a|b|c/ == /[abc]/ |
| 582 | |
| 583 | The last example points out that character classes are like |
| 584 | alternations of characters. At a given character position, the first |
| 585 | alternative that allows the regexp match to succeed will be the one |
| 586 | that matches. |
| 587 | |
| 588 | =head2 Grouping things and hierarchical matching |
| 589 | |
| 590 | Alternation allows a regexp to choose among alternatives, but by |
| 591 | itself it is unsatisfying. The reason is that each alternative is a whole |
| 592 | regexp, but sometime we want alternatives for just part of a |
| 593 | regexp. For instance, suppose we want to search for housecats or |
| 594 | housekeepers. The regexp C<housecat|housekeeper> fits the bill, but is |
| 595 | inefficient because we had to type C<house> twice. It would be nice to |
| 596 | have parts of the regexp be constant, like C<house>, and some |
| 597 | parts have alternatives, like C<cat|keeper>. |
| 598 | |
| 599 | The I<grouping> metacharacters C<()> solve this problem. Grouping |
| 600 | allows parts of a regexp to be treated as a single unit. Parts of a |
| 601 | regexp are grouped by enclosing them in parentheses. Thus we could solve |
| 602 | the C<housecat|housekeeper> by forming the regexp as |
| 603 | C<house(cat|keeper)>. The regexp C<house(cat|keeper)> means match |
| 604 | C<house> followed by either C<cat> or C<keeper>. Some more examples |
| 605 | are |
| 606 | |
| 607 | /(a|b)b/; # matches 'ab' or 'bb' |
| 608 | /(ac|b)b/; # matches 'acb' or 'bb' |
| 609 | /(^a|b)c/; # matches 'ac' at start of string or 'bc' anywhere |
| 610 | /(a|[bc])d/; # matches 'ad', 'bd', or 'cd' |
| 611 | |
| 612 | /house(cat|)/; # matches either 'housecat' or 'house' |
| 613 | /house(cat(s|)|)/; # matches either 'housecats' or 'housecat' or |
| 614 | # 'house'. Note groups can be nested. |
| 615 | |
| 616 | /(19|20|)\d\d/; # match years 19xx, 20xx, or the Y2K problem, xx |
| 617 | "20" =~ /(19|20|)\d\d/; # matches the null alternative '()\d\d', |
| 618 | # because '20\d\d' can't match |
| 619 | |
| 620 | Alternations behave the same way in groups as out of them: at a given |
| 621 | string position, the leftmost alternative that allows the regexp to |
| 622 | match is taken. So in the last example at the first string position, |
| 623 | C<"20"> matches the second alternative, but there is nothing left over |
| 624 | to match the next two digits C<\d\d>. So Perl moves on to the next |
| 625 | alternative, which is the null alternative and that works, since |
| 626 | C<"20"> is two digits. |
| 627 | |
| 628 | The process of trying one alternative, seeing if it matches, and |
| 629 | moving on to the next alternative, while going back in the string |
| 630 | from where the previous alternative was tried, if it doesn't, is called |
| 631 | I<backtracking>. The term 'backtracking' comes from the idea that |
| 632 | matching a regexp is like a walk in the woods. Successfully matching |
| 633 | a regexp is like arriving at a destination. There are many possible |
| 634 | trailheads, one for each string position, and each one is tried in |
| 635 | order, left to right. From each trailhead there may be many paths, |
| 636 | some of which get you there, and some which are dead ends. When you |
| 637 | walk along a trail and hit a dead end, you have to backtrack along the |
| 638 | trail to an earlier point to try another trail. If you hit your |
| 639 | destination, you stop immediately and forget about trying all the |
| 640 | other trails. You are persistent, and only if you have tried all the |
| 641 | trails from all the trailheads and not arrived at your destination, do |
| 642 | you declare failure. To be concrete, here is a step-by-step analysis |
| 643 | of what Perl does when it tries to match the regexp |
| 644 | |
| 645 | "abcde" =~ /(abd|abc)(df|d|de)/; |
| 646 | |
| 647 | =over 4 |
| 648 | |
| 649 | =item Z<>0 |
| 650 | |
| 651 | Start with the first letter in the string 'a'. |
| 652 | |
| 653 | =item Z<>1 |
| 654 | |
| 655 | Try the first alternative in the first group 'abd'. |
| 656 | |
| 657 | =item Z<>2 |
| 658 | |
| 659 | Match 'a' followed by 'b'. So far so good. |
| 660 | |
| 661 | =item Z<>3 |
| 662 | |
| 663 | 'd' in the regexp doesn't match 'c' in the string - a dead |
| 664 | end. So backtrack two characters and pick the second alternative in |
| 665 | the first group 'abc'. |
| 666 | |
| 667 | =item Z<>4 |
| 668 | |
| 669 | Match 'a' followed by 'b' followed by 'c'. We are on a roll |
| 670 | and have satisfied the first group. Set $1 to 'abc'. |
| 671 | |
| 672 | =item Z<>5 |
| 673 | |
| 674 | Move on to the second group and pick the first alternative |
| 675 | 'df'. |
| 676 | |
| 677 | =item Z<>6 |
| 678 | |
| 679 | Match the 'd'. |
| 680 | |
| 681 | =item Z<>7 |
| 682 | |
| 683 | 'f' in the regexp doesn't match 'e' in the string, so a dead |
| 684 | end. Backtrack one character and pick the second alternative in the |
| 685 | second group 'd'. |
| 686 | |
| 687 | =item Z<>8 |
| 688 | |
| 689 | 'd' matches. The second grouping is satisfied, so set $2 to |
| 690 | 'd'. |
| 691 | |
| 692 | =item Z<>9 |
| 693 | |
| 694 | We are at the end of the regexp, so we are done! We have |
| 695 | matched 'abcd' out of the string "abcde". |
| 696 | |
| 697 | =back |
| 698 | |
| 699 | There are a couple of things to note about this analysis. First, the |
| 700 | third alternative in the second group 'de' also allows a match, but we |
| 701 | stopped before we got to it - at a given character position, leftmost |
| 702 | wins. Second, we were able to get a match at the first character |
| 703 | position of the string 'a'. If there were no matches at the first |
| 704 | position, Perl would move to the second character position 'b' and |
| 705 | attempt the match all over again. Only when all possible paths at all |
| 706 | possible character positions have been exhausted does Perl give |
| 707 | up and declare S<C<$string =~ /(abd|abc)(df|d|de)/;>> to be false. |
| 708 | |
| 709 | Even with all this work, regexp matching happens remarkably fast. To |
| 710 | speed things up, Perl compiles the regexp into a compact sequence of |
| 711 | opcodes that can often fit inside a processor cache. When the code is |
| 712 | executed, these opcodes can then run at full throttle and search very |
| 713 | quickly. |
| 714 | |
| 715 | =head2 Extracting matches |
| 716 | |
| 717 | The grouping metacharacters C<()> also serve another completely |
| 718 | different function: they allow the extraction of the parts of a string |
| 719 | that matched. This is very useful to find out what matched and for |
| 720 | text processing in general. For each grouping, the part that matched |
| 721 | inside goes into the special variables C<$1>, C<$2>, etc. They can be |
| 722 | used just as ordinary variables: |
| 723 | |
| 724 | # extract hours, minutes, seconds |
| 725 | if ($time =~ /(\d\d):(\d\d):(\d\d)/) { # match hh:mm:ss format |
| 726 | $hours = $1; |
| 727 | $minutes = $2; |
| 728 | $seconds = $3; |
| 729 | } |
| 730 | |
| 731 | Now, we know that in scalar context, |
| 732 | S<C<$time =~ /(\d\d):(\d\d):(\d\d)/>> returns a true or false |
| 733 | value. In list context, however, it returns the list of matched values |
| 734 | C<($1,$2,$3)>. So we could write the code more compactly as |
| 735 | |
| 736 | # extract hours, minutes, seconds |
| 737 | ($hours, $minutes, $second) = ($time =~ /(\d\d):(\d\d):(\d\d)/); |
| 738 | |
| 739 | If the groupings in a regexp are nested, C<$1> gets the group with the |
| 740 | leftmost opening parenthesis, C<$2> the next opening parenthesis, |
| 741 | etc. Here is a regexp with nested groups: |
| 742 | |
| 743 | /(ab(cd|ef)((gi)|j))/; |
| 744 | 1 2 34 |
| 745 | |
| 746 | If this regexp matches, C<$1> contains a string starting with |
| 747 | C<'ab'>, C<$2> is either set to C<'cd'> or C<'ef'>, C<$3> equals either |
| 748 | C<'gi'> or C<'j'>, and C<$4> is either set to C<'gi'>, just like C<$3>, |
| 749 | or it remains undefined. |
| 750 | |
| 751 | For convenience, Perl sets C<$+> to the string held by the highest numbered |
| 752 | C<$1>, C<$2>,... that got assigned (and, somewhat related, C<$^N> to the |
| 753 | value of the C<$1>, C<$2>,... most-recently assigned; i.e. the C<$1>, |
| 754 | C<$2>,... associated with the rightmost closing parenthesis used in the |
| 755 | match). |
| 756 | |
| 757 | |
| 758 | =head2 Backreferences |
| 759 | |
| 760 | Closely associated with the matching variables C<$1>, C<$2>, ... are |
| 761 | the I<backreferences> C<\g1>, C<\g2>,... Backreferences are simply |
| 762 | matching variables that can be used I<inside> a regexp. This is a |
| 763 | really nice feature; what matches later in a regexp is made to depend on |
| 764 | what matched earlier in the regexp. Suppose we wanted to look |
| 765 | for doubled words in a text, like 'the the'. The following regexp finds |
| 766 | all 3-letter doubles with a space in between: |
| 767 | |
| 768 | /\b(\w\w\w)\s\g1\b/; |
| 769 | |
| 770 | The grouping assigns a value to \g1, so that the same 3-letter sequence |
| 771 | is used for both parts. |
| 772 | |
| 773 | A similar task is to find words consisting of two identical parts: |
| 774 | |
| 775 | % simple_grep '^(\w\w\w\w|\w\w\w|\w\w|\w)\g1$' /usr/dict/words |
| 776 | beriberi |
| 777 | booboo |
| 778 | coco |
| 779 | mama |
| 780 | murmur |
| 781 | papa |
| 782 | |
| 783 | The regexp has a single grouping which considers 4-letter |
| 784 | combinations, then 3-letter combinations, etc., and uses C<\g1> to look for |
| 785 | a repeat. Although C<$1> and C<\g1> represent the same thing, care should be |
| 786 | taken to use matched variables C<$1>, C<$2>,... only I<outside> a regexp |
| 787 | and backreferences C<\g1>, C<\g2>,... only I<inside> a regexp; not doing |
| 788 | so may lead to surprising and unsatisfactory results. |
| 789 | |
| 790 | |
| 791 | =head2 Relative backreferences |
| 792 | |
| 793 | Counting the opening parentheses to get the correct number for a |
| 794 | backreference is error-prone as soon as there is more than one |
| 795 | capturing group. A more convenient technique became available |
| 796 | with Perl 5.10: relative backreferences. To refer to the immediately |
| 797 | preceding capture group one now may write C<\g{-1}>, the next but |
| 798 | last is available via C<\g{-2}>, and so on. |
| 799 | |
| 800 | Another good reason in addition to readability and maintainability |
| 801 | for using relative backreferences is illustrated by the following example, |
| 802 | where a simple pattern for matching peculiar strings is used: |
| 803 | |
| 804 | $a99a = '([a-z])(\d)\g2\g1'; # matches a11a, g22g, x33x, etc. |
| 805 | |
| 806 | Now that we have this pattern stored as a handy string, we might feel |
| 807 | tempted to use it as a part of some other pattern: |
| 808 | |
| 809 | $line = "code=e99e"; |
| 810 | if ($line =~ /^(\w+)=$a99a$/){ # unexpected behavior! |
| 811 | print "$1 is valid\n"; |
| 812 | } else { |
| 813 | print "bad line: '$line'\n"; |
| 814 | } |
| 815 | |
| 816 | But this doesn't match, at least not the way one might expect. Only |
| 817 | after inserting the interpolated C<$a99a> and looking at the resulting |
| 818 | full text of the regexp is it obvious that the backreferences have |
| 819 | backfired. The subexpression C<(\w+)> has snatched number 1 and |
| 820 | demoted the groups in C<$a99a> by one rank. This can be avoided by |
| 821 | using relative backreferences: |
| 822 | |
| 823 | $a99a = '([a-z])(\d)\g{-1}\g{-2}'; # safe for being interpolated |
| 824 | |
| 825 | |
| 826 | =head2 Named backreferences |
| 827 | |
| 828 | Perl 5.10 also introduced named capture groups and named backreferences. |
| 829 | To attach a name to a capturing group, you write either |
| 830 | C<< (?<name>...) >> or C<< (?'name'...) >>. The backreference may |
| 831 | then be written as C<\g{name}>. It is permissible to attach the |
| 832 | same name to more than one group, but then only the leftmost one of the |
| 833 | eponymous set can be referenced. Outside of the pattern a named |
| 834 | capture group is accessible through the C<%+> hash. |
| 835 | |
| 836 | Assuming that we have to match calendar dates which may be given in one |
| 837 | of the three formats yyyy-mm-dd, mm/dd/yyyy or dd.mm.yyyy, we can write |
| 838 | three suitable patterns where we use 'd', 'm' and 'y' respectively as the |
| 839 | names of the groups capturing the pertaining components of a date. The |
| 840 | matching operation combines the three patterns as alternatives: |
| 841 | |
| 842 | $fmt1 = '(?<y>\d\d\d\d)-(?<m>\d\d)-(?<d>\d\d)'; |
| 843 | $fmt2 = '(?<m>\d\d)/(?<d>\d\d)/(?<y>\d\d\d\d)'; |
| 844 | $fmt3 = '(?<d>\d\d)\.(?<m>\d\d)\.(?<y>\d\d\d\d)'; |
| 845 | for my $d qw( 2006-10-21 15.01.2007 10/31/2005 ){ |
| 846 | if ( $d =~ m{$fmt1|$fmt2|$fmt3} ){ |
| 847 | print "day=$+{d} month=$+{m} year=$+{y}\n"; |
| 848 | } |
| 849 | } |
| 850 | |
| 851 | If any of the alternatives matches, the hash C<%+> is bound to contain the |
| 852 | three key-value pairs. |
| 853 | |
| 854 | |
| 855 | =head2 Alternative capture group numbering |
| 856 | |
| 857 | Yet another capturing group numbering technique (also as from Perl 5.10) |
| 858 | deals with the problem of referring to groups within a set of alternatives. |
| 859 | Consider a pattern for matching a time of the day, civil or military style: |
| 860 | |
| 861 | if ( $time =~ /(\d\d|\d):(\d\d)|(\d\d)(\d\d)/ ){ |
| 862 | # process hour and minute |
| 863 | } |
| 864 | |
| 865 | Processing the results requires an additional if statement to determine |
| 866 | whether C<$1> and C<$2> or C<$3> and C<$4> contain the goodies. It would |
| 867 | be easier if we could use group numbers 1 and 2 in second alternative as |
| 868 | well, and this is exactly what the parenthesized construct C<(?|...)>, |
| 869 | set around an alternative achieves. Here is an extended version of the |
| 870 | previous pattern: |
| 871 | |
| 872 | if($time =~ /(?|(\d\d|\d):(\d\d)|(\d\d)(\d\d))\s+([A-Z][A-Z][A-Z])/){ |
| 873 | print "hour=$1 minute=$2 zone=$3\n"; |
| 874 | } |
| 875 | |
| 876 | Within the alternative numbering group, group numbers start at the same |
| 877 | position for each alternative. After the group, numbering continues |
| 878 | with one higher than the maximum reached across all the alternatives. |
| 879 | |
| 880 | =head2 Position information |
| 881 | |
| 882 | In addition to what was matched, Perl also provides the |
| 883 | positions of what was matched as contents of the C<@-> and C<@+> |
| 884 | arrays. C<$-[0]> is the position of the start of the entire match and |
| 885 | C<$+[0]> is the position of the end. Similarly, C<$-[n]> is the |
| 886 | position of the start of the C<$n> match and C<$+[n]> is the position |
| 887 | of the end. If C<$n> is undefined, so are C<$-[n]> and C<$+[n]>. Then |
| 888 | this code |
| 889 | |
| 890 | $x = "Mmm...donut, thought Homer"; |
| 891 | $x =~ /^(Mmm|Yech)\.\.\.(donut|peas)/; # matches |
| 892 | foreach $exp (1..$#-) { |
| 893 | print "Match $exp: '${$exp}' at position ($-[$exp],$+[$exp])\n"; |
| 894 | } |
| 895 | |
| 896 | prints |
| 897 | |
| 898 | Match 1: 'Mmm' at position (0,3) |
| 899 | Match 2: 'donut' at position (6,11) |
| 900 | |
| 901 | Even if there are no groupings in a regexp, it is still possible to |
| 902 | find out what exactly matched in a string. If you use them, Perl |
| 903 | will set C<$`> to the part of the string before the match, will set C<$&> |
| 904 | to the part of the string that matched, and will set C<$'> to the part |
| 905 | of the string after the match. An example: |
| 906 | |
| 907 | $x = "the cat caught the mouse"; |
| 908 | $x =~ /cat/; # $` = 'the ', $& = 'cat', $' = ' caught the mouse' |
| 909 | $x =~ /the/; # $` = '', $& = 'the', $' = ' cat caught the mouse' |
| 910 | |
| 911 | In the second match, C<$`> equals C<''> because the regexp matched at the |
| 912 | first character position in the string and stopped; it never saw the |
| 913 | second 'the'. |
| 914 | |
| 915 | If your code is to run on Perl versions earlier than |
| 916 | 5.20, it is worthwhile to note that using C<$`> and C<$'> |
| 917 | slows down regexp matching quite a bit, while C<$&> slows it down to a |
| 918 | lesser extent, because if they are used in one regexp in a program, |
| 919 | they are generated for I<all> regexps in the program. So if raw |
| 920 | performance is a goal of your application, they should be avoided. |
| 921 | If you need to extract the corresponding substrings, use C<@-> and |
| 922 | C<@+> instead: |
| 923 | |
| 924 | $` is the same as substr( $x, 0, $-[0] ) |
| 925 | $& is the same as substr( $x, $-[0], $+[0]-$-[0] ) |
| 926 | $' is the same as substr( $x, $+[0] ) |
| 927 | |
| 928 | As of Perl 5.10, the C<${^PREMATCH}>, C<${^MATCH}> and C<${^POSTMATCH}> |
| 929 | variables may be used. These are only set if the C</p> modifier is |
| 930 | present. Consequently they do not penalize the rest of the program. In |
| 931 | Perl 5.20, C<${^PREMATCH}>, C<${^MATCH}> and C<${^POSTMATCH}> are available |
| 932 | whether the C</p> has been used or not (the modifier is ignored), and |
| 933 | C<$`>, C<$'> and C<$&> do not cause any speed difference. |
| 934 | |
| 935 | =head2 Non-capturing groupings |
| 936 | |
| 937 | A group that is required to bundle a set of alternatives may or may not be |
| 938 | useful as a capturing group. If it isn't, it just creates a superfluous |
| 939 | addition to the set of available capture group values, inside as well as |
| 940 | outside the regexp. Non-capturing groupings, denoted by C<(?:regexp)>, |
| 941 | still allow the regexp to be treated as a single unit, but don't establish |
| 942 | a capturing group at the same time. Both capturing and non-capturing |
| 943 | groupings are allowed to co-exist in the same regexp. Because there is |
| 944 | no extraction, non-capturing groupings are faster than capturing |
| 945 | groupings. Non-capturing groupings are also handy for choosing exactly |
| 946 | which parts of a regexp are to be extracted to matching variables: |
| 947 | |
| 948 | # match a number, $1-$4 are set, but we only want $1 |
| 949 | /([+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?)/; |
| 950 | |
| 951 | # match a number faster , only $1 is set |
| 952 | /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)/; |
| 953 | |
| 954 | # match a number, get $1 = whole number, $2 = exponent |
| 955 | /([+-]?\ *(?:\d+(?:\.\d*)?|\.\d+)(?:[eE]([+-]?\d+))?)/; |
| 956 | |
| 957 | Non-capturing groupings are also useful for removing nuisance |
| 958 | elements gathered from a split operation where parentheses are |
| 959 | required for some reason: |
| 960 | |
| 961 | $x = '12aba34ba5'; |
| 962 | @num = split /(a|b)+/, $x; # @num = ('12','a','34','a','5') |
| 963 | @num = split /(?:a|b)+/, $x; # @num = ('12','34','5') |
| 964 | |
| 965 | In Perl 5.22 and later, all groups within a regexp can be set to |
| 966 | non-capturing by using the new C</n> flag: |
| 967 | |
| 968 | "hello" =~ /(hi|hello)/n; # $1 is not set! |
| 969 | |
| 970 | See L<perlre/"n"> for more information. |
| 971 | |
| 972 | =head2 Matching repetitions |
| 973 | |
| 974 | The examples in the previous section display an annoying weakness. We |
| 975 | were only matching 3-letter words, or chunks of words of 4 letters or |
| 976 | less. We'd like to be able to match words or, more generally, strings |
| 977 | of any length, without writing out tedious alternatives like |
| 978 | C<\w\w\w\w|\w\w\w|\w\w|\w>. |
| 979 | |
| 980 | This is exactly the problem the I<quantifier> metacharacters C<?>, |
| 981 | C<*>, C<+>, and C<{}> were created for. They allow us to delimit the |
| 982 | number of repeats for a portion of a regexp we consider to be a |
| 983 | match. Quantifiers are put immediately after the character, character |
| 984 | class, or grouping that we want to specify. They have the following |
| 985 | meanings: |
| 986 | |
| 987 | =over 4 |
| 988 | |
| 989 | =item * |
| 990 | |
| 991 | C<a?> means: match 'a' 1 or 0 times |
| 992 | |
| 993 | =item * |
| 994 | |
| 995 | C<a*> means: match 'a' 0 or more times, i.e., any number of times |
| 996 | |
| 997 | =item * |
| 998 | |
| 999 | C<a+> means: match 'a' 1 or more times, i.e., at least once |
| 1000 | |
| 1001 | =item * |
| 1002 | |
| 1003 | C<a{n,m}> means: match at least C<n> times, but not more than C<m> |
| 1004 | times. |
| 1005 | |
| 1006 | =item * |
| 1007 | |
| 1008 | C<a{n,}> means: match at least C<n> or more times |
| 1009 | |
| 1010 | =item * |
| 1011 | |
| 1012 | C<a{n}> means: match exactly C<n> times |
| 1013 | |
| 1014 | =back |
| 1015 | |
| 1016 | Here are some examples: |
| 1017 | |
| 1018 | /[a-z]+\s+\d*/; # match a lowercase word, at least one space, and |
| 1019 | # any number of digits |
| 1020 | /(\w+)\s+\g1/; # match doubled words of arbitrary length |
| 1021 | /y(es)?/i; # matches 'y', 'Y', or a case-insensitive 'yes' |
| 1022 | $year =~ /^\d{2,4}$/; # make sure year is at least 2 but not more |
| 1023 | # than 4 digits |
| 1024 | $year =~ /^\d{4}$|^\d{2}$/; # better match; throw out 3-digit dates |
| 1025 | $year =~ /^\d{2}(\d{2})?$/; # same thing written differently. |
| 1026 | # However, this captures the last two |
| 1027 | # digits in $1 and the other does not. |
| 1028 | |
| 1029 | % simple_grep '^(\w+)\g1$' /usr/dict/words # isn't this easier? |
| 1030 | beriberi |
| 1031 | booboo |
| 1032 | coco |
| 1033 | mama |
| 1034 | murmur |
| 1035 | papa |
| 1036 | |
| 1037 | For all of these quantifiers, Perl will try to match as much of the |
| 1038 | string as possible, while still allowing the regexp to succeed. Thus |
| 1039 | with C</a?.../>, Perl will first try to match the regexp with the C<a> |
| 1040 | present; if that fails, Perl will try to match the regexp without the |
| 1041 | C<a> present. For the quantifier C<*>, we get the following: |
| 1042 | |
| 1043 | $x = "the cat in the hat"; |
| 1044 | $x =~ /^(.*)(cat)(.*)$/; # matches, |
| 1045 | # $1 = 'the ' |
| 1046 | # $2 = 'cat' |
| 1047 | # $3 = ' in the hat' |
| 1048 | |
| 1049 | Which is what we might expect, the match finds the only C<cat> in the |
| 1050 | string and locks onto it. Consider, however, this regexp: |
| 1051 | |
| 1052 | $x =~ /^(.*)(at)(.*)$/; # matches, |
| 1053 | # $1 = 'the cat in the h' |
| 1054 | # $2 = 'at' |
| 1055 | # $3 = '' (0 characters match) |
| 1056 | |
| 1057 | One might initially guess that Perl would find the C<at> in C<cat> and |
| 1058 | stop there, but that wouldn't give the longest possible string to the |
| 1059 | first quantifier C<.*>. Instead, the first quantifier C<.*> grabs as |
| 1060 | much of the string as possible while still having the regexp match. In |
| 1061 | this example, that means having the C<at> sequence with the final C<at> |
| 1062 | in the string. The other important principle illustrated here is that, |
| 1063 | when there are two or more elements in a regexp, the I<leftmost> |
| 1064 | quantifier, if there is one, gets to grab as much of the string as |
| 1065 | possible, leaving the rest of the regexp to fight over scraps. Thus in |
| 1066 | our example, the first quantifier C<.*> grabs most of the string, while |
| 1067 | the second quantifier C<.*> gets the empty string. Quantifiers that |
| 1068 | grab as much of the string as possible are called I<maximal match> or |
| 1069 | I<greedy> quantifiers. |
| 1070 | |
| 1071 | When a regexp can match a string in several different ways, we can use |
| 1072 | the principles above to predict which way the regexp will match: |
| 1073 | |
| 1074 | =over 4 |
| 1075 | |
| 1076 | =item * |
| 1077 | |
| 1078 | Principle 0: Taken as a whole, any regexp will be matched at the |
| 1079 | earliest possible position in the string. |
| 1080 | |
| 1081 | =item * |
| 1082 | |
| 1083 | Principle 1: In an alternation C<a|b|c...>, the leftmost alternative |
| 1084 | that allows a match for the whole regexp will be the one used. |
| 1085 | |
| 1086 | =item * |
| 1087 | |
| 1088 | Principle 2: The maximal matching quantifiers C<?>, C<*>, C<+> and |
| 1089 | C<{n,m}> will in general match as much of the string as possible while |
| 1090 | still allowing the whole regexp to match. |
| 1091 | |
| 1092 | =item * |
| 1093 | |
| 1094 | Principle 3: If there are two or more elements in a regexp, the |
| 1095 | leftmost greedy quantifier, if any, will match as much of the string |
| 1096 | as possible while still allowing the whole regexp to match. The next |
| 1097 | leftmost greedy quantifier, if any, will try to match as much of the |
| 1098 | string remaining available to it as possible, while still allowing the |
| 1099 | whole regexp to match. And so on, until all the regexp elements are |
| 1100 | satisfied. |
| 1101 | |
| 1102 | =back |
| 1103 | |
| 1104 | As we have seen above, Principle 0 overrides the others. The regexp |
| 1105 | will be matched as early as possible, with the other principles |
| 1106 | determining how the regexp matches at that earliest character |
| 1107 | position. |
| 1108 | |
| 1109 | Here is an example of these principles in action: |
| 1110 | |
| 1111 | $x = "The programming republic of Perl"; |
| 1112 | $x =~ /^(.+)(e|r)(.*)$/; # matches, |
| 1113 | # $1 = 'The programming republic of Pe' |
| 1114 | # $2 = 'r' |
| 1115 | # $3 = 'l' |
| 1116 | |
| 1117 | This regexp matches at the earliest string position, C<'T'>. One |
| 1118 | might think that C<e>, being leftmost in the alternation, would be |
| 1119 | matched, but C<r> produces the longest string in the first quantifier. |
| 1120 | |
| 1121 | $x =~ /(m{1,2})(.*)$/; # matches, |
| 1122 | # $1 = 'mm' |
| 1123 | # $2 = 'ing republic of Perl' |
| 1124 | |
| 1125 | Here, The earliest possible match is at the first C<'m'> in |
| 1126 | C<programming>. C<m{1,2}> is the first quantifier, so it gets to match |
| 1127 | a maximal C<mm>. |
| 1128 | |
| 1129 | $x =~ /.*(m{1,2})(.*)$/; # matches, |
| 1130 | # $1 = 'm' |
| 1131 | # $2 = 'ing republic of Perl' |
| 1132 | |
| 1133 | Here, the regexp matches at the start of the string. The first |
| 1134 | quantifier C<.*> grabs as much as possible, leaving just a single |
| 1135 | C<'m'> for the second quantifier C<m{1,2}>. |
| 1136 | |
| 1137 | $x =~ /(.?)(m{1,2})(.*)$/; # matches, |
| 1138 | # $1 = 'a' |
| 1139 | # $2 = 'mm' |
| 1140 | # $3 = 'ing republic of Perl' |
| 1141 | |
| 1142 | Here, C<.?> eats its maximal one character at the earliest possible |
| 1143 | position in the string, C<'a'> in C<programming>, leaving C<m{1,2}> |
| 1144 | the opportunity to match both C<m>'s. Finally, |
| 1145 | |
| 1146 | "aXXXb" =~ /(X*)/; # matches with $1 = '' |
| 1147 | |
| 1148 | because it can match zero copies of C<'X'> at the beginning of the |
| 1149 | string. If you definitely want to match at least one C<'X'>, use |
| 1150 | C<X+>, not C<X*>. |
| 1151 | |
| 1152 | Sometimes greed is not good. At times, we would like quantifiers to |
| 1153 | match a I<minimal> piece of string, rather than a maximal piece. For |
| 1154 | this purpose, Larry Wall created the I<minimal match> or |
| 1155 | I<non-greedy> quantifiers C<??>, C<*?>, C<+?>, and C<{}?>. These are |
| 1156 | the usual quantifiers with a C<?> appended to them. They have the |
| 1157 | following meanings: |
| 1158 | |
| 1159 | =over 4 |
| 1160 | |
| 1161 | =item * |
| 1162 | |
| 1163 | C<a??> means: match 'a' 0 or 1 times. Try 0 first, then 1. |
| 1164 | |
| 1165 | =item * |
| 1166 | |
| 1167 | C<a*?> means: match 'a' 0 or more times, i.e., any number of times, |
| 1168 | but as few times as possible |
| 1169 | |
| 1170 | =item * |
| 1171 | |
| 1172 | C<a+?> means: match 'a' 1 or more times, i.e., at least once, but |
| 1173 | as few times as possible |
| 1174 | |
| 1175 | =item * |
| 1176 | |
| 1177 | C<a{n,m}?> means: match at least C<n> times, not more than C<m> |
| 1178 | times, as few times as possible |
| 1179 | |
| 1180 | =item * |
| 1181 | |
| 1182 | C<a{n,}?> means: match at least C<n> times, but as few times as |
| 1183 | possible |
| 1184 | |
| 1185 | =item * |
| 1186 | |
| 1187 | C<a{n}?> means: match exactly C<n> times. Because we match exactly |
| 1188 | C<n> times, C<a{n}?> is equivalent to C<a{n}> and is just there for |
| 1189 | notational consistency. |
| 1190 | |
| 1191 | =back |
| 1192 | |
| 1193 | Let's look at the example above, but with minimal quantifiers: |
| 1194 | |
| 1195 | $x = "The programming republic of Perl"; |
| 1196 | $x =~ /^(.+?)(e|r)(.*)$/; # matches, |
| 1197 | # $1 = 'Th' |
| 1198 | # $2 = 'e' |
| 1199 | # $3 = ' programming republic of Perl' |
| 1200 | |
| 1201 | The minimal string that will allow both the start of the string C<^> |
| 1202 | and the alternation to match is C<Th>, with the alternation C<e|r> |
| 1203 | matching C<e>. The second quantifier C<.*> is free to gobble up the |
| 1204 | rest of the string. |
| 1205 | |
| 1206 | $x =~ /(m{1,2}?)(.*?)$/; # matches, |
| 1207 | # $1 = 'm' |
| 1208 | # $2 = 'ming republic of Perl' |
| 1209 | |
| 1210 | The first string position that this regexp can match is at the first |
| 1211 | C<'m'> in C<programming>. At this position, the minimal C<m{1,2}?> |
| 1212 | matches just one C<'m'>. Although the second quantifier C<.*?> would |
| 1213 | prefer to match no characters, it is constrained by the end-of-string |
| 1214 | anchor C<$> to match the rest of the string. |
| 1215 | |
| 1216 | $x =~ /(.*?)(m{1,2}?)(.*)$/; # matches, |
| 1217 | # $1 = 'The progra' |
| 1218 | # $2 = 'm' |
| 1219 | # $3 = 'ming republic of Perl' |
| 1220 | |
| 1221 | In this regexp, you might expect the first minimal quantifier C<.*?> |
| 1222 | to match the empty string, because it is not constrained by a C<^> |
| 1223 | anchor to match the beginning of the word. Principle 0 applies here, |
| 1224 | however. Because it is possible for the whole regexp to match at the |
| 1225 | start of the string, it I<will> match at the start of the string. Thus |
| 1226 | the first quantifier has to match everything up to the first C<m>. The |
| 1227 | second minimal quantifier matches just one C<m> and the third |
| 1228 | quantifier matches the rest of the string. |
| 1229 | |
| 1230 | $x =~ /(.??)(m{1,2})(.*)$/; # matches, |
| 1231 | # $1 = 'a' |
| 1232 | # $2 = 'mm' |
| 1233 | # $3 = 'ing republic of Perl' |
| 1234 | |
| 1235 | Just as in the previous regexp, the first quantifier C<.??> can match |
| 1236 | earliest at position C<'a'>, so it does. The second quantifier is |
| 1237 | greedy, so it matches C<mm>, and the third matches the rest of the |
| 1238 | string. |
| 1239 | |
| 1240 | We can modify principle 3 above to take into account non-greedy |
| 1241 | quantifiers: |
| 1242 | |
| 1243 | =over 4 |
| 1244 | |
| 1245 | =item * |
| 1246 | |
| 1247 | Principle 3: If there are two or more elements in a regexp, the |
| 1248 | leftmost greedy (non-greedy) quantifier, if any, will match as much |
| 1249 | (little) of the string as possible while still allowing the whole |
| 1250 | regexp to match. The next leftmost greedy (non-greedy) quantifier, if |
| 1251 | any, will try to match as much (little) of the string remaining |
| 1252 | available to it as possible, while still allowing the whole regexp to |
| 1253 | match. And so on, until all the regexp elements are satisfied. |
| 1254 | |
| 1255 | =back |
| 1256 | |
| 1257 | Just like alternation, quantifiers are also susceptible to |
| 1258 | backtracking. Here is a step-by-step analysis of the example |
| 1259 | |
| 1260 | $x = "the cat in the hat"; |
| 1261 | $x =~ /^(.*)(at)(.*)$/; # matches, |
| 1262 | # $1 = 'the cat in the h' |
| 1263 | # $2 = 'at' |
| 1264 | # $3 = '' (0 matches) |
| 1265 | |
| 1266 | =over 4 |
| 1267 | |
| 1268 | =item Z<>0 |
| 1269 | |
| 1270 | Start with the first letter in the string 't'. |
| 1271 | |
| 1272 | =item Z<>1 |
| 1273 | |
| 1274 | The first quantifier '.*' starts out by matching the whole |
| 1275 | string 'the cat in the hat'. |
| 1276 | |
| 1277 | =item Z<>2 |
| 1278 | |
| 1279 | 'a' in the regexp element 'at' doesn't match the end of the |
| 1280 | string. Backtrack one character. |
| 1281 | |
| 1282 | =item Z<>3 |
| 1283 | |
| 1284 | 'a' in the regexp element 'at' still doesn't match the last |
| 1285 | letter of the string 't', so backtrack one more character. |
| 1286 | |
| 1287 | =item Z<>4 |
| 1288 | |
| 1289 | Now we can match the 'a' and the 't'. |
| 1290 | |
| 1291 | =item Z<>5 |
| 1292 | |
| 1293 | Move on to the third element '.*'. Since we are at the end of |
| 1294 | the string and '.*' can match 0 times, assign it the empty string. |
| 1295 | |
| 1296 | =item Z<>6 |
| 1297 | |
| 1298 | We are done! |
| 1299 | |
| 1300 | =back |
| 1301 | |
| 1302 | Most of the time, all this moving forward and backtracking happens |
| 1303 | quickly and searching is fast. There are some pathological regexps, |
| 1304 | however, whose execution time exponentially grows with the size of the |
| 1305 | string. A typical structure that blows up in your face is of the form |
| 1306 | |
| 1307 | /(a|b+)*/; |
| 1308 | |
| 1309 | The problem is the nested indeterminate quantifiers. There are many |
| 1310 | different ways of partitioning a string of length n between the C<+> |
| 1311 | and C<*>: one repetition with C<b+> of length n, two repetitions with |
| 1312 | the first C<b+> length k and the second with length n-k, m repetitions |
| 1313 | whose bits add up to length n, etc. In fact there are an exponential |
| 1314 | number of ways to partition a string as a function of its length. A |
| 1315 | regexp may get lucky and match early in the process, but if there is |
| 1316 | no match, Perl will try I<every> possibility before giving up. So be |
| 1317 | careful with nested C<*>'s, C<{n,m}>'s, and C<+>'s. The book |
| 1318 | I<Mastering Regular Expressions> by Jeffrey Friedl gives a wonderful |
| 1319 | discussion of this and other efficiency issues. |
| 1320 | |
| 1321 | |
| 1322 | =head2 Possessive quantifiers |
| 1323 | |
| 1324 | Backtracking during the relentless search for a match may be a waste |
| 1325 | of time, particularly when the match is bound to fail. Consider |
| 1326 | the simple pattern |
| 1327 | |
| 1328 | /^\w+\s+\w+$/; # a word, spaces, a word |
| 1329 | |
| 1330 | Whenever this is applied to a string which doesn't quite meet the |
| 1331 | pattern's expectations such as S<C<"abc ">> or S<C<"abc def ">>, |
| 1332 | the regex engine will backtrack, approximately once for each character |
| 1333 | in the string. But we know that there is no way around taking I<all> |
| 1334 | of the initial word characters to match the first repetition, that I<all> |
| 1335 | spaces must be eaten by the middle part, and the same goes for the second |
| 1336 | word. |
| 1337 | |
| 1338 | With the introduction of the I<possessive quantifiers> in Perl 5.10, we |
| 1339 | have a way of instructing the regex engine not to backtrack, with the |
| 1340 | usual quantifiers with a C<+> appended to them. This makes them greedy as |
| 1341 | well as stingy; once they succeed they won't give anything back to permit |
| 1342 | another solution. They have the following meanings: |
| 1343 | |
| 1344 | =over 4 |
| 1345 | |
| 1346 | =item * |
| 1347 | |
| 1348 | C<a{n,m}+> means: match at least C<n> times, not more than C<m> times, |
| 1349 | as many times as possible, and don't give anything up. C<a?+> is short |
| 1350 | for C<a{0,1}+> |
| 1351 | |
| 1352 | =item * |
| 1353 | |
| 1354 | C<a{n,}+> means: match at least C<n> times, but as many times as possible, |
| 1355 | and don't give anything up. C<a*+> is short for C<a{0,}+> and C<a++> is |
| 1356 | short for C<a{1,}+>. |
| 1357 | |
| 1358 | =item * |
| 1359 | |
| 1360 | C<a{n}+> means: match exactly C<n> times. It is just there for |
| 1361 | notational consistency. |
| 1362 | |
| 1363 | =back |
| 1364 | |
| 1365 | These possessive quantifiers represent a special case of a more general |
| 1366 | concept, the I<independent subexpression>, see below. |
| 1367 | |
| 1368 | As an example where a possessive quantifier is suitable we consider |
| 1369 | matching a quoted string, as it appears in several programming languages. |
| 1370 | The backslash is used as an escape character that indicates that the |
| 1371 | next character is to be taken literally, as another character for the |
| 1372 | string. Therefore, after the opening quote, we expect a (possibly |
| 1373 | empty) sequence of alternatives: either some character except an |
| 1374 | unescaped quote or backslash or an escaped character. |
| 1375 | |
| 1376 | /"(?:[^"\\]++|\\.)*+"/; |
| 1377 | |
| 1378 | |
| 1379 | =head2 Building a regexp |
| 1380 | |
| 1381 | At this point, we have all the basic regexp concepts covered, so let's |
| 1382 | give a more involved example of a regular expression. We will build a |
| 1383 | regexp that matches numbers. |
| 1384 | |
| 1385 | The first task in building a regexp is to decide what we want to match |
| 1386 | and what we want to exclude. In our case, we want to match both |
| 1387 | integers and floating point numbers and we want to reject any string |
| 1388 | that isn't a number. |
| 1389 | |
| 1390 | The next task is to break the problem down into smaller problems that |
| 1391 | are easily converted into a regexp. |
| 1392 | |
| 1393 | The simplest case is integers. These consist of a sequence of digits, |
| 1394 | with an optional sign in front. The digits we can represent with |
| 1395 | C<\d+> and the sign can be matched with C<[+-]>. Thus the integer |
| 1396 | regexp is |
| 1397 | |
| 1398 | /[+-]?\d+/; # matches integers |
| 1399 | |
| 1400 | A floating point number potentially has a sign, an integral part, a |
| 1401 | decimal point, a fractional part, and an exponent. One or more of these |
| 1402 | parts is optional, so we need to check out the different |
| 1403 | possibilities. Floating point numbers which are in proper form include |
| 1404 | 123., 0.345, .34, -1e6, and 25.4E-72. As with integers, the sign out |
| 1405 | front is completely optional and can be matched by C<[+-]?>. We can |
| 1406 | see that if there is no exponent, floating point numbers must have a |
| 1407 | decimal point, otherwise they are integers. We might be tempted to |
| 1408 | model these with C<\d*\.\d*>, but this would also match just a single |
| 1409 | decimal point, which is not a number. So the three cases of floating |
| 1410 | point number without exponent are |
| 1411 | |
| 1412 | /[+-]?\d+\./; # 1., 321., etc. |
| 1413 | /[+-]?\.\d+/; # .1, .234, etc. |
| 1414 | /[+-]?\d+\.\d+/; # 1.0, 30.56, etc. |
| 1415 | |
| 1416 | These can be combined into a single regexp with a three-way alternation: |
| 1417 | |
| 1418 | /[+-]?(\d+\.\d+|\d+\.|\.\d+)/; # floating point, no exponent |
| 1419 | |
| 1420 | In this alternation, it is important to put C<'\d+\.\d+'> before |
| 1421 | C<'\d+\.'>. If C<'\d+\.'> were first, the regexp would happily match that |
| 1422 | and ignore the fractional part of the number. |
| 1423 | |
| 1424 | Now consider floating point numbers with exponents. The key |
| 1425 | observation here is that I<both> integers and numbers with decimal |
| 1426 | points are allowed in front of an exponent. Then exponents, like the |
| 1427 | overall sign, are independent of whether we are matching numbers with |
| 1428 | or without decimal points, and can be 'decoupled' from the |
| 1429 | mantissa. The overall form of the regexp now becomes clear: |
| 1430 | |
| 1431 | /^(optional sign)(integer | f.p. mantissa)(optional exponent)$/; |
| 1432 | |
| 1433 | The exponent is an C<e> or C<E>, followed by an integer. So the |
| 1434 | exponent regexp is |
| 1435 | |
| 1436 | /[eE][+-]?\d+/; # exponent |
| 1437 | |
| 1438 | Putting all the parts together, we get a regexp that matches numbers: |
| 1439 | |
| 1440 | /^[+-]?(\d+\.\d+|\d+\.|\.\d+|\d+)([eE][+-]?\d+)?$/; # Ta da! |
| 1441 | |
| 1442 | Long regexps like this may impress your friends, but can be hard to |
| 1443 | decipher. In complex situations like this, the C<//x> modifier for a |
| 1444 | match is invaluable. It allows one to put nearly arbitrary whitespace |
| 1445 | and comments into a regexp without affecting their meaning. Using it, |
| 1446 | we can rewrite our 'extended' regexp in the more pleasing form |
| 1447 | |
| 1448 | /^ |
| 1449 | [+-]? # first, match an optional sign |
| 1450 | ( # then match integers or f.p. mantissas: |
| 1451 | \d+\.\d+ # mantissa of the form a.b |
| 1452 | |\d+\. # mantissa of the form a. |
| 1453 | |\.\d+ # mantissa of the form .b |
| 1454 | |\d+ # integer of the form a |
| 1455 | ) |
| 1456 | ([eE][+-]?\d+)? # finally, optionally match an exponent |
| 1457 | $/x; |
| 1458 | |
| 1459 | If whitespace is mostly irrelevant, how does one include space |
| 1460 | characters in an extended regexp? The answer is to backslash it |
| 1461 | S<C<'\ '>> or put it in a character class S<C<[ ]>>. The same thing |
| 1462 | goes for pound signs: use C<\#> or C<[#]>. For instance, Perl allows |
| 1463 | a space between the sign and the mantissa or integer, and we could add |
| 1464 | this to our regexp as follows: |
| 1465 | |
| 1466 | /^ |
| 1467 | [+-]?\ * # first, match an optional sign *and space* |
| 1468 | ( # then match integers or f.p. mantissas: |
| 1469 | \d+\.\d+ # mantissa of the form a.b |
| 1470 | |\d+\. # mantissa of the form a. |
| 1471 | |\.\d+ # mantissa of the form .b |
| 1472 | |\d+ # integer of the form a |
| 1473 | ) |
| 1474 | ([eE][+-]?\d+)? # finally, optionally match an exponent |
| 1475 | $/x; |
| 1476 | |
| 1477 | In this form, it is easier to see a way to simplify the |
| 1478 | alternation. Alternatives 1, 2, and 4 all start with C<\d+>, so it |
| 1479 | could be factored out: |
| 1480 | |
| 1481 | /^ |
| 1482 | [+-]?\ * # first, match an optional sign |
| 1483 | ( # then match integers or f.p. mantissas: |
| 1484 | \d+ # start out with a ... |
| 1485 | ( |
| 1486 | \.\d* # mantissa of the form a.b or a. |
| 1487 | )? # ? takes care of integers of the form a |
| 1488 | |\.\d+ # mantissa of the form .b |
| 1489 | ) |
| 1490 | ([eE][+-]?\d+)? # finally, optionally match an exponent |
| 1491 | $/x; |
| 1492 | |
| 1493 | or written in the compact form, |
| 1494 | |
| 1495 | /^[+-]?\ *(\d+(\.\d*)?|\.\d+)([eE][+-]?\d+)?$/; |
| 1496 | |
| 1497 | This is our final regexp. To recap, we built a regexp by |
| 1498 | |
| 1499 | =over 4 |
| 1500 | |
| 1501 | =item * |
| 1502 | |
| 1503 | specifying the task in detail, |
| 1504 | |
| 1505 | =item * |
| 1506 | |
| 1507 | breaking down the problem into smaller parts, |
| 1508 | |
| 1509 | =item * |
| 1510 | |
| 1511 | translating the small parts into regexps, |
| 1512 | |
| 1513 | =item * |
| 1514 | |
| 1515 | combining the regexps, |
| 1516 | |
| 1517 | =item * |
| 1518 | |
| 1519 | and optimizing the final combined regexp. |
| 1520 | |
| 1521 | =back |
| 1522 | |
| 1523 | These are also the typical steps involved in writing a computer |
| 1524 | program. This makes perfect sense, because regular expressions are |
| 1525 | essentially programs written in a little computer language that specifies |
| 1526 | patterns. |
| 1527 | |
| 1528 | =head2 Using regular expressions in Perl |
| 1529 | |
| 1530 | The last topic of Part 1 briefly covers how regexps are used in Perl |
| 1531 | programs. Where do they fit into Perl syntax? |
| 1532 | |
| 1533 | We have already introduced the matching operator in its default |
| 1534 | C</regexp/> and arbitrary delimiter C<m!regexp!> forms. We have used |
| 1535 | the binding operator C<=~> and its negation C<!~> to test for string |
| 1536 | matches. Associated with the matching operator, we have discussed the |
| 1537 | single line C<//s>, multi-line C<//m>, case-insensitive C<//i> and |
| 1538 | extended C<//x> modifiers. There are a few more things you might |
| 1539 | want to know about matching operators. |
| 1540 | |
| 1541 | =head3 Prohibiting substitution |
| 1542 | |
| 1543 | If you change C<$pattern> after the first substitution happens, Perl |
| 1544 | will ignore it. If you don't want any substitutions at all, use the |
| 1545 | special delimiter C<m''>: |
| 1546 | |
| 1547 | @pattern = ('Seuss'); |
| 1548 | while (<>) { |
| 1549 | print if m'@pattern'; # matches literal '@pattern', not 'Seuss' |
| 1550 | } |
| 1551 | |
| 1552 | Similar to strings, C<m''> acts like apostrophes on a regexp; all other |
| 1553 | C<m> delimiters act like quotes. If the regexp evaluates to the empty string, |
| 1554 | the regexp in the I<last successful match> is used instead. So we have |
| 1555 | |
| 1556 | "dog" =~ /d/; # 'd' matches |
| 1557 | "dogbert =~ //; # this matches the 'd' regexp used before |
| 1558 | |
| 1559 | |
| 1560 | =head3 Global matching |
| 1561 | |
| 1562 | The final two modifiers we will discuss here, |
| 1563 | C<//g> and C<//c>, concern multiple matches. |
| 1564 | The modifier C<//g> stands for global matching and allows the |
| 1565 | matching operator to match within a string as many times as possible. |
| 1566 | In scalar context, successive invocations against a string will have |
| 1567 | C<//g> jump from match to match, keeping track of position in the |
| 1568 | string as it goes along. You can get or set the position with the |
| 1569 | C<pos()> function. |
| 1570 | |
| 1571 | The use of C<//g> is shown in the following example. Suppose we have |
| 1572 | a string that consists of words separated by spaces. If we know how |
| 1573 | many words there are in advance, we could extract the words using |
| 1574 | groupings: |
| 1575 | |
| 1576 | $x = "cat dog house"; # 3 words |
| 1577 | $x =~ /^\s*(\w+)\s+(\w+)\s+(\w+)\s*$/; # matches, |
| 1578 | # $1 = 'cat' |
| 1579 | # $2 = 'dog' |
| 1580 | # $3 = 'house' |
| 1581 | |
| 1582 | But what if we had an indeterminate number of words? This is the sort |
| 1583 | of task C<//g> was made for. To extract all words, form the simple |
| 1584 | regexp C<(\w+)> and loop over all matches with C</(\w+)/g>: |
| 1585 | |
| 1586 | while ($x =~ /(\w+)/g) { |
| 1587 | print "Word is $1, ends at position ", pos $x, "\n"; |
| 1588 | } |
| 1589 | |
| 1590 | prints |
| 1591 | |
| 1592 | Word is cat, ends at position 3 |
| 1593 | Word is dog, ends at position 7 |
| 1594 | Word is house, ends at position 13 |
| 1595 | |
| 1596 | A failed match or changing the target string resets the position. If |
| 1597 | you don't want the position reset after failure to match, add the |
| 1598 | C<//c>, as in C</regexp/gc>. The current position in the string is |
| 1599 | associated with the string, not the regexp. This means that different |
| 1600 | strings have different positions and their respective positions can be |
| 1601 | set or read independently. |
| 1602 | |
| 1603 | In list context, C<//g> returns a list of matched groupings, or if |
| 1604 | there are no groupings, a list of matches to the whole regexp. So if |
| 1605 | we wanted just the words, we could use |
| 1606 | |
| 1607 | @words = ($x =~ /(\w+)/g); # matches, |
| 1608 | # $words[0] = 'cat' |
| 1609 | # $words[1] = 'dog' |
| 1610 | # $words[2] = 'house' |
| 1611 | |
| 1612 | Closely associated with the C<//g> modifier is the C<\G> anchor. The |
| 1613 | C<\G> anchor matches at the point where the previous C<//g> match left |
| 1614 | off. C<\G> allows us to easily do context-sensitive matching: |
| 1615 | |
| 1616 | $metric = 1; # use metric units |
| 1617 | ... |
| 1618 | $x = <FILE>; # read in measurement |
| 1619 | $x =~ /^([+-]?\d+)\s*/g; # get magnitude |
| 1620 | $weight = $1; |
| 1621 | if ($metric) { # error checking |
| 1622 | print "Units error!" unless $x =~ /\Gkg\./g; |
| 1623 | } |
| 1624 | else { |
| 1625 | print "Units error!" unless $x =~ /\Glbs\./g; |
| 1626 | } |
| 1627 | $x =~ /\G\s+(widget|sprocket)/g; # continue processing |
| 1628 | |
| 1629 | The combination of C<//g> and C<\G> allows us to process the string a |
| 1630 | bit at a time and use arbitrary Perl logic to decide what to do next. |
| 1631 | Currently, the C<\G> anchor is only fully supported when used to anchor |
| 1632 | to the start of the pattern. |
| 1633 | |
| 1634 | C<\G> is also invaluable in processing fixed-length records with |
| 1635 | regexps. Suppose we have a snippet of coding region DNA, encoded as |
| 1636 | base pair letters C<ATCGTTGAAT...> and we want to find all the stop |
| 1637 | codons C<TGA>. In a coding region, codons are 3-letter sequences, so |
| 1638 | we can think of the DNA snippet as a sequence of 3-letter records. The |
| 1639 | naive regexp |
| 1640 | |
| 1641 | # expanded, this is "ATC GTT GAA TGC AAA TGA CAT GAC" |
| 1642 | $dna = "ATCGTTGAATGCAAATGACATGAC"; |
| 1643 | $dna =~ /TGA/; |
| 1644 | |
| 1645 | doesn't work; it may match a C<TGA>, but there is no guarantee that |
| 1646 | the match is aligned with codon boundaries, e.g., the substring |
| 1647 | S<C<GTT GAA>> gives a match. A better solution is |
| 1648 | |
| 1649 | while ($dna =~ /(\w\w\w)*?TGA/g) { # note the minimal *? |
| 1650 | print "Got a TGA stop codon at position ", pos $dna, "\n"; |
| 1651 | } |
| 1652 | |
| 1653 | which prints |
| 1654 | |
| 1655 | Got a TGA stop codon at position 18 |
| 1656 | Got a TGA stop codon at position 23 |
| 1657 | |
| 1658 | Position 18 is good, but position 23 is bogus. What happened? |
| 1659 | |
| 1660 | The answer is that our regexp works well until we get past the last |
| 1661 | real match. Then the regexp will fail to match a synchronized C<TGA> |
| 1662 | and start stepping ahead one character position at a time, not what we |
| 1663 | want. The solution is to use C<\G> to anchor the match to the codon |
| 1664 | alignment: |
| 1665 | |
| 1666 | while ($dna =~ /\G(\w\w\w)*?TGA/g) { |
| 1667 | print "Got a TGA stop codon at position ", pos $dna, "\n"; |
| 1668 | } |
| 1669 | |
| 1670 | This prints |
| 1671 | |
| 1672 | Got a TGA stop codon at position 18 |
| 1673 | |
| 1674 | which is the correct answer. This example illustrates that it is |
| 1675 | important not only to match what is desired, but to reject what is not |
| 1676 | desired. |
| 1677 | |
| 1678 | (There are other regexp modifiers that are available, such as |
| 1679 | C<//o>, but their specialized uses are beyond the |
| 1680 | scope of this introduction. ) |
| 1681 | |
| 1682 | =head3 Search and replace |
| 1683 | |
| 1684 | Regular expressions also play a big role in I<search and replace> |
| 1685 | operations in Perl. Search and replace is accomplished with the |
| 1686 | C<s///> operator. The general form is |
| 1687 | C<s/regexp/replacement/modifiers>, with everything we know about |
| 1688 | regexps and modifiers applying in this case as well. The |
| 1689 | C<replacement> is a Perl double-quoted string that replaces in the |
| 1690 | string whatever is matched with the C<regexp>. The operator C<=~> is |
| 1691 | also used here to associate a string with C<s///>. If matching |
| 1692 | against C<$_>, the S<C<$_ =~>> can be dropped. If there is a match, |
| 1693 | C<s///> returns the number of substitutions made; otherwise it returns |
| 1694 | false. Here are a few examples: |
| 1695 | |
| 1696 | $x = "Time to feed the cat!"; |
| 1697 | $x =~ s/cat/hacker/; # $x contains "Time to feed the hacker!" |
| 1698 | if ($x =~ s/^(Time.*hacker)!$/$1 now!/) { |
| 1699 | $more_insistent = 1; |
| 1700 | } |
| 1701 | $y = "'quoted words'"; |
| 1702 | $y =~ s/^'(.*)'$/$1/; # strip single quotes, |
| 1703 | # $y contains "quoted words" |
| 1704 | |
| 1705 | In the last example, the whole string was matched, but only the part |
| 1706 | inside the single quotes was grouped. With the C<s///> operator, the |
| 1707 | matched variables C<$1>, C<$2>, etc. are immediately available for use |
| 1708 | in the replacement expression, so we use C<$1> to replace the quoted |
| 1709 | string with just what was quoted. With the global modifier, C<s///g> |
| 1710 | will search and replace all occurrences of the regexp in the string: |
| 1711 | |
| 1712 | $x = "I batted 4 for 4"; |
| 1713 | $x =~ s/4/four/; # doesn't do it all: |
| 1714 | # $x contains "I batted four for 4" |
| 1715 | $x = "I batted 4 for 4"; |
| 1716 | $x =~ s/4/four/g; # does it all: |
| 1717 | # $x contains "I batted four for four" |
| 1718 | |
| 1719 | If you prefer 'regex' over 'regexp' in this tutorial, you could use |
| 1720 | the following program to replace it: |
| 1721 | |
| 1722 | % cat > simple_replace |
| 1723 | #!/usr/bin/perl |
| 1724 | $regexp = shift; |
| 1725 | $replacement = shift; |
| 1726 | while (<>) { |
| 1727 | s/$regexp/$replacement/g; |
| 1728 | print; |
| 1729 | } |
| 1730 | ^D |
| 1731 | |
| 1732 | % simple_replace regexp regex perlretut.pod |
| 1733 | |
| 1734 | In C<simple_replace> we used the C<s///g> modifier to replace all |
| 1735 | occurrences of the regexp on each line. (Even though the regular |
| 1736 | expression appears in a loop, Perl is smart enough to compile it |
| 1737 | only once.) As with C<simple_grep>, both the |
| 1738 | C<print> and the C<s/$regexp/$replacement/g> use C<$_> implicitly. |
| 1739 | |
| 1740 | If you don't want C<s///> to change your original variable you can use |
| 1741 | the non-destructive substitute modifier, C<s///r>. This changes the |
| 1742 | behavior so that C<s///r> returns the final substituted string |
| 1743 | (instead of the number of substitutions): |
| 1744 | |
| 1745 | $x = "I like dogs."; |
| 1746 | $y = $x =~ s/dogs/cats/r; |
| 1747 | print "$x $y\n"; |
| 1748 | |
| 1749 | That example will print "I like dogs. I like cats". Notice the original |
| 1750 | C<$x> variable has not been affected. The overall |
| 1751 | result of the substitution is instead stored in C<$y>. If the |
| 1752 | substitution doesn't affect anything then the original string is |
| 1753 | returned: |
| 1754 | |
| 1755 | $x = "I like dogs."; |
| 1756 | $y = $x =~ s/elephants/cougars/r; |
| 1757 | print "$x $y\n"; # prints "I like dogs. I like dogs." |
| 1758 | |
| 1759 | One other interesting thing that the C<s///r> flag allows is chaining |
| 1760 | substitutions: |
| 1761 | |
| 1762 | $x = "Cats are great."; |
| 1763 | print $x =~ s/Cats/Dogs/r =~ s/Dogs/Frogs/r =~ |
| 1764 | s/Frogs/Hedgehogs/r, "\n"; |
| 1765 | # prints "Hedgehogs are great." |
| 1766 | |
| 1767 | A modifier available specifically to search and replace is the |
| 1768 | C<s///e> evaluation modifier. C<s///e> treats the |
| 1769 | replacement text as Perl code, rather than a double-quoted |
| 1770 | string. The value that the code returns is substituted for the |
| 1771 | matched substring. C<s///e> is useful if you need to do a bit of |
| 1772 | computation in the process of replacing text. This example counts |
| 1773 | character frequencies in a line: |
| 1774 | |
| 1775 | $x = "Bill the cat"; |
| 1776 | $x =~ s/(.)/$chars{$1}++;$1/eg; # final $1 replaces char with itself |
| 1777 | print "frequency of '$_' is $chars{$_}\n" |
| 1778 | foreach (sort {$chars{$b} <=> $chars{$a}} keys %chars); |
| 1779 | |
| 1780 | This prints |
| 1781 | |
| 1782 | frequency of ' ' is 2 |
| 1783 | frequency of 't' is 2 |
| 1784 | frequency of 'l' is 2 |
| 1785 | frequency of 'B' is 1 |
| 1786 | frequency of 'c' is 1 |
| 1787 | frequency of 'e' is 1 |
| 1788 | frequency of 'h' is 1 |
| 1789 | frequency of 'i' is 1 |
| 1790 | frequency of 'a' is 1 |
| 1791 | |
| 1792 | As with the match C<m//> operator, C<s///> can use other delimiters, |
| 1793 | such as C<s!!!> and C<s{}{}>, and even C<s{}//>. If single quotes are |
| 1794 | used C<s'''>, then the regexp and replacement are |
| 1795 | treated as single-quoted strings and there are no |
| 1796 | variable substitutions. C<s///> in list context |
| 1797 | returns the same thing as in scalar context, i.e., the number of |
| 1798 | matches. |
| 1799 | |
| 1800 | =head3 The split function |
| 1801 | |
| 1802 | The C<split()> function is another place where a regexp is used. |
| 1803 | C<split /regexp/, string, limit> separates the C<string> operand into |
| 1804 | a list of substrings and returns that list. The regexp must be designed |
| 1805 | to match whatever constitutes the separators for the desired substrings. |
| 1806 | The C<limit>, if present, constrains splitting into no more than C<limit> |
| 1807 | number of strings. For example, to split a string into words, use |
| 1808 | |
| 1809 | $x = "Calvin and Hobbes"; |
| 1810 | @words = split /\s+/, $x; # $word[0] = 'Calvin' |
| 1811 | # $word[1] = 'and' |
| 1812 | # $word[2] = 'Hobbes' |
| 1813 | |
| 1814 | If the empty regexp C<//> is used, the regexp always matches and |
| 1815 | the string is split into individual characters. If the regexp has |
| 1816 | groupings, then the resulting list contains the matched substrings from the |
| 1817 | groupings as well. For instance, |
| 1818 | |
| 1819 | $x = "/usr/bin/perl"; |
| 1820 | @dirs = split m!/!, $x; # $dirs[0] = '' |
| 1821 | # $dirs[1] = 'usr' |
| 1822 | # $dirs[2] = 'bin' |
| 1823 | # $dirs[3] = 'perl' |
| 1824 | @parts = split m!(/)!, $x; # $parts[0] = '' |
| 1825 | # $parts[1] = '/' |
| 1826 | # $parts[2] = 'usr' |
| 1827 | # $parts[3] = '/' |
| 1828 | # $parts[4] = 'bin' |
| 1829 | # $parts[5] = '/' |
| 1830 | # $parts[6] = 'perl' |
| 1831 | |
| 1832 | Since the first character of $x matched the regexp, C<split> prepended |
| 1833 | an empty initial element to the list. |
| 1834 | |
| 1835 | If you have read this far, congratulations! You now have all the basic |
| 1836 | tools needed to use regular expressions to solve a wide range of text |
| 1837 | processing problems. If this is your first time through the tutorial, |
| 1838 | why not stop here and play around with regexps a while.... S<Part 2> |
| 1839 | concerns the more esoteric aspects of regular expressions and those |
| 1840 | concepts certainly aren't needed right at the start. |
| 1841 | |
| 1842 | =head1 Part 2: Power tools |
| 1843 | |
| 1844 | OK, you know the basics of regexps and you want to know more. If |
| 1845 | matching regular expressions is analogous to a walk in the woods, then |
| 1846 | the tools discussed in Part 1 are analogous to topo maps and a |
| 1847 | compass, basic tools we use all the time. Most of the tools in part 2 |
| 1848 | are analogous to flare guns and satellite phones. They aren't used |
| 1849 | too often on a hike, but when we are stuck, they can be invaluable. |
| 1850 | |
| 1851 | What follows are the more advanced, less used, or sometimes esoteric |
| 1852 | capabilities of Perl regexps. In Part 2, we will assume you are |
| 1853 | comfortable with the basics and concentrate on the advanced features. |
| 1854 | |
| 1855 | =head2 More on characters, strings, and character classes |
| 1856 | |
| 1857 | There are a number of escape sequences and character classes that we |
| 1858 | haven't covered yet. |
| 1859 | |
| 1860 | There are several escape sequences that convert characters or strings |
| 1861 | between upper and lower case, and they are also available within |
| 1862 | patterns. C<\l> and C<\u> convert the next character to lower or |
| 1863 | upper case, respectively: |
| 1864 | |
| 1865 | $x = "perl"; |
| 1866 | $string =~ /\u$x/; # matches 'Perl' in $string |
| 1867 | $x = "M(rs?|s)\\."; # note the double backslash |
| 1868 | $string =~ /\l$x/; # matches 'mr.', 'mrs.', and 'ms.', |
| 1869 | |
| 1870 | A C<\L> or C<\U> indicates a lasting conversion of case, until |
| 1871 | terminated by C<\E> or thrown over by another C<\U> or C<\L>: |
| 1872 | |
| 1873 | $x = "This word is in lower case:\L SHOUT\E"; |
| 1874 | $x =~ /shout/; # matches |
| 1875 | $x = "I STILL KEYPUNCH CARDS FOR MY 360" |
| 1876 | $x =~ /\Ukeypunch/; # matches punch card string |
| 1877 | |
| 1878 | If there is no C<\E>, case is converted until the end of the |
| 1879 | string. The regexps C<\L\u$word> or C<\u\L$word> convert the first |
| 1880 | character of C<$word> to uppercase and the rest of the characters to |
| 1881 | lowercase. |
| 1882 | |
| 1883 | Control characters can be escaped with C<\c>, so that a control-Z |
| 1884 | character would be matched with C<\cZ>. The escape sequence |
| 1885 | C<\Q>...C<\E> quotes, or protects most non-alphabetic characters. For |
| 1886 | instance, |
| 1887 | |
| 1888 | $x = "\QThat !^*&%~& cat!"; |
| 1889 | $x =~ /\Q!^*&%~&\E/; # check for rough language |
| 1890 | |
| 1891 | It does not protect C<$> or C<@>, so that variables can still be |
| 1892 | substituted. |
| 1893 | |
| 1894 | C<\Q>, C<\L>, C<\l>, C<\U>, C<\u> and C<\E> are actually part of |
| 1895 | double-quotish syntax, and not part of regexp syntax proper. They will |
| 1896 | work if they appear in a regular expression embedded directly in a |
| 1897 | program, but not when contained in a string that is interpolated in a |
| 1898 | pattern. |
| 1899 | |
| 1900 | Perl regexps can handle more than just the |
| 1901 | standard ASCII character set. Perl supports I<Unicode>, a standard |
| 1902 | for representing the alphabets from virtually all of the world's written |
| 1903 | languages, and a host of symbols. Perl's text strings are Unicode strings, so |
| 1904 | they can contain characters with a value (codepoint or character number) higher |
| 1905 | than 255. |
| 1906 | |
| 1907 | What does this mean for regexps? Well, regexp users don't need to know |
| 1908 | much about Perl's internal representation of strings. But they do need |
| 1909 | to know 1) how to represent Unicode characters in a regexp and 2) that |
| 1910 | a matching operation will treat the string to be searched as a sequence |
| 1911 | of characters, not bytes. The answer to 1) is that Unicode characters |
| 1912 | greater than C<chr(255)> are represented using the C<\x{hex}> notation, because |
| 1913 | \x hex (without curly braces) doesn't go further than 255. (Starting in Perl |
| 1914 | 5.14, if you're an octal fan, you can also use C<\o{oct}>.) |
| 1915 | |
| 1916 | /\x{263a}/; # match a Unicode smiley face :) |
| 1917 | |
| 1918 | B<NOTE>: In Perl 5.6.0 it used to be that one needed to say C<use |
| 1919 | utf8> to use any Unicode features. This is no more the case: for |
| 1920 | almost all Unicode processing, the explicit C<utf8> pragma is not |
| 1921 | needed. (The only case where it matters is if your Perl script is in |
| 1922 | Unicode and encoded in UTF-8, then an explicit C<use utf8> is needed.) |
| 1923 | |
| 1924 | Figuring out the hexadecimal sequence of a Unicode character you want |
| 1925 | or deciphering someone else's hexadecimal Unicode regexp is about as |
| 1926 | much fun as programming in machine code. So another way to specify |
| 1927 | Unicode characters is to use the I<named character> escape |
| 1928 | sequence C<\N{I<name>}>. I<name> is a name for the Unicode character, as |
| 1929 | specified in the Unicode standard. For instance, if we wanted to |
| 1930 | represent or match the astrological sign for the planet Mercury, we |
| 1931 | could use |
| 1932 | |
| 1933 | $x = "abc\N{MERCURY}def"; |
| 1934 | $x =~ /\N{MERCURY}/; # matches |
| 1935 | |
| 1936 | One can also use "short" names: |
| 1937 | |
| 1938 | print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n"; |
| 1939 | print "\N{greek:Sigma} is an upper-case sigma.\n"; |
| 1940 | |
| 1941 | You can also restrict names to a certain alphabet by specifying the |
| 1942 | L<charnames> pragma: |
| 1943 | |
| 1944 | use charnames qw(greek); |
| 1945 | print "\N{sigma} is Greek sigma\n"; |
| 1946 | |
| 1947 | An index of character names is available on-line from the Unicode |
| 1948 | Consortium, L<http://www.unicode.org/charts/charindex.html>; explanatory |
| 1949 | material with links to other resources at |
| 1950 | L<http://www.unicode.org/standard/where>. |
| 1951 | |
| 1952 | The answer to requirement 2) is that a regexp (mostly) |
| 1953 | uses Unicode characters. The "mostly" is for messy backward |
| 1954 | compatibility reasons, but starting in Perl 5.14, any regex compiled in |
| 1955 | the scope of a C<use feature 'unicode_strings'> (which is automatically |
| 1956 | turned on within the scope of a C<use 5.012> or higher) will turn that |
| 1957 | "mostly" into "always". If you want to handle Unicode properly, you |
| 1958 | should ensure that C<'unicode_strings'> is turned on. |
| 1959 | Internally, this is encoded to bytes using either UTF-8 or a native 8 |
| 1960 | bit encoding, depending on the history of the string, but conceptually |
| 1961 | it is a sequence of characters, not bytes. See L<perlunitut> for a |
| 1962 | tutorial about that. |
| 1963 | |
| 1964 | Let us now discuss Unicode character classes, most usually called |
| 1965 | "character properties". These are represented by the |
| 1966 | C<\p{name}> escape sequence. Closely associated is the C<\P{name}> |
| 1967 | property, which is the negation of the C<\p{name}> one. For |
| 1968 | example, to match lower and uppercase characters, |
| 1969 | |
| 1970 | $x = "BOB"; |
| 1971 | $x =~ /^\p{IsUpper}/; # matches, uppercase char class |
| 1972 | $x =~ /^\P{IsUpper}/; # doesn't match, char class sans uppercase |
| 1973 | $x =~ /^\p{IsLower}/; # doesn't match, lowercase char class |
| 1974 | $x =~ /^\P{IsLower}/; # matches, char class sans lowercase |
| 1975 | |
| 1976 | (The "Is" is optional.) |
| 1977 | |
| 1978 | There are many, many Unicode character properties. For the full list |
| 1979 | see L<perluniprops>. Most of them have synonyms with shorter names, |
| 1980 | also listed there. Some synonyms are a single character. For these, |
| 1981 | you can drop the braces. For instance, C<\pM> is the same thing as |
| 1982 | C<\p{Mark}>, meaning things like accent marks. |
| 1983 | |
| 1984 | The Unicode C<\p{Script}> property is used to categorize every Unicode |
| 1985 | character into the language script it is written in. For example, |
| 1986 | English, French, and a bunch of other European languages are written in |
| 1987 | the Latin script. But there is also the Greek script, the Thai script, |
| 1988 | the Katakana script, etc. You can test whether a character is in a |
| 1989 | particular script with, for example C<\p{Latin}>, C<\p{Greek}>, |
| 1990 | or C<\p{Katakana}>. To test if it isn't in the Balinese script, you |
| 1991 | would use C<\P{Balinese}>. |
| 1992 | |
| 1993 | What we have described so far is the single form of the C<\p{...}> character |
| 1994 | classes. There is also a compound form which you may run into. These |
| 1995 | look like C<\p{name=value}> or C<\p{name:value}> (the equals sign and colon |
| 1996 | can be used interchangeably). These are more general than the single form, |
| 1997 | and in fact most of the single forms are just Perl-defined shortcuts for common |
| 1998 | compound forms. For example, the script examples in the previous paragraph |
| 1999 | could be written equivalently as C<\p{Script=Latin}>, C<\p{Script:Greek}>, |
| 2000 | C<\p{script=katakana}>, and C<\P{script=balinese}> (case is irrelevant |
| 2001 | between the C<{}> braces). You may |
| 2002 | never have to use the compound forms, but sometimes it is necessary, and their |
| 2003 | use can make your code easier to understand. |
| 2004 | |
| 2005 | C<\X> is an abbreviation for a character class that comprises |
| 2006 | a Unicode I<extended grapheme cluster>. This represents a "logical character": |
| 2007 | what appears to be a single character, but may be represented internally by more |
| 2008 | than one. As an example, using the Unicode full names, e.g., S<C<A + COMBINING |
| 2009 | RING>> is a grapheme cluster with base character C<A> and combining character |
| 2010 | S<C<COMBINING RING>>, which translates in Danish to A with the circle atop it, |
| 2011 | as in the word E<Aring>ngstrom. |
| 2012 | |
| 2013 | For the full and latest information about Unicode see the latest |
| 2014 | Unicode standard, or the Unicode Consortium's website L<http://www.unicode.org> |
| 2015 | |
| 2016 | As if all those classes weren't enough, Perl also defines POSIX-style |
| 2017 | character classes. These have the form C<[:name:]>, with C<name> the |
| 2018 | name of the POSIX class. The POSIX classes are C<alpha>, C<alnum>, |
| 2019 | C<ascii>, C<cntrl>, C<digit>, C<graph>, C<lower>, C<print>, C<punct>, |
| 2020 | C<space>, C<upper>, and C<xdigit>, and two extensions, C<word> (a Perl |
| 2021 | extension to match C<\w>), and C<blank> (a GNU extension). The C<//a> |
| 2022 | modifier restricts these to matching just in the ASCII range; otherwise |
| 2023 | they can match the same as their corresponding Perl Unicode classes: |
| 2024 | C<[:upper:]> is the same as C<\p{IsUpper}>, etc. (There are some |
| 2025 | exceptions and gotchas with this; see L<perlrecharclass> for a full |
| 2026 | discussion.) The C<[:digit:]>, C<[:word:]>, and |
| 2027 | C<[:space:]> correspond to the familiar C<\d>, C<\w>, and C<\s> |
| 2028 | character classes. To negate a POSIX class, put a C<^> in front of |
| 2029 | the name, so that, e.g., C<[:^digit:]> corresponds to C<\D> and, under |
| 2030 | Unicode, C<\P{IsDigit}>. The Unicode and POSIX character classes can |
| 2031 | be used just like C<\d>, with the exception that POSIX character |
| 2032 | classes can only be used inside of a character class: |
| 2033 | |
| 2034 | /\s+[abc[:digit:]xyz]\s*/; # match a,b,c,x,y,z, or a digit |
| 2035 | /^=item\s[[:digit:]]/; # match '=item', |
| 2036 | # followed by a space and a digit |
| 2037 | /\s+[abc\p{IsDigit}xyz]\s+/; # match a,b,c,x,y,z, or a digit |
| 2038 | /^=item\s\p{IsDigit}/; # match '=item', |
| 2039 | # followed by a space and a digit |
| 2040 | |
| 2041 | Whew! That is all the rest of the characters and character classes. |
| 2042 | |
| 2043 | =head2 Compiling and saving regular expressions |
| 2044 | |
| 2045 | In Part 1 we mentioned that Perl compiles a regexp into a compact |
| 2046 | sequence of opcodes. Thus, a compiled regexp is a data structure |
| 2047 | that can be stored once and used again and again. The regexp quote |
| 2048 | C<qr//> does exactly that: C<qr/string/> compiles the C<string> as a |
| 2049 | regexp and transforms the result into a form that can be assigned to a |
| 2050 | variable: |
| 2051 | |
| 2052 | $reg = qr/foo+bar?/; # reg contains a compiled regexp |
| 2053 | |
| 2054 | Then C<$reg> can be used as a regexp: |
| 2055 | |
| 2056 | $x = "fooooba"; |
| 2057 | $x =~ $reg; # matches, just like /foo+bar?/ |
| 2058 | $x =~ /$reg/; # same thing, alternate form |
| 2059 | |
| 2060 | C<$reg> can also be interpolated into a larger regexp: |
| 2061 | |
| 2062 | $x =~ /(abc)?$reg/; # still matches |
| 2063 | |
| 2064 | As with the matching operator, the regexp quote can use different |
| 2065 | delimiters, e.g., C<qr!!>, C<qr{}> or C<qr~~>. Apostrophes |
| 2066 | as delimiters (C<qr''>) inhibit any interpolation. |
| 2067 | |
| 2068 | Pre-compiled regexps are useful for creating dynamic matches that |
| 2069 | don't need to be recompiled each time they are encountered. Using |
| 2070 | pre-compiled regexps, we write a C<grep_step> program which greps |
| 2071 | for a sequence of patterns, advancing to the next pattern as soon |
| 2072 | as one has been satisfied. |
| 2073 | |
| 2074 | % cat > grep_step |
| 2075 | #!/usr/bin/perl |
| 2076 | # grep_step - match <number> regexps, one after the other |
| 2077 | # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ... |
| 2078 | |
| 2079 | $number = shift; |
| 2080 | $regexp[$_] = shift foreach (0..$number-1); |
| 2081 | @compiled = map qr/$_/, @regexp; |
| 2082 | while ($line = <>) { |
| 2083 | if ($line =~ /$compiled[0]/) { |
| 2084 | print $line; |
| 2085 | shift @compiled; |
| 2086 | last unless @compiled; |
| 2087 | } |
| 2088 | } |
| 2089 | ^D |
| 2090 | |
| 2091 | % grep_step 3 shift print last grep_step |
| 2092 | $number = shift; |
| 2093 | print $line; |
| 2094 | last unless @compiled; |
| 2095 | |
| 2096 | Storing pre-compiled regexps in an array C<@compiled> allows us to |
| 2097 | simply loop through the regexps without any recompilation, thus gaining |
| 2098 | flexibility without sacrificing speed. |
| 2099 | |
| 2100 | |
| 2101 | =head2 Composing regular expressions at runtime |
| 2102 | |
| 2103 | Backtracking is more efficient than repeated tries with different regular |
| 2104 | expressions. If there are several regular expressions and a match with |
| 2105 | any of them is acceptable, then it is possible to combine them into a set |
| 2106 | of alternatives. If the individual expressions are input data, this |
| 2107 | can be done by programming a join operation. We'll exploit this idea in |
| 2108 | an improved version of the C<simple_grep> program: a program that matches |
| 2109 | multiple patterns: |
| 2110 | |
| 2111 | % cat > multi_grep |
| 2112 | #!/usr/bin/perl |
| 2113 | # multi_grep - match any of <number> regexps |
| 2114 | # usage: multi_grep <number> regexp1 regexp2 ... file1 file2 ... |
| 2115 | |
| 2116 | $number = shift; |
| 2117 | $regexp[$_] = shift foreach (0..$number-1); |
| 2118 | $pattern = join '|', @regexp; |
| 2119 | |
| 2120 | while ($line = <>) { |
| 2121 | print $line if $line =~ /$pattern/; |
| 2122 | } |
| 2123 | ^D |
| 2124 | |
| 2125 | % multi_grep 2 shift for multi_grep |
| 2126 | $number = shift; |
| 2127 | $regexp[$_] = shift foreach (0..$number-1); |
| 2128 | |
| 2129 | Sometimes it is advantageous to construct a pattern from the I<input> |
| 2130 | that is to be analyzed and use the permissible values on the left |
| 2131 | hand side of the matching operations. As an example for this somewhat |
| 2132 | paradoxical situation, let's assume that our input contains a command |
| 2133 | verb which should match one out of a set of available command verbs, |
| 2134 | with the additional twist that commands may be abbreviated as long as |
| 2135 | the given string is unique. The program below demonstrates the basic |
| 2136 | algorithm. |
| 2137 | |
| 2138 | % cat > keymatch |
| 2139 | #!/usr/bin/perl |
| 2140 | $kwds = 'copy compare list print'; |
| 2141 | while( $cmd = <> ){ |
| 2142 | $cmd =~ s/^\s+|\s+$//g; # trim leading and trailing spaces |
| 2143 | if( ( @matches = $kwds =~ /\b$cmd\w*/g ) == 1 ){ |
| 2144 | print "command: '@matches'\n"; |
| 2145 | } elsif( @matches == 0 ){ |
| 2146 | print "no such command: '$cmd'\n"; |
| 2147 | } else { |
| 2148 | print "not unique: '$cmd' (could be one of: @matches)\n"; |
| 2149 | } |
| 2150 | } |
| 2151 | ^D |
| 2152 | |
| 2153 | % keymatch |
| 2154 | li |
| 2155 | command: 'list' |
| 2156 | co |
| 2157 | not unique: 'co' (could be one of: copy compare) |
| 2158 | printer |
| 2159 | no such command: 'printer' |
| 2160 | |
| 2161 | Rather than trying to match the input against the keywords, we match the |
| 2162 | combined set of keywords against the input. The pattern matching |
| 2163 | operation S<C<$kwds =~ /\b($cmd\w*)/g>> does several things at the |
| 2164 | same time. It makes sure that the given command begins where a keyword |
| 2165 | begins (C<\b>). It tolerates abbreviations due to the added C<\w*>. It |
| 2166 | tells us the number of matches (C<scalar @matches>) and all the keywords |
| 2167 | that were actually matched. You could hardly ask for more. |
| 2168 | |
| 2169 | =head2 Embedding comments and modifiers in a regular expression |
| 2170 | |
| 2171 | Starting with this section, we will be discussing Perl's set of |
| 2172 | I<extended patterns>. These are extensions to the traditional regular |
| 2173 | expression syntax that provide powerful new tools for pattern |
| 2174 | matching. We have already seen extensions in the form of the minimal |
| 2175 | matching constructs C<??>, C<*?>, C<+?>, C<{n,m}?>, and C<{n,}?>. Most |
| 2176 | of the extensions below have the form C<(?char...)>, where the |
| 2177 | C<char> is a character that determines the type of extension. |
| 2178 | |
| 2179 | The first extension is an embedded comment C<(?#text)>. This embeds a |
| 2180 | comment into the regular expression without affecting its meaning. The |
| 2181 | comment should not have any closing parentheses in the text. An |
| 2182 | example is |
| 2183 | |
| 2184 | /(?# Match an integer:)[+-]?\d+/; |
| 2185 | |
| 2186 | This style of commenting has been largely superseded by the raw, |
| 2187 | freeform commenting that is allowed with the C<//x> modifier. |
| 2188 | |
| 2189 | Most modifiers, such as C<//i>, C<//m>, C<//s> and C<//x> (or any |
| 2190 | combination thereof) can also be embedded in |
| 2191 | a regexp using C<(?i)>, C<(?m)>, C<(?s)>, and C<(?x)>. For instance, |
| 2192 | |
| 2193 | /(?i)yes/; # match 'yes' case insensitively |
| 2194 | /yes/i; # same thing |
| 2195 | /(?x)( # freeform version of an integer regexp |
| 2196 | [+-]? # match an optional sign |
| 2197 | \d+ # match a sequence of digits |
| 2198 | ) |
| 2199 | /x; |
| 2200 | |
| 2201 | Embedded modifiers can have two important advantages over the usual |
| 2202 | modifiers. Embedded modifiers allow a custom set of modifiers to |
| 2203 | I<each> regexp pattern. This is great for matching an array of regexps |
| 2204 | that must have different modifiers: |
| 2205 | |
| 2206 | $pattern[0] = '(?i)doctor'; |
| 2207 | $pattern[1] = 'Johnson'; |
| 2208 | ... |
| 2209 | while (<>) { |
| 2210 | foreach $patt (@pattern) { |
| 2211 | print if /$patt/; |
| 2212 | } |
| 2213 | } |
| 2214 | |
| 2215 | The second advantage is that embedded modifiers (except C<//p>, which |
| 2216 | modifies the entire regexp) only affect the regexp |
| 2217 | inside the group the embedded modifier is contained in. So grouping |
| 2218 | can be used to localize the modifier's effects: |
| 2219 | |
| 2220 | /Answer: ((?i)yes)/; # matches 'Answer: yes', 'Answer: YES', etc. |
| 2221 | |
| 2222 | Embedded modifiers can also turn off any modifiers already present |
| 2223 | by using, e.g., C<(?-i)>. Modifiers can also be combined into |
| 2224 | a single expression, e.g., C<(?s-i)> turns on single line mode and |
| 2225 | turns off case insensitivity. |
| 2226 | |
| 2227 | Embedded modifiers may also be added to a non-capturing grouping. |
| 2228 | C<(?i-m:regexp)> is a non-capturing grouping that matches C<regexp> |
| 2229 | case insensitively and turns off multi-line mode. |
| 2230 | |
| 2231 | |
| 2232 | =head2 Looking ahead and looking behind |
| 2233 | |
| 2234 | This section concerns the lookahead and lookbehind assertions. First, |
| 2235 | a little background. |
| 2236 | |
| 2237 | In Perl regular expressions, most regexp elements 'eat up' a certain |
| 2238 | amount of string when they match. For instance, the regexp element |
| 2239 | C<[abc}]> eats up one character of the string when it matches, in the |
| 2240 | sense that Perl moves to the next character position in the string |
| 2241 | after the match. There are some elements, however, that don't eat up |
| 2242 | characters (advance the character position) if they match. The examples |
| 2243 | we have seen so far are the anchors. The anchor C<^> matches the |
| 2244 | beginning of the line, but doesn't eat any characters. Similarly, the |
| 2245 | word boundary anchor C<\b> matches wherever a character matching C<\w> |
| 2246 | is next to a character that doesn't, but it doesn't eat up any |
| 2247 | characters itself. Anchors are examples of I<zero-width assertions>: |
| 2248 | zero-width, because they consume |
| 2249 | no characters, and assertions, because they test some property of the |
| 2250 | string. In the context of our walk in the woods analogy to regexp |
| 2251 | matching, most regexp elements move us along a trail, but anchors have |
| 2252 | us stop a moment and check our surroundings. If the local environment |
| 2253 | checks out, we can proceed forward. But if the local environment |
| 2254 | doesn't satisfy us, we must backtrack. |
| 2255 | |
| 2256 | Checking the environment entails either looking ahead on the trail, |
| 2257 | looking behind, or both. C<^> looks behind, to see that there are no |
| 2258 | characters before. C<$> looks ahead, to see that there are no |
| 2259 | characters after. C<\b> looks both ahead and behind, to see if the |
| 2260 | characters on either side differ in their "word-ness". |
| 2261 | |
| 2262 | The lookahead and lookbehind assertions are generalizations of the |
| 2263 | anchor concept. Lookahead and lookbehind are zero-width assertions |
| 2264 | that let us specify which characters we want to test for. The |
| 2265 | lookahead assertion is denoted by C<(?=regexp)> and the lookbehind |
| 2266 | assertion is denoted by C<< (?<=fixed-regexp) >>. Some examples are |
| 2267 | |
| 2268 | $x = "I catch the housecat 'Tom-cat' with catnip"; |
| 2269 | $x =~ /cat(?=\s)/; # matches 'cat' in 'housecat' |
| 2270 | @catwords = ($x =~ /(?<=\s)cat\w+/g); # matches, |
| 2271 | # $catwords[0] = 'catch' |
| 2272 | # $catwords[1] = 'catnip' |
| 2273 | $x =~ /\bcat\b/; # matches 'cat' in 'Tom-cat' |
| 2274 | $x =~ /(?<=\s)cat(?=\s)/; # doesn't match; no isolated 'cat' in |
| 2275 | # middle of $x |
| 2276 | |
| 2277 | Note that the parentheses in C<(?=regexp)> and C<< (?<=regexp) >> are |
| 2278 | non-capturing, since these are zero-width assertions. Thus in the |
| 2279 | second regexp, the substrings captured are those of the whole regexp |
| 2280 | itself. Lookahead C<(?=regexp)> can match arbitrary regexps, but |
| 2281 | lookbehind C<< (?<=fixed-regexp) >> only works for regexps of fixed |
| 2282 | width, i.e., a fixed number of characters long. Thus |
| 2283 | C<< (?<=(ab|bc)) >> is fine, but C<< (?<=(ab)*) >> is not. The |
| 2284 | negated versions of the lookahead and lookbehind assertions are |
| 2285 | denoted by C<(?!regexp)> and C<< (?<!fixed-regexp) >> respectively. |
| 2286 | They evaluate true if the regexps do I<not> match: |
| 2287 | |
| 2288 | $x = "foobar"; |
| 2289 | $x =~ /foo(?!bar)/; # doesn't match, 'bar' follows 'foo' |
| 2290 | $x =~ /foo(?!baz)/; # matches, 'baz' doesn't follow 'foo' |
| 2291 | $x =~ /(?<!\s)foo/; # matches, there is no \s before 'foo' |
| 2292 | |
| 2293 | The C<\C> is unsupported in lookbehind, because the already |
| 2294 | treacherous definition of C<\C> would become even more so |
| 2295 | when going backwards. |
| 2296 | |
| 2297 | Here is an example where a string containing blank-separated words, |
| 2298 | numbers and single dashes is to be split into its components. |
| 2299 | Using C</\s+/> alone won't work, because spaces are not required between |
| 2300 | dashes, or a word or a dash. Additional places for a split are established |
| 2301 | by looking ahead and behind: |
| 2302 | |
| 2303 | $str = "one two - --6-8"; |
| 2304 | @toks = split / \s+ # a run of spaces |
| 2305 | | (?<=\S) (?=-) # any non-space followed by '-' |
| 2306 | | (?<=-) (?=\S) # a '-' followed by any non-space |
| 2307 | /x, $str; # @toks = qw(one two - - - 6 - 8) |
| 2308 | |
| 2309 | |
| 2310 | =head2 Using independent subexpressions to prevent backtracking |
| 2311 | |
| 2312 | I<Independent subexpressions> are regular expressions, in the |
| 2313 | context of a larger regular expression, that function independently of |
| 2314 | the larger regular expression. That is, they consume as much or as |
| 2315 | little of the string as they wish without regard for the ability of |
| 2316 | the larger regexp to match. Independent subexpressions are represented |
| 2317 | by C<< (?>regexp) >>. We can illustrate their behavior by first |
| 2318 | considering an ordinary regexp: |
| 2319 | |
| 2320 | $x = "ab"; |
| 2321 | $x =~ /a*ab/; # matches |
| 2322 | |
| 2323 | This obviously matches, but in the process of matching, the |
| 2324 | subexpression C<a*> first grabbed the C<a>. Doing so, however, |
| 2325 | wouldn't allow the whole regexp to match, so after backtracking, C<a*> |
| 2326 | eventually gave back the C<a> and matched the empty string. Here, what |
| 2327 | C<a*> matched was I<dependent> on what the rest of the regexp matched. |
| 2328 | |
| 2329 | Contrast that with an independent subexpression: |
| 2330 | |
| 2331 | $x =~ /(?>a*)ab/; # doesn't match! |
| 2332 | |
| 2333 | The independent subexpression C<< (?>a*) >> doesn't care about the rest |
| 2334 | of the regexp, so it sees an C<a> and grabs it. Then the rest of the |
| 2335 | regexp C<ab> cannot match. Because C<< (?>a*) >> is independent, there |
| 2336 | is no backtracking and the independent subexpression does not give |
| 2337 | up its C<a>. Thus the match of the regexp as a whole fails. A similar |
| 2338 | behavior occurs with completely independent regexps: |
| 2339 | |
| 2340 | $x = "ab"; |
| 2341 | $x =~ /a*/g; # matches, eats an 'a' |
| 2342 | $x =~ /\Gab/g; # doesn't match, no 'a' available |
| 2343 | |
| 2344 | Here C<//g> and C<\G> create a 'tag team' handoff of the string from |
| 2345 | one regexp to the other. Regexps with an independent subexpression are |
| 2346 | much like this, with a handoff of the string to the independent |
| 2347 | subexpression, and a handoff of the string back to the enclosing |
| 2348 | regexp. |
| 2349 | |
| 2350 | The ability of an independent subexpression to prevent backtracking |
| 2351 | can be quite useful. Suppose we want to match a non-empty string |
| 2352 | enclosed in parentheses up to two levels deep. Then the following |
| 2353 | regexp matches: |
| 2354 | |
| 2355 | $x = "abc(de(fg)h"; # unbalanced parentheses |
| 2356 | $x =~ /\( ( [^()]+ | \([^()]*\) )+ \)/x; |
| 2357 | |
| 2358 | The regexp matches an open parenthesis, one or more copies of an |
| 2359 | alternation, and a close parenthesis. The alternation is two-way, with |
| 2360 | the first alternative C<[^()]+> matching a substring with no |
| 2361 | parentheses and the second alternative C<\([^()]*\)> matching a |
| 2362 | substring delimited by parentheses. The problem with this regexp is |
| 2363 | that it is pathological: it has nested indeterminate quantifiers |
| 2364 | of the form C<(a+|b)+>. We discussed in Part 1 how nested quantifiers |
| 2365 | like this could take an exponentially long time to execute if there |
| 2366 | was no match possible. To prevent the exponential blowup, we need to |
| 2367 | prevent useless backtracking at some point. This can be done by |
| 2368 | enclosing the inner quantifier as an independent subexpression: |
| 2369 | |
| 2370 | $x =~ /\( ( (?>[^()]+) | \([^()]*\) )+ \)/x; |
| 2371 | |
| 2372 | Here, C<< (?>[^()]+) >> breaks the degeneracy of string partitioning |
| 2373 | by gobbling up as much of the string as possible and keeping it. Then |
| 2374 | match failures fail much more quickly. |
| 2375 | |
| 2376 | |
| 2377 | =head2 Conditional expressions |
| 2378 | |
| 2379 | A I<conditional expression> is a form of if-then-else statement |
| 2380 | that allows one to choose which patterns are to be matched, based on |
| 2381 | some condition. There are two types of conditional expression: |
| 2382 | C<(?(condition)yes-regexp)> and |
| 2383 | C<(?(condition)yes-regexp|no-regexp)>. C<(?(condition)yes-regexp)> is |
| 2384 | like an S<C<'if () {}'>> statement in Perl. If the C<condition> is true, |
| 2385 | the C<yes-regexp> will be matched. If the C<condition> is false, the |
| 2386 | C<yes-regexp> will be skipped and Perl will move onto the next regexp |
| 2387 | element. The second form is like an S<C<'if () {} else {}'>> statement |
| 2388 | in Perl. If the C<condition> is true, the C<yes-regexp> will be |
| 2389 | matched, otherwise the C<no-regexp> will be matched. |
| 2390 | |
| 2391 | The C<condition> can have several forms. The first form is simply an |
| 2392 | integer in parentheses C<(integer)>. It is true if the corresponding |
| 2393 | backreference C<\integer> matched earlier in the regexp. The same |
| 2394 | thing can be done with a name associated with a capture group, written |
| 2395 | as C<< (<name>) >> or C<< ('name') >>. The second form is a bare |
| 2396 | zero-width assertion C<(?...)>, either a lookahead, a lookbehind, or a |
| 2397 | code assertion (discussed in the next section). The third set of forms |
| 2398 | provides tests that return true if the expression is executed within |
| 2399 | a recursion (C<(R)>) or is being called from some capturing group, |
| 2400 | referenced either by number (C<(R1)>, C<(R2)>,...) or by name |
| 2401 | (C<(R&name)>). |
| 2402 | |
| 2403 | The integer or name form of the C<condition> allows us to choose, |
| 2404 | with more flexibility, what to match based on what matched earlier in the |
| 2405 | regexp. This searches for words of the form C<"$x$x"> or C<"$x$y$y$x">: |
| 2406 | |
| 2407 | % simple_grep '^(\w+)(\w+)?(?(2)\g2\g1|\g1)$' /usr/dict/words |
| 2408 | beriberi |
| 2409 | coco |
| 2410 | couscous |
| 2411 | deed |
| 2412 | ... |
| 2413 | toot |
| 2414 | toto |
| 2415 | tutu |
| 2416 | |
| 2417 | The lookbehind C<condition> allows, along with backreferences, |
| 2418 | an earlier part of the match to influence a later part of the |
| 2419 | match. For instance, |
| 2420 | |
| 2421 | /[ATGC]+(?(?<=AA)G|C)$/; |
| 2422 | |
| 2423 | matches a DNA sequence such that it either ends in C<AAG>, or some |
| 2424 | other base pair combination and C<C>. Note that the form is |
| 2425 | C<< (?(?<=AA)G|C) >> and not C<< (?((?<=AA))G|C) >>; for the |
| 2426 | lookahead, lookbehind or code assertions, the parentheses around the |
| 2427 | conditional are not needed. |
| 2428 | |
| 2429 | |
| 2430 | =head2 Defining named patterns |
| 2431 | |
| 2432 | Some regular expressions use identical subpatterns in several places. |
| 2433 | Starting with Perl 5.10, it is possible to define named subpatterns in |
| 2434 | a section of the pattern so that they can be called up by name |
| 2435 | anywhere in the pattern. This syntactic pattern for this definition |
| 2436 | group is C<< (?(DEFINE)(?<name>pattern)...) >>. An insertion |
| 2437 | of a named pattern is written as C<(?&name)>. |
| 2438 | |
| 2439 | The example below illustrates this feature using the pattern for |
| 2440 | floating point numbers that was presented earlier on. The three |
| 2441 | subpatterns that are used more than once are the optional sign, the |
| 2442 | digit sequence for an integer and the decimal fraction. The DEFINE |
| 2443 | group at the end of the pattern contains their definition. Notice |
| 2444 | that the decimal fraction pattern is the first place where we can |
| 2445 | reuse the integer pattern. |
| 2446 | |
| 2447 | /^ (?&osg)\ * ( (?&int)(?&dec)? | (?&dec) ) |
| 2448 | (?: [eE](?&osg)(?&int) )? |
| 2449 | $ |
| 2450 | (?(DEFINE) |
| 2451 | (?<osg>[-+]?) # optional sign |
| 2452 | (?<int>\d++) # integer |
| 2453 | (?<dec>\.(?&int)) # decimal fraction |
| 2454 | )/x |
| 2455 | |
| 2456 | |
| 2457 | =head2 Recursive patterns |
| 2458 | |
| 2459 | This feature (introduced in Perl 5.10) significantly extends the |
| 2460 | power of Perl's pattern matching. By referring to some other |
| 2461 | capture group anywhere in the pattern with the construct |
| 2462 | C<(?group-ref)>, the I<pattern> within the referenced group is used |
| 2463 | as an independent subpattern in place of the group reference itself. |
| 2464 | Because the group reference may be contained I<within> the group it |
| 2465 | refers to, it is now possible to apply pattern matching to tasks that |
| 2466 | hitherto required a recursive parser. |
| 2467 | |
| 2468 | To illustrate this feature, we'll design a pattern that matches if |
| 2469 | a string contains a palindrome. (This is a word or a sentence that, |
| 2470 | while ignoring spaces, interpunctuation and case, reads the same backwards |
| 2471 | as forwards. We begin by observing that the empty string or a string |
| 2472 | containing just one word character is a palindrome. Otherwise it must |
| 2473 | have a word character up front and the same at its end, with another |
| 2474 | palindrome in between. |
| 2475 | |
| 2476 | /(?: (\w) (?...Here be a palindrome...) \g{-1} | \w? )/x |
| 2477 | |
| 2478 | Adding C<\W*> at either end to eliminate what is to be ignored, we already |
| 2479 | have the full pattern: |
| 2480 | |
| 2481 | my $pp = qr/^(\W* (?: (\w) (?1) \g{-1} | \w? ) \W*)$/ix; |
| 2482 | for $s ( "saippuakauppias", "A man, a plan, a canal: Panama!" ){ |
| 2483 | print "'$s' is a palindrome\n" if $s =~ /$pp/; |
| 2484 | } |
| 2485 | |
| 2486 | In C<(?...)> both absolute and relative backreferences may be used. |
| 2487 | The entire pattern can be reinserted with C<(?R)> or C<(?0)>. |
| 2488 | If you prefer to name your groups, you can use C<(?&name)> to |
| 2489 | recurse into that group. |
| 2490 | |
| 2491 | |
| 2492 | =head2 A bit of magic: executing Perl code in a regular expression |
| 2493 | |
| 2494 | Normally, regexps are a part of Perl expressions. |
| 2495 | I<Code evaluation> expressions turn that around by allowing |
| 2496 | arbitrary Perl code to be a part of a regexp. A code evaluation |
| 2497 | expression is denoted C<(?{code})>, with I<code> a string of Perl |
| 2498 | statements. |
| 2499 | |
| 2500 | Be warned that this feature is considered experimental, and may be |
| 2501 | changed without notice. |
| 2502 | |
| 2503 | Code expressions are zero-width assertions, and the value they return |
| 2504 | depends on their environment. There are two possibilities: either the |
| 2505 | code expression is used as a conditional in a conditional expression |
| 2506 | C<(?(condition)...)>, or it is not. If the code expression is a |
| 2507 | conditional, the code is evaluated and the result (i.e., the result of |
| 2508 | the last statement) is used to determine truth or falsehood. If the |
| 2509 | code expression is not used as a conditional, the assertion always |
| 2510 | evaluates true and the result is put into the special variable |
| 2511 | C<$^R>. The variable C<$^R> can then be used in code expressions later |
| 2512 | in the regexp. Here are some silly examples: |
| 2513 | |
| 2514 | $x = "abcdef"; |
| 2515 | $x =~ /abc(?{print "Hi Mom!";})def/; # matches, |
| 2516 | # prints 'Hi Mom!' |
| 2517 | $x =~ /aaa(?{print "Hi Mom!";})def/; # doesn't match, |
| 2518 | # no 'Hi Mom!' |
| 2519 | |
| 2520 | Pay careful attention to the next example: |
| 2521 | |
| 2522 | $x =~ /abc(?{print "Hi Mom!";})ddd/; # doesn't match, |
| 2523 | # no 'Hi Mom!' |
| 2524 | # but why not? |
| 2525 | |
| 2526 | At first glance, you'd think that it shouldn't print, because obviously |
| 2527 | the C<ddd> isn't going to match the target string. But look at this |
| 2528 | example: |
| 2529 | |
| 2530 | $x =~ /abc(?{print "Hi Mom!";})[dD]dd/; # doesn't match, |
| 2531 | # but _does_ print |
| 2532 | |
| 2533 | Hmm. What happened here? If you've been following along, you know that |
| 2534 | the above pattern should be effectively (almost) the same as the last one; |
| 2535 | enclosing the C<d> in a character class isn't going to change what it |
| 2536 | matches. So why does the first not print while the second one does? |
| 2537 | |
| 2538 | The answer lies in the optimizations the regex engine makes. In the first |
| 2539 | case, all the engine sees are plain old characters (aside from the |
| 2540 | C<?{}> construct). It's smart enough to realize that the string 'ddd' |
| 2541 | doesn't occur in our target string before actually running the pattern |
| 2542 | through. But in the second case, we've tricked it into thinking that our |
| 2543 | pattern is more complicated. It takes a look, sees our |
| 2544 | character class, and decides that it will have to actually run the |
| 2545 | pattern to determine whether or not it matches, and in the process of |
| 2546 | running it hits the print statement before it discovers that we don't |
| 2547 | have a match. |
| 2548 | |
| 2549 | To take a closer look at how the engine does optimizations, see the |
| 2550 | section L<"Pragmas and debugging"> below. |
| 2551 | |
| 2552 | More fun with C<?{}>: |
| 2553 | |
| 2554 | $x =~ /(?{print "Hi Mom!";})/; # matches, |
| 2555 | # prints 'Hi Mom!' |
| 2556 | $x =~ /(?{$c = 1;})(?{print "$c";})/; # matches, |
| 2557 | # prints '1' |
| 2558 | $x =~ /(?{$c = 1;})(?{print "$^R";})/; # matches, |
| 2559 | # prints '1' |
| 2560 | |
| 2561 | The bit of magic mentioned in the section title occurs when the regexp |
| 2562 | backtracks in the process of searching for a match. If the regexp |
| 2563 | backtracks over a code expression and if the variables used within are |
| 2564 | localized using C<local>, the changes in the variables produced by the |
| 2565 | code expression are undone! Thus, if we wanted to count how many times |
| 2566 | a character got matched inside a group, we could use, e.g., |
| 2567 | |
| 2568 | $x = "aaaa"; |
| 2569 | $count = 0; # initialize 'a' count |
| 2570 | $c = "bob"; # test if $c gets clobbered |
| 2571 | $x =~ /(?{local $c = 0;}) # initialize count |
| 2572 | ( a # match 'a' |
| 2573 | (?{local $c = $c + 1;}) # increment count |
| 2574 | )* # do this any number of times, |
| 2575 | aa # but match 'aa' at the end |
| 2576 | (?{$count = $c;}) # copy local $c var into $count |
| 2577 | /x; |
| 2578 | print "'a' count is $count, \$c variable is '$c'\n"; |
| 2579 | |
| 2580 | This prints |
| 2581 | |
| 2582 | 'a' count is 2, $c variable is 'bob' |
| 2583 | |
| 2584 | If we replace the S<C< (?{local $c = $c + 1;})>> with |
| 2585 | S<C< (?{$c = $c + 1;})>>, the variable changes are I<not> undone |
| 2586 | during backtracking, and we get |
| 2587 | |
| 2588 | 'a' count is 4, $c variable is 'bob' |
| 2589 | |
| 2590 | Note that only localized variable changes are undone. Other side |
| 2591 | effects of code expression execution are permanent. Thus |
| 2592 | |
| 2593 | $x = "aaaa"; |
| 2594 | $x =~ /(a(?{print "Yow\n";}))*aa/; |
| 2595 | |
| 2596 | produces |
| 2597 | |
| 2598 | Yow |
| 2599 | Yow |
| 2600 | Yow |
| 2601 | Yow |
| 2602 | |
| 2603 | The result C<$^R> is automatically localized, so that it will behave |
| 2604 | properly in the presence of backtracking. |
| 2605 | |
| 2606 | This example uses a code expression in a conditional to match a |
| 2607 | definite article, either 'the' in English or 'der|die|das' in German: |
| 2608 | |
| 2609 | $lang = 'DE'; # use German |
| 2610 | ... |
| 2611 | $text = "das"; |
| 2612 | print "matched\n" |
| 2613 | if $text =~ /(?(?{ |
| 2614 | $lang eq 'EN'; # is the language English? |
| 2615 | }) |
| 2616 | the | # if so, then match 'the' |
| 2617 | (der|die|das) # else, match 'der|die|das' |
| 2618 | ) |
| 2619 | /xi; |
| 2620 | |
| 2621 | Note that the syntax here is C<(?(?{...})yes-regexp|no-regexp)>, not |
| 2622 | C<(?((?{...}))yes-regexp|no-regexp)>. In other words, in the case of a |
| 2623 | code expression, we don't need the extra parentheses around the |
| 2624 | conditional. |
| 2625 | |
| 2626 | If you try to use code expressions where the code text is contained within |
| 2627 | an interpolated variable, rather than appearing literally in the pattern, |
| 2628 | Perl may surprise you: |
| 2629 | |
| 2630 | $bar = 5; |
| 2631 | $pat = '(?{ 1 })'; |
| 2632 | /foo(?{ $bar })bar/; # compiles ok, $bar not interpolated |
| 2633 | /foo(?{ 1 })$bar/; # compiles ok, $bar interpolated |
| 2634 | /foo${pat}bar/; # compile error! |
| 2635 | |
| 2636 | $pat = qr/(?{ $foo = 1 })/; # precompile code regexp |
| 2637 | /foo${pat}bar/; # compiles ok |
| 2638 | |
| 2639 | If a regexp has a variable that interpolates a code expression, Perl |
| 2640 | treats the regexp as an error. If the code expression is precompiled into |
| 2641 | a variable, however, interpolating is ok. The question is, why is this an |
| 2642 | error? |
| 2643 | |
| 2644 | The reason is that variable interpolation and code expressions |
| 2645 | together pose a security risk. The combination is dangerous because |
| 2646 | many programmers who write search engines often take user input and |
| 2647 | plug it directly into a regexp: |
| 2648 | |
| 2649 | $regexp = <>; # read user-supplied regexp |
| 2650 | $chomp $regexp; # get rid of possible newline |
| 2651 | $text =~ /$regexp/; # search $text for the $regexp |
| 2652 | |
| 2653 | If the C<$regexp> variable contains a code expression, the user could |
| 2654 | then execute arbitrary Perl code. For instance, some joker could |
| 2655 | search for S<C<system('rm -rf *');>> to erase your files. In this |
| 2656 | sense, the combination of interpolation and code expressions I<taints> |
| 2657 | your regexp. So by default, using both interpolation and code |
| 2658 | expressions in the same regexp is not allowed. If you're not |
| 2659 | concerned about malicious users, it is possible to bypass this |
| 2660 | security check by invoking S<C<use re 'eval'>>: |
| 2661 | |
| 2662 | use re 'eval'; # throw caution out the door |
| 2663 | $bar = 5; |
| 2664 | $pat = '(?{ 1 })'; |
| 2665 | /foo${pat}bar/; # compiles ok |
| 2666 | |
| 2667 | Another form of code expression is the I<pattern code expression>. |
| 2668 | The pattern code expression is like a regular code expression, except |
| 2669 | that the result of the code evaluation is treated as a regular |
| 2670 | expression and matched immediately. A simple example is |
| 2671 | |
| 2672 | $length = 5; |
| 2673 | $char = 'a'; |
| 2674 | $x = 'aaaaabb'; |
| 2675 | $x =~ /(??{$char x $length})/x; # matches, there are 5 of 'a' |
| 2676 | |
| 2677 | |
| 2678 | This final example contains both ordinary and pattern code |
| 2679 | expressions. It detects whether a binary string C<1101010010001...> has a |
| 2680 | Fibonacci spacing 0,1,1,2,3,5,... of the C<1>'s: |
| 2681 | |
| 2682 | $x = "1101010010001000001"; |
| 2683 | $z0 = ''; $z1 = '0'; # initial conditions |
| 2684 | print "It is a Fibonacci sequence\n" |
| 2685 | if $x =~ /^1 # match an initial '1' |
| 2686 | (?: |
| 2687 | ((??{ $z0 })) # match some '0' |
| 2688 | 1 # and then a '1' |
| 2689 | (?{ $z0 = $z1; $z1 .= $^N; }) |
| 2690 | )+ # repeat as needed |
| 2691 | $ # that is all there is |
| 2692 | /x; |
| 2693 | printf "Largest sequence matched was %d\n", length($z1)-length($z0); |
| 2694 | |
| 2695 | Remember that C<$^N> is set to whatever was matched by the last |
| 2696 | completed capture group. This prints |
| 2697 | |
| 2698 | It is a Fibonacci sequence |
| 2699 | Largest sequence matched was 5 |
| 2700 | |
| 2701 | Ha! Try that with your garden variety regexp package... |
| 2702 | |
| 2703 | Note that the variables C<$z0> and C<$z1> are not substituted when the |
| 2704 | regexp is compiled, as happens for ordinary variables outside a code |
| 2705 | expression. Rather, the whole code block is parsed as perl code at the |
| 2706 | same time as perl is compiling the code containing the literal regexp |
| 2707 | pattern. |
| 2708 | |
| 2709 | The regexp without the C<//x> modifier is |
| 2710 | |
| 2711 | /^1(?:((??{ $z0 }))1(?{ $z0 = $z1; $z1 .= $^N; }))+$/ |
| 2712 | |
| 2713 | which shows that spaces are still possible in the code parts. Nevertheless, |
| 2714 | when working with code and conditional expressions, the extended form of |
| 2715 | regexps is almost necessary in creating and debugging regexps. |
| 2716 | |
| 2717 | |
| 2718 | =head2 Backtracking control verbs |
| 2719 | |
| 2720 | Perl 5.10 introduced a number of control verbs intended to provide |
| 2721 | detailed control over the backtracking process, by directly influencing |
| 2722 | the regexp engine and by providing monitoring techniques. As all |
| 2723 | the features in this group are experimental and subject to change or |
| 2724 | removal in a future version of Perl, the interested reader is |
| 2725 | referred to L<perlre/"Special Backtracking Control Verbs"> for a |
| 2726 | detailed description. |
| 2727 | |
| 2728 | Below is just one example, illustrating the control verb C<(*FAIL)>, |
| 2729 | which may be abbreviated as C<(*F)>. If this is inserted in a regexp |
| 2730 | it will cause it to fail, just as it would at some |
| 2731 | mismatch between the pattern and the string. Processing |
| 2732 | of the regexp continues as it would after any "normal" |
| 2733 | failure, so that, for instance, the next position in the string or another |
| 2734 | alternative will be tried. As failing to match doesn't preserve capture |
| 2735 | groups or produce results, it may be necessary to use this in |
| 2736 | combination with embedded code. |
| 2737 | |
| 2738 | %count = (); |
| 2739 | "supercalifragilisticexpialidocious" =~ |
| 2740 | /([aeiou])(?{ $count{$1}++; })(*FAIL)/i; |
| 2741 | printf "%3d '%s'\n", $count{$_}, $_ for (sort keys %count); |
| 2742 | |
| 2743 | The pattern begins with a class matching a subset of letters. Whenever |
| 2744 | this matches, a statement like C<$count{'a'}++;> is executed, incrementing |
| 2745 | the letter's counter. Then C<(*FAIL)> does what it says, and |
| 2746 | the regexp engine proceeds according to the book: as long as the end of |
| 2747 | the string hasn't been reached, the position is advanced before looking |
| 2748 | for another vowel. Thus, match or no match makes no difference, and the |
| 2749 | regexp engine proceeds until the entire string has been inspected. |
| 2750 | (It's remarkable that an alternative solution using something like |
| 2751 | |
| 2752 | $count{lc($_)}++ for split('', "supercalifragilisticexpialidocious"); |
| 2753 | printf "%3d '%s'\n", $count2{$_}, $_ for ( qw{ a e i o u } ); |
| 2754 | |
| 2755 | is considerably slower.) |
| 2756 | |
| 2757 | |
| 2758 | =head2 Pragmas and debugging |
| 2759 | |
| 2760 | Speaking of debugging, there are several pragmas available to control |
| 2761 | and debug regexps in Perl. We have already encountered one pragma in |
| 2762 | the previous section, S<C<use re 'eval';>>, that allows variable |
| 2763 | interpolation and code expressions to coexist in a regexp. The other |
| 2764 | pragmas are |
| 2765 | |
| 2766 | use re 'taint'; |
| 2767 | $tainted = <>; |
| 2768 | @parts = ($tainted =~ /(\w+)\s+(\w+)/; # @parts is now tainted |
| 2769 | |
| 2770 | The C<taint> pragma causes any substrings from a match with a tainted |
| 2771 | variable to be tainted as well. This is not normally the case, as |
| 2772 | regexps are often used to extract the safe bits from a tainted |
| 2773 | variable. Use C<taint> when you are not extracting safe bits, but are |
| 2774 | performing some other processing. Both C<taint> and C<eval> pragmas |
| 2775 | are lexically scoped, which means they are in effect only until |
| 2776 | the end of the block enclosing the pragmas. |
| 2777 | |
| 2778 | use re '/m'; # or any other flags |
| 2779 | $multiline_string =~ /^foo/; # /m is implied |
| 2780 | |
| 2781 | The C<re '/flags'> pragma (introduced in Perl |
| 2782 | 5.14) turns on the given regular expression flags |
| 2783 | until the end of the lexical scope. See |
| 2784 | L<re/"'E<sol>flags' mode"> for more |
| 2785 | detail. |
| 2786 | |
| 2787 | use re 'debug'; |
| 2788 | /^(.*)$/s; # output debugging info |
| 2789 | |
| 2790 | use re 'debugcolor'; |
| 2791 | /^(.*)$/s; # output debugging info in living color |
| 2792 | |
| 2793 | The global C<debug> and C<debugcolor> pragmas allow one to get |
| 2794 | detailed debugging info about regexp compilation and |
| 2795 | execution. C<debugcolor> is the same as debug, except the debugging |
| 2796 | information is displayed in color on terminals that can display |
| 2797 | termcap color sequences. Here is example output: |
| 2798 | |
| 2799 | % perl -e 'use re "debug"; "abc" =~ /a*b+c/;' |
| 2800 | Compiling REx 'a*b+c' |
| 2801 | size 9 first at 1 |
| 2802 | 1: STAR(4) |
| 2803 | 2: EXACT <a>(0) |
| 2804 | 4: PLUS(7) |
| 2805 | 5: EXACT <b>(0) |
| 2806 | 7: EXACT <c>(9) |
| 2807 | 9: END(0) |
| 2808 | floating 'bc' at 0..2147483647 (checking floating) minlen 2 |
| 2809 | Guessing start of match, REx 'a*b+c' against 'abc'... |
| 2810 | Found floating substr 'bc' at offset 1... |
| 2811 | Guessed: match at offset 0 |
| 2812 | Matching REx 'a*b+c' against 'abc' |
| 2813 | Setting an EVAL scope, savestack=3 |
| 2814 | 0 <> <abc> | 1: STAR |
| 2815 | EXACT <a> can match 1 times out of 32767... |
| 2816 | Setting an EVAL scope, savestack=3 |
| 2817 | 1 <a> <bc> | 4: PLUS |
| 2818 | EXACT <b> can match 1 times out of 32767... |
| 2819 | Setting an EVAL scope, savestack=3 |
| 2820 | 2 <ab> <c> | 7: EXACT <c> |
| 2821 | 3 <abc> <> | 9: END |
| 2822 | Match successful! |
| 2823 | Freeing REx: 'a*b+c' |
| 2824 | |
| 2825 | If you have gotten this far into the tutorial, you can probably guess |
| 2826 | what the different parts of the debugging output tell you. The first |
| 2827 | part |
| 2828 | |
| 2829 | Compiling REx 'a*b+c' |
| 2830 | size 9 first at 1 |
| 2831 | 1: STAR(4) |
| 2832 | 2: EXACT <a>(0) |
| 2833 | 4: PLUS(7) |
| 2834 | 5: EXACT <b>(0) |
| 2835 | 7: EXACT <c>(9) |
| 2836 | 9: END(0) |
| 2837 | |
| 2838 | describes the compilation stage. C<STAR(4)> means that there is a |
| 2839 | starred object, in this case C<'a'>, and if it matches, goto line 4, |
| 2840 | i.e., C<PLUS(7)>. The middle lines describe some heuristics and |
| 2841 | optimizations performed before a match: |
| 2842 | |
| 2843 | floating 'bc' at 0..2147483647 (checking floating) minlen 2 |
| 2844 | Guessing start of match, REx 'a*b+c' against 'abc'... |
| 2845 | Found floating substr 'bc' at offset 1... |
| 2846 | Guessed: match at offset 0 |
| 2847 | |
| 2848 | Then the match is executed and the remaining lines describe the |
| 2849 | process: |
| 2850 | |
| 2851 | Matching REx 'a*b+c' against 'abc' |
| 2852 | Setting an EVAL scope, savestack=3 |
| 2853 | 0 <> <abc> | 1: STAR |
| 2854 | EXACT <a> can match 1 times out of 32767... |
| 2855 | Setting an EVAL scope, savestack=3 |
| 2856 | 1 <a> <bc> | 4: PLUS |
| 2857 | EXACT <b> can match 1 times out of 32767... |
| 2858 | Setting an EVAL scope, savestack=3 |
| 2859 | 2 <ab> <c> | 7: EXACT <c> |
| 2860 | 3 <abc> <> | 9: END |
| 2861 | Match successful! |
| 2862 | Freeing REx: 'a*b+c' |
| 2863 | |
| 2864 | Each step is of the form S<C<< n <x> <y> >>>, with C<< <x> >> the |
| 2865 | part of the string matched and C<< <y> >> the part not yet |
| 2866 | matched. The S<C<< | 1: STAR >>> says that Perl is at line number 1 |
| 2867 | in the compilation list above. See |
| 2868 | L<perldebguts/"Debugging Regular Expressions"> for much more detail. |
| 2869 | |
| 2870 | An alternative method of debugging regexps is to embed C<print> |
| 2871 | statements within the regexp. This provides a blow-by-blow account of |
| 2872 | the backtracking in an alternation: |
| 2873 | |
| 2874 | "that this" =~ m@(?{print "Start at position ", pos, "\n";}) |
| 2875 | t(?{print "t1\n";}) |
| 2876 | h(?{print "h1\n";}) |
| 2877 | i(?{print "i1\n";}) |
| 2878 | s(?{print "s1\n";}) |
| 2879 | | |
| 2880 | t(?{print "t2\n";}) |
| 2881 | h(?{print "h2\n";}) |
| 2882 | a(?{print "a2\n";}) |
| 2883 | t(?{print "t2\n";}) |
| 2884 | (?{print "Done at position ", pos, "\n";}) |
| 2885 | @x; |
| 2886 | |
| 2887 | prints |
| 2888 | |
| 2889 | Start at position 0 |
| 2890 | t1 |
| 2891 | h1 |
| 2892 | t2 |
| 2893 | h2 |
| 2894 | a2 |
| 2895 | t2 |
| 2896 | Done at position 4 |
| 2897 | |
| 2898 | =head1 BUGS |
| 2899 | |
| 2900 | Code expressions, conditional expressions, and independent expressions |
| 2901 | are I<experimental>. Don't use them in production code. Yet. |
| 2902 | |
| 2903 | =head1 SEE ALSO |
| 2904 | |
| 2905 | This is just a tutorial. For the full story on Perl regular |
| 2906 | expressions, see the L<perlre> regular expressions reference page. |
| 2907 | |
| 2908 | For more information on the matching C<m//> and substitution C<s///> |
| 2909 | operators, see L<perlop/"Regexp Quote-Like Operators">. For |
| 2910 | information on the C<split> operation, see L<perlfunc/split>. |
| 2911 | |
| 2912 | For an excellent all-around resource on the care and feeding of |
| 2913 | regular expressions, see the book I<Mastering Regular Expressions> by |
| 2914 | Jeffrey Friedl (published by O'Reilly, ISBN 1556592-257-3). |
| 2915 | |
| 2916 | =head1 AUTHOR AND COPYRIGHT |
| 2917 | |
| 2918 | Copyright (c) 2000 Mark Kvale |
| 2919 | All rights reserved. |
| 2920 | |
| 2921 | This document may be distributed under the same terms as Perl itself. |
| 2922 | |
| 2923 | =head2 Acknowledgments |
| 2924 | |
| 2925 | The inspiration for the stop codon DNA example came from the ZIP |
| 2926 | code example in chapter 7 of I<Mastering Regular Expressions>. |
| 2927 | |
| 2928 | The author would like to thank Jeff Pinyan, Andrew Johnson, Peter |
| 2929 | Haworth, Ronald J Kimball, and Joe Smith for all their helpful |
| 2930 | comments. |
| 2931 | |
| 2932 | =cut |
| 2933 | |