[perl #128747] Fix line number bug with s//<<END/e

In commit 6745174b561 I changed the multi_open and multi_close
parser pastruct members (known as PL_multi_open/close in toke.c) from
char to UV.

I failed to change the localization code in S_sublex_start:

    SAVEI8(PL_multi_close);

So on big-endian architectures only the most significant byte would be
localized.  That meant that effectively no localization would happen
for ASCII string delimiters.

In S_sublex_done:

	LEAVE;
	if (PL_multi_close == '<')
	    PL_parser->herelines += l - PL_multi_end;

That LEAVE undoes the localization.  '<' for PL_multi_close is a spe-
cial value that can only happen for here-docs.   The ->herelines line
makes sure that line numbers are correct after a here-doc.

What ended up happening was that s//<<END/e would throw off line num-
bers after the here-doc body.  PL_multi_close would end up being set
to '<', not '/', when the lexer was finishing up the s///, so it
treated it like a here-doc and screwed things up.  This resulted in
the test failures in ticket #128747.

I found that we also had a bug on little-endian machines.  But to get
the localization of the *least* sigificant byte to screw things up,
you have to try something other than s//<<END/e:

use utf8;
<<END;
${
#line 57
qq || }
END
warn;  # line 59

Replace the pipes with lightning bolts:

use utf8;
<<END;
${
#line 57
qq ϟϟ }
END
warn;  # line 7

and you get line 7 instead of 59.  In this case, the inner construct
has a delimiter character whose code is > 255, but only the lower
8 bits get localized.  So when the localization unwinds, you get
ord("ϟ") & 0xff | ord("<") instead of just ord("<"), resulting in the
here-doc line number handling being skipped.

This commit fixes the localization and adds the little-endian test.

diff --git a/t/uni/parser.t b/t/uni/parser.t
index ad905b0..a428786 100644 (file)
--- a/t/uni/parser.t
+++ b/t/uni/parser.t
@@ -9,7 +9,7 @@ BEGIN {
     skip_all_without_unicode_tables();
 }
 
-plan (tests => 52);
+plan (tests => 53);
 
 use utf8;
 use open qw( :utf8 :std );
@@ -228,3 +228,18 @@ like( $@, qr/Bad name after Ｆｏｏ'/, 'Bad name after Ｆｏｏ\'' );
 
         {stderr => 1}, "RT# 124216");
 }
+
+
+# New tests go here ^^^^^
+
+# Keep this test last, as it will mess up line number reporting for any
+# subsequent tests.
+
+<<END;
+${
+#line 57
+qq ϟϟ }
+END
+is __LINE__, 59, '#line directive and qq with uni delims inside heredoc';
+
+# Put new tests above the line number tests.

diff --git a/toke.c b/toke.c
index bf2ac05..2c87688 100644 (file)
--- a/toke.c
+++ b/toke.c
@@ -2365,7 +2365,7 @@ S_sublex_push(pTHX)
        SAVEI32(PL_parser->herelines);
        PL_parser->herelines = 0;
     }
-    SAVEI8(PL_multi_close);
+    SAVEIV(PL_multi_close);
     SAVEPPTR(PL_bufptr);
     SAVEPPTR(PL_bufend);
     SAVEPPTR(PL_oldbufptr);