Dest buffer needs to be bigger for utf16_to_utf8()

These undocumented functions require the destination buffer to have the
worst case size.  However that size (previously listed as 3/2 * input)
is wrong for EBCDIC.  Correct the comments, and the single use of these
in core.

These functions do not have a way to avoid overflowing, which strikes me
as wrong.

diff --git a/ext/XS-APItest/APItest.xs b/ext/XS-APItest/APItest.xs
index 99a3957..ffdc56c 100644 (file)
--- a/ext/XS-APItest/APItest.xs
+++ b/ext/XS-APItest/APItest.xs
@@ -2885,7 +2885,7 @@ utf16_to_utf8 (sv, ...)
            len = SvUV(ST(1));
        }
        /* Mortalise this right now, as we'll be testing croak()s  */
-       dest = sv_2mortal(newSV(len * 3 / 2 + 1));
+       dest = sv_2mortal(newSV(len * 2 + 1));
        if (ix) {
            utf16_to_utf8_reversed(source, (U8 *)SvPVX(dest), len, &got);
        } else {

diff --git a/toke.c b/toke.c
index da3ecdb..fd97292 100644 (file)
--- a/toke.c
+++ b/toke.c
@@ -11908,7 +11908,10 @@ S_utf16_textfilter(pTHX_ int idx, SV *sv, int maxlen)
 
        chars = SvCUR(utf16_buffer) >> 1;
        have = SvCUR(utf8_buffer);
-       SvGROW(utf8_buffer, have + chars * 3 + 1);
+
+        /* Assume the worst case size as noted by the functions: twice the
+         * number of input bytes */
+       SvGROW(utf8_buffer, have + chars * 4 + 1);
 
        if (reverse) {
            end = utf16_to_utf8_reversed((U8*)SvPVX(utf16_buffer),

diff --git a/utf8.c b/utf8.c
index 6107348..b731780 100644 (file)
--- a/utf8.c
+++ b/utf8.c
@@ -2364,10 +2364,19 @@ Perl_bytes_to_utf8(pTHX_ const U8 *s, STRLEN *lenp)
 }
 
 /*
- * Convert native (big-endian) or reversed (little-endian) UTF-16 to UTF-8.
+ * Convert native (big-endian) UTF-16 to UTF-8.  For reversed (little-endian),
+ * use utf16_to_utf8_reversed().
  *
- * Destination must be pre-extended to 3/2 source.  Do not use in-place.
- * We optimize for native, for obvious reasons. */
+ * UTF-16 requires 2 bytes for every code point below 0x10000; otherwise 4 bytes.
+ * UTF-8 requires 1-3 bytes for every code point below 0x1000; otherwise 4 bytes.
+ * UTF-EBCDIC requires 1-4 bytes for every code point below 0x1000; otherwise 4-5 bytes.
+ *
+ * These functions don't check for overflow.  The worst case is every code
+ * point in the input is 2 bytes, and requires 4 bytes on output.  (If the code
+ * is never going to run in EBCDIC, it is 2 bytes requiring 3 on output.)  Therefore the
+ * destination must be pre-extended to 2 times the source length.
+ *
+ * Do not use in-place.  We optimize for native, for obvious reasons. */
 
 U8*
 Perl_utf16_to_utf8(pTHX_ U8* p, U8* d, I32 bytelen, I32 *newlen)