value (UV), a double (NV), a string (PV), and another scalar (SV).
("PV" stands for "Pointer Value". You might think that it is misnamed
because it is described as pointing only to strings. However, it is
-possible to have it point to other things For example, it could point
+possible to have it point to other things. For example, it could point
to an array of UVs. But,
using it for non-strings requires care, as the underlying assumption of
much of the internals is that PVs are just for strings. Often, for
-example, a trailing NUL is tacked on automatically. The non-string use
+example, a trailing C<NUL> is tacked on automatically. The non-string use
is documented only in this paragraph.)
The seven routines are:
In the unlikely case of a SV requiring more complex initialization, you
can create an empty SV with newSV(len). If C<len> is 0 an empty SV of
type NULL is returned, else an SV of type PV is returned with len + 1 (for
-the NUL) bytes of storage allocated, accessible via SvPVX. In both cases
+the C<NUL>) bytes of storage allocated, accessible via SvPVX. In both cases
the SV has the undef value.
SV *sv = newSV(0); /* no storage allocated */
allow Perl to calculate the length by using C<sv_setpv> or by specifying
0 as the second argument to C<newSVpv>. Be warned, though, that Perl will
determine the string's length by using C<strlen>, which depends on the
-string terminating with a NUL character, and not otherwise containing
+string terminating with a C<NUL> character, and not otherwise containing
NULs.
The arguments of C<sv_setpvf> are processed like C<sprintf>, and the
The C<sv_set*()> functions are not generic enough to operate on values
that have "magic". See L<Magic Virtual Tables> later in this document.
-All SVs that contain strings should be terminated with a NUL character.
-If it is not NUL-terminated there is a risk of
+All SVs that contain strings should be terminated with a C<NUL> character.
+If it is not C<NUL>-terminated there is a risk of
core dumps and corruptions from code which passes the string to C
-functions or system calls which expect a NUL-terminated string.
-Perl's own functions typically add a trailing NUL for this reason.
+functions or system calls which expect a C<NUL>-terminated string.
+Perl's own functions typically add a trailing C<NUL> for this reason.
Nevertheless, you should be very careful when you pass a string stored
in an SV to a C function or system call.
used in this case. But that can be quite inefficient because C<PL_na> must
be accessed in thread-local storage in threaded Perl. In any case, remember
that Perl allows arbitrary strings of data that may both contain NULs and
-might not be terminated by a NUL.
+might not be terminated by a C<NUL>.
Also remember that C doesn't allow you to safely say C<foo(SvPV(s, len),
len);>. It might work with your
which will determine if more memory needs to be allocated. If so, it will
call the function C<sv_grow>. Note that C<SvGROW> can only increase, not
decrease, the allocated memory of an SV and that it does not automatically
-add space for the trailing NUL byte (perl's own string functions typically do
+add space for the trailing C<NUL> byte (perl's own string functions typically do
C<SvGROW(sv, len + 1)>).
+If you want to write to an existing SV's buffer and set its value to a
+string, use SvPV_force() or one of its variants to force the SV to be
+a PV. This will remove any of various types of non-stringness from
+the SV while preserving the content of the SV in the PV. This can be
+used, for example, to append data from an API function to a buffer
+without extra copying:
+
+ (void)SvPVbyte_force(sv, len);
+ s = SvGROW(sv, len + needlen + 1);
+ /* something that modifies up to needlen bytes at s+len, but
+ modifies newlen bytes
+ eg. newlen = read(fd, s + len, needlen);
+ ignoring errors for these examples
+ */
+ s[len + newlen] = '\0';
+ SvCUR_set(sv, len + newlen);
+ SvUTF8_off(sv);
+ SvSETMAGIC(sv);
+
+If you already have the data in memory or if you want to keep your
+code simple, you can use one of the sv_cat*() variants, such as
+sv_catpvn(). If you want to insert anywhere in the string you can use
+sv_insert() or sv_insert_flags().
+
+If you don't need the existing content of the SV, you can avoid some
+copying with:
+
+ sv_setpvn(sv, "", 0);
+ s = SvGROW(sv, needlen + 1);
+ /* something that modifies up to needlen bytes at s, but modifies
+ newlen bytes
+ eg. newlen = read(fd, s. needlen);
+ */
+ s[newlen] = '\0';
+ SvCUR_set(sv, newlen);
+ SvPOK_only(sv); /* also clears SVf_UTF8 */
+ SvSETMAGIC(sv);
+
+Again, if you already have the data in memory or want to avoid the
+complexity of the above, you can use sv_setpvn().
+
+If you have a buffer allocated with Newx() and want to set that as the
+SV's value, you can use sv_usepvn_flags(). That has some requirements
+if you want to avoid perl re-allocating the buffer to fit the trailing
+NUL:
+
+ Newx(buf, somesize+1, char);
+ /* ... fill in buf ... */
+ buf[somesize] = '\0';
+ sv_usepvn_flags(sv, buf, somesize, SV_SMAGIC | SV_HAS_TRAILING_NUL);
+ /* buf now belongs to perl, don't release it */
+
If you have an SV and want to know what kind of data Perl thinks is stored
in it, you can use the following macros to check the type of SV you have.
at C<SvPVX(sv) - SvIV(sv)> in memory and the PV pointer is pointing
into the middle of this allocated storage.
-This is best demonstrated by example:
+This is best demonstrated by example. Normally copy-on-write will prevent
+the substitution from operator from using this hack, but if you can craft a
+string for which copy-on-write is not possible, you can see it in play. In
+the current implementation, the final byte of a string buffer is used as a
+copy-on-write reference count. If the buffer is not big enough, then
+copy-on-write is skipped. First have a look at an empty string:
- % ./perl -Ilib -MDevel::Peek -le '$a="12345"; $a=~s/.//; Dump($a)'
- SV = PVIV(0x8128450) at 0x81340f0
+ % ./perl -Ilib -MDevel::Peek -le '$a=""; $a .= ""; Dump $a'
+ SV = PV(0x7ffb7c008a70) at 0x7ffb7c030390
+ REFCNT = 1
+ FLAGS = (POK,pPOK)
+ PV = 0x7ffb7bc05b50 ""\0
+ CUR = 0
+ LEN = 10
+
+Notice here the LEN is 10. (It may differ on your platform.) Extend the
+length of the string to one less than 10, and do a substitution:
+
+ % ./perl -Ilib -MDevel::Peek -le '$a=""; $a.="123456789"; $a=~s/.//; Dump($a)'
+ SV = PV(0x7ffa04008a70) at 0x7ffa04030390
REFCNT = 1
FLAGS = (POK,OOK,pPOK)
- IV = 1 (OFFSET)
- PV = 0x8135781 ( "1" . ) "2345"\0
- CUR = 4
- LEN = 5
+ OFFSET = 1
+ PV = 0x7ffa03c05b61 ( "\1" . ) "23456789"\0
+ CUR = 8
+ LEN = 9
-Here the number of bytes chopped off (1) is put into IV, and
-C<Devel::Peek::Dump> helpfully reminds us that this is an offset. The
+Here the number of bytes chopped off (1) is shown next as the OFFSET. The
portion of the string between the "real" and the "fake" beginnings is
shown in parentheses, and the values of C<SvCUR> and C<SvLEN> reflect
-the fake beginning, not the real one.
+the fake beginning, not the real one. (The first character of the string
+buffer happens to have changed to "\1" here, not "1", because the current
+implementation stores the offset count in the string buffer. This is
+subject to change.)
Something similar to the offset hack is performed on AVs to enable
efficient shifting and splicing off the beginning of the array; while
stored in your SV. The "p" stands for private.
There are various ways in which the private and public flags may differ.
-For example, a tied SV may have a valid underlying value in the IV slot
-(so SvIOKp is true), but the data should be accessed via the FETCH
-routine rather than directly, so SvIOK is false. Another is when
+For example, in perl 5.16 and earlier a tied SV may have a valid
+underlying value in the IV slot (so SvIOKp is true), but the data
+should be accessed via the FETCH routine rather than directly,
+so SvIOK is false. (In perl 5.18 onwards, tied scalars use
+the flags the same way as untied scalars.) Another is when
numeric conversion has occurred and precision has been lost: only the
private flag is set on 'lossy' values. So when an NV is converted to an
IV with loss, SvIOKp, SvNOKp and SvNOK will be set, while SvIOK wont be.
If the order of C<sv_setiv> and C<sv_setpv> had been reversed, then the
macro C<SvPOK_on> would need to be called instead of C<SvIOK_on>.
+=head2 Read-Only Values
+
+In Perl 5.16 and earlier, copy-on-write (see the next section) shared a
+flag bit with read-only scalars. So the only way to test whether
+C<sv_setsv>, etc., will raise a "Modification of a read-only value" error
+in those versions is:
+
+ SvREADONLY(sv) && !SvIsCOW(sv)
+
+Under Perl 5.18 and later, SvREADONLY only applies to read-only variables,
+and, under 5.20, copy-on-write scalars can also be read-only, so the above
+check is incorrect. You just want:
+
+ SvREADONLY(sv)
+
+If you need to do this check often, define your own macro like this:
+
+ #if PERL_VERSION >= 18
+ # define SvTRULYREADONLY(sv) SvREADONLY(sv)
+ #else
+ # define SvTRULYREADONLY(sv) (SvREADONLY(sv) && !SvIsCOW(sv))
+ #endif
+
+=head2 Copy on Write
+
+Perl implements a copy-on-write (COW) mechanism for scalars, in which
+string copies are not immediately made when requested, but are deferred
+until made necessary by one or the other scalar changing. This is mostly
+transparent, but one must take care not to modify string buffers that are
+shared by multiple SVs.
+
+You can test whether an SV is using copy-on-write with C<SvIsCOW(sv)>.
+
+You can force an SV to make its own copy of its string buffer by calling C<sv_force_normal(sv)> or SvPV_force_nolen(sv).
+
+If you want to make the SV drop its string buffer, use
+C<sv_force_normal_flags(sv, SV_COW_DROP_PV)> or simply
+C<sv_setsv(sv, NULL)>.
+
+All of these functions will croak on read-only scalars (see the previous
+section for more on those).
+
+To test that your code is behaving correctly and not modifying COW buffers,
+on systems that support L<mmap(2)> (i.e., Unix) you can configure perl with
+C<-Accflags=-DPERL_DEBUG_READONLY_COW> and it will turn buffer violations
+into crashes. You will find it to be marvellously slow, so you may want to
+skip perl's own tests.
+
=head2 Magic Variables
[This section still under construction. Ignore everything here. Post no
-------------------------- ------ -------------
\0 PERL_MAGIC_sv vtbl_sv Special scalar variable
# PERL_MAGIC_arylen vtbl_arylen Array length ($#ary)
- % PERL_MAGIC_rhash (none) extra data for restricted
+ % PERL_MAGIC_rhash (none) Extra data for restricted
hashes
- & PERL_MAGIC_proto (none) my sub prototype CV
+ * PERL_MAGIC_debugvar vtbl_debugvar $DB::single, signal, trace
+ vars
. PERL_MAGIC_pos vtbl_pos pos() lvalue
- : PERL_MAGIC_symtab (none) extra data for symbol
+ : PERL_MAGIC_symtab (none) Extra data for symbol
tables
- < PERL_MAGIC_backref vtbl_backref for weak ref data
- @ PERL_MAGIC_arylen_p (none) to move arylen out of XPVAV
+ < PERL_MAGIC_backref vtbl_backref For weak ref data
+ @ PERL_MAGIC_arylen_p (none) To move arylen out of XPVAV
B PERL_MAGIC_bm vtbl_regexp Boyer-Moore
(fast string search)
c PERL_MAGIC_overload_table vtbl_ovrld Holds overload table
P PERL_MAGIC_tied vtbl_pack Tied array or hash
p PERL_MAGIC_tiedelem vtbl_packelem Tied array or hash element
q PERL_MAGIC_tiedscalar vtbl_packelem Tied scalar or handle
- r PERL_MAGIC_qr vtbl_regexp precompiled qr// regex
+ r PERL_MAGIC_qr vtbl_regexp Precompiled qr// regex
S PERL_MAGIC_sig (none) %SIG hash
s PERL_MAGIC_sigelem vtbl_sigelem %SIG hash element
t PERL_MAGIC_taint vtbl_taint Taintedness
y PERL_MAGIC_defelem vtbl_defelem Shadow "foreach" iterator
variable / smart parameter
vivification
- ] PERL_MAGIC_checkcall vtbl_checkcall inlining/mutation of call
+ \ PERL_MAGIC_lvref vtbl_lvref Lvalue reference
+ constructor
+ ] PERL_MAGIC_checkcall vtbl_checkcall Inlining/mutation of call
to this CV
~ PERL_MAGIC_ext (none) Available for use by
extensions
The perl tie function associates a variable with an object that implements
the various GET, SET, etc methods. To perform the equivalent of the perl
tie function from an XSUB, you must mimic this behaviour. The code below
-carries out the necessary steps - firstly it creates a new hash, and then
+carries out the necessary steps -- firstly it creates a new hash, and then
creates a second hash which it blesses into the class which will implement
the tie methods. Lastly it ties the two hashes together, and returns a
reference to the new tied hash. Note that the code below does NOT call the
For a detailed description of calling conventions from C to Perl,
consult L<perlcall>.
-=head2 Memory Allocation
-
-=head3 Allocation
-
-All memory meant to be used with the Perl API functions should be manipulated
-using the macros described in this section. The macros provide the necessary
-transparency between differences in the actual malloc implementation that is
-used within perl.
-
-It is suggested that you enable the version of malloc that is distributed
-with Perl. It keeps pools of various sizes of unallocated memory in
-order to satisfy allocation requests more quickly. However, on some
-platforms, it may cause spurious malloc or free errors.
-
-The following three macros are used to initially allocate memory :
-
- Newx(pointer, number, type);
- Newxc(pointer, number, type, cast);
- Newxz(pointer, number, type);
-
-The first argument C<pointer> should be the name of a variable that will
-point to the newly allocated memory.
-
-The second and third arguments C<number> and C<type> specify how many of
-the specified type of data structure should be allocated. The argument
-C<type> is passed to C<sizeof>. The final argument to C<Newxc>, C<cast>,
-should be used if the C<pointer> argument is different from the C<type>
-argument.
-
-Unlike the C<Newx> and C<Newxc> macros, the C<Newxz> macro calls C<memzero>
-to zero out all the newly allocated memory.
-
-=head3 Reallocation
-
- Renew(pointer, number, type);
- Renewc(pointer, number, type, cast);
- Safefree(pointer)
-
-These three macros are used to change a memory buffer size or to free a
-piece of memory no longer needed. The arguments to C<Renew> and C<Renewc>
-match those of C<New> and C<Newc> with the exception of not needing the
-"magic cookie" argument.
-
-=head3 Moving
-
- Move(source, dest, number, type);
- Copy(source, dest, number, type);
- Zero(dest, number, type);
-
-These three macros are used to move, copy, or zero out previously allocated
-memory. The C<source> and C<dest> arguments point to the source and
-destination starting points. Perl will move, copy, or zero out C<number>
-instances of the size of the C<type> data structure (using the C<sizeof>
-function).
-
-=head2 PerlIO
-
-The most recent development releases of Perl have been experimenting with
-removing Perl's dependency on the "normal" standard I/O suite and allowing
-other stdio implementations to be used. This involves creating a new
-abstraction layer that then calls whichever implementation of stdio Perl
-was compiled with. All XSUBs should now use the functions in the PerlIO
-abstraction layer and not make any assumptions about what kind of stdio
-is being used.
-
-For a complete description of the PerlIO abstraction, consult L<perlapio>.
-
=head2 Putting a C value on Perl stack
A lot of opcodes (this is an elementary operation in the internal perl
targets for opcodes. A previous version of this document
stated that one can deduce that an SV lives on a scratchpad
by looking on its flags: lexicals have C<SVs_PADMY> set, and
-I<target>s have C<SVs_PADTMP> set. But this have never been fully true.
+I<target>s have C<SVs_PADTMP> set. But this has never been fully true.
C<SVs_PADMY> could be set on a variable that no longer resides in any pad.
While I<target>s do have C<SVs_PADTMP> set, it can also be set on variables
-that have never resided in a pad, but nonetheless act like I<target>s.
+that have never resided in a pad, but nonetheless act like I<target>s. As
+of perl 5.21.5, the C<SVs_PADMY> flag is no longer used and is defined as
+0. C<SvPADMY()> now returns true for anything without C<SVs_PADTMP>.
The correspondence between OPs and I<target>s is not 1-to-1. Different
OPs in the compile tree of the unit can use the same target, if this
The I<target>s on this scratchpad are C<undef>s, but they are already
marked with correct flags.
+=head1 Memory Allocation
+
+=head2 Allocation
+
+All memory meant to be used with the Perl API functions should be manipulated
+using the macros described in this section. The macros provide the necessary
+transparency between differences in the actual malloc implementation that is
+used within perl.
+
+It is suggested that you enable the version of malloc that is distributed
+with Perl. It keeps pools of various sizes of unallocated memory in
+order to satisfy allocation requests more quickly. However, on some
+platforms, it may cause spurious malloc or free errors.
+
+The following three macros are used to initially allocate memory :
+
+ Newx(pointer, number, type);
+ Newxc(pointer, number, type, cast);
+ Newxz(pointer, number, type);
+
+The first argument C<pointer> should be the name of a variable that will
+point to the newly allocated memory.
+
+The second and third arguments C<number> and C<type> specify how many of
+the specified type of data structure should be allocated. The argument
+C<type> is passed to C<sizeof>. The final argument to C<Newxc>, C<cast>,
+should be used if the C<pointer> argument is different from the C<type>
+argument.
+
+Unlike the C<Newx> and C<Newxc> macros, the C<Newxz> macro calls C<memzero>
+to zero out all the newly allocated memory.
+
+=head2 Reallocation
+
+ Renew(pointer, number, type);
+ Renewc(pointer, number, type, cast);
+ Safefree(pointer)
+
+These three macros are used to change a memory buffer size or to free a
+piece of memory no longer needed. The arguments to C<Renew> and C<Renewc>
+match those of C<New> and C<Newc> with the exception of not needing the
+"magic cookie" argument.
+
+=head2 Moving
+
+ Move(source, dest, number, type);
+ Copy(source, dest, number, type);
+ Zero(dest, number, type);
+
+These three macros are used to move, copy, or zero out previously allocated
+memory. The C<source> and C<dest> arguments point to the source and
+destination starting points. Perl will move, copy, or zero out C<number>
+instances of the size of the C<type> data structure (using the C<sizeof>
+function).
+
+=head1 PerlIO
+
+The most recent development releases of Perl have been experimenting with
+removing Perl's dependency on the "normal" standard I/O suite and allowing
+other stdio implementations to be used. This involves creating a new
+abstraction layer that then calls whichever implementation of stdio Perl
+was compiled with. All XSUBs should now use the functions in the PerlIO
+abstraction layer and not make any assumptions about what kind of stdio
+is being used.
+
+For a complete description of the PerlIO abstraction, consult L<perlapio>.
+
=head1 Compiled code
=head2 Code tree
op is a C<LISTOP>, which has any number of children. In this case, the
first child is pointed to by C<op_first> and the last child by
C<op_last>. The children in between can be found by iteratively
-following the C<op_sibling> pointer from the first child to the last.
+following the C<OpSIBLING> pointer from the first child to the last (but
+see below).
-There are also
two other op types: a C<PMOP> holds a regular expression,
+There are also
some other op types: a C<PMOP> holds a regular expression,
and has no children, and a C<LOOP> may or may not have children. If the
C<op_children> field is non-zero, it behaves like a C<LISTOP>. To
complicate matters, if a C<UNOP> is actually a C<null> op after
optimization (see L</Compile pass 2: context propagation>) it will still
have children in accordance with its former type.
+Finally, there is a C<LOGOP>, or logic op. Like a C<LISTOP>, this has one
+or more children, but it doesn't have an C<op_last> field: so you have to
+follow C<op_first> and then the C<OpSIBLING> chain itself to find the
+last child. Instead it has an C<op_other> field, which is comparable to
+the C<op_next> field described below, and represents an alternate
+execution path. Operators like C<and>, C<or> and C<?> are C<LOGOP>s. Note
+that in general, C<op_other> may not point to any of the direct children
+of the C<LOGOP>.
+
+Starting in version 5.21.2, perls built with the experimental
+define C<-DPERL_OP_PARENT> add an extra boolean flag for each op,
+C<op_moresib>. When not set, this indicates that this is the last op in an
+C<OpSIBLING> chain. This frees up the C<op_sibling> field on the last
+sibling to point back to the parent op. Under this build, that field is
+also renamed C<op_sibparent> to reflect its joint role. The macro
+C<OpSIBLING(o)> wraps this special behaviour, and always returns NULL on
+the last sibling. With this build the C<op_parent(o)> function can be
+used to find the parent of any op. Thus for forward compatibility, you
+should always use the C<OpSIBLING(o)> macro rather than accessing
+C<op_sibling> directly.
+
Another way to examine the tree is to use a compiler back-end module, such
as L<B::Concise>.
creates two scopes: the first starts at the C<(> and has C<full == 1>,
the second starts at the C<{> and has C<full == 0>. Both end at the
-C<}>, so calls to C<start> and C<pre/post_end> will match. Anything
+C<}>, so calls to C<start> and C<pre>/C<post_end> will match. Anything
pushed onto the save stack by this hook will be popped just before the
scope ends (between the C<pre_> and C<post_end> hooks, in fact).
is also possible to use the C<BhkDISABLE> and C<BhkENABLE> macros to
temporarily switch entries on and off. You should also be aware that
generally speaking at least one scope will have opened before your
-extension is loaded, so you will see some C<pre/post_end> pairs that
+extension is loaded, so you will see some C<pre>/C<post_end> pairs that
didn't have a matching C<start>.
=head1 Examining internal data structures with the C<dump> functions
to use C<dVAR> in your coding to "declare the global variables"
when you are using them. dTHX does this for you automatically.
-To see whether you have non-const data you can use a BSD-compatible C<nm>:
+To see whether you have non-const data you can use a BSD (or GNU)
+compatible C<nm>:
nm libperl.a | grep -v ' [TURtr] '
-If this displays any C<D> or C<d> symbols, you have non-const data.
+If this displays any C<D> or C<d> symbols (or possibly C<C> or C<c>),
+you have non-const data. The symbols the C<grep> removed are as follows:
+C<Tt> are I<text>, or code, the C<Rr> are I<read-only> (const) data,
+and the C<U> is <undefined>, external symbols referred to.
+
+The test F<t/porting/libperl.t> does this kind of symbol sanity
+checking on C<libperl.a>.
For backward compatibility reasons defining just PERL_GLOBAL_STRUCT
doesn't actually hide all symbols inside a big global struct: some
The IVdf will expand to whatever is the correct format for the IVs.
+Note that there are different "long doubles": Perl will use
+whatever the compiler has.
+
If you are printing addresses of pointers, use UVxf combined
with PTR2UV(), do not use %lx or %p.
=head2 Source Documentation
There's an effort going on to document the internal functions and
-automatically produce reference manuals from them - L<perlapi> is one
+automatically produce reference manuals from them -
- L<perlapi> is one
such manual which details all the functions which are available to XS
writers. L<perlintern> is the autogenerated manual for the functions
which are not part of the API and are supposedly for internal use only.
a variable number of bytes to represent a character. You can learn more
about Unicode and Perl's Unicode model in L<perlunicode>.
+(On EBCDIC platforms, Perl uses instead UTF-EBCDIC, which is a form of
+UTF-8 adapted for EBCDIC platforms. Below, we just talk about UTF-8.
+UTF-EBCDIC is like UTF-8, but the details are different. The macros
+hide the differences from you, just remember that the particular numbers
+and bit patterns presented below will differ in UTF-EBCDIC.)
+
=head2 How can I recognise a UTF-8 string?
You can't. This is because UTF-8 data is stored in bytes just like
non-UTF-8 data. The Unicode character 200, (C<0xC8> for you hex types)
capital E with a grave accent, is represented by the two bytes
C<v196.172>. Unfortunately, the non-Unicode string C<chr(196).chr(172)>
-has that byte sequence as well. So you can't tell just by looking - this
+has that byte sequence as well. So you can't tell just by looking -- this
is what makes Unicode input an interesting problem.
In general, you either have to know what you're dealing with, or you
have to guess. The API function C<is_utf8_string> can help; it'll tell
-you if a string contains only valid UTF-8 characters. However, it can't
-do the work for you. On a character-by-character basis,
-C<is_utf8_char_buf>
+you if a string contains only valid UTF-8 characters, and the chances
+of a non-UTF-8 string looking like valid UTF-8 become very small very
+quickly with increasing string length. On a character-by-character
+basis, C<isUTF8_CHAR>
will tell you whether the current character in a string is valid UTF-8.
=head2 How does UTF-8 represent Unicode characters?
byte, just like good ol' ASCII. Character 128 is stored as
C<v194.128>; this continues up to character 191, which is
C<v194.191>. Now we've run out of bits (191 is binary
-C<10111111>) so we move on; 192 is C<v195.128>. And
+C<10111111>) so we move on; character 192 is C<v195.128>. And
so it goes on, moving to three bytes at character 2048.
+L<perlunicode/Unicode Encodings> has pictures of how this works.
Assuming you know you're dealing with a UTF-8 string, you can find out
how long the first character in it is with the C<UTF8SKIP> macro:
All bytes in a multi-byte UTF-8 character will have the high bit set,
so you can test if you need to do something special with this
-character like this (the UTF8_IS_INVARIANT() is a macro that tests
+character like this (the C<UTF8_IS_INVARIANT()> is a macro that tests
whether the byte is encoded as a single byte even in UTF-8):
U8 *utf;
value of the character; the inverse function C<uvchr_to_utf8> is available
for putting a UV into UTF-8:
- if (!UTF8_IS_INVARIANT(uv))
+ if (!UVCHR_IS_INVARIANT(uv))
/* Must treat this as UTF8 */
utf8 = uvchr_to_utf8(utf8, uv);
else
that character, you can never match a C<chr(200)> in a non-UTF-8 string.
So don't do that!
+(Note that we don't have to test for invariant characters in the
+examples above. The functions work on any well-formed UTF-8 input.
+It's just that its faster to avoid the function overhead when it's not
+needed.)
+
=head2 How does Perl store UTF-8 strings?
-Currently, Perl deals with Unicode strings and non-Unicode strings
+Currently, Perl deals with UTF-8 strings and non-UTF-8 strings
slightly differently. A flag in the SV, C<SVf_UTF8>, indicates that the
string is internally encoded as UTF-8. Without it, the byte value is the
-codepoint number and vice versa (in other words, the string is encoded
-as iso-8859-1, but C<use feature 'unicode_strings'> is needed to get iso-8859-1
-semantics). You can check and manipulate this flag with the
+codepoint number and vice versa. This flag is only meaningful if the SV
+is C<SvPOK> or immediately after stringification via C<SvPV> or a
+similar macro. You can check and manipulate this flag with the
following macros:
SvUTF8(sv)
SvUTF8_off(sv)
This flag has an important effect on Perl's treatment of the string: if
-Unicode data is not properly distinguished, regular expressions,
+UTF-8 data is not properly distinguished, regular expressions,
C<length>, C<substr> and other string handling operations will have
-undesirable results.
+undesirable (wrong) results.
The problem comes when you have, for instance, a string that isn't
-flagged as UTF-8, and contains a byte sequence that could be UTF-8 -
+flagged as UTF-8, and contains a byte sequence that could be UTF-8 --
especially when combining non-UTF-8 and UTF-8 strings.
-Never forget that the C<SVf_UTF8> flag is separate to the PV value; you
-need be sure you don't accidentally knock it off while you're
+Never forget that the C<SVf_UTF8> flag is separate from the PV value; you
+need to be sure you don't accidentally knock it off while you're
manipulating SVs. More specifically, you cannot expect to do this:
SV *sv;
The C<char*> string does not tell you the whole story, and you can't
copy or reconstruct an SV just by copying the string value. Check if the
-old SV has the UTF8 flag set, and act accordingly:
+old SV has the UTF8 flag set (I<after> the C<SvPV> call), and act
+accordingly:
p = SvPV(sv, len);
- frobnicate(p);
+ is_utf8 = SvUTF8(sv);
+ frobnicate(p, is_utf8);
nsv = newSVpvn(p, len);
- if (SvUTF8(sv))
+ if (is_utf8)
SvUTF8_on(nsv);
-In fact, your C<frobnicate> function should be made aware of whether or
-not it's dealing with UTF-8 data, so that it can handle the string
-appropriately.
+In the above, your C<frobnicate> function has been changed to be made
+aware of whether or not it's dealing with UTF-8 data, so that it can
+handle the string appropriately.
Since just passing an SV to an XS function and copying the data of
the SV is not enough to copy the UTF8 flags, even less right is just
-passing a C<char *> to an XS function.
+passing a S<C<char *>> to an XS function.
+
+For full generality, use the L<C<DO_UTF8>|perlapi/DO_UTF8> macro to see if the
+string in an SV is to be I<treated> as UTF-8. This takes into account
+if the call to the XS function is being made from within the scope of
+L<S<C<use bytes>>|bytes>. If so, the underlying bytes that comprise the
+UTF-8 string are to be exposed, rather than the character they
+represent. But this pragma should only really be used for debugging and
+perhaps low-level testing at the byte level. Hence most XS code need
+not concern itself with this, but various areas of the perl core do need
+to support it.
+
+And this isn't the whole story. Starting in Perl v5.12, strings that
+aren't encoded in UTF-8 may also be treated as Unicode under various
+conditions (see L<perlunicode/ASCII Rules versus Unicode Rules>).
+This is only really a problem for characters whose ordinals are between
+128 and 255, and their behavior varies under ASCII versus Unicode rules
+in ways that your code cares about (see L<perlunicode/The "Unicode Bug">).
+There is no published API for dealing with this, as it is subject to
+change, but you can look at the code for C<pp_lc> in F<pp.c> for an
+example as to how it's currently done.
=head2 How do I convert a string to UTF-8?
If you're mixing UTF-8 and non-UTF-8 strings, it is necessary to upgrade
-one of the strings to UTF-8. If you've got an SV, the easiest way to do
+the non-UTF-8 strings to UTF-8. If you've got an SV, the easiest way to do
this is:
sv_utf8_upgrade(sv);
the string contains any characters above 255 that can't be represented
in a single byte.
+=head2 How do I compare strings?
+
+L<perlapi/sv_cmp> and L<perlapi/sv_cmp_flags> do a lexigraphic
+comparison of two SV's, and handle UTF-8ness properly. Note, however,
+that Unicode specifies a much fancier mechanism for collation, available
+via the L<Unicode::Collate> module.
+
+To just compare two strings for equality/non-equality, you can just use
+L<C<memEQ()>|perlapi/memEQ> and L<C<memNE()>|perlapi/memEQ> as usual,
+except the strings must be both UTF-8 or not UTF-8 encoded.
+
+To compare two strings case-insensitively, use
+L<C<foldEQ_utf8()>|perlapi/foldEQ_utf8> (the strings don't have to have
+the same UTF-8ness).
+
=head2 Is there anything else I need to know?
Not really. Just remember these things:
=item *
-There's no way to tell if a string is UTF-8 or not. You can tell if an SV
-is UTF-8 by looking at its C<SvUTF8> flag. Don't forget to set the flag if
-something should be UTF-8. Treat the flag as part of the PV, even though
-it's not - if you pass on the PV to somewhere, pass on the flag too.
+There's no way to tell if a S<C<char *>> or S<C<U8 *>> string is UTF-8
+or not. But you can tell if an SV is to be treated as UTF-8 by calling
+C<DO_UTF8> on it, after stringifying it with C<SvPV> or a similar
+macro. And, you can tell if SV is actually UTF-8 (even if it is not to
+be treated as such) by looking at its C<SvUTF8> flag (again after
+stringifying it). Don't forget to set the flag if something should be
+UTF-8.
+Treat the flag as part of the PV, even though it's not -- if you pass on
+the PV to somewhere, pass on the flag too.
=item *
=item *
-When writing a character
C<uv> to a UTF-8 string, B<always> use
-C<uvchr_to_utf8>, unless C<UTF8_IS_INVARIANT(uv))> in which case
+When writing a character
UV to a UTF-8 string, B<always> use
+C<uvchr_to_utf8>, unless C<UVCHR_IS_INVARIANT(uv))> in which case
you can use C<*s = uv>.
=item *
This feature is implemented as a new op type, C<OP_CUSTOM>. The Perl
core does not "know" anything special about this op type, and so it will
not be involved in any optimizations. This also means that you can
-define your custom ops to be any op structure - unary, binary, list and
-so on - you like.
+define your custom ops to be any op structure -- unary, binary, list and
+so on -- you like.
It's important to know what custom operators won't do for you. They
won't let you add new syntax to Perl, directly. They won't even let you
https://perl5.git.perl.org / perl5.git / blobdiff