The perl tie function associates a variable with an object that implements
the various GET, SET, etc methods. To perform the equivalent of the perl
tie function from an XSUB, you must mimic this behaviour. The code below
-carries out the necessary steps - firstly it creates a new hash, and then
+carries out the necessary steps -- firstly it creates a new hash, and then
creates a second hash which it blesses into the class which will implement
the tie methods. Lastly it ties the two hashes together, and returns a
reference to the new tied hash. Note that the code below does NOT call the
creates two scopes: the first starts at the C<(> and has C<full == 1>,
the second starts at the C<{> and has C<full == 0>. Both end at the
-C<}>, so calls to C<start> and C<pre/post_end> will match. Anything
+C<}>, so calls to C<start> and C<pre>/C<post_end> will match. Anything
pushed onto the save stack by this hook will be popped just before the
scope ends (between the C<pre_> and C<post_end> hooks, in fact).
is also possible to use the C<BhkDISABLE> and C<BhkENABLE> macros to
temporarily switch entries on and off. You should also be aware that
generally speaking at least one scope will have opened before your
-extension is loaded, so you will see some C<pre/post_end> pairs that
+extension is loaded, so you will see some C<pre>/C<post_end> pairs that
didn't have a matching C<start>.
=head1 Examining internal data structures with the C<dump> functions
=head2 Source Documentation
There's an effort going on to document the internal functions and
-automatically produce reference manuals from them - L<perlapi> is one
+automatically produce reference manuals from them -
- L<perlapi> is one
such manual which details all the functions which are available to XS
writers. L<perlintern> is the autogenerated manual for the functions
which are not part of the API and are supposedly for internal use only.
a variable number of bytes to represent a character. You can learn more
about Unicode and Perl's Unicode model in L<perlunicode>.
+(On EBCDIC platforms, Perl uses instead UTF-EBCDIC, which is a form of
+UTF-8 adapted for EBCDIC platforms. Below, we just talk about UTF-8.
+UTF-EBCDIC is like UTF-8, but the details are different. The macros
+hide the differences from you, just remember that the particular numbers
+and bit patterns presented below will differ in UTF-EBCDIC.)
+
=head2 How can I recognise a UTF-8 string?
You can't. This is because UTF-8 data is stored in bytes just like
non-UTF-8 data. The Unicode character 200, (C<0xC8> for you hex types)
capital E with a grave accent, is represented by the two bytes
C<v196.172>. Unfortunately, the non-Unicode string C<chr(196).chr(172)>
-has that byte sequence as well. So you can't tell just by looking - this
+has that byte sequence as well. So you can't tell just by looking -- this
is what makes Unicode input an interesting problem.
In general, you either have to know what you're dealing with, or you
have to guess. The API function C<is_utf8_string> can help; it'll tell
-you if a string contains only valid UTF-8 characters. However, it can't
-do the work for you. On a character-by-character basis,
-C<isUTF8_CHAR>
+you if a string contains only valid UTF-8 characters, and the chances
+of a non-UTF-8 string looking like valid UTF-8 become very small very
+quickly with increasing string length. On a character-by-character
+basis, C<isUTF8_CHAR>
will tell you whether the current character in a string is valid UTF-8.
=head2 How does UTF-8 represent Unicode characters?
byte, just like good ol' ASCII. Character 128 is stored as
C<v194.128>; this continues up to character 191, which is
C<v194.191>. Now we've run out of bits (191 is binary
-C<10111111>) so we move on; 192 is C<v195.128>. And
+C<10111111>) so we move on; character 192 is C<v195.128>. And
so it goes on, moving to three bytes at character 2048.
+L<perlunicode/Unicode Encodings> has pictures of how this works.
Assuming you know you're dealing with a UTF-8 string, you can find out
how long the first character in it is with the C<UTF8SKIP> macro:
All bytes in a multi-byte UTF-8 character will have the high bit set,
so you can test if you need to do something special with this
-character like this (the UTF8_IS_INVARIANT() is a macro that tests
+character like this (the C<UTF8_IS_INVARIANT()> is a macro that tests
whether the byte is encoded as a single byte even in UTF-8):
U8 *utf;
value of the character; the inverse function C<uvchr_to_utf8> is available
for putting a UV into UTF-8:
- if (!UTF8_IS_INVARIANT(uv))
+ if (!UVCHR_IS_INVARIANT(uv))
/* Must treat this as UTF8 */
utf8 = uvchr_to_utf8(utf8, uv);
else
that character, you can never match a C<chr(200)> in a non-UTF-8 string.
So don't do that!
+(Note that we don't have to test for invariant characters in the
+examples above. The functions work on any well-formed UTF-8 input.
+It's just that its faster to avoid the function overhead when it's not
+needed.)
+
=head2 How does Perl store UTF-8 strings?
-Currently, Perl deals with Unicode strings and non-Unicode strings
+Currently, Perl deals with UTF-8 strings and non-UTF-8 strings
slightly differently. A flag in the SV, C<SVf_UTF8>, indicates that the
string is internally encoded as UTF-8. Without it, the byte value is the
-codepoint number and vice versa (in other words, the string is encoded
-as iso-8859-1, but C<use feature 'unicode_strings'> is needed to get iso-8859-1
-semantics). This flag is only meaningful if the SV is C<SvPOK>
-or immediately after stringification via C<SvPV> or a similar
-macro. You can check and manipulate this flag with the
+codepoint number and vice versa. This flag is only meaningful if the SV
+is C<SvPOK> or immediately after stringification via C<SvPV> or a
+similar macro. You can check and manipulate this flag with the
following macros:
SvUTF8(sv)
SvUTF8_off(sv)
This flag has an important effect on Perl's treatment of the string: if
-Unicode data is not properly distinguished, regular expressions,
+UTF-8 data is not properly distinguished, regular expressions,
C<length>, C<substr> and other string handling operations will have
-undesirable results.
+undesirable (wrong) results.
The problem comes when you have, for instance, a string that isn't
-flagged as UTF-8, and contains a byte sequence that could be UTF-8 -
+flagged as UTF-8, and contains a byte sequence that could be UTF-8 --
especially when combining non-UTF-8 and UTF-8 strings.
-Never forget that the C<SVf_UTF8> flag is separate to the PV value; you
-need be sure you don't accidentally knock it off while you're
+Never forget that the C<SVf_UTF8> flag is separate from the PV value; you
+need to be sure you don't accidentally knock it off while you're
manipulating SVs. More specifically, you cannot expect to do this:
SV *sv;
accordingly:
p = SvPV(sv, len);
- frobnicate(p);
+ is_utf8 = SvUTF8(sv);
+ frobnicate(p, is_utf8);
nsv = newSVpvn(p, len);
- if (SvUTF8(sv))
+ if (is_utf8)
SvUTF8_on(nsv);
-In fact, your C<frobnicate> function should be made aware of whether or
-not it's dealing with UTF-8 data, so that it can handle the string
-appropriately.
+In the above, your C<frobnicate> function has been changed to be made
+aware of whether or not it's dealing with UTF-8 data, so that it can
+handle the string appropriately.
Since just passing an SV to an XS function and copying the data of
the SV is not enough to copy the UTF8 flags, even less right is just
-passing a C<char *> to an XS function.
+passing a S<C<char *>> to an XS function.
+
+For full generality, use the L<C<DO_UTF8>|perlapi/DO_UTF8> macro to see if the
+string in an SV is to be I<treated> as UTF-8. This takes into account
+if the call to the XS function is being made from within the scope of
+L<S<C<use bytes>>|bytes>. If so, the underlying bytes that comprise the
+UTF-8 string are to be exposed, rather than the character they
+represent. But this pragma should only really be used for debugging and
+perhaps low-level testing at the byte level. Hence most XS code need
+not concern itself with this, but various areas of the perl core do need
+to support it.
+
+And this isn't the whole story. Starting in Perl v5.12, strings that
+aren't encoded in UTF-8 may also be treated as Unicode under various
+conditions (see L<perlunicode/ASCII Rules versus Unicode Rules>).
+This is only really a problem for characters whose ordinals are between
+128 and 255, and their behavior varies under ASCII versus Unicode rules
+in ways that your code cares about (see L<perlunicode/The "Unicode Bug">).
+There is no published API for dealing with this, as it is subject to
+change, but you can look at the code for C<pp_lc> in F<pp.c> for an
+example as to how it's currently done.
=head2 How do I convert a string to UTF-8?
If you're mixing UTF-8 and non-UTF-8 strings, it is necessary to upgrade
-one of the strings to UTF-8. If you've got an SV, the easiest way to do
+the non-UTF-8 strings to UTF-8. If you've got an SV, the easiest way to do
this is:
sv_utf8_upgrade(sv);
the string contains any characters above 255 that can't be represented
in a single byte.
+=head2 How do I compare strings?
+
+L<perlapi/sv_cmp> and L<perlapi/sv_cmp_flags> do a lexigraphic
+comparison of two SV's, and handle UTF-8ness properly. Note, however,
+that Unicode specifies a much fancier mechanism for collation, available
+via the L<Unicode::Collate> module.
+
+To just compare two strings for equality/non-equality, you can just use
+L<C<memEQ()>|perlapi/memEQ> and L<C<memNE()>|perlapi/memEQ> as usual,
+except the strings must be both UTF-8 or not UTF-8 encoded.
+
+To compare two strings case-insensitively, use
+L<C<foldEQ_utf8()>|perlapi/foldEQ_utf8> (the strings don't have to have
+the same UTF-8ness).
+
=head2 Is there anything else I need to know?
Not really. Just remember these things:
=item *
-There's no way to tell if a string is UTF-8 or not. You can tell if an SV
-is UTF-8 by looking at its C<SvUTF8> flag after stringifying it
-with C<SvPV> or a similar macro. Don't forget to set the flag if
-something should be UTF-8. Treat the flag as part of the PV, even though
-it's not - if you pass on the PV to somewhere, pass on the flag too.
+There's no way to tell if a S<C<char *>> or S<C<U8 *>> string is UTF-8
+or not. But you can tell if an SV is to be treated as UTF-8 by calling
+C<DO_UTF8> on it, after stringifying it with C<SvPV> or a similar
+macro. And, you can tell if SV is actually UTF-8 (even if it is not to
+be treated as such) by looking at its C<SvUTF8> flag (again after
+stringifying it). Don't forget to set the flag if something should be
+UTF-8.
+Treat the flag as part of the PV, even though it's not -- if you pass on
+the PV to somewhere, pass on the flag too.
=item *
=item *
-When writing a character
C<uv> to a UTF-8 string, B<always> use
-C<uvchr_to_utf8>, unless C<UTF8_IS_INVARIANT(uv))> in which case
+When writing a character
UV to a UTF-8 string, B<always> use
+C<uvchr_to_utf8>, unless C<UVCHR_IS_INVARIANT(uv))> in which case
you can use C<*s = uv>.
=item *
This feature is implemented as a new op type, C<OP_CUSTOM>. The Perl
core does not "know" anything special about this op type, and so it will
not be involved in any optimizations. This also means that you can
-define your custom ops to be any op structure - unary, binary, list and
-so on - you like.
+define your custom ops to be any op structure -- unary, binary, list and
+so on -- you like.
It's important to know what custom operators won't do for you. They
won't let you add new syntax to Perl, directly. They won't even let you
https://perl5.git.perl.org / perl5.git / blobdiff