This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Bump Encode::Unicode $VERSION
[perl5.git] / cpan / Encode / Unicode / Unicode.pm
CommitLineData
f2a2953c
JH
1package Encode::Unicode;
2
df1df145 3use strict;
f2a2953c 4use warnings;
a0d8a30e 5no warnings 'redefine';
f2a2953c 6
c2062197 7our $VERSION = do { my @r = ( q$Revision: 2.10 $ =~ /\d+/g ); sprintf "%d." . "%02d" x $#r, @r };
f2a2953c 8
85982a32 9use XSLoader;
d1256cb1 10XSLoader::load( __PACKAGE__, $VERSION );
df1df145 11
f2a2953c
JH
12#
13# Object Generator 8 transcoders all at once!
14#
df1df145 15
f2a2953c 16require Encode;
10c5ecbb 17
d1256cb1 18our %BOM_Unknown = map { $_ => 1 } qw(UTF-16 UTF-32);
a0d8a30e 19
d1256cb1
RGS
20for my $name (
21 qw(UTF-16 UTF-16BE UTF-16LE
22 UTF-32 UTF-32BE UTF-32LE
23 UCS-2BE UCS-2LE)
24 )
df1df145 25{
d1256cb1 26 my ( $size, $endian, $ucs2, $mask );
f2a2953c 27 $name =~ /^(\w+)-(\d+)(\w*)$/o;
d1256cb1
RGS
28 if ( $ucs2 = ( $1 eq 'UCS' ) ) {
29 $size = 2;
df1df145 30 }
d1256cb1
RGS
31 else {
32 $size = $2 / 8;
33 }
34 $endian = ( $3 eq 'BE' ) ? 'n' : ( $3 eq 'LE' ) ? 'v' : '';
f2a2953c
JH
35 $size == 4 and $endian = uc($endian);
36
d1256cb1
RGS
37 $Encode::Encoding{$name} = bless {
38 Name => $name,
39 size => $size,
40 endian => $endian,
41 ucs2 => $ucs2,
42 } => __PACKAGE__;
df1df145
JH
43}
44
369b9ffe 45use parent qw(Encode::Encoding);
f2a2953c 46
d1256cb1 47sub renew {
a0d8a30e 48 my $self = shift;
d1256cb1
RGS
49 $BOM_Unknown{ $self->name } or return $self;
50 my $clone = bless {%$self} => ref($self);
51 $clone->{renewed}++; # so the caller knows it is renewed.
a0d8a30e 52 return $clone;
f2a2953c
JH
53}
54
0a225b3c 55# There used to be a perl implementation of (en|de)code but with
a0d8a30e 56# XS version is ripe, perl version is zapped for optimal speed
f2a2953c 57
a0d8a30e
DK
58*decode = \&decode_xs;
59*encode = \&encode_xs;
df1df145
JH
60
611;
62__END__
67d7b5ef
JH
63
64=head1 NAME
65
0ab8f81e 66Encode::Unicode -- Various Unicode Transformation Formats
67d7b5ef
JH
67
68=cut
f2a2953c
JH
69
70=head1 SYNOPSIS
71
40bed538 72 use Encode qw/encode decode/;
f2a2953c
JH
73 $ucs2 = encode("UCS-2BE", $utf8);
74 $utf8 = decode("UCS-2BE", $ucs2);
75
76=head1 ABSTRACT
77
78This module implements all Character Encoding Schemes of Unicode that
79are officially documented by Unicode Consortium (except, of course,
80for UTF-8, which is a native format in perl).
81
82=over 4
83
84=item L<http://www.unicode.org/glossary/> says:
85
86I<Character Encoding Scheme> A character encoding form plus byte
1485817e 87serialization. There are Seven character encoding schemes in Unicode:
11067275 88UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32 (UCS-4), UTF-32BE (UCS-4BE) and
1485817e
DK
89UTF-32LE (UCS-4LE), and UTF-7.
90
91Since UTF-7 is a 7-bit (re)encoded version of UTF-16BE, It is not part of
92Unicode's Character Encoding Scheme. It is separately implemented in
93Encode::Unicode::UTF7. For details see L<Encode::Unicode::UTF7>.
f2a2953c
JH
94
95=item Quick Reference
96
97 Decodes from ord(N) Encodes chr(N) to...
98 octet/char BOM S.P d800-dfff ord > 0xffff \x{1abcd} ==
99 ---------------+-----------------+------------------------------
370462a2 100 UCS-2BE 2 N N is bogus Not Available
f2a2953c
JH
101 UCS-2LE 2 N N bogus Not Available
102 UTF-16 2/4 Y Y is S.P S.P BE/LE
103 UTF-16BE 2/4 N Y S.P S.P 0xd82a,0xdfcd
370462a2
RGS
104 UTF-16LE 2/4 N Y S.P S.P 0x2ad8,0xcddf
105 UTF-32 4 Y - is bogus As is BE/LE
106 UTF-32BE 4 N - bogus As is 0x0001abcd
107 UTF-32LE 4 N - bogus As is 0xcdab0100
f2a2953c
JH
108 UTF-8 1-4 - - bogus >= 4 octets \xf0\x9a\af\8d
109 ---------------+-----------------+------------------------------
110
111=back
112
113=head1 Size, Endianness, and BOM
114
0ab8f81e
JH
115You can categorize these CES by 3 criteria: size of each character,
116endianness, and Byte Order Mark.
f2a2953c 117
0ab8f81e 118=head2 by size
f2a2953c
JH
119
120UCS-2 is a fixed-length encoding with each character taking 16 bits.
0ab8f81e
JH
121It B<does not> support I<surrogate pairs>. When a surrogate pair
122is encountered during decode(), its place is filled with \x{FFFD}
123if I<CHECK> is 0, or the routine croaks if I<CHECK> is 1. When a
124character whose ord value is larger than 0xFFFF is encountered,
125its place is filled with \x{FFFD} if I<CHECK> is 0, or the routine
126croaks if I<CHECK> is 1.
127
128UTF-16 is almost the same as UCS-2 but it supports I<surrogate pairs>.
f2a2953c 129When it encounters a high surrogate (0xD800-0xDBFF), it fetches the
0ab8f81e
JH
130following low surrogate (0xDC00-0xDFFF) and C<desurrogate>s them to
131form a character. Bogus surrogates result in death. When \x{10000}
132or above is encountered during encode(), it C<ensurrogate>s them and
133pushes the surrogate pair to the output stream.
f2a2953c 134
11067275 135UTF-32 (UCS-4) is a fixed-length encoding with each character taking 32 bits.
0ab8f81e 136Since it is 32-bit, there is no need for I<surrogate pairs>.
f2a2953c 137
0ab8f81e 138=head2 by endianness
f2a2953c 139
0ab8f81e
JH
140The first (and now failed) goal of Unicode was to map all character
141repertoires into a fixed-length integer so that programmers are happy.
142Since each character is either a I<short> or I<long> in C, you have to
143pay attention to the endianness of each platform when you pass data
144to one another.
f2a2953c
JH
145
146Anything marked as BE is Big Endian (or network byte order) and LE is
0ab8f81e
JH
147Little Endian (aka VAX byte order). For anything not marked either
148BE or LE, a character called Byte Order Mark (BOM) indicating the
149endianness is prepended to the string.
f2a2953c 150
7237418a
RGS
151CAVEAT: Though BOM in utf8 (\xEF\xBB\xBF) is valid, it is meaningless
152and as of this writing Encode suite just leave it as is (\x{FeFF}).
153
f2a2953c
JH
154=over 4
155
fcb875d4 156=item BOM as integer when fetched in network byte order
f2a2953c 157
fcb875d4
JH
158 16 32 bits/char
159 -------------------------
160 BE 0xFeFF 0x0000FeFF
742555bd 161 LE 0xFFFe 0xFFFe0000
fcb875d4 162 -------------------------
f2a2953c
JH
163
164=back
151b5d36 165
0ab8f81e 166This modules handles the BOM as follows.
f2a2953c
JH
167
168=over 4
169
170=item *
171
172When BE or LE is explicitly stated as the name of encoding, BOM is
0ab8f81e 173simply treated as a normal character (ZERO WIDTH NO-BREAK SPACE).
f2a2953c
JH
174
175=item *
176
0ab8f81e
JH
177When BE or LE is omitted during decode(), it checks if BOM is at the
178beginning of the string; if one is found, the endianness is set to
179what the BOM says. If no BOM is found, the routine dies.
f2a2953c
JH
180
181=item *
182
183When BE or LE is omitted during encode(), it returns a BE-encoded
184string with BOM prepended. So when you want to encode a whole text
0ab8f81e
JH
185file, make sure you encode() the whole text at once, not line by line
186or each line, not file, will have a BOM prepended.
f2a2953c
JH
187
188=item *
189
0ab8f81e 190C<UCS-2> is an exception. Unlike others, this is an alias of UCS-2BE.
f2a2953c
JH
191UCS-2 is already registered by IANA and others that way.
192
fdd579e2 193=back
f2a2953c 194
fcb875d4 195=head1 Surrogate Pairs
f2a2953c 196
fcb875d4
JH
197To say the least, surrogate pairs were the biggest mistake of the
198Unicode Consortium. But according to the late Douglas Adams in I<The
199Hitchhiker's Guide to the Galaxy> Trilogy, C<In the beginning the
200Universe was created. This has made a lot of people very angry and
201been widely regarded as a bad move>. Their mistake was not of this
202magnitude so let's forgive them.
f2a2953c
JH
203
204(I don't dare make any comparison with Unicode Consortium and the
c731e18e
JH
205Vogons here ;) Or, comparing Encode to Babel Fish is completely
206appropriate -- if you can only stick this into your ear :)
f2a2953c 207
0ab8f81e 208Surrogate pairs were born when the Unicode Consortium finally
fcb875d4 209admitted that 16 bits were not big enough to hold all the world's
0ab8f81e 210character repertoires. But they already made UCS-2 16-bit. What
f2a2953c
JH
211do we do?
212
0ab8f81e
JH
213Back then, the range 0xD800-0xDFFF was not allocated. Let's split
214that range in half and use the first half to represent the C<upper
215half of a character> and the second half to represent the C<lower
216half of a character>. That way, you can represent 1024 * 1024 =
2171048576 more characters. Now we can store character ranges up to
218\x{10ffff} even with 16-bit encodings. This pair of half-character is
219now called a I<surrogate pair> and UTF-16 is the name of the encoding
220that embraces them.
f2a2953c 221
448e90bb 222Here is a formula to ensurrogate a Unicode character \x{10000} and
f2a2953c
JH
223above;
224
225 $hi = ($uni - 0x10000) / 0x400 + 0xD800;
226 $lo = ($uni - 0x10000) % 0x400 + 0xDC00;
227
228And to desurrogate;
229
230 $uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);
231
fcb875d4 232Note this move has made \x{D800}-\x{DFFF} into a forbidden zone but
40bed538 233perl does not prohibit the use of characters within this range. To perl,
fcb875d4
JH
234every one of \x{0000_0000} up to \x{ffff_ffff} (*) is I<a character>.
235
236 (*) or \x{ffff_ffff_ffff_ffff} if your perl is compiled with 64-bit
0ab8f81e 237 integer support!
f2a2953c 238
f9d05ba3
RGS
239=head1 Error Checking
240
241Unlike most encodings which accept various ways to handle errors,
242Unicode encodings simply croaks.
243
40bed538
DK
244 % perl -MEncode -e'$_ = "\xfe\xff\xd8\xd9\xda\xdb\0\n"' \
245 -e'Encode::from_to($_, "utf16","shift_jis", 0); print'
f9d05ba3 246 UTF-16:Malformed LO surrogate d8d9 at /path/to/Encode.pm line 184.
40bed538
DK
247 % perl -MEncode -e'$a = "BOM missing"' \
248 -e' Encode::from_to($a, "utf16", "shift_jis", 0); print'
f9d05ba3
RGS
249 UTF-16:Unrecognised BOM 424f at /path/to/Encode.pm line 184.
250
251Unlike other encodings where mappings are not one-to-one against
252Unicode, UTFs are supposed to map 100% against one another. So Encode
253is more strict on UTFs.
254
255Consider that "division by zero" of Encode :)
256
f2a2953c
JH
257=head1 SEE ALSO
258
1485817e 259L<Encode>, L<Encode::Unicode::UTF7>, L<http://www.unicode.org/glossary/>,
11067275 260L<http://www.unicode.org/unicode/faq/utf_bom.html>,
f2a2953c 261
b2deda17 262RFC 2781 L<http://www.ietf.org/rfc/rfc2781.txt>,
fdd579e2 263
11067275 264The whole Unicode standard L<http://www.unicode.org/unicode/uni2book/u2.html>
fdd579e2 265
fcb875d4 266Ch. 15, pp. 403 of C<Programming Perl (3rd Edition)>
40bed538 267by Larry Wall, Tom Christiansen, Jon Orwant;
fcb875d4
JH
268O'Reilly & Associates; ISBN 0-596-00027-8
269
fdd579e2 270=cut