Commit | Line | Data |
---|---|---|
66cbab2c KW |
1 | =encoding utf8 |
2 | ||
5f05dabc | 3 | =head1 NAME |
4 | ||
b0c42ed9 | 5 | perllocale - Perl locale handling (internationalization and localization) |
5f05dabc | 6 | |
7 | =head1 DESCRIPTION | |
8 | ||
66cbab2c KW |
9 | In the beginning there was ASCII, the "American Standard Code for |
10 | Information Interchange", which works quite well for Americans with | |
11 | their English alphabet and dollar-denominated currency. But it doesn't | |
12 | work so well even for other English speakers, who may use different | |
13 | currencies, such as the pound sterling (as the symbol for that currency | |
14 | is not in ASCII); and it's hopelessly inadequate for many of the | |
15 | thousands of the world's other languages. | |
16 | ||
17 | To address these deficiencies, the concept of locales was invented | |
18 | (formally the ISO C, XPG4, POSIX 1.c "locale system"). And applications | |
19 | were and are being written that use the locale mechanism. The process of | |
20 | making such an application take account of its users' preferences in | |
21 | these kinds of matters is called B<internationalization> (often | |
22 | abbreviated as B<i18n>); telling such an application about a particular | |
23 | set of preferences is known as B<localization> (B<l10n>). | |
24 | ||
7ee2ae1e | 25 | Perl was extended, starting in 5.004, to support the locale system. This |
66cbab2c KW |
26 | is controlled per application by using one pragma, one function call, |
27 | and several environment variables. | |
28 | ||
29 | Unfortunately, there are quite a few deficiencies with the design (and | |
30 | often, the implementations) of locales, and their use for character sets | |
31 | has mostly been supplanted by Unicode (see L<perlunitut> for an | |
32 | introduction to that, and keep on reading here for how Unicode interacts | |
33 | with locales in Perl). | |
34 | ||
7ee2ae1e | 35 | Perl continues to support the old locale system, and starting in v5.16, |
66cbab2c KW |
36 | provides a hybrid way to use the Unicode character set, along with the |
37 | other portions of locales that may not be so problematic. | |
38 | (Unicode is also creating C<CLDR>, the "Common Locale Data Repository", | |
39 | L<http://cldr.unicode.org/> which includes more types of information than | |
40 | are available in the POSIX locale system. At the time of this writing, | |
41 | there was no CPAN module that provides access to this XML-encoded data. | |
42 | However, many of its locales have the POSIX-only data extracted, and are | |
43 | available at L<http://unicode.org/Public/cldr/latest/>.) | |
44 | ||
45 | =head1 WHAT IS A LOCALE | |
46 | ||
47 | A locale is a set of data that describes various aspects of how various | |
48 | communities in the world categorize their world. These categories are | |
49 | broken down into the following types (some of which include a brief | |
50 | note here): | |
51 | ||
52 | =over | |
53 | ||
54 | =item Category LC_NUMERIC: Numeric formatting | |
55 | ||
56 | This indicates how numbers should be formatted for human readability, | |
57 | for example the character used as the decimal point. | |
58 | ||
59 | =item Category LC_MONETARY: Formatting of monetary amounts | |
60 | ||
61 | =for comment | |
62 | The nbsp below makes this look better | |
63 | ||
64 | E<160> | |
65 | ||
66 | =item Category LC_TIME: Date/Time formatting | |
67 | ||
68 | =for comment | |
69 | The nbsp below makes this look better | |
70 | ||
71 | E<160> | |
72 | ||
73 | =item Category LC_MESSAGES: Error and other messages | |
74 | ||
75 | This for the most part is beyond the scope of Perl | |
76 | ||
77 | =item Category LC_COLLATE: Collation | |
78 | ||
79 | This indicates the ordering of letters for comparision and sorting. | |
80 | In Latin alphabets, for example, "b", generally follows "a". | |
81 | ||
82 | =item Category LC_CTYPE: Character Types | |
83 | ||
84 | This indicates, for example if a character is an uppercase letter. | |
85 | ||
86 | =back | |
87 | ||
88 | More details on the categories are given below in L</LOCALE CATEGORIES>. | |
89 | ||
90 | Together, these categories go a long way towards being able to customize | |
91 | a single program to run in many different locations. But there are | |
92 | deficiencies, so keep reading. | |
5f05dabc | 93 | |
94 | =head1 PREPARING TO USE LOCALES | |
95 | ||
66cbab2c KW |
96 | Perl will not use locales unless specifically requested to (see L</NOTES> below |
97 | for the partial exception of C<write()>). But even if there is such a | |
98 | request, B<all> of the following must be true for it to work properly: | |
5f05dabc | 99 | |
100 | =over 4 | |
101 | ||
102 | =item * | |
103 | ||
104 | B<Your operating system must support the locale system>. If it does, | |
14280422 | 105 | you should find that the setlocale() function is a documented part of |
5f05dabc | 106 | its C library. |
107 | ||
108 | =item * | |
109 | ||
5a964f20 | 110 | B<Definitions for locales that you use must be installed>. You, or |
14280422 DD |
111 | your system administrator, must make sure that this is the case. The |
112 | available locales, the location in which they are kept, and the manner | |
5a964f20 TC |
113 | in which they are installed all vary from system to system. Some systems |
114 | provide only a few, hard-wired locales and do not allow more to be | |
115 | added. Others allow you to add "canned" locales provided by the system | |
116 | supplier. Still others allow you or the system administrator to define | |
14280422 | 117 | and add arbitrary locales. (You may have to ask your supplier to |
5a964f20 | 118 | provide canned locales that are not delivered with your operating |
14280422 | 119 | system.) Read your system documentation for further illumination. |
5f05dabc | 120 | |
121 | =item * | |
122 | ||
123 | B<Perl must believe that the locale system is supported>. If it does, | |
124 | C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is | |
125 | C<define>. | |
126 | ||
127 | =back | |
128 | ||
129 | If you want a Perl application to process and present your data | |
130 | according to a particular locale, the application code should include | |
2ae324a7 | 131 | the S<C<use locale>> pragma (see L<The use locale pragma>) where |
5f05dabc | 132 | appropriate, and B<at least one> of the following must be true: |
133 | ||
134 | =over 4 | |
135 | ||
c052850d | 136 | =item 1 |
5f05dabc | 137 | |
66cbab2c | 138 | B<The locale-determining environment variables (see L</"ENVIRONMENT">) |
5a964f20 | 139 | must be correctly set up> at the time the application is started, either |
ef3087ec | 140 | by yourself or by whomever set up your system account; or |
5f05dabc | 141 | |
c052850d | 142 | =item 2 |
5f05dabc | 143 | |
14280422 DD |
144 | B<The application must set its own locale> using the method described in |
145 | L<The setlocale function>. | |
5f05dabc | 146 | |
147 | =back | |
148 | ||
149 | =head1 USING LOCALES | |
150 | ||
151 | =head2 The use locale pragma | |
152 | ||
14280422 | 153 | By default, Perl ignores the current locale. The S<C<use locale>> |
66cbab2c | 154 | pragma tells Perl to use the current locale for some operations. |
7ee2ae1e | 155 | Starting in v5.16, there is an optional parameter to this pragma: |
66cbab2c KW |
156 | |
157 | use locale ':not_characters'; | |
158 | ||
159 | This parameter allows better mixing of locales and Unicode, and is | |
160 | described fully in L</Unicode and UTF-8>, but briefly, it tells Perl to | |
161 | not use the character portions of the locale definition, that is | |
162 | the C<LC_CTYPE> and C<LC_COLLATE> categories. Instead it will use the | |
163 | native (extended by Unicode) character set. When using this parameter, | |
164 | you are responsible for getting the external character set translated | |
165 | into the native/Unicode one (which it already will be if it is one of | |
166 | the increasingly popular UTF-8 locales). There are convenient ways of | |
167 | doing this, as described in L</Unicode and UTF-8>. | |
c052850d KW |
168 | |
169 | The current locale is set at execution time by | |
170 | L<setlocale()|/The setlocale function> described below. If that function | |
171 | hasn't yet been called in the course of the program's execution, the | |
66cbab2c | 172 | current locale is that which was determined by the L</"ENVIRONMENT"> in |
c052850d KW |
173 | effect at the start of the program, except that |
174 | C<L<LC_NUMERIC|/Category LC_NUMERIC: Numeric Formatting>> is always | |
175 | initialized to the C locale (mentioned under L<Finding locales>). | |
70709c68 KW |
176 | If there is no valid environment, the current locale is undefined. It |
177 | is likely, but not necessarily, the "C" locale. | |
c052850d KW |
178 | |
179 | The operations that are affected by locale are: | |
5f05dabc | 180 | |
181 | =over 4 | |
182 | ||
66cbab2c KW |
183 | =item B<Under C<use locale ':not_characters';>> |
184 | ||
185 | =over 4 | |
186 | ||
187 | =item * | |
188 | ||
189 | B<Format declarations> (format()) use C<LC_NUMERIC> | |
190 | ||
191 | =item * | |
192 | ||
193 | B<The POSIX date formatting function> (strftime()) uses C<LC_TIME>. | |
194 | ||
195 | =back | |
196 | ||
197 | =for comment | |
198 | The nbsp below makes this look better | |
199 | ||
200 | E<160> | |
201 | ||
202 | =item B<Under just plain C<use locale;>> | |
203 | ||
204 | The above operations are affected, as well as the following: | |
205 | ||
206 | =over 4 | |
207 | ||
5f05dabc | 208 | =item * |
209 | ||
14280422 DD |
210 | B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>) and |
211 | the POSIX string collation functions strcoll() and strxfrm() use | |
5a964f20 TC |
212 | C<LC_COLLATE>. sort() is also affected if used without an |
213 | explicit comparison function, because it uses C<cmp> by default. | |
14280422 | 214 | |
5a964f20 | 215 | B<Note:> C<eq> and C<ne> are unaffected by locale: they always |
de108802 | 216 | perform a char-by-char comparison of their scalar operands. What's |
14280422 DD |
217 | more, if C<cmp> finds that its operands are equal according to the |
218 | collation sequence specified by the current locale, it goes on to | |
de108802 RGS |
219 | perform a char-by-char comparison, and only returns I<0> (equal) if the |
220 | operands are char-for-char identical. If you really want to know whether | |
5a964f20 | 221 | two strings--which C<eq> and C<cmp> may consider different--are equal |
14280422 DD |
222 | as far as collation in the locale is concerned, see the discussion in |
223 | L<Category LC_COLLATE: Collation>. | |
5f05dabc | 224 | |
225 | =item * | |
226 | ||
14280422 DD |
227 | B<Regular expressions and case-modification functions> (uc(), lc(), |
228 | ucfirst(), and lcfirst()) use C<LC_CTYPE> | |
5f05dabc | 229 | |
5f05dabc | 230 | =back |
231 | ||
66cbab2c | 232 | =back |
5f05dabc | 233 | |
5a964f20 | 234 | The default behavior is restored with the S<C<no locale>> pragma, or |
ef3087ec | 235 | upon reaching the end of the block enclosing C<use locale>. |
66cbab2c KW |
236 | Note that C<use locale> and C<use locale ':not_characters'> may be |
237 | nested, and that what is in effect within an inner scope will revert to | |
238 | the outer scope's rules at the end of the inner scope. | |
5f05dabc | 239 | |
5a964f20 | 240 | The string result of any operation that uses locale |
14280422 DD |
241 | information is tainted, as it is possible for a locale to be |
242 | untrustworthy. See L<"SECURITY">. | |
5f05dabc | 243 | |
244 | =head2 The setlocale function | |
245 | ||
14280422 DD |
246 | You can switch locales as often as you wish at run time with the |
247 | POSIX::setlocale() function: | |
5f05dabc | 248 | |
249 | # This functionality not usable prior to Perl 5.004 | |
250 | require 5.004; | |
251 | ||
252 | # Import locale-handling tool set from POSIX module. | |
253 | # This example uses: setlocale -- the function call | |
254 | # LC_CTYPE -- explained below | |
255 | use POSIX qw(locale_h); | |
256 | ||
14280422 | 257 | # query and save the old locale |
5f05dabc | 258 | $old_locale = setlocale(LC_CTYPE); |
259 | ||
260 | setlocale(LC_CTYPE, "fr_CA.ISO8859-1"); | |
261 | # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1" | |
262 | ||
263 | setlocale(LC_CTYPE, ""); | |
264 | # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG | |
265 | # environment variables. See below for documentation. | |
266 | ||
267 | # restore the old locale | |
268 | setlocale(LC_CTYPE, $old_locale); | |
269 | ||
14280422 DD |
270 | The first argument of setlocale() gives the B<category>, the second the |
271 | B<locale>. The category tells in what aspect of data processing you | |
272 | want to apply locale-specific rules. Category names are discussed in | |
66cbab2c | 273 | L</LOCALE CATEGORIES> and L</"ENVIRONMENT">. The locale is the name of a |
14280422 DD |
274 | collection of customization information corresponding to a particular |
275 | combination of language, country or territory, and codeset. Read on for | |
276 | hints on the naming of locales: not all systems name locales as in the | |
277 | example. | |
278 | ||
502a173a JH |
279 | If no second argument is provided and the category is something else |
280 | than LC_ALL, the function returns a string naming the current locale | |
281 | for the category. You can use this value as the second argument in a | |
282 | subsequent call to setlocale(). | |
283 | ||
284 | If no second argument is provided and the category is LC_ALL, the | |
285 | result is implementation-dependent. It may be a string of | |
c052850d | 286 | concatenated locale names (separator also implementation-dependent) |
f979aebc | 287 | or a single locale name. Please consult your setlocale(3) man page for |
502a173a JH |
288 | details. |
289 | ||
290 | If a second argument is given and it corresponds to a valid locale, | |
291 | the locale for the category is set to that value, and the function | |
292 | returns the now-current locale value. You can then use this in yet | |
293 | another call to setlocale(). (In some implementations, the return | |
294 | value may sometimes differ from the value you gave as the second | |
295 | argument--think of it as an alias for the value you gave.) | |
5f05dabc | 296 | |
297 | As the example shows, if the second argument is an empty string, the | |
298 | category's locale is returned to the default specified by the | |
299 | corresponding environment variables. Generally, this results in a | |
5a964f20 | 300 | return to the default that was in force when Perl started up: changes |
54310121 | 301 | to the environment made by the application after startup may or may not |
5a964f20 | 302 | be noticed, depending on your system's C library. |
5f05dabc | 303 | |
14280422 DD |
304 | If the second argument does not correspond to a valid locale, the locale |
305 | for the category is not changed, and the function returns I<undef>. | |
5f05dabc | 306 | |
66cbab2c KW |
307 | Note that Perl ignores the current C<LC_CTYPE> and C<LC_COLLATE> locales |
308 | within the scope of a C<use locale ':not_characters'>. | |
309 | ||
f979aebc | 310 | For further information about the categories, consult setlocale(3). |
3e6e419a JH |
311 | |
312 | =head2 Finding locales | |
313 | ||
f979aebc | 314 | For locales available in your system, consult also setlocale(3) to |
5a964f20 TC |
315 | see whether it leads to the list of available locales (search for the |
316 | I<SEE ALSO> section). If that fails, try the following command lines: | |
5f05dabc | 317 | |
318 | locale -a | |
319 | ||
320 | nlsinfo | |
321 | ||
322 | ls /usr/lib/nls/loc | |
323 | ||
324 | ls /usr/lib/locale | |
325 | ||
326 | ls /usr/lib/nls | |
327 | ||
b478f28d JH |
328 | ls /usr/share/locale |
329 | ||
5f05dabc | 330 | and see whether they list something resembling these |
331 | ||
2bdf8add | 332 | en_US.ISO8859-1 de_DE.ISO8859-1 ru_RU.ISO8859-5 |
502a173a | 333 | en_US.iso88591 de_DE.iso88591 ru_RU.iso88595 |
2bdf8add | 334 | en_US de_DE ru_RU |
14280422 | 335 | en de ru |
2bdf8add JH |
336 | english german russian |
337 | english.iso88591 german.iso88591 russian.iso88595 | |
502a173a | 338 | english.roman8 russian.koi8r |
5f05dabc | 339 | |
528d65ad JH |
340 | Sadly, even though the calling interface for setlocale() has been |
341 | standardized, names of locales and the directories where the | |
5a964f20 | 342 | configuration resides have not been. The basic form of the name is |
528d65ad JH |
343 | I<language_territory>B<.>I<codeset>, but the latter parts after |
344 | I<language> are not always present. The I<language> and I<country> | |
345 | are usually from the standards B<ISO 3166> and B<ISO 639>, the | |
346 | two-letter abbreviations for the countries and the languages of the | |
347 | world, respectively. The I<codeset> part often mentions some B<ISO | |
348 | 8859> character set, the Latin codesets. For example, C<ISO 8859-1> | |
349 | is the so-called "Western European codeset" that can be used to encode | |
350 | most Western European languages adequately. Again, there are several | |
351 | ways to write even the name of that one standard. Lamentably. | |
5f05dabc | 352 | |
14280422 DD |
353 | Two special locales are worth particular mention: "C" and "POSIX". |
354 | Currently these are effectively the same locale: the difference is | |
5a964f20 TC |
355 | mainly that the first one is defined by the C standard, the second by |
356 | the POSIX standard. They define the B<default locale> in which | |
14280422 | 357 | every program starts in the absence of locale information in its |
5a964f20 | 358 | environment. (The I<default> default locale, if you will.) Its language |
14280422 | 359 | is (American) English and its character codeset ASCII. |
c052850d KW |
360 | B<Warning>. The C locale delivered by some vendors may not |
361 | actually exactly match what the C standard calls for. So beware. | |
5f05dabc | 362 | |
14280422 DD |
363 | B<NOTE>: Not all systems have the "POSIX" locale (not all systems are |
364 | POSIX-conformant), so use "C" when you need explicitly to specify this | |
365 | default locale. | |
5f05dabc | 366 | |
3e6e419a JH |
367 | =head2 LOCALE PROBLEMS |
368 | ||
5a964f20 | 369 | You may encounter the following warning message at Perl startup: |
3e6e419a JH |
370 | |
371 | perl: warning: Setting locale failed. | |
372 | perl: warning: Please check that your locale settings: | |
373 | LC_ALL = "En_US", | |
374 | LANG = (unset) | |
375 | are supported and installed on your system. | |
376 | perl: warning: Falling back to the standard locale ("C"). | |
377 | ||
5a964f20 TC |
378 | This means that your locale settings had LC_ALL set to "En_US" and |
379 | LANG exists but has no value. Perl tried to believe you but could not. | |
380 | Instead, Perl gave up and fell back to the "C" locale, the default locale | |
381 | that is supposed to work no matter what. This usually means your locale | |
382 | settings were wrong, they mention locales your system has never heard | |
383 | of, or the locale installation in your system has problems (for example, | |
384 | some system files are broken or missing). There are quick and temporary | |
385 | fixes to these problems, as well as more thorough and lasting fixes. | |
3e6e419a JH |
386 | |
387 | =head2 Temporarily fixing locale problems | |
388 | ||
5a964f20 | 389 | The two quickest fixes are either to render Perl silent about any |
3e6e419a JH |
390 | locale inconsistencies or to run Perl under the default locale "C". |
391 | ||
392 | Perl's moaning about locale problems can be silenced by setting the | |
900bd440 JH |
393 | environment variable PERL_BADLANG to a zero value, for example "0". |
394 | This method really just sweeps the problem under the carpet: you tell | |
395 | Perl to shut up even when Perl sees that something is wrong. Do not | |
396 | be surprised if later something locale-dependent misbehaves. | |
3e6e419a JH |
397 | |
398 | Perl can be run under the "C" locale by setting the environment | |
5a964f20 TC |
399 | variable LC_ALL to "C". This method is perhaps a bit more civilized |
400 | than the PERL_BADLANG approach, but setting LC_ALL (or | |
401 | other locale variables) may affect other programs as well, not just | |
402 | Perl. In particular, external programs run from within Perl will see | |
3e6e419a | 403 | these changes. If you make the new settings permanent (read on), all |
f979aebc | 404 | programs you run see the changes. See L<"ENVIRONMENT"> for |
5a964f20 | 405 | the full list of relevant environment variables and L<USING LOCALES> |
e05ffc7d | 406 | for their effects in Perl. Effects in other programs are |
5a964f20 | 407 | easily deducible. For example, the variable LC_COLLATE may well affect |
b432a672 | 408 | your B<sort> program (or whatever the program that arranges "records" |
3e6e419a JH |
409 | alphabetically in your system is called). |
410 | ||
5a964f20 TC |
411 | You can test out changing these variables temporarily, and if the |
412 | new settings seem to help, put those settings into your shell startup | |
413 | files. Consult your local documentation for the exact details. For in | |
414 | Bourne-like shells (B<sh>, B<ksh>, B<bash>, B<zsh>): | |
3e6e419a JH |
415 | |
416 | LC_ALL=en_US.ISO8859-1 | |
417 | export LC_ALL | |
418 | ||
5a964f20 TC |
419 | This assumes that we saw the locale "en_US.ISO8859-1" using the commands |
420 | discussed above. We decided to try that instead of the above faulty | |
421 | locale "En_US"--and in Cshish shells (B<csh>, B<tcsh>) | |
3e6e419a JH |
422 | |
423 | setenv LC_ALL en_US.ISO8859-1 | |
c47ff5f1 | 424 | |
c406981e JH |
425 | or if you have the "env" application you can do in any shell |
426 | ||
427 | env LC_ALL=en_US.ISO8859-1 perl ... | |
428 | ||
5a964f20 | 429 | If you do not know what shell you have, consult your local |
3e6e419a JH |
430 | helpdesk or the equivalent. |
431 | ||
432 | =head2 Permanently fixing locale problems | |
433 | ||
5a964f20 TC |
434 | The slower but superior fixes are when you may be able to yourself |
435 | fix the misconfiguration of your own environment variables. The | |
3e6e419a JH |
436 | mis(sing)configuration of the whole system's locales usually requires |
437 | the help of your friendly system administrator. | |
438 | ||
5a964f20 TC |
439 | First, see earlier in this document about L<Finding locales>. That tells |
440 | how to find which locales are really supported--and more importantly, | |
441 | installed--on your system. In our example error message, environment | |
442 | variables affecting the locale are listed in the order of decreasing | |
443 | importance (and unset variables do not matter). Therefore, having | |
444 | LC_ALL set to "En_US" must have been the bad choice, as shown by the | |
445 | error message. First try fixing locale settings listed first. | |
3e6e419a | 446 | |
5a964f20 TC |
447 | Second, if using the listed commands you see something B<exactly> |
448 | (prefix matches do not count and case usually counts) like "En_US" | |
449 | without the quotes, then you should be okay because you are using a | |
450 | locale name that should be installed and available in your system. | |
4a4eefd0 | 451 | In this case, see L<Permanently fixing your system's locale configuration>. |
3e6e419a | 452 | |
4a4eefd0 | 453 | =head2 Permanently fixing your system's locale configuration |
3e6e419a | 454 | |
5a964f20 | 455 | This is when you see something like: |
3e6e419a JH |
456 | |
457 | perl: warning: Please check that your locale settings: | |
458 | LC_ALL = "En_US", | |
459 | LANG = (unset) | |
460 | are supported and installed on your system. | |
461 | ||
462 | but then cannot see that "En_US" listed by the above-mentioned | |
5a964f20 TC |
463 | commands. You may see things like "en_US.ISO8859-1", but that isn't |
464 | the same. In this case, try running under a locale | |
465 | that you can list and which somehow matches what you tried. The | |
3e6e419a | 466 | rules for matching locale names are a bit vague because |
e05ffc7d | 467 | standardization is weak in this area. See again the |
13a2d996 | 468 | L<Finding locales> about general rules. |
3e6e419a | 469 | |
b687b08b | 470 | =head2 Fixing system locale configuration |
3e6e419a | 471 | |
5a964f20 TC |
472 | Contact a system administrator (preferably your own) and report the exact |
473 | error message you get, and ask them to read this same documentation you | |
474 | are now reading. They should be able to check whether there is something | |
475 | wrong with the locale configuration of the system. The L<Finding locales> | |
476 | section is unfortunately a bit vague about the exact commands and places | |
477 | because these things are not that standardized. | |
3e6e419a | 478 | |
5f05dabc | 479 | =head2 The localeconv function |
480 | ||
14280422 DD |
481 | The POSIX::localeconv() function allows you to get particulars of the |
482 | locale-dependent numeric formatting information specified by the current | |
483 | C<LC_NUMERIC> and C<LC_MONETARY> locales. (If you just want the name of | |
484 | the current locale for a particular category, use POSIX::setlocale() | |
5a964f20 | 485 | with a single parameter--see L<The setlocale function>.) |
5f05dabc | 486 | |
487 | use POSIX qw(locale_h); | |
5f05dabc | 488 | |
489 | # Get a reference to a hash of locale-dependent info | |
490 | $locale_values = localeconv(); | |
491 | ||
492 | # Output sorted list of the values | |
493 | for (sort keys %$locale_values) { | |
14280422 | 494 | printf "%-20s = %s\n", $_, $locale_values->{$_} |
5f05dabc | 495 | } |
496 | ||
14280422 | 497 | localeconv() takes no arguments, and returns B<a reference to> a hash. |
5a964f20 | 498 | The keys of this hash are variable names for formatting, such as |
502a173a | 499 | C<decimal_point> and C<thousands_sep>. The values are the |
cea6626f | 500 | corresponding, er, values. See L<POSIX/localeconv> for a longer |
502a173a JH |
501 | example listing the categories an implementation might be expected to |
502 | provide; some provide more and others fewer. You don't need an | |
503 | explicit C<use locale>, because localeconv() always observes the | |
504 | current locale. | |
5f05dabc | 505 | |
5a964f20 TC |
506 | Here's a simple-minded example program that rewrites its command-line |
507 | parameters as integers correctly formatted in the current locale: | |
5f05dabc | 508 | |
ef3087ec KW |
509 | # See comments in previous example |
510 | require 5.004; | |
511 | use POSIX qw(locale_h); | |
512 | ||
513 | # Get some of locale's numeric formatting parameters | |
514 | my ($thousands_sep, $grouping) = | |
515 | @{localeconv()}{'thousands_sep', 'grouping'}; | |
516 | ||
517 | # Apply defaults if values are missing | |
518 | $thousands_sep = ',' unless $thousands_sep; | |
519 | ||
520 | # grouping and mon_grouping are packed lists | |
521 | # of small integers (characters) telling the | |
522 | # grouping (thousand_seps and mon_thousand_seps | |
523 | # being the group dividers) of numbers and | |
524 | # monetary quantities. The integers' meanings: | |
525 | # 255 means no more grouping, 0 means repeat | |
526 | # the previous grouping, 1-254 means use that | |
527 | # as the current grouping. Grouping goes from | |
528 | # right to left (low to high digits). In the | |
529 | # below we cheat slightly by never using anything | |
530 | # else than the first grouping (whatever that is). | |
531 | if ($grouping) { | |
532 | @grouping = unpack("C*", $grouping); | |
533 | } else { | |
534 | @grouping = (3); | |
535 | } | |
536 | ||
537 | # Format command line params for current locale | |
538 | for (@ARGV) { | |
539 | $_ = int; # Chop non-integer part | |
540 | 1 while | |
541 | s/(\d)(\d{$grouping[0]}($|$thousands_sep))/$1$thousands_sep$2/; | |
542 | print "$_"; | |
543 | } | |
544 | print "\n"; | |
5f05dabc | 545 | |
74c76037 | 546 | =head2 I18N::Langinfo |
4bbcc6e8 JH |
547 | |
548 | Another interface for querying locale-dependent information is the | |
e1020413 | 549 | I18N::Langinfo::langinfo() function, available at least in Unix-like |
4bbcc6e8 JH |
550 | systems and VMS. |
551 | ||
74c76037 JH |
552 | The following example will import the langinfo() function itself and |
553 | three constants to be used as arguments to langinfo(): a constant for | |
554 | the abbreviated first day of the week (the numbering starts from | |
555 | Sunday = 1) and two more constants for the affirmative and negative | |
556 | answers for a yes/no question in the current locale. | |
4bbcc6e8 | 557 | |
74c76037 | 558 | use I18N::Langinfo qw(langinfo ABDAY_1 YESSTR NOSTR); |
4bbcc6e8 | 559 | |
ef3087ec KW |
560 | my ($abday_1, $yesstr, $nostr) |
561 | = map { langinfo } qw(ABDAY_1 YESSTR NOSTR); | |
4bbcc6e8 | 562 | |
74c76037 | 563 | print "$abday_1? [$yesstr/$nostr] "; |
4bbcc6e8 | 564 | |
74c76037 JH |
565 | In other words, in the "C" (or English) locale the above will probably |
566 | print something like: | |
567 | ||
e05ffc7d | 568 | Sun? [yes/no] |
4bbcc6e8 JH |
569 | |
570 | See L<I18N::Langinfo> for more information. | |
571 | ||
5f05dabc | 572 | =head1 LOCALE CATEGORIES |
573 | ||
5a964f20 TC |
574 | The following subsections describe basic locale categories. Beyond these, |
575 | some combination categories allow manipulation of more than one | |
576 | basic category at a time. See L<"ENVIRONMENT"> for a discussion of these. | |
5f05dabc | 577 | |
578 | =head2 Category LC_COLLATE: Collation | |
579 | ||
66cbab2c KW |
580 | In the scope of S<C<use locale>> (but not a |
581 | C<use locale ':not_characters'>), Perl looks to the C<LC_COLLATE> | |
5a964f20 | 582 | environment variable to determine the application's notions on collation |
b4ffc3db TC |
583 | (ordering) of characters. For example, "b" follows "a" in Latin |
584 | alphabets, but where do "E<aacute>" and "E<aring>" belong? And while | |
585 | "color" follows "chocolate" in English, what about in Spanish? | |
5f05dabc | 586 | |
60f0fa02 JH |
587 | The following collations all make sense and you may meet any of them |
588 | if you "use locale". | |
589 | ||
590 | A B C D E a b c d e | |
35316ca3 | 591 | A a B b C c D d E e |
60f0fa02 JH |
592 | a A b B c C d D e E |
593 | a b c d e A B C D E | |
594 | ||
f1cbbd6e | 595 | Here is a code snippet to tell what "word" |
5a964f20 | 596 | characters are in the current locale, in that locale's order: |
5f05dabc | 597 | |
598 | use locale; | |
35316ca3 | 599 | print +(sort grep /\w/, map { chr } 0..255), "\n"; |
5f05dabc | 600 | |
14280422 DD |
601 | Compare this with the characters that you see and their order if you |
602 | state explicitly that the locale should be ignored: | |
5f05dabc | 603 | |
604 | no locale; | |
35316ca3 | 605 | print +(sort grep /\w/, map { chr } 0..255), "\n"; |
5f05dabc | 606 | |
607 | This machine-native collation (which is what you get unless S<C<use | |
608 | locale>> has appeared earlier in the same block) must be used for | |
609 | sorting raw binary data, whereas the locale-dependent collation of the | |
b0c42ed9 | 610 | first example is useful for natural text. |
5f05dabc | 611 | |
14280422 DD |
612 | As noted in L<USING LOCALES>, C<cmp> compares according to the current |
613 | collation locale when C<use locale> is in effect, but falls back to a | |
de108802 | 614 | char-by-char comparison for strings that the locale says are equal. You |
14280422 DD |
615 | can use POSIX::strcoll() if you don't want this fall-back: |
616 | ||
617 | use POSIX qw(strcoll); | |
618 | $equal_in_locale = | |
619 | !strcoll("space and case ignored", "SpaceAndCaseIgnored"); | |
620 | ||
621 | $equal_in_locale will be true if the collation locale specifies a | |
5a964f20 | 622 | dictionary-like ordering that ignores space characters completely and |
9e3a2af8 | 623 | which folds case. |
14280422 | 624 | |
5a964f20 | 625 | If you have a single string that you want to check for "equality in |
14280422 DD |
626 | locale" against several others, you might think you could gain a little |
627 | efficiency by using POSIX::strxfrm() in conjunction with C<eq>: | |
628 | ||
629 | use POSIX qw(strxfrm); | |
630 | $xfrm_string = strxfrm("Mixed-case string"); | |
631 | print "locale collation ignores spaces\n" | |
632 | if $xfrm_string eq strxfrm("Mixed-casestring"); | |
633 | print "locale collation ignores hyphens\n" | |
634 | if $xfrm_string eq strxfrm("Mixedcase string"); | |
635 | print "locale collation ignores case\n" | |
636 | if $xfrm_string eq strxfrm("mixed-case string"); | |
637 | ||
638 | strxfrm() takes a string and maps it into a transformed string for use | |
de108802 | 639 | in char-by-char comparisons against other transformed strings during |
14280422 | 640 | collation. "Under the hood", locale-affected Perl comparison operators |
de108802 | 641 | call strxfrm() for both operands, then do a char-by-char |
5a964f20 | 642 | comparison of the transformed strings. By calling strxfrm() explicitly |
14280422 | 643 | and using a non locale-affected comparison, the example attempts to save |
5a964f20 | 644 | a couple of transformations. But in fact, it doesn't save anything: Perl |
2ae324a7 | 645 | magic (see L<perlguts/Magic Variables>) creates the transformed version of a |
5a964f20 | 646 | string the first time it's needed in a comparison, then keeps this version around |
14280422 | 647 | in case it's needed again. An example rewritten the easy way with |
e38874e2 | 648 | C<cmp> runs just about as fast. It also copes with null characters |
14280422 | 649 | embedded in strings; if you call strxfrm() directly, it treats the first |
5a964f20 TC |
650 | null it finds as a terminator. don't expect the transformed strings |
651 | it produces to be portable across systems--or even from one revision | |
e38874e2 DD |
652 | of your operating system to the next. In short, don't call strxfrm() |
653 | directly: let Perl do it for you. | |
14280422 | 654 | |
5a964f20 | 655 | Note: C<use locale> isn't shown in some of these examples because it isn't |
14280422 DD |
656 | needed: strcoll() and strxfrm() exist only to generate locale-dependent |
657 | results, and so always obey the current C<LC_COLLATE> locale. | |
5f05dabc | 658 | |
659 | =head2 Category LC_CTYPE: Character Types | |
660 | ||
66cbab2c KW |
661 | In the scope of S<C<use locale>> (but not a |
662 | C<use locale ':not_characters'>), Perl obeys the C<LC_CTYPE> locale | |
14280422 DD |
663 | setting. This controls the application's notion of which characters are |
664 | alphabetic. This affects Perl's C<\w> regular expression metanotation, | |
f1cbbd6e GS |
665 | which stands for alphanumeric characters--that is, alphabetic, |
666 | numeric, and including other special characters such as the underscore or | |
667 | hyphen. (Consult L<perlre> for more information about | |
14280422 | 668 | regular expressions.) Thanks to C<LC_CTYPE>, depending on your locale |
b4ffc3db TC |
669 | setting, characters like "E<aelig>", "E<eth>", "E<szlig>", and |
670 | "E<oslash>" may be understood as C<\w> characters. | |
5f05dabc | 671 | |
2c268ad5 | 672 | The C<LC_CTYPE> locale also provides the map used in transliterating |
68dc0745 | 673 | characters between lower and uppercase. This affects the case-mapping |
5a964f20 TC |
674 | functions--lc(), lcfirst, uc(), and ucfirst(); case-mapping |
675 | interpolation with C<\l>, C<\L>, C<\u>, or C<\U> in double-quoted strings | |
676 | and C<s///> substitutions; and case-independent regular expression | |
e38874e2 DD |
677 | pattern matching using the C<i> modifier. |
678 | ||
5a964f20 TC |
679 | Finally, C<LC_CTYPE> affects the POSIX character-class test |
680 | functions--isalpha(), islower(), and so on. For example, if you move | |
681 | from the "C" locale to a 7-bit Scandinavian one, you may find--possibly | |
682 | to your surprise--that "|" moves from the ispunct() class to isalpha(). | |
ef3087ec KW |
683 | Unfortunately, this creates big problems for regular expressions. "|" still |
684 | means alternation even though it matches C<\w>. | |
5f05dabc | 685 | |
14280422 DD |
686 | B<Note:> A broken or malicious C<LC_CTYPE> locale definition may result |
687 | in clearly ineligible characters being considered to be alphanumeric by | |
e199995e | 688 | your application. For strict matching of (mundane) ASCII letters and |
5a964f20 | 689 | digits--for example, in command strings--locale-aware applications |
e199995e | 690 | should use C<\w> with the C</a> regular expression modifier. See L<"SECURITY">. |
5f05dabc | 691 | |
692 | =head2 Category LC_NUMERIC: Numeric Formatting | |
693 | ||
2095dafa RGS |
694 | After a proper POSIX::setlocale() call, Perl obeys the C<LC_NUMERIC> |
695 | locale information, which controls an application's idea of how numbers | |
696 | should be formatted for human readability by the printf(), sprintf(), and | |
697 | write() functions. String-to-numeric conversion by the POSIX::strtod() | |
5a964f20 | 698 | function is also affected. In most implementations the only effect is to |
b4ffc3db | 699 | change the character used for the decimal point--perhaps from "." to ",". |
5a964f20 | 700 | These functions aren't aware of such niceties as thousands separation and |
2095dafa | 701 | so on. (See L<The localeconv function> if you care about these things.) |
5a964f20 | 702 | |
3cf03d68 | 703 | Output produced by print() is also affected by the current locale: it |
3cf03d68 JH |
704 | corresponds to what you'd get from printf() in the "C" locale. The |
705 | same is true for Perl's internal conversions between numeric and | |
706 | string formats: | |
5f05dabc | 707 | |
2095dafa RGS |
708 | use POSIX qw(strtod setlocale LC_NUMERIC); |
709 | ||
710 | setlocale LC_NUMERIC, ""; | |
14280422 | 711 | |
5f05dabc | 712 | $n = 5/2; # Assign numeric 2.5 to $n |
713 | ||
35316ca3 | 714 | $a = " $n"; # Locale-dependent conversion to string |
5f05dabc | 715 | |
35316ca3 | 716 | print "half five is $n\n"; # Locale-dependent output |
5f05dabc | 717 | |
718 | printf "half five is %g\n", $n; # Locale-dependent output | |
719 | ||
14280422 DD |
720 | print "DECIMAL POINT IS COMMA\n" |
721 | if $n == (strtod("2,5"))[0]; # Locale-dependent conversion | |
5f05dabc | 722 | |
4bbcc6e8 JH |
723 | See also L<I18N::Langinfo> and C<RADIXCHAR>. |
724 | ||
5f05dabc | 725 | =head2 Category LC_MONETARY: Formatting of monetary amounts |
726 | ||
e199995e | 727 | The C standard defines the C<LC_MONETARY> category, but not a function |
5a964f20 | 728 | that is affected by its contents. (Those with experience of standards |
b0c42ed9 | 729 | committees will recognize that the working group decided to punt on the |
14280422 | 730 | issue.) Consequently, Perl takes no notice of it. If you really want |
e05ffc7d KW |
731 | to use C<LC_MONETARY>, you can query its contents--see |
732 | L<The localeconv function>--and use the information that it returns in your | |
733 | application's own formatting of currency amounts. However, you may well | |
734 | find that the information, voluminous and complex though it may be, still | |
735 | does not quite meet your requirements: currency formatting is a hard nut | |
13a2d996 | 736 | to crack. |
5f05dabc | 737 | |
4bbcc6e8 JH |
738 | See also L<I18N::Langinfo> and C<CRNCYSTR>. |
739 | ||
5f05dabc | 740 | =head2 LC_TIME |
741 | ||
5a964f20 | 742 | Output produced by POSIX::strftime(), which builds a formatted |
5f05dabc | 743 | human-readable date/time string, is affected by the current C<LC_TIME> |
744 | locale. Thus, in a French locale, the output produced by the C<%B> | |
745 | format element (full month name) for the first month of the year would | |
5a964f20 | 746 | be "janvier". Here's how to get a list of long month names in the |
5f05dabc | 747 | current locale: |
748 | ||
749 | use POSIX qw(strftime); | |
14280422 DD |
750 | for (0..11) { |
751 | $long_month_name[$_] = | |
752 | strftime("%B", 0, 0, 0, 1, $_, 96); | |
5f05dabc | 753 | } |
754 | ||
5a964f20 | 755 | Note: C<use locale> isn't needed in this example: as a function that |
14280422 DD |
756 | exists only to generate locale-dependent results, strftime() always |
757 | obeys the current C<LC_TIME> locale. | |
5f05dabc | 758 | |
4bbcc6e8 | 759 | See also L<I18N::Langinfo> and C<ABDAY_1>..C<ABDAY_7>, C<DAY_1>..C<DAY_7>, |
2a2bf5f4 | 760 | C<ABMON_1>..C<ABMON_12>, and C<ABMON_1>..C<ABMON_12>. |
4bbcc6e8 | 761 | |
5f05dabc | 762 | =head2 Other categories |
763 | ||
5a964f20 TC |
764 | The remaining locale category, C<LC_MESSAGES> (possibly supplemented |
765 | by others in particular implementations) is not currently used by | |
98a6f11e | 766 | Perl--except possibly to affect the behavior of library functions |
767 | called by extensions outside the standard Perl distribution and by the | |
768 | operating system and its utilities. Note especially that the string | |
769 | value of C<$!> and the error messages given by external utilities may | |
770 | be changed by C<LC_MESSAGES>. If you want to have portable error | |
265f5c4a | 771 | codes, use C<%!>. See L<Errno>. |
14280422 DD |
772 | |
773 | =head1 SECURITY | |
774 | ||
5a964f20 | 775 | Although the main discussion of Perl security issues can be found in |
14280422 DD |
776 | L<perlsec>, a discussion of Perl's locale handling would be incomplete |
777 | if it did not draw your attention to locale-dependent security issues. | |
5a964f20 TC |
778 | Locales--particularly on systems that allow unprivileged users to |
779 | build their own locales--are untrustworthy. A malicious (or just plain | |
14280422 DD |
780 | broken) locale can make a locale-aware application give unexpected |
781 | results. Here are a few possibilities: | |
782 | ||
783 | =over 4 | |
784 | ||
785 | =item * | |
786 | ||
787 | Regular expression checks for safe file names or mail addresses using | |
5a964f20 | 788 | C<\w> may be spoofed by an C<LC_CTYPE> locale that claims that |
14280422 DD |
789 | characters such as "E<gt>" and "|" are alphanumeric. |
790 | ||
791 | =item * | |
792 | ||
e38874e2 DD |
793 | String interpolation with case-mapping, as in, say, C<$dest = |
794 | "C:\U$name.$ext">, may produce dangerous results if a bogus LC_CTYPE | |
795 | case-mapping table is in effect. | |
796 | ||
797 | =item * | |
798 | ||
14280422 DD |
799 | A sneaky C<LC_COLLATE> locale could result in the names of students with |
800 | "D" grades appearing ahead of those with "A"s. | |
801 | ||
802 | =item * | |
803 | ||
5a964f20 | 804 | An application that takes the trouble to use information in |
14280422 | 805 | C<LC_MONETARY> may format debits as if they were credits and vice versa |
5a964f20 | 806 | if that locale has been subverted. Or it might make payments in US |
14280422 DD |
807 | dollars instead of Hong Kong dollars. |
808 | ||
809 | =item * | |
810 | ||
811 | The date and day names in dates formatted by strftime() could be | |
812 | manipulated to advantage by a malicious user able to subvert the | |
5a964f20 | 813 | C<LC_DATE> locale. ("Look--it says I wasn't in the building on |
14280422 DD |
814 | Sunday.") |
815 | ||
816 | =back | |
817 | ||
818 | Such dangers are not peculiar to the locale system: any aspect of an | |
5a964f20 | 819 | application's environment which may be modified maliciously presents |
14280422 | 820 | similar challenges. Similarly, they are not specific to Perl: any |
5a964f20 | 821 | programming language that allows you to write programs that take |
14280422 DD |
822 | account of their environment exposes you to these issues. |
823 | ||
5a964f20 TC |
824 | Perl cannot protect you from all possibilities shown in the |
825 | examples--there is no substitute for your own vigilance--but, when | |
14280422 | 826 | C<use locale> is in effect, Perl uses the tainting mechanism (see |
5a964f20 | 827 | L<perlsec>) to mark string results that become locale-dependent, and |
14280422 | 828 | which may be untrustworthy in consequence. Here is a summary of the |
5a964f20 | 829 | tainting behavior of operators and functions that may be affected by |
14280422 DD |
830 | the locale: |
831 | ||
832 | =over 4 | |
833 | ||
551e1d92 RB |
834 | =item * |
835 | ||
836 | B<Comparison operators> (C<lt>, C<le>, C<ge>, C<gt> and C<cmp>): | |
14280422 DD |
837 | |
838 | Scalar true/false (or less/equal/greater) result is never tainted. | |
839 | ||
551e1d92 RB |
840 | =item * |
841 | ||
842 | B<Case-mapping interpolation> (with C<\l>, C<\L>, C<\u> or C<\U>) | |
e38874e2 DD |
843 | |
844 | Result string containing interpolated material is tainted if | |
66cbab2c | 845 | C<use locale> (but not S<C<use locale ':not_characters'>>) is in effect. |
e38874e2 | 846 | |
551e1d92 RB |
847 | =item * |
848 | ||
849 | B<Matching operator> (C<m//>): | |
14280422 DD |
850 | |
851 | Scalar true/false result never tainted. | |
852 | ||
5a964f20 | 853 | Subpatterns, either delivered as a list-context result or as $1 etc. |
66cbab2c KW |
854 | are tainted if C<use locale> (but not S<C<use locale ':not_characters'>>) |
855 | is in effect, and the subpattern regular | |
e38874e2 | 856 | expression contains C<\w> (to match an alphanumeric character), C<\W> |
6b0ac556 OK |
857 | (non-alphanumeric character), C<\s> (whitespace character), or C<\S> |
858 | (non whitespace character). The matched-pattern variable, $&, $` | |
e38874e2 DD |
859 | (pre-match), $' (post-match), and $+ (last match) are also tainted if |
860 | C<use locale> is in effect and the regular expression contains C<\w>, | |
861 | C<\W>, C<\s>, or C<\S>. | |
14280422 | 862 | |
551e1d92 RB |
863 | =item * |
864 | ||
865 | B<Substitution operator> (C<s///>): | |
14280422 | 866 | |
e38874e2 | 867 | Has the same behavior as the match operator. Also, the left |
66cbab2c KW |
868 | operand of C<=~> becomes tainted when C<use locale> |
869 | (but not S<C<use locale ':not_characters'>>) is in effect if modified as | |
870 | a result of a substitution based on a regular | |
e38874e2 | 871 | expression match involving C<\w>, C<\W>, C<\s>, or C<\S>; or of |
7b8d334a | 872 | case-mapping with C<\l>, C<\L>,C<\u> or C<\U>. |
14280422 | 873 | |
551e1d92 RB |
874 | =item * |
875 | ||
876 | B<Output formatting functions> (printf() and write()): | |
14280422 | 877 | |
3cf03d68 JH |
878 | Results are never tainted because otherwise even output from print, |
879 | for example C<print(1/7)>, should be tainted if C<use locale> is in | |
880 | effect. | |
14280422 | 881 | |
551e1d92 RB |
882 | =item * |
883 | ||
884 | B<Case-mapping functions> (lc(), lcfirst(), uc(), ucfirst()): | |
14280422 | 885 | |
66cbab2c KW |
886 | Results are tainted if C<use locale> (but not |
887 | S<C<use locale ':not_characters'>>) is in effect. | |
14280422 | 888 | |
551e1d92 RB |
889 | =item * |
890 | ||
891 | B<POSIX locale-dependent functions> (localeconv(), strcoll(), | |
14280422 DD |
892 | strftime(), strxfrm()): |
893 | ||
894 | Results are never tainted. | |
895 | ||
551e1d92 RB |
896 | =item * |
897 | ||
898 | B<POSIX character class tests> (isalnum(), isalpha(), isdigit(), | |
14280422 DD |
899 | isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(), |
900 | isxdigit()): | |
901 | ||
902 | True/false results are never tainted. | |
903 | ||
904 | =back | |
905 | ||
906 | Three examples illustrate locale-dependent tainting. | |
907 | The first program, which ignores its locale, won't run: a value taken | |
54310121 | 908 | directly from the command line may not be used to name an output file |
14280422 DD |
909 | when taint checks are enabled. |
910 | ||
911 | #/usr/local/bin/perl -T | |
912 | # Run with taint checking | |
913 | ||
54310121 | 914 | # Command line sanity check omitted... |
14280422 DD |
915 | $tainted_output_file = shift; |
916 | ||
917 | open(F, ">$tainted_output_file") | |
918 | or warn "Open of $untainted_output_file failed: $!\n"; | |
919 | ||
920 | The program can be made to run by "laundering" the tainted value through | |
5a964f20 TC |
921 | a regular expression: the second example--which still ignores locale |
922 | information--runs, creating the file named on its command line | |
14280422 DD |
923 | if it can. |
924 | ||
925 | #/usr/local/bin/perl -T | |
926 | ||
927 | $tainted_output_file = shift; | |
928 | $tainted_output_file =~ m%[\w/]+%; | |
929 | $untainted_output_file = $&; | |
930 | ||
931 | open(F, ">$untainted_output_file") | |
932 | or warn "Open of $untainted_output_file failed: $!\n"; | |
933 | ||
5a964f20 | 934 | Compare this with a similar but locale-aware program: |
14280422 DD |
935 | |
936 | #/usr/local/bin/perl -T | |
937 | ||
938 | $tainted_output_file = shift; | |
939 | use locale; | |
940 | $tainted_output_file =~ m%[\w/]+%; | |
941 | $localized_output_file = $&; | |
942 | ||
943 | open(F, ">$localized_output_file") | |
944 | or warn "Open of $localized_output_file failed: $!\n"; | |
945 | ||
946 | This third program fails to run because $& is tainted: it is the result | |
5a964f20 | 947 | of a match involving C<\w> while C<use locale> is in effect. |
5f05dabc | 948 | |
949 | =head1 ENVIRONMENT | |
950 | ||
951 | =over 12 | |
952 | ||
953 | =item PERL_BADLANG | |
954 | ||
14280422 | 955 | A string that can suppress Perl's warning about failed locale settings |
54310121 | 956 | at startup. Failure can occur if the locale support in the operating |
5a964f20 | 957 | system is lacking (broken) in some way--or if you mistyped the name of |
900bd440 JH |
958 | a locale when you set up your environment. If this environment |
959 | variable is absent, or has a value that does not evaluate to integer | |
960 | zero--that is, "0" or ""-- Perl will complain about locale setting | |
961 | failures. | |
5f05dabc | 962 | |
14280422 DD |
963 | B<NOTE>: PERL_BADLANG only gives you a way to hide the warning message. |
964 | The message tells about some problem in your system's locale support, | |
965 | and you should investigate what the problem is. | |
5f05dabc | 966 | |
967 | =back | |
968 | ||
969 | The following environment variables are not specific to Perl: They are | |
14280422 DD |
970 | part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale() method |
971 | for controlling an application's opinion on data. | |
5f05dabc | 972 | |
973 | =over 12 | |
974 | ||
975 | =item LC_ALL | |
976 | ||
5a964f20 | 977 | C<LC_ALL> is the "override-all" locale environment variable. If |
5f05dabc | 978 | set, it overrides all the rest of the locale environment variables. |
979 | ||
528d65ad JH |
980 | =item LANGUAGE |
981 | ||
982 | B<NOTE>: C<LANGUAGE> is a GNU extension, it affects you only if you | |
983 | are using the GNU libc. This is the case if you are using e.g. Linux. | |
e1020413 | 984 | If you are using "commercial" Unixes you are most probably I<not> |
22b6f60d JH |
985 | using GNU libc and you can ignore C<LANGUAGE>. |
986 | ||
987 | However, in the case you are using C<LANGUAGE>: it affects the | |
988 | language of informational, warning, and error messages output by | |
989 | commands (in other words, it's like C<LC_MESSAGES>) but it has higher | |
96090e4f | 990 | priority than C<LC_ALL>. Moreover, it's not a single value but |
22b6f60d JH |
991 | instead a "path" (":"-separated list) of I<languages> (not locales). |
992 | See the GNU C<gettext> library documentation for more information. | |
528d65ad | 993 | |
5f05dabc | 994 | =item LC_CTYPE |
995 | ||
996 | In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type | |
997 | locale. In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG> | |
998 | chooses the character type locale. | |
999 | ||
1000 | =item LC_COLLATE | |
1001 | ||
14280422 DD |
1002 | In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation |
1003 | (sorting) locale. In the absence of both C<LC_ALL> and C<LC_COLLATE>, | |
1004 | C<LANG> chooses the collation locale. | |
5f05dabc | 1005 | |
1006 | =item LC_MONETARY | |
1007 | ||
14280422 DD |
1008 | In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the monetary |
1009 | formatting locale. In the absence of both C<LC_ALL> and C<LC_MONETARY>, | |
1010 | C<LANG> chooses the monetary formatting locale. | |
5f05dabc | 1011 | |
1012 | =item LC_NUMERIC | |
1013 | ||
1014 | In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format | |
1015 | locale. In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG> | |
1016 | chooses the numeric format. | |
1017 | ||
1018 | =item LC_TIME | |
1019 | ||
14280422 DD |
1020 | In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time |
1021 | formatting locale. In the absence of both C<LC_ALL> and C<LC_TIME>, | |
1022 | C<LANG> chooses the date and time formatting locale. | |
5f05dabc | 1023 | |
1024 | =item LANG | |
1025 | ||
14280422 DD |
1026 | C<LANG> is the "catch-all" locale environment variable. If it is set, it |
1027 | is used as the last resort after the overall C<LC_ALL> and the | |
5f05dabc | 1028 | category-specific C<LC_...>. |
1029 | ||
1030 | =back | |
1031 | ||
7e4353e9 RGS |
1032 | =head2 Examples |
1033 | ||
1034 | The LC_NUMERIC controls the numeric output: | |
1035 | ||
ef3087ec KW |
1036 | use locale; |
1037 | use POSIX qw(locale_h); # Imports setlocale() and the LC_ constants. | |
1038 | setlocale(LC_NUMERIC, "fr_FR") or die "Pardon"; | |
1039 | printf "%g\n", 1.23; # If the "fr_FR" succeeded, probably shows 1,23. | |
7e4353e9 RGS |
1040 | |
1041 | and also how strings are parsed by POSIX::strtod() as numbers: | |
1042 | ||
ef3087ec KW |
1043 | use locale; |
1044 | use POSIX qw(locale_h strtod); | |
1045 | setlocale(LC_NUMERIC, "de_DE") or die "Entschuldigung"; | |
1046 | my $x = strtod("2,34") + 5; | |
1047 | print $x, "\n"; # Probably shows 7,34. | |
7e4353e9 | 1048 | |
5f05dabc | 1049 | =head1 NOTES |
1050 | ||
1051 | =head2 Backward compatibility | |
1052 | ||
b0c42ed9 | 1053 | Versions of Perl prior to 5.004 B<mostly> ignored locale information, |
5a964f20 TC |
1054 | generally behaving as if something similar to the C<"C"> locale were |
1055 | always in force, even if the program environment suggested otherwise | |
1056 | (see L<The setlocale function>). By default, Perl still behaves this | |
1057 | way for backward compatibility. If you want a Perl application to pay | |
1058 | attention to locale information, you B<must> use the S<C<use locale>> | |
062ca197 KW |
1059 | pragma (see L<The use locale pragma>) or, in the unlikely event |
1060 | that you want to do so for just pattern matching, the | |
70709c68 KW |
1061 | C</l> regular expression modifier (see L<perlre/Character set |
1062 | modifiers>) to instruct it to do so. | |
b0c42ed9 JH |
1063 | |
1064 | Versions of Perl from 5.002 to 5.003 did use the C<LC_CTYPE> | |
5a964f20 TC |
1065 | information if available; that is, C<\w> did understand what |
1066 | were the letters according to the locale environment variables. | |
b0c42ed9 JH |
1067 | The problem was that the user had no control over the feature: |
1068 | if the C library supported locales, Perl used them. | |
1069 | ||
1070 | =head2 I18N:Collate obsolete | |
1071 | ||
5a964f20 | 1072 | In versions of Perl prior to 5.004, per-locale collation was possible |
b0c42ed9 JH |
1073 | using the C<I18N::Collate> library module. This module is now mildly |
1074 | obsolete and should be avoided in new applications. The C<LC_COLLATE> | |
1075 | functionality is now integrated into the Perl core language: One can | |
1076 | use locale-specific scalar data completely normally with C<use locale>, | |
1077 | so there is no longer any need to juggle with the scalar references of | |
1078 | C<I18N::Collate>. | |
5f05dabc | 1079 | |
14280422 | 1080 | =head2 Sort speed and memory use impacts |
5f05dabc | 1081 | |
1082 | Comparing and sorting by locale is usually slower than the default | |
14280422 DD |
1083 | sorting; slow-downs of two to four times have been observed. It will |
1084 | also consume more memory: once a Perl scalar variable has participated | |
1085 | in any string comparison or sorting operation obeying the locale | |
1086 | collation rules, it will take 3-15 times more memory than before. (The | |
1087 | exact multiplier depends on the string's contents, the operating system | |
1088 | and the locale.) These downsides are dictated more by the operating | |
1089 | system's implementation of the locale system than by Perl. | |
5f05dabc | 1090 | |
e38874e2 DD |
1091 | =head2 write() and LC_NUMERIC |
1092 | ||
903eb63f NT |
1093 | If a program's environment specifies an LC_NUMERIC locale and C<use |
1094 | locale> is in effect when the format is declared, the locale is used | |
1095 | to specify the decimal point character in formatted output. Formatted | |
1096 | output cannot be controlled by C<use locale> at the time when write() | |
1097 | is called. | |
e38874e2 | 1098 | |
5f05dabc | 1099 | =head2 Freely available locale definitions |
1100 | ||
66cbab2c KW |
1101 | The Unicode CLDR project extracts the POSIX portion of many of its |
1102 | locales, available at | |
1103 | ||
1104 | http://unicode.org/Public/cldr/latest/ | |
1105 | ||
08d7a6b2 LB |
1106 | There is a large collection of locale definitions at: |
1107 | ||
1108 | http://std.dkuug.dk/i18n/WG15-collection/locales/ | |
1109 | ||
1110 | You should be aware that it is | |
14280422 | 1111 | unsupported, and is not claimed to be fit for any purpose. If your |
5a964f20 | 1112 | system allows installation of arbitrary locales, you may find the |
14280422 DD |
1113 | definitions useful as they are, or as a basis for the development of |
1114 | your own locales. | |
5f05dabc | 1115 | |
14280422 | 1116 | =head2 I18n and l10n |
5f05dabc | 1117 | |
b0c42ed9 JH |
1118 | "Internationalization" is often abbreviated as B<i18n> because its first |
1119 | and last letters are separated by eighteen others. (You may guess why | |
1120 | the internalin ... internaliti ... i18n tends to get abbreviated.) In | |
1121 | the same way, "localization" is often abbreviated to B<l10n>. | |
14280422 DD |
1122 | |
1123 | =head2 An imperfect standard | |
1124 | ||
1125 | Internationalization, as defined in the C and POSIX standards, can be | |
1126 | criticized as incomplete, ungainly, and having too large a granularity. | |
1127 | (Locales apply to a whole process, when it would arguably be more useful | |
1128 | to have them apply to a single thread, window group, or whatever.) They | |
1129 | also have a tendency, like standards groups, to divide the world into | |
1130 | nations, when we all know that the world can equally well be divided | |
e199995e | 1131 | into bankers, bikers, gamers, and so on. |
5f05dabc | 1132 | |
b310b053 JH |
1133 | =head1 Unicode and UTF-8 |
1134 | ||
7ee2ae1e KW |
1135 | The support of Unicode is new starting from Perl version v5.6, and more fully |
1136 | implemented in version v5.8 and later. See L<perluniintro>. It is | |
66cbab2c | 1137 | strongly recommended that when combining Unicode and locale (starting in |
7ee2ae1e | 1138 | v5.16), you use |
66cbab2c KW |
1139 | |
1140 | use locale ':not_characters'; | |
1141 | ||
1142 | When this form of the pragma is used, only the non-character portions of | |
1143 | locales are used by Perl, for example C<LC_NUMERIC>. Perl assumes that | |
1144 | you have translated all the characters it is to operate on into Unicode | |
1145 | (actually the platform's native character set (ASCII or EBCDIC) plus | |
1146 | Unicode). For data in files, this can conveniently be done by also | |
1147 | specifying | |
1148 | ||
1149 | use open ':locale'; | |
1150 | ||
1151 | This pragma arranges for all inputs from files to be translated into | |
1152 | Unicode from the current locale as specified in the environment (see | |
1153 | L</ENVIRONMENT>), and all outputs to files to be translated back | |
1154 | into the locale. (See L<open>). On a per-filehandle basis, you can | |
1155 | instead use the L<PerlIO::locale> module, or the L<Encode::Locale> | |
1156 | module, both available from CPAN. The latter module also has methods to | |
1157 | ease the handling of C<ARGV> and environment variables, and can be used | |
1158 | on individual strings. Also, if you know that all your locales will be | |
1159 | UTF-8, as many are these days, you can use the L<B<-C>|perlrun/-C> | |
1160 | command line switch. | |
1161 | ||
1162 | This form of the pragma allows essentially seamless handling of locales | |
1163 | with Unicode. The collation order will be Unicode's. It is strongly | |
1164 | recommended that when you need to order and sort strings that you use | |
1165 | the standard module L<Unicode::Collate> which gives much better results | |
1166 | in many instances than you can get with the old-style locale handling. | |
1167 | ||
7ee2ae1e | 1168 | For pre-v5.16 Perls, or if you use the locale pragma without the |
66cbab2c KW |
1169 | C<:not_characters> parameter, Perl tries to work with both Unicode and |
1170 | locales--but there are problems. | |
1171 | ||
1172 | Perl does not handle multi-byte locales in this case, such as have been | |
1173 | used for various | |
dc4bfc4b KW |
1174 | Asian languages, such as Big5 or Shift JIS. However, the increasingly |
1175 | common multi-byte UTF-8 locales, if properly implemented, may work | |
1176 | reasonably well (depending on your C library implementation) in this | |
1177 | form of the locale pragma, simply because both | |
1178 | they and Perl store characters that take up multiple bytes the same way. | |
1179 | However, some, if not most, C library implementations may not process | |
1180 | the characters in the upper half of the Latin-1 range (128 - 255) | |
1181 | properly under LC_CTYPE. To see if a character is a particular type | |
1182 | under a locale, Perl uses the functions like C<isalnum()>. Your C | |
1183 | library may not work for UTF-8 locales with those functions, instead | |
1184 | only working under the newer wide library functions like C<iswalnum()>. | |
e199995e KW |
1185 | |
1186 | Perl generally takes the tack to use locale rules on code points that can fit | |
66cbab2c KW |
1187 | in a single byte, and Unicode rules for those that can't (though this |
1188 | isn't uniformly applied, see the note at the end of this section). This | |
1189 | prevents many problems in locales that aren't UTF-8. Suppose the locale | |
1190 | is ISO8859-7, Greek. The character at 0xD7 there is a capital Chi. But | |
1191 | in the ISO8859-1 locale, Latin1, it is a multiplication sign. The POSIX | |
1192 | regular expression character class C<[[:alpha:]]> will magically match | |
1193 | 0xD7 in the Greek locale but not in the Latin one. | |
e199995e KW |
1194 | |
1195 | However, there are places where this breaks down. Certain constructs are | |
b4ffc3db TC |
1196 | for Unicode only, such as C<\p{Alpha}>. They assume that 0xD7 always has its |
1197 | Unicode meaning (or the equivalent on EBCDIC platforms). Since Latin1 is a | |
1198 | subset of Unicode and 0xD7 is the multiplication sign in both Latin1 and | |
1199 | Unicode, C<\p{Alpha}> will never match it, regardless of locale. A similar | |
1200 | issue occurs with C<\N{...}>. It is therefore a bad idea to use C<\p{}> or | |
66cbab2c KW |
1201 | C<\N{}> under plain C<use locale>--I<unless> you can guarantee that the |
1202 | locale will be a ISO8859-1. Use POSIX character classes instead. | |
1203 | ||
1204 | Another problem with this approach is that operations that cross the | |
1205 | single byte/multiple byte boundary are not well-defined, and so are | |
1206 | disallowed. (This boundary is between the codepoints at 255/256.). | |
1207 | For example, lower casing LATIN CAPITAL LETTER Y WITH DIAERESIS (U+0178) | |
1208 | should return LATIN SMALL LETTER Y WITH DIAERESIS (U+00FF). But in the | |
1209 | Greek locale, for example, there is no character at 0xFF, and Perl | |
1210 | has no way of knowing what the character at 0xFF is really supposed to | |
1211 | represent. Thus it disallows the operation. In this mode, the | |
1212 | lowercase of U+0178 is itself. | |
1213 | ||
1214 | The same problems ensue if you enable automatic UTF-8-ification of your | |
e199995e | 1215 | standard file handles, default C<open()> layer, and C<@ARGV> on non-ISO8859-1, |
b4ffc3db TC |
1216 | non-UTF-8 locales (by using either the B<-C> command line switch or the |
1217 | C<PERL_UNICODE> environment variable; see L<perlrun>). | |
1218 | Things are read in as UTF-8, which would normally imply a Unicode | |
1219 | interpretation, but the presence of a locale causes them to be interpreted | |
1220 | in that locale instead. For example, a 0xD7 code point in the Unicode | |
1221 | input, which should mean the multiplication sign, won't be interpreted by | |
66cbab2c | 1222 | Perl that way under the Greek locale. This is not a problem |
b4ffc3db | 1223 | I<provided> you make certain that all locales will always and only be either |
66cbab2c | 1224 | an ISO8859-1, or, if you don't have a deficient C library, a UTF-8 locale. |
b4ffc3db TC |
1225 | |
1226 | Vendor locales are notoriously buggy, and it is difficult for Perl to test | |
1227 | its locale-handling code because this interacts with code that Perl has no | |
1228 | control over; therefore the locale-handling code in Perl may be buggy as | |
66cbab2c KW |
1229 | well. (However, the Unicode-supplied locales should be better, and |
1230 | there is a feed back mechanism to correct any problems. See | |
1231 | L</Freely available locale definitions>.) | |
1232 | ||
7ee2ae1e | 1233 | If you have Perl v5.16, the problems mentioned above go away if you use |
66cbab2c | 1234 | the C<:not_characters> parameter to the locale pragma (except for vendor |
7ee2ae1e | 1235 | bugs in the non-character portions). If you don't have v5.16, and you |
66cbab2c KW |
1236 | I<do> have locales that work, using them may be worthwhile for certain |
1237 | specific purposes, as long as you keep in mind the gotchas already | |
1238 | mentioned. For example, if the collation for your locales works, it | |
1239 | runs faster under locales than under L<Unicode::Collate>; and you gain | |
1240 | access to such things as the local currency symbol and the names of the | |
7ee2ae1e | 1241 | months and days of the week. (But to hammer home the point, in v5.16, |
66cbab2c KW |
1242 | you get this access without the downsides of locales by using the |
1243 | C<:not_characters> form of the pragma.) | |
1244 | ||
1245 | Note: The policy of using locale rules for code points that can fit in a | |
1246 | byte, and Unicode rules for those that can't is not uniformly applied. | |
7ee2ae1e | 1247 | Pre-v5.12, it was somewhat haphazard; in v5.12 it was applied fairly |
66cbab2c | 1248 | consistently to regular expression matching except for bracketed |
7ee2ae1e KW |
1249 | character classes; in v5.14 it was extended to all regex matches; and in |
1250 | v5.16 to the casing operations such as C<"\L"> and C<uc()>. For | |
66cbab2c KW |
1251 | collation, in all releases, the system's C<strxfrm()> function is called, |
1252 | and whatever it does is what you get. | |
b310b053 | 1253 | |
5f05dabc | 1254 | =head1 BUGS |
1255 | ||
1256 | =head2 Broken systems | |
1257 | ||
5a964f20 | 1258 | In certain systems, the operating system's locale support |
2bdf8add | 1259 | is broken and cannot be fixed or used by Perl. Such deficiencies can |
b4ffc3db | 1260 | and will result in mysterious hangs and/or Perl core dumps when |
2bdf8add | 1261 | C<use locale> is in effect. When confronted with such a system, |
7f2de2d2 | 1262 | please report in excruciating detail to <F<perlbug@perl.org>>, and |
b4ffc3db | 1263 | also contact your vendor: bug fixes may exist for these problems |
2bdf8add JH |
1264 | in your operating system. Sometimes such bug fixes are called an |
1265 | operating system upgrade. | |
5f05dabc | 1266 | |
1267 | =head1 SEE ALSO | |
1268 | ||
b310b053 JH |
1269 | L<I18N::Langinfo>, L<perluniintro>, L<perlunicode>, L<open>, |
1270 | L<POSIX/isalnum>, L<POSIX/isalpha>, | |
4bbcc6e8 JH |
1271 | L<POSIX/isdigit>, L<POSIX/isgraph>, L<POSIX/islower>, |
1272 | L<POSIX/isprint>, L<POSIX/ispunct>, L<POSIX/isspace>, | |
1273 | L<POSIX/isupper>, L<POSIX/isxdigit>, L<POSIX/localeconv>, | |
1274 | L<POSIX/setlocale>, L<POSIX/strcoll>, L<POSIX/strftime>, | |
1275 | L<POSIX/strtod>, L<POSIX/strxfrm>. | |
5f05dabc | 1276 | |
1277 | =head1 HISTORY | |
1278 | ||
b0c42ed9 | 1279 | Jarkko Hietaniemi's original F<perli18n.pod> heavily hacked by Dominic |
5a964f20 | 1280 | Dunlop, assisted by the perl5-porters. Prose worked over a bit by |
c052850d | 1281 | Tom Christiansen, and updated by Perl 5 porters. |