Better support for =for

[perl5.git] / pod / perllocale.pod

=head1 NAME

perllocale - Perl locale handling (internationlization)

=head1 DESCRIPTION

Perl supports language-specific notions of data such as "is this a
letter", "what is the upper-case equivalent of this letter", and
"which of these letters comes first".  These are important issues,
especially for languages other than English - but also for English: it
would be very naÔve to think that C<A-Za-z> defines all the "letters".
Perl is also aware that some character other than '.' may be preferred
as a decimal point, and that output date representations may be
language-specific.

Perl can understand language-specific data via the standardized
(ISO C, XPG4, POSIX 1.c) method called "the locale system".
The locale system is controlled per application using a pragma, one
function call, and several environment variables.

B<NOTE>: This feature is new in Perl 5.004, and does not apply unless
an application specifically requests it - see L<Backward
compatibility>.

=head1 PREPARING TO USE LOCALES

If Perl applications are to be able to understand and present your
data correctly according a locale of your choice, B<all> of the following
must be true:

=over 4

=item *

B<Your operating system must support the locale system>.  If it does,
you should find that the C<setlocale> function is a documented part of
its C library.

=item *

B<Definitions for the locales which you use must be installed>.  You,
or your system administrator, must make sure that this is the case.
The available locales, the location in which they are kept, and the
manner in which they are installed, vary from system to system.  Some
systems provide only a few, hard-wired, locales, and do not allow more
to be added; others allow you to add "canned" locales provided by the
system supplier; still others allow you or the system administrator
to define and add arbitrary locales.  (You may have to ask your
supplier to provide canned locales whch are not delivered with your
operating system.)  Read your system documentation for further
illumination.

=item *

B<Perl must believe that the locale system is supported>.  If it does,
C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is
C<define>.

=back

If you want a Perl application to process and present your data
according to a particular locale, the application code should include
the S<C<use locale>> pragma (L<The use locale Pragma>) where
appropriate, and B<at least one> of the following must be true:

=over 4

=item *

B<The locale-determining environment variables (see L<ENVIRONMENT>) must
be correctly set up>, either by yourself, or by the person who set up
your system account, at the time the application is started.

=item *

B<The application must set its own locale> using the method described
in L<The C<setlocale> function>.

=back

=head1 USING LOCALES

=head2 The use locale pragma

By default, Perl ignores the current locale.  The S<C<use locale>> pragma
tells Perl to use the current locale for some operations:

=over 4

=item *

B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>)
use C<LC_COLLATE>.  The C<sort> function is also affected if it is
used without an explicit comparison function because it uses C<cmp> by
default.

B<Note:> The C<eq> and C<ne> operators are unaffected by the locale:
they always perform a byte-by-byte comparison of their scalar
arguments.  If you really want to know if two strings - which C<eq>
may consider different - are equal as far as collation is concerned,
use something like

    !("space and case ignored" cmp "SpaceAndCaseIgnored")

(which would be true if the collation locale specified a
dictionary-like ordering).

I<Editor's note:> I am right about C<eq> and C<ne>, aren't I?

=item *

B<Regular expressions and case-modification functions> (C<uc>,
C<lc>, C<ucfirst>, and C<lcfirst>) use C<LC_CTYPE>

=item *

B<The formatting functions> (C<printf> and C<sprintf>) use
C<LC_NUMERIC>

=item *

B<The POSIX date formatting function> (C<strftime>) uses C<LC_TIME>.

=back

C<LC_COLLATE>, C<LC_CTYPE>, and so on, are discussed further in
L<LOCALE CATEGORIES>.

The default behaviour returns with S<C<no locale>> or on reaching the end
of the enclosing block.

Note that the result of any operation that uses locale information is
tainted (see L<perlsec.pod>), since locales can be created by
unprivileged users on some systems.

=head2 The setlocale function

You can switch locales as often as you wish at runtime with the
C<POSIX::setlocale> function:

        # This functionality not usable prior to Perl 5.004
        require 5.004;

        # Import locale-handling tool set from POSIX module.
        # This example uses: setlocale -- the function call
        #                    LC_CTYPE -- explained below
        use POSIX qw(locale_h);

        # query and save the old locale.
        $old_locale = setlocale(LC_CTYPE);

        setlocale(LC_CTYPE, "fr_CA.ISO8859-1");
        # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1"

        setlocale(LC_CTYPE, "");
        # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG
        # environment variables.  See below for documentation.

        # restore the old locale
        setlocale(LC_CTYPE, $old_locale);

The first argument of C<setlocale> gives the B<category>, the second
the B<locale>.  The category tells in what aspect of data processing
you want to apply locale-specific rules.  Category names are discussed
in L<LOCALE CATEGORIES> and L<ENVIRONMENT>.  The locale is the name of
a collection of customization information corresponding to a paricular
combination of language, country or territory, and codeset.  Read on
for hints on the naming of locales: not all systems name locales as in
the example.

If no second argument is provided, the function returns a string
naming the current locale for the category.  You can use this value as
the second argument in a subsequent call to C<setlocale>.  If a second
argument is given and it corresponds to a valid locale, the locale for
the category is set to that value, and the function returns the
now-current locale value.  You can use this in a subsequent call to
C<setlocale>.  (In some implementations, the return value may sometimes
differ from the value you gave as the second argument - think of it as
an alias for the value that you gave.)

As the example shows, if the second argument is an empty string, the
category's locale is returned to the default specified by the
corresponding environment variables.  Generally, this results in a
return to the default which was in force when Perl started up: changes
to the environment made by the application after start-up may or may
not be noticed, depending on the implementation of your system's C
library.

If the second argument does not correspond to a valid locale, the
locale for the category is not changed, and the function returns
C<undef>.

For further information about the categories, consult
L<setlocale(3)>.  For the locales available in your system,
also consult L<setlocale(3)> and see whether it leads you
to the list of the available locales (search for the C<SEE ALSO>
section).  If that fails, try the following command lines:

        locale -a

        nlsinfo

        ls /usr/lib/nls/loc

        ls /usr/lib/locale

        ls /usr/lib/nls

and see whether they list something resembling these

        en_US.ISO8859-1         de_DE.ISO8859-1         ru_RU.ISO8859-5
        en_US                   de_DE                   ru_RU
        en                      de                      ru
        english                 german                  russian
        english.iso88591        german.iso88591         russian.iso88595

Sadly, even though the calling interface for C<setlocale> has been
standardized, the names of the locales have not.  The form of the name
is usually I<language_country>B</>I<territory>B<.>I<codeset>, but the
latter parts are not always present.

Two special locales are worth particular mention: "C" and
"POSIX".  Currently these are effectively the same locale: the
difference is mainly that the first one is defined by the C standard
and the second by the POSIX standard.  What they define is the
B<default locale> in which every program starts in the absence of
locale information in its environment.  (The default default locale,
if you will.)  Its language is (American) English and its character
codeset ASCII.

B<NOTE>: Not all systems have the "POSIX" locale (not all systems
are POSIX-conformant), so use "C" when you need explicitly to
specify this default locale.

=head2 The localeconv function

The C<POSIX::localeconv> function allows you to get particulars of the
locale-dependent numeric formatting information specified by the
current C<LC_NUMERIC> and C<LC_MONETARY> locales.  (If you just want
the name of the current locale for a particular category, use
C<POSIX::setlocale> with a single parameter - see L<The setlocale
function>.)

        use POSIX qw(locale_h);
        use locale;

        # Get a reference to a hash of locale-dependent info
        $locale_values = localeconv();

        # Output sorted list of the values
        for (sort keys %$locale_values) {
                printf "%-20s = %s\n", $_, $locale_values->{$_}
        }

C<localeconv> takes no arguments, and returns B<a reference to> a
hash.  The keys of this hash are formatting variable names such as
C<decimal_point> and C<thousands_sep>; the values are the
corresponding values.  See L<POSIX (3)/localeconv> for a longer
example, which lists all the categories an implementation might be
expected to provide; some provide more and others fewer, however.

I<Editor's note:> I can't work out whether C<POSIX::localeconv>
correctly obeys C<use locale> and C<no locale>.  In my opinion, it
should, if only to be consistent with other locale stuff - although
it's hardly a show-stopper if it doesn't.  Could someone check,
please?

Here's a simple-minded example program which rewrites its command line
parameters as integers formatted correctly in the current locale:

        # See comments in previous example
        require 5.004;
        use POSIX qw(locale_h);
        use locale;

        # Get some of locale's numeric formatting parameters
        my ($thousands_sep, $grouping) =
            @{localeconv()}{'thousands_sep', 'grouping'};

        # Apply defaults if values are missing
        $thousands_sep = ',' unless $thousands_sep;
        $grouping = 3 unless $grouping;

        # Format command line params for current locale
        for (@ARGV)
        {
            $_ = int; # Chop non-integer part
            1 while
                s/(\d)(\d{$grouping}($|$thousands_sep))/$1$thousands_sep$2/;
            print "$_ ";
        }
        print "\n";

I<Editor's note:> Like all the examples, this needs testing on systems
which, unlike mine, have non-toy implementations of locale handling.

=head1 LOCALE CATEGORIES

The subsections which follow descibe basic locale categories.  As well
as these, there are some combination categories which allow the
manipulation of of more than one basic category at a time.  See
L<ENVIRONMENT VARIABLES> for a discussion of these.

=head2 Category LC_COLLATE: Collation

When in the scope of S<C<use locale>>, Perl looks to the B<LC_COLLATE>
environment variable to determine the application's notions on the
collation (ordering) of characters.  ('B' follows 'A' in Latin
alphabets, but where do '¡' and '\9f' belong?)

Here is a code snippet that will tell you what are the alphanumeric
characters in the current locale, in the locale order:

        use locale;
        print +(sort grep /\w/, map { chr() } 0..255), "\n";

I<Editor's note:> The original example had C<setlocale(LC_COLLATE, "")>
prior to C<print ...>.  I think this is wrong: as soon as you utter
S<C<use locale>>, the default behaviour of C<sort> (well, C<cmp>, really)
becomes locale-aware.  The locale it's aware of is the current locale
which, unless you've changed it yourself, is the default locale
defined by your environment.

Compare this with the characters that you see and their order if you state
explicitly that the locale should be ignored:

        no locale;
        print +(sort grep /\w/, map { chr() } 0..255), "\n";

This machine-native collation (which is what you get unless S<C<use
locale>> has appeared earlier in the same block) must be used for
sorting raw binary data, whereas the locale-dependent collation of the
first example is useful for written text.

B<NOTE>: In some locales some characters may have no collation value
at all - for example, if '-' is such a character, 'relocate' and
're-locate' may be considered to be equal to each other, and so sort
to the same position.

=head2 Category LC_CTYPE: Character Types

When in the scope of S<C<use locale>>, Perl obeys the C<LC_CTYPE> locale
setting.  This controls the application's notion of which characters
are alphabetic.  This affects Perl's C<\w> regular expression
metanotation, which stands for alphanumeric characters - that is,
alphabetic and numeric characters.  (Consult L<perlre> for more
information about regular expressions.)  Thanks to C<LC_CTYPE>,
depending on your locale setting, characters like '\90', '\8a',
'þ', and '¯' may be understood as C<\w> characters.

C<LC_CTYPE> also affects the POSIX character-class test functions -
C<isalpha>, C<islower> and so on.  For example, if you move from the
"C" locale to a 7-bit Scandinavian one, you may find - possibly to
your surprise -that "|" moves from the C<ispunct> class to C<isalpha>.

I<Editor's note:> I can't work out whether the C<POSIX::is...> stuff
correctly obeys C<use locale> and C<no locale>.  In my opinion, they
should.  Could someone check, please?

B<Note:> A broken or malicious C<LC_CTYPE> locale definition may
result in clearly ineligible characters being considered to be
alphanumeric by your application.  For strict matching of (unaccented)
letters and digits - for example, in command strings - locale-aware
applications should use C<\w> inside a C<no locale> block.

=head2 Category LC_NUMERIC: Numeric Formatting

When in the scope of S<C<use locale>>, Perl obeys the C<LC_NUMERIC>
locale information which controls application's idea of how numbers
should be formatted for human readability by the C<printf>, C<fprintf>,
and C<write> functions.  String to numeric conversion by the
C<POSIX::strtod> function is also affected.  In most impementations
the only effect is to change the character used for the decimal point
- perhaps from '.'  to ',': these functions aren't aware of such
niceties as thousands separation and so on.  (See L<The localeconv
function> if you care about these things.)

I<Editor's note:> I can't work out whether C<POSIX::strtod> correctly
obeys C<use locale> and C<no locale>.  In my opinion, it should -
although it's hardly a show-stopper if it doesn't.  Could someone
check, please?

Note that output produced by C<print> is B<never> affected by the
current locale: it is independent of whether C<use locale> or C<no
locale> is in effect, and corresponds to what you'd get from C<printf>
in the "C" locale.  The same is true for Perl's internal conversions
between numeric and string formats:

        use POSIX qw(strtod);
        use locale;
        $n = 5/2;   # Assign numeric 2.5 to $n

        $a = " $n"; # Locale-independent conversion to string

        print "half five is $n\n";       # Locale-independent output

        printf "half five is %g\n", $n;  # Locale-dependent output

        print "DECIMAL POINT IS COMMA\n" # Locale-dependent conversion
            if $n == (strtod("2,5"))[0];

=head2 Category LC_MONETARY: Formatting of monetary amounts

The C standard defines the C<LC_MONETARY> category, but no function
that is affected by its contents.  (Those with experience of standards
committees will recognise that the working group decided to punt on
the issue.)  Consequently, Perl takes no notice of it.  If you really
want to use C<LC_MONETARY>, you can query its contents - see L<The
localeconv function> - and use the information that it returns in your
application's own formating of currency amounts.  However, you may
well find that the information, though voluminous and complex, does
not quite meet your requirements: currency formatting is a hard nut to
crack.

=head2 LC_TIME

The output produced by C<POSIX::strftime>, which builds a formatted
human-readable date/time string, is affected by the current C<LC_TIME>
locale.  Thus, in a French locale, the output produced by the C<%B>
format element (full month name) for the first month of the year would
be "janvier".  Here's how to get a list of the long month names in the
current locale:

        use POSIX qw(strftime);
        use locale;
        for (0..11)
        {
            $long_month_name[$_] = strftime("%B", 0, 0, 0, 1, $_, 96);
        }

I<Editor's note:> Unchecked in "alien" locales: my system can't do
French...

=head2 Other categories

The remaining locale category, C<LC_MESSAGES> (possibly supplemented by
others in particular implementations) is not currently used by Perl -
except possibly to affect the behaviour of library functions called
by extensions which are not part of the standard Perl distribution.

=head1 ENVIRONMENT

=over 12

=item PERL_BADLANG

A string that controls whether Perl warns in its startup about failed
locale settings.  This can happen if the locale support in the
operating system is lacking (broken) is some way.  If this string has
an integer value differing from zero, Perl will not complain.

B<NOTE>: This is just hiding the warning message.  The message tells
about some problem in your system's locale support and you should
investigate what the problem is.

=back

The following environment variables are not specific to Perl: They are
part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale method to
control an application's opinion on data.

=over 12

=item LC_ALL

C<LC_ALL> is the "override-all" locale environment variable. If it is
set, it overrides all the rest of the locale environment variables.

=item LC_CTYPE

In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type
locale.  In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG>
chooses the character type locale.

=item LC_COLLATE

In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation (sorting)
locale.  In the absence of both C<LC_ALL> and C<LC_COLLATE>, C<LANG>
chooses the collation locale.

=item LC_MONETARY

In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the montary formatting
locale.  In the absence of both C<LC_ALL> and C<LC_MONETARY>, C<LANG>
chooses the monetary formatting locale.

=item LC_NUMERIC

In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format
locale.  In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG>
chooses the numeric format.

=item LC_TIME

In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time formatting
locale.  In the absence of both C<LC_ALL> and C<LC_TIME>, C<LANG>
chooses the date and time formatting locale.

=item LANG

C<LANG> is the "catch-all" locale environment variable. If it is set,
it is used as the last resort after the overall C<LC_ALL> and the
category-specific C<LC_...>.

=back

=head1 NOTES

=head2 Backward compatibility

Versions of Perl prior to 5.004 ignored locale information, generally
behaving as if something similar to the C<"C"> locale (see L<The
setlocale function>) was always in force, even if the program
environment suggested otherwise.  By default, Perl still behaves this
way so as to maintain backward compatibility.  If you want a Perl
application to pay attention to locale information, you B<must> use
the S<C<use locale>> pragma (see L<The S<C<use locale>> Pragma>) to
instruct it to do so.

=head2 Sort speed

Comparing and sorting by locale is usually slower than the default
sorting; factors of 2 to 4 have been observed.  It will also consume
more memory: while a Perl scalar variable is participating in any
string comparison or sorting operation and obeying the locale
collation rules it will take about 3-15 (the exact value depends on
the operating system and the locale) times more memory than normally.
These downsides are dictated more by the operating system
implementation of the locale system than by Perl.

=head2 I18N:Collate

In Perl 5.003 (and later development releases prior to 5.003_06),
per-locale collation was possible using the C<I18N::Collate> library
module.  This is now mildly obsolete and should be avoided in new
applications.  The C<LC_COLLATE> functionality is integrated into the
Perl core language and one can use locale-specific scalar data
completely normally - there is no need to juggle with the scalar
references of C<I18N::Collate>.

=head2 An imperfect standard

Internationalization, as defined in the C and POSIX standards, can be
criticized as incomplete, ungainly, and having too large a
granularity.  (Locales apply to a whole process, when it would
arguably be more useful to have them apply to a single thread, window
group, or whatever.)  They also have a tendency, like standards
groups, to divide the world into nations, when we all know that the
world can equally well be divided into bankers, bikers, gamers, and so
on.  But, for now, it's the only standard we've got.  This may be
construed as a bug.

=head2 Freely available locale definitions

There is a large collection of locale definitions at
C<ftp://dkuug.dk/i18n/WG15-collection>.  You should be aware that they
are unsupported, and are not claimed to be fit for any purpose.  If
your system allows the installation of arbitrary locales, you may find
them useful as they are, or as a basis for the development of your own
locales.

=head2 i18n and l10n

Internationalization is often abbreviated as B<i18n> because its first
and last letters are separated by eighteen others.  You can also talk of
localization (B<l10n>), the process of tailoring an
internationalizated application for use in a particular locale.

=head1 BUGS

=head2 Broken systems

In certain system environments the operating system's locale support
is broken and cannot be fixed or used by Perl.  Such deficiencies can
and will result in mysterious hangs and/or Perl core dumps.  One
example is IRIX before release 6.2, in which the C<LC_COLLATE> support
simply does not work.  When confronted with such a system, please
report in excruciating detail to C<perlbug@perl.com>, and complain to
your vendor: maybe some bug fixes exist for these problems in your
operating system.  Sometimes such bug fixes are called an operating
system upgrade.

=head2 Rendering of this documentation

This manual page contains non-ASCII characters, which should all be
rendered as accented letters, and which should make some sort of sense
in context.  If this is not the case, your system is probably not
using the ISO 8859-1 character set which was used to write them,
and/or your formatting, display, and printing software are not
correctly mapping them to your host's character set.  If this annoys
you, and if you can convince yourself that it is due to a bug in one
of Perl's various C<pod2>... utilities, by all means report it as a
Perl bug.  Otherwise, pausing only to curse anyone who ever invented
yet another character set, see if you can make it handle ISO 8859-1
sensibly.

=head1 SEE ALSO

L<POSIX (3)/isalnum>, L<POSIX (3)/isalpha>, L<POSIX (3)/isdigit>,
L<POSIX (3)/isgraph>, L<POSIX (3)/islower>, L<POSIX (3)/isprint>,
L<POSIX (3)/ispunct>, L<POSIX (3)/isspace>, L<POSIX (3)/isupper>,
L<POSIX (3)/isxdigit>, L<POSIX (3)/localeconv>, L<POSIX (3)/setlocale>,
L<POSIX (3)/strtod>

I<Editor's note:> That looks horrible after going through C<pod2man>.
But I do want to call out all thse sectins by name.  What should I
have done?

=head1 HISTORY

Perl 5.003's F<perli18n.pod> heavily hacked by Dominic Dunlop.

Last update:
Mon Dec 16 14:13:10 WET 1996