This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
[DOC PATCH] Add perl4 warning messages to perldiag.pod
[perl5.git] / pod / perlguts.pod
CommitLineData
a0d0e21e
LW
1=head1 NAME
2
954c1994 3perlguts - Introduction to the Perl API
a0d0e21e
LW
4
5=head1 DESCRIPTION
6
b3b6085d
PP
7This document attempts to describe how to use the Perl API, as well as
8containing some info on the basic workings of the Perl core. It is far
9from complete and probably contains many errors. Please refer any
10questions or comments to the author below.
a0d0e21e 11
0a753a76
PP
12=head1 Variables
13
5f05dabc 14=head2 Datatypes
a0d0e21e
LW
15
16Perl has three typedefs that handle Perl's three main data types:
17
18 SV Scalar Value
19 AV Array Value
20 HV Hash Value
21
d1b91892 22Each typedef has specific routines that manipulate the various data types.
a0d0e21e
LW
23
24=head2 What is an "IV"?
25
954c1994 26Perl uses a special typedef IV which is a simple signed integer type that is
5f05dabc 27guaranteed to be large enough to hold a pointer (as well as an integer).
954c1994 28Additionally, there is the UV, which is simply an unsigned IV.
a0d0e21e 29
d1b91892 30Perl also uses two special typedefs, I32 and I16, which will always be at
954c1994
GS
31least 32-bits and 16-bits long, respectively. (Again, there are U32 and U16,
32as well.)
a0d0e21e 33
54310121 34=head2 Working with SVs
a0d0e21e
LW
35
36An SV can be created and loaded with one command. There are four types of
a7dfe00a
MS
37values that can be loaded: an integer value (IV), a double (NV),
38a string (PV), and another scalar (SV).
a0d0e21e 39
9da1e3b5 40The six routines are:
a0d0e21e
LW
41
42 SV* newSViv(IV);
43 SV* newSVnv(double);
08105a92
GS
44 SV* newSVpv(const char*, int);
45 SV* newSVpvn(const char*, int);
46fc3d4c 46 SV* newSVpvf(const char*, ...);
a0d0e21e
LW
47 SV* newSVsv(SV*);
48
deb3007b 49To change the value of an *already-existing* SV, there are seven routines:
a0d0e21e
LW
50
51 void sv_setiv(SV*, IV);
deb3007b 52 void sv_setuv(SV*, UV);
a0d0e21e 53 void sv_setnv(SV*, double);
08105a92
GS
54 void sv_setpv(SV*, const char*);
55 void sv_setpvn(SV*, const char*, int)
46fc3d4c 56 void sv_setpvf(SV*, const char*, ...);
9abd00ed 57 void sv_setpvfn(SV*, const char*, STRLEN, va_list *, SV **, I32, bool);
a0d0e21e
LW
58 void sv_setsv(SV*, SV*);
59
60Notice that you can choose to specify the length of the string to be
9da1e3b5
MUN
61assigned by using C<sv_setpvn>, C<newSVpvn>, or C<newSVpv>, or you may
62allow Perl to calculate the length by using C<sv_setpv> or by specifying
630 as the second argument to C<newSVpv>. Be warned, though, that Perl will
64determine the string's length by using C<strlen>, which depends on the
9abd00ed
GS
65string terminating with a NUL character.
66
67The arguments of C<sv_setpvf> are processed like C<sprintf>, and the
68formatted output becomes the value.
69
70C<sv_setpvfn> is an analogue of C<vsprintf>, but it allows you to specify
71either a pointer to a variable argument list or the address and length of
72an array of SVs. The last argument points to a boolean; on return, if that
73boolean is true, then locale-specific information has been used to format
c2611fb3 74the string, and the string's contents are therefore untrustworthy (see
9abd00ed
GS
75L<perlsec>). This pointer may be NULL if that information is not
76important. Note that this function requires you to specify the length of
77the format.
78
7c906a97 79STRLEN is an integer type (Size_t, usually defined as size_t in
00aadd71 80config.h) guaranteed to be large enough to represent the size of
7c906a97
AD
81any string that perl can handle.
82
9da1e3b5
MUN
83The C<sv_set*()> functions are not generic enough to operate on values
84that have "magic". See L<Magic Virtual Tables> later in this document.
a0d0e21e 85
a3cb178b
GS
86All SVs that contain strings should be terminated with a NUL character.
87If it is not NUL-terminated there is a risk of
5f05dabc
PP
88core dumps and corruptions from code which passes the string to C
89functions or system calls which expect a NUL-terminated string.
90Perl's own functions typically add a trailing NUL for this reason.
91Nevertheless, you should be very careful when you pass a string stored
92in an SV to a C function or system call.
93
a0d0e21e
LW
94To access the actual value that an SV points to, you can use the macros:
95
96 SvIV(SV*)
954c1994 97 SvUV(SV*)
a0d0e21e
LW
98 SvNV(SV*)
99 SvPV(SV*, STRLEN len)
1fa8b10d 100 SvPV_nolen(SV*)
a0d0e21e 101
954c1994 102which will automatically coerce the actual scalar type into an IV, UV, double,
a0d0e21e
LW
103or string.
104
105In the C<SvPV> macro, the length of the string returned is placed into the
1fa8b10d
JD
106variable C<len> (this is a macro, so you do I<not> use C<&len>). If you do
107not care what the length of the data is, use the C<SvPV_nolen> macro.
108Historically the C<SvPV> macro with the global variable C<PL_na> has been
109used in this case. But that can be quite inefficient because C<PL_na> must
110be accessed in thread-local storage in threaded Perl. In any case, remember
111that Perl allows arbitrary strings of data that may both contain NULs and
112might not be terminated by a NUL.
a0d0e21e 113
ce2f5d8f
KA
114Also remember that C doesn't allow you to safely say C<foo(SvPV(s, len),
115len);>. It might work with your compiler, but it won't work for everyone.
116Break this sort of statement up into separate assignments:
117
b2f5ed49 118 SV *s;
ce2f5d8f
KA
119 STRLEN len;
120 char * ptr;
b2f5ed49 121 ptr = SvPV(s, len);
ce2f5d8f
KA
122 foo(ptr, len);
123
07fa94a1 124If you want to know if the scalar value is TRUE, you can use:
a0d0e21e
LW
125
126 SvTRUE(SV*)
127
128Although Perl will automatically grow strings for you, if you need to force
129Perl to allocate more memory for your SV, you can use the macro
130
131 SvGROW(SV*, STRLEN newlen)
132
133which will determine if more memory needs to be allocated. If so, it will
134call the function C<sv_grow>. Note that C<SvGROW> can only increase, not
5f05dabc
PP
135decrease, the allocated memory of an SV and that it does not automatically
136add a byte for the a trailing NUL (perl's own string functions typically do
8ebc5c01 137C<SvGROW(sv, len + 1)>).
a0d0e21e
LW
138
139If you have an SV and want to know what kind of data Perl thinks is stored
140in it, you can use the following macros to check the type of SV you have.
141
142 SvIOK(SV*)
143 SvNOK(SV*)
144 SvPOK(SV*)
145
146You can get and set the current length of the string stored in an SV with
147the following macros:
148
149 SvCUR(SV*)
150 SvCUR_set(SV*, I32 val)
151
cb1a09d0
AD
152You can also get a pointer to the end of the string stored in the SV
153with the macro:
154
155 SvEND(SV*)
156
157But note that these last three macros are valid only if C<SvPOK()> is true.
a0d0e21e 158
d1b91892
AD
159If you want to append something to the end of string stored in an C<SV*>,
160you can use the following functions:
161
08105a92 162 void sv_catpv(SV*, const char*);
e65f3abd 163 void sv_catpvn(SV*, const char*, STRLEN);
46fc3d4c 164 void sv_catpvf(SV*, const char*, ...);
9abd00ed 165 void sv_catpvfn(SV*, const char*, STRLEN, va_list *, SV **, I32, bool);
d1b91892
AD
166 void sv_catsv(SV*, SV*);
167
168The first function calculates the length of the string to be appended by
169using C<strlen>. In the second, you specify the length of the string
46fc3d4c 170yourself. The third function processes its arguments like C<sprintf> and
9abd00ed
GS
171appends the formatted output. The fourth function works like C<vsprintf>.
172You can specify the address and length of an array of SVs instead of the
173va_list argument. The fifth function extends the string stored in the first
174SV with the string stored in the second SV. It also forces the second SV
175to be interpreted as a string.
176
177The C<sv_cat*()> functions are not generic enough to operate on values that
178have "magic". See L<Magic Virtual Tables> later in this document.
d1b91892 179
a0d0e21e
LW
180If you know the name of a scalar variable, you can get a pointer to its SV
181by using the following:
182
4929bf7b 183 SV* get_sv("package::varname", FALSE);
a0d0e21e
LW
184
185This returns NULL if the variable does not exist.
186
d1b91892 187If you want to know if this variable (or any other SV) is actually C<defined>,
a0d0e21e
LW
188you can call:
189
190 SvOK(SV*)
191
9cde0e7f 192The scalar C<undef> value is stored in an SV instance called C<PL_sv_undef>. Its
a0d0e21e
LW
193address can be used whenever an C<SV*> is needed.
194
9cde0e7f
GS
195There are also the two values C<PL_sv_yes> and C<PL_sv_no>, which contain Boolean
196TRUE and FALSE values, respectively. Like C<PL_sv_undef>, their addresses can
a0d0e21e
LW
197be used whenever an C<SV*> is needed.
198
9cde0e7f 199Do not be fooled into thinking that C<(SV *) 0> is the same as C<&PL_sv_undef>.
a0d0e21e
LW
200Take this code:
201
202 SV* sv = (SV*) 0;
203 if (I-am-to-return-a-real-value) {
204 sv = sv_2mortal(newSViv(42));
205 }
206 sv_setsv(ST(0), sv);
207
208This code tries to return a new SV (which contains the value 42) if it should
04343c6d 209return a real value, or undef otherwise. Instead it has returned a NULL
a0d0e21e 210pointer which, somewhere down the line, will cause a segmentation violation,
9cde0e7f 211bus error, or just weird results. Change the zero to C<&PL_sv_undef> in the first
5f05dabc 212line and all will be well.
a0d0e21e
LW
213
214To free an SV that you've created, call C<SvREFCNT_dec(SV*)>. Normally this
3fe9a6f1 215call is not necessary (see L<Reference Counts and Mortality>).
a0d0e21e 216
94dde4fb
SC
217=head2 Offsets
218
219Perl provides the function C<sv_chop> to efficiently remove characters
220from the beginning of a string; you give it an SV and a pointer to
221somewhere inside the the PV, and it discards everything before the
222pointer. The efficiency comes by means of a little hack: instead of
223actually removing the characters, C<sv_chop> sets the flag C<OOK>
224(offset OK) to signal to other functions that the offset hack is in
225effect, and it puts the number of bytes chopped off into the IV field
226of the SV. It then moves the PV pointer (called C<SvPVX>) forward that
00aadd71 227many bytes, and adjusts C<SvCUR> and C<SvLEN>.
94dde4fb
SC
228
229Hence, at this point, the start of the buffer that we allocated lives
230at C<SvPVX(sv) - SvIV(sv)> in memory and the PV pointer is pointing
231into the middle of this allocated storage.
232
233This is best demonstrated by example:
234
235 % ./perl -Ilib -MDevel::Peek -le '$a="12345"; $a=~s/.//; Dump($a)'
236 SV = PVIV(0x8128450) at 0x81340f0
237 REFCNT = 1
238 FLAGS = (POK,OOK,pPOK)
239 IV = 1 (OFFSET)
240 PV = 0x8135781 ( "1" . ) "2345"\0
241 CUR = 4
242 LEN = 5
243
244Here the number of bytes chopped off (1) is put into IV, and
245C<Devel::Peek::Dump> helpfully reminds us that this is an offset. The
246portion of the string between the "real" and the "fake" beginnings is
247shown in parentheses, and the values of C<SvCUR> and C<SvLEN> reflect
248the fake beginning, not the real one.
249
319cef53
SC
250Something similar to the offset hack is perfomed on AVs to enable
251efficient shifting and splicing off the beginning of the array; while
252C<AvARRAY> points to the first element in the array that is visible from
253Perl, C<AvALLOC> points to the real start of the C array. These are
254usually the same, but a C<shift> operation can be carried out by
255increasing C<AvARRAY> by one and decreasing C<AvFILL> and C<AvLEN>.
256Again, the location of the real start of the C array only comes into
257play when freeing the array. See C<av_shift> in F<av.c>.
258
d1b91892 259=head2 What's Really Stored in an SV?
a0d0e21e
LW
260
261Recall that the usual method of determining the type of scalar you have is
5f05dabc 262to use C<Sv*OK> macros. Because a scalar can be both a number and a string,
d1b91892 263usually these macros will always return TRUE and calling the C<Sv*V>
a0d0e21e
LW
264macros will do the appropriate conversion of string to integer/double or
265integer/double to string.
266
267If you I<really> need to know if you have an integer, double, or string
268pointer in an SV, you can use the following three macros instead:
269
270 SvIOKp(SV*)
271 SvNOKp(SV*)
272 SvPOKp(SV*)
273
274These will tell you if you truly have an integer, double, or string pointer
d1b91892 275stored in your SV. The "p" stands for private.
a0d0e21e 276
9e9796d6
JH
277The are various ways in which the private and public flags may differ.
278For example, a tied SV may have a valid underlying value in the IV slot
279(so SvIOKp is true), but the data should be accessed via the FETCH
280routine rather than directly, so SvIOK is false. Another is when
281numeric conversion has occured and precision has been lost: only the
282private flag is set on 'lossy' values. So when an NV is converted to an
283IV with loss, SvIOKp, SvNOKp and SvNOK will be set, while SvIOK wont be.
284
07fa94a1 285In general, though, it's best to use the C<Sv*V> macros.
a0d0e21e 286
54310121 287=head2 Working with AVs
a0d0e21e 288
07fa94a1
JO
289There are two ways to create and load an AV. The first method creates an
290empty AV:
a0d0e21e
LW
291
292 AV* newAV();
293
54310121 294The second method both creates the AV and initially populates it with SVs:
a0d0e21e
LW
295
296 AV* av_make(I32 num, SV **ptr);
297
5f05dabc 298The second argument points to an array containing C<num> C<SV*>'s. Once the
54310121 299AV has been created, the SVs can be destroyed, if so desired.
a0d0e21e 300
54310121 301Once the AV has been created, the following operations are possible on AVs:
a0d0e21e
LW
302
303 void av_push(AV*, SV*);
304 SV* av_pop(AV*);
305 SV* av_shift(AV*);
306 void av_unshift(AV*, I32 num);
307
308These should be familiar operations, with the exception of C<av_unshift>.
309This routine adds C<num> elements at the front of the array with the C<undef>
310value. You must then use C<av_store> (described below) to assign values
311to these new elements.
312
313Here are some other functions:
314
5f05dabc 315 I32 av_len(AV*);
a0d0e21e 316 SV** av_fetch(AV*, I32 key, I32 lval);
a0d0e21e 317 SV** av_store(AV*, I32 key, SV* val);
a0d0e21e 318
5f05dabc
PP
319The C<av_len> function returns the highest index value in array (just
320like $#array in Perl). If the array is empty, -1 is returned. The
321C<av_fetch> function returns the value at index C<key>, but if C<lval>
322is non-zero, then C<av_fetch> will store an undef value at that index.
04343c6d
GS
323The C<av_store> function stores the value C<val> at index C<key>, and does
324not increment the reference count of C<val>. Thus the caller is responsible
325for taking care of that, and if C<av_store> returns NULL, the caller will
326have to decrement the reference count to avoid a memory leak. Note that
327C<av_fetch> and C<av_store> both return C<SV**>'s, not C<SV*>'s as their
328return value.
d1b91892 329
a0d0e21e 330 void av_clear(AV*);
a0d0e21e 331 void av_undef(AV*);
cb1a09d0 332 void av_extend(AV*, I32 key);
5f05dabc
PP
333
334The C<av_clear> function deletes all the elements in the AV* array, but
335does not actually delete the array itself. The C<av_undef> function will
336delete all the elements in the array plus the array itself. The
adc882cf
GS
337C<av_extend> function extends the array so that it contains at least C<key+1>
338elements. If C<key+1> is less than the currently allocated length of the array,
339then nothing is done.
a0d0e21e
LW
340
341If you know the name of an array variable, you can get a pointer to its AV
342by using the following:
343
4929bf7b 344 AV* get_av("package::varname", FALSE);
a0d0e21e
LW
345
346This returns NULL if the variable does not exist.
347
04343c6d
GS
348See L<Understanding the Magic of Tied Hashes and Arrays> for more
349information on how to use the array access functions on tied arrays.
350
54310121 351=head2 Working with HVs
a0d0e21e
LW
352
353To create an HV, you use the following routine:
354
355 HV* newHV();
356
54310121 357Once the HV has been created, the following operations are possible on HVs:
a0d0e21e 358
08105a92
GS
359 SV** hv_store(HV*, const char* key, U32 klen, SV* val, U32 hash);
360 SV** hv_fetch(HV*, const char* key, U32 klen, I32 lval);
a0d0e21e 361
5f05dabc
PP
362The C<klen> parameter is the length of the key being passed in (Note that
363you cannot pass 0 in as a value of C<klen> to tell Perl to measure the
364length of the key). The C<val> argument contains the SV pointer to the
54310121 365scalar being stored, and C<hash> is the precomputed hash value (zero if
5f05dabc
PP
366you want C<hv_store> to calculate it for you). The C<lval> parameter
367indicates whether this fetch is actually a part of a store operation, in
368which case a new undefined value will be added to the HV with the supplied
369key and C<hv_fetch> will return as if the value had already existed.
a0d0e21e 370
5f05dabc
PP
371Remember that C<hv_store> and C<hv_fetch> return C<SV**>'s and not just
372C<SV*>. To access the scalar value, you must first dereference the return
373value. However, you should check to make sure that the return value is
374not NULL before dereferencing it.
a0d0e21e
LW
375
376These two functions check if a hash table entry exists, and deletes it.
377
08105a92
GS
378 bool hv_exists(HV*, const char* key, U32 klen);
379 SV* hv_delete(HV*, const char* key, U32 klen, I32 flags);
a0d0e21e 380
5f05dabc
PP
381If C<flags> does not include the C<G_DISCARD> flag then C<hv_delete> will
382create and return a mortal copy of the deleted value.
383
a0d0e21e
LW
384And more miscellaneous functions:
385
386 void hv_clear(HV*);
a0d0e21e 387 void hv_undef(HV*);
5f05dabc
PP
388
389Like their AV counterparts, C<hv_clear> deletes all the entries in the hash
390table but does not actually delete the hash table. The C<hv_undef> deletes
391both the entries and the hash table itself.
a0d0e21e 392
d1b91892
AD
393Perl keeps the actual data in linked list of structures with a typedef of HE.
394These contain the actual key and value pointers (plus extra administrative
395overhead). The key is a string pointer; the value is an C<SV*>. However,
396once you have an C<HE*>, to get the actual key and value, use the routines
397specified below.
398
a0d0e21e
LW
399 I32 hv_iterinit(HV*);
400 /* Prepares starting point to traverse hash table */
401 HE* hv_iternext(HV*);
402 /* Get the next entry, and return a pointer to a
403 structure that has both the key and value */
404 char* hv_iterkey(HE* entry, I32* retlen);
405 /* Get the key from an HE structure and also return
406 the length of the key string */
cb1a09d0 407 SV* hv_iterval(HV*, HE* entry);
a0d0e21e
LW
408 /* Return a SV pointer to the value of the HE
409 structure */
cb1a09d0 410 SV* hv_iternextsv(HV*, char** key, I32* retlen);
d1b91892
AD
411 /* This convenience routine combines hv_iternext,
412 hv_iterkey, and hv_iterval. The key and retlen
413 arguments are return values for the key and its
414 length. The value is returned in the SV* argument */
a0d0e21e
LW
415
416If you know the name of a hash variable, you can get a pointer to its HV
417by using the following:
418
4929bf7b 419 HV* get_hv("package::varname", FALSE);
a0d0e21e
LW
420
421This returns NULL if the variable does not exist.
422
8ebc5c01 423The hash algorithm is defined in the C<PERL_HASH(hash, key, klen)> macro:
a0d0e21e 424
a0d0e21e 425 hash = 0;
ab192400
GS
426 while (klen--)
427 hash = (hash * 33) + *key++;
87275199 428 hash = hash + (hash >> 5); /* after 5.6 */
ab192400 429
87275199 430The last step was added in version 5.6 to improve distribution of
ab192400 431lower bits in the resulting hash value.
a0d0e21e 432
04343c6d
GS
433See L<Understanding the Magic of Tied Hashes and Arrays> for more
434information on how to use the hash access functions on tied hashes.
435
1e422769
PP
436=head2 Hash API Extensions
437
438Beginning with version 5.004, the following functions are also supported:
439
440 HE* hv_fetch_ent (HV* tb, SV* key, I32 lval, U32 hash);
441 HE* hv_store_ent (HV* tb, SV* key, SV* val, U32 hash);
c47ff5f1 442
1e422769
PP
443 bool hv_exists_ent (HV* tb, SV* key, U32 hash);
444 SV* hv_delete_ent (HV* tb, SV* key, I32 flags, U32 hash);
c47ff5f1 445
1e422769
PP
446 SV* hv_iterkeysv (HE* entry);
447
448Note that these functions take C<SV*> keys, which simplifies writing
449of extension code that deals with hash structures. These functions
450also allow passing of C<SV*> keys to C<tie> functions without forcing
451you to stringify the keys (unlike the previous set of functions).
452
453They also return and accept whole hash entries (C<HE*>), making their
454use more efficient (since the hash number for a particular string
4a4eefd0
GS
455doesn't have to be recomputed every time). See L<perlapi> for detailed
456descriptions.
1e422769
PP
457
458The following macros must always be used to access the contents of hash
459entries. Note that the arguments to these macros must be simple
460variables, since they may get evaluated more than once. See
4a4eefd0 461L<perlapi> for detailed descriptions of these macros.
1e422769
PP
462
463 HePV(HE* he, STRLEN len)
464 HeVAL(HE* he)
465 HeHASH(HE* he)
466 HeSVKEY(HE* he)
467 HeSVKEY_force(HE* he)
468 HeSVKEY_set(HE* he, SV* sv)
469
470These two lower level macros are defined, but must only be used when
471dealing with keys that are not C<SV*>s:
472
473 HeKEY(HE* he)
474 HeKLEN(HE* he)
475
04343c6d
GS
476Note that both C<hv_store> and C<hv_store_ent> do not increment the
477reference count of the stored C<val>, which is the caller's responsibility.
478If these functions return a NULL value, the caller will usually have to
479decrement the reference count of C<val> to avoid a memory leak.
1e422769 480
a0d0e21e
LW
481=head2 References
482
d1b91892
AD
483References are a special type of scalar that point to other data types
484(including references).
a0d0e21e 485
07fa94a1 486To create a reference, use either of the following functions:
a0d0e21e 487
5f05dabc
PP
488 SV* newRV_inc((SV*) thing);
489 SV* newRV_noinc((SV*) thing);
a0d0e21e 490
5f05dabc 491The C<thing> argument can be any of an C<SV*>, C<AV*>, or C<HV*>. The
07fa94a1
JO
492functions are identical except that C<newRV_inc> increments the reference
493count of the C<thing>, while C<newRV_noinc> does not. For historical
494reasons, C<newRV> is a synonym for C<newRV_inc>.
495
496Once you have a reference, you can use the following macro to dereference
497the reference:
a0d0e21e
LW
498
499 SvRV(SV*)
500
501then call the appropriate routines, casting the returned C<SV*> to either an
d1b91892 502C<AV*> or C<HV*>, if required.
a0d0e21e 503
d1b91892 504To determine if an SV is a reference, you can use the following macro:
a0d0e21e
LW
505
506 SvROK(SV*)
507
07fa94a1
JO
508To discover what type of value the reference refers to, use the following
509macro and then check the return value.
d1b91892
AD
510
511 SvTYPE(SvRV(SV*))
512
513The most useful types that will be returned are:
514
515 SVt_IV Scalar
516 SVt_NV Scalar
517 SVt_PV Scalar
5f05dabc 518 SVt_RV Scalar
d1b91892
AD
519 SVt_PVAV Array
520 SVt_PVHV Hash
521 SVt_PVCV Code
5f05dabc
PP
522 SVt_PVGV Glob (possible a file handle)
523 SVt_PVMG Blessed or Magical Scalar
524
525 See the sv.h header file for more details.
d1b91892 526
cb1a09d0
AD
527=head2 Blessed References and Class Objects
528
529References are also used to support object-oriented programming. In the
530OO lexicon, an object is simply a reference that has been blessed into a
531package (or class). Once blessed, the programmer may now use the reference
532to access the various methods in the class.
533
534A reference can be blessed into a package with the following function:
535
536 SV* sv_bless(SV* sv, HV* stash);
537
538The C<sv> argument must be a reference. The C<stash> argument specifies
3fe9a6f1 539which class the reference will belong to. See
2ae324a7 540L<Stashes and Globs> for information on converting class names into stashes.
cb1a09d0
AD
541
542/* Still under construction */
543
544Upgrades rv to reference if not already one. Creates new SV for rv to
8ebc5c01
PP
545point to. If C<classname> is non-null, the SV is blessed into the specified
546class. SV is returned.
cb1a09d0 547
08105a92 548 SV* newSVrv(SV* rv, const char* classname);
cb1a09d0 549
e1c57cef 550Copies integer, unsigned integer or double into an SV whose reference is C<rv>. SV is blessed
8ebc5c01 551if C<classname> is non-null.
cb1a09d0 552
08105a92 553 SV* sv_setref_iv(SV* rv, const char* classname, IV iv);
e1c57cef 554 SV* sv_setref_uv(SV* rv, const char* classname, UV uv);
08105a92 555 SV* sv_setref_nv(SV* rv, const char* classname, NV iv);
cb1a09d0 556
5f05dabc 557Copies the pointer value (I<the address, not the string!>) into an SV whose
8ebc5c01 558reference is rv. SV is blessed if C<classname> is non-null.
cb1a09d0 559
08105a92 560 SV* sv_setref_pv(SV* rv, const char* classname, PV iv);
cb1a09d0 561
8ebc5c01
PP
562Copies string into an SV whose reference is C<rv>. Set length to 0 to let
563Perl calculate the string length. SV is blessed if C<classname> is non-null.
cb1a09d0 564
e65f3abd 565 SV* sv_setref_pvn(SV* rv, const char* classname, PV iv, STRLEN length);
cb1a09d0 566
9abd00ed
GS
567Tests whether the SV is blessed into the specified class. It does not
568check inheritance relationships.
569
08105a92 570 int sv_isa(SV* sv, const char* name);
9abd00ed
GS
571
572Tests whether the SV is a reference to a blessed object.
573
574 int sv_isobject(SV* sv);
575
576Tests whether the SV is derived from the specified class. SV can be either
577a reference to a blessed object or a string containing a class name. This
578is the function implementing the C<UNIVERSAL::isa> functionality.
579
08105a92 580 bool sv_derived_from(SV* sv, const char* name);
9abd00ed 581
00aadd71 582To check if you've got an object derived from a specific class you have
9abd00ed
GS
583to write:
584
585 if (sv_isobject(sv) && sv_derived_from(sv, class)) { ... }
cb1a09d0 586
5f05dabc 587=head2 Creating New Variables
cb1a09d0 588
5f05dabc
PP
589To create a new Perl variable with an undef value which can be accessed from
590your Perl script, use the following routines, depending on the variable type.
cb1a09d0 591
4929bf7b
GS
592 SV* get_sv("package::varname", TRUE);
593 AV* get_av("package::varname", TRUE);
594 HV* get_hv("package::varname", TRUE);
cb1a09d0
AD
595
596Notice the use of TRUE as the second parameter. The new variable can now
597be set, using the routines appropriate to the data type.
598
5f05dabc
PP
599There are additional macros whose values may be bitwise OR'ed with the
600C<TRUE> argument to enable certain extra features. Those bits are:
cb1a09d0 601
5f05dabc 602 GV_ADDMULTI Marks the variable as multiply defined, thus preventing the
54310121 603 "Name <varname> used only once: possible typo" warning.
07fa94a1
JO
604 GV_ADDWARN Issues the warning "Had to create <varname> unexpectedly" if
605 the variable did not exist before the function was called.
cb1a09d0 606
07fa94a1
JO
607If you do not specify a package name, the variable is created in the current
608package.
cb1a09d0 609
5f05dabc 610=head2 Reference Counts and Mortality
a0d0e21e 611
54310121
PP
612Perl uses an reference count-driven garbage collection mechanism. SVs,
613AVs, or HVs (xV for short in the following) start their life with a
55497cff 614reference count of 1. If the reference count of an xV ever drops to 0,
07fa94a1 615then it will be destroyed and its memory made available for reuse.
55497cff
PP
616
617This normally doesn't happen at the Perl level unless a variable is
5f05dabc
PP
618undef'ed or the last variable holding a reference to it is changed or
619overwritten. At the internal level, however, reference counts can be
55497cff
PP
620manipulated with the following macros:
621
622 int SvREFCNT(SV* sv);
5f05dabc 623 SV* SvREFCNT_inc(SV* sv);
55497cff
PP
624 void SvREFCNT_dec(SV* sv);
625
626However, there is one other function which manipulates the reference
07fa94a1
JO
627count of its argument. The C<newRV_inc> function, you will recall,
628creates a reference to the specified argument. As a side effect,
629it increments the argument's reference count. If this is not what
630you want, use C<newRV_noinc> instead.
631
632For example, imagine you want to return a reference from an XSUB function.
633Inside the XSUB routine, you create an SV which initially has a reference
634count of one. Then you call C<newRV_inc>, passing it the just-created SV.
5f05dabc
PP
635This returns the reference as a new SV, but the reference count of the
636SV you passed to C<newRV_inc> has been incremented to two. Now you
07fa94a1
JO
637return the reference from the XSUB routine and forget about the SV.
638But Perl hasn't! Whenever the returned reference is destroyed, the
639reference count of the original SV is decreased to one and nothing happens.
640The SV will hang around without any way to access it until Perl itself
641terminates. This is a memory leak.
5f05dabc
PP
642
643The correct procedure, then, is to use C<newRV_noinc> instead of
faed5253
JO
644C<newRV_inc>. Then, if and when the last reference is destroyed,
645the reference count of the SV will go to zero and it will be destroyed,
07fa94a1 646stopping any memory leak.
55497cff 647
5f05dabc 648There are some convenience functions available that can help with the
54310121 649destruction of xVs. These functions introduce the concept of "mortality".
07fa94a1
JO
650An xV that is mortal has had its reference count marked to be decremented,
651but not actually decremented, until "a short time later". Generally the
652term "short time later" means a single Perl statement, such as a call to
54310121 653an XSUB function. The actual determinant for when mortal xVs have their
07fa94a1
JO
654reference count decremented depends on two macros, SAVETMPS and FREETMPS.
655See L<perlcall> and L<perlxs> for more details on these macros.
55497cff
PP
656
657"Mortalization" then is at its simplest a deferred C<SvREFCNT_dec>.
658However, if you mortalize a variable twice, the reference count will
659later be decremented twice.
660
00aadd71
NIS
661"Mortal" SVs are mainly used for SVs that are placed on perl's stack.
662For example an SV which is created just to pass a number to a called sub
663is made mortal to have it cleaned up automatically when stack is popped.
664Similarly results returned by XSUBs (which go in the stack) are often
665made mortal.
a0d0e21e
LW
666
667To create a mortal variable, use the functions:
668
669 SV* sv_newmortal()
670 SV* sv_2mortal(SV*)
671 SV* sv_mortalcopy(SV*)
672
00aadd71 673The first call creates a mortal SV (with no value), the second converts an existing
5f05dabc
PP
674SV to a mortal SV (and thus defers a call to C<SvREFCNT_dec>), and the
675third creates a mortal copy of an existing SV.
00aadd71
NIS
676Because C<sv_newmortal> gives the new SV no value,it must normally be given one
677via C<sv_setpv>, C<sv_setiv> etc. :
678
679 SV *tmp = sv_newmortal();
680 sv_setiv(tmp, an_integer);
681
682As that is multiple C statements it is quite common so see this idiom instead:
683
684 SV *tmp = sv_2mortal(newSViv(an_integer));
685
686
687You should be careful about creating mortal variables. Strange things
688can happen if you make the same value mortal within multiple contexts,
689or if you make a variable mortal multiple times. Thinking of "Mortalization"
690as deferred C<SvREFCNT_dec> should help to minimize such problems.
691For example if you are passing an SV which you I<know> has high enough REFCNT
692to survive its use on the stack you need not do any mortalization.
693If you are not sure then doing an C<SvREFCNT_inc> and C<sv_2mortal>, or
694making a C<sv_mortalcopy> is safer.
a0d0e21e 695
54310121 696The mortal routines are not just for SVs -- AVs and HVs can be
faed5253 697made mortal by passing their address (type-casted to C<SV*>) to the
07fa94a1 698C<sv_2mortal> or C<sv_mortalcopy> routines.
a0d0e21e 699
5f05dabc 700=head2 Stashes and Globs
a0d0e21e 701
aa689395
PP
702A "stash" is a hash that contains all of the different objects that
703are contained within a package. Each key of the stash is a symbol
704name (shared by all the different types of objects that have the same
705name), and each value in the hash table is a GV (Glob Value). This GV
706in turn contains references to the various objects of that name,
707including (but not limited to) the following:
cb1a09d0 708
a0d0e21e
LW
709 Scalar Value
710 Array Value
711 Hash Value
a3cb178b 712 I/O Handle
a0d0e21e
LW
713 Format
714 Subroutine
715
9cde0e7f 716There is a single stash called "PL_defstash" that holds the items that exist
5f05dabc
PP
717in the "main" package. To get at the items in other packages, append the
718string "::" to the package name. The items in the "Foo" package are in
9cde0e7f 719the stash "Foo::" in PL_defstash. The items in the "Bar::Baz" package are
5f05dabc 720in the stash "Baz::" in "Bar::"'s stash.
a0d0e21e 721
d1b91892 722To get the stash pointer for a particular package, use the function:
a0d0e21e 723
08105a92 724 HV* gv_stashpv(const char* name, I32 create)
a0d0e21e
LW
725 HV* gv_stashsv(SV*, I32 create)
726
727The first function takes a literal string, the second uses the string stored
d1b91892 728in the SV. Remember that a stash is just a hash table, so you get back an
cb1a09d0 729C<HV*>. The C<create> flag will create a new package if it is set.
a0d0e21e
LW
730
731The name that C<gv_stash*v> wants is the name of the package whose symbol table
732you want. The default package is called C<main>. If you have multiply nested
d1b91892
AD
733packages, pass their names to C<gv_stash*v>, separated by C<::> as in the Perl
734language itself.
a0d0e21e
LW
735
736Alternately, if you have an SV that is a blessed reference, you can find
737out the stash pointer by using:
738
739 HV* SvSTASH(SvRV(SV*));
740
741then use the following to get the package name itself:
742
743 char* HvNAME(HV* stash);
744
5f05dabc
PP
745If you need to bless or re-bless an object you can use the following
746function:
a0d0e21e
LW
747
748 SV* sv_bless(SV*, HV* stash)
749
750where the first argument, an C<SV*>, must be a reference, and the second
751argument is a stash. The returned C<SV*> can now be used in the same way
752as any other SV.
753
d1b91892
AD
754For more information on references and blessings, consult L<perlref>.
755
54310121 756=head2 Double-Typed SVs
0a753a76
PP
757
758Scalar variables normally contain only one type of value, an integer,
759double, pointer, or reference. Perl will automatically convert the
760actual scalar data from the stored type into the requested type.
761
762Some scalar variables contain more than one type of scalar data. For
763example, the variable C<$!> contains either the numeric value of C<errno>
764or its string equivalent from either C<strerror> or C<sys_errlist[]>.
765
766To force multiple data values into an SV, you must do two things: use the
767C<sv_set*v> routines to add the additional scalar type, then set a flag
768so that Perl will believe it contains more than one type of data. The
769four macros to set the flags are:
770
771 SvIOK_on
772 SvNOK_on
773 SvPOK_on
774 SvROK_on
775
776The particular macro you must use depends on which C<sv_set*v> routine
777you called first. This is because every C<sv_set*v> routine turns on
778only the bit for the particular type of data being set, and turns off
779all the rest.
780
781For example, to create a new Perl variable called "dberror" that contains
782both the numeric and descriptive string error values, you could use the
783following code:
784
785 extern int dberror;
786 extern char *dberror_list;
787
4929bf7b 788 SV* sv = get_sv("dberror", TRUE);
0a753a76
PP
789 sv_setiv(sv, (IV) dberror);
790 sv_setpv(sv, dberror_list[dberror]);
791 SvIOK_on(sv);
792
793If the order of C<sv_setiv> and C<sv_setpv> had been reversed, then the
794macro C<SvPOK_on> would need to be called instead of C<SvIOK_on>.
795
796=head2 Magic Variables
a0d0e21e 797
d1b91892
AD
798[This section still under construction. Ignore everything here. Post no
799bills. Everything not permitted is forbidden.]
800
d1b91892
AD
801Any SV may be magical, that is, it has special features that a normal
802SV does not have. These features are stored in the SV structure in a
5f05dabc 803linked list of C<struct magic>'s, typedef'ed to C<MAGIC>.
d1b91892
AD
804
805 struct magic {
806 MAGIC* mg_moremagic;
807 MGVTBL* mg_virtual;
808 U16 mg_private;
809 char mg_type;
810 U8 mg_flags;
811 SV* mg_obj;
812 char* mg_ptr;
813 I32 mg_len;
814 };
815
816Note this is current as of patchlevel 0, and could change at any time.
817
818=head2 Assigning Magic
819
820Perl adds magic to an SV using the sv_magic function:
821
08105a92 822 void sv_magic(SV* sv, SV* obj, int how, const char* name, I32 namlen);
d1b91892
AD
823
824The C<sv> argument is a pointer to the SV that is to acquire a new magical
825feature.
826
827If C<sv> is not already magical, Perl uses the C<SvUPGRADE> macro to
645c22ef
DM
828convert C<sv> to type C<SVt_PVMG>. Perl then continues by adding new magic
829to the beginning of the linked list of magical features. Any prior entry
830of the same type of magic is deleted. Note that this can be overridden,
831and multiple instances of the same type of magic can be associated with an
832SV.
d1b91892 833
54310121
PP
834The C<name> and C<namlen> arguments are used to associate a string with
835the magic, typically the name of a variable. C<namlen> is stored in the
836C<mg_len> field and if C<name> is non-null and C<namlen> >= 0 a malloc'd
d1b91892
AD
837copy of the name is stored in C<mg_ptr> field.
838
839The sv_magic function uses C<how> to determine which, if any, predefined
840"Magic Virtual Table" should be assigned to the C<mg_virtual> field.
cb1a09d0 841See the "Magic Virtual Table" section below. The C<how> argument is also
14befaf4
DM
842stored in the C<mg_type> field. The value of C<how> should be chosen
843from the set of macros C<PERL_MAGIC_foo> found perl.h. Note that before
645c22ef 844these macros were added, Perl internals used to directly use character
14befaf4
DM
845literals, so you may occasionally come across old code or documentation
846referrring to 'U' magic rather than C<PERL_MAGIC_uvar> for example.
d1b91892
AD
847
848The C<obj> argument is stored in the C<mg_obj> field of the C<MAGIC>
849structure. If it is not the same as the C<sv> argument, the reference
850count of the C<obj> object is incremented. If it is the same, or if
645c22ef 851the C<how> argument is C<PERL_MAGIC_arylen>, or if it is a NULL pointer,
14befaf4 852then C<obj> is merely stored, without the reference count being incremented.
d1b91892 853
cb1a09d0
AD
854There is also a function to add magic to an C<HV>:
855
856 void hv_magic(HV *hv, GV *gv, int how);
857
858This simply calls C<sv_magic> and coerces the C<gv> argument into an C<SV>.
859
860To remove the magic from an SV, call the function sv_unmagic:
861
862 void sv_unmagic(SV *sv, int type);
863
864The C<type> argument should be equal to the C<how> value when the C<SV>
865was initially made magical.
866
d1b91892
AD
867=head2 Magic Virtual Tables
868
869The C<mg_virtual> field in the C<MAGIC> structure is a pointer to a
870C<MGVTBL>, which is a structure of function pointers and stands for
871"Magic Virtual Table" to handle the various operations that might be
872applied to that variable.
873
874The C<MGVTBL> has five pointers to the following routine types:
875
876 int (*svt_get)(SV* sv, MAGIC* mg);
877 int (*svt_set)(SV* sv, MAGIC* mg);
878 U32 (*svt_len)(SV* sv, MAGIC* mg);
879 int (*svt_clear)(SV* sv, MAGIC* mg);
880 int (*svt_free)(SV* sv, MAGIC* mg);
881
882This MGVTBL structure is set at compile-time in C<perl.h> and there are
883currently 19 types (or 21 with overloading turned on). These different
884structures contain pointers to various routines that perform additional
885actions depending on which function is being called.
886
887 Function pointer Action taken
888 ---------------- ------------
889 svt_get Do something after the value of the SV is retrieved.
890 svt_set Do something after the SV is assigned a value.
891 svt_len Report on the SV's length.
892 svt_clear Clear something the SV represents.
893 svt_free Free any extra storage associated with the SV.
894
895For instance, the MGVTBL structure called C<vtbl_sv> (which corresponds
14befaf4 896to an C<mg_type> of C<PERL_MAGIC_sv>) contains:
d1b91892
AD
897
898 { magic_get, magic_set, magic_len, 0, 0 }
899
14befaf4
DM
900Thus, when an SV is determined to be magical and of type C<PERL_MAGIC_sv>,
901if a get operation is being performed, the routine C<magic_get> is
902called. All the various routines for the various magical types begin
903with C<magic_>. NOTE: the magic routines are not considered part of
904the Perl API, and may not be exported by the Perl library.
d1b91892
AD
905
906The current kinds of Magic Virtual Tables are:
907
14befaf4
DM
908 mg_type
909 (old-style char and macro) MGVTBL Type of magic
910 -------------------------- ------ ----------------------------
911 \0 PERL_MAGIC_sv vtbl_sv Special scalar variable
912 A PERL_MAGIC_overload vtbl_amagic %OVERLOAD hash
913 a PERL_MAGIC_overload_elem vtbl_amagicelem %OVERLOAD hash element
914 c PERL_MAGIC_overload_table (none) Holds overload table (AMT)
915 on stash
916 B PERL_MAGIC_bm vtbl_bm Boyer-Moore (fast string search)
917 D PERL_MAGIC_regdata vtbl_regdata Regex match position data
918 (@+ and @- vars)
919 d PERL_MAGIC_regdatum vtbl_regdatum Regex match position data
920 element
921 E PERL_MAGIC_env vtbl_env %ENV hash
922 e PERL_MAGIC_envelem vtbl_envelem %ENV hash element
923 f PERL_MAGIC_fm vtbl_fm Formline ('compiled' format)
924 g PERL_MAGIC_regex_global vtbl_mglob m//g target / study()ed string
925 I PERL_MAGIC_isa vtbl_isa @ISA array
926 i PERL_MAGIC_isaelem vtbl_isaelem @ISA array element
927 k PERL_MAGIC_nkeys vtbl_nkeys scalar(keys()) lvalue
928 L PERL_MAGIC_dbfile (none) Debugger %_<filename
929 l PERL_MAGIC_dbline vtbl_dbline Debugger %_<filename element
930 m PERL_MAGIC_mutex vtbl_mutex ???
645c22ef 931 o PERL_MAGIC_collxfrm vtbl_collxfrm Locale collate transformation
14befaf4
DM
932 P PERL_MAGIC_tied vtbl_pack Tied array or hash
933 p PERL_MAGIC_tiedelem vtbl_packelem Tied array or hash element
934 q PERL_MAGIC_tiedscalar vtbl_packelem Tied scalar or handle
935 r PERL_MAGIC_qr vtbl_qr precompiled qr// regex
936 S PERL_MAGIC_sig vtbl_sig %SIG hash
937 s PERL_MAGIC_sigelem vtbl_sigelem %SIG hash element
938 t PERL_MAGIC_taint vtbl_taint Taintedness
939 U PERL_MAGIC_uvar vtbl_uvar Available for use by extensions
940 v PERL_MAGIC_vec vtbl_vec vec() lvalue
941 x PERL_MAGIC_substr vtbl_substr substr() lvalue
942 y PERL_MAGIC_defelem vtbl_defelem Shadow "foreach" iterator
943 variable / smart parameter
944 vivification
945 * PERL_MAGIC_glob vtbl_glob GV (typeglob)
946 # PERL_MAGIC_arylen vtbl_arylen Array length ($#ary)
947 . PERL_MAGIC_pos vtbl_pos pos() lvalue
948 < PERL_MAGIC_backref vtbl_backref ???
949 ~ PERL_MAGIC_ext (none) Available for use by extensions
d1b91892 950
68dc0745
PP
951When an uppercase and lowercase letter both exist in the table, then the
952uppercase letter is used to represent some kind of composite type (a list
953or a hash), and the lowercase letter is used to represent an element of
14befaf4
DM
954that composite type. Some internals code makes use of this case
955relationship.
956
957The C<PERL_MAGIC_ext> and C<PERL_MAGIC_uvar> magic types are defined
958specifically for use by extensions and will not be used by perl itself.
959Extensions can use C<PERL_MAGIC_ext> magic to 'attach' private information
960to variables (typically objects). This is especially useful because
961there is no way for normal perl code to corrupt this private information
962(unlike using extra elements of a hash object).
963
964Similarly, C<PERL_MAGIC_uvar> magic can be used much like tie() to call a
965C function any time a scalar's value is used or changed. The C<MAGIC>'s
bdbeb323
SM
966C<mg_ptr> field points to a C<ufuncs> structure:
967
968 struct ufuncs {
a9402793
AB
969 I32 (*uf_val)(pTHX_ IV, SV*);
970 I32 (*uf_set)(pTHX_ IV, SV*);
bdbeb323
SM
971 IV uf_index;
972 };
973
974When the SV is read from or written to, the C<uf_val> or C<uf_set>
14befaf4
DM
975function will be called with C<uf_index> as the first arg and a pointer to
976the SV as the second. A simple example of how to add C<PERL_MAGIC_uvar>
1526ead6
AB
977magic is shown below. Note that the ufuncs structure is copied by
978sv_magic, so you can safely allocate it on the stack.
979
980 void
981 Umagic(sv)
982 SV *sv;
983 PREINIT:
984 struct ufuncs uf;
985 CODE:
986 uf.uf_val = &my_get_fn;
987 uf.uf_set = &my_set_fn;
988 uf.uf_index = 0;
14befaf4 989 sv_magic(sv, 0, PERL_MAGIC_uvar, (char*)&uf, sizeof(uf));
5f05dabc 990
14befaf4
DM
991Note that because multiple extensions may be using C<PERL_MAGIC_ext>
992or C<PERL_MAGIC_uvar> magic, it is important for extensions to take
993extra care to avoid conflict. Typically only using the magic on
994objects blessed into the same class as the extension is sufficient.
995For C<PERL_MAGIC_ext> magic, it may also be appropriate to add an I32
996'signature' at the top of the private data area and check that.
5f05dabc 997
ef50df4b
GS
998Also note that the C<sv_set*()> and C<sv_cat*()> functions described
999earlier do B<not> invoke 'set' magic on their targets. This must
1000be done by the user either by calling the C<SvSETMAGIC()> macro after
1001calling these functions, or by using one of the C<sv_set*_mg()> or
1002C<sv_cat*_mg()> functions. Similarly, generic C code must call the
1003C<SvGETMAGIC()> macro to invoke any 'get' magic if they use an SV
1004obtained from external sources in functions that don't handle magic.
4a4eefd0 1005See L<perlapi> for a description of these functions.
189b2af5
GS
1006For example, calls to the C<sv_cat*()> functions typically need to be
1007followed by C<SvSETMAGIC()>, but they don't need a prior C<SvGETMAGIC()>
1008since their implementation handles 'get' magic.
1009
d1b91892
AD
1010=head2 Finding Magic
1011
1012 MAGIC* mg_find(SV*, int type); /* Finds the magic pointer of that type */
1013
1014This routine returns a pointer to the C<MAGIC> structure stored in the SV.
1015If the SV does not have that magical feature, C<NULL> is returned. Also,
54310121 1016if the SV is not of type SVt_PVMG, Perl may core dump.
d1b91892 1017
08105a92 1018 int mg_copy(SV* sv, SV* nsv, const char* key, STRLEN klen);
d1b91892
AD
1019
1020This routine checks to see what types of magic C<sv> has. If the mg_type
68dc0745
PP
1021field is an uppercase letter, then the mg_obj is copied to C<nsv>, but
1022the mg_type field is changed to be the lowercase letter.
a0d0e21e 1023
04343c6d
GS
1024=head2 Understanding the Magic of Tied Hashes and Arrays
1025
14befaf4
DM
1026Tied hashes and arrays are magical beasts of the C<PERL_MAGIC_tied>
1027magic type.
9edb2b46
GS
1028
1029WARNING: As of the 5.004 release, proper usage of the array and hash
1030access functions requires understanding a few caveats. Some
1031of these caveats are actually considered bugs in the API, to be fixed
1032in later releases, and are bracketed with [MAYCHANGE] below. If
1033you find yourself actually applying such information in this section, be
1034aware that the behavior may change in the future, umm, without warning.
04343c6d 1035
1526ead6
AB
1036The perl tie function associates a variable with an object that implements
1037the various GET, SET etc methods. To perform the equivalent of the perl
1038tie function from an XSUB, you must mimic this behaviour. The code below
1039carries out the necessary steps - firstly it creates a new hash, and then
1040creates a second hash which it blesses into the class which will implement
1041the tie methods. Lastly it ties the two hashes together, and returns a
1042reference to the new tied hash. Note that the code below does NOT call the
1043TIEHASH method in the MyTie class -
1044see L<Calling Perl Routines from within C Programs> for details on how
1045to do this.
1046
1047 SV*
1048 mytie()
1049 PREINIT:
1050 HV *hash;
1051 HV *stash;
1052 SV *tie;
1053 CODE:
1054 hash = newHV();
1055 tie = newRV_noinc((SV*)newHV());
1056 stash = gv_stashpv("MyTie", TRUE);
1057 sv_bless(tie, stash);
899e16d0 1058 hv_magic(hash, (GV*)tie, PERL_MAGIC_tied);
1526ead6
AB
1059 RETVAL = newRV_noinc(hash);
1060 OUTPUT:
1061 RETVAL
1062
04343c6d
GS
1063The C<av_store> function, when given a tied array argument, merely
1064copies the magic of the array onto the value to be "stored", using
1065C<mg_copy>. It may also return NULL, indicating that the value did not
9edb2b46
GS
1066actually need to be stored in the array. [MAYCHANGE] After a call to
1067C<av_store> on a tied array, the caller will usually need to call
1068C<mg_set(val)> to actually invoke the perl level "STORE" method on the
1069TIEARRAY object. If C<av_store> did return NULL, a call to
1070C<SvREFCNT_dec(val)> will also be usually necessary to avoid a memory
1071leak. [/MAYCHANGE]
04343c6d
GS
1072
1073The previous paragraph is applicable verbatim to tied hash access using the
1074C<hv_store> and C<hv_store_ent> functions as well.
1075
1076C<av_fetch> and the corresponding hash functions C<hv_fetch> and
1077C<hv_fetch_ent> actually return an undefined mortal value whose magic
1078has been initialized using C<mg_copy>. Note the value so returned does not
9edb2b46
GS
1079need to be deallocated, as it is already mortal. [MAYCHANGE] But you will
1080need to call C<mg_get()> on the returned value in order to actually invoke
1081the perl level "FETCH" method on the underlying TIE object. Similarly,
04343c6d
GS
1082you may also call C<mg_set()> on the return value after possibly assigning
1083a suitable value to it using C<sv_setsv>, which will invoke the "STORE"
9edb2b46 1084method on the TIE object. [/MAYCHANGE]
04343c6d 1085
9edb2b46 1086[MAYCHANGE]
04343c6d
GS
1087In other words, the array or hash fetch/store functions don't really
1088fetch and store actual values in the case of tied arrays and hashes. They
1089merely call C<mg_copy> to attach magic to the values that were meant to be
1090"stored" or "fetched". Later calls to C<mg_get> and C<mg_set> actually
1091do the job of invoking the TIE methods on the underlying objects. Thus
9edb2b46 1092the magic mechanism currently implements a kind of lazy access to arrays
04343c6d
GS
1093and hashes.
1094
1095Currently (as of perl version 5.004), use of the hash and array access
1096functions requires the user to be aware of whether they are operating on
9edb2b46
GS
1097"normal" hashes and arrays, or on their tied variants. The API may be
1098changed to provide more transparent access to both tied and normal data
1099types in future versions.
1100[/MAYCHANGE]
04343c6d
GS
1101
1102You would do well to understand that the TIEARRAY and TIEHASH interfaces
1103are mere sugar to invoke some perl method calls while using the uniform hash
1104and array syntax. The use of this sugar imposes some overhead (typically
1105about two to four extra opcodes per FETCH/STORE operation, in addition to
1106the creation of all the mortal variables required to invoke the methods).
1107This overhead will be comparatively small if the TIE methods are themselves
1108substantial, but if they are only a few statements long, the overhead
1109will not be insignificant.
1110
d1c897a1
IZ
1111=head2 Localizing changes
1112
1113Perl has a very handy construction
1114
1115 {
1116 local $var = 2;
1117 ...
1118 }
1119
1120This construction is I<approximately> equivalent to
1121
1122 {
1123 my $oldvar = $var;
1124 $var = 2;
1125 ...
1126 $var = $oldvar;
1127 }
1128
1129The biggest difference is that the first construction would
1130reinstate the initial value of $var, irrespective of how control exits
1131the block: C<goto>, C<return>, C<die>/C<eval> etc. It is a little bit
1132more efficient as well.
1133
1134There is a way to achieve a similar task from C via Perl API: create a
1135I<pseudo-block>, and arrange for some changes to be automatically
1136undone at the end of it, either explicit, or via a non-local exit (via
1137die()). A I<block>-like construct is created by a pair of
b687b08b
TC
1138C<ENTER>/C<LEAVE> macros (see L<perlcall/"Returning a Scalar">).
1139Such a construct may be created specially for some important localized
1140task, or an existing one (like boundaries of enclosing Perl
1141subroutine/block, or an existing pair for freeing TMPs) may be
1142used. (In the second case the overhead of additional localization must
1143be almost negligible.) Note that any XSUB is automatically enclosed in
1144an C<ENTER>/C<LEAVE> pair.
d1c897a1
IZ
1145
1146Inside such a I<pseudo-block> the following service is available:
1147
13a2d996 1148=over 4
d1c897a1
IZ
1149
1150=item C<SAVEINT(int i)>
1151
1152=item C<SAVEIV(IV i)>
1153
1154=item C<SAVEI32(I32 i)>
1155
1156=item C<SAVELONG(long i)>
1157
1158These macros arrange things to restore the value of integer variable
1159C<i> at the end of enclosing I<pseudo-block>.
1160
1161=item C<SAVESPTR(s)>
1162
1163=item C<SAVEPPTR(p)>
1164
1165These macros arrange things to restore the value of pointers C<s> and
1166C<p>. C<s> must be a pointer of a type which survives conversion to
1167C<SV*> and back, C<p> should be able to survive conversion to C<char*>
1168and back.
1169
1170=item C<SAVEFREESV(SV *sv)>
1171
1172The refcount of C<sv> would be decremented at the end of
26d9b02f
JH
1173I<pseudo-block>. This is similar to C<sv_2mortal> in that it is also a
1174mechanism for doing a delayed C<SvREFCNT_dec>. However, while C<sv_2mortal>
1175extends the lifetime of C<sv> until the beginning of the next statement,
1176C<SAVEFREESV> extends it until the end of the enclosing scope. These
1177lifetimes can be wildly different.
1178
1179Also compare C<SAVEMORTALIZESV>.
1180
1181=item C<SAVEMORTALIZESV(SV *sv)>
1182
1183Just like C<SAVEFREESV>, but mortalizes C<sv> at the end of the current
1184scope instead of decrementing its reference count. This usually has the
1185effect of keeping C<sv> alive until the statement that called the currently
1186live scope has finished executing.
d1c897a1
IZ
1187
1188=item C<SAVEFREEOP(OP *op)>
1189
1190The C<OP *> is op_free()ed at the end of I<pseudo-block>.
1191
1192=item C<SAVEFREEPV(p)>
1193
1194The chunk of memory which is pointed to by C<p> is Safefree()ed at the
1195end of I<pseudo-block>.
1196
1197=item C<SAVECLEARSV(SV *sv)>
1198
1199Clears a slot in the current scratchpad which corresponds to C<sv> at
1200the end of I<pseudo-block>.
1201
1202=item C<SAVEDELETE(HV *hv, char *key, I32 length)>
1203
1204The key C<key> of C<hv> is deleted at the end of I<pseudo-block>. The
1205string pointed to by C<key> is Safefree()ed. If one has a I<key> in
1206short-lived storage, the corresponding string may be reallocated like
1207this:
1208
9cde0e7f 1209 SAVEDELETE(PL_defstash, savepv(tmpbuf), strlen(tmpbuf));
d1c897a1 1210
c76ac1ee 1211=item C<SAVEDESTRUCTOR(DESTRUCTORFUNC_NOCONTEXT_t f, void *p)>
d1c897a1
IZ
1212
1213At the end of I<pseudo-block> the function C<f> is called with the
c76ac1ee
GS
1214only argument C<p>.
1215
1216=item C<SAVEDESTRUCTOR_X(DESTRUCTORFUNC_t f, void *p)>
1217
1218At the end of I<pseudo-block> the function C<f> is called with the
1219implicit context argument (if any), and C<p>.
d1c897a1
IZ
1220
1221=item C<SAVESTACK_POS()>
1222
1223The current offset on the Perl internal stack (cf. C<SP>) is restored
1224at the end of I<pseudo-block>.
1225
1226=back
1227
1228The following API list contains functions, thus one needs to
1229provide pointers to the modifiable data explicitly (either C pointers,
00aadd71 1230or Perlish C<GV *>s). Where the above macros take C<int>, a similar
d1c897a1
IZ
1231function takes C<int *>.
1232
13a2d996 1233=over 4
d1c897a1
IZ
1234
1235=item C<SV* save_scalar(GV *gv)>
1236
1237Equivalent to Perl code C<local $gv>.
1238
1239=item C<AV* save_ary(GV *gv)>
1240
1241=item C<HV* save_hash(GV *gv)>
1242
1243Similar to C<save_scalar>, but localize C<@gv> and C<%gv>.
1244
1245=item C<void save_item(SV *item)>
1246
1247Duplicates the current value of C<SV>, on the exit from the current
1248C<ENTER>/C<LEAVE> I<pseudo-block> will restore the value of C<SV>
1249using the stored value.
1250
1251=item C<void save_list(SV **sarg, I32 maxsarg)>
1252
1253A variant of C<save_item> which takes multiple arguments via an array
1254C<sarg> of C<SV*> of length C<maxsarg>.
1255
1256=item C<SV* save_svref(SV **sptr)>
1257
1258Similar to C<save_scalar>, but will reinstate a C<SV *>.
1259
1260=item C<void save_aptr(AV **aptr)>
1261
1262=item C<void save_hptr(HV **hptr)>
1263
1264Similar to C<save_svref>, but localize C<AV *> and C<HV *>.
1265
1266=back
1267
1268The C<Alias> module implements localization of the basic types within the
1269I<caller's scope>. People who are interested in how to localize things in
1270the containing scope should take a look there too.
1271
0a753a76 1272=head1 Subroutines
a0d0e21e 1273
68dc0745 1274=head2 XSUBs and the Argument Stack
5f05dabc
PP
1275
1276The XSUB mechanism is a simple way for Perl programs to access C subroutines.
1277An XSUB routine will have a stack that contains the arguments from the Perl
1278program, and a way to map from the Perl data structures to a C equivalent.
1279
1280The stack arguments are accessible through the C<ST(n)> macro, which returns
1281the C<n>'th stack argument. Argument 0 is the first argument passed in the
1282Perl subroutine call. These arguments are C<SV*>, and can be used anywhere
1283an C<SV*> is used.
1284
1285Most of the time, output from the C routine can be handled through use of
1286the RETVAL and OUTPUT directives. However, there are some cases where the
1287argument stack is not already long enough to handle all the return values.
1288An example is the POSIX tzname() call, which takes no arguments, but returns
1289two, the local time zone's standard and summer time abbreviations.
1290
1291To handle this situation, the PPCODE directive is used and the stack is
1292extended using the macro:
1293
924508f0 1294 EXTEND(SP, num);
5f05dabc 1295
924508f0
GS
1296where C<SP> is the macro that represents the local copy of the stack pointer,
1297and C<num> is the number of elements the stack should be extended by.
5f05dabc 1298
00aadd71 1299Now that there is room on the stack, values can be pushed on it using C<PUSHs>
484ce0c5 1300macro. The values pushed will often need to be "mortal" (See L</Reference Counts and Mortality>).
5f05dabc 1301
00aadd71
NIS
1302 PUSHs(sv_2mortal(newSViv(an_integer)))
1303 PUSHs(sv_2mortal(newSVpv("Some String",0)))
1304 PUSHs(sv_2mortal(newSVnv(3.141592)))
5f05dabc
PP
1305
1306And now the Perl program calling C<tzname>, the two values will be assigned
1307as in:
1308
1309 ($standard_abbrev, $summer_abbrev) = POSIX::tzname;
1310
1311An alternate (and possibly simpler) method to pushing values on the stack is
00aadd71 1312to use the macro:
5f05dabc 1313
5f05dabc
PP
1314 XPUSHs(SV*)
1315
00aadd71 1316This macro automatically adjust the stack for you, if needed. Thus, you
5f05dabc 1317do not need to call C<EXTEND> to extend the stack.
00aadd71
NIS
1318
1319Despite their suggestions in earlier versions of this document the macros
1320C<PUSHi>, C<PUSHn> and C<PUSHp> are I<not> suited to XSUBs which return
1321multiple results, see L</Putting a C value on Perl stack>.
5f05dabc
PP
1322
1323For more information, consult L<perlxs> and L<perlxstut>.
1324
1325=head2 Calling Perl Routines from within C Programs
a0d0e21e
LW
1326
1327There are four routines that can be used to call a Perl subroutine from
1328within a C program. These four are:
1329
954c1994
GS
1330 I32 call_sv(SV*, I32);
1331 I32 call_pv(const char*, I32);
1332 I32 call_method(const char*, I32);
1333 I32 call_argv(const char*, I32, register char**);
a0d0e21e 1334
954c1994 1335The routine most often used is C<call_sv>. The C<SV*> argument
d1b91892
AD
1336contains either the name of the Perl subroutine to be called, or a
1337reference to the subroutine. The second argument consists of flags
1338that control the context in which the subroutine is called, whether
1339or not the subroutine is being passed arguments, how errors should be
1340trapped, and how to treat return values.
a0d0e21e
LW
1341
1342All four routines return the number of arguments that the subroutine returned
1343on the Perl stack.
1344
954c1994
GS
1345These routines used to be called C<perl_call_sv> etc., before Perl v5.6.0,
1346but those names are now deprecated; macros of the same name are provided for
1347compatibility.
1348
1349When using any of these routines (except C<call_argv>), the programmer
d1b91892
AD
1350must manipulate the Perl stack. These include the following macros and
1351functions:
a0d0e21e
LW
1352
1353 dSP
924508f0 1354 SP
a0d0e21e
LW
1355 PUSHMARK()
1356 PUTBACK
1357 SPAGAIN
1358 ENTER
1359 SAVETMPS
1360 FREETMPS
1361 LEAVE
1362 XPUSH*()
cb1a09d0 1363 POP*()
a0d0e21e 1364
5f05dabc
PP
1365For a detailed description of calling conventions from C to Perl,
1366consult L<perlcall>.
a0d0e21e 1367
5f05dabc 1368=head2 Memory Allocation
a0d0e21e 1369
86058a2d
GS
1370All memory meant to be used with the Perl API functions should be manipulated
1371using the macros described in this section. The macros provide the necessary
1372transparency between differences in the actual malloc implementation that is
1373used within perl.
1374
1375It is suggested that you enable the version of malloc that is distributed
5f05dabc 1376with Perl. It keeps pools of various sizes of unallocated memory in
07fa94a1
JO
1377order to satisfy allocation requests more quickly. However, on some
1378platforms, it may cause spurious malloc or free errors.
d1b91892
AD
1379
1380 New(x, pointer, number, type);
1381 Newc(x, pointer, number, type, cast);
1382 Newz(x, pointer, number, type);
1383
07fa94a1 1384These three macros are used to initially allocate memory.
5f05dabc
PP
1385
1386The first argument C<x> was a "magic cookie" that was used to keep track
1387of who called the macro, to help when debugging memory problems. However,
07fa94a1
JO
1388the current code makes no use of this feature (most Perl developers now
1389use run-time memory checkers), so this argument can be any number.
5f05dabc
PP
1390
1391The second argument C<pointer> should be the name of a variable that will
1392point to the newly allocated memory.
d1b91892 1393
d1b91892
AD
1394The third and fourth arguments C<number> and C<type> specify how many of
1395the specified type of data structure should be allocated. The argument
1396C<type> is passed to C<sizeof>. The final argument to C<Newc>, C<cast>,
1397should be used if the C<pointer> argument is different from the C<type>
1398argument.
1399
1400Unlike the C<New> and C<Newc> macros, the C<Newz> macro calls C<memzero>
1401to zero out all the newly allocated memory.
1402
1403 Renew(pointer, number, type);
1404 Renewc(pointer, number, type, cast);
1405 Safefree(pointer)
1406
1407These three macros are used to change a memory buffer size or to free a
1408piece of memory no longer needed. The arguments to C<Renew> and C<Renewc>
1409match those of C<New> and C<Newc> with the exception of not needing the
1410"magic cookie" argument.
1411
1412 Move(source, dest, number, type);
1413 Copy(source, dest, number, type);
1414 Zero(dest, number, type);
1415
1416These three macros are used to move, copy, or zero out previously allocated
1417memory. The C<source> and C<dest> arguments point to the source and
1418destination starting points. Perl will move, copy, or zero out C<number>
1419instances of the size of the C<type> data structure (using the C<sizeof>
1420function).
a0d0e21e 1421
5f05dabc 1422=head2 PerlIO
ce3d39e2 1423
5f05dabc
PP
1424The most recent development releases of Perl has been experimenting with
1425removing Perl's dependency on the "normal" standard I/O suite and allowing
1426other stdio implementations to be used. This involves creating a new
1427abstraction layer that then calls whichever implementation of stdio Perl
68dc0745 1428was compiled with. All XSUBs should now use the functions in the PerlIO
5f05dabc
PP
1429abstraction layer and not make any assumptions about what kind of stdio
1430is being used.
1431
1432For a complete description of the PerlIO abstraction, consult L<perlapio>.
1433
8ebc5c01 1434=head2 Putting a C value on Perl stack
ce3d39e2
IZ
1435
1436A lot of opcodes (this is an elementary operation in the internal perl
1437stack machine) put an SV* on the stack. However, as an optimization
1438the corresponding SV is (usually) not recreated each time. The opcodes
1439reuse specially assigned SVs (I<target>s) which are (as a corollary)
1440not constantly freed/created.
1441
0a753a76 1442Each of the targets is created only once (but see
ce3d39e2
IZ
1443L<Scratchpads and recursion> below), and when an opcode needs to put
1444an integer, a double, or a string on stack, it just sets the
1445corresponding parts of its I<target> and puts the I<target> on stack.
1446
1447The macro to put this target on stack is C<PUSHTARG>, and it is
1448directly used in some opcodes, as well as indirectly in zillions of
1449others, which use it via C<(X)PUSH[pni]>.
1450
1bd1c0d5
SC
1451Because the target is reused, you must be careful when pushing multiple
1452values on the stack. The following code will not do what you think:
1453
1454 XPUSHi(10);
1455 XPUSHi(20);
1456
1457This translates as "set C<TARG> to 10, push a pointer to C<TARG> onto
1458the stack; set C<TARG> to 20, push a pointer to C<TARG> onto the stack".
1459At the end of the operation, the stack does not contain the values 10
1460and 20, but actually contains two pointers to C<TARG>, which we have set
1461to 20. If you need to push multiple different values, use C<XPUSHs>,
1462which bypasses C<TARG>.
1463
1464On a related note, if you do use C<(X)PUSH[npi]>, then you're going to
1465need a C<dTARG> in your variable declarations so that the C<*PUSH*>
00aadd71 1466macros can make use of the local variable C<TARG>.
1bd1c0d5 1467
8ebc5c01 1468=head2 Scratchpads
ce3d39e2 1469
54310121 1470The question remains on when the SVs which are I<target>s for opcodes
5f05dabc
PP
1471are created. The answer is that they are created when the current unit --
1472a subroutine or a file (for opcodes for statements outside of
1473subroutines) -- is compiled. During this time a special anonymous Perl
ce3d39e2
IZ
1474array is created, which is called a scratchpad for the current
1475unit.
1476
54310121 1477A scratchpad keeps SVs which are lexicals for the current unit and are
ce3d39e2
IZ
1478targets for opcodes. One can deduce that an SV lives on a scratchpad
1479by looking on its flags: lexicals have C<SVs_PADMY> set, and
1480I<target>s have C<SVs_PADTMP> set.
1481
54310121
PP
1482The correspondence between OPs and I<target>s is not 1-to-1. Different
1483OPs in the compile tree of the unit can use the same target, if this
ce3d39e2
IZ
1484would not conflict with the expected life of the temporary.
1485
2ae324a7 1486=head2 Scratchpads and recursion
ce3d39e2
IZ
1487
1488In fact it is not 100% true that a compiled unit contains a pointer to
1489the scratchpad AV. In fact it contains a pointer to an AV of
1490(initially) one element, and this element is the scratchpad AV. Why do
1491we need an extra level of indirection?
1492
1493The answer is B<recursion>, and maybe (sometime soon) B<threads>. Both
1494these can create several execution pointers going into the same
1495subroutine. For the subroutine-child not write over the temporaries
1496for the subroutine-parent (lifespan of which covers the call to the
1497child), the parent and the child should have different
1498scratchpads. (I<And> the lexicals should be separate anyway!)
1499
5f05dabc
PP
1500So each subroutine is born with an array of scratchpads (of length 1).
1501On each entry to the subroutine it is checked that the current
ce3d39e2
IZ
1502depth of the recursion is not more than the length of this array, and
1503if it is, new scratchpad is created and pushed into the array.
1504
1505The I<target>s on this scratchpad are C<undef>s, but they are already
1506marked with correct flags.
1507
0a753a76
PP
1508=head1 Compiled code
1509
1510=head2 Code tree
1511
1512Here we describe the internal form your code is converted to by
1513Perl. Start with a simple example:
1514
1515 $a = $b + $c;
1516
1517This is converted to a tree similar to this one:
1518
1519 assign-to
1520 / \
1521 + $a
1522 / \
1523 $b $c
1524
7b8d334a 1525(but slightly more complicated). This tree reflects the way Perl
0a753a76
PP
1526parsed your code, but has nothing to do with the execution order.
1527There is an additional "thread" going through the nodes of the tree
1528which shows the order of execution of the nodes. In our simplified
1529example above it looks like:
1530
1531 $b ---> $c ---> + ---> $a ---> assign-to
1532
1533But with the actual compile tree for C<$a = $b + $c> it is different:
1534some nodes I<optimized away>. As a corollary, though the actual tree
1535contains more nodes than our simplified example, the execution order
1536is the same as in our example.
1537
1538=head2 Examining the tree
1539
1540If you have your perl compiled for debugging (usually done with C<-D
1541optimize=-g> on C<Configure> command line), you may examine the
1542compiled tree by specifying C<-Dx> on the Perl command line. The
1543output takes several lines per node, and for C<$b+$c> it looks like
1544this:
1545
1546 5 TYPE = add ===> 6
1547 TARG = 1
1548 FLAGS = (SCALAR,KIDS)
1549 {
1550 TYPE = null ===> (4)
1551 (was rv2sv)
1552 FLAGS = (SCALAR,KIDS)
1553 {
1554 3 TYPE = gvsv ===> 4
1555 FLAGS = (SCALAR)
1556 GV = main::b
1557 }
1558 }
1559 {
1560 TYPE = null ===> (5)
1561 (was rv2sv)
1562 FLAGS = (SCALAR,KIDS)
1563 {
1564 4 TYPE = gvsv ===> 5
1565 FLAGS = (SCALAR)
1566 GV = main::c
1567 }
1568 }
1569
1570This tree has 5 nodes (one per C<TYPE> specifier), only 3 of them are
1571not optimized away (one per number in the left column). The immediate
1572children of the given node correspond to C<{}> pairs on the same level
1573of indentation, thus this listing corresponds to the tree:
1574
1575 add
1576 / \
1577 null null
1578 | |
1579 gvsv gvsv
1580
1581The execution order is indicated by C<===E<gt>> marks, thus it is C<3
15824 5 6> (node C<6> is not included into above listing), i.e.,
1583C<gvsv gvsv add whatever>.
1584
9afa14e3
SC
1585Each of these nodes represents an op, a fundamental operation inside the
1586Perl core. The code which implements each operation can be found in the
1587F<pp*.c> files; the function which implements the op with type C<gvsv>
1588is C<pp_gvsv>, and so on. As the tree above shows, different ops have
1589different numbers of children: C<add> is a binary operator, as one would
1590expect, and so has two children. To accommodate the various different
1591numbers of children, there are various types of op data structure, and
1592they link together in different ways.
1593
1594The simplest type of op structure is C<OP>: this has no children. Unary
1595operators, C<UNOP>s, have one child, and this is pointed to by the
1596C<op_first> field. Binary operators (C<BINOP>s) have not only an
1597C<op_first> field but also an C<op_last> field. The most complex type of
1598op is a C<LISTOP>, which has any number of children. In this case, the
1599first child is pointed to by C<op_first> and the last child by
1600C<op_last>. The children in between can be found by iteratively
1601following the C<op_sibling> pointer from the first child to the last.
1602
1603There are also two other op types: a C<PMOP> holds a regular expression,
1604and has no children, and a C<LOOP> may or may not have children. If the
1605C<op_children> field is non-zero, it behaves like a C<LISTOP>. To
1606complicate matters, if a C<UNOP> is actually a C<null> op after
1607optimization (see L</Compile pass 2: context propagation>) it will still
1608have children in accordance with its former type.
1609
0a753a76
PP
1610=head2 Compile pass 1: check routines
1611
8870b5c7
GS
1612The tree is created by the compiler while I<yacc> code feeds it
1613the constructions it recognizes. Since I<yacc> works bottom-up, so does
0a753a76
PP
1614the first pass of perl compilation.
1615
1616What makes this pass interesting for perl developers is that some
1617optimization may be performed on this pass. This is optimization by
8870b5c7 1618so-called "check routines". The correspondence between node names
0a753a76
PP
1619and corresponding check routines is described in F<opcode.pl> (do not
1620forget to run C<make regen_headers> if you modify this file).
1621
1622A check routine is called when the node is fully constructed except
7b8d334a 1623for the execution-order thread. Since at this time there are no
0a753a76
PP
1624back-links to the currently constructed node, one can do most any
1625operation to the top-level node, including freeing it and/or creating
1626new nodes above/below it.
1627
1628The check routine returns the node which should be inserted into the
1629tree (if the top-level node was not modified, check routine returns
1630its argument).
1631
1632By convention, check routines have names C<ck_*>. They are usually
1633called from C<new*OP> subroutines (or C<convert>) (which in turn are
1634called from F<perly.y>).
1635
1636=head2 Compile pass 1a: constant folding
1637
1638Immediately after the check routine is called the returned node is
1639checked for being compile-time executable. If it is (the value is
1640judged to be constant) it is immediately executed, and a I<constant>
1641node with the "return value" of the corresponding subtree is
1642substituted instead. The subtree is deleted.
1643
1644If constant folding was not performed, the execution-order thread is
1645created.
1646
1647=head2 Compile pass 2: context propagation
1648
1649When a context for a part of compile tree is known, it is propagated
a3cb178b 1650down through the tree. At this time the context can have 5 values
0a753a76
PP
1651(instead of 2 for runtime context): void, boolean, scalar, list, and
1652lvalue. In contrast with the pass 1 this pass is processed from top
1653to bottom: a node's context determines the context for its children.
1654
1655Additional context-dependent optimizations are performed at this time.
1656Since at this moment the compile tree contains back-references (via
1657"thread" pointers), nodes cannot be free()d now. To allow
1658optimized-away nodes at this stage, such nodes are null()ified instead
1659of free()ing (i.e. their type is changed to OP_NULL).
1660
1661=head2 Compile pass 3: peephole optimization
1662
1663After the compile tree for a subroutine (or for an C<eval> or a file)
1664is created, an additional pass over the code is performed. This pass
1665is neither top-down or bottom-up, but in the execution order (with
7b8d334a 1666additional complications for conditionals). These optimizations are
0a753a76
PP
1667done in the subroutine peep(). Optimizations performed at this stage
1668are subject to the same restrictions as in the pass 2.
1669
1ba7f851
PJ
1670=head2 Pluggable runops
1671
1672The compile tree is executed in a runops function. There are two runops
1673functions in F<run.c>. C<Perl_runops_debug> is used with DEBUGGING and
1674C<Perl_runops_standard> is used otherwise. For fine control over the
1675execution of the compile tree it is possible to provide your own runops
1676function.
1677
1678It's probably best to copy one of the existing runops functions and
1679change it to suit your needs. Then, in the BOOT section of your XS
1680file, add the line:
1681
1682 PL_runops = my_runops;
1683
1684This function should be as efficient as possible to keep your programs
1685running as fast as possible.
1686
9afa14e3
SC
1687=head1 Examining internal data structures with the C<dump> functions
1688
1689To aid debugging, the source file F<dump.c> contains a number of
1690functions which produce formatted output of internal data structures.
1691
1692The most commonly used of these functions is C<Perl_sv_dump>; it's used
1693for dumping SVs, AVs, HVs, and CVs. The C<Devel::Peek> module calls
1694C<sv_dump> to produce debugging output from Perl-space, so users of that
00aadd71 1695module should already be familiar with its format.
9afa14e3
SC
1696
1697C<Perl_op_dump> can be used to dump an C<OP> structure or any of its
1698derivatives, and produces output similiar to C<perl -Dx>; in fact,
1699C<Perl_dump_eval> will dump the main root of the code being evaluated,
1700exactly like C<-Dx>.
1701
1702Other useful functions are C<Perl_dump_sub>, which turns a C<GV> into an
1703op tree, C<Perl_dump_packsubs> which calls C<Perl_dump_sub> on all the
1704subroutines in a package like so: (Thankfully, these are all xsubs, so
1705there is no op tree)
1706
1707 (gdb) print Perl_dump_packsubs(PL_defstash)
1708
1709 SUB attributes::bootstrap = (xsub 0x811fedc 0)
1710
1711 SUB UNIVERSAL::can = (xsub 0x811f50c 0)
1712
1713 SUB UNIVERSAL::isa = (xsub 0x811f304 0)
1714
1715 SUB UNIVERSAL::VERSION = (xsub 0x811f7ac 0)
1716
1717 SUB DynaLoader::boot_DynaLoader = (xsub 0x805b188 0)
1718
1719and C<Perl_dump_all>, which dumps all the subroutines in the stash and
1720the op tree of the main root.
1721
954c1994 1722=head1 How multiple interpreters and concurrency are supported
ee072b34 1723
ee072b34
GS
1724=head2 Background and PERL_IMPLICIT_CONTEXT
1725
1726The Perl interpreter can be regarded as a closed box: it has an API
1727for feeding it code or otherwise making it do things, but it also has
1728functions for its own use. This smells a lot like an object, and
1729there are ways for you to build Perl so that you can have multiple
1730interpreters, with one interpreter represented either as a C++ object,
1731a C structure, or inside a thread. The thread, the C structure, or
1732the C++ object will contain all the context, the state of that
1733interpreter.
1734
54aff467 1735Three macros control the major Perl build flavors: MULTIPLICITY,
4d1ff10f 1736USE_5005THREADS and PERL_OBJECT. The MULTIPLICITY build has a C structure
54aff467 1737that packages all the interpreter state, there is a similar thread-specific
4d1ff10f 1738data structure under USE_5005THREADS, and the (now deprecated) PERL_OBJECT
a7486cbb 1739build has a C++ class to maintain interpreter state. In all three cases,
54aff467
GS
1740PERL_IMPLICIT_CONTEXT is also normally defined, and enables the
1741support for passing in a "hidden" first argument that represents all three
651a3225 1742data structures.
54aff467
GS
1743
1744All this obviously requires a way for the Perl internal functions to be
ee072b34
GS
1745C++ methods, subroutines taking some kind of structure as the first
1746argument, or subroutines taking nothing as the first argument. To
1747enable these three very different ways of building the interpreter,
1748the Perl source (as it does in so many other situations) makes heavy
1749use of macros and subroutine naming conventions.
1750
54aff467 1751First problem: deciding which functions will be public API functions and
00aadd71 1752which will be private. All functions whose names begin C<S_> are private
954c1994
GS
1753(think "S" for "secret" or "static"). All other functions begin with
1754"Perl_", but just because a function begins with "Perl_" does not mean it is
00aadd71
NIS
1755part of the API. (See L</Internal Functions>.) The easiest way to be B<sure> a
1756function is part of the API is to find its entry in L<perlapi>.
1757If it exists in L<perlapi>, it's part of the API. If it doesn't, and you
1758think it should be (i.e., you need it for your extension), send mail via
a422fd2d 1759L<perlbug> explaining why you think it should be.
ee072b34
GS
1760
1761Second problem: there must be a syntax so that the same subroutine
1762declarations and calls can pass a structure as their first argument,
1763or pass nothing. To solve this, the subroutines are named and
1764declared in a particular way. Here's a typical start of a static
1765function used within the Perl guts:
1766
1767 STATIC void
1768 S_incline(pTHX_ char *s)
1769
1770STATIC becomes "static" in C, and is #define'd to nothing in C++.
1771
651a3225
GS
1772A public function (i.e. part of the internal API, but not necessarily
1773sanctioned for use in extensions) begins like this:
ee072b34
GS
1774
1775 void
1776 Perl_sv_setsv(pTHX_ SV* dsv, SV* ssv)
1777
1778C<pTHX_> is one of a number of macros (in perl.h) that hide the
1779details of the interpreter's context. THX stands for "thread", "this",
1780or "thingy", as the case may be. (And no, George Lucas is not involved. :-)
1781The first character could be 'p' for a B<p>rototype, 'a' for B<a>rgument,
a7486cbb
JH
1782or 'd' for B<d>eclaration, so we have C<pTHX>, C<aTHX> and C<dTHX>, and
1783their variants.
ee072b34 1784
a7486cbb
JH
1785When Perl is built without options that set PERL_IMPLICIT_CONTEXT, there is no
1786first argument containing the interpreter's context. The trailing underscore
ee072b34
GS
1787in the pTHX_ macro indicates that the macro expansion needs a comma
1788after the context argument because other arguments follow it. If
1789PERL_IMPLICIT_CONTEXT is not defined, pTHX_ will be ignored, and the
54aff467
GS
1790subroutine is not prototyped to take the extra argument. The form of the
1791macro without the trailing underscore is used when there are no additional
ee072b34
GS
1792explicit arguments.
1793
54aff467 1794When a core function calls another, it must pass the context. This
a7486cbb 1795is normally hidden via macros. Consider C<sv_setsv>. It expands into
ee072b34
GS
1796something like this:
1797
1798 ifdef PERL_IMPLICIT_CONTEXT
c52f9dcd 1799 define sv_setsv(a,b) Perl_sv_setsv(aTHX_ a, b)
ee072b34
GS
1800 /* can't do this for vararg functions, see below */
1801 else
c52f9dcd 1802 define sv_setsv Perl_sv_setsv
ee072b34
GS
1803 endif
1804
1805This works well, and means that XS authors can gleefully write:
1806
1807 sv_setsv(foo, bar);
1808
1809and still have it work under all the modes Perl could have been
1810compiled with.
1811
1812Under PERL_OBJECT in the core, that will translate to either:
1813
1814 CPerlObj::Perl_sv_setsv(foo,bar); # in CPerlObj functions,
1815 # C++ takes care of 'this'
1816 or
1817
1818 pPerl->Perl_sv_setsv(foo,bar); # in truly static functions,
1819 # see objXSUB.h
1820
1821Under PERL_OBJECT in extensions (aka PERL_CAPI), or under
4d1ff10f 1822MULTIPLICITY/USE_5005THREADS with PERL_IMPLICIT_CONTEXT in both core
a7486cbb 1823and extensions, it will become:
ee072b34
GS
1824
1825 Perl_sv_setsv(aTHX_ foo, bar); # the canonical Perl "API"
1826 # for all build flavors
1827
1828This doesn't work so cleanly for varargs functions, though, as macros
1829imply that the number of arguments is known in advance. Instead we
1830either need to spell them out fully, passing C<aTHX_> as the first
1831argument (the Perl core tends to do this with functions like
1832Perl_warner), or use a context-free version.
1833
1834The context-free version of Perl_warner is called
1835Perl_warner_nocontext, and does not take the extra argument. Instead
1836it does dTHX; to get the context from thread-local storage. We
1837C<#define warner Perl_warner_nocontext> so that extensions get source
1838compatibility at the expense of performance. (Passing an arg is
1839cheaper than grabbing it from thread-local storage.)
1840
1841You can ignore [pad]THX[xo] when browsing the Perl headers/sources.
1842Those are strictly for use within the core. Extensions and embedders
1843need only be aware of [pad]THX.
1844
a7486cbb
JH
1845=head2 So what happened to dTHR?
1846
1847C<dTHR> was introduced in perl 5.005 to support the older thread model.
1848The older thread model now uses the C<THX> mechanism to pass context
1849pointers around, so C<dTHR> is not useful any more. Perl 5.6.0 and
1850later still have it for backward source compatibility, but it is defined
1851to be a no-op.
1852
ee072b34
GS
1853=head2 How do I use all this in extensions?
1854
1855When Perl is built with PERL_IMPLICIT_CONTEXT, extensions that call
1856any functions in the Perl API will need to pass the initial context
1857argument somehow. The kicker is that you will need to write it in
1858such a way that the extension still compiles when Perl hasn't been
1859built with PERL_IMPLICIT_CONTEXT enabled.
1860
1861There are three ways to do this. First, the easy but inefficient way,
1862which is also the default, in order to maintain source compatibility
1863with extensions: whenever XSUB.h is #included, it redefines the aTHX
1864and aTHX_ macros to call a function that will return the context.
1865Thus, something like:
1866
1867 sv_setsv(asv, bsv);
1868
4375e838 1869in your extension will translate to this when PERL_IMPLICIT_CONTEXT is
54aff467 1870in effect:
ee072b34 1871
2fa86c13 1872 Perl_sv_setsv(Perl_get_context(), asv, bsv);
ee072b34 1873
54aff467 1874or to this otherwise:
ee072b34
GS
1875
1876 Perl_sv_setsv(asv, bsv);
1877
1878You have to do nothing new in your extension to get this; since
2fa86c13 1879the Perl library provides Perl_get_context(), it will all just
ee072b34
GS
1880work.
1881
1882The second, more efficient way is to use the following template for
1883your Foo.xs:
1884
c52f9dcd
JH
1885 #define PERL_NO_GET_CONTEXT /* we want efficiency */
1886 #include "EXTERN.h"
1887 #include "perl.h"
1888 #include "XSUB.h"
ee072b34
GS
1889
1890 static my_private_function(int arg1, int arg2);
1891
c52f9dcd
JH
1892 static SV *
1893 my_private_function(int arg1, int arg2)
1894 {
1895 dTHX; /* fetch context */
1896 ... call many Perl API functions ...
1897 }
ee072b34
GS
1898
1899 [... etc ...]
1900
c52f9dcd 1901 MODULE = Foo PACKAGE = Foo
ee072b34 1902
c52f9dcd 1903 /* typical XSUB */
ee072b34 1904
c52f9dcd
JH
1905 void
1906 my_xsub(arg)
1907 int arg
1908 CODE:
1909 my_private_function(arg, 10);
ee072b34
GS
1910
1911Note that the only two changes from the normal way of writing an
1912extension is the addition of a C<#define PERL_NO_GET_CONTEXT> before
1913including the Perl headers, followed by a C<dTHX;> declaration at
1914the start of every function that will call the Perl API. (You'll
1915know which functions need this, because the C compiler will complain
1916that there's an undeclared identifier in those functions.) No changes
1917are needed for the XSUBs themselves, because the XS() macro is
1918correctly defined to pass in the implicit context if needed.
1919
1920The third, even more efficient way is to ape how it is done within
1921the Perl guts:
1922
1923
c52f9dcd
JH
1924 #define PERL_NO_GET_CONTEXT /* we want efficiency */
1925 #include "EXTERN.h"
1926 #include "perl.h"
1927 #include "XSUB.h"
ee072b34
GS
1928
1929 /* pTHX_ only needed for functions that call Perl API */
1930 static my_private_function(pTHX_ int arg1, int arg2);
1931
c52f9dcd
JH
1932 static SV *
1933 my_private_function(pTHX_ int arg1, int arg2)
1934 {
1935 /* dTHX; not needed here, because THX is an argument */
1936 ... call Perl API functions ...
1937 }
ee072b34
GS
1938
1939 [... etc ...]
1940
c52f9dcd 1941 MODULE = Foo PACKAGE = Foo
ee072b34 1942
c52f9dcd 1943 /* typical XSUB */
ee072b34 1944
c52f9dcd
JH
1945 void
1946 my_xsub(arg)
1947 int arg
1948 CODE:
1949 my_private_function(aTHX_ arg, 10);
ee072b34
GS
1950
1951This implementation never has to fetch the context using a function
1952call, since it is always passed as an extra argument. Depending on
1953your needs for simplicity or efficiency, you may mix the previous
1954two approaches freely.
1955
651a3225
GS
1956Never add a comma after C<pTHX> yourself--always use the form of the
1957macro with the underscore for functions that take explicit arguments,
1958or the form without the argument for functions with no explicit arguments.
ee072b34 1959
a7486cbb
JH
1960=head2 Should I do anything special if I call perl from multiple threads?
1961
1962If you create interpreters in one thread and then proceed to call them in
1963another, you need to make sure perl's own Thread Local Storage (TLS) slot is
1964initialized correctly in each of those threads.
1965
1966The C<perl_alloc> and C<perl_clone> API functions will automatically set
1967the TLS slot to the interpreter they created, so that there is no need to do
1968anything special if the interpreter is always accessed in the same thread that
1969created it, and that thread did not create or call any other interpreters
1970afterwards. If that is not the case, you have to set the TLS slot of the
1971thread before calling any functions in the Perl API on that particular
1972interpreter. This is done by calling the C<PERL_SET_CONTEXT> macro in that
1973thread as the first thing you do:
1974
1975 /* do this before doing anything else with some_perl */
1976 PERL_SET_CONTEXT(some_perl);
1977
1978 ... other Perl API calls on some_perl go here ...
1979
ee072b34
GS
1980=head2 Future Plans and PERL_IMPLICIT_SYS
1981
1982Just as PERL_IMPLICIT_CONTEXT provides a way to bundle up everything
1983that the interpreter knows about itself and pass it around, so too are
1984there plans to allow the interpreter to bundle up everything it knows
1985about the environment it's running on. This is enabled with the
a7486cbb 1986PERL_IMPLICIT_SYS macro. Currently it only works with PERL_OBJECT
4d1ff10f 1987and USE_5005THREADS on Windows (see inside iperlsys.h).
ee072b34
GS
1988
1989This allows the ability to provide an extra pointer (called the "host"
1990environment) for all the system calls. This makes it possible for
1991all the system stuff to maintain their own state, broken down into
1992seven C structures. These are thin wrappers around the usual system
1993calls (see win32/perllib.c) for the default perl executable, but for a
1994more ambitious host (like the one that would do fork() emulation) all
1995the extra work needed to pretend that different interpreters are
1996actually different "processes", would be done here.
1997
1998The Perl engine/interpreter and the host are orthogonal entities.
1999There could be one or more interpreters in a process, and one or
2000more "hosts", with free association between them.
2001
a422fd2d
SC
2002=head1 Internal Functions
2003
2004All of Perl's internal functions which will be exposed to the outside
2005world are be prefixed by C<Perl_> so that they will not conflict with XS
2006functions or functions used in a program in which Perl is embedded.
2007Similarly, all global variables begin with C<PL_>. (By convention,
2008static functions start with C<S_>)
2009
2010Inside the Perl core, you can get at the functions either with or
2011without the C<Perl_> prefix, thanks to a bunch of defines that live in
2012F<embed.h>. This header file is generated automatically from
2013F<embed.pl>. F<embed.pl> also creates the prototyping header files for
2014the internal functions, generates the documentation and a lot of other
2015bits and pieces. It's important that when you add a new function to the
2016core or change an existing one, you change the data in the table at the
2017end of F<embed.pl> as well. Here's a sample entry from that table:
2018
2019 Apd |SV** |av_fetch |AV* ar|I32 key|I32 lval
2020
2021The second column is the return type, the third column the name. Columns
2022after that are the arguments. The first column is a set of flags:
2023
2024=over 3
2025
2026=item A
2027
2028This function is a part of the public API.
2029
2030=item p
2031
2032This function has a C<Perl_> prefix; ie, it is defined as C<Perl_av_fetch>
2033
2034=item d
2035
2036This function has documentation using the C<apidoc> feature which we'll
2037look at in a second.
2038
2039=back
2040
2041Other available flags are:
2042
2043=over 3
2044
2045=item s
2046
a7486cbb
JH
2047This is a static function and is defined as C<S_whatever>, and usually
2048called within the sources as C<whatever(...)>.
a422fd2d
SC
2049
2050=item n
2051
2052This does not use C<aTHX_> and C<pTHX> to pass interpreter context. (See
2053L<perlguts/Background and PERL_IMPLICIT_CONTEXT>.)
2054
2055=item r
2056
2057This function never returns; C<croak>, C<exit> and friends.
2058
2059=item f
2060
2061This function takes a variable number of arguments, C<printf> style.
2062The argument list should end with C<...>, like this:
2063
2064 Afprd |void |croak |const char* pat|...
2065
a7486cbb 2066=item M
a422fd2d 2067
00aadd71 2068This function is part of the experimental development API, and may change
a422fd2d
SC
2069or disappear without notice.
2070
2071=item o
2072
2073This function should not have a compatibility macro to define, say,
2074C<Perl_parse> to C<parse>. It must be called as C<Perl_parse>.
2075
2076=item j
2077
2078This function is not a member of C<CPerlObj>. If you don't know
2079what this means, don't use it.
2080
2081=item x
2082
2083This function isn't exported out of the Perl core.
2084
2085=back
2086
2087If you edit F<embed.pl>, you will need to run C<make regen_headers> to
2088force a rebuild of F<embed.h> and other auto-generated files.
2089
6b4667fc 2090=head2 Formatted Printing of IVs, UVs, and NVs
9dd9db0b 2091
6b4667fc
A
2092If you are printing IVs, UVs, or NVS instead of the stdio(3) style
2093formatting codes like C<%d>, C<%ld>, C<%f>, you should use the
2094following macros for portability
9dd9db0b 2095
c52f9dcd
JH
2096 IVdf IV in decimal
2097 UVuf UV in decimal
2098 UVof UV in octal
2099 UVxf UV in hexadecimal
2100 NVef NV %e-like
2101 NVff NV %f-like
2102 NVgf NV %g-like
9dd9db0b 2103
6b4667fc
A
2104These will take care of 64-bit integers and long doubles.
2105For example:
2106
c52f9dcd 2107 printf("IV is %"IVdf"\n", iv);
6b4667fc
A
2108
2109The IVdf will expand to whatever is the correct format for the IVs.
9dd9db0b 2110
8908e76d
JH
2111If you are printing addresses of pointers, use UVxf combined
2112with PTR2UV(), do not use %lx or %p.
2113
2114=head2 Pointer-To-Integer and Integer-To-Pointer
2115
2116Because pointer size does not necessarily equal integer size,
2117use the follow macros to do it right.
2118
c52f9dcd
JH
2119 PTR2UV(pointer)
2120 PTR2IV(pointer)
2121 PTR2NV(pointer)
2122 INT2PTR(pointertotype, integer)
8908e76d
JH
2123
2124For example:
2125
c52f9dcd
JH
2126 IV iv = ...;
2127 SV *sv = INT2PTR(SV*, iv);
8908e76d
JH
2128
2129and
2130
c52f9dcd
JH
2131 AV *av = ...;
2132 UV uv = PTR2UV(av);
8908e76d 2133
a422fd2d
SC
2134=head2 Source Documentation
2135
2136There's an effort going on to document the internal functions and
2137automatically produce reference manuals from them - L<perlapi> is one
2138such manual which details all the functions which are available to XS
2139writers. L<perlintern> is the autogenerated manual for the functions
2140which are not part of the API and are supposedly for internal use only.
2141
2142Source documentation is created by putting POD comments into the C
2143source, like this:
2144
2145 /*
2146 =for apidoc sv_setiv
2147
2148 Copies an integer into the given SV. Does not handle 'set' magic. See
2149 C<sv_setiv_mg>.
2150
2151 =cut
2152 */
2153
2154Please try and supply some documentation if you add functions to the
2155Perl core.
2156
2157=head1 Unicode Support
2158
2159Perl 5.6.0 introduced Unicode support. It's important for porters and XS
2160writers to understand this support and make sure that the code they
2161write does not corrupt Unicode data.
2162
2163=head2 What B<is> Unicode, anyway?
2164
2165In the olden, less enlightened times, we all used to use ASCII. Most of
2166us did, anyway. The big problem with ASCII is that it's American. Well,
2167no, that's not actually the problem; the problem is that it's not
2168particularly useful for people who don't use the Roman alphabet. What
2169used to happen was that particular languages would stick their own
2170alphabet in the upper range of the sequence, between 128 and 255. Of
2171course, we then ended up with plenty of variants that weren't quite
2172ASCII, and the whole point of it being a standard was lost.
2173
2174Worse still, if you've got a language like Chinese or
2175Japanese that has hundreds or thousands of characters, then you really
2176can't fit them into a mere 256, so they had to forget about ASCII
2177altogether, and build their own systems using pairs of numbers to refer
2178to one character.
2179
2180To fix this, some people formed Unicode, Inc. and
2181produced a new character set containing all the characters you can
2182possibly think of and more. There are several ways of representing these
2183characters, and the one Perl uses is called UTF8. UTF8 uses
2184a variable number of bytes to represent a character, instead of just
b3b6085d 2185one. You can learn more about Unicode at http://www.unicode.org/
a422fd2d
SC
2186
2187=head2 How can I recognise a UTF8 string?
2188
2189You can't. This is because UTF8 data is stored in bytes just like
2190non-UTF8 data. The Unicode character 200, (C<0xC8> for you hex types)
2191capital E with a grave accent, is represented by the two bytes
2192C<v196.172>. Unfortunately, the non-Unicode string C<chr(196).chr(172)>
2193has that byte sequence as well. So you can't tell just by looking - this
2194is what makes Unicode input an interesting problem.
2195
2196The API function C<is_utf8_string> can help; it'll tell you if a string
2197contains only valid UTF8 characters. However, it can't do the work for
2198you. On a character-by-character basis, C<is_utf8_char> will tell you
2199whether the current character in a string is valid UTF8.
2200
2201=head2 How does UTF8 represent Unicode characters?
2202
2203As mentioned above, UTF8 uses a variable number of bytes to store a
2204character. Characters with values 1...128 are stored in one byte, just
2205like good ol' ASCII. Character 129 is stored as C<v194.129>; this
a31a806a 2206continues up to character 191, which is C<v194.191>. Now we've run out of
a422fd2d
SC
2207bits (191 is binary C<10111111>) so we move on; 192 is C<v195.128>. And
2208so it goes on, moving to three bytes at character 2048.
2209
2210Assuming you know you're dealing with a UTF8 string, you can find out
2211how long the first character in it is with the C<UTF8SKIP> macro:
2212
2213 char *utf = "\305\233\340\240\201";
2214 I32 len;
2215
2216 len = UTF8SKIP(utf); /* len is 2 here */
2217 utf += len;
2218 len = UTF8SKIP(utf); /* len is 3 here */
2219
2220Another way to skip over characters in a UTF8 string is to use
2221C<utf8_hop>, which takes a string and a number of characters to skip
2222over. You're on your own about bounds checking, though, so don't use it
2223lightly.
2224
2225All bytes in a multi-byte UTF8 character will have the high bit set, so
2226you can test if you need to do something special with this character
2227like this:
2228
2229 UV uv;
2230
2231 if (utf & 0x80)
2232 /* Must treat this as UTF8 */
2233 uv = utf8_to_uv(utf);
2234 else
2235 /* OK to treat this character as a byte */
2236 uv = *utf;
2237
2238You can also see in that example that we use C<utf8_to_uv> to get the
2239value of the character; the inverse function C<uv_to_utf8> is available
2240for putting a UV into UTF8:
2241
2242 if (uv > 0x80)
2243 /* Must treat this as UTF8 */
2244 utf8 = uv_to_utf8(utf8, uv);
2245 else
2246 /* OK to treat this character as a byte */
2247 *utf8++ = uv;
2248
2249You B<must> convert characters to UVs using the above functions if
2250you're ever in a situation where you have to match UTF8 and non-UTF8
2251characters. You may not skip over UTF8 characters in this case. If you
2252do this, you'll lose the ability to match hi-bit non-UTF8 characters;
2253for instance, if your UTF8 string contains C<v196.172>, and you skip
2254that character, you can never match a C<chr(200)> in a non-UTF8 string.
2255So don't do that!
2256
2257=head2 How does Perl store UTF8 strings?
2258
2259Currently, Perl deals with Unicode strings and non-Unicode strings
2260slightly differently. If a string has been identified as being UTF-8
2261encoded, Perl will set a flag in the SV, C<SVf_UTF8>. You can check and
2262manipulate this flag with the following macros:
2263
2264 SvUTF8(sv)
2265 SvUTF8_on(sv)
2266 SvUTF8_off(sv)
2267
2268This flag has an important effect on Perl's treatment of the string: if
2269Unicode data is not properly distinguished, regular expressions,
2270C<length>, C<substr> and other string handling operations will have
2271undesirable results.
2272
2273The problem comes when you have, for instance, a string that isn't
2274flagged is UTF8, and contains a byte sequence that could be UTF8 -
2275especially when combining non-UTF8 and UTF8 strings.
2276
2277Never forget that the C<SVf_UTF8> flag is separate to the PV value; you
2278need be sure you don't accidentally knock it off while you're
2279manipulating SVs. More specifically, you cannot expect to do this:
2280
2281 SV *sv;
2282 SV *nsv;
2283 STRLEN len;
2284 char *p;
2285
2286 p = SvPV(sv, len);
2287 frobnicate(p);
2288 nsv = newSVpvn(p, len);
2289
2290The C<char*> string does not tell you the whole story, and you can't
2291copy or reconstruct an SV just by copying the string value. Check if the
2292old SV has the UTF8 flag set, and act accordingly:
2293
2294 p = SvPV(sv, len);
2295 frobnicate(p);
2296 nsv = newSVpvn(p, len);
2297 if (SvUTF8(sv))
2298 SvUTF8_on(nsv);
2299
2300In fact, your C<frobnicate> function should be made aware of whether or
2301not it's dealing with UTF8 data, so that it can handle the string
2302appropriately.
2303
2304=head2 How do I convert a string to UTF8?
2305
2306If you're mixing UTF8 and non-UTF8 strings, you might find it necessary
2307to upgrade one of the strings to UTF8. If you've got an SV, the easiest
2308way to do this is:
2309
2310 sv_utf8_upgrade(sv);
2311
2312However, you must not do this, for example:
2313
2314 if (!SvUTF8(left))
2315 sv_utf8_upgrade(left);
2316
2317If you do this in a binary operator, you will actually change one of the
b1866b2d 2318strings that came into the operator, and, while it shouldn't be noticeable
a422fd2d
SC
2319by the end user, it can cause problems.
2320
2321Instead, C<bytes_to_utf8> will give you a UTF8-encoded B<copy> of its
2322string argument. This is useful for having the data available for
b1866b2d 2323comparisons and so on, without harming the original SV. There's also
a422fd2d
SC
2324C<utf8_to_bytes> to go the other way, but naturally, this will fail if
2325the string contains any characters above 255 that can't be represented
2326in a single byte.
2327
2328=head2 Is there anything else I need to know?
2329
2330Not really. Just remember these things:
2331
2332=over 3
2333
2334=item *
2335
2336There's no way to tell if a string is UTF8 or not. You can tell if an SV
2337is UTF8 by looking at is C<SvUTF8> flag. Don't forget to set the flag if
2338something should be UTF8. Treat the flag as part of the PV, even though
2339it's not - if you pass on the PV to somewhere, pass on the flag too.
2340
2341=item *
2342
2343If a string is UTF8, B<always> use C<utf8_to_uv> to get at the value,
2344unless C<!(*s & 0x80)> in which case you can use C<*s>.
2345
2346=item *
2347
2348When writing to a UTF8 string, B<always> use C<uv_to_utf8>, unless
2349C<uv < 0x80> in which case you can use C<*s = uv>.
2350
2351=item *
2352
2353Mixing UTF8 and non-UTF8 strings is tricky. Use C<bytes_to_utf8> to get
2354a new string which is UTF8 encoded. There are tricks you can use to
2355delay deciding whether you need to use a UTF8 string until you get to a
2356high character - C<HALF_UPGRADE> is one of those.
2357
2358=back
2359
53e06cf0
SC
2360=head1 Custom Operators
2361
2362Custom operator support is a new experimental feature that allows you do
2363define your own ops. This is primarily to allow the building of
2364interpreters for other languages in the Perl core, but it also allows
2365optimizations through the creation of "macro-ops" (ops which perform the
2366functions of multiple ops which are usually executed together, such as
2367C<gvsv, gvsv, add>.) Currently, this feature must be enabled with the C
2368flag C<-DPERL_CUSTOM_OPS>.
2369
2370Enabling the feature will create a new op type, C<OP_CUSTOM>. The Perl
2371core does not "know" anything special about this op type, and so it will
2372not be involved in any optimizations. This also means that you can
2373define your custom ops to be any op structure - unary, binary, list and
2374so on - you like.
2375
2376It's important to know what custom operators won't do for you. They
2377won't let you add new syntax to Perl, directly. They won't even let you
2378add new keywords, directly. In fact, they won't change the way Perl
2379compiles a program at all. You have to do those changes yourself, after
2380Perl has compiled the program. You do this either by manipulating the op
2381tree using a C<CHECK> block and the C<B::Generate> module, or by adding
2382a custom peephole optimizer with the C<optimize> module.
2383
2384When you do this, you replace ordinary Perl ops with custom ops by
2385creating ops with the type C<OP_CUSTOM> and the C<pp_addr> of your own
2386PP function. This should be defined in XS code, and should look like
2387the PP ops in C<pp_*.c>. You are responsible for ensuring that your op
2388takes the appropriate number of values from the stack, and you are
2389responsible for adding stack marks if necessary.
2390
2391You should also "register" your op with the Perl interpreter so that it
2392can produce sensible error and warning messages. Since it is possible to
2393have multiple custom ops within the one "logical" op type C<OP_CUSTOM>,
2394Perl uses the value of C<< o->op_ppaddr >> as a key into the
2395C<PL_custom_op_descs> and C<PL_custom_op_names> hashes. This means you
2396need to enter a name and description for your op at the appropriate
2397place in the C<PL_custom_op_names> and C<PL_custom_op_descs> hashes.
2398
2399Forthcoming versions of C<B::Generate> (version 1.0 and above) should
2400directly support the creation of custom ops by name; C<Opcodes::Custom>
2401will provide functions which make it trivial to "register" custom ops to
2402the Perl interpreter.
2403
954c1994 2404=head1 AUTHORS
e89caa19 2405
954c1994
GS
2406Until May 1997, this document was maintained by Jeff Okamoto
2407<okamoto@corp.hp.com>. It is now maintained as part of Perl itself
2408by the Perl 5 Porters <perl5-porters@perl.org>.
cb1a09d0 2409
954c1994
GS
2410With lots of help and suggestions from Dean Roehrich, Malcolm Beattie,
2411Andreas Koenig, Paul Hudson, Ilya Zakharevich, Paul Marquess, Neil
2412Bowers, Matthew Green, Tim Bunce, Spider Boardman, Ulrich Pfeifer,
2413Stephen McCamant, and Gurusamy Sarathy.
cb1a09d0 2414
954c1994 2415API Listing originally by Dean Roehrich <roehrich@cray.com>.
cb1a09d0 2416
954c1994
GS
2417Modifications to autogenerate the API listing (L<perlapi>) by Benjamin
2418Stuhl.
cb1a09d0 2419
954c1994 2420=head1 SEE ALSO
cb1a09d0 2421
954c1994 2422perlapi(1), perlintern(1), perlxs(1), perlembed(1)