This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Move more URLs from http:// to https://
[perl5.git] / pod / perlpacktut.pod
CommitLineData
34babc16
JH
1=head1 NAME
2
3perlpacktut - tutorial on C<pack> and C<unpack>
4
5=head1 DESCRIPTION
6
7C<pack> and C<unpack> are two functions for transforming data according
8to a user-defined template, between the guarded way Perl stores values
47b6252e 9and some well-defined representation as might be required in the
34babc16
JH
10environment of a Perl program. Unfortunately, they're also two of
11the most misunderstood and most often overlooked functions that Perl
12provides. This tutorial will demystify them for you.
13
14
15=head1 The Basic Principle
16
17Most programming languages don't shelter the memory where variables are
18stored. In C, for instance, you can take the address of some variable,
19and the C<sizeof> operator tells you how many bytes are allocated to
20the variable. Using the address and the size, you may access the storage
21to your heart's content.
22
23In Perl, you just can't access memory at random, but the structural and
24representational conversion provided by C<pack> and C<unpack> is an
25excellent alternative. The C<pack> function converts values to a byte
26sequence containing representations according to a given specification,
27the so-called "template" argument. C<unpack> is the reverse process,
28deriving some values from the contents of a string of bytes. (Be cautioned,
29however, that not all that has been packed together can be neatly unpacked -
30a very common experience as seasoned travellers are likely to confirm.)
31
32Why, you may ask, would you need a chunk of memory containing some values
33in binary representation? One good reason is input and output accessing
34some file, a device, or a network connection, whereby this binary
35representation is either forced on you or will give you some benefit
36in processing. Another cause is passing data to some system call that
37is not available as a Perl function: C<syscall> requires you to provide
38parameters stored in the way it happens in a C program. Even text processing
39(as shown in the next section) may be simplified with judicious usage
40of these two functions.
41
42To see how (un)packing works, we'll start with a simple template
43code where the conversion is in low gear: between the contents of a byte
44sequence and a string of hexadecimal digits. Let's use C<unpack>, since
47b6252e 45this is likely to remind you of a dump program, or some desperate last
34babc16
JH
46message unfortunate programs are wont to throw at you before they expire
47into the wild blue yonder. Assuming that the variable C<$mem> holds a
48sequence of bytes that we'd like to inspect without assuming anything
49about its meaning, we can write
50
51 my( $hex ) = unpack( 'H*', $mem );
52 print "$hex\n";
53
54whereupon we might see something like this, with each pair of hex digits
55corresponding to a byte:
56
57 41204d414e204120504c414e20412043414e414c2050414e414d41
58
47b6252e 59What was in this chunk of memory? Numbers, characters, or a mixture of
34babc16
JH
60both? Assuming that we're on a computer where ASCII (or some similar)
61encoding is used: hexadecimal values in the range C<0x40> - C<0x5A>
47f22e19 62indicate an uppercase letter, and C<0x20> encodes a space. So we might
34babc16
JH
63assume it is a piece of text, which some are able to read like a tabloid;
64but others will have to get hold of an ASCII table and relive that
65firstgrader feeling. Not caring too much about which way to read this,
66we note that C<unpack> with the template code C<H> converts the contents
67of a sequence of bytes into the customary hexadecimal notation. Since
68"a sequence of" is a pretty vague indication of quantity, C<H> has been
69defined to convert just a single hexadecimal digit unless it is followed
70by a repeat count. An asterisk for the repeat count means to use whatever
71remains.
72
73The inverse operation - packing byte contents from a string of hexadecimal
74digits - is just as easily written. For instance:
75
4e848ca9 76 my $s = pack( 'H2' x 10, 30..39 );
34babc16
JH
77 print "$s\n";
78
79Since we feed a list of ten 2-digit hexadecimal strings to C<pack>, the
80pack template should contain ten pack codes. If this is run on a computer
81with ASCII character coding, it will print C<0123456789>.
82
34babc16
JH
83=head1 Packing Text
84
85Let's suppose you've got to read in a data file like this:
86
87 Date |Description | Income|Expenditure
9567c12e 88 01/24/2001 Zed's Camel Emporium 1147.99
34babc16
JH
89 01/28/2001 Flea spray 24.99
90 01/29/2001 Camel rides to tourists 235.00
91
92How do we do it? You might think first to use C<split>; however, since
93C<split> collapses blank fields, you'll never know whether a record was
94income or expenditure. Oops. Well, you could always use C<substr>:
95
96 while (<>) {
97 my $date = substr($_, 0, 11);
98 my $desc = substr($_, 12, 27);
99 my $income = substr($_, 40, 7);
100 my $expend = substr($_, 52, 7);
101 ...
102 }
103
104It's not really a barrel of laughs, is it? In fact, it's worse than it
105may seem; the eagle-eyed may notice that the first field should only be
10610 characters wide, and the error has propagated right through the other
107numbers - which we've had to count by hand. So it's error-prone as well
108as horribly unfriendly.
109
110Or maybe we could use regular expressions:
7207e29d 111
34babc16
JH
112 while (<>) {
113 my($date, $desc, $income, $expend) =
114 m|(\d\d/\d\d/\d{4}) (.{27}) (.{7})(.*)|;
115 ...
116 }
117
118Urgh. Well, it's a bit better, but - well, would you want to maintain
119that?
120
121Hey, isn't Perl supposed to make this sort of thing easy? Well, it does,
122if you use the right tools. C<pack> and C<unpack> are designed to help
123you out when dealing with fixed-width data like the above. Let's have a
124look at a solution with C<unpack>:
125
126 while (<>) {
127 my($date, $desc, $income, $expend) = unpack("A10xA27xA7A*", $_);
128 ...
129 }
130
131That looks a bit nicer; but we've got to take apart that weird template.
132Where did I pull that out of?
133
134OK, let's have a look at some of our data again; in fact, we'll include
135the headers, and a handy ruler so we can keep track of where we are.
136
137 1 2 3 4 5
138 1234567890123456789012345678901234567890123456789012345678
139 Date |Description | Income|Expenditure
140 01/28/2001 Flea spray 24.99
141 01/29/2001 Camel rides to tourists 235.00
142
47b6252e 143From this, we can see that the date column stretches from column 1 to
34babc16
JH
144column 10 - ten characters wide. The C<pack>-ese for "character" is
145C<A>, and ten of them are C<A10>. So if we just wanted to extract the
146dates, we could say this:
147
148 my($date) = unpack("A10", $_);
149
150OK, what's next? Between the date and the description is a blank column;
151we want to skip over that. The C<x> template means "skip forward", so we
152want one of those. Next, we have another batch of characters, from 12 to
15338. That's 27 more characters, hence C<A27>. (Don't make the fencepost
154error - there are 27 characters between 12 and 38, not 26. Count 'em!)
155
156Now we skip another character and pick up the next 7 characters:
157
158 my($date,$description,$income) = unpack("A10xA27xA7", $_);
159
160Now comes the clever bit. Lines in our ledger which are just income and
161not expenditure might end at column 46. Hence, we don't want to tell our
162C<unpack> pattern that we B<need> to find another 12 characters; we'll
163just say "if there's anything left, take it". As you might guess from
164regular expressions, that's what the C<*> means: "use everything
165remaining".
166
167=over 3
168
169=item *
170
171Be warned, though, that unlike regular expressions, if the C<unpack>
172template doesn't match the incoming data, Perl will scream and die.
173
174=back
175
176
177Hence, putting it all together:
178
555bd962
BG
179 my ($date, $description, $income, $expend) =
180 unpack("A10xA27xA7xA*", $_);
34babc16
JH
181
182Now, that's our data parsed. I suppose what we might want to do now is
183total up our income and expenditure, and add another line to the end of
184our ledger - in the same format - saying how much we've brought in and
185how much we've spent:
186
187 while (<>) {
555bd962
BG
188 my ($date, $desc, $income, $expend) =
189 unpack("A10xA27xA7xA*", $_);
34babc16
JH
190 $tot_income += $income;
191 $tot_expend += $expend;
192 }
193
194 $tot_income = sprintf("%.2f", $tot_income); # Get them into
195 $tot_expend = sprintf("%.2f", $tot_expend); # "financial" format
196
197 $date = POSIX::strftime("%m/%d/%Y", localtime);
198
199 # OK, let's go:
200
555bd962
BG
201 print pack("A10xA27xA7xA*", $date, "Totals",
202 $tot_income, $tot_expend);
34babc16
JH
203
204Oh, hmm. That didn't quite work. Let's see what happened:
205
9567c12e 206 01/24/2001 Zed's Camel Emporium 1147.99
34babc16
JH
207 01/28/2001 Flea spray 24.99
208 01/29/2001 Camel rides to tourists 1235.00
209 03/23/2001Totals 1235.001172.98
210
211OK, it's a start, but what happened to the spaces? We put C<x>, didn't
212we? Shouldn't it skip forward? Let's look at what L<perlfunc/pack> says:
213
214 x A null byte.
215
216Urgh. No wonder. There's a big difference between "a null byte",
217character zero, and "a space", character 32. Perl's put something
218between the date and the description - but unfortunately, we can't see
219it!
220
221What we actually need to do is expand the width of the fields. The C<A>
222format pads any non-existent characters with spaces, so we can use the
223additional spaces to line up our fields, like this:
224
555bd962
BG
225 print pack("A11 A28 A8 A*", $date, "Totals",
226 $tot_income, $tot_expend);
34babc16
JH
227
228(Note that you can put spaces in the template to make it more readable,
229but they don't translate to spaces in the output.) Here's what we got
230this time:
231
9567c12e 232 01/24/2001 Zed's Camel Emporium 1147.99
34babc16
JH
233 01/28/2001 Flea spray 24.99
234 01/29/2001 Camel rides to tourists 1235.00
235 03/23/2001 Totals 1235.00 1172.98
236
237That's a bit better, but we still have that last column which needs to
238be moved further over. There's an easy way to fix this up:
239unfortunately, we can't get C<pack> to right-justify our fields, but we
240can get C<sprintf> to do it:
241
242 $tot_income = sprintf("%.2f", $tot_income);
243 $tot_expend = sprintf("%12.2f", $tot_expend);
244 $date = POSIX::strftime("%m/%d/%Y", localtime);
555bd962
BG
245 print pack("A11 A28 A8 A*", $date, "Totals",
246 $tot_income, $tot_expend);
34babc16
JH
247
248This time we get the right answer:
249
250 01/28/2001 Flea spray 24.99
251 01/29/2001 Camel rides to tourists 1235.00
252 03/23/2001 Totals 1235.00 1172.98
253
254So that's how we consume and produce fixed-width data. Let's recap what
255we've seen of C<pack> and C<unpack> so far:
256
257=over 3
258
259=item *
260
261Use C<pack> to go from several pieces of data to one fixed-width
262version; use C<unpack> to turn a fixed-width-format string into several
263pieces of data.
264
265=item *
266
267The pack format C<A> means "any character"; if you're C<pack>ing and
268you've run out of things to pack, C<pack> will fill the rest up with
269spaces.
270
271=item *
272
273C<x> means "skip a byte" when C<unpack>ing; when C<pack>ing, it means
274"introduce a null byte" - that's probably not what you mean if you're
275dealing with plain text.
276
277=item *
278
279You can follow the formats with numbers to say how many characters
280should be affected by that format: C<A12> means "take 12 characters";
281C<x6> means "skip 6 bytes" or "character 0, 6 times".
282
283=item *
284
285Instead of a number, you can use C<*> to mean "consume everything else
286left".
287
288B<Warning>: when packing multiple pieces of data, C<*> only means
289"consume all of the current piece of data". That's to say
290
291 pack("A*A*", $one, $two)
292
293packs all of C<$one> into the first C<A*> and then all of C<$two> into
294the second. This is a general principle: each format character
295corresponds to one piece of data to be C<pack>ed.
296
297=back
298
299
300
301=head1 Packing Numbers
302
303So much for textual data. Let's get onto the meaty stuff that C<pack>
304and C<unpack> are best at: handling binary formats for numbers. There is,
305of course, not just one binary format - life would be too simple - but
306Perl will do all the finicky labor for you.
307
308
309=head2 Integers
310
311Packing and unpacking numbers implies conversion to and from some
312I<specific> binary representation. Leaving floating point numbers
313aside for the moment, the salient properties of any such representation
314are:
315
316=over 4
317
318=item *
319
320the number of bytes used for storing the integer,
321
322=item *
323
324whether the contents are interpreted as a signed or unsigned number,
325
326=item *
327
328the byte ordering: whether the first byte is the least or most
329significant byte (or: little-endian or big-endian, respectively).
330
331=back
332
333So, for instance, to pack 20302 to a signed 16 bit integer in your
334computer's representation you write
335
336 my $ps = pack( 's', 20302 );
337
338Again, the result is a string, now containing 2 bytes. If you print
339this string (which is, generally, not recommended) you might see
340C<ON> or C<NO> (depending on your system's byte ordering) - or something
341entirely different if your computer doesn't use ASCII character encoding.
342Unpacking C<$ps> with the same template returns the original integer value:
343
344 my( $s ) = unpack( 's', $ps );
345
346This is true for all numeric template codes. But don't expect miracles:
47b6252e 347if the packed value exceeds the allotted byte capacity, high order bits
34babc16
JH
348are silently discarded, and unpack certainly won't be able to pull them
349back out of some magic hat. And, when you pack using a signed template
350code such as C<s>, an excess value may result in the sign bit
351getting set, and unpacking this will smartly return a negative value.
352
35316 bits won't get you too far with integers, but there is C<l> and C<L>
354for signed and unsigned 32-bit integers. And if this is not enough and
355your system supports 64 bit integers you can push the limits much closer
356to infinity with pack codes C<q> and C<Q>. A notable exception is provided
357by pack codes C<i> and C<I> for signed and unsigned integers of the
358"local custom" variety: Such an integer will take up as many bytes as
359a local C compiler returns for C<sizeof(int)>, but it'll use I<at least>
36032 bits.
361
362Each of the integer pack codes C<sSlLqQ> results in a fixed number of bytes,
363no matter where you execute your program. This may be useful for some
364applications, but it does not provide for a portable way to pass data
365structures between Perl and C programs (bound to happen when you call
366XS extensions or the Perl function C<syscall>), or when you read or
367write binary files. What you'll need in this case are template codes that
368depend on what your local C compiler compiles when you code C<short> or
369C<unsigned long>, for instance. These codes and their corresponding
370byte lengths are shown in the table below. Since the C standard leaves
371much leeway with respect to the relative sizes of these data types, actual
372values may vary, and that's why the values are given as expressions in
373C and Perl. (If you'd like to use values from C<%Config> in your program
374you have to import it with C<use Config>.)
375
376 signed unsigned byte length in C byte length in Perl
f8b4d74f
WL
377 s! S! sizeof(short) $Config{shortsize}
378 i! I! sizeof(int) $Config{intsize}
379 l! L! sizeof(long) $Config{longsize}
d832b8f6 380 q! Q! sizeof(long long) $Config{longlongsize}
34babc16
JH
381
382The C<i!> and C<I!> codes aren't different from C<i> and C<I>; they are
383tolerated for completeness' sake.
384
385
386=head2 Unpacking a Stack Frame
387
388Requesting a particular byte ordering may be necessary when you work with
47f22e19 389binary data coming from some specific architecture whereas your program could
34babc16
JH
390run on a totally different system. As an example, assume you have 24 bytes
391containing a stack frame as it happens on an Intel 8086:
392
393 +---------+ +----+----+ +---------+
394 TOS: | IP | TOS+4:| FL | FH | FLAGS TOS+14:| SI |
395 +---------+ +----+----+ +---------+
396 | CS | | AL | AH | AX | DI |
397 +---------+ +----+----+ +---------+
398 | BL | BH | BX | BP |
399 +----+----+ +---------+
400 | CL | CH | CX | DS |
401 +----+----+ +---------+
402 | DL | DH | DX | ES |
403 +----+----+ +---------+
404
405First, we note that this time-honored 16-bit CPU uses little-endian order,
406and that's why the low order byte is stored at the lower address. To
9dc383df 407unpack such a (unsigned) short we'll have to use code C<v>. A repeat
34babc16
JH
408count unpacks all 12 shorts:
409
410 my( $ip, $cs, $flags, $ax, $bx, $cd, $dx, $si, $di, $bp, $ds, $es ) =
411 unpack( 'v12', $frame );
412
413Alternatively, we could have used C<C> to unpack the individually
414accessible byte registers FL, FH, AL, AH, etc.:
415
416 my( $fl, $fh, $al, $ah, $bl, $bh, $cl, $ch, $dl, $dh ) =
417 unpack( 'C10', substr( $frame, 4, 10 ) );
418
419It would be nice if we could do this in one fell swoop: unpack a short,
420back up a little, and then unpack 2 bytes. Since Perl I<is> nice, it
421proffers the template code C<X> to back up one byte. Putting this all
422together, we may now write:
423
424 my( $ip, $cs,
425 $flags,$fl,$fh,
426 $ax,$al,$ah, $bx,$bl,$bh, $cx,$cl,$ch, $dx,$dl,$dh,
427 $si, $di, $bp, $ds, $es ) =
428 unpack( 'v2' . ('vXXCC' x 5) . 'v5', $frame );
429
49704364
LW
430(The clumsy construction of the template can be avoided - just read on!)
431
47f22e19 432We've taken some pains to construct the template so that it matches
34babc16
JH
433the contents of our frame buffer. Otherwise we'd either get undefined values,
434or C<unpack> could not unpack all. If C<pack> runs out of items, it will
47f22e19
WL
435supply null strings (which are coerced into zeroes whenever the pack code
436says so).
34babc16
JH
437
438
439=head2 How to Eat an Egg on a Net
440
441The pack code for big-endian (high order byte at the lowest address) is
442C<n> for 16 bit and C<N> for 32 bit integers. You use these codes
443if you know that your data comes from a compliant architecture, but,
444surprisingly enough, you should also use these pack codes if you
445exchange binary data, across the network, with some system that you
446know next to nothing about. The simple reason is that this
447order has been chosen as the I<network order>, and all standard-fearing
448programs ought to follow this convention. (This is, of course, a stern
449backing for one of the Lilliputian parties and may well influence the
450political development there.) So, if the protocol expects you to send
451a message by sending the length first, followed by just so many bytes,
452you could write:
453
454 my $buf = pack( 'N', length( $msg ) ) . $msg;
455
456or even:
457
458 my $buf = pack( 'NA*', length( $msg ), $msg );
459
460and pass C<$buf> to your send routine. Some protocols demand that the
461count should include the length of the count itself: then just add 4
5a0de581 462to the data length. (But make sure to read L</"Lengths and Widths"> before
34babc16
JH
463you really code this!)
464
465
59f20ca5
MHM
466=head2 Byte-order modifiers
467
468In the previous sections we've learned how to use C<n>, C<N>, C<v> and
469C<V> to pack and unpack integers with big- or little-endian byte-order.
470While this is nice, it's still rather limited because it leaves out all
471kinds of signed integers as well as 64-bit integers. For example, if you
472wanted to unpack a sequence of signed big-endian 16-bit integers in a
473platform-independent way, you would have to write:
474
475 my @data = unpack 's*', pack 'S*', unpack 'n*', $buf;
476
c4ecfaf1 477This is ugly. As of Perl 5.9.2, there's a much nicer way to express your
59f20ca5
MHM
478desire for a certain byte-order: the C<E<gt>> and C<E<lt>> modifiers.
479C<E<gt>> is the big-endian modifier, while C<E<lt>> is the little-endian
480modifier. Using them, we could rewrite the above code as:
481
482 my @data = unpack 's>*', $buf;
483
484As you can see, the "big end" of the arrow touches the C<s>, which is a
485nice way to remember that C<E<gt>> is the big-endian modifier. The same
486obviously works for C<E<lt>>, where the "little end" touches the code.
487
488You will probably find these modifiers even more useful if you have
489to deal with big- or little-endian C structures. Be sure to read
5a0de581 490L</"Packing and Unpacking C Structures"> for more on that.
59f20ca5 491
34babc16
JH
492
493=head2 Floating point Numbers
494
495For packing floating point numbers you have the choice between the
59f20ca5
MHM
496pack codes C<f>, C<d>, C<F> and C<D>. C<f> and C<d> pack into (or unpack
497from) single-precision or double-precision representation as it is provided
498by your system. If your systems supports it, C<D> can be used to pack and
aacf4ea2
JH
499unpack (C<long double>) values, which can offer even more resolution
500than C<f> or C<d>. B<Note that there are different long double formats.>
501
502C<F> packs an C<NV>, which is the floating point type used by Perl
503internally.
504
505There is no such thing as a network representation for reals, so if
506you want to send your real numbers across computer boundaries, you'd
507better stick to text representation, possibly using the hexadecimal
508float format (avoiding the decimal conversion loss), unless you're
509absolutely sure what's on the other end of the line. For the even more
510adventuresome, you can use the byte-order modifiers from the previous
511section also on floating point codes.
34babc16
JH
512
513
514
515=head1 Exotic Templates
516
517
518=head2 Bit Strings
519
520Bits are the atoms in the memory world. Access to individual bits may
521have to be used either as a last resort or because it is the most
522convenient way to handle your data. Bit string (un)packing converts
523between strings containing a series of C<0> and C<1> characters and
524a sequence of bytes each containing a group of 8 bits. This is almost
525as simple as it sounds, except that there are two ways the contents of
526a byte may be written as a bit string. Let's have a look at an annotated
527byte:
528
529 7 6 5 4 3 2 1 0
530 +-----------------+
531 | 1 0 0 0 1 1 0 0 |
532 +-----------------+
533 MSB LSB
534
535It's egg-eating all over again: Some think that as a bit string this should
536be written "10001100" i.e. beginning with the most significant bit, others
537insist on "00110001". Well, Perl isn't biased, so that's why we have two bit
538string codes:
539
540 $byte = pack( 'B8', '10001100' ); # start with MSB
541 $byte = pack( 'b8', '00110001' ); # start with LSB
542
543It is not possible to pack or unpack bit fields - just integral bytes.
544C<pack> always starts at the next byte boundary and "rounds up" to the
545next multiple of 8 by adding zero bits as required. (If you do want bit
546fields, there is L<perlfunc/vec>. Or you could implement bit field
547handling at the character string level, using split, substr, and
548concatenation on unpacked bit strings.)
549
550To illustrate unpacking for bit strings, we'll decompose a simple
551status register (a "-" stands for a "reserved" bit):
552
553 +-----------------+-----------------+
554 | S Z - A - P - C | - - - - O D I T |
555 +-----------------+-----------------+
556 MSB LSB MSB LSB
557
558Converting these two bytes to a string can be done with the unpack
559template C<'b16'>. To obtain the individual bit values from the bit
47f22e19 560string we use C<split> with the "empty" separator pattern which dissects
34babc16
JH
561into individual characters. Bit values from the "reserved" positions are
562simply assigned to C<undef>, a convenient notation for "I don't care where
563this goes".
564
49704364 565 ($carry, undef, $parity, undef, $auxcarry, undef, $zero, $sign,
34babc16 566 $trace, $interrupt, $direction, $overflow) =
47f22e19 567 split( //, unpack( 'b16', $status ) );
34babc16
JH
568
569We could have used an unpack template C<'b12'> just as well, since the
570last 4 bits can be ignored anyway.
571
572
573=head2 Uuencoding
574
99f65d01 575Another odd-man-out in the template alphabet is C<u>, which packs a
34babc16
JH
576"uuencoded string". ("uu" is short for Unix-to-Unix.) Chances are that
577you won't ever need this encoding technique which was invented to overcome
578the shortcomings of old-fashioned transmission mediums that do not support
579other than simple ASCII data. The essential recipe is simple: Take three
580bytes, or 24 bits. Split them into 4 six-packs, adding a space (0x20) to
581each. Repeat until all of the data is blended. Fold groups of 4 bytes into
582lines no longer than 60 and garnish them in front with the original byte count
583(incremented by 0x20) and a C<"\n"> at the end. - The C<pack> chef will
584prepare this for you, a la minute, when you select pack code C<u> on the menu:
585
586 my $uubuf = pack( 'u', $bindat );
587
588A repeat count after C<u> sets the number of bytes to put into an
589uuencoded line, which is the maximum of 45 by default, but could be
590set to some (smaller) integer multiple of three. C<unpack> simply ignores
591the repeat count.
592
593
594=head2 Doing Sums
595
596An even stranger template code is C<%>E<lt>I<number>E<gt>. First, because
597it's used as a prefix to some other template code. Second, because it
598cannot be used in C<pack> at all, and third, in C<unpack>, doesn't return the
599data as defined by the template code it precedes. Instead it'll give you an
600integer of I<number> bits that is computed from the data value by
601doing sums. For numeric unpack codes, no big feat is achieved:
602
603 my $buf = pack( 'iii', 100, 20, 3 );
604 print unpack( '%32i3', $buf ), "\n"; # prints 123
605
606For string values, C<%> returns the sum of the byte values saving
607you the trouble of a sum loop with C<substr> and C<ord>:
608
609 print unpack( '%32A*', "\x01\x10" ), "\n"; # prints 17
610
611Although the C<%> code is documented as returning a "checksum":
612don't put your trust in such values! Even when applied to a small number
613of bytes, they won't guarantee a noticeable Hamming distance.
614
615In connection with C<b> or C<B>, C<%> simply adds bits, and this can be put
616to good use to count set bits efficiently:
617
618 my $bitcount = unpack( '%32b*', $mask );
619
620And an even parity bit can be determined like this:
621
622 my $evenparity = unpack( '%1b*', $mask );
623
624
625=head2 Unicode
626
627Unicode is a character set that can represent most characters in most of
628the world's languages, providing room for over one million different
629characters. Unicode 3.1 specifies 94,140 characters: The Basic Latin
630characters are assigned to the numbers 0 - 127. The Latin-1 Supplement with
631characters that are used in several European languages is in the next
632range, up to 255. After some more Latin extensions we find the character
47b6252e 633sets from languages using non-Roman alphabets, interspersed with a
34babc16 634variety of symbol sets such as currency symbols, Zapf Dingbats or Braille.
71c89d21 635(You might want to visit L<https://www.unicode.org/> for a look at some of
34babc16
JH
636them - my personal favourites are Telugu and Kannada.)
637
638The Unicode character sets associates characters with integers. Encoding
639these numbers in an equal number of bytes would more than double the
47b6252e 640requirements for storing texts written in Latin alphabets.
34babc16
JH
641The UTF-8 encoding avoids this by storing the most common (from a western
642point of view) characters in a single byte while encoding the rarer
643ones in three or more bytes.
644
2575c402
JW
645Perl uses UTF-8, internally, for most Unicode strings.
646
647So what has this got to do with C<pack>? Well, if you want to compose a
648Unicode string (that is internally encoded as UTF-8), you can do so by
649using template code C<U>. As an example, let's produce the Euro currency
650symbol (code number 0x20AC):
34babc16
JH
651
652 $UTF8{Euro} = pack( 'U', 0x20AC );
2575c402 653 # Equivalent to: $UTF8{Euro} = "\x{20ac}";
34babc16 654
2575c402
JW
655Inspecting C<$UTF8{Euro}> shows that it contains 3 bytes:
656"\xe2\x82\xac". However, it contains only 1 character, number 0x20AC.
657The round trip can be completed with C<unpack>:
34babc16
JH
658
659 $Unicode{Euro} = unpack( 'U', $UTF8{Euro} );
660
2575c402
JW
661Unpacking using the C<U> template code also works on UTF-8 encoded byte
662strings.
663
34babc16
JH
664Usually you'll want to pack or unpack UTF-8 strings:
665
666 # pack and unpack the Hebrew alphabet
667 my $alefbet = pack( 'U*', 0x05d0..0x05ea );
668 my @hebrew = unpack( 'U*', $utf );
669
2575c402 670Please note: in the general case, you're better off using
8e179dd8
P
671L<C<Encode::decode('UTF-8', $utf)>|Encode/decode> to decode a UTF-8
672encoded byte string to a Perl Unicode string, and
673L<C<Encode::encode('UTF-8', $str)>|Encode/encode> to encode a Perl Unicode
674string to UTF-8 bytes. These functions provide means of handling invalid byte
2575c402 675sequences and generally have a friendlier interface.
34babc16 676
47f22e19
WL
677=head2 Another Portable Binary Encoding
678
679The pack code C<w> has been added to support a portable binary data
680encoding scheme that goes way beyond simple integers. (Details can
f979aebc 681be found at L<http://Casbah.org/>, the Scarab project.) A BER (Binary Encoded
47f22e19
WL
682Representation) compressed unsigned integer stores base 128
683digits, most significant digit first, with as few digits as possible.
684Bit eight (the high bit) is set on each byte except the last. There
685is no size limit to BER encoding, but Perl won't go to extremes.
686
687 my $berbuf = pack( 'w*', 1, 128, 128+1, 128*128+127 );
688
689A hex dump of C<$berbuf>, with spaces inserted at the right places,
690shows 01 8100 8101 81807F. Since the last byte is always less than
691128, C<unpack> knows where to stop.
692
34babc16 693
49704364
LW
694=head1 Template Grouping
695
696Prior to Perl 5.8, repetitions of templates had to be made by
697C<x>-multiplication of template strings. Now there is a better way as
698we may use the pack codes C<(> and C<)> combined with a repeat count.
699The C<unpack> template from the Stack Frame example can simply
700be written like this:
701
702 unpack( 'v2 (vXXCC)5 v5', $frame )
703
704Let's explore this feature a little more. We'll begin with the equivalent of
705
706 join( '', map( substr( $_, 0, 1 ), @str ) )
707
708which returns a string consisting of the first character from each string.
709Using pack, we can write
710
711 pack( '(A)'.@str, @str )
712
713or, because a repeat count C<*> means "repeat as often as required",
714simply
715
716 pack( '(A)*', @str )
717
718(Note that the template C<A*> would only have packed C<$str[0]> in full
719length.)
ffc145e8 720
49704364
LW
721To pack dates stored as triplets ( day, month, year ) in an array C<@dates>
722into a sequence of byte, byte, short integer we can write
723
724 $pd = pack( '(CCS)*', map( @$_, @dates ) );
725
726To swap pairs of characters in a string (with even length) one could use
727several techniques. First, let's use C<x> and C<X> to skip forward and back:
728
729 $s = pack( '(A)*', unpack( '(xAXXAx)*', $s ) );
730
731We can also use C<@> to jump to an offset, with 0 being the position where
732we were when the last C<(> was encountered:
733
734 $s = pack( '(A)*', unpack( '(@1A @0A @2)*', $s ) );
735
736Finally, there is also an entirely different approach by unpacking big
737endian shorts and packing them in the reverse byte order:
738
739 $s = pack( '(v)*', unpack( '(n)*', $s );
740
741
34babc16
JH
742=head1 Lengths and Widths
743
744=head2 String Lengths
745
746In the previous section we've seen a network message that was constructed
747by prefixing the binary message length to the actual message. You'll find
748that packing a length followed by so many bytes of data is a
749frequently used recipe since appending a null byte won't work
47b6252e 750if a null byte may be part of the data. Here is an example where both
34babc16
JH
751techniques are used: after two null terminated strings with source and
752destination address, a Short Message (to a mobile phone) is sent after
753a length byte:
754
755 my $msg = pack( 'Z*Z*CA*', $src, $dst, length( $sm ), $sm );
756
757Unpacking this message can be done with the same template:
758
759 ( $src, $dst, $len, $sm ) = unpack( 'Z*Z*CA*', $msg );
760
47b6252e 761There's a subtle trap lurking in the offing: Adding another field after
34babc16
JH
762the Short Message (in variable C<$sm>) is all right when packing, but this
763cannot be unpacked naively:
764
765 # pack a message
766 my $msg = pack( 'Z*Z*CA*C', $src, $dst, length( $sm ), $sm, $prio );
7207e29d 767
34babc16
JH
768 # unpack fails - $prio remains undefined!
769 ( $src, $dst, $len, $sm, $prio ) = unpack( 'Z*Z*CA*C', $msg );
770
771The pack code C<A*> gobbles up all remaining bytes, and C<$prio> remains
772undefined! Before we let disappointment dampen the morale: Perl's got
773the trump card to make this trick too, just a little further up the sleeve.
774Watch this:
775
776 # pack a message: ASCIIZ, ASCIIZ, length/string, byte
777 my $msg = pack( 'Z* Z* C/A* C', $src, $dst, $sm, $prio );
778
779 # unpack
780 ( $src, $dst, $sm, $prio ) = unpack( 'Z* Z* C/A* C', $msg );
781
782Combining two pack codes with a slash (C</>) associates them with a single
783value from the argument list. In C<pack>, the length of the argument is
784taken and packed according to the first code while the argument itself
785is added after being converted with the template code after the slash.
786This saves us the trouble of inserting the C<length> call, but it is
787in C<unpack> where we really score: The value of the length byte marks the
788end of the string to be taken from the buffer. Since this combination
f8b4d74f 789doesn't make sense except when the second pack code isn't C<a*>, C<A*>
34babc16
JH
790or C<Z*>, Perl won't let you.
791
792The pack code preceding C</> may be anything that's fit to represent a
793number: All the numeric binary pack codes, and even text codes such as
794C<A4> or C<Z*>:
795
796 # pack/unpack a string preceded by its length in ASCII
797 my $buf = pack( 'A4/A*', "Humpty-Dumpty" );
798 # unpack $buf: '13 Humpty-Dumpty'
799 my $txt = unpack( 'A4/A*', $buf );
800
47b6252e 801C</> is not implemented in Perls before 5.6, so if your code is required to
7b0ac457 802work on ancient Perls you'll need to C<unpack( 'Z* Z* C')> to get the length,
47b6252e
NC
803then use it to make a new unpack string. For example
804
555bd962
BG
805 # pack a message: ASCIIZ, ASCIIZ, length, string, byte
806 # (5.005 compatible)
47b6252e
NC
807 my $msg = pack( 'Z* Z* C A* C', $src, $dst, length $sm, $sm, $prio );
808
809 # unpack
810 ( undef, undef, $len) = unpack( 'Z* Z* C', $msg );
811 ($src, $dst, $sm, $prio) = unpack ( "Z* Z* x A$len C", $msg );
812
813But that second C<unpack> is rushing ahead. It isn't using a simple literal
814string for the template. So maybe we should introduce...
34babc16
JH
815
816=head2 Dynamic Templates
817
818So far, we've seen literals used as templates. If the list of pack
819items doesn't have fixed length, an expression constructing the
49704364
LW
820template is required (whenever, for some reason, C<()*> cannot be used).
821Here's an example: To store named string values in a way that can be
822conveniently parsed by a C program, we create a sequence of names and
823null terminated ASCII strings, with C<=> between the name and the value,
824followed by an additional delimiting null byte. Here's how:
34babc16 825
49704364 826 my $env = pack( '(A*A*Z*)' . keys( %Env ) . 'C',
47f22e19
WL
827 map( { ( $_, '=', $Env{$_} ) } keys( %Env ) ), 0 );
828
829Let's examine the cogs of this byte mill, one by one. There's the C<map>
830call, creating the items we intend to stuff into the C<$env> buffer:
831to each key (in C<$_>) it adds the C<=> separator and the hash entry value.
832Each triplet is packed with the template code sequence C<A*A*Z*> that
49704364 833is repeated according to the number of keys. (Yes, that's what the C<keys>
fe854a6f 834function returns in scalar context.) To get the very last null byte,
47f22e19
WL
835we add a C<0> at the end of the C<pack> list, to be packed with C<C>.
836(Attentive readers may have noticed that we could have omitted the 0.)
34babc16
JH
837
838For the reverse operation, we'll have to determine the number of items
839in the buffer before we can let C<unpack> rip it apart:
840
47b6252e 841 my $n = $env =~ tr/\0// - 1;
49704364 842 my %env = map( split( /=/, $_ ), unpack( "(Z*)$n", $env ) );
34babc16 843
47b6252e 844The C<tr> counts the null bytes. The C<unpack> call returns a list of
47f22e19 845name-value pairs each of which is taken apart in the C<map> block.
34babc16
JH
846
847
49704364
LW
848=head2 Counting Repetitions
849
850Rather than storing a sentinel at the end of a data item (or a list of items),
851we could precede the data with a count. Again, we pack keys and values of
852a hash, preceding each with an unsigned short length count, and up front
853we store the number of pairs:
854
855 my $env = pack( 'S(S/A* S/A*)*', scalar keys( %Env ), %Env );
856
857This simplifies the reverse operation as the number of repetitions can be
858unpacked with the C</> code:
859
860 my %env = unpack( 'S/(S/A* S/A*)', $env );
861
862Note that this is one of the rare cases where you cannot use the same
863template for C<pack> and C<unpack> because C<pack> can't determine
864a repeat count for a C<()>-group.
865
866
aa51dd41
MB
867=head2 Intel HEX
868
869Intel HEX is a file format for representing binary data, mostly for
870programming various chips, as a text file. (See
71c89d21
MM
871L<https://en.wikipedia.org/wiki/.hex> for a detailed description, and
872L<https://en.wikipedia.org/wiki/SREC_(file_format)> for the Motorola
aa51dd41
MB
873S-record format, which can be unravelled using the same technique.)
874Each line begins with a colon (':') and is followed by a sequence of
875hexadecimal characters, specifying a byte count I<n> (8 bit),
876an address (16 bit, big endian), a record type (8 bit), I<n> data bytes
877and a checksum (8 bit) computed as the least significant byte of the two's
878complement sum of the preceding bytes. Example: C<:0300300002337A1E>.
879
880The first step of processing such a line is the conversion, to binary,
881of the hexadecimal data, to obtain the four fields, while checking the
882checksum. No surprise here: we'll start with a simple C<pack> call to
883convert everything to binary:
884
885 my $binrec = pack( 'H*', substr( $hexrec, 1 ) );
886
887The resulting byte sequence is most convenient for checking the checksum.
888Don't slow your program down with a for loop adding the C<ord> values
889of this string's bytes - the C<unpack> code C<%> is the thing to use
890for computing the 8-bit sum of all bytes, which must be equal to zero:
891
892 die unless unpack( "%8C*", $binrec ) == 0;
893
894Finally, let's get those four fields. By now, you shouldn't have any
895problems with the first three fields - but how can we use the byte count
896of the data in the first field as a length for the data field? Here
897the codes C<x> and C<X> come to the rescue, as they permit jumping
898back and forth in the string to unpack.
899
900 my( $addr, $type, $data ) = unpack( "x n C X4 C x3 /a", $bin );
901
902Code C<x> skips a byte, since we don't need the count yet. Code C<n> takes
903care of the 16-bit big-endian integer address, and C<C> unpacks the
904record type. Being at offset 4, where the data begins, we need the count.
905C<X4> brings us back to square one, which is the byte at offset 0.
906Now we pick up the count, and zoom forth to offset 4, where we are
907now fully furnished to extract the exact number of data bytes, leaving
908the trailing checksum byte alone.
909
910
911
34babc16
JH
912=head1 Packing and Unpacking C Structures
913
914In previous sections we have seen how to pack numbers and character
915strings. If it were not for a couple of snags we could conclude this
916section right away with the terse remark that C structures don't
917contain anything else, and therefore you already know all there is to it.
918Sorry, no: read on, please.
919
59f20ca5
MHM
920If you have to deal with a lot of C structures, and don't want to
921hack all your template strings manually, you'll probably want to have
922a look at the CPAN module C<Convert::Binary::C>. Not only can it parse
923your C source directly, but it also has built-in support for all the
924odds and ends described further on in this section.
925
34babc16
JH
926=head2 The Alignment Pit
927
928In the consideration of speed against memory requirements the balance
929has been tilted in favor of faster execution. This has influenced the
930way C compilers allocate memory for structures: On architectures
931where a 16-bit or 32-bit operand can be moved faster between places in
932memory, or to or from a CPU register, if it is aligned at an even or
933multiple-of-four or even at a multiple-of eight address, a C compiler
934will give you this speed benefit by stuffing extra bytes into structures.
935If you don't cross the C shoreline this is not likely to cause you any
47b6252e
NC
936grief (although you should care when you design large data structures,
937or you want your code to be portable between architectures (you do want
938that, don't you?)).
34babc16
JH
939
940To see how this affects C<pack> and C<unpack>, we'll compare these two
941C structures:
942
943 typedef struct {
944 char c1;
945 short s;
946 char c2;
947 long l;
948 } gappy_t;
949
950 typedef struct {
951 long l;
952 short s;
953 char c1;
954 char c2;
955 } dense_t;
956
957Typically, a C compiler allocates 12 bytes to a C<gappy_t> variable, but
958requires only 8 bytes for a C<dense_t>. After investigating this further,
959we can draw memory maps, showing where the extra 4 bytes are hidden:
960
961 0 +4 +8 +12
962 +--+--+--+--+--+--+--+--+--+--+--+--+
963 |c1|xx| s |c2|xx|xx|xx| l | xx = fill byte
964 +--+--+--+--+--+--+--+--+--+--+--+--+
965 gappy_t
966
967 0 +4 +8
968 +--+--+--+--+--+--+--+--+
969 | l | h |c1|c2|
970 +--+--+--+--+--+--+--+--+
971 dense_t
972
973And that's where the first quirk strikes: C<pack> and C<unpack>
974templates have to be stuffed with C<x> codes to get those extra fill bytes.
975
976The natural question: "Why can't Perl compensate for the gaps?" warrants
977an answer. One good reason is that C compilers might provide (non-ANSI)
978extensions permitting all sorts of fancy control over the way structures
979are aligned, even at the level of an individual structure field. And, if
980this were not enough, there is an insidious thing called C<union> where
981the amount of fill bytes cannot be derived from the alignment of the next
982item alone.
983
984OK, so let's bite the bullet. Here's one way to get the alignment right
985by inserting template codes C<x>, which don't take a corresponding item
986from the list:
987
988 my $gappy = pack( 'cxs cxxx l!', $c1, $s, $c2, $l );
989
990Note the C<!> after C<l>: We want to make sure that we pack a long
47b6252e
NC
991integer as it is compiled by our C compiler. And even now, it will only
992work for the platforms where the compiler aligns things as above.
993And somebody somewhere has a platform where it doesn't.
994[Probably a Cray, where C<short>s, C<int>s and C<long>s are all 8 bytes. :-)]
34babc16
JH
995
996Counting bytes and watching alignments in lengthy structures is bound to
997be a drag. Isn't there a way we can create the template with a simple
998program? Here's a C program that does the trick:
999
1000 #include <stdio.h>
1001 #include <stddef.h>
1002
1003 typedef struct {
1004 char fc1;
1005 short fs;
1006 char fc2;
1007 long fl;
1008 } gappy_t;
1009
1010 #define Pt(struct,field,tchar) \
1011 printf( "@%d%s ", offsetof(struct,field), # tchar );
1012
d832b8f6 1013 int main() {
34babc16
JH
1014 Pt( gappy_t, fc1, c );
1015 Pt( gappy_t, fs, s! );
1016 Pt( gappy_t, fc2, c );
1017 Pt( gappy_t, fl, l! );
1018 printf( "\n" );
1019 }
1020
1021The output line can be used as a template in a C<pack> or C<unpack> call:
1022
1023 my $gappy = pack( '@0c @2s! @4c @8l!', $c1, $s, $c2, $l );
1024
1025Gee, yet another template code - as if we hadn't plenty. But
1026C<@> saves our day by enabling us to specify the offset from the beginning
1027of the pack buffer to the next item: This is just the value
1028the C<offsetof> macro (defined in C<E<lt>stddef.hE<gt>>) returns when
1029given a C<struct> type and one of its field names ("member-designator" in
1030C standardese).
1031
49704364
LW
1032Neither using offsets nor adding C<x>'s to bridge the gaps is satisfactory.
1033(Just imagine what happens if the structure changes.) What we really need
1034is a way of saying "skip as many bytes as required to the next multiple of N".
1035In fluent Templatese, you say this with C<x!N> where N is replaced by the
1036appropriate value. Here's the next version of our struct packaging:
1037
1038 my $gappy = pack( 'c x!2 s c x!4 l!', $c1, $s, $c2, $l );
1039
1040That's certainly better, but we still have to know how long all the
1041integers are, and portability is far away. Rather than C<2>,
1042for instance, we want to say "however long a short is". But this can be
1043done by enclosing the appropriate pack code in brackets: C<[s]>. So, here's
1044the very best we can do:
1045
1046 my $gappy = pack( 'c x![s] s c x![l!] l!', $c1, $s, $c2, $l );
1047
34babc16 1048
59f20ca5
MHM
1049=head2 Dealing with Endian-ness
1050
1051Now, imagine that we want to pack the data for a machine with a
1052different byte-order. First, we'll have to figure out how big the data
1053types on the target machine really are. Let's assume that the longs are
105432 bits wide and the shorts are 16 bits wide. You can then rewrite the
1055template as:
1056
1057 my $gappy = pack( 'c x![s] s c x![l] l', $c1, $s, $c2, $l );
1058
1059If the target machine is little-endian, we could write:
1060
1061 my $gappy = pack( 'c x![s] s< c x![l] l<', $c1, $s, $c2, $l );
1062
1063This forces the short and the long members to be little-endian, and is
1064just fine if you don't have too many struct members. But we could also
1065use the byte-order modifier on a group and write the following:
1066
1067 my $gappy = pack( '( c x![s] s c x![l] l )<', $c1, $s, $c2, $l );
1068
1069This is not as short as before, but it makes it more obvious that we
1070intend to have little-endian byte-order for a whole group, not only
1071for individual template codes. It can also be more readable and easier
1072to maintain.
1073
1074
34babc16
JH
1075=head2 Alignment, Take 2
1076
1077I'm afraid that we're not quite through with the alignment catch yet. The
1078hydra raises another ugly head when you pack arrays of structures:
1079
1080 typedef struct {
1081 short count;
1082 char glyph;
1083 } cell_t;
1084
1085 typedef cell_t buffer_t[BUFLEN];
1086
1087Where's the catch? Padding is neither required before the first field C<count>,
1088nor between this and the next field C<glyph>, so why can't we simply pack
1089like this:
1090
1091 # something goes wrong here:
1092 pack( 's!a' x @buffer,
1093 map{ ( $_->{count}, $_->{glyph} ) } @buffer );
1094
1095This packs C<3*@buffer> bytes, but it turns out that the size of
1096C<buffer_t> is four times C<BUFLEN>! The moral of the story is that
1097the required alignment of a structure or array is propagated to the
1098next higher level where we have to consider padding I<at the end>
1099of each component as well. Thus the correct template is:
1100
1101 pack( 's!ax' x @buffer,
1102 map{ ( $_->{count}, $_->{glyph} ) } @buffer );
1103
47b6252e
NC
1104=head2 Alignment, Take 3
1105
1106And even if you take all the above into account, ANSI still lets this:
1107
1108 typedef struct {
1109 char foo[2];
1110 } foo_t;
34babc16 1111
47b6252e
NC
1112vary in size. The alignment constraint of the structure can be greater than
1113any of its elements. [And if you think that this doesn't affect anything
1114common, dismember the next cellphone that you see. Many have ARM cores, and
1115the ARM structure rules make C<sizeof (foo_t)> == 4]
34babc16
JH
1116
1117=head2 Pointers for How to Use Them
1118
1119The title of this section indicates the second problem you may run into
1120sooner or later when you pack C structures. If the function you intend
1121to call expects a, say, C<void *> value, you I<cannot> simply take
1122a reference to a Perl variable. (Although that value certainly is a
1123memory address, it's not the address where the variable's contents are
1124stored.)
1125
1126Template code C<P> promises to pack a "pointer to a fixed length string".
1127Isn't this what we want? Let's try:
1128
1129 # allocate some storage and pack a pointer to it
1130 my $memory = "\x00" x $size;
1131 my $memptr = pack( 'P', $memory );
1132
1133But wait: doesn't C<pack> just return a sequence of bytes? How can we pass this
1134string of bytes to some C code expecting a pointer which is, after all,
1135nothing but a number? The answer is simple: We have to obtain the numeric
1136address from the bytes returned by C<pack>.
1137
1138 my $ptr = unpack( 'L!', $memptr );
1139
1140Obviously this assumes that it is possible to typecast a pointer
1141to an unsigned long and vice versa, which frequently works but should not
1142be taken as a universal law. - Now that we have this pointer the next question
1143is: How can we put it to good use? We need a call to some C function
1144where a pointer is expected. The read(2) system call comes to mind:
1145
1146 ssize_t read(int fd, void *buf, size_t count);
1147
1148After reading L<perlfunc> explaining how to use C<syscall> we can write
1149this Perl function copying a file to standard output:
1150
b98b4a71 1151 require 'syscall.ph'; # run h2ph to generate this file
34babc16
JH
1152 sub cat($){
1153 my $path = shift();
1154 my $size = -s $path;
1155 my $memory = "\x00" x $size; # allocate some memory
1156 my $ptr = unpack( 'L', pack( 'P', $memory ) );
1157 open( F, $path ) || die( "$path: cannot open ($!)\n" );
1158 my $fd = fileno(F);
1159 my $res = syscall( &SYS_read, fileno(F), $ptr, $size );
1160 print $memory;
1161 close( F );
1162 }
1163
1164This is neither a specimen of simplicity nor a paragon of portability but
1165it illustrates the point: We are able to sneak behind the scenes and
1166access Perl's otherwise well-guarded memory! (Important note: Perl's
1167C<syscall> does I<not> require you to construct pointers in this roundabout
1168way. You simply pass a string variable, and Perl forwards the address.)
1169
1170How does C<unpack> with C<P> work? Imagine some pointer in the buffer
1171about to be unpacked: If it isn't the null pointer (which will smartly
1172produce the C<undef> value) we have a start address - but then what?
1173Perl has no way of knowing how long this "fixed length string" is, so
1174it's up to you to specify the actual size as an explicit length after C<P>.
1175
1176 my $mem = "abcdefghijklmn";
1177 print unpack( 'P5', pack( 'P', $mem ) ); # prints "abcde"
1178
1179As a consequence, C<pack> ignores any number or C<*> after C<P>.
1180
1181
1182Now that we have seen C<P> at work, we might as well give C<p> a whirl.
1183Why do we need a second template code for packing pointers at all? The
1184answer lies behind the simple fact that an C<unpack> with C<p> promises
1185a null-terminated string starting at the address taken from the buffer,
1186and that implies a length for the data item to be returned:
1187
1188 my $buf = pack( 'p', "abc\x00efhijklmn" );
1189 print unpack( 'p', $buf ); # prints "abc"
1190
1191
1192
1193Albeit this is apt to be confusing: As a consequence of the length being
1194implied by the string's length, a number after pack code C<p> is a repeat
1195count, not a length as after C<P>.
1196
1197
1198Using C<pack(..., $x)> with C<P> or C<p> to get the address where C<$x> is
1199actually stored must be used with circumspection. Perl's internal machinery
1200considers the relation between a variable and that address as its very own
1201private matter and doesn't really care that we have obtained a copy. Therefore:
1202
1203=over 4
1204
1205=item *
1206
1207Do not use C<pack> with C<p> or C<P> to obtain the address of variable
1208that's bound to go out of scope (and thereby freeing its memory) before you
1209are done with using the memory at that address.
1210
1211=item *
1212
1213Be very careful with Perl operations that change the value of the
1214variable. Appending something to the variable, for instance, might require
1215reallocation of its storage, leaving you with a pointer into no-man's land.
1216
1217=item *
1218
1219Don't think that you can get the address of a Perl variable
1220when it is stored as an integer or double number! C<pack('P', $x)> will
1221force the variable's internal representation to string, just as if you
1222had written something like C<$x .= ''>.
1223
1224=back
1225
1226It's safe, however, to P- or p-pack a string literal, because Perl simply
1227allocates an anonymous variable.
1228
1229
1230
1231=head1 Pack Recipes
1232
1233Here are a collection of (possibly) useful canned recipes for C<pack>
1234and C<unpack>:
1235
1236 # Convert IP address for socket functions
1237 pack( "C4", split /\./, "123.4.5.6" );
1238
1239 # Count the bits in a chunk of memory (e.g. a select vector)
1240 unpack( '%32b*', $mask );
1241
1242 # Determine the endianness of your system
1243 $is_little_endian = unpack( 'c', pack( 's', 1 ) );
1244 $is_big_endian = unpack( 'xc', pack( 's', 1 ) );
1245
1246 # Determine the number of bits in a native integer
1247 $bits = unpack( '%32I!', ~0 );
1248
1249 # Prepare argument for the nanosleep system call
1250 my $timespec = pack( 'L!L!', $secs, $nanosecs );
1251
f8b4d74f
WL
1252For a simple memory dump we unpack some bytes into just as
1253many pairs of hex digits, and use C<map> to handle the traditional
1254spacing - 16 bytes to a line:
1255
34babc16 1256 my $i;
49704364
LW
1257 print map( ++$i % 16 ? "$_ " : "$_\n",
1258 unpack( 'H2' x length( $mem ), $mem ) ),
f8b4d74f 1259 length( $mem ) % 16 ? "\n" : '';
34babc16
JH
1260
1261
47f22e19
WL
1262=head1 Funnies Section
1263
1264 # Pulling digits out of nowhere...
1265 print unpack( 'C', pack( 'x' ) ),
1266 unpack( '%B*', pack( 'A' ) ),
1267 unpack( 'H', pack( 'A' ) ),
1268 unpack( 'A', unpack( 'C', pack( 'A' ) ) ), "\n";
1269
1270 # One for the road ;-)
1271 my $advice = pack( 'all u can in a van' );
1272
1273
34babc16
JH
1274=head1 Authors
1275
1276Simon Cozens and Wolfgang Laun.
1277