| 1 | =head1 NAME |
| 2 | |
| 3 | perlpacktut - tutorial on C<pack> and C<unpack> |
| 4 | |
| 5 | =head1 DESCRIPTION |
| 6 | |
| 7 | C<pack> and C<unpack> are two functions for transforming data according |
| 8 | to a user-defined template, between the guarded way Perl stores values |
| 9 | and some well-defined representation as might be required in the |
| 10 | environment of a Perl program. Unfortunately, they're also two of |
| 11 | the most misunderstood and most often overlooked functions that Perl |
| 12 | provides. This tutorial will demystify them for you. |
| 13 | |
| 14 | |
| 15 | =head1 The Basic Principle |
| 16 | |
| 17 | Most programming languages don't shelter the memory where variables are |
| 18 | stored. In C, for instance, you can take the address of some variable, |
| 19 | and the C<sizeof> operator tells you how many bytes are allocated to |
| 20 | the variable. Using the address and the size, you may access the storage |
| 21 | to your heart's content. |
| 22 | |
| 23 | In Perl, you just can't access memory at random, but the structural and |
| 24 | representational conversion provided by C<pack> and C<unpack> is an |
| 25 | excellent alternative. The C<pack> function converts values to a byte |
| 26 | sequence containing representations according to a given specification, |
| 27 | the so-called "template" argument. C<unpack> is the reverse process, |
| 28 | deriving some values from the contents of a string of bytes. (Be cautioned, |
| 29 | however, that not all that has been packed together can be neatly unpacked - |
| 30 | a very common experience as seasoned travellers are likely to confirm.) |
| 31 | |
| 32 | Why, you may ask, would you need a chunk of memory containing some values |
| 33 | in binary representation? One good reason is input and output accessing |
| 34 | some file, a device, or a network connection, whereby this binary |
| 35 | representation is either forced on you or will give you some benefit |
| 36 | in processing. Another cause is passing data to some system call that |
| 37 | is not available as a Perl function: C<syscall> requires you to provide |
| 38 | parameters stored in the way it happens in a C program. Even text processing |
| 39 | (as shown in the next section) may be simplified with judicious usage |
| 40 | of these two functions. |
| 41 | |
| 42 | To see how (un)packing works, we'll start with a simple template |
| 43 | code where the conversion is in low gear: between the contents of a byte |
| 44 | sequence and a string of hexadecimal digits. Let's use C<unpack>, since |
| 45 | this is likely to remind you of a dump program, or some desperate last |
| 46 | message unfortunate programs are wont to throw at you before they expire |
| 47 | into the wild blue yonder. Assuming that the variable C<$mem> holds a |
| 48 | sequence of bytes that we'd like to inspect without assuming anything |
| 49 | about its meaning, we can write |
| 50 | |
| 51 | my( $hex ) = unpack( 'H*', $mem ); |
| 52 | print "$hex\n"; |
| 53 | |
| 54 | whereupon we might see something like this, with each pair of hex digits |
| 55 | corresponding to a byte: |
| 56 | |
| 57 | 41204d414e204120504c414e20412043414e414c2050414e414d41 |
| 58 | |
| 59 | What was in this chunk of memory? Numbers, characters, or a mixture of |
| 60 | both? Assuming that we're on a computer where ASCII (or some similar) |
| 61 | encoding is used: hexadecimal values in the range C<0x40> - C<0x5A> |
| 62 | indicate an uppercase letter, and C<0x20> encodes a space. So we might |
| 63 | assume it is a piece of text, which some are able to read like a tabloid; |
| 64 | but others will have to get hold of an ASCII table and relive that |
| 65 | firstgrader feeling. Not caring too much about which way to read this, |
| 66 | we note that C<unpack> with the template code C<H> converts the contents |
| 67 | of a sequence of bytes into the customary hexadecimal notation. Since |
| 68 | "a sequence of" is a pretty vague indication of quantity, C<H> has been |
| 69 | defined to convert just a single hexadecimal digit unless it is followed |
| 70 | by a repeat count. An asterisk for the repeat count means to use whatever |
| 71 | remains. |
| 72 | |
| 73 | The inverse operation - packing byte contents from a string of hexadecimal |
| 74 | digits - is just as easily written. For instance: |
| 75 | |
| 76 | my $s = pack( 'H2' x 10, 30..39 ); |
| 77 | print "$s\n"; |
| 78 | |
| 79 | Since we feed a list of ten 2-digit hexadecimal strings to C<pack>, the |
| 80 | pack template should contain ten pack codes. If this is run on a computer |
| 81 | with ASCII character coding, it will print C<0123456789>. |
| 82 | |
| 83 | =head1 Packing Text |
| 84 | |
| 85 | Let's suppose you've got to read in a data file like this: |
| 86 | |
| 87 | Date |Description | Income|Expenditure |
| 88 | 01/24/2001 Zed's Camel Emporium 1147.99 |
| 89 | 01/28/2001 Flea spray 24.99 |
| 90 | 01/29/2001 Camel rides to tourists 235.00 |
| 91 | |
| 92 | How do we do it? You might think first to use C<split>; however, since |
| 93 | C<split> collapses blank fields, you'll never know whether a record was |
| 94 | income or expenditure. Oops. Well, you could always use C<substr>: |
| 95 | |
| 96 | while (<>) { |
| 97 | my $date = substr($_, 0, 11); |
| 98 | my $desc = substr($_, 12, 27); |
| 99 | my $income = substr($_, 40, 7); |
| 100 | my $expend = substr($_, 52, 7); |
| 101 | ... |
| 102 | } |
| 103 | |
| 104 | It's not really a barrel of laughs, is it? In fact, it's worse than it |
| 105 | may seem; the eagle-eyed may notice that the first field should only be |
| 106 | 10 characters wide, and the error has propagated right through the other |
| 107 | numbers - which we've had to count by hand. So it's error-prone as well |
| 108 | as horribly unfriendly. |
| 109 | |
| 110 | Or maybe we could use regular expressions: |
| 111 | |
| 112 | while (<>) { |
| 113 | my($date, $desc, $income, $expend) = |
| 114 | m|(\d\d/\d\d/\d{4}) (.{27}) (.{7})(.*)|; |
| 115 | ... |
| 116 | } |
| 117 | |
| 118 | Urgh. Well, it's a bit better, but - well, would you want to maintain |
| 119 | that? |
| 120 | |
| 121 | Hey, isn't Perl supposed to make this sort of thing easy? Well, it does, |
| 122 | if you use the right tools. C<pack> and C<unpack> are designed to help |
| 123 | you out when dealing with fixed-width data like the above. Let's have a |
| 124 | look at a solution with C<unpack>: |
| 125 | |
| 126 | while (<>) { |
| 127 | my($date, $desc, $income, $expend) = unpack("A10xA27xA7A*", $_); |
| 128 | ... |
| 129 | } |
| 130 | |
| 131 | That looks a bit nicer; but we've got to take apart that weird template. |
| 132 | Where did I pull that out of? |
| 133 | |
| 134 | OK, let's have a look at some of our data again; in fact, we'll include |
| 135 | the headers, and a handy ruler so we can keep track of where we are. |
| 136 | |
| 137 | 1 2 3 4 5 |
| 138 | 1234567890123456789012345678901234567890123456789012345678 |
| 139 | Date |Description | Income|Expenditure |
| 140 | 01/28/2001 Flea spray 24.99 |
| 141 | 01/29/2001 Camel rides to tourists 235.00 |
| 142 | |
| 143 | From this, we can see that the date column stretches from column 1 to |
| 144 | column 10 - ten characters wide. The C<pack>-ese for "character" is |
| 145 | C<A>, and ten of them are C<A10>. So if we just wanted to extract the |
| 146 | dates, we could say this: |
| 147 | |
| 148 | my($date) = unpack("A10", $_); |
| 149 | |
| 150 | OK, what's next? Between the date and the description is a blank column; |
| 151 | we want to skip over that. The C<x> template means "skip forward", so we |
| 152 | want one of those. Next, we have another batch of characters, from 12 to |
| 153 | 38. That's 27 more characters, hence C<A27>. (Don't make the fencepost |
| 154 | error - there are 27 characters between 12 and 38, not 26. Count 'em!) |
| 155 | |
| 156 | Now we skip another character and pick up the next 7 characters: |
| 157 | |
| 158 | my($date,$description,$income) = unpack("A10xA27xA7", $_); |
| 159 | |
| 160 | Now comes the clever bit. Lines in our ledger which are just income and |
| 161 | not expenditure might end at column 46. Hence, we don't want to tell our |
| 162 | C<unpack> pattern that we B<need> to find another 12 characters; we'll |
| 163 | just say "if there's anything left, take it". As you might guess from |
| 164 | regular expressions, that's what the C<*> means: "use everything |
| 165 | remaining". |
| 166 | |
| 167 | =over 3 |
| 168 | |
| 169 | =item * |
| 170 | |
| 171 | Be warned, though, that unlike regular expressions, if the C<unpack> |
| 172 | template doesn't match the incoming data, Perl will scream and die. |
| 173 | |
| 174 | =back |
| 175 | |
| 176 | |
| 177 | Hence, putting it all together: |
| 178 | |
| 179 | my ($date, $description, $income, $expend) = |
| 180 | unpack("A10xA27xA7xA*", $_); |
| 181 | |
| 182 | Now, that's our data parsed. I suppose what we might want to do now is |
| 183 | total up our income and expenditure, and add another line to the end of |
| 184 | our ledger - in the same format - saying how much we've brought in and |
| 185 | how much we've spent: |
| 186 | |
| 187 | while (<>) { |
| 188 | my ($date, $desc, $income, $expend) = |
| 189 | unpack("A10xA27xA7xA*", $_); |
| 190 | $tot_income += $income; |
| 191 | $tot_expend += $expend; |
| 192 | } |
| 193 | |
| 194 | $tot_income = sprintf("%.2f", $tot_income); # Get them into |
| 195 | $tot_expend = sprintf("%.2f", $tot_expend); # "financial" format |
| 196 | |
| 197 | $date = POSIX::strftime("%m/%d/%Y", localtime); |
| 198 | |
| 199 | # OK, let's go: |
| 200 | |
| 201 | print pack("A10xA27xA7xA*", $date, "Totals", |
| 202 | $tot_income, $tot_expend); |
| 203 | |
| 204 | Oh, hmm. That didn't quite work. Let's see what happened: |
| 205 | |
| 206 | 01/24/2001 Zed's Camel Emporium 1147.99 |
| 207 | 01/28/2001 Flea spray 24.99 |
| 208 | 01/29/2001 Camel rides to tourists 1235.00 |
| 209 | 03/23/2001Totals 1235.001172.98 |
| 210 | |
| 211 | OK, it's a start, but what happened to the spaces? We put C<x>, didn't |
| 212 | we? Shouldn't it skip forward? Let's look at what L<perlfunc/pack> says: |
| 213 | |
| 214 | x A null byte. |
| 215 | |
| 216 | Urgh. No wonder. There's a big difference between "a null byte", |
| 217 | character zero, and "a space", character 32. Perl's put something |
| 218 | between the date and the description - but unfortunately, we can't see |
| 219 | it! |
| 220 | |
| 221 | What we actually need to do is expand the width of the fields. The C<A> |
| 222 | format pads any non-existent characters with spaces, so we can use the |
| 223 | additional spaces to line up our fields, like this: |
| 224 | |
| 225 | print pack("A11 A28 A8 A*", $date, "Totals", |
| 226 | $tot_income, $tot_expend); |
| 227 | |
| 228 | (Note that you can put spaces in the template to make it more readable, |
| 229 | but they don't translate to spaces in the output.) Here's what we got |
| 230 | this time: |
| 231 | |
| 232 | 01/24/2001 Zed's Camel Emporium 1147.99 |
| 233 | 01/28/2001 Flea spray 24.99 |
| 234 | 01/29/2001 Camel rides to tourists 1235.00 |
| 235 | 03/23/2001 Totals 1235.00 1172.98 |
| 236 | |
| 237 | That's a bit better, but we still have that last column which needs to |
| 238 | be moved further over. There's an easy way to fix this up: |
| 239 | unfortunately, we can't get C<pack> to right-justify our fields, but we |
| 240 | can get C<sprintf> to do it: |
| 241 | |
| 242 | $tot_income = sprintf("%.2f", $tot_income); |
| 243 | $tot_expend = sprintf("%12.2f", $tot_expend); |
| 244 | $date = POSIX::strftime("%m/%d/%Y", localtime); |
| 245 | print pack("A11 A28 A8 A*", $date, "Totals", |
| 246 | $tot_income, $tot_expend); |
| 247 | |
| 248 | This time we get the right answer: |
| 249 | |
| 250 | 01/28/2001 Flea spray 24.99 |
| 251 | 01/29/2001 Camel rides to tourists 1235.00 |
| 252 | 03/23/2001 Totals 1235.00 1172.98 |
| 253 | |
| 254 | So that's how we consume and produce fixed-width data. Let's recap what |
| 255 | we've seen of C<pack> and C<unpack> so far: |
| 256 | |
| 257 | =over 3 |
| 258 | |
| 259 | =item * |
| 260 | |
| 261 | Use C<pack> to go from several pieces of data to one fixed-width |
| 262 | version; use C<unpack> to turn a fixed-width-format string into several |
| 263 | pieces of data. |
| 264 | |
| 265 | =item * |
| 266 | |
| 267 | The pack format C<A> means "any character"; if you're C<pack>ing and |
| 268 | you've run out of things to pack, C<pack> will fill the rest up with |
| 269 | spaces. |
| 270 | |
| 271 | =item * |
| 272 | |
| 273 | C<x> means "skip a byte" when C<unpack>ing; when C<pack>ing, it means |
| 274 | "introduce a null byte" - that's probably not what you mean if you're |
| 275 | dealing with plain text. |
| 276 | |
| 277 | =item * |
| 278 | |
| 279 | You can follow the formats with numbers to say how many characters |
| 280 | should be affected by that format: C<A12> means "take 12 characters"; |
| 281 | C<x6> means "skip 6 bytes" or "character 0, 6 times". |
| 282 | |
| 283 | =item * |
| 284 | |
| 285 | Instead of a number, you can use C<*> to mean "consume everything else |
| 286 | left". |
| 287 | |
| 288 | B<Warning>: when packing multiple pieces of data, C<*> only means |
| 289 | "consume all of the current piece of data". That's to say |
| 290 | |
| 291 | pack("A*A*", $one, $two) |
| 292 | |
| 293 | packs all of C<$one> into the first C<A*> and then all of C<$two> into |
| 294 | the second. This is a general principle: each format character |
| 295 | corresponds to one piece of data to be C<pack>ed. |
| 296 | |
| 297 | =back |
| 298 | |
| 299 | |
| 300 | |
| 301 | =head1 Packing Numbers |
| 302 | |
| 303 | So much for textual data. Let's get onto the meaty stuff that C<pack> |
| 304 | and C<unpack> are best at: handling binary formats for numbers. There is, |
| 305 | of course, not just one binary format - life would be too simple - but |
| 306 | Perl will do all the finicky labor for you. |
| 307 | |
| 308 | |
| 309 | =head2 Integers |
| 310 | |
| 311 | Packing and unpacking numbers implies conversion to and from some |
| 312 | I<specific> binary representation. Leaving floating point numbers |
| 313 | aside for the moment, the salient properties of any such representation |
| 314 | are: |
| 315 | |
| 316 | =over 4 |
| 317 | |
| 318 | =item * |
| 319 | |
| 320 | the number of bytes used for storing the integer, |
| 321 | |
| 322 | =item * |
| 323 | |
| 324 | whether the contents are interpreted as a signed or unsigned number, |
| 325 | |
| 326 | =item * |
| 327 | |
| 328 | the byte ordering: whether the first byte is the least or most |
| 329 | significant byte (or: little-endian or big-endian, respectively). |
| 330 | |
| 331 | =back |
| 332 | |
| 333 | So, for instance, to pack 20302 to a signed 16 bit integer in your |
| 334 | computer's representation you write |
| 335 | |
| 336 | my $ps = pack( 's', 20302 ); |
| 337 | |
| 338 | Again, the result is a string, now containing 2 bytes. If you print |
| 339 | this string (which is, generally, not recommended) you might see |
| 340 | C<ON> or C<NO> (depending on your system's byte ordering) - or something |
| 341 | entirely different if your computer doesn't use ASCII character encoding. |
| 342 | Unpacking C<$ps> with the same template returns the original integer value: |
| 343 | |
| 344 | my( $s ) = unpack( 's', $ps ); |
| 345 | |
| 346 | This is true for all numeric template codes. But don't expect miracles: |
| 347 | if the packed value exceeds the allotted byte capacity, high order bits |
| 348 | are silently discarded, and unpack certainly won't be able to pull them |
| 349 | back out of some magic hat. And, when you pack using a signed template |
| 350 | code such as C<s>, an excess value may result in the sign bit |
| 351 | getting set, and unpacking this will smartly return a negative value. |
| 352 | |
| 353 | 16 bits won't get you too far with integers, but there is C<l> and C<L> |
| 354 | for signed and unsigned 32-bit integers. And if this is not enough and |
| 355 | your system supports 64 bit integers you can push the limits much closer |
| 356 | to infinity with pack codes C<q> and C<Q>. A notable exception is provided |
| 357 | by pack codes C<i> and C<I> for signed and unsigned integers of the |
| 358 | "local custom" variety: Such an integer will take up as many bytes as |
| 359 | a local C compiler returns for C<sizeof(int)>, but it'll use I<at least> |
| 360 | 32 bits. |
| 361 | |
| 362 | Each of the integer pack codes C<sSlLqQ> results in a fixed number of bytes, |
| 363 | no matter where you execute your program. This may be useful for some |
| 364 | applications, but it does not provide for a portable way to pass data |
| 365 | structures between Perl and C programs (bound to happen when you call |
| 366 | XS extensions or the Perl function C<syscall>), or when you read or |
| 367 | write binary files. What you'll need in this case are template codes that |
| 368 | depend on what your local C compiler compiles when you code C<short> or |
| 369 | C<unsigned long>, for instance. These codes and their corresponding |
| 370 | byte lengths are shown in the table below. Since the C standard leaves |
| 371 | much leeway with respect to the relative sizes of these data types, actual |
| 372 | values may vary, and that's why the values are given as expressions in |
| 373 | C and Perl. (If you'd like to use values from C<%Config> in your program |
| 374 | you have to import it with C<use Config>.) |
| 375 | |
| 376 | signed unsigned byte length in C byte length in Perl |
| 377 | s! S! sizeof(short) $Config{shortsize} |
| 378 | i! I! sizeof(int) $Config{intsize} |
| 379 | l! L! sizeof(long) $Config{longsize} |
| 380 | q! Q! sizeof(long long) $Config{longlongsize} |
| 381 | |
| 382 | The C<i!> and C<I!> codes aren't different from C<i> and C<I>; they are |
| 383 | tolerated for completeness' sake. |
| 384 | |
| 385 | |
| 386 | =head2 Unpacking a Stack Frame |
| 387 | |
| 388 | Requesting a particular byte ordering may be necessary when you work with |
| 389 | binary data coming from some specific architecture whereas your program could |
| 390 | run on a totally different system. As an example, assume you have 24 bytes |
| 391 | containing a stack frame as it happens on an Intel 8086: |
| 392 | |
| 393 | +---------+ +----+----+ +---------+ |
| 394 | TOS: | IP | TOS+4:| FL | FH | FLAGS TOS+14:| SI | |
| 395 | +---------+ +----+----+ +---------+ |
| 396 | | CS | | AL | AH | AX | DI | |
| 397 | +---------+ +----+----+ +---------+ |
| 398 | | BL | BH | BX | BP | |
| 399 | +----+----+ +---------+ |
| 400 | | CL | CH | CX | DS | |
| 401 | +----+----+ +---------+ |
| 402 | | DL | DH | DX | ES | |
| 403 | +----+----+ +---------+ |
| 404 | |
| 405 | First, we note that this time-honored 16-bit CPU uses little-endian order, |
| 406 | and that's why the low order byte is stored at the lower address. To |
| 407 | unpack such a (unsigned) short we'll have to use code C<v>. A repeat |
| 408 | count unpacks all 12 shorts: |
| 409 | |
| 410 | my( $ip, $cs, $flags, $ax, $bx, $cd, $dx, $si, $di, $bp, $ds, $es ) = |
| 411 | unpack( 'v12', $frame ); |
| 412 | |
| 413 | Alternatively, we could have used C<C> to unpack the individually |
| 414 | accessible byte registers FL, FH, AL, AH, etc.: |
| 415 | |
| 416 | my( $fl, $fh, $al, $ah, $bl, $bh, $cl, $ch, $dl, $dh ) = |
| 417 | unpack( 'C10', substr( $frame, 4, 10 ) ); |
| 418 | |
| 419 | It would be nice if we could do this in one fell swoop: unpack a short, |
| 420 | back up a little, and then unpack 2 bytes. Since Perl I<is> nice, it |
| 421 | proffers the template code C<X> to back up one byte. Putting this all |
| 422 | together, we may now write: |
| 423 | |
| 424 | my( $ip, $cs, |
| 425 | $flags,$fl,$fh, |
| 426 | $ax,$al,$ah, $bx,$bl,$bh, $cx,$cl,$ch, $dx,$dl,$dh, |
| 427 | $si, $di, $bp, $ds, $es ) = |
| 428 | unpack( 'v2' . ('vXXCC' x 5) . 'v5', $frame ); |
| 429 | |
| 430 | (The clumsy construction of the template can be avoided - just read on!) |
| 431 | |
| 432 | We've taken some pains to construct the template so that it matches |
| 433 | the contents of our frame buffer. Otherwise we'd either get undefined values, |
| 434 | or C<unpack> could not unpack all. If C<pack> runs out of items, it will |
| 435 | supply null strings (which are coerced into zeroes whenever the pack code |
| 436 | says so). |
| 437 | |
| 438 | |
| 439 | =head2 How to Eat an Egg on a Net |
| 440 | |
| 441 | The pack code for big-endian (high order byte at the lowest address) is |
| 442 | C<n> for 16 bit and C<N> for 32 bit integers. You use these codes |
| 443 | if you know that your data comes from a compliant architecture, but, |
| 444 | surprisingly enough, you should also use these pack codes if you |
| 445 | exchange binary data, across the network, with some system that you |
| 446 | know next to nothing about. The simple reason is that this |
| 447 | order has been chosen as the I<network order>, and all standard-fearing |
| 448 | programs ought to follow this convention. (This is, of course, a stern |
| 449 | backing for one of the Lilliputian parties and may well influence the |
| 450 | political development there.) So, if the protocol expects you to send |
| 451 | a message by sending the length first, followed by just so many bytes, |
| 452 | you could write: |
| 453 | |
| 454 | my $buf = pack( 'N', length( $msg ) ) . $msg; |
| 455 | |
| 456 | or even: |
| 457 | |
| 458 | my $buf = pack( 'NA*', length( $msg ), $msg ); |
| 459 | |
| 460 | and pass C<$buf> to your send routine. Some protocols demand that the |
| 461 | count should include the length of the count itself: then just add 4 |
| 462 | to the data length. (But make sure to read L<"Lengths and Widths"> before |
| 463 | you really code this!) |
| 464 | |
| 465 | |
| 466 | =head2 Byte-order modifiers |
| 467 | |
| 468 | In the previous sections we've learned how to use C<n>, C<N>, C<v> and |
| 469 | C<V> to pack and unpack integers with big- or little-endian byte-order. |
| 470 | While this is nice, it's still rather limited because it leaves out all |
| 471 | kinds of signed integers as well as 64-bit integers. For example, if you |
| 472 | wanted to unpack a sequence of signed big-endian 16-bit integers in a |
| 473 | platform-independent way, you would have to write: |
| 474 | |
| 475 | my @data = unpack 's*', pack 'S*', unpack 'n*', $buf; |
| 476 | |
| 477 | This is ugly. As of Perl 5.9.2, there's a much nicer way to express your |
| 478 | desire for a certain byte-order: the C<E<gt>> and C<E<lt>> modifiers. |
| 479 | C<E<gt>> is the big-endian modifier, while C<E<lt>> is the little-endian |
| 480 | modifier. Using them, we could rewrite the above code as: |
| 481 | |
| 482 | my @data = unpack 's>*', $buf; |
| 483 | |
| 484 | As you can see, the "big end" of the arrow touches the C<s>, which is a |
| 485 | nice way to remember that C<E<gt>> is the big-endian modifier. The same |
| 486 | obviously works for C<E<lt>>, where the "little end" touches the code. |
| 487 | |
| 488 | You will probably find these modifiers even more useful if you have |
| 489 | to deal with big- or little-endian C structures. Be sure to read |
| 490 | L<"Packing and Unpacking C Structures"> for more on that. |
| 491 | |
| 492 | |
| 493 | =head2 Floating point Numbers |
| 494 | |
| 495 | For packing floating point numbers you have the choice between the |
| 496 | pack codes C<f>, C<d>, C<F> and C<D>. C<f> and C<d> pack into (or unpack |
| 497 | from) single-precision or double-precision representation as it is provided |
| 498 | by your system. If your systems supports it, C<D> can be used to pack and |
| 499 | unpack (C<long double>) values, which can offer even more resolution |
| 500 | than C<f> or C<d>. B<Note that there are different long double formats.> |
| 501 | |
| 502 | C<F> packs an C<NV>, which is the floating point type used by Perl |
| 503 | internally. |
| 504 | |
| 505 | There is no such thing as a network representation for reals, so if |
| 506 | you want to send your real numbers across computer boundaries, you'd |
| 507 | better stick to text representation, possibly using the hexadecimal |
| 508 | float format (avoiding the decimal conversion loss), unless you're |
| 509 | absolutely sure what's on the other end of the line. For the even more |
| 510 | adventuresome, you can use the byte-order modifiers from the previous |
| 511 | section also on floating point codes. |
| 512 | |
| 513 | |
| 514 | |
| 515 | =head1 Exotic Templates |
| 516 | |
| 517 | |
| 518 | =head2 Bit Strings |
| 519 | |
| 520 | Bits are the atoms in the memory world. Access to individual bits may |
| 521 | have to be used either as a last resort or because it is the most |
| 522 | convenient way to handle your data. Bit string (un)packing converts |
| 523 | between strings containing a series of C<0> and C<1> characters and |
| 524 | a sequence of bytes each containing a group of 8 bits. This is almost |
| 525 | as simple as it sounds, except that there are two ways the contents of |
| 526 | a byte may be written as a bit string. Let's have a look at an annotated |
| 527 | byte: |
| 528 | |
| 529 | 7 6 5 4 3 2 1 0 |
| 530 | +-----------------+ |
| 531 | | 1 0 0 0 1 1 0 0 | |
| 532 | +-----------------+ |
| 533 | MSB LSB |
| 534 | |
| 535 | It's egg-eating all over again: Some think that as a bit string this should |
| 536 | be written "10001100" i.e. beginning with the most significant bit, others |
| 537 | insist on "00110001". Well, Perl isn't biased, so that's why we have two bit |
| 538 | string codes: |
| 539 | |
| 540 | $byte = pack( 'B8', '10001100' ); # start with MSB |
| 541 | $byte = pack( 'b8', '00110001' ); # start with LSB |
| 542 | |
| 543 | It is not possible to pack or unpack bit fields - just integral bytes. |
| 544 | C<pack> always starts at the next byte boundary and "rounds up" to the |
| 545 | next multiple of 8 by adding zero bits as required. (If you do want bit |
| 546 | fields, there is L<perlfunc/vec>. Or you could implement bit field |
| 547 | handling at the character string level, using split, substr, and |
| 548 | concatenation on unpacked bit strings.) |
| 549 | |
| 550 | To illustrate unpacking for bit strings, we'll decompose a simple |
| 551 | status register (a "-" stands for a "reserved" bit): |
| 552 | |
| 553 | +-----------------+-----------------+ |
| 554 | | S Z - A - P - C | - - - - O D I T | |
| 555 | +-----------------+-----------------+ |
| 556 | MSB LSB MSB LSB |
| 557 | |
| 558 | Converting these two bytes to a string can be done with the unpack |
| 559 | template C<'b16'>. To obtain the individual bit values from the bit |
| 560 | string we use C<split> with the "empty" separator pattern which dissects |
| 561 | into individual characters. Bit values from the "reserved" positions are |
| 562 | simply assigned to C<undef>, a convenient notation for "I don't care where |
| 563 | this goes". |
| 564 | |
| 565 | ($carry, undef, $parity, undef, $auxcarry, undef, $zero, $sign, |
| 566 | $trace, $interrupt, $direction, $overflow) = |
| 567 | split( //, unpack( 'b16', $status ) ); |
| 568 | |
| 569 | We could have used an unpack template C<'b12'> just as well, since the |
| 570 | last 4 bits can be ignored anyway. |
| 571 | |
| 572 | |
| 573 | =head2 Uuencoding |
| 574 | |
| 575 | Another odd-man-out in the template alphabet is C<u>, which packs a |
| 576 | "uuencoded string". ("uu" is short for Unix-to-Unix.) Chances are that |
| 577 | you won't ever need this encoding technique which was invented to overcome |
| 578 | the shortcomings of old-fashioned transmission mediums that do not support |
| 579 | other than simple ASCII data. The essential recipe is simple: Take three |
| 580 | bytes, or 24 bits. Split them into 4 six-packs, adding a space (0x20) to |
| 581 | each. Repeat until all of the data is blended. Fold groups of 4 bytes into |
| 582 | lines no longer than 60 and garnish them in front with the original byte count |
| 583 | (incremented by 0x20) and a C<"\n"> at the end. - The C<pack> chef will |
| 584 | prepare this for you, a la minute, when you select pack code C<u> on the menu: |
| 585 | |
| 586 | my $uubuf = pack( 'u', $bindat ); |
| 587 | |
| 588 | A repeat count after C<u> sets the number of bytes to put into an |
| 589 | uuencoded line, which is the maximum of 45 by default, but could be |
| 590 | set to some (smaller) integer multiple of three. C<unpack> simply ignores |
| 591 | the repeat count. |
| 592 | |
| 593 | |
| 594 | =head2 Doing Sums |
| 595 | |
| 596 | An even stranger template code is C<%>E<lt>I<number>E<gt>. First, because |
| 597 | it's used as a prefix to some other template code. Second, because it |
| 598 | cannot be used in C<pack> at all, and third, in C<unpack>, doesn't return the |
| 599 | data as defined by the template code it precedes. Instead it'll give you an |
| 600 | integer of I<number> bits that is computed from the data value by |
| 601 | doing sums. For numeric unpack codes, no big feat is achieved: |
| 602 | |
| 603 | my $buf = pack( 'iii', 100, 20, 3 ); |
| 604 | print unpack( '%32i3', $buf ), "\n"; # prints 123 |
| 605 | |
| 606 | For string values, C<%> returns the sum of the byte values saving |
| 607 | you the trouble of a sum loop with C<substr> and C<ord>: |
| 608 | |
| 609 | print unpack( '%32A*', "\x01\x10" ), "\n"; # prints 17 |
| 610 | |
| 611 | Although the C<%> code is documented as returning a "checksum": |
| 612 | don't put your trust in such values! Even when applied to a small number |
| 613 | of bytes, they won't guarantee a noticeable Hamming distance. |
| 614 | |
| 615 | In connection with C<b> or C<B>, C<%> simply adds bits, and this can be put |
| 616 | to good use to count set bits efficiently: |
| 617 | |
| 618 | my $bitcount = unpack( '%32b*', $mask ); |
| 619 | |
| 620 | And an even parity bit can be determined like this: |
| 621 | |
| 622 | my $evenparity = unpack( '%1b*', $mask ); |
| 623 | |
| 624 | |
| 625 | =head2 Unicode |
| 626 | |
| 627 | Unicode is a character set that can represent most characters in most of |
| 628 | the world's languages, providing room for over one million different |
| 629 | characters. Unicode 3.1 specifies 94,140 characters: The Basic Latin |
| 630 | characters are assigned to the numbers 0 - 127. The Latin-1 Supplement with |
| 631 | characters that are used in several European languages is in the next |
| 632 | range, up to 255. After some more Latin extensions we find the character |
| 633 | sets from languages using non-Roman alphabets, interspersed with a |
| 634 | variety of symbol sets such as currency symbols, Zapf Dingbats or Braille. |
| 635 | (You might want to visit L<http://www.unicode.org/> for a look at some of |
| 636 | them - my personal favourites are Telugu and Kannada.) |
| 637 | |
| 638 | The Unicode character sets associates characters with integers. Encoding |
| 639 | these numbers in an equal number of bytes would more than double the |
| 640 | requirements for storing texts written in Latin alphabets. |
| 641 | The UTF-8 encoding avoids this by storing the most common (from a western |
| 642 | point of view) characters in a single byte while encoding the rarer |
| 643 | ones in three or more bytes. |
| 644 | |
| 645 | Perl uses UTF-8, internally, for most Unicode strings. |
| 646 | |
| 647 | So what has this got to do with C<pack>? Well, if you want to compose a |
| 648 | Unicode string (that is internally encoded as UTF-8), you can do so by |
| 649 | using template code C<U>. As an example, let's produce the Euro currency |
| 650 | symbol (code number 0x20AC): |
| 651 | |
| 652 | $UTF8{Euro} = pack( 'U', 0x20AC ); |
| 653 | # Equivalent to: $UTF8{Euro} = "\x{20ac}"; |
| 654 | |
| 655 | Inspecting C<$UTF8{Euro}> shows that it contains 3 bytes: |
| 656 | "\xe2\x82\xac". However, it contains only 1 character, number 0x20AC. |
| 657 | The round trip can be completed with C<unpack>: |
| 658 | |
| 659 | $Unicode{Euro} = unpack( 'U', $UTF8{Euro} ); |
| 660 | |
| 661 | Unpacking using the C<U> template code also works on UTF-8 encoded byte |
| 662 | strings. |
| 663 | |
| 664 | Usually you'll want to pack or unpack UTF-8 strings: |
| 665 | |
| 666 | # pack and unpack the Hebrew alphabet |
| 667 | my $alefbet = pack( 'U*', 0x05d0..0x05ea ); |
| 668 | my @hebrew = unpack( 'U*', $utf ); |
| 669 | |
| 670 | Please note: in the general case, you're better off using |
| 671 | Encode::decode_utf8 to decode a UTF-8 encoded byte string to a Perl |
| 672 | Unicode string, and Encode::encode_utf8 to encode a Perl Unicode string |
| 673 | to UTF-8 bytes. These functions provide means of handling invalid byte |
| 674 | sequences and generally have a friendlier interface. |
| 675 | |
| 676 | =head2 Another Portable Binary Encoding |
| 677 | |
| 678 | The pack code C<w> has been added to support a portable binary data |
| 679 | encoding scheme that goes way beyond simple integers. (Details can |
| 680 | be found at L<http://Casbah.org/>, the Scarab project.) A BER (Binary Encoded |
| 681 | Representation) compressed unsigned integer stores base 128 |
| 682 | digits, most significant digit first, with as few digits as possible. |
| 683 | Bit eight (the high bit) is set on each byte except the last. There |
| 684 | is no size limit to BER encoding, but Perl won't go to extremes. |
| 685 | |
| 686 | my $berbuf = pack( 'w*', 1, 128, 128+1, 128*128+127 ); |
| 687 | |
| 688 | A hex dump of C<$berbuf>, with spaces inserted at the right places, |
| 689 | shows 01 8100 8101 81807F. Since the last byte is always less than |
| 690 | 128, C<unpack> knows where to stop. |
| 691 | |
| 692 | |
| 693 | =head1 Template Grouping |
| 694 | |
| 695 | Prior to Perl 5.8, repetitions of templates had to be made by |
| 696 | C<x>-multiplication of template strings. Now there is a better way as |
| 697 | we may use the pack codes C<(> and C<)> combined with a repeat count. |
| 698 | The C<unpack> template from the Stack Frame example can simply |
| 699 | be written like this: |
| 700 | |
| 701 | unpack( 'v2 (vXXCC)5 v5', $frame ) |
| 702 | |
| 703 | Let's explore this feature a little more. We'll begin with the equivalent of |
| 704 | |
| 705 | join( '', map( substr( $_, 0, 1 ), @str ) ) |
| 706 | |
| 707 | which returns a string consisting of the first character from each string. |
| 708 | Using pack, we can write |
| 709 | |
| 710 | pack( '(A)'.@str, @str ) |
| 711 | |
| 712 | or, because a repeat count C<*> means "repeat as often as required", |
| 713 | simply |
| 714 | |
| 715 | pack( '(A)*', @str ) |
| 716 | |
| 717 | (Note that the template C<A*> would only have packed C<$str[0]> in full |
| 718 | length.) |
| 719 | |
| 720 | To pack dates stored as triplets ( day, month, year ) in an array C<@dates> |
| 721 | into a sequence of byte, byte, short integer we can write |
| 722 | |
| 723 | $pd = pack( '(CCS)*', map( @$_, @dates ) ); |
| 724 | |
| 725 | To swap pairs of characters in a string (with even length) one could use |
| 726 | several techniques. First, let's use C<x> and C<X> to skip forward and back: |
| 727 | |
| 728 | $s = pack( '(A)*', unpack( '(xAXXAx)*', $s ) ); |
| 729 | |
| 730 | We can also use C<@> to jump to an offset, with 0 being the position where |
| 731 | we were when the last C<(> was encountered: |
| 732 | |
| 733 | $s = pack( '(A)*', unpack( '(@1A @0A @2)*', $s ) ); |
| 734 | |
| 735 | Finally, there is also an entirely different approach by unpacking big |
| 736 | endian shorts and packing them in the reverse byte order: |
| 737 | |
| 738 | $s = pack( '(v)*', unpack( '(n)*', $s ); |
| 739 | |
| 740 | |
| 741 | =head1 Lengths and Widths |
| 742 | |
| 743 | =head2 String Lengths |
| 744 | |
| 745 | In the previous section we've seen a network message that was constructed |
| 746 | by prefixing the binary message length to the actual message. You'll find |
| 747 | that packing a length followed by so many bytes of data is a |
| 748 | frequently used recipe since appending a null byte won't work |
| 749 | if a null byte may be part of the data. Here is an example where both |
| 750 | techniques are used: after two null terminated strings with source and |
| 751 | destination address, a Short Message (to a mobile phone) is sent after |
| 752 | a length byte: |
| 753 | |
| 754 | my $msg = pack( 'Z*Z*CA*', $src, $dst, length( $sm ), $sm ); |
| 755 | |
| 756 | Unpacking this message can be done with the same template: |
| 757 | |
| 758 | ( $src, $dst, $len, $sm ) = unpack( 'Z*Z*CA*', $msg ); |
| 759 | |
| 760 | There's a subtle trap lurking in the offing: Adding another field after |
| 761 | the Short Message (in variable C<$sm>) is all right when packing, but this |
| 762 | cannot be unpacked naively: |
| 763 | |
| 764 | # pack a message |
| 765 | my $msg = pack( 'Z*Z*CA*C', $src, $dst, length( $sm ), $sm, $prio ); |
| 766 | |
| 767 | # unpack fails - $prio remains undefined! |
| 768 | ( $src, $dst, $len, $sm, $prio ) = unpack( 'Z*Z*CA*C', $msg ); |
| 769 | |
| 770 | The pack code C<A*> gobbles up all remaining bytes, and C<$prio> remains |
| 771 | undefined! Before we let disappointment dampen the morale: Perl's got |
| 772 | the trump card to make this trick too, just a little further up the sleeve. |
| 773 | Watch this: |
| 774 | |
| 775 | # pack a message: ASCIIZ, ASCIIZ, length/string, byte |
| 776 | my $msg = pack( 'Z* Z* C/A* C', $src, $dst, $sm, $prio ); |
| 777 | |
| 778 | # unpack |
| 779 | ( $src, $dst, $sm, $prio ) = unpack( 'Z* Z* C/A* C', $msg ); |
| 780 | |
| 781 | Combining two pack codes with a slash (C</>) associates them with a single |
| 782 | value from the argument list. In C<pack>, the length of the argument is |
| 783 | taken and packed according to the first code while the argument itself |
| 784 | is added after being converted with the template code after the slash. |
| 785 | This saves us the trouble of inserting the C<length> call, but it is |
| 786 | in C<unpack> where we really score: The value of the length byte marks the |
| 787 | end of the string to be taken from the buffer. Since this combination |
| 788 | doesn't make sense except when the second pack code isn't C<a*>, C<A*> |
| 789 | or C<Z*>, Perl won't let you. |
| 790 | |
| 791 | The pack code preceding C</> may be anything that's fit to represent a |
| 792 | number: All the numeric binary pack codes, and even text codes such as |
| 793 | C<A4> or C<Z*>: |
| 794 | |
| 795 | # pack/unpack a string preceded by its length in ASCII |
| 796 | my $buf = pack( 'A4/A*', "Humpty-Dumpty" ); |
| 797 | # unpack $buf: '13 Humpty-Dumpty' |
| 798 | my $txt = unpack( 'A4/A*', $buf ); |
| 799 | |
| 800 | C</> is not implemented in Perls before 5.6, so if your code is required to |
| 801 | work on older Perls you'll need to C<unpack( 'Z* Z* C')> to get the length, |
| 802 | then use it to make a new unpack string. For example |
| 803 | |
| 804 | # pack a message: ASCIIZ, ASCIIZ, length, string, byte |
| 805 | # (5.005 compatible) |
| 806 | my $msg = pack( 'Z* Z* C A* C', $src, $dst, length $sm, $sm, $prio ); |
| 807 | |
| 808 | # unpack |
| 809 | ( undef, undef, $len) = unpack( 'Z* Z* C', $msg ); |
| 810 | ($src, $dst, $sm, $prio) = unpack ( "Z* Z* x A$len C", $msg ); |
| 811 | |
| 812 | But that second C<unpack> is rushing ahead. It isn't using a simple literal |
| 813 | string for the template. So maybe we should introduce... |
| 814 | |
| 815 | =head2 Dynamic Templates |
| 816 | |
| 817 | So far, we've seen literals used as templates. If the list of pack |
| 818 | items doesn't have fixed length, an expression constructing the |
| 819 | template is required (whenever, for some reason, C<()*> cannot be used). |
| 820 | Here's an example: To store named string values in a way that can be |
| 821 | conveniently parsed by a C program, we create a sequence of names and |
| 822 | null terminated ASCII strings, with C<=> between the name and the value, |
| 823 | followed by an additional delimiting null byte. Here's how: |
| 824 | |
| 825 | my $env = pack( '(A*A*Z*)' . keys( %Env ) . 'C', |
| 826 | map( { ( $_, '=', $Env{$_} ) } keys( %Env ) ), 0 ); |
| 827 | |
| 828 | Let's examine the cogs of this byte mill, one by one. There's the C<map> |
| 829 | call, creating the items we intend to stuff into the C<$env> buffer: |
| 830 | to each key (in C<$_>) it adds the C<=> separator and the hash entry value. |
| 831 | Each triplet is packed with the template code sequence C<A*A*Z*> that |
| 832 | is repeated according to the number of keys. (Yes, that's what the C<keys> |
| 833 | function returns in scalar context.) To get the very last null byte, |
| 834 | we add a C<0> at the end of the C<pack> list, to be packed with C<C>. |
| 835 | (Attentive readers may have noticed that we could have omitted the 0.) |
| 836 | |
| 837 | For the reverse operation, we'll have to determine the number of items |
| 838 | in the buffer before we can let C<unpack> rip it apart: |
| 839 | |
| 840 | my $n = $env =~ tr/\0// - 1; |
| 841 | my %env = map( split( /=/, $_ ), unpack( "(Z*)$n", $env ) ); |
| 842 | |
| 843 | The C<tr> counts the null bytes. The C<unpack> call returns a list of |
| 844 | name-value pairs each of which is taken apart in the C<map> block. |
| 845 | |
| 846 | |
| 847 | =head2 Counting Repetitions |
| 848 | |
| 849 | Rather than storing a sentinel at the end of a data item (or a list of items), |
| 850 | we could precede the data with a count. Again, we pack keys and values of |
| 851 | a hash, preceding each with an unsigned short length count, and up front |
| 852 | we store the number of pairs: |
| 853 | |
| 854 | my $env = pack( 'S(S/A* S/A*)*', scalar keys( %Env ), %Env ); |
| 855 | |
| 856 | This simplifies the reverse operation as the number of repetitions can be |
| 857 | unpacked with the C</> code: |
| 858 | |
| 859 | my %env = unpack( 'S/(S/A* S/A*)', $env ); |
| 860 | |
| 861 | Note that this is one of the rare cases where you cannot use the same |
| 862 | template for C<pack> and C<unpack> because C<pack> can't determine |
| 863 | a repeat count for a C<()>-group. |
| 864 | |
| 865 | |
| 866 | =head2 Intel HEX |
| 867 | |
| 868 | Intel HEX is a file format for representing binary data, mostly for |
| 869 | programming various chips, as a text file. (See |
| 870 | L<http://en.wikipedia.org/wiki/.hex> for a detailed description, and |
| 871 | L<http://en.wikipedia.org/wiki/SREC_(file_format)> for the Motorola |
| 872 | S-record format, which can be unravelled using the same technique.) |
| 873 | Each line begins with a colon (':') and is followed by a sequence of |
| 874 | hexadecimal characters, specifying a byte count I<n> (8 bit), |
| 875 | an address (16 bit, big endian), a record type (8 bit), I<n> data bytes |
| 876 | and a checksum (8 bit) computed as the least significant byte of the two's |
| 877 | complement sum of the preceding bytes. Example: C<:0300300002337A1E>. |
| 878 | |
| 879 | The first step of processing such a line is the conversion, to binary, |
| 880 | of the hexadecimal data, to obtain the four fields, while checking the |
| 881 | checksum. No surprise here: we'll start with a simple C<pack> call to |
| 882 | convert everything to binary: |
| 883 | |
| 884 | my $binrec = pack( 'H*', substr( $hexrec, 1 ) ); |
| 885 | |
| 886 | The resulting byte sequence is most convenient for checking the checksum. |
| 887 | Don't slow your program down with a for loop adding the C<ord> values |
| 888 | of this string's bytes - the C<unpack> code C<%> is the thing to use |
| 889 | for computing the 8-bit sum of all bytes, which must be equal to zero: |
| 890 | |
| 891 | die unless unpack( "%8C*", $binrec ) == 0; |
| 892 | |
| 893 | Finally, let's get those four fields. By now, you shouldn't have any |
| 894 | problems with the first three fields - but how can we use the byte count |
| 895 | of the data in the first field as a length for the data field? Here |
| 896 | the codes C<x> and C<X> come to the rescue, as they permit jumping |
| 897 | back and forth in the string to unpack. |
| 898 | |
| 899 | my( $addr, $type, $data ) = unpack( "x n C X4 C x3 /a", $bin ); |
| 900 | |
| 901 | Code C<x> skips a byte, since we don't need the count yet. Code C<n> takes |
| 902 | care of the 16-bit big-endian integer address, and C<C> unpacks the |
| 903 | record type. Being at offset 4, where the data begins, we need the count. |
| 904 | C<X4> brings us back to square one, which is the byte at offset 0. |
| 905 | Now we pick up the count, and zoom forth to offset 4, where we are |
| 906 | now fully furnished to extract the exact number of data bytes, leaving |
| 907 | the trailing checksum byte alone. |
| 908 | |
| 909 | |
| 910 | |
| 911 | =head1 Packing and Unpacking C Structures |
| 912 | |
| 913 | In previous sections we have seen how to pack numbers and character |
| 914 | strings. If it were not for a couple of snags we could conclude this |
| 915 | section right away with the terse remark that C structures don't |
| 916 | contain anything else, and therefore you already know all there is to it. |
| 917 | Sorry, no: read on, please. |
| 918 | |
| 919 | If you have to deal with a lot of C structures, and don't want to |
| 920 | hack all your template strings manually, you'll probably want to have |
| 921 | a look at the CPAN module C<Convert::Binary::C>. Not only can it parse |
| 922 | your C source directly, but it also has built-in support for all the |
| 923 | odds and ends described further on in this section. |
| 924 | |
| 925 | =head2 The Alignment Pit |
| 926 | |
| 927 | In the consideration of speed against memory requirements the balance |
| 928 | has been tilted in favor of faster execution. This has influenced the |
| 929 | way C compilers allocate memory for structures: On architectures |
| 930 | where a 16-bit or 32-bit operand can be moved faster between places in |
| 931 | memory, or to or from a CPU register, if it is aligned at an even or |
| 932 | multiple-of-four or even at a multiple-of eight address, a C compiler |
| 933 | will give you this speed benefit by stuffing extra bytes into structures. |
| 934 | If you don't cross the C shoreline this is not likely to cause you any |
| 935 | grief (although you should care when you design large data structures, |
| 936 | or you want your code to be portable between architectures (you do want |
| 937 | that, don't you?)). |
| 938 | |
| 939 | To see how this affects C<pack> and C<unpack>, we'll compare these two |
| 940 | C structures: |
| 941 | |
| 942 | typedef struct { |
| 943 | char c1; |
| 944 | short s; |
| 945 | char c2; |
| 946 | long l; |
| 947 | } gappy_t; |
| 948 | |
| 949 | typedef struct { |
| 950 | long l; |
| 951 | short s; |
| 952 | char c1; |
| 953 | char c2; |
| 954 | } dense_t; |
| 955 | |
| 956 | Typically, a C compiler allocates 12 bytes to a C<gappy_t> variable, but |
| 957 | requires only 8 bytes for a C<dense_t>. After investigating this further, |
| 958 | we can draw memory maps, showing where the extra 4 bytes are hidden: |
| 959 | |
| 960 | 0 +4 +8 +12 |
| 961 | +--+--+--+--+--+--+--+--+--+--+--+--+ |
| 962 | |c1|xx| s |c2|xx|xx|xx| l | xx = fill byte |
| 963 | +--+--+--+--+--+--+--+--+--+--+--+--+ |
| 964 | gappy_t |
| 965 | |
| 966 | 0 +4 +8 |
| 967 | +--+--+--+--+--+--+--+--+ |
| 968 | | l | h |c1|c2| |
| 969 | +--+--+--+--+--+--+--+--+ |
| 970 | dense_t |
| 971 | |
| 972 | And that's where the first quirk strikes: C<pack> and C<unpack> |
| 973 | templates have to be stuffed with C<x> codes to get those extra fill bytes. |
| 974 | |
| 975 | The natural question: "Why can't Perl compensate for the gaps?" warrants |
| 976 | an answer. One good reason is that C compilers might provide (non-ANSI) |
| 977 | extensions permitting all sorts of fancy control over the way structures |
| 978 | are aligned, even at the level of an individual structure field. And, if |
| 979 | this were not enough, there is an insidious thing called C<union> where |
| 980 | the amount of fill bytes cannot be derived from the alignment of the next |
| 981 | item alone. |
| 982 | |
| 983 | OK, so let's bite the bullet. Here's one way to get the alignment right |
| 984 | by inserting template codes C<x>, which don't take a corresponding item |
| 985 | from the list: |
| 986 | |
| 987 | my $gappy = pack( 'cxs cxxx l!', $c1, $s, $c2, $l ); |
| 988 | |
| 989 | Note the C<!> after C<l>: We want to make sure that we pack a long |
| 990 | integer as it is compiled by our C compiler. And even now, it will only |
| 991 | work for the platforms where the compiler aligns things as above. |
| 992 | And somebody somewhere has a platform where it doesn't. |
| 993 | [Probably a Cray, where C<short>s, C<int>s and C<long>s are all 8 bytes. :-)] |
| 994 | |
| 995 | Counting bytes and watching alignments in lengthy structures is bound to |
| 996 | be a drag. Isn't there a way we can create the template with a simple |
| 997 | program? Here's a C program that does the trick: |
| 998 | |
| 999 | #include <stdio.h> |
| 1000 | #include <stddef.h> |
| 1001 | |
| 1002 | typedef struct { |
| 1003 | char fc1; |
| 1004 | short fs; |
| 1005 | char fc2; |
| 1006 | long fl; |
| 1007 | } gappy_t; |
| 1008 | |
| 1009 | #define Pt(struct,field,tchar) \ |
| 1010 | printf( "@%d%s ", offsetof(struct,field), # tchar ); |
| 1011 | |
| 1012 | int main() { |
| 1013 | Pt( gappy_t, fc1, c ); |
| 1014 | Pt( gappy_t, fs, s! ); |
| 1015 | Pt( gappy_t, fc2, c ); |
| 1016 | Pt( gappy_t, fl, l! ); |
| 1017 | printf( "\n" ); |
| 1018 | } |
| 1019 | |
| 1020 | The output line can be used as a template in a C<pack> or C<unpack> call: |
| 1021 | |
| 1022 | my $gappy = pack( '@0c @2s! @4c @8l!', $c1, $s, $c2, $l ); |
| 1023 | |
| 1024 | Gee, yet another template code - as if we hadn't plenty. But |
| 1025 | C<@> saves our day by enabling us to specify the offset from the beginning |
| 1026 | of the pack buffer to the next item: This is just the value |
| 1027 | the C<offsetof> macro (defined in C<E<lt>stddef.hE<gt>>) returns when |
| 1028 | given a C<struct> type and one of its field names ("member-designator" in |
| 1029 | C standardese). |
| 1030 | |
| 1031 | Neither using offsets nor adding C<x>'s to bridge the gaps is satisfactory. |
| 1032 | (Just imagine what happens if the structure changes.) What we really need |
| 1033 | is a way of saying "skip as many bytes as required to the next multiple of N". |
| 1034 | In fluent Templatese, you say this with C<x!N> where N is replaced by the |
| 1035 | appropriate value. Here's the next version of our struct packaging: |
| 1036 | |
| 1037 | my $gappy = pack( 'c x!2 s c x!4 l!', $c1, $s, $c2, $l ); |
| 1038 | |
| 1039 | That's certainly better, but we still have to know how long all the |
| 1040 | integers are, and portability is far away. Rather than C<2>, |
| 1041 | for instance, we want to say "however long a short is". But this can be |
| 1042 | done by enclosing the appropriate pack code in brackets: C<[s]>. So, here's |
| 1043 | the very best we can do: |
| 1044 | |
| 1045 | my $gappy = pack( 'c x![s] s c x![l!] l!', $c1, $s, $c2, $l ); |
| 1046 | |
| 1047 | |
| 1048 | =head2 Dealing with Endian-ness |
| 1049 | |
| 1050 | Now, imagine that we want to pack the data for a machine with a |
| 1051 | different byte-order. First, we'll have to figure out how big the data |
| 1052 | types on the target machine really are. Let's assume that the longs are |
| 1053 | 32 bits wide and the shorts are 16 bits wide. You can then rewrite the |
| 1054 | template as: |
| 1055 | |
| 1056 | my $gappy = pack( 'c x![s] s c x![l] l', $c1, $s, $c2, $l ); |
| 1057 | |
| 1058 | If the target machine is little-endian, we could write: |
| 1059 | |
| 1060 | my $gappy = pack( 'c x![s] s< c x![l] l<', $c1, $s, $c2, $l ); |
| 1061 | |
| 1062 | This forces the short and the long members to be little-endian, and is |
| 1063 | just fine if you don't have too many struct members. But we could also |
| 1064 | use the byte-order modifier on a group and write the following: |
| 1065 | |
| 1066 | my $gappy = pack( '( c x![s] s c x![l] l )<', $c1, $s, $c2, $l ); |
| 1067 | |
| 1068 | This is not as short as before, but it makes it more obvious that we |
| 1069 | intend to have little-endian byte-order for a whole group, not only |
| 1070 | for individual template codes. It can also be more readable and easier |
| 1071 | to maintain. |
| 1072 | |
| 1073 | |
| 1074 | =head2 Alignment, Take 2 |
| 1075 | |
| 1076 | I'm afraid that we're not quite through with the alignment catch yet. The |
| 1077 | hydra raises another ugly head when you pack arrays of structures: |
| 1078 | |
| 1079 | typedef struct { |
| 1080 | short count; |
| 1081 | char glyph; |
| 1082 | } cell_t; |
| 1083 | |
| 1084 | typedef cell_t buffer_t[BUFLEN]; |
| 1085 | |
| 1086 | Where's the catch? Padding is neither required before the first field C<count>, |
| 1087 | nor between this and the next field C<glyph>, so why can't we simply pack |
| 1088 | like this: |
| 1089 | |
| 1090 | # something goes wrong here: |
| 1091 | pack( 's!a' x @buffer, |
| 1092 | map{ ( $_->{count}, $_->{glyph} ) } @buffer ); |
| 1093 | |
| 1094 | This packs C<3*@buffer> bytes, but it turns out that the size of |
| 1095 | C<buffer_t> is four times C<BUFLEN>! The moral of the story is that |
| 1096 | the required alignment of a structure or array is propagated to the |
| 1097 | next higher level where we have to consider padding I<at the end> |
| 1098 | of each component as well. Thus the correct template is: |
| 1099 | |
| 1100 | pack( 's!ax' x @buffer, |
| 1101 | map{ ( $_->{count}, $_->{glyph} ) } @buffer ); |
| 1102 | |
| 1103 | =head2 Alignment, Take 3 |
| 1104 | |
| 1105 | And even if you take all the above into account, ANSI still lets this: |
| 1106 | |
| 1107 | typedef struct { |
| 1108 | char foo[2]; |
| 1109 | } foo_t; |
| 1110 | |
| 1111 | vary in size. The alignment constraint of the structure can be greater than |
| 1112 | any of its elements. [And if you think that this doesn't affect anything |
| 1113 | common, dismember the next cellphone that you see. Many have ARM cores, and |
| 1114 | the ARM structure rules make C<sizeof (foo_t)> == 4] |
| 1115 | |
| 1116 | =head2 Pointers for How to Use Them |
| 1117 | |
| 1118 | The title of this section indicates the second problem you may run into |
| 1119 | sooner or later when you pack C structures. If the function you intend |
| 1120 | to call expects a, say, C<void *> value, you I<cannot> simply take |
| 1121 | a reference to a Perl variable. (Although that value certainly is a |
| 1122 | memory address, it's not the address where the variable's contents are |
| 1123 | stored.) |
| 1124 | |
| 1125 | Template code C<P> promises to pack a "pointer to a fixed length string". |
| 1126 | Isn't this what we want? Let's try: |
| 1127 | |
| 1128 | # allocate some storage and pack a pointer to it |
| 1129 | my $memory = "\x00" x $size; |
| 1130 | my $memptr = pack( 'P', $memory ); |
| 1131 | |
| 1132 | But wait: doesn't C<pack> just return a sequence of bytes? How can we pass this |
| 1133 | string of bytes to some C code expecting a pointer which is, after all, |
| 1134 | nothing but a number? The answer is simple: We have to obtain the numeric |
| 1135 | address from the bytes returned by C<pack>. |
| 1136 | |
| 1137 | my $ptr = unpack( 'L!', $memptr ); |
| 1138 | |
| 1139 | Obviously this assumes that it is possible to typecast a pointer |
| 1140 | to an unsigned long and vice versa, which frequently works but should not |
| 1141 | be taken as a universal law. - Now that we have this pointer the next question |
| 1142 | is: How can we put it to good use? We need a call to some C function |
| 1143 | where a pointer is expected. The read(2) system call comes to mind: |
| 1144 | |
| 1145 | ssize_t read(int fd, void *buf, size_t count); |
| 1146 | |
| 1147 | After reading L<perlfunc> explaining how to use C<syscall> we can write |
| 1148 | this Perl function copying a file to standard output: |
| 1149 | |
| 1150 | require 'syscall.ph'; # run h2ph to generate this file |
| 1151 | sub cat($){ |
| 1152 | my $path = shift(); |
| 1153 | my $size = -s $path; |
| 1154 | my $memory = "\x00" x $size; # allocate some memory |
| 1155 | my $ptr = unpack( 'L', pack( 'P', $memory ) ); |
| 1156 | open( F, $path ) || die( "$path: cannot open ($!)\n" ); |
| 1157 | my $fd = fileno(F); |
| 1158 | my $res = syscall( &SYS_read, fileno(F), $ptr, $size ); |
| 1159 | print $memory; |
| 1160 | close( F ); |
| 1161 | } |
| 1162 | |
| 1163 | This is neither a specimen of simplicity nor a paragon of portability but |
| 1164 | it illustrates the point: We are able to sneak behind the scenes and |
| 1165 | access Perl's otherwise well-guarded memory! (Important note: Perl's |
| 1166 | C<syscall> does I<not> require you to construct pointers in this roundabout |
| 1167 | way. You simply pass a string variable, and Perl forwards the address.) |
| 1168 | |
| 1169 | How does C<unpack> with C<P> work? Imagine some pointer in the buffer |
| 1170 | about to be unpacked: If it isn't the null pointer (which will smartly |
| 1171 | produce the C<undef> value) we have a start address - but then what? |
| 1172 | Perl has no way of knowing how long this "fixed length string" is, so |
| 1173 | it's up to you to specify the actual size as an explicit length after C<P>. |
| 1174 | |
| 1175 | my $mem = "abcdefghijklmn"; |
| 1176 | print unpack( 'P5', pack( 'P', $mem ) ); # prints "abcde" |
| 1177 | |
| 1178 | As a consequence, C<pack> ignores any number or C<*> after C<P>. |
| 1179 | |
| 1180 | |
| 1181 | Now that we have seen C<P> at work, we might as well give C<p> a whirl. |
| 1182 | Why do we need a second template code for packing pointers at all? The |
| 1183 | answer lies behind the simple fact that an C<unpack> with C<p> promises |
| 1184 | a null-terminated string starting at the address taken from the buffer, |
| 1185 | and that implies a length for the data item to be returned: |
| 1186 | |
| 1187 | my $buf = pack( 'p', "abc\x00efhijklmn" ); |
| 1188 | print unpack( 'p', $buf ); # prints "abc" |
| 1189 | |
| 1190 | |
| 1191 | |
| 1192 | Albeit this is apt to be confusing: As a consequence of the length being |
| 1193 | implied by the string's length, a number after pack code C<p> is a repeat |
| 1194 | count, not a length as after C<P>. |
| 1195 | |
| 1196 | |
| 1197 | Using C<pack(..., $x)> with C<P> or C<p> to get the address where C<$x> is |
| 1198 | actually stored must be used with circumspection. Perl's internal machinery |
| 1199 | considers the relation between a variable and that address as its very own |
| 1200 | private matter and doesn't really care that we have obtained a copy. Therefore: |
| 1201 | |
| 1202 | =over 4 |
| 1203 | |
| 1204 | =item * |
| 1205 | |
| 1206 | Do not use C<pack> with C<p> or C<P> to obtain the address of variable |
| 1207 | that's bound to go out of scope (and thereby freeing its memory) before you |
| 1208 | are done with using the memory at that address. |
| 1209 | |
| 1210 | =item * |
| 1211 | |
| 1212 | Be very careful with Perl operations that change the value of the |
| 1213 | variable. Appending something to the variable, for instance, might require |
| 1214 | reallocation of its storage, leaving you with a pointer into no-man's land. |
| 1215 | |
| 1216 | =item * |
| 1217 | |
| 1218 | Don't think that you can get the address of a Perl variable |
| 1219 | when it is stored as an integer or double number! C<pack('P', $x)> will |
| 1220 | force the variable's internal representation to string, just as if you |
| 1221 | had written something like C<$x .= ''>. |
| 1222 | |
| 1223 | =back |
| 1224 | |
| 1225 | It's safe, however, to P- or p-pack a string literal, because Perl simply |
| 1226 | allocates an anonymous variable. |
| 1227 | |
| 1228 | |
| 1229 | |
| 1230 | =head1 Pack Recipes |
| 1231 | |
| 1232 | Here are a collection of (possibly) useful canned recipes for C<pack> |
| 1233 | and C<unpack>: |
| 1234 | |
| 1235 | # Convert IP address for socket functions |
| 1236 | pack( "C4", split /\./, "123.4.5.6" ); |
| 1237 | |
| 1238 | # Count the bits in a chunk of memory (e.g. a select vector) |
| 1239 | unpack( '%32b*', $mask ); |
| 1240 | |
| 1241 | # Determine the endianness of your system |
| 1242 | $is_little_endian = unpack( 'c', pack( 's', 1 ) ); |
| 1243 | $is_big_endian = unpack( 'xc', pack( 's', 1 ) ); |
| 1244 | |
| 1245 | # Determine the number of bits in a native integer |
| 1246 | $bits = unpack( '%32I!', ~0 ); |
| 1247 | |
| 1248 | # Prepare argument for the nanosleep system call |
| 1249 | my $timespec = pack( 'L!L!', $secs, $nanosecs ); |
| 1250 | |
| 1251 | For a simple memory dump we unpack some bytes into just as |
| 1252 | many pairs of hex digits, and use C<map> to handle the traditional |
| 1253 | spacing - 16 bytes to a line: |
| 1254 | |
| 1255 | my $i; |
| 1256 | print map( ++$i % 16 ? "$_ " : "$_\n", |
| 1257 | unpack( 'H2' x length( $mem ), $mem ) ), |
| 1258 | length( $mem ) % 16 ? "\n" : ''; |
| 1259 | |
| 1260 | |
| 1261 | =head1 Funnies Section |
| 1262 | |
| 1263 | # Pulling digits out of nowhere... |
| 1264 | print unpack( 'C', pack( 'x' ) ), |
| 1265 | unpack( '%B*', pack( 'A' ) ), |
| 1266 | unpack( 'H', pack( 'A' ) ), |
| 1267 | unpack( 'A', unpack( 'C', pack( 'A' ) ) ), "\n"; |
| 1268 | |
| 1269 | # One for the road ;-) |
| 1270 | my $advice = pack( 'all u can in a van' ); |
| 1271 | |
| 1272 | |
| 1273 | =head1 Authors |
| 1274 | |
| 1275 | Simon Cozens and Wolfgang Laun. |
| 1276 | |