This is a live mirror of the Perl 5 development currently hosted at https://github.com/perl/perl5
Integrate mainline.
[perl5.git] / pod / perliol.pod
CommitLineData
50b80e25
NIS
1
2=head1 NAME
3
4perliol - C API for Perl's implementation of IO in Layers.
5
6=head1 SYNOPSIS
7
8 /* Defining a layer ... */
9 #include <perliol.h>
10
11
12=head1 DESCRIPTION
13
9d799145
NIS
14This document describes the behavior and implementation of the PerlIO
15abstraction described in L<perlapio> when C<USE_PERLIO> is defined (and
16C<USE_SFIO> is not).
50b80e25
NIS
17
18=head2 History and Background
19
9d799145
NIS
20The PerlIO abstraction was introduced in perl5.003_02 but languished as
21just an abstraction until perl5.7.0. However during that time a number
22of perl extentions switched to using it, so the API is mostly fixed to
23maintain (source) compatibility.
50b80e25 24
9d799145
NIS
25The aim of the implementation is to provide the PerlIO API in a flexible
26and platform neutral manner. It is also a trial of an "Object Oriented
27C, with vtables" approach which may be applied to perl6.
50b80e25
NIS
28
29=head2 Layers vs Disciplines
30
9d799145
NIS
31Initial discussion of the ability to modify IO streams behaviour used
32the term "discipline" for the entities which were added. This came (I
33believe) from the use of the term in "sfio", which in turn borrowed it
34from "line disciplines" on Unix terminals. However, this document (and
35the C code) uses the term "layer".
36
37This is, I hope, a natural term given the implementation, and should avoid
38connotations that are inherent in earlier uses of "discipline" for things
39which are rather different.
50b80e25
NIS
40
41=head2 Data Structures
42
43The basic data structure is a PerlIOl:
44
45 typedef struct _PerlIO PerlIOl;
46 typedef struct _PerlIO_funcs PerlIO_funcs;
47 typedef PerlIOl *PerlIO;
48
49 struct _PerlIO
50 {
51 PerlIOl * next; /* Lower layer */
52 PerlIO_funcs * tab; /* Functions for this layer */
53 IV flags; /* Various flags for state */
54 };
55
9d799145
NIS
56A C<PerlIOl *> is a pointer to to the struct, and the I<application> level
57C<PerlIO *> is a pointer to a C<PerlIOl *> - i.e. a pointer to a pointer to
58the struct. This allows the application level C<PerlIO *> to remain
59constant while the actual C<PerlIOl *> underneath changes. (Compare perl's
60C<SV *> which remains constant while its C<sv_any> field changes as the
61scalar's type changes.) An IO stream is then in general represented as a
62pointer to this linked-list of "layers".
50b80e25 63
9d799145 64It should be noted that because of the double indirection in a C<PerlIO *>,
11e1c8f2
PP
65a C<< &(perlio-E<gt>next) >> "is" a C<PerlIO *>, and so to some degree
66at least one layer can use the "standard" API on the next layer down.
50b80e25
NIS
67
68A "layer" is composed of two parts:
69
70=over 4
71
72=item 1. The functions and attributes of the "layer class".
73
74=item 2. The per-instance data for a particular handle.
75
76=back
77
78=head2 Functions and Attributes
79
9d799145
NIS
80The functions and attributes are accessed via the "tab" (for table)
81member of C<PerlIOl>. The functions (methods of the layer "class") are
82fixed, and are defined by the C<PerlIO_funcs> type. They are broadly the
83same as the public C<PerlIO_xxxxx> functions:
50b80e25
NIS
84
85 struct _PerlIO_funcs
86 {
87 char * name;
88 Size_t size;
89 IV kind;
90 IV (*Fileno)(PerlIO *f);
91 PerlIO * (*Fdopen)(PerlIO_funcs *tab, int fd, const char *mode);
92 PerlIO * (*Open)(PerlIO_funcs *tab, const char *path, const char *mode);
93 int (*Reopen)(const char *path, const char *mode, PerlIO *f);
94 IV (*Pushed)(PerlIO *f,const char *mode,const char *arg,STRLEN len);
95 IV (*Popped)(PerlIO *f);
96 /* Unix-like functions - cf sfio line disciplines */
97 SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
98 SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
99 SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
100 IV (*Seek)(PerlIO *f, Off_t offset, int whence);
101 Off_t (*Tell)(PerlIO *f);
102 IV (*Close)(PerlIO *f);
103 /* Stdio-like buffered IO functions */
104 IV (*Flush)(PerlIO *f);
105 IV (*Fill)(PerlIO *f);
106 IV (*Eof)(PerlIO *f);
107 IV (*Error)(PerlIO *f);
108 void (*Clearerr)(PerlIO *f);
109 void (*Setlinebuf)(PerlIO *f);
110 /* Perl's snooping functions */
111 STDCHAR * (*Get_base)(PerlIO *f);
112 Size_t (*Get_bufsiz)(PerlIO *f);
113 STDCHAR * (*Get_ptr)(PerlIO *f);
114 SSize_t (*Get_cnt)(PerlIO *f);
115 void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
116 };
117
9d799145
NIS
118The first few members of the struct give a "name" for the layer, the
119size to C<malloc> for the per-instance data, and some flags which are
120attributes of the class as whole (such as whether it is a buffering
121layer), then follow the functions which fall into four basic groups:
50b80e25
NIS
122
123=over 4
124
125=item 1. Opening and setup functions
126
127=item 2. Basic IO operations
128
129=item 3. Stdio class buffering options.
130
131=item 4. Functions to support Perl's traditional "fast" access to the buffer.
132
133=back
134
135A layer does not have to implement all the functions, but the whole table has
136to be present. Unimplemented slots can be NULL (which will will result in an error
137when called) or can be filled in with stubs to "inherit" behaviour from
138a "base class". This "inheritance" is fixed for all instances of the layer,
139but as the layer chooses which stubs to populate the table, limited
140"multiple inheritance" is possible.
141
142=head2 Per-instance Data
143
144The per-instance data are held in memory beyond the basic PerlIOl struct,
145by making a PerlIOl the first member of the layer's struct thus:
146
147 typedef struct
148 {
149 struct _PerlIO base; /* Base "class" info */
150 STDCHAR * buf; /* Start of buffer */
151 STDCHAR * end; /* End of valid part of buffer */
152 STDCHAR * ptr; /* Current position in buffer */
153 Off_t posn; /* Offset of buf into the file */
154 Size_t bufsiz; /* Real size of buffer */
155 IV oneword; /* Emergency buffer */
156 } PerlIOBuf;
157
158In this way (as for perl's scalars) a pointer to a PerlIOBuf can be treated
159as a pointer to a PerlIOl.
160
161=head2 Layers in action.
162
163 table perlio unix
164 | |
165 +-----------+ +----------+ +--------+
166 PerlIO ->| |--->| next |--->| NULL |
167 +-----------+ +----------+ +--------+
168 | | | buffer | | fd |
169 +-----------+ | | +--------+
170 | | +----------+
171
172
173The above attempts to show how the layer scheme works in a simple case.
9d799145
NIS
174The application's C<PerlIO *> points to an entry in the table(s)
175representing open (allocated) handles. For example the first three slots
176in the table correspond to C<stdin>,C<stdout> and C<stderr>. The table
177in turn points to the current "top" layer for the handle - in this case
178an instance of the generic buffering layer "perlio". That layer in turn
179points to the next layer down - in this case the lowlevel "unix" layer.
50b80e25 180
9d799145
NIS
181The above is roughly equivalent to a "stdio" buffered stream, but with
182much more flexibility:
50b80e25
NIS
183
184=over 4
185
186=item *
187
9d799145
NIS
188If Unix level C<read>/C<write>/C<lseek> is not appropriate for (say)
189sockets then the "unix" layer can be replaced (at open time or even
190dynamically) with a "socket" layer.
50b80e25
NIS
191
192=item *
193
194Different handles can have different buffering schemes. The "top" layer
195could be the "mmap" layer if reading disk files was quicker using C<mmap>
196than C<read>. An "unbuffered" stream can be implemented simply by
197not having a buffer layer.
198
199=item *
200
201Extra layers can be inserted to process the data as it flows through.
9d799145
NIS
202This was the driving need for including the scheme in perl 5.7.0+ - we
203needed a mechanism to allow data to be translated bewteen perl's
204internal encoding (conceptually at least Unicode as UTF-8), and the
205"native" format used by the system. This is provided by the
206":encoding(xxxx)" layer which typically sits above the buffering layer.
50b80e25
NIS
207
208=item *
209
210A layer can be added that does "\n" to CRLF translation. This layer can be used
211on any platform, not just those that normally do such things.
212
213=back
214
215=head2 Per-instance flag bits
216
9d799145
NIS
217The generic flag bits are a hybrid of C<O_XXXXX> style flags deduced from
218the mode string passed to C<PerlIO_open()>, and state bits for typical buffer
50b80e25
NIS
219layers.
220
9d799145 221=over 4
50b80e25
NIS
222
223=item PERLIO_F_EOF
224
225End of file.
226
227=item PERLIO_F_CANWRITE
228
3039a93d 229Writes are permitted, i.e. opened as "w" or "r+" or "a", etc.
50b80e25
NIS
230
231=item PERLIO_F_CANREAD
232
3039a93d 233Reads are permitted i.e. opened "r" or "w+" (or even "a+" - ick).
50b80e25
NIS
234
235=item PERLIO_F_ERROR
236
9d799145 237An error has occured (for C<PerlIO_error()>)
50b80e25
NIS
238
239=item PERLIO_F_TRUNCATE
240
241Truncate file suggested by open mode.
242
243=item PERLIO_F_APPEND
244
245All writes should be appends.
246
247=item PERLIO_F_CRLF
248
11e1c8f2
PP
249Layer is performing Win32-like "\n" mapped to CR,LF for output and CR,LF
250mapped to "\n" for input. Normally the provided "crlf" layer is the only
251layer that need bother about this. C<PerlIO_binmode()> will mess with this
9d799145
NIS
252flag rather than add/remove layers if the C<PERLIO_K_CANCRLF> bit is set
253for the layers class.
50b80e25
NIS
254
255=item PERLIO_F_UTF8
256
3039a93d 257Data written to this layer should be UTF-8 encoded; data provided
50b80e25
NIS
258by this layer should be considered UTF-8 encoded. Can be set on any layer
259by ":utf8" dummy layer. Also set on ":encoding" layer.
260
261=item PERLIO_F_UNBUF
262
263Layer is unbuffered - i.e. write to next layer down should occur for
264each write to this layer.
265
266=item PERLIO_F_WRBUF
267
268The buffer for this layer currently holds data written to it but not sent
269to next layer.
270
271=item PERLIO_F_RDBUF
272
273The buffer for this layer currently holds unconsumed data read from
274layer below.
275
276=item PERLIO_F_LINEBUF
277
9d799145
NIS
278Layer is line buffered. Write data should be passed to next layer down
279whenever a "\n" is seen. Any data beyond the "\n" should then be
280processed.
50b80e25
NIS
281
282=item PERLIO_F_TEMP
283
9d799145 284File has been C<unlink()>ed, or should be deleted on C<close()>.
50b80e25
NIS
285
286=item PERLIO_F_OPEN
287
288Handle is open.
289
290=item PERLIO_F_FASTGETS
291
9d799145
NIS
292This instance of this layer supports the "fast C<gets>" interface.
293Normally set based on C<PERLIO_K_FASTGETS> for the class and by the
294existance of the function(s) in the table. However a class that
50b80e25
NIS
295normally provides that interface may need to avoid it on a
296particular instance. The "pending" layer needs to do this when
9d799145
NIS
297it is pushed above an layer which does not support the interface.
298(Perl's C<sv_gets()> does not expect the streams fast C<gets> behaviour
50b80e25
NIS
299to change during one "get".)
300
301=back
302
303=head2 Methods in Detail
304
305=over 4
306
307=item IV (*Fileno)(PerlIO *f);
308
9d799145
NIS
309Returns the Unix/Posix numeric file decriptor for the handle. Normally
310C<PerlIOBase_fileno()> (which just asks next layer down) will suffice
311for this.
50b80e25
NIS
312
313=item PerlIO * (*Fdopen)(PerlIO_funcs *tab, int fd, const char *mode);
314
9d799145 315Should (perhaps indirectly) call C<PerlIO_allocate()> to allocate a slot
50b80e25
NIS
316in the table and associate it with the given numeric file descriptor,
317which will be open in an manner compatible with the supplied mode string.
318
319=item PerlIO * (*Open)(PerlIO_funcs *tab, const char *path, const char *mode);
320
9d799145
NIS
321Should attempt to open the given path and if that succeeds then (perhaps
322indirectly) call C<PerlIO_allocate()> to allocate a slot in the table and
323associate it with the layers information for the opened file.
50b80e25
NIS
324
325=item int (*Reopen)(const char *path, const char *mode, PerlIO *f);
326
9d799145
NIS
327Re-open the supplied C<PerlIO *> to connect it to C<path> in C<mode>.
328Returns as success flag. Perl does not use this and L<perlapio> marks it
329as subject to change.
50b80e25
NIS
330
331=item IV (*Pushed)(PerlIO *f,const char *mode,const char *arg,STRLEN len);
332
9d799145
NIS
333Called when the layer is pushed onto the stack. The C<mode> argument may
334be NULL if this occurs post-open. The C<arg> and C<len> will be present
335if an argument string was passed. In most cases this should call
336C<PerlIOBase_pushed()> to convert C<mode> into the appropriate
337C<PERLIO_F_XXXXX> flags in addition to any actions the layer itself takes.
50b80e25
NIS
338
339=item IV (*Popped)(PerlIO *f);
340
9d799145
NIS
341Called when the layer is popped from the stack. A layer will normally be
342popped after C<Close()> is called. But a layer can be popped without being
343closed if the program is dynamically managing layers on the stream. In
344such cases C<Popped()> should free any resources (buffers, translation
345tables, ...) not held directly in the layer's struct.
50b80e25
NIS
346
347=item SSize_t (*Read)(PerlIO *f, void *vbuf, Size_t count);
348
349Basic read operation. Returns actual bytes read, or -1 on an error.
350Typically will call Fill and manipulate pointers (possibly via the API).
9d799145
NIS
351C<PerlIOBuf_read()> may be suitable for derived classes which provide
352"fast gets" methods.
50b80e25
NIS
353
354=item SSize_t (*Unread)(PerlIO *f, const void *vbuf, Size_t count);
355
9d799145
NIS
356A superset of stdio's C<ungetc()>. Should arrange for future reads to
357see the bytes in C<vbuf>. If there is no obviously better implementation
358then C<PerlIOBase_unread()> provides the function by pushing a "fake"
359"pending" layer above the calling layer.
50b80e25
NIS
360
361=item SSize_t (*Write)(PerlIO *f, const void *vbuf, Size_t count);
362
363Basic write operation. Returns bytes written or -1 on an error.
364
365=item IV (*Seek)(PerlIO *f, Off_t offset, int whence);
366
9d799145
NIS
367Position the file pointer. Should normally call its own C<Flush> method and
368then the C<Seek> method of next layer down.
50b80e25
NIS
369
370=item Off_t (*Tell)(PerlIO *f);
371
9d799145
NIS
372Return the file pointer. May be based on layers cached concept of
373position to avoid overhead.
50b80e25
NIS
374
375=item IV (*Close)(PerlIO *f);
376
9d799145
NIS
377Close the stream. Should normally call C<PerlIOBase_close()> to flush
378itself and close layers below, and then deallocate any data structures
379(buffers, translation tables, ...) not held directly in the data
380structure.
50b80e25
NIS
381
382=item IV (*Flush)(PerlIO *f);
383
9d799145
NIS
384Should make stream's state consistent with layers below. That is, any
385buffered write data should be written, and file position of lower layers
386adjusted for data read fron below but not actually consumed.
50b80e25
NIS
387
388=item IV (*Fill)(PerlIO *f);
389
390The buffer for this layer should be filled (for read) from layer below.
391
392=item IV (*Eof)(PerlIO *f);
393
9d799145 394Return end-of-file indicator. C<PerlIOBase_eof()> is normally sufficient.
50b80e25
NIS
395
396=item IV (*Error)(PerlIO *f);
397
9d799145 398Return error indicator. C<PerlIOBase_error()> is normally sufficient.
50b80e25
NIS
399
400=item void (*Clearerr)(PerlIO *f);
401
9d799145
NIS
402Clear end-of-file and error indicators. Should call C<PerlIOBase_clearerr()>
403to set the C<PERLIO_F_XXXXX> flags, which may suffice.
50b80e25
NIS
404
405=item void (*Setlinebuf)(PerlIO *f);
406
407Mark the stream as line buffered.
408
409=item STDCHAR * (*Get_base)(PerlIO *f);
410
411Allocate (if not already done so) the read buffer for this layer and
412return pointer to it.
413
414=item Size_t (*Get_bufsiz)(PerlIO *f);
415
9d799145 416Return the number of bytes that last C<Fill()> put in the buffer.
50b80e25
NIS
417
418=item STDCHAR * (*Get_ptr)(PerlIO *f);
419
3039a93d 420Return the current read pointer relative to this layer's buffer.
50b80e25
NIS
421
422=item SSize_t (*Get_cnt)(PerlIO *f);
423
424Return the number of bytes left to be read in the current buffer.
425
426=item void (*Set_ptrcnt)(PerlIO *f,STDCHAR *ptr,SSize_t cnt);
427
428Adjust the read pointer and count of bytes to match C<ptr> and/or C<cnt>.
429The application (or layer above) must ensure they are consistent.
430(Checking is allowed by the paranoid.)
431
432=back
433
434
435=head2 Core Layers
436
437The file C<perlio.c> provides the following layers:
438
439=over 4
440
441=item "unix"
442
9d799145
NIS
443A basic non-buffered layer which calls Unix/POSIX C<read()>, C<write()>,
444C<lseek()>, C<close()>. No buffering. Even on platforms that distinguish
445between O_TEXT and O_BINARY this layer is always O_BINARY.
50b80e25
NIS
446
447=item "perlio"
448
9d799145
NIS
449A very complete generic buffering layer which provides the whole of
450PerlIO API. It is also intended to be used as a "base class" for other
451layers. (For example its C<Read()> method is implemented in terms of the
452C<Get_cnt()>/C<Get_ptr()>/C<Set_ptrcnt()> methods).
50b80e25 453
9d799145
NIS
454"perlio" over "unix" provides a complete replacement for stdio as seen
455via PerlIO API. This is the default for USE_PERLIO when system's stdio
456does not permit perl's "fast gets" access, and which do not distinguish
457between C<O_TEXT> and C<O_BINARY>.
50b80e25
NIS
458
459=item "stdio"
460
9d799145
NIS
461A layer which provides the PerlIO API via the layer scheme, but
462implements it by calling system's stdio. This is (currently) the default
463if system's stdio provides sufficient access to allow perl's "fast gets"
464access and which do not distinguish between C<O_TEXT> and C<O_BINARY>.
50b80e25
NIS
465
466=item "crlf"
467
9d799145
NIS
468A layer derived using "perlio" as a base class. It provides Win32-like
469"\n" to CR,LF translation. Can either be applied above "perlio" or serve
470as the buffer layer itself. "crlf" over "unix" is the default if system
471distinguishes between C<O_TEXT> and C<O_BINARY> opens. (At some point
472"unix" will be replaced by a "native" Win32 IO layer on that platform,
473as Win32's read/write layer has various drawbacks.) The "crlf" layer is
474a reasonable model for a layer which transforms data in some way.
50b80e25
NIS
475
476=item "mmap"
477
9d799145
NIS
478If Configure detects C<mmap()> functions this layer is provided (with
479"perlio" as a "base") which does "read" operations by mmap()ing the
480file. Performance improvement is marginal on modern systems, so it is
481mainly there as a proof of concept. It is likely to be unbundled from
482the core at some point. The "mmap" layer is a reasonable model for a
483minimalist "derived" layer.
50b80e25
NIS
484
485=item "pending"
486
9d799145
NIS
487An "internal" derivative of "perlio" which can be used to provide
488Unread() function for layers which have no buffer or cannot be bothered.
489(Basically this layer's C<Fill()> pops itself off the stack and so resumes
490reading from layer below.)
50b80e25
NIS
491
492=item "raw"
493
9d799145
NIS
494A dummy layer which never exists on the layer stack. Instead when
495"pushed" it actually pops the stack(!), removing itself, and any other
496layers until it reaches a layer with the class C<PERLIO_K_RAW> bit set.
50b80e25
NIS
497
498=item "utf8"
499
9d799145
NIS
500Another dummy layer. When pushed it pops itself and sets the
501C<PERLIO_F_UTF8> flag on the layer which was (and now is once more) the top
502of the stack.
50b80e25
NIS
503
504=back
505
9d799145
NIS
506In addition F<perlio.c> also provides a number of C<PerlIOBase_xxxx()>
507functions which are intended to be used in the table slots of classes
508which do not need to do anything special for a particular method.
50b80e25
NIS
509
510=head2 Extension Layers
511
512Layers can made available by extension modules.
513
514=over 4
515
516=item "encoding"
517
518 use Encoding;
519
520makes this layer available. It is an example of a layer which takes an argument.
521as it is called as:
522
523 open($fh,"<:encoding(iso-8859-7)",$pathname)
524
525=back
526
527
528=cut
529
530
531