Add OS-specific modules in Module::Corelist generation script

[perl5.git] / Porting / repository.pod

=head1 NAME

repository - Using the Perl repository

=head1 Synopsis

First, we assume here that you have already decided that you will
need B<write> access to the repository.  If all you need is B<read>
access, there are much better ways to access the most current state of
the perl repository, or explore individual files and patches therein.
See L<perlhack> for details.

This document describes what a Perl Porter needs to do to start using
the Perl repository.

=head1 Prerequisites

You'll need to get hold of the following software.

=over 4

=item Perforce

Download a perforce client from:

   http://www.perforce.com/perforce/loadprog.html

You'll probably also want to look at:

   http://www.perforce.com/perforce/technical.html

where you can look at or download its documentation.

=item ssh

If you don't already have access to an ssh client, then look at its
home site C<http://www.cs.hut.fi/ssh> which mentions ftp sites from
which it's available. You only need to build the client parts (ssh
and ssh-keygen should suffice).

If you're on Windows then you might like to obtain Cygwin from:

    http://cygwin.com/

which contains an ssh client.  (MSYS also contains an ssh client
but it seems to time-out and disconnect from the server and doesn't
understand the ServerAliveInterval setting described later that can
be used to stop Cygwin's ssh client from doing this.)

Alternatively, the "plink" program, part of PuTTY:

    http://www.chiark.greenend.org.uk/~sgtatham/putty/

should also work fine for Windows users.

=back

=head1 Creating an SSH Key Pair

If you already use ssh and want to use the same key pair for perl
repository access then you can skip the rest of this section.
Otherwise, generate an ssh key pair for use with the repository
by typing the command

    ssh-keygen

After generating a key pair and testing it, ssh-keygen will ask you
to enter a filename in which to save the key. The default it offers
will be the file F<~/.ssh/identity> which is suitable unless you
particularly want to keep separate ssh identities for some reason.
If so, you could save the perl repository private key in the file
F<~/.ssh/perl>, for example, but I will use the standard filename
in the remainder of the examples of this document.

After typing in the filename, it will prompt you to type in a
passphrase. The private key will itself be encrypted so that it is
usable only when that passphrase is typed. (When using ssh, you will
be prompted when it requires a pass phrase to unlock a private key.)
If you provide a blank passphrase then no passphrase will be needed
to unlock the key and, as a consequence, anyone who gains access to
the key file gains access to accounts protected with that key
(barring additional configuration to restrict access by IP address).

When you have typed the passphrase in twice, ssh-keygen will confirm
where it has saved the private key (in the filename you gave and
with permissions set to be only readable by you), what your public
key is (don't worry: you don't need to memorise it) and where it
has saved the corresponding public key. The public key is saved in
a filename corresponding to your private key's filename but with
".pub" appended, usually F<~/.ssh/identity.pub>. That public key
can be (but need not be) world readable. It is not used by your
own system at all.

Note that the above process creates a key pair for ssh protocol 1.
You can request ssh protocol 2 (RSA) instead if you prefer (if your
particular ssh client supports it), via the command

    ssh-keygen -t rsa

This will create private/public identity files called F<~/.ssh/id_rsa>
and F<~/.ssh/id_rsa.pub> respectively.  Protocol 2 offers a higher
level of security than protocol 1.  This is not required for access to
the Perl repository -- ssh is used for authentication rather than
encryption (the Perl sources are open anyway) -- but either protocol
is supported by the server.

B<IMPORTANT NOTE FOR CYGWIN USERS:>  In order to make the private key
files only readable by you you must include the string "ntea" in the
"CYGWIN" environment variable in the shell used to run C<chmod(1)>,
and in the shell used to run the ssh client itself later.  If "CYGWIN"
doesn't contain "ntea" then it will appear to the ssh client that the
file permissions are not set correctly, in which case the files will be
ignored and you won't be able to connect.

=head1 Notifying the Repository Keeper

Mail the contents of that public key file to the keeper of the perl
repository (see L</Contact Information> below).
When the key is added to the repository host's configuration file,
you will be able to connect to it with ssh by using the corresponding
private key file (after unlocking it with your chosen passphrase).

There is no harm in creating both protocol 1 and protocol 2 keys and
mailing them both in.  That way you'll be able to connect using either
protocol, which may be useful if you later find yourself using a client
that only supports one or the other protocol.

=head1 Connecting to the Repository

Connections to the repository are made by using ssh to provide a
TCP "tunnel" rather than by using ssh to login to or invoke any
ordinary commands on the repository.

The ssh (secure shell) protocol runs over port number 22, so if you
have a firewall installed at the client end then you must ensure that
it is configured to allow you to make an outgoing connection to port 22
on sickle.activestate.com.

When you want to start a session using the repository, use the command:

    ssh -l perlrep -f -q -x -L 1666:127.0.0.1:1666 sickle.activestate.com foo

If you are not using the default filename of F<~/.ssh/identity> or
F<~/.ssh/id_rsa> to hold your perl repository private key then you'll
need to add the option B<-i filename> to tell ssh where it is. Unless
you chose a blank passphrase for that private key, ssh will prompt you
for the passphrase to unlock that key. Then ssh will fork and put itself
in the background, returning you (silently) to your shell prompt.

Note that the first time you connect you may see a message like
"The authenticity of host 'sickle.activestate.com' can't be established,"
and asking you if you want to continue.  Just answer yes and sickle's
details will be cached in a F<known_hosts> or F<known_hosts2> file.  You
will not see that message again unless you delete the cache file.

The tunnel for repository access is now ready for use.

For the sake of completeness (and for the case where the chosen
port of 1666 is already in use on your machine), I'll briefly
describe what all those ssh arguments are for.

=over 4

=item B<-l perlrep>

Use a remote username of perlrep. (The account on the repository which
provides the end-point of the ssh tunnel is named "perlrep".)

=item B<-f>

Tells ssh to fork and remain running in the background. Since ssh
is only being used for its tunnelling capabilities, the command
that ssh runs never does any I/O and can sit silently in the
background.

=item B<-q>

Tells ssh to be quiet. Without this option, ssh will output a
message each time you use a p4 command (since each p4 command
tunnels over the ssh connection to reach the repository).

=item B<-x>

Tells ssh not to bother to set up a tunnel for X11 connections.
The repository doesn't allow this anyway.

=item B<-L 1666:127.0.0.1:1666>

This is the important option. It tells ssh to listen out for
connections made to port 1666 on your local machine. When such
a connection is made, the ssh client tells the remote side
(the corresponding ssh daemon on the repository) to make a
connection to IP address 127.0.0.1, port 1666. Data flowing
along that connection is tunnelled over the ssh connection
(encrypted). The perforce daemon running on the repository
only accepts connections from localhost and that is exactly
where ssh-tunnelled connections appear to come from.

If port 1666 is already in use on your machine then you can
choose any non-privileged port (a number between 1024 and 65535)
which happens to be free on your machine. It's the first of the
three colon separated values that you should change. Picking
port 2345 would mean changing the option to
B<-L 2345:127.0.0.1:1666>. Whatever port number you choose should
be used for the value of the P4PORT environment variable (q.v.).

=item sickle.activestate.com

This is the canonical name of the host on which the perl repository
resides.

=item foo

This is a dummy place holder argument. Without an argument
here, ssh will try to perform an interactive login to the
repository which is not allowed. Ordinarily, this argument
is for the one-off command which is to be executed on the
remote host. However, the repository's ssh configuration
file uses the "command=" option to force a particular
command to run so the actual value of the argument is
ignored. The command that's actually run merely pauses and
waits for the ssh connection to drop, then exits.

=back

=head1 Problems

You should normally get a prompt that asks for the passphrase
for your RSA key when you connect with the ssh command shown
above.  If you see a prompt that looks like:

    perlrep@sickle.activestate.com's password:

Then you either don't have a F<~/.ssh/identity> or F<~/.ssh/id_rsa>
file corresponding to your public key, or that file is not readable.
Fix the problem and try again.  Alternatively, some ssh implementations
will fail to verify your RSA key if the key if readable by others.
Just lower the permissions to make the key readable to yourself.

If you only had the public key file for one protocol installed at the
server end then make sure your client is using the corresponding
protocol.  An ssh client that supports protocol 2 will probably choose
that by default, which will fail if the server end only has your public
key file for protocol 1.  Some ssh clients have "-1" and "-2" arguments
to force which protocol to use.

The "-v" (verbose) flag can be useful for seeing what protocol your
client is actually trying to connect with, and for spotting any other
problems.  The flag can be specified multiple times to increase
verbosity.  Note that specifying the "-q" flag as well might override
your request for verbose output, so drop the "-q" flag when trying this.

If you're using the Cygwin ssh client on Windows then you will probably
find that the connection times out after a short period of inactivity.
You will have to keep re-entering your passphrase to reconnect, which
gets annoying after a while.  In order to prevent these time-outs from
happening place the following two lines in the file F<~/.ssh/config>:

	Host sickle.activestate.com
	ServerAliveInterval 120

This causes the ssh client to send a message to the server every 120
seconds to check that the server is still alive.  The client will not
disconnect unless "ServerAliveCountMax" many of these messages go
unanswered.  Run C<man ssh_config> for more details.  Note also that
this option applies to protocol version 2 only.

=head1 Using the Perforce Client

Remember to read the documentation for Perforce. You need
to make sure that three environment variable are set
correctly before using the p4 client with the perl repository.

=over 4

=item P4PORT

Set this to localhost:1666 (the port for your ssh client to listen on)
unless that port is already in use on your host. If it is, see
the section above on the B<-L 1666:127.0.0.1:1666> option to ssh.

=item P4CLIENT

The value of this is the name by which Perforce knows your
host's workspace. You need to pick a name (normally, your
Perforce username, a dash, and your host's short name)
when you first start using the perl repository and then
stick with it.

Perforce keeps track of the files you have on your machine.  It
does this through your client. When you first sync a version of a
file, the file comes from the server to your machine.  If you sync
the same file again the server does nothing because it
knows you already have the file.

You should NOT use the same client on different machines.  If you do
you probably won't get the files you expect, and may end up with
nasty corruption.  Perforce allows you to have as many clients as
you want. For example, sally-home, sally-openbsd, sally-laptop.

Also, never change the client's root and view at the same time.
See C<http://www.perforce.com/perforce/doc.002/manuals/p4guide/04_details.html#1048341>

If you have multiple hosts sharing the same directory structure
via NFS then you may be able to get away with only one client name,
but be careful.

The C<p4 clients> command lists all currently known clients.

=item P4USER

This is the username by which perforce knows you. Use your
username if you have a well known or obvious one or else pick
a new one which other perl5-porters will recognise. There is
a licence limit on the number of these usernames, so be sure not
to use more than one.

It is very important to set a password for your Perforce username,
or else anyone can impersonate you.  Use the C<p4 passwd> command
to do this.  Once a password is set for your account, you'll need
to tell Perforce what it is. You can do this by setting the
environment variable P4PASSWD, or you can use the C<-P> flag
with the C<p4> command.

There are a few techniques you can use to avoid having to either
set an environment variable or type the password on every command.
One is to create a shell alias, for example, in bash, add something like
    alias p4='p4 -P secret'
to your F<.bash_profile> file.  Another way is to create a small shell
script, for example
    #!/bin/bash
    p4 -P secret $@
And use this instead of running C<p4> directly.

With either of these, be sure the file containing your password
(the F<.bash_profile> or shell script file) is only readable by you.

The C<p4 users> command lists all currently known users.

=back

Note that on Windows P4PORT and P4USER are requested when installing
Perforce.  They are stored in the registry, so they do not need to be
set in the environment.

Once these three environment variables are set, you can use the
perforce p4 client exactly as described in its documentation.

After setting these variables and connecting to the repository
for the first time, you should use the C<p4 user> command to
set a valid email address for yourself.  Messages to the commit list
are sent (faked) from whatever email address you set here.

Also use the C<p4 client> command to specify your workspace
specifications for each individual client from which you will interact
with the repository.  The P4CLIENT environment variable, of course,
needs to be set to one of these client workspace names.

=head1 Ending a Repository Session

When you have finished a session using the repository, you
should kill off the ssh client process to break the tunnel.
Since ssh forked itself into the background, you'll need to use
something like ps with the appropriate options to find the ssh
process and then kill it manually. The default signal of
SIGTERM is fine.

=head1 Overview of the Repository

Please read at least the introductory sections of the Perforce
User Guide (and perhaps the Quick Start Guide as well) before
reading this section.

Every repository user typically "owns" a "branch" of the mainline
code in the repository.  They hold the "pumpkin" for things in this
area, and are usually the only user who will modify files there.
This is not strictly enforced in order to allow the flexibility
of other users stealing the pumpkin for short periods with the
owner's permission.

Here is (part of) the current structure of the repository:

    /----+-----perl                  - Mainline development (bleadperl)
         +-----perlio                - PerlIO Pumpkin's Perl
         +-----vmsperl               - VMS Pumpkin's Perl
         +-----maint-5.004------perl - Maintenance branches
         +-----maint-5.005------perl
         +-----maint-5.6--------perl
         +-----maint-5.8--------perl
         +-----pureperl---------pureperl

Perforce uses a branching model that simply tracks relationships
between files.  It does not care about directories at all, so
any file can be a branch of any other file--the fully qualified
depot path name (of the form //depot/foo/bar.c) uniquely determines
a file for the purpose of establishing branching relationships.
Since a branch usually involves hundreds of files, such relationships
are typically specified en masse using a branch map (try `p4 help branch`).
`p4 branches` lists the existing branches that have been set up.
`p4 branch -o branchname` can be used to view the map for a particular
branch, if you want to determine the ancestor for a particular set of
files.

The mainline (aka "trunk") code in the Perl repository is under
"//depot/perl/...".  Most branches typically map its entire
contents under a directory that goes by the same name as the branch
name.  Thus the contents of the perlio branch are to be found
in //depot/perlio.

Run `p4 client` to specify how the repository contents should map to
your local disk.  Most users will typically have a client map that
includes at least their entire branch and the contents of the mainline.

Run `p4 changes -l -m10` to check on the activity in the repository.
//depot/perl/Porting/genlog is useful to get an annotated changelog
that shows files and branches.  You can use this listing to determine
if there are any changes in the mainline that you need to merge into
your own branch.  A typical merging session looks like this:

    % cd ~/p4view/perlio
    % p4 integrate -b perlio     # to bring parent changes into perlio
    % p4 resolve -am ./...        # auto merge the changes
    % p4 resolve ./...           # manual merge conflicting changes
    % p4 submit ./...            # check in

If the owner of the mainline wants to bring the changes in perlio
back into the mainline, they do:

    % p4 integrate -r -b perlio
    ...

Generating a patch for change#42 is done as follows:

    % p4genpatch 42 > change-42.patch

F<p4genpatch> is to be found in //depot/perl/Porting/.

The usual routine to apply a patch is

    % p4 edit file.c file.h
    % patch < patch.txt

(any necessary, re-Configure, make regen_headers, make clean, etc, here)

    % make all test

(preferably make all test in several platforms and under several
different Configurations)

    % while unhappy
    do
      $EDITOR
      make all test
    done
    % p4 submit

Other useful Perforce commands

    % p4 describe -du 12345 # show change 12345

Note: the output of "p4 describe" is not in proper diff format, use
the F<Porting/p4genpatch> to get a diff-compatible format.
(Note that it may be easier to get one already prepared: grep
L<perlhack> for APC, and append eg "/diffs/12345.gz" to one of the
URLs to get a usable patch.)

    % p4 diff -se ./...     # have I modified something but forgotten
                            # to "p4 edit", easy faux pas with autogenerated
                            # files like proto.h, or if one forgets to
                            # look carefully which files a patch modifies
    % p4 sync file.h        # if someone else has modified file.h
    % p4 opened             # which files are opened (p4 edit) by me
    % p4 opened -a          # which files are opened by anybody
    % p4 diff -du file.c    # what changes have I done
    % p4 revert file.h      # never mind my changes
    % p4 sync -f argh.c     # forcibly synchronize your file
                            # from the repository
    % p4 diff -sr | p4 -x - revert
                            # throw away (opened but) unchanged files
                            # (in Perforce it's a little bit too easy
                            # to checkin unchanged files)

Integrate patch 12345 from the mainline to the maint-5.6 branch:
(you have to in the directory that has both the mainline and
the maint-5.6/perl as subdirectories)

    % p4 integrate -d perl/...@12345,12345 maint-5.6/perl/...

Integrate patches 12347-12350 from the perlio branch to the mainline:

    % p4 integrate -d perlio/...@12347,12350 perl/...

=head1 Contact Information

The mail alias E<lt>perl-repository-keepers@perl.orgE<gt> can be used to reach
all current users of the repository.

The repository keeper is currently Philippe M. Chiasson
E<lt>gozer@ActiveState.comE<gt>.

=head1 AUTHORS

Malcolm Beattie, E<lt>mbeattie@sable.ox.ac.ukE<gt>, 24 June 1997.

Gurusamy Sarathy, E<lt>gsar@activestate.comE<gt>, 8 May 1999.

Slightly updated by Simon Cozens, E<lt>simon@brecon.co.ukE<gt>, 3 July 2000.

More updates by Jarkko Hietaniemi, E<lt>jhi@iki.fiE<gt>, 28 June 2001.

Perforce clarifications by Randall Gellens, E<lt>rcg@users.sourceforge.netE<gt>, 12 July 2001.

Windows-related updates by Steve Hay E<lt>shay@cpan.orgE<gt>, 23 July 2004
and 08 Aug 2005.

=cut