============
PyPyBuilder
============

What is this?
=============

PyPyBuilder is an application that allows people to build PyPy instances on
demand. If you have a nice idle machine connected to the Internet, and don't
mind us 'borrowing' it every once in a while, you can start up the client
script (in bin/client) and have the server send compile jobs to your machine.
If someone requests a build of PyPy that is not already available on the PyPy
website, and your machine is capable of making such a build, the server may ask
your machine to create it. If enough people participate, with diverse enough
machines, a 'build farm' is created.

Quick usage instructions
========================

For the impatient, that just want to get started, some quick instructions.

First you'll need to have a checkout of the 'buildtool' package, that can
be found here::

  https://codespeak.net/svn/pypy/build/buildtool

To start a compilation, run (from the buildtool root directory)::

  $ ./bin/startcompile.py [options] <email address>

where the options can be found by using --help, and the email address will be
used to send mail to once the compilation is finished.

To start a build server, to participate in the build farm, do::

  $ ./bin/buildserver.py

That's it for the compilation script and build server, if you have your own
project and want to set up your own meta server, you'll have to be a bit more
patient and read the details below...

Components
==========

The application consists of 3 main components: a meta server component, a
client component that handles compilations (let's call this a 'build server')
and a small client component to start compile jobs (which we'll call
'requesting clients' for now).

The server waits for build server to register, and for compile job
requests. When participating clients register, they pass the server information
about what compilations the system can handle (system info), and a set of
options to use for compilation (compile info).

When now a requesting client requests a compilation job, the server checks
whether a suitable binary is already available based on the system and compile
info, and if so returns that. If there isn't one, the server walks through a
list of connected participating clients to see if one of them can handle the
job, and if so dispatches the compilation. If there's no participating client
to handle the job, it gets queued until there is.

If a client crashes during compilation, the build is restarted, or error
information is sent to the logs and requesting client, depending on the type of
error. As long as no compilation error occurs (read: on disconnects, system
errors, etc.) compilation will be retried until a build is available.

Once a build is available, the server will send an email to all clients waiting
for the build (it could be that more than one person asked for some build at
the same time!).

Configuration
=============

There are several aspects to configuration on this system. Of course, for the
meta server, build server and startcompile components there is configuration
for the host and port to connect to, and there is some additional configuration
for things like which mailhost to use (only applies to the server), but also
there is configuration data passed around to determine what client is picked,
and what the client needs to compile exactly.

Config file
-----------

The host/port configuration etc. can be found in the file 'config.py' in the
build tool dir. There are several things that can be configured here, mostly
related to what application to build, and where to build it. Please read the
file carefully when setting up a new build network, or when participating for
compilation, because certain items (e.g. the svnpath_to_url function, or the
client_checkers) can make the system a lot less secure when not configured
properly.

Note that all client-related configuration is done from command-line switches,
so the configuration file is supposed to be changed on a per-project basis:
unless you have specific needs, use a test version of the build tool, or are
working on another project than PyPy, you will not want to modify the it.

System configuration
--------------------

This information is used by the client and startcompile components. On the
participating clients this information is retrieved by querying the system, on
the requesting clients the system values are used by default, but may be
overridden (so a requesting client running an x86 can still request PPC builds,
for instance). The clients compare their own system config to that of a build
request, and will (should) refuse a build if it can not be executed because
of incompatibilities.

Compilation configuration
-------------------------

The third form of configuration is that of the to-be-built application itself,
its compilation arguments. This configuration is only provided by the
requesting clients, build servers can examine the information and refuse a
compilation based on this configuration (just like with the system config, see
'client_checkers' in 'config.py'). Compilation configuration can be controlled
using command-line arguments (use 'bin/startcompile.py --help' for an
overview).

Build tool options
------------------

Yet another part of the configuration are the options that are used by the
startcompile.py script itself: the user can specify what SVN path (relative to
a certain base path) and what Subversion revision is desired.  The revision can
either be specified exactly, or as a range of versions.

Installation
============

Build Server
------------

Installing the system should not be required: just run './bin/buildserver' to
start. Note that it depends on the `py lib`_ (as does the rest of PyPy).

When starting a build server with PyPy's default configuration, it will connect
to a meta server we have running in codespeak.net.

Meta Server
-----------

Also for the server there's no real setup required, and again there's a 
dependency on the `py lib`_. Starting it is done by running
'./bin/metaserver'.

Running a compile job
---------------------

Again installation is not required, just run './bin/startcompile.py [options]
<email>' (see --help for the options) to start. Again, you need to have the
`py lib`_ installed.

Normally the codespeak.net meta server will be used when this script is issued.

.. _`py lib`: http://codespeak.net/py

Using the build tool for other projects
=======================================

The code for the build tool is meant to be generic. Using it for other projects
than PyPy (for which it was originally written) is relatively straight-forward:
just change the configuration, and implement a build client script (probably
highly resembling bin/buildserver.py).

Note that there is a test project in 'tool/build/testproject' that can serve
as an example.

Prerequisities
--------------

Your project can use the build tool if:

  * it can be built from Python

    Of course this is a rather vague requirement: theoretically _anything_ can
    be built from Python; it's just a matter of integrating it into the tool
    properly... A project that can entirely be built from Python code (like
    PyPy) is easier to integrate than something that is built from the command
    line, though (although implementing that won't be very hard either, see
    the test project for instance).

  * it is located in Subversion

    The build tool makes very little hard-coded assumptions, but having code
    in Subversion is one of them. There are several locations in the code where
    SVN is assumed: the command line options (see `build tool options`_),
    the server (which checks SVN urls for validity, and converts HEAD revision
    requests to actual revision ids) and and build client (which checks out the
    data) all make this assumption, changing to a different revision control
    system is currently not easy and unsupported (but who knows what the future
    will bring).

  * it uses PyPy's config mechanism

    PyPy has a very nice, generic configuration mechanism (essentially wrapper
    OptionParser stuff) that makes dealing with fragmented configuration
    and command-line options a lot easier. This mechanism is used by the build
    tool: it assumes configuration is provided in this format. If your project
    uses this configuration mechanism already, you can provide the root Config
    object from config.compile_config; if not it should be fairly straight-
    forward to wrap your existing configuration with the PyPy stuff.

Basically that's it: if your project is stored in SVN, and you don't mind using
Python a bit, it shouldn't be too hard to get things going (note that more
documentation about this subject will follow in the future).

Web Front-End
=============

To examine the status of the meta server, connected build servers and build
requests, there is a web server available. This can be started using
'./bin/webserver' and uses port 8080 by default (override in
config.py).

The web server presents a number of different pages:

  * / and /metaserverstatus - meta server status

    this displays a small list of information about the meta server, such
    as the amount of connected build servers, the amount of builds available,
    the amount of waiting clients, etc.

  * /buildservers - connected build servers

    this page contains a list of all connected build servers, system
    information and what build they're currently working on (if any)

  * /builds - a list of builds

    here you'll find a list of all builds, both done and in-progress and
    queued ones, with links to the details pages, the date they were
    requested and their status

  * /build/<id> - build details

    the 'build' (virtual) directory contains pages of information for each
    build - each of those pages displays status information, time requested,
    time started and finished (if appropriate), links to the zip and logs,
    and system and compile information

There's a build tool status web server for the meta server on codespeak.net
available at http://codespeak.net/pypy/buildstatus/.

More info
=========

For more information, bug reports, patches, etc., please send an email to 
guido@merlinux.de.

