Savannah the next generation

A set of packages to install, upgrade and operate a development hosting facility

This document is in the public domain.

September 2001


Table of Contents

Goals
Implementation

Goals

  • Decentralized. Any given machine running Savannah can host a list of projects (with pointers to the machines hosting the projects), read-only copies of projects (with pointers to the machines hosting the read-write instance of the project), read-write projects. The read-only projects are mirrored from the machine hosting the read-write instance of the project or other machines hosting a read-only copy. The format used is based on XML and defined by CoopX (http://coopx.eu.org). The number of read-only and read-write projects hosted on a given machine is controlled by the maintainer of the machine, depending on the available resources. For instance, an average machine with lots of bandwidth could host at most one hundred read-write projects and ten thousand read-only projects. At a given time, only one machine hosts a read-write project. It is the responsibility of the project maintainers to ensure this. Should a machine become unavailable for some reason, the administrators of the project can toggle the read-only flag of one of the read-only mirrors and start using it as a replacement. Such a decentralized setup is not the most efficient replication system but it is definitely simple and requires very little effort.

  • Multi machine. A single machine may not be powerful enough for someone willing to host a large number of projects. Planning a multi machine setup in advance is mostly impossible because many strategies can be implemented depending on the needs. For instance a site like SourceForge will not implement the same strategies than a site like Geocities because the usage pattern is not the same. A Debian package repository will be used to store the packages that implement various strategies. They can all coexist together. Let's say, for instance, that the package Savannah-single implements a single machine setup and depends on exim, cvs and bind. There can also be a Savannah-multi-cvs, Savannah-multi-exim, Savannah-multi-bind, each installed on a separate machine and that configure the corresponding service for a multi-machine setup.

  • Packaging. Every bit is packaged for Debian. Although it may be desirable to package it for other GNU/Linux distributions, this is not our goal. The cooperative nature of Debian and the package dependency management are essential. When the set of packages reach a stable state it will be possible to contribute them to Debian without a need to find an agreement with a company. The dependency management allows to consistently use a large set of packages. The packages will be produced in a source specific to the project. This source (currently http://france.fsfeurope.org/debian/) contains three types of packages: copy of existing Debian packages slightly modified for the needs of Savannah, packages of unfinished software written for Savannah, packages whose sole purpose is to bind a set of packages together and configure them. The modified copy of existing Debian packages are only stored while waiting for the Debian maintainer or the upstream of the software to integrate the change. It is never meant to be permanent. Communication is always established with the Debian maintainer and the upstream before the modified copy is built. Should we fail to do that, Savannah will become a concurrent Debian distribution and this is to be avoided at all cost. The software specific to Savannah is packaged at a very early stage. Basically the package is done whenever the software performs at least one task. Properly packaging a software takes a lot of time and failing to do it from the very beginning is asking for troubles. It also allows developpers to use the software and contribute to it in an organized way. Finally, since Savannah is a collection of many packages it allows to debug and architecture their interdepencies while writing code.

  • Templates and internationalization. User visible content is stored in templates. The preferred format for HTML pages is XHTML. Every string displayed is stored using gettext files and obeys the system locale.

  • Upgradable. Each release provides an upgrade procedure from the previous version. The upgrade may imply a planned down-time, Savannah is committed to provide an upgrade facility that would work without stopping the service. The XML dump of the database will be used to upgrade the data using XSL files. The database structure itself is upgraded with a set of SQL orders. The associated services (smtp, cvs, etc.) are reset to their initial state (empty) and the backend scripts are run on the upgraded database to restore the desired state.

  • Export/Import using CoopX. CoopX provides a set of XML based schemas to represent all the data related to a project. This includes the description of a user (P3P), contact information (XMLvCard) and custom made schemas for the CVS tree and the bug tracker information for instance. The goal of CoopX is not to promote a standard but to use them where possible. The bug tracker information, for instance, is codified using a DTD with some documentation to understand it. There are no plans to submit it to W3C or other similar organizations. The CoopX format is used for upgrades and for communication between distinct Savannah platforms.

  • Simplicity. The motto is: keep it simple, stupid. Inventing complex solutions, over-generalization, implementing features for future use are common pitfalls that lead to unnecessary complex software. Finding a simple solution often requires more efforts than accepting a complex one.

  • Modularity. The myth of a perfectly modular software is hard to kill. Modularity is an advantage and a burden. Deciding that a feature will live in a module implies that we have to define and maintain an interface for it. This is a significant effort that has to be rapidly compensated by the actual use of various modules. If the perspective of using more than one module is too far in the future, we might as well not bother about it. We need modules but we should not abuse them.

Implementation

The following items are implemented, all are optionals except for the authentication system and the import/export functions.

  • Authentication: anyone can create an account with associated personal data such as full name and public keys (P3P). A logged in user can create a group (or project but in term of authentication it means creating a group to which users can be added). A set of permission bits is granted to each application by the authentication system, the application decides which semantic a specific bit has. A set of users within a group have the ability to modify the permission bits. (priority 0).

  • Export and import projects using the CoopX XML based format that contains all the data necessary to move projects from one platform to another. (priority 0).

  • CVS with ssh/kerberos/gserver/pserver access (priority 1).

  • WebCVS, ViewCVS (priority 1).

  • Tracker declined in support request, bug tracking, patch manager and task manager (priority 1).

  • Forum (priority 2).

  • Shell account (priority 2).

  • Poll (priority 2).

  • HTML pages managed from CVS (priority 1).

  • HTML pages managed from shell account (priority 2).

  • Mailing lists (priority 1).

  • Calendar (priority 2).

  • Contact database (priority 2).

  • Statistics (priority 1).

  • DNS management (priority 1).

  • FTP (priority 1).

  • rsync (priority 1).

  • SQL database access (priority 2).

  • php (priority 2).

1 Core
The core is the server that implements interactions with the user and backend scripts. It can be used thru a web based interface or an xmlrpc interface. phpGroupWare is used as a code base. It provides templates, authentication, interface to SQL database, xmlrpc, session management, permission management with groups and acl. The development version of phpGroupWare (0.9.13) is preferred and close interaction with the development team is maintained. A set of phpGroupWare applications are developed or re-used to implement the Savannah core. Architectural decisions that require changes of the phpGroupWare code base are first discussed with the development team. The changes are carefully evaluated and reduced to the minimal. The most important source of changes comes from the fact that Savannah is project oriented where phpGroupWare is user oriented. The Savannah application implements the user registration, project registration, project membership, project moderation, activation/deactivation of other applications for the project, the home page, searching projects and members, export/import of projects, read-only/read-write toggle of projects. Each functionality available to a project (CVS, WebCVS etc.) is implemented in the core by an existing phpGroupWare application where possible, with a new application if nothing is available. Forking an existing application is not an option : coordination with the maintainer of the application allows to implement the missing functionalities. If an application is of general interest, its inclusion in the phpGroupWare project is discussed before the application is implemented.
2 Documentation
Quick start, reference manual, user guide, implementor guide. The quick start is ten pages at most, including screen shots of the installation process. The intended audience is a GNU/Linux system administrator with basic knowledge of the various services installed and an understanding of the purpose of Savannah. The reference manual documents every component of Savannah and how they work together. It is the ultimate source of information for anyone willing to operate, grow or fix a machine running Savannah. The intended audience is a GNU/Linux system administrator with good knowledge of the various services installed and advanced user of Savannah. The user guide explains in detail how to interact with a machine running Savannah, how to setup a project on it, and how to manage the project. The intended audience is a GNU/Linux user with basic understanding of the GNU/Linux commands or a developer with basic understanding of the GNU/Linux development tools. The implementor guide won't have a fixed structure and provides basic pointers to help volunteers find their way in the project. Only when the project becomes mature enough will the implementor guide be re-worked in a structured way. The intended audience is a skilled GNU/Linux system administrator and developer. All documentation is based on DocBook and also available in texinfo and man pages formats.
3 Back end scripts
Bind the authentication information in the database to the system files/services. Import and export projects automatically or upon request. Update, configure system services for CVS, WebCVS/ViewCVS, trackers, HTML over CVS, mailing lists, statistics, DNS, rsync, FTP, according to the content of the database. Backend scripts have two options to interact with phpgroupware. Either thru the XML dump of the projects or thru xmlrpc. Queries are never run directly on the phpgroupware database. Services with priority two are not meant to be usable in the first stage. Scripts are non intrusive as far as authentication is concerned. It means that user accounts and groups unrelated to Savannah may be created manually by the system administration and are not overriden by Savannah. All backend scripts are designed to be installed independently, the only mandatory scripts being authentication and import/export. Libraries common to all scripts is developed to factorize and normalize interaction with phpgroupware. All scripts are designed to be able to run on a machine that is not the same machine as other services.
4 System administration
Each system service is installed and configured so that it can be used by the backend scripts. This may involve to modify/customize the original distribution in some cases such as WebCVS/ViewCVS, mailing lists, kernel to allow more group by users + services coping with glibc lossage, DNS to access the database instead of a script, chroot'ed environment for security, reverse proxy ability for FTP service etc. Such modifications are packaged and generalized in such a way that they can be accepted by the Debian developer responsible of the package and the upstream maintainers. Interaction with the Debian developer and the upstream maintainer is done before modifying the software.
5 Packaging
The whole set of packages needed to run a Savannah instance on a specific machine can be installed on a Debian GNU/Linux unstable distribution using apt-get install task-Savannah and answering debconf questions on a maximum of five screens. Prior to this, a line has to be added to the sources.list files so that the necessary modified packaged are found if needed. The task-Savannah package is included in the unstable branch and must therefore be able to operate consistently on the basis of unmodified packages. The applications added to the code base, each backend script and modified services have their own package. All packages based on phpgroupware use xmlrpc to configure phpgroupware so that no web based interaction is required from the system administrator. The rationale behind creating a source in addition to the standard unstable is that it can contain packages modified in a way that is completely specific to Savannah. Although this should be kept to the minimum it won't be possible to avoid all cases. For instance a kernel patched with NGROUPS_MAX raised to 512 may be of no interest to Debian in general. Or a chroot'ed MySQL server with a quota patch.

1 Core

The core is the server that implements interactions with the user and backend scripts. It can be used thru a web based interface or an xmlrpc interface. phpGroupWare is used as a code base. It provides templates, authentication, interface to SQL database, xmlrpc, session management, permission management with groups and acl. The development version of phpGroupWare (0.9.13) is preferred and close interaction with the development team is maintained. A set of phpGroupWare applications are developed or re-used to implement the Savannah core. Architectural decisions that require changes of the phpGroupWare code base are first discussed with the development team. The changes are carefully evaluated and reduced to the minimal. The most important source of changes comes from the fact that Savannah is project oriented where phpGroupWare is user oriented. The Savannah application implements the user registration, project registration, project membership, project moderation, activation/deactivation of other applications for the project, the home page, searching projects and members, export/import of projects, read-only/read-write toggle of projects. Each functionality available to a project (CVS, WebCVS etc.) is implemented in the core by an existing phpGroupWare application where possible, with a new application if nothing is available. Forking an existing application is not an option : coordination with the maintainer of the application allows to implement the missing functionalities. If an application is of general interest, its inclusion in the phpGroupWare project is discussed before the application is implemented.

2 Documentation

Quick start, reference manual, user guide, implementor guide. The quick start is ten pages at most, including screen shots of the installation process. The intended audience is a GNU/Linux system administrator with basic knowledge of the various services installed and an understanding of the purpose of Savannah. The reference manual documents every component of Savannah and how they work together. It is the ultimate source of information for anyone willing to operate, grow or fix a machine running Savannah. The intended audience is a GNU/Linux system administrator with good knowledge of the various services installed and advanced user of Savannah. The user guide explains in detail how to interact with a machine running Savannah, how to setup a project on it, and how to manage the project. The intended audience is a GNU/Linux user with basic understanding of the GNU/Linux commands or a developer with basic understanding of the GNU/Linux development tools. The implementor guide won't have a fixed structure and provides basic pointers to help volunteers find their way in the project. Only when the project becomes mature enough will the implementor guide be re-worked in a structured way. The intended audience is a skilled GNU/Linux system administrator and developer. All documentation is based on DocBook and also available in texinfo and man pages formats.

3 Back end scripts

Bind the authentication information in the database to the system files/services. Import and export projects automatically or upon request. Update, configure system services for CVS, WebCVS/ViewCVS, trackers, HTML over CVS, mailing lists, statistics, DNS, rsync, FTP, according to the content of the database. Backend scripts have two options to interact with phpgroupware. Either thru the XML dump of the projects or thru xmlrpc. Queries are never run directly on the phpgroupware database. Services with priority two are not meant to be usable in the first stage. Scripts are non intrusive as far as authentication is concerned. It means that user accounts and groups unrelated to Savannah may be created manually by the system administration and are not overriden by Savannah. All backend scripts are designed to be installed independently, the only mandatory scripts being authentication and import/export. Libraries common to all scripts is developed to factorize and normalize interaction with phpgroupware. All scripts are designed to be able to run on a machine that is not the same machine as other services.

4 System administration

Each system service is installed and configured so that it can be used by the backend scripts. This may involve to modify/customize the original distribution in some cases such as WebCVS/ViewCVS, mailing lists, kernel to allow more group by users + services coping with glibc lossage, DNS to access the database instead of a script, chroot'ed environment for security, reverse proxy ability for FTP service etc. Such modifications are packaged and generalized in such a way that they can be accepted by the Debian developer responsible of the package and the upstream maintainers. Interaction with the Debian developer and the upstream maintainer is done before modifying the software.

5 Packaging

The whole set of packages needed to run a Savannah instance on a specific machine can be installed on a Debian GNU/Linux unstable distribution using apt-get install task-Savannah and answering debconf questions on a maximum of five screens. Prior to this, a line has to be added to the sources.list files so that the necessary modified packaged are found if needed. The task-Savannah package is included in the unstable branch and must therefore be able to operate consistently on the basis of unmodified packages. The applications added to the code base, each backend script and modified services have their own package. All packages based on phpgroupware use xmlrpc to configure phpgroupware so that no web based interaction is required from the system administrator. The rationale behind creating a source in addition to the standard unstable is that it can contain packages modified in a way that is completely specific to Savannah. Although this should be kept to the minimum it won't be possible to avoid all cases. For instance a kernel patched with NGROUPS_MAX raised to 512 may be of no interest to Debian in general. Or a chroot'ed MySQL server with a quota patch.