dulwich/README.swift

Openstack Swift as backend for Dulwich
======================================
Fabien Boucher <fabien.boucher@enovance.com>

The module dulwich/contrib/swift.py implements dulwich.repo.BaseRepo
in order to being compatible with Openstack Swift.
We can then use Dulwich as server (Git server) and instead of using
a regular POSIX file system to store repository objects we use the
object storage Swift via its own API.

c Git client <---> Dulwich server <---> Openstack Swift API

This implementation is still a work in progress and we can say that
is a Beta version so you need to be prepared to find bugs.

Configuration file
------------------

We need to provide some configuration values in order to let Dulwich
talk and authenticate against Swift. The following config file must
be used as template:

    [swift]
    # Authentication URL (Keystone or Swift)
    auth_url = http://127.0.0.1:5000/v2.0
    # Authentication version to use
    auth_ver = 2
    # The tenant and username separated by a semicolon
    username = admin;admin
    # The user password
    password = pass
    # The Object storage region to use (auth v2) (Default RegionOne)
    region_name = RegionOne
    # The Object storage endpoint URL to use (auth v2) (Default internalURL)
    endpoint_type = internalURL
    # Concurrency to use for parallel tasks (Default 10)
    concurrency = 10
    # Size of the HTTP pool (Default 10)
    http_pool_length = 10
    # Timeout delay for HTTP connections (Default 20)
    http_timeout = 20
    # Chunk size to read from pack (Bytes) (Default 12228)
    chunk_length = 12228
    # Cache size (MBytes) (Default 20)
    cache_length = 20


Note that for now we use the same tenant to perform the requests
against Swift. Therefor there is only one Swift account used
for storing repositories. Each repository will be contained in
a Swift container.

How to start unittest
---------------------

There is no need to have a Swift cluster running to run the unitests.
Just run the following command in the Dulwich source directory:

    $ PYTHONPATH=. python -m dulwich.contrib.test_swift

How to start functional tests
-----------------------------

We provide some basic tests to perform smoke tests against a real Swift
cluster. To run those functional tests you need a properly configured
configuration file. The tests can be run as follow:

    $ DULWICH_SWIFT_CFG=/etc/swift-dul.conf PYTHONPATH=. python -m dulwich.contrib.test_swift_smoke

How to install
--------------

Install the Dulwich library via the setup.py. The dependencies will be
automatically retrieved from pypi:

    $ python ./setup.py install

How to run the server
---------------------

Start the server using the following command:

    $ python -m dulwich.contrib.swift daemon -c /etc/swift-dul.conf -l 127.0.0.1

Note that a lot of request will be performed against the Swift
cluster so it is better to start the Dulwich server as close
as possible of the Swift proxy. The best solution is to run
the server on the Swift proxy node to reduce the latency.

How to use
----------

Once you have validated that the functional tests is working as expected and
the server is running we can init a bare repository. Run this
command with the name of the repository to create:

    $ python -m dulwich.contrib.swift init -c /etc/swift-dul.conf edeploy

The repository name will be the container that will contain all the Git
objects for the repository. Then standard c Git client can be used to
perform operations againt this repository.

As an example we can clone the previously empty bare repository:

    $ git clone git://localhost/edeploy

Then push an existing project in it:

    $ git clone https://github.com/enovance/edeploy.git edeployclone
    $ cd edeployclone
    $ git remote add alt git://localhost/edeploy
    $ git push alt master
    $ git ls-remote alt
    9dc50a9a9bff1e232a74e365707f22a62492183e        HEAD
    9dc50a9a9bff1e232a74e365707f22a62492183e        refs/heads/master

The other Git commands can be used the way you do usually against
a regular repository.

Note the daemon subcommands starts a Git server listening for the
Git protocol. Therefor there is no authentication or encryption
at all between the cGIT client and the GIT server (Dulwich).

Note on the .info file for pack object
--------------------------------------

The Swift interface of Dulwich relies only on the pack format
to store Git objects. Instead of using only an index (pack-sha.idx)
along with the pack, we add a second file (pack-sha.info). This file
is automatically created when a client pushes some references on the
repository. The purpose of this file is to speed up pack creation
server side when a client fetches some references. Currently this
.info format is not optimized and may change in future.