fs3 Developer Guide

Thomas Davis

01/27/2010


Table of Contents

1. Introduction
1. Purpose
2. Scope
3. Revision History
2. Project Overview
1. What is fs3?
Supported Amazon S3 Features
Unsupported Amazon S3 Features
2. Where is it?
3. License
4. Bugs, features requests and support?
3. Installation
1. System Requirements
2. Single Machine and Client Installation
3. Web Service Installation
4. Core Architecture
1. Use Amazon's S3 Documentation
2. Objectives
3. Transaction Flow
Local Transactions
Remote Transactions
Development Transactions
4. Design Details
5. Authentication
1. Security Documentation
2. Authentication Basics
3. fs3KeyManager
6. Development
1. Checking out the Source
2. Coding Conventions
7. Writing Documentation
1. phpDocumentor
Writing PHP Docs
Installation
Building the Documentation
2. DocBook v5.0
Building the Documentation
3. UML Diagrams
8. Testing
1. The PHPUnit Test Tool
Installing PHPUnit
2. Running Tests
All Automated Tests
A Single Test Suite
All Automated Tests
A. Resources
Bibliography

List of Figures

3.1. Rest Deployment
4.1. Local Transaction Flow Use Case
4.2. Remote Transaction Flow Use Case
4.3. Development Transaction Flow Use Case
4.4. Core API

List of Tables

1.1. Revision History

Chapter 1. Introduction

1. Purpose

The purpose of this document is to describe the fs3 project, its design and usage.

2. Scope

This document assumes that the reader understands PHP and has a basic understanding of what Amazon's S3 service is.

3. Revision History

Table 1.1. Revision History

VersionDateDescription
<A>01/27/2010initial Release
<B>02/02/2010Added new method information
   

Chapter 2. Project Overview

1. What is fs3?

fs3 is an Amazon S3 compatible client and server implementation enabling central ized file storage web services ideal for multi-server applications. Users can de velop potential S3-based applications using only a typical Apache-PHP environmen t.

fs3 is not a replacement for Amazon's S3 service. fs3 runs on a user-controlled server and the API is a subset of Amazon's offering focusing on file management with real files, not databases. The objective is to provide the minimal key services, fast and reliably, usinga typical Apache web server installation. Amazon offers a very robust service with customer support and more features.

Web sites that have the space and bandwidth can use fs3 for file management scalability with multiple servers and have the ability to move to Amazon's solution when demand requires it.

The fs3 remote client fs3RemoteBucket is compaible with Amazon's S3 service, so switching between the fs3 server and Amazon's system will be easier. Amazon S3 clients may be compaitble with fs3 out of the box.

fs3 is not a replacement for Amazon's S3 service! It is intended to allow developers and engineers build robust systems for use on service clusters without the expense of using the clusters until it is actually needed. It is because of Amazon's robust design that it was chosen as a model platform.

Supported Amazon S3 Features

These operations are all that most applications require.

  • Object GET requests using REST API.

  • Object PUT requests using REST API.

  • Object DELETE requests using REST API.

  • Bucket GET requests using REST API with prefix option.

  • Bucket PUT requests using REST API.

  • Bucket DELETE requests using REST API.

  • Authentication is a direct model of Amazon's.

  • Error reporting.

Unsupported Amazon S3 Features

Future support is exapected for most of this. However, these are not nessessary for most applications.

  • The SOAP API.

  • Use of some query options in the REST API.

  • REST API HEAD/COPY/POST Object requests.

  • Use of object names that cannot be saved as filenames.

3. License

The MIT License

Copyright (c) 2010 Thomas Davis (sunsetbrew)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

4. Bugs, features requests and support?

Please report bugs, request features or support using the tracker at http://sourceforge.net/tracker/?group_id=300336. Using this interface will help others who may see the issue and help the project team prioritize and remember it.

As it says in the license, this software is "AS IS". While the project team wants to provide the best software, it does not constitute a waver in the license.

Chapter 3. Installation

1. System Requirements

  • PHP version 5 or newer built with CURL.

2. Single Machine and Client Installation

  • Extract the fs3 archive somewhere on the target system.

  • From within the existing web application use the fs3RemoteBucket for connecting to a remotely hosted web service. Refer to examples/client.php for reference.

  • Alternatively fs3LocalBucket can be used for connecting to a local repository. Moreover, a fs3RemoteBucket can be added as a read-only master to enable a very nice development connection which grabs the data remotely while only adding files to local repository.

3. Web Service Installation

Figure 3.1. Rest Deployment

Rest Deployment

  • Extract the fs3 archive somewhere on the target system.

  • Copy the examples/server.php into the Apache web server docroot.

  • Copy the examples/htaccess.cfg into the Apache web server docroot as .htaccess. This will redirect all requests to the server.php script.

  • Create a repository directory to store the objects in. It must be writable by the web server, but does not need to be in the docroot.

  • Update the configuration of server.php.

  • For production use, the fs3MockKeyManager in server.php must be configured to contain the usable keys.

Chapter 4. Core Architecture

1. Use Amazon's S3 Documentation

This project is based on the Amazon Simple Storage Service. Please refer to the See the Amazon Simple Storage Service Developer Guide [AMAZON-S3] for more details on how the client/server communications works.

2. Objectives

  • Provide a fast and simple file storage service for use with any number of clients.

  • Provide support for developing against large production repositories without the need for copying the respository. This method must protect and forbid modication the production data.

  • Maintain web service compatibility with Amazon's S3 service such that using fs3 can be a migration path to use S3.

  • Easy to deploy and use.

  • Do not require a database.

  • Backwards compatibility must be supported in future versions.

  • Use Amazon's authentication framework.

  • The API must be stateless. It must not prevent the use of distributed solutions.

  • The API must minimize use of system resources.

3. Transaction Flow

Local Transactions

Local transactions take place through immediate function calls just like any software library.

Figure 4.1. Local Transaction Flow Use Case

Local Transaction Flow Use Case

  • User calls a method on fs3LocalBucket which performs actions against a local directory and returns the result synchronously to the user.

Remote Transactions

Remote transactions take place over a network connection. The flow includes two different fs3Bucket implementations and one service.

Figure 4.2. Remote Transaction Flow Use Case

Remote Transaction Flow Use Case

  • User calls a method on fs3RemoteBucket which uses a network based protocol (REST) to communicate the request to a remote service.

  • The remote service then makes the appropriate method call on fs3LocalBucket which performs actions against a local directory.

  • The remote service takes the result and returns the result synchronously to the user.

Development Transactions

A development is a mix of Local and Remote transactions so that all writing is limited to local but reading is done remotely if not found locally first. This is near perfect arrangment for projects with huge file stores and a nead to ensure development instances do not write to the production file stores.

Figure 4.3. Development Transaction Flow Use Case

Development Transaction Flow Use Case

  • User calls a method on fs3LocalBucket which performs actions against a local directory and returns the result synchronously to the user.

  • If the call is to getObject() and the Object cannot be found locally, the fs3LocalBucket will call its read-only master assigned fs3RemoteBucket to get the object remotely. This uses a network based protocol (REST) to communicate the request to a remote service.

  • The remote service then makes the appropriate method call on its own fs3LocalBucket which performs actions against its local directory.

  • The remote service takes the result and returns the result synchronously to the user.

4. Design Details

Figure 4.4. Core API

Core API

  • fs3Bucket - This interface defines all the methods needed to support object storage and retrieval.

  • fs3KeyManager - This interface defines all the methods needed to cross reference between keys and secrets for request authentication.

  • fs3Server - This class encapsulates server-side request handling.

  • fs3MockKeyManager - This class is a simple array based implementation of key manager for use with fs3Server.

  • fs3RemoteBucket - This class implements client-side interface of the API to a remote fs3 or Amazon S3 service.

  • fs3ObjectInfo - This class implements a plain object containing general object information.

  • fs3LocalBucket - This class implments the S3 repository using the local file system. In this context, a bucket is simply a directory and the objects are files in that directory tree.

  • fs3Exception - This class encapsulates the all fs3 exceptions. As of this writing, it is the same as the default Exception with a different name.

  • fs3ErrorCodes - This class encapsulates all of Amazon's S3 error codes. The class has contants for all known errors and methods for coverting from error name, error codes and HTTP response codes.

  • fs3Utils - This class ecapulates code that is common to both the client and server implmentations. There is no real boundary for this class.

Chapter 5. Authentication

The user authentication state machine.

1. Security Documentation

This section covers the configuration of authentication keys on the server.

This API replicated Amazon's S3 method for authenication. Please refer to the Amazon Simple Storage Service Developer Guide [AMAZON-S3] for more details on how the authentication works.

2. Authentication Basics

Authentication is composed of 4 basic portions.

  • Key: string used to identify the user. Very much like a name only it will look more like a junk string. This key is known by the client and sent ot the server as part of the message. The server then uses this key to lookup the secret.

  • Secret: 40 character string used to create a "signature". There is one secret per key. It is known by the client and never sent in the message. The server can find it using the key.

  • Signature: a string alorithmically gernerated from the message contents and the secret. It is sent along with the message to the server. The server then creates its own signature from the secret it looks up and compares them to ensure there is an exact match.

  • Message Contents: Information from the message used when creating a signature. This includes the date, resource name, checksum, amazon headers, content type and HTTP method.

3. fs3KeyManager

Server-side only

The fs3KeyManager interface defines the methods the fs3Server will use to cross reference between keys and secrets.

The class fs3MockKeyManager provides the most trival of implmentation using only an array for cross referencing. However, the data still must be populated from somewhere. It is up to the user to do this.

It would be nice to have a database to store keys in or at least a data file; however, the goal of this project is to keep it simple. It is easy enough to implement the one method interface.

Chapter 6. Development

This section for the developers working in the fs3 project.

1. Checking out the Source

The source code is stored in the Subversion resposity hosted by Source Forge. There are a wide variety of tools available for managing code in Subversion.

The active respository is located at https://fs3.svn.sourceforge.net/svnroot/fs3/trunk/ and is free for any person or company to check out. Like any software project, authorization is required to check in changes. Changes are encouraged and completely welcome. Just contact the project owner to get the needed authorization.

2. Coding Conventions

Coding and style conventions can leads to a reduction in creativity and fun. Developers learn what other people have done and make their own informed choices. Thus far, the majority of conventions used on this project are described in article PHP Application Development Part One [FELLS-PHP]. It is only there to help the developer understand why some things were done a certain way.

Chapter 7. Writing Documentation

This section describes the documentation strategy.

1. phpDocumentor

phpDocumentor is used to process the API documentation directly from the PHP source code.

Writing PHP Docs

phpDocumentor behaves much like JavaDoc and Doc++. It looks for comment blocks starting with /** and uses them for documentation. It also has the smarts to pick apart the source code itself even without comment blocks. Finally it searches comment blocks for tags that start with the @ symbol. The tags can tell it what to do or provide special information like what an argument type and meaning is. See the phpDocumentor web site [PHPDOC-ORG] for more information.

Installation

  
  pear install XML_Parser 
  pear install XML_Util 
  pear install XML_Beautifier 
  pear install PhpDocumentor
      

Building the Documentation

The installation process will create documentation in the dist/htdocs/fs3Api directory.

  
    make clean 
    make phpdoc 
      

2. DocBook v5.0

DocBook version 5.0 is the file format for the User and Developer guides. It is an XML format that is easy to maintain and convert to many other formats. There are a lot of editors for DocBook documents. One such editor is the XMLmind XML Editor which was used to create this document. See DocBook.org [DOCBOOK-ORG] from more information.

Building the Documentation

The installation process will create documentation in the dist/htdocs directory.

  
    make clean 
    make userGuide 
    make devGuide
      

3. UML Diagrams

There are so many of these tools out there and attempts to choose just one, especially one that runs on all platforms to make design diagrams easier failed. It was hard to find a good free one. Since the original developer already owned a license to OmniGraffle, that is in use for now.

Chapter 8. Testing

The How To Test Guide.

1. The PHPUnit Test Tool

This project uses pear package PHPUnit to run the unit tests. See the PHPUnit web site [PHPUNIT-DE] for more details on how to use it.

Installing PHPUnit

 pear channel-discover pear.phpunit.de
 pear install phpunit/PHPUnit

2. Running Tests

All Automated Tests

By all tests we mean, all tests that don't need require connection.

 cd tests
 phpunit --verbose AllTests

A Single Test Suite

The test suites are in the tests directory and have names ending with Test.php by convention.

 cd tests
 phpunit FileIWantToTest.php

All Automated Tests

By all tests we mean, all tests that don't need require connection.

 cd tests
 phpunit --verbose AllTests

Appendix A. Resources

Table of Contents

Bibliography

This section contains a list of references used in this document.

Bibliography

Online Resources

[AMAZON-S3] Amazon Web Services LLC. Amazon Simple Storage Service Developer Guide.

[PHPDOC-ORG] Joshua Eichorn. phpDocumentor.

[PHPUNIT-DE] Sebastian Bergmann. PHPUnit.

[DOCBOOK-ORG] DocBook.org. OASIS DocBook Technical Committee.

[FELLS-PHP] David Fells. PHP Application Development Part One. Developer Shed. 03 March 2005.