• Replication and auditing

• Secure communication and look at authorization within a distributed archive

• internal prototype, document results in TR

Version 2 Goals - assuming preservation approach is feasable

• review all v1 components, identify success/failure
  • identify which components need re-written
• Secure communication and look at authorization roles
• Modularize data transfer
• pluggable low-level data store (tsm, srb, etc)
• releaseable beta

• All tomcat/servlet based

• Develop manager
• Develop data nodes

Comments

• cryptographically self-naming packages. Pakcages are identified by sha-256 digest in this prototype.

• Automatic replication of packages

• • package sets are grouped together to allow for nodes to validate data in the sets

• Ingestion of prior-created packages according to pre-defined storage classes.

• • Manager is not authoritative for any package level infomration, merely a cache.
• Create storage classes for placement of packages into fileset groups
• Store location of filesets and fileset groups

• • receive fileset and fileset group information from fileset group masters

• nodes contain filesets which belong to fileset groups.
  • fileset groups actively manage packages stored within.

• • files have minimal types attaches (metadata, manifest, data)

• contains checksum, uuid.

• Work out details on fileset based replication
• pick a uuid - use sha-256 digest of packages for now.

• Replication occurs at the entire fileset level.

• The manager is disposable,
  • all information stored is useful for active preservation
  • Manager is not authoritative for any package - level infomration, merely a cache.
• Create replication plan from ingestion requirements and inform appropriate nodes

• Nodes are authoritative for package information.

• • Packages are just storage containers, and are not aware of contextual information outside it's uuid.

• Creating peer-peer management functionality (reserach project)

ADAPT Middle Layer Prototype

Goals

• Automatic replication of packages according to package requirements
• Create data nodes that support preservation functionality
• Node are able to perform preservation actions unassisted (replication, checking, etc)
• Create simple disposable manager to trigger preservation actions, because nodes do not take initiative
• Scale to 100 nodes
• Focus on data nodes

External services to provide

• Ingestion of prior-created packages according to client-specified requirements
• Query status of packages (acknowledge replication)
• URL based access to packages and items contained within packages
• Location service for packages (on manager)

Technology

• Raw sockets for bulk transfer
• web service for control information (axis)
  • security layered in later w/ wss4j as appropriate
• all tomcat/servlet based

Work Plan

First Steps

• Specify inter-node and external communication (wsdl docs)
• Define Package Layout
• pick a uuid

Components

Manager

• Create replication plan from ingestion requirements
• Store location of nodes
• Cache location of packages and provide lookup service
  • receive updates of package locations from nodes
  • Cached package plans
• Triggers data purges as instructed.
• triggers replication based on file plans as replicas disapper, invalidated, etc

Nodes

• provide data storage
  • self-validate data stored in data store's
• cache package metadata (ID, checksums, etc)
• contains a plan manager to schedule data push to other nodes
  • plans fullfillments supplied from Manager
  • priority given to certain actions (first replica...)
• Access servlet (url) to packages and items in a package
• Allow for external challange of a package's integrity
• Activity history (purge, add, etc...) on pakcages

Package

• Compound object made up of many files
• Simple format, similiar to tar (id,size,data,id,size,data....)
• Contains plan, checksum, uuid.
• package named with global uuid, but internal file id's are package unique only
• 64-bit size

Items that will not be addressed

• Creating metadata and access interfaces outside of demos (PAWN will publish to these)
• Creating of an AIP and organizing metadata/data (it's just data in a package)
• Access control, calls will be secured later with ws-security as time permits
• Versioning and format migration of data.