• sync partial directories (dearp)

• sync partial directories (dearp)

• remove individual directories from monitoring. (dearp)
• Create report showing which items are not currently active. (dearp)

• Create report showing which items are not currently active. (dearp)

• Fix ordering of items on status screen. (dearp)
• Link errors to directory or file (dearp)
  • logging handles this, just need to finish gui. masterfile and replicafile in gui finished, replica site and master site needs done still.
  • Filter logs based on collection

• replica sites and master site must be federated.
• Account listed in the master site is the one used for replication to all child sites. It must exist on all replica sites

A common practice that works for replicating is to create a second account rooted in the replica zone, give this account read access to the master collection, use this account for the master and replica account. This provides a degree of safety in that this account cannot change the original copy.

• Name - descriptive name for the collection
• SRB Account - account that will be used to replicate this collection. Account must have read access to master collection
• path - SRB directory containing the master collection.

• SRB Account - account to use for browsing and selecting a resource only (not used during replication)
• resource - destination resource
• path - destination path.

• • SRB Setup
  • Collection Configuration
  • Event Logs

Files

• schema-export.sql: SQL Schema
• srb-monitor.war: webapp
• context.xml: webapp configuration

3. Install webapp

Install tomcat on your server and verify it's working. Shutdown the server and place a copy of the srb-monitor.war file in your TOMCAT/webapps directory. Place a copy of the configuration file srb-monitor.xml in your TOMCAT/conf/Catalina/localhost directory. Open up the configuration file and edit the following settings.

...
    <Parameter name="hibernate.connection.username" value="monitor" />
    <Parameter name="hibernate.connection.password" value="YOUR_PASSWORD" />
    <Parameter name="hibernate.connection.url" value="jdbc:mysql://localhost/srbmon" />
...
...
    <Realm className="org.apache.catalina.realm.JDBCRealm"
        driverName="org.gjt.mm.mysql.Driver"
        connectionURL="jdbc:mysql://localhost/srbmon" 
            connectionName="monitor" connectionPassword="YOUR_PASSWORD"
        userTable="USERS" userNameCol="username" userCredCol="password"
        userRoleTable="USER_ROLES" roleNameCol="rolename"
    />
...

• connectionPassword and hibernate.connection.password should be set to the password you used above
• if your database is running on a different server, or you used a username other than monitor when creating your database, update the above sections accordingly

If you haven't already done so, you will need to place a copy of the mysql connector in your tomcat TOMCAT/common/lib directory so the monitor can use the database for authentication.

You can now restart tomcat and browse to http://localhost:8080/srb-monitor where localhost is the name of the machine you installed the monitor on.

4. Register your first collection

When the monitor starts up, you will see a status screen with no collections.

• Click on SRB Configuration. You will be prompted for a username and password, enter 'admin' for both. Fill in the six blanks with an account that has read access to your collection. Click Add when finished.
• Click on Collection Configuration. Enter a name of your collection, and choose the path in the SRB where you collection is located. When finished click Add.
• Click on Add Replica on the left. Select a resource that will store your replica, and then select the destination path for your collection. Click Add when finished.

5. Start the sync

Click on the Status tab, click your collection, then click Synchronize Mirrors at the bottom. After a few seconds, you should see files in your collection listed, and the counts for replicated items start to increase.

• Status - view current status and browse collections
• Collections - register new collections and replica's for those collections
• SRB Configuration - Register accounts that will be used to access collections.
• Event Log - detailed event list of everything the replica monitor has done.
• Accounts - Control access to the replica monitor.

• Collection Name: descriptive name of the collection, preceded with a red icon showing no syncing is in progress or a green icon showing a sync is actively running.
• Site Status - State of the master site as of the last sync.
• Total Files - Count of monitored files, this will not be accurate until after the first complete sync has finished.

• Stop / Start sync - Start or start the syncing of master site to replica sites.
• Collection Configuration - Go to the collection configuration screen for this collection.
• Browse - browse and display the current status of any file in the collection

Collection Configuration

Event Logs

The event logs track every action that occurs while syncing a collection. As the logs can grow quite large, there are a number of filtering options available. The beginning and end of every sync is recorded as well as any new files or new replicas that were created and any errors that may have occurred. For a new file, you will generally see 3 entries per file, an entry for a new master file discovered, and two entries for the beginning and ending of the replica creation.

• Errors - show only errors that occurred during transfer. This is any error from master site to replica
• Missing Files - Logged when a master file has gone missing.
• New Master Items - Logged when a new master file has been found
• New Replica - show when a new replica was created.
• Sync Start/Stop - show when syncs started and stopped, good for showing over time involved in syncing a collection
• Collections - limit the displayed errors to a given collection.

• Show per page: display 20, 50, or 100 log items per page
• |< >| - start at beginning or end of logs
• << >> - show previous or next page

• accounts-new-web-small.png:
  

• accounts-roles-web-small.png:
  

The settings that most will want to change are:

    <!-- Time after which a replica check is needed, in hours -->
    <Parameter name="edu.umiacs.checkTime" value="168" />
    <!-- Max number of running replica's, if set to 0 or blank, then no limit -->
    <Parameter name="edu.umiacs.maxSyncThreads" value="5" />
    <!-- Max number of attempts to copy a file if we get an error -->
    <Parameter name="edu.umiacs.maxRetry" value="5" />
    <!-- Should we schedule sync's or require them to be run manually -->
    <Parameter name="edu.umiacs.autoSync" value="false" />

• • low priority, startup cost even in large (2m collections) is only a few hours.

• • Filter logs based on collection

• login for admin functions (web.xml filter...), and accounts creation page (dearp)

• Add stop replication button

• Highlight currently selected item in status window
• remove individual directories from monitoring.

• • logging handles this, just need to finish gui. masterfile and replicafile in gui finished, replica site and master site needs done still.
  • Filter logs based on collection
• Create report showing which items are not currently active.

• Files

2. Create database

Create a new database called 'srbmon', grant permissions and setup table structure.

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2 to server version: 4.1.20

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> create database srbmon;
Query OK, 1 row affected (0.01 sec)

mysql> grant all on srbmon.* to monitor@localhost identified by 'YOUR_PASSWORD';
Query OK, 0 rows affected (0.00 sec)

mysql> use srbmon;
Database changed
mysql> source WHERE_YOU_DOWNLOADED_SCHEMA.sql;
ERROR 1146 (42S02): Table 'srbmon.ACTIVITY_LOG_ENTRY' doesn't exist
ERROR 1146 (42S02): Table 'srbmon.ACTIVITY_LOG_ENTRY' doesn't exist
...
...
Query OK, 0 rows affected (0.00 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> exit
Bye

Change WHERE_YOU_DOWNLOADED_SCHEMA.sql to correspond to the location you downloaded the schema listed below. Set YOUR_PASSWORD to a password to use for this database.

Files

• schema-export.sql: SQL Schema

• Fix ordering of items on status screen.

• get total file count prior to starting replica
• Make interface IE proof
• Move hibernate config into context.xml

• get total file count prior to starting replica

• when determining what replica to run, choose the one w/ the greatest time since last checked.

• Check hourly for stuff that has passed it's check time, while respecting max run

1. Requirements

• MySql (tested on 4.1)
• Apache/Jarkata Tomcat (5.5 or higher)

2. Create database

Create a new database called 'srbmon'

3. Install webapp

Install tomcat on your server and verify it's working. Shutdown the server and place a copy of the xxx.war file in your webapps directory. Place a copy of the configuration file xxx.xml in your conf/Catalina/localhost directory. Open up the configuration file and edit the following settings

• Move hibernate config into context.xml

• login for admin functions (web.xml filter...), and accounts creation page (dearp)

• • logging handles this, just need to finish gui. masterfile and replicafile in gui finished, replica site and master site needs done still

• Fix bug where files don't get marked active if they are replicated

• Quick Setup
• Usage
  • Status
  • Adding Collections
  • Viewing Errors
  • Account Management
  • System Configuration
• Implementation

Quick Setup

Usage

Status

Adding Collections

Viewing Errors

Account Management

System Configuration

• Make interface IE proof
• login for admin functions (web.xml filter...) (dearp)
• Checkpointing during sync of collections in case of restart or crash. This should be done per directory.
• Add stop replication button
• Allow collections and replica's to have excluded directories
• Link errors to directory or file
• Fix ordering of items on status screen.
• when determining what replica to run, choose the one w/ the greatest time since last checked.

Logging

Implementation

The Monitor consists of three parts, first a database backend that tracks all monitored files, configuration, and logging information. Second a sync thread that runs across each collection, looking for new items and logging any changes. Third an interface for configuring collections and viewing the results of a synchronization.

+

SRB Replication Monitor

• Overview
• TODO

Overview

The SRB Replica monitor is a simple webapp that will watch registered directories and ensure that copies exist at designated mirrors. The monitor stores enough information to know if files have been removed from the master site and when the last time a file was seen. In addition any action that the webapp takes on files is logged. The monitor does NOT do any type of integrity checking, this is the responsibility of additional components.

TODO

• Move hibernate config into context.xml
• status monitoring for running threads
• start threads in background from webapp startup
• what happens when replicas are removed when a thread is running.
• Clean up the interface.
• bug - cross-zone copies don't always work, why
• login for admin functions (filter...)

-- MikeSmorul - 25 Aug 2006