Firemap Loader

The firemap database loader program is responsible for incrementally updating the database with new active fire data as it becomes available from the satellite. Since there probably will be a time when different versions of fire data will be available at the same time (e.g., MODIS collection 4, MODIS collection 5) the loader can be configured to load from multiple sources and configured differently for each source. A fire data source is called a fire version in this system and is identified by a prefix as specified in the firemap-schema documentation.

When invoked, the program retreives the current list of fire versions from the database. If a fire version is marked as ingestible, the loader retreives a list from the input source and loads the file into the database if it is new or updates the database if the file has changed. If a change is done to the database, a line will be printed out to the log file and an entry inserted into the load_log table in the database. Any errors that may occur are logged to the specified log file.

Configuration

The program only takes one argument: The path to a properties configuration file. The required database configuration parameters are as follows:

• edu.umd.geog.firms.firemap.Database.Interface: Interface implementation class to use: should be edu.umd.geog.firms.firemap.dbi.postgis.DatabaseInterfaceImpl
• edu.umd.geog.firms.firemap.Database.User: Name of the user to be used when authenticating with the database.
• edu.umd.geog.firms.firemap.Database.Password: Password to be used when authenticating with the database.
• edu.umd.geog.firms.firemap.Database.URL: URL used to connect to the database. Consult the PostgreSQL JDBC guide on the exact format for this, but it is basically in the form of jdbc:postgresql://hostname/database

The loader program has the following configuration parameters:

• edu.umd.geog.firms.firemap.Loader.LogFile: Path of where a log file should be written.
• edu.umd.geog.firms.firemap.Loader.LogLevel.Engine: Set to DEBUG to print out debugging information as the loader does its job. Default is INFO.
• edu.umd.geog.firms.firemap.Loader.LogLevel.Database: Set to DEBUG to print out all the SQL sent to the database, Default is FATAL.

Each fire version in the database marked as ingestible must have the following parameters (where ${prefix} is the fire version prefix):

• edu.umd.geog.firms.firemap.Loader.${prefix}.Driver: Fully qualified name of the load driver class implementation to use. Either edu.umd.geog.firms.firemap.loader.impl.FTPLoadDriver or edu.umd.geog.firms.firemap.loader.impl.FileSystemLoadDriver.
• edu.umd.geog.firms.firemap.Loader.${prefix}.Parser: Fully qualified name of the data set file name parser implementation to use. Should be set to edu.umd.geog.firms.firemap.loader.impl.StandardFireDataSetNameParser. This is the class that parses the date from the filename of the file--change this class if the filename format changes.

For loading fire data from an FTP server, the Driver class implementation should be set to edu.umd.geog.firms.firemap.loader.impl.FTPLoadDriver and the following parameters need to be specified:

• edu.umd.geog.firms.firemap.Loader.${prefix}.FTP.Host: Host of the FTP server to connect to.
• edu.umd.geog.firms.firemap.Loader.${prefix}.FTP.User: Name of the user to use when authenticating with the FTP server.
• edu.umd.geog.firms.firemap.Loader.${prefix}.FTP.Password: Password to use when authenticating with the FTP server.
• edu.umd.geog.firms.firemap.Loader.${prefix}.FTP.Directory: Directory on the FTP server that contains the fire data set files.

For loading fire data from a local filesystem, the Driver class implementation should be set to edu.umd.geog.firms.firemap.loader.impl.FileSystemLoadDriver and the following parameter needs to be specified:

• edu.umd.geog.firms.firemap.Loader.${prefix}.FileSystem.Directory: Directory that contains the fire data set files.

Building from source

To build from source, check out the following projects from the repository:

• firemap-loader
• firemap-schema
• util

The following libraries are also required:

• org.postgresql.jdbc3: PostgreSQL JDBC Driver
• org.apache.commons.net: Provides a FTP interface
• org.apache.oro: Required by the apache commons net.
• org.apache.log4j: Logging library
• com.google.gwt: Required to build firemap-schema

Build the firemap-loader project and a a firemap-loader.jar file and lib directory should appear in the dist directory found in the project's directory. To deploy the program, both the jar file and library directory are needed (the jar must be in the lib's parent directory). Feel free to repackage as needed--this is the way that Netbeans decides to do it. To run as is, simple be in the directory where the jar file is and run:

java -jar firemap-loader.jar

Setup on agni

The loader is currently installed on agni at:

/home/firemap/firemap-loader

In a production environment, it should be placed outside the home directory (in this case, under /export/firemap).

The loader is invoked every hour to check for new data via a cron job. The following entry has been added (all ONE line):

0 * * * * /export/firemap/opt/jdk/bin/java 
-jar /home/firemap/firemap-loader/firemap-loader.jar
/export/firemap/system/etc/firemap/firemap.properties >/dev/null 2>&1

Notes

• The loader will try to load everything it can but will not retry processing on an item if there is a transient error.

-- MikeMcGann - 25 Jan 2007
to top