========================================================================
Project: lcatest
File: README
Author: James Bordner (jbordner@cosmos.ucsd.edu)
Created: 2006-11-07
Updated: 2007-01-04
========================================================================

This is the main README file for the Software Test Environment
"lcatest". More detailed documentation can be found in the doc/
subdirectory.

------------------------------------------------------------------------
FILES
------------------------------------------------------------------------

README Basic documentation.
VERSION lcatest version
~/.lcatestrc Configuration file (in home directory)
lcatest.css HTML Cascading Style Sheet for webpages.
lcatest.config Configuration file created by lcatest

------------------------------------------------------------------------
DIRECTORIES
------------------------------------------------------------------------

bin Python code implementing lcatest
doc Complete documentation of lcatest and Enzo tests
machines Machine settings
projects Project definitions
runs Test run results
tests Test definitions
work Internal lcatest files. Analogous to /proc

------------------------------------------------------------------------
DOWNLOAD
------------------------------------------------------------------------

1. Initialize CVSROOT environment variable

% setenv CVSROOT :pserver:anonymous@jbpc.ucsd.edu:/usr/local/cvs_lca

2. Log into the CVS repository

% cvs login

3. Checkout lcatest

% cvs checkout lcatest

4. PRUNE OUT OLD EMPTY DIRECTORIES. Things may fail if you don't!
This is because of how CVS works (or, rather, how it doesn't).
From the lcatest directory:

% cvs update -d -P

------------------------------------------------------------------------
INSTALL
------------------------------------------------------------------------

First you should compile lcatest and generate the Latex
documentation.

1. Compile lcatest and generate documentation

% make

2. Tell lcatest where it is

2a. lcatest uses the file ~/.lcatestrc for storing the path to
lcatest.

2b. This value can be overridden using the LCATEST_PATH environment
variable.

% setenv LCATEST_PATH <full path to lcatest>

3. Initialize lcatest

% cd bin
% ./lcatest init

Enter project name: [Enzo|enzo-code]
Enter machine name: [sdsc-datastar|paolo-cluster|sdsc-teragrid]
Enter user name: [e.g. Bordner]

The "Enzo" project is the "LLNL" branch of Enzo; the "enzo-code"
project is the "old CVS" branch of Enzo.

------------------------------------------------------------------------
DOCUMENTATION
------------------------------------------------------------------------

Documentation is in lcatest/doc. There should be two PostScript
documents available after the INSTALL step.

1. doc/lcatest.ps documents how to use lcatest software.

2. doc/enzotest.ps documents tests specific to Enzo or enzo-code.

------------------------------------------------------------------------
NEW MACHINE
------------------------------------------------------------------------

Machine settings are stored in the "machines" subdirectory.

The convention for naming machines is <organization>-<machine>
in all lower-case. E.g. sdsc-datastar.

An organization may be a person, e.g. paolo-cluster.

The easiest way to add a new machine is to find a similar machine
in the subdirectory, copy the directory (minus CVS
subdirectories), and edit files as needed.

1. Create a subdirectory for the machine.

% mkdir machines/sdsc-datastar

2. Create a configuration file with the required variables.

% cat > machines/sdsc-datastar/sdsc-datastar.config
cpp_path /usr/lib/cpp -P -traditional-cpp

3. Create a script template file "interactive.template" for running
interactive jobs.

% cat > machines/sdsc-datastar/interactive.template
poe EXE ARGS -nodes NODES -tasks_per_node PROCS_PER_NODE -rmpool 1

4. Create a script template file "batch.template" for running
batch jobs. [NOT IMPLEMENTED YET]

% vi machines/sdsc-datastar/batch.template

...

------------------------------------------------------------------------
NEW PROJECT
------------------------------------------------------------------------

Project definitions are stored in the projects/ subdirectory.

A project is an application, such as Enzo or ZEUS.

Multiple projects can exist for the same code, for example
different versions of Enzo with different makefiles.

1. Create a subdirectory for the project.

% mkdir projects/enzo-code

2. Create a configuration file for the project

% vi projects/enzo-code/enzo-code.config

executable enzo
branch stable
path ~/enzo/enzo-code
src_dir amr_mpi/src
bin_dir bin

------------------------------------------------------------------------
NEW TEST
------------------------------------------------------------------------

Test definitions are in the tests/ subdirectory.

Tests are named Tnnnn.

Tests may be linked--this is done through dependencies.

1. Create a subdirectory for the test.

% mkdir tests/T0162

2. Create a configuration file for the test.

% vi tests/T0162/T0162.config

(see other tests and documentation in doc/

3. Create script templates for each step in the test. Only
step 6 is required.

3.1. Create script for configuring the application before compiling:

% vi tests/T0162/1-configure.script

./configure

3.2. Create script for compiling the application

% 2. vi tests/T0162/2-compile.script

./gmake clean
./gmake
./gmake install

3.3. Create script for preparing for running, e.g. run inits,
ring, etc.

% 3. vi tests/t0162/3-prepare.script

3.4. Create script for executing the application

% 4. vi tests/t0162/4-execute.script

3.5. Create script for post-processing the output, including
creating output data files, plots, etc.

% 5. vi tests/t0162/5-filter.script

3.6. Create a (short) script for determining whether the test
is successful or not, and return "Pass" or "Fail".

% 6. vi tests/T0162/6-evaluate.script

------------------------------------------------------------------------
RUNNING TESTS
------------------------------------------------------------------------

Runs are made within the lcatest subdirectory structure, so when
running on parallel machines, you should move the lcatest directory
to the parallel filesystem (e.g. gpfs).

REMEMBER TO UPDATE THE "path" VARIABLE IN ~/.lcatestrc ACCORDINGLY.

The main executable is lcatest/bin/lcatest.

1. Basic help information.

% lcatest

lcatest
lcatest init
lcatest run [argument ...]
lcatest html
lcatest help [command]

2. Detailed help information

% lcatest help run

lcatest run -category <category1> ... Specify categories
lcatest run -level <level> Specify highest level
lcatest run -force_test <test1> ... Run specified tests
lcatest run -only_test <test1> ... Run only the specified tests
lcatest run -skip <test1> ... Skip specific tests
lcatest run -check Check only (dry run)
lcatest run -summary <summary> Provide run summary

3. Running tests

3.1 Run every test available. NOT RECOMMENDED.

% lcatest run

3.2 Running "fast" level 1 tests

% lcatest run -level 1

3.3 Running all tests in a category

% lcatest run -category TestSuite:Run

3.4 Running all "fast" tests in a category

% lcatest run -category TestSuite:Run -level 1

3.5 Don't actually perform the runs, just list them.

% lcatest run -category TestSuite:Run -level 1 -check

4. Regenerate web pages

% lctest html

5. Remove web pages

% lctest clean

------------------------------------------------------------------------
REPORTING BUGS
------------------------------------------------------------------------

Although lcatest is designed to help find bugs in applications,
there are inevitably bugs in lcatest itself. A bug will look
something like this:

% lcatest html

Traceback (most recent call last):
File "lcatest", line 115, in ?
Command.html(options);
File "/home/bordner/lcatest/bin/Command.py", line 356, in html
Machine.write_html()
File "/home/bordner/lcatest/bin/Machine.py", line 110, in write_html
Global.set_machine (machine,read_config(machine))
File "/home/bordner/lcatest/bin/Machine.py", line 72, in read_config
return lca.read_file(path(machine) + "/" + machine + ".config")
File "/home/bordner/lcatest/bin/lca.py", line 115, in read_file
Error.throw ("nonexistent file", file)
NameError: global name 'Error' is not defined

If you get an error like this:

1. Ensure you're running the latest version of lcatest

% cd lcatest
% cvs update
% make

2. If you still get the error, email the complete error message
traceback to the author. It may not be sufficient for me to
track down the bug, but it is necessary.

% mail jbordner@cosmos.ucsd.edu

Please also e-mail me if you have code suggestions, or if you
find mistakes, inadequacies, or omissions in the documentation.