PressurelessCollapse test seems broken

[sskory]

PressurelessCollapse seems to be broken. Here's what I get on Kraken:

stderr

sskory@kraken-pwd1:~/l/PressurelessCollapse> aprun -n 1 ./enzo.exe -d PressurelessCollapse.enzo > 01.out
warning: the following parameter line was not interpreted:
SubcycleSafetyFactor   = 2       // 
****** GetUnits:  1.000000e+00 1.000000e+00 1.000000e+00 1.000000e+00 *******
CWD /lustre/scratch/sskory/PressurelessCollapse
Global Dir set to /lustre/scratch/sskory/PressurelessCollapse
PressurelessCollapse: check CellFlaggingMethod.
InitializeExternalBoundaryFace
SimpleConstantBoundary FALSE
End of set exterior
Successfully read in parameter file PressurelessCollapse.enzo.
INITIALIZATION TIME =   4.29391861e-03
DATA dump: /lustre/scratch/sskory/PressurelessCollapse/DD0000/data
WriteAllData: writing group file /lustre/scratch/sskory/PressurelessCollapse/DD0000/data0000.cpu0000
DATA dump: dumpdirname=(/lustre/scratch/sskory/PressurelessCollapse/DD0000) == unixresult=0
Continuation Flag = 1
 stop in euler with e < 0
aborting job:
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0

stdout

MPI_Init: NumberOfProcessors = 1
=========================
Enzo SVN Branch   http://mngrid.ucsd.edu/svn/Enzo/devel/trunk
Enzo SVN Revision 1822
=========================
InitializeNew: Starting problem initialization.
InitializeNew: Finished problem initialization.
Initialize Exterior
ExtBndry: BoundaryRank = 1
ExtBndry: GridDimension = 106 
ExtBndry: NumberOfBaryonFields = 3
InitializeNew: Initial grid hierarchy set
InitializeNew: Partition Initial Grid 0
RebuildHierarchy: level = 0
CommunicationTransferParticles: moved = 0
TopGrid dt = 0.370000     time = 0    cycle = 0
Level[0]: dt = 0.37  0  (0.37/0.37)
n=0  dim=0  ldim=2  CycleNumber=0
 euler: dnu <= 0:                        4                        1
  -9.4091000473794849E-010   1.0000000000000000E-010    37.00000000000000      
   0.3700000000000000       -0.1418286001479890       -0.1418286001479890     
    0.000000000000000        1.0000000000000000E-010   9.9999999999999991E-011
    0.000000000000000        0.1418404323075552        0.4231674606151089     
    0.000000000000000        9.9999999999999995E-021   9.9999999999999995E-021
    0.000000000000000        1.0000000000000000E-010   9.9999999999999991E-011
    0.000000000000000        0.1418404323075552        0.4231674606151089     
    0.000000000000000        9.9999999999999995E-021   9.9999999999999995E-021
    0.000000000000000        0.1418404323075552        0.4231674606151089     
    0.000000000000000        9.9999999999999995E-021   9.9999999999999995E-021
    0.000000000000000        1.0000000000000000E-010   9.9999999999999991E-011
    0.000000000000000        1.0000000000000000E-010    1.000000000000000     
    0.000000000000000        0.1418404323075552        0.4231674606151089     
    0.000000000000000        0.1418404323075552        0.4231674606151089     
    0.000000000000000        9.9999999999999995E-021   9.9999999999999995E-021
    0.000000000000000        9.9999999999999995E-021   9.9999999999999995E-021
   1.0000000000000000E-010   1.0000000000000000E-010    1.000000000000000     
    0.000000000000000         0.000000000000000         0.000000000000000     
    0.000000000000000         0.000000000000000         0.000000000000000     
   1.0000000000000000E-010   1.0000000000000000E-010   1.0000000000000000E-010
    0.000000000000000         0.000000000000000         0.000000000000000     
    0.000000000000000         3.066820158001193         6.082746558001162     
==================
=== ENZO ERROR ===   euler.src: 491   node 0
==================
Application 171075 exit codes: 255
Application 171075 resources: utime 0, stime 0

gmake show-config

   MACHINE: NICS Kraken

   PARAMETER_MAX_SUBGRIDS:       100000
   PARAMETER_MAX_BARYONS:        20
   PARAMETER_MAX_TASKS_PER_NODE: 8

   CONFIG_PRECISION:             64
   CONFIG_PARTICLES:             64
   CONFIG_INTEGERS:              64
   CONFIG_INITS:                 64
   CONFIG_IO:                    64
   CONFIG_USE_MPI:               yes
   CONFIG_OBJECT_MODE:           64
   CONFIG_TASKMAP:               no
   CONFIG_PACKED_AMR:            yes
   CONFIG_PACKED_MEM:            no
   CONFIG_JBPERF:                no
   CONFIG_PAPI:                  no
   CONFIG_UNIGRID_TRANSPOSE:     no
   CONFIG_OOC_BOUNDARY:          no
   CONFIG_OPT:                   debug
   CONFIG_TESTING:               no
   CONFIG_ISOBCS:                yes
   CONFIG_TPVEL:                 no

PressurelessCollapse.enzo

#
# AMR PROBLEM DEFINITION FILE: Pressureless collapse
#
#  define problem
#
ProblemType            = 21      // Pressureless collapse
TopGridRank            = 1
TopGridDimensions      = 100
SelfGravity            = 1       // gravity on
TopGridGravityBoundary = 1       // Isolated BCs
LeftFaceBoundaryCondition  = 1    // outflow ?
RightFaceBoundaryCondition = 1    // outflow ?
PressureFree           = 1       // turn off pressure
#
#  set I/O and stop/start parameters
#
StopTime               = 0.37
dtDataDump             = 0.4
#
#  set hydro parameters
#
Gamma                  = 1.4
CourantSafetyNumber    = 0.05    // needs to be lower for pressurefree
PPMDiffusionParameter  = 0       // diffusion off
#
#  set grid refinement parameters
#
StaticHierarchy           = 1    // dynamic hierarchy
MaximumRefinementLevel    = 1    // use up to 2 levels
RefineBy                  = 4    // refinement factor
MinimumSlopeForRefinement = 0.2  // set this to <= 0.2 to refine CD
#
#  set some global parameters
#
SubcycleSafetyFactor   = 2       // 
tiny_number            = 1.0e-10 // fixes velocity slope problem
MinimumEfficiency      = 0.4     // better value for 1d than 0.2

[sskory]

Brian writes:

does this happen on systems other than Kraken (DataStar? or Abe, for example - these are totally different architectures)?

I don't have access to Abe, but on Datastar the result is even more odd. This looks like a library problem. I get this regardless of optimization level.

DCFT : 2538-2030
The transform length (ARG NO. 8) is not an allowed value. The next higher
allowed value is (210).

DCFT : 2538-2099
End of input argument error reporting. For more information, refer to
Engineering and Scientific Subroutine Library Guide and Reference (SA22-7904).

DCFT : 2538-2604
Execution terminating due to error count for error number 2099.

DCFT : 2538-2605
Message summary:      2030     - 1

DCFT : 2538-2605
Message summary:      2099     - 1
STOP 1

Does it happen for other sets of compilation options, or if you vary the parameters in the parameter file? If you reduce the optimization level of the compilers, does it still happen?

Oddly, if I have opt-debug on Kraken, it dies, but if I have -O2 or -O3 it works. I played with the input parameters in the file (Courant, grid size, Gamma), and nothing makes it work (this was using no optimization).

However, looking at the outputs by hand, nothing is happening in these 'successful' runs. The baryon density stays constant across data dumps, the velocity doesn't change. So there is no collapse. The initial data dump looks after inspecting grid::PressurelessCollapseInitializeGrid, the fields are set up the way they should be.

[sskory]

A quick test on Ranger shows the same euler.src problem at no and -O2 levels of optimization.

[sskory]

This seems to work after I fixed Grid_PPMDirectEuler.C, see changeset 1937.

[sskory]

Frick. It's not fixed.

[sskory]

I've decided that this bug comes down to the fact that there is something wrong with isolated boundary conditions. So, I'm going to put this on the back burner right now.