Changes between Version 2 and Version 3 of Tutorials/RunCosmologySimulation
- Timestamp:
- 08/26/08 12:44:23 (5 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Tutorials/RunCosmologySimulation
v2 v3 1 1 = How to run a cosmology simulation = 2 2 3 In order to run an example cosmology simulation you will need to recompile the code with MPI ON. Refer to the section above on compiling Enzo to see how to do that. After recompiling with MPI on, you should create a directory to run the simulation in. This is because Enzo cosmology simulations create quite a few output files, so it's best to store them in their own directory. For the purposes of this example I'm going to assume that you have created a directory called EnzoTestSim in your home directory. You should then download these two files: Example_Cosmology_Sim.inits and Example_Cosmology_Sim.param which are the initial conditions parameter file and the simulation parameter file, respectively. Put these two files in that directory, along with the enzo binary, the ring binary, and the initial conditions generator binary (the binaries are named enzo, ring and inits, respectively). 4 Creating initial conditions 3 In order to run a cosmology simulation, you'll need to [wiki:BuildingEnzo build enzo.exe, inits.exe and ring.exe]. inits creates the initial conditions for your simulation, and ring splits up the root grid which is necessary if you're using parallel IO. Once you have built the three executables, put them in a common directory where you will run your test simulation. You will also save the inits and param files (shown and discussed below) in this directory. 5 4 6 The first step in preparing the simulation is to create the initial conditions. The file Example_Cosmology_Sim.inits is a text file which contains a list of parameter file names with their associated values. These values tell the initial conditions generator useful information like the simulation box size, the cosmological parameters and the size of the root grid. The code then takes that information and creates a set of initial conditions. inits is run by typing this command: 5 == Creating initial conditions == 7 6 8 user02:~/EnzoTestSim150% ./inits -d Example_Cosmology_Sim.inits 7 The first step in preparing the simulation is to create the initial conditions. The file inits uses is a text file which contains a list of parameter file names with their associated values. These values tell the initial conditions generator useful information like the simulation box size, the cosmological parameters and the size of the root grid. The code then takes that information and creates a set of initial conditions. Here is an example inits file: 8 9 10 {{{ 11 # 12 # Generates initial grid and particle fields for a multi-grid 13 # CDM simulation (top grid) 14 # 15 # Cosmology Parameters 16 # 17 CosmologyOmegaBaryonNow = 0.044 18 CosmologyOmegaMatterNow = 0.27 19 CosmologyOmegaLambdaNow = 0.73 20 CosmologyComovingBoxSize = 10.0 // in Mpc/h 21 CosmologyHubbleConstantNow = 0.71 // in units of 100 km/s/Mpc 22 CosmologyInitialRedshift = 60 23 # 24 # Power spectrum Parameters 25 # 26 27 PowerSpectrumType = 11 28 PowerSpectrumSigma8 = 0.9 29 PowerSpectrumPrimordialIndex = 1.0 30 PowerSpectrumRandomSeed = -584783758 31 # 32 # Grid info 33 # 34 Rank = 3 35 GridDims = 32 32 32 36 InitializeGrids = 1 37 GridRefinement = 1 38 # 39 # Particle info 40 # 41 ParticleDims = 32 32 32 42 InitializeParticles = 1 43 ParticleRefinement = 1 44 # 45 # Overall field parameters 46 # 47 # 48 # Names 49 # 50 ParticlePositionName = ParticlePositions 51 ParticleVelocityName = ParticleVelocities 52 GridDensityName = GridDensity 53 GridVelocityName = GridVelocities 54 }}} 55 56 inits is run by typing this command: 57 58 {{{% ./inits.exe -d Example_Cosmology_Sim.inits}}} 9 59 10 60 inits will produce some output to the screen to tell you what it is doing, and will write five files: GridDensity, GridVelocities, ParticlePositions, ParticleVelocities and PowerSpectrum.out. The first four files contain information on initial conditions for the baryon and dark matter componenets of the simulation, and are HDF 5 files. The last file is an ascii file which contains information on the power spectrum used to generate the initial conditions. 11 Parallel IO - the ring tool12 61 13 This simulation is quite small. The root grid is only 32 cells on a side and we allow a maximum of three levels of mesh refinement. Still, we will use the ring tool, since it is important for larger simulations of sizes typically used for doing science.62 It is also possible to run cosmology simulations using initial nested subgrids. 14 63 15 The ring tool is part of the Enzo parallel IO (input-output) scheme. Examine the last section of the parameter file for this example simulation and you will see: 64 == Parallel IO - the ring tool == 16 65 66 This simulation is quite small. The root grid is only 32 cells on a side and we allow a maximum of three levels of mesh refinement. Still, we will use the ring tool, since it is important for larger simulations of sizes typically used for doing science. Additionally, if you wish to run with 64 or more processors, you are required to use [wiki:HowDoesParallelRootGridIOwork parallel root grid IO], so it is beneficial to learn. 67 68 The ring tool is part of the Enzo parallel IO (input-output) scheme. Examine the last section of the parameter file (see below) for this example simulation and you will see: 69 70 {{{ 17 71 # 18 72 # IO parameters … … 20 74 ParallelRootGridIO = 1 21 75 ParallelParticleIO = 1 76 }}} 22 77 23 78 These two parameters turn on parallel IO for both grids and particles. In a serial IO simulation where multiple processors are being used, the master processor reads in all of the grid and particle initial condition information and parcels out portions of the data to the other processors. Similarly, all simulation output goes through the master processor as well. This is fine for relatively small simulations using only a few processors, but slows down the code considerably when a huge simulation is being run on hundreds of processors. Turning on the parallel IO options allows each processor to perform its own IO, which greatly decreases the amount of time the code spends performing IO. … … 25 80 The process for parallelizing grid and particle information is quite different. Since we know exactly where every grid cell in a structured Eulerian grid is in space, and these cells are stored in a regular and predictable order in the initial conditions files, turning on ParallelRootGridIO simply tells each processor to figure out which portions of the arrays in the GridDensity and GridVelocities belong to it, and then read in only that part of the file. The particle files (ParticlePositions and ParticleVelocities) store the particle information in no particular order, so in order to efficiently parallelize the particle IO the ring tool is used. ring is run on the same number of processors as the simulation that you intend to run, and can be used right before the simulation itself is run. In ring, each processor reads in an equal fraction of the particle position and velocity information into a list, flags the particles that belong in its simulation spatial domain, and then passes its portion of the total list on to another processor. After each portion of the list has made its way to every processor, each processor then collects all of the particle and velocity information that belongs to it and writes them out into files called PPos.nnnn and PVel.nnnn, where nnnn is the processor number. Turning on the ParallelParticleIO flag in the Enzo parameter file instructs Enzo to look for these files. 26 81 27 For the purpose of this example, I'm going to run ring and Enzo on 4 processors, but as an interactive job on Titan. One enters interactive parallel mode by typing: 82 There are lots more details on [wiki:HowDoesParallelRootGridIOwork this page]. 28 83 29 user02:~/EnzoTestSim151% qsub -I -V -l walltime=00:30:00,nodes=2:ppn=2:prod 84 For the purpose of this example, I'm going to run ring and Enzo on 4 processors. The number of processors used in an MPI job is set differently on each machine, so you'll have to figure out how that works for you. To start an interactive run, it might look something like this: 30 85 31 This tells the machine that I want two compute nodes with two processors on each node (for a total of 4 processors) for a half hour of wall clock time. See this web page for more information on running interactive jobs on Titan. You may have to wait a bit until nodes become available, and then you will probably be left in your home directory, so you will have to change back to the EnzoTestSim subdirectory. You then run ring on the particle files by typing: 86 {{{% qsub -I -V -l walltime=00:30:00,size=4}}} 32 87 33 user02:~/EnzoTestSim154% vmirun ./ring ParticlePositions ParticleVelocities 88 This tells the machine that I want four processors total for a half hour of wall clock time. You may have to wait a bit until nodes become available, and then you will probably start out back in your home directory. You then run ring on the particle files by typing something like this: 34 89 35 This will then produce some output to your screen, and will generate 8 files: PPos.0000 through PPos.0003 and PVel.0000 through PVel.0003. Note that if you are using a different machine or platform, you may use something other than vmirun for MPI-parallel applications. Consult your system administrator or system documentation for more information. 90 {{{% aprun -n 4 ./ring.exe pv ParticlePositions ParticleVelocities}}} 91 92 This will then produce some output to your screen, and will generate 8 files: PPos.0000 through PPos.0003 and PVel.0000 through PVel.0003. 36 93 37 94 Congratulations, you're now ready to run your cosmology simulation! 38 Running an Enzo cosmology simulation39 95 40 After all of this preparation, running the simulation itself should be straightforward. You start enzo by typing: 96 == Running an Enzo cosmology simulation == 41 97 42 user02:~/EnzoTestSim155% vmirun ./enzo -d Example_Cosmology_Sim.param > output.log 98 After all of this preparation, running the simulation itself should be straightforward. First, you need to have an enzo parameter file. Here is an example compatible with the inits file above: 43 99 44 The simulation will now run. The -d flag ensures a great deal of output, so we redirect it into a log file called output.log for later examination. This particular simulation should take approximately ten minutes to run on 4 processors on Titan. When the simulation is done, enzo will display the message "Successful run, exiting." 100 {{{ 101 # 102 # AMR PROBLEM DEFINITION FILE: Cosmology Simulation (amr version) 103 # 104 # define problem 105 # 106 ProblemType = 30 // cosmology simulation 107 TopGridRank = 3 108 TopGridDimensions = 32 32 32 109 SelfGravity = 1 // gravity on 110 TopGridGravityBoundary = 0 // Periodic BC for gravity 111 LeftFaceBoundaryCondition = 3 3 3 // same for fluid 112 RightFaceBoundaryCondition = 3 3 3 113 # 114 # problem parameters 115 # 116 CosmologySimulationOmegaBaryonNow = 0.044 117 CosmologySimulationOmegaCDMNow = 0.226 118 CosmologyOmegaMatterNow = 0.27 119 CosmologyOmegaLambdaNow = 0.73 120 CosmologySimulationDensityName = GridDensity 121 CosmologySimulationVelocity1Name = GridVelocities 122 CosmologySimulationVelocity2Name = GridVelocities 123 CosmologySimulationVelocity3Name = GridVelocities 124 CosmologySimulationParticlePositionName = ParticlePositions 125 CosmologySimulationParticleVelocityName = ParticleVelocities 126 CosmologySimulationNumberOfInitialGrids = 1 127 # 128 # define cosmology parameters 129 # 130 ComovingCoordinates = 1 // Expansion ON 131 CosmologyHubbleConstantNow = 0.71 // in km/s/Mpc 132 CosmologyComovingBoxSize = 10.0 // in Mpc/h 133 CosmologyMaxExpansionRate = 0.015 // maximum allowed delta(a)/a 134 CosmologyInitialRedshift = 60.0 // 135 CosmologyFinalRedshift = 3.0 // 136 GravitationalConstant = 1 // this must be true for cosmology 137 # 138 # set I/O and stop/start parameters 139 # 140 CosmologyOutputRedshift[0] = 25.0 141 CosmologyOutputRedshift[1] = 10.0 142 CosmologyOutputRedshift[2] = 5.0 143 CosmologyOutputRedshift[3] = 3.0 144 # 145 # set hydro parameters 146 # 147 Gamma = 1.6667 148 PPMDiffusionParameter = 0 // diffusion off 149 DualEnergyFormalism = 1 // use total & internal energy 150 InterpolationMethod = 1 // SecondOrderA 151 CourantSafetyNumber = 0.5 152 ParticleCourantSafetyNumber = 0.8 153 FluxCorrection = 1 154 ConservativeInterpolation = 0 155 HydroMethod = 0 156 # 157 # set cooling parameters 158 # 159 RadiativeCooling = 0 160 MultiSpecies = 0 161 RadiationFieldType = 0 162 StarParticleCreation = 0 163 StarParticleFeedback = 0 164 # 165 # set grid refinement parameters 166 # 167 StaticHierarchy = 0 // AMR turned on! 168 MaximumRefinementLevel = 3 169 MaximumGravityRefinementLevel = 3 170 RefineBy = 2 171 CellFlaggingMethod = 2 4 172 MinimumEfficiency = 0.35 173 MinimumOverDensityForRefinement = 4.0 4.0 174 MinimumMassForRefinementLevelExponent = -0.1 175 MinimumEnergyRatioForRefinement = 0.4 45 176 46 Enzo is a complicated code, with a similarly complicated output format. See the Enzo User Guide page on the Enzo output format for more information on the data outputs. 177 # 178 # set some global parameters 179 # 180 GreensFunctionMaxNumber = 100 // # of greens function at any one time 181 182 183 # 184 # IO parameters 185 # 186 187 ParallelRootGridIO = 1 188 ParallelParticleIO = 1 189 }}} 190 191 Once you've saved this, you start enzo by typing: 192 193 {{{% aprun -n 4 ./enzo.exe -d Example_Cosmology_Sim.param > output.log}}} 194 195 The simulation will now run. The -d flag ensures a great deal of output, so we redirect it into a log file called output.log for later examination. This particular simulation shouldn't take too long, so you can run this in the same 30 minute interactive job you started when you ran inits. When the simulation is done, enzo will display the message "Successful run, exiting." 47 196 48 197 Congratulations! If you've made it this far, you have now successfully run a simulation using Enzo!
