NetLogo 6.2.2 User Manual: BehaviorSpace Guide

# BehaviorSpace Guide

NetLogo 6.2.2 User Manual

This guide has three parts:

- What is BehaviorSpace?: A general description of the tool, including the ideas and principles behind it.
- How It Works: Walks you through how to use the tool and highlights its most commonly used features.
- Advanced Usage: How to use BehaviorSpace from the command line, or from your own Java code.

## What is BehaviorSpace?

BehaviorSpace is a software tool integrated with NetLogo that allows you to perform experiments with models.

BehaviorSpace runs a model many times, systematically varying the model’s settings and recording the results of each model run. This process is sometimes called “parameter sweeping”. It lets you explore the model’s “space” of possible behaviors and determine which combinations of settings cause the behaviors of interest.

If your computer has multiple processor cores, then by default, model runs will happen in parallel, one per core.

### Why BehaviorSpace?

The need for this type of experiment is revealed by the following observations. Models often have many settings, each of which can take a range of values. Together they form what in mathematics is called a parameter space for the model, whose dimensions are the number of settings, and in which every point is a particular combination of values. Running a model with different settings (and sometimes even the same ones) can lead to drastically different behavior in the system being modeled. So, how are you to know which particular configuration of values, or types of configurations, will yield the kind of behavior you are interested in? This amounts to the question of where in its huge, multi-dimension parameter space does your model perform best?

For example, suppose you want speedy synchronization from the agents in the Fireflies model. The model has four sliders – number, cycle-length, flash-length and flashes-to-reset – that have approximately 2000, 100, 10 and 3 possible values, respectively. That means there are 2000 * 100 * 10 * 3 = 6,000,000 possible combinations of slider values! Trying combinations one at a time is hardly an efficient way to learn which one will evoke the speediest synchronization.

BehaviorSpace offers you a much better way to solve this problem. If you specify a subset of values from the ranges of each slider, it will run the model with each possible combination of those values and, during each model run, record the results. In doing so, it samples the model’s parameter space – not exhaustively, but enough so that you will be able to see relationships form between different sliders and the behavior of the system. After all the runs are over, a dataset is generated which you can open in a different tool, such as a spreadsheet, database, or scientific visualization application, and explore.

By enabling you to explore the entire “space” of behaviors a model can exhibit, BehaviorSpace can be a powerful assistant to the modeler.

## How It Works

To begin using BehaviorSpace, open your model, then choose the BehaviorSpace item on NetLogo’s Tools menu.

### Managing experiment setups

The dialog that opens lets you create, edit, duplicate, delete, and run experiment setups. Experiments are listed by name and how by model runs the experiment will consist of.

Experiment setups are considered part of a NetLogo model and are saved as part of the model.

To create a new experiment setup, press the “New” button.

### Creating an experiment setup

In the new dialog that appears, you can specify the following information. Note that you don’t always need to specify everything; some parts can be left blank, or left with their default values, depending on your needs.

Experiment name: If you have multiple experiments, giving them different names will help you keep them straight.

Vary variables as follows: This is where you specify which settings you want varied, and what values you want them to take. Variables can include sliders, switches, choosers, and any global variables in your model.

Variables can also include `max-pxcor`, `min-pxcor`, `max-pycor` and `min-pycor`, `world-width`, `world-height` and `random-seed`. These are not, strictly speaking, variables, but BehaviorSpace lets you vary them as if they were. Varying the world dimensions lets you explore the effect of world size upon your model. Since setting `world-width` and `world-height` does not necessarily define the bounds of the world how they are varied depends on the location of the origin. If the origin is centered, BehaviorSpace will keep it centered so the values `world-width` or `world-height` must be odd. If one of the bounds is at zero that bound will be kept at zero and the other bound will move, for example if you start with a world with `min-pxcor = 0` `max-pxcor = 10` and you vary `world-width` like this:

> `["world-width" [11 1 14]]`

`min-pxcor` will stay at zero and `max-pxcor` will set to 11, 12, and 13 for each of the runs. If neither of these conditions are true, the origin is not centered, nor at the edge of the world you cannot vary `world-height` or `world-width` directly but you should vary `max-pxcor`, `max-pycor`, `min-pxcor` and `min-pycor` instead.

Varying `random-seed` lets you repeat runs by using a known seed for the NetLogo random number generator. Note that you’re also free to use the `random-seed` command in your experiment’s setup commands. For more information on random seeds, see the Random Numbers section of the Programming Guide.

You may specify values either by listing the values you want used, or by specifying that you want to try every value within a given range. For example, to give a slider named `number` every value from 100 to 1000 in increments of 50, you would enter:

> `["number" [100 50 1000]]`

Or, to give it only the values of 100, 200, 400, and 800, you would enter:

> `["number" 100 200 400 800]`

Be careful with the brackets here. Note that there are fewer square brackets in the second example. Including or not including this extra set of brackets is how you tell BehaviorSpace whether you are listing individual values, or specifying a range.

Also note that the double quotes around the variable names are required.

You can vary as many settings as you want, including just one, or none at all. Any settings that you do not vary will retain their current values. Not varying any settings is useful if you just want to do many runs with the current settings.

What order you list the variables in determines what order the runs will be done in. All values for a later variable will be tried before moving to the next value for an earlier variable. So for example if you vary both x and y from 1 to 3, and x is listed first, then the order of model runs will be: x=1 y=1, x=1 y=2, x=1 y=3, x=2 y=1, and so on.

Repetitions: Sometimes the behavior of a model can vary a lot from run to run even if the settings don’t change, if the model uses random numbers. If you want to run the model more than once at each combination of settings, enter a higher number.

Measure runs using these reporters: This is where you specify what data you want to collect from each run. For example, if you wanted to record how the population of turtles rose and fell during each run, you would enter:

> `count turtles`

You can enter one reporter, or several, or none at all. If you enter several, each reporter must be on a line by itself, for example:

> `count frogs` `count mice` `count birds`

If you don’t enter any reporters, the runs will still take place. This is useful if you want to record the results yourself your own way, such as with the `export-world` command.

Measure runs at every step: Normally NetLogo will measure model runs at every step, using the reporters you entered in the previous box. If you’re doing very long model runs, you might not want all that data. Uncheck this box if you only want to measure each run after it ends.

Setup commands: These commands will be used to begin each model run. Typically, you will enter the name of a procedure that sets up the model, typically `setup`. But it is also possible to include other commands as well.

Go commands: These commands will be run over and over again to advance to the model to the next “step”. Typically, this will be the name of a procedure, such as `go`, but you may include any commands you like.

Stop condition: This lets you do model runs of varying length, ending each run when a certain condition becomes true. For example, suppose you wanted each run to last until there were no more turtles. Then you would enter:

> `not any? turtles`

If you want the length of runs to all be of a fixed length, just leave this blank.

The run may also stop because the go commands use the `stop` command, in the same way that `stop` can be used to stop a forever button. The `stop` command may be used directly in the go commands, or in a procedure called directly by the go commands. (The intent is that the same `go` procedure should work both in a button and in a BehaviorSpace experiment.) Note that the step in which `stop` is used is considered to have been aborted, so no results will be recorded for that step. Therefore, the stopping test should be at the beginning of the go commands or procedure, not at the end.

Final commands: These are any extra commands that you want run once, when the run ends. Usually this is left blank, but you might use it to call the `export-world` command or record the results of the run in some other way.

Time limit: This lets you set a fixed maximum length for each run. If you don’t want to set any maximum, but want the length of the runs to be controlled by the stop condition instead, enter 0.

### Special primitives for BehaviorSpace experiments

Currently there are only two, `behaviorspace-run-number` and `behaviorspace-experiment-name`. The run number reported by the former primitive matches the run number used in the results files generated by BehaviorSpace. The experiment name reported by the latter matches the name with which the experiment was set up.

### Running an experiment

When you’re done setting up your experiment, press the “OK” button, followed by the “Run” button. A dialog titled “Run options” will appear.

#### Run options: formats

The run options dialog lets you select the formats you would like the data from your experiment saved in. Data is collected for each run or step, according to the setting of Measure runs at every step option. In either case, the initial state of the system is recorded, after the setup commands run but before the go commands run for the first time.

Both Table output and Spreadsheet output will include a header section that has rows for 1) the NetLogo app version used, 2) the name of the NetLogo model file used, 3) the name of the BehaviorSpace experiment used, 4) the date and time at the start, and 5) the dimensions of the world used at the start.

The Table output format lists each measurement step from each run in its own row, with each metric in a separate column. The measurement rows will appear in the order they happen in real time. With the parallel runs option the measurements may appear in a mixed order as multiple runs can happen simultaneously. To help identify which run a row belongs to, there is a column titled `[run number]` along with a column titled `[step]` that indicates the measurement number for that run. Each row also includes the values used for the variables during initial setup of the run; these values are the same for each of that run’s measurement rows. Table output data is written to the file as each run completes.

The Table output format is good to use when you want to further process the data using another tool, such as importing into a database, a statistics package, or a spreadsheet application for analysis.

In the below Table output sample image the header section is in red, the run number and step columns are in blue, the initial values of the variables for the run are in green, and the measurement metric data is in purple.

The Spreadsheet output format lists each metric for each run in a separate column, with each row corresponding to a measurement step that applies to all runs. If one run finishes before another due to a stop condition, then its measurement steps after that point will be blank. At the top of the file there is a `[run number]` row that will have the run number repeated for each metric at the top of the file. After that are rows for the initial values given to each variable for the run. The spreadsheet output also calculates the min, mean, max, and final values for each metric and lists those after the initial values before the individual run measurements. Then it lists the actual number of steps a run went through in a `[steps]` row.

The Spreadsheet output data makes it easier to quickly compare runs against each other, as they will be aligned vertically when imported into a spreadsheet application. It can also make generating comparative graphs of results with initial variable changes across runs easy for the same reason.

It is important to note that Spreadsheet output data is not written to the results file until the experiment finishes. Since the data is stored in memory until the experiment is done, very large experiments could run out of memory. You should disable Spreadsheet output unless you really want it. Also, if anything interrupts the experiment no spreadsheet results will be written. Possible sources of interruptions would be runtime errors in the model, running out of memory, system crashes, or power outages. For long experiments you may want to also enable Table output format as a precaution so that if something happens and you get no Spreadsheet output you’ll at least get partial data output.

In the below Spreadsheet output sample image the header section is in red, the run number row is in blue, the initial values of the variables for the run are in green, the extra calculated metrics are in orange, and the measurement metric data is in purple.

After selecting your output formats, BehaviorSpace will prompt you for the name of a file to save the results to. The default name ends in “.csv”. You can change it to any name you want, but don’t leave off the “.csv” part; that indicates the file is a Comma Separated Values (CSV) file. This is a plain-text data format that is readable by any text editor as well as by most popular spreadsheet and database programs.

#### Run options: parallel runs

The run options dialog also lets you select whether you want multiple model runs to happen in parallel, and if so, how many are allowed to be simultaneously active. This number will default to the number of processor cores in your computer.

There are a few cautions associated with parallel runs.

First, if multiple runs are active, only one of them will be in the “foreground” and cause the view and plots to update. The other runs will happen invisibly in the background.

Second, invisible background runs can’t use primitives that only work in the GUI. For example, a background run can’t make a movie.

Third, since parallel runs progress independently of each other, table format output may contain interleaved, out-of-order results. When you analyze your table data, you may wish to sort it by run number first. (Spreadsheet format output is not affected by this issue, since it is not written until the experiment completes or is aborted.)

Fourth, using all available processor cores may make your computer slow to use for other tasks while the experiment is running.

Fifth, doing runs in parallel will multiply the experiment’s memory requirements accordingly. You may need to increase NetLogo’s memory ceiling (see this FAQ entry).

Sixth, each parallel run will get its own world for the model to run in. This world is not cleared automatically by BehaviorSpace if a parallel run gets re-used for another repitition, which happens quite frequently. This means, for example, if you do `ask patches [ set pcolor red ]` in one run and do not use `clear-all` or `clear-patches` in the setup commands of the next run, then the patches will all still be red. In general using `clear-all` before each run would be a best practice, but there are times when you might not want to, such as loading data from a file that doesn’t change run-to-run. Just be careful with whatever data is not cleared out.

Seventh, there is a very, very small chance that at startup multiple parallel runs could wind up with the same random number generator state if they startup at the eact same moment in time. This means the runs would produce identical output for all random operations and likely the same results. This would have a chance to happen when running on very fast processors and with lots of parallel runs at once. If you need to make sure this doesn’t impact your results, you can add `random-seed new-seed` to your setup commands to re-generate a new unique random seed for each run. In fact, storing the `new-seed` as a global variable so you can output it with the rest of your results would let you re-run a run later on by manually using that value to set the `random-seed`.

#### Observing runs

After you complete the run options dialog, another dialog will appear, titled “Running Experiment”. In this dialog, you’ll see a progress report of how many runs have been completed so far and how much time has passed. If you entered any reporters for measuring the runs, and if you left the “Measure runs at every step” box checked, then you’ll see a plot of how they vary over the course of each run.

You can also watch the runs in the main NetLogo window. (If the “Running Experiment” dialog is in the way, just move it to a different place on the screen.) The view and plots will update as the model runs. If you don’t need to see them update, then use the checkboxes in the “Running Experiment” dialog to turn the updating off. This will make the experiment go faster.

If you want to stop your experiment before it’s finished, press the “Abort” button. Any results generated so far will still be saved.

When all the runs have finished, the experiment is complete.

## Advanced Usage

### Running from the command line

It is possible to run BehaviorSpace experiments “headless”, that is, from the command line, without any graphical user interface (GUI). This is useful for automating runs on a single machine or a cluster of machines.

No Java programming is required. Experiment setups can be created in the GUI and then run later from the command line, or, if you prefer, you can create or edit experiment setups directly using XML.

#### How to use it

Run NetLogo using the included command line script. This is found in the root directory of your NetLogo installation and is named `netlogo-headless.sh` on Mac and Linux and `netlogo-headless.bat` on Windows. The netlogo-headless script supports the following arguments:

- `--model <path>`: pathname of model to open (required)
- `--setup-file <path>`: read experiment setups from this file instead of the model file
- `--experiment <name>`: name of experiment to run
- `--table <path>`: pathname to send table output to (or `-` for standard output)
- `--spreadsheet <path>`: pathname to send table output to (or `-` for standard output)
- `--threads <number>`: use this many threads to do model runs in parallel, or 1 to disable parallel runs. defaults to one thread per processor.
- `--min-pxcor <number>`: override world size setting in model file
- `--max-pxcor <number>`: override world size setting in model file
- `--min-pycor <number>`: override world size setting in model file
- `--max-pycor <number>`: override world size setting in model file

`--model` is required. If you don’t specify `--experiment`, you must specify `--setup-file`. By default no results are generated, so you’ll usually want to specify either `--table` or `--spreadsheet`, or both. If you specify any of the world dimensions, you must specify all four.

Note: The remainder of this guide uses `netlogo-headless.sh` to refer to the NetLogo Headless launch script. If you are using Windows, please substitute `netlogo-headless.bat` for `netlogo-headless.sh` in each example.

#### Examples

It is easiest if you create your experiment setup ahead of time in the GUI, so it is saved as part of the model. To run an experiment setup saved in a model, here is an example command line, run from the NetLogo 6.2.2 installation folder so the paths to the `netlogo-headless.sh` script and `Wolf Sheep Simple 5.nlogo` model are correct.

    # on macOS use "models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" instead
    ./netlogo-headless.sh \
      --model "app/models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" \
      --experiment "Wolf Sheep Simple model analysis" \
      --table -

For this to work, Java (version 1.8 or later) must be available. You can make Java available to headless in either of two ways

1. Set the `JAVA_HOME` environment variable to the path to the Java installation. This is the directory of the Java installation which contains a `bin` directory.
2. Add the directory containing the Java executable to the `PATH` environment variable

If `JAVA_HOME` is defined, netlogo-headless will run NetLogo using the Java that it points to, ignoring the version of Java available on the path.

After the named experiment has run, the results are sent to standard output in table format, as CSV. `-` is how you specify standard output instead of output to a file.

When running netlogo headless, it forces the system property `java.awt.headless` to be true. This tells Java to run in headless mode, allowing NetLogo to run on machines when a graphical display is not available.

The required `--model` argument is used to specify the model file you want to open.

The `--experiment` argument is used to specify the name of the experiment you want to run. (At the time you create an experiment setup in the GUI, you assign it a name.)

Here’s another example that shows some additional, optional arguments:

    # on macOS use "models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" instead
    ./netlogo-headless.sh \
      --model "app/models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" \
      --experiment "Wolf Sheep Simple model analysis" \
      --max-pxcor 5 \
      --min-pxcor -5 \
      --max-pycor 5 \
      --min-pycor -5

Note the use of the optional `--max-pxcor`, `--max-pycor`, etc. arguments to specify a different world size than that saved in the model. (It’s also possible for the experiment setup to specify values for the world dimensions; if they are specified by the experiment setup, then there is no need to specify them on the command line.)

Since neither `--table` nor `--spreadsheet` is specified, no results will be generated. This is useful if the experiment setup generates all the output you need by some other means, such as exporting world files or writing to a text file.

Yet another example:

    # on macOS use "models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" instead
    ./netlogo-headless.sh \
      --model "app/models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" \
      --experiment "Wolf Sheep Simple model analysis" \
      --table wsp5-table-output.csv \
      --spreadsheet wsp5-spreadsheet-output.csv

The optional `--table <filename>` argument specifies that output should be generated in a table format and written to the given file as CSV data. If `-` is specified as the filename, than the output is sent to the standard system output stream. Table data is written as it is generated, with each complete run.

The optional `--spreadsheet <filename>` argument specified that spreadsheet output should be generated and written to the given file as CSV data. If `-` is specified as the filename, than the output is sent to the standard system output stream. Spreadsheet data is not written out until all runs in the experiment are finished.

Note that it is legal to specify both `--table` and `--spreadsheet`, and if you do, both kinds of output file will be generated.

Here is an example that shows how to run an experiment setup which is stored in a separate XML file, instead of in the model file (see below for more information on the XML file format). This assumes you’ve created a `my-wsp-setups.xml` file with a `My WSP Exploration" experiment and placed it in your home directory.

    # on macOS use "models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" instead
    ./netlogo-headless.sh \
      --model "app/models/IABM Textbook/chapter 4/Wolf Sheep Simple 5.nlogo" \
      --setup-file ~/my-wsp-setups.xml \
      --experiment "My WSP Exploration"

If the XML file contains more than one experiment setup, it is necessary to use the `--experiment` argument to specify the name of the setup to use.

In order to run a NetLogo 3D experiment, run headless with the `--3D` argument, for example:

    # on macOS use "models/3D/Sample Models/GasLab/GasLab Free Gas 3D.nlogo3d" instead
    ./netlogo-headless.sh \
      --3D \
      --model "app/models/3D/Sample Models/GasLab/GasLab Free Gas 3D.nlogo3d" \
      --experiment "100 runs" \
      --table -

Note that you should supply a 3D model and there are also 3D arguments `--max-pzcor <number>` and `--min-pzcor <number>`.

The next section has information on how to create standalone experiment setup files using XML.

### Setting up experiments in XML

We don’t yet have detailed documentation on authoring experiment setups in XML, but if you already have some familiarity with XML, then the following pointers may be enough to get you started.

The structure of BehaviorSpace experiment setups in XML is determined by a Document Type Definition (DTD) file. The DTD is stored in NetLogo.jar, as `system/behaviorspace.dtd`. (JAR files are also zip files, so you can extract the DTD from the JAR using Java’s “jar” utility or with any program that understands zip format.)

The easiest way to learn what setups look like in XML, though, is to author a few of them in BehaviorSpace’s GUI, save the model, and then examine the resulting .nlogo file in a text editor. The experiment setups are stored towards the end of the .nlogo file, in a section that begins and ends with a `experiments` tag. Example:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE experiments SYSTEM "behaviorspace.dtd">
    <experiments>
      <experiment name="My WSP Exploration" repetitions="5" runMetricsEveryStep="false">
        <setup>setup</setup>
        <go>go</go>
        <timeLimit steps="2000"/>
        <metric>count wolves</metric>
        <metric>count sheep</metric>
        <metric>sum [grass-amount] of patches</metric>
        <enumeratedValueSet variable="energy-gain-from-grass">
          <value value="2"/>
        </enumeratedValueSet>
        <steppedValueSet variable="number-of-wolves" first="5" step="1" last="15"/>
        <enumeratedValueSet variable="movement-cost">
          <value value="0.5"/>
        </enumeratedValueSet>
        <enumeratedValueSet variable="energy-gain-from-sheep">
          <value value="5"/>
        </enumeratedValueSet>
        <enumeratedValueSet variable="number-of-sheep">
          <value value="500"/>
        </enumeratedValueSet>
        <enumeratedValueSet variable="grass-regrowth-rate">
          <value value="0.3"/>
        </enumeratedValueSet>
      </experiment>
    </experiments>

In this example, only one experiment setup is given, but you can put as many as you want between the beginning and ending `experiments` tags.

Between looking at the DTD, and looking at examples you create in the GUI, it will hopefully be apparent how to use the tags to specify different kind of experiments. The DTD specifies which tags are required and which are optional, which may be repeated and which may not, and so forth.

When XML for experiment setups is included in a model file, it does not begin with any XML headers, because not the whole file is XML, only part of it. If you keep experiment setups in their own file, separate from the model file, then the extension on the file should be .xml not .nlogo, and you’ll need to begin the file with proper XML headers, as follows:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE experiments SYSTEM "behaviorspace.dtd">

The second line must be included exactly as shown. In the first line, you may specify a different encoding than `UTF-8`, such as `ISO-8859-1`.

### Adjusting JVM Parameters

Opening the NetLogo Headless launcher script will show the options used to launch java when running NetLogo Headless. You can adjust various JVM parameters in this script. You may also pass in Java properties starting with `-D` to the launcher.

Note the use of `-Xmx` to specify a maximum heap size of one gigabyte. If you don’t specify a maximum heap size, you will get your VM’s default size, which may be unusably small. (One gigabyte is an arbitrary size which should be more than large enough for most models; you can specify a different limit if you want.)

Note the use of `-Dfile.encoding=UTF-8`. This forces all file I/O to use UTF-8 encoding. Doing so ensures that NetLogo can load all models consistently, and that file-* primitives work consistently on all platforms, including models containing Unicode characters.

### Controlling API

If BehaviorSpace is not sufficient for your needs, a possible alternative is to use our Controlling API, which lets you write Java code that controls NetLogo. The API lets you run BehaviorSpace experiments from Java code, or, you can write custom code that controls NetLogo more directly to do BehaviorSpace-like things. See the Controlling section of the User Manual for further details on both possibilities.