## Jun 25: IntelliSweep for Parameter Testing

Category: Computer Science

Posted by: Aaron

Typically when we explore the outputs of agent-based models we do a parameter sweep: change the values of the parameters according to some pre-defined scheme and take measurements of the outcome from each of those parameter sets. Typically our models are stochastic so we do multiple trials of each set of parameters. Similar approaches are used for many other modeling techniques (including deterministic ones) to measure sensitivities and generate comparative statics. This post focuses on an alternative to the standard parameter sweep that intelligently searches through the parameter space to find the regions where the outcome is most variable, i.e. the interesting parts.

The standard parameter sweep technique requires modelers to specify the values of parameters over intervals and to test each of the combinations of parameters. Typically people will choose regular intervals over their parameters' dimensions; e.g. sweep the values 1, 2, 3, 4, and 5. Those may be good flag posts if you don't have any information about the more turbulent areas of your parameter space, and starting there may be a good idea. One could also start by sampling random values within that 1-5 range, or logarithmically distributed values, or use a chaotic parameter sampling technique, or whatever...that won't matter much to what I'm saying here, but it will make a big difference depending on your model. Whatever the case, the first step of this technique is to sample a few values (at least 3) of each of your parameters via your favorite distribution.

The next step is to map the measures of the outcomes (whatever yours are) for those parameter sweeps. That outcome space will have areas of smoothness (maybe) and areas of very rough terrain (maybe). Once you see where the output is more volatile given changes in input, you'll want to sample that region with greater resolution. You might still want to sample within the smooth regions because there may be spikes within the smooth regions, but you know you can't sample everything and that region is expected to give less returns on your simulation investment. To find the smooth and rough and moderate areas of your output space you need a measure of output volatility to guide your refinement decisions. There are lots of obvious options, but I want to point out the difference between an absolute measure and a relative measure.

To use an absolute measure you would need to specify a threshold for the difference between two neighboring points on the output map. Take two sets of input parameters that have a change in just one input parameter. These inputs map to two different outputs (i.e. measures of the final configuration) and if the difference between the output measures is greater than your threshold then pick an intermediate value for the varying input parameter and run your model on the intermediate value. Which intermediate value you should choose will again refer to your personal sampling preferences, the default being the mean value of the previous inputs. And you should note that you can use the same threshold for all parameter sweeps because the threshold is on defined on the values of the outputs. The result of this approach will be an arbitrary number of samplings depending on the actual variability in outputs upon inputs so this is good if you want a thorough search of the space but don't know what is required to achieve this.

If you have a fixed amount of time to run sweeps (which translates roughly into a fixed number of trials) then using a relative measure to inform where to search parameters with greater resolution is better. Instead of resampling if a region is sufficiently potentially interesting, a relative measure resamples one region if it is more interesting than other regions. Here's how it works. You first specify a number of samples (n > 3) to take for a parameter, and then you sample at least three points according to your favorite scheme. To determine which region to sample next, you compare the values of your output measure at your output points (to start that's those three points), determine the differences between all adjacent points, and then resample within the region with the greatest output measure difference. Using this technique you may not be sampling all the potentially interesting regions (as you would with an absolute measure), but you are collecting data on the best regions you can given your time constraints.

Note that for both absolute and relative approaches the sampling values generated for parameter

An IntelliSweep can easily be implemented within your model if you already have your model written in java, python, C++, Mathematica, etc. It could also be created as a separate program (a script) that controls your drone runs; and this would just be an adjustment to some existing techniques. In Netlogo you could build this in with considerable effort, but it's not part of the Behavior Space tool built in to sweep parameters. I'm

[Note: This technique is under consideration for inclusion in the BehaviorSearch NetLogo addon project. Go to behaviorsearch.org to let them know you'd like to use it. And while you're there learn about the nice searching capability already built into that tool.]

The standard parameter sweep technique requires modelers to specify the values of parameters over intervals and to test each of the combinations of parameters. Typically people will choose regular intervals over their parameters' dimensions; e.g. sweep the values 1, 2, 3, 4, and 5. Those may be good flag posts if you don't have any information about the more turbulent areas of your parameter space, and starting there may be a good idea. One could also start by sampling random values within that 1-5 range, or logarithmically distributed values, or use a chaotic parameter sampling technique, or whatever...that won't matter much to what I'm saying here, but it will make a big difference depending on your model. Whatever the case, the first step of this technique is to sample a few values (at least 3) of each of your parameters via your favorite distribution.

The next step is to map the measures of the outcomes (whatever yours are) for those parameter sweeps. That outcome space will have areas of smoothness (maybe) and areas of very rough terrain (maybe). Once you see where the output is more volatile given changes in input, you'll want to sample that region with greater resolution. You might still want to sample within the smooth regions because there may be spikes within the smooth regions, but you know you can't sample everything and that region is expected to give less returns on your simulation investment. To find the smooth and rough and moderate areas of your output space you need a measure of output volatility to guide your refinement decisions. There are lots of obvious options, but I want to point out the difference between an absolute measure and a relative measure.

**Absolute Measure**To use an absolute measure you would need to specify a threshold for the difference between two neighboring points on the output map. Take two sets of input parameters that have a change in just one input parameter. These inputs map to two different outputs (i.e. measures of the final configuration) and if the difference between the output measures is greater than your threshold then pick an intermediate value for the varying input parameter and run your model on the intermediate value. Which intermediate value you should choose will again refer to your personal sampling preferences, the default being the mean value of the previous inputs. And you should note that you can use the same threshold for all parameter sweeps because the threshold is on defined on the values of the outputs. The result of this approach will be an arbitrary number of samplings depending on the actual variability in outputs upon inputs so this is good if you want a thorough search of the space but don't know what is required to achieve this.

**Relative Measures**If you have a fixed amount of time to run sweeps (which translates roughly into a fixed number of trials) then using a relative measure to inform where to search parameters with greater resolution is better. Instead of resampling if a region is sufficiently potentially interesting, a relative measure resamples one region if it is more interesting than other regions. Here's how it works. You first specify a number of samples (n > 3) to take for a parameter, and then you sample at least three points according to your favorite scheme. To determine which region to sample next, you compare the values of your output measure at your output points (to start that's those three points), determine the differences between all adjacent points, and then resample within the region with the greatest output measure difference. Using this technique you may not be sampling all the potentially interesting regions (as you would with an absolute measure), but you are collecting data on the best regions you can given your time constraints.

Note that for both absolute and relative approaches the sampling values generated for parameter

*x*with parameter*y*=*a*will likely be different than the sampling points for*x*with parameter*y*=*b*because the more volatile regions of the output space along one input dimension vary differently in different contexts of the other parameters This would all be done automatically by the sampling algorithm so it's not more work, but it will complicate how one reports parameter sweep results because it doesn't produce a graph paper-like distribution of points.An IntelliSweep can easily be implemented within your model if you already have your model written in java, python, C++, Mathematica, etc. It could also be created as a separate program (a script) that controls your drone runs; and this would just be an adjustment to some existing techniques. In Netlogo you could build this in with considerable effort, but it's not part of the Behavior Space tool built in to sweep parameters. I'm

**sure**these sorts of ideas are already implemented in numerical studies of dynamical systems and even by other agent-based modelers and genetic programmers. I haven't seen anything like this outlined before, and so maybe you haven't either and reading this was worthwhile. If you know of a technique like this being utilized somewhere already, I'd greatly appreciate it if you sent me a link or reference or whatever. And if you'd like to use this technique and would like some help in setting it up then feel free to contact me.[Note: This technique is under consideration for inclusion in the BehaviorSearch NetLogo addon project. Go to behaviorsearch.org to let them know you'd like to use it. And while you're there learn about the nice searching capability already built into that tool.]