Parametrized Simulations

Let us consider that our objective is to study how a given DTN performs (e.g., in terms of message delivery ratio) when three different routing protocols, say Epidemic, PRoPHET, and Spray-and-Wait (SnW), are used. In our previous tutorial, we learned how to simulate a simple scenario using the ONE simulator. For the current study, we can, for example, simply change the routing protocol in the settings file three times, and simulate the scenarios after each change. Done!

Now, let us try to extend our study. Let us consider two mobility models: Random Waypoint (RWP) and Random Walk (RW). We want to compare the delivery ratio of messages obtained for each routing policy and each mobility model used.

We can, again, take a similar approach. We can keep changing the movement model in the settings file every time. Here, we have 3 * 2 = 6 different combinations of settings:

• Epidemic, RWP
• PRoPHET, RWP
• SnW, RWP
• Epidemic, RW
• PRoPHET, RW
• SnW, RW

So, this time we need to run the simulation six times.

Suppose, we are also interested to observe the behavior when the nodes have buffer sizes of 2M, 5M, 10M, 15M, and 20M. From our above calculation it follows that this time we have 3 * 2 * 5 = 30 different scenarios. And why not vary the number of nodes itself, say from 10 to 70, in steps of 10? That would amount to 30 * 7 = 210 scenarios.

This leads to the following question — how to manage simulations when the number of scenarios keep increasing? Surely, running them manually every time becomes tedious even when the number of parameters varied is slightly increased.

The ONE simulator comes with an excellent feature — it allows automating several simulation scenarios and therefore, eliminates the need for running each scenario manually. Continuing with the above example, the relevant contents of the settings file would look like:

Scenario.name = %%Group.movementModel%%_%%Group.router%%_%%Group.bufferSize%%
Group.movementModel = [RandomWalk; RandomWaypoint;]
Group.router = [EpidemicRouter; ProphetRouter; SprayAndWaitRouter;]
Group.bufferSize = [2M; 5M; 10M; 15M; 20M;]

To simulate all these scenarios, type in the following in the terminal:

./one.sh -b 30 multiple_scenarios.txt

The above command executes the simulator in the batch mode ("-b") and the GUI is not displayed. The number 30 tells how many times the simulator should run. The final argument is the name of the settings file. Notice that, our settings file specifies exactly 30 unique scenarios. Any report generated would have unique file names since our Scenario.name is always unique.

It is important to note that while running parametrized simulations, one should set the Scenario.name parameter appropriately. For example, in the aforementioned settings file, if we set the scenario name as:

To simulate all these scenarios, type in the following in the terminal:

Scenario.name = sample_scenario

then irrespective of how many times the simulation is executed in batch mode, only a single report file will be generated by the name sample_scenario_MessageStatsReport.txt (or any other report files specified). Therefore, it is essential that the scenario name setting contains all desired parameters of the simulation to generate appropriate reports.

Caution:

If you have developed a new routing or movement or any other module that has one or more (non-final) static member(s), you need to take extra care. If such a module is used in the batch mode of simulation for multiple, say 10 times, the static member(s) would retain the value(s) initialized at the first time. For example, if the static member is assigned a new ArrayList  all the 10 simulation instances would retain such an assignment. In reality,  you perhaps would want a new ArrayList created at every run-index of the simulation.

To mitigate such a scenario, static members should not be initialized inside a constructor. Rather, use a static block to initialize them. To mitigate this issue, you should provide a reset() method for your class. As an example, consider the following code snippet from the BusControlSystem class in the movement package:

static {
    DTNSim.registerForReset(BusControlSystem.class
        .getCanonicalName());
    reset();
}

public static void reset() {
    systems = new HashMap<Integer, BusControlSystem>();
}

The registerForReset() method specifies that the concerned class must be reset at the beginning of any simulation. The reset() method does the actual resetting task. Inside this method, you should initialize all static members to be used.  In the above example, the reset() method prevents the systems member from using the same HashMap over and over again for the successive simulations executed in batch mode.

Further details on how resetting actually works can be found in the DTNSim class. The said problem with static members, however, does not arise if you are not running the simulation multiple times, i.e., either not using batch mode or using -b 1.

Continuing with our previous example, if one looks at the order of the execution of the above scenarios, one would find something like this:

1. RandomWalk_EpidemicRouter_2M
2. RandomWaypoint_ProphetRouter_5M
3. RandomWalk_SprayAndWaiRoutert_10M
4. RandomWaypoint_EpidemicRouter_15M

and so on. This order of execution is not similar to the nested for loops. Rather, the "array" of values of each parameter is treated somewhat like a cyclic array, and during each iteration, the value pointed by the current index is selected.

This results in a problem. Let us consider the modified settings file:

Scenario.name = %%Group.movementModel%%_%%Group.router%%
Group.movementModel = [RandomWalk; RandomWaypoint;]
Group.router = [EpidemicRouter; ProphetRouter;]

And, say, we run the simulation as:

./one.sh -b 4 multiple_scenarios.txt

In this case, although four simulations would be performed, only two scenarios would be simulated (instead of four, as desired)! The two simulated scenarios would be: RandomWalk_EpidemicRouter and RandomWaypoint_ProphetRouter.

The ONE simulator team has mentioned a simple work around for this. Let us consider that there are n parameters in our simulation namely, P1, P2, …, Pn. Let, v1, v2, …, vn, respectively, denote the number of values specified for each parameter. Thus, to simulate v1 * v2 * … * vn unique scenarios, one should ensure that the pairwise GCD of v1, v2, …, vn is 1. In our first example, we had GCD(2, 3) = 1, GCD(3, 5) = 1 and GCD(2, 5) = 1, and, therefore, resulted in 30 unique scenarios. In other words, the numbers in (v1, v2, …, vn) should be pairwise coprime.

In the latter example, however, we had GCD(2, 2) = 2, which does not satisfy the necessary criteria. As a consequence, all (four) simulation scenarios were not executed.

If this confuses you, simply specify prime number of values for each desired simulation parameter. However, if the number of such parameters is high, this would result in executing excess and unwanted scenarios!

Finally, let me mention a small trick here. Let us consider that there are 4 parameters say, P1, P2, P3 and P4. Let, the # of values for each parameter be 3, 4, 6, 9. Further, let us consider that the last parameter (P4) be the simulation duration. Note that although the numbers (3, 4, 6, 9) are unique, they are not pairwise coprime: GCD(3, 6) = 3. So, let us consider their nearest prime numbers: (3, 5, 7, 11). Clearly, all these are pairwise coprime numbers.

However, note that since we have increased the # of values of P4 from 9 to 11, total number of scenarios to simulate has also increased! This would make the simulations run for a long time.

The trick to save (real) time is to set the simulation duration of the two (or more) extra cases to, say 1 second. That would result in valid simulation scenarios, which, however, would terminate instantly. Another such example would be — if you are considering variable transmission range — to set the radio range to 0 for the extra cases. In such a case, no messages would be exchanged, and the simulation would terminate quite fast. Yet another example would be to use PassiveRouter when you need some extra cases for the routing protocol.


Please also look at question 5 and its answer at the ONE FAQs under the Reports section.

Revision History:


Published: 2013
09 June 2014: Updated example; added few tricks; added link to the ONE FAQ
16 June 2014: Updated description
06 February 2015: Fixed router names in the example code
02 December 2015: Added caution on static fields
06 May 2016: Some minor editing
12 June 2017: Updated explanation for reset()

Previous: The ONE tutorial

Next: Results of Performance Evaluations and Confidence Intervals