Skip to main content

Results of Performance Evaluation and Confidence Intervals

Note: If you are planning to use the Python scripts discussed here, please copy them from the respective Github Gist URLs (linked with the respective script names) in order to avoid malformed indentation, if any. Moreover, you would likely need to adapt these scripts to suit your performance evaluation scenarios.


The ONE simulator provides various reports, which serve as the basis for performance evaluations. For example, measure of different statistics can be obtained from the MessageStatsReport. However, in practice, it is not enough to obtain (and plot) the average value of a statistic, but to show the corresponding confidence interval as well. In this tutorial, we will look at:
  1. Computing the average of a statistic where the sample values are obtained from multiple randomized scenarios. In particular, we will look at obtaining the average value of message delivery ratio from the MessageStatsReport. Henceforth, a random scenario will be alternatively termed as a sample.
  2. Computing the 95% confidence interval (CI) for the average of the delivery ratio.
  3. Plotting the results with GnuPlot.
We consider a simple scenario — let us find and compare the delivery ratio of  messages obtained using the Epidemic and Spray-and-Wait routing protocols for three different areas: 500x500, 1000x1000, and 1500x1500. For each such protocol and area combination, for example, Epidemic with 500x500, we will simulate 5 scenarios using 5 different values of the setting MovementModel.rngSeed. (Note: A tutorial on parametrized simulations has been covered earlier.) Therefore, we have in total 2 * 3 * 5 = 30 scenarios. The content of the simulation settings file is as follows:
We execute the simulations with the scenario file as:
./one.sh -b 30

Once the simulations are over, we 30 MessageStatsReport files would be generated. Now, for each routing protocol and area used here, we consider the 5 samples, and obtain the mean delivery ratio. We repeat this for the other two protocols as well. Based on the sample size, average, and standard deviation, the 95% CI can be calculated. Finally, these data can be used to obtain a nice plot with the GnuPlot.

Let us summarize the contents (and usage) of the code files here. The _gen_stats.py file provides some useful code to compute the average and standard deviation of a list of numbers. It also provides a method to calculate the 95% CI. In short, this script acts as a library to be used by others.

The del_ratio.py script iterates over all the samples for each parameter value, and computes the average delivery ratio of the messages. All such averages can be easily grouped together simply by controlling the "print" statement inside the nested loops. Addition of a new parameters is simple — just add another for loop in the nesting.
To use this script, execute it with python and redirect the output to a file:
python del_ratio.py > data.in

The plots.gp file contains code to plot the results with GnuPlot. In this case, we use the output stored in the "data.in" file obtained above and plot them using bar graphs.
The final result looks like this.
The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.


Unless there is any change of plan (or user demand!), this concludes the final part of the ONE tutorial trilogy. Hope this is useful to the students and researchers. A big thanks to all who have used this tutorial and made it popular.
This blog, however, would be updated at times with different topics.


Previous: Parametrized Simulations

Revision history:

02 Apr 2017: Updated code listing via GitHub Gist
05 May 2016: Revised content
07 Aug 2013: Published

Comments

  1. thanks.. pLEASE make more advanced tutorials..

    ReplyDelete
    Replies
    1. Hi Ganesh,

      Thanks for your feedback! Hope something more could be added in the future.

      Delete
  2. Thank you for the tutorial. It help me to understand ONE sim.

    ReplyDelete
  3. another awesome tutorial , I really greatful and appreciate your effort

    ReplyDelete
  4. how to fix the number of message generated during the simulation??

    ReplyDelete
  5. Can u tell me how to setup Working Day Movement Model ? in One simultor

    ReplyDelete
    Replies
    1. Look at the default_settings.txt file that ships with the source code of the simulator.

      Delete
  6. This comment has been removed by the author.

    ReplyDelete

Post a Comment

Popular posts from this blog

Specifying Source and Destination of Messages

One of the frequently asked questions in the community is how to specify which particular nodes would act as source(s) and destination(s) of the messages created in the ONE simulator. The simulator, in fact, provides a pair of settings (shown below in bold face) aimed for this particular purpose.

Let us consider that there are $n + 1$ nodes in an OMN.  Further, let the nodes with addresses from $x$ to $y$, both inclusive, would create messages. The nodes in the range $w$ to $z$, both inclusive, would be the destinations of those messages, where $0 \le x \le y \le n$, and $0 \le w \le z \le n$. Then, the corresponding simulation scenario can be configured as follows.

## Message creation parameters # How many event generators Events.nrof = 1 # Class of the first event generator Events1.class = MessageEventGenerator # (Following settings are specific for the MessageEventGenerator class) # Creation interval in seconds (one new message every 25 to 35 seconds) Events1.interval = 25,35 # Me…

Effects of Buffer Size on Delay Tolerant Routing

In this post, we look at how buffer size affects, if at all, the performance of the routing protocols in DTNs. For this purpose, we will consider the following five routing protocols:
EpidemicPROPHETSpray-and-Wait (SnW) First Contact (FC) Direct Delivery (DD)  Detailed discussion of these protocols is scoped out here. We just note that in case of Epidemic, there is unlimited replication of the messages. In PROPHET, however, the replication is usually less than that of Epidemic. On the other hand, SnW has a fixed limit (L) on possible number of replications of a message. Finally, FC and DD involve message forwarding -- not replication. So, in the latter cases, there is always a single copy of any message in the DTN.

We will consider the buffer sizes from 20 MB to 180 MB, both inclusive, in steps of 20 MB so that we have total 9 different buffer sizes. We will use the real-life connection traces from Infocom'06. Therefore, we will need to simulate 5 * 9 = 45 scenarios to get the rel…

Controlling Transmission Range from within the Simulation

While simulating scenarios with the ONE simulator, one typically defines one or more network interfaces, and add them to the nodes as required. This use case prevails in most of the scenarios. However, a drawback here is that different network interfaces are mutually incompatible — an interface of type 1 can't communicate with any interface not of type 1.

Under certain circumstances, it might be required to control the transmission range of one or more network interfaces dynamically from within the simulation. For example, in one of my works, "On emotional aspects in Mission-Oriented Opportunistic Networks", I have considered the case where users occasionally turn off their device radios based on their contemporary emotions. In particular, the following shows how to set the radio range to 0: ModuleCommunicationBus comBus = host.getComBus(); // Store the original radio range the first time it is reset if (this.originalRadioRange == -1) { this.originalRadioRange = Double…