Skip to main content

Results of Performance Evaluation and Confidence Intervals

Note: If you are planning to use the Python scripts discussed here, please copy them from the respective Github Gist URLs (linked with the respective script names) in order to avoid malformed indentation, if any. Moreover, you would likely need to adapt these scripts to suit your performance evaluation scenarios.

The ONE simulator provides various reports, which serve as the basis for performance evaluations. For example, measure of different statistics can be obtained from the MessageStatsReport. However, in practice, it is not enough to obtain (and plot) the average value of a statistic, but to show the corresponding confidence interval as well. In this tutorial, we will look at:
  1. Computing the average of a statistic where the sample values are obtained from multiple randomized scenarios. In particular, we will look at obtaining the average value of message delivery ratio from the MessageStatsReport. Henceforth, a random scenario will be alternatively termed as a sample.
  2. Computing the 95% confidence interval (CI) for the average of the delivery ratio.
  3. Plotting the results with GnuPlot.
We consider a simple scenario — let us find and compare the delivery ratio of  messages obtained using the Epidemic and Spray-and-Wait routing protocols for three different areas: 500x500, 1000x1000, and 1500x1500. For each such protocol and area combination, for example, Epidemic with 500x500, we will simulate 5 scenarios using 5 different values of the setting MovementModel.rngSeed. (Note: A tutorial on parametrized simulations has been covered earlier.) Therefore, we have in total 2 * 3 * 5 = 30 scenarios. The content of the simulation settings file is as follows:
We execute the simulations with the scenario file as:
./ -b 30

Once the simulations are over, we 30 MessageStatsReport files would be generated. Now, for each routing protocol and area used here, we consider the 5 samples, and obtain the mean delivery ratio. We repeat this for the other two protocols as well. Based on the sample size, average, and standard deviation, the 95% CI can be calculated. Finally, these data can be used to obtain a nice plot with the GnuPlot.

Let us summarize the contents (and usage) of the code files here. The file provides some useful code to compute the average and standard deviation of a list of numbers. It also provides a method to calculate the 95% CI. In short, this script acts as a library to be used by others.

The script iterates over all the samples for each parameter value, and computes the average delivery ratio of the messages. All such averages can be easily grouped together simply by controlling the "print" statement inside the nested loops. Addition of a new parameters is simple — just add another for loop in the nesting.
To use this script, execute it with python and redirect the output to a file:
python >

The file contains code to plot the results with GnuPlot. In this case, we use the output stored in the "" file obtained above and plot them using bar graphs.
The final result looks like this.
The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.

Unless there is any change of plan (or user demand!), this concludes the final part of the ONE tutorial trilogy. Hope this is useful to the students and researchers. A big thanks to all who have used this tutorial and made it popular.
This blog, however, would be updated at times with different topics.

Previous: Parametrized Simulations

Revision history:

02 Apr 2017: Updated code listing via GitHub Gist
05 May 2016: Revised content
07 Aug 2013: Published


  1. thanks.. pLEASE make more advanced tutorials..

    1. Hi Ganesh,

      Thanks for your feedback! Hope something more could be added in the future.

  2. Thanks for this tutorial. Good job.

  3. Thank you for the tutorial. It help me to understand ONE sim.

  4. another awesome tutorial , I really greatful and appreciate your effort

  5. how to fix the number of message generated during the simulation??

  6. Can u tell me how to setup Working Day Movement Model ? in One simultor

    1. Look at the default_settings.txt file that ships with the source code of the simulator.

  7. This comment has been removed by the author.

  8. Great post! No fluff, straight to the point and so fact-filled it’s hard to comment without repeating the point you’ve already made.
    Good stuff!

    Read my Latest Post

  9. Dear Barun how r u? is there any mechanism to add incentives on existing routing algorithms


Post a Comment

Popular posts from this blog

Text Highlighting in Latex

While preparing a manuscript with Latex, it is often useful to highlight the changes made in the current revision with a different color. This can be achieved using the \ textcolor command provided by Latex. For example, \textcolor {red}{Hello World} would display the string "Hello World" in red color. However, the final/published copy of the manuscript does not contain any highlighted text. Therefore, if a large volume of changes were made, it becomes tiresome at the end to find and remove all the individual portions of highlighted text. This can be circumvented by defining a utility command to switch highlighting on and off as desired. In the following, we define a new Latex command, highlighttext , for this purpose. The command takes only a single argument—the text to be highlighted.     \usepackage {color}    % For highlighting changes in this version with red color   \newcommand { \highlighttext }[1] { \textcolor {red}{#1}}   % Remove all text highlighting

The ONE Knowledge Base

The Opportunistic Network Environment (ONE) simulator has become largely popular and widely used in research on DTNs and Opportunistic Networks. The simulator has an associated friendly community, where users can ask questions and get their doubts clarified. Indeed, the community is vibrant and its members are useful. However, there are two issues with this community. First, the archives of the community's mailing list are not public. This is unlike many other similar lists hosted using Mailman, which are public and, therefore, can be searched using Google. Hence, time and again people have asked the question -- how to search the email archives? The second issue is largely due to the first. Due to the lack of searching facility over the archives, often same question tend to be asked by the new users repeatedly. People here are patient enough to reply to them. But such necessity could be entirely discounted had there been a facility to search the archives. In particular, it w

The ONE KB has a new home

The ONE Knowledge Base is now hosted at If you are unaware, the ONE KB allows you to search the old email archives of the simulator's community. Therefore, if you have any question related to simulation, you may query the existing database at the above link. Chances are good that your question might already have been answered previously. If not, you can still post an email to the community's mailing list. Have you tried the ONE KB already? How was your experience? Was it helpful? Let me know in the comments!