Skip to main content

Results of Performance Evaluation and Confidence Intervals

Note: If you are planning to use the Python scripts discussed here, please copy them from the respective Github Gist URLs (linked with the respective script names) in order to avoid malformed indentation, if any. Moreover, you would likely need to adapt these scripts to suit your performance evaluation scenarios.


The ONE simulator provides various reports, which serve as the basis for performance evaluations. For example, measure of different statistics can be obtained from the MessageStatsReport. However, in practice, it is not enough to obtain (and plot) the average value of a statistic, but to show the corresponding confidence interval as well. In this tutorial, we will look at:
  1. Computing the average of a statistic where the sample values are obtained from multiple randomized scenarios. In particular, we will look at obtaining the average value of message delivery ratio from the MessageStatsReport. Henceforth, a random scenario will be alternatively termed as a sample.
  2. Computing the 95% confidence interval (CI) for the average of the delivery ratio.
  3. Plotting the results with GnuPlot.
We consider a simple scenario — let us find and compare the delivery ratio of  messages obtained using the Epidemic and Spray-and-Wait routing protocols for three different areas: 500x500, 1000x1000, and 1500x1500. For each such protocol and area combination, for example, Epidemic with 500x500, we will simulate 5 scenarios using 5 different values of the setting MovementModel.rngSeed. (Note: A tutorial on parametrized simulations has been covered earlier.) Therefore, we have in total 2 * 3 * 5 = 30 scenarios. The content of the simulation settings file is as follows:
We execute the simulations with the scenario file as:
./one.sh -b 30

Once the simulations are over, we 30 MessageStatsReport files would be generated. Now, for each routing protocol and area used here, we consider the 5 samples, and obtain the mean delivery ratio. We repeat this for the other two protocols as well. Based on the sample size, average, and standard deviation, the 95% CI can be calculated. Finally, these data can be used to obtain a nice plot with the GnuPlot.

Let us summarize the contents (and usage) of the code files here. The _gen_stats.py file provides some useful code to compute the average and standard deviation of a list of numbers. It also provides a method to calculate the 95% CI. In short, this script acts as a library to be used by others.

The del_ratio.py script iterates over all the samples for each parameter value, and computes the average delivery ratio of the messages. All such averages can be easily grouped together simply by controlling the "print" statement inside the nested loops. Addition of a new parameters is simple — just add another for loop in the nesting.
To use this script, execute it with python and redirect the output to a file:
python del_ratio.py > data.in

The plots.gp file contains code to plot the results with GnuPlot. In this case, we use the output stored in the "data.in" file obtained above and plot them using bar graphs.
The final result looks like this.
The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.


Unless there is any change of plan (or user demand!), this concludes the final part of the ONE tutorial trilogy. Hope this is useful to the students and researchers. A big thanks to all who have used this tutorial and made it popular.
This blog, however, would be updated at times with different topics.


Previous: Parametrized Simulations

Revision history:

02 Apr 2017: Updated code listing via GitHub Gist
05 May 2016: Revised content
07 Aug 2013: Published

Comments

  1. thanks.. pLEASE make more advanced tutorials..

    ReplyDelete
    Replies
    1. Hi Ganesh,

      Thanks for your feedback! Hope something more could be added in the future.

      Delete
  2. Thank you for the tutorial. It help me to understand ONE sim.

    ReplyDelete
  3. another awesome tutorial , I really greatful and appreciate your effort

    ReplyDelete
  4. how to fix the number of message generated during the simulation??

    ReplyDelete
  5. Can u tell me how to setup Working Day Movement Model ? in One simultor

    ReplyDelete
    Replies
    1. Look at the default_settings.txt file that ships with the source code of the simulator.

      Delete
  6. This comment has been removed by the author.

    ReplyDelete

Post a Comment

Popular posts from this blog

Specifying Source and Destination of Messages

One of the frequently asked questions in the community is how to specify which particular nodes would act as source(s) and destination(s) of the messages created in the ONE simulator. The simulator, in fact, provides a pair of settings (shown below in bold face) aimed for this particular purpose.

Let us consider that there are $n + 1$ nodes in an OMN.  Further, let the nodes with addresses from $x$ to $y$, both inclusive, would create messages. The nodes in the range $w$ to $z$, both inclusive, would be the destinations of those messages, where $0 \le x \le y \le n$, and $0 \le w \le z \le n$. Then, the corresponding simulation scenario can be configured as follows.

## Message creation parameters # How many event generators Events.nrof = 1 # Class of the first event generator Events1.class = MessageEventGenerator # (Following settings are specific for the MessageEventGenerator class) # Creation interval in seconds (one new message every 25 to 35 seconds) Events1.interval = 25,35 # Me…

Text Highlighting in Latex

While preparing a manuscript with Latex, it is often useful to highlight the changes made in the current revision with a different color. This can be achieved using the\textcolor command provided by Latex. For example,\textcolor{red}{Hello World}would display the string "Hello World" in red color.

However, the final/published copy of the manuscript does not contain any highlighted text. Therefore, if a large volume of changes were made, it becomes tiresome at the end to find and remove all the individual portions of highlighted text.

This can be circumvented by defining a utility command to switch highlighting on and off as desired. In the following, we define a new Latex command, highlighttext, for this purpose. The command takes only a single argument -- the text that it should highlight.

    \usepackage{color}

% For highlighting changes in this version with red color
    \newcommand{\highlighttext}[1] {\textcolor{red}{#1}}
    % Remove all text highlighting
    % Useful to gener…

IllegalArgumentException in Simulations

At times, users have reported of runtime exceptions while running a simulation that uses PRoPHET. The error signature looks like the following.

Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeHi(Unknown Source) at java.util.TimSort.mergeAt(Unknown Source) at java.util.TimSort.mergeCollapse(Unknown Source) at java.util.TimSort.sort(Unknown Source) at java.util.Arrays.sort(Unknown Source) at java.util.ArrayList.sort(Unknown Source) at java.util.Collections.sort(Unknown Source)
It may be noted that the default sorting algorithm has changed since Java 7 — rather than Merge sort, Tim sort is now used by default. However, it seems like it is possible to override the default sorting algorithm in Java using the following argument to the JVM.
-Djava.util.Arrays.useLegacyMergeSort=true
The execution command in the one.sh (or one.bat) file should be updated with the above argument to something like this:

j…