Barun Saha's blog on AI, GenAI, DTNs, and other networks.
Search This Blog
Results of Performance Evaluation and Confidence Intervals
Note: If you are planning to use the Python scripts discussed here, please copy them from the respective Github Gist URLs (linked with the respective script names) in order to avoid malformed indentation, if any. Moreover, you would likely need to adapt these scripts to suit your performance evaluation scenarios.
The ONE simulator provides various reports, which serve as the basis for performance evaluations. For example, measure of different statistics can be obtained from the MessageStatsReport. However, in practice, it is not enough to obtain (and plot) the average value of a statistic, but to show the corresponding confidence interval as well. In this tutorial, we will look at:
Computing the average of a statistic where the sample values are obtained from multiple randomized scenarios. In particular, we will look at obtaining the average value of message delivery ratio from the MessageStatsReport. Henceforth, a random scenario will be alternatively termed as a sample.
Computing the 95% confidence interval (CI) for the average of the delivery ratio.
Plotting the results with GnuPlot.
We consider a simple scenario — let us find and compare the delivery ratio of messages obtained using the Epidemic and Spray-and-Wait routing protocols for three different areas: 500x500, 1000x1000, and 1500x1500. For each such protocol and area combination, for example, Epidemic with 500x500, we will simulate 5 scenarios using 5 different values of the setting MovementModel.rngSeed. (Note: A tutorial on parametrized simulations has been covered earlier.) Therefore, we have in total 2 * 3 * 5 = 30 scenarios. The content of the simulation settings file is as follows:
We execute the simulations with the scenario file as:
./one.sh -b 30
Once the simulations are over, we 30 MessageStatsReport files would be generated. Now, for each routing protocol and area used here, we consider the 5 samples, and obtain the mean delivery ratio. We repeat this for the other two protocols as well. Based on the sample size, average, and standard deviation, the 95% CI can be calculated. Finally, these data can be used to obtain a nice plot with the GnuPlot.
Let us summarize the contents (and usage) of the code files here. The _gen_stats.py file provides some useful code to compute the average and standard deviation of a list of numbers. It also provides a method to calculate the 95% CI. In short, this script acts as a library to be used by others.
The del_ratio.py script iterates over all the samples for each parameter value, and computes the average delivery ratio of the messages. All such averages can be easily grouped together simply by controlling the "print" statement inside the nested loops. Addition of a new parameters is simple — just add another for loop in the nesting.
To use this script, execute it with python and redirect the output to a file:
python del_ratio.py > data.in
The plots.gp file contains code to plot the results with GnuPlot. In this case, we use the output stored in the "data.in" file obtained above and plot them using bar graphs.
The final result looks like this.
The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.
Unless there is any change of plan (or user demand!), this concludes the final part of the ONE tutorial trilogy. Hope this is useful to the students and researchers. A big thanks to all who have used this tutorial and made it popular.
This blog, however, would be updated at times with different topics.
While preparing a manuscript with Latex, it is often useful to highlight the changes made in the current revision with a different color. This can be achieved using the \ textcolor command provided by Latex. For example, \textcolor {red}{Hello World} would display the string "Hello World" in red color. However, the final/published copy of the manuscript does not contain any highlighted text. Therefore, if a large volume of changes were made, it becomes tiresome at the end to find and remove all the individual portions of highlighted text. This can be circumvented by defining a utility command to switch highlighting on and off as desired. In the following, we define a new Latex command, highlighttext , for this purpose. The command takes only a single argument—the text to be highlighted. \usepackage {color} % For highlighting changes in this version with red color \newcommand { \highlighttext }[1] { \textcolor {red}{#1}} % Remove...
A snapshot of the Aya collection (Bengali) . Image taken from HuggingFace. In February 2024, Cohere launched Aya , a multilingual Large Language Model (LLM). Alongside, a set of datasets used to train Aya has also been released. For example, the aya_dataset consists around 205K examples annotated by humans. On the other hand, the recently released aya_collection_language_split is a gigantic dataset with more than 500 million data points spread across more than 100 languages. As the name suggests, this dataset is split by language. For example, all data points in Bengali, irrespective of the underlying task, can be found in a single split. Apart from the original human-annotated examples from the aya_dataset, aya_collection_language_split also contains a lot of translated and templated data. The dataset is released using an Apache-2.0 license, allowing academic and commercial use. The Bengali Language Split Each language split in the Aya collection has three splits. The Bengali split,...
One of the frequently asked questions in the community is how to specify which particular nodes would act as source(s) and destination(s) of the messages created in the ONE simulator. The simulator, in fact, provides a pair of settings (shown below in bold face) aimed for this particular purpose. Let us consider that there are $n + 1$ nodes in an OMN. Further, let the nodes with addresses from $x$ to $y$, both inclusive, would create messages. The nodes in the range $w$ to $z$, both inclusive, would be the destinations of those messages, where $0 \le x \le y \le n$, and $0 \le w \le z \le n$. Then, the corresponding simulation scenario can be configured as follows. ## Message creation parameters # How many event generators Events.nrof = 1 # Class of the first event generator Events1.class = MessageEventGenerator # (Following settings are specific for the MessageEventGenerator class) # Creation interval in seconds (one new message every 25 to 35 seconds) Events1.interval ...
The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.
thanks.. pLEASE make more advanced tutorials..
ReplyDeleteHi Ganesh,
DeleteThanks for your feedback! Hope something more could be added in the future.
Thanks for this tutorial. Good job.
ReplyDeleteThanks Sharif!
DeleteThank you for the tutorial. It help me to understand ONE sim.
ReplyDeleteThank u very much
ReplyDeleteanother awesome tutorial , I really greatful and appreciate your effort
ReplyDeletehow to fix the number of message generated during the simulation??
ReplyDeleteCan u tell me how to setup Working Day Movement Model ? in One simultor
ReplyDeleteLook at the default_settings.txt file that ships with the source code of the simulator.
DeleteThis comment has been removed by the author.
ReplyDeleteDear Barun how r u? is there any mechanism to add incentives on existing routing algorithms
ReplyDelete