Results of Performance Evaluation and Confidence Intervals

Note: If you are planning to use the Python scripts discussed here, please copy them from the respective Github Gist URLs (linked with the respective script names) in order to avoid malformed indentation, if any. Moreover, you would likely need to adapt these scripts to suit your performance evaluation scenarios.


The ONE simulator provides various reports, which serve as the basis for performance evaluations. For example, measure of different statistics can be obtained from the MessageStatsReport. However, in practice, it is not enough to obtain (and plot) the average value of a statistic, but to show the corresponding confidence interval as well. In this tutorial, we will look at:

1. Computing the average of a statistic where the sample values are obtained from multiple randomized scenarios. In particular, we will look at obtaining the average value of message delivery ratio from the MessageStatsReport. Henceforth, a random scenario will be alternatively termed as a sample.
2. Computing the 95% confidence interval (CI) for the average of the delivery ratio.
3. Plotting the results with GnuPlot.

We consider a simple scenario — let us find and compare the delivery ratio of  messages obtained using the Epidemic and Spray-and-Wait routing protocols for three different areas: 500x500, 1000x1000, and 1500x1500. For each such protocol and area combination, for example, Epidemic with 500x500, we will simulate 5 scenarios using 5 different values of the setting MovementModel.rngSeed. (Note: A tutorial on parametrized simulations has been covered earlier.) Therefore, we have in total 2 * 3 * 5 = 30 scenarios. The content of the simulation settings file is as follows:

	#
	# Performance comparison of Epidemic and Snw for different area dimensions
	#
	# Group.router: 2
	# MovementModel.worldSize: 3
	# MovementModel.rngSeed: 5
	#
	# Total scenarios = 2 * 3 * 5 = 30
	#
	## Scenario settings
	Scenario.name = scenario_%%Group.router%%_area-%%MovementModel.worldSize%%_rng-%%MovementModel.rngSeed%%
	Scenario.simulateConnections = true
	Scenario.updateInterval = 1.0
	# 43200s == 12h
	Scenario.endTime = 43200
	Scenario.nrofHostGroups = 1
	## Interface-specific settings:
	# type : which interface class the interface belongs to
	# For different types, the sub-parameters are interface-specific
	# For SimpleBroadcastInterface, the parameters are:
	# transmitSpeed : transmit speed of the interface (bytes per second)
	# transmitRange : range of the interface (meters)
	# "Bluetooth" interface for all nodes
	btInterface.type = SimpleBroadcastInterface
	# Transmit speed of 2 Mbps = 250kBps
	# IEEE 802.11b: ~ 11 Mbits/s for 200 m
	btInterface.transmitSpeed = 250k
	btInterface.transmitRange = 10
	## Group-specific settings:
	# groupID : Group's identifier. Used as the prefix of host names
	# nrofHosts: number of hosts in the group
	# movementModel: movement model of the hosts (valid class name from movement package)
	# waitTime: minimum and maximum wait times (seconds) after reaching destination
	# speed: minimum and maximum speeds (m/s) when moving on a path
	# bufferSize: size of the message buffer (bytes)
	# router: router used to route messages (valid class name from routing package)
	# activeTimes: Time intervals when the nodes in the group are active (start1, end1, start2, end2, ...)
	# msgTtl : TTL (minutes) of the messages created by this host group, default=infinite
	## Group and movement model specific settings
	# pois: Points Of Interest indexes and probabilities (poiIndex1, poiProb1, poiIndex2, poiProb2, ... )
	# for ShortestPathMapBasedMovement
	# okMaps : which map nodes are OK for the group (map file indexes), default=all
	# for all MapBasedMovent models
	# routeFile: route's file path - for MapRouteMovement
	# routeType: route's type - for MapRouteMovement
	# Common settings for all groups
	Group.movementModel = RandomWalk
	Group.router = [EpidemicRouter; SprayAndWaitRouter;]
	SprayAndWaitRouter.binaryMode = true
	SprayAndWaitRouter.nrofCopies = 8
	# All nodes have the bluetooth interface
	Group.nrofInterfaces = 1
	Group.interface1 = btInterface
	# Speeds
	Group.speed = 2,5
	# Message TTL of 300 minutes (5 hours)
	Group.msgTtl = 300
	Group.nrofHosts = 50
	Group.groupID = n
	## Movement model settings
	# seed for movement models' pseudo random number generator (default = 0)
	MovementModel.rngSeed = [1; 2; 3; 4; 5;]
	# World's size for Movement Models without implicit size (width, height; meters)
	MovementModel.worldSize = [500,500; 1000,1000; 1500,1500;]
	# How long time to move hosts in the world before real simulation
	## Message creation parameters
	# How many event generators
	Events.nrof = 1
	# Class of the first event generator
	Events1.class = MessageEventGenerator
	# (following settings are specific for the MessageEventGenerator class)
	# Creation interval in seconds (one new message every 30 to 60 second)
	Events1.interval = 30,60
	# Message sizes (500kB - 1MB)
	Events1.size = 1k
	# range of message source/destination addresses
	Events1.hosts = 0, 50
	# Message ID prefix
	Events1.prefix = M
	## Reports - all report names have to be valid report classes
	# how many reports to load
	Report.nrofReports = 1
	Report.reportDir = my_reports/blog_tutorial
	# Report classes to load
	Report.report1 = MessageStatsReport
	## Optimization settings -- these affect the speed of the simulation
	## see World class for details.
	Optimization.cellSizeMult = 5
	Optimization.randomizeUpdateOrder = true

view raw default_settings.txt hosted with ❤ by GitHub

We execute the simulations with the scenario file as:

./one.sh -b 30

Once the simulations are over, we 30 MessageStatsReport files would be generated. Now, for each routing protocol and area used here, we consider the 5 samples, and obtain the mean delivery ratio. We repeat this for the other two protocols as well. Based on the sample size, average, and standard deviation, the 95% CI can be calculated. Finally, these data can be used to obtain a nice plot with the GnuPlot.

Let us summarize the contents (and usage) of the code files here. The _gen_stats.py file provides some useful code to compute the average and standard deviation of a list of numbers. It also provides a method to calculate the 95% CI. In short, this script acts as a library to be used by others.

	import csv
	'''
	A simple module with a method to get the value of a statistic from the MessageStatsReport of the ONE simulator. Also provides a method to compute the 95% confidence interval from a set of sample values.
	'''
	__author__ = "Barun Kumar Saha"
	__copyright__ = "Copyright 2013, Barun Kumar Saha"
	__license__ = "MIT"
	__version__ = "1.0"
	# Average of a list of numbers
	def get_average(numbers = []):
	avg = 0.0
	n = len(numbers)
	for i in xrange(0, n):
	avg += numbers[i]
	avg /= n
	return avg
	# Std. Dev. of a list of numbers
	def get_std_dev(num = []):
	n = len(num)
	avg = get_average(num)
	variance = 0.0
	for i in xrange(0, n):
	variance += (num[i] - avg) ** 2
	variance /= n
	std = variance ** 0.5
	return std
	# Get a named statistic from the MessageStats report file
	def get_stat(file_name, stat_name = 'delivery_prob'):
	result = 0.0
	with open(file_name, 'r') as report:
	reader = csv.reader(report, delimiter = ' ')
	for line in reader:
	if line[0].find(stat_name) == 0:
	result = float(line[1])
	break
	return result
	#
	# t-distribution table
	#
	#Tail Probabilities
	#One Tail 0.10 0.05 0.025 0.01 0.005 0.001 0.0005
	#Two Tails 0.20 0.10 0.05 0.02 0.01 0.002 0.001
	#-------+---------------------------------------------------------+-----
	# D 1 | 3.078 6.314 12.71 31.82 63.66 318.3 637 | 1
	# E 2 | 1.886 2.920 4.303 6.965 9.925 22.330 31.6 | 2
	# G 3 | 1.638 2.353 3.182 4.541 5.841 10.210 12.92 | 3
	# R 4 | 1.533 2.132 2.776 3.747 4.604 7.173 8.610 | 4
	# E 5 | 1.476 2.015 2.571 3.365 4.032 5.893 6.869 | 5
	# E 6 | 1.440 1.943 2.447 3.143 3.707 5.208 5.959 | 6
	# S 7 | 1.415 1.895 2.365 2.998 3.499 4.785 5.408 | 7
	# 8 | 1.397 1.860 2.306 2.896 3.355 4.501 5.041 | 8
	# O 9 | 1.383 1.833 2.262 2.821 3.250 4.297 4.781 | 9
	# F 10 | 1.372 1.812 2.228 2.764 3.169 4.144 4.587 | 10
	# 11 | 1.363 1.796 2.201 2.718 3.106 4.025 4.437 | 11
	# F 12 | 1.356 1.782 2.179 2.681 3.055 3.930 4.318 | 12
	# R 13 | 1.350 1.771 2.160 2.650 3.012 3.852 4.221 | 13
	# E 14 | 1.345 1.761 2.145 2.624 2.977 3.787 4.140 | 14
	# E 15 | 1.341 1.753 2.131 2.602 2.947 3.733 4.073 | 15
	# D 16 | 1.337 1.746 2.120 2.583 2.921 3.686 4.015 | 16
	# O 17 | 1.333 1.740 2.110 2.567 2.898 3.646 3.965 | 17
	# M 18 | 1.330 1.734 2.101 2.552 2.878 3.610 3.922 | 18
	#
	# Get CI of a mean
	# Currently hard coded for sample size = 10, 95% CI
	## 95% only
	__t_values = {
	1: 12.71,
	2: 4.303,
	3: 3.182,
	4: 2.776,
	5: 2.571,
	6: 2.447,
	7: 2.365,
	8: 2.306,
	9: 2.262,
	10: 2.228,
	11: 2.201,
	12: 2.179,
	13: 2.160,
	14: 2.145,
	15: 2.131,
	16: 2.120,
	17: 2.110,
	18: 2.101,
	}
	def confidence_interval_mean(sample_size, sample_sd):
	'''Only 95% CI'''
	# If sample_size < 30 and population SD is unknown, use t distribution
	# Else use std. normal distribution
	delta = 0
	root_n = sample_size ** 0.5
	if sample_size < 30:
	df = sample_size - 1
	# t for 95% CI and df = 10 - 1 = 9
	t = __t_values[df]
	delta = t * sample_sd / root_n
	else:
	delta = 1.96 * sample_sd / root_n
	return delta

view raw _gen_stats.py hosted with ❤ by GitHub

The del_ratio.py script iterates over all the samples for each parameter value, and computes the average delivery ratio of the messages. All such averages can be easily grouped together simply by controlling the "print" statement inside the nested loops. Addition of a new parameters is simple — just add another for loop in the nesting.

	#!/usr/bin/python
	'''
	A simple to compute the delivery ratio of the messages (with 95% CI) under
	various scenarios.
	'''
	__author__ = "Barun Kumar Saha"
	__copyright__ = "Copyright 2013, Barun Kumar Saha"
	__license__ = "MIT"
	__version__ = "1.0"
	import csv
	import _gen_stats as gs
	import math
	# Routers used
	routers = ('EpidemicRouter', 'SprayAndWaitRouter',)
	areas = ('500,500', '1000,1000', '1500,1500',)
	rng_max = 5
	for router in routers:
	print '# Router', router
	for area in areas:
	del_ratio = []
	for i in xrange(1, rng_max+1):
	fname = 'scenario_%s_area-%s_rng-%d_MessageStatsReport.txt' % (router, area, i,)
	del_ratio.append(gs.get_stat(fname))
	# Average delivery ratio
	avg = gs.get_average(del_ratio)
	sd = gs.get_std_dev(del_ratio)
	ci = gs.confidence_interval_mean(rng_max, sd)
	print '%s %.2f %.4f' % (area, avg, ci,)
	# Any two data sets for Gnuplot must be separated by exactly 2 blank lines
	print '\n'

view raw del_ratio.py hosted with ❤ by GitHub

To use this script, execute it with python and redirect the output to a file:

python del_ratio.py > data.in

The plots.gp file contains code to plot the results with GnuPlot. In this case, we use the output stored in the "data.in" file obtained above and plot them using bar graphs.

	set terminal postscript eps enhanced color solid colortext 9
	set output 'del_ratio.eps'
	set style fill pattern border
	set style histogram errorbars gap 1 lw 3.5
	set style data histograms
	set boxwidth 0.9
	set bars 1
	set xtics font "Times-Roman, 17"
	set ytics 0.2 font "Times-Roman, 17"
	set xlabel "Area (sq m)" font "Times-Roman, 18"
	set ylabel "Delivery ratio" font "Times-Roman, 18.5"
	#set xrange[-0.5:2.5]
	plot 'data.in' using 2:3:xtic(1) index 0 title "Epidemic" fs pattern 0 lt 3 lw 2, \
	'' using 2:3:xtic(1) index 1 title "SnW" fs pattern 1 lt 4 lw 2

view raw plots.gp hosted with ❤ by GitHub

The final result looks like this.

The image shown above is in PNG format, which is alright for display in a web page. However, if you are writing a research article, I strongly recommend that you export your plots as EPS figures and include them from your Latex file.

---

Unless there is any change of plan (or user demand!), this concludes the final part of the ONE tutorial trilogy. Hope this is useful to the students and researchers. A big thanks to all who have used this tutorial and made it popular.
This blog, however, would be updated at times with different topics.

---

Previous: Parametrized Simulations

Revision history:

02 Apr 2017: Updated code listing via GitHub Gist
05 May 2016: Revised content
07 Aug 2013: Published