Skip to main content

Commonly Used Metrics for Performance Evaluation


The following metrics are commonly used when evaluating scenarios related to DTN protocols.
  • Delivery ratio of the messages,
  • Average message delivery latency
  • Overhead ratio (of the underlying routing mechanism)
Suppose that $M$ be the set of all messages created in the network and $M_d$ be the set of all messages delivered. Then, the delivery ratio is computed as $|M_d| / |M|$.

Now let the $i^{th}$ delivered message was created at time $c_i$ and delivered at time $d_i$. Then the average message delivery latency is computed as $(\sum_{i = 1}^{|M_d|} (d_i - c_i)) / |M_d|$. Note that, in Statistics, mean, median and mode are all the measures of average. But "loosely speaking", unless otherwise specified, we refer to the "mean" value when we say "average." Nevertheless, the MessageStatsReport in the ONE simulator provides a measure of both the mean and median values wherever appropriate.

One may refer the above metric as "end-to-end delay." Personally, I think such usage is inappropriate given that, by definition, DTNs typically lack end-to-end paths.

Finally, let $r_i$ be the number of replications of any message $m_i \in M$. Then the overhead ratio is determined as $(\sum_{i = 1}^{|M|} r_i - |M_d|) / M_d$.

The above definitions are generic. However, if you feel that they are helpful and wish to incorporate into your research article, you may consider citing it as:

B. K. Saha (2014, Mar.) Commonly Used Metrics for Performance Evaluation. Accessed: DD Mon. YYYY. [Online]. Available: http://delay-tolerant-networks.blogspot.com/2014/03/commonly-used-metrics.html

Comments

Popular posts from this blog

Specifying Source and Destination of Messages

One of the frequently asked questions in the community is how to specify which particular nodes would act as source(s) and destination(s) of the messages created in the ONE simulator. The simulator, in fact, provides a pair of settings (shown below in bold face) aimed for this particular purpose.

Let us consider that there are $n + 1$ nodes in an OMN.  Further, let the nodes with addresses from $x$ to $y$, both inclusive, would create messages. The nodes in the range $w$ to $z$, both inclusive, would be the destinations of those messages, where $0 \le x \le y \le n$, and $0 \le w \le z \le n$. Then, the corresponding simulation scenario can be configured as follows.

## Message creation parameters # How many event generators Events.nrof = 1 # Class of the first event generator Events1.class = MessageEventGenerator # (Following settings are specific for the MessageEventGenerator class) # Creation interval in seconds (one new message every 25 to 35 seconds) Events1.interval = 25,35 # Me…

Effects of Buffer Size on Delay Tolerant Routing

In this post, we look at how buffer size affects, if at all, the performance of the routing protocols in DTNs. For this purpose, we will consider the following five routing protocols:
EpidemicPROPHETSpray-and-Wait (SnW) First Contact (FC) Direct Delivery (DD)  Detailed discussion of these protocols is scoped out here. We just note that in case of Epidemic, there is unlimited replication of the messages. In PROPHET, however, the replication is usually less than that of Epidemic. On the other hand, SnW has a fixed limit (L) on possible number of replications of a message. Finally, FC and DD involve message forwarding -- not replication. So, in the latter cases, there is always a single copy of any message in the DTN.

We will consider the buffer sizes from 20 MB to 180 MB, both inclusive, in steps of 20 MB so that we have total 9 different buffer sizes. We will use the real-life connection traces from Infocom'06. Therefore, we will need to simulate 5 * 9 = 45 scenarios to get the rel…

Controlling Transmission Range from within the Simulation

While simulating scenarios with the ONE simulator, one typically defines one or more network interfaces, and add them to the nodes as required. This use case prevails in most of the scenarios. However, a drawback here is that different network interfaces are mutually incompatible — an interface of type 1 can't communicate with any interface not of type 1.

Under certain circumstances, it might be required to control the transmission range of one or more network interfaces dynamically from within the simulation. For example, in one of my works, "On emotional aspects in Mission-Oriented Opportunistic Networks", I have considered the case where users occasionally turn off their device radios based on their contemporary emotions. In particular, the following shows how to set the radio range to 0: ModuleCommunicationBus comBus = host.getComBus(); // Store the original radio range the first time it is reset if (this.originalRadioRange == -1) { this.originalRadioRange = Double…