next up previous
Next: Applications and Extensions of Up: Sensitivity of Dependency Detection Previous: G-Test Sensitivity to Execution

Empirically Testing for Data Sensitivity.

We know that the value of G increases linearly with increases in the number of patterns in the execution traces, but only if the ratios in the contingency table remain the same, as the number of patterns increases. In trying to decide how many execution traces to gather, we also need to know whether the results will be vulnerable to noise, which is more apparent with few patterns. Unlike the sensitivity to total number of patterns, the sensitivity of the G-test to noise is complicated.

We can evaluate empirically whether, in practice, getting slightly more or fewer execution traces would have significantly changed which dependencies were detected in execution traces gathered from Phoenix. We do so by seeing how many of the dependencies would not have been detected if the counts in row one in the contingency table varied by a small amount. For example, if the contingency table in Table 1 had [52,35,240,643] instead of [52,33,240,643], then G=40.42, which is not much different than the value for Table 1 of G=42.86. To determine whether the dependencies detected in the execution traces are vulnerable to noise, we can do the following test: 1) construct the contingency table for dependencies detected in execution traces, 2) vary, one at a time, the counts of row one, column one and row one, column two by tex2html_wrap_inline319 , and 3) run a G-test on the resulting contingency tables. By tweaking the contingency table cell values in this manner, we check the sensitivity of G to noise in the data. Both columns of the first row were varied because some of the first column counts were 1, which makes it impossible to test whether a lower ratio of first column to second column might not have influenced the value of G more than a higher ratio. We tweaked the counts by tex2html_wrap_inline319 because many contingency tables contained cell counts of less than 5, varying by tex2html_wrap_inline319 spans that range.

   tex2html_wrap_inline337
Table 2: Dependencies remaining after tweaking contingency tables. The table includes the number of dependencies remaining after tweaking over the total number of dependencies found in the execution traces from the four experiments.

Table 2 shows how many of the dependencies found in each of four sets of execution traces for Phoenix would remain if their contingency tables are so tweaked. About 65% of the dependencies remain after tweaking their contingency table values, meaning that the counts in the first row of the contingency table can be changed by tex2html_wrap_inline319 without dropping the significance of G below the level of table118 . So, 35% of the dependencies detected would disappear if a few patterns more or less were included in the execution traces. Based on this testing of execution traces from Phoenix, dependency detection is sensitive to small differences in the content of the execution traces. Most of the dependencies that were vulnerable to the tweaking were based on execution traces that included few instances of the precursor/failure pattern, 23 out of 44 or 52% of the dependencies that disappeared were based on contingency tables in which one of the counts in the first row was less than five.

The implication of the sensitivity of dependency detection to noise in the execution traces is that rare patterns are especially sensitive to noise and so should be viewed skeptically. One must interpret the results of dependency detection with care: if ``sensitive" dependencies are discarded, then rare events may remain undetected; at the same time, one does not wish to chase chimeras. Interpreting dependencies requires weighing false positives against misses. If we are trying to identify dependencies between precursors that occur rarely or failures that occur rarely, then additional effort should be expended to get enough execution traces to ensure that the dependency is not due to noise.


next up previous
Next: Applications and Extensions of Up: Sensitivity of Dependency Detection Previous: G-Test Sensitivity to Execution

adele howe
Wed Oct 23 13:21:04 MDT 1996