Two sample T-test
Notation
Suppose that \(Y_{ij}\) is the response for subjects \(i=1,\ldots, n_j\) from population \(j=1,2\).
Use of the term treatment or group instead of population
Here the treatment is \(j=1\) microbial transplant vs \(j=2\) placebo.
We assume
\[Y_{ij}\text{ i.i.d. } N(\mu_j,\sigma^2)\;\;\;i=1,\ldots,n_i\;j=1,2.\]
Note, that we assume equal variances homoscedastic
(Unequal variances are referred to as heteroscedastic)
Hypotheses
Test \[ H_0: \mu_1 = \mu_2 \] against \[ H_1: \mu_1 \neq \mu_2 .\]
\(H_1\) is again the research hypothesis: the average relative abundance of Staphylococcus spp. is different upon microbial transplant then upon placebo treatment.
\(H_0\) and \(H_1\) can also be specified in terms of the effect size between the two treatments, \(\mu_1-\mu_2\) \[H_0: \mu_1-\mu_2 = 0,\] \[H_1: \mu_1-\mu_2 \neq 0.\]
We can estimate the effect size using the difference in sample means: \[\hat \mu_1-\hat \mu_2=\bar Y_1 -\bar Y_2.\]
Variance estimator
The experimental units are independent so the sample means are also independent and the variance on the difference is
\[\text{Var}_{\bar Y_1 -\bar Y_2}=\frac{\sigma^2}{n_1}+\frac{\sigma^2}{n_2}=\sigma^2 \left(\frac{1}{n_1}+\frac{1}{n_2}\right).\]
And the standard error becomes \[\sigma_{\bar Y_1 -\bar Y_2}=\sigma\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}.\]
The variance can be estimated within each group using the sample variance:
\[S_1^2 = \frac{1}{n_1-1}\sum_{i=1}^{n_1} (Y_{i1}-\bar{Y}_1)^2.\]
\[S_2^2 = \frac{1}{n_2-1}\sum_{i=1}^{n_2} (Y_{i2}-\bar{Y}_2)^2.\]
But, if we assume equal variances \(\sigma_1^2=\sigma_2^2=\sigma^2\) than we can estimate the variance more precise by using all observations in both groups. This variance estimator is also referred to as the pooled variance estimator: \(S^2_p\).
So \(S_1^2\) en \(S_2^2\) are estimators of the same parameter \(\sigma^2\).
And we can combine them into one estimator based on all \(n_1+n_2\) observations:
\[ S_p^2 = \frac{n_1-1}{n_1+n_2-2} S_1^2 + \frac{n_2-1}{n_1+n_2-2} S_2^2 = \frac{1}{n_1+n_2-2}\sum_{j=1}^2\sum_{i=1}^{n_j} (Y_{ij} - \bar{Y}_j)^2.\]
\[ S_p^2= \sum\limits_{j=1}^2\sum\limits_{i=1}^{n_j} \frac{(Y_{ij}-\bar{Y}_{.j})^2}{n_1+n_2-2}\]
The pooled variance estimator uses the squared deviations of the observations from their group mean and has \(n_1+n_2-2\) degrees of freedom.
Test statistic
Two-sample \(t\)-teststatistiek:
\[T = \frac{\bar{Y}_1-\bar{Y}_2}{\sqrt{\frac{S_p^2}{n_1}+\frac{S_p^2}{n_2}}} =
\frac{\bar{Y}_1 - \bar{Y}_2}{S_p\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}.\]
The statistic T follows a t-distribution with \(n_1+n_2-2\) under \(H_0\) is all data are independent, normally distributed and have equal variances.
Armpit example
We can implement the test in R:
t.test(rel~trt,data=ap,var.equal=TRUE)
Two Sample t-test
data: rel by trt
t = -5.0334, df = 18, p-value = 8.638e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-31.53191 -12.96072
sample estimates:
mean in group placebo mean in group transplant
44.15496 66.40127
On the \(5\%\) significance level we reject the null hypothesis in favor of the alternative hypothesis and conclude that the relative abundance of Staphylococcus spp. is on average extreme significant larger is in transplantation group than in the placebo group.
If there is no effect of the transplant we have a probability of less then 9 in \(100000\) to observe a test statistic in a random sample that is at least as extreme as what we observed in the armpit experiment.
This is extremely rare under \(H_0\).
If \(H_1\) is correct, we expect that the test statistic is larger in absolute value and expect small p-values. Hence we decide that there is a lot of evidence against \(H_0\) in favour of \(H_1\).
Good statistical practice is to report the \(p\)-value, but also effect size along with its confidence interval. So that we can judge the statistical significance and the biological relevance.
Conclusion
On average the relative abundance of Staphylococcus spp. in the microbiome of the armpit in the transplant group is extremely significantly different from that in the placebo group (\(p<<0.001\)). The relative abundance of Staphylococcus spp. is on average 22.2% larger in the transplant group than in the placebo group (95% CI [13.0,31.5]%).
Assumptions
Validity of t-test depends on distributional assumptions:
- Independence (design)
- One-sample t-test: normality of the observations
- Paired t-test: normality of the difference
- Two-sample t-test: Normality of the observations in both groups, and equal variances.
If the assumptions are not met, the null distribution does not follow a t-distribution, and, the p-values and critical values are incorrect.
To construct confidence intervals we also rely on these assumptions.
Evaluate normality
There also exist hypothesis tests (goodness-of-fit test), but their null hypothesis is that the data are normally distributed so we make a weak conclusion!
- Kolmogorov-Smirnov, Shapiro-Wilk en Anderson-Darling.
- In small samples they have a low power
- In large samples they often flag very small deviations as significant
Recommendation
Start with graphical exploration of the data and keep the sample size in mind to avoid overinterpretation of the plots.
If you have doubts, use simulation where you simulate data with the same sample size from a Normal distribution with the same mean and variance as the one that you observed in the sample
If you observed deviations of normality check in the literature how sensitive your method is such deviations of normality. (e.g. T-tests for instance are rather insensitive to deviations as long as the distribution of the data is symmetric.)
In large samples you can resort to the central limit theorem.
You might resort to transformations of the response.
Homoscedasticity
Boxplots: The box size is the inter quartile range (IQR) a robust estimator of the variance.
If the differences are not large \(\rightarrow\) homoscedasticiteit
Again you can use simulation to get insight in the differences you can expect.
Formal F-test can be used to compare the variances, but again under the null you assume equal variances, so the same criticism as for normality tests applies here.
Welch modified t-test
If the data are heteroscedastic, you can use a Welch two-sample T-test, which no longer uses the pooled variance estimator.
\[T = \frac{\bar{Y}_1 - \bar{Y}_2}{\sqrt{\frac{S^2_1}{n_1}+\frac{S^2_2}{n_2}}}\] with \(S^2_1\) en \(S^2_2\) the sample variances in both groups.
This statistic follows approximately a t-distribution with a number of degrees of freedom between \(\text{min}(n_1-1,n_2-1)\) and \(n_1+n_2-2\).
In R the degrees of freedom are estimated using the Welch- Satterthwaite approximation. You can do this by using the t.test
function with argument var.equal=FALSE
.
t.test(rel~trt,data=ap,var.equal=FALSE)
Welch Two Sample t-test
data: rel by trt
t = -5.0334, df = 15.892, p-value = 0.0001249
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-31.62100 -12.87163
sample estimates:
mean in group placebo mean in group transplant
44.15496 66.40127
Note that you can see that the Welch T-test is adopted in the title. The adjusted degrees of freedom are \(df = 17.876\) \(\pm\) to that of the conventional T-test, because the variances are approximately equal.
How to report?
Rule of thumb:
Report an estimate together with its confidence interval (and its p-value)
- The result of the test can be derived of the confidence interval
- It allows the reader to judge scientific relevance.
Welch Two Sample t-test
data: rel by trt
t = -5.0334, df = 15.892, p-value = 0.0001249
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-31.62100 -12.87163
sample estimates:
mean in group placebo mean in group transplant
44.15496 66.40127
The result of an \(\alpha\)-level t-test is equivalent with comparing the effect size under \(H_0\) with the \(1-\alpha\) CI.
An effect can be extremely statistically significant, but scientifically irrelevant. With a CI you will spot this.
LS0tCnRpdGxlOiAiNS4gU3RhdGlzdGljYWwgSW5mZXJlbmNlOiBUd28tc2FtcGxlIHQtdGVzdCIgICAKYXV0aG9yOiAiTGlldmVuIENsZW1lbnQiCmRhdGU6ICJzdGF0T21pY3MsIEdoZW50IFVuaXZlcnNpdHkgKGh0dHBzOi8vc3RhdG9taWNzLmdpdGh1Yi5pbykiCm91dHB1dDoKICAgIGh0bWxfZG9jdW1lbnQ6CiAgICAgIGNvZGVfZG93bmxvYWQ6IHRydWUgICAgCiAgICAgIHRoZW1lOiBjb3NtbwogICAgICB0b2M6IHRydWUKICAgICAgdG9jX2Zsb2F0OiB0cnVlCiAgICAgIGhpZ2hsaWdodDogdGFuZ28KICAgICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChpbmNsdWRlID0gVFJVRSwgY29tbWVudCA9IE5BLCBlY2hvID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgIG1lc3NhZ2UgPSBGQUxTRSwgd2FybmluZyA9IEZBTFNFKQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojU21lbGx5IGFybXBpdCBleGFtcGxlCgotIFNtZWxseSBhcm1waXRzIGFyZSBub3QgY2F1c2VkIGJ5IHN3ZWF0LCBpdHNlbGYuIFRoZSBzbWVsbCBpcyBjYXVzZWQgYnkgc3BlY2lmaWMgbWljcm8tb3JnYW5pc21zIGJlbG9uZ2luZyB0byB0aGUgZ3JvdXAgb2YgKkNvcnluZWJhY3Rlcml1bSBzcHAuKiB0aGF0IG1ldGFib2xpc2Ugc3dlYXQuCkFub3RoZXIgZ3JvdXAgb2YgYWJ1bmRhbnQgYmFjdGVyaWEgYXJlIHRoZSAqU3RhcGh5bG9jb2NjdXMgc3BwLiosIHRoZXNlIGJhY3RlcmlhIGRvIG5vdCBtZXRhYm9saXNlIHN3ZWF0IGluIHNtZWxseSBjb21wb3VuZHMuCgotIFRoZSBDTUVULWdyb3VwIGF0IEdoZW50IFVuaXZlcnNpdHkgZG9lcyByZXNlYXJjaCB0byBvbiB0cmFuc3BsYW50aW5nIHRoZSBhcm1waXQgbWljcm9iaW9tZSB0byBzYXZlIHBlb3BsZSB3aXRoIHNtZWxseSBhcm1waXRzLgoKLSBQcm9wb3NlZCBUaGVyYXB5OgogIAkxLiBSZW1vdmUgYXJtcGl0LW1pY3JvYmlvbWUgd2l0aCBhbnRpYmlvdGljcwogICAgMi4gSW5mbHVlbmNlIGFybXBpdCBtaWNyb2Jpb21lIHdpdGggbWljcm9iaWFsICB0cmFuc3BsYW50IChodHRwczovL3lvdXR1LmJlLzlSSUZ5cUxYZFZ3KQoKLSBFeHBlcmltZW50OgoKICAgIC0gMjAgc3ViamVjdHMgd2l0aCBzbWVsbHkgYXJtcGl0cyBhcmUgYXR0cmlidXRlZCB0byBvbmUgb2YgdHdvIHRyZWF0bWVudCBncm91cHMKICAgIC0gcGxhY2VibyAob25seSBhbnRpYmlvdGljcykKICAgIC0gdHJhbnNwbGFudCAoYW50aWJpb3RpY2EgZm9sbG93ZWQgYnkgbWljcm9iaWFsIHRyYW5zcGxhbnQpLgogICAgLSBUaGUgbWljcm9iaW9tZSBpcyBzYW1wbGVkIDYgd2Vla3MgdXBvbiB0aGUgdHJlYXRtZW50CiAgICAtIFRoZSByZWxhdGl2ZSBhYnVuZGFuY2Ugb2YgKlN0YXBoeWxvY29jY3VzIHNwcC4qIG9uICpDb3J5bmViYWN0ZXJpdW0gc3BwLiogKyAqU3RhcGh5bG9jb2NjdXMgc3BwLiogaW4gdGhlIG1pY3JvYmlvbWUgaXMgbWVhc3VyZWQgdmlhIERHR0UgKCpEZW5hdHVyaW5nIEdyYWRpZW50IEdlbCBFbGVjdHJvcGhvcmVzaXMqKS4KCi0tLQoKIyMgSW1wb3J0IHRoZSBkYXRhCgpgYGB7cn0KYXA8LXJlYWRfY3N2KCJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vR1RQQi9QU0xTMjAvbWFzdGVyL2RhdGEvYXJtcGl0LmNzdiIpCmFwCmBgYAoKIyMgRGF0YSBleHBsb3JhdGlvbgoKV2UgcGxvdCB0aGUgZGlyZWN0IHJlbGF0aXZlIGFidW5kYW5jZXMgaW4gZnVuY3Rpb24gb2YgdGhlIHRyZWF0bWVudCBncm91cC4gV2l0aCB0aGUgZ2dwbG90MiBsaWJyYXJ5IHdlIGNhbiBlYXNpbHkgYnVpbGQgcGxvdHMgYnkgYWRkaW5nIGxheWVycy4KCmBgYHtyfQphcCAlPiUgIGdncGxvdChhZXMoeD10cnQseT1yZWwpKSArIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArIGdlb21fcG9pbnQocG9zaXRpb249ImppdHRlciIpCgphcCAlPiUgZ2dwbG90KGFlcyhzYW1wbGU9cmVsKSkgKyAKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpICsKICBmYWNldF93cmFwKH50cnQpCmBgYAoKLS0tCgojIFR3byBzYW1wbGUgVC10ZXN0CiMjIE5vdGF0aW9uCgpTdXBwb3NlIHRoYXQgJFlfe2lqfSQgaXMgdGhlIHJlc3BvbnNlIGZvciBzdWJqZWN0cyAkaT0xLFxsZG90cywgbl9qJCBmcm9tIHBvcHVsYXRpb24gJGo9MSwyJC4KClVzZSBvZiB0aGUgdGVybSAqKnRyZWF0bWVudCoqIG9yICoqZ3JvdXAqKiBpbnN0ZWFkIG9mIHBvcHVsYXRpb24KCkhlcmUgdGhlIHRyZWF0bWVudCBpcyAkaj0xJCBtaWNyb2JpYWwgdHJhbnNwbGFudCB2cyAkaj0yJCBwbGFjZWJvLiAKCldlIGFzc3VtZSAKCiQkWV97aWp9XHRleHR7IGkuaS5kLiB9IE4oXG11X2osXHNpZ21hXjIpXDtcO1w7aT0xLFxsZG90cyxuX2lcO2o9MSwyLiQkCgpOb3RlLCB0aGF0IHdlIGFzc3VtZSBlcXVhbCB2YXJpYW5jZXMgKipob21vc2NlZGFzdGljKioKCihVbmVxdWFsIHZhcmlhbmNlcyBhcmUgcmVmZXJyZWQgdG8gYXMgICoqaGV0ZXJvc2NlZGFzdGljKiopCgotLS0KCiMjIEh5cG90aGVzZXMKClRlc3QgJCQgSF8wOiBcbXVfMSA9IFxtdV8yICQkCmFnYWluc3QKJCQgIEhfMTogXG11XzEgXG5lcSBcbXVfMiAuJCQKCiRIXzEkIGlzIGFnYWluIHRoZSByZXNlYXJjaCBoeXBvdGhlc2lzOiB0aGUgYXZlcmFnZSByZWxhdGl2ZSBhYnVuZGFuY2Ugb2YgKlN0YXBoeWxvY29jY3VzIHNwcC4qIGlzIGRpZmZlcmVudCB1cG9uIG1pY3JvYmlhbCB0cmFuc3BsYW50IHRoZW4gdXBvbiBwbGFjZWJvIHRyZWF0bWVudC4gIAoKJEhfMCQgYW5kICRIXzEkIGNhbiBhbHNvIGJlIHNwZWNpZmllZCBpbiB0ZXJtcyBvZiB0aGUgZWZmZWN0IHNpemUgYmV0d2VlbiB0aGUgdHdvIHRyZWF0bWVudHMsICRcbXVfMS1cbXVfMiQKJCRIXzA6IFxtdV8xLVxtdV8yID0gMCwkJAokJEhfMTogXG11XzEtXG11XzIgXG5lcSAwLiQkCgpXZSBjYW4gZXN0aW1hdGUgdGhlIGVmZmVjdCBzaXplIHVzaW5nIHRoZSBkaWZmZXJlbmNlIGluIHNhbXBsZSBtZWFuczoKJCRcaGF0IFxtdV8xLVxoYXQgXG11XzI9XGJhciBZXzEgLVxiYXIgWV8yLiQkCgotLS0KCiMjIFZhcmlhbmNlIGVzdGltYXRvciAKClRoZSBleHBlcmltZW50YWwgdW5pdHMgYXJlIGluZGVwZW5kZW50IHNvIHRoZSBzYW1wbGUgbWVhbnMgYXJlIGFsc28gaW5kZXBlbmRlbnQgYW5kIHRoZSB2YXJpYW5jZSBvbiB0aGUgZGlmZmVyZW5jZSBpcyAgCiQkXHRleHR7VmFyfV97XGJhciBZXzEgLVxiYXIgWV8yfT1cZnJhY3tcc2lnbWFeMn17bl8xfStcZnJhY3tcc2lnbWFeMn17bl8yfT1cc2lnbWFeMiBcbGVmdChcZnJhY3sxfXtuXzF9K1xmcmFjezF9e25fMn1ccmlnaHQpLiQkCgpBbmQgdGhlIHN0YW5kYXJkIGVycm9yIGJlY29tZXMKJCRcc2lnbWFfe1xiYXIgWV8xIC1cYmFyIFlfMn09XHNpZ21hXHNxcnR7XGZyYWN7MX17bl8xfStcZnJhY3sxfXtuXzJ9fS4kJAoKVGhlIHZhcmlhbmNlIGNhbiBiZSBlc3RpbWF0ZWQgd2l0aGluIGVhY2ggZ3JvdXAgdXNpbmcgdGhlIHNhbXBsZSB2YXJpYW5jZToKCiQkU18xXjIgPSBcZnJhY3sxfXtuXzEtMX1cc3VtX3tpPTF9XntuXzF9IChZX3tpMX0tXGJhcntZfV8xKV4yLiQkCgokJFNfMl4yID0gXGZyYWN7MX17bl8yLTF9XHN1bV97aT0xfV57bl8yfSAoWV97aTJ9LVxiYXJ7WX1fMileMi4kJAoKQnV0LCBpZiB3ZSBhc3N1bWUgZXF1YWwgdmFyaWFuY2VzICRcc2lnbWFfMV4yPVxzaWdtYV8yXjI9XHNpZ21hXjIkIHRoYW4gd2UgY2FuIGVzdGltYXRlIHRoZSB2YXJpYW5jZSBtb3JlIHByZWNpc2UgYnkgdXNpbmcgYWxsIG9ic2VydmF0aW9ucyBpbiBib3RoIGdyb3Vwcy4gVGhpcyB2YXJpYW5jZSBlc3RpbWF0b3IgaXMgYWxzbyByZWZlcnJlZCB0byBhcyB0aGUgKnBvb2xlZCB2YXJpYW5jZSBlc3RpbWF0b3I6ICRTXjJfcCQqLgoKU28gJFNfMV4yJCBlbiAkU18yXjIkIGFyZSBlc3RpbWF0b3JzIG9mIHRoZSBzYW1lIHBhcmFtZXRlciAkXHNpZ21hXjIkLgoKQW5kIHdlIGNhbiBjb21iaW5lIHRoZW0gaW50byBvbmUgZXN0aW1hdG9yIGJhc2VkIG9uIGFsbCAkbl8xK25fMiQgb2JzZXJ2YXRpb25zOgoKJCQgIFNfcF4yID0gXGZyYWN7bl8xLTF9e25fMStuXzItMn0gU18xXjIgKyBcZnJhY3tuXzItMX17bl8xK25fMi0yfSBTXzJeMiA9IFxmcmFjezF9e25fMStuXzItMn1cc3VtX3tqPTF9XjJcc3VtX3tpPTF9XntuX2p9IChZX3tpan0gLSBcYmFye1l9X2opXjIuJCQKCiQkIFNfcF4yPSBcc3VtXGxpbWl0c197aj0xfV4yXHN1bVxsaW1pdHNfe2k9MX1ee25fan0gXGZyYWN7KFlfe2lqfS1cYmFye1l9X3suan0pXjJ9e25fMStuXzItMn0kJAoKClRoZSBwb29sZWQgdmFyaWFuY2UgZXN0aW1hdG9yIHVzZXMgdGhlIHNxdWFyZWQgZGV2aWF0aW9ucyBvZiB0aGUgb2JzZXJ2YXRpb25zIGZyb20gdGhlaXIgZ3JvdXAgbWVhbiBhbmQgaGFzICRuXzErbl8yLTIkIGRlZ3JlZXMgb2YgZnJlZWRvbS4gCgotLS0KCiMjIFRlc3Qgc3RhdGlzdGljIApUd28tc2FtcGxlICR0JC10ZXN0c3RhdGlzdGllazoKCiQkVCA9IFxmcmFje1xiYXJ7WX1fMS1cYmFye1l9XzJ9e1xzcXJ0e1xmcmFje1NfcF4yfXtuXzF9K1xmcmFje1NfcF4yfXtuXzJ9fX0gPQogIFxmcmFje1xiYXJ7WX1fMSAtIFxiYXJ7WX1fMn17U19wXHNxcnR7XGZyYWN7MX17bl8xfStcZnJhY3sxfXtuXzJ9fX0uJCQKClRoZSBzdGF0aXN0aWMgVCBmb2xsb3dzIGEgdC1kaXN0cmlidXRpb24gd2l0aCAkbl8xK25fMi0yJCB1bmRlciAkSF8wJCBpcyBhbGwgZGF0YSBhcmUgaW5kZXBlbmRlbnQsIG5vcm1hbGx5IGRpc3RyaWJ1dGVkIGFuZCBoYXZlIGVxdWFsIHZhcmlhbmNlcy4KCi0tLQoKIyMgQXJtcGl0IGV4YW1wbGUKCldlIGNhbiBpbXBsZW1lbnQgdGhlIHRlc3QgaW4gUjoKCmBgYHtyfQp0LnRlc3QocmVsfnRydCxkYXRhPWFwLHZhci5lcXVhbD1UUlVFKQpgYGAKCk9uIHRoZSAkNVwlJCBzaWduaWZpY2FuY2UgbGV2ZWwgd2UgcmVqZWN0IHRoZSBudWxsIGh5cG90aGVzaXMgaW4gZmF2b3Igb2YgdGhlIGFsdGVybmF0aXZlIGh5cG90aGVzaXMgYW5kIGNvbmNsdWRlIHRoYXQgdGhlIHJlbGF0aXZlIGFidW5kYW5jZSBvZiAgKlN0YXBoeWxvY29jY3VzIHNwcC4qIGlzIG9uIGF2ZXJhZ2UgZXh0cmVtZSBzaWduaWZpY2FudCBsYXJnZXIgaXMgaW4gdHJhbnNwbGFudGF0aW9uIGdyb3VwIHRoYW4gaW4gdGhlIHBsYWNlYm8gZ3JvdXAuICAgCgpJZiB0aGVyZSBpcyBubyBlZmZlY3Qgb2YgdGhlIHRyYW5zcGxhbnQgd2UgaGF2ZSBhIHByb2JhYmlsaXR5IG9mIGxlc3MgdGhlbiA5IGluICQxMDAwMDAkIHRvIG9ic2VydmUgYSB0ZXN0IHN0YXRpc3RpYyBpbiBhIHJhbmRvbSBzYW1wbGUgdGhhdCBpcyBhdCBsZWFzdCBhcyBleHRyZW1lIGFzIHdoYXQgd2Ugb2JzZXJ2ZWQgaW4gdGhlIGFybXBpdCBleHBlcmltZW50LiAKClRoaXMgaXMgZXh0cmVtZWx5IHJhcmUgdW5kZXIgJEhfMCQuIAoKSWYgJEhfMSQgaXMgY29ycmVjdCwgd2UgZXhwZWN0IHRoYXQgdGhlIHRlc3Qgc3RhdGlzdGljIGlzIGxhcmdlciBpbiBhYnNvbHV0ZSB2YWx1ZSBhbmQgZXhwZWN0IHNtYWxsIHAtdmFsdWVzLiBIZW5jZSB3ZSBkZWNpZGUgdGhhdCB0aGVyZSBpcyBhIGxvdCBvZiBldmlkZW5jZSBhZ2FpbnN0ICRIXzAkIGluIGZhdm91ciBvZiAkSF8xJC4gIAoKKipHb29kIHN0YXRpc3RpY2FsIHByYWN0aWNlKiogaXMgdG8gcmVwb3J0IHRoZSAkcCQtdmFsdWUsIGJ1dCBhbHNvICBlZmZlY3Qgc2l6ZSBhbG9uZyB3aXRoIGl0cyBjb25maWRlbmNlIGludGVydmFsLiBTbyB0aGF0IHdlIGNhbiBqdWRnZSB0aGUgc3RhdGlzdGljYWwgc2lnbmlmaWNhbmNlIGFuZCB0aGUgYmlvbG9naWNhbCByZWxldmFuY2UuCgojIyMgQ29uY2x1c2lvbgoKT24gYXZlcmFnZSB0aGUgcmVsYXRpdmUgYWJ1bmRhbmNlIG9mICpTdGFwaHlsb2NvY2N1cyBzcHAuKiBpbiB0aGUgbWljcm9iaW9tZSBvZiB0aGUgYXJtcGl0IGluIHRoZSB0cmFuc3BsYW50IGdyb3VwIGlzIGV4dHJlbWVseSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBmcm9tIHRoYXQgaW4gdGhlIHBsYWNlYm8gZ3JvdXAgKCRwPDwwLjAwMSQpLiBUaGUgcmVsYXRpdmUgYWJ1bmRhbmNlIG9mICpTdGFwaHlsb2NvY2N1cyBzcHAuKiBpcyBvbiBhdmVyYWdlIGByIHJvdW5kKGRpZmYodC50ZXN0KHJlbH50cnQsZGF0YT1hcCx2YXIuZXF1YWw9VFJVRSkkZXN0aW1hdGUpLDEpYCUgbGFyZ2VyIGluIHRoZSB0cmFuc3BsYW50IGdyb3VwIHRoYW4gaW4gdGhlIHBsYWNlYm8gZ3JvdXAgKDk1XCUgQ0kgW2ByIHBhc3RlKGZvcm1hdCgtdC50ZXN0KHJlbH50cnQsZGF0YT1hcCx2YXIuZXF1YWw9VFJVRSkkY29uZi5pbnRbMjoxXSxkaWdpdHM9Mixuc21hbGw9MSksY29sbGFwc2U9IiwiKWBdJSkuCgotLS0KCiMgQXNzdW1wdGlvbnMKClZhbGlkaXR5IG9mIHQtdGVzdCBkZXBlbmRzIG9uIGRpc3RyaWJ1dGlvbmFsIGFzc3VtcHRpb25zOgoKLSBJbmRlcGVuZGVuY2UgKGRlc2lnbikKLSBPbmUtc2FtcGxlIHQtdGVzdDogbm9ybWFsaXR5IG9mIHRoZSBvYnNlcnZhdGlvbnMgCi0gUGFpcmVkIHQtdGVzdDogbm9ybWFsaXR5IG9mIHRoZSBkaWZmZXJlbmNlICAKLSBUd28tc2FtcGxlIHQtdGVzdDogTm9ybWFsaXR5IG9mIHRoZSBvYnNlcnZhdGlvbnMgaW4gYm90aCBncm91cHMsIGFuZCBlcXVhbCB2YXJpYW5jZXMuCgpJZiB0aGUgYXNzdW1wdGlvbnMgYXJlIG5vdCBtZXQsIHRoZSBudWxsIGRpc3RyaWJ1dGlvbiBkb2VzIG5vdCBmb2xsb3cgYSB0LWRpc3RyaWJ1dGlvbiwgYW5kLCB0aGUgcC12YWx1ZXMgYW5kIGNyaXRpY2FsIHZhbHVlcyBhcmUgaW5jb3JyZWN0LiAgCgpUbyBjb25zdHJ1Y3QgY29uZmlkZW5jZSBpbnRlcnZhbHMgd2UgYWxzbyByZWx5IG9uIHRoZXNlIGFzc3VtcHRpb25zLgoKLSBXZSB1c2VkIHF1YW50aWxlcyBmcm9tIHRoZSB0LWRpc3RyaWJ1dGlvbiB0byBjYWxjdWxhdGUgdGhlIGxvd2VyIGFuZCB1cHBlciBsaW1pdC4gCgotIFRoZSBjb3JyZWN0IGNvdmVyYWdlIG9mIHRoZSBDSSBkZXBlbmRzIG9uIHRoZXNlIGFzc3VtcHRpb25zIAoKLS0tCgojIyBFdmFsdWF0ZSBub3JtYWxpdHkgCiAKIC0gQm94cGxvdHMgYW5kIGhpc3RvZ3JhbXM6IHNoYXBlIG9mIGRpc3RyaWJ1dGlvbiBhbmQgb3V0bGllcnMKIAogLSBRUS1wbG90cwoKVGhlcmUgYWxzbyBleGlzdCBoeXBvdGhlc2lzIHRlc3RzIChnb29kbmVzcy1vZi1maXQgdGVzdCksIGJ1dCB0aGVpciBudWxsIGh5cG90aGVzaXMgaXMgdGhhdCB0aGUgZGF0YSBhcmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgc28gd2UgbWFrZSBhIHdlYWsgY29uY2x1c2lvbiEgIAoKICAtIEtvbG1vZ29yb3YtU21pcm5vdiwgU2hhcGlyby1XaWxrIGVuIEFuZGVyc29uLURhcmxpbmcuCiAgLSBJbiBzbWFsbCBzYW1wbGVzIHRoZXkgaGF2ZSBhIGxvdyBwb3dlcgogIC0gSW4gbGFyZ2Ugc2FtcGxlcyB0aGV5IG9mdGVuIGZsYWcgdmVyeSBzbWFsbCBkZXZpYXRpb25zIGFzIHNpZ25pZmljYW50CgpSZWNvbW1lbmRhdGlvbiAKCi0gU3RhcnQgd2l0aCBncmFwaGljYWwgZXhwbG9yYXRpb24gb2YgdGhlIGRhdGEgYW5kIGtlZXAgdGhlIHNhbXBsZSBzaXplIGluIG1pbmQgdG8gYXZvaWQgb3ZlcmludGVycHJldGF0aW9uIG9mIHRoZSBwbG90cy4gCgotIElmIHlvdSBoYXZlIGRvdWJ0cywgdXNlIHNpbXVsYXRpb24gd2hlcmUgeW91IHNpbXVsYXRlIGRhdGEgd2l0aCB0aGUgc2FtZSBzYW1wbGUgc2l6ZSBmcm9tIGEgTm9ybWFsIGRpc3RyaWJ1dGlvbiB3aXRoIHRoZSBzYW1lIG1lYW4gYW5kIHZhcmlhbmNlIGFzIHRoZSBvbmUgdGhhdCB5b3Ugb2JzZXJ2ZWQgaW4gdGhlIHNhbXBsZSAKCi0gSWYgeW91IG9ic2VydmVkIGRldmlhdGlvbnMgb2Ygbm9ybWFsaXR5IGNoZWNrIGluIHRoZSBsaXRlcmF0dXJlIGhvdyBzZW5zaXRpdmUgeW91ciBtZXRob2QgaXMgc3VjaCBkZXZpYXRpb25zIG9mIG5vcm1hbGl0eS4gKGUuZy4gVC10ZXN0cyBmb3IgaW5zdGFuY2UgYXJlIHJhdGhlciBpbnNlbnNpdGl2ZSB0byBkZXZpYXRpb25zIGFzIGxvbmcgYXMgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgZGF0YSBpcyBzeW1tZXRyaWMuKQoKLSBJbiBsYXJnZSBzYW1wbGVzIHlvdSBjYW4gcmVzb3J0IHRvIHRoZSBjZW50cmFsIGxpbWl0IHRoZW9yZW0uIAoKLSBZb3UgbWlnaHQgcmVzb3J0IHRvIHRyYW5zZm9ybWF0aW9ucyBvZiB0aGUgcmVzcG9uc2UuIAoKLS0tIAoKIyMgSG9tb3NjZWRhc3RpY2l0eQoKLSBCb3hwbG90czogVGhlIGJveCBzaXplIGlzIHRoZSBpbnRlciBxdWFydGlsZSByYW5nZSAoSVFSKSBhIHJvYnVzdCBlc3RpbWF0b3Igb2YgdGhlIHZhcmlhbmNlLiAKCi0gSWYgdGhlIGRpZmZlcmVuY2VzIGFyZSBub3QgbGFyZ2UgJFxyaWdodGFycm93JCBob21vc2NlZGFzdGljaXRlaXQgCgotIEFnYWluIHlvdSBjYW4gdXNlIHNpbXVsYXRpb24gdG8gZ2V0IGluc2lnaHQgaW4gdGhlIGRpZmZlcmVuY2VzIHlvdSBjYW4gZXhwZWN0LiAKCi0gRm9ybWFsIEYtdGVzdCBjYW4gYmUgdXNlZCB0byBjb21wYXJlIHRoZSB2YXJpYW5jZXMsIGJ1dCBhZ2FpbiB1bmRlciB0aGUgbnVsbCB5b3UgYXNzdW1lIGVxdWFsIHZhcmlhbmNlcywgc28gdGhlIHNhbWUgY3JpdGljaXNtIGFzIGZvciBub3JtYWxpdHkgdGVzdHMgYXBwbGllcyBoZXJlLiAKCi0tLQoKIyMgV2VsY2ggbW9kaWZpZWQgdC10ZXN0CgpJZiB0aGUgZGF0YSBhcmUgaGV0ZXJvc2NlZGFzdGljLCB5b3UgY2FuIHVzZSBhIFdlbGNoIHR3by1zYW1wbGUgVC10ZXN0LCB3aGljaCBubyBsb25nZXIgdXNlcyB0aGUgcG9vbGVkIHZhcmlhbmNlIGVzdGltYXRvci4gCgokJFQgPSAgXGZyYWN7XGJhcntZfV8xIC0gXGJhcntZfV8yfXtcc3FydHtcZnJhY3tTXjJfMX17bl8xfStcZnJhY3tTXjJfMn17bl8yfX19JCQKd2l0aCAkU14yXzEkIGVuICRTXjJfMiQgdGhlIHNhbXBsZSB2YXJpYW5jZXMgaW4gYm90aCBncm91cHMuICAKClRoaXMgc3RhdGlzdGljIGZvbGxvd3MgYXBwcm94aW1hdGVseSBhIHQtZGlzdHJpYnV0aW9uIHdpdGggYSBudW1iZXIgb2YgZGVncmVlcyBvZiBmcmVlZG9tIGJldHdlZW4gICRcdGV4dHttaW59KG5fMS0xLG5fMi0xKSQgYW5kICRuXzErbl8yLTIkLgoKSW4gUiB0aGUgZGVncmVlcyBvZiBmcmVlZG9tIGFyZSBlc3RpbWF0ZWQgdXNpbmcgdGhlIFdlbGNoLSBTYXR0ZXJ0aHdhaXRlIGFwcHJveGltYXRpb24uIFlvdSBjYW4gZG8gdGhpcyBieSB1c2luZyB0aGUgYHQudGVzdGAgZnVuY3Rpb24gd2l0aCBhcmd1bWVudCBgdmFyLmVxdWFsPUZBTFNFYC4KCmBgYHtyfQp0LnRlc3QocmVsfnRydCxkYXRhPWFwLHZhci5lcXVhbD1GQUxTRSkKYGBgCgpOb3RlIHRoYXQgeW91IGNhbiBzZWUgdGhhdCB0aGUgV2VsY2ggVC10ZXN0IGlzIGFkb3B0ZWQgaW4gdGhlIHRpdGxlLiBUaGUgYWRqdXN0ZWQgZGVncmVlcyBvZiBmcmVlZG9tIGFyZSAkZGYgPSAxNy44NzYkICRccG0kIHRvIHRoYXQgb2YgdGhlIGNvbnZlbnRpb25hbCBULXRlc3QsIGJlY2F1c2UgdGhlIHZhcmlhbmNlcyBhcmUgYXBwcm94aW1hdGVseSBlcXVhbC4gCgotLS0KCiMgSG93IHRvIHJlcG9ydD8gCgotIEluIHRoZSBzY2llbnRpZmljIGxpdGVyYXR1cmUgdGhlcmUgaXMgdG9vIG11Y2ggYXR0ZW50aW9uIGZvciBwLXZhbHVlcwoKLSBJdCBpcyBtdWNoIG1vcmUgaW5mb3JtYXRpdmUgdG8gY29tYmluZSBhbiBlc3RpbWF0ZSB3aXRoIGl0cyBjb25maWRlbmNlIGludGVydmFsLiAKCioqUnVsZSBvZiB0aHVtYioqOgoKUmVwb3J0IGFuIGVzdGltYXRlIHRvZ2V0aGVyIHdpdGggaXRzICBjb25maWRlbmNlIGludGVydmFsIChhbmQgaXRzIHAtdmFsdWUpIAoKMS4gVGhlIHJlc3VsdCBvZiB0aGUgdGVzdCBjYW4gYmUgZGVyaXZlZCBvZiB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbCAKMi4gSXQgYWxsb3dzIHRoZSByZWFkZXIgdG8ganVkZ2UgKipzY2llbnRpZmljIHJlbGV2YW5jZSoqLgoKYGBge3J9CnQudGVzdChyZWx+dHJ0LGRhdGE9YXApCmBgYAoKVGhlIHJlc3VsdCBvZiBhbiAkXGFscGhhJC1sZXZlbCB0LXRlc3QgaXMgZXF1aXZhbGVudCB3aXRoIGNvbXBhcmluZyB0aGUgZWZmZWN0IHNpemUgdW5kZXIgJEhfMCQgd2l0aCB0aGUgJDEtXGFscGhhJCBDSS4gCgpBbiBlZmZlY3QgY2FuIGJlIGV4dHJlbWVseSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50LCBidXQgc2NpZW50aWZpY2FsbHkgaXJyZWxldmFudC4gV2l0aCBhIENJIHlvdSB3aWxsIHNwb3QgdGhpcy4gCgoKLS0tCgojIFtIb21lXShodHRwczovL2d0cGIuZ2l0aHViLmlvL1BTTFMyMC8pIHstfQo=