To experiment more with the concepts of ANOVA, we will perform a short analysis on the NHANES dataset.
The NHANES dataset
The National Health and Nutrition Examination Survey (NHANES) contains data that has been collected since 1960. For this tutorial, we will make use of the data that were collected between 2009 and 2012, for 10.000 U.S. civilians. The dataset contains a large number of physical, demographic, nutritional and life-style-related parameters.
Goal
In the NHANES dataset, one of the columns is named HealthGen
. HealthGen is a self-reported rating of a participant’s health in general terms. HealthGen is reported for participants aged 12 years or older. It is a factor with the following levels: Excellent, Vgood, Good, Fair, or Poor.
We want to test whether or not the mean systolic blood pressure value (take column BPSys1
) is equal between the five self-reported health categories. To this end, we will use an ANOVA analysis (if the required assumptions are met).
Load the required libraries
Data import
NHANES <- read_csv("https://raw.githubusercontent.com/GTPB/PSLS20/master/data/NHANES.csv")
glimpse(NHANES[1:10])
## Observations: 10,000
## Variables: 10
## $ ID <dbl> 51624, 51624, 51624, 51625, 51630, 51638, 51646, 51647,…
## $ SurveyYr <chr> "2009_10", "2009_10", "2009_10", "2009_10", "2009_10", …
## $ Gender <chr> "male", "male", "male", "male", "female", "male", "male…
## $ Age <dbl> 34, 34, 34, 4, 49, 9, 8, 45, 45, 45, 66, 58, 54, 10, 58…
## $ AgeDecade <chr> "30-39", "30-39", "30-39", "0-9", "40-49", "0-9", "0-9"…
## $ AgeMonths <dbl> 409, 409, 409, 49, 596, 115, 101, 541, 541, 541, 795, 7…
## $ Race1 <chr> "White", "White", "White", "Other", "White", "White", "…
## $ Race3 <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
## $ Education <chr> "High School", "High School", "High School", NA, "Some …
## $ MaritalStatus <chr> "Married", "Married", "Married", NA, "LivePartner", NA,…
Data Exploration
NHANES %>%
ggplot(aes(x=HealthGen,y=BPSys1,fill=HealthGen)) +
scale_fill_brewer(palette="RdGy") +
theme_bw() +
geom_boxplot(outlier.shape=NA) +
#geom_jitter(width = 0.2,size=0.01) + ## omitted as it makes the plot messy
ggtitle("Boxplot of the systolic bloodpressure for each health category") +
ylab("Systolic blood pressure (mmHg)") +
stat_summary(fun.y=mean, geom="point", shape=5, size=3, color="black", fill="black")
## Warning: Removed 1763 rows containing non-finite values (stat_boxplot).
## Warning: Removed 1763 rows containing non-finite values (stat_summary).
This plot is not ideal; it would be far more intuitive if the health categories were ordered properly (i.e., Poor –> excellent). In addition, we observe a sixth “category” of NA values.
To improve the plot you should:
- Filter out subjects with NA values for HealthGen or BPSys1
- Set HealthGen to a factor and relevel it to Poor –> Excellent
The second task can be achieved by using the mutate
, as.factor
and fct_relevel
functions.
NHANES <- NHANES %>%
filter(!is.na(HealthGen),!is.na(BPSys1)) %>%
mutate(HealthGen = as.factor(HealthGen)) %>%
mutate(HealthGen =fct_relevel(HealthGen,c("Poor","Fair","Good","Vgood","Excellent")))
NHANES %>%
ggplot(aes(x=HealthGen,y=BPSys1,fill=HealthGen)) +
scale_fill_brewer(palette="RdGy") +
theme_bw() +
geom_boxplot(outlier.shape=NA) +
ggtitle("Boxplot of the systolic bloodpressure for each health category") +
ylab("Systolic blood pressure (mmHg)") +
stat_summary(fun.y=mean, geom="point", shape=5, size=3, color="black", fill="black")
Check the assumptions for ANOVA
To study whether or not the observed difference between the average systolic blood pressure values of the different health groups are significant, we may perform an ANOVA.
The null hypothesis of ANOVA states that: \(H0\): The mean systolic blood pressure is equal between the different health groups.
The alternative hypothesis of ANOVA states that: \(HA\): The mean systolic blood pressure for at least one health group is different from the mean systolic blood pressure in at least one other health group.
Before we may proceed with the analysis, we must make sure that all assumptions for ANOVA are met. ANOVA has three assumptions:
- The observations are independent of each other (in all groups)
- The data (BPSys1) must be normally distributed (in all groups)
- The variability within all groups is similar
Assumption of independence
The first assumption is met; there shoud be no specific pattterns of dependence.
Assumption of normality
For the second assumption, we must check normality in each group.
NHANES %>%
ggplot(aes(sample=BPSys1)) +
geom_qq() +
geom_qq_line() +
facet_grid(~HealthGen)
The data does not appear to be normally distributed for each group. It seems to have a heavy right tail. We can perform a log transformation on the data.
NHANES %>%
mutate(BPSys1_log = log(BPSys1)) %>%
ggplot(aes(sample=BPSys1_log)) +
geom_qq() +
geom_qq_line() +
facet_grid(~HealthGen)
While the log transformation improved the distributions somewaht, the data still does not appear to be normally distributed for each group. However, we do have a very large number of observations per group:
## Count the number of observations per treatment
NHANES %>%
count(HealthGen)
## # A tibble: 5 x 2
## HealthGen n
## <fct> <int>
## 1 Poor 184
## 2 Fair 934
## 3 Good 2803
## 4 Vgood 2367
## 5 Excellent 848
As such, we may rely on the cental limit theorem. Remember, the cental limit theorem that when the number of observations is sufficiently large (i.e. >100), we will assume that the distribution of the sample mean will approximate a normal distribution, even if the underlying data is not normally distributed.
Analysis
ANOVA
fit <- lm(log(BPSys1)~HealthGen, NHANES)
fit_anova <- anova(fit)
fit_anova
## Analysis of Variance Table
##
## Response: log(BPSys1)
## Df Sum Sq Mean Sq F value Pr(>F)
## HealthGen 4 1.908 0.4771 25.788 < 2.2e-16 ***
## Residuals 7131 131.931 0.0185
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
print(paste("Not-rounded p-value:", fit_anova$`Pr(>F)`[1]))
## [1] "Not-rounded p-value: 2.99356182478463e-21"
The p-value of the ANOVA analysis is extremely significant (p-value = 2.994e-21), so we reject the null hypothesis that the mean egg length is equal between the different bird types. We can say that the mean egg length is significantly different between at least two bird types on the 5% significance level.
Based on this analysis, we do not yet know between which particular bird types there is a significant difference. To study this, we will perfrom the Tuckey post-hoc analysis.
Post-hoc analysis
library(multcomp, quietly = TRUE)
mcp <- glht(fit, linfct = mcp(HealthGen = "Tukey"))
summary(mcp)
##
## Simultaneous Tests for General Linear Hypotheses
##
## Multiple Comparisons of Means: Tukey Contrasts
##
##
## Fit: lm(formula = log(BPSys1) ~ HealthGen, data = NHANES)
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>|t|)
## Fair - Poor == 0 -0.015828 0.010971 -1.443 0.57595
## Good - Poor == 0 -0.036762 0.010351 -3.551 0.00299 **
## Vgood - Poor == 0 -0.059415 0.010410 -5.708 < 0.001 ***
## Excellent - Poor == 0 -0.055476 0.011062 -5.015 < 0.001 ***
## Good - Fair == 0 -0.020934 0.005139 -4.074 < 0.001 ***
## Vgood - Fair == 0 -0.043587 0.005256 -8.293 < 0.001 ***
## Excellent - Fair == 0 -0.039648 0.006452 -6.145 < 0.001 ***
## Vgood - Good == 0 -0.022653 0.003797 -5.966 < 0.001 ***
## Excellent - Good == 0 -0.018714 0.005331 -3.511 0.00355 **
## Excellent - Vgood == 0 0.003938 0.005444 0.723 0.94608
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)
##
## Simultaneous Confidence Intervals
##
## Multiple Comparisons of Means: Tukey Contrasts
##
##
## Fit: lm(formula = log(BPSys1) ~ HealthGen, data = NHANES)
##
## Quantile = 2.684
## 95% family-wise confidence level
##
##
## Linear Hypotheses:
## Estimate lwr upr
## Fair - Poor == 0 -0.015828 -0.045274 0.013617
## Good - Poor == 0 -0.036762 -0.064545 -0.008979
## Vgood - Poor == 0 -0.059415 -0.087355 -0.031475
## Excellent - Poor == 0 -0.055476 -0.085167 -0.025786
## Good - Fair == 0 -0.020934 -0.034727 -0.007141
## Vgood - Fair == 0 -0.043587 -0.057694 -0.029480
## Excellent - Fair == 0 -0.039648 -0.056965 -0.022332
## Vgood - Good == 0 -0.022653 -0.032844 -0.012462
## Excellent - Good == 0 -0.018714 -0.033022 -0.004407
## Excellent - Vgood == 0 0.003938 -0.010672 0.018549
Conclusion
We have found an extremely significant dependence (p-value = 2.994e-21), between the mean systolic blood pressure and the health group on the global 5% significance level.
The mean logarithm of systolic blood pressure in the self-reported health category Poor
is significantly higher as compared three other groups:
- the
Good
group (adjusted p-value = < 0.001, mean difference = -0.036762 mmHg, 95% CI [-0.034725; -0.007142])
- the
Vgood
group (adjusted p-value = < 0.001, mean difference = -0.059415 mmHg, 95% CI [-0.087352; -0.031477])
- the
Excellent
group (adjusted p-value = < 0.001, mean difference = -0.059415 mmHg, 95% CI [-0.085164; -0.025789])
The mean logarithm of systolic blood pressure in the self-reported health category Fair
is significantly higher as compared three other groups:
- the
Good
group (adjusted p-value = 0.00317, mean difference = -0.020934 mmHg, 95% CI [-0.064542; -0.008982])
- the
Vgood
group (adjusted p-value = < 0.001, mean difference = -0.043587 mmHg, 95% CI [-0.057692; -0.029481])
- the
Excellent
group (adjusted p-value = < 0.001, mean difference = -0.039648 mmHg, 95% CI [-0.056963; -0.022333])
The mean logarithm of systolic blood pressure in the self-reported health category Good
is significantly higher as compared two other groups:
- the
Vgood
group (adjusted p-value = < 0.001, mean difference = -0.022653 mmHg, 95% CI [-0.032843; -0.012463])
- the
Excellent
group (adjusted p-value = 0.00362, mean difference = -0.018714 mmHg, 95% CI [-0.033021; -0.004408])
We do not find enough evidence to claim a difference in systolic blood pressure levels between the other groups.
Note that in order to interpret the outcomes on the original scale, we should backtransform the outcomes with the exp()
functions (interpretation on the geometric mean).
LS0tCnRpdGxlOiAiVHV0b3JpYWwgNy4yOiBBTk9WQSBpbiB0aGUgTkhBTkVTIGRhdGFzZXQiICAgCm91dHB1dDoKICAgIGh0bWxfZG9jdW1lbnQ6CiAgICAgIGNvZGVfZG93bmxvYWQ6IHRydWUgICAgCiAgICAgIHRoZW1lOiBjb3NtbwogICAgICB0b2M6IHRydWUKICAgICAgdG9jX2Zsb2F0OiB0cnVlCiAgICAgIGhpZ2hsaWdodDogdGFuZ28KICAgICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCi0tLQoKVG8gZXhwZXJpbWVudCBtb3JlIHdpdGggdGhlIGNvbmNlcHRzIG9mIEFOT1ZBLCB3ZSB3aWxsIHBlcmZvcm0gYQpzaG9ydCBhbmFseXNpcyBvbiB0aGUgTkhBTkVTIGRhdGFzZXQuCgojIFRoZSBOSEFORVMgZGF0YXNldAoKVGhlIE5hdGlvbmFsIEhlYWx0aCBhbmQgTnV0cml0aW9uIEV4YW1pbmF0aW9uIFN1cnZleSAoTkhBTkVTKQpjb250YWlucyBkYXRhIHRoYXQgaGFzIGJlZW4gY29sbGVjdGVkIHNpbmNlIDE5NjAuIEZvciB0aGlzIHR1dG9yaWFsLAp3ZSB3aWxsIG1ha2UgdXNlIG9mIHRoZSBkYXRhIHRoYXQgd2VyZSBjb2xsZWN0ZWQgYmV0d2VlbiAyMDA5IGFuZCAKMjAxMiwgZm9yIDEwLjAwMCBVLlMuIGNpdmlsaWFucy4gVGhlIGRhdGFzZXQgY29udGFpbnMgYSBsYXJnZSBudW1iZXIgb2YKcGh5c2ljYWwsIGRlbW9ncmFwaGljLCBudXRyaXRpb25hbCBhbmQgbGlmZS1zdHlsZS1yZWxhdGVkIHBhcmFtZXRlcnMuCgojIEdvYWwKCkluIHRoZSBOSEFORVMgZGF0YXNldCwgb25lIG9mIHRoZSBjb2x1bW5zIGlzIG5hbWVkIGBIZWFsdGhHZW5gLiAKSGVhbHRoR2VuIGlzIGEgc2VsZi1yZXBvcnRlZCByYXRpbmcgb2YgYSBwYXJ0aWNpcGFudOKAmXMgaGVhbHRoIAppbiBnZW5lcmFsIHRlcm1zLiBIZWFsdGhHZW4gaXMgcmVwb3J0ZWQgZm9yIHBhcnRpY2lwYW50cyBhZ2VkIDEyCnllYXJzIG9yIG9sZGVyLiBJdCBpcyBhIGZhY3RvciB3aXRoIHRoZSBmb2xsb3dpbmcgbGV2ZWxzOgpFeGNlbGxlbnQsIFZnb29kLCBHb29kLCBGYWlyLCBvciBQb29yLgoKV2Ugd2FudCB0byB0ZXN0IHdoZXRoZXIgb3Igbm90IHRoZSBtZWFuIHN5c3RvbGljIGJsb29kIHByZXNzdXJlIAp2YWx1ZSAodGFrZSBjb2x1bW4gYEJQU3lzMWApIGlzIGVxdWFsIGJldHdlZW4gdGhlIGZpdmUgc2VsZi1yZXBvcnRlZApoZWFsdGggY2F0ZWdvcmllcy4gVG8gdGhpcyBlbmQsIHdlIHdpbGwgdXNlIGFuIEFOT1ZBIGFuYWx5c2lzCihpZiB0aGUgcmVxdWlyZWQgYXNzdW1wdGlvbnMgYXJlIG1ldCkuCgojIExvYWQgdGhlIHJlcXVpcmVkIGxpYnJhcmllcwoKYGBge3IsIG1lc3NhZ2UgPSBGQUxTRX0KbGlicmFyeSh0aWR5dmVyc2UpCmBgYAoKIyBEYXRhIGltcG9ydCAgCgpgYGB7ciwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KTkhBTkVTIDwtIHJlYWRfY3N2KCJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vR1RQQi9QU0xTMjAvbWFzdGVyL2RhdGEvTkhBTkVTLmNzdiIpCmdsaW1wc2UoTkhBTkVTWzE6MTBdKQpgYGAKCiMgRGF0YSBFeHBsb3JhdGlvbgoKYGBge3J9Ck5IQU5FUyAlPiUKICBnZ3Bsb3QoYWVzKHg9SGVhbHRoR2VuLHk9QlBTeXMxLGZpbGw9SGVhbHRoR2VuKSkgKyAKICBzY2FsZV9maWxsX2JyZXdlcihwYWxldHRlPSJSZEd5IikgKwogIHRoZW1lX2J3KCkgKwogIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArCiAgI2dlb21faml0dGVyKHdpZHRoID0gMC4yLHNpemU9MC4wMSkgKyAjIyBvbWl0dGVkIGFzIGl0IG1ha2VzIHRoZSBwbG90IG1lc3N5CiAgZ2d0aXRsZSgiQm94cGxvdCBvZiB0aGUgc3lzdG9saWMgYmxvb2RwcmVzc3VyZSBmb3IgZWFjaCBoZWFsdGggY2F0ZWdvcnkiKSArCiAgeWxhYigiU3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgKG1tSGcpIikgKyAKICBzdGF0X3N1bW1hcnkoZnVuLnk9bWVhbiwgZ2VvbT0icG9pbnQiLCBzaGFwZT01LCBzaXplPTMsIGNvbG9yPSJibGFjayIsIGZpbGw9ImJsYWNrIikKYGBgCgpUaGlzIHBsb3QgaXMgbm90IGlkZWFsOyBpdCB3b3VsZCBiZSBmYXIgbW9yZSBpbnR1aXRpdmUgaWYKdGhlIGhlYWx0aCBjYXRlZ29yaWVzIHdlcmUgb3JkZXJlZCBwcm9wZXJseSAoaS5lLiwgUG9vciAtLT4gZXhjZWxsZW50KS4KSW4gYWRkaXRpb24sIHdlIG9ic2VydmUgYSBzaXh0aCAiY2F0ZWdvcnkiIG9mIE5BIHZhbHVlcy4KClRvIGltcHJvdmUgdGhlIHBsb3QgeW91IHNob3VsZDoKCjEuIEZpbHRlciBvdXQgc3ViamVjdHMgd2l0aCBOQSB2YWx1ZXMgZm9yIEhlYWx0aEdlbiBvciBCUFN5czEKMi4gU2V0IEhlYWx0aEdlbiB0byBhIGZhY3RvciBhbmQgcmVsZXZlbCBpdCB0byBQb29yIC0tPiBFeGNlbGxlbnQKClRoZSBzZWNvbmQgdGFzayBjYW4gYmUgYWNoaWV2ZWQgYnkgdXNpbmcgdGhlIGBtdXRhdGVgLCBgYXMuZmFjdG9yYAphbmQgYGZjdF9yZWxldmVsYCBmdW5jdGlvbnMuCgpgYGB7cn0KTkhBTkVTIDwtIE5IQU5FUyAlPiUKICBmaWx0ZXIoIWlzLm5hKEhlYWx0aEdlbiksIWlzLm5hKEJQU3lzMSkpICU+JQogIG11dGF0ZShIZWFsdGhHZW4gPSBhcy5mYWN0b3IoSGVhbHRoR2VuKSkgJT4lCiAgbXV0YXRlKEhlYWx0aEdlbiA9ZmN0X3JlbGV2ZWwoSGVhbHRoR2VuLGMoIlBvb3IiLCJGYWlyIiwiR29vZCIsIlZnb29kIiwiRXhjZWxsZW50IikpKQpgYGAKCmBgYHtyfQpOSEFORVMgJT4lCiAgZ2dwbG90KGFlcyh4PUhlYWx0aEdlbix5PUJQU3lzMSxmaWxsPUhlYWx0aEdlbikpICsgCiAgc2NhbGVfZmlsbF9icmV3ZXIocGFsZXR0ZT0iUmRHeSIpICsKICB0aGVtZV9idygpICsKICBnZW9tX2JveHBsb3Qob3V0bGllci5zaGFwZT1OQSkgKwogIGdndGl0bGUoIkJveHBsb3Qgb2YgdGhlIHN5c3RvbGljIGJsb29kcHJlc3N1cmUgZm9yIGVhY2ggaGVhbHRoIGNhdGVnb3J5IikgKwogIHlsYWIoIlN5c3RvbGljIGJsb29kIHByZXNzdXJlIChtbUhnKSIpICsgCiAgc3RhdF9zdW1tYXJ5KGZ1bi55PW1lYW4sIGdlb209InBvaW50Iiwgc2hhcGU9NSwgc2l6ZT0zLCBjb2xvcj0iYmxhY2siLCBmaWxsPSJibGFjayIpCmBgYAoKIyMgQ2hlY2sgdGhlIGFzc3VtcHRpb25zIGZvciBBTk9WQQoKVG8gc3R1ZHkgd2hldGhlciBvciBub3QgdGhlIG9ic2VydmVkIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGUKYXZlcmFnZSBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSB2YWx1ZXMgb2YgdGhlIGRpZmZlcmVudCBoZWFsdGggZ3JvdXBzCmFyZSBzaWduaWZpY2FudCwgd2UgbWF5IHBlcmZvcm0gYW4gQU5PVkEuCgpUaGUgbnVsbCBoeXBvdGhlc2lzIG9mIEFOT1ZBIHN0YXRlcyB0aGF0OgokSDAkOiBUaGUgbWVhbiBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSBpcyBlcXVhbCBiZXR3ZWVuIHRoZSBkaWZmZXJlbnQgaGVhbHRoIGdyb3Vwcy4KClRoZSBhbHRlcm5hdGl2ZSBoeXBvdGhlc2lzIG9mIEFOT1ZBIHN0YXRlcyB0aGF0OgokSEEkOiBUaGUgbWVhbiBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSBmb3IgYXQgbGVhc3Qgb25lIGhlYWx0aCBncm91cCBpcyBkaWZmZXJlbnQgCmZyb20gdGhlIG1lYW4gc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgaW4gYXQgbGVhc3Qgb25lIG90aGVyIGhlYWx0aCBncm91cC4KCkJlZm9yZSB3ZSBtYXkgcHJvY2VlZCB3aXRoIHRoZSBhbmFseXNpcywgd2UgbXVzdCBtYWtlIHN1cmUgdGhhdCBhbGwKYXNzdW1wdGlvbnMgZm9yIEFOT1ZBIGFyZSBtZXQuIEFOT1ZBIGhhcyB0aHJlZSBhc3N1bXB0aW9uczoKCjEuIFRoZSBvYnNlcnZhdGlvbnMgYXJlIGluZGVwZW5kZW50IG9mIGVhY2ggb3RoZXIgKGluIGFsbCBncm91cHMpCjIuIFRoZSBkYXRhIChCUFN5czEpIG11c3QgYmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgKGluIGFsbCBncm91cHMpCjMuIFRoZSB2YXJpYWJpbGl0eSB3aXRoaW4gYWxsIGdyb3VwcyBpcyBzaW1pbGFyCgojIyMgQXNzdW1wdGlvbiBvZiBpbmRlcGVuZGVuY2UKClRoZSBmaXJzdCBhc3N1bXB0aW9uIGlzIG1ldDsgdGhlcmUgc2hvdWQgYmUgbm8gc3BlY2lmaWMKcGF0dHRlcm5zIG9mIGRlcGVuZGVuY2UuCgojIyMgQXNzdW1wdGlvbiBvZiBub3JtYWxpdHkKCkZvciB0aGUgc2Vjb25kIGFzc3VtcHRpb24sIHdlIG11c3QgY2hlY2sgbm9ybWFsaXR5IGluIGVhY2ggZ3JvdXAuCgpgYGB7cn0KTkhBTkVTICU+JSAKICBnZ3Bsb3QoYWVzKHNhbXBsZT1CUFN5czEpKSArCiAgZ2VvbV9xcSgpICsKICBnZW9tX3FxX2xpbmUoKSArIAogIGZhY2V0X2dyaWQofkhlYWx0aEdlbikKYGBgCgpUaGUgZGF0YSBkb2VzIG5vdCBhcHBlYXIgdG8gYmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgZm9yCmVhY2ggZ3JvdXAuIEl0IHNlZW1zIHRvIGhhdmUgYSBoZWF2eSByaWdodCB0YWlsLiBXZSBjYW4KcGVyZm9ybSBhIGxvZyB0cmFuc2Zvcm1hdGlvbiBvbiB0aGUgZGF0YS4KCmBgYHtyfQpOSEFORVMgJT4lIAogIG11dGF0ZShCUFN5czFfbG9nID0gbG9nKEJQU3lzMSkpICU+JQogIGdncGxvdChhZXMoc2FtcGxlPUJQU3lzMV9sb2cpKSArCiAgZ2VvbV9xcSgpICsKICBnZW9tX3FxX2xpbmUoKSArIAogIGZhY2V0X2dyaWQofkhlYWx0aEdlbikKYGBgCgpXaGlsZSB0aGUgbG9nIHRyYW5zZm9ybWF0aW9uIGltcHJvdmVkIHRoZSBkaXN0cmlidXRpb25zIHNvbWV3YWh0LAp0aGUgZGF0YSBzdGlsbCBkb2VzIG5vdCBhcHBlYXIgdG8gYmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgZm9yCmVhY2ggZ3JvdXAuIEhvd2V2ZXIsIHdlIGRvIGhhdmUgYSB2ZXJ5IGxhcmdlIG51bWJlciBvZgpvYnNlcnZhdGlvbnMgcGVyIGdyb3VwOgoKYGBge3J9CiMjIENvdW50IHRoZSBudW1iZXIgb2Ygb2JzZXJ2YXRpb25zIHBlciB0cmVhdG1lbnQKTkhBTkVTICU+JQogIGNvdW50KEhlYWx0aEdlbikKYGBgCgpBcyBzdWNoLCB3ZSBtYXkgcmVseSBvbiB0aGUgY2VudGFsIGxpbWl0IHRoZW9yZW0uClJlbWVtYmVyLCB0aGUgY2VudGFsIGxpbWl0IHRoZW9yZW0gdGhhdCB3aGVuIHRoZSBudW1iZXIKb2Ygb2JzZXJ2YXRpb25zIGlzIHN1ZmZpY2llbnRseSBsYXJnZSAoaS5lLiA+MTAwKSwgd2UgCndpbGwgYXNzdW1lIHRoYXQgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgc2FtcGxlIG1lYW4gd2lsbCAKYXBwcm94aW1hdGUgYSBub3JtYWwgZGlzdHJpYnV0aW9uLCBldmVuIGlmIHRoZSB1bmRlcmx5aW5nCmRhdGEgaXMgbm90IG5vcm1hbGx5IGRpc3RyaWJ1dGVkLgoKIyBBbmFseXNpcwoKIyMgQU5PVkEKCmBgYHtyfQpmaXQgPC0gbG0obG9nKEJQU3lzMSl+SGVhbHRoR2VuLCBOSEFORVMpCmZpdF9hbm92YSA8LSBhbm92YShmaXQpCmZpdF9hbm92YQpwcmludChwYXN0ZSgiTm90LXJvdW5kZWQgcC12YWx1ZToiLCBmaXRfYW5vdmEkYFByKD5GKWBbMV0pKQpgYGAKClRoZSBwLXZhbHVlIG9mIHRoZSBBTk9WQSBhbmFseXNpcyBpcyBleHRyZW1lbHkgc2lnbmlmaWNhbnQKKHAtdmFsdWUgPSBgciBmb3JtYXQoZml0X2Fub3ZhJCJQcig+RikiWzFdLGRpZ2l0cz00KWApLCAKc28gd2UgcmVqZWN0IHRoZSBudWxsIGh5cG90aGVzaXMgdGhhdCB0aGUgbWVhbiAKZWdnIGxlbmd0aCBpcyBlcXVhbCBiZXR3ZWVuIHRoZSBkaWZmZXJlbnQgYmlyZCB0eXBlcy4KV2UgY2FuIHNheSB0aGF0IHRoZSBtZWFuIGVnZyBsZW5ndGggaXMgc2lnbmlmaWNhbnRseSBkaWZmZXJlbnQKYmV0d2VlbiBhdCBsZWFzdCB0d28gYmlyZCB0eXBlcyBvbiB0aGUgNSUgc2lnbmlmaWNhbmNlIGxldmVsLgoKQmFzZWQgb24gdGhpcyBhbmFseXNpcywgd2UgZG8gbm90IHlldCBrbm93IGJldHdlZW4gd2hpY2ggcGFydGljdWxhcgpiaXJkIHR5cGVzIHRoZXJlIGlzIGEgc2lnbmlmaWNhbnQgZGlmZmVyZW5jZS4gVG8gc3R1ZHkgdGhpcywgd2Ugd2lsbApwZXJmcm9tIHRoZSBUdWNrZXkgcG9zdC1ob2MgYW5hbHlzaXMuCgojIyBQb3N0LWhvYyBhbmFseXNpcwoKYGBge3IsbWVzc2FnZT1GQUxTRX0KbGlicmFyeShtdWx0Y29tcCwgcXVpZXRseSA9IFRSVUUpCm1jcCA8LSBnbGh0KGZpdCwgbGluZmN0ID0gbWNwKEhlYWx0aEdlbiA9ICJUdWtleSIpKQpzdW1tYXJ5KG1jcCkKY29uZmludChtY3ApCmBgYAoKIyBDb25jbHVzaW9uCgpXZSBoYXZlIGZvdW5kIGFuIGV4dHJlbWVseSBzaWduaWZpY2FudCBkZXBlbmRlbmNlIChwLXZhbHVlID0gYHIgZm9ybWF0KGZpdF9hbm92YSQiUHIoPkYpIlsxXSxkaWdpdHM9NClgKSwgCmJldHdlZW4gdGhlIG1lYW4gc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgYW5kIHRoZSBoZWFsdGggZ3JvdXAKb24gdGhlIGdsb2JhbCA1JSBzaWduaWZpY2FuY2UgbGV2ZWwuCgpUaGUgbWVhbiBsb2dhcml0aG0gb2Ygc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgaW4gdGhlIHNlbGYtcmVwb3J0ZWQgaGVhbHRoCmNhdGVnb3J5IGBQb29yYCBpcyBzaWduaWZpY2FudGx5IGhpZ2hlciBhcyBjb21wYXJlZCB0aHJlZSBvdGhlciBncm91cHM6CgotIHRoZSBgR29vZGAgZ3JvdXAgKGFkanVzdGVkIHAtdmFsdWUgPSA8IDAuMDAxLCBtZWFuIGRpZmZlcmVuY2UgPSAtMC4wMzY3NjIgbW1IZywgOTUlIENJIFstMC4wMzQ3MjU7IC0wLjAwNzE0Ml0pCi0gdGhlIGBWZ29vZGAgZ3JvdXAgKGFkanVzdGVkIHAtdmFsdWUgPSA8IDAuMDAxLCBtZWFuIGRpZmZlcmVuY2UgPSAtMC4wNTk0MTUgbW1IZywgOTUlIENJIFstMC4wODczNTI7IC0wLjAzMTQ3N10pCi0gdGhlIGBFeGNlbGxlbnRgIGdyb3VwIChhZGp1c3RlZCBwLXZhbHVlID0gPCAwLjAwMSwgbWVhbiBkaWZmZXJlbmNlID0gLTAuMDU5NDE1IG1tSGcsIDk1JSBDSSBbLTAuMDg1MTY0OyAtMC4wMjU3ODldKQoKVGhlIG1lYW4gbG9nYXJpdGhtIG9mICBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSBpbiB0aGUgc2VsZi1yZXBvcnRlZCBoZWFsdGgKY2F0ZWdvcnkgYEZhaXJgIGlzIHNpZ25pZmljYW50bHkgaGlnaGVyIGFzIGNvbXBhcmVkIHRocmVlIG90aGVyIGdyb3VwczoKCi0gdGhlIGBHb29kYCBncm91cCAoYWRqdXN0ZWQgcC12YWx1ZSA9IDAuMDAzMTcsIG1lYW4gZGlmZmVyZW5jZSA9IC0wLjAyMDkzNCAgbW1IZywgOTUlIENJIFstMC4wNjQ1NDI7IC0wLjAwODk4Ml0pCi0gdGhlIGBWZ29vZGAgZ3JvdXAgKGFkanVzdGVkIHAtdmFsdWUgPSA8IDAuMDAxLCBtZWFuIGRpZmZlcmVuY2UgPSAtMC4wNDM1ODcgbW1IZywgOTUlIENJIFstMC4wNTc2OTI7IC0wLjAyOTQ4MV0pCi0gdGhlIGBFeGNlbGxlbnRgIGdyb3VwIChhZGp1c3RlZCBwLXZhbHVlID0gPCAwLjAwMSwgbWVhbiBkaWZmZXJlbmNlID0gLTAuMDM5NjQ4IG1tSGcsIDk1JSBDSSBbLTAuMDU2OTYzOyAtMC4wMjIzMzNdKQoKVGhlIG1lYW4gbG9nYXJpdGhtIG9mIHN5c3RvbGljIGJsb29kIHByZXNzdXJlIGluIHRoZSBzZWxmLXJlcG9ydGVkIGhlYWx0aApjYXRlZ29yeSBgR29vZGAgaXMgc2lnbmlmaWNhbnRseSBoaWdoZXIgYXMgY29tcGFyZWQgdHdvIG90aGVyIGdyb3VwczoKCi0gdGhlIGBWZ29vZGAgZ3JvdXAgKGFkanVzdGVkIHAtdmFsdWUgPSA8IDAuMDAxLCBtZWFuIGRpZmZlcmVuY2UgPSAtMC4wMjI2NTMgbW1IZywgOTUlIENJIFstMC4wMzI4NDM7IC0wLjAxMjQ2M10pCi0gdGhlIGBFeGNlbGxlbnRgIGdyb3VwIChhZGp1c3RlZCBwLXZhbHVlID0gMC4wMDM2MiwgbWVhbiBkaWZmZXJlbmNlID0gLTAuMDE4NzE0IG1tSGcsIDk1JSBDSSBbLTAuMDMzMDIxOyAtMC4wMDQ0MDhdKQoKV2UgZG8gbm90IGZpbmQgZW5vdWdoIGV2aWRlbmNlIHRvIGNsYWltIGEgZGlmZmVyZW5jZSBpbiBzeXN0b2xpYwpibG9vZCBwcmVzc3VyZSBsZXZlbHMgYmV0d2VlbiB0aGUgb3RoZXIgZ3JvdXBzLgoKTm90ZSB0aGF0IGluIG9yZGVyIHRvIGludGVycHJldCB0aGUgb3V0Y29tZXMgb24gdGhlIG9yaWdpbmFsIHNjYWxlLAp3ZSBzaG91bGQgYmFja3RyYW5zZm9ybSB0aGUgb3V0Y29tZXMgd2l0aCB0aGUgYGV4cCgpYCBmdW5jdGlvbnMgCihpbnRlcnByZXRhdGlvbiBvbiB0aGUgZ2VvbWV0cmljIG1lYW4pLgoKCgoKCgoKCg==