In this tutorial, you will learn how to explore paired data.

1 The captopril dataset

The captopril dataset stems from a small experiment with 15 patients with hypertension. For each patient systolic and diasystolic blood pressure measurements where taken before and after administering captopril.

2 Import the data

library(tidyverse)
captopril <- read_csv("https://raw.githubusercontent.com/GTPB/PSLS20/master/data/captopril.txt")
## Parsed with column specification:
## cols(
##   id = col_double(),
##   SBPb = col_double(),
##   DBPb = col_double(),
##   SBPa = col_double(),
##   DBPa = col_double()
## )
head(captopril)
## # A tibble: 6 x 5
##      id  SBPb  DBPb  SBPa  DBPa
##   <dbl> <dbl> <dbl> <dbl> <dbl>
## 1     1   210   130   201   125
## 2     2   169   122   165   121
## 3     3   187   124   166   121
## 4     4   160   104   157   106
## 5     5   167   112   147   101
## 6     6   176   101   145    85

3 Data visualization

One straightforward possibility to visualize the four types of blood pressure values is by adopting the gather function from tidyverse. It will reshape the dataframe, such that we have have a single variable type, which points at one of the four blood pressure types, and bp, which points at the actual value for each type for each patient.

captopril %>% 
  gather(type,bp,-id)
## # A tibble: 60 x 3
##       id type     bp
##    <dbl> <chr> <dbl>
##  1     1 SBPb    210
##  2     2 SBPb    169
##  3     3 SBPb    187
##  4     4 SBPb    160
##  5     5 SBPb    167
##  6     6 SBPb    176
##  7     7 SBPb    185
##  8     8 SBPb    206
##  9     9 SBPb    173
## 10    10 SBPb    146
## # … with 50 more rows

3.1 Barplot

A Barplot is a plot that you will commonly find in papers.

captopril %>% 
  gather(type,bp,-id) %>% 
  group_by(type) %>%
     summarize_at("bp",
               list(mean=~mean(.,na.rm=TRUE),
                    sd=~sd(.,na.rm=TRUE),
                    n=function(x) x%>%is.na%>%`!`%>%sum)) %>%
  mutate(se=sd/sqrt(n)) %>%
  ggplot(aes(x=type,y=mean,fill=type)) + 
  scale_fill_brewer(palette="RdGy") +
  theme_bw() +
  geom_bar(stat="identity") + 
  geom_errorbar(aes(ymin=mean-se, ymax=mean+se),width=.2) +
  ggtitle("Barplot of different blood pressure measures") +
  ylab("blood pressure (mmHg)")

A barplot, however, is not very informative. The height of the bars only provides us with information of the mean blood pressure. However, we don’t see the actual underlying values, so we for instance don’t have any information on the spread of the data. It is usually more informative to represent to underlying values as raw as possible. Boxplots are ideal for this!

Note that it is possible to add the raw data on the barplot, but we still would not see any measures of the spread, such as the interquartile range.

3.2 Boxplot

With the boxplot, we get a lot of useful information. We immediately see multiple features of the spread of the data, such as the median, the 25% and 75% quantiles and outliers. Since we only have 15 raw values (patients), we can easily add them to the plot without getting messy.

captopril %>% 
  gather(type,bp,-id) %>% 
  ggplot(aes(x=type,y=bp,fill=type)) + 
  scale_fill_brewer(palette="RdGy") +
  theme_bw() +
  geom_boxplot(outlier.shape=NA) + 
  geom_jitter(width = 0.2) +
  ggtitle("Boxplot of different blood pressure measures") +
  ylab("blood pressure (mmHg)") + stat_summary(fun.y=mean, geom="point", shape=5, size=3, color="black", fill="black")

In terms of interpretation, we can see that the median systolic and diastolic blood pressure values are lower after treatment with captopril than before. If we want to have visual inference on the mean values (cfr. class on t-test), we can add them to the plot stat_summary function.

4 Paired data

An important feature of this dataset is that it contains paired data; for each patient, we have blood pressure values (systolic and diastolic) before and after treatment with captopril.

We first plot the systolic blood pressure measurements. We make a plot with the data points for the two systolic blood pressure measurements in the patient id.

captopril %>% 
  gather(type,bp,-id) %>% 
  filter(type%in%c("SBPa","SBPb")) %>%
  ggplot(aes(x=id,y=bp,color=type)) +
  geom_point() +
  scale_color_manual(values = c("darkgreen","red"))

We see that for all patients, the systolic blood pressure is lower after captopril treatment than before. Note that we could not see this from the boxplot, directly.

Another good way to visualise the paired data is by using a line plot that is grouped by subject. Note, that we make a factor of the type and that we choose order of the levels so that the level “b” before becomes before the level “a”. Otherwise the after measurements will appear first in the plot.

captopril %>% 
  gather(type,bp,-id) %>%
  mutate(type=factor(type,levels=c("SBPb","SBPa","DBPb","DBPa"))) %>%
  filter(type%in%c("SBPa","SBPb")) %>%
  ggplot(aes(x=type,y=bp)) +
  geom_line(aes(group = id)) +
  geom_point()

captopril %>% 
  gather(type,bp,-id) %>%
  mutate(type=factor(type,levels=c("SBPb","SBPa","DBPb","DBPa"))) %>%
  filter(type%in%c("DBPa","DBPb")) %>%
  ggplot(aes(x=type,y=bp)) +
  geom_line(aes(group = id)) +
  geom_point()

Analogously, we may immediately difference the after measurement from the before measurement because we have paired data.

captopril %>% 
  mutate(bp_diff = SBPa-SBPb) %>%
  ggplot(aes(x="",y=bp_diff)) +
  geom_boxplot(outlier.shape=NA) +
  geom_jitter(width = 0.2) + 
  ggtitle("Boxplot of the difference in blood pressure") +
  ylab("blood pressure difference (mmHg)") + stat_summary(fun.y=mean, geom="point", shape=5, size=3, color="black", fill="black") +
  theme_bw()+
  ylim(-40,10) +
  geom_hline(yintercept = 0, color="red") # adds horizontal line to plot

The mean difference is lower than zero. It seems that on average the captopril treatment has lowered the blood pressure by about 20 mmHg.

5 Assumptions: QQ-plot

In the tutorial on hypothesis testing we will assess if this difference is statistically significant. We will introduce a statistical test that relies on the assumption that the differences are normally distributed. We can check this with a QQ-plot:

captopril %>% 
  mutate(bp_diff = SBPa-SBPb) %>%
  select(bp_diff) %>%
  ggplot(aes(sample=bp_diff)) +
  geom_qq() +
  geom_qq_line()

The plot shows no large deviations of the straight line indicating that the data are approximately normally distributed.

6 Descriptive statistics

Descriptive statistics for all blood pressure measurements are found below.

captopril %>% 
  gather(type,bp,-id) %>% 
  group_by(type) %>%
     summarize_at("bp",
               list(mean=~mean(.,na.rm=TRUE),
                    sd=~sd(.,na.rm=TRUE),
                    n=function(x) x%>%is.na%>%`!`%>%sum))%>%
  mutate(se=sd/sqrt(n))
## # A tibble: 4 x 5
##   type   mean    sd     n    se
##   <chr> <dbl> <dbl> <int> <dbl>
## 1 DBPa   103.  12.6    15  3.24
## 2 DBPb   112.  10.5    15  2.70
## 3 SBPa   158   20.0    15  5.16
## 4 SBPb   177.  20.6    15  5.31

It is also very useful to calculate the descriptive statistics for the systolic blood pressure difference because we can assess the effect size of administering captopril for every patient.

captopril %>% 
  mutate(bp_diff = SBPa-SBPb) %>%
     summarize_at("bp_diff",
               list(mean=~mean(.,na.rm=TRUE),
                    sd=~sd(.,na.rm=TRUE),
                    n=function(x) x%>%is.na%>%`!`%>%sum)) %>%
  mutate(se=sd/sqrt(n))
## # A tibble: 1 x 4
##    mean    sd     n    se
##   <dbl> <dbl> <int> <dbl>
## 1 -18.9  9.03    15  2.33

7 Wrap-up

Next, time if you explore paired data, you can immediately adopt the line plot and the boxplot of the differences from Section 3.3. so as to acknowledge the pairing and perform the descriptive statistics in section 3.4. It might also be useful to plot boxplots of the raw blood pressure measurements to get an idea on their varability.

It is not useful to report the barplot.

7.1 Advantage of paired design

Every patient acts as their own control: we can estimate the effect of the treatment for each patient

captopril %>% 
  ggplot(aes(SBPb,SBPa)) +
  geom_point() + 
  ggtitle(paste("correlation = ",round(cor(captopril$SBPb,captopril$SBPa),3)))

captopril %>% 
  mutate(bp_diff = SBPa-SBPb) %>%
  select(SBPb,SBPa,bp_diff) %>%
  summarize_all(sd)
## # A tibble: 1 x 3
##    SBPb  SBPa bp_diff
##   <dbl> <dbl>   <dbl>
## 1  20.6  20.0    9.03

Calculate variance based on fact that data are paired.

captopril %>% summarise(sd_diff=sqrt(sd(SBPb)^2 + sd(SBPa)^2-2*sd(SBPa)*sd(SBPb)*cor(SBPa,SBPb)))
## # A tibble: 1 x 1
##   sd_diff
##     <dbl>
## 1    9.03

Note, that the standard deviations on the blood pressure measurements are more than twice as large as the standard deviations on the difference. This is because each patient acts as their own control. As such, we can correct for their baseline blood pressure when estimating the effect of the treatment.


LS0tCnRpdGxlOiAiVHV0b3JpYWwgNC4zOiBFeHBsb3JpbmcgdGhlIGNhcHRvcHJpbCBkYXRhc2V0IiAgIApvdXRwdXQ6CiAgICBodG1sX2RvY3VtZW50OgogICAgICBjb2RlX2Rvd25sb2FkOiB0cnVlICAgIAogICAgICB0aGVtZTogY29zbW8KICAgICAgdG9jOiB0cnVlCiAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICBoaWdobGlnaHQ6IHRhbmdvCiAgICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQotLS0KCkluIHRoaXMgdHV0b3JpYWwsIHlvdSB3aWxsIGxlYXJuIGhvdyB0byBleHBsb3JlIHBhaXJlZCBkYXRhLiAKCiMgVGhlIGNhcHRvcHJpbCBkYXRhc2V0CgpUaGUgY2FwdG9wcmlsIGRhdGFzZXQgc3RlbXMgZnJvbSBhIHNtYWxsIGV4cGVyaW1lbnQgd2l0aAoxNSBwYXRpZW50cyB3aXRoIGh5cGVydGVuc2lvbi4gCkZvciBlYWNoIHBhdGllbnQgc3lzdG9saWMgYW5kIGRpYXN5c3RvbGljIGJsb29kIHByZXNzdXJlIG1lYXN1cmVtZW50cyB3aGVyZSB0YWtlbiBiZWZvcmUgYW5kIGFmdGVyIGFkbWluaXN0ZXJpbmcgY2FwdG9wcmlsLgoKCiMgSW1wb3J0IHRoZSBkYXRhCgoKYGBge3IsIG1lc3NhZ2U9RkFMU0V9CmxpYnJhcnkodGlkeXZlcnNlKQpgYGAKCmBgYHtyfQpjYXB0b3ByaWwgPC0gcmVhZF9jc3YoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9HVFBCL1BTTFMyMC9tYXN0ZXIvZGF0YS9jYXB0b3ByaWwudHh0IikKYGBgCgpgYGB7cn0KaGVhZChjYXB0b3ByaWwpCmBgYAoKCiMgRGF0YSB2aXN1YWxpemF0aW9uCgpPbmUgc3RyYWlnaHRmb3J3YXJkIHBvc3NpYmlsaXR5IHRvIHZpc3VhbGl6ZSB0aGUgZm91ciB0eXBlcwpvZiBibG9vZCBwcmVzc3VyZSB2YWx1ZXMgaXMgYnkgYWRvcHRpbmcgdGhlIGBnYXRoZXJgCmZ1bmN0aW9uIGZyb20gdGlkeXZlcnNlLiBJdCB3aWxsIHJlc2hhcGUgdGhlIGRhdGFmcmFtZSwKc3VjaCB0aGF0IHdlIGhhdmUgaGF2ZSBhIHNpbmdsZSB2YXJpYWJsZSBgdHlwZWAsIHdoaWNoIApwb2ludHMgYXQgb25lIG9mIHRoZSBmb3VyIGJsb29kIHByZXNzdXJlIHR5cGVzLCBhbmQgYGJwYCwKd2hpY2ggcG9pbnRzIGF0IHRoZSBhY3R1YWwgdmFsdWUgZm9yIGVhY2ggdHlwZSAKZm9yIGVhY2ggcGF0aWVudC4KCmBgYHtyfQpjYXB0b3ByaWwgJT4lIAogIGdhdGhlcih0eXBlLGJwLC1pZCkKYGBgCgojIyBCYXJwbG90CgpBIEJhcnBsb3QgaXMgYSBwbG90IHRoYXQgeW91IHdpbGwgY29tbW9ubHkgZmluZCBpbiBwYXBlcnMuCgpgYGB7cn0KY2FwdG9wcmlsICU+JSAKICBnYXRoZXIodHlwZSxicCwtaWQpICU+JSAKICBncm91cF9ieSh0eXBlKSAlPiUKICAgICBzdW1tYXJpemVfYXQoImJwIiwKICAgICAgICAgICAgICAgbGlzdChtZWFuPX5tZWFuKC4sbmEucm09VFJVRSksCiAgICAgICAgICAgICAgICAgICAgc2Q9fnNkKC4sbmEucm09VFJVRSksCiAgICAgICAgICAgICAgICAgICAgbj1mdW5jdGlvbih4KSB4JT4laXMubmElPiVgIWAlPiVzdW0pKSAlPiUKICBtdXRhdGUoc2U9c2Qvc3FydChuKSkgJT4lCiAgZ2dwbG90KGFlcyh4PXR5cGUseT1tZWFuLGZpbGw9dHlwZSkpICsgCiAgc2NhbGVfZmlsbF9icmV3ZXIocGFsZXR0ZT0iUmRHeSIpICsKICB0aGVtZV9idygpICsKICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIpICsgCiAgZ2VvbV9lcnJvcmJhcihhZXMoeW1pbj1tZWFuLXNlLCB5bWF4PW1lYW4rc2UpLHdpZHRoPS4yKSArCiAgZ2d0aXRsZSgiQmFycGxvdCBvZiBkaWZmZXJlbnQgYmxvb2QgcHJlc3N1cmUgbWVhc3VyZXMiKSArCiAgeWxhYigiYmxvb2QgcHJlc3N1cmUgKG1tSGcpIikKYGBgCgpBIGJhcnBsb3QsIGhvd2V2ZXIsIGlzIG5vdCB2ZXJ5IGluZm9ybWF0aXZlLiBUaGUgaGVpZ2h0IG9mIHRoZSBiYXJzCm9ubHkgcHJvdmlkZXMgdXMgd2l0aCBpbmZvcm1hdGlvbiBvZiB0aGUgbWVhbiBibG9vZCBwcmVzc3VyZS4KSG93ZXZlciwgd2UgZG9uJ3Qgc2VlIHRoZSBhY3R1YWwgdW5kZXJseWluZyB2YWx1ZXMsIHNvIHdlIGZvcgppbnN0YW5jZSBkb24ndCBoYXZlIGFueSBpbmZvcm1hdGlvbiBvbiB0aGUgc3ByZWFkIG9mIHRoZSBkYXRhLgpJdCBpcyB1c3VhbGx5IG1vcmUgaW5mb3JtYXRpdmUgdG8gcmVwcmVzZW50IHRvIHVuZGVybHlpbmcgCnZhbHVlcyBhcyBfcmF3XyBhcyBwb3NzaWJsZS4gQm94cGxvdHMgYXJlIGlkZWFsIGZvciB0aGlzIQoKTm90ZSB0aGF0IGl0IGlzIHBvc3NpYmxlIHRvIGFkZCB0aGUgcmF3IGRhdGEgb24KdGhlIGJhcnBsb3QsIGJ1dCB3ZSBzdGlsbCB3b3VsZCBub3Qgc2VlIGFueSBtZWFzdXJlcyBvZiB0aGUKc3ByZWFkLCBzdWNoIGFzIHRoZSBpbnRlcnF1YXJ0aWxlIHJhbmdlLgoKIyMgQm94cGxvdCAKCldpdGggdGhlIGJveHBsb3QsIHdlIGdldCBhIGxvdCBvZiB1c2VmdWwgaW5mb3JtYXRpb24uCldlIGltbWVkaWF0ZWx5IHNlZSBtdWx0aXBsZSBmZWF0dXJlcyBvZiB0aGUgc3ByZWFkIG9mCnRoZSBkYXRhLCBzdWNoIGFzIHRoZSBtZWRpYW4sIHRoZSAyNSUgYW5kIDc1JSBxdWFudGlsZXMKYW5kIG91dGxpZXJzLiBTaW5jZSB3ZSBvbmx5IGhhdmUgMTUgcmF3IHZhbHVlcyAocGF0aWVudHMpLAp3ZSBjYW4gZWFzaWx5IGFkZCB0aGVtIHRvIHRoZSBwbG90IHdpdGhvdXQgZ2V0dGluZyBtZXNzeS4KCmBgYHtyfQpjYXB0b3ByaWwgJT4lIAogIGdhdGhlcih0eXBlLGJwLC1pZCkgJT4lIAogIGdncGxvdChhZXMoeD10eXBlLHk9YnAsZmlsbD10eXBlKSkgKyAKICBzY2FsZV9maWxsX2JyZXdlcihwYWxldHRlPSJSZEd5IikgKwogIHRoZW1lX2J3KCkgKwogIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArIAogIGdlb21faml0dGVyKHdpZHRoID0gMC4yKSArCiAgZ2d0aXRsZSgiQm94cGxvdCBvZiBkaWZmZXJlbnQgYmxvb2QgcHJlc3N1cmUgbWVhc3VyZXMiKSArCiAgeWxhYigiYmxvb2QgcHJlc3N1cmUgKG1tSGcpIikgKyBzdGF0X3N1bW1hcnkoZnVuLnk9bWVhbiwgZ2VvbT0icG9pbnQiLCBzaGFwZT01LCBzaXplPTMsIGNvbG9yPSJibGFjayIsIGZpbGw9ImJsYWNrIikKYGBgCgoKSW4gdGVybXMgb2YgaW50ZXJwcmV0YXRpb24sIHdlIGNhbiBzZWUgdGhhdCB0aGUgbWVkaWFuIApzeXN0b2xpYyBhbmQgZGlhc3RvbGljIGJsb29kIHByZXNzdXJlIHZhbHVlcyBhcmUgbG93ZXIgCmFmdGVyIHRyZWF0bWVudCB3aXRoIGNhcHRvcHJpbCB0aGFuIGJlZm9yZS4gSWYgd2Ugd2FudCB0bwpoYXZlIHZpc3VhbCBpbmZlcmVuY2Ugb24gdGhlIG1lYW4gdmFsdWVzIChjZnIuIGNsYXNzIG9uIHQtdGVzdCksCndlIGNhbiBhZGQgdGhlbSB0byB0aGUgcGxvdCBgc3RhdF9zdW1tYXJ5YCBmdW5jdGlvbi4KCiMgUGFpcmVkIGRhdGEKCkFuIGltcG9ydGFudCBmZWF0dXJlIG9mIHRoaXMgZGF0YXNldCBpcyB0aGF0IGl0IGNvbnRhaW5zCnBhaXJlZCBkYXRhOyBmb3IgZWFjaCBwYXRpZW50LCB3ZSBoYXZlIGJsb29kIHByZXNzdXJlIHZhbHVlcwooc3lzdG9saWMgYW5kIGRpYXN0b2xpYykgYmVmb3JlIGFuZCBhZnRlciB0cmVhdG1lbnQgd2l0aApjYXB0b3ByaWwuCgpXZSBmaXJzdCBwbG90IHRoZSBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSBtZWFzdXJlbWVudHMuIApXZSBtYWtlIGEgcGxvdCB3aXRoIHRoZSBkYXRhIHBvaW50cyBmb3IgdGhlIHR3byBzeXN0b2xpYyBibG9vZCBwcmVzc3VyZSBtZWFzdXJlbWVudHMgaW4gdGhlIHBhdGllbnQgaWQuIAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2F0aGVyKHR5cGUsYnAsLWlkKSAlPiUgCiAgZmlsdGVyKHR5cGUlaW4lYygiU0JQYSIsIlNCUGIiKSkgJT4lCiAgZ2dwbG90KGFlcyh4PWlkLHk9YnAsY29sb3I9dHlwZSkpICsKICBnZW9tX3BvaW50KCkgKwogIHNjYWxlX2NvbG9yX21hbnVhbCh2YWx1ZXMgPSBjKCJkYXJrZ3JlZW4iLCJyZWQiKSkKYGBgCgpXZSBzZWUgdGhhdCBmb3IgYWxsIHBhdGllbnRzLCB0aGUgc3lzdG9saWMgYmxvb2QKcHJlc3N1cmUgaXMgbG93ZXIgYWZ0ZXIgY2FwdG9wcmlsIHRyZWF0bWVudCB0aGFuIGJlZm9yZS4KTm90ZSB0aGF0IHdlIGNvdWxkIG5vdCBzZWUgdGhpcyBmcm9tIHRoZSBib3hwbG90LCBkaXJlY3RseS4KCkFub3RoZXIgZ29vZCB3YXkgdG8gdmlzdWFsaXNlIHRoZSBwYWlyZWQgZGF0YSBpcyBieSB1c2luZyBhIGxpbmUgcGxvdCB0aGF0IGlzIGdyb3VwZWQgCmJ5IHN1YmplY3QuIE5vdGUsIHRoYXQgd2UgbWFrZSBhIGZhY3RvciBvZiB0aGUgdHlwZSBhbmQgdGhhdCB3ZSBjaG9vc2Ugb3JkZXIgb2YgCnRoZSBsZXZlbHMgc28gdGhhdCB0aGUgbGV2ZWwgImIiIGJlZm9yZSBiZWNvbWVzIGJlZm9yZSB0aGUgbGV2ZWwgImEiLiBPdGhlcndpc2UgCnRoZSBhZnRlciBtZWFzdXJlbWVudHMgd2lsbCBhcHBlYXIgZmlyc3QgaW4gdGhlIHBsb3QuIAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2F0aGVyKHR5cGUsYnAsLWlkKSAlPiUKICBtdXRhdGUodHlwZT1mYWN0b3IodHlwZSxsZXZlbHM9YygiU0JQYiIsIlNCUGEiLCJEQlBiIiwiREJQYSIpKSkgJT4lCiAgZmlsdGVyKHR5cGUlaW4lYygiU0JQYSIsIlNCUGIiKSkgJT4lCiAgZ2dwbG90KGFlcyh4PXR5cGUseT1icCkpICsKICBnZW9tX2xpbmUoYWVzKGdyb3VwID0gaWQpKSArCiAgZ2VvbV9wb2ludCgpCmBgYAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2F0aGVyKHR5cGUsYnAsLWlkKSAlPiUKICBtdXRhdGUodHlwZT1mYWN0b3IodHlwZSxsZXZlbHM9YygiU0JQYiIsIlNCUGEiLCJEQlBiIiwiREJQYSIpKSkgJT4lCiAgZmlsdGVyKHR5cGUlaW4lYygiREJQYSIsIkRCUGIiKSkgJT4lCiAgZ2dwbG90KGFlcyh4PXR5cGUseT1icCkpICsKICBnZW9tX2xpbmUoYWVzKGdyb3VwID0gaWQpKSArCiAgZ2VvbV9wb2ludCgpCmBgYAoKQW5hbG9nb3VzbHksIHdlIG1heSBpbW1lZGlhdGVseSBkaWZmZXJlbmNlIHRoZSBgYWZ0ZXJgIG1lYXN1cmVtZW50IGZyb20gdGhlCmBiZWZvcmVgIG1lYXN1cmVtZW50IGJlY2F1c2Ugd2UgaGF2ZSBwYWlyZWQgZGF0YS4KCmBgYHtyfQpjYXB0b3ByaWwgJT4lIAogIG11dGF0ZShicF9kaWZmID0gU0JQYS1TQlBiKSAlPiUKICBnZ3Bsb3QoYWVzKHg9IiIseT1icF9kaWZmKSkgKwogIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArCiAgZ2VvbV9qaXR0ZXIod2lkdGggPSAwLjIpICsgCiAgZ2d0aXRsZSgiQm94cGxvdCBvZiB0aGUgZGlmZmVyZW5jZSBpbiBibG9vZCBwcmVzc3VyZSIpICsKICB5bGFiKCJibG9vZCBwcmVzc3VyZSBkaWZmZXJlbmNlIChtbUhnKSIpICsgc3RhdF9zdW1tYXJ5KGZ1bi55PW1lYW4sIGdlb209InBvaW50Iiwgc2hhcGU9NSwgc2l6ZT0zLCBjb2xvcj0iYmxhY2siLCBmaWxsPSJibGFjayIpICsKICB0aGVtZV9idygpKwogIHlsaW0oLTQwLDEwKSArCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gMCwgY29sb3I9InJlZCIpICMgYWRkcyBob3Jpem9udGFsIGxpbmUgdG8gcGxvdApgYGAKClRoZSBtZWFuIGRpZmZlcmVuY2UgaXMgbG93ZXIgdGhhbiB6ZXJvLiBJdCBzZWVtcyAKdGhhdCBvbiBhdmVyYWdlIHRoZSBjYXB0b3ByaWwgdHJlYXRtZW50IGhhcyBsb3dlcmVkCnRoZSBibG9vZCBwcmVzc3VyZSBieSBhYm91dCAyMCBtbUhnLiAKCiMgQXNzdW1wdGlvbnM6IFFRLXBsb3QKCkluIHRoZSB0dXRvcmlhbCBvbiBoeXBvdGhlc2lzIHRlc3Rpbmcgd2Ugd2lsbCBhc3Nlc3MgaWYgdGhpcyBkaWZmZXJlbmNlIGlzIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQuIApXZSB3aWxsIGludHJvZHVjZSBhIHN0YXRpc3RpY2FsIHRlc3QgdGhhdCByZWxpZXMgb24gdGhlIGFzc3VtcHRpb24gdGhhdCB0aGUgZGlmZmVyZW5jZXMgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLiAKV2UgY2FuIGNoZWNrIHRoaXMgd2l0aCBhIFFRLXBsb3Q6IAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgbXV0YXRlKGJwX2RpZmYgPSBTQlBhLVNCUGIpICU+JQogIHNlbGVjdChicF9kaWZmKSAlPiUKICBnZ3Bsb3QoYWVzKHNhbXBsZT1icF9kaWZmKSkgKwogIGdlb21fcXEoKSArCiAgZ2VvbV9xcV9saW5lKCkKYGBgCgpUaGUgcGxvdCBzaG93cyBubyBsYXJnZSBkZXZpYXRpb25zIG9mIHRoZSBzdHJhaWdodCBsaW5lIGluZGljYXRpbmcgdGhhdCB0aGUgZGF0YSBhcmUgYXBwcm94aW1hdGVseSBub3JtYWxseSBkaXN0cmlidXRlZC4gCgojRGVzY3JpcHRpdmUgc3RhdGlzdGljcwoKRGVzY3JpcHRpdmUgc3RhdGlzdGljcyBmb3IgYWxsIGJsb29kIHByZXNzdXJlIG1lYXN1cmVtZW50cyBhcmUgZm91bmQgYmVsb3cuIAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2F0aGVyKHR5cGUsYnAsLWlkKSAlPiUgCiAgZ3JvdXBfYnkodHlwZSkgJT4lCiAgICAgc3VtbWFyaXplX2F0KCJicCIsCiAgICAgICAgICAgICAgIGxpc3QobWVhbj1+bWVhbiguLG5hLnJtPVRSVUUpLAogICAgICAgICAgICAgICAgICAgIHNkPX5zZCguLG5hLnJtPVRSVUUpLAogICAgICAgICAgICAgICAgICAgIG49ZnVuY3Rpb24oeCkgeCU+JWlzLm5hJT4lYCFgJT4lc3VtKSklPiUKICBtdXRhdGUoc2U9c2Qvc3FydChuKSkKYGBgCgpJdCBpcyBhbHNvIHZlcnkgdXNlZnVsIHRvIGNhbGN1bGF0ZSB0aGUgZGVzY3JpcHRpdmUgc3RhdGlzdGljcyBmb3IgdGhlIHN5c3RvbGljIGJsb29kIHByZXNzdXJlIGRpZmZlcmVuY2UgYmVjYXVzZSB3ZSBjYW4gYXNzZXNzIHRoZSBlZmZlY3Qgc2l6ZSBvZiBhZG1pbmlzdGVyaW5nIGNhcHRvcHJpbCBmb3IgZXZlcnkgcGF0aWVudC4gCgpgYGB7cn0KY2FwdG9wcmlsICU+JSAKICBtdXRhdGUoYnBfZGlmZiA9IFNCUGEtU0JQYikgJT4lCiAgICAgc3VtbWFyaXplX2F0KCJicF9kaWZmIiwKICAgICAgICAgICAgICAgbGlzdChtZWFuPX5tZWFuKC4sbmEucm09VFJVRSksCiAgICAgICAgICAgICAgICAgICAgc2Q9fnNkKC4sbmEucm09VFJVRSksCiAgICAgICAgICAgICAgICAgICAgbj1mdW5jdGlvbih4KSB4JT4laXMubmElPiVgIWAlPiVzdW0pKSAlPiUKICBtdXRhdGUoc2U9c2Qvc3FydChuKSkKYGBgCgoKIyBXcmFwLXVwIAoKTmV4dCwgdGltZSBpZiB5b3UgZXhwbG9yZSBwYWlyZWQgZGF0YSwgeW91IGNhbiBpbW1lZGlhdGVseSBhZG9wdCB0aGUgbGluZSBwbG90IGFuZCB0aGUgYm94cGxvdCBvZiB0aGUgZGlmZmVyZW5jZXMgZnJvbSBTZWN0aW9uIDMuMy4gc28gYXMgdG8gYWNrbm93bGVkZ2UgdGhlIHBhaXJpbmcgYW5kIHBlcmZvcm0gdGhlIGRlc2NyaXB0aXZlIHN0YXRpc3RpY3MgaW4gc2VjdGlvbiAzLjQuIApJdCBtaWdodCBhbHNvIGJlIHVzZWZ1bCB0byBwbG90IGJveHBsb3RzIG9mIHRoZSByYXcgYmxvb2QgcHJlc3N1cmUgbWVhc3VyZW1lbnRzIHRvIGdldCBhbiBpZGVhIG9uIHRoZWlyIHZhcmFiaWxpdHkuCgpJdCBpcyBub3QgdXNlZnVsIHRvIHJlcG9ydCB0aGUgYmFycGxvdC4gCgojIyBBZHZhbnRhZ2Ugb2YgcGFpcmVkIGRlc2lnbgoKRXZlcnkgcGF0aWVudCBhY3RzIGFzIHRoZWlyIG93biBjb250cm9sOiB3ZSBjYW4gZXN0aW1hdGUgdGhlIGVmZmVjdCBvZiB0aGUgdHJlYXRtZW50IApmb3IgZWFjaCBwYXRpZW50IAoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2dwbG90KGFlcyhTQlBiLFNCUGEpKSArCiAgZ2VvbV9wb2ludCgpICsgCiAgZ2d0aXRsZShwYXN0ZSgiY29ycmVsYXRpb24gPSAiLHJvdW5kKGNvcihjYXB0b3ByaWwkU0JQYixjYXB0b3ByaWwkU0JQYSksMykpKQpgYGAKCmBgYHtyfQpjYXB0b3ByaWwgJT4lIAogIG11dGF0ZShicF9kaWZmID0gU0JQYS1TQlBiKSAlPiUKICBzZWxlY3QoU0JQYixTQlBhLGJwX2RpZmYpICU+JQogIHN1bW1hcml6ZV9hbGwoc2QpCmBgYAoKQ2FsY3VsYXRlIHZhcmlhbmNlIGJhc2VkIG9uIGZhY3QgdGhhdCBkYXRhIGFyZSBwYWlyZWQuCgpgYGB7cn0KY2FwdG9wcmlsICU+JSBzdW1tYXJpc2Uoc2RfZGlmZj1zcXJ0KHNkKFNCUGIpXjIgKyBzZChTQlBhKV4yLTIqc2QoU0JQYSkqc2QoU0JQYikqY29yKFNCUGEsU0JQYikpKQpgYGAKCk5vdGUsIHRoYXQgdGhlIHN0YW5kYXJkIGRldmlhdGlvbnMgb24gdGhlIGJsb29kIHByZXNzdXJlIG1lYXN1cmVtZW50cyBhcmUgbW9yZSB0aGFuCnR3aWNlIGFzIGxhcmdlIGFzIHRoZSBzdGFuZGFyZCBkZXZpYXRpb25zIG9uIHRoZSBkaWZmZXJlbmNlLiAKVGhpcyBpcyBiZWNhdXNlIGVhY2ggcGF0aWVudCBhY3RzIGFzIHRoZWlyIG93biBjb250cm9sLiBBcyBzdWNoLCB3ZSBjYW4gY29ycmVjdCBmb3IKdGhlaXIgYmFzZWxpbmUgYmxvb2QgcHJlc3N1cmUgd2hlbiBlc3RpbWF0aW5nIHRoZSBlZmZlY3Qgb2YgdGhlIHRyZWF0bWVudC4KCi0tLQoKIyBbSG9tZV0oaHR0cHM6Ly9ndHBiLmdpdGh1Yi5pby9QU0xTMjAvKSB7LX0K