1 Comparison of \(g\) groups

  • Extend \(F\)-test from a one-way ANOVA to non-parametric alternatives.

2 DMH Voorbeeld

Assess genotoxicity of 1,2-dimethylhydrazine dihydrochloride (DMH) (EU directive)

  • 24 rats
  • four groups with daily DMH dose
  • control
  • low
  • medium
  • high

  • Genotoxicity in liver using comet assay on 150 liver cells per rat.
  • Are there differences in DNA damage due to DMH dose?

2.1 Comet Assay:

  • Visualise DNA strand breaks
  • Length comet tail is a proxy for strand breaks.
Comet assay

Comet assay

dna <- read_delim("https://raw.githubusercontent.com/GTPB/PSLS20/master/data/dna.txt",delim=" ")
dna$dose <- as.factor(dna$dose)
dna
# A tibble: 24 x 3
   id    length dose 
   <chr>  <dbl> <fct>
 1 Rat1    19.3 0    
 2 Rat2    18.9 0    
 3 Rat3    18.6 0    
 4 Rat4    19.0 0    
 5 Rat5    17.1 0    
 6 Rat6    18.8 0    
 7 Rat7    55.4 1.25 
 8 Rat8    59.2 1.25 
 9 Rat9    59.1 1.25 
10 Rat10   52.1 1.25 
# ... with 14 more rows
dna %>% 
  ggplot(aes(x=dose,y=length,fill=dose))+
  geom_boxplot() +
  geom_point(position="jitter")

dna %>%
  ggplot(aes(sample=length)) +
  geom_qq() + 
  geom_qq_line() +
  facet_wrap(~dose)

  • Strong indication that data in control group has a lower variance.
  • 6 observations per group are too few to check the assumptions
plot(lm(length~dose,data=dna))

3 Kruskal-Wallis Rank Test

  • The Kruskal-Wallis Rank Test (KW-test) is a non-parameteric alternative for ANOVA F-test.

  • Classical \(F\)-teststatistic can be written as \[ F = \frac{\text{SST}/(g-1)}{\text{SSE}/(n-g)} = \frac{\text{SST}/(g-1)}{(\text{SSTot}-\text{SST})/(n-g)} , \]
  • with \(g\) the number of groups.

  • SSTot depends only on outcomes \(\mathbf{y}\) and will not vary in permutation test.

  • SST can be used as statistic \[\text{SST}=\sum_{j=1}^t n_j(\bar{Y}_j-\bar{Y})^2\]

  • The KW test statistic is based on SST on rank-transformed outcomes1, \[ \text{SST} = \sum_{j=1}^g n_j \left(\bar{R}_j - \bar{R}\right)^2 = \sum_{j=1}^t n_j \left(\bar{R}_j - \frac{n+1}{2}\right)^2 , \]
  • with \(\bar{R}_j\) the mean of the ranks in group \(j\), and \(\bar{R}\) the mean of all ranks, \[ \bar{R} = \frac{1}{n}(1+2+\cdots + n) = \frac{1}{n}\frac{1}{2}n(n+1) = \frac{n+1}{2}. \]
  • The KW teststatistic is given by \[ KW = \frac{12}{n(n+1)} \sum_{j=1}^g n_j \left(\bar{R}_j - \frac{n+1}{2}\right)^2. \]
  • The factor \(\frac{12}{n(n+1)}\) is used so that \(KW\) has a simple asymptotic null distribution. In particular under \(H_0\), given thart \(\min(n_1,\ldots, n_g)\rightarrow \infty\), \[ KW \rightarrow \chi^2_{t-1}. \]

  • The exact KW-test can be executed by calculating the permutation null distribution (that only depends on \(n_1, \ldots, n_g\)) to test \[H_0: f_1=\ldots=f_g \text{ vs } H_1: \text{ at least two means are different}.\]

  • In order to allow \(H_1\) to be formulated in terms of means, the assumption of locations shift should be valid.
  • For DMH example this is not the case.
  • If location-shift is invalid, we have to formulate \(H_1\) in terms of probabilistic indices: \[H_0: f_1=\ldots=f_g \text{ vs } H_1: \exists\ j,k \in \{1,\ldots,g\} : \text{P}\left[Y_j\geq Y_k\right]\neq 0.5\]

3.1 DNA Damage Example

kruskal.test(length~dose,data=dna)

    Kruskal-Wallis rank sum test

data:  length by dose
Kruskal-Wallis chi-squared = 14, df = 3, p-value = 0.002905
  • On the \(5\%\) level of significance we can reject the null hypothesis.

  • R-functie kruskal.test only returns the asymptotic approximation for \(p\)-values.

  • With only 6 observaties per groep, this is not a good approximation of the \(p\)-value

  • With the coin R package we can calculate the exacte \(p\)-value

library(coin)
kwPerm <- kruskal_test(length~dose,data=dna,
            distribution=approximate(B=100000))
kwPerm

    Approximative Kruskal-Wallis Test

data:  length by dose (0, 1.25, 2.5, 5)
chi-squared = 14, p-value = 0.00039
  • We conclude that the difference in the distribution of the DNA damages due to the DMH dose is extremely significantly different.

  • Posthoc analysis with WMW tests.

pairwise.wilcox.test(dna$length,dna$dose)

    Pairwise comparisons using Wilcoxon rank sum test 

data:  dna$length and dna$dose 

     0     1.25  2.5  
1.25 0.013 -     -    
2.5  0.013 0.818 -    
5    0.013 0.721 0.788

P value adjustment method: holm 
  • All DMH behandelingen are significantly different from the control.
  • The DMH are not significantly different from one another.
  • U1 does not occur in the pairwise.wilcox.test output. Point estimate on probability on higher DNA-damage?
nGroup <- table(dna$dose)
probInd <- combn(levels(dna$dose),2,function(x) 
         {
         test <- wilcox.test(length~dose,subset(dna,dose%in%x))
         return(test$statistic/prod(nGroup[x]))
         }
         )
names(probInd) <- combn(levels(dna$dose),2,paste,collapse="vs")
probInd
  0vs1.25    0vs2.5      0vs5 1.25vs2.5   1.25vs5    2.5vs5 
0.0000000 0.0000000 0.0000000 0.4444444 0.2777778 0.3333333 

Because there are doubts on the location-shift model we draw our conclusions in terms of the probabilistic index.

3.1.1 Conclusion

  • There is an extremely significant difference in in the distribution of the DNA-damage measurements due to the treatment with DMH (\(p<0.001\) KW-test).
  • DNA-damage is more likely upon DMH treatment than in the control treatment (all p=0.013, WMW-testen).
  • The probability on higher DNA-damage upon exposure to DMH is 100% (Calculation of a CI on the probabilistic index is beyond the scope of the course)
  • There are no significant differences in the distributions of the comit-lengths among the treatment with the different DMH concentrations (\(p=\) 0.72-0.82).
  • DMH shows already genotoxic effects at low dose.
  • (Alle paarswise tests are gecorrected for multiple testing using Holm’s methode).


  1. we assume that no ties are available

LS0tCnRpdGxlOiAiOS4gTm9uLXBhcmFtZXRyaWMgc3RhdGlzdGljcyAtIEtydXNrYWwgV2FsbGlzIiAgCmF1dGhvcjogIkxpZXZlbiBDbGVtZW50IgpkYXRlOiAic3RhdE9taWNzLCBHaGVudCBVbml2ZXJzaXR5IChodHRwczovL3N0YXRvbWljcy5naXRodWIuaW8pIgpvdXRwdXQ6CiAgICBodG1sX2RvY3VtZW50OgogICAgICBjb2RlX2Rvd25sb2FkOiB0cnVlICAgIAogICAgICB0aGVtZTogY29zbW8KICAgICAgdG9jOiB0cnVlCiAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICBoaWdobGlnaHQ6IHRhbmdvCiAgICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoaW5jbHVkZSA9IFRSVUUsIGNvbW1lbnQgPSBOQSwgZWNobyA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgICBtZXNzYWdlID0gRkFMU0UsIHdhcm5pbmcgPSBGQUxTRSkKbGlicmFyeSh0aWR5dmVyc2UpCnNldC5zZWVkKDE0MCkKYGBgCgoKIyBDb21wYXJpc29uIG9mICRnJCBncm91cHMKCi0gRXh0ZW5kICAkRiQtdGVzdCBmcm9tIGEgb25lLXdheSBBTk9WQSB0byBub24tcGFyYW1ldHJpYyBhbHRlcm5hdGl2ZXMuCgojIERNSCBWb29yYmVlbGQKCkFzc2VzcyBnZW5vdG94aWNpdHkgb2YgMSwyLWRpbWV0aHlsaHlkcmF6aW5lIGRpaHlkcm9jaGxvcmlkZSAoRE1IKSAgKEVVIGRpcmVjdGl2ZSkKCi0gMjQgcmF0cyAKLSBmb3VyIGdyb3VwcyB3aXRoIGRhaWx5IERNSCBkb3NlCiAgLSBjb250cm9sCiAgLSBsb3cKICAtIG1lZGl1bSAKICAtIGhpZ2gKCi0gR2Vub3RveGljaXR5IGluIGxpdmVyIHVzaW5nIGNvbWV0IGFzc2F5IG9uIDE1MCBsaXZlciBjZWxscyBwZXIgcmF0LgotIEFyZSB0aGVyZSBkaWZmZXJlbmNlcyBpbiBETkEgZGFtYWdlIGR1ZSB0byBETUggZG9zZT8KCiMjIENvbWV0IEFzc2F5OgoKLSBWaXN1YWxpc2UgRE5BIHN0cmFuZCBicmVha3MKLSBMZW5ndGggY29tZXQgdGFpbCBpcyBhIHByb3h5IGZvciBzdHJhbmQgYnJlYWtzLiAKCiFbQ29tZXQgYXNzYXldKGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9HVFBCL1BTTFMyMC9naC1wYWdlcy9hc3NldHMvZmlncy9jb21ldC5qcGcpeyB3aWR0aD01MCUgfQoKCmBgYHtyfQpkbmEgPC0gcmVhZF9kZWxpbSgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL0dUUEIvUFNMUzIwL21hc3Rlci9kYXRhL2RuYS50eHQiLGRlbGltPSIgIikKZG5hJGRvc2UgPC0gYXMuZmFjdG9yKGRuYSRkb3NlKQpkbmEKYGBgCgoKYGBge3J9CmRuYSAlPiUgCiAgZ2dwbG90KGFlcyh4PWRvc2UseT1sZW5ndGgsZmlsbD1kb3NlKSkrCiAgZ2VvbV9ib3hwbG90KCkgKwogIGdlb21fcG9pbnQocG9zaXRpb249ImppdHRlciIpCgpkbmEgJT4lCiAgZ2dwbG90KGFlcyhzYW1wbGU9bGVuZ3RoKSkgKwogIGdlb21fcXEoKSArIAogIGdlb21fcXFfbGluZSgpICsKICBmYWNldF93cmFwKH5kb3NlKQpgYGAKCi0gU3Ryb25nIGluZGljYXRpb24gdGhhdCBkYXRhIGluIGNvbnRyb2wgZ3JvdXAgaGFzIGEgbG93ZXIgdmFyaWFuY2UuCi0gNiBvYnNlcnZhdGlvbnMgcGVyIGdyb3VwIGFyZSB0b28gZmV3IHRvIGNoZWNrIHRoZSBhc3N1bXB0aW9ucyAKCmBgYHtyfQpwbG90KGxtKGxlbmd0aH5kb3NlLGRhdGE9ZG5hKSkKYGBgCgojIEtydXNrYWwtV2FsbGlzIFJhbmsgVGVzdAoKLSBUaGUgS3J1c2thbC1XYWxsaXMgUmFuayBUZXN0IChLVy10ZXN0KSBpcyBhICBub24tcGFyYW1ldGVyaWMgYWx0ZXJuYXRpdmUgIGZvciBBTk9WQSBGLXRlc3QuICAKICAKLSAgQ2xhc3NpY2FsICRGJC10ZXN0c3RhdGlzdGljIGNhbiBiZSB3cml0dGVuIGFzCiAgXFsKICAgIEYgPSBcZnJhY3tcdGV4dHtTU1R9LyhnLTEpfXtcdGV4dHtTU0V9LyhuLWcpfSA9IFxmcmFje1x0ZXh0e1NTVH0vKGctMSl9eyhcdGV4dHtTU1RvdH0tXHRleHR7U1NUfSkvKG4tZyl9ICwKICBcXQotICB3aXRoICRnJCB0aGUgbnVtYmVyIG9mIGdyb3Vwcy4gCiAKLSBTU1RvdCBkZXBlbmRzIG9ubHkgb24gIG91dGNvbWVzICRcbWF0aGJme3l9JCBhbmQgd2lsbCBub3QgdmFyeSBpbiBwZXJtdXRhdGlvbiB0ZXN0LiAKCi0gU1NUIGNhbiBiZSB1c2VkIGFzIHN0YXRpc3RpYwogJCRcdGV4dHtTU1R9PVxzdW1fe2o9MX1edCBuX2ooXGJhcntZfV9qLVxiYXJ7WX0pXjIkJAoKCi0gIFRoZSBLVyB0ZXN0IHN0YXRpc3RpYyBpcyBiYXNlZCBvbiBTU1Qgb24gcmFuay10cmFuc2Zvcm1lZCBvdXRjb21lc15bd2UgYXNzdW1lIHRoYXQgbm8gKnRpZXMqIGFyZSBhdmFpbGFibGVdLAogIFxbCiAgICAgXHRleHR7U1NUfSA9IFxzdW1fe2o9MX1eZyBuX2ogXGxlZnQoXGJhcntSfV9qIC0gXGJhcntSfVxyaWdodCleMiA9IFxzdW1fe2o9MX1edCBuX2ogXGxlZnQoXGJhcntSfV9qIC0gXGZyYWN7bisxfXsyfVxyaWdodCleMiAsCiAgXF0KLSAgd2l0aCAkXGJhcntSfV9qJCB0aGUgbWVhbiBvZiB0aGUgcmFua3MgaW4gZ3JvdXAgJGokLCBhbmQgJFxiYXJ7Un0kIHRoZSBtZWFuIG9mIGFsbCByYW5rcywKICBcWwogICAgXGJhcntSfSA9IFxmcmFjezF9e259KDErMitcY2RvdHMgKyBuKSA9IFxmcmFjezF9e259XGZyYWN7MX17Mn1uKG4rMSkgPSBcZnJhY3tuKzF9ezJ9LgogIFxdCi0gIFRoZSBLVyB0ZXN0c3RhdGlzdGljIGlzIGdpdmVuIGJ5CiAgXFsKICAgIEtXID0gXGZyYWN7MTJ9e24obisxKX0gIFxzdW1fe2o9MX1eZyBuX2ogXGxlZnQoXGJhcntSfV9qIC0gXGZyYWN7bisxfXsyfVxyaWdodCleMi4KICBcXQotICBUaGUgZmFjdG9yICRcZnJhY3sxMn17bihuKzEpfSQgaXMgdXNlZCBzbyB0aGF0ICRLVyQgaGFzIGEgc2ltcGxlIGFzeW1wdG90aWMgbnVsbCBkaXN0cmlidXRpb24uIEluIHBhcnRpY3VsYXIgdW5kZXIgJEhfMCQsIGdpdmVuIHRoYXJ0ICRcbWluKG5fMSxcbGRvdHMsIG5fZylccmlnaHRhcnJvdyBcaW5mdHkkLCAKICBcWwogICAgS1cgIFxyaWdodGFycm93IFxjaGleMl97dC0xfS4KICBcXQoKLSAgVGhlIGV4YWN0IEtXLXRlc3QgY2FuIGJlIGV4ZWN1dGVkIGJ5IGNhbGN1bGF0aW5nIHRoZSBwZXJtdXRhdGlvbiBudWxsIGRpc3RyaWJ1dGlvbiAodGhhdCBvbmx5IGRlcGVuZHMgb24gJG5fMSwgXGxkb3RzLCBuX2ckKSB0byB0ZXN0CiAgJCRIXzA6IGZfMT1cbGRvdHM9Zl9nIFx0ZXh0eyB2cyB9IEhfMTogXHRleHR7IGF0IGxlYXN0IHR3byBtZWFucyBhcmUgZGlmZmVyZW50fS4kJCAKICAKLSBJbiBvcmRlciB0byBhbGxvdyAkSF8xJCB0byBiZSBmb3JtdWxhdGVkIGluIHRlcm1zIG9mIG1lYW5zLCB0aGUgYXNzdW1wdGlvbiBvZiBsb2NhdGlvbnMgc2hpZnQgc2hvdWxkIGJlIHZhbGlkLgotIEZvciBETUggZXhhbXBsZSB0aGlzIGlzIG5vdCB0aGUgY2FzZS4gICAKLSBJZiBsb2NhdGlvbi1zaGlmdCBpcyBpbnZhbGlkLCB3ZSBoYXZlIHRvIGZvcm11bGF0ZSAkSF8xJCBpbiB0ZXJtcyBvZiBwcm9iYWJpbGlzdGljIGluZGljZXM6CiAgJCRIXzA6IGZfMT1cbGRvdHM9Zl9nIFx0ZXh0eyB2cyB9IEhfMTogXGV4aXN0c1wgaixrIFxpbiBcezEsXGxkb3RzLGdcfSA6IFx0ZXh0e1B9XGxlZnRbWV9qXGdlcSBZX2tccmlnaHRdXG5lcSAwLjUkJAoKCiMjIEROQSBEYW1hZ2UgRXhhbXBsZQoKYGBge3J9CmtydXNrYWwudGVzdChsZW5ndGh+ZG9zZSxkYXRhPWRuYSkKYGBgCgotIE9uIHRoZSAkNVwlJCBsZXZlbCBvZiBzaWduaWZpY2FuY2Ugd2UgY2FuIHJlamVjdCB0aGUgbnVsbCBoeXBvdGhlc2lzLiAKICAKLSBSLWZ1bmN0aWUgYGtydXNrYWwudGVzdGAgb25seSByZXR1cm5zIHRoZSBhc3ltcHRvdGljIGFwcHJveGltYXRpb24gZm9yICRwJC12YWx1ZXMuIAoKLSBXaXRoIG9ubHkgNiBvYnNlcnZhdGllcyBwZXIgZ3JvZXAsIHRoaXMgaXMgbm90IGEgZ29vZCBhcHByb3hpbWF0aW9uIG9mIHRoZSAkcCQtdmFsdWUKCi0gIFdpdGggdGhlIGBjb2luYCBSIHBhY2thZ2Ugd2UgY2FuIGNhbGN1bGF0ZSB0aGUgZXhhY3RlICRwJC12YWx1ZQoKYGBge3Isd2FybmluZz1GQUxTRSxtZXNzYWdlPUZBTFNFfQpsaWJyYXJ5KGNvaW4pCmt3UGVybSA8LSBrcnVza2FsX3Rlc3QobGVuZ3RofmRvc2UsZGF0YT1kbmEsCgkJICAgIGRpc3RyaWJ1dGlvbj1hcHByb3hpbWF0ZShCPTEwMDAwMCkpCmt3UGVybQpgYGAKCi0gV2UgY29uY2x1ZGUgdGhhdCB0aGUgZGlmZmVyZW5jZSBpbiB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSBETkEgZGFtYWdlcyBkdWUgdG8gdGhlIERNSCBkb3NlIGlzIGV4dHJlbWVseSBzaWduaWZpY2FudGx5IGRpZmZlcmVudC4gCgotIFBvc3Rob2MgYW5hbHlzaXMgd2l0aCBXTVcgdGVzdHMuIAoKYGBge3J9CnBhaXJ3aXNlLndpbGNveC50ZXN0KGRuYSRsZW5ndGgsZG5hJGRvc2UpCmBgYAoKLSBBbGwgRE1IIGJlaGFuZGVsaW5nZW4gYXJlIHNpZ25pZmljYW50bHkgZGlmZmVyZW50IGZyb20gdGhlIGNvbnRyb2wuIAotIFRoZSBETUggYXJlIG5vdCBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBmcm9tIG9uZSBhbm90aGVyLiAgCi0gVTEgZG9lcyBub3Qgb2NjdXIgaW4gdGhlIGBwYWlyd2lzZS53aWxjb3gudGVzdGAgb3V0cHV0LiBQb2ludCBlc3RpbWF0ZSBvbiBwcm9iYWJpbGl0eSBvbiBoaWdoZXIgRE5BLWRhbWFnZT8KCmBgYHtyLCBlY2hvPUZBTFNFfQpwYWlyV2lsY294IDwtIHBhaXJ3aXNlLndpbGNveC50ZXN0KGRuYSRsZW5ndGgsZG5hJGRvc2UpCmBgYAoKYGBge3J9Cm5Hcm91cCA8LSB0YWJsZShkbmEkZG9zZSkKcHJvYkluZCA8LSBjb21ibihsZXZlbHMoZG5hJGRvc2UpLDIsZnVuY3Rpb24oeCkgCgkJIHsKCQkgdGVzdCA8LSB3aWxjb3gudGVzdChsZW5ndGh+ZG9zZSxzdWJzZXQoZG5hLGRvc2UlaW4leCkpCgkJIHJldHVybih0ZXN0JHN0YXRpc3RpYy9wcm9kKG5Hcm91cFt4XSkpCgkJIH0KCQkgKQpuYW1lcyhwcm9iSW5kKSA8LSBjb21ibihsZXZlbHMoZG5hJGRvc2UpLDIscGFzdGUsY29sbGFwc2U9InZzIikKcHJvYkluZApgYGAKCkJlY2F1c2UgdGhlcmUgYXJlIGRvdWJ0cyBvbiB0aGUgbG9jYXRpb24tc2hpZnQgbW9kZWwgd2UgZHJhdyBvdXIgY29uY2x1c2lvbnMgaW4gdGVybXMgb2YgdGhlIHByb2JhYmlsaXN0aWMgaW5kZXguIAoKIyMjIENvbmNsdXNpb24KCi0gVGhlcmUgaXMgYW4gZXh0cmVtZWx5IHNpZ25pZmljYW50IGRpZmZlcmVuY2UgaW4gaW4gdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgRE5BLWRhbWFnZSBtZWFzdXJlbWVudHMgZHVlIHRvIHRoZSB0cmVhdG1lbnQgd2l0aCBETUggICgkcDwwLjAwMSQgS1ctdGVzdCkuCi0gRE5BLWRhbWFnZSBpcyBtb3JlIGxpa2VseSB1cG9uIERNSCB0cmVhdG1lbnQgdGhhbiBpbiB0aGUgY29udHJvbCB0cmVhdG1lbnQgKGFsbCBwPTAuMDEzLCBXTVctdGVzdGVuKS4gCi0gVGhlIHByb2JhYmlsaXR5IG9uIGhpZ2hlciBETkEtZGFtYWdlIHVwb24gZXhwb3N1cmUgdG8gRE1IIGlzIDEwMCUgKENhbGN1bGF0aW9uIG9mIGEgQ0kgb24gdGhlIHByb2JhYmlsaXN0aWMgaW5kZXggaXMgYmV5b25kIHRoZSBzY29wZSBvZiB0aGUgY291cnNlKSAKLSBUaGVyZSBhcmUgbm8gc2lnbmlmaWNhbnQgZGlmZmVyZW5jZXMgaW4gdGhlIGRpc3RyaWJ1dGlvbnMgb2YgdGhlIGNvbWl0LWxlbmd0aHMgYW1vbmcgdGhlIHRyZWF0bWVudCB3aXRoIHRoZSBkaWZmZXJlbnQgRE1IIGNvbmNlbnRyYXRpb25zICgkcD0kIGByIHBhc3RlKGZvcm1hdChyYW5nZShwYWlyV2lsY294JHAudmFsdWVbLC0xXSxuYS5ybT1UUlVFKSxkaWdpdD0yKSxjb2xsYXBzZT0iLSIpYCkuIAotIERNSCBzaG93cyBhbHJlYWR5IGdlbm90b3hpYyBlZmZlY3RzIGF0IGxvdyBkb3NlLiAKLSAoQWxsZSBwYWFyc3dpc2UgdGVzdHMgYXJlIGdlY29ycmVjdGVkIGZvciBtdWx0aXBsZSB0ZXN0aW5nIHVzaW5nIEhvbG0ncyBtZXRob2RlKS4KCgotLS0KCiMgW0hvbWVdKGh0dHBzOi8vZ3RwYi5naXRodWIuaW8vUFNMUzIwLykgey19Cg==