class: center, middle, inverse, title-slide # Explanatory Methods I ## Statistical Inference and Linear Regression ### <b>rstudio::</b>conf(2022) --- class: left, middle, rstudio-logo, bigfont ## Aims of this module &#9989; Understand the concept of statistical inference - Briefly review some important statistical concepts - Review an example of a statistical hypothesis test &#9989; Learn our first explanatory modeling method - Review simple and multiple Linear Regression - Learn some foundational concepts that apply to other model types --- class: left, middle, rstudio-logo # Statistical Inference --- class: left, middle, rstudio-logo ## Samples and Populations In the vast majority of situations when we apply statistics to a problem, we are being asked to draw a conclusion about a *population*, but we only have data on a *sample* subset of that population. What might be the sample and what might be the population in each of these problems? 1. A political election forecast 2. A market research survey for a grocery chain 3. An employer trying to understand if compensation levels may be a factor in employee retention No matter what we see in a sample, we can never be 100% certain that we would see the same in the population, but sometimes we can be certain enough to *infer* that we will see the same in the population. The mathematics behind this process is known as *statistical inference*. --- class: left, middle, rstudio-logo ## Let's look at an example Let's take the charity donation dataset that you worked on in the exercises in the last module, and let's determine the mean age of ten randomly selected people who made donations. Note that the age of a person is considered a *random variable*, in that each persons age is independently drawn from the same overall distribution. ```r url <- "https://peopleanalytics-regression-book.org/data/charity_donation.csv" charity_data <- read.csv(url) # select ages of ten people at random set.seed(123) # <- ensures we can reproduce random_ages <- sample(charity_data$age, 100) #take mean (mean_age <- mean(random_ages)) ``` ``` ## [1] 45.64 ``` So can we conclude that people who donate to this charity have a mean age of 45.64? --- class: left, middle, rstudio-logo ## Repeated sampling tells us about a pattern we can expect If we were to do this process 1000 times, and draw a density plot, this is what it would look like. You can see the code for this in the source of this document. <img src="2-inference_and_linear_regression_files/figure-html/unnamed-chunk-2-1.png" height="350" style="float:left; padding:10px" /> Our density plot shows the following properties: * Has the shape of a normal (Gaussian) distribution * Blue line as the mean value (the mean of the sample means) - most likely value for the population mean * Red dashed lines as +/- 1 standard deviations - 69% probability that population mean is in this range * Brown dashed lines as +/- 1.96 standard deviations - 95% probability that population mean is in this range --- class: left, middle, rstudio-logo ## This observation leads to some important concepts 1. The expected mean over many samples of a random variable is itself a random variable, with a mean value and a distribution around that mean. 2. Given this distribution, we call a standard deviation above or below the expected mean of a random variable a *standard error (SE)* for the population mean 3. We call a probability range around the expected mean of a random variable a *confidence interval* for the population mean. This confidence interval corresponds to specified multiples of standard errors. As a consequence of this, if we know the mean of our sample, we can use the expected distribution of that mean to determine the standard error and the confidence interval. Often as a 'rule of thumb', practitioners will regard +/- 2 standard errors as the 95% confidence interval. --- class: left, middle, rstudio-logo ## Hypothesis testing Now imagine someone asks us a question about differences between populations. For example: do men and women that donate to our charity have a different average age? We do not have the data for all men and women that donate, but we do have the data from our sample dataset. Let's calculate the mean age of men and of women in that data set. ```r age_m <- charity_data$age[charity_data$gender == "M"] mean(age_m) ``` ``` ## [1] 46.49007 ``` ```r age_f <- charity_data$age[charity_data$gender == "F"] mean(age_f) ``` ``` ## [1] 47.37438 ``` So in our sample there is an age difference of about 0.9 years on average. We now need to determine if that age difference is large enough to infer that there is a difference in the average age of all men who donate vs all women who donate. --- class: left, middle, rstudio-logo ## Process of statistically testing a hypothesis of difference First we note that the difference of two random variables is a random variable, so in this case we can expect the difference between the mean age of men and the mean age of women to obey an expected sampling distribution with a standard error and a 95% confidence interval for the population value. So we can use the following process to test our whether we can infer that there is a difference: * Assume as a starting point that the difference in the population is actually zero. This is called the *null hypothesis*. * Use the sample to calculate the expected mean, standard error and 95% confidence interval for the difference between the mean ages in the population. * Determine whether or not zero lies inside or outside the 95% confidence interval. If it lies outside, this means that there is less than a 5% chance that this sample would occur if the null hypothesis were true. * If there is less than a 5% chance that this sample would occur if the null hypothesis were true, we infer that the null hypothesis should be rejected and we conclude the *alternative hypothesis*. * Otherwise we do not infer that the null hypothesis can be rejected. --- class: left, middle, rstudio-logo ## Welch's `\(t\)`-test of difference in means Welch's `\(t\)`-test will perform all of these steps for you. It will calculate the expected distribution of the difference in means based on your samples, it will compute the 95% confidence interval, and it will tell you where zero lies on the distribution. In this case it tells us that we cannot infer a rejection of the null hypothesis. ```r t.test(age_m, age_f) ``` ``` ## ## Welch Two Sample t-test ## ## data: age_m and age_f ## t = -0.53486, df = 339.75, p-value = 0.5931 ## alternative hypothesis: true difference in means is not equal to 0 ## 95 percent confidence interval: ## -4.136438 2.367802 ## sample estimates: ## mean of x mean of y ## 46.49007 47.37438 ``` --- class: left, middle, rstudio-logo ## `\(\alpha\)` and the `\(p\)`-value The `\(p\)`-value returned by a hypothesis test represents the likelihood of seeing this sample difference or a larger difference if the null hypothesis were true. <img src="2-inference_and_linear_regression_files/figure-html/unnamed-chunk-6-1.png" height="350" style="float:left; padding:10px" /> * The red dashed lines represent the points at which the difference is zero (first line is F - M, second line is M - F) * This splits the density curve into 3 segments. * The total area of A + C represents the `\(p\)`-value - it is the likelihood that samples with this difference or a larger difference would occur assuming there was a zero difference in the population. * If this `\(p\)`-value is less than a specified standard known as `\(\alpha\)`, we can infer a rejection of the null hypothesis. Usually `\(\alpha = 0.05\)`. --- class: left, middle, rstudio-logo ## Exercise - Hypothesis testing for a difference in means For our next short exercise, we will do some practice on running a hypothesis test for a difference in means. Go to our [RStudio Cloud workspace](https://rstudio.cloud/spaces/230780/join?access_code=7cXJKFU1KUuuZGLwBVQpLG3dIxPUD3jak3ZQmESh) and start **Assignment 02A - Statistical Inference**. Let's work on **Exercises 1 and 2**. --- class: left, middle, rstudio-logo ## Using regression models for statistical inference Using similar principles to the simple hypothesis test we have just seen, we use regression modeling techniques on samples in order to make inferences about populations. In most modeling scenarios we will have the following at the outset: 1. A sample `\(S\)` of observations taken from a population `\(P\)` 2. A dataset containing data for our sample set `\(S\)`. The data includes a set of input variables `\(x_1, x_2, ..., x_p\)` and an outcome variable `\(y\)`. 3. A model type to relate `\(x_1, x_2, ..., x_p\)` to `\(y\)`. We select this model type for the most part based on the scale of our outcome data `\(y\)`. We will use our sample dataset to estimate the parameters for the model type we have selected, and we will use these parameters to make inferences about the following: 1. Which input variables influence the outcome for the population? 2. To what degree does each input variable influence the outcome? 3. How much of the outcome is explained by all our input variables? --- class: left, middle, rstudio-logo ## Choosing your model type Regression model types are usually chosen based on the scale of the outcome variable `\(y\)`. <table class=" lightable-minimal" style='font-family: "Trebuchet MS", verdana, sans-serif; margin-left: auto; margin-right: auto;'> <thead> <tr> <th style="text-align:left;"> Outcome </th> <th style="text-align:left;"> Model </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;background-color: lightblue !important;"> Continuous scale (eg money, height, weight) </td> <td style="text-align:left;background-color: lightblue !important;"> Linear regression </td> </tr> <tr> <td style="text-align:left;"> Binary scale (Yes/No) </td> <td style="text-align:left;"> Binomial logistic regression </td> </tr> <tr> <td style="text-align:left;"> Nominal category scale (eg A, B, C) </td> <td style="text-align:left;"> Multinomial logistic regression </td> </tr> <tr> <td style="text-align:left;"> Ordinal category scale (eg Low, Medium, High) </td> <td style="text-align:left;"> Ordinal logistic regression </td> </tr> <tr> <td style="text-align:left;"> Time dependent binary scale </td> <td style="text-align:left;"> Survival/proportional hazard regression </td> </tr> </tbody> </table> --- class: left, middle, rstudio-logo # Linear regression modeling --- class: left, middle, rstudio-logo ## Equation of a straight line *Linear regression* assumes that the mean of the outcome variable `\(y\)` over many observations has a straight line relationship with the input variables `\(x_1, x_2, ..., x_p\)`. You may remember the equation of a straight line from high school: $$ y = mx + c $$ where `\(m\)` is the gradient of the line and `\(c\)` is the `\(y\)`-intercept. <img src="2-inference_and_linear_regression_files/figure-html/unnamed-chunk-8-1.png" width="300" height="300" style="display: block; margin: auto;" /> --- class: left, middle, rstudio-logo ## Simple linear regression *Simple linear regression* refers to the case where there is only one input variable and only one outcome variable. This is an easy technique to visualize. Given a set of data, we plot the `\(x\)` and `\(y\)` values, and then we find the *line of best fit*. Let's imagine doing this for fifty observations of the examination scores of some students on a four year progam, with our input variable `\(x\)` being their score in Year 3 (out of a total of 200) and our outcome variable `\(y\)` being their Final year score (out of a total of 300). <img src="2-inference_and_linear_regression_files/figure-html/unnamed-chunk-9-1.png" width="300" height="300" style="float:left; padding:10px" style="display: block; margin: auto;" /> The line of best fit for this sample has a gradient of 0.54 and an intercept of 84.58. Therefore, using our sample to model for the relationship between Year 3 and Final year test scores, we can conclude: * Students who scored zero in Year 3 will score 84.58 in their final year on average. * Every point scored in Year 3 is associated with 0.54 points scored in the final year on average. --- class: left, middle, rstudio-logo ## Inferring from the sample to the population Now we want to infer this relationship from our sample to all students (past, present and future). As we learned earlier in this module, there will be uncertainty about what the gradient and the intercept will be for the population, so our line of best fit will have a 95% confidence interval for the intercept and for the gradient. <img src="2-inference_and_linear_regression_files/figure-html/unnamed-chunk-10-1.png" width="300" height="300" style="float:left; padding:10px" style="display: block; margin: auto;" /> Therefore, we can infer the following for all students (past, present and future) with 95% confidence: * Students who score zero in Year 3 will score between 52.8 and 116.36 on average in their final year. * Every point scored in Year 3 is associated with between 0.24 and 0.83 points on average in the final year. --- class: left, middle, rstudio-logo ## Estimating the line of best fit in R Let's estimate our model using the full set of sample data instead of just 50 data points. In general, to estimate any model in R, you need (at a minimum) a model function, some data and a formula. In this case: * Model is linear model - `lm()` function in R * Data is `ugtests` * Formula takes the form `Outcome ~ Input`, in this case `Final ~ Yr3` for a simple regression ```r # get data url <- "https://peopleanalytics-regression-book.org/data/ugtests.csv" ugtests <- read.csv(url) # estimate model and save model object model <- lm(data = ugtests, formula = Final ~ Yr3) ``` --- class: left, middle, rstudio-logo ## Interpreting the coefficients The intercept and gradient are collectively known as the *coefficients* of the model. The model output in R is a named list with many components, one of which contains the estimates of the coefficients. ```r model$coefficients ``` ``` ## (Intercept) Yr3 ## 56.2668876 0.8817947 ``` We can be 95% certain that the gradient is non-zero if the confidence interval for `Yr3` does not contain zero. You can use the `confint()` function on your model to view this: ```r confint(model) ``` ``` ## 2.5 % 97.5 % ## (Intercept) 49.4184585 63.1153168 ## Yr3 0.8197207 0.9438687 ``` We can say that there is a *statistically significant* association between Year 3 scores and final year scores. We can use the estimated coefficient to illustrate the extent to which Year 3 scores influence final year scores. For example, *each point scored in Year 3 is associated with approximately 0.88 points in the final year*. --- class: left, middle, rstudio-logo ## Exercise - Simple linear regression For our next short exercise, we will do some practice on running a simple linear regression model and interpreting the coefficients. Go to our [RStudio Cloud workspace](https://rstudio.cloud/spaces/230780/join?access_code=7cXJKFU1KUuuZGLwBVQpLG3dIxPUD3jak3ZQmESh) and start **Assignment 02B - Linear regression**. Let's work on **Exercises 1 and 2**. --- class: left, middle, rstudio-logo ## Multiple regression *Multiple regression* refers to a situation when an outcome is being modeled against more than one input variable. Each input variable has its own gradient coefficient which helps determine the influence that input variable has on the outcome. For multiple linear regression, we assume: $$ y = \alpha + \beta_1x_1 + \beta_2x_2 + ... + \beta_px_p $$ where `\(\alpha\)` is the intercept and each `\(\beta_k\)` is the coefficient for input variable `\(x_k\)`. `\(\beta_k\)` represents the change in `\(y\)` associated with a unit increase in `\(x_k\)` **assuming no change in the other input variables**. --- class: left, middle, rstudio-logo ## Running a multiple linear regression model Let's relate our final year examination score to all previous years. We simply extend our formula using the `+` symbol to include multiple input variables. ```r full_model <- lm(data = ugtests, formula = Final ~ Yr1 + Yr2 + Yr3) full_model$coefficients ``` ``` ## (Intercept) Yr1 Yr2 Yr3 ## 14.14598945 0.07602621 0.43128539 0.86568123 ``` ```r confint(full_model) ``` ``` ## 2.5 % 97.5 % ## (Intercept) 3.39187185 24.9001071 ## Yr1 -0.05227936 0.2043318 ## Yr2 0.36749170 0.4950791 ## Yr3 0.80850142 0.9228610 ``` --- class: left, middle, rstudio-logo ## Interpreting a multiple regression model The key thing to remember is that every coefficient estimate assumes no change in the value of the other input variables. * Individuals with zero in all three previous years can expect a score of between 3.39 and 24.9 on average in the final year * For individuals with the same scores in Years 1 and 2, an additional point in Year 3 is associated with between 0.81 and 0.92 additional points on average in the final year * For individuals with the same scores in Years 1 and 3, an additional point in Year 2 is associated with between 0.37 and 0.5 additional points on average in the final year * For individuals with the same scores in Years 2 and 3, there is no significant association of Year 1 score with final year score on average --- class: left, middle, rstudio-logo ## Using categorical input variables If an input variable is categorical, you need to check that it has the right data type or the model will not understand it. ```r url <- "https://peopleanalytics-regression-book.org/data/sociological_data.csv" socio_data <- read.csv(url) str(socio_data) ``` ``` ## 'data.frame': 2618 obs. of 9 variables: ## $ annual_income_ppp: int 66417 55124 55124 54238 53573 52688 52245 51138 49810 49367 ... ## $ average_wk_hrs : int 50 50 50 50 50 51 51 51 51 51 ... ## $ education_months : int 157 156 155 154 150 147 143 141 137 134 ... ## $ region : chr "Southern Asia" "Southern Asia" "Southern Asia" "Southern Asia" ... ## $ job_type : chr "Unskilled" "Unskilled" "Unskilled" "Unskilled" ... ## $ gender : chr "F" "F" "F" "F" ... ## $ family_size : int 5 5 5 5 4 5 5 5 4 4 ... ## $ work_distance : int 3 0 3 0 0 0 0 0 2 0 ... ## $ languages : int 1 1 1 1 1 1 1 1 1 1 ... ``` If we want to use `region`, `job_type` or `gender` as input variables, we need to change them into factors. You can do this using `as.factor()`, but many modeling function in R will automatically determine if character string columns should be considered to be categorical, and will automatically convert the data type for you. --- class: left, middle, rstudio-logo ## Interpreting categorical input variables Your model will assume a reference (default) category and then it will estimate the influence that a change to one of the other categories will have on the outcome variable. ```r # model with a categorical and a numerical input socio_model <- lm(annual_income_ppp ~ education_months + gender, socio_data) socio_model$coefficients ``` ``` ## (Intercept) education_months genderM ## 15513.2010 269.8212 16762.4734 ``` If you wish you can relevel the variable to control which value to use as reference. ```r # relevel gender to use M as the reference socio_data$gender <- as.factor(socio_data$gender) |> relevel(ref = "M") # rerun model socio_model <- lm(annual_income_ppp ~ education_months + gender, socio_data) socio_model$coefficients ``` ``` ## (Intercept) education_months genderF ## 32275.6744 269.8212 -16762.4734 ``` --- class: left, middle, rstudio-logo ## Model summary Most regression models in R have a `summary()` function. ```r # summary of student test score model summary(full_model) ``` ``` ## ## Call: ## lm(formula = Final ~ Yr1 + Yr2 + Yr3, data = ugtests) ## ## Residuals: ## Min 1Q Median 3Q Max ## -92.638 -20.349 0.001 18.954 98.489 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 14.14599 5.48006 2.581 0.00999 ** ## Yr1 0.07603 0.06538 1.163 0.24519 ## Yr2 0.43129 0.03251 13.267 < 2e-16 *** ## Yr3 0.86568 0.02914 29.710 < 2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 30.43 on 971 degrees of freedom ## Multiple R-squared: 0.5303, Adjusted R-squared: 0.5289 ## F-statistic: 365.5 on 3 and 971 DF, p-value: < 2.2e-16 ``` --- class: left, middle, rstudio-logo ## Understanding model fit It's never the case in people analytics that an outcome variable can be fully explained by a set of input variables, so it is of interest to understand *how much* of it is explained by the input variables we are using. This gives us a sense of how well we have explained our outcome. The `\(R^2\)` of a linear regression model represents the proportion of the variance of the outcome variable that is explained by the input variables. It takes a value between 0 (a 'null' model where the input variables have no explanatory power) and 1 (a perfect fit). The R-squared is part of the summary function for linear models. ```r summary(full_model)$r.squared ``` ``` ## [1] 0.5303275 ``` --- class: left, middle, rstudio-logo ## Is your model 'better than nothing'? Depending on the fit and the sample size, its possible that your model is not significantly different from a 'null model'. A model with a high `\(R^2\)` on a handful of observations might actually be useless. Conversely, a model with a low `\(R^2\)` on a large data set might be better than nothing. The *F-statistic* helps determine the likelihood that your model is better than a null model. The `\(p\)`-value of the F-statistic tests the hypothesis that the model is no different from a null model. You can see the results of this hypothesis test at the end of the model summary. --- class: left, middle, rstudio-logo ## Model simplification If an input variable is not considered significant in your model, it can be removed without a significant loss in fit. It is always advisable to remove non-significant variables from models in order to have a simpler model. This is known as *model parsimony*. ```r simpler_model <- lm(Final ~ Yr3 + Yr2, ugtests) summary(full_model)$r.squared ``` ``` ## [1] 0.5303275 ``` ```r summary(simpler_model)$r.squared ``` ``` ## [1] 0.5296734 ``` --- class: left, middle, rstudio-logo ## Assumption of normality of residuals Remember that our linear relationship represents the expected mean outcome over many observations of the input variables. Individual observations will not precisely match this expected outcome, and so each observation is expected to have some error. $$ `\begin{equation} y_i = \alpha + \beta_1x_{1i} + ... + \beta_px_{pi} + \epsilon_i \end{equation}` $$ These errors `\(\epsilon_i\)` are known as **residuals**. Linear regression assumes that **residuals are normally distributed with a mean of zero**. This is a necessary assumption for the mathematics of the model to work. If your data does not have 'normal-looking' residuals, linear regression may not be the best model to use. --- class: left, middle, rstudio-logo ## Assumption of normality of residuals (Visual representation) <center> <div id="htmlwidget-09a6c607b230247ec7cd" style="width:504px;height:400px;" class="plotly html-widget"></div> <script type="application/json" data-for="htmlwidget-09a6c607b230247ec7cd">{"x":{"visdat":{"e21612e6340":["function () ","plotlyVisDat"]},"cur_data":"e21612e6340","attrs":{"e21612e6340":{"x":[50,54.1666666666667,58.3333333333333,62.5,66.6666666666667,70.8333333333333,75,79.1666666666667,83.3333333333333,87.5,91.6666666666667,95.8333333333333,100,104.166666666667,108.333333333333,112.5,116.666666666667,120.833333333333,125,129.166666666667,133.333333333333,137.5,141.666666666667,145.833333333333,150],"y":[-11.1731901323413,2.37836226796413,15.9299146682696,29.481467068575,43.0330194688804,56.5845718691859,70.1361242694913,83.6876766697968,97.2392290701022,110.790781470408,124.342333870713,137.893886271019,151.445438671324,164.996991071629,178.548543471935,192.10009587224,205.651648272546,219.203200672851,232.754753073157,246.306305473462,259.857857873767,273.409410274073,286.960962674378,300.512515074684,314.064067474989],"z":[[6.8757669310426e-07,1.90462735994011e-06,3.19345461164552e-06,1.09049475318485e-05,2.12700743228464e-05,4.22479431296527e-05,6.22981351923622e-05,8.17347895233864e-05,9.73072250223269e-05,8.92535457252145e-05,7.51183064016232e-05,5.36627121280837e-05,3.18393913849933e-05,1.38733939141235e-05,5.27823021476772e-06,2.43331553628381e-06,1.36712388473098e-06,5.65143263491273e-07,7.44846702615408e-08,2.23200394615994e-09,2.84576509584747e-12,1.16757660860987e-16,1.8073463184976e-22,1.18073711384036e-29,2.84993754905266e-38],[4.99579118872134e-07,1.39802122302984e-06,2.93278005686488e-06,1.04866680017396e-05,2.06721728345015e-05,4.0476950935072e-05,6.31694569976868e-05,8.61889390547646e-05,0.000104026287300377,0.000100731816376931,8.69355402121767e-05,6.35202454631078e-05,3.82188661793598e-05,1.98559101430219e-05,7.53308077897322e-06,3.63282613275852e-06,1.39109386718213e-06,5.02886886449144e-07,1.85253066701122e-07,7.89468186280331e-09,1.19717361696595e-11,7.06925564375382e-16,1.7521648365058e-21,1.55777462941634e-28,4.25850544854416e-37],[1.91858906157905e-07,5.80821567137255e-07,2.314969906379e-06,8.79415924033501e-06,1.77355486151007e-05,3.39915337833823e-05,5.77475963586082e-05,8.35321049663884e-05,0.000103261747179814,0.000107285618240798,9.59226076544404e-05,7.22828215034475e-05,4.43084660822144e-05,2.68522465022087e-05,1.04150909185101e-05,5.34411924473573e-06,1.43486297399444e-06,4.05036054100413e-07,3.29143643933904e-07,1.81868590332826e-08,4.10413191276043e-11,3.8435492396651e-15,1.25407571285578e-20,1.23589951291077e-27,5.31160177931823e-36],[4.00024540672876e-08,2.43632949985022e-07,2.02704463811862e-06,7.19935928299754e-06,1.51182999499463e-05,2.89373574684076e-05,5.0560631100792e-05,7.66936078143997e-05,9.84770931157294e-05,0.000107517423963351,0.000100388245782417,7.85839553787009e-05,4.92886991559206e-05,3.16855424239552e-05,1.33852369254201e-05,7.5847938857335e-06,1.99460126978636e-06,4.95330756602533e-07,4.18741934574371e-07,3.54708549170122e-08,1.24925755909033e-10,1.49027773146005e-14,5.29420169007568e-20,6.57991013895767e-27,8.24483091716485e-34],[6.93015999332264e-09,2.816989384896e-07,2.0113782041505e-06,5.65399818333394e-06,1.27913615578622e-05,2.51313940633772e-05,4.22623548204788e-05,6.73299391507219e-05,9.22650386713636e-05,0.000105163289777217,0.000104184396386251,8.58342109410258e-05,5.56387404392501e-05,3.63461013533431e-05,1.74549376012771e-05,1.07634830412367e-05,3.17338810839917e-06,8.02209496255881e-07,4.83800583487733e-07,6.12604200049092e-08,2.65370307443788e-10,3.38465690562518e-14,1.34883029226392e-19,2.99632538600944e-25,1.90179581062847e-31],[3.60311472836539e-09,3.17823664808672e-07,1.87342104873282e-06,4.48510487852433e-06,1.09796661119168e-05,2.21175073289139e-05,3.66015421275368e-05,6.07030973889899e-05,8.88021243097883e-05,0.000104227755821089,0.000107790313417981,9.28282298863892e-05,6.28281515243575e-05,4.12813120131085e-05,2.21431122542897e-05,1.31493268242708e-05,4.343887914937e-06,1.20151341173566e-06,5.33084636307131e-07,7.70614009272466e-08,3.45666304124605e-10,4.59755429296821e-14,1.7055267992313e-18,3.49565430882344e-23,2.58476494486802e-29],[2.16247613882525e-09,2.04536405731638e-07,1.27876759743728e-06,3.2247750112187e-06,8.58208490811603e-06,1.75086235984292e-05,3.13183062952809e-05,5.32755839571984e-05,8.33770756880709e-05,9.95493555901109e-05,0.000105739759371306,9.56241696829567e-05,6.94753486708561e-05,4.59740335450214e-05,2.76899621733412e-05,1.39839540227897e-05,5.41907865959166e-06,1.75644218162253e-06,5.63689410939448e-07,7.52278835807628e-08,3.26375047725572e-10,1.67666000000646e-13,9.89366485805076e-17,2.31144698638559e-21,2.50667053851764e-27],[7.3105258991261e-10,7.42239420535582e-08,6.96357319772693e-07,2.23107059179967e-06,6.64005702340198e-06,1.37429116604123e-05,2.77713382146609e-05,4.82037819406153e-05,8.03807164136631e-05,9.75220327275996e-05,0.00010610342035391,0.000100869842986938,7.89656675507546e-05,5.31473911535514e-05,3.45608113861072e-05,1.54032635902953e-05,6.73630745636332e-06,2.56867651766094e-06,7.0802241080936e-07,7.84462607370391e-08,5.05551882369306e-10,4.35585295374466e-12,3.35805624375268e-15,9.12898922465506e-20,2.19170697812863e-25],[1.33423881654672e-10,1.83193198746819e-08,4.02464681295008e-07,1.59195429523546e-06,5.22638413621255e-06,1.12091525604568e-05,2.38132499812698e-05,4.20762778410973e-05,7.27243136497259e-05,9.05450905540012e-05,0.000101687575113772,0.000101694961573644,8.58697106345752e-05,6.02165921722714e-05,3.99290365941108e-05,1.78558699439154e-05,8.00736664790583e-06,3.70690829092997e-06,1.11530338006022e-06,1.0600820523874e-07,3.57910347708999e-09,7.68110003929454e-11,6.27338422318636e-14,2.73965456334965e-18,1.64180871146896e-23],[1.4490286080107e-11,5.38666844094609e-09,2.70534312591215e-07,1.13830079742597e-06,4.12288852056298e-06,9.45473630586526e-06,1.99888401452314e-05,3.61941700738945e-05,6.40295326185052e-05,8.31015977766409e-05,9.73707291193553e-05,0.00010363604708693,9.49788808613515e-05,6.99694051013824e-05,4.67305745572133e-05,2.24907523067784e-05,9.99664840973529e-06,5.39408512953625e-06,1.79406222746918e-06,1.93914824539862e-07,2.96848771719221e-08,7.29975382678879e-10,7.25587904955176e-13,7.42714234330849e-17,2.03898076091071e-21],[1.82085799144724e-12,2.68908415991328e-09,1.86234596984887e-07,7.87877809468462e-07,3.16288776333757e-06,7.79781342342506e-06,1.71440124070377e-05,3.20808751969221e-05,5.71412739118782e-05,7.70462228922743e-05,9.23549790477381e-05,0.000104330135707136,0.000100922924746366,7.61804581269632e-05,5.30517627535547e-05,2.87783494841506e-05,1.2895835360104e-05,7.11362898201315e-06,2.44415417918131e-06,4.36239402877897e-07,1.44895896001883e-07,3.8733944121458e-09,6.75554498944312e-12,1.48126611956151e-15,7.13035340099722e-19],[1.47977015255831e-12,1.77031901156621e-09,1.01010275541467e-07,4.09960736271386e-07,1.98298796345166e-06,5.48684270089354e-06,1.42330119280204e-05,2.81599538686203e-05,4.95583347148505e-05,6.97081979548248e-05,8.37994231303116e-05,0.000101917711780767,0.000102912281872489,7.86089143410519e-05,5.96141607570482e-05,3.78397110258295e-05,1.74246217561912e-05,9.22310860724443e-06,3.13786541922225e-06,8.95300434940145e-07,3.90248057844988e-07,1.33398261517803e-08,5.60229815245887e-11,1.74034812735967e-14,1.8162403055218e-16],[6.43154987647502e-12,3.85183996040397e-09,8.10209530159027e-08,1.60762622466527e-07,1.11364613261211e-06,3.71525450373158e-06,1.17633461508796e-05,2.51068381308113e-05,4.3852927486974e-05,6.52095121544233e-05,7.96964505116157e-05,0.000102237868325055,0.000106898485218851,8.38790101317926e-05,6.72037038472498e-05,4.64375087242527e-05,2.24021124740602e-05,1.1766540858529e-05,3.70950833051452e-06,1.2520559733349e-06,6.26024576490346e-07,3.91842138681028e-08,3.24627614960733e-10,1.15577631391938e-13,2.48633532599993e-14],[2.12338707838239e-11,1.18549324952369e-08,1.86932653454105e-07,1.15656017634166e-07,6.65474477272894e-07,2.55831971731024e-06,8.20479767350173e-06,2.00007229761377e-05,3.58498465814272e-05,5.86912103690238e-05,7.68723948642437e-05,9.85835025154023e-05,0.000106452885312867,8.94940469968428e-05,7.14385248831692e-05,4.96601824828264e-05,2.63089552990659e-05,1.44242702558346e-05,3.79370488932522e-06,1.24300778107155e-06,7.65139840031739e-07,1.01043913256547e-07,1.10854742979053e-09,6.96400538146223e-13,1.79809048975835e-12],[3.77040374838779e-11,2.09780614429014e-08,3.25137390443209e-07,1.50685559091296e-07,4.28918266321998e-07,1.80956975807292e-06,4.75728116339923e-06,1.50225713272767e-05,2.87000657788765e-05,5.35962367440356e-05,7.74161926681321e-05,9.73012963128833e-05,0.000108966780536825,9.89999604767587e-05,7.76899737309021e-05,5.33588426503737e-05,3.10202888439931e-05,1.76724270364142e-05,4.324763883009e-06,1.33454646247457e-06,9.24907298009105e-07,1.88055358510089e-07,2.30022824851185e-09,1.2338263188411e-11,6.86400920547485e-11],[3.53687147470242e-11,1.96755647605486e-08,3.04696146313122e-07,1.34980969036633e-07,2.83127524585818e-07,1.39827120116192e-06,3.10846776102171e-06,1.17656489028328e-05,2.41846466814389e-05,4.78842981934306e-05,7.41602550367286e-05,9.33317751864799e-05,0.000109005000256426,0.000102853422171041,8.1700352642642e-05,5.71666976165503e-05,3.42021957801407e-05,1.92921585536415e-05,6.51140500783248e-06,2.16460973653471e-06,1.13341781242823e-06,2.67237444862798e-07,3.64458797400238e-09,2.3259004752086e-10,1.38306446440325e-09],[1.75131845416731e-11,9.7424865108684e-09,1.50865795829538e-07,6.63406185255135e-08,1.62039451690627e-07,1.1466016790551e-06,2.66961729384708e-06,9.48842076509923e-06,2.11638142517655e-05,4.11454909843801e-05,6.77538924035441e-05,8.74820034334687e-05,0.00010818993655592,0.000103972738018822,8.62282829865162e-05,6.32725358154235e-05,3.78601187015278e-05,2.05775917696595e-05,1.0682916219583e-05,3.87775575648941e-06,1.47210246298478e-06,3.54669624719682e-07,5.371779101127e-09,2.46781446586661e-09,1.47097696247867e-08],[4.57729366841649e-12,2.5463219551902e-09,3.94305708219714e-08,1.74050612068947e-08,8.51065917616378e-08,9.31260365137111e-07,2.44893590003211e-06,7.96550560139104e-06,1.85740001209363e-05,3.62192235607647e-05,6.22019961716192e-05,8.26129455429458e-05,0.000106420570273265,0.00010476457374849,9.23208416164808e-05,7.09896323295889e-05,4.41359442630433e-05,2.39780725270502e-05,1.45655199880595e-05,5.63966047005341e-06,1.90570425339704e-06,4.1001217786595e-07,7.46846903974606e-09,1.40660399729073e-08,8.25799899383743e-08],[6.31465260642694e-13,3.51280461191865e-10,5.43968701526521e-09,2.46253127348177e-09,3.99735586531137e-08,5.88822614847003e-07,1.90235143461072e-06,6.27448237691483e-06,1.41060683290939e-05,3.06175249233663e-05,5.34263300895247e-05,7.32079516601759e-05,9.61822496051991e-05,9.98912712315589e-05,9.70181344605532e-05,7.8653711923894e-05,5.37614599534561e-05,3.05288880827579e-05,1.76886221322113e-05,7.43305882428815e-06,2.49599538447671e-06,4.20518778826e-07,2.23948220421969e-08,4.47483495796474e-08,2.44726047707561e-07],[4.59819249287202e-14,2.5579478066194e-11,3.96107486222653e-10,2.00480823747185e-10,1.310112264643e-08,2.83947995394878e-07,1.3708728198309e-06,4.92812195618398e-06,1.02002188562107e-05,2.63707904736614e-05,4.67236677814033e-05,6.63845042360802e-05,8.82777189924528e-05,9.87316979742898e-05,0.000104739875985932,8.90297669924988e-05,6.53659034064009e-05,3.85022542524836e-05,2.12182599132962e-05,9.6107354263661e-06,3.35234324936895e-06,5.4977806130166e-07,1.25954585584407e-07,9.23070909527656e-08,3.82970461517754e-07],[1.76734700877131e-15,9.83164887823177e-13,1.52249055905907e-11,1.1600163311572e-11,2.85990639287494e-09,1.25332417393001e-07,9.29964663809341e-07,3.76785176745659e-06,7.55365099941859e-06,2.17503270102008e-05,4.04750681765823e-05,5.95820128242469e-05,7.97986725188741e-05,9.7545765443621e-05,0.000106694749708791,9.4441667730606e-05,7.05799909022709e-05,4.25141040465066e-05,2.38388481424164e-05,1.17047107031207e-05,4.43249587089608e-06,1.00730325057934e-06,4.75112753304148e-07,1.62140076924297e-07,3.16952850548925e-07],[3.58553120568649e-17,1.99461021261448e-14,3.08900527686958e-13,6.54756264947702e-13,5.17221721821115e-10,4.67361246934692e-08,5.62263199349306e-07,2.74161110239343e-06,5.66837379312864e-06,1.71800227988889e-05,3.51591653252123e-05,5.40895014231813e-05,7.29347250876127e-05,9.93063132500086e-05,0.000109118556657178,0.000100709919599,7.53422449007661e-05,4.66676877220542e-05,2.74274417108681e-05,1.47188132271859e-05,5.99756017780311e-06,1.7773066207134e-06,1.0337453012145e-06,2.49214669881453e-07,1.3971997143116e-07],[3.83956638082773e-19,2.1359285277482e-16,3.30908148392539e-15,4.03877759789547e-14,9.07814821436557e-11,1.67660694086622e-08,4.24088203909314e-07,2.0866218399816e-06,4.37687929110722e-06,1.38579956910155e-05,3.06390813996787e-05,4.93069224222278e-05,6.71899799904662e-05,9.81403270119051e-05,0.000109585794367837,0.000103987422254077,7.97186959110555e-05,5.21098408651898e-05,3.14835823836956e-05,1.83285969337375e-05,7.49471652498913e-06,2.49322988452967e-06,1.49352035686556e-06,2.72418008266116e-07,3.39205590488331e-08],[2.17024043280851e-21,1.2072922322952e-18,1.87462883293395e-17,3.45218015388249e-15,2.98320694104094e-11,1.60992375697756e-08,4.80167871702803e-07,1.60074154763379e-06,3.1506402519333e-06,1.07289516897672e-05,2.48538410266129e-05,4.24829555857871e-05,5.90485864528805e-05,8.9991083947827e-05,0.000104733734134605,0.000101539182561215,8.25018624123909e-05,5.87379022317092e-05,3.59017939541076e-05,2.2727145993314e-05,8.95038488832585e-06,3.17945315334993e-06,1.86425763641774e-06,2.18650037056814e-07,5.31204319561305e-09],[6.47486200080307e-24,3.60192930124475e-21,5.85328453663162e-20,1.33829990043005e-15,2.88330114358132e-11,1.95351809460002e-08,5.18891041794024e-07,1.22680021033493e-06,2.19787510733496e-06,8.14081522396844e-06,1.94700754174088e-05,3.52151380338585e-05,4.98627593904573e-05,7.80664889847776e-05,9.42460464838207e-05,9.26786502528796e-05,7.88184488452405e-05,5.94186979102325e-05,3.64807691984474e-05,2.41898803932594e-05,9.29199879165071e-06,3.40328805706941e-06,1.96318144550752e-06,1.57974707130835e-07,9.56759645627429e-10]],"alpha_stroke":1,"sizes":[10,100],"spans":[1,20],"type":"surface","showscale":false,"inherit":true}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"Year 3"},"yaxis":{"title":"Final"},"zaxis":{"title":"Density","showticklabels":false}},"hovermode":"closest","showlegend":false},"source":"A","config":{"modeBarButtonsToAdd":["hoverclosest","hovercompare"],"showSendToCloud":false},"data":[{"colorbar":{"title":"","ticklen":2},"colorscale":[["0","rgba(68,1,84,1)"],["0.0416666666666667","rgba(70,19,97,1)"],["0.0833333333333333","rgba(72,32,111,1)"],["0.125","rgba(71,45,122,1)"],["0.166666666666667","rgba(68,58,128,1)"],["0.208333333333333","rgba(64,70,135,1)"],["0.25","rgba(60,82,138,1)"],["0.291666666666667","rgba(56,93,140,1)"],["0.333333333333333","rgba(49,104,142,1)"],["0.375","rgba(46,114,142,1)"],["0.416666666666667","rgba(42,123,142,1)"],["0.458333333333333","rgba(38,133,141,1)"],["0.5","rgba(37,144,140,1)"],["0.541666666666667","rgba(33,154,138,1)"],["0.583333333333333","rgba(39,164,133,1)"],["0.625","rgba(47,174,127,1)"],["0.666666666666667","rgba(53,183,121,1)"],["0.708333333333333","rgba(79,191,110,1)"],["0.75","rgba(98,199,98,1)"],["0.791666666666667","rgba(119,207,85,1)"],["0.833333333333333","rgba(147,214,70,1)"],["0.875","rgba(172,220,52,1)"],["0.916666666666667","rgba(199,225,42,1)"],["0.958333333333333","rgba(226,228,40,1)"],["1","rgba(253,231,37,1)"]],"showscale":false,"x":[50,54.1666666666667,58.3333333333333,62.5,66.6666666666667,70.8333333333333,75,79.1666666666667,83.3333333333333,87.5,91.6666666666667,95.8333333333333,100,104.166666666667,108.333333333333,112.5,116.666666666667,120.833333333333,125,129.166666666667,133.333333333333,137.5,141.666666666667,145.833333333333,150],"y":[-11.1731901323413,2.37836226796413,15.9299146682696,29.481467068575,43.0330194688804,56.5845718691859,70.1361242694913,83.6876766697968,97.2392290701022,110.790781470408,124.342333870713,137.893886271019,151.445438671324,164.996991071629,178.548543471935,192.10009587224,205.651648272546,219.203200672851,232.754753073157,246.306305473462,259.857857873767,273.409410274073,286.960962674378,300.512515074684,314.064067474989],"z":[[6.8757669310426e-07,1.90462735994011e-06,3.19345461164552e-06,1.09049475318485e-05,2.12700743228464e-05,4.22479431296527e-05,6.22981351923622e-05,8.17347895233864e-05,9.73072250223269e-05,8.92535457252145e-05,7.51183064016232e-05,5.36627121280837e-05,3.18393913849933e-05,1.38733939141235e-05,5.27823021476772e-06,2.43331553628381e-06,1.36712388473098e-06,5.65143263491273e-07,7.44846702615408e-08,2.23200394615994e-09,2.84576509584747e-12,1.16757660860987e-16,1.8073463184976e-22,1.18073711384036e-29,2.84993754905266e-38],[4.99579118872134e-07,1.39802122302984e-06,2.93278005686488e-06,1.04866680017396e-05,2.06721728345015e-05,4.0476950935072e-05,6.31694569976868e-05,8.61889390547646e-05,0.000104026287300377,0.000100731816376931,8.69355402121767e-05,6.35202454631078e-05,3.82188661793598e-05,1.98559101430219e-05,7.53308077897322e-06,3.63282613275852e-06,1.39109386718213e-06,5.02886886449144e-07,1.85253066701122e-07,7.89468186280331e-09,1.19717361696595e-11,7.06925564375382e-16,1.7521648365058e-21,1.55777462941634e-28,4.25850544854416e-37],[1.91858906157905e-07,5.80821567137255e-07,2.314969906379e-06,8.79415924033501e-06,1.77355486151007e-05,3.39915337833823e-05,5.77475963586082e-05,8.35321049663884e-05,0.000103261747179814,0.000107285618240798,9.59226076544404e-05,7.22828215034475e-05,4.43084660822144e-05,2.68522465022087e-05,1.04150909185101e-05,5.34411924473573e-06,1.43486297399444e-06,4.05036054100413e-07,3.29143643933904e-07,1.81868590332826e-08,4.10413191276043e-11,3.8435492396651e-15,1.25407571285578e-20,1.23589951291077e-27,5.31160177931823e-36],[4.00024540672876e-08,2.43632949985022e-07,2.02704463811862e-06,7.19935928299754e-06,1.51182999499463e-05,2.89373574684076e-05,5.0560631100792e-05,7.66936078143997e-05,9.84770931157294e-05,0.000107517423963351,0.000100388245782417,7.85839553787009e-05,4.92886991559206e-05,3.16855424239552e-05,1.33852369254201e-05,7.5847938857335e-06,1.99460126978636e-06,4.95330756602533e-07,4.18741934574371e-07,3.54708549170122e-08,1.24925755909033e-10,1.49027773146005e-14,5.29420169007568e-20,6.57991013895767e-27,8.24483091716485e-34],[6.93015999332264e-09,2.816989384896e-07,2.0113782041505e-06,5.65399818333394e-06,1.27913615578622e-05,2.51313940633772e-05,4.22623548204788e-05,6.73299391507219e-05,9.22650386713636e-05,0.000105163289777217,0.000104184396386251,8.58342109410258e-05,5.56387404392501e-05,3.63461013533431e-05,1.74549376012771e-05,1.07634830412367e-05,3.17338810839917e-06,8.02209496255881e-07,4.83800583487733e-07,6.12604200049092e-08,2.65370307443788e-10,3.38465690562518e-14,1.34883029226392e-19,2.99632538600944e-25,1.90179581062847e-31],[3.60311472836539e-09,3.17823664808672e-07,1.87342104873282e-06,4.48510487852433e-06,1.09796661119168e-05,2.21175073289139e-05,3.66015421275368e-05,6.07030973889899e-05,8.88021243097883e-05,0.000104227755821089,0.000107790313417981,9.28282298863892e-05,6.28281515243575e-05,4.12813120131085e-05,2.21431122542897e-05,1.31493268242708e-05,4.343887914937e-06,1.20151341173566e-06,5.33084636307131e-07,7.70614009272466e-08,3.45666304124605e-10,4.59755429296821e-14,1.7055267992313e-18,3.49565430882344e-23,2.58476494486802e-29],[2.16247613882525e-09,2.04536405731638e-07,1.27876759743728e-06,3.2247750112187e-06,8.58208490811603e-06,1.75086235984292e-05,3.13183062952809e-05,5.32755839571984e-05,8.33770756880709e-05,9.95493555901109e-05,0.000105739759371306,9.56241696829567e-05,6.94753486708561e-05,4.59740335450214e-05,2.76899621733412e-05,1.39839540227897e-05,5.41907865959166e-06,1.75644218162253e-06,5.63689410939448e-07,7.52278835807628e-08,3.26375047725572e-10,1.67666000000646e-13,9.89366485805076e-17,2.31144698638559e-21,2.50667053851764e-27],[7.3105258991261e-10,7.42239420535582e-08,6.96357319772693e-07,2.23107059179967e-06,6.64005702340198e-06,1.37429116604123e-05,2.77713382146609e-05,4.82037819406153e-05,8.03807164136631e-05,9.75220327275996e-05,0.00010610342035391,0.000100869842986938,7.89656675507546e-05,5.31473911535514e-05,3.45608113861072e-05,1.54032635902953e-05,6.73630745636332e-06,2.56867651766094e-06,7.0802241080936e-07,7.84462607370391e-08,5.05551882369306e-10,4.35585295374466e-12,3.35805624375268e-15,9.12898922465506e-20,2.19170697812863e-25],[1.33423881654672e-10,1.83193198746819e-08,4.02464681295008e-07,1.59195429523546e-06,5.22638413621255e-06,1.12091525604568e-05,2.38132499812698e-05,4.20762778410973e-05,7.27243136497259e-05,9.05450905540012e-05,0.000101687575113772,0.000101694961573644,8.58697106345752e-05,6.02165921722714e-05,3.99290365941108e-05,1.78558699439154e-05,8.00736664790583e-06,3.70690829092997e-06,1.11530338006022e-06,1.0600820523874e-07,3.57910347708999e-09,7.68110003929454e-11,6.27338422318636e-14,2.73965456334965e-18,1.64180871146896e-23],[1.4490286080107e-11,5.38666844094609e-09,2.70534312591215e-07,1.13830079742597e-06,4.12288852056298e-06,9.45473630586526e-06,1.99888401452314e-05,3.61941700738945e-05,6.40295326185052e-05,8.31015977766409e-05,9.73707291193553e-05,0.00010363604708693,9.49788808613515e-05,6.99694051013824e-05,4.67305745572133e-05,2.24907523067784e-05,9.99664840973529e-06,5.39408512953625e-06,1.79406222746918e-06,1.93914824539862e-07,2.96848771719221e-08,7.29975382678879e-10,7.25587904955176e-13,7.42714234330849e-17,2.03898076091071e-21],[1.82085799144724e-12,2.68908415991328e-09,1.86234596984887e-07,7.87877809468462e-07,3.16288776333757e-06,7.79781342342506e-06,1.71440124070377e-05,3.20808751969221e-05,5.71412739118782e-05,7.70462228922743e-05,9.23549790477381e-05,0.000104330135707136,0.000100922924746366,7.61804581269632e-05,5.30517627535547e-05,2.87783494841506e-05,1.2895835360104e-05,7.11362898201315e-06,2.44415417918131e-06,4.36239402877897e-07,1.44895896001883e-07,3.8733944121458e-09,6.75554498944312e-12,1.48126611956151e-15,7.13035340099722e-19],[1.47977015255831e-12,1.77031901156621e-09,1.01010275541467e-07,4.09960736271386e-07,1.98298796345166e-06,5.48684270089354e-06,1.42330119280204e-05,2.81599538686203e-05,4.95583347148505e-05,6.97081979548248e-05,8.37994231303116e-05,0.000101917711780767,0.000102912281872489,7.86089143410519e-05,5.96141607570482e-05,3.78397110258295e-05,1.74246217561912e-05,9.22310860724443e-06,3.13786541922225e-06,8.95300434940145e-07,3.90248057844988e-07,1.33398261517803e-08,5.60229815245887e-11,1.74034812735967e-14,1.8162403055218e-16],[6.43154987647502e-12,3.85183996040397e-09,8.10209530159027e-08,1.60762622466527e-07,1.11364613261211e-06,3.71525450373158e-06,1.17633461508796e-05,2.51068381308113e-05,4.3852927486974e-05,6.52095121544233e-05,7.96964505116157e-05,0.000102237868325055,0.000106898485218851,8.38790101317926e-05,6.72037038472498e-05,4.64375087242527e-05,2.24021124740602e-05,1.1766540858529e-05,3.70950833051452e-06,1.2520559733349e-06,6.26024576490346e-07,3.91842138681028e-08,3.24627614960733e-10,1.15577631391938e-13,2.48633532599993e-14],[2.12338707838239e-11,1.18549324952369e-08,1.86932653454105e-07,1.15656017634166e-07,6.65474477272894e-07,2.55831971731024e-06,8.20479767350173e-06,2.00007229761377e-05,3.58498465814272e-05,5.86912103690238e-05,7.68723948642437e-05,9.85835025154023e-05,0.000106452885312867,8.94940469968428e-05,7.14385248831692e-05,4.96601824828264e-05,2.63089552990659e-05,1.44242702558346e-05,3.79370488932522e-06,1.24300778107155e-06,7.65139840031739e-07,1.01043913256547e-07,1.10854742979053e-09,6.96400538146223e-13,1.79809048975835e-12],[3.77040374838779e-11,2.09780614429014e-08,3.25137390443209e-07,1.50685559091296e-07,4.28918266321998e-07,1.80956975807292e-06,4.75728116339923e-06,1.50225713272767e-05,2.87000657788765e-05,5.35962367440356e-05,7.74161926681321e-05,9.73012963128833e-05,0.000108966780536825,9.89999604767587e-05,7.76899737309021e-05,5.33588426503737e-05,3.10202888439931e-05,1.76724270364142e-05,4.324763883009e-06,1.33454646247457e-06,9.24907298009105e-07,1.88055358510089e-07,2.30022824851185e-09,1.2338263188411e-11,6.86400920547485e-11],[3.53687147470242e-11,1.96755647605486e-08,3.04696146313122e-07,1.34980969036633e-07,2.83127524585818e-07,1.39827120116192e-06,3.10846776102171e-06,1.17656489028328e-05,2.41846466814389e-05,4.78842981934306e-05,7.41602550367286e-05,9.33317751864799e-05,0.000109005000256426,0.000102853422171041,8.1700352642642e-05,5.71666976165503e-05,3.42021957801407e-05,1.92921585536415e-05,6.51140500783248e-06,2.16460973653471e-06,1.13341781242823e-06,2.67237444862798e-07,3.64458797400238e-09,2.3259004752086e-10,1.38306446440325e-09],[1.75131845416731e-11,9.7424865108684e-09,1.50865795829538e-07,6.63406185255135e-08,1.62039451690627e-07,1.1466016790551e-06,2.66961729384708e-06,9.48842076509923e-06,2.11638142517655e-05,4.11454909843801e-05,6.77538924035441e-05,8.74820034334687e-05,0.00010818993655592,0.000103972738018822,8.62282829865162e-05,6.32725358154235e-05,3.78601187015278e-05,2.05775917696595e-05,1.0682916219583e-05,3.87775575648941e-06,1.47210246298478e-06,3.54669624719682e-07,5.371779101127e-09,2.46781446586661e-09,1.47097696247867e-08],[4.57729366841649e-12,2.5463219551902e-09,3.94305708219714e-08,1.74050612068947e-08,8.51065917616378e-08,9.31260365137111e-07,2.44893590003211e-06,7.96550560139104e-06,1.85740001209363e-05,3.62192235607647e-05,6.22019961716192e-05,8.26129455429458e-05,0.000106420570273265,0.00010476457374849,9.23208416164808e-05,7.09896323295889e-05,4.41359442630433e-05,2.39780725270502e-05,1.45655199880595e-05,5.63966047005341e-06,1.90570425339704e-06,4.1001217786595e-07,7.46846903974606e-09,1.40660399729073e-08,8.25799899383743e-08],[6.31465260642694e-13,3.51280461191865e-10,5.43968701526521e-09,2.46253127348177e-09,3.99735586531137e-08,5.88822614847003e-07,1.90235143461072e-06,6.27448237691483e-06,1.41060683290939e-05,3.06175249233663e-05,5.34263300895247e-05,7.32079516601759e-05,9.61822496051991e-05,9.98912712315589e-05,9.70181344605532e-05,7.8653711923894e-05,5.37614599534561e-05,3.05288880827579e-05,1.76886221322113e-05,7.43305882428815e-06,2.49599538447671e-06,4.20518778826e-07,2.23948220421969e-08,4.47483495796474e-08,2.44726047707561e-07],[4.59819249287202e-14,2.5579478066194e-11,3.96107486222653e-10,2.00480823747185e-10,1.310112264643e-08,2.83947995394878e-07,1.3708728198309e-06,4.92812195618398e-06,1.02002188562107e-05,2.63707904736614e-05,4.67236677814033e-05,6.63845042360802e-05,8.82777189924528e-05,9.87316979742898e-05,0.000104739875985932,8.90297669924988e-05,6.53659034064009e-05,3.85022542524836e-05,2.12182599132962e-05,9.6107354263661e-06,3.35234324936895e-06,5.4977806130166e-07,1.25954585584407e-07,9.23070909527656e-08,3.82970461517754e-07],[1.76734700877131e-15,9.83164887823177e-13,1.52249055905907e-11,1.1600163311572e-11,2.85990639287494e-09,1.25332417393001e-07,9.29964663809341e-07,3.76785176745659e-06,7.55365099941859e-06,2.17503270102008e-05,4.04750681765823e-05,5.95820128242469e-05,7.97986725188741e-05,9.7545765443621e-05,0.000106694749708791,9.4441667730606e-05,7.05799909022709e-05,4.25141040465066e-05,2.38388481424164e-05,1.17047107031207e-05,4.43249587089608e-06,1.00730325057934e-06,4.75112753304148e-07,1.62140076924297e-07,3.16952850548925e-07],[3.58553120568649e-17,1.99461021261448e-14,3.08900527686958e-13,6.54756264947702e-13,5.17221721821115e-10,4.67361246934692e-08,5.62263199349306e-07,2.74161110239343e-06,5.66837379312864e-06,1.71800227988889e-05,3.51591653252123e-05,5.40895014231813e-05,7.29347250876127e-05,9.93063132500086e-05,0.000109118556657178,0.000100709919599,7.53422449007661e-05,4.66676877220542e-05,2.74274417108681e-05,1.47188132271859e-05,5.99756017780311e-06,1.7773066207134e-06,1.0337453012145e-06,2.49214669881453e-07,1.3971997143116e-07],[3.83956638082773e-19,2.1359285277482e-16,3.30908148392539e-15,4.03877759789547e-14,9.07814821436557e-11,1.67660694086622e-08,4.24088203909314e-07,2.0866218399816e-06,4.37687929110722e-06,1.38579956910155e-05,3.06390813996787e-05,4.93069224222278e-05,6.71899799904662e-05,9.81403270119051e-05,0.000109585794367837,0.000103987422254077,7.97186959110555e-05,5.21098408651898e-05,3.14835823836956e-05,1.83285969337375e-05,7.49471652498913e-06,2.49322988452967e-06,1.49352035686556e-06,2.72418008266116e-07,3.39205590488331e-08],[2.17024043280851e-21,1.2072922322952e-18,1.87462883293395e-17,3.45218015388249e-15,2.98320694104094e-11,1.60992375697756e-08,4.80167871702803e-07,1.60074154763379e-06,3.1506402519333e-06,1.07289516897672e-05,2.48538410266129e-05,4.24829555857871e-05,5.90485864528805e-05,8.9991083947827e-05,0.000104733734134605,0.000101539182561215,8.25018624123909e-05,5.87379022317092e-05,3.59017939541076e-05,2.2727145993314e-05,8.95038488832585e-06,3.17945315334993e-06,1.86425763641774e-06,2.18650037056814e-07,5.31204319561305e-09],[6.47486200080307e-24,3.60192930124475e-21,5.85328453663162e-20,1.33829990043005e-15,2.88330114358132e-11,1.95351809460002e-08,5.18891041794024e-07,1.22680021033493e-06,2.19787510733496e-06,8.14081522396844e-06,1.94700754174088e-05,3.52151380338585e-05,4.98627593904573e-05,7.80664889847776e-05,9.42460464838207e-05,9.26786502528796e-05,7.88184488452405e-05,5.94186979102325e-05,3.64807691984474e-05,2.41898803932594e-05,9.29199879165071e-06,3.40328805706941e-06,1.96318144550752e-06,1.57974707130835e-07,9.56759645627429e-10]],"type":"surface","frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"shinyEvents":["plotly_hover","plotly_click","plotly_selected","plotly_relayout","plotly_brushed","plotly_brushing","plotly_clickannotation","plotly_doubleclick","plotly_deselect","plotly_afterplot","plotly_sunburstclick"],"base_url":"https://plot.ly"},"evals":[],"jsHooks":[]}</script> </center> --- class: left, middle, rstudio-logo ## Other key watchouts The following complicating factors can render the results of a linear regression model untrustworthy. It is advisable to check for these before interpreting your model: 1. **Collinearity - Highly correlated input variables**: These can result in inaccurate inferences about the importance of input variables. Correlations of over 0.7 should set off serious alarm bells. It is best to experiment with models that do not include both variables. 2. **Non-linear relationships**: If the relationship is clearly non-linear, then applying linear regression may be inappropriate and may result in an over simplified model with a poor fit. This is called *underfitting*. Analysis of model error terms (*residuals*) can quickly spot if a linear model is inappropriate. 3. **Unusual observations**: Extreme observations can impact model estimates. Extreme outcome values are known as *outliers* while extreme input variable values are known as *high leverage points*. It is advisable to remove these if there are reasons to believe that data quality issues could be at play. --- class: left, middle, rstudio-logo ## Exercise - Multiple linear regression For our next short exercise, we will do some practice on running and interpreting a multiple linear regressio model. Go to our [RStudio Cloud workspace](https://rstudio.cloud/spaces/230780/join?access_code=7cXJKFU1KUuuZGLwBVQpLG3dIxPUD3jak3ZQmESh) and start **Assignment 02B - Linear regression**. Let's work on **Exercises 3, 4 and 5**. --- class: left, middle, rstudio-logo # &#127857; Lunchtime! &#128523;