\n
![\"\"](\"http:\/\/benediktehinger.de\/blog\/science\/upload\/sites\/2\/2017\/01\/Rplot01.png\")
<\/a><\/p>\nWith regression one predicts X2 based on X1 (X2~X1 in R-Formula writing) or vice versa. With principal component (or total least squares) one tries to quantify the relation between the two. To completely understand the difference, image what quantity is reduced in the three cases.<\/p>\n
In regression, we reduce the residuals in direction of the dependent variable. With principal components, we find the line, that has the smallest error orthogonal to the regression line. See the following image for a visual illustration. For me it becomes interesting if you plot a scatter plot of two independent variables, i.e. you would usually report the correlation coefficient. The ‘correct’ line accompaniyng the correlation coefficient would be the principal component (‘correct’ as it is also agnostic to the order of the signals).<\/p>\n How to draw the line, eLife 2013<\/a> “`R set.seed(2) dat = data.frame(x1 = dat[,1],x2 = dat[,2])<\/p>\n # Calculate the first principle component x1tox2 = coef(lm(x1~x2,dat)) # We want this to draw the neat orthogonal lines. segmeData = rbind(data.frame(x=dat[,1],y=dat[,2],xend=points[1,],yend=points[2,],type = ‘Principal Component’), ggplot(aes(x1,x2),data=dat)+geom_point()+ ggplot(aes(x1,x2),data=dat)+geom_point()+ I made some graphs that show the relation between X1~X2 (X2 predicts X1), X2~X1 (X1 predicts X2) and the first principal component (direction with highest variance, also called total least squares). The line you fit with a principal component is not the same line as in a regression (either predicting X2 by X1 [X2~X1] or X1 by X2 [X1~X2]. This is quite well known (see references below). With regression one predicts X2 based on X1 (X2~X1 in R-Formula writing) or vice versa. With principal component (or total least squares) one tries to quantify the relation between the two. To completely…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-183","post","type-post","status-publish","format-standard","hentry","category-blog"],"_links":{"self":[{"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/posts\/183","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/comments?post=183"}],"version-history":[{"count":0,"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/posts\/183\/revisions"}],"wp:attachment":[{"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/media?parent=183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/categories?post=183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/benediktehinger.de\/blog\/science\/wp-json\/wp\/v2\/tags?post=183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n<\/a><\/p>\nFurther information:<\/h4>\n
\nGelman, Hill – Data Analysis using Regression, p.58
\n Also check out the nice blogpost from Martin Johnsson doing practically the same thing but three years earlier \ud83d\ude09 <\/a><\/p>\nSource<\/h4>\n
\nlibrary(ggplot2)
\nlibrary(magrittr)
\nlibrary(plyr)<\/p>\n
\ncorrCoef = 0.5 # sample from a multivariate normal, 10 datapoints
\ndat = MASS::mvrnorm(10,c(0,0),Sigma = matrix(c(1,corrCoef,2,corrCoef),2,2))
\ndat[,1] = dat[,1] – mean(dat[,1]) # it makes life easier for the princomp
\ndat[,2] = dat[,2] – mean(dat[,2])<\/p>\n
\n# see http:\/\/stats.stackexchange.com\/questions\/13152\/how-to-perform-orthogonal-regression-total-least-squares-via-pca
\nv = dat%>%prcomp%$%rotation
\nx1x2cor = bCor = v[2,1]\/v[1,1]\n
\nx2tox1 = coef(lm(x2~x1,dat))
\nslopeData =data.frame(slope = c(x1x2cor,1\/x1tox2[2],x2tox1[2]),type=c(‘Principal Component’,’X1~X2′,’X2~X1′))<\/p>\n
\npointOnLine = function(inp){
\n # y = a*x + c (c=0)
\n # yOrth = -(1\/a)*x + d
\n # yOrth = b*x + d
\n x0 = inp[1]\n y0 = inp[2]\n a = x1x2cor
\n b = -(1\/a)
\n c = 0
\n d = y0 – b*x0
\n x = (d-c)\/(a-b)
\n y = -(1\/a)*x+d
\n return(c(x,y))
\n}
\npoints = apply(dat,1,FUN=pointOnLine)<\/p>\n
\n data.frame(x=dat[,1],y=dat[,2],yend=dat[,1]*x2tox1[2],xend=dat[,1],type=’X2~X1′),
\n data.frame(x=dat[,1],y=dat[,2],yend=dat[,2],xend=dat[,2]*x1tox2[2],type=’X1~X2′))<\/p>\n
\n geom_abline( data=slopeData,aes(slope = slope,intercept=0,color=type))+
\n theme_minimal(20)+coord_equal()<\/p>\n
\n geom_abline( data=slopeData,aes(slope = slope,intercept=0,color=type))+
\n geom_segment(data=segmeData,aes(x=x,y=y,xend=xend,yend=yend,color=type))+facet_grid(.~type)+coord_equal()+theme_minimal(20)
\n“`<\/p>\n","protected":false},"excerpt":{"rendered":"