{"id":139,"date":"2020-01-24T13:38:20","date_gmt":"2020-01-24T13:38:20","guid":{"rendered":"http:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/?p=139"},"modified":"2021-11-08T10:25:55","modified_gmt":"2021-11-08T10:25:55","slug":"factor-analysis-and-interpretability","status":"publish","type":"post","link":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/2020\/01\/24\/factor-analysis-and-interpretability\/","title":{"rendered":"Factor analysis and Interpretability"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">So, we had the STOR-i conference last week. 2 days of interesting talks, free wine, and somewhat awkward photos.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"twitter-tweet\" data-width=\"550\" data-dnt=\"true\"><p lang=\"en\" dir=\"ltr\">Our 2020 edition of the STOR-i conference commences today! We are glad to have talks with wonderful academic and industry professionals alongside STOR-i alumni and PhD students! <a href=\"https:\/\/twitter.com\/hashtag\/Statistics?src=hash&amp;ref_src=twsrc%5Etfw\">#Statistics<\/a> <a href=\"https:\/\/twitter.com\/hashtag\/OperationalResearch?src=hash&amp;ref_src=twsrc%5Etfw\">#OperationalResearch<\/a> <a href=\"https:\/\/t.co\/CMWFKlKSVJ\">pic.twitter.com\/CMWFKlKSVJ<\/a><\/p>&mdash; STOR-i (@STORiCDT) <a href=\"https:\/\/twitter.com\/STORiCDT\/status\/1215211531933188097?ref_src=twsrc%5Etfw\">January 9, 2020<\/a><\/blockquote><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script>\n<\/div><figcaption>Exhibit A<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Of the presentations, I was intrigued by one presented by Dolores Romero Morales, of the Copenhagen Business School (her personal website is <a href=\"https:\/\/www.cbs.dk\/en\/research\/departments-and-centres\/department-of-economics\/staff\/drmeco\">here<\/a>) mentioning a paper entitled &#8220;Enhancing Interpretability in Factor Analysis by Means of Mathematical Optimization&#8221;. Sadly, she ran out of time to go into this during the talk, so I decided to do a post discussing this paper.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">But firstly, a background in factor analysis.<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Factor analysis is a statistical method to try and reduce the number of important variables in a linear regression model. In a standard linear regression model, you have a number of variables, which you can see\/observe, and you assume have a direct effect on the output.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"773\" src=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Linear-regression-diagram-1024x773.png\" alt=\"\" class=\"wp-image-140\" srcset=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Linear-regression-diagram-1024x773.png 1024w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Linear-regression-diagram-300x226.png 300w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Linear-regression-diagram-768x580.png 768w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Linear-regression-diagram.png 1463w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Basic idea behind linear regression<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In Factor analysis, there are a number of latent variables called &#8216;Factors&#8217;. These are unseen variables, which affect the seen variables you would use in your linear regression. The factors affect this variables in the manner of:<\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\">Variable = Effect of Factor 1 + Effect of Factor 2 + &#8230;+ Some Random Error<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You then use the variables to determine their effect on the outcome.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"578\" src=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-1024x578.png\" alt=\"\" class=\"wp-image-142\" srcset=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-1024x578.png 1024w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-300x169.png 300w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-768x434.png 768w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-1536x867.png 1536w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1-1600x903.png 1600w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Diagram-1.png 1993w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>The basic idea behind factor analysis.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">An example is a study in which researchers are trying to model the income of participants. One can imagine that some unobservable qualities &#8211; such as intelligence and work ethic &#8211; would influence how much a person will earn. However, we can&#8217;t measure these &#8211; we can only measure things such as university results and years of work experience. The idea is that the (unbserved) factors of intelligence and work ethic will influence the (observed) variables of university results and years of work experience, for example:<\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\">University results = Effect of Intelligence + Effect of Work Ethic + Random error<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Then, you use the university results and work experience to model income, such as:<\/p>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\">Income = Effect of University Results + Effect of Work Experience + Some more random error<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"589\" src=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-1024x589.png\" alt=\"\" class=\"wp-image-146\" srcset=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-1024x589.png 1024w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-300x173.png 300w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-768x442.png 768w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-1536x884.png 1536w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example-1600x921.png 1600w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/Factor-Analysis-Example.png 1955w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>The structure of the variables<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Factor analysis can be used to:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Examine if a hypothesised structure exists in the data &#8211; our toy example is an example of this<\/li><li>Examine the data to see if there is any structure present.<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The latter example is known as <strong>exploratory factor analysis<\/strong>. The paper I&#8217;m going to look at today looks at an extension of this.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Enhancing Interpretability in Factor Analysis by Means of Mathematical Optimization <\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;m not going to lie, I was very excited when I heard the title of this paper. Communication in mathematics and statistics is always a problem. Data Science, just through the sheer number of observations and variables is particularly susceptible to this.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The paper is trying to solve the following problem. Let&#8217;s say in our example above, that you didn&#8217;t have any idea what the two factors were. So you plug your data into your software package of choice to do some factor analysis<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"337\" src=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/rprogramming.png\" alt=\"\" class=\"wp-image-148\" srcset=\"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/rprogramming.png 600w, https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-content\/uploads\/sites\/8\/2020\/01\/rprogramming-300x169.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><figcaption>Hint: You should choose R<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The issue is, the model <s>R<\/s> your software package fits isn&#8217;t guaranteed in any way to fit factors that make intuitive sense. For example, in our example above, you could get a model which has 13 factors, of which 9 affect university results and 7 affect work experience, with no real pattern between them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While there are many different ways of dealing with the interpretability of factors, the (extremely condensed) idea of the paper is that you can assign the different explanatory variables to clusters. Then, you can force the model to fit factors that match these clusters. Therefore, you can be sure that you will end up with factors that &#8220;make sense&#8221;.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Example &#8211; California Reservoir Levels<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The authors in the paper trial their method on a dataset of California reservoir levels (from Taeb et al. 2017). They assign one variable to each cluster, chosing:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Palmer Drought Severity Index<\/li><li>Colorado River Discharge<\/li><li>State-Wide Consumer Price Index<\/li><li>Hydroelectric Power<\/li><li>State-Wide Number of Agricultural Workers<\/li><li>Sierra Nevada Snow Pack<\/li><li>Temperature<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">as the seven variables, with one variable in each cluster. Therefore, when fitting the factors, we can tell that they will be related to one of these variables.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In fact, they find that the best fitting comes from having 2 factors &#8211; one for Hydroelectric power and one for temperature.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Issues with the paper<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;m torn here. On the whole, while I&#8217;m sympathetic to any attempt to make data science more accessible, I&#8217;m a bit confused by this paper (ironic, really). They assign each factor to a specific variable to make the model easier to interpret. But once this is done, it isn&#8217;t clear why this isn&#8217;t simply a linear regression. The whole idea is that factor analysis tries to find some hidden factors that govern the way the explanatory variables work. But for this approach, the factors are assigned to seen explanatory variables anyway, so it&#8217;s not clear what this process achieves. Furthermore, the code used in the paper isn&#8217;t available online, so it is hard to replicate what they&#8217;ve done and work out what they did for yourself.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, I would be very interested to see how fitting models with these interpretability clusters to fitting standard exploratory factor analysis models. Having previously worked trying to explain complex data analysis to non-technical people, the idea that you could explain that your explanatory variables are governed by a simple, understandable process is golden. However, until I give this more of a go myself, I&#8217;m not sure how much this will add to the process.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, we had the STOR-i conference last week. 2 days of interesting talks, free wine, and somewhat awkward photos. Of&hellip;<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[6,3,4,5],"class_list":["post-139","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-communication","tag-data-science","tag-factor-analysis","tag-statistics"],"_links":{"self":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/posts\/139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/comments?post=139"}],"version-history":[{"count":10,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/posts\/139\/revisions"}],"predecessor-version":[{"id":303,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/posts\/139\/revisions\/303"}],"wp:attachment":[{"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/media?parent=139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/categories?post=139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lancaster.ac.uk\/stor-i-student-sites\/hamish-thorburn\/wp-json\/wp\/v2\/tags?post=139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}