How to produce normalized scores in R

I like little bits of code that make analysis simple and here's a nice one that uses the plyr package. Say you have some data. It could be sales figures from shops or exam scores from schools around the local area  or in this case votes for political parties and you want to compare how they've changed over time. You could look at the raw numbers there's nothing wrong in that but if you normalize the data it can be easier to make comparisions both to the same case over time and between cases

data <-mutate(data, LABscale = round((Labour/11559735)*100,0))

Just to go through this if we start with mutate. This is the function that takes data from your dataframe lets you change it and then with <- tacks it back on the end. You put your dataframe name as the first thing after the open bracket. In this case it's data and then follow it with a comma. LABscale is the name of the new normalized variable. Labour is the variable from the dataframe we're interested in as this contains the number of votes Labour got at each general election from 1992. So to normalize the data with 1992 as the benchmark we divide by the number of votes Labour got in 1992 ie 11,559,735 and times by 100.

The use of the round function with the 0 after the comma at the end is to specify the amount of numbers after the decimal point we want. In this case it's 0 as we want an integer ie 108 rather than 108.3467843 ect as that doesn't really add anything to people's understanding.

Anyway this should get you something like this.

  LABscale
1      100
2      117
3       93
4       86
5       74

Using R to compare FTSE 100 bank share price performance

Click picture for bigger image. I think R Function of the Day is a great blog for people learning R and I wanted to use the stockPortfolio package to compare share price performance. In particular I wanted to see how the Chancellor is protecting our public investments in RBS and Lloyds especially compared to other banks in the FTSE 100. As it turns out the worst performaning banks are RBS and Lloyds since Mr Osborne took office are the one's he's nominally responsible for. Code below:

> require(stockPortfolio)

> ticker<-c("RBS.L","STAN.L","LLOY.L","HSBA.L", "BARC.L")
> returns<-getReturns(ticker, "week", start="2010-05-12") 

> plot(returns, main="FTSE 100 Bank share price performance since Osborne became Chancellor")

From Tweets to Polls

Sometimes a paper hits the spot. Previously I had been thinking about how accurate twitter sentiment analysis actually is, especially how does it compare to traditional polling? So what should I find but this excellent paper spot on the same subject. The answer it appears is not too bad but we're talking complement to rather than replacement of traditional polling.

From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series  by Brendan O’Connor, Ramnath Balasubramanyan, Bryan R. Routledge, Noah A. Smith

Abstract
We connect measures of public opinion measured from polls with sentiment measured from text. We analyze several surveys on consumer confidence and political opinion over the 2008 to 2009 period, and find they correlate to sentiment word frequencies in contemporaneous Twitter messages. While our results vary across datasets, in several cases the correlations are as high as 80%, and capture important large-scale trends. The results highlight the potential of text streams as a substitute
and supplement for traditional polling.


Link to pdf

Ken vs Boris: Can Twitter point towards the result?

I tried to grab the latest 1500 tweets mentioning Boris Johnson and then again with Ken Livingstone and got back about 1450 each . There are more Boris tweets out there which is why his period of time coverage is shorter. This was done with TwitteR.

Now using the tm package I found:

Terms most frequently associated with "Boris Johnson"

> findFreqTerms(boris.dtm, lowfreq=50) [1] "2012" "boris" [3] "brits" "campaign" [5] "child" "ginger" [7] "hair" "harry" [9] "hustings" "johnson" [11] "johnsons" "london" [13] "looks" "love" [15] "mayor" "mayoral" [17] "mayoroflondon" "olympic" [19] "olympics" "prince" [21] "sheeran" "spanking" [23] "ticket" "transparency"

Terms most frequently associated with "Ken Livingstone"

> findFreqTerms(ken.dtm, lowfreq=50) [1] "admitted" "backboris" [3] "banker" "banquier" [5] "beacon" "boris" [7] "boriss" "brian" [9] "campaign" "candidate" [11] "cutoutandkeep" "cycle" [13] "davehill" "election" [15] "equality" "extraordinary" [17] "free" "gay" [19] "guide" "hang" [21] "hire" "hustings" [23] "idea" "ill"

Then I used the sentiment package to classify the tweets into neutral, positive or negative ones and this was displayed using ggplot2.

> table(classify_polarity(borisemot, algorithm="voter",verbose=FALSE)) 0.500001 1 272 1178 1.999996000008e-06 1e-06 41 2628 500001 negative 231 41 neutral positive 1178 231

table(classify_polarity(kentext, algorithm="voter",verbose=FALSE)) 0.200000319999872 6.66666222222519e-07 9.99999000001e-07 129 37 100 negative neutral positive 586 585 316

Points to make

1) Boris is getting more coverage than Ken.
2) I'd want to know if the Ken Negative is so high due to wider "non political" public mood or politically converted Conservative tweeters 
3) In future analysis it could be a good idea to restrict the sample to tweeters than only mention a candidate once. If they mention a candidate 10 times lets face it they've probably already decided how to vote.
4) I'm not convinced Boris's neutral tweets are all neutral.
5) This analysis would benefit from being done every day to pick up trends
6) Potentially has a much faster turnaround time than telephone and internet polling.
7) This would only work on large races like the London mayoral election. I doubt the Doncaster mayoral election would generate the same level of interest on Twitter.
8) Ideally sentiment should be weighted by followers. If someone tweets 200 really negative tweets but they only have 5 followers what does it matter in the grand scheme of things. Not a lot. 
9) The most closely associated terms need expert analysis who knows about the specifics otherwise it looks randomn

So who's going to win?

If you'd have asked me before I would have expected Boris's negative ratings to be higher than they are. That Ken's positives are higher than Boris's although marginally may reflect the reality of a revived centre left after their exit from government in 2010.  Looking at the polls before starting this I thought it was too close to call. After, if the method of analysis works and that is a massive if I would lean on the Boris side of too close to call. But if you make judgements based on this data as to who is going to win in May then you're an idiot. So more evidence required.

This post has been brought to you with the greatfully received help from Jeffrey Breen, Heuristic Andrew and the sentiment package. Check them out.

What happens to the #dropthebill hashtag when the PM mentions his NHS reforms?

As every political geek will know midday on Wednesday when Parliament is in session is Prime Minister's Questions. This week Labour Opposition Leader Ed Miliband led on the NHS reforms and how did twitter react? As you can see the Prime Minister's answers weren't convincing for his opponents on Twitter as there was huge spike in the  use of the #dropthebill hashtag. Interestingly the effect lasted after PMQ's has finished.

>require(twitteR)
> tweets = searchTwitter("#dropthebill", n=1500)
> length(tweets)
[1] 1499
> library(plyr)
> tweets.df = ldply(tweets, function(t) t$toDataFrame() )
> names(tweets.df)
 [1] "text" "favorited" "replyToSN" "created" [5] "truncated" "replyToSID" "id" "replyToUID" [9] "statusSource" "screenName"
 > range(tweets.df$created) [1] "2012-02-22 08:09:59 UTC" "2012-02-22 16:28:16 UTC"
> require(ggplot2)
> qplot(tweets.df$created, data=tweets.df, geom="histogram", main="#dropthebill on 22nd February 2012", xlab="Time", binwidth=1200)

to the K-means and beyond ...

One of the rarer types of articles in journals are the sort that seek to do a potted history of a discipline or sub discipline and while they may not make a new contribution to knowledge they are a great resource for people wanting to get up to speed with something.

A great example of such an article is Data Clustering: 50 Years Beyond K-Means by Anil K. Jain (pdf file) As K-means is still one of the most popular algorithms it just goes to show as with other areas of statistics that if your way of doing things arrives first and becomes commonly adopted it'll often trundle along even if better alternatives appear in the literature.

How to trim a mean in R

Good news! My secondhand copy of Fundamentals of Modern Statistical Methods by Rand R. Wilcox has winged its way safely across the Atlantic. In celebration here's a how to do trimmed means in R. Really it's quite simple  but as you can see it can have a big effect on the results.

> #Lets find some data to look at the mean and the trimmed mean
> firstlotofdata <- c(14,13,10,11,19,14,12,12,12,16,18,16,13,14,15,18,17,16,14,11)
> #Some data that also has some outliers could also be helpful
> secondlotofdata <-c(15,18,11,11,12,12,19,17,14,13,12,18,17,16,17,14,11,12,875,4572)
> firstlotofdata
 [1] 14 13 10 11 19 14 12 12 12 16 18 16 13 14 15 18 17
[18] 16 14 11
> secondlotofdata
 [1]   15   18   11   11   12   12   19   17   14   13
[11]   12   18   17   16   17   14   11   12  875 4572
> mean(firstlotofdata)
[1] 14.25
> mean(secondlotofdata)
[1] 285.3
> mean(firstlotofdata, 0.1)
[1] 14.1875
> mean(secondlotofdata, 0.1)
[1] 14.8125
> mean(secondlotofdata, 0.05)
[1] 62.38889
> length(secondlotofdata)
[1] 20

As you can see the first lot of data doesn't produce a lot of variation between the mean and trimmed mean as the data is nicely bunched together and free from outliers. This is as expected as there is no difference between a mean and a trimmed mean when the data fit a Gaussian distribution.

The difference is evident as you can see from the second lot of data which has two massive outliers which hugely distort the result. So if you want to trim the mean by removing 20% of cases you code mean(mydata, 0.1) and this takes 10% of cases from both the top and bottom of the distribution. With a length of 20 this would mean removing 4 cases. As you can see 0.05 only removes 2 cases keeping in the 875 results in the trimmed mean of over 60.