www.fgks.org   »   [go: up one dir, main page]

current events Feed

October 31, 2018

Spooky! Gravedigger in R

Need something to distract the kids while they're waiting to head out Trick-or-Treating? You could have them try out a Creepy Computer Game in R! Engineer and social scientist Dr Peter Provost translated one of the old BASIC games from the classic book Creepy Computer Games into R: just have them type in this file (and yes, they do need to type it character-by-character to get the full experience) and then source it into R to play the game. The goal is to move your character * to the exit at the SE corner of the graveyard populated by gravestones + while avoiding the skeletons X.

Scary computer games

For extra credit, see if they can figure out the bug that causes the move counter to go down, instead of up. It might help to refer to the orginal BASIC sources -- the book is online at the publisher's website as a PDF here.

Gravedigger

The Lucid Manager: Celebrate Halloween with Creepy Computer Games in R

Posted by at 08:38 in current events, R | | Comments (0)

October 18, 2018

Maryland's Bridge Safety, reported using R

A front-page story in the Baltimore Sun reported last week on the state of the bridges in Maryland. Among the report's findings:

  • 5.4% of bridges are classified in "poor" or "structurally deficient" condition
  • 13% of bridges in the city of Baltimore are in "poor" condition


Baltimore sun

Those findings were the result of analysis of Federal infrastructure data by reporter Christine Zhang. The analysis was performed using R and documented in a Jupyter Notebook published on Github. The raw data included almost 50 variables including type, location, ownership, and inspection dates and ratings, and required a fair bit of processing with tidyverse functions to extract the Marlyland-specific statistics above. The analysis also turned up an unusual owner for one of the bridges: this one — an access road to the Goddard Space Flight Center — is owned by NASA.

Bridge

You can read the story in the Baltimore Sun, or check out the R analysis in the Github repo linked below.

Github (Baltimore Sun): Maryland bridges analysis (via Sharon Machlis)

Posted by at 16:23 in current events, R | | Comments (0)

September 14, 2018

How many deaths were caused by the hurricane in Puerto Rico?

President Trump is once again causing distress by downplaying the number of deaths caused by Hurricane Maria's devastation of Puerto Rico last year. Official estimates initially put the death toll at 15 before raising it to 64 months later, but it was clear even then that those numbers were absurdly low. The government of Puerto Rico commissioned an official report from the Millikan Institute of Public Health at George Washington University (GWU) to obtain a more accurate estimate, and with its interim publication official toll stands at 2,975. 

Why were the initial estimates so low? I read the interim GWU report to find out. The report itself is clearly written, quite detailed, and composed by an expert team of social and medical scientists, demographers, epidemiologists and biostatisticians, and I find its analysis and conclusions compelling. (Sadly however the code and data behind the analysis have not yet been released; hopefully they will become available when the final report is published.) In short:

  • In the earliest days of the hurricane, the death-recording office was closed and without power, which suppressed the official count.
  • Even once death certificates were collected, it became clear that officials throughout Puerto Rico has not been trained on how to record deaths in the event of a natural disaster, and most deaths were not attributed correctly in official records. 

Given these deficiencies in the usual data used to calculate death tolls (death certificates) the GWU team used a different approach to calculate the death toll. The basis of the method was to estimate excess mortality, in other words, how many deaths occurred in the post-Maria period compared to the number of deaths that would have been expected if it had never happened. This calculation required two quantitative studies:

  • An estimate of what the population would have been if the hurricane hadn't happened. This was based on a GLM model of monthly data from the prior years, accounting for factors including recorded population, normal emigration and mortality rates.
  • The total number of deaths in the post-Maria period, based on death certificates from the Puerto Rico government (irrespective of how the cause of death was coded).
  • (A third study examined the communication protocols before, during and after the disaster. This study did not affect the quantiative conclusions, but formed the basis of some of the report's recommendations.)

The difference between the actual mortality, and the estimated "normal" mortality formed the basis for the estimate of excess deaths attributed to the hurricane. You can see those estimates of excess deaths one month, three months, and five months after the event in the table below; the last column represents the current official estimate.

Excess mortality

These results are consistent in scale with another earlier study by Nishant Kishore et al. (The data and R code behind this study is available on GitHub.) This study attempted to quantify deaths attributed to the hurricane directly, by visiting 3299 randomly chosen households across Puerto Rico. At each household, inhabitants were asked about any household members who had died and their cause of death (related to or unrelated to the hurricane), and whether anyone had left Puerto Rico because of the hurricane. From this survey, the paper's authors extrapolated the number hurricane-related deaths to the entire island. The headline estimate of 4,625 at three months is somewhat larger than the middle column of the study above, but due to the small number of recorded deaths in the survey sample the 95% confidence interval is also much larger: 793 to 8498 excess deaths. (Gelman's blog has some good discussion of this earlier study, including some commentary from the authors.)

With two independent studies reporting excess deaths well into the thousands attributable directly to Hurricane Maria, it's a fair question to ask whether a more effective response before and after the storm could have reduced the scale of this human tragedy.

Milken Institute School of Public Health: Study to Estimate the Excess Deaths from Hurricane Maria in Puerto Rico

Posted by at 05:49 in current events, R, statistics | | Comments (10)

September 07, 2018

Who wrote that anonymous NYT op-ed? Text similarity analyses with R

In US politics news, the New York Times took the unusual step this week of publishing an anonymous op-ed from a current member of the White House (assumed to be a cabinet member or senior staffer). Speculation about the identity of the author is, of course, rife. Much of the attention has focused on the use of specific words in the article, but can data science provide additional clues? In the last 48 hours, several R users have employed text similarity analysis to try and identify likely culprits.

David Robinson followed a similar process to the one he used to identify which of President Trump's tweets were written by him personally (as opposed to a staffer). He downloaded the tweets written by 69 Cabinet members and WH staff who maintain accounts on on Twitter (about 1.5 million words total). Then, he used the TD-IDF (term-frequency inverse-document-frequency) distance measure to find the rate at which certain words are more frequently (or less frequently) by the each subject. Comparing those rates to the rate of use in the op-ed. After ruling out the VP (and the President himself), this analysis suggests Secretary of State Pompeo as a suspect, in part due to his preference for the word "malign".

David Robinson's analysis
David Robinson's analysis

Michael Kearney also used tweets as the comparison text, but limited the suspect pool to cabinet members only. He compared the text of those tweets to the NYT op-ed on using 107 numeric features, such as use of capitalizaion, punctuation, whitespace, and sentence length and structure. On that basis, again ruling out the President and VP, the closest match is Secretary Pompeo.

Michael Kearney's analysis
Michael Kearney's analysis

Kanishka Misra also used tweets from cabinet members, but used the Burows' Delta metric to compare their text to that of the op-ed; those tweets with similar absolute word frequencies to the op-ed would be "closest" by this measure. Secretary of Transportation Elaine Chao is the closest match using this method.

Kanishka Misra's analysis
Kanishka Misra's analysis

The authors take great pains to point out the limitations of these analyses: not all possible authors are active on Twitter; some of those likely don't write all their own tweets; Twitter is a medium that uses very different language than a newspaper editorial; and it's even possible the author was deliberately concealing their authorship through their language choices. But even so, the use of data has highlighted some unusual turns of phrase in the original op-ed that might have otherwise gone unobserved.

In each case the analysis was performed using the R language, and the authors make use of a common set of R packages to assist with the analysis:

  • rvest to scrape data from the NYT website,
  • rtweet, to download the text of tweets from likely suspects,
  • tidytext, to tokenize and otherwise prepare the texts for analysis, and
  • widyr to reshape the data for distance calculations

The full R code behind the analysis is provided by each author, and if nothing else provides interesting examples of different methods for text similarity analysis. These articles are a great demonstration of the power of rapid, in-depth data science with R: the op-ed was only published two days ago!

 

Posted by at 12:26 in current events, data science, R | | Comments (3)

April 18, 2018

Uber overtakes taxis in New York City

In an update to his analysis of taxi and ride-share trips, Todd Schnieder reports that the number of daily Uber rides exceeds the number of taxi rides in New York City, as of November 2017.

Uber-nyc

The data is provided by the New York Taxi and Limousine Commission, and the analysis and graphics were created using the open-source R language. (The code and data for the analysis are available on Github.) You can find a wealth of detailed analysis of rides in New York, including some gorgeous maps of pick-up and drop-off locations, at the link below.

Todd W. Schneider: Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance

 

Posted by at 08:52 in current events, graphics, R | | Comments (0)

March 29, 2018

BotRNot: An R app to detect Twitter bots

Twitter's bot problem is well documented, influencing discourse on divisive topics like politics and civil rights. But it's getting harder and harder to spot such nefarious bots, who often borrow biographies and tweets from real (and often stolen) profiles to evade detection. (The New York Times recently published an outstanding feature on bots and follower factories.) 

Can we distinguish bots from real users using data science? Journalism professor Mike Kearney has developed an R package and Shiny web application that analyzes the tweets of a given user to calculate the probability that user is a bot. For example, it correctly detects that @TwoHeadlines is a bot:

Botornot

It does fairly well according to my tests, with @MagicRealismBot (99.8%), @RealPressSecBot (99.8%),  @NYT4thDownBot (99.3%) and @threat_update (100%) all correctly identified as bots. I'm also pleased to report that my own Twitter account @revodavid is classified as human (2.6% chance of botness). The app has also been used to identify a potential bot interjecting in the gun control debate. On the other hand, @RlangTip is classified as a bot (98.8%), despite my being reliably informed that it's written by humans. 

To detect bots, the app applies a gradient boosting machine learning model implemented using the gbm package in R. Factors used to identify bot-like behavior include the user's number of followers and followed accounts, the bio, and use of hashtags, @-mentions, and capital letters in account's last 100 tweets.

Try out the BotOrNot app at the link below, and let us know of any surprises in the results in the comments.

ShinyApps: {botrnot}

Posted by at 14:39 in current events, R | | Comments (2)

January 31, 2018

Analysis of Trump's State of the Union Speech, with R

President Trump's State of the Union speech was last night, and it seemed to me it dragged on a bit. That's because it was apparently the slowest SOTU speech in history, based on average number of words spoken per minute. The chart below, based on these data, was created by Josh Katz. (The R code for the chart is here. Note the use of his needs package to load the required R packages.)

Trump sotu

Peter Aldhous at Buzzfeed analyzed last night's State of the Union speech as well. Not only was the speech delivered slowly, the script was at a 9-year-old's reading level, following a long-term decline in the complexity of State of the Union speeches.

SOTU-reading-level

You can find more analysis of this and past SOTU speeches at the link below. The R code behind the analysis can be found here. (You may also want to check out this analysis of Trump's Twitter activity and the associated R code.)

Buzzfeed: “I Have The Best Words.” How Trump’s First SOTU Compares To All The Others.

 

Posted by at 14:42 in current events, graphics, R | | Comments (5)

December 25, 2017

Merry Christmas and Happy New Year!

The Revolutions team is celebrating Christmas today, and we're taking a break with family and enjoying good food. And given the number of Eggnogs that are being prepared — thanks to Hadley Wickham's eggnogr Shiny app — it might be a good idea to take the rest of the week off as well. (You can find the R source behind the eggnogr app here.)

Eggnogr

We'll be back again in the New Year, on January 2. Enjoy the holiday season, and a happy New Year to all.

Posted by at 05:00 in current events, R | | Comments (0)

December 01, 2017

A case study in messy data analysis: the Australian same-sex marriage survey

Last month the Australian people signaled their approval of legalizing same-sex marriage by a 62%:38% margin in a national survey. (On a personal note, I was elated and relieved by the result: my husband and I have discussed eventually retiring to Australia, and with this decision our marriage would be recognized there.) While fears of a surprise Brexit-like electoral backlash proved unfounded, researchers including R user Miles McBain explored the results for correlations to demographic variables. This process wasn't as simple as it might have been though: the Australian Bureau of Statistics released the results as a pair of Excel files that violate just about every good practice for sharing data in spreadsheets:

Survey-results

Miles shares the R code he used to extract useful data from this spreadsheet as a blog post that makes a great case study in dealing with messy data using R. The post demonstrates how he used the read_excel function (from readxl package) to extract specific sub-tables from the spreadsheet by specifying row and column ranges, and then use the dplyr package to clean up and merge the data. If you want to explore the data yourself, you can find the R code and the source data in this Github repository.

In a follow-up post, Miles combines the same-sex marriage survey data with Australian Census data to explore various demographic relationships. Unlike the US Census data (which is easily accessible in R thanks to the tidycensus package), there's no interface package for Australian Census data. (Selected tables are available in the Census2016 package, however.) Instead, Miles demonstrates how to use R to download and extract data from the the "Census DataPacks" (CSV data files and Excel data dictionaries) provided by the Australian Bureau of Statistics.  Yet more data wrangling allows Miles to create summary charts of the responses, such as this chart of proportion voting No by percent of the district population declaring a religious affiliation, broken down by state. As you may expect, those districts with more religious populations voted No at greater rates.

Religious-affliliation

Both of these post provide great examples of working with government data, which is often provided in inconvenient formats with messy structures. Follow the links below for step-by-step guides, including the R code used to extract the data, structure it for analysis, and create useful charts.

Medium (Miles McBain): Tidying the Australian Same Sex Marriage Postal Survey Data with R; Combining Australian Census data with the Same Sex Marriage Postal Survey in R

Posted by at 11:23 in current events, data science, graphics, packages, R | | Comments (3)

November 23, 2017

Happy Thanksgiving!

Today is Thanksgiving Day here in the US, so we're taking the rest of the week off to enjoy the time with family.

HappyThanksgiving

Even if you don't celebrate Thanksgiving, today is still an excellent day to give thanks to the volunteers who have contributed to the R project and its ecosystem. In particular, give thanks to the R Core Group, whose tireless dedication — in several cases over a period of more than 20 years — was and remains critical to the success and societal contributions of the language we use and love: R. You can contribute financially by becoming a Supporting Member of the R Foundation.

Posted by at 08:00 in current events, R | | Comments (0)

Next »