Revolutions: applications

applications

February 07, 2019

Create data visualizations like BBC News with the BBC's R Cookbook

If you're looking for a guide to making publication-ready data visualizations in R, check out the BBC Visual and Data Journalism cookbook for R graphics. Announced in a BBC blog post this week, it provides scripts for making line charts, bar charts, and other visualizations like those below used in the BBC's data journalism.

The cookbook is based around the bbplot package (available on Github), which has been in use at the BBC since March 2018 for creating graphics in the BBC News graphics style. Rather than drafting graphics in R and then relying another graphics tool or a design team for publication, R has been a "game changer" for the BBC. It offers more creativity and control while saving time and effort, especially when the chart needs to be reproduced every time the underlying data change. The BBC offers a six-week course for its data journalists internally, and you can learn as well with the free cookbook published by the BBC, linked below.

Github (BBC): BBC Visual and Data Journalism cookbook for R graphics

Posted by David Smith at 13:58 in applications, graphics, R | Permalink | Comments (2)

November 09, 2018

T-mobile uses R for Customer Service AI

T-Mobile, the global telecommunication company, is using R in production to automatically classify text messages to customer service and route them to an agent that can help. The AI@T-mobile team used the keras library in R to build a natural language processing engine with Tensorflow, and deployed it to production as a docker container. The MRAN Time Machine ensures the container gets fixed R package versions for reproducibility.

While you may think first of Python for building production AI models, the keras library in R has already proven to be a first-class interface for building AI models. The T-mobile team was able to build this system in just four months and with a small budget because they were able to explore in R and immediately deploy in R:

"Despite being an incredibly popular language for exploratory analysis, data scientists are repeatedly told that R is not sufficient for machine learning – especially if those ML models are destined for production. Though much exploratory analysis and modeling is done in R, these models must be rebuilt in Python to meet DevOps and enterprise requirements. Our team doesn’t see the value in this double work. We explore in R and immediately deploy in R. Our APIs are neural network models using R and TensorFlow in docker containers that are small and maintainable enough to make our DevOps team happy!"

If you'd like to try this out yourself, the T-mobile team has published an open source version of their message-classification container on Github, and you can read the details in the blog post by team leads at the link below.

T-Mobile blog: Enterprise Web Services with Neural Networks Using R and TensorFlow

Posted by David Smith at 05:50 in AI, applications, other industry, R | Permalink | Comments (1)

October 12, 2018

The Economist's Big Mac Index is calculated with R

The Economist's Big Mac Index (also described on Wikipedia if you're not a subscriber) was created (somewhat tongue-in-cheek) as a measure to compare the purchasing power of money in different countries. Since Big Macs are available just about everywhere in the world, the price of a Big Mac in Sweden — expressed in US dollars — gives an American traveler a sense of how much more expensive things will be in Stockholm. And comparing the price of a Big Mac in several countries converted to a single baseline currency is a measure of how over-valued (or undervalued) those other currencies are compared to that baseline.

Since its inception in 1986, the Big Mac Index has been compiled and calculated manually, twice a year. But starting with the most recent published index (July 2018, shown above), the index is now calculated with R. This is the first example of a new program at The Economist to publish the data and methods behind its journalism, and here the data and code behind the Big Mac Index have been published as a Github repository. The method behind the index is provided as a richly-commented Jupyter Notebook, where you can also find some additional charts and within-currency analyses not published in the main index.

The repository is published under an open MIT license, meaning you can remix the code to create a new index on prices of another international commodity, provided you can find the data. Find everything you need at the Github repository linked below.

Github (TheEconomist): Data and methodology for the Big Mac index

Posted by David Smith at 12:28 in applications, R | Permalink | Comments (1)

September 21, 2018

Applications of R presented at EARL London 2018

During the EARL (Enterprise Applications of the R Language) conference in London last week, the organizers asked me how I thought the conference had changed over the years. (This is the conference's fifth year, and I'd been to each one.) My response was that it reflected the increasing maturity of R in the enterprise. The early years featured many presentations that were about using R in research and the challenges (both technical and procedural) for integrating that research into the day-to-day processes of the business. This year though, just about every presentation was about R in production, as a mainstream part of the operational infrastructure for analytics.

That theme began in earnest with Garret Grolemund's keynote presentation on the consequences of scientific research that can't be replicated independently. (This slide, based on this 2016 JAMA paper was an eye-opener for me.) The R language has been at the forefront of providing the necessary tools and infrastructure to remove barriers to reproducible research in science, and the RMarkdown package and its streamlined integration in RStudio, is particularly helpful. Garret's keynote was recorded but the video isn't yet available; when it's published I highly recommend taking the time to watch this excellent speech.

The rest of the program was jam-packed with a variety of applications of R in industry as well. I couldn't see them all (it was a three-track conference), but every one I attended demonstrated mature, production-scale applications of R to solve difficult business problems with data. Here are just a few examples:

Rainmakers, a market research firm, uses R to estimate the potential market for new products. They make use of the officer package to automate the generation of Powerpoint reports.
Geolytix uses R to help companies choose locations for new stores that maximize profitability while reducing the risk of cannibalizing sales from other nearby locations. They use SQL Server ML Services to deploy these models to their clients.
Dyson uses R (and the prophet package) to forecast the expected sales of new models of vacuum cleaners, hand dryers, and other products, so that the manufacturing plants can ramp up (or down) production as needed.
Google uses R to design and analyze the results of customer surveys, to make the best decisions of which product features to invest in next.
N Brown Group, a fashion retailer, uses R to analyze online product reviews from customers.
Marks and Spencer uses R to increase revenues by optimizing the products shown and featured in the online store.
PartnerRe, the reinsurance firm, has built an analytics team around R, and uses Shiny to deploy R-based applications throughout the company.
Amazon Web Services uses containerized applications with R to identify customers who need additional onboarding assistance or who may be dissatisfied, and to detect fraud.
Microsoft uses R and Spark (via sparklyr in HDInsight) to support marketing efforts for Xbox, Windows and Surface with propensity modeling, to identify who is most likely to respond to an offer.

You can see may more examples at the list of speakers linked below. (I blogged about my own talk at EARL earlier this week.) Click through to see detailed summaries and (in most cases) a link to download slides.

EARL London 2018: Speakers

Posted by David Smith at 11:57 in applications, events, R | Permalink | Comments (0)

August 28, 2018

Videos from NYC R Conference

The videos from the NYC R conference have been published, and there are so many great talks there to explore. I highly recommend checking them out: you'll find a wealth of interesting R applications, informative deep dives on using R (and a few other applications as well), and some very entertaining deliveries. In this post, I wanted to highlight a couple of talks in particular.

The talk by Jonathan Hersh (Chapman University), Applying Deep Learning to Satellite Images to Estimate Violence in Syria and Poverty in Mexico is both fascinating and a real technical achievement. In the first part of the talk, he uses convolutional neural networks to identify buildings, roads, and vegetation in satellite images, and use that to predict poverty rates across all of Mexico. In the second part, he uses a similar technique to identify violent hotspots in Syria, this time by identifying damaged buildings and roads from the satellite images. In both cases, analyzing satellite images is a lot less expensive (and more consistent) than collecting samples at the ground level. The neural networks were implemented in Theano running on a GPU-enabled Microsoft Azure server.

Moving from applications to platforms, my colleague Marck Vaisman gave a great talk on R for Big Data in the cloud. His talk offers some practical considerations related to working with large data sets in R, and details some cloud technologies for big data that you can connect with R, like Spark and sparklyr (which is easy to use in the cloud with Azure Databricks).

My talk on R and Minecraft is there as well, if you want to check it out. That talk and all of the rest from the conference are available for viewing at the link below.

New York R Conference: 2018

Posted by David Smith at 14:54 in AI, applications, events, Microsoft, R | Permalink | Comments (0)

June 27, 2018

The Financial Times and BBC use R for publication graphics

While graphics guru Edward Tufte recently claimed that "R coders and users just can't do words on graphics and typography" and need additonal tools to make graphics that aren't "clunky", data journalists at major publications beg to differ. The BBC has been creating graphics "purely in R" for some time, with a typography style matching that of the BBC website. Senior BBC Data Journalist Christine Jeavans offers several examples, including this chart of life expectancy differences between men and women:

... and this chart on gender pay gaps at large British banks:

Meanwhile, the chart below was made for the Financial Times using just R and the ggplot2 package, "down to the custom FT font and the white bar in the top left", according to data journalist John Burn-Murdoch.

There are also entire collections devoted to recreating Tufte's own visualizations in R, presumably meeting his typography standards. Tufte later clarified saying "Problem is not code, problem is published practice", which is true of any programming environment, which is why it was strange that he'd call out R in particular.

Posted by David Smith at 14:56 in applications, graphics, R | Permalink | Comments (6)

February 22, 2018

Analyzing accelerometer data with R

Using your smartphone (any modern phone with a built-in accelerometer should work), visit the Cast Your Spell page created by Nick Strayer. (If you need to type it to your phone browser directly, here's a shortlink: bit.ly/castspell .) Scroll down and click the "Press To Cast!" button, and then wave your phone like a wand using one of the shapes shown.

The app will attempt to detect which of the four "spells" you gestured. It was pretty confident in its detection when I cast "Incendio", but your mileage may vary depending on your wizarding ability and the underlying categorization model.

Nick Strayer described how he built this application in a presentation at Data Day Texas last month. The app itself was built using Shiny with the shinysense package (on Github) to collect movement data from the phone. Nick trained a convolutional neural network model (from his own casting gesture data) using the keras package to classify gestures into one of the four "spells". (Interesting side note: because CNNs aren't time-dependent, you can gesture in reverse and still pass the classification test.)

It's almost like magic! For the complete details on how the Cast your Spell app was constructed, see Nick Strayer's presentation at the link below.

Data Day Texas 2018: Making Magic with Keras and Shiny (Nick Strayer)

Posted by David Smith at 12:15 in AI, applications, R | Permalink | Comments (1)

January 03, 2018

Make your R code run faster

There are lots of tricks you can use to make R code run faster: use more efficient data structures; vectorize your R code; offload complex data management tasks to databases. Emily Robinson shares many of these R performance tips in a case study on A/B testing for Etsy. The tips are just as valuable as the process Emily shares for evaluating them — and also the process of asking the R community for help. Check out her post, linked below.

Hooked on Data: Making R Code Faster : A Case Study

Posted by David Smith at 09:00 in applications, beginner tips, high-performance computing, R | Permalink | Comments (0)

November 16, 2017

The City of Chicago uses R to issue beach safety alerts

Among the many interesting talks I saw a the Domino Data Science Pop-Up in Chicago earlier this week was the presentation by Gene Lynes and Nick Lucius from the City of Chicago. The City of Chicago Tech Plan encourages smart communities and open government, and as part of that initiative the city has undertaken dozens of open-source, open-data projects in areas such as food safety inspections, preventing the spread of West Nile virus, and keeping sidewalks clear of snow.

This talk was on the Clear Water initiative, a project to monitor the water quality of Chicago's many public beaches on Lake Michigan, and to issue safety alerts (or in serious cases, beach closures) when E Coli levels in the water get too high. The problem is that E Coli levels can change rapidly: water levels can be normal for weeks, and then spike for a single day. But traditional culture tests take many days to return results, and while rapid DNA-based tests do exist, it's not possible conduct these tests daily at every beach.

The solution is to build a predictive model, which uses meteorological data and rapid DNA tests for some beaches, combined with historical (culture-based) evaluations of water quality, to predict E Coli levels at all beaches every day. The analysis is performed using R (you can find the R code at this Github repository).

The analysis was developed in conjunction with citizen scientists at Chi Hack Night and statisticians from DePaul University. In 2017, the model was piloted in production in Chicago to issue beach safety alerts and to create a live map of beach water quality. This new R-based model predicted 60 additional occurrences of poor water quality, compared with the process used in prior years.

Still, water quality is hard to predict: once you have the slower test data and an actual result to compare with, that's an accuracy rate of 38%, with fewer than 2% false alarms. (The city plans to use clustering techniques to further improve that number.) That model uses rapid DNA testing at five beaches to predict all beaches along Lake Michigan. A Shiny app (linked below) lets you explore the impact of testing at a different set of beaches, and adjusting the false positive rate, on the overall accuracy of the model.

You can find more details about the City of Chicago Clear Water initiative at the link below.

City of Chicago: Clear Water

Posted by David Smith at 14:27 in applications, government, R | Permalink | Comments (1)

October 18, 2017

Is it faster to take a bike or taxi in NYC?

Taxis are plentiful and convenient in New York City, but the city is also served by a wide network of commuter bicycles (Citi Bikes). If you need to get from, say, the West Village to the Garment District, are you better off time-wise hailing a cab, or heading over to the nearest Citi Bike station? Data scientist Todd W. Schnieder crunched the number on travel times for both taxis and Citi Bikes to figure out which was better. Neither is universally the best, but for some trips taxis are most often the fastest, and for others bikes are faster. An interactive map (created with R) allows you to select the time of day and an origin neighborhood, and the map will then tell you the fraction of the time (according to the historical data) that a Citi Bike will outpace a taxi.

Click the image above to go to Todd's site and interact with the map, where you'll also find lots of other analysis comparing the two modes of travel. The source code (in R and SQL) for the analysis is available on GitHub.

Todd W. Schneider: When Are Citi Bikes Faster Than Taxis in New York City?

Posted by David Smith at 06:18 in applications, graphics, R | Permalink | Comments (0)

Revolutions

Milestones in AI, Machine Learning, Data Science, and visualization with R and Python since 2008