www.fgks.org   »   [go: up one dir, main page]

packages Feed

February 21, 2019

Developments in AzureR

by Hong Ooi, senior data scientist, Microsoft Azure

The AzureR packages have now been on CRAN for a couple of months, so I thought I'd provide an update on developments in the works.

First, AAD authentication has been moved into a new package, AzureAuth, so that people who just want OAuth tokens can get it without any other baggage. This has many new features:

  • Supports both AAD v1.0 and v2.0
  • Tokens are cached in a user-specific directory using the rappdirs package, typically c:\users\<username>\local\AzureR on Windows and ~/.local/share/AzureR on Linux
  • Supports 4 authentication methods: authorization_code, device_code, client_credentials and resource_grant
  • Supports logging in with a username or with a certificate

In the longer term, the hope is for AzureAuth to be something like the R equivalent of the ADAL client libraries. Things to add include dSTS, federated logins, and more.

AzureRMR 2.0 has a new login framework and no longer requires you to create a service principal (although you can still provide your own SP if desired). Running create_azure_login() logs you into Azure interactively, and caches your credentials; in subsequent sessions, run get_azure_login() to login without having to reauthenticate.

AzureStor 2.0 has several new features mostly for more efficient uploading and downloading:

  • Parallel file transfers, using a pool of background processes. This greatly improves the transfer speed when working with lots of small files.
  • Transfer files to or from a local connection. This lets you transfer in-memory R objects without having to create a temporary file first.
  • Experimental interface to AzCopy version 10. This lets you do essentially anything that AzCopy can do, from within R. (Note: AzCopy 10 is quite a different beast to AzCopy 8.x.)
  • A new framework of generic methods, to organise all the various storage-type-specific functions.

A new AzureKusto package is in the works, for working with Kusto/Azure Data Explorer. This includes:

  • All basic functionality including querying, engine management commands, and ingesting
  • A dplyr interface written by Alex Kyllo.

AzureStor 2.0 is now on CRAN, and the others should also be there over the next few weeks. As always, if you run into any problems using the packages, feel free to contact me.

Posted by at 08:30 in Microsoft, packages, R | | Comments (1)

January 15, 2019

Use foreach with HPC schedulers thanks to the future package

The future package is a powerful and elegant cross-platform framework for orchestrating asynchronous computations in R. It's ideal for working with computations that take a long time to complete; that would benefit from using distributed, parallel frameworks to make them complete faster; and that you'd rather not have locking up your interactive R session. You can get a good sense of the future package from its introductory vignette or from this eRum 2018 presentation by author by Henrik Bengtsson (video embedded below), but at its simplest it allows constructs in R like this:

a %<-% slow_calculation(1:50)
b %<-% slow_calculation(51:100)
a+b

The idea here is that slow_calculation is an R function that takes a lot of time, but with the special %<-% assignment operator the computation begins and the R interpreter is ready immediately. The first two lines of R code above take essentially zero time to execute. The futures package farms off those computations to another process or even a remote system (you specify which with a preceding plan call), and R will only halt when the result is needed, as in the third line above. This is beneficial in Bengtsson's own work, where he uses the future package to parallelize cancer research on DNA sequences on high-performance computing (HPC) clusters.

The future package supports a wide variety of computation frameworks including parallel local R sessions, remote R sessions, and cluster computing frameworks. (If you can't use any of these, it falls back to evaluating the expressions locally, in sequence.) The future package also works in concert other parallel programming systems already available in R. For example, it provides future_lapply as a futurized analog of lapply, which will use whatever computation plan you have defined to run the computations in parallel.

The future package also extends the foreach package thanks to the updated doFuture package. By using registerDoFuture as the foreach backend, your loops can use any computation plan provided by the future package to run the iterations in parallel. (The same applies to R packages that use foreach internally, notably the caret package.) This means you can now use foreach with any of the HPC schedulers supported by future, which includes TORQUE, Slurm, and OpenLava. So if you you share a Slurm HPC cluster with colleagues in your department, you can queue up a parallel simulation on the cluster using code like this:

library("doFuture")
registerDoFuture()
library("future.batchtools")
plan(batchjobs_slurm)

mu <- 1.0
sigma <- 2.0
x <- foreach(i = 1:3, .export = c("mu", "sigma")) %dopar% {
  rnorm(i, mean = mu, sd = sigma)
}

The future package is available on CRAN now, and works consistently on Windows, Mac and Linux systems. You can learn more in the video at the end of this post, or in the recent blog update linked below.

JottR: Maintenance Updates of Future Backends and doFuture

Posted by at 15:25 in high-performance computing, packages, R | | Comments (0)

December 18, 2018

AzureStor: an R package for working with Azure storage

by Hong Ooi, senior data scientist, Microsoft Azure

A few weeks ago, I introduced the AzureR family of packages for working with Azure in R. Since then, I’ve also written articles on how to use AzureRMR to interact with Azure Resource Manager, how to use AzureVM to manage virtual machines, and how to use AzureContainers to deploy R functions with Azure Kubernetes Service. This article is the next in the series, and covers AzureStor: an interface to Azure storage.

The Resource Manager interface: creating and deleting storage accounts

AzureStor implements an interface to Azure Resource Manager, which you can use manage storage accounts: creating them, retrieving them, deleting them, and so forth. This is done via the appropriate methods of the az_resource_group class. For example, the following code shows how you might create a new storage account from scratch.

library(AzureStor)

# get the resource group for the storage account
rg <- AzureRMR::az_rm$
    new(tenant="{tenant_id}", app="{app_id}", password="{password}")$
    get_subscription("{subscription_id}")$
    get_resource_group("myresourcegroup")

# create the storage account
# by default, this will be in the resource group's region
rg$create_storage_account("mynewstorage")

Without any options, this will create a storage account with the following parameters:

  • General purpose account (all storage types supported)
  • Locally redundant storage (LRS) replication
  • Hot access tier (for blob storage)
  • HTTPS connection required for access

You can change these by setting the arguments to create_storage_account(). For example, to create an account with geo-redundant storage replication and the default blob access tier set to “cool”:

Continue reading "AzureStor: an R package for working with Azure storage" »

Posted by at 08:18 in Microsoft, packages, R | | Comments (0)

December 03, 2018

AzureVM: managing virtual machines in Azure

by Hong Ooi, senior data scientist, Microsoft Azure

This is the next article in my series on AzureR, a family of packages for working with Azure in R. I’ll give a short introduction on how to use AzureVM to manage Azure virtual machines, and in particular Data Science Virtual Machines (DSVMs).

Creating a VM

Creating a VM is as simple as using the create_vm method, which is available as part of the az_subscription and az_resource_group classes.

library(AzureRMR)
library(AzureVM)

## using the subscription method
sub <- az_rm$
    new(tenant="{tenant_id}", app="{app_id}", password="{password}")$
    get_subscription("{subscription_id}")

myNewVM <- sub$create_vm("myNewVM",
    location="australiaeast",
    username="datascience",
    passkey="fAs30q-2a5vF!Z")  # be sure to choose a strong password!


## using the resource group method
rg <- sub$create_resource_group("myresourcegroup",
    location="australiaeast")

myOtherVM <- rg$create_vm("myOtherVM",
    username="datascience",
    passkey="l3Kgrf21%?0DFm")

Without any other options, this will create a Windows Server 2016 Data Science Virtual Machine, which is pre-installed with several tools useful for analytics: Python, R (and RStudio), Tensorflow, XGBoost, SQL Server, and so on. The size will be a Standard DS3 V2 VM, which has 4 cores, 14GB of memory, 1TB primary disk, and up to 16x28GB data disks.

Continue reading "AzureVM: managing virtual machines in Azure" »

Posted by at 08:00 in Microsoft, packages, R | | Comments (0)

November 26, 2018

AzureRMR: an R interface to Azure Resource Manager

In a previous article I announced AzureR, a new family of packages for working with Azure from R. This article goes into more detail on how you can use AzureRMR, the base package of the AzureR family, to manage resources with Azure Resource Manager.

Before you begin

The first thing you have to do is create a service principal. This is a security identity used by apps, services and automation tools for programmatic access to Azure resources. You can think of it as a “user identity” (login and password or certificate) with a role and permissions to access your resources. Creating a service principal is a one-time task, after which you can reuse it for all your AzureRMR sessions.

There are various ways to create a service principal. The easiest is to have your network administrator create it for you (which is probably also the best option in a corporate environment), but if you’re doing it yourself, you can do it via the Azure cloud shell.

Continue reading "AzureRMR: an R interface to Azure Resource Manager" »

Posted by at 08:00 in Microsoft, packages, R | | Comments (0)

November 08, 2018

AzureR: R packages to control Azure services

by Hong Ooi, senior data scientist, Microsoft Azure

This post is to announce a new family of packages we have developed as part of the CloudyR project for talking to Azure from R: AzureR.

As background, some of you may remember the AzureSMR package, which was written a few years back as an R interface to Azure. AzureSMR was very successful and gained a significant number of users, but it was never meant to be maintainable in the long term. As more features were added it became more unwieldy until its design limitations became impossible to ignore.

The AzureR family is a refactoring/rewrite of AzureSMR that aims to fix the earlier package's shortcomings.

The core package of the family is AzureRMR, which provides a lightweight yet powerful interface to Azure Resource Manager. It handles authentication (including automatically renewing when a session token expires), managing resource groups, and working with individual resources and templates. It also calls the Resource Manager REST API directly, so you don't need to have PowerShell or Python installed; it depends only on commonly used R packages like httr, jsonlite and R6.

Here's what some code using AzureRMR looks like. Note that it uses R6 classes to represent Azure objects, hence the use of $ to call methods.

library(AzureRMR)

az <- az_rm$new(tenant="{tenant_id}",
    app="{app_id}",
    password="{password}")

# get a subscription and resource group
sub <- az$get_subscription("{subscription_id}")
rg <- sub$get_resource_group("rgName")

# get a resource (storage account)
stor <- rg$get_resource(type="Microsoft.Storage/storageAccounts",
    name="myStorage")

# method chaining works too
stor <- az$
    get_subscription("{subscription_id}")$
    get_resource_group("rgName")$
    get_resource(type="Microsoft.Storage/storageAccounts",
        name="myStorage")

# create a new resource group and resource
rg2 <- sub$create_resource_group("newRgName", location="westus")

stor2 <- rg2$create_resource(type="Microsoft.Storage/storageAccounts",
    name="myNewStorage",
    kind="Storage", 
    sku=list(name="Standard_LRS", tier="Standard"))

# delete them
stor2$delete(confirm=FALSE)
rg2$delete(confirm=FALSE)

You can use AzureRMR to work with any Azure service, but a key idea behind it is that it's extensible: you can write new packages that provide extra functionality relevant to specific Azure offerings. This deals with one of the main shortcomings of AzureSMR: as a monolithic package, it became steadily more difficult to add new features as time went on. The packages that extend AzureRMR in this way constitute the AzureR family.

Currently, the AzureR family includes the following packages:

  • AzureVM is a package for working with virtual machines and virtual machine clusters. It allows you to easily deploy, manage and delete a VM from R. A number of templates are provided based on the Data Science Virtual Machine, or you can also supply your own template.
  • AzureStor is a package for working with storage accounts. In addition to a Resource Manager interface for creating and deleting storage accounts, it also provides a client interface for working with the storage itself. You can upload and download files and blobs, list file shares and containers, list files within a share, and so on. It supports authenticated access to storage via either a key or a shared access signature (SAS).
  • AzureContainers is a package for working with containers in Azure: specifically, it provides an interface to Azure Container Instances, Azure Container Registry and Azure Kubernetes Service. You can easily create an image, push it to an ACR repo, and then deploy it as a service to AKS. It also provides lightweight shells to docker, kubectl and helm (assuming these are installed).

I'll be writing future blog posts to describe each of these in further detail.

All the AzureR packages are part of the CloudyR Project, which aims to make R work better with cloud services of all kinds. They are not (yet) on CRAN, but you can obtain them with devtools::install_github.

library(devtools)
install_github("cloudyr/AzureRMR")
install_github("cloudyr/AzureVM")
install_github("cloudyr/AzureStor")
install_github("cloudyr/AzureContainers")

If you use Azure with R, I hope you'll find these packages to be of significant benefit to your workflow. If you have any questions or comments, please contact me at hongooi@microsoft.com.

Ed Note: This is the first post in a series. For more detail on the individual packages, here are links to the followup posts:

Posted by at 08:50 in announcements, Microsoft, packages, R | | Comments (0)

November 06, 2018

Working with US Census Data in R

If you need data about the American populace, there's no source more canonical than the US Census Bureau. The bureau publishes a wide range of public sets, and not just from the main Census conducted every 10 years: there are more than 100 additional surveys and programs published as well. To help R users access this rich source of data, Ari Lamstein and Logan Powell have published A Guide to Working with US Census Data in R, a publication of the R Consortium Census Working Group.

The guide provides an overview of the data available from the US census bureau and various tools available in R to access and analyze it. The guide notes that there are 22 R packages for working with census data, and cites as being particularly useful:

  • tigris, for working with shape files of census regions (census data is may be aggregated to any of a number of levels as shown int the diagram below)
  • acs, for downloading and managing data from the decennial census and the American Community Survey
  • choroplethr and choroplethrMaps, for mapping data (including census data) by region
  • tidycensus, to extract census data as tidy data frames
  • censusapi, for extracting data using the Census API
  • ipumsr, to extract US census data in a form that can be compared with data from other countries 
Census-geo

You can find the complete guide at the link below.

R Consortium: A Guide to Working with US Census Data in R

Posted by at 16:03 in government, packages, R | | Comments (1)

September 26, 2018

3-D shadow maps in R: the rayshader package

Data scientists often work with geographic data that needs to be visualized on a map, and sometimes the maps themselves are the data. The data is often located in two-dimensional space (latitude and longitude), but for some applications we have a third dimension as well: elevation. We could represent the elevations using contours, color, or 3-D perspective, but with the new rayshader package for R by Tyler Morgan-Wall, it's easy to visualize such maps as 3-D relief maps complete with shadows, perspective and depth of field:

Tyler describes the rayshader package in a gorgeous blog post: his goal was to generate 3-D representations of landscape data that "looked like a paper weight". (Incidentally, you can use this package to produce actual paper weights with 3-D printing.) To this end, he went beyond simply visualizing a 3-D surface in rgl and added a rectangular "base" to the surface as well as shadows cast by the geographic features. He also added support for detecting (or specifying) a water level: useful for representing lakes or oceans (like the map of the Monterey submarine canyon shown below) and for visualizing the effect of changing water levels like this animation of draining Lake Mead.

Raytracer

The rayshader package is implemented as an independent R package; it doesn't require any external 3-D graphics software to work. Not only does that make it easy to install and use, but it also means that the underlying computations are available for specialized data analysis tasks. For example, research analyst David Waldran used a LIDAR scan of downtown Indianapolis to create (with the lidR package) a 3-D map of the buildings, and then used the ray_shade function to simulate the shadows cast by the buildings at various times during a winter's day. Averaging those shadows yields this map of the shadiest winter spots in Indianapolis:

Indianapolis Winter

The rayshader package is available for download now from your local CRAN mirror. You can also find an overview, and the latest version of the package at the Github repository linked below.

Github (tylermorganwall): rayshader, R Package for Producing and Visualizing Hillshaded Maps from Elevation Matrices, in 2D and 3D

 

Posted by at 13:37 in graphics, packages, R | | Comments (1)

August 22, 2018

Using gganimate to illustrate the luminance illusion

Many illusions are based on the fact that our perceptions of color or brightness of an object are highly dependent on the background surrounding the object. For example, in this image (an example of the Cornsweet illusion) the upper and lower blocks are exactly the same color, according to the pixels displayed on your screen.

Here's another simpler representation of the principle, created by Colin Fay (in response to this video made with colored paper). In the animation below, the rectangle moving to the left to right remains the same color throughout (a middling gray). But as the background around it changes, our perception of the color changes as well.

Lightness_illusion

Colin created this animation in R using the gganimate package (available on GitHub from author Thomas Lin Pederson), and the process is delightfully simple. It begins with a chart of 10 "points", each being the same grey square equally spaced across the shaded background. Then, a simple command animates the transitions from one point to the next, and interpolates between them smoothly:

library(gganimate)
gg_animated <- gg + 
  transition_time(t) + 
  ease_aes('linear')

You can find the complete R source code behind the animation at the blog post linked below, along with an interesting discussion of luminance and how you should consider it when choosing color scales for your data visualizations.

RTask: Remaking ‘Luminance-gradient-dependent lightness illusion’ with R

Posted by at 14:21 in graphics, packages, R | | Comments (0)

August 16, 2018

Make R speak

Every wanted to make R talk to you? Now you can, with the mscstts package by John Muschelli. It provides an interface to the Microsoft Cognitive Services Text-to-Speech API (hence the name) in Azure, and you can use it to convert any short piece of text to a playable audio file, rendering it as speech using a number of different voices.

Before you can generate speech yourself, you'll need a Bing Speech API key. If you don't already have an Azure account, you can generate a free 7-day API key in seconds by visiting the Try Cognitive Services page, selecting "Speech APIs", and clicking "Get API Key" for "Bing Speech":

Bing speech

No credit card is needed; all you need is a Microsoft, Facebook, LinkedIn or Github account. (If you need permanent keys, you can create an Azure account here which you can use for 5000 free Speech API calls per month, and also get $200 in free credits for use with any Azure service.)

Once you have your key (you'll actually get two, but you can use either one) you will call the ms_synthesize function to convert text up to 1,000 characters or so into mp3 data, which you can then save to a file. (See lines 8-10 in the speakR.R script, below.) Then, you can play the file with any MP3 player on your system. On Windows, the easiest way to do that is to call start on the MP3 file itself, which will use your system default. (You'll need to modify line 11 of the script to work non-Windows systems.)

Saving the data to a file and then invoking a player got a bit tedious for me, so I created the say function in the script below to automate the process. Let's see it in action:

Note that you can choose from a number of accents and spoken languages (including British English, Canadian French, Chinese and Japanese), and the gender of the voice (though both female and male voices aren't available for all languages).  You can even modify volume, pitch, speaking rate, and even the pronunciation of individual words using the SSML standard. (This does mean you can't use characters recognized as SSML in your text, which is why the say function below filters out < > and / first.) 

The mscstts package is available for download now from your favorite CRAN mirror, and you can find the latest development version on Github. Many thanks to John Muschelli for putting this handy package together!

Posted by at 15:59 in Microsoft, packages, R | | Comments (0)

Next »