Revolutions: Microsoft

Microsoft

Microsoft365R: an R interface to the Microsoft 365 suite

February 09, 2021

I’m very happy to announce Microsoft365R, a package for working with the Microsoft 365 (formerly known as Office 365) suite of cloud services. Microsoft365R extends the interface to the Microsoft Graph API provided by the AzureGraph package to provide a lightweight yet powerful interface to SharePoint and OneDrive, with support for Teams and Outlook soon to come.

Microsoft365R is now available on CRAN, or you can install the development version from GitHub with devtools::install_github("Azure/Microsoft365R").

Authentication

The first time you call one of the Microsoft365R functions (see below), it will use your Internet browser to authenticate with Azure Active Directory (AAD), in a similar manner to other web apps. You will get a dialog box asking for permission to access your information.

Microsoft365R is registered as an app in the “aicatr” AAD tenant. Because it needs read/write access to groups and SharePoint sites, you’ll need an admin to grant it access to your tenant. Alternatively, if the environment variable CLIMICROSOFT365_AADAPPID is set, Microsoft365R will use its value as the app ID for authenticating; or you can specify the app ID as an argument when calling the functions below. See also this issue at the GitHub repo for some possible workarounds.

OneDrive

To access your personal OneDrive, call the personal_onedrive() function, and to access OneDrive for Business call business_onedrive(). Both functions return an R6 client object of class ms_drive, which has methods for working with files and folders. Note that OneDrive for Business is technically part of SharePoint, and requires a Microsoft 365 Business subscription.

od <- personal_onedrive()
odb <- business_onedrive(tenant="mycompany")

# use the device code authentication flow in RStudio Server
od <- personal_onedrive(auth_type="device_code")

# list files and folders
od$list_items()
od$list_items("Documents")

# upload and download files
od$download_file("Documents/myfile.docx")
od$upload_file("somedata.xlsx")

# create a folder
od$create_folder("Documents/newfolder")

You can open a file or folder in your browser with the open_item() method. For example, a Word document or Excel spreadsheet will open in Word or Excel Online, and a folder will be shown in OneDrive.

od$open_item("Documents/myfile.docx")

You can get and set the metadata properties for a file or folder with get_item_properties() and set_item_properties(). For the latter, provide the new properties as named arguments to the method. Not all properties can be changed; some, like the file size and last modified date, are read-only. You can also retrieve an object representing the file or folder with get_item(), which has methods appropriate for drive items.

od$get_item_properties("Documents/myfile.docx")

# rename a file -- version control via filename is bad, mmkay
od$set_item_properties("Documents/myfile.docx", name="myfile version 2.docx")

# alternatively, you can call the file object's update() method
item <- od$get_item("Documents/myfile.docx")
item$update(name="myfile version 2.docx")

SharePoint

To access a SharePoint site, use the sharepoint_site() function and provide the site URL or ID.

site <- sharepoint_site("https://myaadtenant.sharepoint.com/sites/my-site-name")

The client object has methods to retrieve drives (document libraries) and lists. To show all drives in a site, use the list_drives() method, and to retrieve a specific drive, use get_drive(). Each drive is an object of class ms_drive, just like the OneDrive clients above.

# list of all document libraries under this site
site$list_drives()

# default document library
drv <- site$get_drive()

# same methods as for OneDrive
drv$list_items()
drv$open_item("teamproject/plan.xlsx")

To show all lists in a site, use the get_lists() method, and to retrieve a specific list, use get_list() and supply either the list name or ID.

site$get_lists()
lst <- site$get_list("my-list")

You can retrieve the items in a list as a data frame, with list_items(). This has arguments filter and select to do row and column subsetting respectively. filter should be an OData expression provided as a string, and select should be a string containing a comma-separated list of columns. Any column names in the filter expression must be prefixed with fields/ to distinguish them from item metadata.

# return a data frame containing all list items
lst$list_items()

# get subset of rows and columns
lst$list_items(
    filter="startsWith(fields/firstname, 'John')",
    select="firstname,lastname,title"
)

Finally, you can retrieve subsites with list_subsites() and get_subsite(). These also return SharePoint site objects, so all the methods above are available for a subsite.

Future plans

Currently, Microsoft365R supports OneDrive and SharePoint Online; future updates will add the ability to post to Teams channels and send emails via Outlook. You can also provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.

Posted by Hong Ooi at 09:30 in announcements, Microsoft, packages, R | Permalink | Comments (0)

April 15, 2021

Microsoft365R 2.1.0 with Outlook support now on CRAN

by Hong Ooi

I’m happy to announce that Microsoft365R 2.1.0 is now on CRAN with Outlook email support! Here’s a quick summary of the new features:

Send, reply to and forward emails, optionally composed with blastula or emayili
Copy and move emails between folders
Create, delete, copy and move folders
Add, remove and download attachments

Here’s a sample of how to write an email using blastula:

library(Microsoft365R)

# 1st one is for your personal Microsoft account
# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()

# compose an email with blastula
library(blastula)
bl_body <- "## Hello!

This is an email message that was generated by the blastula package.

We can use **Markdown** formatting with the `md()` function.

Cheers,

The blastula team"

bl_em <- compose_email(
    body=md(bl_body),
    footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
                        to="bob@example.com")

# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()

And on the other side, here’s a sample of how to work with the emails in your inbox:

# list the most recent emails in your inbox
emlst <- outl$list_emails()

# get the most recent email
em <- emlst[[1]]

# list and download attachments
em$list_attachments()
em$download_attachment("mydatafile.csv")

# reply to it
em$create_reply("Replying from R")$send()

# list the folders in your account
outl$list_folders()

# move the move recent email to a given folder
folder <- outl$get_folder("My project folder")
em$move(folder)

In addition, this release fixes a bug in the list_files() method for OneDrive/Sharepoint drives and drive items, and adds the ability to create nested drive folders in one call.

Please note that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)

If you have any feedback or comments, you can email me or open an issue at the repo.

Posted by Hong Ooi at 09:30 in Microsoft, packages, R | Permalink | Comments (0)

March 30, 2021

Outlook client support in Microsoft365R available for beta test

by Hong Ooi

This is an announcement that a beta Outlook email client is now part of the Microsoft365R package. You can install it from the GitHub repository with:

devtools::install_github("Azure/Microsoft365R")

The client provides the following features:

Send, reply to and forward emails, optionally composed with blastula or emayili
Copy and move emails between folders
Create, delete, copy and move folders
Add, remove and download attachments

The plan is to submit this to CRAN sometime next month, after a period of public testing. Please give it a try and give me your feedback: either via email or by opening an issue at the repo.

Here’s a small sample of the client in action (taken from the README):

library(Microsoft365R)

# 1st one is for your personal Microsoft account
# 2nd is for your work & school account
outl <- get_personal_outlook()
outlb <- get_business_outlook()

# compose an email with blastula
library(blastula)
bl_body <- "## Hello!

This is an email message that was generated by the blastula package.

We can use **Markdown** formatting with the `md()` function.

Cheers,

The blastula team"

bl_em <- compose_email(
    body=md(bl_body),
    footer=md("sent via Microsoft365R")
)
em <- outl$create_email(bl_em, subject="Hello from R",
                        to="bob@example.com")

# add an attachment and send it
em$add_attachment("mydocument.docx")
em$send()

# list the most recent emails in your inbox
emlst <- outl$list_emails()

# reply to the most recent email
emlst[[1]]$
    create_reply("Replying from R")$
    send()

Please note though, that if you’re using one of the workarounds mentioned in the authentication vignette, they won’t work with Outlook. You’ll need to get the Microsoft365R app approved for your tenant, or alternatively, if you have admin rights you can create your own tenant with the required permissions. (This applies if you’re using Microsoft365R at work; if you’re using it at home on your personal account, you shouldn’t have any problems.)

Posted by Hong Ooi at 09:30 in Microsoft, packages, R | Permalink | Comments (0)

March 10, 2021

Teams support in Microsoft365R

by Hong Ooi

I’m happy to announce that version 2.0 of Microsoft365R, the R interface to Microsoft 365, is now on CRAN! This version adds support for Microsoft Teams, a much-requested feature.

To access a team in Microsoft Teams, use the get_team() function and provide the team name or ID. You can also list the teams you’re in with list_teams(). These return objects of R6 class ms_team, which has methods for working with channels and drives.

list_teams()
team <- get_team("My team")

# list the channels in a team (including your private channels)
team$list_channels()

# get the primary channel for a team
team$get_channel()

# get a specific channel
team$get_channel("My channel")

# drives for a team
team$list_drives()
team$get_drive()

A drive is an ms_drive object, so if you’re already using Microsoft365R to interface with OneDrive and SharePoint document libraries, you already know how to use a team’s drives. Each team will generally have at least one drive, and possibly two: the default “Shared Documents” drive, which is where uploaded files are stored, and the “Teams Wiki Data” drive, if the team has a wiki. Each team channel will usually also have an associated folder in each drive.

drv <- team$get_drive()

# one folder per channel
drv$list_files()

# upload will appear in Files tab of "My channel" in the Teams client
drv$upload_file("myfile.csv", "My channel/myfile.csv")

Channels

A team object has methods for listing, retrieving, creating and deleting channels. However you should not create and delete channels unnecessarily, since Teams tracks all channels ever created, even after you delete them. In turn, a channel object has methods for listing and sending messages, and uploading and deleting files.

Channel messages

Teams channels are semi-threaded. Getting the list of messages for a channel retrieves only the first message in each thread; to get an entire thread, you get the starting message and then retrieve the replies to it. Note that channels don’t have nested replies, so you can’t reply to a reply—only to the starting message.

The body of a message is part of the list of properties returned from the host, and can be found in the properties field of the object. Other properties include metadata such as the author, date, list of attachments, etc.

chan <- team$get_channel()

# retrieve most recent messages from the server
msgs <- chan$list_messages()

# get the latest message by ID
msg <- chan$get_message(msgs[[1]]$properties$id)

# body of the message
msg$properties$body

# 10 most recent replies
repl_list <- msg$list_replies(n=10)

# body of an individual reply
repl_list[[1]]$properties$body

You can send a message to a channel as plain text (the default) or HTML. A message can also include attachments and inline images.

# sending messages to a channel
chan$send_message("Hello from R")
chan$send_message(
    "<div>Hello from <em>R</em></div>", content_type="html")

# attachments and inline images
chan$send_message("Hello with attachments",
    attachments=c("intro.md", "myfile.csv"))
chan$send_message("",
    content_type="html", inline="graph.png")

# send a reply to a message
msg <- chan$send_message("Starting a new thread in R")
msg$send_reply("Reply from R")

Currently, Microsoft365R only supports messaging in channels. Support for chats between individuals may come later.

Channel files

Uploading a file to a channel will place it in the channel’s drive folder. The channel object itself provides convenience functions to list, upload and download files. It also provides a get_folder() method to retrieve the folder for the channel, as an ms_drive_item object; this object has more general methods for working with files.

# files for the channel
chan$list_files()

# upload a file to the channel
chan$upload_file("myfile.docx")

# open the uploaded document for editing in Word Online
chan_folder <- chan$get_folder()
item <- chan_folder$get_item("myfile.docx")
item$open()

# download it again
item$download(overwrite=TRUE)

Providing Feedback

Let us know how this works for you! You can provide feedback and make feature requests by opening an issue at the repo, or by emailing me at hongooi73 (@) gmail.com.

Posted by Hong Ooi at 09:30 in Microsoft, packages, R | Permalink | Comments (2)

January 21, 2021

AzureCosmosR: interface to Azure Cosmos DB

by Hong Ooi

Last week, I announced AzureCosmosR, an R interface to Azure Cosmos DB, a fully-managed NoSQL database service in Azure. This post gives a short rundown on the main features of AzureCosmosR.

Explaining what Azure Cosmos DB is can be tricky, so here’s an excerpt from the official description:

Azure Cosmos DB is a fully managed NoSQL database for modern app development. Single-digit millisecond response times, and automatic and instant scalability, guarantee speed at any scale. Business continuity is assured with SLA-backed availability and enterprise-grade security. App development is faster and more productive thanks to turnkey multi region data distribution anywhere in the world, open source APIs and SDKs for popular languages. As a fully managed service, Azure Cosmos DB takes database administration off your hands with automatic management, updates and patching. It also handles capacity management with cost-effective serverless and automatic scaling options that respond to application needs to match capacity with demand.

Among other features, Azure Cosmos DB is notable in that it supports multiple data models and APIs. When you create a new Cosmos DB account, you specify which API you want to use: SQL/core API, which lets you use a dialect of T-SQL to query and manage tables and documents; MongoDB; Azure table storage; Cassandra; or Gremlin (graph). AzureCosmosR provides a comprehensive interface to the SQL API, as well as bridges to the MongoDB and table storage APIs. On the Resource Manager side, AzureCosmosR extends the AzureRMR class framework to allow creating and managing Cosmos DB accounts.

AzureCosmosR is now available on CRAN. You can also install the development version from GitHub, with devtools::install_github("Azure/AzureCosmosR").

SQL interface

The meat of AzureCosmosR is a suite of methods to work with databases, containers (tables) and documents (rows) using the SQL API.

library(dplyr)
library(AzureCosmosR)

# endpoint object for this account
endp <- cosmos_endpoint(
    "https://myaccount.documents.azure.com:443/",
    key="mykey"
)

# all databases in this account
list_cosmos_databases(endp)

# a specific database
db <- get_cosmos_database(endp, "mydatabase")

# create a new container and upload the Star Wars dataset from dplyr
cont <- create_cosmos_container(db, "mycontainer", partition_key="sex")
bulk_import(cont, starwars)

query_documents(cont, "select * from mycontainer")

# an array select: all characters who appear in ANH
query_documents(cont,
    "select c.name
        from mycontainer c
        where array_contains(c.films, 'A New Hope')")

You can easily create and execute JavaScript stored procedures and user-defined functions:

proc <- create_stored_procedure(
    cont,
    "helloworld",
    'function () {
        var context = getContext();
        var response = context.getResponse();
        response.setBody("Hello, World");
    }'
)

exec_stored_procedure(proc)

create_udf(cont, "times2", "function(x) { return 2*x; }")

query_documents(cont, "select udf.times2(c.height) from cont c")

Aggregates take some extra work, as the Cosmos DB REST API currently only has limited support for cross-partition queries. Set by_pkrange=TRUE in the query_documents call, which will run the query on each partition key range (physical partition) and return a list of data frames. You can then process the list to obtain an overall result.

# average height by sex, by pkrange
df_lst <- query_documents(cont,
    "select c.gender, count(1) n, avg(c.height) height
        from mycontainer c
        group by c.gender",
    by_pkrange=TRUE
)

# combine pkrange results
df_lst %>%
    bind_rows(.id="pkrange") %>%
    group_by(gender) %>%
    summarise(height=weighted.mean(height, n))

Full support for cross-partition queries, including aggregates, may come in a future version of AzureCosmosR.

Other client interfaces

MongoDB

You can query data in a MongoDB-enabled Cosmos DB instance using the mongolite package. AzureCosmosR provides a simple bridge to facilitate this.

endp <- cosmos_mongo_endpoint(
    "https://myaccount.mongo.cosmos.azure.com:443/",
    key="mykey"
)

# a mongolite::mongo object
conn <- cosmos_mongo_connection(endp, "mycollection", "mydatabase")
conn$find("{}")

For more information on working with MongoDB, see the mongolite documentation.

Table storage

You can work with data in a table storage-enabled Cosmos DB instance using the AzureTableStor package.

endp <- AzureTableStor::table_endpoint(
    "https://myaccount.table.cosmos.azure.com:443/",
    key="mykey"
)

tab <- AzureTableStor::storage_table(endp, "mytable")
AzureTableStor::list_table_entities(tab, filter="firstname eq 'Satya'")

Further information

A good introduction to Azure Cosmos DB can be found here, or you can browse the official documentation. If you have any questions or feedback about the AzureCosmosR package, you can open an issue or email me at hongooi73 (@) gmail.com.

Posted by Hong Ooi at 09:30 in big data, Microsoft, R | Permalink | Comments (0)

January 13, 2021

What's new with AzureR

by Hong Ooi

This is an update on what’s been happening with the AzureR suite of packages. First, you may have noticed that just before the holiday season, the packages were updated on CRAN to change their maintainer email to a non-Microsoft address. This is because I’ve left Microsoft for a role at Westpac bank here in Australia; while I’m sad to be leaving, I do intend to continue maintaining and updating the packages.

To that end, here are the changes that have recently been submitted to CRAN, or will be shortly:

AzureAuth now allows obtaining tokens for the “organizations” and “consumers” generic tenants, in addition to “common”. This should make it easier to authenticate users who are using a personal account, as opposed to a work & school account. This is now live on CRAN.

AzureStor gains convenience functions for transferring data in several commonly-used formats: RDS (as created by saveRDS/readRDS), RData (as created by save/load), CSV, CSV2, and tab-delimited. These work via connections and so don’t create temporary files on disk.

AzureRMR makes it easier to retrieve subresources by adding a get_subresource method to the az_resource class. For example, if res is a storage account resource, you can obtain the subresource for a particular blob container with res$get_subresource(type="blobServices/default/containers", name="containername"). It now authenticates using AAD v2.0 by default.

AzureGraph has had several changes:

Like AzureRMR, it also now authenticates using AAD v2.0 by default, making it more convenient to use with personal accounts (which require AAD v2.0).
get_graph_login is much more flexible, letting you choose authentication parameters like the permission scope(s), app ID, and authorisation method.
The internals have undergone a refactoring to enhance extensibility.

All of these changes are put to good use in the new Microsoft365R package, which is intended to be an interface to Microsoft 365 (formerly known as Office 365). Currently, it enables you to access your data stored in OneDrive (personal or business) and SharePoint Online, using the Microsoft Graph API. This will be released to CRAN in the next few weeks; later versions may add support for other Microsoft 365 services.

The new AzureCosmosR package is an interface to Azure Cosmos DB. Like the other interface packages in AzureR, it provides both management plane and data plane access, in the latter case to the SQL (core) API. You can easily create and manage Cosmos DB accounts, run queries, and create and execute stored procedures. Like Microsoft365R, this should be released to CRAN in the next few weeks.

If you have any comments or feedback, feel free to drop me a line at hongooi73@gmail.com.

Posted by Hong Ooi at 09:30 in Microsoft, R | Permalink | Comments (0)

December 17, 2020

Azure Functions with R and plumber

Azure Functions is a cloud service that allows you to deploy “serverless” microservices that are triggered by events (timers, HTTP POST events, etc) and automatically scale to serve demand while minimizing latency. The service natively supports functions written in C#, Java, JavaScript, PowerShell, Python and TypeScript, and now supports other languages as well thanks to the launch last week of custom handlers for Azure Functions.

A new tutorial walks you through the process of creating a custom handler for a “hello world” R function. The process is fairly straightforward: use a couple of Azure CLI commands to set up a project on your local machine and create Azure resources, write a “handler” script in R to provide a Web service, and push a Docker container with the Azure Functions runtime, R engine, and packages needed to implement your Function. Then, when you want to update your Function all you need to do is push a new version of the container image with the updated R code. The video below shows a brief demo of the process in action:

The tutorial uses the httpuv package to implement a stripped-down Web server to implement the Function, but you can make things easier for yourself (at a small cost to performance) by using the plumber package. With plumber, you can easily annotate an R function you already have, and make it into a web service suitable for Azure Functions.

In this GitHub repository you’ll find code that implements an Azure Function to predict from a GLM model trained with the caret package, and also a Shiny app to consume the function. Follow the instructions in the README.md file to deploy your Function, and then launch the associated Shiny app. As you adjust the parameters on the left side of the application, the chart on the right updates in real time with an estimate of the probability that a car accident with those parameters would be fatal.

It’s important to note that the model prediction is not being generated by the Shiny app: rather, it’s being generated by an Azure Function running R in the cloud. That means you could integrate the model estimate into any application written in any language: a mobile app, or an IoT service, or anything that can call an HTTP endpoint. Furthermore, you don’t need to worry how many apps are running or how often estimates will be requested by the app: Azure Functions will automatically scale to meet the demand as needed.

Thanks to plumber, the “handler function” providing the Web service is very simple. All I needed to do was to decorate an existing function that made a prediction based on parameters with a couple of comments to make it into an HTTP POST endpoint compatible with Azure Functions.

#* Predict probability of fatality from params in body
#* @post /api/accident
function(params) {
  model_path <- "."
  model <- readRDS(file.path(model_path, "model.rds"))
  method <- model$method

  message(paste(method, "model loaded"))
  
  prediction <- predict(model, newdata=params, type="prob")[,"dead"]

  return(prediction)
}

The only other trick is to create a container containing R and the packages to support plumber and your model, but if you’re familiar with Dockerfiles this should be quite straightforward.

If you’d like to try it out yourself, you’ll find complete instructions at the repository linked below.

GitHub (revodavid): Azure Functions for R with Custom Handlers

Posted by David Smith at 09:30 in Microsoft, mlops, R | Permalink | Comments (1)

December 11, 2020

R at Microsoft

I was my great pleasure yesterday to be a presenter in the "Why R Webinar" series, on the topic R at Microsoft. In the talk (which you can watch below) I recounted the history of Microsoft's acquisition of Revolution Analytics, and the various way the Microsoft supports R: its membership of the R Consortium, integration with many products (including demos of Azure ML Service with GitHub Actions, and Azure Functions), and how Microsoft has driven adoption of R in large organizations by making R "IT approved". Many thanks to the Why R Foundation for hosting the talk, and to everyone who asked questions and prompted a lively discussion at the end:

You can also find links and resources associated with the talk, including the slides, at the GitHub repository linked below.

GitHub (revodavid): R at Microsoft

Posted by David Smith at 10:39 in Microsoft, mlops, R | Permalink | Comments (0)

November 25, 2020

Attend the Create:Data free online event, December 7

The Microsoft Create: series is back again, now with Create: Data!

Join us for a half-day of conversations at Microsoft Create: Data and connect with the experts and community to learn and discuss everything data - from the upcoming trends to best practices and data for good.

Join our virtual event Create: Data

Date: 7 December 2020

Time: 8:00AM – 11:10AM PDT / 4:00PM – 7:10PM GMT

Join us!

In this event you will hear from our keynote speakers Arun Ulagaratchagan and Heather Newman share how to drive a data culture in a world of remote everything. Learn about Apache Spark, Data Pipelines and how to pick the right distributed database from other notable community leaders like Holden Karau, Cheryl Adams, Tim Berglund, Jacek Laskowski, Jasmine Greenaway and Abhishek Gupta.

Our exciting line-up of panellists, Eve Pardi, Ruth Yakubu, Adi Shacham-Shavit and Anna Hoffman will share how data can be used for good and we will wrap up with a closing keynote by Saveen Reddy and Simon Whiteley.

Skill-up

At Microsoft, our mission is to empower every person and every organization on the planet to achieve more. Join us for a unique workshop led by Adi Polak and Simon Whiteley to skill you up, where you will learn about Data Analytics, SQL and Data Visualization. Most importantly, you will gain hands-on experience with running interactive queries using Azure Synapse SQL Serverless! We'll reward you with limited Azure Heroes Badgers upon completion of the workshop!

Create: Data is part of #Dataweek

Join us for a full week of celebrating everything Data & Data Technologies! Data week includes expert panel discussions on industry trends & the future of data, AMA events with Microsoft Engineers and Product Managers, plus hands-on demonstrations and workshops with the engineers who built the tools! Join us for a week of Data-Related learning! Learn about what’s planned for you this #dataweek.

Create Data: Register here

Posted by David Smith at 09:00 in big data, events, Microsoft | Permalink | Comments (0)

November 18, 2020

AzureTableStor: R interface to Azure table storage service

by Hong Ooi

I’m pleased to announce that the AzureTableStor package, providing a simple yet powerful interface to the Azure table storage service, is now on CRAN. This is something that many people have requested since the initial release of the AzureR packages nearly two years ago.

Azure table storage is a service that stores structured NoSQL data in the cloud, providing a key/attribute store with a schemaless design. Because table storage is schemaless, it’s easy to adapt your data as the needs of your application evolve. Access to table storage data is fast and cost-effective for many types of applications, and is typically lower in cost than traditional SQL for similar volumes of data.

You can use table storage to store flexible datasets like user data for web applications, address books, device information, or other types of metadata your service requires. You can store any number of entities in a table, and a storage account may contain any number of tables, up to the capacity limit of the storage account.

AzureTableStor builds on the functionality provided by the AzureStor package. The table storage service is available both as part of general Azure storage and via Azure Cosmos DB; AzureTableStor is able to work with either.

Tables

AzureTableStor provides a table_endpoint function that is the analogue of AzureStor’s blob_endpoint, file_endpoint and adls_endpoint functions. There are methods for retrieving, creating, listing and deleting tables within the endpoint.

library(AzureTableStor)

# storage account endpoint
endp <- table_endpoint(
    "https://mystorageacct.table.core.windows.net",
    key="mykey")
# Cosmos DB w/table API endpoint
endp <- table_endpoint(
    "https://mycosmosdb.table.cosmos.azure.com:443",
    key="mykey")

create_storage_table(endp, "mytable")
list_storage_tables(endp)
tab <- storage_table(endp, "mytable")

Entities

In table storage jargon, an entity is a row in a table. The columns of the table are properties. Note that table storage does not enforce a schema; that is, individual entities in a table can have different properties. An entity is identified by its RowKey and PartitionKey properties, which must be unique for each entity.

AzureTableStor provides the following functions to work with data in a table:

insert_table_entity: inserts a row into the table.
update_table_entity: updates a row with new data, or inserts a new row if it doesn’t already exist.
get_table_entity: retrieves an individual row from the table.
delete_table_entity: deletes a row from the table.
import_table_entities: inserts multiple rows of data from a data frame into the table.

insert_table_entity(tab, list(
    RowKey="row1",
    PartitionKey="partition1",
    firstname="Bill",
    lastname="Gates"
))

get_table_entity(tab, "row1", "partition1")

# we can import to the same table as above:
# table storage doesn't enforce a schema
import_table_entities(tab, mtcars,
    row_key=row.names(mtcars),
    partition_key=as.character(mtcars$cyl))

list_table_entities(tab)
list_table_entities(tab, filter="firstname eq 'Satya'")
list_table_entities(tab, filter="RowKey eq 'Toyota Corolla'")

Batch transactions

With the exception of import_table_entities, all of the above entity functions work on a single row of data. Table storage provides a batch execution facility, which lets you bundle up single-row operations into a single transaction that will be executed atomically. In the jargon, this is known as an entity group transaction. import_table_entities is an example of an entity group transaction: it bundles up multiple rows of data into batch jobs, which is much more efficient than sending each row individually to the server.

The create_table_operation, create_batch_transaction and do_batch_transaction functions let you perform entity group transactions. Here is an example of a simple batch insert. The actual import_table_entities function is more complex as it can also handle multiple partition keys and more than 100 rows of data.

ir <- subset(iris, Species == "setosa")

# property names must be valid C# variable names
names(ir) <- sub("\\.", "_", names(ir))

# create the PartitionKey and RowKey properties
ir$PartitionKey <- ir$Species
ir$RowKey <- sprintf("%03d", seq_len(nrow(ir)))

# generate the array of insert operations: 1 per row
ops <- lapply(seq_len(nrow(ir)), function(i)
    create_table_operation(endp, "mytable", body=ir[i, ],
                           http_verb="POST")))

# create a batch transaction and send it to the endpoint
bat <- create_batch_transaction(endp, ops)
do_batch_transaction(bat)

If you have any feedback, or to report bugs with the package, please contact me at hongooi73@gmail.com or open an issue on GitHub.

Posted by Hong Ooi at 09:30 in Microsoft, packages, R | Permalink | Comments (0)

September 23, 2020

Microsoft R Open 4.0.2 now available

Microsoft R Open 4.0.2 has been released, combining the latest R language engine with multi-processor performance and tools for managing R packages reproducibly. You can download Microsoft R Open 4.0.2 for Windows,and Linux from MRAN now. Microsoft R Open is 100% compatible with all R version 4 scripts and packages, and works with all your favorite R interfaces and development environments.

This update brings R version 4 to Microsoft R Open for the first time. This update includes many new features for the R language and system, along with improved performance and memory usage. Note that you will need to install any R packages you wish to use with MRO 4.0.2. By default, MRO installs packages from a static CRAN snapshot taken on July 16 2020.

We hope you find Microsoft R Open useful, and if you have any comments or questions please visit the Microsoft R Open forum. You can follow the development of Microsoft R Open at the MRO GitHub repository. To download Microsoft R Open, simply follow the link below.

MRAN: Download Microsoft R Open

Posted by David Smith at 16:30 in announcements, Microsoft, R | Permalink | Comments (4)

Milestones in AI, Machine Learning, Data Science, and visualization with R and Python since 2008

Microsoft

Authentication

OneDrive

SharePoint

Future plans

April 15, 2021

March 30, 2021

March 10, 2021

Channels

Channel messages

Channel files

Providing Feedback

January 21, 2021

SQL interface

Other client interfaces

MongoDB

Table storage

Further information

January 13, 2021

December 17, 2020

December 11, 2020

November 25, 2020

November 18, 2020

Tables

Entities

Batch transactions

September 23, 2020

Information

Search Revolutions Blog

R links

Recommended Sites