Digitization 101

In response to a complaint we received under the US Digital Millennium Copyright Act, we have removed 2 result(s) from this page. If you wish, you may read the DMCA complaint that caused the removal(s) at LumenDatabase.org. 

This quote seems to be all anyone needs in order to be tempted to read this article:

...the idea that somewhere at Google there is a database containing 25-million books and nobody is allowed to read them. It’s like that scene at the end of the first Indiana Jones movie where they put the Ark of the Covenant back on a shelf somewhere, lost in the chaos of a vast warehouse.

Photo from NIH Library

• Using google translate in a novel setting
• Finding help on academic assignments 

• The literacy of information
• Knowing what the information is...
• How to use and interact with the information 
• Knowing how to use information in "hand to hand" combat

• Survey of 2225 US-English Internet users, 90.5% do not onion how to "find" on a page.
• 51.1% of 545 US-English Internet using teachers do not know how to "find" on page.

• Learn how to ask the right questions.
• Know what tools are available.

• Understand space of information available
• Can search effectively
• Can understand how to interpret the results

• Take a class...continue to learn more
• Become more aware
• Subscribe to 
• Understand
• Play a serious game 
• Teach a class

Dr. Russell's contact info

Tim Wu writes:

Today, the project sits in a kind of limbo. On one hand, Google has scanned an impressive thirty million volumes, putting it in a league with the world’s larger libraries (the library of Congress has around thirty-seven million books). 

I remember when everyone was talking about Google Books. It grabbed our imaginations. As a digitization consultant, I thought it was something we'd be able to learn from. However, the ability to scan millions of books ran into the reality of authors' rights.

If you've not thought about this project for a while, this article sums it all up nicely. Of course, what it doesn't say is whether we'll ever get the access to those 35,000,000 volumes in the way that we want.

I heard today that the classic version of Dialog® is on its last legs. All of the databases that were available in the old version of Dialog are now available in ProQuest Dialog. While I recognize that the command line version of Dialog is not what today's searchers want to use...and while I recognize that our technology has gotten better...I mourn.

Many librarians learned how to search for electronic information on the Dialog system.  We learned commands, tricks, and short cuts.  We memorized details about specific databases and consulted documentation to double-check details so that we didn't spend extra time or money online.  We spoke in phrases peppered with file numbers and field names.  We shared stories of commands (or searches)  gone wrong and the charges that they caused.  And we smiled.

In talking to someone from ProQuest today, I used the phrase "under the hood."  She immediately thought I was referring to a 2001 article by Carol Tenopir entitled "Why I still teach Dialog" (Library Journal, 126(8), 35-36), which is available through ProQuest.  In it, Tenopir writes:

For new students in an LIS program, DialogClassic helps them understand the workings of the systems they will be searching, teaching, or designing. 

This is true.  It is also true that teaching someone to search Dialog's old command line interface takes patience. The learner must be willing to try and fail, and learn from those failures as well as the successes.  It is also true that most - if not all - of our searching these days is not done with a command prompt, but through some Google-like interface or an advanced search screen.  It is difficult to teach students something that they will not use after the class is over.  Face it, no one is going to run home and search using a command prompt just for fun!

However, I believe that understanding the old ways helps you understand how we do things now. All of us that used Dialog in the old days have a knowledge about the system that our younger counterparts will never have.  Although hard to quantify, I would argue that there is value to that knowledge.

The Power of Full-Text 

I remember when full-text records came to Dialog and the power that came with it.  I no longer had to use a document delivery service to obtain the full-text.  Not only was that a cost saving, it also meant I could get the full-text to my client faster.  Initially the full-text was not searchable.  When it became searchable, it was revolutionary!  Now we take full-text searching for granted.  And instead of have full-text that is in ASCII, we have full-text that is presented in PDFs with graphics, etc., intact.  The addition of full-text has been due to re-typing, scanning, and other methods.  Some of it has come with added errors. All of it has been appreciated.

By the way, could we look at those companies that produced the Dialog databases as being early pioneers in digitization?  Yes, I think so.

First Dialog, then Google

We can also argue that Dialog paved the way for many services, including Google.  I remember working with programmers on DR-LINK, a product of TextWise, and calling upon my Dialog knowledge in helping them make sense of the files that had to be turned into coherent databases.  The command level search had taught me much about file structures and expectations. I'm sure that others that have built services have had the same revelations not from using fancy interfaces, but from "getting their hands dirty" at the command prompt.

In Memoriam

I wonder if ProQuest will throw a virtual event when DialogClassic is finally turned off?  Or perhaps some of us "old Dialog searchers" will just find a way to gather, light candles, and tell stories of a great system that began it all...

Like many others, I was mesmerized by Google's digitization program and then by the litigation that followed.  The Copyright Clearance Center's Beyond the Book podcast has covered Google's efforts, the lawsuit, and the aftermath and given many of us a concise summary of what is occurring.  This 15-minute podcast entitled "The Battle for the Books" is an interview with journalist and attorney Jeff Roberts provides an overview of it all and some of the elements that you may not have considered (e.g., "East Coast against West Coast, Manhattan Island versus Silicon Valley, technology upstarts taking on publishing elites").  If you have not kept up with the lawsuit, this will be you quickly up-to-date and likely leave you wanting to know more.

This podcast is also, in essence, an introduction to Roberts book The Battle for the Books: Inside Google's Gambit to Create the World's Biggest Library.The book is available for the Kindle, which points at something that occurred over the last 10+ years; Google is not the dominate purveyor of books that we thought it might become.  Instead Amazon marshaled its forces and created an ebook distribution service that many of us use.

In talking about the disposition of The Authors Guild et al v. Google Inc. and American Society of Media Photographers et al v. Google Inc., this Reuters article notes:

Thousands of authors can sue Google Inc. in a class-action lawsuit over its plan to create the world's largest digital book library, a federal judge ruled on Thursday.

This decision today comes more than a year after Judge Chin "cited antitrust and copyright concerns in rejecting a proposed comprehensive $125 million settlement, saying it went 'too far' in letting Google effectively conduct 'wholesale copying of copyrighted works without permission.'"

I'm sure more people will be writing about this and I look forward to what they have to say.

Addendum (6/5/2012): The Copyright Clearance Center has written an article on this.

Reading the article, this text stood out to me:

Google isn't saying whether it has pulled back from its longstanding goal of collecting all of the world's knowledge. Some of its digitization efforts have shifted to Europe. Much of the company's public focus lately has been not on mass digitization but on how to use individuals' data to create more focused advertising and online browsing. Meanwhile, a copyright-infringement lawsuit brought against it by authors' and publishers' groups drags on. 

The company has been digitizing books since 2006 and converted millions of books to digital form. It could be that Google has hit a maturation point or it could be something else.  For example, has Google gotten what it needs out of this effort (whatever that might be)?  (There were rumors early on what why Google might be doing this work, e.g., making its search engine better or capturing more eyeballs on the Internet, and maybe those needs have been satisfied.)  

I cannot let the last Wednesday in 2011 go by without looking backwards over the last 12 months. What stands out amid the growing din of the "news"?

Google

• Google shut down its newspaper digitization program. (post) This was one of many things that Google did away with in 2011, in an effort to rid itself of those products and services that have not had the desired impact. (article)  Of course, after its long shopping spree, something was bound to be let go. (post)
• Google's amended Book Search settlement was rejected. (post) This was, of course, a surprise to no one.

HathiTrust

• In 2011, the Authors Guild turned its attention to the digitization work that the HathiTrust had been engaged in. (post)  The trial is scheduled to begin in November 2012. (article)

Georgia State 

• I keep thinking that the copyright lawsuit against members of the George State University administration should be settled by now.  The judge was expected to release his decision in early fall.  I've searched for any recent news and found none. Because so many colleges and universities are using digital course reserves, this will have far reaching implications. (related blog post)

And...yes...those all (above) have to do with copyright. 

Kenneth Crews (post), Clifford Lynch (post) and Henrik de Gyor 

• I am always amazed by the people I get to talk to...from Henrik de Gyor (Another DAM Podcast), who is a fellow blogger, to people like Clifford Lynch and Kenny Crews. While these weren't news highlights for you, they were for me!

Martin King III and Ambassador Andrew Young 

•  In April, I was invited to a meeting with Ambassador Andrew Young and Martin King III.  The photo on the right was taken on my iPhone and you can see Ambassador Young checking his iPhone!
• Is this digitization related?  Yes.  JPMorgan Chase has been working with the King Center to digitization over one million documents.  (article)  This fall, Syracuse University's library announced plans to digitize audio and video materials in the King Center archive. (article)  The idea for SU to get involved with digitizing materials at the King Center was born at this meeting.
• There were other digitization-related ideas that came out of this meeting, and I hope they come to fruition.
• The lesson...you never know who is interested in digitization! The project of your dreams may be waiting for you in the next meeting that you attend.

Amazing Digitization Programs

• There are many amazing digitization programs going on now and they people involved aren't always who you would imagine.  For example, it's JPMorgan Chase that is working with the King Center on its digitization efforts. JPMC didn't hand the project off to someone else, instead they learned what needed to be done, how to do it, and then got to work.
• Among the programs that I should be following more closely is the Digital Public Library of America.  This sounds like an effort that more people and organizations need to know about and get involved in. 
• The need to handle "big data" - which can be created through digitization - is growing, and so some of the "projects" people need to get involved in are around analysis, open access, preservation, etc.  These projects may not be glamorous, but they are definitely necessary.
• We have so much born digital content now that comes to us in a variety of way, that digitization doesn't have that "oh wow" affect on people. People are concerned about ebooks, new apps, tablet computers, smartphones, etc.  Digitization remains important when people look for something from the non-computer era, but that isn't something that people do every day.  Does this mean that we should digitize less?  No.  But it does mean that we need to continue to educate people about why it is important.

On a Personal Note

• I wrote more than 130 blog posts this year in Digitization 101.  While that will sounds like a lot to some people, actually my blogging has slowed down...and my focus has shifted.  I find myself drawn more to copyright concerns these days, even though the topic of digitization is important to me (and my teaching).   In 2012, look for a continued stream of posts on digitization, digital libraries, copyright, etc., but don't be surprised if you see a greater proportion of blog posts on copyright.
• Teaching at Syracuse University has kept me quite busy.  (Sometimes too busy!)  Yet this was a prolific year for me in terms of publications.  In March, The Information and Knowledge Professional's Career Handbook written by Ulla de Stricker and I was released and has received positive reviews.  In April, The Atlas of New Librarianship, which was written and edited by David Lankes, was released.  It includes a section on "special librarians" written by Ruth Kneale and I.  And finally, Academic Entrepreneurship and Community Engagement: Scholarship in Action and the Syracuse Miracle which contains a chapter that I wrote.
• I've done my best to enjoy every day!  I hope you've done the same.
  

The research in this 14 minute TED video was made possible by Google's book digitization efforts. The two researchers present this serious information in a fun way and demonstrate that reading isn't the only thing these digitized books are good for.

The case of Tasini v. New York Times, which deals with the copyrights of freelance writers, was first settled in 1997.  Because of various appeals and rulings, the settlement was not final and:

On August 17, the Second Circuit Court of Appeals sent the parties in the long-running class action suit known shorthand as Freelance back to the drawing board, rejecting an $18 million settlement struck in 2005.

This article talked about the current status and what it means for:

• For Publishers & Database Operators
• For the Plaintiffs
• For Libraries and the Public
• For the Google Settlement 

If you have article databases that you or license, are a freelance author yourself, or are just interested in how this could impact Google, then take time to read this article.

It will be interesting to note what happens next...and when....

Yes, Google is shutting down one of its digitization efforts.  In a statement to Search Engine Land, a Google spokesperson said:

Users can continue to search digitized newspapers at http://news.google.com/archivesearch, but we don’t plan to introduce any further features or functionality to the Google News Archives and we are no longer accepting new microfilm or digital files for processing.

Google's efforts were in partnership with several North American newspapers, ProQuest and Heritage Microfilm, according to a 2008 news report.

In reporting on Google's decision, the Boston Phoenix wrote:

News Archive was generally a good deal for newspapers -- especially smaller ones like ours, who couldn't afford the tens or hundreds of thousands of dollars it would have cost to digitally scan and index our archives -- and a decent bet for Google. It threaded a loophole for newspapers, who, in putting pre-internet archives online, generally would have had to sort out tricky rights issues with freelancers -- but were thought to have escaped those obligations due to the method with which Google posted the archives. (Instead of posting the articles as pure text, Google posted searchable image files of the actual newspaper pages.) Google reportedly used its Maps technology to decipher the scrawl of ancient newsprint and microfilm; but newspapers are infamously more difficult to index than books, thanks to layout complexities such as columns and jumps, which require humans or intense algorithmic juju to decode. Here's two wild guesses: the process may have turned out to be harder than Google anticipated. Or it may have turned out that the resulting pages drew far fewer eyeballs than anyone expected.

The lesson is that jumping on the Google bandwagon can be good thing, if the wagon keeps on moving. A lesson that those involved in Microsoft's book digitization program also learned the hard way.

---

Addendum (10:53 a.m.): Gary Price at INFOdocket wrote a good piece on this.  Price noted:

New leadership is in place at Google and new leadership can often bring changes. This is likely one of them.

Gizmodo has an article and link to video (below).  The article states:

Google [employees] work under a caste system: employees wear different colored badges to indicate their status. That's not new or surprising, and doesn't really mean much, other than that you can spot a Google intern (literally) a half a mile away. They're the ones in green. White badges are full-timers, and red badges—numbering in the thousands—are contractors...there are the yellow badges. A class of employee that exists largely apart from the rest of the Google hive, sequestered to building 3.1459~, denied the benefits that nearly everyone else shares in.

As it turns out, those with yellow badges are reportedly scanning materials for Google Book Search.

The video is Andrew Norman Wilson discussing how he stumbled upon this information.  In the video, Wilson's delivery is purposefully flat, perhaps to take the emotion out of the piece and leave you focused on the words and images (both on the screen and those being created in your mind).  It is both a news story and an art piece. 

If you decide to watch the video, be sure to read the Gizmodo article too for a slightly different perspective.

BTW Andrew Wilson is a 2006 graduate of Syracuse University's Newhouse School of Communications, which makes me even more interested in the video and story. (Go Orange!)

Lois Wasoff recommended this readings during a Copyright Clearance Center webinar on this topic (March 30, 2011):

• Inside Judge Chin's Opinion 
• Book Ruling Cuts Options for Google
• Google Block: The Google Books settlement should teach the company humility. It won't.
• A Digital Library Better Than Google’s

This one-hour free webinar is being presented by the Copyright Clearance Center.  As their web site says:

As you very likely know, on March 22, Judge Chin issued his long awaited opinion in the Google Book Search settlement proceedings, rejecting the Amended Settlement Agreement (ASA) proposed by the Authors Guild, AAP and Google.

On Wednesday, March 30, 2011, 12:00pm EST, join copyright expert Lois Wasoff and me as we analyze this highly-anticipated decision, what it means for those affected by the proposed settlement, and what is likely to happen next.

To register for this free, one-hour event click here.

In his 48-page opinion, Judge Chin discussed the various objections before the court, including concerns regarding copyright, international law, antitrust, privacy and the class action/procedural aspects of the case – ultimately concluding that the ASA is not “fair, adequate and reasonable” as required for court approval of a settlement.

Judge Chin did, however, leave the door open for the parties to renegotiate and resubmit the settlement, urging them to consider adopting an “opt-in” rather than “opt-out” model which would ameliorate many of the concerns raised in the objections.

1992 participants in CIL this year including exhibits, speakers, etc. Will go over 2000 with onsite registrations.  People from:

• 44 states plus DC
• 14 countries outside the US

The keynote presenter from Google tried to take the red-eye to get here for this morning's session and failed!  The flight ran late, so Roy Tennant, Dick Kaser,Stephen Abrams and Marshall Breeding filled in admirably.  Roy was the moderator. (He will speak during lunch, however, and it will be streamed and archived.)

First question – What are some of the implications they see with the new players in the ebook environment?

Marshall – Wasn’t long ago that digitizing the world books was inconceivable. But you can’t count on Google for everything! Google has the resources to go into the world’s best libraries and digitize them. A world changing phenomenon. He looks at it from the aspects of discovery. You can now search across books. Search for and search inside. The challenge is this search aspect. The discoverability is more exciting.

Stephen – What are the unintended consequences? Is this really about books? What are the consequences of putting all of these books online? Immersive entertainment experience versus the information finding experience. Disaggregation of books. What is the difference between chapters, books, articles? The books are discoverable, but it is the ability to search in that will change things.

Dick – Analogies to the journal digitization that occurred. There was a controversy when the first libraries signed with Google. Remember that Google is a vendor. Interested in the impact of digitization on what libraries do. With this content outside of the library catalogues, what is left for the library catalogue to do?

Libraries can digitize materials from their community and from their online collection – data, archival records. National Archives received money from Ancestry.com…so getting money from companies is not out of the question.

Roy – Years ago, he argued that the Library of Congress would not be fully digitized. He is now willing to eat his hat!

Stephen – Why do you digitize books? Demand Media has said (1) To put advertising within reading. (2) Use the books to control search engine results. In an advertising-based company (Google), why digitize books?

Marshall – Libraries have more books due to the books that are digitixzed. Enticing for smaller libraries. Have to do a “deal with the devil” in order to do it. Thinking of the Google Book Settlement, do libraries get what they want and need?

Internet Archive approaches digitization in a more library friendly way. Yes, libraries have to pay for part of the digitization that the Internet Archive does. However, the results are more frienedly. WE need to find the right deals that benefit libraries and library users the most.

Dick – What is the sttus of the books being digitized by Google? Can we really see the content from all of the books? Are some books still under copyright protection?

The publishers are digitizing books very rapidly. There is a vast rush to digitize books and make them available to ereaders. There is an ebook standard. Also HMTL5 – more flexibility.

Stephen – Sometimes in 2011 the Supreme Court will make a decision about the in-copyright books being digitized by the Google Books Project. The Google Books Project will limit access to the content at the libraries who have access to it. Libraries will need to pay for additional access –printing/saving. This is a major disruption in the force.

Roy – Harper Collins – the ramifications of leasing access to content.

Dick – IN the Netherlands the 26 loans is seen as fair. (odd) Lending ebooks scares publishers.

Marshall – Libraries are based on legal models that apply only to books. What is the library’s role when things are streamed? What is the library’s role with things are published digitally? There is a real struggle between what the publishers are worried about and what the libraries understand how to do. Most books will soon be in digital form.

Dick – ebooks are in place in many research centers. Lack of DRM.

Stephen – The Harper Collins solutions is a version of whack-a-mole. The two biggest publishers won’t let libraries have their ebooks at all (i.e., Simon Schuster). Why are we upset with Harper Collins? There are others that we need to be concerned with.

Obama Administration announced that it is going to refocus on copyright.

Roy – What kind of impact of having these massive collections of ebooks will have on our print collections? Can we ship more books to storage? Can we get rid of a few?

Marshall – A shift in how we handle backfiles. Put things in remote storage. The working collections will become smaller. The vast amounts of materials, if discoverable, will be available in some way. Long form readng is different than reading articles.

Stephen – Do I want Amazon to control what goes on an ereader? Do we want Apple to control how we read/comment on books? Do we want someone to have that type of power? How many libraries would allow one person to ban a book? Yet we allow Jeff Bezos to ban books. Why is the library profession silent on this issue?

One of my students found this program and it is impressive.  Imagine being able to walk through an art museum without physically being there?  That is what Google allows you to do.  Google used its street view technology to make this happen.  While street view isn't the easiest thing to use, it does give you a capability not found elsewhere.  Go ahead...try it out!

Since reading my student's blog post, Kenneth Crews has also blogged about the Google Art Project from a copyright perspective.  My student has also followed up with a blog post about copyright and terms of use in regards to the program. 

While I'm thrilled that Google has done this, I do wonder about its motives.  Is this just a cool use of street view? Is this an experiment (project) or something that Google will truly stick with (program)?  Is Google purposefully expanding its domain/reach?  And...what is the catch?  mmm....

The TechCrunch blog post and original graphic from Scores.org are both worth looking at since the text and comments do not mirror each other.  What is fascinating is that Google has acquired SO much more than any of us realize, and this list is not complete.  For example, it's acquistion of the Papers of Record is not shown. And it does not show partnerships, which is what has impacted Google's digitization efforts.

You'll wonder about the colors, etc., and what it all means.  There is a key at the top.  Blue seems to signify that the acquisition was integrated into Google's offerings.  Note that some companies were acquired and left on their own.

By the way, someone has constructed a table showing the acquisitions of Google, Microsoft, Yahoo and Amazon, whom we might now want to call the 'big four'.


Research by Scores.org

My blog post on July 20 about the Copyright Clearance Center and Beyond the Book content on the Google Book Settlement was sparked by a book I received in the mail from Chris Kenneally at Beyond the Book of transcripts from several Lois Wasoff webinars.  In a Twitter conversation, Kenneally noted that the book was something they were testing and that all of the transcripts were on the CCC web site.  That sounded like a challenge to me!  So I set off to find the transcripts that are in the book, which also lead me to addition content relevant to the Google Book Settlement.

So I was inspired by what the CCC had done, and the Kenneally was ten inspired by my blog post.  Hence, the book is now available as a PDF on the Beyond the Book web site. {Chris, thank you!}

As a side note, the CCC and Beyond the Book create a ton of content on copyright that is available for free.  Chris Kenneally does weekly podcasts at Beyond the Book about copyright, publishing, and new ways of thinking about the content we create.  On the CCC web site, look under the education section for tools, news, guidelines, reports and more.  You might want to check it out and see what would be useful to you.

• Before Google ‘Fairness’ Hearing, Wasoff Reviews Latest, Recorded February 12, 2009 (This file has two different dates associated with it, so this date may not be correct.)
• What Authors and Publishers Need to Know as May 5th Approaches presented by Lois Wasoff, Esq., Recorded on April 14, 2009
• What's Next for Authors and Publishers? An update from Lois Wasoff, Esq., Recorded on June 22, 2009
•  Lois Wasoff Returns to Discuss What's Next for the Google Settlement, Recorded Sept 23, 2009
• Google Book Settlement Amended: Copyright Law Expert Lois Wasoff Highlights the Changes, Recorded on November 18, 2009
• Google Book Settlement: Copyright Law Expert Lois Wasoff Reviews the Amended Settlement, Recorded on December 10, 2009
• Reactions to Google's Amended Settlement Agreement: Copyright Law Expert Lois Wasoff discusses the latest reaction and commentary, Recorded on February 12, 2010

In addition, these transcripts may be of interest:

• A Conversation with Michael Healy, Executive Director of the Book Industry Study Group and expected Executive Director of the Book Rights Registry (outline of benefits)
• Allan Adler, Vice President of the Association of American Publishers (AAP) Speaks About the Settlement with CCC

Would be cool if the CCC repackaged some of these into a ebook or something?  Seems like content that should be repurposed....