Brewster Kahle's Blog

I never used the tools of Big Data and artificial intelligence to build profiles of masses of people. And on purpose. I realized what could be done– the power in knowing too much, the temptation to manipulate. But now others have crossed that line. Total Information Awareness; Facebook, Equifax. Now we see the problems, now we have a problem, and we need ideas on where to go from here.

I started collecting databases in the early 1980s at the Artificial Intelligence Lab at MIT and at Thinking Machines in the 1980s. I knew what could be done as we built supercomputers.

The artist David Byrne wrote, “In the Future, there will be so much going on, no one will be able to keep track of it.”   His statement haunted me in the early 1980s, I felt it was importantly wrong. “No one” person would could keep track, but computers could. I read a book in the late 1970’s about how computers were used in South African apartheid. It turns out it was very simple record keeping computer, a couple of mainframes from England (IBM did not want to sell to them) was all it took to keep a population of millions under control.

I spent time at library school in the 1980’s and studying the information habits of high priced consultants at KPMG Peat Marwick. I wanted to know what people asked about. Often it was as simple as “people, products, companies”. I was up for helping with the products and companies, but not the people.

When we released the first Internet publishing system’s software, Wide Area Information Servers, into the public domain in April of 1992, I included an essay with every copy titled “The Ethics of Digital Librarianship.”  A kind of guide, a warning, to those that would start to accumulate information on users in the form of usage logs– unwitting traces of what others were thinking about. Usage logs give intimate insights into individuals, and the Internet addresses they come from can allow a user to know whose thoughts they are.

Some think I came up with the term “Big Data” in the early 1990’s, I could have been the one, it is hard to know. But I certainly promoted it. If I did coin it, then where it came from playing with Laurie Anderson’s song title “Big Science.”   In that song, released in 1982, Laurie warns of

“Every man, every man for himself
All in favor say aye
Big Science. Hallelujah.”

“Big data” came after databases and before data mining. Profiling, redlining, targeting. Data analytics. I knew that big data unleashed by libertarian free-for-all thought ( “what can be built must be built” type of thing), we could get mass manipulation by those that could afford it, those that wanted it: Intelligence agencies, corporations, and those wishing to influence elections.

There would be those that would cross the line to build profiles on a massive scale– Facebook, Cambridge Analytica–  but I was not going to join them. I was not going to use these powerful tools for this.

As we have built large datasets at the Internet Archive, there are some we can let anyone do anything with, but most allow aggregation of information about people, things that make us uncomfortable letting anyone do just do anything with it. Do we need an ethics board to sift proposals as they do in medical studies?  People are sold that there is no “personally identifiable information” in these large datasets, even after they are “anonymized.”  Maybe they want to believe it. But it is almost certainly not true.

I don’t know of a technical way to keep us from making databases of profiles, or restraining manipulation. We need to keep ourselves back from those edges. But those edges are justified by many with money, with calls for “security” and other righteous causes.

We have the tools of Big Data now, will we keep our humanity?  I am still looking for answers, and how we can help. Any ideas?

I am in China and AliPay is everywhere.  With your smartphone (which everyone has at least one of) you scan the QR code at convenience stores, in your taxi, at the subway, anywhere– to instantly pay for things.  Many people do not carry cash anymore. AliPay, a service of Alibaba the massive Amazon-like company, and wechat-pay, backed again by WeChat, is almost the only way people pay for small bills and large.

But it is more than just a convenient credit card system in a land that did not go to credit cards. AliPay seems to have very very low payment threshold making new businesses possible.

Every restaurant in Hang Zhou I have been in has a phone recharging station full of batteries you can rent for free for the first 30 minutes, and 14 cents/hour thereafter.  So a typical charge is 14 cents, and it is still profitable to make a business around this. Visa card fees are often a minimum of 30 to 50 cents, so charging 14 cents would not work.

Rental bicycles are everywhere here.  Unlike the states, they are not in big theft-proof

Rental Batteries with AliPay

docks, but rather are basic bikes that lay around everywhere.  Every street corner, in front of most restaurants, everywhere.   You scan the code on the bike which unlocks it, and off you go. People subscribe to one of the many companies that have bikes everywhere, and it costs $28/year, and most rides are free (unless you keep the bike longer than the normal hour or so, and which case it costs 14 cents per hour).

And the rental bike companies are private, not city subsidized.  This means there is  competition and as I said, they are everywhere. The payment system is only part of why this works (also the bikes are inexpensive and there is enough trust that these do not need to be locked down).

The payment system is making new things possible.

But…

there is a privacy trade-off to how this is done.  Alibaba knows everything you pay for, everywhere you go, and can infer everyone you socialize with. They are explicit about this and display your “social credit score”– how trustable you are based on datamining this.  This score shows up on your phone. This score is being used by lenders to figure out who to lend to, by landlords to figure out who to rent to.  And the government is looking to use an extension of this to figure out how good a citizen you are.  This is super creepy to Americans, and to some Chinese who will be frank.  Many Chinese do not see the problem with this, or at least will not say so to a visiting American.

Apple-Pay and Google-Pay have launched, and I guess that means Amazon-pay will follow.  Combined with Apple’s convenient fingerprint reading log-in system, and now facial recognition, your phone is your “PIN”, your personal identification number.  And your phone is bio-metrically fixed on your real world identity. Already Apple or Google, app companies, and your phone company know who you socialize with, much of what you say to them, and soon everything you buy, where, and with whom.

This can go very wrong.    But, there could be another way:

Introducing Crypto-pay! (boy, do we need a better name than that– “Open-Pay”?)

Imagine Ali-pay but as an open system, and a private system, and competitive system. Fortunately we have many of the pieces already to do this, but it has not been done.

Crypto-pay users would have apps on their phone and merchants would have QR codes.  The QR code encodes a web URL for that store, this could be as simple as a bitcoin address, but it can also be a more complicated little web app to handle the rental bikes or rental batteries.   The App on the phone would ask the user for how much money to pay and they punch it in, hit confirm, and the store’s cashregister (which is a phone) instantly says “paid.” The store does not need to know who paid them.

Lets take another easy app for Crypto-pay– paying in a restaurant– you could get a paper bill which is a QR code on it that encodes the store name and the amount.  Scan, confirm, presto– paid.   Or, as most restaurants here in China do, you scan the QR code on your table, it brings up a menu on your phone, you pick what you want, they bring it, and then at the end you say “confirm” and it is paid for.  No paper menus, no waitstaff taking the order, no paper receipt.  Living in the future.   But with Crypto-pay it can be as private as paying with cash.

The particular Crypto-pay app on your phone could be one of many competing applications that come from multiple vendors.   I use BreadWallet on my phone (disclosure: I have a tiny investment in that company) which makes spending bitcoin really really easy.  But as we have seen Bitcoin’s transaction fees and delays make it inappropriate for this application.

So bitcoin may not be it, but imagining a system that has sub-penny transaction costs for tiny transactions (and maybe higher fees for larger transactions) and a few second transaction times is not impossible technically.

So an open and competitive system– with lots of players providing merchant services, lots of players creating user apps, even with different block chain technologies is possible.

But will it happen?  It starts with imagining a future with lots of winners, and then getting a level playing field to do it on.  It will require Apple and Google to not lock out apps of their stores that rival apple-pay and google-pay (or maybe we could do it completely with web based phone-apps, but lets start easy).  It will require regulators to not be heavy handed as they have been with bitcoin.   It will require investment in startups, it will take neutral organizations to help with standards.

Hard to accomplish, yes.  But just imagine– a system that has the combined benefits of cash and debit cards.  And not creepy.

I want it, anyone else?

Do you think more often of your date of birth or date of death?
Do you think of the start or the end?
Do you think of the project or the deadline?
Do you think of your journey or your legacy?
How much substance is there in your soul verses value in your impact on others?

I imagine these are the differences between the Collector and the Digital Librarian. The collector seems to want to experience, to learn and also to share. The collector discovers, understands and reveals to others.

The digital librarian has no long term memory other than what is captured in the library. Where the collector lives for life’s expressive expanses, a digital librarian is designing for not-being-here-anymore. When a doctor says, as the end nears, “you should get your affairs in order…” it is gloomy, foreboding, and tragic to the collector. The Digital Librarian says, “That is all I ever do.”

It is not that the Digital Librarian does not want to live forever—in fact that may be the driving emotion—it is just the method to live forever is not corporal, it is informational. We fight mortality through trying to share, and share permanently.

In creating the Great 78 Project, I have wanted to keep the notes of what records were in whose collections. I believe this may be the most important thing—more important than the recordings—what records were together?

If we want to understand a time or a life, it is made up of those groupings. As a Digital Librarian I want to illuminate for others those lives, those perspectives — I want to not lose those past lives through reorganization. But I don’t think I will be the one to learn from these lives, those choices, those perspectives. It will be other people, or even machines that will learn from these assemblies.

Bill Dunn said in the mid 1980s, “The metadata is more important than the data itself.” Astonishing—how did he know? He came up with the term “metadata” with Mitch Kapor around that time.

Collections are metadata and metadata of great value if these reflect a life’s choices. Those life’s choices may be the most valuable part of the Great 78 collection.

As a Digital Librarian, I feel I should, I must preserve this, share this.

But it is not for me, it passes through me. I am a Digital Librarian, not a collector.

I hope I do a good job during my brief stay on this earth.

Announced a month ago with 15,000 digitized “sides” of 78 RPM records, I am trying to understand it and to understand myself through it. B George of the Archive of Contemporary music is also the Internet Archive’s music curator but he does not know as much about 78s because he specializes in LPs — mostly from the 1980s on. So we have donated collections and he is going through them and picking large sections to be digitized. The slant on our contribution of recordings to this project really comes from the collections that have come in.

Barrie Thorpe collected and donated to the Batavia Public Library in Illinois decades go, we have that collection of maybe 40,000 records. “Tercat” is from Rhode Island. A collection of polka, others…

I am interested in keeping the collections together because I think that is where much of the value is, but I am not sure. Maybe these are not original collections but remixes by the next generation.

I want to know what the early 20th century sounded like. Midwest, different countries, different social classes, different immigrant communities and their loves and fears. I am not looking for the great record, the unfound gem, I don’t think. At this point I am looking for ‘discovery’, for inspiration, for leads I can follow up on, maybe I am looking for a rationale for spending time and money digitizing this stuff.

As a librarian, I see justification through others — is it useful? Did someone say “thank you” or “I love what you do”? But what is this collection or collection of collections? “Selection” is easily muted if you go for “comprehensive” — an easy out.

But what is this? How about a “modern discography”? A reference collection that is more than a listing in a book of the releases of a particular label — more than one that has pictures of the labels. This one has the digitized sound recordings.

This “reference collection” moniker works because we do not really care if we have the physical disc (though that is nice because we can go back to it to re-record or study it, or at least know it is safe). What we want is it to be findable with a click.

For instance, I am reading a book called, “Do Not Sell at Any Price: The Wild, Obsessive Hunt for the World’s Rarest 78rpm Records” by Amanda Petrusich. In it, she talks of many records, performers, labels, collectors, re-issues. I want to click on the paper page and hear it. I want citations to turn into blue links so I can more deeply understand what she is talking about. Is this weird? I don’t think so. Footnotes were always supposed to be hyperlinks, we just did not have the tech with paper. But now we have ebooks — let’s bring the citations to life. To do this we need a reference collection, ta-da.

But there is something more for me personally — I get a thrill as the materials come online. Each disk is a revelation. It feels like when I was in college and I would buy a used record for $3 and bring it home and play it — “retail therapy.” Ownership and discovery and possession. We just made a Twitter feed out of the newly available digitized 78s… we will see if anyone else likes it.

I also like playing a list of 78s based on a search — all hillbilly, all yodeling, all by a performer. It is less curated than a compilation LP or CD. It is serendipitous. It is kind of random. It has discovery feelings. Maybe if it is a new way to find things more like how Youtube goes from one video to the next. A new thing?

So what is this? A reference collection? A collector’s dream? A discovery radio station? The soundtrack of the early 20th century? All Good. All Fun.

All told, I would say the Great 78 Project is building a reference collection.

I am upgrading the security of my communications while still being easy to use. I thought I would share what I currently use in case it is helpful to copy and I would appreciate comments.

I want end-to-end encryption so nobody can intercept what I am saying (unless they have infected my phone or computer, but that is another issue), and bonus points for making it so that it is unknown who I am communicating with and when (private metadata and traffic). Skype, phonecalls, sms/texts, slack and email are now known to not be private (at least by default) thanks to Edward Snowden. This is too bad since I still use these.  (Slack is not end-to-end encrypted even for direct messages, which it could and should.) So far I have only partially achieved the first step: end-to-end encryption. I am migrating to:

• Signal for point-to-point instant messaging replacing sms and skype. Free software, free of cost, and open source, works on smart phones, and with a chrome-based desktop Signal app on my Mac (which is what I mostly use).   It uses phone numbers as identifiers, which is kind of a pain.  EFF friend called this “best of breed” for security. Small development staff. I have donated.
• appear.in for 1-on-1 and small group video chat that is end-to-end encrypted replacing Skype. This does not require a download or an account. Go to the homepage, type a bunch of characters to make a meeting room, then send the resulting url to someone and they can use that throw-away meeting room.  Super easy. Uses webrtc (now standard in browsers), and https with it, they say it is end-to-end encrypted.   They have a iphone app as well, but don’t know about security. This does not seemed designed for super high security, but seems to be pretty good.
• zoom.us for larger group video chats replacing Webex. Free of cost for most of my uses, easy to use (requires download, but is super easy) . It says it is end-to-end encrypted with a little lock icon when in use and encrypted.
• Facetime occasionally on my iphone replacing cellphone calls to friends with an iphone.  Apple says that it is end-to-end encrypted.
• Thunderbird + Enigmail to sign all email, receive encrypted email, and sometimes sending encypted Email, with an organizational email server (archive.org not gmail).  Enigmail is moderately hard to set up, I had help in a meetup.  Cost free, and I believe free and open source software. I am donating.
• encrypted notes file (the mac Notes app) on my mac for high priority secure notes. It syncs the encrypted file with my iphone via icloud.
• Breadwallet, bitcoin wallet on my iphone, for small amounts of bitcoin for casual purchases.  Super easy and a full wallet (does not hang off a server). Love this wallet. Cost free.  I invested a tiny amount of money in the company– great guys.
• Torbrowser for private web browsing beyond Firefox’s Private browsing feature.  Free and open source software, cost free. I have donated.

Any comments or ideas are welcome. I realize have traded off security for ease of use.  I hope stronger tools get easier and I suggest we all invest in tools based on donations and development help.  I wish I knew my mac and iphone were not compromised.  Not sure how to do that.

I have tried ricochet as an instant messaging client that secures who I am talking to via Tor, easy to use, but few I know use it, so I don’t use it often.  I have tried encrypting my email using pgp via enigmail but have run into trouble with others being able to read it, so I do not encrypt email by default. As an aside, encryption is related in a funny way to content-addressible systems, which is a different subject, but this is magic and the future.

—– From a commenter: —–

Web search:  DuckDuckGo or StartPage.com.     (thank you, Reinout)

I thought I was so clever– instead of buying all the parts for our books scanners from industrial vendors then assembling&testing&reshipping from Richmond California, we could send many parts to where they were needed and assemble in our scanning centers.  We wanted to buy all the parts through amazon.com, all on one amazon business account, which would make it easy to track.  We thought we could try this for a scanning center in Hong Kong.

Well, here are the problems we encountered:
1) the prices for cameras and lenses were 30% more than if we bought from selected stores we could bargain with,
2) some amazon.com vendors would not ship internationally,
3) most electronics vendors had limits on how many you could buy, like 2 or 4 or 5. What a pain. Even “amazon basics” were limited,
4) a couple of the vendors only had a couple in stock,
5) shipping to Hong Kong means deliveries take at least 9-10 days because they come from the US,
6) shipping to Hong Kong is expensive because there is no Prime or free shipping or local delivery.

All in all it took hours after we had figured out which ones we wanted.  (thank you Salem!)

So we ended up making many sub-accounts for our business account to get around some limits, have to wait for restocking from others, and shifting around vendors. On the positive side, we could use our business credit card to get a kickback in miles, and we also used smile.amazon.com to try to get a kickback to Internet Archive.   We should have used the affiliate code, which would might have given us a bit more of a kickback.

Oh, well.  Amazon.com is meant for individual consumers– and for that it works well.   For this application, not so much.

-brewster

It is conjuring from the ether, it is wishing things into your hands, it is just saying its name and it will appear. It is pure magic. And it may become a very important part of our future– in the Decentralized Web and beyond.

“The great thing of the web is that now knowledge has an address,” said Peter Lyman, the University Librarian of UC Berkeley 20 years ago of the URL, which means that people can build easily on other’s knowledge. Now we can add something: “Content addressibility means that knowledge has a name.” A name can be better than an address because addresses sometimes become obsolete. (Peter Lyman was one of the first board members of the Internet Archive and, I believe, came up with the term “Born-Digital” to describe materials had never been printed– a new thing in 1996).

What am I talking about? This might sound to simple to stand up these big claims, but bear with me. This is one of the big things I have learned from the Decentralized Web work.

Content Addressing starts by processing a digital file into a “hash” which is roughly 64Bytes, or 64 character long string of numbers (using sha256). This hash is has amazing properties– given a hash you can confirm that a digital file matches it, further given a hash it is very very difficult to create the digital file. And, here is the kicker, given a hash it is almost impossible to create a second digital file that matches it, but was not exactly the same as the original.

Therefore, a “hash” is a name for a file in the sense that if you have a hash you are looking for, and someone hands you a file, you can confirm it, and you do not have to trust who gave it to you– they can not fake or counterfeit the file. The file either has the same hash or not.

That the hash is very short, like 64 characters, and can name a multigigabyte file means that moving around these hashes is very efficient. The Internet Archive has 17 petabytes of web data, but all of the hashes are only 22terabytes. Therefore to give every web object a unique name, it only takes 0.1% of the size.

So, with a hash, one can address content directly, ask for it by name, and confirm if what you are given matches. The most common application of this is in the BitTorrent system, but it is widely used. In bittorrent, one can start with a “magnet link”, which is a hash, and asked the “decentralized hash table” DHT, and it will help you retrieve the file that matches that hash, in this case a “torrent” file. A torrent file, in turn, contains a list of hashes of pieces of files that can then be retrieved through the bittorrent protocol, and after this magic is done, then you will have a set of files on our hard drive that came from 10s or thousands of others all over the net.

Therefore, if there are others on the peer-to-peer network that are serving files, and you have a hash, then you can ask the network to give you that matching file or piece of a file, and there can be no counterfeiting.

Magic.

Why this can be important that materials can be served from many places, served from libraries and archives, and keep permanently available long after the original server is gone. I think of it as a way to have the same book be in many libraries, and even if the publisher goes away, and several of the libraries merge, you still have a chance to get the book. This is different from a website, where if the website goes away, you are either out of luck or if something like the Wayback Machine has a copy, you are saved, but you have to trust us. So in a way, this hash idea is bringing back some nice features back from the printed era. A much more reliable system of digital publishing is possible in this way.

(This is how IPFS, Zeronet, DAT, and just about every decentralized system works, but I think still it is under-appreciated magic. Next miracle I will describe is how cryptographically signed files can bring us the next step: updatable digital files that are served from everywhere and nowhere.)