Content Addressing is Magic

It is conjuring from the ether, it is wishing things into your hands, it is just saying its name and it will appear. It is pure magic. And it may become a very important part of our future– in the Decentralized Web and beyond.

“The great thing of the web is that now knowledge has an address,” said Peter Lyman, the University Librarian of UC Berkeley 20 years ago of the URL, which means that people can build easily on other’s knowledge. Now we can add something: “Content addressibility means that knowledge has a name.” A name can be better than an address because addresses sometimes become obsolete. (Peter Lyman was one of the first board members of the Internet Archive and, I believe, came up with the term “Born-Digital” to describe materials had never been printed– a new thing in 1996).

What am I talking about? This might sound to simple to stand up these big claims, but bear with me. This is one of the big things I have learned from the Decentralized Web work.

Content Addressing starts by processing a digital file into a “hash” which is roughly 64Bytes, or 64 character long string of numbers (using sha256). This hash is has amazing properties– given a hash you can confirm that a digital file matches it, further given a hash it is very very difficult to create the digital file. And, here is the kicker, given a hash it is almost impossible to create a second digital file that matches it, but was not exactly the same as the original.

Therefore, a “hash” is a name for a file in the sense that if you have a hash you are looking for, and someone hands you a file, you can confirm it, and you do not have to trust who gave it to you– they can not fake or counterfeit the file. The file either has the same hash or not.

That the hash is very short, like 64 characters, and can name a multigigabyte file means that moving around these hashes is very efficient. The Internet Archive has 17 petabytes of web data, but all of the hashes are only 22terabytes. Therefore to give every web object a unique name, it only takes 0.1% of the size.

So, with a hash, one can address content directly, ask for it by name, and confirm if what you are given matches. The most common application of this is in the BitTorrent system, but it is widely used. In bittorrent, one can start with a “magnet link”, which is a hash, and asked the “decentralized hash table” DHT, and it will help you retrieve the file that matches that hash, in this case a “torrent” file. A torrent file, in turn, contains a list of hashes of pieces of files that can then be retrieved through the bittorrent protocol, and after this magic is done, then you will have a set of files on our hard drive that came from 10s or thousands of others all over the net.

Therefore, if there are others on the peer-to-peer network that are serving files, and you have a hash, then you can ask the network to give you that matching file or piece of a file, and there can be no counterfeiting.


Why this can be important that materials can be served from many places, served from libraries and archives, and keep permanently available long after the original server is gone. I think of it as a way to have the same book be in many libraries, and even if the publisher goes away, and several of the libraries merge, you still have a chance to get the book. This is different from a website, where if the website goes away, you are either out of luck or if something like the Wayback Machine has a copy, you are saved, but you have to trust us. So in a way, this hash idea is bringing back some nice features back from the printed era. A much more reliable system of digital publishing is possible in this way.

(This is how IPFS, Zeronet, DAT, and just about every decentralized system works, but I think still it is under-appreciated magic. Next miracle I will describe is how cryptographically signed files can bring us the next step: updatable digital files that are served from everywhere and nowhere.)

Posted in Uncategorized | 4 Comments

Successful German Model for Permanently Tenant-Friendly Housing

Started in 1989 and now made up of 107 apartment buildings in Germany, a network of housing projects called Mietshäuser Syndikat, have locked themselves into a structure that allows almost complete autonomy for each housing project, with a couple of exceptions: the building can not be sold or condo-ized without the permission of a central organization.

This form of “some rights reserved” housing provides more security for the tenants from market-based rent fluctuations and evictions.    It also provides a structure and advice network for those wanting to build tenant housing associations that will own and control their buildings over a long term.

In the United States, I have seen people drawn to the urban land trust structure for some of these advantages.   This German model has the advantage of a fixed, and low, fee structure (2.5 cents per square foot per month) and a transparent and very limited set of rights reserved for the central entity.   In this way, the central entity is deliberately limited to a small function to keep autonomy in the housing projects themselves.

I have been interested in restricting new debt being put a member building, but that is a step further than this model does.   In the case of Foundation Housing, the benefits are allocated to non-profit use, either by accumulating assets or lower-than-market usage fees.    So this German system is a step in these directions, and given their rapid uptake, a successful step.

I hope this ideas spreads!

Thank you to Ben Woosley for pointing this out to me.


The joint venture



Posted in Housing, Uncategorized | 2 Comments

Custom School Short Video Description and Short Video by Kid having done it



And, for more good news, Logan got into Cornell a year earlier than his years would suggest, and he is doing well there.   So custom schooling has been working well.

Posted in Uncategorized | Comments Off on Custom School Short Video Description and Short Video by Kid having done it

Paper on systems like Foundation Housing

Housing for non-profit workers does not seem to be represented in this paper, but union housing, or permanently affordable housing is.   And hence, I suggest worth reading.

The paper from 1996 usefully describes experiences with Community Land Trusts in urban areas and Mutual Homes Association (which I have loved since discovering Richmond’s successful Atchison Village). 

This paper brings context to those working on a “third sector” (not public housing, or for-profit housing).

I liked this paper because it gave real examples of success and failures and tried to draw conclusions and trends.  Unfortunately, most of the groups they studied were under 10 years old.   I hope they update this study, or if anyone knows of an update, I would like to see it.    Unvarnished experience is valuable.




Posted in Housing | 1 Comment

Divertissement for Warming Orchestra #D4

Having just gone to the symphony tonight, I would like to propose a new piece of music, called Divertissement for Warming Orchestra.

Here is the “score”: It is a form of call-and-response.  When the orchestra is warming up, any player plays a short segment of a familiar tune.  Then someone else in the orchestra responds, maybe with the next part, maybe something as a riff.  For instance a small part of twinkle twinkle, or Alice’s Restaurant, or Gilligan’s Island theme, or Star Wars, or …

The little back and forth can go on for no longer than 30 seconds, and not be obvious.  It has to just tickle the ear, fire a neuron, and then be gone.  If the conductor looks like she might come on, then it is to stop.  The musicians are not to show any indication they are doing this, so this piece is to be heard but not seen.

Anonymous, crowd sourced, guerrilla music might just make being in the audience before the performance really fun.

If you believe this piece has been played, maybe tweet about it with hashtag #D4   (as in “Divertissement for”).  If your orchestra is banned from playing this piece, then also tweet #D4.

We will know this piece is successful if it is banned in symphonies in several cities beginning with C.

Posted in Uncategorized | Tagged | Comments Off on Divertissement for Warming Orchestra #D4

Locking the Web Open: A Call for a Decentralized Web

(Short form article, Short lecture, Long lecture, demo of a fraction of the idea of a distributed website (or paste this link in maelstrom))

Over the last 25 years, millions of people have poured creativity and knowledge into the World Wide Web. New features have been added and dramatic flaws have emerged based on the original simple design. I would like to suggest we could now build a new Web on top of the existing Web that secures what we want most out of an expressive communication tool without giving up its inclusiveness. I believe we can do something quite counter-intuitive: We can lock the Web open.

One of my heroes, Larry Lessig, famously said “Code is Law.” The way we code the web will determine the way we live online. So we need to bake our values into our code. Freedom of expression needs to be baked into our code. Privacy should be baked into our code. Universal access to all knowledge. But right now, those values are not embedded in the Web.
It turns out that the World Wide Web is quite fragile. But it is huge. At the Internet Archive we collect one billion pages a week. We now know that Web pages only last about 100 days on average before they change or disappear. They blink on and off in their servers.

02-blockedworldAnd the Web is massively accessible– unless you live in China. The Chinese government has blocked the Internet Archive, the New York Times, and other sites from its citizens. And other countries block their citizens’ access as well every once in a while. So the Web is not reliably accessible.

03-spyingAnd the Web isn’t private. People, corporations, countries can spy on what you are reading. And they do. We now know, thanks to Edward Snowden, that Wikileaks readers were selected for targeting by the National Security Agency and the UK’s equivalent just because those organizations could identify those Web browsers that visited the site and identify the people likely to be using those browsers. In the library world, we know how important it is to protect reader privacy. Rounding people up for the things that they’ve read has a long and dreadful history. So we need a Web that is better than it is now in order to protect reader privacy.

04-spendBut the Web is fun. The Web is so easy to use and inviting that millions of people are putting interesting things online; in many ways pouring a digital representation of their lives into the Web. New features are being invented and added into the technology because one does not need permission to create in this system. All in all, the openness of the Web has led to the participation of many.

05-distributedWe got one of the three things right. But we need a Web that is reliable, a Web that is private, while keeping the Web fun. I believe it is time to take that next step: I believe we can now build a Web reliable, private and fun all at the same time. To get these features, we need to build a “Distributed Web.”

Imagine “Distributed Web” sites that are as easy to setup and use as WordPress blogs, Wikimedia sites, or even Facebook pages, but have these properties. But how? First, a bit about what is meant by a “distributed system.”

06-escherContrast the current Web to the Internet—the network of pipes on top of which the World Wide Web sits. The Internet was designed so that if any one piece goes out, it will still function. If some of the routers that sort and transmit packets are knocked out, then the system is designed to automatically reroute the packets through the working parts of the system. While it is possible to knock out so much that you create a chokepoint in the Internet fabric, for most circumstances it is designed to survive hardware faults and slowdowns. Therefore, the Internet can be described as a “distributed system” because it routes around problems and automatically rebalances loads.

The Web is not distributed in this way. While different websites are located all over the world, in most cases, any particular website has only one physical location. Therefore, if the hardware in that particular location is down then no one can see that website. In this way, the Web is centralized: if someone controls the hardware of a website or the communication line to a website, then they control all the uses of that website.

In this way, the Internet is a truly distributed system, while the Web is not.

Distributed systems are typically more difficult to design than centralized ones. At a recent talk by Vint Cerf, sponsored by the California Academy of Sciences, Cerf said that he spent much of 1974 in an office with two other engineers working on the protocols to support a distributed Internet system, to make it such that there are no central points of control.

07-networkHere’s another way of thinking about distributed systems: take the Amazon Cloud. The Amazon Cloud is made up of computers in datacenters all over the world. The data stored in this cloud can be copied from computer to computer in these different places, avoiding machines that are not working, as well as getting the data closer to users and replicating it as it is increasingly used. This has turned out to be a great idea. What if we could make the next generation Web work like that, but across the entire Internet, like an enormous Amazon Cloud?

In part, it would be based on peer-to-peer technology—a system that isn’t dependent on a central host or the policies of one particular country. In a peer-to-peer model, those who are using the distributed Web are also providing some of the bandwidth and storage to run it.

Instead of one Web server per website we would have many. The more people or organizations that are involved in the distributed Web, the more redundant, safe, and fast it will become.

08-privacyAnd it also needs to be private—so no one knows what you are reading. The bits will be distributed—across the net—so no one can track the readers of a site from a single point or connection. Absolute privacy may be difficult to achieve, but we can make the next Web much more secure.

The next generation Web also needs a distributed authentication system without centralized usernames and passwords. That’s where encryption comes in to provide a robust but private identity system.

We’d also want to bring in some other features if we’re going to redo this Web.

09-memoryThis time the Web should have a memory. We would like to build in a form of versioning, so the Web is archived through time. The Web would no longer exist in a land of the perpetual present.

On library shelves, we have past editions of books, but on the Web, you don’t have past editions of websites. Everyday is a new day, unless you know to use the Internet Archive’s Wayback Machine, which may have copies of previous versions. Where the Wayback Machine was created after-the-fact to solve this problem of the current Web, in this next iteration we can build versions into the basic fabric of the Distributed Web to provide a history and reliability to our growing digital heritage.

We could also add a feature that has long been missing from the Web: easy mechanisms for readers to pay writers. With the support of easy payments, the Distributed Web could evolve richer business models than the current advertising and large-scale ecommerce systems.

Adding redundancy based on distributed copies, storing versions, and a payment system could reinforce the reliability and longevity of a new Web infrastructure.

Plus it needs to be fun—malleable enough to spur the imaginations of millions of inventors. This new Web could be an inviting system that welcomes people to share their stories and ideas, as well as be a technology platform that one can add to and change without having to ask permission– allowing technological change just for the fun of it.

10-componentsHow can we build this new Distributed Web? There have been many advances since the start of the Web in 1992 that will be helpful.

We have computers that are a thousand times faster. We have JavaScript that allows us to run sophisticated code in the browser. So now, many more people can help to build it.

Public key encryption systems were illegal to distribute in the early 90’s, but are now legal, so we can use them for authentication and privacy. With strong cryptography, communications can be made safe in transit and can be signed so that forgery is much more difficult.

We have Block Chain technology that enables the Bitcoin community to have a global database with no central point of control.

And we have virtual currencies such as Bitcoin, which could make micropayments work in a distributed environment. Many other projects have pushed the limits of distributed systems giving us building blocks for a Distributed Web.

11-spaceodysseyI’ve seen each of the necessary pieces work independently, but never pulled together into a new Web.

I suggest we need a bold goal, one that is understandable and achievable. Something that we might be able to rally around, and have multiple groups contribute to, in order to build an easy to use Distributed Web.

What about WordPress, but distributed? WordPress is a very popular toolkit that millions have used to build websites. My blog,, for instance, is built on the open source WordPress software installed on a server at the Internet Archive. Free to use, and free to host, this toolkit enables anyone to select from a set of template designs and modify it to give it a unique look. Then the original creator can appoint users to play roles such as administrator, editor, or commenter. Those with these different privileges can, in turn, grant privileges to others as appropriate. And then the writers can post articles or images to its pages or change the look and feel of the site.

A WordPress website, traditionally, would then be hosted on a computer of the creator’s choice, either on, or on other sites offering hosting, or even on their own computer because the underlying software is available open source as well. This is where WordPress is not “distributed,” in the sense we are talking about earlier. If the organization hosting the site does not like the material, or it is blocked in another country, or goes out of business, then the website will not be available. Even major companies, such as Apple, Google, and Yahoo, have taken down whole systems hosting millions of user’s websites, often with little notice.

We would like to allow anyone to build a WordPress website–that has themes and different people with different roles, fun to go to and add to, free to create—which is also distributed in a way that is private and reliable.

We would want it to work in all browsers with no add-on’s or modifications. We would want to refer to a distributed website with a simple name like and it needs to be fast.

We would need users to be able to log in without having to have many websites know their usernames and passwords, or have a central site, like Facebook or Google, control their online credentials. In other words, we need a distributed identity system.

Additionally, we would like to have payments work in the Distributed Web. We would like to enable anyone to pay anyone else, akin to leaving a tip, or paying a suggested amount for reading an article or watching a movie. Thus people could get paid for publishing on this Distributed Web.

In addition, we would want to have saved versions of websites, and dependable archives to make this distributed websites reliable.

How can we build this system?
Way to Build the Distributed Web: an Example

Please bear with me as I to try to argue that this is possible using an amalgam of existing or near-existing technologies.

A piece of this system could be a peer-to-peer system such as Bittorrent. Storing and retrieving files in a distributed way has been commonplace for years with Bittorrent. While downloading custom software is not ideal, it shows this function can be done and done for millions of people. Bittorrent is kind of magic, where typing a long number that is a unique identifier for a file or set of files will cause it to appear on your machine. Pieces of the desired file will come from other computers that had previously retrieved those files and therefore store them on their computers. In this way, the readers of files become the servers of those files. There are millions of users of Bittorrent sharing everything from commercial movies, to free software, to library materials. The Internet Archive, for instance, offers petabytes of files to the public using the Bittorrent protocol so that users have the option to retrieve files from the Internet Archive or from other users who might be closer.

Using Bittorrent as part of the Distributed Web to share the files is working in prototype form now. Bittorrent Incorporated’s peer-to-peer powered Web browser Maelstrom is now in alpha release. With this browser, a files of files can be distributed using Bittorrent. Using this early version, I demonstrated at a conference last month a static version of my blog, being served by people around the Internet.

13-bkblogNotice in this image, that the Web address starts with bittorrent:// and then a long number. This is how the website was retrieved from the Bittorrent network.

Another system, IPFS, designed and implemented Juan Benet, is an open source and has some of the same peer-to-peer characteristics but has some added enhancements. Juan took my blog and in a few minutes put it into his system, showing that system is also working. One of the major features this system offers over Bittorrent is that updates to the blog can be discovered and distributed naturally through the system. Therefore, as people would add comments and posts to a blog, these can be retrieved without having to get a new identifier.

13.2-bkblogNotice in this case, the Web address refers to the localhost, meaning that it is retrieving the pages using a computer program running on my laptop, which is operating the peer-to-peer functionality.

Other distributed systems are in different stages of development, which will certainly be useful. Many of these systems are listed at the end of this paper.

Therefore the idea of storing and retrieving files that are part of a distributed website is now a reality in prototype form. But there are still some pieces missing.

14-javascriptBuilding Seamlessly on Top of the Existing Web

One feature that would greatly ease adoption would be to have distributed websites work seamlessly in reader’s browsers without any add-ons, plug-ins, or downloads–just click and see.

This is important because software on phones, tablets, and laptops are becoming more difficult to install without the permission of a company, such as Apple. Fortunately, it is easy to distribute JavaScript as part of Web pages, and this will likely be supported for a long time because it is important to sites such as Google Docs and Google Maps.

16-wordpressJavaScript running in users’ browsers as a kind of application platform is now possible and usable. I was surprised to find that JavaScript is now powerful enough to emulate older computers in the browser. For instance, you can now run a IBM PC emulator running MSDos 3.1 running a computer game just by clicking on a weblink to go to a webpage. The game Oregon Trail, or Prince of Persia, or old arcade games are now available on the Internet Archive and have been played by millions of people. The way this works is that others have made emulators of the underlying machines in the programming language C, and then that code cross-compiles it into JavaScript. So, when a user goes to and clicks to run it– it downloads a JavaScript program that boots an emulator of an old IBM PC or an Apple 2 in the browser. Then it reads a floppy, in this case a virtual floppy, and then runs that program in the emulator so that you are basically experiencing that old computer interface. It was a strange mind twist for me to download and run a whole machine emulator in a browser. Since JavaScript is capable enough to do that, then we can build the mechanism we need for the Distributed Web in JavaScript.

To run a distributed system in the browser, we need one more feature. The code running in the browser must be able to connect to other browsers that are running the same system. Basically we need to make it so that a browser can contact another browser instead of going to a server. This is now achievable based on a new standard, web-RTC, that was created to allow video conferencing and multiplayer games.

With the underlying speed of modern machines, the maturity of a coding system like JavaScript, and the peer-to-peer features supported in browsers, we seem to have all the pieces we need to support a Distributed Web on top of the current Web without any downloads, plug-ins, or add-ons.

There is an additional advantage to building the Distributed Web in JavaScript: it can be changed and added to by many people independently. In fact, different websites might use different Distributed Web systems all interoperably on the Internet at one time. It does not require coordination or relationships with the browser manufacturers to make changes to how the Distributed Web works. Features can be added, subtracted, and experimented with in parallel, without permission. The Distributed Web could evolve much faster than current Web technologies and yet still be interoperable.

Distributed Websites that have Search Engines and Databases

Since WordPress sites have search and database functions for selecting posts from particular months and with particular tags, to be fully functional, we need our distributed websites to have these features as well. In the current Web, programs running on a server support these features, so that when the user types a few words into a search box, it is sent to the server, and then a program runs on the server to create the page that is then transmitted back to the browser. In the Distributed Web there are no servers, there are only static files that are retrieved from a peer-to-peer network. Luckily some of the files of the website can themselves be computer code in the form of JavaScript. All of the computation then happens in the browser based on those files.

Fortunately this is possible because a search engine and the index can be packaged as files that can be downloaded to a browser and run in the browser. This feature has been achieved in the demonstrations based on Bittorrent as well as IPFS mentioned before; the programmer, Marcel van der Peijl, used an open source tool js-search to take the pages of my blog site to create an index plus search engine in JavaScript. For my site, the resulting JavaScript page was one megabyte, which is large, but not too large for broadband computer users. To make this more usable, he only loads this code after the page the user requested has been displayed, so in most cases the user would not notice the delay.

This approach will work for most blogs, but maybe the largest ones will need more sophistication.

Therefore, we can have distributed websites that include dynamic elements such as search engines and databases.

Adding New Posts and Other Changes to a Distributed Website

A key feature of a WordPress site is adding comments or posts. This is trickier in a distributed setting than in centralized systems because updates have to be made in many places. In the WordPress application we do not need the website to be up-to-the-second for every reader, but we need to propagate changes quickly.

Bittorrent has a facility called “mutable torrents” which allows updates, but currently this requires a centralized facility to keep track of the newest version. This has the disadvantage of making the user contact a central server to find the most up-to-date version. This central server could be watched or controlled by a third party.

Another peer-to-peer file sharing system, IPFS, on the other hand, has a truly distributed facility for supporting updates and versions. IPFS is a very clever system that has some of the features of a Unix file system, but one that supports versions. How this works is some of the genius of this system. Since we have seen that a distributed WordPress site can be made out of files, which may contain images and text as well as code that can then be retrieved and played in the browser, a distributed file system could hold and transmit required files.

So there are solutions, even in a distributed way, to have millions of updates and not have to resort to central control or central notification that could impact our goal of protecting reader privacy.

17-waybackmachineThe Wayback Machine of the Distributed Web

The Wayback Machine is a free service of the Internet Archive that allows people to see past versions of websites. We do this by periodically visiting a website and downloading and storing the webpages being offered.

A Wayback Machine for the Distributed Web could store versions as they are created because it is easy to recognize what has changed and store it. This way, the Wayback Machine would have some significant advantages over the current one—it could always be up to date and it could help serve the current website and past versions to users without their even knowing it. This way the user would not need to go to the website to access the Wayback Machine. It would just start serving versions of the website on request, including the current version. If it did not have those files, then it could find them from other servers to add to the archive. Therefore, the Wayback Machine would be a form of host for the current version of the website, since it would participate in offering files to the readers. The Wayback Machine would therefore make the Distributed Web more reliable.

If someone referred to a past version of a website, and if the Wayback Machine had those files, it would serve those as well. In this way, the Wayback Machine would become more tightly integrated into the Distributed Web.

Many Wayback Machines could be run by many different organizations in a smooth way. As more groups participate, the more reliable and robust this system would become.

There is another significant advantage to the Wayback Machine application in the Distributed Web: it would archive and serve fully functional websites, not just snapshots of what it looked like through time. All of the functionality would be served, so its search and database functions would be supportable forever and in past versions. This way, the distributed websites would live on in time and space even if there were a disruption in hosting or authorship.

In this way, a library, such as the Internet Archive, could preserve and provide access to websites that are no longer maintained, or where the authors have moved on to other projects. This is similar to what libraries have done with professor’s research papers—offering enduring access to past works so that people can learn from them.

Therefore the Distributed Web would have a major advantage because it could be easily archived and served in a distributed and enduring way.

Fast performance

By having institutions such as the Internet Archive offering access to distributed websites, the users will get a more reliable service, but it could also help provide better performance. Since there are other organizations that are also motivated to provide fast and reliable access for their users, others could help replicate the data and make the Distributed Web more robust. Internet Service Providers (ISP’s), for example, want their users to have a good Web experience and would be likely to serve as a close and fast host for their users. This would also help save those companies on bandwidth bills because more of their traffic would be local. In this way, there can be cultural institutions as well as commercial organizations that have incentive to replicate parts of the Distributed Web, thus increasing reliability and performance for users.

Surveillance and Censorship

Since the Distributed Web would have users and repositories all over the world, both hosted by institutions and by other readers of the Distributed Web, some of the techniques for surveillance and censorship would become more difficult. For instance, the so-called Great Firewall of China blocks access to some websites outside of China by watching all traffic on its borders and filtering based on which websites are being accessed. Since a distributed website does not have a single location it would be more difficult to monitor or block its use. Furthermore, if one copy gets behind a firewall of this kind, then it can be replicated inside, making censorship more difficult.

The encryption used in this traffic may make it difficult to even know which files are being requested in the first place. Therefore, some of the existing systems of surveillance and censorship will not be as easy to conduct in the Distributed Web.

18-blockchainEasy Names of Distributed Websites

We also want easy-to-remember names for distributed websites. When the Internet was first designed, there were IP addresses that were strings of numbers such as These were not easy to remember so a naming system was created called the Domain Name System (DNS), that allowed someone to remember names such as “” instead of an address. The Web, being built on the Internet, used these in its universal resource locators, such as

In the Distributed Web, we have a similar problem with long, hard-to-remember numbers. In the implementations described above for both Bittorrent and IPFS, a webpage is an unique, incomprehensible string such as: 88f775eea02293b407e4b22c69d387cb9bbf50b8 or /ipfs/QmavE42xtK1VovJFVTVkCR5Jdf761QWtxmvak9Zx718TVr. It would be much more convenient if we had a string such as https://brewstersblog.arc.

The domain name service could be used for this purpose and would probably be a good starting point because it would leverage a large investment in technology and investments by society in regulating who gets what names. The Distributed Web could also incorporate new naming systems that would exist alongside the DNS to support new approaches to naming and the technologies to support them.

One distributed naming system that currently exists is called Namecoin, and it is an open source system built on a Bitcoin-like Blockchain, which is in itself a distributed system. To understand Namecoin, lets start with some of the characteristics of the Blockchain technology.

The Blockchain is a form of distributed database that is used to store the ledger under Bitcoin and similar systems. It is very clever in how it maintains consistency even when none of the participants trust each other. People submit “transactions” by signing them with their private cryptographic keys, and offer a financial tip to those who compete to operate the Blockchain consistency system; they are the so-called “miners.” The Blockchain then is a way to register transactions that everyone can see and everyone agrees to. In the case of Namecoin, the Blockchain is used to register a claim for a name and the long number with which it will be associated.

In this way, people can register a name and address pair in the Blockchain and others can look it up in a distributed manner. Unfortunately looking up a name is a time-consuming process, but at least it is certain who registers a name first. Increasing performance can be another task.

Another system that could be used for this is the Distributed Hash Table, or DHT, which is central to the way Bittorrent works. This is another distributed system for looking up a name.

So if this is done correctly, we can have easy-to-remember names resolve to distributed websites quickly, securely, and privately.

Furthermore, there could be registrars that charge for new names, and in return offer services such as fast servers and permanent archives. This could be a new business model that helps support the system.

To have a distributed naming system work in current browsers, without modification, we need a way to resolve the name to an address in JavaScript without contacting the server. Fortunately, there is a mechanism to do this using an anchor tag such as

Therefore we can have a simple system for naming distributed websites without losing privacy or reliability.

Distributed Identity

To know who is allowed to update a blog, we need a system to register administrators and then to authenticate someone as being that person. That is achieved on current WordPress sites when a user creates an account with a username and password using a Web page. This is kept in a database on the server. If a similar system could be implemented with a distributed webpage that operates the database, we could make the system more secure and easier for people to use.

Another way current websites often work is one logs in using one’s Google, Facebook, or Twitter account information. This way a user does not have to give a password to many different sites, but it has the disadvantage that large corporations know a great deal about one’s behavior online.

A better system might be one that uses cryptography to allow users to create multiple account credentials and use these without necessarily tying them back to their persons. That way people would have control over who knows what about them, and if they wanted to walk away from an account, that would work as well.

This could use what is called public key encryption, which uses special math functions to create pairs of public and private keys. The private key is used to sign documents in such a way that anyone using the public key, which is publicly known, can verify that it was correctly signed. No one else can forge a document. Thus, if posts were signed on a Distributed Web, then the readers can verify that it is the particular user that has the authority to perform that action and the website never needs to know a user’s password or private keys.

19-bitcoinMaking Money by Publishing on the Distributed Web

Public-private key pairs are central to how Bitcoin works, and this fact can be useful. In Bitcoin, a public key is used as the account name such as 1KAHLE1taA85EXaVm1XuVYtbGp839MyEzB. With Bitcoin, people can create as many accounts as they want to. An account really has an effect only when someone has created a transaction using it, and thereby depositing Bitcoins into that account. Anyone can deposit money (Bitcoins) into an account, but only the holder of the private key can transfer the money out of the account to another account.

If the Distributed Web uses the same math function for creating public and private keys that Bitcoin does, then the Distributed Web’s identity system will be compatible with Bitcoin accounts. This has an interesting advantage that anyone could leave a tip for any writer on the Distributed Web because his public key would be his Bitcoin account. In this way, we could make it easy for payments, even very small ones, to be made in the Distributed Web.

I believe it would be even possible to use Bitcoin-like technology to require a payment before a reader can decode a file, say a movie. In this way, we may have a distributed way to sell digital files on the Internet without any central clearinghouse. It would still be possible to rip someone off by buying a file, decoding it, and then redistributing it, but this is true now. What would be different is that it would be easy to make micropayments and full purchases on the Distributed Web without third parties getting involved or taking a slice. Automated tipping could even be installed to try micropayments as a default behavior.

Locking the Web Open

In conclusion, through the last 25 years, people have poured their lives and dreams into the World Wide Web, yielding a library and communication tool that is unprecedented in scale. We can now build a stronger tool on top of the current Web to offer added reliability, privacy, and fun.

Our new Web would be reliable because it would be hosted in many places, and multiple versions. Also, people could even make money, so there could be extra incentive to publish in the Distributed Web.

It would be more private because it would be more difficult to monitor who is reading a particular website. Using cryptography for the identity system makes it less related to personal identity, so there is an ability to walk away without being personally targeted.

And it could be as fun as it is malleable and extendable. With no central entities to regulate the evolution of the Distributed Web, the possibilities are much broader.

Fortunately, the needed technologies are now available in JavaScript, Bitcoin, IPFS/Bittorrent, Namecoin, and others. We do not need to wait for Apple, Microsoft or Google to allow us to build this.

What we need to do now is bring together technologists, visionaries, and philanthropists to build such a system that has no central points of control. Building this as a truly open project could in itself be done in a distributed way, allowing many people and many projects to participate toward a shared goal of a Distributed Web.

20-distributedwebTogether we can lock the Web open.

We can make openness irrevocable.

We can bake the First Amendment into the code itself, for the benefit of all.

We can build this.

We can build it together.

Previous writings on this subject:

Decentralized and Distributed systems and communities: Maelstrom by Bittorrent, MaidSafe, Namecoin / Ethereum, Bitcoin for payments, Proof of Storage (blockchain), Oceanstore, I2p, IPFS, Storj, Peer5, Tahoe-LAFS, Twister, Peerjs / Web RTC, BitcoinJS,,

Posted in Uncategorized | 19 Comments

How about 3 billion people, all living the good life?

We talk about rolling back our pollution production to 1990 levels, we talk about solving world poverty, we talk about reducing water use.   Why don’t we talk about rolling back our population explosion?  What if we rolled back the human population to 1960’s level of 3 billion?  Might this not help us solve all of this?     1960 did not feel like an uninhabited world– what if we went back to that population level?  I am not suggesting a war or anything, just a decrease in the same way we increased– naturally, but with determination.

It is possible that the world could sustain 3 billion people living on earth, and all of these people living the good life.   Could we have the courage and the resolve to find a peaceful way to achieve this?   First of all it takes discussing it.

So, for a moment, lets explore what the earth would be like with modern technology, but with 3 billion people.    Lets explore what would happen if most or all of these 3 billion got to live “the good life,” or more exactly, got to live the life they wanted to live.     What is the good life?   Healthy food, clean water and air, good medical care, shelter, desirable professions and reasonable workdays.   Time off for family and fun.   I find the upper middle class life in the United States pretty desirable.   What if everyone had the opportunity and choice to live that way?  Could we make it work?

Reuter’s reports that the world’s middle class has not grown much since 1960, but the number and percentage of people in poverty has grown greatly.   What if we could have the 3 billion people on earth be middle class and above.   This would mean we would replace the haves-and-have-nots with the haves-and-have-mores.

With the current technology developments we soon will no longer need poor people to make the gadgets for the haves– we have robots coming online at a very rapid pace.   I visited the Tesla factory a month ago and it is a sea of robots.  There were lots of people there, but they were not doing the robot-like work– they were talking with each other and watching over the consoles.   Soon our phones will also be assembled by robots, especially if the price of labor went up.     People will work, but they will not have to work like robots.

We do not need poor people to toil in farms to feed us either.   Some people will like to farm, which is terrific, but we will not need to import people far from their homes, pay exploitative wages and give them few citizens rights, just to grow the food for the good life.    Already corn, wheat, and soy are tended mostly by machines.   Fruits and vegetables can also be tended by machines if there were not people willing to pay a reasonable wage to people that choose to do this.   With better technology we could illiminate most of the need for pesticides and destructive fertilizers.   With better sensors we don’t need to use as much water.   We can take care of the basics with machines, and leave the fun work for people, the work they choose to do.   What we need to do is not have to keep growing exponentially to feed a surging population, each expecting better and better food.

Other low paid jobs can be replaced and are being replaced by machines:   customer service, retail, taxi drivers, garbage removal, and the like.    Good riddance.   So people can live a good life without an underclass to serve them.   If people want to have those jobs, they can, and they might just if people are willing to pay them enough so they could have a good life.   The key is here, that they do not have to.

Why 3 billion?   It is a pretty arbitrary number, but might have some things going for it.    For instance, if we take the top-earning 3 billion of the current 7 billion people now, and look at the water, energy, land they use, that could be an approximation of what we might use if there were 3 billion people living the good life.    We need to develop ways to conserve water and energy, and I am assuming these will continue.

World energy consumption has tripled to quadrupled since 1960.   Since the population has about doubled since 1960, and the well-off use most of the energy, then we may not bring down our energy use much from current levels, but it would not rise above it.

World food production has grown since 1960, some unsustainably, and returning to 1960 level population means we would all be fed, and fed well.   And we must do this with less strain on the Earth, and by having a smaller population would help.

The earth might be able to sustain 3 billion people living the good life.

How would we reduce our population?

First we have to want to.    First it starts with a goal, and an explaination.  This essay is my attempt to at least bring up the subject.

Each country could strive to get their population back to where it was in 1960. Each would come up with different ideas on how.

Then we need to have a way to do it.   China implemented the ‘one child policy‘ in 1980 which has helped them bend their population curve.    Indonesia did a program called “2 is enough” which has helped somewhat.   We will need to continue these programs and spread them.   Waiting for infant mortality or disease to sweep through is not a good future, we need to commit ourselves to a one child policy until we get the population back to where we think it can be.

This requires popular will and political will.   Currently, most countries are not strong enough to even have the conversation much less implement it.   China is the exception to the rule, not the rule.    I think of what would happen to an American politician that proposed a one-child policy, or even a “two is enough” policy– probably nothing good.

So we need to start it at the popular levels, bring up the discussion.   Get the meme rolling.   Maybe 3 billion is not the right number, maybe it is 2 billion,  maybe it is 4– but lets have the discussion.

Starting now.







Posted in Food, Health | 1 Comment

Locking the Web Open, a Call for a Distributed Web

Presentation by Brewster Kahle, Internet Archive Digital Librarian at Ford Foundation NetGain gathering, — a call from 5 top foundations to think big about prospects for our digital future. (via )

Hi, I’m Brewster Kahle, Founder of the Internet Archive. For 25 years we’ve been building this fabulous thing—the Web. I want to talk to you today about how can we Lock the Web Open.

CodeLawOne of my heroes, Larry Lessig, famously said that “Code is Law.” The way we code the Web will determine the way we live online. So we need to bake our values into our code.

Freedom of expression needs to be baked into our code. Privacy should be baked into our code. Universal access to all knowledge. But right now, those values are not embedded in the Web.

IA_serversIt turns out that the World Wide Web is very fragile. But it is huge. At the Internet Archive we collect 1 billion pages a week. We now know that Web pages only last about 100 days on average before they change or disappear. They blink on and off in their servers.

map_China_RussiaAnd the Web is massively accessible, unless you live in China. The Chinese government has blocked the Internet Archive, the New York Times, and other sites from its citizens. And so do other countries every once in a while.

Censorship_flic.kr_p_gZZRQvSo the Web is not reliableAnd the Web isn’t private. People, corporations, countries can spy on what you are reading. And they do. We now know that Wikileaks readers were targeted by the NSA and the UK’s equivalent. We, in the library world, know the value of reader privacy.

It-is-FunBut the Web is fun. We got one of the three things right. So we need a Web that is Reliable, Private but is still Fun. I believe it is time to take that next step. And It’s within our reach.

Imagine “Distributed Web” sites that are as functional as Word Press blogs, Wikimedia sites, or even Facebook. But How?

Tubes_flic_kr_p_89HvvdContrast the current Web to the internet—the network of pipes that the World Wide Web sits on top of. The internet was designed so that if any one piece goes out, it will still function. The internet is a truly distributed system. What we need is a Next Generation Web; a truly distributed Web.

Peer2PeerHere’s a way of thinking about it: Take the Amazon Cloud. The Amazon Cloud works by distributing your data. Moving it from computer to computer—shifting machines in case things go down, getting it closer to users, and replicating it as it is used more. That’s a great idea. What if we could make the Next Generation Web work that, but across the entire internet, like an enormous Amazon Cloud?

In part, it would be based on Peer-to-peer technology—systems that aren’t dependent on a central host or the policies of one particular country. In peer-to-peer models, those who are using the distributed Web are also providing some of the bandwidth and storage to run it.

Instead of one web server per website we would have many. The more people or organizations that are involved in the distributed Web, the safer and faster it will become. The next generation Web also needs a distributed authentication system without centralized log-in and passwords. That’s where encryption comes in.

Private-e1423717177408And it also needs to be Private—so no one knows what you are reading. The bits will be distributed—across the Net—so no one can track you from a central portal.

MemoryAnd this time the Web should have a memory. We’d build in a form of versioning, so the Web is archived thru time. The Web would no longer exist in a land of the perpetual present.

Plus it still needs to be Fun—malleable enough spur the imaginations of a millions of inventors. How do we know that it can work? There have been many advances since the birth of the Web in 1992.

Blockchain_JavaWe have computers that are 1000 times faster. We have JAVAScript that allows us to run sophisticated code in the browser. So now readers of the distributed web could help build it. Public key encryption is now legal, so we can use it for authentication and privacy. And we have Block Chain technology that enables the Bitcoin community to have a global database with no central point of control.

NewWebI’ve seen each of these pieces work independently, but never pulled together into a new Web. That is what I am challenging us to do.

Funders, and leaders, and visionaries– This can be a Big Deal. And it’s not being done yet! By understanding where we are headed, we can pave the path.

DistributedWebLarry Lessig’s equation was Code = Law. We could bake the First Amendment into the code of a next generation Web.

We can lock the web open.
Making openness irrevocable.
We can build this.
We can do it together.

Delivered February 11, 2015 at the Ford Foundation-hosted gathering: NetGain, Working Together for a Stronger Digital Society

Posted in Uncategorized | 3 Comments

The Myth of the Self-Made Man, or maybe better said: Thank You.

My dear sons, Caslon and Logan–

Those that proclaim themelves “Self Made”, I am convinced, are either ignorant or marketers. Ignorant of the many people that help us along, push us along, often without recognition. Or marketers that are purposefully simplifying the story so as to sell something, in this case, themselves.

One proverb is “we make our own luck”, which, like all cliches has some truth to it but misses the point: which is that circumstances and people outside of our control make up the vast majority of the “luck” that lands us where we are. Gender, economic cycle, “being at the right place at the right time”, an unseen opportunity becoming available because someone else offended yet another person at just the right time, at the least the right time for you.

For myself, I feel lucky. And in my case because I was lucky. Born white, male, upper-middle class, to a functional family, in a country whose economy was soaring (because the rest of the world had been bombed flat 15 years before)– I was off to a easy start. Being a math geek at the fortunate time when computers were on the rise let me learn from a high school friend, Rob Bedichek, who hand-wired his own computer out of logic chips and switches. Then having the combined miracle of supportive teachers and an unseen college admissions officer that let a barely top-20%-of-his-class guy into MIT, gave me a leg up that I have not been able to live down completely.

Top career advice came a few times, one from Professor Gerry Sussman when I called to ask him to hire me into his lab so I could learn to make chips to protect the privacy of all phonecallers. He said, “I don’t know you, why should I hire you? Just come in, start working, and if you are any good then someone will hire you.” Little rough, kicked me in the butt, and his lab delivered– best advice ever.

Another was from Marvin Minsky and Danny Hillis: try to do your big idea, maybe in steps, but with the big idea always in mind– you may not achieve it, but achieving a goal is overrated. It is the journey and your fellow travelers that are the point. As Laurie Anderson pointed out from Moby Dick– it might not turn out so well for those that get their whale.

Outdated implicit advice from my parents: keep your head down. They lived this advice based on their growing up in the McCarthy era United States when trying new things could get you blacklisted, and did tank many people’s possible careers. But I found being bold to have worked well in my era, more ‘open’ the better, more giving, more straightforward the better. People then understood what I wanted to do, and then could more easily help. (This approach reached a logical conclusion, but tragically, with Aaron Swartz, someone who lived a completely open-source life, and worked for the public good, but in his case, he was crushed by institutions around him and was driven to suicide.)

Why do I get a beautiful view overlooking the San Francisco Bay this February morning? Why do I get to go sailing with a beautiful wife most weekends? Thank you.

So what is the point? I have been invested in, I have been made by my communities. At most we can be worth investing in, and appreciative of the support we receive at the same time as we invest in and help others along their paths. Few will win the lottery, but we can take satisfaction, and happiness, in the successes around us.

Thank you, all.

Posted in Education | Tagged , | 2 Comments

When I First Realized I was being Lied to…

When I first realized I was being lied to, systematically lied to, and by the government, I felt upset, then felt duped, and it started me thinking– how far does this lying go?

It was in the beginning of college and it was the government’s messages about the Vietnam war and marijuana.    I found out that it was not just a matter of point of view, of older-wiser people teaching lessons I was resisting.   No, it was flat out lies.   Things they knew were wrong, but were saying were true.    Lies.     It was hard to take.

This would have been 1978, and I was 18 years old when it felt like a light was turned on in the room.    It may sound like I was naive or unusually sheltered but I don’t think I was that abnormal.    I was taught that police were to be feared and respected, their tactics might be harsh, but it was a grownup world and their motivations were mature.

But it was the stinging realization that these lies made a big difference in people’s lives, my life, that made the lies stabbed me, then made me doubt, question, and distrust the powerful.   Shifting from thinking of power rather than maturity.

The drug messages in the 70’s was pervasive…   if you start with marijuana you will end up on heroin and in a gutter.   Marijuana made you crazy and would lead to birth defects.    It was the movies we were shown, it was on the tv news and in newspapers, it was in underground books that were circulated like “Go Ask Alice” that were lies all the way through.   And that was not all.

The Vietnam police action (it was not a war, they said) was necessary to stop a domino strategy of communist world domination.    Resisting was unamerican, and ungrateful for not fulfilling our social contract– not doing the Right Thing.    Vietnam was a puppet of China, we were always about to win, we don’t want our boys to have died in vein.   As I found it– I was being lied to.  Systematically, knowingly, and with grave consequences.

I started to, as they say, “Question Authority”, and I found more and more holes in the logic and rottenness in the motivations.    I looked for answers in philosophy classes, maybe they could help me figure out if I should register for the draft– I took a class on social contracts studying Hobbes Locke and Rousseau, but the Leviathan seemed to be a justification, and an a-historic justification for absolute monarchs.     I studied western religions at a divinity school, but those approaches did not seem to pass the logic tests.    Only Zen Buddhist practice seemed to avoid obvious shortfalls, but only gave general guidance.   Maybe that was the best we could do.

Reading real scientific studies of the effects of drugs on the brain was a way to find very different answers from those in Time magazine and the evening news that purported to be built on the same evidence.   Scientific writing, and the scientists behind them, while limited, seemed to at least not just be making things up to justify the agenda of the powerful.

Maybe this is just growing up, but I don’t think it has to be this way.   We do not need to have to teach our children from a young age that they are being consistently lied to by powerful entities, that they are being sold things that are bad for them, that they should fear the police and not believe them because the police are encouraged to lie to get confessions.

We can do better than this, we can build and live in societies where we do not have to constantly question secret motives.    We can dis-empower the institutional structures that profit through deception.    Large corporations and governments seem to have incentives to take shortcuts and deceive.   Maybe we could replace their functions with responsive and local organizations that are transparent and straightforward.   Invest in those we trust and teach our children that they do not need to accept deceit as “just the way it is.”

As a kid, being caught lying was a big deal that came with consequences.    Lets have that apply to grownups too.

[reading later, there is this nytimes article that tells of this fellows learning he was lied to about the vietnam war when he went to college.]

Posted in Education, Uncategorized | Comments Off on When I First Realized I was being Lied to…