What happens to your data if your experiment fails? If the results turn out different than you thought? If there’s nothing to publish?

Don’t throw it away. Your dead end might be another person’s missing link.

Wired Magazine has an essay by Thomas Goetz on this “dark data”.

He says that while storing huge amounts of data can be an issue, the real problem is the culture of science:

More and more, research is funded by commercial entities, which deem any results proprietary. And even among fair-minded academics, the pressures of time, tender, and tenure can make openness an afterthought. If their research is successful, many academics guard their data like Gollum, wringing all the publication opportunities they can out of it over years. If the research doesn’t pan out, there’s a strong incentive to move on, ASAP, and a disincentive to linger in eddies that may not advance one’s job prospects.

But Goetz says that the dark data phenomenon isn’t limited to science:

Getting science comfortable with exposing its dark data is really just the beginning. Once you start looking for it, dark data is everywhere: It’s locked away in out-of-print books and orphaned art, the stuff that Creative Commons and Google Book Search have been bringing to light. Speaking of which: Hey, Google! Know all those research projects your employees do that the company will never green-light? How about letting the rest of the world take a crack at them?

The challenge is to find a way to share the data. How about the Comprehensive Knowledge Archive Network I blogged earlier?

Rufus Pollock must have had a busy week. A few days after the Open Knowledge Foundation he’s involved in launched the Comprehensive Knowledge Archive Network, Rufus (who is a PhD candidate at Cambridge) has published a paper. From an economist’s perspective, he is looking at the optimal term for copyright:

Abstract
The optimal level for copyright has been a matter for extensive debate over the last decade. This paper contributes several new results on this issue divided into two parts. In the first, a parsimonious theoretical model is used to prove several novel propositions about the optimal level of protection. Specifically, we demonstrate that (a) optimal copyright falls as the costs of production go down (for example as a result of digitization) and that (b) the optimal level of copyright will, in general, fall over time. The second part of the paper focuses on the specific case of copyright term. Using a simple model we characterise optimal term as a function of a few key parameters. We estimate this function using a combination of new and existing data on recordings and books and find an optimal term of around fourteen years. This is substantially shorter than any current copyright term and implies that existing copyright terms are too long.

This sounds very interesting. Go get the paper here (pdf), read it, pick holes in it, and see if it holds up!

via BoingBoing, netzpolitik.org

Today, the UK-based Open Knowledge Foundation has launched the “Comprehensive Knowledge Archive Network” (CKAN). What this is?

CKAN is a registry of open knowledge packages and projects — be that a set of Shakespeare’s works, a global population density database, the voting records of MPs, or 30 years of US patents.

CKAN is the place to search for open knowledge resources as well as register your own. Those familiar with freshmeat (a registry of open source software), CPAN (Perl) or PyPI (python package index) can think of CKAN as providing an analogous service for open knowledge.

[...]

we are looking for people to register ‘packages’ that is collections with some kind of structure rather than individual items. So a substantial set of photos, a datasets of all kinds, the writings of Shakespeare but not an individual blog, or your flickr photo collection (unless it is really big!).

“Open” meaning here that anyone is free to use, enhance, modify and distribute the things in this archive, similar to Free Software. The data and information contained in this archive network either has to be in the public domain (not subject to copyright), or distributed under a free licence (e.g. Wikipedia articles, which use the GNU Free Documentation Licence, or works under a free Creative Commons licence).

The discussion on how to organise knowledge best is far from ended, so this is just a basic infrastructure. Yet it already contains a few things of value:

Shakespeare’s works, a global population density database, the voting records of MPs, or 30 years of US patents.

I’m really looking forward to how this will develop. Like any self-respecting Internet innovation these days, it is a public beta, to be improved over time. But looking at how valuable similar archive networks have become as a resource – for example, the Comprehensive Tex Archive Network (CTAN) for the TeX typesetting system -, this could become really, really good.

Knowledge Ecology International (formerly CPTech), a Washington DC-based NGO that is a real hub in the debate on Access to Knowledge, is doing the right thing and putting out a journal, Knowledge Ecology Studies. Sounds promising:

KE Studies is an online publication that focuses on the creation, dissemination and access to knowledge goods. It is a multidisciplinary journal that draws on a number of specialties: sciences, technologies, public policies, the laws of intellectual property, business, free speech and privacy, telecommunications and other related knowledge disciplines.KE Studies strives to be a publication in the classic sense of the word, but with new opportunities for authors and readers. The papers and other forms of published materials examine new developments, alternatives and social implications. We also review policy proposals as well as books, articles, media, and any other type of knowledge good.

Unlike traditional journals, which focus primarily on lengthy scholarly articles on specific fields, KE Studies focuses on pieces of different format –articles, short research papers, commentaries written by experts in various fields about production, dissemination and access to knowledge goods broadly defined including medical innovation policies, legal issues, educational issues among others.

In addition, KE Studies strives to be a publication with a fast turn around and inclusive to non-published authors from all over the world. We publish pieces on unscheduled timing with an end of the year compilation volume.

In a pre-published piece for The New Yorker, Jeffrey Toobin writes about the Google Library Project. This is a different project from the Google Book Search project which caused some controversy in the past. Google Book Search is a commercial project that Google launched in which it links search with “snippets’ of book content and, of course, advertisement. The Google Library Project is however supposed to be the incarnation of Google’s corporate motto that it wants to make the world knowledge accessible to all, free. This all sounds very philanthropic but we might want to have a good look at what is going on.

As can be imagined, publishers and authors have been up in arms about this project as these groups are afraid that Google is riding rough-shod over copy-right laws (and their business’ “raison d’etre”). And they are right to do so as in principle this is what Google is doing. Google has however found some powerful partners to do this with. Although some of these are not willing to go as far as violating copy-right law some others are.

This project is in fact not a novelty and the problems encountered are all well known. Libraries around the world have been looking into digitizing projects for more then a decade and projects such as Project Gutenberg have been around for quite a while as well. The problem these two have encountered is money. To undertake projects this size a huge budget is necessary, which is something that publicly funded organizations, in the case of libraries, or organizations which hinge on public support such as the Project Gutenberg do not have.

There are also other initiatives such as the Open Content Alliance, a Microsoft sponsored outfit, but these are trying to play be the rules and will most probably loose out to Google in the long run as Google is prepared to inject millions into to this, in making deals with the publishers and authors and investing in technology and manpower. By making these trade deals and by investing in digitizing technology R&D, in short; cutting out the competition, Google will in effect emerge as the monopolist in this field. A possibly highly profitable field. It remains to be seen whether they will honor their pledge of not infesting the Google Library Project with advertisement and other harassements, as unlike for instance libraries they do not seem to adhere to a “Socrates oath” of providing information virtually free and without bias (remember google.cn).

Somehow in looking at this issue, I get the feeling that the majority of people do not see the difference between ‘open content’ and ‘open access’. The Google Library Project is truly a good idea, a noble one even, if they truly will provide unhindered and unbiased access to the digitized content of their books database. However I remain sceptical as I doubt whether they will really be able to capture the world knowledge for one and secondly, looking at Google’s track-record, whether they will really provide open access to it. We need to remember; Google is a company, a very good advertising company.

People called it a “cool story” (via Lessig’s blog via Veni Markowski’s blog) when the Bulgarian Foreign Ministry announced it will start using a Creative Commons 2.5 license for all of its publications. Creative Commons licenses allow authors to determine which rights they want to share and which they would rather reserve – in a more fine-grained and sensible way than the default copyright that is attached to everything we write.
What is a bit curious is that we file the decision by a public body to make its resources available to the public, under “cool story” rather than “boring mundane expected”. Maybe one day nobody will raise a mouse-finger …

We released this policy brief a few weeks ago in prepration for the Symposium on Open Standards and the new intellectual property rights paradigms that are coming out of the various ‘open’ movements (open science, open source software, open medicine, etc.).

Because the policy brief is a bit hidden away on the merit website, I thought it made sense to post a link to it from the A2K blog. Please note that to get a list of relevant UNU publications that we have mentioned on the blog, you can select the “publications” category link in the navigation menu.

The policy brief is here (http://www.merit.unu.edu/publications/pb/unu_pb_2006_01.pdf)