I have been making this argument lately by raising the concern that the free software/culture movement is being outflanked by a land grab for data. Free software is only one corner piece of this puzzle – to complete the jigsaw we need the corners of free data, in a free format. If I can get my data back out in a free format, do I really care which calendar service I am using (if the data is meant to be public and shared, that is)? As the applications become commodities, data is king.

http://alchemicalmusings.org/2006/03/12/saints-in-the-church-of-writely/

I also just saw this quote in a circa 1997 Oracle MDM whitepaper: “It has been said that data outlasts applications.” (see http://www.oracle.com/master-data-management/oracle-master-data.pdf ).

In my opinion, the following is the most valuable statement among your slides and deserves much attention and discussion:

“Much of the value in our data will be unexpected and unintended, therefore we should engineer for serendipity.” – Ian Davis

Data may be important, code may be important, but the ability to incorporate into engineering certain methodologies to enhance the ability to read and understand significance in data is magic – especially in a manner that does not excessively limit potential meaning in data through preconceptions. Data is in the eye of the beholder, at least as related to using data for decision-making.

How do we ask questions that help us to engineer for serendipity? This, in my opinion, is a hugely significant question for all of us that we ought not obscure through an open data vs open source argument. I think this question should be investigated on its own.

How do we add value to data by engineering for serendipity?

This fits in with another comment you made:

“Network effects arise when the act of participation makes the entire network more useful for everyone.” – Ian Davis

This is an excellent statement! Most of us know how the example of social networking fits in to this statement. We know that the more people who participate in social networking, the more valuable that network can become for each of us. We know how the web fits in, as for example, your presenting these ideas on a public web page makes the web more useful for each of us. But not enough of us understand networking effects in raw data, especially within a corporate business environment. Not enough of us understand that keeping data in a spreadsheet within one department may add insufficient value to the organization because there can be few serendipitous value creations since few people will be exposed to the data. Placing the spreadsheet on an intranet might not be the most efficient method of producing the network effects that are possible.

Combining network effects with engineering for serendipity is a good recipe for changing the world by making data not only open, but also useful and positioning it squarely in the eye of the beholder.

I’d very much like to pursue this further.

What “open” are you referring to in “open data” in the context of this conjecture? As you say, legal restrictions expire, eventually. Do you mean data in open formats and open standards? Those are defined by a legal component (lack of patent encumberance, which is more certain to expire if present anyway than are copyright restrictions), documentation, and sometimes — open source reference implementation.

If pressed I’d argue the whole “open” stack requires free/open source software (even if all software isn’t), thus in a sense “open” software is more important than “open” data (whether data is more important than software is a somewhat different argument). For something along those lines (briefly, open formats/standards was a losing battle without source code … the article at the link tries to argue that we’re in danger of losing the battle again as code moves to servers), see http://lists.canonical.org/pipermail/kragen-tol/2006-July/000818.html … note that this is from a person who considers closed formats unethical http://lists.canonical.org/pipermail/kragen-tol/2002-July/000725.html

Or maybe by “open data” you merely mean data which is accessible, eg downloadable. Your comments toward the end contrasting having lost your data and having lost your code seem to indicate that this is what you mean. Ok, but this again seems rather facile, at least not without some context. For example, you could argue that it is more important for governments to make their data available than it is for governments to use open source software.

Software comes and goes, but data sticks around. The data is actually more expensive to produce or replace than software.

One of the unfortunate things about much of our data is that it was sort of fitted to the idiosyncracies of a particular piece of software that was going to be used to display it when it was created. Values were put in certain fields because they would then be displayed (or not displayed) by certain software, in an idiosyncratic ad hoc way. But the data has long outlasted the software whose behaviors it was molded to.

You’d think catalogers would be pleased to have computer programmers acknowledging that what they do (data control and generation) is more important than what we do!

You seem more at the evangelist end of the CTO spectrum, and this debate will hopefully become more commonplace and thought provoking.

Looking forward to more semantic serendipity!

i’d even go a bit further and propose that whenever you have to create code to access/manage some data, this must not take longer than – say – 1/10 of the time necessary to create the data. and maybe we should put a hard limit of 5 years on that, too, because in most parts of the programming world, that’s about the time it takes to make technologies obsolete.

thanx for getting back on this and your detailed explanation of your perspective.

From my perspective, your arguments seem to be build on two assumptions:

1. data is static
2. code is cheap to build

About 2: in science this is typically not the case. Surely, writing a XML parser is easy enough, and writing a simple text editor likewise. However, much of the scientific code output is the outcome of several years of development. Believe me, it is not cheap to rebuild scientific software. The market is scientific programmers is so small, the costs for rebuilding software is quite. Unlike with desktop software, where there is a huge user market, and redoing it is trivial.

Unfortunately, it is common to not distinguish between desktop software and scientific software. People typically do not understand what makes development of new scientific algorithms more difficult than writing a desktop tool. A choice of CSV versus XML is a non-issue in scientific software design; Proper representation of the scientific problem, that requires a lot of domain knowledge, and a lot of testing against that knowledge. These scientific algorithms easily outlast a lot of the data around.

In short, the development of scientific software which reflects new scientific insights is certainly not by default cheaper than measuring the data again.

About 1) However, there is another reason why data does not outlast code. Street map data of 100 years ago surely has some historical value, but I leave it to the reader how useful it is to current problems. Likewise, chemical data measured 50 years ago is certainly not as useful as it is now: measurements we make now are so much more accurate and more precise. Infrared (IR) spectroscopy was the main identification method in organic chemistry until NMR came around. The latter practically made IR obsolete, though it often still is used to back up the presence of a certain chemical fragment. I’m sure this is even more the case in biological sciences.

Old scientific data is nice, but to use it, you just have to redo your experiment (it’s exponentially cheaper to do the measuring now, then when it was originally done), gaining much more information. Who cares about the speed of light measured 100 years ago. That data did simply not last. Scientific data does not last (and I do not think telephone numbers of 5 years ago are useful either). The assumption that data is static and lasts is flawed. The assumption that measuring things is prohibitively expensive is flawed too. Well, if you don’t have code, it surely is.

Bottom line is that either can be less or more expensive to reproduce: the data and the code. At least in science.

Egon