This site has moved to
The posts below are backup copies from the new site.

May 9, 2013

'Big Data Needs a Big Theory to Go with It'

Travel day, so some quick ones before heading out:
Big Data Needs a Big Theory to Go with It, by Geoffrey West: As the world becomes increasingly complex and interconnected, some of our biggest challenges have begun to seem intractable. What should we do about uncertainty in the financial markets? How can we predict energy supply and demand? How will climate change play out? ... To bring scientific rigor to the challenges of our time, we need to develop a deeper understanding of complexity itself.
What does this mean? Complexity comes into play when there are many parts that can interact in many different ways so that the whole takes on a life of its own: it adapts and evolves in response to changing conditions. It can be prone to sudden and seemingly unpredictable changes—a market crash is the classic example. One or more trends can reinforce other trends in a “positive feedback loop” until things swiftly spiral out of control and cross a tipping point...
The digital revolution is driving much of the increasing complexity..., but this technology also presents an opportunity..., enormous amounts of data. ... The trouble is, we don't have a unified, conceptual framework for addressing questions of complexity. ... “Big data” without a “big theory” to go with it loses much of its potency and usefulness...  We now need to ask if our age can produce universal laws of complexity...
We won't predict when the next financial crash will occur, but we ought to be able to assign a probability of one occurring in the next few years. The field is in the midst of a broad synthesis of scientific disciplines, helping reverse the trend toward fragmentation and specialization, and is groping toward a more unified, holistic framework for tackling society's big questions. The future of the human enterprise may well depend on it.

No comments: