Latest News from Paul Miller

Discussing Virtual Machine Interoperability with the Open Data Center Alliance

Thu, 16 May 2013 23:00:00 EDT

The Open Data Center Alliance (ODCA) is holding its Forecast event in San Francisco in June, and I’ve been invited to moderate the panel discussing Virtual Machine Interoperability. As moderator, I’ll be far more interested in facilitating insights from panel and audience than in wittering on about what I think, so I wanted to use this blog post to begin getting some of the issues clear in my mind. What is VM interoperability, and why does it matter? From time to time, I write about Open Data. This has nothing to do with that. The Open Data Center Alliance is interested in data centres, not data. The Alliance was established back in 2010 with Intel driving things forward, and now claims over 300 member organisations, including the likes of BMW, Lockheed Martin, Microsoft, Deutsche Bank and Marriott Hotels. According to the ODCA website, we came together to deliver a unified voice for emerging data center and cloud computing requirements. Our mission is to speed the migration to cloud computing by enabling the solution and service ecosystem to address IT requirements with the highest level of interoperability and standards. Much of the Alliance’s work involves identifying customer requirements and capturing these in a series of usage [...]

Unpicking the multi-cloud at GigaOM Structure

Wed, 15 May 2013 11:46:43 EDT

Last month, RightScale’s State of the Cloud report got me thinking about the rise of multi-cloud solutions. Next month, I’ll be moderating a Mapping Session at GigaOM’s Structure event to work out how, where, when, why and if this trend is going to prove significant. Hybrid clouds, in which one public cloud and one private cloud are used together, are becoming increasingly common solutions to a range of business challenges. RightScale’s figures suggest growing interest in something more complex and, potentially, more interesting; multi-cloud. In a multi-cloud arrangement, customers build solutions combining one or more public clouds with one or more private clouds. This has the potential to significantly increase complexity, without necessarily delivering a comparable increase in value. In my post last month, I suggested that many of these multi-cloud deployments were essentially accidental. A quick email exchange with RightScale shows that their survey respondents would appear to disagree; multi-cloud, for them, is a conscious business decision. I’m intrigued, and so were GigaOM. So we’re putting on a Mapping Session at next month’s Structure conference to explore the issue further. I’ll be joined at the front of the room by fellow GigaOM Pro Analysts Ben Kepes and David Linthicum, as well [...]

Getting it right with data attribution

Fri, 10 May 2013 12:33:55 EDT

There have always, it seems, been people for whom attribution and citation really matter. Some of them passionately engage in arguments that last months or years, debating the merits of comma placement in written citations for the work of others. Bizarre, right? But, as we all become increasingly dependent upon data sourced from third parties, aspects of this rather esoteric pastime are beginning to matter to a far broader audience. Products, recommendations, decisions and entire businesses are being constructed on top of data sourced from trusted partners, from new data brokers, from crowdsourced communities, or simply plucked from across the open web. Without an understanding of where that data came from, and how it was collected, interpreted or maintained, all of those products, recommendations, decisions and businesses stand upon very shaky foundations indeed. Data attribution is increasingly important, but it will be essential to make sure that the rules, tools and norms which emerge are both lightweight and pragmatic. Now is not the time to get heavy-handed and pedantic about where the comma goes. Former colleague Leigh Dodds recently offered a useful discussion of the rationale behind data attribution. Early on, he describes the related (and, often, sloppily interchangeable) notions [...]

Seeking Simplicity’s Sweet Spot

Thu, 09 May 2013 10:10:54 EDT

Albert Einstein, you may have heard, was a clever man. He scribbled equations on blackboards, thought big thoughts, and all of that. But, allegedly, he also said Everything should be made as simple as possible, but not simpler. These words have resonated with me recently, as I’ve heard pitches from one company after another, all of which are trying to cut through the complexity of data to make it accessible. Their goals appear laudable, but all too often I find myself wondering how simple this stuff can be? If we make it too simple, do we run the risk of unleashing a flood of half-baked ‘analysis,’ undertaken by people who really shouldn’t be allowed near a calculator, let alone a Hadoop cluster? On the other hand there’s a persuasive argument to be made for democratising access to data and tools, freeing organisations from over-reliance upon their new High Priests of Data. Every data question should not require a data scientist, but maybe we really shouldn’t be making it too easy for people to tackle the hard questions without support from someone who knows what they’re doing. How simple, then, is too simple? And can we use data in a different way; [...]

Find the data, aggregate the data, make the data useful

Fri, 03 May 2013 15:12:55 EDT

I was in New York in March, taking part in GigaOM’s Structure:Data event. As usual on these trips, I spent the day before the event walking around the city, soaking up some air, getting rained on, using coffee to stay awake, and meeting with a number of local companies. Of the companies I met that day, one stood out. And this week, that same company was recognised by others when it won TechCrunch Disrupt NY. That company was Enigma. Enigma pulls data from tens of thousands of public data sets, and then offers up an interface that makes it pretty straightforward to trawl through the whole lot in search of the data points that you actually need. As the company’s Marc DaCosta introduced it, a “search and discovery platform for public data.” At present, everything in Enigma is publicly available data. It’s mostly from the USA right now, and is acquired by a combination of screen scraping/ crawling of .gov sites… and calling government agencies to request that they ship CDs of offline material. DaCosta stresses that the data is all — theoretically — available to any US citizen, but that it’s not really that easy for them to access. Even with [...]

Cloud Database Company Xeround, and a Tale of Evolving Business Models

Fri, 03 May 2013 07:15:00 EDT

Last night, cloud database company Xeround announced that they’re shutting down the version of their service hosted in public clouds such as Amazon, Rackspace, GreenQloud, and others. Users of the free service have until 8 May to move elsewhere, whilst paying customers have until 15 May. The company describes this as an attempt to “re-focus,” with the implication that other parts of the business remain viable. It’s never easy to admit mistakes and kill products, but the ability to do so is an essential part of running a business that’s viable for the long haul. Xeround’s announcement needn’t be interpreted as the end of the company, or the end of databases running in the public cloud. The challenge now is one of persuading staff, investors and customers to move past the short-term pain and uncertainty, and to get behind the new direction with conviction. Xeround was founded back in 2005, initially delivering scalable data management solutions into the data centres of telcos such as T-Mobile. The company’s work on scaling MySQL and offering it in the cloud brought it to wider attention, and they seemed to maintain momentum by adding additional cloud partners. Last year, the company rolled out a [...]

Survey lifts covers on Cloud Promiscuity: good thing, bad thing, or who cares?

Mon, 29 Apr 2013 07:14:24 EDT

Figures from RightScale‘s latest State of the Cloud Report (free registration required) suggest “a strong interest in multi-cloud strategies” amongst respondents. The rationale for hybrid cloud (mixing a public cloud service like Amazon’s with something running in your own data centre, colocation site or hosting facility) is reasonably well understood, but why might companies choose to use more than one public cloud and/or more than one private cloud? Surveys. Along with the loathsome infographic, they have become the default tool of the lazy PR person. I get loads of them, pretty much every day. And, almost without fail, they are a shocking waste of everyone’s time. Three out of the four people we gave some money to said our product was best. I don’t care. And I’m definitely not going to write about it. Every now and then, though, one that’s worth a little attention comes along. RightScale’s latest State of the Cloud, released last week, is one of those. The sample is relatively small (just 625) and, as Network World‘s Brandon Butler emphasises, they came from a community to [sic] people that had somehow, some way interacted with RightSacle [sic] in the past — they had registered on the company’s website, dropped a business card in a [...]

Visualisation – the key that unlocks data’s value?

Fri, 26 Apr 2013 07:44:09 EDT

As the Big Data hype machine continues its relentless attempt to gobble everything in its path, new business units and entire new domains buying into the promise find themselves faced with unanticipated data volume and complexity. They see the potential for data-based decision making, but still face (short-term?) challenges in actually managing, analysing or interpreting the data they now collect. Early iterations of core tools such as Hadoop were raw and unpolished, driving the emergence of a niche group of developers and data analysts with the specialist skills to cope. Those tools become easier to use with each new release, which goes some way toward countering panicked claims of a massively yawning skills gap. There is, of course, still a need for people with specialist knowledge, hard-won skills, and painfully gained experience. But you no longer (if you ever really did) need to install a Hadoop cluster with your bare hands and juggle complex statistical formulae in your head to benefit from the growing prevalence of data in all aspects of business. At the ‘softer’ end of the market, specifically, there has been an explosion of new startups rushing to offer tools that make it easier to create visualisations and [...]

Not Quite Ready to Live in the Cloud

Sun, 21 Apr 2013 11:30:00 EDT

Google’s impressive Chromebook Pixel is just the latest in a series of devices which are trying to entice users to compute in a different way. With (almost) ubiquitous connectivity, and an increasing reliance upon web-based services for mail, calendars, document creation and more, might we be reaching a point at which the browser really can be our means of accessing everything? Philosophically, the idea resonates. And yet, although I am not a power user who needs to regularly process video or edit high resolution images (the usual excuses for not embracing the Chromebook vision), I still remain uncomfortable with giving up my non-browser tools. Despite living and working in the cloud, I find that locally installed client software continues to deliver real value. Maybe, the next time I upgrade a computer, I need to try installing nothing more than a browser for a week or two, and see if it’s as painful as I feel it could be… The cloud powers my business. The cloud is what I talk to clients about, it’s what I write about, it’s what people pay me to know about. The cloud (and, more generally, the web) make it possible for me to work with clients [...]

OpenStack Summit – thoughts from Portland

Thu, 18 Apr 2013 18:28:09 EDT

OpenStack has come a long way since the project was first unveiled at OSCon back in 2010. This week, almost 3,000 people gathered in Portland, Oregon, to continue the job of defining, debating, developing, and delivering the code upon which the OpenStack community depends. Alongside the developers, though, there were some early signs of tangible adoption. During Monday’s Analyst Day, and in sessions throughout the conference, we began to see evidence of fledgling steps being taken beyond the early adopters on the bleeding edge. As Hewlett Packard’s Florian Otel remarked in his session, “it’s called the bleeding edge for a reason.” Some people want to be there. Some people need to be there. The majority of adopters, however, simply want robust, credible, dependable, and wide-ranging solutions that work. After a period in which OpenStack’s hype, perhaps, outpaced its reality, might we finally be seeing some credible proof points beginning to emerge? The fledgling OpenStack Foundation clearly recognises its own need to move beyond an introspective community of tech enthusiasts playing with an interesting new toy. The video (embedded below) which loudly greeted the audience for Tuesday’s opening keynote was not subtle. If OpenStack is to succeed, the e.n.t.e.r.p.r.i.s.e is where it must [...]

To Dublin, in search of evidence

Mon, 15 Apr 2013 08:40:54 EDT

I travelled to Ireland last week, to attend the second meeting of the European Data Forum (EDF). The EDF provided travel support for my trip, and I am grateful to them for that. I was searching for evidence of ways in which smart use of data is having a transformative effect upon European businesses. Although some of those effects could be inferred from the presentations and hallway conversations, I was disappointed by the lack of focus upon explicitly enumerating these benefits. Plenty of people described what they did. Plenty of people showed pictures of their work, and their results. Very few told us how their efforts made anything better, cheaper, quicker, or more responsive, and that was a shame. This evidence really should exist. Was I at the wrong event? Was I talking to the wrong people? Or was I failing to see something that was actually right before my eyes? So, with my disappointment out of the way, what was good? Actually, plenty. Before getting there, though, some background. The EDF describes itself as a meeting place for industry, research, policymakers and community initiatives to discuss the challenges of Big Data and the emerging Data Economy and to develop suitable [...]

Doing the DataBeat

Tue, 09 Apr 2013 16:05:39 EDT

For the past two years, Ben Kepes and I have helped the team at VentureBeat assemble the programme for their annual Cloud Computing event, CloudBeat. It looks as though we may end up doing something similar with them this year, as CloudBeat moves from Redwood City to downtown San Francisco, and from November to September. But that’s for a subsequent post. This year, VentureBeat — and I — are trying something new; a conference all about data. The name, unsurprisingly, is DataBeat. The venue, Redwood City’s Sofitel. The date, 4-5 December 2013. I’m doing this one without Ben, and I’m also taking more of a strategic role this time around; less about content advising and much more about content deciding. It should be fun, and we’ve just opened the call for participation. So if you have a story to tell, get that form filled in to let us know! So, in a world where there seems to be some Big Data event or another every single day, why bother? I’m certainly not interested in putting on yet another me-too event, riding on the back of Big Data’s cacophonous hype and near-bottomless marketing budgets. Talking all of this through with VentureBeat Founder and Editor [...]

Is Infochimps running from the Data Market business?

Thu, 28 Feb 2013 06:24:55 EST

Infochimps is one of the early champions of the data market business, and one that I’ve followed for several years. As I mentioned in my last post on the topic, the company has recently begun to pivot towards delivery of their (compelling) Enterprise Cloud big data analysis offering, with the company’s data market origins slipping further and further down the home page. And now, it seems that they’re taking the next step. A friend drew my attention to Infochimps’ stated intention to stop providing API access to a number of data sets from today, 28 February. Historically, Infochimps has offered several ways to access data through their marketplace; 5,731 datasets, hosted by Infochimps and made available for download 5,303 datasets, catalogued by Infochimps but made available for download somewhere else 14 (yes, fourteen) datasets, available for online query via Infochimps’ API There’s no sign (yet) that the 10,000 or more data sets catalogued by Infochimps and available for download from themselves or their partners are going away. The 13 datasets affected (a dataset of 60,000 UFO sightings will still be available via API) represent a mere drop in the ocean, and maybe we shouldn’t care. But it’s the fact that this [...]

Discussing Data Markets in New York City

Tue, 05 Feb 2013 11:21:00 EST

As part of GigaOM’s Structure:Data Conference (taking place in New York City on 20-21 March), Jo Maitland and I are going to host a Mapping Session on Data Marketplaces. What are they, what are they doing, why do they matter, and how does their future look? The session is intended to be highly interactive, so attendees should come armed with questions, opinions, data, and perspectives to share. It is free to registered Structure:Data attendees, but space is limited so sign up here if interested. About a year ago, I began a series of podcasts to explore the scale and nature of the opportunity facing a growing collection of so-called Data Markets. I spoke with the most visible champions of the idea such as Iceland’s DataMarket, Infochimps, Factual and Microsoft, as well as smaller companies like AggData with different perspectives to share. Some of the conclusions from the series appeared in a report (subscription required) for GigaOM Pro, and I’ve continued to watch the space, and to provide more detailed assessments of it to several clients. Companies like Factual and DataMarket continue to grow. Factual adds new data and products with regularity, whilst DataMarket has opened a US office and – with [...]

[Some of] what you need to know about the cloud for 2013

Thu, 03 Jan 2013 07:23:00 EST

Towards the end of last year, David Linthicum and I joined GigaOM’s Adam Lesser on a skype chat to take a look back at cloud successes and failures in 2012, and forward to cloud opportunities in 2013. GigaOM released the conversation as a podcast this morning. Amazon, Rackspace, Google, OpenStack, DropBox, and more get a look-in during a wide-ranging conversation about various ways in which the cloud is shifting consumer, small business, and enterprise attitudes and behaviours. Have a listen, and let us know if you agree with our analysis and predictions for the coming year. Image by Flickr user Rob Chandler.

Big Data as Core, Big Data as Context, and Big Data as Buzzword Bingo

Thu, 20 Dec 2012 10:06:05 EST

It’s neither particularly newsworthy nor insightful to suggest that ‘Big Data’ gets everywhere these days, but two recent items reminded me of the gulf between credible execution of a big data play and the more questionable tacking of the big data meme onto an otherwise useful product. Christmas is coming. Which means skating, and pantomimes (Captain Jack! And the Krankies!), and surprisingly expensive daughter shops, and pie with chicken and banana. But in amongst that lot, the weekend’s email and RSS brought news of an ideal solution to store, manage and archive big data and a service built specifically for Fortune 1000 enterprises who want to rapidly explore how big data technology can unlock revenue from their data. (both with my emphasis) Infochimps has been around since 2009, and I’ve been following them with interest. CTO and Co-Founder Flip Kromer and I recorded podcasts in 2009 and early 2012, and we continue to meet up from time to time. From humble beginnings, the company grew to become one of a handful of credible Data Market offerings, before moving on to contribute key pieces of code to projects such as VMware’s Serengeti. Earlier this year, Infochimps’ broader ambitions began to become public [...]

Hewlett Packard: A Tale of Many Clouds

Fri, 14 Dec 2012 07:00:00 EST

Hewlett Packard used its Discover event in Frankfurt last week to reassert the company’s cloud credentials. Public, private, hybrid; HP is painting pictures that encompass them all, whilst seeking to protect hardware revenues and reassure conservative executives at some of its largest and most profitable customers. But HP has been here before, making bold claims and telling people what they wanted to hear about an HP cloud upon which enterprises could depend. This time, will the company deliver? Earlier this year, satirical news site The Onion took a cruel but funny swipe at HP’s cloud pretensions. HP, the sketch suggested, had the answers, the technology, and a lot of cloud. The company has done — and continues to do — a lot right in this space, but it really did bring this derision upon itself. Mixed messaging, repeated announcements of amazing new cloud services that never quite saw the light of day, an endless stream of apparent strategy U-turns that must surely have left long-time HP executives as dizzy as those trying to understand their intentions? None of this helped HP. But now, Windows Azure is apparently behind us. PalmOS (or whatever it’s called these days) is no longer a glue to [...]

‘Autonomy Inside’ matters at Hewlett Packard

Mon, 10 Dec 2012 10:21:10 EST

The Hewlett Packard marketing machine was busy last week, assuring the world that the company’s £7.1bn ($11.7bn) acquisition of Autonomy still made sense despite an eye-watering financial write down and unseemly public squabbling with the Cambridge company’s former management. HP CEO Meg Whitman used her keynote at HP Discover in Frankfurt to assert that the technology giant was “100% committed to Autonomy’s technologies,” whilst almost everywhere we went in the Messe‘s halls we encountered Autonomy pixie dust spread liberally across HP’s portfolio of products and services. That Autonomy powers fraud prevention and e-discovery services is no surprise. Its apparently pivotal role in augmented reality magazine ads, fault diagnosis in laptops, and something elusive in the printer division was, perhaps, more of a stretch. And yet, it was here that HP staff got excited. It was here, too, that the benefit of tight integration within a hardware, software and services behemoth could make real sense. And yet, the beauty and logic of this match at times appeared just a little too perfect. Is HP and Autonomy really the match made in heaven that was implied? Do Autonomy and HP Vertica really fit together like peas in a pod? Are there no rough edges [...]

Dropbox, Google Drive, Apple iCloud, Microsoft SkyDrive; maybe they’re not apples after all?

Wed, 21 Nov 2012 09:21:15 EST

Cloud storage product Dropbox is one of those tools that users tend to rave about. It’s deceptively simple. It’s pretty reliable. The value proposition is immediately apparent. It has paid tiers of usage that bring additional storage but (like other freemium beacons such as Evernote) the free offering is rich enough to be compelling, engaging, and valuable. However, as Apple, Google and Microsoft start bundling very similar capabilities right into their latest operating systems, how can Dropbox (or any of its many peers) manage to keep attracting new customers? Maybe all the articles and blog posts that lump these products together and label them as ‘just’ alternative cloud storage solutions are missing the point? Maybe they’re addressing fundamentally different problems, and maybe that offers room for differentiation as the market becomes clearer. The late Steve Jobs once, famously, described Dropbox as ‘a feature, not a product.’ A feature for which he was apparently willing to part with as much as $800 million, but still just a feature. And, from Apple’s perspective, cloud storage is just a feature. It’s a feature that strengthens the Apple product ecosystem. It’s a feature that makes it that little bit harder to seriously consider buying [...]

My GigaOM week – report published, fireside chat videos, and more

Fri, 19 Oct 2012 07:49:39 EDT

It’s been a pretty GigaOM-focussed week. To begin the week, my GigaOM Pro report on Europe’s Helix Nebula cloud project was published. Then Tuesday and Wednesday were dominated by activities in and around GigaOM’s first conference on this side of the Atlantic; Structure Europe, held in Amsterdam. I moderated two sessions on the main stage on Tuesday, and worked with Kris Tuttle to guide conversation during a breakout workshop on Wednesday. The report, Helix Nebula and the Future of Europe’s Cloud, takes a look at the interesting combination of demand-side consumers of cloud (ESA, CERN and EMBL) with supply-side providers of infrastructure (CloudSigma and a host of others), in a project that seeks to raise and address the issues associated with procuring and consuming cloud services at scale from an ecosystem of providers. The report also explores some of the ways in which this collaboration serves as a useful blueprint for Europe’s wider cloud ambitions, as laid out in last month’s European Cloud Strategy document. My first conference session was a conversation with Ignacio Llorente of the Open Nebula project. Less well known, perhaps, than other open source clouds such as OpenStack, CloudStack, and Eucalyptus, Open Nebula is nevertheless building a steady [...]

My GigaOM Pro report on metered computing

Fri, 05 Oct 2012 16:39:33 EDT

I’ve just had another report published on GigaOM Pro. This one, Metered IT: the path to utility computing, is about the importance of defining common ways of describing and measuring computing resources. Until you can do that, it isn’t really feasible to compare the true cost of running jobs on different IT infrastructures. The piece was underwritten by 6fusion, a company with an interesting offering in the space. As usual, though, GigaOM retained full editorial control over the process of preparing the report. Image by Flickr user Micah Taylor

Tesco uses data for more than just loyalty cards

Mon, 01 Oct 2012 08:59:15 EDT

Tesco. Poster-child of the data-driven retail experience, thanks to its loyalty card and early work with DunnHumby. But Tom Hebbert didn’t talk about that. Instead, he focused on their supply chain work. Rough notes from the session follow. Big Data projects deliver huge returns at Tesco; improving promotions to ensure 30% fewer gaps on shelves, predicting the weather and behaviour to deliver £6million less food wastage in the summer, £50million less stock in warehouses, optimising store operations to give £30million less wastage. Tesco – 2,979 stores, 30,000+ products in a big store, 23 depots in the UK, 50million cases delivered into stores each week, 16million of which contain perishable foodstuffs. Weather A ten degree rise in temperature means 300% more barbeque meat. 45% more lettuce. 50% more coleslaw, 25% fewer sprouts. But weather is not straightforward, and it is very localised. It also has different effects dependent upon the day of the week, and the location. A city-centre store will see an uplift in sandwiches (to eat outside) on a warm weekday… and almost no effect at all on a warm weekend. And basic facts around temperature, sunshine etc don’t tell the whole story. A 16 degree sunny Saturday in [...]

Data Journalism at The Guardian

Mon, 01 Oct 2012 06:26:59 EDT

UK newspaper, The Guardian, has done some pioneering work to use data, and to engage readers in exploring data to share their own insights. The paper’s Simon Rogers and Google’s Kathryn Hurley shared some of the lessons at Strata this morning. Rough notes follow. Not going to talk about big projects like riots and Wikileaks and MP’s expenses… Going to talk about the day-to-day process of hacking around with data. Open data journalism – more than just Google spreadsheets. Much more of a two-way process than simply writing and disseminating stories. Numbers need context. Journalists need the skills to interpret, probe, and tell a data-backed story. First rule of what we do – find the key data behind a story and make it public. Guardian Datablog and Data Store used to push out data relevant to the main news stories of the week. Lots of data is available, but it’s locked up in a wide range of data sets. A lot of the Data team’s work is involved with pulling freely available data together in one place – making it comparable and useful. Get past raw numbers, and show how they have changed over time. Measures and units and groupings change, so [...]

George Dyson does Strata

Mon, 01 Oct 2012 05:31:07 EDT

George Dyson (no, nothing to do with vacuum cleaners) delivered the final session before coffee here at Strata in London this morning. Rapidly jotted notes follow… When he was invited to speak, he didn’t know what big data was. Made up his own definition… “when the human cost of making the decision of throwing something away became higher than the machine cost of continuing to store it.” That works… :-) 100 years since birth of Alan Turing. Died in 1954, at the dawn of this world. In 1953, there were 53kb of random access storage… globally. In 1959, Richard Feynman anticipated growth… “why can’t we write the contents of the Encyclopaedia Britannica on the head of a pin?” Now we can… Idea formalised (more or less) as Moore’s Law. Last week’s New York Times piece… alarmist view that data centres will consume all available power. It’s not that bad. Huge improvements on the way, says Dyson. Exchequer tallies “the first data centre?” The first time that money ‘existed in two places at the same time…’ [What about clay tablets from Mesopotamia, etc?] Thomas Hobbes, in 1656, anticipated importance of data… and binary. Also Leibnitz… “told us 400 years ago how [...]

O’Reilly’s Strata comes to Europe, with a very British opening

Mon, 01 Oct 2012 04:32:54 EDT

O’Reilly’s Big Data extravaganza, Strata, left its native U.S. for the first time this week, coming to London for two days of data; the big, the open, the structured, the unstructured, and the undecided. Whilst many of the companies and issues are the same, whether you’re in London, California or New York City, there are also plenty of ways in which local philosophy, approach, and mindset impact upon what’s possible – and what’s important. O’Reilly delivered this message loud and clear this morning, devoting the first two sessions of the opening morning to stories from the UK’s open data-enthusiastic government. Rather raw notes follow… First up, Liam Maxwell (@liammax) from the Cabinet Office. What is the government doing about data, open data, and the role of IT in all of that? We spend 1% of GDP on government IT systems. But we want to deliver efficiencies, and enable the UK economy to grow. Transparency is important; for example, local government must publish any expenditure over £500… originally, it was meant to be £5,000. £500 is better! Open Government Partnership – open government makes sense. It’s become an orthodoxy. Also keen on open source. But not because open source is necessarily best. [...]

The Next Big Thing: WeeData

Fri, 28 Sep 2012 08:00:00 EDT

‘Big Data’ has a problem, and that problem is its name. Dig deep into the big data ecosystem, or spend any time at all talking with its practitioners, and you should quickly start hitting the Vs. Initially Volume, Velocity and Variety, the Vs rapidly bred like rabbits. Now we have a plethora of new V-words, including Value, Veracity, and more. Every new presentation on big data, it seems, feels obligated to add a V to the pile. But by latching onto the ‘big’ part of the name, and reinforcing that with the ‘volume’ V, we become distracted and run the risk of rapidly missing the point. The implication from a whole industry is that size matters. Bigger is better. If you don’t collect everything, you’re woefully out of touch. And if you’re not counting in petas, exas, zettas or yottas, how on earth do you live with the shame?

A guest post on VentureBeat

Tue, 25 Sep 2012 16:11:12 EDT

As part of the work I’m doing to help assemble the programme for this year’s CloudBeat conference (you should come, you’ll love it), I had a guest post published on VentureBeat earlier today. The big vendor extravaganzas – the Dreamforces and OpenWorlds and I/Os of the world – wrap attendees in a warm and fuzzy blanket of free swag and free beer and free rock concerts. They might – just for a moment – succeed in persuading you that they have everything you could ever need. They do cloud. They know cloud. Some, unbelievably, will even sell you the cloud in a box! The real world is more complex, and more hybrid. Few companies, big or small, will buy everything they need from a single IT supplier. And that’s what the post is about. Yes, it’s a plug for CloudBeat (where we’ll be assembling some great stories of customer success)… but the underlying point is valid whether you consider coming along to CloudBeat or not. But if you do want to come to California at the end of November, then I’ll see you there… Image by Flickr user ‘Ardonik‘

Data Markets, revisited

Fri, 24 Aug 2012 12:03:45 EDT

Earlier this year, I conducted a series of podcasts with some of the leading lights in the Data Market business. We delved into the things that differentiated them from one another, and we searched for the areas of commonality that might provide some boundaries to the rather fluid concept of a market for a non-rival good like data. Through ten separate conversations, I had opportunities to talk with old friends and acquaintances, and to explore the ideas of people whom I had previously only admired from afar. Some of the insights from that process have now been written up in a report for GigaOM Pro; Data Markets: In search of new business models. As the blurb states, From information on U.S. census returns to the location of every Starbucks in Canada, the demand for data to support decision making is increasing. Fittingly, a number of new data markets have emerged in the past few years that provide access to this data. A wide range of companies exists in this space, and often there are more differences than similarities in the various products on offer, not to mention the many different financial models. This report describes the basics of a data [...]

Commentary: Amazon Remains the Cloud's Top Dog

Sun, 19 Aug 2012 20:30:00 EDT

In the competitive world of cloud-based computing infrastructure, Amazon remains top dog. It’s highly visible, its footprint is almost global, it incrementally adds features or cuts prices to keep competitors on their toes, and it generally manages to meet most people’s needs, most of the time. It may not always offer the lowest prices, or the best support, or the fastest processing, or the friendliest management console, but it consistently manages to meet customer demand with an offer that is more or less ‘good enough.’ It is a convenient choice, and it’s the standard against which its multitude of competitors will typically be measured. Amazon is Asda or Carrefour or WalMart to the competition’s Tesco, Marks & Spencer, Fortnum & Mason, Whole Foods, and neighbourhood deli. Amazon is not, of course, the only game in town. Far from it. Rackspace comes a close second by most measures, big names such as HP and Google are becoming increasingly serious contenders, and there is a healthy – and growing – crop of smaller providers that differentiate themselves by geography, by price, by customer domain, by specific features, and more. Indeed, I looked at some of those differentiations recently. But every one of the cloud infrastructure providers [...]

CloudBeat is Back

Tue, 14 Aug 2012 06:05:00 EDT

Ben Kepes and I had a load of fun last year, helping the team at VentureBeat put on their inaugural cloud computing event, CloudBeat. Clearly we did something right whilst having fun, as they’ve invited us back to reprise our content advising/ programme shaping role again this year. Right at the end of November, we’ll once again be doing what we can to assemble a stellar cast of cloud companies and their customers at the Sofitel in Redwood City, just south of San Francisco. As the blurb states, Unlike other cloud conferences, CloudBeat brings IT executives who use cloud technologies onto the same stage as the companies making those technologies. It’s a rare chance to learn what really works, who’s buying what, and where the industry is going. Ben and I both feel that there’s a real need to ensure that the stories of customer success (and failure) get heard, instead of just having the cloud companies tell us how wonderful they’re going to make the future. How do the adopters of cloud solutions have to change? What can they do that they couldn’t achieve with other approaches? Is it about saving money, or going green, or being agile, or [...]

Thinking about Open Data, with a little help from the Data Hub

Tue, 31 Jul 2012 12:45:29 EDT

Continuing to explore the adoption of explicit Open Data licenses, I’ve been having a trawl through some of the data in the Open Knowledge Foundation‘s Data Hub. I’m disappointed – but not surprised – by the extent to which widely applicable Open Data licenses are (not!) being applied. For those who are impatient or already aware of the background, feel free to skip straight to the results. For the rest of you, let me begin with a little background and an explicit description of my methodology. Background Open Data is, increasingly, recognised as being A Good Thing. Governments are releasing data, making them more accountable, (possibly) saving themselves money by avoiding the need to endlessly answer Freedom of Information requests, and providing the foundation upon which a whole new generation of websites and mobile apps are being built. Museums and Libraries are releasing data, increasing visibility of their collections and freeing these institutional collections from their decades-long self imposed exile in the ghetto of their own web sites. Scientists are beginning to release their data, making it far easier for their peers to engage in that fundamental principle of science; the reproduction of published results. Open Data is good, and useful, and valuable, and [...]

GigaOM Pro report on Hadoop and cluster management

Mon, 23 Jul 2012 08:57:29 EDT

My latest piece of work for GigaOM Pro just went live. Scaling Hadoop clusters: the role of cluster management is available to GigaOM Pro subscribers, and was underwritten by StackIQ. Thanks to everyone who took the time to speak with me during the preparation of this report. As the blurb describes, From Facebook to Johns Hopkins University, organizations are coping with the challenge of processing unprecedented volumes of data. It is possible to manually build, run and maintain a large cluster and to use it to run applications such as Hadoop. However, many of the processes involved are repetitive, time-consuming and error-prone. So IT managers (and companies like IBM and Dell) are increasingly turning to cluster-management solutions capable of automating a wide range of tasks associated with cluster creation, management and maintenance. This report provides an introduction to Hadoop and then turns to more-complicated matters like ensuring efficient infrastructure and exploring the role of cluster management. Also included is an analysis of different cluster-management tools from Rocks to Apachi Ambari and how to integrate them with Hadoop. Compulsory picture of an elephant as it’s a Hadoop story provided by Flickr user Brian Snelson. Related articles The unsexy side of big [...]

The Americans are Coming

Sat, 21 Jul 2012 13:09:26 EDT

This October, two great US events are making their first forays into Europe. O’Reilly‘s big data conference, Strata, reaches London on 1-2 October. Then GigaOM‘s cloud computing event, Structure, hits Amsterdam on 16-17 October. I’ve attended both in the States (see disclaimer), and look forward to seeing how each sets about fusing the best elements of a successful American event with local speakers, local issues, and local sensibilities. Good events don’t always cross the Atlantic (or, indeed, the Pacific, the Rockies, or the Channel!) very well, but I have high hopes for these two. I’ve never been particularly keen on the huge trade shows that fill Moscone or Olympia or RAI, or that take over large chunks of the Vegas Strip for a week. I’ve rarely been in a position where I’ve wanted (or needed) to buy or sell in that environment. More importantly, I have enough USB sticks, t-shirts, mugs and pens, and no longer use mouse mats, so what use is all the swag? No one seems to give away the large umbrella that I would actually like to receive, to replace a broken one. Smaller events, more focused upon contact and ideas and connections are far more appealing. [...]

Crunching the Numbers in Search of a Greener Cloud

Fri, 20 Jul 2012 07:45:00 EDT

Although sometimes portrayed as a big computer in the sky, the reality of cloud computing is far more mundane. Clouds run on physical hardware, located in data centres, connected to one another and to their customers via high speed networks. All of that hardware must be powered and cooled, and all of those offices must be lit. Whilst many data centre operators continue to make welcome strides toward increasing the efficiency of their buildings, machines and processes, these advances remain a drop in the ocean next to the environmental implications of choices made about power source. With access to good information, might it be possible for users of the cloud to make choices that save themselves money, whilst at the same time saving (a bit of) the planet? Greenpeace has consistently drawn attention to the importance of energy choices in evaluating the environmental credentials of data centres, with 2011′s How Dirty Is Your Data? report continuing to polarise arguments after more than a year. The most efficient modern data centres deploy an impressive arsenal of tricks to save energy (and therefore money), and to burnish their green credentials. They use the most efficient modern processors, heat offices with waste server [...]

Survey: How open is your data?

Wed, 11 Jul 2012 06:42:09 EDT

Back in 2006 as we rolled out the first public draft of the Talis Community Licence, the world of data licensing seemed a simple place. Today, the Open Knowledge Foundation‘s Data Hub contains 3,888 data sets, many of which are explicitly licensed with respect to the Open Definition. But many are still not explicitly licensed. Over at the UK Government, there are 8,619 data sets today, and an assertion that “in general, the data is licensed under the Open Government License.” Too much still isn’t, of course, but they’re getting there. And then there are the many, many more data sets out on the web, not registered with repositories like the Data Hub or data.gov.uk at all. More than four years on, how are we really doing? As a scoping exercise for a larger project that I might be undertaking, I’d be really grateful if you could take a moment to fill in this brief survey [which will open in a new window or tab]. It simply sets out to assess the relative proportions of data that are not openly licensed, that are implicitly open, explicitly open with some home-grown statement, or explicitly open and using a recognised data license like CC0 or one of the Open [...]

A new host, a new design

Tue, 10 Jul 2012 09:31:31 EDT

As visitors to the Cloud of Data site have probably spotted, it’s got a new design today. Under the hood it’s also moved to a new hosting company, and the team at ZippyKid have been very helpful during the transition. We’re still on WordPress, and everything appears to be working as it should, but I’ll be keeping an eye on things as DNS records and the like settle down. If you spot any broken bits, silly CCS errors, or other oddities, please do let me know. I’ll gradually be refreshing a lot of the background content on the site, some of which hasn’t changed since the original site was set up back in 2008. I also have a pipeline of blog posts that I’ll be pushing out over the next few weeks as other things quieten down a bit. I’m quite pleased with the new site (although some CSS oddities caused a lot of head-scratching), and hope you like it too. Comments, criticisms, observations all welcome as usual.

Thinking about Data Gravity

Mon, 02 Jul 2012 10:31:28 EDT

Dave McCrory introduced his idea of Data Gravity with a blog post back in 2010. The core idea was — and is — interesting, and got some traction from sites like ReadWriteWeb, ZDNet and GigaOM. More recently, Data Gravity featured in this year’s EMC World keynote. But beyond the observation that large or valuable agglomerations of data [...]

Google’s Knowledge Graph Bringing Semantics to the Masses

Mon, 21 May 2012 08:00:00 EDT

With Facebook’s IPO upon us, the timing of Google’s latest press blitz should probably be regarded with a healthy dose of suspicion, but the unveiling of the Knowledge Graph is an important step in Google’s journey — and a reaffirmation of values diluted by recent dalliances in social networking. Writing for The Atlantic, Alexis [...]

CloudCamp reaches Leeds on 14 June

Wed, 16 May 2012 08:10:21 EDT

The global CloudCamp movement continues to grow, with events over the next few weeks in Denmark, Germany, Nigeria, Ghana, Kenya, and across the United States. And now, I’m very pleased to announce that the English city of Leeds is joining the party. CloudCamp events have been taking place in the UK for years, and the [...]

Independent Cloud Providers: Introducing Brightbox

Mon, 23 Apr 2012 03:00:00 EDT

Amazon’s globe-encircling cloud infrastructure is compelling to many. From Virginia to California, from Ireland to Singapore, and from Japan to Brazil; wherever you find yourself there’s a local instance of the same familiar set of services. And, in all likelihood, Australia will soon be added to the list. For those primarily interested in just serving both Europe [...]

Hubris and the Data Scientist

Fri, 20 Apr 2012 02:00:00 EDT

ReadWriteWeb‘s Joe Brockmeier captures a recurring issue from last week’s O’Reilly Strata conference, asking “Can Big Data replace domain expertise?” According to Brockmeier, the audience (of data scientists) apparently narrowly agreed that their arsenal of tools and algorithms trumped the knowledge and experience of the meteorologists, financiers, and retailers to whose domains data scientists are increasingly [...]

Surely the computer should do that?

Thu, 12 Apr 2012 12:49:54 EDT

We have become accustomed to the simple yet all-powerful search box. ‘Advanced’ search options and arcane query syntaxes have largely been replaced by the learned behaviour of throwing some words at Google*, ignoring the sponsored links, and (usually) finding what we want somewhere in the first 5-10 proper results. A Google search is certainly impressive [...]

Solar power in the data centre – solution or window dressing?

Sun, 18 Mar 2012 14:55:16 EDT

Most of us recognise that the Earth is warming and that — despite our planet’s temperatures having dramatically risen and fallen before — we humans must accept some measure of responsibility for the current changes. Already consuming at least 1.1-1.5% of global power, and only forecast to grow ever-more rapacious, the data centres that power our information [...]

Shiny New Cloud Standards Effort From OASIS

Mon, 05 Mar 2012 03:00:00 EST

Open standards body OASIS recently unveiled yet another shiny new standards effort. The OASIS Topology and Orchestration Specification for Cloud Applications (TOSCA) Technical Committee hopes to make it “easier to deploy cloud applications without vendor lock-in,” and to support moving from one cloud to another.

Data Market Chat: Microsoft’s Windows Azure Marketplace

Sat, 03 Mar 2012 08:30:00 EST

As CEO Steve Ballmer has noted more than once, Microsoft’s future plans see the company going “all in” with the cloud. The company’s cloud play, Azure, offers the capabilities that we might expect from a cloud, and includes infrastructure such as virtual machines and storage as well as the capability to host and run software.

Data Market Chat: Rufus Pollock and Irina Bolychevsky discuss the Open Knowledge Foundation and CKAN

Thu, 01 Mar 2012 12:00:49 EST

The Open Knowledge Foundation (OKFN) promotes the creation, dissemination and use of ‘open knowledge.’ As part of this activity OKFN developed a data repository called CKAN, and has seen this become increasingly important to a range of data dissemination activities such as data.gov.uk and publicdata.eu. In this podcast I talk with OKFN Director Rufus Pollock [...]

Data Market Chat: Leigh Dodds Discusses Kasabi

Thu, 01 Mar 2012 04:00:00 EST

Kasabi sees its role very much as an enabler of aggregation. Rather than focusing, as some data markets do, on simply providing access to data sets, Kasabi is betting on the power of being able to combine and recombine data in compelling new ways.

Data Market Chat: Shion Deysarkar discusses Datafiniti

Wed, 31 Dec 1969 19:00:00 EST

Houston-based data startup Datafiniti takes a different attitude to gathering the data that it sells. Rather than negotiate complex deals with the owners of data sets, the company relies upon techniques borrowed from Grid Computing in order to crawl the public web and harvest interesting data along the way. CEO Shion Deysarkar talks about the [...]

Data Market Chat: Shion Deysarkar discusses Datafiniti

Wed, 31 Dec 1969 19:00:00 EST

Data Market Chat: Shion Deysarkar discusses Datafiniti

Wed, 31 Dec 1969 19:00:00 EST