Dark data, and how frustrating it is that we can’t see the forest from the trees

Posted by Fred Oliveira on January 4, 2012 | Comments (0)

It sucks that this is 2012 and that there’s no good way to organize and collect the things we do, share and like (online or otherwise). In the last hour I added things to instapaper, pinboard, gimmebar, and a notebook I carry. I took photos on my iPhone, that I will one day move to a computer, then an external hard-drive, then eventually forget about. I like collecting things, because it gives me the sense that one day, someday, I’ll be able to look back and connect the dots. There’s meaning in the things that I’ve collected, but that meaning is lost without the tool to capture it.

There’s certainly meaning in the over 70.000 songs I played on Last.fm, the things I tweeted, the images I saved on FFFFOUND and Gimmebar, the books I’ve read, the bookmarks I added to Pinboard (and before that, Delicious). I know the meaning is in there, lurking beneath the surface of a seemingly infinite number of services; it’s a huge graph of things I care about, and the invisible connections between them. It is huge, I am certain, but I still can’t see it.

I’m calling this “dark data” because much like dark matter, we know it exists even if we can’t grasp it. Dark matter is 83% of the universe and we can’t see it. Dark data is everything we do and we still can’t see it.

I’ve been thinking about this problem of dark data for a while, sketching solutions, putting some into code (certainly not enough code). But I’m not there yet, and sadly, services like Gimme Bar, Bundlr, Evernote are not there yet either. There’s a ton of potential in making meaning out of all our photos, our tweets, our listened songs and our shared articles. Building tools to see the forest from the trees – now there’s something worth doing.

Looking back, moving forward

Posted by Fred Oliveira on December 27, 2011 | Comments (1)

The end of each year is always a good time to look back in time. Yesterday, I did. Unable to sleep, I picked up one of my A Book Apart books (Ethan Marcotte’s Responsive Web Design, for the curious) and suddenly remembered the old A List Apart design. If you’ve been in the web industry for a while, you probably remember it too – yellow, orange, awesome.

It’s been 13 years since Zeldman‘s own ALA became a website – it was back in 98 and I was 16 years old, surfing on a crap modem, collecting images from random web pages. My favorite website at the time was NBA.com, for some reason. I had been coding for a few years already, albeit never for the web, influenced by what was in hindsight the single most important contribution to my career: a Spectrum ZX and a BASIC manual, both given to me by my parents when I was 6. I remember the moment perfectly – where I was, what my parents were wearing. Somehow that memory remains while others have faded. I remember one of my first pieces of code too: it asked for my name, said hello, and kept my phone numbers. I remember writing a simple text-based game for my sister, which she remembers well too. Again – some memories remain, while others fade.

Looking back lets you connect the dots, and that I have. Telling people how I got to where I am today is then relatively easy – I simply look back, and connect the dots I remember. Looking forward, however, is a futile exercise. I can speak of my passions and where they might lead me, but I can never be certain of anything. I’m not even certain I’ll be around tomorrow – although I suspect I will be. So I will keep suspecting I’ll be around – and while I am, I’ll try to make a dent in things. The universe, sure, but the little things too. Because when I look back and connect the dots, it was the small dents in things, the pokes in some direction that got me here. Like that Spectrum that hooked me on computers, ALA that got me thinking about the web, and the many things (and people) that followed, I hope to dent and poke (people and things) going forward.

Looking forward isn’t possible, but moving forward most certainly is. So I’ll keep looking back, and moving forward. Hopefully, making a dent in things as I go.

Buffy, social, and Google

Posted by Fred Oliveira on November 22, 2011 | Comments (0)

AllthingsD reports that Facebook is working on a mobile phone, internally called “Buffy” – unsurprising news anyone who has been following the industry. There are, however, a number of interesting things lurking between the lines that are important to think about at this point.

Facebook is clearly taking advantage of their lead when it comes to social. Despite the surprising growth numbers on Google+ (Vic Gundotra and Sergey Brin talked a little bit about how positive they were at Web 2.0 Expo), they still have a lot of catching up to do. Google’s in a tough spot. Here’s why:

Even though they’re the clear market lead in search and online advertising, Google is losing what’s possibly the most important race they’re competing in. Social networking have become a layer on top of which all the services we use stand. From the movies we watch, to the music we listen to, to the books we read and the websites we visit, chances are high that these choices emerge out of online social interactions. Now if you think about it, Google doesn’t control any major platform supporting this kind of social interaction – with the possible exception of Youtube.

I can’t think of the last acquisition Google made in the last few years that tipped the social scale in their favor. Blogger’s acquisition is now long in the past, and since then Google has sadly been playing catch-up. By not acquiring Twitter a couple of years ago and instead focusing on products like Wave and Buzz (two social initiatives out of Mountain View that ended up failing), Google became a 3rd spot player in what might be the most critical market in IT going forward.

My guess is that we’re going to see some heavy priority changes out of Google (the social bonus suggested by Larry Page when he took his new job as CEO is a key indicator of that), and possibly more acquisitions in the social space.

So let me circle back to Buffy. Facebook is large enough now that they can focus on taking on a new market (much like Apple did with music, then phones, then tablets), and they’re comfortably sitting in the lead as a social network. Until now, however, Facebook had to rely on third party platforms (like iOS and Android) to bring their non-desktop experience to their users. Not after their phone comes out. They’re now taking a lesson from players like Apple and cutting the middle man altogether – and with a bit of irony, they’re forking Android in order to build their new mobile OS.

This whole thing makes for an interesting imaginary Venn diagram. Facebook, Twitter and Google are in Social; Apple, Amazon, Google and Facebook are in mobile; None of these companies is naive enough to only focus on their old core business. The new core business for all of them is having access to how we communicate, how we access data, and more importantly, how we share that data back with others. Social, search and mobile used to all be different things – but that’s certainly not the case anymore.

All design and content © Fred Oliveira 2007-2012, unless otherwise specified.