m.e.driscoll: data utopian

eight golden rules of interface design

Mon, 14 May 2012 15:34:00 -0400

As we dedicate an increasing fraction of our time interacting with software — from airport check-in terminals and parking meters, to desktop and mobile applications — digital interface design is becoming as important as physical architecture in improving our experience of the world.

Here are Professor Ben Schneiderman’s Eight Golden rules for optimally designing that experience (drawn from his classic text, Designing the User Interface):

1 Strive for consistency.
Consistent sequences of actions should be required in similar situations; identical terminology should be used in prompts, menus, and help screens; and consistent commands should be employed throughout.

2 Enable frequent users to use shortcuts.
As the frequency of use increases, so do the user’s desires to reduce the number of interactions and to increase the pace of interaction. Abbreviations, function keys, hidden commands, and macro facilities are very helpful to an expert user.

3 Offer informative feedback.
For every operator action, there should be some system feedback. For frequent and minor actions, the response can be modest, while for infrequent and major actions, the response should be more substantial.

4 Design dialog to yield closure.
Sequences of actions should be organized into groups with a beginning, middle, and end. The informative feedback at the completion of a group of actions gives the operators the satisfaction of accomplishment, a sense of relief, the signal to drop contingency plans and options from their minds, and an indication that the way is clear to prepare for the next group of actions.

5 Offer simple error handling.
As much as possible, design the system so the user cannot make a serious error. If an error is made, the system should be able to detect the error and offer simple, comprehensible mechanisms for handling the error.

6 Permit easy reversal of actions.
This feature relieves anxiety, since the user knows that errors can be undone; it thus encourages exploration of unfamiliar options. The units of reversibility may be a single action, a data entry, or a complete group of actions.

7 Support internal locus of control.
Experienced operators strongly desire the sense that they are in charge of the system and that the system responds to their actions. Design the system to make users the initiators of actions rather than the responders.

8 Reduce short-term memory load.
The limitation of human information processing in short-term memory requires that displays be kept simple, multiple page displays be consolidated, window-motion frequency be reduced, and sufficient training time be allotted for codes, mnemonics, and sequences of actions.

the rise of the technical VC

Sat, 05 May 2012 22:12:00 -0400

Silicon Valley’s first big bang of innovation occurred in 1957, when eight engineers left Shockley Transistor to form FairChild Semiconductor. Back then, the idea of engineers being entrusted as founders of a business was heretical. Forty-one firms were asked to invest, but “none of them were interested”, according to Arthur Rock.

The idea that engineers without MBAs can be successful founders has changed, but what about engineers acting as investors? In my experience, the majority of investment professionals on Sand Hill road are still non-technical.

But that is changing, in two ways.

First, several young prominent venture capitalists who have technical degrees are rising to the top of their profession. Folks such as Kevin Efrusy (MSEE and BSEE from Stanford) and Jeremy Levine (CS degree from Duke) are ranked #9 and #10, respectively, on this year’s Midas List of top investors. And at #1 this year is Jim Breyer, who earned a CS degree from Stanford, and having just turned 50 is still youthful by VC standards.

Secondly, as technical founders have made their fortunes, many of them have joined the investing class. Marc Andreessen and Reid Hoffman, two successful technical founders turned investors, were the second and third top investors in 2012.

And the Midas List doesn’t cover the funding arena where the influence of technical founders is greatest: angel investing. Many of the world’s most successful non-professional investors — Jeff Bezos, Max Levchin, Andy Bechtolsheim, Paul Graham, Bill Joy, and Marc Benioff — have, with their spare change and spare time, outperformed entire funds.

Silicon Valley’s venture capital community is undergoing the same “revenge of the nerds” phenomenon that its businesses underwent in the 1960s and 70s. Technical founders are launching companies, earning returns, and then spotting new start-ups to invest in — increasingly without needing surrogates carrying MBAs.

Or perhaps more accurately, whereas the technical class was previously seen as serving the business class, now it is the business class that serves the technical class. Mark Zuckerberg’s having a controlling share of Facebook is testament to this new reality.

The rise of the technical VC is part of a larger macro-trend that Marc Andreessen cogently captured in five words: software is eating the world.

One vertical after another — from media, travel, and (soon I hope) health care and education — is being transformed by information technology. Those who conceive, develop, and understand software are the new masters of the universe. And everyone else — lawyers, bankers, janitors — are their servants.

CEOs and VCs are learning to code not because their curiosity inspires it, but because their careers depend on it.

dna dating

Mon, 09 Apr 2012 00:07:00 -0400

A recent start-up, Yoke.me, is attempting to build a better dating engine using Big Data and algorithms. But what mix of data could best be used to algorithmically identify an optimal mate? Photos, favorite albums, and religious beliefs are a start.

But how about DNA?

A couple of years ago at SciFoo, Toby Segaran, Meredith Carpenter, and I brainstormed about creating a start-up that would do just this. We dubbed it GeneHarmony.

Here’s how it would work: to become a member, you submit a saliva sample to our genomics facility, which sequences all of your genetic quirks (since most of us share DNA which is 99.6% similar, we need only sequence the differences).

Once sequenced, your genome would be scanned against all other members, with a focus on genes that are known to be predictive of mate compatibility, and return a rank-ordered list of potential dates.

The principal of “opposites attract” is mirrored at the DNA level. Studies show that individuals who are genetically dissimilar are significantly more likely to marry (the inverse of “why you shouldn’t marry your cousin.”)

So much of mating is an elaborate system to uncover genetic signals. Many factors which are considered attractive — facial symmetry, body shape, intelligence, body odor — are ways in which humans tell suitors “I have good genes.” DNA dating could cut through these perceptual inefficiencies and get right to the genetic point.

Even better, members’ experiences could be tracked and fed back into the genetic database to create better dating models. One could even tune the parameters depending on the kind of relationship sought: are you a 22 year-old thrill-seeker looking for fun, or an aging bachelor seeking marriage and stability?

Of course, the privacy issues raised by such a service are massive. What if the site was used to settle a paternity lawsuit? Or used to target advertisements? Facebook’s privacy issues appear trivial by comparison.

And yet, for most of us, selecting a partner is the most consequential decision of our lives. Why shouldn’t we leverage all of the science and technology we have to improve that choice?

the data science debate: domain expertise or machine learning?

Mon, 05 Mar 2012 03:50:00 -0500

(Photo credit: O’Reilly Radar - See Link to Full Video)

This past Tuesday evening at Strata I moderated an Oxford-Style debate between six of the top data scientists in Silicon Valley and beyond. The motion debated was:

“In data science, domain expertise is more important than machine learning skill.”

The topic emerged from conversations over dinner the previous night, with Kaggle’s Jeremy Howard, LinkedIn’s Monica Rogati, and some pre-debate musings of Google’s Hal Varian.

To constrain the question, we added an additional clarification: which of these would you favor more in hiring your company’s first data scientist?

Arguing in favor of the motion (e.g. favoring domain expertise) were:

Drew Conway, Ph.D. Candidate at NYU, Data Scientist at IA Ventures
DJ Patil, Data Scientist in Residence at Greylock Partners
Amy Heineike, Director of Mathematics at Quid

Weighing in against the motion (e.g. favoring machine learning skills) were:

Pete Warden, CTO of JetPac
Pete Skomoroch, Principal Data Scientist at LinkedIn
Toby Segaran, Author of Collective Intelligence and Google Engineer

When the Strata audience was initially polled, the vote was 53 to 40 in favor of domain expertise. Then the debate began with comments from the audience.

The Audience: s/MachineLearning/DomainExpertise is Easy

We heard from Daniel Tunkelang, who argued in favor of domain expertise, stating that it was easier to learn statistics and machine learning than to acquire a lifetime of expertise and intuition (perhaps it comes easy to Dr. Tunkelang, but I’m not sure how many who have attempted to consume the Elements of Statistical Learning on their own would agree).

Claudia Perlich, a three-time winner of the KDD Nuggets competition, stood up and shared how she had won contests in domains as varied as “breast cancer, movie prediction, and sales performance – and I can tell you I knew next to nothing about those things when I started.”

The panelists were then asked to weigh in with their thoughts.

The Panelists: Our Opponents Have Made Our Points for Us

Drew Conway, whose popular Data Science Venn Diagram includes “substantive expertise” as one of its components (and truth be told, “math & statistics knowledge”) advocated that asking good questions is the most critical element in a data science project. And the ability to ask good questions requires domain understanding.

Toby Segaran relayed a story about work I had done using social network analysis for modeling telco customer churn. He went on to say that, “Mike, a domain expert in almost nothing, actually outperformed the domain experts.” (ed. note: Thanks for the backhanded compliment, Toby :) ).

DJ Patil read from the original LinkedIn Data Science job posting, arguing that machine learning skills were not even mentioned. Rather they were seeking those who had curiosity and the ability to rapidly acquire domain expertise in the area of social network analysis. He cited their hire of a theoretical physicist from Stanford, Jonathan Goldman — who did the initial groundbreaking work on the People You May Know algorithm — as evidence that machine learning skills were not important.

Pete Skomoroch fired back that “since machine learning and physics are both just mathematics” that Jonathan was actually just a machine learning expert by another name. Those skills, said Skomoroch, helped him tackle and ultimately succeed in a domain in which he had little prior expertise.

Pete Warden, arguing for machine learning skills, cited his own experience at JetPac, his new travel site, where identifying high quality user photos was a high priority. They hosted a competition on Kaggle, the machine learning contest platform, and in three weeks had built a quality ranking algorithm for just $5,000.

Amy Heineike then retorted that Pete Warden had actually made the case against himself. In outsourcing their machine learning, she claimed, they underscored the importance of the one thing they could not outsource: their own domain expertise.

Toby Segaran agreed that company founders have excellent domain expertise: that is why they started their companies. But when hiring a first data scientist, they need to hire for what they don’t have: machine learning skills. (Zing!)

Pete Skomoroch ended the debate with a rhetorical question, asking the audience to consider the most successful companies in recent years: was human intuition or was it analytics driving them?

The Verdict: Let Us All Now Hail Our Machine Learning Overlords

In the end, the audience was polled again, and the results were tabulated in parallel by the panel (using what I like to call ManReduce), the verdict was: 52 for domain expertise, 55 for machine learning.

Like any good debate topic, there is merit on both sides of the domain expertise versus machine learning proposition. As Hal Varian said when we asked him before the panel: “it depends on the structure of the problem.” And in fairness to the debate panelists, they did not choose their positions: we assigned teams fifteen minutes before we went on stage.

One of the conclusions reached was that, when a problem is well-structured (or to Drew Conway’s point, when a good question is posed), it is much easier for machine learning to succeed. Kaggle’s strength as a contest platform is that domain experts have already framed the problem: they choose the features of the data to use (feature engineering or “feature creation”, as Monica Rogati calls it) as well as the criteria for success. This is the first, hardest step in any data science project. After this, machine learners can step in and develop the best algorithms for classifying and predicting new data (or, less usefully, explaining old data).

Thus who you decide to hire as your first data scientist — a domain expert or a machine learner — might be as simple as this: could you currently prepare your data for a Kaggle competition? If so, then hire a machine learner. If not, hire a data scientist who has the domain expertise and the data hacking skills to get you there.

(Thanks to O’Reilly Media, and Strata organizers Edd Dumbill and Alistair Croll — who suggested the Oxford Debate format — for hosting a terrific conference).

start-ups belong in cities

Thu, 23 Feb 2012 13:21:00 -0500

Last Saturday, I woke up and walked down to my favorite coffee shop in San Francisco, SightGlass coffee in SoMa.

I met up with a couple of entrepreneurs pitching an amazing idea, and while ordering some mind-buzzingly-good drip coffee, ran into a mentor of mine.

I write this because, while these interactions could have happened in the suburbs of Silicon Valley — whether the Coupa Cafe in Palo Alto or Red Rock in Mountain View — they are quintessentially enabled by four qualities of a city like San Francisco:

neighborhoods that mix commerce and living, that “serve more than one primary function”
blocks that are walkable, short and broken up with alleyways and side streets
buildings which are a diversity of the old and new, luxury and low-rent
people are prevalent and sufficiently concentrated

These four qualities enable the unique vibrancy of urban neighborhoods, and were laid out by Jane Jacobs in her magnum opus “The Death and Life of Great American Cities.”

I know that, for historical reasons technology start-ups began in Silicon Valley. But there is something tragic about watching 22-year-old software engineers waiting on city corners for buses to take them to work in the suburbs.

Especially when San Francisco is undergoing a renaissance of technology firms, driven by a few forces:

anchor firms like Twitter, Splunk, DropBox, Zynga, and Square
flourishing of start-up neighborhoods like SoMa, and now DogPatch
early stage VC firms with a strong SF presence, like True Ventures and OATV

To that end, I’m a huge supporter of the sf.citi initiative which is helping strengthen the community of hackers, entrepreneurs, and firms who recognize the unique advantages that a city provides.

To the aspiring young engineers thinking of coming West, I say: don’t settle for a bagels and Wi-Fi ride to an office park or even a campus. Come to a Great American City, we have amazing start-ups for you to join.

To the entrepreneurs and their investors: curse the overpriced rents in San Francisco but recognize that efficient markets sometimes express a valid point. Where there is high value there is high cost, and as Richard Florida has observed, the world’s brightest and most talented people flock to cities. So invest in them, their start-ups and the cities — San Francisco, New York, Chicago, London, Beijing — where they want to live.

ETL: the coal mining of the information age

Sun, 11 Dec 2011 15:22:00 -0500

“If I were starting a NoSQL-in-the-enterprise startup, I would focus on ETL. ETL is a mess, and is a precursor for any fancy uses of data.” - @jaykreps

“@jaykreps ETL is the coal mining of the information age: dirty, important work that fuels the economy.” - @peteskomoroch

One of the largest obstacles facing companies who seek to derive value from data isn’t data’s size. It’s data’s dirtiness.

It’s been said before: 80% of the effort that goes into a data science project is extracting, transforming, and loading (ETL’ing) data into a system where it can be analyzed.

This challenge is not simply a consequence of poorly structured data: free form text records are now mostly rare. Yet there remains bewildering variety within well-structured, regular data.

Take the basic dimension of time, an attribute that nearly every data set contains. A date can be expressed as POSIX or ISO8601 strings or a Unix epoch integer, among myriad other forms:

Sat Dec 10 10:37:13 PST
2011-12-11T18:37:13.0+0000
1323599850

And dates are just the beginning. There are country codes, currency symbols, geospatial coordinates, and language indicators. Beyond the data itself, there how it is delimited and encoded (including XML, the clamshell plastic packaging of data formats).

Data platform businesses create value by reducing the friction of data flow among participants. They do this with standards. The financial services industry, the most mature of data verticals, has defined symbologies for equities and other tradeable instruments. Consumer goods have UPC barcodes. Governments have national postal codes.

The manufacturing world has long appreciated the value of interchangeable parts, in lowering the costs of creating everything from electronics to airplanes. Historically, standards arise in one of two ways: through the de jure recommendation of a consortium, or through the de facto adoption of a market leader’s schema.

As the industrial revolution of data continues to unfold, we need data platforms and standards’ bodies to facilitate “interchangeable data”. These will accelerate the growth of a new breed of data-driven applications and services. Clean coal mining may be a fantasy, but clean data mining may yet be possible.

why everyone should be a medical data donor

Sun, 11 Dec 2011 03:10:00 -0500

What happens to your medical records when you die? Gil Elbaz thinks you ought to donate them to science, a thought he shared with a technology audience this past week.

It’s a fascinating idea. But why wait until you’re dead? In the age of the quantified self, why shouldn’t you be able to give your DNA sequence, your diet, and your disease diagnoses to science while you’re alive? Unlike your organs, you can donate your data away and yet still keep it.

We have companies collecting vast swaths of data about our buying, browsing, and clicking habits to sell us more stuff. But when it comes to understanding what behaviors keep us healthy, it’s a rocky landscape of HIPAA-regulated, technologically-challenged health insurers and providers. We collect so much data about what makes us click, yet so little about makes us tick.

There are pockets of hope. Sites such as PatientsLikeMe — which as this writing has 122,640 patients and over a thousand conditions — and Ginger.io are green sprouts in a bottom-up, democratizing data movement for health.

Nearly eight out of ten people on the planet earth now own a mobile phone. These phones send so-called “heartbeat” data to cell towers every few seconds. Imagine if, instead, we had the true heartbeat data of the humans carrying those phones? A simple cardiac signal can betray a host of health issues, from stress and aging to a warning of impending stroke or heart attack.

I know that I’m not alone in being willing to give my data to medical science. If the Fitbit or Jawbone UP had a checkbox that read “donate my data”, and the receiving institution was a trusted one, it could be the beginning of a valuable data bank. If the Red Cross can convince us to stick needles in our arms to give blood, certainly we can endure bracelets on our wrists to give data.

lies, damned lies, and social media statistics

Sun, 20 Nov 2011 15:54:00 -0500

Social media statistics — shares, retweets, and likes — reflect content’s value the way a funhouse mirror reflects one’s looks: grotesquely. As the web lines its halls with social mirrors, these distortions are influencing the content we create and consume.

One need look no further than the headlines at Hacker News for a gallery of the grotesque: ”N Reasons…”, “Why X is Wrong”, “Free Y”, and “How Z.. Cancer”. Many of these stories are explicitly crafted to achieve fifteen seconds of fame.

I plead guilty of this seduction — with @jkottke telling me off as proof — because it’s tempting to believe that metrics are an honest measure of value. They’re not.

Social Media Statistics are Biased

Hacker News readers are not a representative audience. Because of the frenzied frequency with which they flood the voting booths of cyberspace, their influence is outsized — and perversely enough, in inverse proportion to their attention spans.

We need a balance against these biases. A retweet from @timoreilly means more than one from @lolz69. Klout has attempted, with some ignominy, to measure online influence. If we weighted retweet counts by influence, we might have a better measure of an article’s impact.

Time matters too. All content is a zero until someone reacts, so we need to gauge the speed of +1s or shares, not just the total.

And positive feedback loops are everywhere. We end up reading and sharing the same few dozen articles every day, not because these are always the most valuable, but because once they’ve bubbled up into the meme pool, they get recirculated and amplified.

Be a First Follower

The strongest signal of quality should be the content itself, not its number of shares or comments. If you keep an open mind, you’ll encounter that joy of discovery once so integral to the web. Lovely gems still lurk out there.

Being the first follower takes a smidgeon of bravery. So ignore what other people think and share something no one else has. You’ll be a democratizing force.

Connect with People, Don’t Collect Them

Few of us share our ideas, photographs, and experiences online solely to collect followers. We do so to convince, to delight, to connect with people.

If you’re a creator, never confuse numbers with the value of your creative output. Resist the urge to chase some earlier success. If you create something of lasting value, which has staying power after the initial spasms of interest have passed, you will engage with your audience in a way that few metrics reveal.

Blogging to boost your follower count is like launching a start-up to build your bank balance: it rarely works. Instead, focus passionately on creating value, and the rest will come.

what to feed the mythical machine learning beast?

Sat, 19 Nov 2011 18:52:00 -0500

One of the holy grails of machine learning is the creation of a system that can “read the web” and learn from it, as Isaac Newton read Euclid’s Elements and taught himself geometry.

Imagine a mythical beast that could speed-read one-hundred million pages per second, consuming every Wikipedia entry, every scientific article on arxiv.org, every out-of-copyright scanned book, and beyond just indexing that information, could actually reason with it.

Building an intelligent machine isn’t ~~hard~~ impossible. It’s building a learning machine, one that mirrors the magic by which a teenager learns to drive a car, play chess, or do calculus in a period of a few dozen hours — that’s the magic that we haven’t yet figured out.

But I wonder if some of our challenges in creating this mythical learning machine lie with what we’re trying to feed the beast. After all, the web of documents was written for human consumption. Natural language is a lossy compression algorithm; it maps the massive varieties of our experiences into semantic text. A high-frequency sensory stream of sights, sounds, and experiences gets hashed into “cold sidewalks are slippery.”

To that end, if we want machines to reason about our world, let’s stop giving them our digested cud of content. Let’s provide them direct experience, via the sensor streams that our instrumented planet is emitting via weather stations, transit networks, electrical grids, smart phones, fitbits, and GPS devices. With that data, machines might begin to intuit relationships between weather and sidewalk slips — in forms that are beyond our own human minds to comprehend.

It’s data, not documents, that the mythical machine learning beast will eat.

"We spend more time working than we do on almost any other activity in our lives. People want all..."

Sat, 12 Nov 2011 13:50:13 -0500

““We spend more time working than we do on almost any other activity in our lives. People want all that time to mean something.””

- Laslo Bock, “Passion, Not Perks”

beyond hadoop: fast queries from big data

Sat, 12 Nov 2011 13:28:00 -0500

There’s an unspoken truth lurking behind the scourge of Big Data and the heralding of Hadoop as its savior:

While Hadoop shines as a processing platform, it is awkward as a query tool.

Hive was developed by the folks at Facebook in 2008, as a means of providing an easy-to-use, SQL-like query language that would compile to MapReduce code. A year later, Hive was responsible for 95% of the Hadoop jobs run on Facebook’s servers. This is consistent with another observation made by Cloudera’s Jeff Hammerbacher: when Hive is installed on a client’s Hadoop cluster, its overall usage increases tenfold.

That data-heavy businesses can achieve visibility into the terabytes of logs that they generate is, at a primary level, a major step forward. Before the Hadoop era, this was difficult to impossible without a major engineering investment. Thus Hadoop has solved the challenge of economically processing data at scale. Hive has solved the challenge of hand-writing Hadoop queries.

But there remains a painful challenge that Hive and Hadoop does not solve for: speed.

A Powerful But Lumbering Elephant

Hadoop does not respond anywhere close to “human time”, a term that describes response thresholds acceptable to a human user, typically on the order of seconds. Larry Ellison and his marketing mavens invoke a similar theme when pitching their wares as “analytics at the speed of thought.”

Nonetheless, this sluggishness is not the fault of Hive or Hadoop per se. If a business user asks a question about a year’s worth of data with Hive, a set of MapReduce jobs will dutifully scan and process, in parallel, terabytes of data to obtain the answer. It’s neither the commodity hardware that most Hadoop clusters use nor some of its IO indulgences while executing processes, that are to blame. These are the low-order performance bits.

And while Hadoop jobs do have a fairly constant overhead — with a lower bound in the range of 15 seconds — this is often considered trivial within the context of the minutes or hours that most full jobs are expected to take.

The higher-order bits affecting query performance are: (i) the size of the data being scanned, (ii) the nature of storage, e.g. whether it is kept on disk or in memory, and (iii) the degree of parallelization.

An Emerging Design Pattern: Distill, then Store

As a result, a common design pattern is emerging among data-heavy firms: Hadoop is used as a pre-processing tool to generate summarized data cubes, which is then loaded into an in-memory, parallelized database — be it Oracle Exalytics, Netezza, Greenplum or even Microsoft SQL Server. Occasionally, a traditional database query layer can be bypassed altogether, and summary data cubes can be loaded directly into a desktop analytics tool such as Qlikview, Spotfire, or Tableau.

At my start-up Metamarkets, we have embraced this design pattern and the role that Hadoop plays in preparing data for fast queries. Our particular bag of tricks is best described by the three principles of Druid:

Distill: We roll data data up the coarsest grain at which a user might have reasonable interest. Put simply, it is rare that one is concerned with individual events at one-second time frames. Rolling up to groups of events, with a select set of dimensions and at minutely or hourly granularity, can distill raw data’s footprint down to 1/100th of its original size.

Distribute: While this summarized data is spread across multiple nodes in our cluster, the queries against this data are also distributed and parallelized. In our quest to break into the “human time” threshold, we have increased this parallelization to as many as 1000 cores, allowing each query to hit a large percentage of nodes on our cluster. In our experience, CPUs are rarely the bottleneck for systems serving human clients, even for a cluster serving hundreds of users concurrently.

Keep in Memory: We share Curt Monash’s sentiment that traditional databases will eventually end up in RAM , as memory costs continue to fall. In-memory analytics are popular because they are fast, often 100x to 1000x faster than disk. This dramatic performance kick is what makes Qlikview such a popular desktop tool.

The end result of these three techniques, each of which independently delivers between a 10 and 1000-fold improvement, is a platform that can run in seconds what previously took minutes or even hours in Hive.

This approach, for which we know we are not alone in pursuing, achieves performance that exceeds or matches any of the big box retailers at a considerably lower price point.

The commoditization wave that began with massive data processing, initiated by Hadoop, is migrating upwards towards query architectures. Thus the competitive differentiators are shifting away from large-scale data management and towards what might be called Big Analytics, where the next battle for profits will be fought.

(reblogged from a version I wrote at the Metamarkets blog).

four threats to oracle, data's big box retailer

Fri, 07 Oct 2011 21:59:00 -0400

This week’s Oracle World was bracketed by two events. First: the unveiling of Oracle Exalytics, a beefy in-memory appliance dedicated to large-scale analytics, during Larry Ellison’s opening keynote. Second: the undressing of Oracle’s cloud computing initiatives by Marc Benioff, SalesForce’s CEO, and the unceremonious cancellation of his keynote on Wednesday morning.

Both events highlight that when it comes to Big Data, analytics and cloud computing, Oracle is on the wrong side of history.

Startups don’t use Oracle

To glimpse the future of the data stack, Oracle need look no further than its own backyard, to what Silicon Valley start-ups are embracing: the distributed processing ecosystem of Hadoop, NoSQL data stores like MongoDB, and cloud platforms like Amazon’s web services. As Marc Andreessen said last week, “Not a single one of our startups uses Oracle.”

The truth is, Oracle can’t support the kind of technology stacks embraced by startups — open-source software, elastic architectures, commodity hardware grids — because it cannibalizes revenue from their existing lines of business.

“I don’t care if our commodity X86 business goes to zero,” Ellison said in Oracle’s last earnings call, “We don’t make money selling that.”

This commoditization wave has sent others, including HP, fleeing from hardware, but it has driven Oracle into the breach as a big box retailer: they’re attempting to capture higher margins on sales of SPARC architectures.

But history is not on Oracle’s side. Here are four realities that Oracle must face to maintain its unassailable position as the world’s leading data firm:

Threat #1: The future of data is distributed

“Lots of little servers everywhere, lots of little databases everywhere. Your information got hopelessly fragmented in the process.” – from Matthew Symonds book Softwar (p. 38).

This is how Larry Ellison described the technology landscape of the 1990s, and his personal jihad against complexity has deepened Oracle’s distrust of distributed computing.

But the tide of data isn’t turning back, and the scale is too large to contain in any box; Big Data, on the scale of hundreds of terabytes to petabytes, must be distributed across “lots of little servers.” The most viable tool available today for processing and persisting Big Data is Hadoop.

Whether at the data layer — or a level above, at analytics — firms must adapt to this distributed reality and build tools that enable parallelized, many-to-many migration of data between nodes on Hadoop and those on their own platforms.

Threat #2: The future of computing is elastic

Metal server boxes don’t bend or expand; they are inelastic, both physically and economically. In contrast, the needs of businesses are highly elastic; as companies grow, they shouldn’t have to unpack and install boxes to meet their compute needs, any more than they should install generators for more electricity.

Computing is a utility, compute cycles are fungible, and firms want to pay for what they need, when it’s needed, like electricity.

The ability to scale storage and compute capacity up or down, within minutes, is liberating for individuals and cost-effective for organizations, but it is impossible with a “cloud in a box.” It is only enabled by a true cloud computing infrastructure, with virtualization and dynamic provisioning from a common pool of resources.

Threat #3: The future of applications is not on the desktop

Despite Oracle having developed the first pure network computer in 1996 (or perhaps because of this), far too many of Oracle’s supporting business applications are delivered via the desktop, rather than via web browsers.

By comparison, Cloudera has created a rich web-based application for managing and monitoring all aspects of Hadoop clusters; Amazon Web Services has a fully-featured web console for interacting with its offerings; and Salesforce’s products are almost exclusively web-driven.

The expressivity afforded by web browsers has risen dramatically in the last two years, particularly with the emergence of Javascript as the lingua franca of web application development, and improvements in Javascript engines.

The same trend from desktop to browser also extends into mobile devices. An increasingly large fraction of computing occurs on smart phones and tablets, and forward-thinking firms, like Dropbox, have built applications that cater to this reality.

Threat #4: The future of analytics is beautiful

The decades of disappointment with business intelligence tools isn’t due only to their lack of brains (such that they’ve now fled to the fresh moniker of “business analytics”), but also the absence of beauty. Data is beautiful, as any reader of Edward Tufte can attest.

When visualized thoughtfully and artfully, data has an almost hymnal power to persuade decision makers. And when exploring data of high complexity and dimensionality, the kind that lives in Oracle’s databases, tools that accelerate the “mean time to pretty chart” are essential.

In addition, analytics tool users are right to expect a smooth user experience on a par with other tools, whether photo editing or word processing, when they are creating and exploring data visualizations.

Yet amidst all of Oracle’s presentations and marketing materials about big data and analytics, one finds not a single dashboard or visualization to stir the senses.

While Spotfire and Tableau are notable exceptions to this critique, on the whole, the tools that dot the Oracle landscape lack either brains or beauty.

Enterprises will be slow to wake up to these realities, and Oracle will continue to profit handsomely from their slumber.

Fin: Oracle is ripe for attack by data services

The opportunities abound to chip away at the massive market share that Oracle now holds, providing data services to start-ups who won’t buy Oracle’s capital intensive boxes, and helping medium-sized businesses migrate to flexible, cost-effective, cloud-based alternatives.

(An earlier version of post was published as a guest column at GigaOm).

the secret guild of silicon valley

Fri, 19 Aug 2011 05:20:00 -0400

A couple of weeks ago, I was drinking beer in San Francisco with friends when someone quipped:

“You have too many hipsters, you won’t scale like that. Hire some fat guys who know C++.”

It’s funny, but it got me thinking. Who are the “fat guys who know C++”, or as someone else put it, “the guys with neckbeards, who keep Google’s servers running”? And why is it that if you encounter one, it’s like pulling on a thread, and they all seem to know each other?

The reason is because the top engineers in Silicon Valley, whether they realize it or not, are part of a secret Guild. They are a confraternity of craftsmen who share a set of traits:

Their craft is creating software
Their tools of choice are C, C++, and Java — not Javascript or PHP
They wear ironic t-shirts, and that is the outer limit of their fashion sense
They’re not hipsters who live in the Mission or even in the city; they live near a CalTrain stop, somewhere on the Peninsula
They meet for Game Night on Thursdays to play Settlers of Catan
They are passive, logical, and Spock-like

They aren’t interested in tweeting, blogging, or giving talks at conferences. They care about building and shipping code. They’re more likely to be found in IRC chat rooms, filing JIRAs for Apache projects, or spinning out Github repos in their spare time.

They are part of a nomadic band of software tradesmen, who have mentored one another over the last four decades in Silicon Valley, and they have quietly, steadily built the infrastructure behind the world’s most successful companies. When they leave — as they have places like Netscape, Sun, and Yahoo — the firms they leave behind wither and die.

If you want to build a technology company, you’ll need to hire them, but you’ll never find a member of the Guild through a recruiter. They are being cold-called, cold-emailed, and cold-LinkedIn-messaged on a daily basis by recruiters, but their response will be similarly cold.

A true member of the Guild is only ever an IM away from a new job at Facebook, Google, or the long archipelago of start-ups their fellow members are busy building. Outwardly successful companies that fail to draw engineers from the Guild will struggle with the performance and stability of their technology — as LinkedIn did in its early days and as Twitter did until recently.

It’s rare for an entrepreneur or executive to earn membership in the Guild, for that requires a path of apprenticeship that few have the talent or stamina for. But it’s possible to earn the respect of the Guild, and to convince its members that your company is a hall where they can gather daily to mentor and develop their craft.

It begins with having an engineering-led culture, where technology decisions are made on their technical merits, never on personal grounds. It also means allowing craftsmen to solve problems by creating new tools, rather than with just a labored application of the old. These are values that Google and Facebook, two veritable Guild halls of the Valley, tout to any engineer who asks.

Finally, the implicit compact that the Guild makes with a company is that their efforts will not be in vain. The most powerfully attractive force for the Guild is the promise of building a product that will get into the happy hands of hundreds, thousands, or millions. This is the coveted currency that even companies that have struggled to build an engineering reputation, like foursquare, can offer.

The Guild of Silicon Valley is largely invisible, but their affiliations have determined the rise and fall of technology giants. The start-ups who recognize the unsung talents of its members today will be tomorrow’s success stories.

[ Addendum: George E.P. Box said “All models are wrong. Some models are useful.” While my tongue-in-cheek model of the anti-hipster Guild of Engineers has angered those who interpret it literally, my rhetorical goal is to make a point: that the hard work of engineering isn’t glamorous, and is often invisible to the media or the reigning pop culture of start-ups you’ll find in San Francisco. If you want to build a successful technology company, you would do well to target the experienced folks who have been honing their craft in the trenches of Silicon Valley for the last few decades, and those whom they’ve mentored. ]

The Big Data Stack, from my piece, Building Data Startups at...

Wed, 17 Aug 2011 21:50:12 -0400

The Big Data Stack, from my piece, Building Data Startups at O’Reilly Radar.

Slides from my presentation from O’Reilly’s Strata...

Thu, 26 May 2011 21:08:37 -0400

Slides from my presentation from O’Reilly’s Strata Online Conference on May 25, 2010.

"information is the new oil."

Mon, 16 May 2011 04:08:00 -0400

This past February, I moderated an event at Stanford about Predictive Analytics. I led with a brief introduction, followed by a discussion with Omar Tawakol, CEO of Bluekai, Scott Burke of Yahoo!, Matt Barkoff, VP at Badgeville, and Theresia Gouw Ranzetta, Partner at Accel Partners.

See the YouTube video here.

node.js and the javascript age

Tue, 19 Apr 2011 03:03:00 -0400

Three months ago, we decided to tear down the framework we were using for our dashboard, Python’s Django, and rebuild it entirely in server-side Javascript, using node.js. (If there is ever a time in a start-ups life to remodel parts of your infrastructure, it’s early on, when your range of motion is highest.)

This decision was driven by a realization: the LAMP stack is dead. In the two decades since its birth, there have been fundamental shifts in the web’s make-up of content, protocols, servers, and clients. Together, these mark three ages of the web.

Read Full Post at Metamarkets

color: the cinderella of dataviz

Tue, 19 Apr 2011 02:47:00 -0400

Color is one of the most abused and neglected tools in data visualization. It is abused when we make poor color choices; it is neglected when we rely on poor software defaults. Yet despite its historically poor treatment at the hands of engineers and end-users alike, if used wisely, color is unrivaled as a visualization tool.

Read Full Post at Dataspora

"The final project in every college CS course should be: contribute a feature to an open-source..."

Tue, 19 Apr 2011 02:21:31 -0400

“The final project in every college CS course should be: contribute a feature to an open-source project.”

- via Twitter.

the rise of the data web

Tue, 19 Apr 2011 01:50:00 -0400

The future of the web is data, not documents. The web has evolved from Tim Berners-Lee’s original vision of“some big, virtual documentation system in the sky” into an vibrant ecosystem of data where documents — and human actors — will play an ever smaller role.

Read Full Post at Dataspora