Apache Druid Turns 10: The Untold Origin Story

“We did something crazy: we rolled our own database.” – Eric Tschetter, creator of Druid

Ten years ago today, the Druid data store was introduced to the world by Eric Tschetter, its creator, working at a small start-up named Metamarkets.  Eric had left LinkedIn six months earlier to join us as the first full-time employee, and I was the CTO and co-founder, working in a shoebox office[1] off South Park in San Francisco.  In his blog post, Introducing Druid: Real-Time Analytics at a Billion Rows Per Second, he shared the rationale for Druid’s creation:

“Here at Metamarkets we have developed a web-based analytics console that supports drill-downs and roll-ups of high dimensional data sets – comprising billions of events – in real-time. This [post introduces] Druid, the data store that powers our console. Over the last twelve months, we tried and failed to achieve scale and speed with relational databases (Greenplum, InfoBright, MySQL) and NoSQL offerings (HBase). So instead we did something crazy: we rolled our own database. Druid is the distributed, in-memory OLAP data store that resulted.”

The initial responses from HackerNews were predictably skeptical:

  • “It’s always tempting to build it yourself. “
  • “They should have just used QlikView.”
  • “HANA has been doing this for at least 5 years now.”

Ignoring the naysayers,  Eric continued to lead the engineering team in building out Druid as the core engine for the Metamarkets platform.  A year and half later, in October 2012, we open sourced Druid in a talk we gave at O’Reilly Strata’s conference [2]. Open source Druid has now been adopted by hundreds of leading companies around the globe, notably Netflix [3], Lyft [4], eBay [5], Netflix [6], Salesforce[7], Pinterest [8], Yahoo! [9], and Snap [10].

While most of this is public, there is one piece of history about Druid that hasn’t previously been shared. Before Metamarkets’ acquisition by Snap in 2017, I retained a few keepsakes from the early days. One of them was an email titled, quite simply, “Druid — the spec” from February 7, 2011. It is 78 lines and 553 words long. It lays out a simple proposal for Druid’s architecture, with a postscript “I’m going to take this project on as a background thread, working on it whenever there aren’t other more pressing things to deal with.” In the subsequent eight weeks, Eric not only wrote Druid but he pushed it into production. He sunset our HBase cluster on April 4, 2011, replacing it with the first Druid service. That original Druid cluster has been continuously operational for over a decade, having processed over 100 trillion events, 100 billion queries, and 1000s of end users in its lifetime.

Druid architecture — “the spec”

---------- Forwarded message ---------
From: Eric Tschetter <cheddar@metamarketsgroup.com>
Date: Mon, Feb 7, 2011 at 6:09 PM
Subject: Druid -- the spec
To: <eng@metamarketsgroup.com>
Given what we've learned about our data needs and various operational
issues, I think we need the following backend in order to support our
dashboarding and analytics front-end needs.

I'll start with the requirements:

1) Fast scans of the data
2) Operationally simple to manage


So, in order to meet these requirements, the initial implementation should be

1) In-memory: the only way to ensure fast scans is to remove disk from
the equation as much as possible.  This starts with everything in
memory
2) Able to push code to data: this means to me that it should be
implemented on the JVM as it provides functionality for distributing
code and the various scripting languages on the JVM allow for a wide
variety of languages.  This is because the only way we are going to
get fast scans of data is to be able to push code down to data in
parallel.
3) Symmetrical (all nodes should be able to do the same job that other nodes do)


What I propose is:

1) Take "beta" style data and produce an "index" which is the beta
style data indexed for all of the dims.  Partition the indexes by
timestamp (daily for now) and store them on s3
2) Build up a simple Java process that, when requested (HTTP) for data
in a specific time range, will go and pull the index and load it in
memory
-- there are a number of optimizations that can be done to reduce
latency here, but we'll start really simple
3) Build a fat client that will consistently hash over the current
instances and request data in parallel to as many nodes as it needs to

What will not be in v1:

Pushing of code.  At the beginning, there will be a set of operations
that are pre-implemented and clients will be limited.  Eventually, if
this works out as hoped we will add that ability.

Benefits:

1) Simple to manage and deploy a new cluster of these things
-- Just fire up some java processes on nodes that have access to s3
-- Capacity expansion is as simple as adding new machines
-- Vertical scalability possible
2) Fast (hopefully)
-- Initial tests seem to indicate that it will be fast enough, but
this is yet to be seen.
3) Changes the problem of more dimensions from a
combinatorial(exponential) problem into a multiplicative problem
-- Makes it significantly faster to backfill
4) Allows for more flexibility in the future
5) Possible to turn this into a real-time aggregating and serving system as well
6) All dimensions available
7) Ability to do or/and queries inside of a specific dimension

Downsides:

1) A new system
-- I don't think we have a choice, though
2) Indexing implies the existence of a schema.
-- Our data isn't schema-less anyway, so I think this is acceptable


I obviously think there is more up-side than down-side, but it's my
proposal so there's most likely some bias.  Please provide feedback.
Specifically, if there are any more downsides that people can think
of, it will be good to have them out and known ahead of time.

Barring objections that might be raised, I'm going to take this
project on as a background thread, working on it whenever there aren't
other more pressing things to deal with.

--Eric

While I could share my own interpretations about the modest beginnings of technology innovation, often stories and source materials speak for themselves. That’s the appeal of the collections of stories in books like Revolution in the Valley and Founders at Work. I hope this story of Druid’s origins will be valuable to others out there who have a crazy idea for a new software architecture, and steel them with the confidence that their contribution could indeed make a dent in the technology universe.

Footnotes

[1] The shoebox was 300 Brannan Street, San Francisco. It was packed to the gills with startups, like many office buildings around South Park at the time. It was an unofficial, physical incubator where the rents were (then) affordable and the amenities were few.  Guillermo Rauch, who went out to create Next.js and Vercel, was downstairs from us; Daniel Gross, was then a 19-year prodigy and media darling working on Greplin (eventually Cue) down the hall.  The entire building reeked of a sickly sweet odor from its first floor restaurant, aptly named Ozone Thai.

[2] Beyond Hadoop: Fast Ad-Hoc Queries on Big Data Michael Driscoll, Eric Tschetter. O’Reilly Strata Conference 2012. I remember the night before the Strata talk, our head of sales & marketing, Eric, and I were holed up in a Manhattan hotel room rehearsing while Eric pored over the code base to clear it for public release.

[3] How Superset and Druid Power Real-Time Analytics at AirBnB. (June 2017) by Maxime Beauchemin, YouTube.

[4] Streaming SQL and Druid at Lyft. (August 2018) by Arup Malakar, YouTube.

[5] Monitoring at eBay with Druid (May 2019). Mohan Garadi, eBay Blog.

[6] How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience (March 2020). Ben Sykes, Netflix Tech Blog.

[7] Salesforce – Delivering High-Quality Insights Interactively Using Apache Druid at Salesforce (2020). Dun Lu, Salesforce Engineering Blog.

[8] Powering Pinterest ad analytics with Apache Druid (Jan 2020). Filip Jaros and Weihong Wang, Pinterest Engineering Blog.

[9] Yahoo Casts Real-Time OLAP Queries with Druid (Aug 2015). Datanami.

[10] Data analytics and processing at Snap (Sep 2018). Charles Allen, Slideshare.

Apache, Apache Druid, Druid, the Apache Druid logo, Apache Superset, and Superset are registered trademarks or trademarks of Apache Software Foundation.

Operational intelligence and the new frontier of data

Always-on businesses such as global retailers, social media apps, transportation platforms, and financial marketplaces have mission-critical use cases that require real-time decisions on operational event streams.

Target’s supply chains must adapt to changes in store inventory, Snap’s new app launches must be debugged, Lyft’s drivers must be predictively routed to riders, and at Paypal, fraudulent payments must be flagged and blocked.

These use cases for logistics, app monitoring, and fraud detection aren’t science fiction, they’re real-world examples powered by an emerging technology stack that combines event stream processing, fast OLAP databases, interactive dashboards and machine-learning applications.  Fueled by real-time data coming from instrumented products and services, this technology stack is driving a distinct category of analytics called operational intelligence, which is complementary to traditional business intelligence.

I believe the need for operational intelligence will dramatically expand in the coming years.  In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.

Operational intelligence: why now?

A global digital nervous system is emerging

One consequence of ubiquitous computing and digital transformation is the promise of visibility into the previously invisible.  There are now an estimated 20 billion connected devices on the planet, encompassing everything from smartphones and cars, to lamp posts and laundry machines.  Previously unobservable actions in the analog world — a package delivery, a store purchase, a taxi ride — now throw off digital signals in the form of a barcode scan, a payment gateway request, or a stream of GPS heartbeats.

Collectively, these connected devices are the sensory scaffold for an emerging global digital nervous system, whose potential we are only just beginning to explore.  However, some of the near-term opportunities are analogous to the advantages of our own nervous system:  faster reaction times and better decision-making with richer sensory data.

Event streams are the new frontier of data

While smart sensors have been widely deployed and digital services are more finely instrumented than ever, businesses are just now building the software infrastructure to capture and process the event streams from their products and services.

With this first wave of data plumbing complete, businesses now have an opportunity to build new kinds of workflows. For example, traditionally businesses compile and update a set of KPIs on a weekly or daily basis. What happens when it’s possible to track these metrics up-to-the-minute and down to an individual line item or customer? It means that small issues can be addressed before they become big challenges. Shipped software used to take months to detect and correct issues, but today Netflix continuously monitors the performance of new versions of its software and can revert software for specific users when issues are detected. Similarly, Lyft tracks rider conversion rates at a city block level and redirects drivers to a particular block when those rates drop.

Intelligence pushed to the edge: accelerating decision loops

Sensor data is generated on the edge by users on their devices, fleets of GPS-instrumented vehicles and servers in a data center. This data is then streamed over the cloud to a central processing system to synthesize KPIs.  However, if the insights stopped here, with a downsloping graph on the dashboard of a business analyst, the value of these insights is limited.

One of the hallmarks of operational intelligence is the speed of feedback to the edge. In the real-life examples above, Netflix and Lyft aren’t waiting hours or days to make adjustments, they are acting within seconds to minutes.  As operational intelligence systems mature, human analysts are excised from these decision loops, and these course-corrections are automated, powered by machine-learning algorithms.

To support the accelerated decision-cycles, businesses need to transform their technology stacks.

How operational and business intelligence are different

Thinking, fast and slow

The distinction between operational and business intelligence is analogous to the distinction between fast and slow thinking, as characterized by psychologist Daniel Kahneman in his paper Thinking, Fast and Slow. One system operates quickly and automatically for simple decisions and the other leverages slow and effortful deliberation for complex decisions). Per Kahneman’s own style, we share a few examples below that illustrate different use cases for operational and business intelligence.

Operational intelligenceBusiness intelligence
As a Lyft marketplace optimization algorithm, should I direct more drivers to Logan airport to increase conversions?As Lyft’s head of HR, should we hire more drivers in Boston this month, to match rising demand there?
As a Netflix DevOps engineer, should I roll back this morning’s software release for tablets running Android version 10.0.0_r52, so we can reduce crash rates?As a Netflix product manager, should we delay the Android feature launch a few days for further testing?
As a Pinterest campaign manager, should I increase our advertising budget by 5% in the next hour on YouTube?As a director of marketing, how should we allocate advertising budget across YouTube and Instagram next quarter?

Ultimately, the output of operational and business intelligence are decisions. Operational intelligence fuels fast, frequent decisions on real-time and near-time data by hands-on operators. Business intelligence drives complex decisions that occur daily or weekly, on fairly complete data sets by managers. We further unpack some of the distinguishing features below.

Operational intelligenceBusiness intelligence
Decision features
FrequencyIntraday, continuousDaily, weekly, quarterly
ScopeNarrow, localizedBroad, macro
EffortLowHigh
Decision makers
Organization rolesOperatorsManagers
ProcessDecentralizedCentralized
Algorithmic decisionsFrequentRare
Data inputs
SourcesRaw event streamsData lakes
FreshnessUp to real-timeUpdated hourly or daily
History retainedWeeks to monthsYears to forever
Tools
Database engineOLAP databasesCloud warehouses
Speed of queriesSub-second to secondsMinutes to hours
Dashboard typeAd hoc, interactiveStructured, limited interactivity

Managers can be operators too, albeit on a much higher level.  Some of the biggest consumers of operational intelligence tools are CEOs, who track and react to intraday global KPIs for their businesses.  Operational KPIs often have an organizational hierarchy, and at every level — from a district manager to field service provider — there are component KPIs that require continuous monitoring and decision-making.

Why operational intelligence requires new technologies‍

Operational intelligence provides a set of decision-making capabilities that are complementary to business intelligence.  But its unique performance requirements also demand a novel stack of distinct technologies which are complementary and sit adjacent to existing business intelligence stacks.

Analytics technology stacks can be thought of as data flowing into a three-layered cake consisting of ETL, databases, and applications.  The requirements for an operational intelligence stack is that it supports:

  • high speed of data from ETL to application
  • high frequency, low-latency queries between the database and application layer

In the diagram below we illustrate two common examples for technologies used in operational and business intelligence stacks.

On the left hand side we see an operational intelligence stack. The high speed requirements of operational intelligence are met by ensuring the data stays in flight and never hits disk. This is made possible with a combination of Kafka, a streaming processing engine like Flink, and Apache Druid. This is not achievable with the data lake architecture shown on the right-hand side of our diagram, where data lands in the cloud data lake (S3), is transformed by Spark batch processing jobs, and finally loaded into a data warehouse. 

Regarding the requirement of high frequency, low latency queries at the application layer, data warehouses are simply not optimized for these access patterns.  In the best case scenario, a data warehouse is able to achieve low latency queries, but at significantly higher cost per query.  Warehouses like Snowflake optimize for low-cost storage, storing data on disk, while OLAP data stores like Apache Druid optimize for low-cost queries, storing data mostly in memory.

Retrofitting an existing business intelligence stack for operational intelligence capabilities will deliver lower performance at a higher cost; while these costs may not be apparent at small scale, at large scale they can become unsustainable.

Synergies between operational and business intelligence

While the illustration above shows operational and business intelligence as parallel, independent technology stacks, the reality is that the most successful data architectures intertwine these.  Traditional business intelligence applications such as Looker and Tableau can benefit from the performance gains offered by the data stores favored by operational intelligence stacks.  Similarly, operational intelligence stacks can benefit from the cost efficiencies of batch processing, particularly when doing historical restatements.

As data infrastructure technologies evolve, it’s uncertain whether a single stack will emerge to serve both operational and business intelligence use cases, or whether these stacks will diverge further.  In the present, however, the experiences of the world’s leading digital companies indicate that the era of operational intelligence has dawned and will shine for many years to come.

Savor the surprises

Alexander Fleming, Photo Credit Historic UK

I was recently listening to Mike Maples interview Andy Rachleff about the search for product-market fit, which is fitting, since Andy coined the term. Andy recounted a pearl of wisdom that Scott Cook had once given him: when doing customer research, savor the surprises. A few years into founding Intuit, Scott uncovered that while QuickBooks was created for personal finance, half their users were businesses. Why? Because most small businesses lacked formal accounting expertise, they preferred simple software.

Savoring surprises is a simple but powerful framing, because it forces reflection. It’s a great question for a job interview (“When you first got to Google, what surprised you?”) or at a cocktail hour (“What surprised you most about Tokyo?”). Who would want to hire or hang out with someone who answers “Nothing”?

Surprises are the bits of data we don’t expect. It is cognitively taxing to retain these bits, rather than burying them to confirm what we think we already know. Savoring surprise is at the heart of the beginner’s mindset. And it is the essence of learning and discovery.

In September 1928, a scientist returned from a two-week vacation and found a mold had contaminated his bacterial culture, and unexpectedly, killed the bacteria around it. Alexander Fleming savored this surprise, rather than ignore it, and it ultimately led to his discovery of penicillin. As he later put it “One sometimes finds what one is not looking for.”

Geoffrey Moore on disrupting business with data intelligence

[A]ll the great business disrupters of the past decade [–] Amazon, Google, Microsoft, Apple, Tesla, Uber, Airbnb, Netflix—they are all running Systems of Observation against the data flows they are privileged to access or host, and then feeding them into Systems of Intelligence to extract insights from them.

Geoffrey MooreIntelligent Computing Systems: How will Enterprise Architecture Evolve?

Colin Ware on cognition

Thinking is not something that goes on entirely, or even mostly, inside people’s heads. Little intellectual work is accomplished with our eyes and ears closed. Most cognition is done as a kind of interaction with cognitive tools, pencils and paper, calculators, and, increasingly, computer-based intellectual supports and information systems. Neither is cognition mostly accomplished alone with a computer. It occurs as a process in systems containing many people and many cognitive tools. Since the beginning of science, diagrams, mathematical notations, and writing have been essential tools of the scientist. Now we have powerful interactive analytic tools, such as MATLAB, Maple, Mathematica, and S-PLUS, together with databases. The entire fields of genomics and proteomics are built on computer storage and analytic tools.

Colin Ware. Information Visualization: Perception for Design.

design principles for data pipelines

image

(Image: ‘Tower of Babel’ by Pieter The Elder Bruegel, 1563)

Underinvestment in and misunderstanding of ETL is a silent killer in organizations.  It’s why reports are often delayed, why answers across systems rarely agree, and why more than 50% of corporate business intelligence initiatives fail.

ETL is hard because data is messy.  Even the most common attribute of data, time, has thousands of accepted dialects: “Sat Mar 1 10:12:53 PST,” “2014-03-01 18:12:53 +00:00” and “1393697578” are all equivalent.  And there’s a growing chorus of other sources with even less consistency:  geo-coordinates, user agent strings, country codes, and currencies. Each new data type is a layer of bricks in our collective, digital tower of Babel.

It’s no wonder that a CIO recently confessed to me that he’d spent tens of millions of dollars a year on the reliable, repeatable transformation of data – and that some of Silicon Valley’s smartest minds (Joe Hellerstein and Jeff Heer’s Trifacta, Wes McKinney’s Datapad) are tackling this challenge.

As someone who has spent much of my career wrestling ETL’s demons, here are five secrets for keeping cool inside data’s infernos:

1. Stay Close To The Source

Journalists know that to get the truth, go to primary sources. The same is true for ETL. The closer you are to the origin of the data, the fewer dependencies you will have on filtered or intermediate versions, and the less chance that something will break.

Beyond fidelity, closeness to data sources conveys speed: tapping into a raw source means data feeds can be processed and acted upon in minutes, not hours or days.

The best ETL pipelines resemble tributaries feeding rivers, not bridges connecting islands.

2. Avoid Processed Data

Like food, data is best when it’s fresh, you know the source, and it’s minimally processed.  This last piece key: in order to crunch huge quantities of data, one common approach is sampling.  Twitter provides a spritzer stream that is < 1% sample of all tweets; in my world of programmatic advertising, many marketplaces provide a 1% feed of bid requests to buyers.

Sampling can be great for rapid prototyping (and Knuth’s reservoir sampling algorithm is both beautiful and useful), but in my real-world experience, I’ve rarely found a sampling approach that didn’t backfire on me at some point. Critical population metrics – like maxes, mins, and frequency counts – become impossible to recover once a data stream has been put through the shredder.

In era when bandwidth is cheap and computing resources are vast, you may choose to summarize or sketch your data – but don’t sample it.

3. Embrace (And Enforce) Standards

In the early days of the railroads, as many as a dozen distinct track gauges, ranging from a width between the inside rails of 2 to nearly 10 feet, had proliferated across North America, Europe, Africa and Asia. Owing to the difficulties of non-interoperable trains and carriages, as well as continuous transport across regions, a standard width was eventually adopted at the suggestion of a British civil engineer named George Stephenson. Today, approximately 60% of the world’s lines use this gauge.

Just as with the railroads two centuries before, systems that embrace and enforce standards will succeed, and those who invent their own proprietary binary formats will suffer.

4. Let Business Questions Drive Data Collection

Too many organizations, upon recognizing that they’ve got data challenges, decide to undertake a grand data-unification project. Noble in intentions, cheered by vendors and engineers alike, these efforts seek to funnel every source of data in the organization into a massive central platform (today, it’s usually a Hadoop cluster). The implicit assumption is that “once we have all the data, we can answer any question we’d like.” It’s an approach that’s doomed to fail.  There’s always more data available than can be collected, so choosing what to crunch can only be made in the context of business questions.

Laying down ETL pipe is wrist-straining work, so avoid building pipelines and drilling data wells where no business inquiry will ever visit.

5. Less Data Extraction, More API Action

Sometimes working with the nuts and bolts of data is a necessity, but for a growing class of problems it’s possible to get out of the data handling business entirely.  Take contact lists, email, and digital documents: for years, IT departments suffered through migrations of these assets from silo to silo. Today, cloud applications like Salesforce, Gmail, and Box make this somebody else’s problem.

A maturing ecosystem of SaaS applications expose APIs for acting on – without extracting – cloud-managed data.  These interfaces will allow developers and organizations to focus less on handling data and more on the activities of their core businesses.

(An earlier version of this essay appeared on February 27, 2014 in AdExchanger).

let us now praise data engineers

image

In the last year, the data scientist has been called “the sexiest job of the 21st century.”  But if data is the new oil, and data scientists are its petrochemical high priests, who are the oil riggers?  Who are the roughnecks doing the dirty work to get data pipelines flowing, unpacking bytes, transforming formats, loading databases?

They are the data engineers, and their brawny skills are more critical than ever.  As the era of Big Data pivots from research to development, from theoretical blueprints to concrete infrastructure, the notional demand for data science is being dwarfed by the true need for data engineering.

A stark but recurring reality in the business world is this: when it comes to working with data, statistics and mathematics are rarely the rate-limiting elements in moving the needle of value.  Most firms’ unwashed masses of data sit far lower on Maslow’s hierarchy at the level of basic nurture and shelter.  What is needed for this data isn’t philosophy, religion, or science – what’s needed is basic, scalable infrastructure.

It’s the data engineers who can build this infrastructure, and they represent the true talent shortage of Silicon Valley and beyond.  Their unsexy but critical skills include crafting Hadoop pipelines, programming of job schedulers, and parsing broad classes of data – timestamps, currencies, lat & long coordinates – which are the screws, bolts, and ball bearings in the industrial age of data.

Let us now praise these unsung heroes, the data engineers, who are building the invisible but essential digital underground.

the psychology of the enterprise buyer

image

Consumer startups like Facebook, Twitter, Pinterest, and even DropBox are built by founders who wanted to “make something cool” for their own benefit. Their teams intuitively understand what works because they are their own target audience: young, tech-savvy people looking for better ways to connect, share, and organize their digital stuff.

When it comes to buyer psychology, corporations are not people

By contrast, the challenge for enterprise startups, is that corporations are not really people (their legal personhood aside) — and certainly not our people.

When you’re hungry for lunch, you go and buy a sandwich for a few dollars. When an enterprise is hungry for lunch, it solicits bids from multiple catering companies, negotiates for weeks to months, and signs a contract for a few million dollars.

This gap between the psychology of enterprises and the startups that sell to them is a challenge that consumer startups do not face. Worse, early team members in startups have limited enterprise experience; they are a poor fit to the process-orientation and risk-aversion (or to put it more kindly, risk-balancing) that is rewarded at the higher levels of corporate environments.

Less Goldilocks, more Dunder-Mifflin

Lacking this enterprise DNA, younger startups often build their sales processes in the image of how startups buy rather than how enterprises buy. When startups seek to purchase a software solution, they favor simple, scalable pricing: click a box, swipe a credit card, and start running. Hence the canonical three-column SaaS pricing page (call it Goldilocks pricing) that you see at many SaaS companies—where the middle column invariably feels “just right.”

But large enterprise buyers are less adventure-embracing Goldilocks, and more The Office’s Dunder Mifflin. They require more than three sizes of self-serve, they don’t do click-through contracts, and they rarely pay with credit cards. The reasons are both economic and cultural. Economically, as buying decisions grow larger, the cost of sales — product customization, negotiated contracts, and invoicing — become marginally small. Culturally, Fortune 500 companies expect to have a relationship.

As Box CEO Aaron Levie recently told me, “Look, when Coca Cola writes you a big check, they want to meet you in person.”

Silicon Valley IT is not enterprise IT

Startups also often underestimate the importance of professional services and training for enterprises. They believe every company has a cadre of engineers smart enough to set up and tailor an application accordingly, and business users who can quickly figure it out — whether it be Google Analytics, Hubspot, or Expensify — and get up and running.

But this is not the case in most enterprises. The success of firms like RedHat, MySQL AB, and more recently, Cloudera, testify to the enormous value lies in integration and support, even when that software – whether Linux, MySQL, or Hadoop – is free and open-source.

Seasoned sales executives: The “growth hackers” of enterprise startups

As the venture investing pendulum swings back towards enterprise technology companies, founders and venture capitalists will need to augment their teams with sales executives who can nimbly step around the often woolly, sometimes mammoth challenges of contract negotiations, channel partnerships, and client services engagements. These experienced leaders will be the “growth hackers” of the enterprise realm.

This essay originally appeared in VentureBeat. 

the fuel of founders: curiosity & impatience

“On a scale of 1-10 of impatience, the best entrepreneurs are an 11.” – Tom Stemberg, Founder of Staples

Curiosity and impatience make for great founder traits, but they often pull in different directions.

Curiosity compels you to sit and study a problem, to voraciously consume every article and reference you can find to wrap your head around a big idea or an imagined future (self-driving cars, space elevators, or self-destructing sexts).

Impatience gets you up out of your chair to do something about it: hire, fundraise, sell, and evangelize.

Curiosity is for academics, impatience for executives, but start-up founders need to be both dreamers and doers, straddling the world of ideas and realities.

Robert Oppenheimer, the American Prometheus behind the first atomic bomb, was a dreamer – but he was also impatient.  His colleague Murray Gellman said he lacked the ability to sit still:    

“Germans call it ‘Sitzfleisch’, ‘sitting flesh’ when you sit on a chair.  As far as I know, he never wrote a long paper or did a long calculation, anything of that kind.  He didn’t have the patience for that… [But] he inspired other people to do things, and his influence was fantastic." 

Impatience is the very opposite of Sitzfleisch, and without it, the Manhattan Project would have yielded nothing more than chalk dust.

Curiosity is what drew Steve Jobs to sit in on calligraphy classes at Reed; inspired Larry Ellison to study chip design at U. Chicago; compelled Bill Gates to cram for economics courses at Harvard; lured Larry and Sergey to pursue computer science Ph.D.s at Stanford.  

Impatience is what drove them all to drop out and start Apple, Oracle, Microsoft, and Google.

Silicon Valley’s cult of the drop-out pays homage to impatience – who has time for school when you’re building a billion-dollar business? – but gives short shrift to curiosity which is the heart of innovation.

Nothing fires a healthy impatience more than the desire to see a big idea, born of deep curiosity, brought to life.  As Steve Jobs said, "remembering that you are going to die” is a great motivator.