(Image: ‘Tower of Babel’ by Pieter The Elder Bruegel, 1563)
Underinvestment in and misunderstanding of ETL is a silent killer in organizations. It’s why reports are often delayed, why answers across systems rarely agree, and why more than 50% of corporate business intelligence initiatives fail.
In the last year, the data scientist has been called "the sexiest job of the 21st century.” But if data is the new oil, and data scientists are its petrochemical high priests, who are the oil riggers? Who are the roughnecks doing the dirty work to get data pipelines flowing, unpacking bytes, transforming formats, loading databases?
They are the data engineers, and their brawny skills are more critical than ever. As the era of Big Data pivots from research to development, from theoretical blueprints to concrete infrastructure, the notional demand for data science is being dwarfed by the true need for data engineering.
Consumer startups like Facebook, Twitter, Pinterest, and even DropBox are built by founders who wanted to “make something cool” for their own benefit. Their teams intuitively understand what works because they are their own target audience: young, tech-savvy people looking for better ways to connect, share, and organize their digital stuff.
"On a scale of 1-10 of impatience, the best entrepreneurs are an 11." - Tom Stemberg, Founder of Staples
Curiosity and impatience make for great founder traits, but they often pull in different directions.
Curiosity compels you to sit and study a problem, to voraciously consume every article and reference you can find to wrap your head around a big idea or an imagined future (self-driving cars, space elevators, or self-destructing sexts).
Impatience gets you up out of your chair to do something about it: hire, fundraise, sell, and evangelize.
Curiosity is for academics, impatience for executives, but start-up founders need to be both dreamers and doers, straddling the world of ideas and realities.
(Image credit: A.Koblin for RadioHead)
This is a phrase that has stuck with me since Tim O’Reilly uttered some form of it two years ago. Tim was talking about online cartography, saying it’s not the maps that matter: it’s getting to our destination. Maps are a half-step short of that goal. And in a world of navigational algorithms and self-driving cars, maps become less useful as tools.
Likewise, data visualization is a halfway house: a stopping place on the path from data to decision.
“ You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.”
As we dedicate an increasing fraction of our time interacting with software — from airport check-in terminals and parking meters, to desktop and mobile applications — digital interface design is becoming as important as physical architecture in improving our experience of the world.
Here are Professor Ben Schneiderman’s Eight Golden rules for optimally designing that experience (drawn from his classic text, Designing the User Interface):
Silicon Valley’s first big bang of innovation occurred in 1957, when eight engineers left Shockley Transistor to form FairChild Semiconductor. Back then, the idea of engineers being entrusted as founders of a business was heretical. Forty-one firms were asked to invest, but “none of them were interested”, according to Arthur Rock.
The idea that engineers without MBAs can be successful founders has changed, but what about engineers acting as investors? In my experience, the majority of investment professionals on Sand Hill road are still non-technical.
But that is changing, in two ways.
(L to R: Mike Driscoll, Drew Conway, DJ Patil, Amy Heineike, Pete Skomoroch, Pete Warden, Toby Segaran. Credit: O’Reilly - Link to Video)
This past Tuesday evening at Strata I moderated an Oxford-Style debate between six of the top data scientists in Silicon Valley and beyond. The motion debated was:
"In data science, domain expertise is more important than machine learning skill."