Technology Inheritance

Someone once told me that people tend to move between companies using the flying wedge formation: a scout shows up to learn the culture and opportunities, and then an increasingly large contingent of previous coworkers and friends stream in.

This jives with my experience, likely the direct consequence of most hiring systems’ heavy reliance on referrals (incidentally, I think what Slack is attempting to do to move their hiring pipeline away from referal dependence is one of the more interesting ideas in tech recruiting today).

One of the fascinating side-effects of group movement is that they tend to come with unexpected stowaways: their favorite technology stack. This causes certain technology stacks to propagate from company to company.

Who has used Clusto? Well, there is essentially no chance you’ve even heard of this tool unless your company inherited Digg’s infrastructure stack (I believe Linden Lab is one of the few companies that adopted it organically). If you have worked at such a company, especially in the 2011 to 2014 range, then you very likely did use it, or at least strongly considered it!

(If so, you may have developed feelings about Clusto.)

Clusto’s just the tip of the inherited stack thought! The full Digg infrastructure stack looks like:

1. Python for scripting.
2. Clusto for infrastructure metadata.
3. Puppet for role configuration.
4. FAI (aka Fully Automatic Installed) for provisiong servers.
5. HAProxy running locally on every node as a load balancer.
6. A healthy distrust of DNS.

I’m optimistic that there aren’t many new companies getting founded with this specific technology stack, but if you looked at the companies where ex-Digg engineers joined as early engineers, you’ll continue to see some of the hallmarks propagate, dilluting over time.

This isn’t unique to Digg (if you’ve used NSQ, your chances of having a coworker from Bitly is rather high), but part of a larger pattern that you’ll see everywhere as you start looking for it. I suspect it’s even an optimal behavior, in terms of picking tools you’re familiar and comfortable with, instead of investing in technology decisions that are fairly unlikely to make or break a young company.

For some technologies though, and I think Clusto is likely in this category in 2017, it does create a hazy second-life for open source technologies that never achieve organic adoption. They end up with adoption returns that mirror venture capitalists financial returns: probably 90% of the engineers using Clusto are at Uber (even though I’m pretty sure there aren’t any ex-Digg engineers there anymore), with limited survival in the broader marketplace of ideas. (Consul and EtcD, along with the Puppet, SaltStack and Chef specific tooling, feel like they’re today’s winners.)

Whatis your team’s technology inheritance?