Skill Cap

I've just started to read Corman's Algorithms, and in the first chapter a thought occurred to me.

There was a recent post about how inter-connected the Facebook codebase is. First of all, I found it quite surprising how much code there is, but I assume a good chunk of it is for scalability, which is certainly important (it's basically a mystery to me how facebook works at all, given that there's no way their user database can exist as a single object on a single machine).

However, thinking further, and thinking about the sorts of companies I see out there, it seems like there are a lot of people solving the same kinds of problems. And between one thing and another, I started wondering: how much of this is because our languages for expressing common problems haven't matured?

There's a cap on median engineer intelligence (it doesn't really matter what this means, but for the sake of argument, let's say the median engineer working in web technologies has an IQ no higher than 130, or two sigma above the population). Engineers are in a weird spot inasmuch as they work in one of the few industries where your job is very similar to your training and tool-building. A teacher can take classes on being a better teacher, and make resources to make teaching better and easier, but at the end of the day if they don't get in front of a classroom, they're not doing their job.

An engineer, on the other hand, can write a framework to make their work easier, learn (or create) a domain-specific language to speed up one aspect of their job that's repetitive, and at the end of the day have in hand new knowledge, work that pays the bills, and a tool they can release for other developers to use. This probably goes a long way in explaining the galapagosation of technologies, especially web technologies.

At some point, the learning curve of understanding how everything goes together becomes too challenging. Thus, our median engineer can't get much more efficient, because there are too many things to keep track of, and too many things that can break, and he wants to understand his toolchain and not look a fool because he can't tell his boss why the database got borked.

The thing is, we already have specialists (Database Engineers, SysAdmins, Front-end, Back-end, Project Engineers, the list goes on), and the role of "DevOps" kind of papers over the fact that at some scale, a company should be concerned with differentiation of responsibilities. All engineers should have some idea of how their work might affect other domains, but the job of optimization should lie in the hands of experts. (Note here Conway's Law, which states that "organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations".)

Is it possible to make a framework which would make engineers more productive, and capable of expressing higher-order concepts without having to worry about the entire stack?

One attempt at an answer: the sorts of problems that companies are focused on solving largely center around CRUD, certainly. We're getting to a point where most of the things people want to accomplish involve finding a collection of rows in a database, and presenting them in a way that makes sense to users. Visual pattern matching, just-in-time data delivery, etc, are all very important to making a successful application. But when all the code you use on the front-end is visible to users (via JavaScript) and the actions people take are generally similar across the board (CRUD), creating a niche relies on largely on branding and network effects.

The original "killer app" on Facebook was (probably) checking out the pictures of cute girls in your classes, and bragging about your awesomeness in an attempt to improve social standing, and not much has changed. They leverage more aspects of homo now, ten years on—the desire to be informed, the desire to feel social without the costs—but these things depend on the network that was bootstrapped via "cool, attractive young people".

And yes, I'm being a bit glib, but the problems that Facebook solves in interesting ways mostly center around speed, availability, scale, and presentation. The CRUD core is kind of… boring?

The company that eats Amazon's lunch, by contrast, will make it easier to actually shop, and not just find something close. This is presentation and filtering layer technology, and Amazon's existence hasn't shut the door on other online marketplaces the way Facebook has largely supplanted all other entrants to vanilla social networking.

Amazon's power has come, in turn, from its infrastructure and its internal APIs. Most engineers, when they hear "Amazon" now, will think more naturally of AWS than of the original online bookseller. Sure, Amazon will always mean books on some level, but they drive so. much. of the web that I wouldn't be surprised if that becomes their core business (just like I joke about Google becoming primarily a car manufacturer).

[A funny sidenote: how many online stores, I wonder, are running on AWS on the backend? Amazon is pretty content-agnostic, even when the instances they're hosting would compete with their original niche.]

Meanwhile, technology stacks are maturing—redis for cache, capistrano for server management, docker for deployment—thus freeing engineers from having to roll their own, so to speak. Interesting companies are getting more meta (sidenote: I really need to play with SquareSpace to see what they enable people to do… but it's possible that they'll kill the indie web dev)… and I don't actually know what the engineers are doing at the rest of these companies? Is it all just gluing frameworks together, whiling away time until the company is aquihired?

And there's something empowering to playing the "Full Stack Engineer" game as long as you can. Outside Project Management, FSEs are among the few who can say they're able to realize a vision, which the parts-and-glue, division-of-labor engineers probably can't.

Other than inertia, then, I suspect a primary reason this industry hasn't settled on a single stack is because as long as there's capital flowing, people are having fun at work—and occasionally making money doing so. If software engineers collectively decided they wanted to change the world in a more meaningful way, they certainly could (or at least could make a good stab at it). However, at that point, the majority of the industry would be consigning themselves to grunt work, the sort of work that mostly appeals to milquetoast personalities with pocket protectors, and late-career people with families and hobbies.

A business idea, then, would be "do whatever would make software engineering boring, the fastest, for the most people". That would pull the rug out from under the entire field for a while, though, until the middle-road engineers could figure out a way to build on top of this new stack and have fun doing so.