Despite my half-vow last week to try to write more, it doesn't seem to have worked out that way. I only consider myself to have missed a single day (Friday), but that's enough.

Thursday I left off having torn out my user auth and dropped in devise; I was getting frustrated with it by the late hour I left, whenever that was. I finally got it working today, and even though it's one of those things that's really powerful once it's up and running, there's so much going on with rails and devise metaprogramming that it was hard to pin down where the weird behaviors were coming from.

Long story short: getting ajax sign-in working is highly non-trivial in devise, because it has a lot of expectations for how you'll be using it. There's no easy way to shove the user's account data down the pipe when logging in via ajax, and it's not clear from the documentation what's breaking.

It turns out, though, that you can call respond_with with a :location argument, and rails does not respect that unless the :format is html. So when trying to debug what appears to be a straightforward statement, using the suggestions written by the devise authors, you're led to fixate on a couple small lines of code that actually have nothing to do with your problem.

This is the double-edged sword of using a framework. Once you figure out enough of the gotchas to get things moving, you get a lot of reliability and power for free, but you're forced to either use things how they intend you to (in the given examples and configuration options) or hack up enough of the framework to bend it to your will.

Oh, well; that's out of the way. I may have to generate a toy project using straight rails and vanilla devise authentication to see how it's supposed to work, because I'm not entirely happy with what I've got now. That seems to be the way of this latest phase of my final project: I've been adding things and discovering the limitations of a lot of the more robust options out there, and it's annoying.

One of the things I added to my portfolio recently is a set of progress bars representing my knowledge of different tools. I was reluctant to do this, because any progress number seems quite arbitrary, but I'm willing to venture that one thing that represents knowledge level is when you begin to get frustrated with a tool.

In fact, I'll put this forward: if we're all being honest, able to use something while using the documentation might be 10% knowledge; 20% would, then, be, able to complete simple tasks without reference to the documentation. 30% might be the point where you start saying "oh, this is neat!". At 40% you feel at ease; 50% would be the point where you have the documentation open, but you don't rely on it. By 60% you forget to even open the documentation, but at 70% you've got it open again because you're trying to do something that the tool just doesn't make easy. At 80% you're saying "oh, for god's sake"… at 90% you're patching the tool.

It's rough, but it's something. With this framework, my resume is a bit of a lie, but it's close to accurate so I suppose I can leave it. And by this measure, I'm at 80% with some things I wrote as 70%, and 40% on some 50%s.

I'm okay with this, in the end.

I've just started to read Corman's Algorithms, and in the first chapter a thought occurred to me.

There was a recent post about how inter-connected the Facebook codebase is. First of all, I found it quite surprising how much code there is, but I assume a good chunk of it is for scalability, which is certainly important (it's basically a mystery to me how facebook works at all, given that there's no way their user database can exist as a single object on a single machine).

However, thinking further, and thinking about the sorts of companies I see out there, it seems like there are a lot of people solving the same kinds of problems. And between one thing and another, I started wondering: how much of this is because our languages for expressing common problems haven't matured?

There's a cap on median engineer intelligence (it doesn't really matter what this means, but for the sake of argument, let's say the median engineer working in web technologies has an IQ no higher than 130, or two sigma above the population). Engineers are in a weird spot inasmuch as they work in one of the few industries where your job is very similar to your training and tool-building. A teacher can take classes on being a better teacher, and make resources to make teaching better and easier, but at the end of the day if they don't get in front of a classroom, they're not doing their job.

An engineer, on the other hand, can write a framework to make their work easier, learn (or create) a domain-specific language to speed up one aspect of their job that's repetitive, and at the end of the day have in hand new knowledge, work that pays the bills, and a tool they can release for other developers to use. This probably goes a long way in explaining the galapagosation of technologies, especially web technologies.

At some point, the learning curve of understanding how everything goes together becomes too challenging. Thus, our median engineer can't get much more efficient, because there are too many things to keep track of, and too many things that can break, and he wants to understand his toolchain and not look a fool because he can't tell his boss why the database got borked.

The thing is, we already have specialists (Database Engineers, SysAdmins, Front-end, Back-end, Project Engineers, the list goes on), and the role of "DevOps" kind of papers over the fact that at some scale, a company should be concerned with differentiation of responsibilities. All engineers should have some idea of how their work might affect other domains, but the job of optimization should lie in the hands of experts. (Note here Conway's Law, which states that "organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations".)

Is it possible to make a framework which would make engineers more productive, and capable of expressing higher-order concepts without having to worry about the entire stack?

One attempt at an answer: the sorts of problems that companies are focused on solving largely center around CRUD, certainly. We're getting to a point where most of the things people want to accomplish involve finding a collection of rows in a database, and presenting them in a way that makes sense to users. Visual pattern matching, just-in-time data delivery, etc, are all very important to making a successful application. But when all the code you use on the front-end is visible to users (via JavaScript) and the actions people take are generally similar across the board (CRUD), creating a niche relies on largely on branding and network effects.

The original "killer app" on Facebook was (probably) checking out the pictures of cute girls in your classes, and bragging about your awesomeness in an attempt to improve social standing, and not much has changed. They leverage more aspects of homo now, ten years on—the desire to be informed, the desire to feel social without the costs—but these things depend on the network that was bootstrapped via "cool, attractive young people".

And yes, I'm being a bit glib, but the problems that Facebook solves in interesting ways mostly center around speed, availability, scale, and presentation. The CRUD core is kind of… boring?

The company that eats Amazon's lunch, by contrast, will make it easier to actually shop, and not just find something close. This is presentation and filtering layer technology, and Amazon's existence hasn't shut the door on other online marketplaces the way Facebook has largely supplanted all other entrants to vanilla social networking.

Amazon's power has come, in turn, from its infrastructure and its internal APIs. Most engineers, when they hear "Amazon" now, will think more naturally of AWS than of the original online bookseller. Sure, Amazon will always mean books on some level, but they drive so. much. of the web that I wouldn't be surprised if that becomes their core business (just like I joke about Google becoming primarily a car manufacturer).

[A funny sidenote: how many online stores, I wonder, are running on AWS on the backend? Amazon is pretty content-agnostic, even when the instances they're hosting would compete with their original niche.]

Meanwhile, technology stacks are maturing—redis for cache, capistrano for server management, docker for deployment—thus freeing engineers from having to roll their own, so to speak. Interesting companies are getting more meta (sidenote: I really need to play with SquareSpace to see what they enable people to do… but it's possible that they'll kill the indie web dev)… and I don't actually know what the engineers are doing at the rest of these companies? Is it all just gluing frameworks together, whiling away time until the company is aquihired?

And there's something empowering to playing the "Full Stack Engineer" game as long as you can. Outside Project Management, FSEs are among the few who can say they're able to realize a vision, which the parts-and-glue, division-of-labor engineers probably can't.

Other than inertia, then, I suspect a primary reason this industry hasn't settled on a single stack is because as long as there's capital flowing, people are having fun at work—and occasionally making money doing so. If software engineers collectively decided they wanted to change the world in a more meaningful way, they certainly could (or at least could make a good stab at it). However, at that point, the majority of the industry would be consigning themselves to grunt work, the sort of work that mostly appeals to milquetoast personalities with pocket protectors, and late-career people with families and hobbies.

A business idea, then, would be "do whatever would make software engineering boring, the fastest, for the most people". That would pull the rug out from under the entire field for a while, though, until the middle-road engineers could figure out a way to build on top of this new stack and have fun doing so.

Today: clearing out the backlog of things that I wanted to do on my homepage.

You know, until AppAcademy, I hadn't seen much purpose for a homepage. I remember the days of the late 90s, when everyone had a homepage with "About Me" sections that (let's be honest) no one really cared about, and then some collection of things that they cared about, like spacecraft of the Federation. (There was one guy who had a homepage dedicated to the Performa 6400, and that I used for years to diagnose issues.)

I may do something similar on my own page in time, but for now it's just a pretty good hub for everything I want people (who might want to hire me) might want to know about me.

In trying to put my best face forward, I've been customizing my hosting. Among the NEAT things I discovered today:

• Bare git repos, and the hooks directory in a git repo, that allows you to auto-deploy files (for instance)
• I have control over all components of my chosen theme, which means I can debug broken scripts (damn designers!)
• It's easy to override the designer's css, especially when he barely uses classes.
• It's easy to keep my version of given plugins up-to-date in order to use the most recent features.

I know, nothing here that's ground-breaking, but there were a couple times today that I found myself exclaiming, "neat!", if only because I figured out a way to get something closer to exactly what I wanted. … well, within a certain definition of "exactly". There are a lot of things I would do differently if I chose to build my theme from scratch, but given the constraints of using Wordpress and (most of) a pre-baked theme, it's pretty cool.

I may still SASS my page up, since SASS is cool, and choose to redo major components of the design (e.g. the font-family), but all-in-all, it's close to where I want it to be to represent me.

One thing I discovered: I haven't been nearly as consistent at posting here as I thought. The calendar widget at the bottom of the page has a LOT more gaps than I'd have believed at first. I shall try to rectify that. One change is that I intend to post more stuff that may not be public up there (here), including blog posts that don't have anything to do with AppAcademy. All in due time.

A summary of the majority of the postings I see:

• We want a full-stack engineer
• With skills in our exact tech stack (modulo one or two things)
• With five years experience

I'm not so stupid that I believe everything they say—with work that's important enough and with enough money coming in, they would find a way to make a non-ideal candidate work—but there is no way that there are enough engineers out there able to fit these criteria.

In any industry (or endeavor) there's going to be some "limiting factor". In chemistry, for instance, you generally try to use excess of your cheap reagent to try to get the highest yield possible of your most expensive reagent. However, one has to wonder, where are all the jobs posting asking for engineers with 10+ years experience? (Note: I'm not looking at the "senior" positions, but even if those are looking for people with 10+ years experience, where are the 15 year positions?)

This is the factor that makes me think, especially, that "5 years" is just a shorthand for "we don't want dummies". I doubt there's any industry where dead weight is as dangerous as it is in software; if you can get your foot in the door elsewhere, you can have a career as a mediocre … whatever… but there's not a lot of room for mediocre in software. If you can't understand the standard processes of abstraction, problem solving (etc), there's not much room for you in software.

However, the career progression for engineers is a bit shady. Salaries go from (slave wages of) about $45k to a healthy median of $80-90, up to about $130… and then what? I know one guy who was a project manager and made a very large salary, but I don't hear a lot about a lot of others.

That's an interesting thought, though: is the next level of abstraction above software engineer project manager? The thing that is more abstract than writing large projects is coordinating teams that write large projects, perhaps?

Back to the point, though: no one wants to bear the responsibility for training, and no one wants to bear the risk of getting that one guy who just will never add value to the organization. So I assume, at this point, that the entry point for engineers is a convoluted process of signalling and trials: internships, pedigree, networking, taking chances on people who seem otherwise intelligent.

So far there's nothing novel here. I'm curious, though:

• Is there an overall model that would be able to take advantage of the (presumable) under-tapped pool of future engineers with aptitude?
• Or, is there a model that would pull more people into the industry and train them to produce more?
• Is there a language (paradigm?) that would make programming accessible to more people, or are we already close to peak utilization of programming-capable people in the population at large?
• If so, is there a language (paradigm?) that would take the finite quantity of engineers and make them more powerful?

What's the skill cap on an engineer?

After a weekend-long breather, I'm searching for work in earnest as of (let's be honest) 11 this morning. Twelve hours later, I left the office, feeling not significantly better than on arrival.

I did a little work on my final project, but nothing of significance. I honestly thought of implementing authentication using Devise—it seems a lot more robust than my home-rolled solution—but I quickly recognized it for what it was, a way to put off taking the next step. Maybe I'll prioritize that down the line, once my app is more feature-rich and I have applied to a few more places. There are a lot of things I can see I could do better; using jQuery's modals in place of Backbone.Modal, for instance… but really, I just want this app to feel like a complete app. In time.

The job search is… well, it feels like the last couple times I've gone looking for work. There are a lot of companies whose missions are practically opaque, and a couple dead-ends—companies that were bought out or maybe even went out of business since they were added to whatever list I'm looking at. But overall, the feeling I get is that there are a lot of companies that are funded to the level they are simply because it's easier for someone to throw a couple mil at a lot of long shots than to get in early on a few sure things. I do wonder about the status of venture capital: are there really that few interesting problems left, that all there is to fund are yet another X?

And yes, if any future hiring manager bothers to read this blog, I understand that I'm pissing on the hand that might want to feed me, and I'm sure there are all sorts of interesting problems that I could find at any employer, but it at seems kind of … pointless? Everyone's doing the same things, at least on the surface, and their marketing copy doesn't do a good job of differentiating them from the competition. Hell, at least half the sites I've seen are using templates or libraries I've seen elsewhere, and it's just like… do you even believe in yourselves? I'm not sure anyone cares how many redbulls you drink per week at your office, and the fact that all your testimonials come from the same thirty day window, eight months ago, says a lot about how dynamic your business is.

Sure, writing copy about yourself is hard—this is something that I'll have to face, on a personal level, in a couple hours or days—but isn't there something meaningful you can say? Or are you all, like the company that specializes in popups on outbound links, just plain evil? I wonder what their conversion rate is?

The strangest things is that I actually believe in marketing, I just don't see a lot of firms that are targeting what seems to be meaningful, reasonable, or sustainable avenues in that area. It's much easier now, with cloud hosting and quick launch templates and tools, to generate a respectable (looking) online presence, and as such it's not as clear now when a company is two bros and a scam compared to the late 90s. And I'm sure that a lot of it has to do with a (relative) lack of creativity and a (somewhat) saturated marketplace; the next real game-changer will probably come out of left field, where no one else thought to look.

Great, now I'm thinking of how to make a better marketing platform. I have some weird ideas, but nothing of any substance; they're all fleeting, like ghosts. Does it merit thinking about? At this point, with basically thirty days of expenses left, probably not. But the money is not going out of advertising any time soon, and I don't have the knee-jerk revulsion for it, conceptually, that others seem to.

I don't mind finance or marketing or mobile app development, per se, but there's just so much crap.

Back to sifting tomorrow.

Well, it's the day, and we have an hour until people should start showing up for project day… and I'm just sitting here, blogging.

I spent the day thinking about what I want to say about my project, and how I'm going to present, and refining my resume. I finally have enough final project under my belt that it's not purely embarrassing, but I know that there's so much more work that I can do before I can consider my project "functional".

On the plus side, my resume is much nicer now, and I am much more comfortable with SASS and @media queries, and it looks nice both on screen and in print. But there is never enough time to do what I want.

Case in point: we have to set up for demo day now. Whelp.

When you're styling… the whole world, styles with you…

Today was devoted to making my app not look nearly as terrible with a whole lot of bootstrap + sass. I guess I can put those on my resume; they're easy enough to use, and (especially) .scss is nice because it's compatible with .css.

This is kind of a last-ditch: there are so many things I need to do, want to do, with my app, and time's up—tomorrow is presentation day. I'm kind of cobbling together a spiel about what I've actually spent the last month doing, which is to say, learning a framework from end to end (two, if you consider Marionette and Backbone to be separate entities), understanding something of asynchronous calls, and getting everything to play nice.

It's hard to believe it's been a month; there's truly never enough time to learn what you want, and everything takes many more man-hours than you think. Am I faster than others? Slower? I can see what sorts of work others have been able to produce in three months (since we started at a/A… crap, time flies) and I don't think I'm that far removed from my peers. I can certainly field a good set of questions about what I've done, but I can't say that I'm markedly better or worse than anyone.

What other metric can I use? I know I keep coming back to my successes relative to those around me; I might be able to find more hours in the week to work, but at the end of all of this, I can't say that anyone is far superlative to anyone else. I know what I'm learning, and every day there was someone who was able to accomplish more than I was, but in a class of (now) 23, that's bound to happen. If anything, the biggest problem I have, consistently, is having expectations that are much too high…

Does that mean that I will learn more, faster, than others? Or does that mean that I will know more esoteric knowledge, but never have as much to show for it? I've never worked with a meaningful production system, but I suspect that there will be far fewer surprises there for me than for some people… but should I have been spending more time learning things with the most marginal utility, instead of learning the details of a few things?

More questions than answers, and I suspect I'll never truly know. What I do know is that it hasn't stopped being interesting, and that I don't have any plans to slow down in the near future.

So, back to the topic at hand: I bootstrapped the heck out of my app, and thereby got rid of a couple major styling issues, but there are a lot of things that still need to be done. I'll really need to do triage tomorrow, especially considering that I need to do things like prepare paper copies of my resume (which looks… acceptable, at least) and, perhaps, get a domain for my heroku app.

(20 minutes later… done)

Well, I'm a little closer to being ready to hire. *gulp*

One of my hangups about continuing work resolved itself over the past couple days. I ended up having a couple of redundant checkins—"Fixes data model", "Data model finally fixed"—but I think I'm there now.

Collections have models, and certain models, like Posts, have collections of their own (in this case, comments). When you pull the data from the server for a non-root page, the data needs somewhere to "hang" so as not to face sync problems. So, when I fetch a post, I shouldn't fetch the class it belongs to, I should instead examine the post's section_id property, and then put that in an existing class's post collection. … Got it?

User -> Sections Collection -> Section Model -> Posts Collection -> Post Model

So, on fetch of the last item in that chain, I need to go to the head of the chain and request that this data goes in the proper place in that chain.

It took me a while, but I think I ironed out all the kinks. It took moving some logic from the models into the collections: if you ask a Post what section (class) it belongs to, it asks its collection, which in turn had that property set upon initialization. (I also needed to start initializing new collections with (null, options) instead of ({}, options) because it turns out {} is a model on its own, just one with no attributes… derp.)

What does this mean? A whole bunch of crazy sync problems disappeared all at once, and I can finally see the light at the end of the tunnel. I have never truly had a Minimum Viable Project (MVP), and I'm getting to the point where that feels possible, and likely to happen soon.

I don't deal with stress well. I like when I know what I'm doing and when things are working well, but struggle to overcome impasses.

I don't know if there's a known thing called "coder's block", but I have, in the past, pooh-poohed writer's block. There's always something to write, in my head, and if I have problems I just read other people's work and get ideas there. However, I've never written a proper story, so I don't think I can properly state that I know what writer's block is like.

On the other hand, I've been dealing with coder's block for the better part of a week.

One reason is that I'm really not excited about the CRUD aspects of this project. I thought I'd be able to plow through much more easily than I have, and while I now am happy with the framework I've developed, the knotty puzzles are pretty much behind me (for now?).

I spent some time cleaning up other things, and there is some value in those. I've learned quite a bit—from playing with Bootstrap to getting familiar with Sass, solidifying my knowledge of Javascript fundamentals, and so forth—and I've put some polish on things that don't really matter? I'm consolidating, but there's a line where you realize that your "polishing" bears a strong resemblance to "work avoidance".

So, you can spend time doing other things, but you can't, because the work you really need to be doing isn't going anywhere. All this stuff would need to have happened in time, but I can't say that it needed to happen now.

By contrast, though, I was spinning my wheels last week. I'm not so much anymore—I've gotten more actual useful work done on my project in two hours today than in any eight last week. Why is that, I wonder?

One of our cohort has a job offer! Neat.

I spent the past two (and a half) days getting up late and working later. I got nervous that I have absolutely nothing to show with regards to portfolio work and decided to do some mini-side projects that I thought would be good examples for my portfolio.

The first of these was cleaning up Rails Lite, which I'm pretty happy with. When we were doing it in class, I was frustrated because of how fragmented the lessons were, but once I consolidated my code a little bit everything made perfect sense. When controller instances inherit from the ControllerBase class, they gain all the powers that it has, including auto-routing and rendering of default templates… it's actually quite clear, and quite neat how it all works. The inheritance tree is also much clearer, now, so I can see that parsing of input parameters and session management happens in quite the sensible manner. It's truly improved my understanding of how Rails works.

One neat side-effect: When I told Tommy that I wanted to include Rails Lite in my portfolio because it made more sense to me, he and I talked for a bit about how rails is organized and it piqued my interest about the project codebase. So I started reading through the source on github, and ended up identifying a bug in the source that will prevent the current cohort from being able to complete the assignment on that class day. (Long story short: the way the project is designed, it will work with version 4.1.x but not version 4.2.) It was interesting to be able to dive into the code like that and know what I was looking for, and I feel more capable for it.

Late Thursday, I started in on a project that I knew I wanted to do: modifying my Snake (game) code to instead implement Tetris. Besides being (I believe) the better game, it was something that I think could set me apart a bit on my portfolio, and it was fun to see JavaScript in a different light now that I'm more versed in the language. I've finished it up now, for the most part, after quite a bit of late-night logic-checking (pieces no longer get stuck in the sidewalls! Yay!) and I'm quite happy to say that I can be proud of my code in a way that I wasn't for Snake. It is my project and it works well and looks like what I envisioned when starting out, more or less.

Curiously, I got the code 90% right straight out of the box when doing the initial implementation. I started it up without the rendering code, and after making a small fix (ensuring that the IIFE was starting correctly), I could step through the game logic and see little boolean pieces falling like they should. It was pretty empowering to have that be the case; writing (mostly) bug-free code with few syntactic errors is a powerful feeling.

I added both projects to my resume, which has been cleaned up a bit, and I'm feeling (almost) ready to go full-in on the job hunt. I know I'm late to the game, but that's quite alright. This break has done me well, and if I can do that with as much determination as I've been able to muster the past two-three days, I should be fine.