So you’re doing quite well in your career as a software developer. You’ve proven your technical chops several times over. You always deliver the work you’re asked to do. Then, sooner or later, the question comes:
“Are you interested in taking on a management role?”
You think about a little. But you dislike office politics. You loathe meetings. You abhor menial administrative tasks (managers do a lot of those, right?).
Even worse, you think to yourself, you can barely stand working for “the man”. No way you could be the man. The man cares more about the “business” than the quality of technical work. The man is out of touch with technical work. And he definitely cares more about the business than the development team.
No, no, no. You’d be betraying everything you stand for. No way you could do that.
“I’d rather stick to the technical track,” you respond. And that’s OK, you’re told, because your company, like other cool companies, has a “dual ladder” system. You can keep getting promoted, keep having more influence, keep making more money, as an individual contributor (IC).
I’ve seen this scene play out over and over again. And sometimes it works out.
Sometimes it doesn’t because the company is lying about the dual ladder. The managers have (or create) a chummy system that works to their advantage. You’re not in the room when important decisions are discussed or made, but they are.
On the other hand, sometimes it doesn’t because the developer believes they’re exempt from “management-like” tasks. “I’m not a manager, so I don’t need to sit in meetings, I don’t need to worry about influencing other people, etc”. I’d like to talk about that a little bit, because it’s a common mistake I see developers make in their career.
Just because you decide not to become a manager, you’re not exempt from technical leadership.
Technical leadership is broader than management. Most modern software development is not done in isolation. It’s a heavily collaborative, cross-functional discipline. The “people” are necessarily a part of the technical system, often times even more important than the code (and definitely less deterministic).
As you progress in your career as an IC, you will need skills like the ability to communicate with people on your team and outside your team, the ability to influence others, the ability to coach and mentor others, and so on. And as you progress further, you will need to think about what makes your team (or your company) effective or ineffective. These are all skills that are required of managers, but are still very useful for ICs. A manager without these skills will probably fail — an IC without them might just have a stunted career.
I’m not saying you need to spend all your time in meetings, or that you need to learn to play corporate politics. But I’ve seen many ICs who are so turned off from anything management-related that they throw the baby out with the bathwater. So don’t shy away from skills that might seem “managerial” just because you’ve decided to grow on the technical ladder — rather, invest in them!
Another (maybe more cynical) reason to not totally throw out leadership is because your company will have people processes, and as you progress, you will more often be on the receiving end of them. Recruiting and hiring? You’ll go through that process. Performance management? You’ll go through that process, too. You’ll witness good management and bad management. You’ll have to manage up. You should understand those processes, why they are designed the way they are, their strengths and weaknesses. And every process has weaknesses.
So just because you’re not a manager, you’re not exempt from leadership. When people come together to achieve something, leadership is always required. Those with more experience, with more capability, must provide it. Even if their title isn’t “manager” and they don’t have people formally reporting to them.
You have the potential to provide leadership, and that potential implies an obligation to use it.
If you’re reading this, chances are your work involves dealing with funnels. Funnels are any flow with several steps where the goal is to move a person forward in some process.
Common funnels you might encounter are:
Marketing funnels: The classic example is customers start with awareness (of what you’re trying to sell), then move to interest (in what you’re selling), consideration, all the way through making an actual purchase.
Product funnels: Getting users on a website or app to take a sequence of successive steps, like signing up and onboarding.
Hiring funnels: You want to hire people for your company, starting from some conversation or job announcement, through to extending an offer, getting it accepted, and the person joining your company.
This article is about one of the biggest mistakes I see people making with funnels: force-feeding. It’s part of a two-part series (in this part I’ll talk about force-feeding and it’s effects, and in the second part, I’ll talk about how to avoid force-feeding).
But before we talk about funnels, let’s talk about ducks (or just skip past the next line break if you’d rather get to the funnels).
Of all the cruel and creative ways humans have found to farm / exploit, gavage is one of the most interesting. Gavage (French word) is a way of force-feeding ducks. A tube is stuck down the duck’s throat, by-passing the beak, esophagus, etc and food (corn and fat) is deposited directly into the duck’s crop.
Now, if you’re the human doing this to a duck, this is a marvelous exploitation. You can increase the weight of the duck and create wonderful delicacies like foie gras to eat or sell. It is a great short-term win for the exploiter.
But, if you’re the duck (or a person who cares about ducks and other living things), this is atrocious. The core input mechanisms of the digestive system (beak, throat) are bypassed, and perhaps more importantly, the appetite is bypassed, too. The duck has no say in the matter. By being force-fed quantities and types of food it wouldn’t normally eat, it becomes obese, its liver swells to an unnatural proportion, and it develops a whole other slew of detrimental physical and psychological effects. I’ll spare you the rest of the gory details, but, needless to say, a duck being force-fed must be slaughtered within a certain window, otherwise it will die from the effects.
Let’s get back to sales, marketing, hiring, and product funnels. Too many people practice gavage with their funnels. They hijack and degrade their funnels by force-feeding them in unnatural ways. This can result in short-term wins, but sooner or later, the chickens come home to roost.
I’ll first talk about how this happens, then I’ll talk about the subtle effects.
Let’s assume we’re tuning a product funnel that involves a subscription-based product:
User sees an ad for the product
User clicks the ad
User lands on your landing page describing your product
User signs up
User pays for the subscription ($9.99 / month)
(some time later) User renews their subscription
Between each of these steps, some subset of users drop off and abandons the funnel, and you can calculate a conversion rate between each step (ie what % of people who see an ad click it, or what % of people who sign up end up paying).
Let’s say that initially, your landing page mentions your pricing. Someone on your team designs an A/B test for a new landing page that doesn’t mention your pricing. The A/B test is positive! A higher percentage of people who land on your landing page are now signing up! Launch it, right? Celebrate!
Clearly, you can see the problem here. By withholding information from users, you’re kicking the can down the funnel. What will probably happen is that your down-funnel conversion rates will drop (ie more users who are now signing up, are not willing to pay, and conversion from sign-up to purchase will drop as soon as they see the price on your paywall).
Now, there’s definitely a balance here. Often, mentioning information like price too early, before you’ve built interest can deter users who might have paid had they had a better understanding of your solution. So timing is critical. But the point is, if by withholding information, you’re simplistically seeing a higher short-term conversion rate up-funnel can be misleading.
The same goes for information like salary or benefits in a hiring funnel. Too many companies and candidates keep that discussion until the end, finding out they’re miles apart after spending a large amount of time and energy.
Reducing Funnel Alignment
In general, withholding information is one way of reducing funnel alignment. Funnel alignment is probably worth its own write-up, since it’s a common mistake people make, but at a high-level, it means that at each step in your funnel, the people making it to the next step are aligned with previous steps (and ultimately, what is offered at the end of the funnel).
Funnel alignment works in two modes, usually depending on how deep a user is in the funnel. In the early stages of your funnel, lack of funnel alignment creates friction. For example, let’s say your product’s ad says: “A modern budgeting tool for the thrifty spender”, but when a user clicks on the ad, the landing pages shows a picture of a family and the text reads: “Sign up to secure your family’s future”.
Now, there are products that can help you accomplish both those things: spend less today and save for your family’s future. But a person who clicked on the first ad may not have a family. Or, they might, but now you’ve created enough ambiguity in their mind about what the product does that they go elsewhere. The two steps are not fully-aligned.
In fact, you can often get a sizable increase in conversion rates just by doing a quick scan of your early-funnel alignment. Does the ad text match the landing page message? Does the landing page message match the calls-to-action?
On the other hand, deeper down the funnel, lack of funnel alignment is less about friction and more about promise delivery. Did your product promise users that they’ll effortlessly save hundreds of dollars by using it? If not, a big chunk of your users may churn after realizing the truth. Did you promise a candidate you just hired a role that doesn’t end up materializing? They’ll probably leave at the first chance they get (and while they’re there, they probably won’t be motivated to do much of a good job).
Removing Good Friction
Growth hackers love the idea of removing “friction” from a funnel. Friction can be cognitive (you’re making someone think too much) or physical (they have to take too many steps).
In general, friction in a funnel is usually a bad thing. Too many steps or effort in your funnel, and you will lose users who might ultimately convert.
But friction can also be good. This is, again, worthy of its own write-up, but here are a few mechanisms by which “good friction” in a funnel can work:
Collecting information (that might be crucial to engaging a user later)
Strengthening resolve/motivation (e.g. reminding a user of certain benefits of a product)
Positive energy (ie celebrating/congratulating the user)
You can often have a step in your funnel that combines several of the above mechanisms (e.g. asking a user to choose their top goals from a list collects information, strengthens resolve by focusing them on the benefits, and provides them positive energy by telling them they’ve just embarked on a path to getting those benefits. In some sense, it also provides information: the list they’re choosing from gives them an idea of the product’s supposed benefits).
So think twice before just cutting steps out of your funnel. It might seem like an easy win, but sometimes it backfires.
Lowering the Bar / Removing Qualification
In many funnels, you’re not just trying to help someone decide to choose you (or your product, company, etc), you’re also trying to choose someone. Funnels are often two-ways.
In the sales world, this is referred to as lead qualification. You have limited bandwidth to spend on leads, and so you try to determine, as early as possible, which leads are potentially going to convert to customers so you can focus your time on them.
In the hiring world, on the other hand, a good interview process performs the same function. A good interview process cuts out candidates who will not be a good fit as early as possible in the process to avoid wasting their time and your company’s time (“a good fit” could be either someone who’s not qualified for the job, or someone who’s really unlikely to accept your offer).
When pressed, sales teams (or hiring teams) will often lower the bar and perform lax qualification to push more leads (or candidates) down the funnel.
The Downsides of Force-Feeding Your Funnel
We’ve talked through a few force-feeding techniques. But what are the downsides?
Firstly, if your funnel requires resources to run, by force-feeding your funnel you’re wasting those resources. This one is obvious, and we discussed it in the qualification section above.
Secondly, you’re often fooling yourself as to the positive effects you’re experiencing. You might see better immediate conversions, but down-stream, the metrics will likely rebound and you’ll see lower conversions. And downstream metrics might be harder to measure, since they have longer feedback loops.
Often, “funnels” extend far beyond what people consider. For example, a hiring funnel isn’t just about hiring someone. It extends after a candidate joins, through their onboarding and into their employment. Hiring the wrong people will result in more attrition, but you’ll probably only find out months later. A product funnel extends after a user uses or purchases your product. They may not be retained later, or they may give you poor word-of-mouth that deters other users.
Finally, you might experience funnel corruption. Your funnel is so over-stuffed that it ceases to be a proper funnel, and causes you to make poor product decisions.
Let’s go back to that subscription product we talked about earlier. Let’s assume the company building that product engages in some drastic funnel gavage. Now, a product manager decides to launch a new feature, and, being data-driven, watches the data closely to decide on success.
That feature might be something like a new periodic email from the product with some updates. The PM expects that email to increase engagement, driving more people back to the product. She launches the A/B test. To her dismay, there’s no engagement lift. In fact, the email is driving unsubscribes and subscription cancellations! This is due to funnel corruption.
Funnels give you feedback, but that feedback is biased by the people you send down your funnel, and if your funnel is corrupt, you’ll be learning the wrong things. In this case, the people paying for the product don’t really want it, and now, every time you try to engage them, instead of getting a positive reaction, you get a negative one (because you’re reminding them about your product, which they’re paying for, but not getting value of).
Funnel corruption can destroy your product, because you end up learning the wrong things and taking the wrong actions. You start optimizing your product for the wrong people
I’ll talk about solutions in the second part of this series. Stay tuned.
One thing people often get wrong about software work is stakeholder management, and one thing people get wrong about stakeholder management is minimizing surprises.
This is actually pretty simple. There’s a design principle known as the Principle of Least Surprises (or Astonishment), which generally states that the behavior of a product should not surprise users. Surprises are bad.
The same principle carries over to project execution and stakeholder management. Generally, any project you work on will have stakeholders:
Colleagues you work with directly. If you’re an engineer, this might be other engineers, product managers, designers, etc.
Other teams that have dependencies on your work (ie your work must get done so their work can get done).
Other teams on which your work depends (ie their work must get done so your work can get done).
People who are inexplicably (from your perspective) excited about or worried about the work you’re doing.
You’ll be a lot more successful, and build a lot more trust, if you minimize surprises with these stakeholders.
Form a mental map of who your stakeholders are, what sort of information they’d want to know and what cadence, and make sure they’re never surprised. For example, it’s bad if your work is delayed, it’s much worse if that’s a surprise. It’s impossible to create zero surprises, but you should be creating near-zero surprises.
And keep on the lookout for unsolicited “checkins”, which can be a leading indicator that people are worried you might surprise them. This isn’t always true, sometimes people just need a status update or a piece of information, but if you’re communicating well and proactively, and people trust you to deliver, the volume of unsolicited checkins you get will be very low.
The most effective engineers I’ve worked with were effective not just because of how they wrote code, but because of things they did when they were not writing code. Here’s my (growing) short list of things you should do before or after you write code.
Understand what you’re building. Do you have specs or requirements? Are they clear? Do you understand what users you’re building for and how they’d use what you’re about to build?
Understand how it might need to change in the future, so you can design for adaptability. Are there any assumptions you’re going to make now that might hinder your ability to adapt your code in the future or expand your functionality.
Understand why you’re building it. If you’re building it, there should be value to your users and the business. What is that value? Do you agree with it?
Understand why build it now. Presumably, there are other things you could spend your time on. Why is this important right now? Is it creating more value than other options? Does it unlock other dependencies that require that this be completed?
Have a plan for how you will build it. Depending on the complexity and the stakeholders, this plan could be a couple sentences you run by a team-mate, like “I’m going to have to modify our data model, which will involve a migration, and change one API call.” For more complicated changes with many stakeholders, this might involve a Request For Comment document, or a fully-fledged Design Document to discuss with your team and other stakeholders. There must be some thought. If you can’t put it in words, you’re not ready to put it in code.
Think through side-effects. Will you stress certain parts of your infrastructure? Have to refactor a part of the code base?
Develop a mental map of stakeholders. Who might you need to check with or communicate with as you build? Your product manager? Designer? Executives/leadership? Someone else on your team who’s irrationally excited about this feature and has been championing it for months? Anyone who would be surprised by any your work, positively or negatively, is a stakeholder.
Note that you might need to do some “pre-work” to figure out some of the above, like #5 and #6. That’s fine. But treat that work as “pre-work” not as actual work.
I’m a big believer in follow-through. None of that “get it done and toss it over the wall crap”. This may sound like it’s not your responsibility, but you should always own the quality of what you’re building.
Instrument some metrics / quantitative results. Depending on your product and number of users, this might be very simple or very complicated. But you want quantitative numbers that show how many people are using what you build and how.
Gather qualitative information. Ask to watch someone use what you built. Use products like FullStory.
Do your own QA. Even if you have product managers, QA testers, etc, own your own quality. Figure out what the testing plan is and decide whether it is sufficient or not. Test it yourself. Dogfood it. Ask to roll out incrementally.
Get feedback from users and stakeholders. Have a list of what was successful and what wasn’t. What can be changed in the future? Have a list, even if it’s not going to get worked on now.
You’ll notice as you do these things more and more, that your work will stand out. People will value working with you. The quality of your work will improve. And most of all, you’ll build better and better judgment for how to do your future work.
What can you do, as a builder, if your job seems to require to employ disrespectful design patterns?
What is Disrespectful Design
First, it’s worth rehashing what Disrespectful Design is. I think there are a few common dimensions:
It’s adversarial. It puts the needs of the product (or the company building the product) clearly ahead of the needs of users, and tries to get users to do something. This post about Reddit has a bunch of examples of that, but in general, you’ll find lots of adversarial designs around sign-up flows or invite flows in social products.
It caters to System 1 over System 2. It uses dark patterns that cater to people’s automatic minds, their instincts/emotions, over their reason and logic. Vanity metrics that appeal to the ego and make users feel good through short dopamine hits. Infinite feeds that keep users mindlessly scrolling without being aware of how much time has passed.
It favors short–term gratification over longer term benefits. You might feel better after while or immediately after using the product, but it probably has some bad long-term effects.
It is paternalistic and often dishonest. It claims to know what’s best for users better than the users themselves. My all-time favorite is: “We track you and invade your privacy, so that when we show you ads, they’re more relevant to you!” but it’s generally more like “You’ll have a better experience if you do X, so we will trick you (not convince you) into doing X. You don’t know what’s best for you, we do.”
Not all products that have these characteristics constitute Disrespectful Design, and there’s a bit of I know it when I see it to it. For example, even products that help people improve over the long-term like fitness apps or meditation apps might employ some of these tools to help people meet their goals. But these apps won’t have all these characteristics.
Why it Happens
A combination of things have overlapped to encourage Disrespectful Design.
The proliferation of “free” products. As the saying goes, if the product is free, you are the product. Free products are often monetized using your attention, your data, and your actions, so they need to get as much of that as possible from you so they can show you more ads or try to get you to take some other action.
An overly “data-driven” culture. You can very easily A/B test your way into a very disrespectful product. You try something, it juices your metrics, so you do more of it. The data becomes a crutch, and often, the things that are more measurable are short-term, shallow metrics.
“Performance”-driven culture.“We’re a meritocracy. If you move the needle, you’ll get rewarded”. Put that together with a data-driven culture, and suddenly you have a lot of builders finding ways to juice short-term metrics so they can get that next promotion or that bonus.
Grow-at-all-costs mentality. No doubt that the venture-capital-backed start-up model has created a lot of success, but it does push companies to a “grow-at-all-costs”, “winner-takes-all” type of mentality, where the key metric is growth and everything else be damned.
What You Can Do
So as a builder, if you find yourself working on a product that employs or plans to employ disrespectful design, here is what you can do:
First, decide what you’re OK with. There’s a spectrum here. For me, making a call-to-action in a flow more prominent in a way that makes it easier for users to find is probably fine. Changing it so users mistakenly click on it? Probably not.
Then, try to quantify the damage. Sure, maybe the short-term metric you’re looking at improved, but take a more holistic view. Look at down-funnel metrics. Maybe that sleazy sign-up page converts more users at that stage, but does it create better-engaged users? Unfortunately, the answer is often yes, but not always. For example, apps like Quora and Reddit that block their mobile browser UIs (or make them crappy) as a way to entice users to download their app do piss off a lot of users, but ultimately, convert enough people to the (much more sticky) mobile app that the metrics make it look like it’s worth it.
Look at long-term metrics. Maybe across the funnel, disrespectful design works in the short-term. But long-term, are you building a healthy base of users? Long-term metrics are, unfortunately, really hard to test. By definition, you’d need to wait a long time to see the results of an A/B test. And, during that period, you have extra product and engineering complexity of having multiple active flows. Finally, in some products, it’s really hard to fully isolate effects between control and treatment populations for an A/B test (ie can you fully track an anonymous user over time / devices / locations? can you prevent effects on one population from leaking to the other population? etc).
Next, some negative effects of Disrespectful Design can’t be quantified, but they can be observed. A company that has a reputation for employing Disrespectful Design might have its ability to hire affected. For example, while Reddit definitely has a really talented team, you can’t look at a thread like this and argue that it’s ability to hire more talented people isn’t negatively affected. You can’t easily put a number on that effect, but if you believe that the people you hire are one of the key determinants of your success as a company, hiring brand is a crucial ingredient.
Finally, know when you’re ready to walk away. Life’s too short, your time and talent too valuable. If you’re not comfortable spending your time and energy on a product because of the way it’s being built, and you can’t change things, look elsewhere. It’s hard to walk away from a well-paying job at a well-known company, but it’s also hard to put a price on that feeling of putting your head down, at the end of a long day, and feeling proud of the work you’ve done.
A lot of people talk compare good code to poetry. Poetry is generally:
Elegant and enjoyable to read.
Of variable information density. Sometimes, a line of poetry can contain multiple meanings. Sometimes, it’s more verbose.
Subjective and potentially ambiguous. Different people reading the same poem might interpret it differently.
But often, good code should be more like an essay:
Of the right information density. It exposes important information and hides unimportant details. Not too verbose, not too dense.
Well-structured and layered. Different layers will expose different levels of detail.
As clear and objective as possible. There is one clear interpretation of what’s happening.
As simple as possible, but not simpler.
Often written and re-written.
If you’re writing hobby code that only you will read and not maintain, write code like poetry. Make magical, whimsical, and fun. Make it clever. Write what flows out of you.
If you’re writing code that others will read and maintain (and as a hint, “future you” is generally an other), write code like an essay. Don’t make it clever or artistic. Make it simple, clear, and easy to read and understand.
It will be read far more often than it was written, and the person reading it probably won’t be looking to enjoy your “art”. They’ll just want to know what the hell is going on so they can get on with their day.
In general, there are two ways to be good at something. You can be born with it (be “a natural”), or you can learn how to get good at it. The mistake I see a lot of people make when they want to get good at something is to find “a natural” and ask them for advice. The problem with that is naturals typically can’t actually tell you how to get good at something if you are not, because by definition, they were never not good at the thing they’re good at.
In that vein, I am not a creative person. I consider myself a pretty solid engineer, manager, and occasional product person, and have generally excelled at tasks that require logic or other structured thinking. Creativity, on the other hand, has always been a struggle for me. I’ve always looked on with admiration (and envy) at friends and colleagues who were naturally creative, but I had been able to get by without being creative.
But as my career has progressed, I’ve found that creativity has become more and more important, especially as I started leading teams. In particular, I needed creativity to solve problems effectively. More complex problems required more creativity.
So here’s my list of how to generate creativity if you’re not a naturally creative person. There’s a common theme, which I’ll get to at the end, but first, let’s talk about the tactics.
Ask multiple people to solve it independently. You and your team agree that you will go off and independently try to solve the problem, then regroup1. Force people to write down their thoughts. Have two or three people write design docs before you implement a system.
Ask multiple people to solve it together. You and your team do a “brainstorming session”. Initially, you cast the net wide: any idea is fair game, and no one can shoot down an idea no matter how crazy it sounds. Then you can start to narrow down the space of possible solutions.
Constrain the problem. How would you solve the problem if you only had one day to solve it? How would you solve it if you wanted to do it without any additional cost or head-count? If you didn’t have to worry about existing product and technical debt?
Un-constrain the problem. How would you solve the problem if you had effectively endless time, money, and people to throw at it?
Invert the problem. Supposed You Wanted to Kill a Lot of Pilots. What if you wanted to achieve the opposite of your intended solution? Does that generate a set of solutions to avoid, or a set of solutions that might be adjacent to them? Are your perceived fears not as bad as you thought?
Change your surroundings. To break out of a mental pattern, sometimes you have to break out of a physical one. Travel can help, but even things like spending time in nature or just in a high-ceiling environment can alter your thinking enough to generate creativity.
Move. Exercise or walk.
Sleep on it. Sometimes a problem just needs more time, and often, when you disconnect and stop actively thinking about a problem, you give your subconscious enough room to interject a solution or your conscious enough time to forget whatever parts of the problem are causing a mental block. Take a long shower. Sleep on it.
Ask people who have solved similar problems. How did they solve them? What other solutions did they consider? What did they learn? Is your circumstance similar to theirs?
Ask people who have solved un-similar problems. Especially in other domains. Just choose a random person who has solved problems successfully in other domains, give them a little bit of context, and see what they say.
Why they work
Ultimately, creativity involves two competing forces—generating solutions and choosing solutions. System 1 dueling with System 2, or the elephant sparring with the rider, or even just “right-brain” vs “left-brain”. Part of your mind evolved to form abstractions, trying to simplify the world around you to efficiently generate conclusions, while part of your mind evolved to think, ideate, ruminate, and even obsess (it’s what makes us human).
The abstraction-forming part of your brain constantly constrains the world and the set of possible solutions to any problem and tries to minimize your intellectual effort. This can hinder creativity. Solutions are eliminated prematurely, and most of the time, that part of your brain is right. But creativity requires a little delay to that process. Ideas that your brain may not consider at all due to preconceived notions, or ideas that your brain would shoot down right away might require a little nurturing before they’re ready to be sized up.
You can also view this through a probability/statistics lens. For example, having multiple independent brains work on a problem, is, in a sense, increasing variance (since the input variables are more independent), and with creativity, you want more variance. On the other hand, while brainstorming together makes the input variables more dependent, it can also increase variance if it creates room for interactions (often non-linear) between the input variables (ie people building on each others’ ideas).
Do it iteratively
Conventional wisdom is often to do the broad, un-constraining activities first, then work to constrain and limit. I find it’s better to do it iteratively. You can still start with un-constraining activities, but it’s not a funnel, it’s a loop or a ping pong match.
As an example, my company recently parted ways with a marketing agency because they weren’t delivering results, but we still had to generate marketing phrases for our product. We staged a friendly, inter-company competition. Everyone came up with an idea for a phrase, and we tested them in ads. The best-performing ad variation performed significantly better compared to our ex-agency’s best performing ads.
But you need to be careful with this one. You don’t want a destructive intra-team competitive dynamic about who’s better at coming up with solutions. Your team needs to have a high amount of trust and you need to reinforce that even when brainstorming independently, the goal is to come up with the best solution as a team.
It is costly to have incompetence in your organization. This is obvious. Someone is not performing or delivering at a high level, your product or service suffers, your business suffers.
It is also obvious that as a secondary effect, your team suffers. Other people have to work harder to correct mistakes and pick up the slack. Everyone has less pride in their workmanship.
The thing is, those costs are easily correctable once the person responsible for incompetence departs (willingly or unwillingly) or corrects (gains the will or skill required to do their work correctly). They’re not structural.
There are more subtle hidden structural costs of incompetence, and what they look like depends on where they occur in the organization.
High-level incompetence, at the managerial or executive level results in excessive subversion. In a healthy organization, people trust management to make management decisions. And, as much as I hate top-down hierarchical management, the role of management is to have a broader, more strategic view than any person or team in the rest of the organization. When managerial incompetence occurs, managerial trust is lost. Management loses its ability to direct, nudge, or otherwise “manage”, and becomes ineffective. Employees usually still know well enough to tell managers what they want to hear, but they won’t actually follow-through effectively.
This becomes a structural problem that can’t easily be resolved by changing a manager or two (or even the entire management team). People become subversive, and management has to resort to some combination of carrots/sticks rather than relying on intrinsic motivation and alignment.
Low-level incompetence, on the other hand, results in over-management / micromanagement. This creates a culture where driven, independent-minded employees have one of two choices: either stop being driven and independent-minded, or leave. Which results in more incompetence, and more over-management. This is structural too, in that it is self-reinforcing. The manager’s instinct is now more command-and-control, more management, more deadlines, more check-ins, etc. But that’s the opposite of what’s actually needed.
This is why it is critical to rectify incompetence as quickly and effectively as possible, at any level of the organization.
Of course, this entire post would be lacking if I didn’t clarify that management is always ultimately responsible. Management has the ability to hire, fire, reward, and punish, and so any incompetence is management’s responsibility.
Most modern software teams have no choice but to spend a lot of effort on their infrastructure, often spinning up entire teams to do so (and sometimes placing their best engineers on those teams).
How did cloud infrastructure get so complicated? Why is AWS such a hot mess of insane configurability and complexity? I ponder this a lot. As a manager of software teams, any minute not spent on building an awesome product kills me a little.
I have a few thoughts.
For a minute, back in 2008-2010, things looked different. Heroku had just been launched, and with a few clicks in a GUI and a git push you had your code deployed and accessible.
Fast forward a decade, and now, we’ve got containers, Kubernetes, so-called “serverless” lambdas. Heroku is still around, but most people outgrow it pretty quickly.
Instead, we end up in the complex world of Amazon Web Services, Google Cloud Platform, Azure, etc with their hundreds of services, each with a tremendous amount of configuration and complexity.
Now, maybe you think I’m exaggerating, because you’ve already spend the time necessary to learn/navigate one of those platforms and/or you enjoy that type of work. But any effort spent not directly delivering value to users is wasted on some level. Infrastructure should be simple, it should just work. But it doesn’t. Even simple tasks like estimating what your infrastructure costs might be are a chore on some of these platforms. There are literal venture-backed startups whose business model is to use reinforcement learning to tweak your infrastructure configuration.
Instead of getting simpler, infrastructure got more complicated. Why?
I have a few theories. The first is creator’s bias. AWS, GCP, etc were all born at FAANG-ish companies, who are outliers in terms of the size of their company, their data, their code base. They build tools they would use themselves. But why would 99% of the world’s engineers need the same complex tooling that the biggest software companies in the world need?
It actually gets worse. Because those companies are viewed as (and often are) technologically ahead, the open source community models its software after them (or they contribute tools back to the open source community). Borg inspires Kubernetes (actually born at Google), Mesos, and Docker. MapReduce inspires the Hadoop ecosystem. And so on.
The second big reason is competitive dynamics. The big cloud providers are locked in an existential competitive battle for “the cloud”. This creates a dynamic where more is better. You’d think that competition in markets would lead to better products, but for enterprise or business-to-business products, often it just leads to an arms race for more features and more complexity. More services, more configurability, and ultimately, more complexity. This is how you stay ahead of the competition and how you win the big contracts. Meanwhile, the average developer is just gasping for air.
The third is Heroku’s decline. Heroku could have been so much more, but instead, they were acquired by Salesforce. Instantly, they lose their ability to innovate and become that classic story of “large company buys startup, startup product stagnates / only gets incremental improvements”. And in this case, Salesforce isn’t exactly a beacon of simplicity when it comes to product (it’s a great product, it’s just clunky and enterprisey… just less so than the competitors it displaced). There are a few bright spots like https://render.com/ which I’m keeping an eye on that could pick up where Heroku left off.
I hope this changes. I respect each of the big cloud platforms as businesses, and know that they have some of the smartest engineers working both on them directly, and building software on top of them. But this is all just tremendously inefficient.
Something interesting has been happening as I’ve been trying to write more about engineering management.
When I wrote advice about micromanaging for managers, a few friends asked me about how to deal with their (micro)manager, so I wrote about how to handle your manager. The latter piece seemed to be a lot more useful. I also wrote about how managers should avoid cognitive biases, and most questions I got were about engineers who felt their managers were victims of those biases. You see the pattern. I write for managers, I get interest from ICs.
It could be that I‘m reaching the wrong audience, but I think there’s something deeper going on.
If you work in tech, you’ve heard of the “dual career ladder”. As engineers grow, they can choose to climb the “technical ladder”, or they can climb the “management ladder”. In fact, companies you interview with will often brag about it so much, you’d think they were the ones who invented it! In reality, like most things modern Silicon Valley thinks it invented, the dual ladder has been around for decades. I found mentions that imply it was first made explicit by post-World War II DuPont (which, if true, would put the concept alongside other notable DuPont inventions like nylon, Kevlar, and the concept of ROI — more on the nylon part, in particular, later).
What’s the purpose of the dual ladder?
Avoiding the Peter Principle. Engineering and management require different skill-sets, so if a technology company always “promotes” its best engineers to managers, it will end up converting great engineers to bad managers.
Attracting, retaining and motivating engineers. If engineers feel like their only path to growth is management, and aren’t interested in it, they might not join, or they might leave, or just stay and not be that motivated.
It’s commonly said that managers of technical teams should understand technology so that they can be effective. It helps them build credibility, coach their staff, and make better technical decisions. I’d like to propose a broader claim: Regardless of which ladder you choose, you should understand the skills needed to ascend the other. Just like managers should understand technical issues, engineers should understand management issues to be effective. In the rest of this piece, I’ll mostly make a case for why this is important, but in future posts, I’ll go deeper on a few selected topics. But before that, let’s do a quick jump into the past.
Remember when we said we’d get back to DuPont and nylon? It’s hard to imagine how the world would be today if DuPont had not been able to offer a brilliant young scientist an opportunity to spend his time doing inventive research on polymers.
Wallace Carothers was teaching Chemistry at Harvard when DuPont approached him and offered him a job as a lab director. He initially declined. It wasn’t that Carothers, who was described as both brilliant and melancholy, enjoyed his teaching job. He actually seemed to dislike teaching. But he just wanted to do research, and had been dealing with bouts of paralyzing depression. He was worried about the industrial pressures of working at DuPont. Nevertheless, DuPont persisted, offering double his salary at Harvard, and explaining that at DuPont, while he would direct a lab, he could spend most of his time inventing. He would write of his days at DuPont:
“Nobody asks any questions as to how I am spending my time or what my plans are. Apparently it is all up to me.”
Carothers’ work led to the invention of both neoprene and nylon, which revolutionized plastic and were used in everything from clothing to military equipment. In fact, nylon became known as the “fiber that won the war” due to its contribution to the Allied victory of World War II.
Despite his success in the lab, Carothers continued to suffer from depression, and tragically took his own life at the age of 41, years before his inventions really took off. But his research was world-changing. It doesn’t seem like DuPont had formalized the “dual ladder” system yet in the 1930’s, and technically he was “director” of his lab, but if DuPont hadn’t offered him money, prestige, and a chance to spend his time doing research and solving technical problems, the world might be a very different place.
Let’s take another example. Meet Arthur Fry, credited with invention of the Post-It Note.
In 1974, Fry was a researcher at 3M, on the technical side of their dual ladder. Frustrated with trying to have bookmarks in his hymnal stay put, he used a unique adhesive developed by Spencer Silver, a fellow chemist at 3M to stick, unstick, and re-stick small pieces of paper, inventing the Post-It Note.
He would go on to get promoted to “corporate researcher”, the highest rung on 3M’s technical ladder, and stay with the company for many more decades. If it wasn’t for the dual ladder, he later said, he’d “have left the company and gone into business for [himself] as an inventor, or joined some small company where they give you a piece of the action.” Instead, he got to spend his time at 3M “staring at the wall or getting [himself] educated” while still “making as much money as a vice president”.
In other words, companies have been finding ways to attract and retain technical talent by providing them with opportunities that don’t force them to become managers or administrators. Of course, the “binary” dual-ladder is a little restrictive, so a lot of companies have more of a “wide ladder” (where there’s a spectrum of management vs. technical contribution) or a “jungle gym” (where mobility between the different types of roles is allowed or encouraged).
Most companies have realized that managers need some understanding of the technical side of things, lest you end up with pointy-haired Dilbert-style managers.
On the other hand, many individual contributors (ICs) don’t necessarily feel the need to understand things on the management side. ICs might feel like management is irrelevant to them, or they might have a cynical view on management. Here’s why I think that’s a big mistake.
Understand management because you need those skills
The software companies of today are a far cry from the industrial R&D labs of DuPont and 3M. Most current software development is more of an engineering discipline than it is a science. Sure, managers and ICs have very different day-to-day activities: managers spend more of their time in meetings and talking to people, and ICs spend more focus-time on technical issues. But, most ICs are not working in isolation in some lab in search of a moment of brilliance that will change the world. Modern software development is a heavily collaborative, cross-functional discipline. The brilliance comes in increments and iterations.
As you progress in your career as an IC, you will need skills like the ability to communicate with people on your team and outside your team, the ability to influence others, the ability to coach and mentor others, and so on. These are all skills that are required of managers, but are still very useful for ICs. A manager without these skills will probably fail — an IC without them might just have a stunted career. I’m not saying you need to spend all your time in meetings, or that you need to learn to play corporate politics. But I’ve seen many ICs who are so turned off from anything management-related that they throw the baby out with the bathwater. So don’t shy away from skills that might seem “managerial” just because you’ve decided to grow on the technical ladder — rather, invest in them!
Understand management because it shapes your system
By system, I mean the people processes in your company or team. You’re likely to be on the receiving end of processes designed primarily by managers. Recruiting and hiring? You’ll go through that process. Performance management? You’ll go through that process, too. You’ll witness good management and bad management. You’ll have to manage up. You should understand those processes, why they are designed the way they are, their strengths and weaknesses. And every process has weaknesses.
Let’s zoom in a little on a topic like performance management. Early on in a start-up’s life, the primary goals are execution and survival. Many start-ups don’t have the time, luxury, or experience to think deeply about performance management too deeply. As an engineer, you work on whatever needs to get done, and if you don’t know how to do it, you try to learn it, and through that, you become a better, more knowledgable engineer. If you’re not doing what the company needs you to do, it’s pretty obvious to you and everyone around you. If you continue to not do what the company needs you to do, you’ll probably be a victim of the “fire fast” mantra. Time horizons are short and feedback is swift but chaotic.
As the company grows and matures, things start to change. The company hires (or transitions) managers and “People Ops” folk, who have the experience (or at least the time and mandate) to think about performance management of people on their teams. The company’s time horizon stretches, which makes planning for the future easier. Someone might propose some structure: a 360-degree performance review process run at some regular cadence (once or twice a year). But how does the company evaluate people? Someone proposes a rubric (or “ladder”, maybe even a dual one). How do you reward people? Someone proposes a compensation policy.
Manager A notices that Manager B is a little more “generous” when evaluating members on his team. She proposes a calibration process, to try and maintain standards and fairness across different teams. After every performance, managers hole up in conference rooms to discuss their proposed evaluations and try to maintain consistency.
These processes differ from company to company, depending on the company’s size, maturity, culture, and overall philosophy. But as an engineer, you will be on the receiving end of one of these systems, with its pro’s and con’s. Even if your primary goal isn’t to get promoted, it’s still useful to understand why your company designed (or stumbled into) that particular process. What dimensions are being evaluated and why? Do managers have autonomy to make their own decisions, or are decisions made by impartial committees?
So how can you learn more about all this?
One way to do this is to just ask your manager. Why do we do “calibrations” as part of performance management? Who decides who gets promoted? What are the dimensions you evaluate people against? Sometimes those processes are well-designed and work as intended, but sometimes they’re not. In either case, you should try to understand them.
I actually love it when someone on my team show interest in these things, and value the feedback I get when I explain the reasoning behind some of the processes we’ve built.
I’m going to be writing more about each of these issues, but I’m going to try to do it a little differently. Firstly, I’m going to try to gather multiple perspectives from others, since different companies approach these issues differently and have learned various things about what works and what doesn’t. But more importantly, when I write, from now on, I’m going to try to write with a perspective from both ladders.