When I joined the Amazon EC2 Networking team, one of the services I owned was a critical system monitoring a large portion of the EC2 network. As a part of my ramp up on the team, I asked for a technical deep dive into how the monitoring system worked.
I was surprised to see that the monitor was not a highly distributed service in the cloud. Instead, it was a relatively simple list of cron jobs and shell scripts. Files and reports were FTP'd around. I initially was taken aback. What the heck?!
I foolishly assumed we needed to quickly replace this with a real system. I was biased by the tools I'd worked with in the past. All real systems I'd worked on over the previous 10+ years had been some type of highly scaled service, usually Java. Shell scripts and cron jobs were used for operational things like log rotations, or simple operational tools. Certainly, not a production system!
But soon enough, it became clear that this use of simple technology was actually pretty brilliant. Cron rarely fails. Shell scripts don't have memory leaks. FTP just works, or it doesn't. If you distribute your system onto all your hosts, and it runs in cron, you can't really have an "outage". Essentially, they'd boiled the requirements down to such a simple solution that they almost eliminated the possibility of monitor fail.
There are multiple interesting lessons to be learned, but I'd like to point out one in particular. I'd had a specific set of experience at work, and so I had specific tools I assumed should be used in each circumstance. If I needed a distributed solution, I assumed that would be a Java service. It was simply what I was used to.
Any time in my career after this point, I always had in the back of my mind that a simpler solution might be available. Something overlooked by us all because we were biased by our experience.
I became a better technologist and a better leader as I grew my toolbox. This was just one example of such growth.
I love the idea that all of us carry around a tool box. Those tools are the tips, tricks, skills, and processes we've learned through our life and careers.
For example, a software engineer would certainly have some programming languages in their toolbox, but also their time management skills, their knowledge of Scrum, and the communication skills they were taught by their mom.
Some people have a very short list of specialized tools, and they're very skilled at using them. This might include a mechanical engineer who has been working on designing car hood latches their entire career, or an expert machine learning scientist. Either way, these employees have narrowed their focus to a very specific set of skills, and they're paid for those specific skills.
Others have a broad set of tools. I'd argue that those employees with a broad set of skills have more opportunities for career opportunities, growth, and less risk.
First, let's discuss the various types of tools I'm referencing.
Well, not all of you program, but as an engineering manager, this comes to mind quickly. Various languages have different patterns of how they solve problems, and those patterns help you think in different ways. Particularly valuable is if you have a variety of language types in your toolbox.
C helps you understand exactly how data is being stored. How various data structures can relate to another. It helps you build an understanding of what is efficient and what isn't.
Relational SQL can help you think in objects. How data can be associated with an entity, and when it makes sense to move data to be separate from said entity. In other words, it helps you think through the positive and negative ramifications of saying that an email address is a part of a "customer", or a separate "email address" entity.
Python or Perl smooth over the nuances of syntax and let you think a bit more about logic flow.
Cobol lets you get better at typing really, really long commands. I'm allowed to make fun of it because it was the second programming language I learned.
I don't know what Haskell teaches you, but I've heard good things. Maybe someday I'll add that one.
People managing helps you understand HR and compensation processes. It helps you see your personal problems from the perspective of a manager. It helps you understand the challenges of delegation and auditing.
Project managing helps you understand deadlines, dependencies, and critical path. It helps you get a better view of the various job families, and who they interact with.
Everyone should have an opportunity to provide customer service for their customer-facing products. Every single time I've seen a non-customer service team do customer service work, they've always been heavily impacted. The bugs you were ignoring feel horrific when encountered by a customer. The features you were excited about are not interesting to your customers. The way customers talk about your product is incredibly different from the way you talk about it. It opens your eyes.
Types of technology
Mobile development has fascinating challenges. The place your code runs is out of your physical control (unlike backend development). The devices have inconsistent specifications, could have low battery, and poor connectivity. Helps you learn defensive programming.
Low-level, large-scale development focuses on tiny efficiencies. Every byte of memory can matter. Every loop is a nightmare. Often minimal amounts of code, but significantly more effort is put into each line. For example, the firmware code in routers. Due to the scale and potential difficulty in deployment, high quality is required in all code.
Web development has an opposite advantage. The impact of outages is usually minor, and deployment is usually easy. This means you can practice rapid prototyping. Speed of development is typically a higher priority than in depth testing.
You never know when you'll have an exciting opportunity. Perhaps it's at a company you admire. Perhaps it's an opportunity to lead a new team. You typically encounter opportunities because you have a slight edge over others. How do you get that edge? Have more tools in your toolbox.
For one simple example, I was managing a backend engineering team which built and maintained Java services. Then during our yearly planning, a project idea was approved which required a new Android application. One of our existing engineers had Android experience. Not a ton, but she knew enough to be valuable. She was handed the opportunity to be a founding member of the new team. It was exciting for her, and a great career opportunity.
Even within your existing team, there are often opportunities.
One of my favorite product managers epitomized the concepts of ownership and bias for action. As a side note, epitomized is such a neat word.
I remember that PM Roosevelt trying to get a pet project of theirs done. A lot of the work was on my team, but a bit of work had to be done by a peer team. It was a small amount of work, but work that only the other team could do. Roosevelt asked nicely, begged, offered favors… everything he could think of to get that team to do the work. They kept saying they didn't have time.
As a product manager, a role which certainly doesn't expect coding abilities, Roosevelt opened up their code base, found the small change he needed, made the change, and offered it in a code review.
"Here. I made the change. Please just deploy it."
While that tactic won't always work, it worked in this case. The team, perhaps feeling bad that the change was much easier than they'd let on, got his change pushed to production.