How to Increase the Agility of Your Organization, Improve Quality, and Reduce Costs
How to Increase the Agility of Your Organization, Improve Quality, and Reduce Costs
Slowing down deployments to add QA is viewed as trading off improved quality for slower speed. I'm going to convince you that it also lowers quality.
We've gotten a crazy amount of snow here in the eastern suburbs of Seattle. I grew up in the Chicago suburbs (and went to school in Northern Michigan), so I'm no stranger to snow. But we're not ready for snow out here. Few plows. No sand or salt tossed onto the road. Many more hills. It swiftly becomes dangerous to move anywhere.
On the other hand, we can appreciate the few days of snow, and marvel at how beautiful things look. It's a nice alternative to our normal dim and wet winter days.
If you're interested in coaching, I have blocked off some days for family travel, but I still have times available now, and early in January. And if you can't find a time which works for you, as always, please let me know!
I hope you are having a lovely week, had a great Thanksgiving, and have all your necessary gifts ready for the Christmas holidays. 😊
How fast does your team / organization / company ship changes into production?
Most modern startups emphasize lightweight automated testing which pushes changes into production quickly. As those companies grow, they begin to change their processes. Many larger companies have added layers of testing and change management, which slow down their path to production. These process changes are usually built over the years as reactions to negative customer impacts.
Your process influences the actions you take. A slow or fast release process is not just about the cadence of releases. It has massive repercussions across your teams, and how you do your work. Tools and processes don't work in a vacuum. They create and encourage certain behaviors.
"it’s not the tool’s fault, it’s how you use it.” That’s a cop out. Tools encourage default behaviors, they dictate patterns and golden paths. - Jason Fried
An implication.
Let's take a step back, and think through the implications of tools and processes changing our behaviors.
Imagine that you have a process which encourages your employees to work as many hours as possible. You verbally reward employees who work long hours. You give them additional compensation, and promotions.
After a year or two, your HR group points out that your employees have trended towards a much higher percentage of young males. Particularly people at higher levels. Those employees tend to be the ones without a family, or with a spouse at home to take care of things.
Everyone is concerned, so they create anti-bias training. They encourage the recruiting team to focus more heavily on women. They fund women in engineering non-profits, create affinity groups, and have regular meetings to review their metrics.
What are they doing? They're fighting against the results of their processes. The system is not biased, but it drives biased behaviors and patterns. They can fight against the results of the system they generated, but it's a costly battle. They're swimming upstream.
The alternative
When you find yourself constantly battling the same problems, you likely have a process or tool creating those problems. You can continue to exert effort to patch the problem, or you can change the root cause of the issue.
In this case, we're referring to issues created by having a slow release cadence. What are the issues? Here's a couple to start:
Bugs can exist in production for a long time. Releases have a high chance of having a major bug, and it's hard to tell which change created each bug.
Your work-in-progress (WIP) is large, and it takes forever to get data / feedback on it from customers.
The above concerns are common at many companies. How are they often addressed?
More manual and automated QA, to reduce bugs getting into production.
More customer surveys and user research to get data pre-launch.
Both of those solutions (I'd call them patches) are employed, and they do slightly mitigate the pain of slow release processes. But then you have a bug reach production. And that added QA makes it take even longer to get your fix into production. And your user research takes a few weeks to run, so you pause your UI work while waiting on the results. It's a spiral which gets worse, as your product gets more complicated.
Instead of patching the symptoms, you need to address the root issue.
How do you fix the bug situation? How do you fix the customer feedback loop, and reduce your WIP?
Launch more frequently. Ok, I'll admit, that feels obvious. But considering the release cycles of larger companies, it's an obvious proposal that many don't follow.
Why?
Launching more often is hard.
I worked with some brilliant engineers at Amazon. I still ran into mental barriers when discussing fundamental changes to existing processes. Here's a simplified version of a conversation I had more than once.
Me: "How often are our releases right now?"
Engineer: "Every 6 weeks."
Me: "Let's cut that down."
Engineer: "I think we could get it to 4 weeks, if we had more people working in parallel on QA."
Me: "What would it take to get it down to 1 week?"
Engineer: "Impossible."
Fundamental changes require a different mindset than incremental changes. They require taking a big step back from your current process to address what is possible, and what it would take to get there.
Before get too wrapped up in how to get there, let's talk about why we'd want to head in the quick release direction.
Fix problems faster
If you run into an issue with a major release, it can take serious time to deploy a fix. You may require many steps of manual testing and deployment steps before you can push out a bug fix. If you miss the train of the next release, you may be waiting literal weeks before a small fix could be deployed.
On the other hand, if you push changes out at the granularity of individual code changes, your fixes can be in production in the same hour you make them.
As this decreases the impact to customers, it decreases the amount you need to avoid those impacts. It becomes less of a big deal to let things go into production when you can fix them quickly.
Smaller problems shipped to production
If you bundle a long period of development into a release, you may break whole features. Long-term branch development is notorious for creating major issues.
Larger changes are naturally more likely to be visible and impactful.
Conversely, the smaller the changes you ship, the less impactful they're likely to be.
You don't want major impact when you launch changes. You want minor continual impacts. This spreads out the impact to customers, spreads out the impact to your operational load, and reduces the chances of any major negative events.