Product managers shouldn't be primarily responsible for testing.
Steps to migrate out of being a manual tester without shunning your responsibility as a person who contributes to building quality software.
Product managers at too many companies are responsible for testing (i.e., manual testing) to ensure software quality when quality shouldn’t be entirely on the PM’s shoulders.
There, I’ve said it! It’s taken me a long time to make this statement. I was wrong when I used to believe PMs should perform more testing. I thought it was good. Learn from my mistakes and learn how to move away from being a manual tester.
Don’t repeat my mistakes
Early in my career, I was a PM at a 50-person startup. Like many startups, we were short of software engineers and there was more work than our team could complete. I also had internalized the belief that PMs should “fill in the gap”, and do whatever was necessary to help the team. In summary:
startup, early stage (values speed over quality)
too few software engineers
too much development work
a culture of must do
In this environment, I felt proud I could pitch in as a manual tester. It was the least I could do and there were benefits. Performing manual testing gave me an in-depth look into the customer experience. It also helped me understand how the software works from back-end to front. I could also build deeper and better relationships with software engineers, who saw me as someone not afraid of getting “dirty”. I also liked learning new technical skills such as reading logs and writing simple scripts. I loved it all. Through manual testing, I could prove I wasn’t just a guy with an idea or spreadsheets, but someone who directly contributed to the finished software that customers use.
But as I developed more experiences as a manual tester, these strengths also became my weakness.
How getting better at manual testing can make you a worse product manager.
As I became a more proficient tester, several things started happening. Mind you, this wasn’t obvious at the time.
Software engineers learn to rely upon you for quality.
If you’re good at a job, it’s only natural for other people to reply on you. Why wouldn’t they? This can create a negative feedback cycle. As software engineers work with you on testing their features, they’ll expect testing is part of your daily job. Soon, engineers will deploy feature branches to staging environments and slack you to test. You become integral to the completion of a development task, maybe even writing sudo-code or code itself. You start thinking about ways to automate your manual testing (i.e., automated testing). When new software engineers join, they’ll look at how work is done, mimicking other software engineers and come to you for their testing needs. There are always more software engineers than PM (or at least there should be). Soon, you’re spending even more time on manual testing (i.e., the negative feedback loop).
Manual testing isn’t scaleable.
It’s not scaleable for you or the product. The more time you spend performing manual testing, the more you’re creating a problem for yourself and the software engineering team. Unfortunately, you don’t see that problem initially, which makes this even worse. Assuming the engineering team grows in size, at some point, you’ll be unable to keep up. This is because you are not a full-time manual tester. Thus, three options present themselves: work even more hours to make up for PM work (there won’t be enough time/burnout), drop other PM responsibilities (bad for the product and your career), or drop manual testing responsibilities (bad for the product quality in the short term).
Good manual tester forestalls the need for software-driven automated testing.
In the beginning, there are few features so manual testing could cover all test scenarios with reasonable quality assurance. However, this is the time when building automated testing is easier. But hey, the engineering team has you, an awesome PM. Besides, software engineers could spend their time building additional features, which you’re likely to champion. By the time you become overwhelmed, well, there are now a lot more features or more complex features. Building an automated test to cover all, that’s now a harder problem and it’s going to take a lot longer.
What to do if you’re in this position?
If you’re spending at least an hour daily performing manual testing, you’re already in the hole. What do you do? While everyone's situation is different, there are four specific tips I can give.
Show people the problem, don’t tell.
Show people, especially the software engineering team you’re working with the problem you face. Record yourself conducting a manual test. Make someone watch how long it takes to create test data and enter everything manually. Count the hours you’re spending. Show people why this repetitive task is reducing your productivity. Even better, ask another software engineer to conduct the manual test duty for a sprint to help reduce your workload so she/he can empathize with your pain.
Reduce new development
Early, I wrote that being a great manual tester delays the work necessary to build automated testing capabilities. Well, that debt is calling to collect. The need to build automated testing is going to eat into the software engineer team’s capacity to build new features. This is a problem that shouldn’t only be allocated to a single person. You need to dedicate focus because you don’t want to drag out this problem, which does mean reducing new feature development. People won’t be happy. In fact, they will be angry. But there is no way around this. The analogy I give is: “If you haven’t been exercising for years, starting up is hard and you can’t just go run a marathon. Still, you have to start slowly, but be consistent. The benefits will come with time even if it fills painful exercising.”
Redirect your energy elsewhere and help people adjust to a new PM.
As the software engineering team focuses on automating testing, you’ll free up time. What do you do with it? You’re going to have to dust off or develop those PM skills that have atrophied. When was the last time you joined or conducted user research with a customer? How about performing a product teardown of a competitor’s product? Or spent time breaking down L1, L2, and L3 metrics? Work here might not only have stopped or never been started, but people might not even be aware of the purpose or value of this work. You’re going to have to redirect yourself while helping other people understand the purpose and value of this work. Don’t overwhelm yourself and others. Instead, pick one specific area (e.g., customer interviews) and start doing that work consistently.
Set expectations on how much and where you’ll perform manual testing.
People don’t deal well with sudden changes. It’s unlikely you’ll be able to stop conducting manual testing overnight. But there are two steps you can take to help reduce your time and effort spent as a tester. First, identify how many hours you’re spending conducting manual testing. That’s the current state. Next, define what’s an acceptable target and this target state shouldn’t be zero hours. Remember, you can and should still contribute to software quality. I think it’s acceptable to spend 1 - 2 hours per week testing. Then, define how you’re spending your precious time testing. Are you testing the same feature repeatedly? If so, why? Are you testing new features for final acceptance? If so, when are you doing this? You should minimize time spent performing repetitive testing. If it’s repetitive, even such as conducting a product release sanity test, it should be automated. Focus your time on new and novel (e.g., new features, unusual test cases, new experiences).
Does the above resonate with you as a PM? Do you disagree or have a point you’d want to elaborate on. Leave a comment by sharing your experience.