Non-reproducible bugs are probably the biggest time-sinks for fast-moving teams.
Even if you assume that an engineer or a QA person finds 3 hard-to-reproduce bugs a week (that’s a conservative estimate), that’s thousands of dollars of time taken away from shipping product capabilities.
It makes all the more sense to invest in a short SOP of sorts when it comes to handling hard-to-reproduce bugs. I’m sharing what I’ve learnt in the past.
Decide which non-reproducible bugs are worth spending more time on
You should surely ignore non-reproducible bugs in less critical paths in your software, but you can’t walk away if you find them in more critical paths. The table below lists my approach to allocating effort - it always makes sense to try your best to reproduce tough bugs in highly critical or medium critical paths, because it definitely means more disruption for your customers (or worse - scary incidents)
Here is a chart on times when you should totally ignore hard-to-reproduce bugs vs when you shouldn’t easily ignore them:
Of course, this is my approach - but I’m happy to hear yours. Do you have a framework that works better - tweet to us @gobeamio :)
Try reproducing the bug and write the steps if you’re able to catch it
Obviously one of the first things you should do is trying to reproduce the bug yourself. I probably do this not on the same browser tab but on another one. The browser tab where the issue occurred could have important console and network logs that might be useful for fixing things.
When trying to reproduce try to think of the possible variables that might have led to the bug occurring in the first place: could be the account or environment, could be the user, could be one specific path in the flow that was different this time that led to the error. This is hard, but this is the only way to deduce what factor led to the unexpected behavior.
If you’re in customer service and a customer is reporting an issue, probably makes more sense to get on a call with them - because your understanding of quirks and nuances in the product must be better than theirs and you can help them capture the right information to enable your engineering team for fixes.
Replay the behaviour on Logrocket or another session replay product
When a hard-to-reproduce bug occurs, you can note the timestamp down, the user details and maybe the environment or account information where the problem occurs. Then, you or your engineers can check the session replay tool your product is embedded with, to replay your activity and understand what happened.
Some teams use Logrocket, some use Sentry, some teams use other products. There’s a whole bunch of session replay products in the market. The only difficulty here is that that you or your engineers will have to take a bit of time to find out the exact session where the bug occurred. Over time, the annoyance and the time taken to locate the right session for every single non-reproducible bug add up and
Record your screen during the QA session
QA sessions last hours. A non-reproducible bug can occur at any moment when you’re on a bug hunt. If your team doesn’t have a session replay product embedded, it makes sense to use something like Beam. Even if they do have something in your product, it often takes a fair bit of time to locate the exact user session where the non-reproducible behaviour happened.
Beam, on the other hand, is a Chrome extension. Once you install it, you can instantly replay the last five minutes of your browser sessions every time you find hard-to-reproduce bugs or behaviour. Once you’ve reviewed the replay, Beam gives you a link with the network and console logs, so you don’t have to spend time exporting HAR files, etc. You just share the link with engineers, and they have everything they need to start fixing the problem.
It’s possible that your organization already provides licenses for tools like Loom. However, you’ll have to remember to launch tools like Loom every time you start a QA session or once you resume after recording a video. This doesn’t sound like a big problem but trust me - I’ve done this and there’s a lot of times I’ve not caught the behaviour that led to the non-reproducible bug because I forgot to resume the recording! Loom also doesn’t capture network or console logs - it’s great for general screen recordings that your training or customer-facing team makes, but it’s not designed for this capability.
Check out Beam instead and you’ll thank us when you run into your next non-reproducible bug :)
Share with your team on Slack or Microsoft Teams
One thing to do is to share details about the unexpected behaviour immediately on your collaboration channels: Slack or Microsoft Teams. You don’t have to push your team for a fix always. If the bug is in a critical path tag it as critical and give some ideas on what you did to reproduce the behaviour. If the bug isn’t in a critical path, mention it.
What happens usually is that other people who might have run into the bug but must have ignored it start participating in the discussion sharing debugging clues, etc. Even a medium priority issue gets fixed this way sooner if you have a team that moves fast.
Add more logs
Without a doubt, it makes more sense to carefully monitor that area in your product where such bugs are emerging. Adding more instrumentation, alerts and logs help get to the bottom sooner.
How do you deal with difficult-to-reproduce bugs?
