Getting Control of those Pesky Bugs

Some teams seem to have a very hard time managing their bug process.

Even teams who implement good processes sometimes have setbacks when budgets get tight and trade-offs are made. In large and small companies, often management pushes for more features so sales has “new stuff” to sell and bug fixes go by the wayside. Sometimes the code itself is not good due to a rush to get something out-the-door. Teams often end up with a huge technical debt: piles of bugs that need fixing; code that is brittle and hard to maintain.

The team I’ve worked with for 15 years has implemented great processes and tools to stay ahead of the game. Through the years we’ve learned what works and what doesn’t. Here’s a list of do’s and don’ts to stay ahead of the “Technical Debt” debacle.

How to avoid Technical Debt in the First Place

Avoidance is the best policy. It’s really not that hard. Here’s what you need to do:

  1. Code it right the first time. There is no such thing as a “working prototype”. Prototypes end up being the real code. They were just implemented using shortcuts and temporary coding techniques and will end up becoming a huge technical debt.

  2. Keep it simple. Don’t build functions that are more complex than needed. Some teams get too creative trying to build elegant and esoteric software that instead ends up hard to maintain with many features that are never utilized by clients. Don’t use rocket science if you aren’t sending up a rocket. Keep it simple.

  3. Incentivize developers to find their own bugs. Make it part of the company culture that developers do their own unit tests instead of assuming that QA and automated testing will take care of finding any issues. First, bugs are cheaper to fix the earlier in the development cycle that they are found. Second, developers who are trained to produce quality code instead of lots of code will end up producing products with less technical debt.

  4. Never release code with “known issues”. If your company is used to putting out release notes that include a list of problems/warnings for the clients, you’re in trouble. Contrary to some manager’s belief, it is not only possible to deliver bug-free code, it’s the only smart thing to do.

    Let’s quantity that. That does not mean we need to test to the level of standards that are required for space software that can’t be repaired in space or for medical tools which, if they error, will cause human harm.

    It means testing to the level that is commercially acceptable and fixing what is found with a reasonable amount of test cycles. Yes, the clients may still encounter some new issues based on different configurations and usage. Fix those in the next maintenance release.

    But we never, never find problems during testing that users will encounter which will impact their work and deliver anyway just to meet a schedule. If we continue to hit those kinds of decision points, it means we are not scheduling enough time for adequate test and related fixes.

  5. Think long-term. Besides thinking about the functions and new features to be available with this release, think about how your clients will use the system long-term and build your software so they can easily implement and then upgrade from one release to another. This was overlooked during the 20th century by most vendors, particularly the big ERP solution providers. The big gorilla ERP software vendors made a large proportion of their profit from services: Implementation and upgrades. I call that spending money on no-value-added services. I think that is wrong and should/can be avoided.

    In this century, with the advent of cloud computing, software vendors are having to figure out how to make it streamline and simple to upgrade. Finally. But if you are building on-premise software or your systems that need to be integrated with other systems and periodically upgraded, think about ways to make it simple for your clients. It will save you a significant amount of bug-fixing time in the long-run.

    If you don’t have the developer expertise to do that, I recommend you look at building your software on a cloud platform like Salesforce that provides upgrade functions and features with the platform.

  6. How to dig yourself out if you have Technical Debt

    What if you find yourself in a group with significant technical debt? If you’ve decided you need to get rid of your huge pile of bugs, you may find yourself spending inordinate amounts of time and effort sifting through the bug calls, listening to client complaints and trying to prioritize which bugs to fix first.

    Instead, start with a process and stick to it. The best way to get out of your hole is to stop digging.

    First you’ll need to decide if you need any major re-architecture projects. If you’re an IT team, you may need to look for a vendor solution to replace your in-house tools. Look for one from a company with good processes and clean software.

    If the problem is a huge pile of unaddressed bugs and customer issues, here’s a good approach:

    • Gather together a cross-functional team with representatives from your services team who represent the client, the developers, product management/marketing and QA.

    • Take one pass through your open bug list pile, mark each one with three identifiers:
    • Severity: How bad is this bug?
    • 1 – Critical. A client’s production system is down or data is being degraded. Must get fixed ASAP.
    • 2 – Serious. The bug is causing or will cause issues for clients.
    • 3 – Non-Critical. This is a cosmetic issue or else buried in a place where users are very unlikely to go.

    • Priority: How soon do we need to fix this bug?
    • 1 – High. Needs to be fixed ASAP.
    • 2 – Medium. Shouldn’t sit forever, fix when we can.
    • 3 – Low. Fix when there’s time. No big rush.
    • Priority and Severity are normally not coupled. Normally a Severity = Critical will also be Priority = High unless, perhaps, the data degradation isn’t occurring because the client hasn’t turned on some related function. Otherwise there’s little linkage. Even a Severity NonCritical bug could be Priority “High” if it is a misspelling that makes the company look bad, for example.

    • Release: What release do we assign it to?
      This step assumes that, in addition to doing your regular enhancement releases or agile iterations, you are putting out regular maintenance releases. The bug fixes that are not associated with new enhancements should be assigned to release “piles”, leaving some bandwidth in each for unexpected issues.

      Assign to upcoming releases based on a combination of Severity and Priority. Lower Severity and Priority items can get put into a maintenance bucket to be worked off after the more important ones are done.

      Why? Because if these are not assigned somewhere, the team continues to thrash through them, again and again. If they get assigned, even if some adjustments are needed, we have a sense of when fixes can be made available for the user.

    • Hold weekly or bi-weekly SDRB Meetings. Once or twice a week, a cross-functional team should meet to review all of the new issues. This should be the highest-level person responsible for the product area. For example, the QA Director, VP Engineering and/or product lead, Director of Product Management, etc. When our company was small, even the CEO sat in periodically. We call this team the SDRB (Software Development Review Board) and they have complete power for making decisions.
    • Together we decide if they are marked with the right Severity and Priority.
    • Together we assign them to a specific release.
    • There. Done.
    • Instead of continuing to look at them and thrash through them again and again, they are ready for developers.

    • Train the Developers in the Process:
    • Only work on bugs assigned to the current release.
    • Never, never change the Severity or Priority or Release of a bug – only the SDRB can do that.
    • If you have a question, assign the bug to the product manager to answer it. It’s helpful if your bug tool captures comments so the product manager can respond and assign it back to the developer.
    • Note in the tool what you did to fix the problem. We also put our bug number in the code itself so later we can tell why changes were made.
    • When the code is checked into the code repository, the bug is marked “resolved”.
    • When it has been tested, the bug is marked “closed”.

    It’s best if you have a good bug tracking tool that allowed you to change assignments and ask questions, capture all comments, and record history. At my company we used the in-house tool SD Tracker. That tool was the impetus for building Software 2020.

    Regardless of what tool you use, everyone needs to use it and it needs to be easy and effective. It can be done with spreadsheets – it’s just a lot harder.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s