How to Set Up a Bug Tracker That Actually Finds Bugs

Who This Is For

If you've ever typed "bug tracker" into a search bar, you already know the problem: too many options, too many features, and too little guidance. This is for the developer who's outgrown a shared spreadsheet or a source-control issue list, and needs a tool that doesn't just store bugs but helps you fix the right ones. You're not building a spacecraft or a radiation machine (and if you are, skip ahead to the warning below). You just want to stop losing track of defects and start shipping cleaner code.

I'm going to walk you through what to look for, how to set up your workflow, and what to do when a bug isn't just a bug but a security vulnerability. By the end, you'll know exactly how to configure your tracker so it earns its keep.

Step 1: Pick a Tracker That Fits, Not One That Impresses

The tool matters less than the workflow you build around it. That's not just my opinion—it's the reason Bugzilla, one of the oldest open-source bug trackers, is still around. It was originally written in Tcl by Terry Weissman in 1998 to replace Netscape's internal system, and it's used by Mozilla and "hundreds or thousands of organizations" (Bugzilla). If a tool built in the 90s can still serve a browser vendor, the secret isn't flashy features—it's solid fundamentals.

So what do you actually need? A tracker that lets you define statuses, assign ownership, and link issues to code. Jira, Azure DevOps, GitHub Issues, and Marker.io are common choices (Atlassian). But don't pick one because it's popular. Pick one that fits your team's size and your willingness to configure. Bugzilla, for example, is free and runs on open-source tools, but it's not pretty. GitHub Issues is dead simple but lacks some advanced workflow controls. The right choice is the one you'll actually use.

Here's a quick comparison to help you decide:

Don't get paralyzed by choice. Start with the simplest tool that gives you a clear status workflow and the ability to add custom fields. You can always migrate later.

Step 2: Set Up a Workflow That Matches Reality

Your workflow should be as simple as possible while still covering the essentials. Atlassian recommends a bare-bones path: Open → In Progress → In Review → Done → Closed, with an optional Triage step (Atlassian). That's a great starting point. Don't add ten statuses because you think you need them. You'll just end up with a tracker that takes more time to update than to fix.

But there are a few extra states you should know about, because they happen in real life. Atlassian lists Rejected, Duplicate, Deferred, Reopened, Not a Bug, and Non-Reproducible as common alternatives (Atlassian). Include them if you find yourself constantly explaining that a bug came back. A proper workflow lets you close a bug as "Not a Bug" without losing the history.

Here's the key: assign bugs to a specific developer, and when a fix is done, reassign it back to the original tester for verification. Any new defects found during that retest should be logged as separate issues (Atlassian). That keeps your data clean and your accountability clear.

One thing that trips up everyone: priority vs. severity. They're not the same. Priority is how urgent it is to fix; severity is how much it breaks things (Atlassian). A cosmetic typo on the homepage might be high priority because it's embarrassing, but low severity because it doesn't stop anyone from using the app. A crash in an obscure report might be high severity but low priority if nobody uses that report. You need both fields. Use a typical scale: priority from Lowest/Trivial to Highest/Blocker, and severity from Fatal/Blocker down to Minor/Trivial (Atlassian).

Step 3: Write a Good Bug Report (Yes, It's Part of the Tool)

The tracker is only as good as the reports you put in it. A complete defect report includes reproduction steps, expected vs. actual behavior, severity and priority, environment details, and visual proof (Atlassian). If you can't reproduce it, say so. Don't make the developer guess.

For web accessibility bugs, you need to test against the W3C's Web Content Accessibility Guidelines (WCAG). WCAG 2.2 has 13 guidelines under four principles—perceivable, operable, understandable, robust—and it's backwards compatible with earlier versions (W3C WAI). So if you're testing for WCAG 2.1, you're still aligned with the latest. Use the conformance levels A, AA, AAA to specify how severe the accessibility issue is (W3C WAI).

Now, a warning: if you're working on safety-critical systems, a bug report might be the only thing standing between a normal day and a disaster. The Therac-25 radiation machine killed and injured patients because of software errors, and the investigation found that the same error existed in the Therac-20, but hardware interlocks mitigated it. The lesson? Reusing software doesn't guarantee safety—safety is a property of the whole system (MIT). So when you write a bug report for something that can hurt people, treat it with the gravity it deserves.

Step 4: Track Security Vulnerabilities Separately

Not all bugs are created equal. Security vulnerabilities need a different kind of tracking. The National Vulnerability Database (NVD) is the U.S. government's repository for vulnerability data, and it enriches every CVE with CVSS scores, CWE types, and CPE applicability statements (NVD). You should adopt that mindset: for each security bug, record the CVSS score, the CWE type, and which platforms are affected.

CVSS v4.0 is the latest version, and it scores vulnerabilities across four metric groups—Base, Threat, Environmental, and Supplemental (FIRST). The Base group is the most important; it includes exploitability and impact metrics. Qualitative ratings map scores to None/Low/Medium/High/Critical (FIRST). Use that to prioritize. A Critical score (9.0–10.0) should get your immediate attention.

But don't just rely on CVSS. Check CISA's Known Exploited Vulnerabilities (KEV) catalog. It's the authoritative list of vulnerabilities that are actually being exploited in the wild, and it includes a remediation due date and whether the vulnerability is used in ransomware campaigns (CISA KEV). If a bug in your system matches a KEV entry, that's your top priority, regardless of CVSS score.

Here's a concrete example: the 2024 CWE Top 25 lists CWE-79 Cross-site Scripting as the top weakness, with a score of 56.92 and only 3 CVEs in the KEV catalog, while CWE-787 Out-of-bounds Write is #2 with 18 KEV CVEs (CWE). If you're tracking a buffer overflow that's actively exploited, that's a bigger deal than a theoretical XSS that nobody's using. Use the KEV catalog to adjust your priorities.

Step 5: Use Your Tracker to Improve Your Process

Your bug tracker isn't just a ledger; it's a goldmine of data. Two metrics worth tracking: defect rejection ratio (rejected defects divided by total reported) and defect leakage ratio (production defects divided by total found) (Atlassian). If your rejection ratio is high, your testers are filing bad reports. If your leakage ratio is high, your testing is missing things.

But don't fall for the "fix everything faster" myth. The largest study on the delayed issue effect, covering 171 projects from 2006–2014, found no evidence that fixing issues later costs more effort (arXiv). So don't assume that every bug must be fixed immediately. Sometimes it's fine to defer.

Still, you should aim to catch bugs before they reach production. NIST research shows that 70–95% of software failures are triggered by just two interacting variables, and practically 100% are caused by six or fewer (NIST). That's why combinatorial testing—testing combinations of two to six variables—can be as good as exhaustive testing (NIST). Use your tracker to note which combinations you've tested, so you know you're not leaving gaps.

What I'd Actually Do

If you're starting from scratch, my recommendation is this: pick a tracker with a simple workflow—Jira or GitHub Issues are fine—and set up these fields: severity, priority, environment, and a link to the code commit. Use the Atlassian workflow: Open → In Progress → In Review → Done → Closed, with a Triage step. That's it. Don't overcomplicate.

For security bugs, create a separate project or label, and require a CVSS score and CWE type. Check the CISA KEV catalog at least weekly to see if any of your bugs are actively exploited. If they are, drop everything.

And for the love of all that is holy, write good reports. A bug without reproduction steps is a guess, not a bug. Your future self—and your team—will thank you.

Sources

• Atlassian (bug life cycle) - https://www.atlassian.com/software/jira/guides/bug-tracking/bug-life-cycle
• Atlassian (issue priority vs severity) - https://www.atlassian.com/software/jira/guides/issues/priorities
• Bugzilla (official site) - https://www.bugzilla.org/about/
• CISA (Known Exploited Vulnerabilities Catalog) - https://www.cisa.gov/known-exploited-vulnerabilities-catalog
• NIST (2010 combination testing news release) - https://www.nist.gov/news-events/news/2010/11/updated-nist-software-uses-combination-testing-catch-bugs-fast-and-easy
• arXiv (Menzies et al. 2016) - https://arxiv.org/abs/1609.04886