Developer Roundtable: Rethinking Testing in Real-World Software Development

When developers think of testing, the knee-jerk reaction is often a push for 100% code coverage. But is coverage alone the best metric? Testing is complex and nuanced, with real-world trade-offs. Here’s how we approach testing with purpose, flexibility, and confidence.

What Are Tests Really For?

The primary purpose of tests is to protect against regressions. Making sure changes don’t break existing features. However, the type of testing you choose often depends on what you’re trying to accomplish:

• Confidence: Tests should build confidence that the software behaves as expected.
• Regression Prevention: Particularly useful are integration tests, which simulate user interactions across the application.
• Performance: Tests can also help ensure that refactoring efforts don’t degrade performance.

Understanding these goals helps avoid getting lost in the weeds of “coverage.” Testing is not about the number of tests but the value they provide in maintaining reliable software.

The Coverage Conundrum: What Do Tests Cover?

Coverage metrics are popular, but high numbers can be misleading. Tests are valuable for what they actually cover in your app:

• Critical Flows: Focus on testing features that are core to your app. If a feature breaks, would it impact users significantly? Start there.
• Edge Cases: Some bugs lurk in unexpected places. Consider testing scenarios that users are less likely to encounter but could cause problems if unhandled.

Visual Regression Testing

Sometimes, bugs are not in the code logic but in the visual presentation. Jim Remsik, the owner of Flagrant, recently started using Backstop for visual regression testing and shared a demo of it with the team. It’s especially helpful for catching subtle changes, like when a menu button’s size decreases, impacting accessibility. Backstop even has a sliding view to compare the current and expected UI, making it easy to spot differences across responsiveness breakpoints (e.g., mobile, tablet, desktop).

Practical Considerations: Balancing Testing Efforts

The approach to testing varies based on the app’s architecture. For instance:

• Front-End Focused Apps: In a React app, controller tests may hold less value. Emphasis is placed on UI and integration tests that align more closely with user interactions.
• Ecosystem-Specific Limitations: Some tools, like Wallaby, can restrict test flexibility, such as supporting only case-sensitive searches.

Another practical approach is clickey-pokey testing (end-to-end testing). Although it requires substantial setup and maintenance, it offers a broad scope that can reveal issues missed by unit tests.

Leave a Trail: Comments in Specs

Comments can often prevent future headaches, particularly in complex specs. Instead of creating entirely new specs, developers may add logic to an existing spec. Adding comments can guide others (or even your future self) to understand what’s being tested, why, and any nuances in setup.

Visual and Interaction Testing: Ensuring the Right Look and Feel

Beyond functionality, testing should ensure that the app looks and interacts as users expect. This might mean verifying that:

• Inputs have correctly associated labels
• Elements are easily identifiable and clickable, as a user would experience them

Visual testing is often neglected but can be crucial, particularly when user interfaces are involved.

Writing Tests for Confidence and Speed

Confidence is the ultimate goal. Tests should reduce anxiety around shipping changes and streamline development. For example, when setting up tests for specific interactions, running them through ChromeDriver can allow you to watch them execute in real-time, making it easier to pinpoint issues.

The Case Against Flaky Tests

A hot take: delete flaky tests. These tests, which pass or fail inconsistently, can do more harm than good, eroding confidence in your testing suite and wasting valuable developer time. Instead, rewrite these tests to be more stable, providing reliable feedback that adds confidence rather than undermining it.

Closing Thoughts

By focusing on the “why” behind testing and making thoughtful choices about what and how to test, developers can create a testing strategy that offers real value. It’s not about achieving 100% coverage but about covering the areas that matter most, giving developers confidence to build and ship robust software.

---

Enjoyed this content? Join our developer roundtable, we meet every other Friday from 10:30 - 11:30 am CT. Hope to see you there!

If you’re looking for a team to help you discover the right thing to build and help you build it, get in touch.