Testing FrontEND Code with AI

Introduction

Everyone knows testing is one of the pillars of quality code, and sometimes it is a fundamental principal dev teams work around - think of Test-Driven Development, Behavior-Driven Development, or even Acceptance Test-Driven Development... I remember the days of attaching screenshots to prove I have tested along with the release ticket... Needless to say I have always been of the opinion that testing is important, but it is very difficult to do for dynamic, rich and complex web applications because adding tests sometimes takes longer to implement the features to begin with, and keeping them up to date is very time-consuming. In highly dynamic, high pressure environments such as prototyping and delivering features quickly, I dare to say it has been almost impossible to diligently meet coverage thresholds defined for a project. These types of apps have been what I've been working on most of my career, so adding useful tests have been extremely difficult and I've mostly added unit tests for important and complex functions, but shied away from end-to-end tests. I have been so focused on feature delivery and ensuring code and product quality in other aspects that I only recently discovered that I've been following the philosophies of the Testing Trophy. Recently I decided I wanted to be more thoughtful about automated testing since AI is perfect for the task. As a dedicated experiment I decided to utilize AI to define and implement rules to conform to the Testing Trophy's ideas, and in this post I will share my experience. 

What is the Testing Trophy?

To understand my testing setup it is important to first understand what the Testing Trophy is and what it isn't. The testing trophy is a concept coined by Kent C. Dodds that classifies not only the kinds of tests to use in a frontend project, but also how many of each kind to use in order to have a solid testing posture. It's a very simple concept but really powerful when incorporated in a real project. The reason I adopted the testing trophy to such an extent is because AI is really good at following rules and iterating until the desired state is met. The testing trophy is actually quite flexible, and each project's goals/rules can be easily defined while still being fluid and easily digestible by AI.

The image below illustrates all the tools I'm using at each of the levels:

Breakdown

The truth is that keeping the testing trophy in good stature in a project is quite tedious and there's no real measurement of how well each level is maintained. However, there are best practices on how to approach the requirement of each level, and by defining this for my agents I have been able to improve the testing footprint in my experiment. I used AI extensively to write my tests, but the key is not to just let it write tests for the sake of meeting the project's desired coverage level. Instead, the core idea is to make sure that each test covers the expected behavior, and catches real-world issues that could occur. Every single test I have followed the red-then-green rule, meaning after the test is created I deliberately let the behavior-based test fail (by changing the code) before fixing it again and letting the test pass.

Roughly this is what I did for each level:

Static: I make sure to run each tool whenever there is a code change - AI catches this immediately and corrects any issues before the task is considered complete.

Unit: pure logic is tested without the need of a browser, and the guideline I have is to extract pure helpers from functions instead of having convoluted components with helper functions in their bodies.

Integration: I ensure each component, hook, and state is tested, and I make sure LLM has a test for each input path; furthermore the goal for the test writer is to resemble real usage. To ensure quality, the name of the test is the contract - it defines the exact behavior that the test covers. A real rule I have is that tests has to make non-vacuous assertions.

E2E: logic specifically isn't tested at this layer - the focus is making sure user-facing behavior has one or more test.

How does it scale?

The key to making this approach scale across my suite of web apps was to extract the foundational configurations to a shared component library. I opted to adding it to my existing Storybook shared library that all the apps were already using, making it a very seamless transition. So the approach from the start was: implement the testing trophy strategy into one project, extract the pieces that should be standardized into the shared library, iterate on it on the initial project, version and finalize it, and start using it on other project. Now, maintenance is done primarily in the shared library, making it quick, easy and streamlined.

What have been the drawbacks/challenges?

Getting the entire flow automated for the most part was time consuming, since it touched a lot of the core setup of the project.

It does take a few more steps now to make quick changes, but it's completely worth the effort for the sense of trust I now have that the changes didn't cause obvious regressions. It truly is the philosophy of the Testing Trophy in action.

Does it catch all the bugs? Nope - there is no silver bullet in IT, not even when following an approach like this. I still do thorough manual testing to ensure features work as expected in the browser.

What have been the benefits?

I will need to spend more time with this setup and refine some of the steps to make it even more streamlined, but the overall testing posture has been great. My code coverage increased dramatically, which means more code is covered by tests, reducing the chances of creating regressions. Having agents coding has also improved because with this setup whenever it makes a change, they add a proper test, make sure it fails first, and then makes sure it passes. Doing this manually would have taken a lot of time, but now it's essentially free. Since AI is not afraid to make code changes and work rapidly, it helps a lot having tests to ensure it didn't inadvertently break something else.

The AI Factor with Architectural Changes

I've done these types of refactors before (making a deep architectural or process change, standardize it and scale it to multiple apps) and it has always been a time consuming endeavor, taking lots of trial and error because there are so many libraries that need to work together, each with their own quirks and requirements. This is where AI really shines - it can easily look up the documentation, find the quirks and workarounds, quickly interpret and troubleshoot obscure error traces and find solutions much faster than a human would.

Conclusion

In this post I shared my experience of using AI to drastically improve my testing confidence. We discussed the Testing Trophy and how I approached adding it to my suite of applications. Testing has never been easier and more streamlined to set up and get right in related projects thanks to AI coding tools. Just like any tool it is not a silver bullet, but when applied thoughtfully and carefully the benefits are fantastic. 

Feedback and questions are welcome - feel free to contact me on LinkedIn.