Viewing Tests as Integral to the Codebase

Too many teams treat testing as a metric rather than an opportunity. A developer is told to write tests, so they do the bare minimum to hit the required coverage percentage. A function runs inside a unit test, the coverage tool marks it as covered, and the developer moves on. The percentage goes up, leadership is satisfied, and the codebase is left with the illusion of quality. But what was actually tested? Too often, the answer is: almost nothing. The logic was executed, but its behavior was never challenged. The function was called, but its failure modes were ignored. The edge cases, error handling, and real-world complexity were never explored. The opportunity to truly exercise the code and ensure it works in every scenario was completely missed. This is a systemic failure in how organizations think about testing. Instead of seeing unit, integration, and end-to-end (E2E) testing as distinct silos, they should recognize that all testing is just exercising the same code. The farther you get from the code, the harder and more expensive it becomes to test. If logic is effectively tested at the unit and integration level, it does not suddenly behave differently at the E2E level. Software is a rational system. A well-tested function does not magically start failing in production unless something external—such as infrastructure or dependencies—introduces instability. When developers treat unit and integration testing as a checkbox exercise, they push the real burden of testing downstream. Bugs that should have been caught in milliseconds by a unit test are now caught minutes or hours later in an integration test, or even days later during E2E testing. Some are not caught at all until they reach production. Organizations then spend exponentially more time and money debugging issues that should never have existed in the first place. The best engineering teams do not chase code coverage numbers. They see testing as an opportunity to build confidence in their software at the lowest possible level. They write tests that ask hard questions of the code, not just ones that execute it. They recognize that when testing is done well at the unit and integration level, their E2E tests become simpler and more reliable—not a desperate last line of defense against failures that should have been prevented. But the very best testers go even further. They recognize the system for what it truly is—a beautiful, interconnected mosaic of logic, data, and dependencies. They do not just react to failures at the UX/UI layer, desperately trying to stop an avalanche of possible combinations. They seek to understand and control the system itself, shaping it in a way that prevents those avalanches from happening in the first place. Organizations that embrace this mindset build more stable systems, ship with more confidence, and spend less time firefighting production issues. #SoftwareTesting #QualityEngineering

Writing automated tests goes beyond confirming whether a specific feature works in isolation....it’s about safeguarding the system's integrity as it scales and evolves. From an architectural perspective, tests act as a critical feedback loop. They validate assumptions, ensure that changes don’t break core functionality, and support scalability. Without tests, every modification becomes a risk, and troubleshooting becomes reactive. Good tests also enforce clarity in both code and design, surfacing weak points early. They’re not just for bug detection they’re for creating systems that are reliable, maintainable, and future-ready. By shifting testing left(incorporating tests early in the development process), you catch issues sooner, reducing the cost of defects and increasing the overall quality of the system. Think of tests as an essential part of your architectural blueprint. They’re not just there to check if individual components work, but to ensure you’re building with confidence, for both today and the future. #SoftwareArchitecture #AutomatedTesting #ShiftLeft #CodeQuality #TechLeadership #Scalability #CleanCode #Agile

During the initial phase of my career in VLSI, I realised that writing Testcases is equally important as Testbench development. A Testcase in any language be it Verilog, VHDL, SystemVerilog, and UVM is not only used to verify the functional correctness and the integrity of the design but also point out areas where the Testbench could be improved. Below are the most important category of Testcases which are most critical: [1] Functional Tests --> In this type of test, the functionality or feature of an IP/module or a subsystem is verified. [2] Register-based Tests --> RW Tests, RO/WO Tests, Default Read/Hard reset Tests, Soft reset tests, Negative RO/WO Tests, Aliasing, Broadcasting, etc [3] Connectivity Tests [4] Clock and Reset Tests [5] Boot up Tests, wake up sequence, training sequence tests. For eg. In the case of DDR – MPC Training, RD DQ Calibration, Command Bus training, Write leveling, etc [6] Command and Sequence-based Tests. [7] Overlapping and Unallocated Region tests. [8] Back-to-back data transfer-based tests. [9] UPF Tests --> Power domain, Level Shifter, clock gating, voltage domain, etc [10] Code Coverage Tests --> In this test toggle, expression, branch, FSM, and conditional coverage holes are measured, and depending on the holes, tests are being written to completely exercise the DUT. [11] Functional Coverage Tests --> In these types of test categories, the functionality of DUT is being measured with the help of bins. There are several ways to do it. If there are coverage holes, more bins are coded to cover those areas, complex scenarios are covered with cross coverage, and bins of intersect functionality. [12] Assertions are basically a check against the design. Basically, these are insertion points within the design which improve the observability and debugging ability. The above are some of the categorizations of tests that need to be applied while checking a design but to achieve all the above features, testcases are broadly classified into the following two types: [1] Directed Testcase: These are the scenarios that the verification engineers can think of or can anticipate. [2] RandomTestcase: These are the scenarios where the maximum amount of bugs can be caught. The random seeds will hit many different use cases which can not be anticipated earlier and has the probability to catch the design issues. Ideally, random tests can be classified into the following two categories: [1] Corner cases --> This is the bug that is only possible to catch when many different scenarios are processed together or they overlap and the best way to catch this type of scenario is to run more repeated regression with more seeds. [2] Stress testing -->These types of tests are useful to check the performance and the scalability of the DUT under multiple concurrent activities and unpredictable scenarios. #vlsi #asic #electricalengineering #semiconductorindustry

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.

In modern software development, writing code is only half the job — testing it is just as critical. But as codebases grow, maintaining strong unit test coverage becomes increasingly challenging. A recent engineering blog from The New York Times explores an interesting approach: using generative AI tools to help scale unit test creation across a large frontend codebase. - The team built an AI-assisted workflow that systematically identifies gaps in test coverage and generates unit tests to fill them. Using a custom coverage analysis tool and carefully designed prompts, the AI proposes new test cases while following strict guardrails — such as never modifying the underlying source code. Engineers then review and refine the generated tests before merging them. - This human-in-the-loop approach proved surprisingly effective. In several projects, test coverage increased from the low double digits to around 80%, while the time engineers spent writing repetitive test scaffolding dropped significantly. The process also follows a simple iterative loop: measure coverage, generate tests, validate results, and repeat. The experiment also highlighted some limitations. AI can hallucinate tests, lose context in large codebases, or produce outputs that require careful review. The takeaway: AI works best as an accelerator — not a replacement — for engineering judgment. As these tools mature, this kind of collaborative workflow may become a practical way for teams to scale reliability without slowing down development. #DataScience #MachineLearning #SoftwareEngineering #AIinEngineering #GenerativeAI #DeveloperProductivity #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gj9fc322

Over the past decade, I've analyzed +300 codebases. (Yes, I've seen more code than I ever wished for). Surprisingly, some of the worst technical debt I've encountered is in test code. Test scripts often receive far less care than application code. When tests are poorly designed and hard to maintain, they stop being a support system and become a productivity bottleneck. There are multiple reasons: * First, we developers tend to make a mental divide between application code (which we know is important to keep clean and easy to maintain) and test code which received considerably less love. * Second, test automation is a fairly recent addition to mainstream software development; we might not yet have learned what well-designed test code looks like. After all, tests have a different purpose and separate constraints compared to application code. Tests smells also tend to be different than the traditional code smells you see in application code. A common example is duplicated-assertion-blocks. (See the image for an interesting example). The duplicated-assertion-blocks smell is a classic example on a lack of encapsulation. The solution is straightforward: encapsulate the test criteria in a custom assert with a descriptive name that can communicate the intent, and re-use the custom assert when your tests call for it. 💡Test data has to be encapsulated just like any other implementation detail. Of course, there's always the counterargument that if we abstract the code too much, the tests become harder to read and understand. That's absolutely true. But it's also true that there's a whole gulf of abstractions between "not at all" and "too much." Perhaps there's a mid-point where we can pay attention to the abstraction level in our tests without going completely overboard with abstraction acrobatics?

An unexpected repercussion of vibe coding that I've noticed: it really magnifies the quality of your developer experience. A lot of folks have compared coding AIs to something like an over-eager intern: energetic and prolific, though frequently lacking in experience and judgment. If you want to get the most out of it, ask yourself: how good is the day-1 experience of interns on your codebase? Because vibe coding forces you to replay that day-1 experience over and over. The quality of a coding agent's output improves dramatically if you give it a good fitness function to optimize for. It's one thing to ask, "just spit out code for this." It's quite another to say, "fix this bug, add tests to this file, make sure lint passes and the code compiles and all the tests pass before you talk to me again." It's vastly more successful with the guard rails of typecheckers and compilers and tests telling it whether it's achieved a minimum acceptable quality bar. If your typical testing flow is "fire up a local server and click around a few pages," then suddenly your high-velocity intern is going to struggle. Most agents won't even be able to *do* that; if they can, there's no clear pass/fail signal, so it has to exercise judgment in deciding whether its code works or not, and recall that this particular intern has terrible judgment. Maybe your tests are flaky or don't run locally. That's gonna ruin your intern's day. Maybe you sometimes need to start a docker container before you run your tests. That's an extra step your intern needs to figure out and will sometimes forget. If you want to be able to leverage background agents, then you need to have a simple and fast path to bring a blank new machine into a state where it can run your tests. If any of the steps involve "just figure it out," your agent will definitely fail. And of course, absolutely none of this is new. This has been true of carbon-based interns before silicon-based ones. These matters of codebase quality have always had an impact on your company's productivity. If you don't invest in the developer experience, you just pay a perpetual tax on the labor of your high-value employees. Predictability breeds velocity. And always has.

If you can't change your code (without changing behaviour) because your tests fail, you've likely got tests at the wrong level. It's not about TDD vs. non-TDD, it's about the idea that your tests protect the behaviour your users expect. Everything else is waste. Worse, it makes your codebase resistant to change, causing frustration and stress. Users in this context are the people who use the system and those who support it. The tests you write should ensure that both of these users can get what they need, and when you change code, they still can. That's the point of the tests. If you have change/fix tests, you can no longer trust that test to tell you the behaviour is still valid. It's highly likely that it isn't. Beyond thinking about your external API service area, also think about what you need to support that application in production. Do you have alerts based on a particular metric or span tag? Then, you should have a test for that metric or tag specifically, but nothing else. If there's a log event that you need to ensure a successful audit, test that log line, not all of them. As a good high-level rule, if the test fails a month after you've released the feature, and you're likely to just ignore, skip or delete the test, chances are good that it's a pointless test. Write better tests, write tests that matter, and are important. Writing those tests before you write the code ensures that you don't assert based on implementation details, but it's sometimes harder to do if you're not used to it. Either way, resist writing harnesses around code, and prefer protecting the behaviour from undesired changes.

There's something missing in almost every data pipeline but is critical for trustworthiness and maintainability. It's unit testing. Unit testing is one of the main ways engineers ensure that code does what it's supposed to do and doesn't break unexpectedly. You may think, but I already have "tests", how is it different? Most data folks think of testing data as a data quality check or audit. These checks look for things like NULLs or duplicates after a pipeline has run. These checks do not validate business logic, they're mainly used to ensure upstream data hasn't changed unexpectedly. On the other hand, a unit test validates if a set of fixed inputs results in fixed outputs. They are quick to run and automatically run in a CI/CD process. Here are some benefits that unit testing provides over audits. - Validate fine grained business logic - Provide examples for how data should be transformed - Catch unintended breakages or changes to logic The last point is super important and the most important factor for maintainable code bases. As the number of contributors to your code base grows, the knowledge of business logic becomes more dilute. So people who make changes may not be familiar with every single line of SQL. This ensures that contributors are aware of the consequences of their changes. Even if you're not a big team, unit tests give you the ability to make changes or refactors more confidently. You may have spent a long time getting certain edge cases right, and unit tests will ensure that those edge cases are still addressed even through major refactors. I've been writing data unit tests since I started using Spark in 2014. It's relatively easy to setup unit tests in Spark because you can run it locally. That's why we've made unit testing a first class citizen every since we launched SQLMesh 2 years ago. I'm happy to see that dbt has also jumped aboard the unit testing train with their recent 1.8 release. Unit testing is definitely a step forward in the right direction for the industry and should become a standard. Unit testing is not prevalent in data because it's difficult to setup. It can be challenging generating test cases because of how tedious it is to get all the inputs and outputs. That's why SQLMesh has the ability to automatically create tests for you with a script. https://lnkd.in/gstDyQ7P Instead of manually specify every input and output, you can just write a couple of SELECT statements with LIMITs and it will populate a test. Another convenience feature that SQLMesh provides is the ability to run unit tests in DuckDB (or other engine). You may not want to run all of your tests in Bigquery or Snowflake because of access issues or cost. SQLMesh can transpile your queries to DuckDB, meaning your tests are fast and can be run anywhere. Do you unit test your data pipelines? https://lnkd.in/dgH_6uzC

Ever shipped firmware that worked flawlessly on your bench — then blew up the moment it hit real hardware? That pain is why system-level TDD exists. Most embedded teams build bottom-up. → Drivers first. → Middleware next. → Application logic last. By the time you hit “system testing,” your design is already locked in concrete. That’s when bugs turn into rewrites. Now flip that model. Start from the top. Write a failing system-level test before touching a single GPIO. Define what the entire system should do — not what one driver should handle. Here’s why it changes everything: → You can run full system tests early, in simulation or emulation, before hardware even shows up. → You validate actual behaviour — not just “does it boot?” → You can automate everything using a proper testing framework so results are consistent, measurable, and repeatable. → You still follow TDD’s rhythm — fail → code → pass → refactor — but now your design is driven by intent, not cleanup. For firmware engineers, that means: → You catch design flaws months earlier. → Your codebase stays modular and testable. → Your CI pipeline becomes your safety net — not an afterthought. Start testing from the system down, not the hardware up. Do that, and you’ll never dread integration week again. #EmbeddedSystems #FirmwareEngineering #TDD #SystemLevelTesting #EmbeddedTesting #CleanCode #CI_CD