How to Implement Quality as Code Practices

𝗧𝗵𝗲 𝗱𝗮𝘀𝗵𝗯𝗼𝗮𝗿𝗱 𝗹𝗼𝗼𝗸𝗲𝗱 𝗳𝗶𝗻𝗲. 𝗧𝗵𝗲 𝗻𝘂𝗺𝗯𝗲𝗿𝘀 𝘄𝗲𝗿𝗲 𝘄𝗿𝗼𝗻𝗴 𝗳𝗼𝗿 𝘁𝗵𝗿𝗲𝗲 𝘄𝗲𝗲𝗸𝘀 𝗯𝗲𝗳𝗼𝗿𝗲 𝗮𝗻𝘆𝗼𝗻𝗲 𝗻𝗼𝘁𝗶𝗰𝗲𝗱. Ep 42 covered monitoring: how you detect problems.  This episode covers how you prevent them from reaching production in the first place. Data quality as code means embedding validation checks directly into your pipeline, not running them after something breaks. 𝗪𝗵𝗮𝘁 𝗺𝗼𝘀𝘁 𝘁𝗲𝗮𝗺𝘀 𝗱𝗼: → Spot-check data manually after a stakeholder complains. → Write one-off SQL queries to investigate. → Fix the issue. Move on. Same problem returns next quarter. 𝗪𝗵𝗮𝘁 "𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝘀 𝗰𝗼𝗱𝗲" 𝗺𝗲𝗮𝗻𝘀: → Assertions in the pipeline. "Order amount is never negative." "Row count within 10% of yesterday." "No duplicate primary keys." These run automatically, every time. → Tests at layer boundaries. Validate at ingestion (is the source clean?), after transformation (did the logic produce expected results?), and before serving (is this safe for consumers?). → Version-controlled checks. Quality rules live in the same repo as pipeline code. They go through PR review. They have history. They evolve with the data. → Fail-fast behavior. When a check fails, the pipeline stops. It is better to deliver a late report than a wrong one. 𝗧𝗼𝗼𝗹𝘀 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝘁𝗵𝗶𝘀 𝗽𝗮𝘁𝘁𝗲𝗿𝗻: → dbt tests: built-in assertions (unique, not_null, accepted_values, relationships) plus custom SQL tests. → Great Expectations: expectation suites with profiling, data docs, and orchestrator integration. → Soda: lightweight checks defined in YAML, designed for pipeline integration. If your only test is eyeballing dashboards, you don't have data quality. You have luck. What quality check would have caught your last data incident earliest? #DataEngineering #DataQuality #DataPipelines

As a client project manager, I consistently found that offshore software development teams from major providers like Infosys, Accenture, IBM, and others delivered software that failed 1/3rd of our UAT tests after the provider's independent dedicated QA teams passed it. And when we got a fix back, it failed at the same rate, meaning some features cycled through Dev/QA/UAT ten times before they worked. I got to know some of the onshore technical leaders from these companies well enough for them to tell me confidentially that we were getting such poor quality because the offshore teams were full of junior developers who didn't know what they were doing and didn't use any modern software engineering practices like Test Driven Development. And their dedicated QA teams couldn't prevent these quality issues because they were full of junior testers who didn't know what they were doing, didn't automate tests and were ordered to test and pass everything quickly to avoid falling behind schedule. So, poor quality development and QA practices were built into the system development process, and independent QA teams didn't fix it. Independent dedicated QA teams are an outdated and costly approach to quality. It's like a car factory that consistently produces defect-ridden vehicles only to disassemble and fix them later. Instead of testing and fixing features at the end, we should build quality into the process from the start. Modern engineering teams do this by working in cross-functional teams. Teams that use test-driven development approaches to define testable requirements and continuously review, test, and integrate their work. This allows them to catch and address issues early, resulting in faster, more efficient, and higher-quality development. In modern engineering teams, QA specialists are quality champions. Their expertise strengthens the team’s ability to build robust systems, ensuring quality is integral to how the product is built from the outset. The old model, where testing is done after development, belongs in the past. Today, quality is everyone’s responsibility—not through role dilution but through shared accountability, collaboration, and modern engineering practices.

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.

I’ve lost count of projects that shipped gorgeous features but relied on messy data assets. The cost always surfaces later when inevitable firefights, expensive backfills, and credibility hits to the data team occur. This is a major reason why I argue we need to incentivize SWEs to treat data as a first-class citizen before they merge code. Here are five ways you can help SWEs make this happen: 1. Treat data as code, not exhaust Data is produced by code (regardless of whether you are the 1st party producer or ingesting from a 3rd party). Many software engineers have minimal visibility into how their logs are used (even the business-critical ones), so you need to make it easy for them to understand their impact. 2. Automate validation at commit time Data contracts enable checks during the CI/CD process when a data asset changes. A failing test should block the merge just like any unit test. Developers receive instant feedback instead of hearing their data team complain about the hundredth data issue with minimal context. 3. Challenge the "move fast and break things" mantra Traditional approaches often postpone quality and governance until after deployment, as shipping fast feels safer than debating data schemas at the outset. Instead, early negotiation shrinks rework, speeds onboarding, and keeps your pipeline clean when the feature's scope changes six months in. Having a data perspective when creating product requirement documents can be a huge unlock! 4. Embed quality checks into your pipeline Track DQ metrics such as null ratios, referential breaks, and out-of-range values on trend dashboards. Observability tools are great for this, but even a set of SQL queries that are triggered can provide value. 5. Don't boil the ocean; Focus on protecting tier 1 data assets first Your most critical but volatile data asset is your top candidate to try these approaches. Ideally, there should be meaningful change as your product or service evolves, but that change can lead to chaos. Making a case for mitigating risk for critical components is an effective way to make SWEs want to pay attention. If you want to fix a broken system, you start at the source of the problem and work your way forward. Not doing this is why so many data teams I talk to feel stuck. What’s one step your team can take to move data quality closer to SWEs? #data #swe #ai

Scrum Guarantees Better Quality Software... What Nonsense! Scrum gets credit for things it doesn’t actually do. One myth is that adopting Scrum will result in higher-quality software. It won’t. Scrum is a framework for managing work, not for ensuring technical excellence. Yes, Scrum can expose problems that impact quality, but it doesn’t prescribe the engineering practices necessary to improve it. Scrum Isn't Magic Scrum provides a structure for iterative work, frequent feedback, and continuous improvement. But it doesn’t dictate how teams must write, test, or deploy code. Scrum defines accountabilities, events, artifacts, but quality comes from what happens inside the iterations, not from the framework itself. If a team writes bad code today, breaking it into 2-week iterations won’t magically make it better. If releases are painful and error prone, sprinting won’t fix that. Quality comes from technical practices, not Scrum rituals. Engineering Practices That Improve Quality Scrum leaves the “how” of software development up to teams, and Agile engineering practices fill the gap. These are (some of) the drivers of high-quality software: CI/CD: Teams that integrate and deploy frequently catch issues earlier, reduce risk, and maintain a steady flow of value. CI/CD pipelines automate testing and deployment, increasing confidence that every increment of software will be production-ready. Test Automation: Scrum encourages frequent delivery, but releasing buggy code faster isn’t an improvement. Automated unit, integration, and end-to-end tests serve as a safety net - maintaining the functional integrity and fitness for purpose of each increment. TDD: TDD forces developers to think about correctness before coding. Writing tests first leads to cleaner, more maintainable code and reduces defects. Pair Programming and Code Reviews: Collaboration helps prevent defects and spreads knowledge across the team. Pair programming produces immediate feedback. Peer reviews catch mistakes before they reach production. Trunk-Based Development: Trunk-based development encourages small, frequent commits to a shared main branch, facilitating integration and improving stability. DoD with Quality Gates: A meaningful DoD doesn’t just say, "The acceptance criteria was met." It checks for code reviews, automated tests, security scans, and performance benchmarks. Without this discipline, think "don't," not “done." Observability and Monitoring: Scrum is silent on what happens after deployment, but software without monitoring is a risk. Logging, tracing, and monitoring help teams detect and fix issues quickly. Don't Sprint Toward Tech Debt A Scrum team without engineering discipline may deliver every sprint, but what they deliver could be faulty, fragile, unmaintainable code. Scrum makes work visible, but it doesn’t make it good. Better software doesn’t come directly from Scrum. It comes from teams that invest in engineering practice excellence.

Use these essential techniques for crafting high-quality code in your data job. Here’s how you can ensure your code meets the highest standards: 1. 𝗙𝗼𝗹𝗹𝗼𝘄 𝗕𝗲𝘀𝘁 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀: Stick to the conventions and standards in your language of choice, whether it's Python, R, or SQL. This includes using meaningful variable names, intending your code into blocks, and organizing code in logical modules.     2. 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁 𝗩𝗲𝗿𝘀𝗶𝗼𝗻 𝗖𝗼𝗻𝘁𝗿𝗼𝗹: Use tools like Git to manage changes and collaborate more effectively. Version control isn’t just for software developers but for every professional writing code.     3. 𝗖𝗼𝗱𝗲 𝗥𝗲𝘃𝗶𝗲𝘄𝘀: Engage with your team to review each other's code. This not only helps catch errors before they become problems but also promotes a culture of learning and knowledge sharing within your team.     4. 𝗪𝗿𝗶𝘁𝗲 𝗧𝗲𝘀𝘁𝘀: Although it might seem over the top for some data projects, writing tests for your code can catch bugs early and save hours of debugging later. It ensures that your code functions as expected and makes modifications safer and more reliable. You can include different tests like unit tests for functions and schema checks for your inputs.     5. 𝗥𝗲𝗳𝗮𝗰𝘁𝗼𝗿 𝗥𝗲𝗴𝘂𝗹𝗮𝗿𝗹𝘆: Revisit and revise your code regularly. Refactoring helps to improve your code efficiency and readability. As you learn and grow, you'll find ways to make your existing code better.     6. 𝗗𝗼𝗰𝘂𝗺𝗲𝗻𝘁 𝗘𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴: Good documentation is not just about commenting within the code. Document your data sources, your algorithms, and your methodologies in an accessible way. This is crucial for when others (or you in the future) need to understand and possibly build on your work. A good place for this additional documentation is architectural decision records placed in your repository.     7. 𝗙𝗼𝗰𝘂𝘀 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 𝗼𝗻 𝗢𝗻𝗲 𝗧𝗵𝗶𝗻𝗴: Make sure your function is doing exactly one thing. Having to use "and" in the function name or the comment describing the function then split it! It will make it much easier to understand and modify later on.     8. 𝗟𝗲𝘃𝗲𝗿𝗮𝗴𝗲 𝗟𝗶𝗻𝘁𝗲𝗿𝘀 𝗮𝗻𝗱 𝗙𝗼𝗿𝗺𝗮𝘁𝘁𝗲𝗿𝘀: Tools that automatically check your code for style issues, errors, or syntactic discrepancies can drastically improve your code quality. They enforce consistency and help you focus on the logic rather than the format. You level up your code from good to great by including these techniques in your workflow. High-quality code means fewer errors, easier maintenance, and more reliable results. What strategies do you use to maintain high code quality? ---------------- ♻️ Share if you find this post useful ➕ Follow for more daily insights on how to grow your career in the data field #dataanalytics #datascience #cleancode #python #codingstandards

Mastering Code Quality: 12 Key Practices for Efficiency and Reliability 1. Use prettification tools like Prettier to standardize code formatting. 2. Employ linters like SonarLint to catch code smells and potential bugs. 3. Configure precommit hooks with Husky to automate checks before commits. 4. Follow SOLID principles for scalable, maintainable code. 5. Avoid memory leaks by managing resources effectively. 6. Apply design patterns for reusable, structured code. 7. Write unit tests to verify code correctness early. 8. Use dependency injection to reduce tight coupling and improve flexibility. 9. Follow DRY principles to avoid code duplication. 10. Perform code reviews for quality control and knowledge sharing. 11. Optimize code for performance with efficient algorithms and data structures. 12. Implement continuous integration for regular, automated testing and integration. What other practices do you use to ensure clean, efficient, and robust code? Share yours below! #SoftwareDevelopment #CodingBestPractices #CleanCode #SoftwareEngineering #CodeQuality #ProgrammingTips #Tech

𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗦𝗼𝗻𝗮𝗿𝗤𝘂𝗯𝗲 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗺𝗲𝘁𝗿𝗶𝗰𝘀? To maximize the effectiveness of SonarQube in maintaining and improving code quality, here are some best practices for SonarQube analysis: 𝟭. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲 𝗦𝗼𝗻𝗮𝗿𝗤𝘂𝗯𝗲 𝗲𝗮𝗿𝗹𝘆 𝗶𝗻 𝘁𝗵𝗲 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗽𝗿𝗼𝗰𝗲𝘀𝘀 Integrate SonarQube into your CI/CD pipeline to ensure that code quality checks are performed automatically with every build. The sooner we start, the more accurate our data will be. It is more accurate than introducing it later in the process; you will get many more false positives as SonarQube analyzes the code as it is written, not after. 𝟮. 𝗗𝗲𝗳𝗶𝗻𝗲 𝗰𝗹𝗲𝗮𝗿-𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗴𝗮𝘁𝗲𝘀 Define quality gates with specific thresholds for critical metrics such as bugs, vulnerabilities, code smells, and coverage. This helps enforce quality standards. Also, configure your CI/CD pipeline to fail builds if the quality gate criteria are not met, ensuring issues are addressed promptly. 𝟯. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲 𝗦𝗼𝗻𝗮𝗿𝗤𝘂𝗯𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗖𝗜/𝗖𝗗 𝘀𝗲𝗿𝘃𝗲𝗿 When we enable the automatic usage of SonarQube during the build and PR process, we get immediate feedback about our code quality. This helps us to improve the code before it is merged into the codebase. This also allows our reports to always be up to date and not rely on any manual process. 𝟰. 𝗣𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘇𝗲 𝗶𝘀𝘀𝘂𝗲𝘀 𝗯𝗮𝘀𝗲𝗱 𝗼𝗻 𝘀𝗲𝘃𝗲𝗿𝗶𝘁𝘆 To maintain your application's stability and security, address critical and significant issues (bugs and vulnerabilities) first. Then, incrementally tackle code smells and minor issues to gradually improve code maintainability without overwhelming the team. 𝟱. 𝗗𝗼𝗻’𝘁 𝗶𝗴𝗻𝗼𝗿𝗲 𝗶𝘀𝘀𝘂𝗲𝘀 If we ignore issues, we will postpone the problem and increase technical debt, which we don’t want to do. We should fix the issues when they happen. If there are issue types we don’t want to fix, we can adjust the SonarQube ruleset and exclude those rules. 𝟲. 𝗠𝗶𝗻𝗶𝗺𝗶𝘇𝗲 𝗰𝗼𝗱𝗲 𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 Review the codebase regularly for duplicates and refactor them into reusable components or functions. SonarQube can help identify these duplications. 𝟳. 𝗠𝗶𝗻𝗶𝗺𝗶𝘇𝗲 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗗𝗲𝗯𝘁 The technical debt ratio is the estimated time to fix code issues divided by the project development time. Aim to keep this ratio below 5% to ensure the project remains manageable. Allocate a portion of the team development time to addressing technical debt. This could involve refactoring, improving test coverage, or resolving code smells. 𝟴. 𝗠𝗮𝗶𝗻𝘁𝗮𝗶𝗻 𝗵𝗶𝗴𝗵 𝘁𝗲𝘀𝘁 𝗰𝗼𝘃𝗲𝗿𝗮𝗴𝗲 Aim for high test coverage, typically 60-80%. This ensures that most of the codebase is tested, reducing the likelihood of bugs slipping through. Use tools like JaCoCo or Cobertura to measure test coverage. #technology #softwareengineering #programming #techworldwithmilan #coding

Why is Quality important in your project? You probably heard about the Crowdstrike outage on the weekend that disrupted airports, banks and stores all around the world. It was caused by Crowdstrike pushing a system file full of zeros to production. How can we avoid this sort of disaster as we manage our own Projects? There are many ways: ➡️ Peer review & Code Inspections: Checking the requirements with a user or reviewing the code with another Developer. ➡️ Continuous Integration: Merging changes into the main test system (often daily) with automatic tests. ➡️ Test Driven Development: Tests are written first, failed, then run again and passed after the solution is coded. ➡️ Unit Tests: Testing each small piece or User Story. ➡️ System Testing: Testing the integrated system as a whole. ➡️ User Acceptance Testing: Testing the system from the User’s point of view. ➡️ Regression Testing: Testing the existing system with the changes to see if they’ve impacted normal operations. ➡️ Production Verification Testing: Testing the change in the live environment after go-live then rolling the change back if it goes wrong. ➡️ Sprint Review: Demonstrating the actual change to the customer or senior users before release. Quality is one of the 10 Project Management Knowledge Areas for a reason. ⭐ What are some of the ways you manage Quality on your own projects? ⭐

Your dev team doesn’t need more code. It needs better systems. This week, I reviewed one of our codebases. What I saw is common across fast-moving startups: – Repos with no consistent naming – Missing or outdated READMEs – Old branches piling up – Hard-coded credentials – No CI checks, tests, or structure – Massive, unmanageable files None of this is about bad developers. It’s about missing systems. Here’s what we did—and what any dev team can implement fast: 1. Organize your repos Clear names. Clean branches. Set main as the default. Protect it. 2. Write solid docs Every repo should explain what it is, how to run it, and what it does. Use ChatGPT to help. 3. Automate the basics Use GitHub Actions. Lint. Run tests. Scan dependencies. Don’t trust code that hasn’t passed checks. 4. Improve PR quality Good titles. Clear descriptions. Screenshots. Auto-delete merged branches. Require reviews. 5. Refactor big files Modular, readable code wins every time. Don’t mix config with logic. Use env variables and lock files. 6. Use AI as your co-pilot I uploaded a full codebase to ChatGPT. In 10 minutes, it flagged issues, suggested tests, and surfaced risks we missed. That’s leverage. The real unlock? Mindset. This isn’t red tape. It’s freedom. With checks in place, I can approve PRs from my phone—because I know the basics are solid. Startups move fast. But only if your codebase can keep up. What dev practice has saved your team the most time or stress? --- Enjoy this? ♻️ Repost it to your network and follow Kevin Henrikson for more. Weekly frameworks on AI, startups, leadership, and scaling. Join 1800+ subscribers today: https://lnkd.in/gstGkhJF