Continuous Integration in Agile

Does your team treat Continuous Integration like a daily chore or a strategic advantage? Continuous Integration (CI) is more than just merging code frequently—it’s about discovering defects as soon as they creep in. The faster you identify a problem, the cheaper it is to fix. With a reliable CI pipeline, every commit triggers automated tests, style checks, and static analysis, giving immediate feedback on code quality. This rapid loop means teams spend less time guessing where an issue originated and more time innovating. When testers contribute meaningful checks to the CI pipeline, they become early guardians of quality, ensuring that code merges don’t degrade the product.

Teams can’t keep the integration environment stable and are asking for a dev environment before integration & production… That’s like saying, “We can’t keep the kitchen clean, so let’s build a second one and hope for less mess.” Classic case of multiplying complexity to avoid addressing root problems. 🔧 Reality check: * Adding environments and branching policies (like long-lived branches) increase cognitive load, create bottlenecks, and delay feedback loops. * Continuous Integration is about behaviour, not tools. You build small, integrate often, test fast, and recover quickly. If you can’t keep one environment clean, adding more won’t magically fix discipline issues. What's next: an environment per feature + long-lived feature branches? 😂 That’s the slow train to merge hell. You're not enabling parallel development, you're fostering parallel universes. And when they collide? Chaos. ✅ Better approach: * Trunk-based development. * Short-lived branches. * Feature toggles. * Invest in fast CI pipelines and “stop-the-line” culture. * One environment, one truth. 𝗧𝗵𝗲 𝗺𝗮𝗻𝘁𝗿𝗮: 𝗳𝗲𝘄𝗲𝗿 𝗲𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀, 𝘁𝗶𝗴𝗵𝘁𝗲𝗿 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗹𝗼𝗼𝗽𝘀, 𝗰𝗹𝗲𝗮𝗻𝗲𝗿 𝗳𝗹𝗼𝘄. So, adding environments isn’t the solution. Want stability? Tackle integration daily, not quarterly. Keep it tight, keep it flowing. #Simplification #CICD #TBDForTheWin

Continuous integration isn't automation. It's a workflow that's enabled by automation. On the surface, CI is where everyone integrates their changes to the trunk at least daily, though more often is recommended. However, much like an iceberg, there's more to it than that. Teams that do this well are also very good at working together to define the behaviors of the next feature, collaborating on the acceptance tests for those features, defining contract changes, etc. They aren't surprised by code changes others have made because they helped define those changes before coding began. They help each other complete work before starting a new task. They don't struggle with code reviews because they helped define the tests for the coding task they are reviewing, and code review comments are seen as improvements, not criticism. This isn't hypothetical. This is how I've worked since learning CI, and how I've helped other teams transition from co-located individuals sharing a backlog to high-performing teams focused on delivering value. It's this disciplined approach to teamwork that makes CI possible, not Github Actions.

→ What if your project could reveal its risks BEFORE they become disasters? 𝐂𝐨𝐧𝐭𝐢𝐧𝐮𝐨𝐮𝐬 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 (𝐂𝐈) is not just developer jargon - it’s a PROJECT MANAGER’S secret weapon. Here’s why it demands your attention: • 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐂𝐈? Developers merge code changes multiple times daily into a shared repository. Each merge triggers automated builds and tests. • 𝐖𝐡𝐲 𝐬𝐡𝐨𝐮𝐥𝐝 𝐩𝐫𝐨𝐣𝐞𝐜𝐭 𝐦𝐚𝐧𝐚𝐠𝐞𝐫𝐬 𝐜𝐚𝐫𝐞? 𝐁𝐞𝐜𝐚𝐮𝐬𝐞 𝐂𝐈: • Detects bugs early, saving time and headache. • Ensures higher code quality consistently. • Provides immediate feedback for swift fixes. • Reduces risky code merges and integration hell. • Fosters team collaboration around a single source of truth. → 𝐇𝐨𝐰 𝐝𝐨𝐞𝐬 𝐂𝐈 𝐟𝐢𝐭 𝐢𝐧𝐭𝐨 𝐲𝐨𝐮𝐫 𝐩𝐫𝐨𝐣𝐞𝐜𝐭 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐫𝐡𝐲𝐭𝐡𝐦? • Plan for integration and testing cycles - blocking buffer time is key. • Allocate resources: automated testing tools, build servers, and infrastructure. • Monitor build health and pipeline bottlenecks continuously. • Use CI outcomes as transparent status updates to stakeholders. → 𝐓𝐡𝐞 𝐠𝐨𝐥𝐝𝐞𝐧 𝐩𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞𝐬 𝐲𝐨𝐮 𝐜𝐚𝐧’𝐭 𝐢𝐠𝐧𝐨𝐫𝐞: • Frequent small code merges to avoid conflicts. • Automated builds & tests catching bugs before they escalate. • Team collaboration sharpened through shared code bases. • Faster, safer delivery by catching issues early. • Dramatically reduced risk - fewer surprises. CI transforms uncertainty into insight.  It shifts your role from firefighting to foresight. Follow Carlos Shoji for more insights on project management

Let's deploy an ML model to production. Step by step ⬇️ 𝗘𝘅𝗮𝗺𝗽𝗹𝗲 Imagine today is your first day as an ML engineer at Uber, and your task is to improve the ML service that predicts the Estimated Time of Arrival (ETA), which gives users an estimate of when their driver will arrive. 𝗔𝘀 𝗮 𝘀𝘁𝗮𝗿𝘁𝗲𝗿 You -> define the problem -> talk to other data engineers and ML engineers in the team -> expand the set of model features, and -> train a promising ML model. You are happy with the results, so it is time to deploy So far you have 4 things: -> the model pickle -> a predict function, that maps input features to output predictions, using the model artifact. -> A FastAPI wrapper around your Python code, to build the REST API, and -> A Dockerfile to package everything in an isolated box. How do you deploy this code as a production ready API? 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻 -> 𝗧𝗵𝗲 𝗠𝗟𝗢𝗽𝘀 𝘄𝗮𝘆 Let’s go step by step 𝗦𝘁𝗲𝗽 𝟭 → Push code to new branch and open Pull Request (PR) Use git and Github/Gitlab to track code changes. I recommend you develop your code in a non-master branch, where you work, and push changes to remote. Once you are happy with the results, you open a Pull Request 𝗦𝘁𝗲𝗽 𝟮 → Continuous Integration (CI) The CI pipeline is a github action that is triggered automatically every time you open a Pull Request. The end goal is to make sure that your new code version, and model artifact are actually good and deserve to be promoted to production. The CI steps in this case are: -> Unit tests, e.g. make sure you feature engineering functions work as expected -> Trains the model and generates the model artifact. -> Validates the ML model performance and possible biases, using a library like PyTest or Giskard At the end, the pipeline checks can either → fail, so you need to work further on your code, or move to another project with higher priori → pass, so your model is ready to be deployed. In this case, the CI pipeline pushes the model to your model registry, from where it can later be deployed, and your branch “my_new_model” gets merged to master. 𝗦𝘁𝗲𝗽 𝟯 → Continuous Deployment (CD) The Continuos Deployment (CD) pipeline is another github action that is triggered automatically after successful completion of the CI pipeline. The role of the CD pipeline is to deploy the Dockerfile you wrote as a production ready API. In this case, our github action -> Pushes the Docker image to Uber’s docker registry, -> Triggers the deployment to your compute platform, for example - Kubernetes, with kubectl - AWS Lambda, or - Serverless platforms, like Beam or Cerebrium that do not require Docker images. ---- Hi there! It's Pau 👋 Every day I share free, hands-on content, on production-grade ML, to help you build real-world ML products. 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and 𝗰𝗹𝗶𝗰𝗸 𝗼𝗻 𝘁𝗵𝗲 🔔 so you don't miss what's coming next #machinelearning #mlops #realworldml

Interview Conversation Role: RTE Topic: Continuous delivery pipeline 👨💼 Interviewer: "Can you explain how the Continuous Delivery Pipeline fits into the SAFe framework?" 👩 Candidate: "It’s the process for delivering software continuously, and it’s divided into phases like development and deployment." 👨💼 Interviewer: "Interesting. Now, imagine a scenario: your teams are consistently missing deadlines in the release phase because of integration issues. Stakeholders are unhappy, and the velocity of delivering business value is dropping. How would you resolve this?" 👩 Candidate: "I’d ask the teams to collaborate better during development to avoid integration issues later." What the RTE should have answered: ------------------------------------------ An RTE must understand that the Continuous Delivery Pipeline is more than a process—it’s a mindset of flow, feedback, and constant improvement. ✍ In this situation, I’d first assess the pipeline’s current state using SAFe’s DevOps Health Radar to identify bottlenecks in integration or deployment phases. For example, are we missing automated testing, or is the staging environment a bottleneck? ✍ I’d facilitate a Value Stream Mapping workshop with all ART teams to identify inefficiencies, improve handoffs, and define actions for smoother integration. ✍ To address stakeholder concerns, I’d introduce frequent System Demos, aligning everyone on incremental progress and ensuring early feedback to avoid surprises during release. ✍ A real-world example: In my previous role, we resolved late integration issues by introducing Continuous Integration checkpoints at least twice a sprint. This improved collaboration and flagged issues early, drastically reducing delays. Why it matters: 💡 The Continuous Delivery Pipeline in SAFe isn’t just about speed—it’s about delivering high-quality, business-aligned value efficiently. 💡 As an RTE, coaching teams to adopt DevOps practices like automation and early integration strengthens predictability, improves quality, and keeps stakeholders confident. #SAFe #RTE #ContinuousDeliveryPipeline #ScaledAgile #AgileLeadership #ReleaseTrainEngineer

Continuous Integration (CI) is a software development practice where code changes are automatically built, tested, and integrated into a shared repository multiple times a day. Here are several reasons why CI is essential in modern software development: 1. **Early Detection of Bugs**: CI enables developers to identify bugs and integration issues early in development. By automatically running tests with each code change, bugs can be caught and fixed quickly, reducing the cost and time associated with resolving them later in the development cycle. 2. **Faster Feedback Loop**: With CI, developers receive immediate feedback on the impact of their code changes. This rapid feedback loop encourages developers to write more reliable code and helps them iterate more efficiently. 3. **Improved Code Quality**: Continuous integration encourages best practices such as writing modular, testable code and adhering to coding standards. This focus on quality leads to more maintainable and robust software. 4. **Reduced Integration Risks**: CI ensures that changes made by different developers work well together. Integration issues are identified and resolved promptly by continuously integrating code into a shared repository, reducing the risk of conflicts during the later stages of development. 5. **Streamlined Development Process**: CI automates the process of building, testing, and deploying software, making it more efficient. Developers can focus on writing code rather than spending time on manual tasks such as building and testing. 6. **Enhanced Collaboration**: CI promotes collaboration among team members by providing a centralized platform for sharing code changes and feedback. It encourages transparency and helps team members stay synchronized throughout the development process. 7. **Support for Agile and DevOps Practices**: CI aligns well with Agile and DevOps principles by enabling frequent, incremental software updates and promoting a culture of collaboration, automation, and continuous improvement. #CICD #Devops

Some commenters on my last post asked how to transition to trunk-based development. It's a good question. I understand that many teams might find moving to trunk-based development challenging due to a number of reasons (environmental and cultural factors included). It can feel like too big a shift from long-lived branches and traditional post-development code reviews. However, it does not need to happen all at once. In fact, it should be treated as an agile project: the key is to adapt gradually, breaking the process into small, manageable steps and refining it over time. As a general guideline, the agile roadmap could be split into a few major milestones: 1. Reduce branch lifespan: If your team currently works with long-lived branches, start by shortening their lifespan to a maximum of 1-2 days. This enables continuous integration and reduces merge conflicts. 2. Embrace social programming (team-focused development): Work collaboratively in pairs, quartets, or mobs so that most code review happens during development. This allows the final review to either be removed entirely or reduced to a quick pre-merge check. 3. Shorten branch life further and remove them if possible: With a reliable local commit and acceptance-phase testing strategy, reduce branch lifespan even further, ideally to just a few hours, with the goal of removing them entirely. This brings the team even closer to continuous integration. 4. Adopt test-driven development: This increases confidence in frequent commits and eliminates the need for lengthy review cycles. A pair or quartet should be able to commit and push small, fully-tested change sets at the end of every red-green-refactor cycle, ideally every few minutes. 5. Improve knowledge sharing and ownership: For teams used to working in fixed pairs or independently, introducing rotating pairs or quartets may feel like a significant shift, but it is highly beneficial for TBD. It enhances knowledge sharing, reduces dependencies, and improves code quality before it reaches the trunk. Start with voluntary rotations, allowing engineers to switch partners at natural points, such as the start of a new feature. Then establish a regular cadence, aiming for pair rotations at least once a day or after major commits. Expand to quartets, where two pairs collaborate, further strengthening knowledge-sharing and enabling collective ownership. Each of these steps above can be implemented as agile iterations. At regular intervals, the team can reflect and adjust. Every successive step can inform the previous, and step 5 can continuously cycle back to step 1 to further refine the process (Kaizen!) The idea is that small and short is always good (at least in software!). #trunkbasedevelopment #testdrivendevelopment #teamfocuseddevelopment #mobprogramming #pairprogramming #softwareteaming #ensembleprogramming #socialprogramming #softwareengineering #softwaredevelopment

Continuous Integration with Ab Initio + Jenkins In the modern data engineering world, agility and automation are everything. Combining Ab Initio with Jenkins brings the best of ETL development and DevOps together, streamlining workflows from requirement gathering to deployment. Here’s how the cycle flows: Requirements → Development → Testing → Deployment Business users and analysts define requirements via JIRA. Developers build and commit code/test cases into Ab Initio EME. Automated pipelines in Jenkins trigger builds, run validations, and integrate test data. Unit, integration, and UAT testing ensure quality and reliability. Once validated, code is marked complete and seamlessly moved to production. Benefits of this CI approach: Faster release cycles with reduced manual intervention Improved collaboration across business, development, and QA Automated validations ensuring higher quality & stability Visibility and traceability across the entire lifecycle Continuous Integration isn’t just about faster delivery — it’s about building confidence in every release. With Ab Initio + Jenkins, organizations can bridge ETL complexity and DevOps speed. #AbInitio #Jenkins #CI_CD #ETL #DataEngineering #Automation #DevOps #ContinuousIntegration #DataPipelines #C2C #DataEngineer #SeniorDataEngineering