IT Auditing Practices

Most AI audit programs test the wrong things. 😓 Not because the people running them don't know what they're doing, but because AI systems require three fundamentally different types of audits, and most programs are only running one. The core problem with applying traditional audit frameworks to AI is that traditional audits operate in a world of deterministic systems. A financial audit verifies whether transactions were recorded correctly. An IT audit verifies whether access controls functioned as designed. There is a correct answer, and you're checking whether reality matches it. AI systems are often probabilistic. A model never achieves perfect accuracy. That requires a different auditing logic entirely. Governing Intelligence by Noah M. Kenney defines three overlapping audit types, each asking a fundamentally different question: 1️⃣ Technical Audit: "Does this system work as designed?" This is the audit most teams are running. It covers model accuracy on holdout test data, performance consistency across demographic subgroups, robustness against adversarial inputs and distribution shift, and edge case behavior. Test sets must be representative of real deployment conditions, not just training conditions. A model tested only on data similar to its training set will look far more capable than it is. Subgroup testing is non-negotiable. 2️⃣ Algorithmic Audit: "Is this system fair, and does it reflect the values we've stated?" This is the audit most teams are not running systematically. It requires defining fairness metrics appropriate to the decision context, measuring whether the model meets those metrics, and acknowledging that no single fairness definition is universally correct. Demographic parity (equal outcome rates across groups), equalized odds (equal error rates across groups), and calibration (equal accuracy of predictions across groups) cannot all be simultaneously satisfied when true outcome rates differ across groups. This is the Impossibility Theorem in practice. The governance obligation isn't to satisfy all fairness metrics, it's to consciously choose which metric applies to your context, document why, and accept accountability for that choice. 3️⃣ Compliance Audit: "Does this system meet the regulatory requirements that apply to it?" This is documentation review, process verification, and regulatory gap analysis. Does the system have a completed Data Protection Impact Assessment? Is technical documentation current and accurate? Were conformity assessment requirements met before deployment? Are incident reporting procedures in place and tested? Are human oversight mechanisms functional rather than ceremonial? The compliance audit catches the gap between what governance documents claim and what governance infrastructure actually exists. Drop a comment on which of the three audit types is most underdeveloped in your program right now? #AIGovernance #AIAudit #GRC #RiskManagement #Compliance

I wish someone had shown me this pyramid on Day 1 of my IT audit career. Would've saved me 6 months of confusion. When I started, I jumped straight to controls. Access reviews. Change management. Backup testing. I was checking boxes. But I had no idea WHY those controls mattered. No one told me to start at the top of the pyramid. The Business. What does this company actually do? How do they make money? What goals are they chasing? Without understanding that, every control I tested felt random. Then one day, my manager asked me: "Chinmay, why this IT Application is in scope for our audit?" I froze. Because I was testing controls in isolation. I never connected controls to IT apps and IT apps to the business process. Great auditors don't start at the bottom of the pyramid. They start at the top. You can't test what you don't understand. This framework changed everything for me. Understand the business → What goals drive this company? Map the core processes → What processes support those goals? Identify the applications → What systems enable those processes? Evaluate IT risks → What can go wrong in those systems? Test the controls → What mitigates those risks? Top to bottom. Always. If you're confused about where to start, save this infographic. Print it. Keep it at your desk. Because the biggest mistake I made wasn't bad testing. It was testing without context. Learn IT audit the way it's actually done. Because clarity is the difference between doing audit and understanding it. Tag someone who needs to see this framework. #itaudit #audit #risk #compliance #internalaudit #cisa #isaca

"💡 What is internal audit? Internal audit is an organizational function that evaluates the adequacy and effectiveness of a company’s risk management, control, and governance processes. It is organizationally independent from senior management and reports directly to the board of directors. 🏛️ The role of internal audit in corporate governance In the Three Lines Model, which is a popular risk governance framework (https://lnkd.in/e8dWA8_D), internal audit serves as the third line and is responsible for providing independent assurance to the board of directors. But internal audit is not the only assurance provider. In many companies, the board also gets reports from compliance, external auditors, etc. To avoid blind spots and “assurance fatigue”, the different assurance activities need to be coordinated (e.g. by using the same terms, taxonomies, and threat models). Internal audit is often responsible for ensuring this coordination. ✅ Why frontier AI developers need an internal audit function Internal audit can identify ineffective or inadequate risk management practices. This is important because, without a deliberate attempt to identify flawed practices, some of them will likely remain unnoticed. For example, developers' model evaluations might be inaccurate or unreliable (see https://lnkd.in/embgeirH) or their information security might be inadequate (see https://lnkd.in/d9MTCgKx). Internal audit can also ensure that the board has a more accurate understanding of the current level of risk and the adequacy of risk management practices. For example, internal audit could verify if the company actually complies with its AI safety framework (see https://lnkd.in/e2ZnMyYT). ❌ Limitations But frontier AI developers should also be aware of key limitations: Internal audit adds friction, it can be captured by senior management, and the benefits depend on the ability of individuals to identify ineffective practices. In light of rapid progress in AI research and development, frontier AI developers need to strengthen their risk governance. Instead of reinventing the wheel, they should follow existing best practices. Although this might not be sufficient, they should not skip this obvious first step." Paper (and summary) are from Jonas Schuett and the Centre for the Governance of AI (GovAI).

Auditing is proposed in laws, regulations, and industry guidelines to mitigate AI risks, but there's a lack of established norms and standardized practices for compliance and assurance audits. Despite varied approaches like adversarial pressure testing and quantitative assessments, consensus on norms and practices is still evolving. The term 'audit' is used broadly to encompass diverse evaluations of algorithmic tools, including pressure-testing by external entities, internal pre-deployment assessments, collaborative audits, and external audits ensuring compliance with legislative or standardized framework requirements. External audits differ from risk or impact assessments in two main aspects. Firstly, algorithmic impact or risk assessments primarily focus on internal evaluations. Secondly, external audits require a conclusive outcome for stakeholders to act upon, while risk or impact assessments usually provide open-ended outputs, such as prioritized lists of risks or impacts. This paper below specifically focuses on 'external audits,' also known as 'compliance audits,' which aim to ensure adherence to specified requirements. This paper introduces the 'criterion audit' as a practical way to do external audits, inspired by how financial audits work. It is defined as: "A criteria-based independent external evaluation E of an algorithmic system S conducted by an auditor A to determine whether the given system S meets the requirements set by a normative framework." The criterion audit is characterized by 4 key features: 1. Standardized Criteria: Transparent evaluation against publicly accessible criteria. 2. Normative Framework: Measuring compliance against a specific normative framework. 3. Auditor Training: Standardized training and accreditation for auditors. 4. Public Disclosure: Results disclosed, ensuring transparency while addressing security concerns. The standard process for a criterion audit includes target scoping, documentation submission, evidence verification, publication of the audit report, and certification of the audited algorithmic system based on the evaluation against normative framework requirements. The paper demonstrates the application of the proposed approach to comply with NYC Local Law 144. The paper stresses that auditors for the criterion audit, like financial auditors, need professional values, subject matter expertise, and rigorous audit processes. It advocates for standardized audit training and suggests combining this with responsible AI education for a comprehensive understanding of complex considerations in algorithm audits. Title: "A Framework for Assurance Audits of Algorithmic Systems": Authors: BABL AI research team, led by Khoa Lam, Dr. Benjamin Lange, and Borhane Blili-Hamelin, PhD. Contributions from Shea Brown, Jovana Davidovic, and Ali Hasan.

Six Guides for Applying ISO/IEC 27002:2022 – Organizational Controls When it comes to information security audits, non-conformities often arise not from a lack of policies, but also from a lack of practical implementation. That is why I have authored a 6 part book series entirely dedicated to the first domain of ISO/IEC 27002:2022 (Organizational Controls). With over 50 certifications across IT disciplines (Governance, Cybersecurity, Networks, Systems, Development, DevSecOps, Cloud, etc.), I decided to go beyond theoretical knowledge to offer concrete, actionable guidance. Each book focuses on a specific set of organizational controls and includes: How to implement each control in practice Tools, templates, and methodologies Audit insights: what auditors actually look for How to close gaps and resolve non-conformities effectively This work proves that certification does not exclude competence; on the contrary, when combined with real-world experience, it builds a strong foundation for excellence and compliance I invite CISOs, auditors, consultants, and infosec professionals worldwide to explore this series. It is not about theory; it is about turning controls into tangible, auditable, and effective actions. Access the full series here : https://lnkd.in/eZFXP2QA

Dear IT Auditors, The Forgotten Step in IT General Controls (ITGC) Reviews Every IT auditor can list the core ITGCs, access management, backups, change management, and job scheduling. These are the foundation of assurance work. But here’s the trap: many ITGC reviews fail, not because the controls are missing, but because accountability is missing. You can perform flawless testing of access rights, backup restores, and system changes. But if no one truly owns the control, your results are only temporary. The moment something breaks, there’s no one accountable to fix it. In ITGC audits, some auditors may skip validating control ownership by not asking questions such as: 📌 Who owns the control, and do they actually know they own it? 📌 Is ownership documented in policies, procedures, or job descriptions? 📌 When failures occur, is there a clear escalation path? 📌 Do control owners receive automated alerts or dashboards to monitor effectiveness? 📌 When staff turnover happens, is ownership reassigned formally? 📌 Are control owners trained regularly so accountability isn’t just “on paper”? A strong ITGC environment isn’t only about design and operation. It’s about execution, which depends entirely on people. Systems don’t keep themselves secure. People do. And when ownership is unclear, every test result is just a snapshot of temporary success. That’s why, in my ITGC reviews, I don’t just ask “Is the control effective?” I ask “Who’s responsible, and are they equipped to own it?” Because in the end, control ownership may the difference between resilience and failure. #ITGC #ITAudit #AccessControls #ChangeManagement #RiskOwnership #ControlEffectiveness #AuditExecution #CyberGRC #CyberYard #CyberVerge

Digesting your Internal Audit Methodology The introduction of new Global IIA Standards this year has prompted many teams to refine and formalize their internal audit approach and processes. For some teams, significant changes are being made to audit planning, data analytics integration, and documenting and reporting of audit results. As a result, methodology manual begins to swell, with more and more pages of required activities added. A concern many CAEs should have is how to make sure all of these methodology changes are not just adopted, but how will they be internalized and stick with the rest of the Internal Audit team. For those teams looking to drive further adoption, here are some ideas they can consider incorporating to make their audit methodology easier to internalized and carry-out. 1. Incorporate more visuals and instructional videos into your Internal Audit methodology. This approach helps reduce lengthy and detailed narrative , while still communicating what needs to be done. Your team is unlikely to read the IA methodology word-for-word, so there's no need to document it in that manner. 2. Socialize the changes before enforcement. Many Internal Audit teams aim to implement methodology changes in Q1 2025. Now is an ideal time to introduce these major changes in team meetings, project status updates, and informal conversations. By the time the changes are implemented, most team members should be thoroughly familiar with them. 3. Have a formal onboarding program. For new team members, dedicate time to formal training during their first week and month. This approach effectively introduces your team's audit methodology, ensuring a thorough understanding from the start. 4. Implement testing or certification for changes. While some team members may initially resist, establishing a process to test staff and seniors on their understanding of new methodologies can prevent mistakes and ensures that everyone grasps how to incorporate the proposed changes effectively. 5. Leverage your Audit Technology. Purpose-built audit management software often includes features like models (pop-up screens that reinforce necessary actions) or gates (automated checks that prevent users from advancing until specific tasks are completed). Consider incorporating these tools to "implement once, enforce many," streamlining the adoption of your new methodology. 6. Limit the frequency of updates. While it's tempting to continuously improve your internal audit methodology, frequent changes can frustrate your team. They may become overwhelmed and start ignoring updates. Instead, restrict changes to once or twice a year. Make these updates significant events, and wait until the next scheduled "change" opportunity to incorporate additional feedback.

✅ How to Perform SOX Control Testing If your company is scaling fast or approaching IPO readiness, mastering SOX control testing is critical. Done right, it builds trust with your auditors and executive team. Done wrong, it creates chaos at quarter-end. Here’s a clear, repeatable playbook every IT Audit or Compliance leader should know 👇 1️⃣ Confirm Scope & Objectives Define significant accounts, systems, and assertions. Align early with external auditors. 2️⃣ Build the Control Universe (RACM) Map controls to risks and assertions. Assign clear owners and evidence expectations. 3️⃣ Define Your Testing Strategy Document nature, timing, and extent of testing. Pre-agree sampling with auditors. 4️⃣ Perform Walkthroughs Trace transactions end-to-end. Confirm control precision, completeness, and accuracy. 5️⃣ Validate IPE (Information Produced by the Entity) Reperform reports and tie outputs back to the system of record. Never assume your data is accurate—prove it. 6️⃣ Test ITGCs (Logical Access, Change Mgmt, Ops) ✅ Password configurations ✅ Admin access reviews ✅ Code change approvals ✅ Backup and job monitoring 7️⃣ Execute Business Process Testing Sample revenue, journal entries, and reconciliations. Validate approvals and timeliness. 8️⃣ Evaluate Exceptions & Report Deficiencies Quantify, classify, and document remediation. Escalate early. 9️⃣ QA Review & Close Independent review, retention, and lessons learned. ⚙️ Common Pitfalls (and Fixes) 🚫 Stale control descriptions → ✅ Rewrite for clarity & precision 🚫 Weak IPE validation → ✅ Tie every report to its data source 🚫 Missing evidence → ✅ Centralize PBC requests and SLAs 🚫 Sampling debates → ✅ Standardize sample size matrix 💡 Pro Tip: Consistency beats perfection. A disciplined weekly cadence, clear evidence trails, and early stakeholder alignment are what separate “audit ready” from “audit reactive.” #SOX #InternalAudit #ITCompliance #TechnologyRisk #AuditLeadership #ITGC #SOX404 #GRC #RiskManagement #CISO

IT internal audit controls: 1. Access Controls: Control: Implement measures to ensure only authorized personnel have access to systems and data. Audit Point: Review user access logs, permissions settings, and authentication mechanisms. Check for instances of unauthorized or inappropriate access. 2. Change Management: Control: All changes to IT systems, especially production environments, should follow a formal change management process. Audit Point: Examine documentation related to system changes. Ensure approvals were obtained and testing was performed before deployment. 3. Backup and Recovery: Control: Regular backups of critical data and systems should be performed. Recovery processes should also be established. Audit Point: Validate the frequency and success rate of backups. Test the recovery process for effectiveness. 4. Network Security: Control: Secure the organization's network through firewalls,intrusion detection systems, and regular vulnerability assessments. Audit Point: Review network security logs and assess the efficacy of security devices. 5. Physical Security: Control: Implement security measures to prevent unauthorized physical access to critical IT infrastructure (e.g., data centers). Audit Point: Inspect physical access logs and security measures in place at data centers and server rooms. 6. Data Encryption: Control: Ensure that sersitive data, especially during transmission, is encrypted. Audit Point: Check encryption standards employed and assess their adequacy based on the sensitivity of the data. 7. Incident Management: Control: Establish a process for identifying, responding to,and reporting security incidents. Audit Point: Review incident logs and assess the organization's response to past incidents. 8. Vendor Management: Control: Vendors with access to the organization's IT systems should adhere to the same security standards. Audit Point: Examine contracts and agreements with vendors. Check for clauses related to IT security and assess vendor compliance. 9. Application Controls: Control: Controls within specific applications to ensure the integrity and accuracy of transactions and data. Audit Point: Test critical transaction flows within applications for any anomalies. 10. Patching and Up-dates: Control: Regularly update and patch IT systems to protect against known vulnerabilities. Audit Point: Review the patch management process. Check for outdated systems. 11. Disaster Recovery and Business Continuity: Control: Develop and maintain a disaster recovery plan. Ensure business continuity even in the face of major IT disruptions. Audit Point: Evaluate the disaster recovery plan's comprehensiveness. Conduct or review results from periodic disaster recovery drills. 12. User Training and Awareness: Control: Regularly train users on IT security best practices and raise awareness about potential threats. Audit Point: Assess the frequency and content of training programs. Check for user awareness and adherence.

Internal Audit Process: 1. Planning Phase Objective: Establish a clear understanding of the audit subject and develop a roadmap (audit program) for executing the audit effectively. Key Activities: > Initial Contact & Information Gathering: Understand the size, responsibilities, and procedures of the audited unit. > Risk Assessment: Performed to identify high-risk areas for focus. > Audit Objectives & Methodology: Defined and documented through the audit program. > Notification Letter: Sent to leadership to inform them of the audit. May include a pre-audit questionnaire or document request list. > Entrance Meeting: Discuss audit scope and objectives. Explain methodology and timeline. Identify scheduling concerns (e.g., staff availability). Encourage input on known risks and areas of concern. 2. Fieldwork Phase Objective: Evaluate internal controls, compliance, and operational effectiveness through testing and inquiry. Key Activities: > Testing & Documentation Review: Examine transactions, records, and procedures. > Staff Interviews: Conducted to gain deeper insights into practices and control execution. > Disruption Minimization: Work is coordinated to limit interference with operations. > Ongoing Communication: Frequent updates and discussions with audit clients. > Collaborative Analysis: Observations and issues are discussed with management to identify root causes and explore solutions. 3. Reporting Phase Objective: Present audit findings, recommendations, and management’s corrective action plans in a formal written report. Key Activities: > Draft Report: Initially shared with local management for review. > Management Response: Required for each recommendation, including: Action plan. Responsible person. Implementation date. > Exit Meeting: Held if needed to address concerns and clarify findings before finalizing the report. > Final Distribution: The final report is sent to Management and Boards. 4. Follow-Up Phase Objective: Ensure that corrective actions are implemented effectively and that issues are resolved. Key Activities: > Verification Procedures: May involve document review, staff interviews, or re-auditing specific processes. > Ongoing Tracking: Open findings are tracked and presented at each Institutional Audit Committee (IAC) meeting. > Escalation for Delays: If action plans miss deadlines, the responsible party must submit a written explanation. Repeated delays require in-person explanation to the IAC.