Pavithra Arunachalam’s Post

We’ve all been there: a manager asks for an inventory or an interface status report, and you’re stuck logging into 20 switches and manually formatting an Excel sheet. In Part 3 of my Nornir series, I’m showing you how to turn that manual chore into a one-click automated workflow: ✅ Parsed Data to CSV: Turn structured TextFSM output into spreadsheets. ✅ Automated Backups: Pulling configs with timestamped filenames. ✅ Error Handling: Making sure one unreachable device doesn't crash your whole script. If you’re ready to move from "running commands" to "generating value," check out the full guide on The Next Hop. https://lnkd.in/eg6hYedd GitHub repository for the series: https://lnkd.in/eiKddxiy #NetworkAutomation #Nornir #Python #NetDevOps #Cisco

𝗧𝗵𝗶𝘀 𝗜𝘀 𝗔 𝗛𝗲𝗮𝗱𝗹𝗶𝗻𝗲 I deployed my Python API to production. All tests passed. I felt good about it. But Monday morning, I got support messages. My API was returning 500 errors for about 30% of requests. My code parsed JSON from an external API. I used the requests library to get user data. But my tests mocked the API response with perfect JSON. Every field was present. The nesting was proper. Exactly what I expected. But not what I got in production. The production API returned incomplete data. Sometimes the profile object was missing. Sometimes the contact email was missing. The JSON was still valid, but structured differently. My tests did not check for this. I wrote happy path mocks. Here's what I learned: - New users may be missing profile data - Deactivated accounts may be missing contact info - International users may have different field names I changed my code to use defensive access patterns. I added tests with missing fields. Now I get the data with defaults instead of assuming the structure. - I use `data.get('profile', {})` to get the profile data - I use `profile.get('name', 'Unknown')` to get the name I also added tests for missing fields: - I test for missing profile data - I test for empty responses You should test for edge cases. Testing only perfect scenarios is not enough. I am still finding edge cases in production. Source: https://lnkd.in/gM5Kz2ej

Database transactions don't prevent two requests from reading the same row simultaneously. Wrapping code in atomic() makes it safe from concurrent modification. That assumption is wrong in a specific and destructive way. atomic() alone doesn't prevent race conditions. It only guarantees atomicity - all or nothing. It says nothing about what other transactions can read while yours is running. select_for_update() is what actually locks the row. How select_for_update() works - 1. When Django executes SELECT ... FOR UPDATE - the database places a lock on that row. 2. Any other transaction attempting to read the same row with select_for_update() blocks - it waits until the first transaction commits or rolls back. 3. Only then does the second transaction proceed - with the updated value. No simultaneous reads. No conflicting writes. The real Catch! select_for_update() must be inside an atomic() block. Always. Outside a transaction - most databases silently ignore the lock entirely. No error. No warning. No lock. nowait and skip_locked can be used for controlling wait behaviour. nowait=True -> if the row is already locked, raise an exception immediately instead of waiting skip_locked=True -> if the row is locked, skip it and move on. Useful for task queues where any available row is acceptable. Takeaway - -> Transactions tell the database - treat these operations as one. -> Locks tell the database - treat this row as mine until I'm done. -> Both are needed. Neither replaces the other. What concurrency bug has cost you the most debugging time - race condition, deadlock, or something else entirely? #Python #Django #BackendDevelopment #SoftwareEngineering

Every backend developer has faced this dilemma: ❌ Auto-increment IDs? You're leaking your business volume to anyone paying attention. order_id=100420 at 10am, 100780 at 2pm = 360 orders in 4 hours. Your growth curve, exposed. 📉 ❌ UUID v4? 128-bit random keys destroy your B+ tree index performance once data outgrows memory. Every insert becomes a cache miss. 😩 ❌ Snowflake-style timestamp IDs? You married the wall clock. NTP step backward? VM resume after snapshot? Your ID generator either emits duplicates or stalls. ⏱️ There's a fourth option — and it sidesteps all three. permid64 generates 64-bit IDs from a simple idea: 🔑 "Uniqueness comes from a counter. The random-looking surface comes from a reversible permutation." A persistent counter provides strictly monotonic uniqueness — no wall clock involved, ever. A bijection over 64 bits maps those sequential values into opaque-looking IDs that reveal nothing about your business volume. The best part? It's FULLY DECODABLE. When an anomalous ID appears in a production log, you can instantly recover the instance and sequence number — no database lookup required. 👇 Quick look at the Python implementation: from permid64 import Id64 gen = Id64.multiplicative(instance_id=42, state_file="orders.state") token = gen.next_base62() // e.g. '3kTMd92Hx7Q' print(f"New Order: ORD_{token}") // Incident tracing: decode instantly meta = gen.decode_base62('3kTMd92Hx7Q') print(f"Instance: {meta.instance_id}, Sequence: {meta.sequence}") You get compact, URL-safe tokens for external use — and full traceability for ops. No trade-off required. 📦 pip install permid64 🔗 Full technical deep-dive (B+ tree analysis, bijection proofs, Feistel network internals) in the comments 👇 #Python #SoftwareEngineering #BackendDevelopment #Database #IDGeneration #Microservices #OpenSource

We need to talk about context windows, because they are fundamentally changing how we debug. 🗄️ As a Data Manager, I spend a lot of time buried in documentation—whether it’s tracing a weird SQL error across multiple tables or trying to understand a massive new Python library. In the past, you couldn't paste a massive dataset or a giant code repository into an AI without it "forgetting" the first half of your prompt. Enter Gemini 3.1 Pro. With its massive context window, I’m no longer feeding it bite-sized snippets. I’m dropping in entire: API documentation PDFs Multi-layered ETL script logs Massive, messy JSON files It doesn’t just "read" them; it retains the architecture of the problem. It’s like having a senior engineer who has already memorized the entire codebase before you even start the meeting. If you are still using AI just to write single lines of code, you are missing out on its ability to understand the entire system.

🚀 Day 19 — Transactional Outbox, Dual-Write Safety, and Reliable Event Delivery One of the easiest ways to break a distributed system is this: ✅ Database write succeeds ❌ Event publish fails Or even worse: ✅ Event publish succeeds ❌ Database write fails At first, both operations look simple. But together, they create one of the most dangerous problems in event-driven systems: 👉 dual-write inconsistency That was my biggest takeaway from Day 19. Because in real systems, when business state changes, the event about that change also matters. And if those two do not stay aligned, downstream systems start operating on incomplete or incorrect reality. Today’s focus was: Transactional Outbox, Dual-Write Safety, and Reliable Event Delivery 📦 What I covered today 📘 ⚠️ Dual-write failure mode fundamentals 🧾 Transactional outbox pattern 🔁 Relay / poller based event publication 📤 At-least-once delivery and retry behavior 🪪 Idempotency implications for consumers ⚖️ Ordering and throughput trade-offs What stood out to me ✅ Writing to the database and broker separately is risky because either side can fail independently ✅ Transactional outbox solves this by storing state change + event intent in the same local transaction ✅ If the transaction commits, both records exist; if it aborts, neither exists ✅ Relay failures should not delete events — they should leave them pending and retry ✅ At-least-once delivery means duplicates are possible, so idempotent consumers are mandatory ✅ Monitoring outbox backlog is important because delayed publishing becomes an operational signal, not just an implementation detail I also implemented a small Transactional Outbox Simulation in Python and Java to make the concept more practical. 🛠️ ➡️ Git: https://lnkd.in/dixudzYY That helped me understand a simple but important idea: 📌 The hardest part is not publishing events 📌 The hardest part is publishing them reliably after state changes 📌 Good distributed design is often about removing small windows where inconsistency can happen This is one of those topics that looks straightforward in diagrams, but becomes much more meaningful when you think about retries, crashes, duplicate publishes, and downstream services depending on those events to stay correct. System Design is slowly becoming less about only changing state and more about making sure every important state change can be observed, published, and recovered reliably. On to Day 20 📈 #SystemDesign #DistributedSystems #BackendEngineering #SoftwareEngineering #ScalableSystems #TransactionalOutbox #EventDrivenArchitecture #ReliableEventDelivery #DualWriteProblem #Microservices #DataConsistency #AtLeastOnceDelivery #Idempotency #OutboxPattern #BackendDevelopment #CloudComputing #TechLearning #EngineeringJourney #SystemArchitecture #Java #Python #Kafka #RabbitMQ #SoftwareArchitecture #DevelopersIndia

I've been playing with Claude Code A LOT and as a data person I love it, but there's one big mistake it's easy to make as a noob. Claude is an eager and autonomous problem solver but it defaults to burning a ton of tokens and doing bespoke computation every time you run it. What do I mean? You ask Claude to pull some data, do some simple transformations, land it in MotherDuck and build a simple report. It does it shockingly fast and the output is shocking good. When you ask it to do the exact same thing the next day it chooses a different path. Maybe it uses different python libraries. It writes fresh SQL that is 10x less efficient at inserting the data. No two runs are alike. You NEED to ask Claude to build the deterministic tools to solve the problem, not solve it directly. Instead of writing the insert fresh every time, make a skill that calls a script. Claude is great at solving problems, but it's terrible at solving the same problem twice.

I noticed this too trying to build a website. Glad to see we are talking about it! Great way to learn on how to use Claude in a smarter way.

Manual utility auditing is traditionally slow, fragmented, and prone to human error. I developed the SPC Portal to bridge the gap between raw data ingestion and real-time operational decision-making. Here’s the breakdown: Unified Ingestion: No more manual downloads. I’ve automated the "monthly grind" by using n8n to programmatically extract Excel attachments from Outlook, normalizing the data into a single PostgreSQL source of truth Automated Benchmarking: The system uses Python to establish "normal" consumption profiles automatically based on historical baselines. Anomaly Detection: The moment consumption deviates, an alert is triggered, allowing the team to investigate spikes before they impact the bottom line. The Result? 100% transparent energy oversight and a significant reduction in financial leakage. Great to see this driving real-world impact for the team! The Stack: Python | Streamlit | PostgreSQL | n8n | Red Hat OpenShift. #DataAnalytics #Automation #Python #ITOperations #Streamlit #n8n #DigitalTransformation #DataEngineering

🚀 Day 20/100: Data Types Deep Dive – Precision, Size & Memory 📊🧠 Today’s learning focused on the science behind data storage in Java. Writing efficient code is not just about logic—it’s about choosing the right data type to optimize memory usage and performance. Here’s a structured breakdown of what I explored: 🏗️ 1. Primitive Data Types – The Core Building Blocks These are predefined types that store actual values directly in memory. 🔢 Numeric (Whole Numbers): byte → 1 byte | Range: -128 to 127 short → 2 bytes | Range: -32,768 to 32,767 int → 4 bytes | Standard integer type long → 8 bytes | Used for large values (L suffix) 🔢 Numeric (Floating-Point): float → 4 bytes | Requires f suffix double → 8 bytes | Default for decimal values 🔤 Non-Numeric: char → 2 bytes | Stores a single Unicode character boolean → JVM-dependent | Represents true or false 🏗️ 2. Non-Primitive Data Types – Reference Types These types store references (memory addresses) rather than actual values: String → Sequence of characters Array → Collection of similar data types Class & Interface → Blueprint for objects 💡 Unlike primitives, their default value is null, and they reside in Heap memory, with references stored in the Stack. 🧠 Key Insight: Primitives → Store actual values (Stack memory) Non-Primitives → Store references to objects (Heap memory) ⚙️ Why This Matters: Choosing the correct data type improves: ✔️ Memory efficiency ✔️ Application performance ✔️ Code reliability at scale 📈 Today reinforced that strong fundamentals in data types are essential for writing optimized, production-ready Java applications. #Day20 #100DaysOfCode #Java #Programming #MemoryManagement #DataTypes #SoftwareEngineering #CodingJourney #JavaDeveloper #10000Coders