Disha Suyogi

Snowflake Intelligence: From Data to Decisions - Atrium's AI: AI projects fail when data quality is poor and siloed. At its core, the Snowflake platform solves this by giving access to unified insights with ...

1

1 Comment

3 Patterns to Productionize Snowflake Cortex Agents. This is the first pattern My “ChartGPT for Snowflake” story It all started with a question I asked at the DFW Snowflake User Group last week. Snowflake had just introduced Snowflake Intelligence, and I got curious about how it could extend beyond Cortex Analyst. With Snowflake Intelligence, I was able to do a Quickstart and spin up an agent that uses tools like Cortex Analyst, Cortex Search Services, and even orchestrations like drafting reports or sending emails. Pretty cool stuff. But my question was — what if people outside of the Snowflake console could use this? What if customers without direct Snowflake access could still query their enterprise data? So, I built a Streamlit app that calls the Snowflake Intelligence agent I created. This way, the agent’s capabilities are available outside Snowflake too. Imagine embedding this into CRMs like Salesforce or into internal dashboards — where teams can ask questions about invoices, analyze support cases, or even salespeople pulling insights directly from their notes. Reason why I nicknamed it “ChartGPT for Snowflake.” is -- Inside Snowflake Intelligence, you get charts by default. But when I pulled the agent out, I only had backend responses — no charts. So, I hacked around it by building my own chart modules in Streamlit. The result? A chat interface that responds with both insights and charts, just like ChatGPT — but for your Snowflake data. #Snowflake #SnowflakeIntelligence #CortexAgent #Streamlit #Innovation

79

6 Comments

https://lnkd.in/eR232ynq Great article by Shweta Shah, PhD Bringing together the Data Needs from a data architecture and governance perspective . Building the data to get the data consumers what they need #generative #predictive #llm #clean #labelled #structured #normalized #scaled #regulatory #gdpr #eu #AI-Ready #datagovernance #privacy #cleandata #usability #contextualcapability #sensitivity #hubandspoke #lineage #metadata #lakehouse #dataarchitecture #aiarchitecture #AIreadiness #datamesh #medallion #semanticlayer #datajourney #compliance #datausers #consumers #KPI #ROI #risk #metadata #sunlife #datastewards #dataowners

8

2 Comments

💡 Real Healthcare Databricks Interview Questions & My Answers (8 Yrs Experience) Sharing 5 real-world scenario-based questions I faced as a Senior Data Engineer in Healthcare 👇 These reflect real production challenges from handling PHI to building scalable Delta pipelines. 1️⃣ How do you ensure HIPAA compliance while processing data in Databricks? 🩺 Answer: “In healthcare projects, I always enforce PHI protection using column-level encryption and access control via Unity Catalog. All PII fields are tokenized before landing in Delta tables, and audit logs are maintained using Databricks audit logs + Azure Monitor. We also integrate with Azure Key Vault for credential rotation and secure secrets management.” 2️⃣ How do you optimize Delta tables storing EHR/Claims data with billions of rows? 📊 Answer: “I apply ZORDER on high-cardinality columns like patient_id or claim_id for faster lookups. Partitioning is done by encounter_date or claim_year to improve pruning. I’ve also used OPTIMIZE with VACPUM scheduling and Delta caching for repeated analytical workloads — achieving a 45% boost in query performance.” 3️⃣ How do you handle incremental updates from HL7/FHIR streaming feeds? ⚙️ Answer: “I built a streaming ingestion using Auto Loader + Structured Streaming. Each micro-batch merges data into the Delta table using MERGE INTO with CDC logic. Watermarking ensures late-arriving updates are captured without duplication. Schema drift is handled via mergeSchema = true since FHIR resources often evolve. 4️⃣ How do you manage schema evolution across Bronze–Silver–Gold layers? 🏗️ Answer: “In our Medallion Architecture, we validate schema consistency at each layer. Bronze ingests raw JSON, Silver applies type casting + deduplication, and Gold provides curated analytics. To manage evolving schemas, I version schemas in Delta logs and maintain a schema registry ensuring backward compatibility.” 5️⃣ How do you build a unified pipeline for batch claims data and real-time clinical events? 🚀 Answer: “I designed a hybrid ingestion framework in Databricks that handles both historical (batch) data and real-time (streaming) HL7 feeds. Batch loads initialize the Delta tables, while streaming updates are applied using foreachBatch. The architecture supports both CDC and append-only streams for lab results, vitals, and patient admissions.” These real-world approaches have helped me build secure, compliant, and high-performing healthcare data platforms using Databricks, Azure, and Delta Lake. #Databricks #HealthcareDataEngineering #Azure #DeltaLake #MedallionArchitecture #CDC #HL7 #FHIR #DataEngineering #InterviewPreparation Kforce Inc IntraEdge Cardinal Health UnitedHealth Group Lakshya Technologies

10

Databricks has open-sourced Dicer, its internal auto-sharding system, Dicer solves a common distributed systems challenge on How you efficiently shard and rebalance data without the operational pain of static partitioning or the inefficiency of fully stateless systems. Why this matters the most ✅ Automatic sharding & rebalancing – No more manual partition management as workloads grow or shrink. ✅ Better resilience – Handles restarts, failures, and autoscaling smoothly, reducing downtime risks. ✅ Lower latency & higher throughput – Smart request routing and state locality improve performance for data-heavy services. ✅ Open source impact – These capabilities are now available beyond Databricks, enabling broader innovation in distributed data platforms. For teams building scalable data processing systems, real-time services, or metadata-heavy platforms, Dicer offers a practical blueprint for balancing performance, reliability, and operational simplicity. #DataEngineering #Databricks #OpenSource #DistributedSystems

8