Saikrishna Pujari

Snowflake: Did I forget to add salt to the biryani? curl --location 'https://lnkd.in/gkGpnU4G' \  --header "Authorization: Bearer abcd" The /api/v2/databases endpoint also returns 404. This confirms that the Snowflake REST API v2 endpoints are not accessible with PAT authentication in your account configuration. Summary: ❌ REST API /api/v2/* endpoints return 404 with PAT Bearer token authentication ✅ Python SDK with PAT works perfectly

6

Another fantastic blog from Ganesh Bharathan on query optimization for Snowflake! Read his analysis with example code below! In the blog he provides a deep dive into the following topics: 🏆 Data modeling and query compilation 🏆 Concurrency and warehouse sizing 🏆 Join explosion 🏆 Large table scans 🏆 Data retention and archiving 🏆 Disk spillage 🏆.Complex queries https://lnkd.in/ergJqeQf

23

1 Comment

🍺 My Architecture Today: Works great. 1. Snowflake External Volume --With credential-vending, this may no longer be required. 2. Catalog Integration with Polaris – REST integration – Automatic synchronization 3. Catalog-Linked Database – We need to provide an External Volume so that we can perform CRUD operations on Iceberg tables from the Snowflake engine. – Snowflake does not follow a hierarchical directory structure; all files are written into a single directory. – Fictitious partition key value issue: For example, if my table has a partition key "st", the AWS directory has some unknown value (in my case, 94). – A Spark rewrite fixes the partition key directory name. – Even my grandma — who passed away 50 years ago — could write Snowflake SQL. – Snowflake RBAC – I have not tested Snowflake masking; I am sure it will work. 4. Spark Engines – CRUD operations – Table maintenance 🥃 🥭 With the new architecture, as per the image below, I’m expecting the following improvements: 1. No need for an External Volume. --Credential vending should take care of it. 2. Support for minimum Iceberg features – Table maintenance – Correct directory structure handling – Assume some organizations do not have Spark available 😡 Btw: Spark is still the Biryani of Iceberg. If you don't know how to cook Biryani(Spark) time to learn. Comments.

23

1 Comment

Claude, ChatGPT, Snowflake Intelligence, Cortex Analyst, Databricks Genie, Zenlytic, TextQL. Think about any AI agent you connect to your data. They all get questions from business users like: “What was the average number of daily active users in the last quarter?” These AI agent connects to Euno’s MCP server to retrieve metadata. That’s when Euno gives these agents the missing context they need to understand your data. ** Just a reminder that Euno is a metadata platform so we only access your metadata ** Euno runs a semantic search and looks for dashboards that use DAU (which is semantically equivalent to Daily Active Users). It filters for certified dashboards using the trust signals you’ve already set in Euno. In this example Euno finds two candidates: 1. DAU used in the User Engagement dashboards 2. DAU_NEW used in the Product Statistics dashboard Both dashboards are certified. But Euno also shows how often each has been used in the past 30 days, along with the owner or creator. With that information, the business user can confidently choose the right one and the AI agent can deliver the correct answer. The same applies to tools like Cursor. Let’s say you’re an engineer and ask: “Can you find me the Tableau dashboard for Customer Success?” Cursor will reach out to Euno’s AI assistant to get that BI context because Cursor is focused on dbt development and doesn’t have access to that layer. You can even go further and ask which dbt models power that dashboard and how those models are calculated. Euno feeds Cursor with that real-time context as you build. It’s not just for business users. It’s for developers too.

25

5 Comments

The term “Hadoop Cluster” can have many meanings. In the most simple sense, it can refer to the core components: HDFS (Hadoop Distributed FileSystem), YARN (Resource scheduler), Tez, and batch MapReduce (MapReduce processing engines). In reality, it is much more. Many other Apache big data tools are usually provided and resource-managed by YARN. It is not uncommon to have HBase (Hadoop column-oriented database), Spark (scalable language), Hive (Hadoop RDBS), Sqoop (database to HDFS tool), and Kafka (data pipelines). Some background processes, like Zookeeper, also provide a way to post cluster-wide resource information. https://lnkd.in/gMfHssUn Every service is installed separately w

2