Vinod Kumar Nerella

The term “Hadoop Cluster” can have many meanings. In the most simple sense, it can refer to the core components: HDFS (Hadoop Distributed FileSystem), YARN (Resource scheduler), Tez, and batch MapReduce (MapReduce processing engines). In reality, it is much more. Many other Apache big data tools are usually provided and resource-managed by YARN. It is not uncommon to have HBase (Hadoop column-oriented database), Spark (scalable language), Hive (Hadoop RDBS), Sqoop (database to HDFS tool), and Kafka (data pipelines). Some background processes, like Zookeeper, also provide a way to post cluster-wide resource information. https://lnkd.in/gMfHssUn Every service is installed separately w

2

A self-evaluating LLM pipeline for custom data extraction.  The idea is a below: 1) LLM #1 extracts structured fields from invoice documents. 2)You manually validate a few samples to create a reference dataset (input + output). 3) LLM #2 then uses this reference to evaluate the consistency or correctness of LLM #1’s outputs. This dual-prompt strategy inspired by Derah Onuorah post last week!! is a clever way to enhance trust in AI-driven workflows but also opens doors to scalable, self-validating automation within the Power Platform ecosystem. https://lnkd.in/ee8X_pfw #aibuilder #powerplatform #powerautomate #llm #powerfuldevs

10

2 Comments

Sovereign AI models were one of the most discussed themes at the recent AI Summit held in Delhi. The core idea is simple but powerful: countries and enterprises want AI systems that are trained, deployed, and governed within their own boundaries — aligned with local languages, regulations, data privacy norms, and cultural context. As highlighted by , the next wave of AI innovation will be driven by models built for local ecosystems — especially Indian languages, public-sector use cases, and enterprise-sensitive workloads. I am keen to explore Sarvam AI APIs to test and build use cases around local language capabilities. Key takeaways: • Data sovereignty is becoming as important as model performance • Governments are investing in national AI infrastructure and compute • Local language models and public-sector applications are accelerating • Enterprises are exploring “sovereign AI stacks” for sensitive workloads Sovereign AI is not about isolation — it’s about control, trust, and strategic independence in an AI-first world.

21

Amazon S3 - One of the biggest storage system! 100s & 1000s of data lakes are built using Amazon S3 today. It is one of the biggest storage system ever built for cloud. In its simplest form, S3 is an object storage service offering high scalability, data availability, security, and performance. Building such a system needs several long lasting considerations. This incredibly detailed blog by Andy Warfield at S3 elaborates on what went into building such a system. Here's a quick glimpse but highly encourage reading the blog/watching the talk to capture granular details. 🌟 Scale Challenges: Operating S3 involves managing vast scale & complexity, with millions of hard drives and numerous microservices interacting. 🌟 Heat Management: S3 faces unique challenges in balancing I/O demand across disks to prevent hotspots and maintain performance. 🌟 Redundancy: S3 uses replication and erasure coding for both durability and load balancing, optimizing for cost and efficiency. 🌟 Replication and Load Balancing: S3 uses data replication not only to protect against disk failures but also to optimize I/O performance by allowing flexible data access, while erasure coding reduces capacity overhead without compromising durability. Blog link in comments. #dataengineering #softwareengineering

43

3 Comments

🍺 My Architecture Today: Works great. 1. Snowflake External Volume --With credential-vending, this may no longer be required. 2. Catalog Integration with Polaris – REST integration – Automatic synchronization 3. Catalog-Linked Database – We need to provide an External Volume so that we can perform CRUD operations on Iceberg tables from the Snowflake engine. – Snowflake does not follow a hierarchical directory structure; all files are written into a single directory. – Fictitious partition key value issue: For example, if my table has a partition key "st", the AWS directory has some unknown value (in my case, 94). – A Spark rewrite fixes the partition key directory name. – Even my grandma — who passed away 50 years ago — could write Snowflake SQL. – Snowflake RBAC – I have not tested Snowflake masking; I am sure it will work. 4. Spark Engines – CRUD operations – Table maintenance 🥃 🥭 With the new architecture, as per the image below, I’m expecting the following improvements: 1. No need for an External Volume. --Credential vending should take care of it. 2. Support for minimum Iceberg features – Table maintenance – Correct directory structure handling – Assume some organizations do not have Spark available 😡 Btw: Spark is still the Biryani of Iceberg. If you don't know how to cook Biryani(Spark) time to learn. Comments.

23

1 Comment

37% ARR growth for an MDM vendor is truly remarkable. That's because the market itself usually only grows around 6-9%, which means that Profisee is clearly doing something right. The success of Profisee speaks to the necessity of MDM in a modern data stack - but more importantly - it speaks to the product vision, the unwavering customer focus, and the commitment of the entire Profisee team. I'm humbled to be a part of this company, and I'm excited to see where things head. Something tells me we're just getting started... #mdm #masterdatamanagement #masterdata

18

1 Comment

AI retrieval is evolving fast. RAG systems are no longer just “retrieve and generate.” They need precision, efficiency, and adaptability. Here are the Top 10 RAG Optimization Techniques for 2026 → Query Understanding • Classify intent before retrieval • Decompose complex queries • Normalize entities → Multi-Stage Retrieval • Generate broad candidate pools with embeddings • Apply reranking for precision • Select top relevant chunks → Chunking Strategies • Structure-aware content segmentation • Adaptive chunk sizes by content density • Semantic topic shift segmentation → Agentic Retrieval Planning • Iterative retrieval with reasoning • Stop based on confidence threshold → Feedback-Driven Index Refresh • Track low confidence outputs • Re-embed documents using feedback → Retrieval-Aware Prompt Engineering • Ground responses on retrieved documents • Separate instructions from factual examples → Hybrid Retrieval Signals • Combine dense vectors, sparse keywords, metadata filters → Context Compression & Distillation • Remove overlaps, summarize long documents • Extract only answer-relevant spans → Domain-Specific Retriever Fine-Tuning • Train on domain datasets with hard negatives • Apply contrastive learning → Evaluation Beyond Accuracy • Measure recall, precision, context utilization • Monitor latency and token efficiency RAG is no longer retrieval-it’s intelligent reasoning over information. ----------------------- If learning feels confusing, the problem isn’t effort. It’s the lack of a roadmap. Tell 𝐒𝐡𝐢𝐟𝐮 your goal. Get a personalized path, analyzed from millions of learning journeys, with curated resources in one place. Discover your path on 𝐒𝐡𝐢𝐟𝐮. 👉 𝐉𝐨𝐢𝐧 the community to stay updated on new 𝐆𝐞𝐧𝐀𝐈-𝐀𝐠𝐞𝐧𝐭𝐢𝐜𝐀𝐈 advancements. Link in comments section 👉 𝐃𝐌 me for 𝐜𝐚𝐫𝐞𝐞𝐫 𝐠𝐮𝐢𝐝𝐚𝐧𝐜𝐞/ 𝐄𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐀𝐈 𝐬𝐞𝐭 𝐮𝐩 Follow Ujjyaini Mitra for more insights on Enterprise Gen AI

70

8 Comments