Harshita Singh

Ever needed to run a regex search on your dataset—only to watch your queries crawl while your users tap their feet? You collect tons of log lines, events, or messages in your favorite analytics database and want to surface exceptions, patterns, or keywords with a quick REGEXP search. But as your data grows, those once-snappy queries start to lag... and lag... and lag. So why does this happen?   A basic regex search forces your database to inspect every value— O(n) time for n records. If you’ve got 100 million rows, even a blazing-fast system starts sweating. Now you start thinking: Can I make this faster with indexing? What options are there? - Start simple: Use a dictionary scan. If your text field has low cardinality—a few repeated values—Pinot’s dictionary scan is O(d), much better than O(n). But if that dictionary is huge? Performance still drags. - Get smarter: Enter Lucene's FST (Finite State Transducer) Index. It builds an automaton (think: compressed trie) that encodes all possible field values as a searchable graph. Now, rather than scanning the whole dataset, It matches your regex just once across the FST— O(k) time for your regex pattern length! Suddenly, 30-second queries finish in milliseconds, no matter the table size. - Still want more? Add inverted index alongside FST. For complex queries—mixing regex and filters—you get lightning-fast bitmap matching plus rapid regex search. Regex is fast, filters are fast, and everything stays real-time. But is indexing worth it? You might worry about extra storage or write overhead. What if you get all of these. Here comes Pinot’s FST(link in comment) and inverted indexes are so efficient you can build them inline, as new data lands in real-time. A tiny storage premium for massive query speedups—an unbeatable trade-off, especially as your workloads scale. Have you hit the same regex wall? What tricks helped you push past it? If you’re in Bangalore and want to geek out on real-time indexing and analytics internals—you would want to be at Meesho tech (https://lnkd.in/gDM2t8EZ) meetup. That’s where the latest Pinot tips and deep dives are brewing! StarTree Meesho Gnanaguru SattanathanJayesh Asrani #DataEngineering #IndexingMatters #BigData #PerformanceOptimization #ApachePinot #MeeshoTech #RealTimeAnalytics

11

1 Comment

🚀 Just wrapped up an incredible 3-day Databricks Workshop focused on the future of Data and AI. Organized by Databricks at Warner Bros. Discovery India Hyderabad, the sessions offered hands-on learning and deep dives into the core pillars of modern data intelligence: Data Engineering, Machine Learning, and Generative AI. 💡 Key Takeaways: 📊 Data Engineering: We explored building robust, scalable pipelines using Delta Lake and the Medallion Architecture. Topics included orchestration with Workflows, and governance via Unity Catalog. 🤖 Machine Learning: From feature engineering and experiment tracking with MLflow to real-time inference with Mosaic AI, Databricks demonstrated how their unified platform accelerates the entire ML lifecycle. 🧠 Generative AI: We saw how Databricks powers agent-based systems that drive accurate, data-driven decisions. The hands-on lab given us explained the end to end flow to deploy any agent to our application. Big thanks to the Databricks team and Shriharsha Jana for the collaborative sessions, live demos, and for creating an open space for innovation and learning. #Databricks #DataEngineering #MachineLearning #GenerativeAI #Workshops #WBD #IIH

18

Is the "Search & Browse" era of E-commerce officially evolving into something entirely new? 🛒🤖 I just finished watching an incredible deep dive with Manav Garg (Together Fund) on Think School. As a developer in e-commerce space, one takeaway really stood out: The transition to Agentic Commerce. For years, the industry has been obsessed with perfecting filters and recommendation engines. But the next big leap isn't a better UI, it’s the "headless" interaction between AI agents. (Timestamp: 18:05) The Shift: From UI to Infrastructure In the near future, customers may not spend 20 minutes scrolling. Instead, personal AI agents will negotiate with brand agents to find the best deal based on a user's specific goals and budget. (Timestamp: 15:08) What this means for the future of our craft: Logistics is the new Front-end: If the "Browse" layer becomes automated, the real competitive edge moves to the physical layer i.e supply chain, speed, and fulfillment infrastructure. This is where vertically integrated platforms really shine. (Timestamp: 21:31) The Rise of "Orchestration": With tools like Cursor and AI-assisted coding, we're moving away from boilerplate and toward building complex "Orchestration Layers" that allow these agents to talk to our platforms seamlessly. (Timestamp: 08:30) Efficiency as a Feature: As we integrate LLMs, the focus moves to making them profitable through smarter token usage and caching, turning "hype" into sustainable margins. (Timestamp: 36:52) We often focus on the 95% of time users spend browsing. (Timestamp: 17:52) But as that time shrinks, the platforms that win will be the ones built for agents, not just humans. Are we ready for Agentic Commerce, or are we still building for a world that's quickly moving past the search bar? Video Source: https://lnkd.in/ghfmbqw3 #Ecommerce #SoftwareDevelopment #AI #AgenticCommerce #ThinkSchool #Innovation

22

💡 Spark jobs shouldn’t be black boxes. At InMobi, I built a custom SparkListener that emits structured, job-level telemetry, giving us precise cost and performance insights without overloading blobs. 🔧 Optimized for scale: • 200k+ jobs/month • Smart aggregation (p95/p99 with T-Digest) • Exported as structured logs (yes, not Prometheus) This approach gives us deep, programmatic insights into Spark jobs across Databricks, Kubernetes, and on-prem environments, be it Ad-hoc tasks or ETL pipelines. If you’re trying to move beyond monitoring into true operational intelligence, this journey might be worth reading. I want to thank Amit Khanna and Srikanth Sundarrajan for their continuous guidance and motivation, along with Archana H, for co-authoring the blog. https://lnkd.in/g9f9J8Sc https://lnkd.in/g9f9J8Sc #ApacheSpark #BigData #Observability #DataEngineering #CostOptimization #Kubernetes #Databricks #Telemetry #InMobi #LinkedInTech #EngineeringLeadership #PlatformEngineering

56

7 Comments

I do not really post a lot about tech stuff, but maybe I can start. Here is a paper I read recently. ROSE: Robust caches for Amazon product search [https://lnkd.in/gPhbgHHQ] Thanks to Arpit Bhayani's paper shelf recommendations. The insights from this are really fascinating for the search world and how Amazon always knows how to find what you are looking for even if you have clumsy fingers like mine. For example if you are looking for shoes, you might type something like: “nike shoe,” “nike shoes,” “nikes shooes.” A normal cache treats these as different, so it gets big and slow, and you miss the cache a lot. Amazon’s ROSE fixes this by caching the intent, not the exact spelling. What ROSE essentially is, is a typo/variant-tolerant cache for product search that groups similar queries into the same “bucket,” so most lookups hit fast without growing memory. It’s deployed in Amazon Search and improved both latency (single-digit ms for most traffic) and business metrics. How this works is even more fascinating. 1. Locality-Sensitive Hashing: Similar queries collide in the same bucket. • Lexical-preserving hashing (character n-grams + minhash) for typos/variants. • Product-type-preserving hashing (weighted minhash) to keep the category intent (e.g., “dishwasher” vs. “dishwasher parts”). 2. Reservoir sampling: caps each bucket so memory stays constant even as queries grow. (Super convenient, and one of the many engineering gotcha's people might miss) 3. Count-based k-selection: avoids pairwise similarity inside buckets; just count collisions across hash tables → near constant-time retrieval. Where we can apply some of these concepts with Gen-AI (which is in many flavors essentially a search problem with extra steps)- Semantic prompt cache: Group paraphrased prompts to reuse an answer/logits/KV cache even when text isn’t identical. Can be helpful in saving your tokens and $. RAG request cache: map similar user questions to one retrieval plan to cut vector searches + reranking cost. [I remember Azure AI search having a RAG cache, but I am not sure how that worked] Tool/agent call cache: Dedupe near-duplicate function/tool invocations (e.g., same API call phrased differently). Eval & feedback loops: Bucket similar generations or errors to reuse critiques/patches. Support/search front-ends to LLMs: Typo-tolerant, intent-stable pre-rewrites before hitting the model, ensure we only use tokens when absolutely needed.

174

6 Comments

AI-Powered Insights Spark a Transatlantic Triumph! Just had a "lightbulb moment" I wanted to share, showcasing the power of applying AI trends across borders! My team in India was grappling with optimizing logistics for a new client. Traditional methods weren't cutting it. Inspired by a recent U.S. based case study on using AI for predictive route optimization (thanks, LinkedIn!), we adapted a similar model to the Indian context. The challenges were different: congested urban areas, diverse road conditions, and varying traffic patterns. But using AI to analyze historical data, real-time traffic feeds, and even social media sentiment around road closures, we were able to predict and dynamically adjust routes. The result? A whopping 18% reduction in delivery times and a 12% decrease in fuel consumption for the client. This success not only saved them money but also significantly reduced their carbon footprint. It's amazing to see how cross-pollination of ideas and the application of global AI trends can lead to localized wins. It highlights the power of a connected and informed professional community! What AI trends are you seeing make a real impact in your industry or region? Share your thoughts in the comments below! Let's learn from each other! #AI #ArtificialIntelligence #Logistics #Innovation #India #UnitedStates #SupplyChain #DataScience #TechTrends #MachineLearning #GlobalCollaboration

4

Shantanu Narayen’s 18 year tenure as CEO at Adobe is a reminder of how rare long-term stewardship has become in Silicon Valley. During his time with the company, the company’s stock increased more than 6x and its business model fundamentally changed. The most important decision of his tenure was moving Adobe from selling boxed software like Creative Suite to a subscription platform through Creative Cloud. At the time it was controversial. Today, it is considered one of the most successful SaaS transitions in the history of enterprise software, turning Photoshop, Illustrator, and Acrobat into one of the most profitable recurring software ecosystems ever built. The respect he commands across the industry says a lot about how difficult that transformation actually was. Microsoft CEO Satya Nadella even posted publicly congratulating Narayen on a “legendary run,” praising both his leadership and the way he expanded what creators and businesses could do with software. In an industry that is often intensely competitive, that kind of recognition from peers is not given lightly. There are only a few CEOs in the world who truly get to define one era of software and then help set the stage for the next. Narayen did it by turning Adobe into a SaaS powerhouse. However, the next leader will have to answer a harder question. What does a creative software company look like when AI becomes part of the creative process itself? #Adobe #Leadership #SaaS #AI #Technology

10

Just heard Amazon Prime laid off so many employees, Because they shipped faster and automated things with AI. And this just scares me but at the same time I feel there’s huge gap between how managerial side is thinking about AI v/s the Reality of AI: Because: 1. AI writes the code but fails at quality, I need to spend time to get good final version 2. Doing LLDs with AI is too hectic, it skips what it doesn’t understand or comes up with solution which isn’t optimised 3. HLDs aren’t even AIs cup of tea yet This just about coding part, there’s lot of more that we do as engineers. So, is it managers illusion which is taking away job ? But whatever it is someone needs to wake these companies to reality and show how important engineers are for the company. Done with listening AI can replace engineers.

193

9 Comments

Ed Donner, Sachin Chandrashekhar 🇮🇳 Building an Agentic AI Email System: I have been experimenting with Agentic AI and the OpenAI SDK to automate email workflows, and it’s been an amazing hands-on learning experience. Here is what I did: 1. Built agents that can draft emails automatically. 2. Created tools to generate email subjects and convert plain text into HTML emails. 3. Implemented a handoff system so the best draft gets selected and sent automatically. Learned how important tool prompting and agent instructions are to make the workflow smooth. It’s exciting to see AI agents working together, almost like a small team, to get a task done. This project really helped me understand how to orchestrate AI for real-world automation, like email campaigns or notifications. Next, I want to explore connecting this system to LinkedIn posts and automated data pipelines—bringing end-to-end automation to life. #OPENAI Agentic SDK #OPENAI #GPT Models # GEMINI #Python #Automation #DataEngineering #AWS Pipelines. Code Snippet: emailer_agent = Agent( name="Email Manager", instructions=instructions, tools=email_tools, model="gpt-4o-mini", handoff_description="Convert an email to HTML and send it") tools = [hr_tool1, hr_tool2] handoffs = [emailer_agent] hr_instructions = ''' You are an HR agent at Cognizant. You are responsible for recruiting Data Engineers. Steps you must follow: 1. Use the HR tools such hr_tool1 and hr_tool2 to generate candidate email drafts. 2. Evaluate all drafts and choose exactly one best draft. 3. Handoff only this single best draft to the emailer_agent for formatting and sending. Critical Rules: - You must always use the HR tools to generate drafts (never write the drafts yourself). - You must never output more than one draft. - You must always handoff exactly one draft to the emailer_agent. - Do not ask clarifying questions or request more input. - Produce the final result in one step with a single handoff.''' hr_manager = Agent( name='hr manager', instructions=hr_instructions, tools = tools, handoffs= handoffs, model = 'gpt-4o-mini' ) with trace('Hr agent'): try: response = await Runner.run(hr_manager, "Send out a Recruiting email addressed to Dear Applicant from OPENAI") except : response = {"final_output": "Could not complete in 2 turns, please retry with higher max_turns."}

3