Jaswanth Thathireddy’s Post

Over the past few days, I’ve been diving into PySpark and distributed data processing concepts. Coming from a background in Python, SQL, and data-driven backend systems, it’s been interesting to see how similar data transformations scale when working with large datasets. I’ve been exploring how Spark handles data processing across clusters and how it fits into real-world data pipelines. Currently focusing on: • Working with Spark DataFrames • Performing transformations (filter, groupBy, joins) • Understanding ETL workflows at scale Still early in the learning process, but it’s a valuable step toward building more scalable data solutions. #PySpark #DataEngineering #BigData #Python #LearningJourney

🚀 Journey to Becoming a Data Scientist — Day 24 Today I continued working on data manipulation using Pandas. 📚 What I learned today • Subsetting data in a DataFrame • Selecting specific columns using [] • Selecting multiple columns at once • Subsetting rows based on conditions • Using loc for label-based selection • Using iloc for position-based selection 📊 What I practiced • Extracted specific columns from datasets • Filtered rows based on conditions • Combined row and column selection • Worked with subsets to analyze relevant data 💡 Key takeaway Subsetting helps in focusing only on the required data, making analysis more efficient and easier to understand. 🚀 Improving step by step with Pandas. #DataScienceJourney #Python #Pandas #DataScience #LearningInPublic #Consistency

SQL vs Python vs PySpark — One Table That Can Save Your Career Hours! If you're in Data Analytics / Data Engineering, you’ve probably asked yourself: 👉 Should I use SQL, Python, or PySpark for this task? So I created a quick comparison cheat sheet that maps common operations across all three. 💡 From: Removing duplicates Handling nulls Aggregations Joins Conditional logic Filtering data ➡️ You can now translate logic across tools instantly #DataAnalytics #DataEngineering #SQL #Python #PySpark #BigData #DataScience #Learning #CareerGrowth #TechSkills #NDAC #MicrosoftFabric Magudeswaran | Ajay Babu | Kaviya | Manikanta | Srinivasareddy | Sreethar M B | Suresh | Maureen Direro | Krishnakanth | Gopi Krishna | Satya Sekhar | RAMA | Santosh J. | Mahesh | Sabyasachi | Sainatha | Veeresh | Shafque | Anirban

🐍 Day 2/30 — Python for Data Engineers Lists & Tuples. These two will follow you everywhere. In my 3 years as a Data Engineer, barely a day passed without using either of these. Here's what I wish someone told me on Day 1: Lists = Dynamic. You'll append rows, filter tables, and loop through pipeline stages. Tuples = Fixed. Every DB record you fetch comes back as a tuple. The one mistake beginners always make 👇 one = (42) ❌ # this is just an int one = (42,) ✅ # THIS is a tuple And the thing that makes Python lists actually powerful: List Comprehension — transform data in one line: active = [t for t, ok in all_tables if ok] That single line replaces 5 lines of for-loop code. 📌 Full cheat sheet in the image — save it for your daily reference. Day 3 tomorrow: Dictionaries & Sets 🔑 Follow Jaswanth Thathireddy if you're learning Python for Data Engineering 👇 #Python #DataEngineering #30DaysOfPython #LearnPython #DataEngineer

🧠 Quiz Answer Reveal Time! ❓ What is Pandas mainly used for? ✅ Correct Answer: B) Data Manipulation Explanation: 👉 Pandas is mainly used for: Cleaning data Filtering data Analyzing datasets 💡 It works with tables using DataFrames Understanding these fundamentals helps build a strong foundation in Data Analytics, Python, SQL, and Business Intelligence. 💡 Small concepts like these are used every day by Data Analysts and Data Engineers. #Python #QuizPython #UpSkill #DataAnalytics #DataAnalyst #TechQuiz #Upskilling #DataEngineering #TechLearning #NattonTechnology #NattonAI #NatonDigital #NattonSkillX

🧠 Quiz Answer Reveal Time! ❓ What is Pandas mainly used for? ✅ Correct Answer: B) Data Manipulation Explanation: 👉 Pandas is mainly used for: Cleaning data Filtering data Analyzing datasets 💡 It works with tables using DataFrames Understanding these fundamentals helps build a strong foundation in Data Analytics, Python, SQL, and Business Intelligence. 💡 Small concepts like these are used every day by Data Analysts and Data Engineers. #Python #QuizPython #UpSkill #DataAnalytics #DataAnalyst #TechQuiz #Upskilling #DataEngineering #TechLearning #NattonTechnology #NattonAI #NatonDigital #NattonSkillX

🧠 Quiz Answer Reveal Time! ❓ What is Pandas mainly used for? ✅ Correct Answer: B) Data Manipulation Explanation: 👉 Pandas is mainly used for: Cleaning data Filtering data Analyzing datasets 💡 It works with tables using DataFrames Understanding these fundamentals helps build a strong foundation in Data Analytics, Python, SQL, and Business Intelligence. 💡 Small concepts like these are used every day by Data Analysts and Data Engineers. #Python #QuizPython #UpSkill #DataAnalytics #DataAnalyst #TechQuiz #Upskilling #DataEngineering #TechLearning #NattonTechnology #NattonAI #NatonDigital #NattonSkillX

🧠 Quiz Answer Reveal Time! ❓ What is Pandas mainly used for? ✅ Correct Answer: B) Data Manipulation Explanation: 👉 Pandas is mainly used for: Cleaning data Filtering data Analyzing datasets 💡 It works with tables using DataFrames Understanding these fundamentals helps build a strong foundation in Data Analytics, Python, SQL, and Business Intelligence. 💡 Small concepts like these are used every day by Data Analysts and Data Engineers. #Python #QuizPython #UpSkill #DataAnalytics #DataAnalyst #TechQuiz #Upskilling #DataEngineering #TechLearning #NattonTechnology #NattonAI #NatonDigital #NattonSkillX

I got paid to NOT build an ML model. Here’s why. SQL > fancy ML models. Fight me. 🫵 Okay hear me out - I've seen teams spend months building ML pipelines... when a 10-line SQL query would've answered the question in 10 minutes. My actual toolkit after 4 years: 🗄️ SQL - find the truth in the data 🐍 Python - automate everything else 🤖 ML - deploy it when SQL genuinely can't do the job The aha moment? They work best in that exact order. Most people jump straight to ML. The pros start with SQL. Where are you in your data journey? 👇 #SQL #Python #MachineLearning #DataScience #HotTake #DataEngineering #TechOpinion #LearningInPublic #BuildingInPublic #DataAnalytics

🐍 Day 5/30 — Python for Data Engineers Conditionals & Loops. How pipelines make decisions. Every pipeline does two things constantly: 1. Makes decisions → skip bad rows, branch on job status, alert on failure 2. Iterates → loop over files, tables, API pages, batches Today's cheat sheet covers both — and a few patterns I use in production every day. The one most engineers miss 👇 for...else — the else block runs only if the loop completed without a break: for stage in pipeline: if stage.failed: break else: notify("All stages passed ✅") And the chunked insert pattern — essential for large loads: for i in range(0, len(rows), 1000): db_insert(rows[i : i + 1000]) Sending 1M rows in one shot will crash your DB. Send them in chunks of 1000. Always. Today's sheet covers: → if / elif / else → Ternary + walrus operator :=  → match/case (Python 3.10+) → for loops with enumerate, zip, break, continue → while loop + retry with backoff → All 3 comprehension types → 4 real DE pipeline patterns 📌 Save the cheat sheet above. Day 6 tomorrow: Error Handling & Exceptions 🛡️ Which loop pattern do you use most in your pipelines? 👇 #Python #DataEngineering #Python #DataEngineering #DataEngineer #LearnPython #BigData #ETL #Coding #TechCommunity #SoftwareEngineering #BackendDevelopment #CloudComputing #AWS #OpenToWork #JobsInFrance #TechJobsFrance #LearnPython #DataEngineer