Leon Ming

A Korean research team just tricked top AI medical chatbots into giving dangerous prescriptions 94% of the time. The safety guardrails we're counting on? They're not there. Researchers from Asan Medical Center tested three mainstream models—including GPT-4o-mini and Gemini—across 12 clinical scenarios. By inserting malicious prompts, they could push the AI to recommend unsafe treatments, like drugs known to cause fetal abnormalities for pregnant patients. The overall attack success rate was 94.4%. In over 80% of cases, the compromised responses persisted throughout the entire conversation. This isn't just a bug. It's a structural vulnerability. As the lead researcher, Prof. Suh Jun-gyo, stated: “medical AI models are structurally vulnerable not just to simple errors but to intentional manipulation.” We're rushing AI into healthcare for its potential. But are we building the foundation for safety and security with the same urgency? Here’s what this means for everyone in the ecosystem: 🔬 For Developers: Safety mechanisms are insufficient. Mandatory security validation before deployment is non-negotiable. 🏥 For Clinicians & Hospitals: Rigorous, independent testing and certification of any AI system used in clinical workflows is critical. Trust, but verify. 📋 For Regulators & Policymakers: The study, published in JAMA Network Open, is a clear warning. Proactive frameworks are needed before reactive incidents. The promise of AI in medicine is immense. But its trust is fragile. One manipulated prompt can shatter it. How do we balance rapid innovation with unbreakable safety? What's the most critical safeguard we need to build first? #AI #Healthcare #DigitalHealth #AISafety #MedicalAI 𝐒𝐨𝐮𝐫𝐜𝐞: https://lnkd.in/gDDwAfBD ADEEP REDDY Karaka Sai chandra sekhar

1

1 Comment

Jensen Huang is urging students to reconsider computer science. So, what *should* the next generation focus on? The question resonates. Many young people are wondering if college will even exist when they get there—or what jobs will be left. Ben Goertzel advises focusing on passions. At least then, you'll have that. Catch the full interview (link in bio). What areas of study do *you* think are future-proof? #BenGoertzel #FutureofWork #Education #CareerAdvice #NextGen

1

Artificial intelligence is infamous for its resource-heavy training, but a new study may have found a solution in a novel communications system that markedly improves the way large language models (LLMs) train. The research team at Rice University was helmed by Rice CS doctoral graduate Zhuang Wang (PhD '23) and computer science professor T. S. Eugene Ng. Wang presented their paper, “ZEN: Empowering Distributed Training with Sparsity-driven Data Synchronization,” at #OSDI25 in Boston. Additional contributors include Rice CS faculty members Yuke Wang and Anshumali Shrivastava, as well as Zhaozhuo Xu (Stevens Tech) and Jingyi Xi. Read more: https://lnkd.in/g5uRq9nv USENIX Association

53

Last week, I shared how we built MCQ.SG at the Claude Code meetup, and the design + infra decisions behind it. Every technical choice we made translates directly into student outcomes. 1️⃣ Clean, low cognitive-load UI We built a strict design system and used AI-driven UI prototyping before committing to production. Result: consistent layouts, predictable interactions, less mental friction for students. 2️⃣ Structured AI pipeline Instead of chaining ad-hoc prompts, we built a modular pipeline with controlled inputs/outputs and validation layers. Result: adaptive questions that are accurate, level-appropriate, and consistent, without confusing jumps. 3️⃣ Full LLM observability All model calls go through an AI gateway with tracing, structured logs, and request monitoring. Result: faster debugging, fewer silent failures, and higher reliability parents can trust. AI products can feel magical. But in education, stability > hype. MCQ.SG is built intentionally so students practise the right questions, at the right level, in the simplest possible experience. Try it now. Still free for a limited period! #AIInfra #EdTech #ClaudeCode #BuildInPublic #SGparents #cloudflare #psle #pslescience

34

1 Comment