We Have a Deployment Problem

Publishing: Where AI models go to die. Very few models make it beyond the publishing stage. Sure everyone's favorite buzzwords are out there like our friends the Large Language Models (LLM)s and in my sector the Image Recognition like ResNet, YOLO, and the always asked in the last 4 years "Have you tried a vision transformer?"

In the AI community we trade these like commodities, adapting them to our purpose--nailing a new high score in a benchmark or publishing a paper about a corner of our fields. All good work, but now, what happens when we want to take the next step and share our work with colleagues who want to use the latest greatest AI model?

Canned response: Just deploy it in Docker on the Cloud(TM)! Piece of Cake until the dreaded three letters in my research show up: I R B. Now many of you will be asking who or what is this IRB monster? Its the Institutional Review Board or the dreaded Data Privacy and Security Board by another name--the AI scientist's nemesis.

Apparently some people feel that it is not very cool to just start flinging personally identifiable information (PII) across the internet, no matter how secure you promise to make your API/Cloud/Docker. And to be fair, they have a point. Once you send me your data, you have to trust me that I will discard it and NOT sell it to Google/Facebook/Anonymous. So, many times Cloud is out of the picture.

"But my old friend," you say. "You certainly can deploy the app locally with Docker" Yes I could get every person to download and setup docker, spin up the container and just make local API calls, no sweat. Then reality kicked me in the head in the form of my boss who reminds me: "Most people don't work on the command line and have limited experience with software, you have to make it easy for them" And I work with M.D.s!

In other words, I have to write a windows app that contains my model and all its functionality. Have you tried writing apps containing AI models lately? And remember no command line and no python (The average desktop has no python and runs windows). Right now that REALLY kills your options. Unless you have a C# or C++ programmer (or possibly Java), things are going to get painful really really fast. In addition: all the data loading and cleaning has to be re-written--all our fancy libraries like Pillow or OpenCV have to be redone (OpenCV at least exists outside python). No more PyTorch or Tensorflow. That make for some steep (but not impossible) problems and is a huge barrier to adoption even of the most novel AI models. Google/Facebook/OpenAI--they all have teams to take care of this. Your regular Data Scientist/Machine Learning Engineer is outta luck!

Right now our choices are to export the model in ONNX format and then use various Microsoft Windows Libraries in C# or C++ to build a deployable app. We've been so fascinated by Cloud/Docker/API that when data security hits you with the "No Data in the Cloud" it leaves you stunned with a "Ok Now What?!" As I poked around looking for assistance, most people were downright confused "Why would you want to do that, Just throw it in Docker and Flask/Cloud" Data Privacy is here to stay and we need alternatives. If you can't send the data, I can send the model, BUT I need to send it in a way you can easily deploy and any instructions that start with "Open a command prompt" are doomed to failure. Most people just don't have those skills. And while not impossible, switching over to C++ or C# is a serious investment that tight strapped groups just don't have--Not to mention just 2 MS Windows Libraries.

I don't have the answers to make this better. Windows isn't going anywhere and the bottom line is unless you are working with fellow AI scientists, your app needs to be deployable on Windows without a python installation (having a GUI is pretty much required). Plus--if we do get the apps on smartphones what happens when your phone losses connection and you need the model to "do something" Your cloud can't help you there--you need the model on the device in an app. Fortunately smartphones run a version of linux--but we have a ways to go to shrink them down and the hardware is getting better (more capacity) But the time is coming! Cloud/Docker/API won't always be the solution!