Data Scientist vs Software Engineer
Lately I’ve been talking to a lot of companies while looking for my next opportunity. Often a question I get asked is: Are you a data scientist? The simple answer is: no. Let me explain my thoughts on what each role is and why they are different.
Data Scientists are like traditional scientists in commercial software development because they hypothesize that something is true and then create experiments to prove their theories. Often, finding the answer leads to many new questions and insights. The key to success is therefore enabling quick iterations. Data scientists don’t want technology getting in their way. Slowing down the feedback loop blocks creative insights. The conscious side of the brain (logic) needs to figure out how to make the computer do something and blocks out the subconscious brain (intuition). Often times data scientists have all the data in front of them but they don’t know what the final outcome will be.
A software engineer usually knows the desired expectations of the system. The challenge is finding the right balance of different and sometimes competing requirements. High throughput systems naively implemented are harder to debug because of the volume of chatter they internally create. Distributing functionality across many different processes allows for high scalability at the cost of reliability if not handled correctly. Software systems are expected to be highly customizable and flexible at the potential cost of understandability in reading the code. Software engineers want to release new functionality as quickly as possible and need to find reliable ways to test it. For the past 10 years I’ve specialized in building systems that address these concerns and that’s why I consider myself to be a software engineer. I’ll be sharing my thoughts on these subjects in later articles posted here.
Finally I believe data scientists and software engineers should have a symbiotic relationship to create a successful company. Data scientists find the golden nuggets that engineers build products around. Engineers create tools and platforms that enable data scientists to iterate quickly and deploy the code safely. Working together they produce better outcomes than each working individually.
I often think that data scientists are much better at thinking about the "why." Software engineers focus a lot more on the "how." They're different skill-sets, in my opinion. When dealing with big data sets, it's really helpful to have a data scientist involved from project inception to make sure you can tease out the "why." Signals that might be important may not be fully considered by engineers designing a system.
Very good post Louis Calisi!