7 key steps involved in Machine Learning Projects
The use of data in business is growing rapidly. Although businesses have been using data for decades, the use of machine learning is considered a relatively new capability. And for those who are not working in the field of data, it can be challenging or difficult to interact with their data team or solution provider.
So, let's understand what all includes in a machine learning project and what you can ask the data team to ensure your expectations will be met.
A machine learning project involves building a machine learning model that can be used to find a solution for a problem statement.
A typical machine learning project involves these 7 steps starting from defining the objective to getting the predictions.
- Define the objective
Defining the objective is fundamental to any project whether it's a construction project or a machine learning project. However, from a data perspective, you must be clear with what you hope to achieve out of this project. It also means you need to be sure of what impact this project can have on the abilities of the business to use the outcome of the project.
Defining the problem statement with as many details as possible is very important. Possibly defining it through use cases that explain the situation, challenges, and the desired outcome from the project. The reason is the value of the output expected is heavily dependent on the problem statement. Therefore being very specific and actionable helps a lot.
- Data Gathering
When it comes to data, you might want to check
- What data is needed to solve this problem,
- If the desired data is available, and
- How you can get the data?
Once the problem statement is clearly defined and understood by the data team, you need to know the data requirement. You could ask these questions to the data team -
- How much historical data do we need, one month, 6 months, one year, etc.
- What all information should the data cover,
- What type of data is needed - text, numbers, dates, etc.
You might also want to check if someone has already collected that data for some other business requirement and if there was any analysis done on the data. This will help the data team to get a jump start with deciding the approach to take to solve the problem.
It's important to know if the data is already available, what sources it comes from. What process is involved to get hold of the data? Also, if some of the required data is not available, what the possible ways are to collect the missing information for example by running some surveys, or conducting interviews with individuals or teams.
- Data Preparation
You will be very fortunate if the data you have got is exactly what the data team asked you to provide.
Most often than not, the data has some inconsistencies, missing values, duplicate values, etc. It is highly critical that the data team has validated the input data and any issues in it are fixed there itself. Otherwise, it's a situation of GIGO - garbage in garbage out, which means you cannot rely on the output or predictions you will get from the model.
This is mostly done using scripts considering the size of data one has to deal with in Machine Learning projects.
- Data exploration
It's the time for the data team to put their detective hat on as at this stage they would want to take a deep dive into the data. This involves exploring if the data has any patterns and trends in it. This step is also called Exploratory Data Analysis (EDA.
This is a considerable amount of work considering the project scope. As per seasoned data scientists, the Pareto principle applies here which is 80% of the time goes into data exploration while constructing the model takes 20 percent of their time.
- Building a Machine Learning Model
There are two terms used 'machine learning algorithm' and 'machine learning model' which are used interchangeably.
A machine learning algorithm is a procedure that is run on data to identify patterns in it. There are various types of algorithms used, for example, linear regression, logistic regression, decision tree, gradient boost, etc. Picking the right algorithm depends on the type of problem being solved.
A machine learning model is an output produced by the algorithm which represents what was learned by a machine through the algorithm.
In the Model Building stage, the data is split into two parts as training data, and testing data. Once a machine is trained using the training data, it is asked to predict the output using the testing data.
- Model Evaluation
After the construction of the model, the comparison of the predicted output and the actual output from the testing data gives the level of accuracy of the model.
There are multiple techniques used to improve the model accuracy popularly known as parameter tuning and cross-validation.
- Predictions
Prediction is nothing but the output of an algorithm after training and testing the machine on historical data and then applying the algorithm to the new data to forecast the likelihood of a particular outcome.
The output could be forecasting a categorical value like 'Success' or 'Failure' or 'Red', 'Amber', 'Green' or it could be a continuous value like forecasting value of a stock.
Summary
The typical process of delivering a machine learning project starts with identifying a problem statement and ends with getting the answer or solution to the problem statement which is nothing but predicting the outcome with a certain probability attached to it.
Predictions from machine learning models should not be treated as rational human opinions. Also, they are not simple columns in a spreadsheet that can be easily verified. Treat the machine learning output as insights to enhance decision-making rather than make the decisions.
Therefore it's important to note that the correct expectations are already set with the sponsors, stakeholders, and users on the ability and challenges associated with the machine learning output.
Hope you found this article useful.
