What are Implicit Parameters in Scala and why are they so Important

Implicit Parameters are just like normal variables in #Scala, but they are defined with an implicit keyword in the beginning. Let's take an example of #SparkSession variable that you usually create in your App.

So how is it so different from normally defined variables? Unlike normally defined variables, implicit variables are stored in the context and whenever you define a function that takes in implicit variables as parameter, Scala checks if the implicit variable is defined, before the call to the function definition.

Defining a function that takes in a implicit variable:

Lets say you want to call the function1, if you call function1 without defining implicit val spark then the compiler will already complain and shows error for you to supply a variable for this parameter. To call this function, you should first define a implicit variable called spark(exactly the name of the variable that the function requires) and call the function function1. Alternatively, there is also a way to supply the implicit variable using another parenthesis when calling function1, you can try it out yourself.

Now that you understand what is implicit variables and how to work with them, some of you would have already found some usefulness of this implicit variable, below is an example:

As a Software Developer, it is very essential to know how important is unit testing and how important it plays a role in development and deployment of the software. Implicit variables are a way to inject dependencies to a function or a class. The above SparkSession is one of the best case defining it's usefulness. If you are working with #spark, you know the Spark Session changes when running locally and when running in cluster, you might have a extra parameter called enableHiveSupport and your master will most likely be yarn like shown below.

For Yarn Cluster (usually, you will define Spark Session, something like this in main):

For Unit Testing, you will most likely change the master and enableHiveSupport and run or you will create a function that returns this SparkSession based on cluster or running it locally. Imagine this: What if you forget to do this and your app goes to production?

There is a simple way to avoid this with implicit variables:

And in your unit test:

This is one example of implicit variables, you can already imagine some other very useful things e.g. some functions might be using spark.sql and they might not run locally and results in errors, so you should separate them and make this function take a implicit Dataset or DataFrame so that you can inject them.