The case of the successful shell script. Part 1.

I am a little behind on my blog posting but I am going to make it a point to post every month. This month is off to an exciting start, I have scheduled some time off to visit family but before I do that I have a couple of things to wrap up at work then I would be a happy engineer trading my sneakers for flip flops even if it is just for a weekend. This will be a two part article, in this article I will post in detail about the problem I faced at work and in the second post I will be able to provide my understanding and my explanation about the peculiar pattern in linux. Feel free to post your solutions to this problem on the comments section.

The problem:

We were told by the cluster management team that our projects were not using provided dependencies instead we were tightly coupling specific versions of Hbase dependencies to our jars. This caused some issues when the cluster team uplifted Hbase dependencies in our dev cluster thus leading to a delay in releasing code and causing panic among the engineers and the leads in my team. Now I was a bit ahead in my task and I was digging around for other random process improvement tasks that will help my team with faster turn around time with releasing code when my lead mentioned this problem to me. To learn more about this I jumped at this opportunity. The team decided to go through the quick fix route i.e. uplift the version of the dependency instead of removing the coupling (Don't ask me why). So I decided to uplift the dependency and was able to deploy the jar to our internal repo and I was getting ready to test the passivity of this uplift in a cluster where the uplift has not taken place. We were trying to get ahead of the cluster uplifts on another node.

Now we have some specific commands to run while we execute our project to see if our code works on the data we have in our cluster. I diligently executed the commands one after another and when I ran the final command that is supposed to run the map reduce jobs I found out what it was blowing out on guava dependency mismatch.

I immediately ran a dependency tree to see if the uplifted version provided an alternate guava version, that would have made this case a whole lot easier but to my dismay the guava versions provided by the previous version of the jar and the uplifted one were the same. I was struggling for around 3 days trying to look at the verbose dependency tree and trying to exclude dependencies in the hope of isolating the problem but nothing worked. Every time I ran the commands I was unable to find the cause of the above illegal access error. Now I did something that every engineer dreads, I asked for help. One of the engineers in my team who successfully executed the command volunteered to take a look. He said he found a shell script that he use to run the commands. I was not sure if that would help but I told him to give it a try. So he added my new uplifted jar to the script and executed the command.

To my amazement, IT WORKED!

The map reduce did not blow up and it provided the exact result the command was supposed to do. Now I told my lead about this peculiar problem and he was not sure as to why this would even happen. To be fair each of the commands only set the variables needed for last command that kicks of the map reduce job. I was not sure as to why this would work when I ran it inside a shell script. I am attaching a mock of what my script contains to show you exactly what I did.

I am now motivated to find the reason why the shell script was successful while running each command individually didn't work. Feel free to share your thoughts on the case of the successful shell script.