From the course: Problem-Solving Strategies for Data Engineers
Join today to access over 25,500 courses taught by industry experts.
Approaches to bug fixing
From the course: Problem-Solving Strategies for Data Engineers
Approaches to bug fixing
- [Instructor] Let's talk about problems with error handling and with back fixing. Typical things that you're going to see is that the errors are in log files, which is a big problem because you don't have an central hub where you can look at the logs, a central collection unit. You don't know where to start so you just try stuff out in the debugging phase. You don't know if the error is in the code or in the data and you don't know how to actually debug the pipeline. These are things that are usually problematic with error handling, with bug fixing. Let's look at the first point here. Let's look at log files. What I recommend is don't make the mistake of using individual files. Set up a central system that where all the logs are going to go, all the messages, something like Elasticsearch that can collect the logs and is a central hub for you to query data, to make queries to Elasticsearch and find errors very quickly. How…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.