Processing Radar Imagery in Python – Thoughts and Conclusions

I have lived in prejudice that preprocessing of Earth Observation data is manageable if I know the required processing steps. Clouds are the problem; atmospheric and geometric corrections are tricky, but correction steps are well-documented.

Currently, I discovered Sentinel 1 data. It is radar data, and the physics behind data acquisition differs from taking a photo using a CCD camera. Radar sensors are active; generally, they send pulses of radio waves into the ground and measure the echo of each received signal. Radar aperture gathers information in multiple modes, and data processing is complex. Why do we bother? Because satellite images we know from mapping services are useful only when the sky is clear, and there are no clouds. It is uncommon in my region, Helsinki, especially in winter. The solution could be radar satellite imagery because radio waves penetrate water clouds, and we can “see” what’s below.

What surprised me was that there are disputes about when to apply a specific preprocessing step to radar imagery. Moreover, there are multiple ways to achieve the result (which could be an input for the AI/ML model). Radar output is complex; data is heavy, and pipeline logic might be fuzzy. We must be cautious. I realized that the complexity of data gathered from radar instruments is close to the complexity of human-made events!

Finding a way to process radar data is an application-specific task that should be done by specialists or after an initial research phase. I’m in a good position because I was not bothered by the task of finding optimal data processing steps, and I received ordered steps from researchers. My role has been to automate a pipeline generating Machine Learning input for specific areas and times.

What I’ve learned so far?

• Working with complex data sources requires moving away from Python. I’ve tested Python packages for radar data processing, but the best solution is the ESA SNAP system and its GPT processing graphs, especially if the research team gives us their input as XML files.
• I use Python for automation, file naming, logs, and orchestration. Python invokes processes written in XML files (using subprocess module and ESA-SNAP GPT command).
• It is important to know if the radar image covers the Area of Interest. Thus, I read KML files. Fiona and Pyogrio throw exceptions when reading KML from an archive, so I’ve written a function using the Zipfile module to parse KML and get the bounding box coordinates of a radar image. Sometimes, taking a step back and using basic Python functionalities is better than wrestling with existing solutions.
• Catching exceptions from Java requires scanning terminal output.
• It’s much easier when each XML file is treated as a Kubeflow component. The program doesn’t need to be a Kubeflow pipeline, but a program structure with modules with a single responsibility helps with debugging.

In summary, my advice for working with complex radar images is to use the best tools for tasks (and it doesn’t mean tools that we like the most), write small processing steps with limited responsibility, and log as much as you can.