Tiny Object Detection in Remote Sensing Images: End-to-End Super-Resolution and Object Detection with Deep Learning

Abstract

In this thesis, we study the problem of detecting small objects on low-resolution (LR) satellite imagery. Small-object detection is a challenging problem, especially from LR images. To tackle the challenge, we propose a method to generate super-resolution images from low-resolution images and simultaneously detect objects from the super-resolution images. A generative adversarial network (GAN)-based model called enhanced super-resolution GAN (ESRGAN) shows remarkable image enhancement performance, but reconstructed images miss high-frequency edge information. Therefore, object detection performance degrades for the small objects on recovered noisy and low-resolution remote sensing images. Inspired by the success of edge enhanced GAN (EEGAN) and ESRGAN, we apply a new edge-enhanced super-resolution GAN (EESRGAN) to improve the image quality of remote sensing images and used different detector networks in an end-to-end manner where detector loss is backpropagated into the EESRGAN to improve the detection performance. We propose an architecture with three components: ESRGAN, Edge Enhancement Network (EEN), and Detection network. We use residual-in-residual dense blocks (RRDB) for both the GAN and EEN, and for the detector network, we use the faster region-based convolutional network (FRCNN) (two-stage detector) and single-shot multi-box detector (SSD) (one stage detector). Extensive~experiments on a public (car overhead with context) dataset and another self-assembled (oil and gas storage tank) satellite dataset show superior performance of our method compared to the standalone state-of-the-art object detectors. While working with the detection problem, we create a GUI tool to label, train, and detect objects from remote sensing images that cover a large area. This GUI makes it easier to create small image tiles from the large satellite images, training the state-of-the-art object detection models, running the detection, and finally obtaining the output geolocation for the detected objects.