Using fastai to Detect Solar Panels From Orthophotos (2/3)

<p dir="auto">Welcome to the second post in this three-part ML project on using <code class="language-r">fastai</code> to detect solar panels. Throughout the miniseries, I'll be covering the following ML project steps:</p> <ol><li dir="auto"><a href="https://appsilon.com/using-ai-to-detect-solar-panels-part-1/" target="_blank" rel="noopener noreferrer">Data collection and preprocessing</a></li><li dir="auto">Training neural network model using <code class="language-r">fastai</code></li><li dir="auto"><a href="https://appsilon.com/using-streamlit-to-deploy-poc-app-part-3/" target="_blank" rel="noopener">Deploying your app using <code class="language-r">streamlit</code></a></li></ol> In this blog, we'll begin building a model to automatically detect solar panels in orthophotos. To do this, we'll focus on creating an image segmentation model using the <code class="language-r">fastai</code> library. Although we'll be using satellite imagery, more specifically - orthophotos, the methods presented here apply to any tasks working with images in <code class="language-r">fastai</code><span style="font-family: Consolas, Monaco, monospace;">.</span> <ul><li><a href="#anchor-1" rel="noopener noreferrer">What is fastai?</a></li><li><a href="#anchor-2" rel="noopener noreferrer">Image segmentation as a machine learning task</a></li><li><a href="#anchor-3" rel="noopener noreferrer">Preparing fastai dataloaders for modeling</a></li><li><a href="#anchor-4" rel="noopener noreferrer">Segmentation model training using fastai</a></li><li><a href="#anchor-5" rel="noopener noreferrer">fastai model serialization</a></li><li><a href="#anchor-6" rel="noopener noreferrer">Prediction results using trained fastai model</a></li><li><a href="#anchor-7" rel="noopener noreferrer">Next Step</a></li></ul> <h2 id="anchor-1">What is fastai?</h2> <p class="language-r"><code class="language-r"><a href="https://github.com/fastai/fastai" target="_blank" rel="noopener noreferrer">fastai</a></code> is a deep learning library created by Jeremy Howard et al. that gives practitioners both high- and low-level components to quickly create advanced PoC models. It's an innovative tool that gives us ease of use without sacrificing flexibility or performance. Since we are working on a vision task, we'll use <code class="language-r">fastai.vision</code>, a part of the greater <code class="language-r">fastai</code> ecosystem. But it is important to note that <code class="language-r">fastai</code> provides convenient interfaces for:</p> <ul><li> <strong>Vision</strong><ul><li>image classification, single and multi-label</li><li>image segmentation</li><li>estimating points in the image</li></ul> </li> <li><strong>Text data</strong> <ul><li>creating of language models</li></ul> </li> <li><strong>Tabular data</strong> <ul><li>convenient creation of embeddings</li></ul> </li> <li><strong>Collaborative filtering</strong> <ul><li>recommender systems</li></ul> </li> </ul> <h2 id="anchor-2">Image segmentation as a machine learning task</h2> A quick reminder from where we finished in the previous post. After preprocessing data, we have two folders <code class="language-r">images</code> and <code class="language-r">masks</code>. The folders contain our preprocessed patches of images (500 x 500px) as well as a <code class="language-r">fill_ratio.csv</code> file containing the percentage of pixels occupied by solar panels in every image. <img class="aligncenter wp-image-8898 size-full" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01fac2829666b590cadca_patch_solars.webp" alt="Solar Panels identified in an orthophoto" width="500" height="500" /> So let's load the necessary libraries and start modeling! Note that <code class="language-r">from fastai.vision.all import *</code> already imports <code class="language-r">pandas</code>, <code class="language-r">numpy</code>, <code class="language-r">Path</code> and more. As you might recall, a lot of images (nearly 85%) don't contain any solar panels. This is a lot of "useless" data to process. So if we can reduce the target data down to only those of value we can save a lot of time. But how do we do this? To do so we'll use a powerful <code class="language-r">pandas</code> function, <code class="language-r">query</code>. Check out<a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html" target="_blank" rel="noopener noreferrer"> query documentation</a>, there are a lot of <a href="https://towardsdatascience.com/12-amazing-pandas-numpy-functions-22e5671a45b8" target="_blank" rel="noopener noreferrer">neat things you can do with pandas query</a>! Since we're using <code class="language-r">fastai</code> in this project, we'll use the <code class="language-r">L</code> class as well. This is <code class="language-r">fastai</code>'s answer to a container that's most similar to Python list - <code class="language-r">numpy</code> one-dimensional ndarray. <code class="language-r">L</code> class supports a lot of useful functionalities like map and filter. You can read about the <code class="language-r">L</code> class in the <a href="https://fastcore.fast.ai/#L" target="_blank" rel="noopener noreferrer">documentation</a>. <blockquote><strong>Looking for effective real-time object recognition? Explore the possibilities of image classification with the <a href="https://appsilon.com/object-detection-yolo-algorithm/" target="_blank" rel="noopener noreferrer">YOLO object detection algorithm</a>.</strong></blockquote> <code class="language-r">Path</code> on the other hand, comes directly from the Python standard library. This feature introduced in Python 3.4 allows easier working with paths on various systems. Check out this great <a href="https://realpython.com/python-pathlib/" target="_blank" rel="noopener noreferrer">tutorial on pathlib</a> paths. What's important for us is that you can create paths by using slash, e.g., Path('.') / 'images'. And the paths will be valid on every system Python is running on! Finally, paths in the csv file describe the mask files while we need paths to image files. Function map applies our lambda function to every element of <code class="language-r">nzero_fill</code> and the second <code class="language-r">map</code> creates <code class="language-r">Path</code> objects from strings. <h2 id="anchor-3">Preparing fastai dataloaders for modeling</h2> Since we have nicely organized data, we would like to use the <code class="language-r">SegmentationDataLoaders.from_label_func</code> function to create dataloaders. To do so, we need a function that will say which image file corresponds to which mask. All that's left to do is to create dataloaders and preview an example batch. <img class="aligncenter size-full wp-image-8897" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01faef638169cb2574b09_example_batch.webp" alt="Orthophoto showing solar panels, indicated by reddish/brown mask" width="743" height="733" /> <h2 id="anchor-4">Segmentation model training using fastai</h2> In general, image segmentation might be a difficult machine learning task to train a model from scratch. Fortunately, we can leverage pre-trained image classification models. With <code class="language-r">fastai</code> all we have to do is create a <code class="language-r">Learner</code> with the <code class="language-r">unet_learner</code> function. Now we can run the <code class="language-r">fine_tune</code> method and leave all training-related issues to <code class="language-r">fastai</code>! Using Nvidia T4 single GPU, each epoch took around 20 minutes. In all, the entire training should last about an hour. Since we are using large images, we have to create small training batches to fit them into our memory. To compensate for the small batches, we use gradient accumulation. This is very easy in <code class="language-r">fastai</code> with its callback system. <img class="aligncenter size-full wp-image-8931" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01fae2a228de4b21f2a80_model-stat.webp" alt="fastai model stats" width="455" height="286" /> To quickly preview the model's performance let's run <code class="language-r">show_results</code>. The parameter <code class="language-r">figsize</code> allows us to make images larger, which is particularly useful if we want to view the results. <img class="aligncenter size-full wp-image-8899" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01faf3c5d0379a4f98e9f_show_results.webp" alt="Orthophoto comparing target to prediction of solar panels" width="740" height="831" /> <h2 id="anchor-5">fastai model serialization</h2> Usually, after we spend our precious resources training the model, we want to save it. It's not unheard of that a Jupyter kernel dies unexpectedly and we have to train from scratch. If you've been lucky enough to escape this fate, congratulations. But trust me when I say, it's no fun. We also plan to use this model in part 3 of the miniseries, so we really ought to save it. <h2 id="anchor-6">Prediction results using trained fastai model</h2> It's always nice to see the decreasing loss values and some predictions on the validation set. For me, what's truly impressive about machine learning is its ability for generalization. Let's take some orthophoto images of Warsaw suburbs and see whether our model will be able to find solar panels in it! <blockquote><strong>Explore our guide to GPU-accelerated <a href="https://appsilon.com/ship-recognition-in-satellite-imagery-part-i/" target="_blank" rel="noopener noreferrer">ship recognition using Keras and R</a>. </strong></blockquote> Here we're simulating usage of the model after its deserialization. Our model returns the predicted mask as a PyTorch tensor. To visualize found solar panels, let's create an alpha mask that will fade out parts of the image that are not panels. <img class="aligncenter size-full wp-image-8900" src="https://webflow-prod-assets.s3.amazonaws.com/6525256482c9e9a06c7a9d3c%2F65b01fb000b05468d71c1e82_suburbs_warsaw_predicted.webp" alt="Alpha mask fading non-solar panel pixels in orthophoto" width="665" height="394" /> <h2 id="anchor-7">Next step</h2> In just a few lines of code, we've trained a PoC model for solar panel detection using <code class="language-r">fastai</code> and performed predictions on our files. We've also serialized the model for future use. Need assistance with object detection, image classification, or satellite image analysis? Contact our <a href="https://appsilon.com/computer-vision/" target="_blank" rel="noopener noreferrer">Computer Vision team</a>. We build models that identify objects, defects, and patterns that human auditors miss. We can provide fast development, going from concept to a working prototype in as little as one week. <h3>Closing notes on fastai</h3> When working with libraries like <code class="language-r">fastai</code> sometimes you have to do odd things like juggling filenames back and forth to prepare appropriate input for provided functions. But this is only a minor inconvenience when compared to accessing the impressive features of <code class="language-r">fastai</code>. Another thing to note, while we trained the model on images 500 x 500px, we've made predictions on images of different sizes and it just worked. But this wasn't magic. <code class="language-r">fastai</code> took care of everything in the background. There are many beneficial reasons to utilize fastai in your project - the best of which you won't actively notice. We recommend you begin with <a href="https://docs.fast.ai/quick_start.html" target="_blank" rel="noopener noreferrer">fastai's quick start</a> and explore their tutorials to get a better grasp of what's possible. Written under <code class="language-r">fastai</code> version 2.4.1.