Helping AI algorithms build object aware maps of the world

30 June 2015 by DigitalGlobe Object Detection Team
Figure 1: Output of the Object Detection system @DigitalGlobe Inc. that was trained to detect planes, parking-lots and storage-tanks using training data generated by the Tomnod labelers.
Figure 1: Output of the Object Detection system @DigitalGlobe Inc. that was trained to detect planes, parking-lots and storage-tanks using training data generated by the Tomnod labelers.

In recent years, there has been a tremendous increase in the amount and resolution of commercially available satellite imagery. This growth has resulted in several novel applications such as precision agriculture, population density estimation, and numerous location based services, just to name a few. An important problem shared by these applications is the accurate and efficient detection of multiple objects (e.g. planes, ships, road-structures etc.) over vast areas captured using satellite imagery. This capability to automatically detect objects of interest could be used in various ways for the betterment of society, including:

Scientists at DigitalGlobe Inc. have developed state-of-the-art Artificial Intelligence (AI) algorithms that can be trained to distinguish between different types of objects. Training such algorithms to recognize an object, say a storage-tank, requires presenting them with multiple examples of storage-tanks. Once the algorithm has learned what a storage-tank looks like, it can then detect other storage-tanks that might be present in the satellite image of a previously unseen area. The following toy example illustrates the basic process of training an algorithm by example and testing it on a previously unseen image:

Figure 2: A toy example to illustrate the idea of example based training of AI algorithms. Multiple images of cats and dogs are shown to the algorithm which learns to distinguish one from the other. Once training is complete, it can classify a previously unseen image into either a dog or a cat.
Figure 2: A toy example to illustrate the idea of example based training of AI algorithms. Multiple images of cats and dogs are shown to the algorithm which learns to distinguish one from the other. Once training is complete, it can classify a previously unseen image into either a dog or a cat.

A major challenge in this workflow is that it can take thousands of training examples before the AI algorithms become good at detecting an object accurately. Collecting these large training data sets can be time consuming and labor intensive. That’s where Tomnod comes to the rescue!

Thanks to the Tomnod Crowd, the Scientists at DigitalGlobe were able to collect one of the largest training datasets for object detection ever collected in the geospatial community consisting of close to two hundred thousand training examples of more than twenty object classes. An example area with labels of multiple object types is shown below:

Figure 3: An example area with labels of multiple object types is shown. In particular, some of the piers are zoomed in for better viewing.
Figure 3: An example area with labels of multiple object types is shown. In particular, some of the piers are zoomed in for better viewing.

Using these labels, the Object Detection system at DigitalGlobe is able to accurately detect multiple object classes accurately and efficiently. An example image of some of the classes detected by the system are shown below:

Figure 4: Example detections. Red polygons denote detection regions and yellow squares or polygons indicate ground truth locations for (a) Parking Lot, (b) Storage Tank, (c) Swimming Pool, (d) Soccer Field, (e) Baseball Field, (f) Tennis Court. (g) Overpass, (h) Bridge, (i) Cul-de-sac, (j) Roundabout, (k) Marina, and (l) Pier
Figure 4: Example detections. Red polygons denote detection regions and yellow squares or polygons indicate ground truth locations for (a) Parking Lot, (b) Storage Tank, (c) Swimming Pool, (d) Soccer Field, (e) Baseball Field, (f) Tennis Court. (g) Overpass, (h) Bridge, (i) Cul-de-sac, (j) Roundabout, (k) Marina, and (l) Pier

Not only can the object detection system highlight the areas where different types of objects are present, it can actually count the number of these objects. An important application of this capability is to generate summary statistics of different objects at a particular location to better understand the behavior of these objects over time. The following figure illustrates the counting results of the object detection system applied to planes at an air-field over four years.

Figure 5: (Top) Counts of large planes in the air-field over four years. (Bottom) Visualization of summary statistic of plane counts over an example runway image in the air-field data-set. Blue and red represent low and high-occupancy areas respectively. This type of information could be useful to detect anomalous patterns of plane movements in different parts of the runway.
Figure 5: (Top) Counts of large planes in the air-field over four years. (Bottom) Visualization of summary statistic of plane counts over an example runway image in the air-field data-set. Blue and red represent low and high-occupancy areas respectively. This type of information could be useful to detect anomalous patterns of plane movements in different parts of the runway.

Going forward, the scientists at DigitalGlobe count on the help of the Tomnod Crowd to collect labels for an ever increasing variety of object types! The long-term goal is to tighten the loop between label collection, algorithm training and detection, in order to enable reliable object detection at scale. This is all part of the Seeing a Better World initiative here at DigitalGlobe: using humans and computers in collaboration in order to extract actionable information on our changing planet.

This work was done by the DigitalGlobe Object Detection team, which consists of a bunch of mad scientists, ninja engineers, and jedi data-wranglers. Their goal is to build state-of-the-art AI systems that can help us make better sense of the dynamically evolving surface of our planet.