As part of a recent campaign, we asked our crowd to classify 693802 property parcels in Adelaide, Australia, in parcels that contain a swimming pool (‘yes’) and parcels that do not (‘no’). The campaign was sponsored by a private company that compiles public and private sector data for a variety of markets including education, public safety, government, telecommunications, and insurance.
Sounds like a simple task for our crowd, right? A pool is pretty easy to see in our imagery.
Swimming pool with bright blue water!
Machine learning for Search Area Reduction
In order to reduce the average number of user votes required per parcel to classify all the parcels in the data set with sufficient confidence, we decided to deploy a supervised machine learning algorithm to help direct the crowd to the parcels where the presence of the pool was likely, and remove from consideration the parcels which most likely did not contain a pool. Since the estimated percentage of properties which contain pools is quite small (<10%), this combination of machine learning and crowdsourcing can lead to a large efficiency gain. You can think of this as having the ability to reliably classify a given number of polygons in a fraction of the time required with a pure crowdsourcing campaign. This opens up interesting posibilities for continent-wide mapping of swimming pools.
Adelaide is marked with the red circle. What about all the other cities in Australia?
For you image mining and machine learning geeks out there, the algorithm we deployed is quite simple. First, we identified the spectral profiles of an open and a blue-tarp-covered swimming pool using multi-spectral, WorldView-2 imagery. For each property polygon, we computed a small number of features, including the minimum spectral angle between each pixel in the polygon and each of the spectral profiles. We then fed each feature vector to a Random Forest Classifier, which we trained using a few thousand ‘yes’ and ‘no’ parcels labeled with high confidence by the crowd.
Spectral signatures of open and blue-tarp-covered pools in an Atmospherically Compensated 8-band WorldView-2 image.
We deployed the classifier on 245090 polygons (out of 693802 total). 35617 of those were classified as ‘yes’ (machine score > 0.5) and the rest as ‘no’ (machine score < 0.5). We had the crowd vote only on the 50000 or so polygons which the machine gave a score > 0.3, in order to weed out the false positives. This led to a 5x search area reduction.
Search area reduction at work: the crowd only has to inspect the parcels on the right.
We collected a random sample of 2500 classified parcels. Out of 69 parcels attributed as having a swimming pool, only 5 were wrong, yielding an indicative commission rate of 7%. However, an additional 22 parcels which contained swimming pools were not identified, yielding an indicative omission rate of 22/(64+22) = 26%!
Our first reaction was that the machine was responsible for the false negatives, i.e., that the crowd was never shown these polygons due to their low machine score. Yet, delving deeper into the results, we discovered that the culprit was the crowd!: the machine was correctly detecting pools which the crowd then misclassified. Here is an example.
Swimming pool with green water and partially hidden by trees!
In this case, six of our users voted ‘no’ and only one voted ‘yes’! Yet the machine was 85% confident that this polygon contained a pool.
This pool is almost completely covered!
In this case, four of our users voted ‘no’ and only one voted ‘yes’. The machine was 90% confident that this polygon contained a pool.
The truth is that, given the available imagery for this region of the world, and the fact that pools come in all sorts of shapes, sizes, water colors (some are even empty!), as well as the fact that many of them are partially or completely covered by tents and tarps, the task of identifying a swimming pool in the polygon is not so simple for a human as initially anticipated.
Examples of swimming pools which the crowd missed and the machine detected.
In order to reduce the pool omission rate, we retrained the machine and redeployed it on a much larger portion of the data set (approximately 500000 polygons). We then used the machine score
Where to next?
We believe that the right combination of humans and machines can lead to unparalleled accuracy image mining at scale. The use case described here indicates that rudimentary object detectors can perform very well when used in conjunction with human operators. We are currently experimenting with more advanced machine learning algorithms based on convolutional neural networks, which we believe will further boost the accuracy of the crowd+machine system.