The new algorithm developed by MIT, “MosAIc,” finds subtle similarities between works of art by scanning images.
The new system, which is in operation at the Metropolitan Museum of Art and the Rijksmuseum in Amsterdam , scans artworks and then uses deep networks to find similarities in pieces across cultures, artists and media that might otherwise have gone unnoticed, according to a blog post from MIT's Computer Science & Artificial Intelligence Laboratory (CSAIL)
But let's see how it works: the user inputs an image and MosAIc finds similar works of art. In one example, the algorithm linked "The Martyrdom of Saint Serapion" by Francisco de Zurbarán, and "The Threatened Swan" by Jan Asselijn.
"These two artists did not correspond, nor did they meet during their lifetimes, but their paintings revealed a rich, hidden structure that underlies both of their works," said MIT CSAIL student Mark Hamilton, lead author of the study.
A particularly difficult aspect of developing MosAIc was creating an algorithm that could find not just similarities in color or style, but also in meaning and subject matter, Hamilton said .The researchers looked at a deep network of “activations,” or features, for each image in the open-access collections of both museums. The distance between the “activations” in the deep web was how the researchers judged similarity.
The MosAIc researchers also used a new search called a “KNN Tree,” which groups images into a tree-like structure. To find the closest match to an image, the algorithm starts at the “trunk” of the cluster and then follows the most promising “branch” until it finds the closest image. The data structure improves itself by allowing the tree to “prune” itself based on image features.
Hamilton said he hopes the work he started at MosAIc can be expanded to other fields, such as the humanities, social sciences and medicine.
«These fields are rich with information that has never been processed with these techniques , and can be a source of great inspiration for both computer scientists and domain experts..»
