The analysis of the millions of works that are part of the cultural heritage and artistic is a work that seems impossible for the human being, but not for the supercomputers. However, machines have to learn to do it first.
the european project Saint George by bicyclecoordinated by the Barcelona Supercomputing Center – National Supercomputing Center (BSC-CNS) in collaboration with the European Foundationstarted in 2019 with the aim of using the supercomputer MareNostrum4 train models artificial intelligence (AI) that contribute to a better interpretation of European cultural heritage, as well as publicizing its wealth, value and the importance of its preservation to the public.
To this end, the project seeks to generate automatic descriptions hundreds of thousands of images from various cultural heritage repositories using natural language processing algorithms and deep learning (deep learning in English).
Now, in a second phase of the project, the researchers launched a collective collaboration campaign or crowdsourcing inside zooniverse, a citizen science portal, to collect thousands of handwritten notes to help better train these AI models. The campaign is completely open and anyone can participate in the analysis of 5,000 European pictorial works.
For the first time, artificial intelligence provides descriptions of images of cultural heritage with greater coverage of themes, objects and iconographic relationships, taking into account the temporal context of creation of the works and the rules of composition of periods and scenes of sacred iconography between centuries XII and XVIII.
“This ambitious project interprets images according to their context for the first time and thus seeks to give the machines a certain common sensewhich is one of the great barriers of AI today”, declares Joaquim Moreresearcher at the BSC and specialist in computational linguistics in the project.
“For example, when the system initially identifies a motorcycle in a 15th century painting of Saint George, it corrects itself and identifies the most plausible object for the time, which is the horse. Subsequently, this adaptation will be made to the cultural context. So, in the Japanese cultural context, what we would call a gentleman in Europe would be a samurai”, he explains.
Thanks to this initiative, it will be possible better analyze the paintingsdetect never-before-seen images or compositions, discover relationships between their elements and interpret their symbolism, helping to develop cultural and dissemination initiatives, such as virtual exhibitions with related paintings from around the world.
Cultural, educational and academic applications
“Our project will allow quick access to cultural information that can be used not only for cultural and social purposes, but also in other sectors such as education and tourism, and possibly also help historians or anthropologists”, he says. Maria Cristina Marineescuresearcher at the BSC and coordinator of the Saint George by bicycle.
“In addition – he concludes – society in general can also benefit from better public services, such as better accessibility to web pages for the visually impaired, narratives that can expose social injustice or integration through collective cultural heritage to help create a more tolerant European identity”.
