Can AI really give us a glimpse into lost masterpieces?

[1945fire[1945Brand claimed three of the most controversial paintings by Gustav Klimt. Commissioned in 1894 for the University of Vienna, the “Faculty Pictures”, as they were called, were different from all previous works by the Austrian Symbolist. Even at its presentation, the critics were in an uproar over their dramatic departure from the aesthetics of the time. Professors at the university rejected them immediately, and Klimt withdrew from the project. The works soon found their way into other collections. During the Second World War they were placed in a castle north of Vienna for safekeeping, but the castle burned down and the paintings probably went with it. Today only a few black and white photographs and writings from this period have survived. Still, I stare straight at her.

Well, not the images themselves. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to revive Klimt’s lost work. It was a tedious process that started with these black and white photos and then incorporated artificial intelligence and lots of information about the painter’s art to try what these lost paintings might have looked like. The results show me Smola and Wallner – and even they are amazed by the fascinating Technicolor images that the AI ​​has produced.

Let’s get one thing straight: no one is saying this AI is bringing back Klimt’s original works. “It is not a process to recreate the actual colors, but to recolor the photographs,” Smola quickly states. “The medium of photography is already an abstraction from the real works.” What machine learning does gives an insight into something that for decades was believed to be lost.

Smola and Wallner find this delightful, but not everyone supports AI in filling these gaps. The idea of ​​machine learning to recreate lost or destroyed works is, like the faculty pictures themselves, controversial. “My main concern is the ethical dimension of the use of machine learning in the context of nature conservation,” says art restorer Ben Fino-Radin, “because of the sheer number of ethical and moral problems that have plagued the field of machine learning.”

The use of technology to revitalize human works of art is of course peppered with sensitive questions. Even if there were perfect AI that could figure out what colors or brushstrokes Klimt might have used, no algorithm can generate the author’s intent. Debates about this have raged for centuries. As early as 1936, before Klimt’s paintings were destroyed, the essayist Walter Benjamin argued against mechanical replication, also in photographs, and said: “Even the most perfect reproduction of a work of art lacks one thing: its presence in time and space, its uniqueness”. Existence wherever it is. ”Benjamin wrote in The work of art in the age of mechanical reproductionhe calls a work “aura. ”For many art lovers, the idea that a computer reproduces this intangible element is absurd, if not impossible.

And yet there is still a lot to learn from what AI can do. The faculty paintings were crucial to Klimt’s artistic development, a crucial bridge between his more traditional earlier paintings and later, more radical works. But how they looked in full color remains a mystery. That is the riddle that Smola and Wellner were trying to solve. Your project, organized by Google Arts and Culture, it wasn’t about perfect reproductions; it was about giving a glimpse of what is missing.

Wallner has developed and trained a three-part algorithm for this. First, a few hundred thousand images of art from the Google Arts and Culture database were fed into the algorithm. This helped him understand objects, works of art, and compositions. Next, it was specially trained in Klimt’s paintings. “This creates a tendency towards his colors and his motifs over time,” explains Wallner. And finally, the AI ​​was fed color references to certain parts of the paintings. But where did these clues come from with no color references to the paintings? Even Klimt expert Smola was surprised at how many details the writings of the time revealed. Because the paintings were considered so dirty and strange, critics tended to describe them in detail, right down to the artist’s choice of colors, he says. “You could call it an irony of history,” says Simon Rein, program director of the project. “The fact that the pictures caused a scandal and were rejected enables us to better restore them due to the large amount of documentation. And those kind of data points, when fed into the algorithm, create a more accurate version of what those paintings likely looked like at the time.”

The key to this accuracy lies in the combination of the algorithm with Smola’s expertise. His research found that Klimt’s work during this period tends to have strong patterns and consistency. The study of existing paintings before and after the painting faculty provided indications of the colors and motifs recurring in his work at the time. Even the surprises Smola and Wallner experienced are backed up by historical evidence. When Klimt first showed his paintings, critics noted that he used a red that was rare in the artist’s palette at the time. but The three ages of womenPainted shortly after the faculty paintings boldly uses a red, one Smola believes it is the same color that caused a turmoil when first seen in the faculty paintings. In another faculty painting, too, writings from this period can be heard across the terrifying green sky. Combining these fonts with Smola’s knowledge of Klimt’s particular palette of greens, when fed into the algorithm, produced one of the first surprising images to come out of AI.