On generation, selection, and the choices that make a work of art

A question at the villa

A few days ago I was in Viareggio, the Tuscan town on the warm Tyrrhenian Sea where I grew up and where all of my ancestors were born (I was not born there myself, but that is a different story). There I took a guided tour of the villa where Giacomo Puccini spent the last years of his life and worked on Turandot, the opera he never finished. Our guide was Luigi Viani, the director of the Fondazione Giacomo Puccini, a warm and generous man who knows the Maestro’s life intimately, with its little-known anecdotes, the quarrels, the cars, and the small vanities of one of the most famous opera composers in the world. At the end of the visit, after he learned what I had done for a living for the past forty-some years, Luigi turned to me and asked a question I had not expected: what would Puccini do with artificial intelligence, if he were alive today?

I did not try to answer on the spot, and instead I went home and kept thinking about it, and eventually I wrote a short text for a talk, in Italian, that Luigi will present at his next conference on Puccini. In preparing it I realized that behind Luigi’s question there was another one, which I had been carrying around for years without quite formulating it: what does an artist actually do, hour by hour, and which of those hours could a machine reasonably take over without taking away the art?

Let me say first that I am not much interested in art made by machines for its own sake. Most of the public conversation about AI and art, though, is stuck on a single image: the machine that paints the picture, writes the poem, or composes the aria. That image is real enough, and I will come back to it, but I think it hides something more interesting. Every artist I have known does two kinds of work: generation, the production of possibilities, and selection, the long and often painful process of deciding which of those possibilities comes closest to what the artist wants to say.  Around both there is a great deal of labor that has little to do with inspiration: research, technical craft, revision, and the business of finding an audience.

Readers of my series From Spark to System will recognize a familiar pattern. There I argued that invention, the spark of something new, is only the beginning, and that innovation is the long work of turning that spark into something of value for real people in the real world. Art follows a similar path. Inspiration is the artist’s invention, the flash that no one else could have had. The innovation is the finished work, shaped, chosen, refined, and carried to an audience, a work that travels through the years and reaches the hearts of people the artist will never meet. For years I have thought that scientists and artists have a lot in common, but I will leave that for a future essay.

If we look at the whole path, from the spark of an idea to the finished work, instead of only at the moment of inspiration, the question of what an artist might want from AI becomes much more concrete, and in some ways much more hopeful for artists, because much of what the machine can do is work they would gladly hand over, while the choices that make the work theirs remain in their hands.

Five hundred frames

Why do I care so much about selection? Because for a good part of my life, alongside my work in speech recognition and dialogue systems, I was a serious photographer (you can see a small selection of my portfolio at robertopieraccini.com, mostly of portraits, travel, and street scenes, and selection was the part of the work that consumed me.

I shot digitally, which meant that the cost of pressing the shutter was practically zero. In a studio session of two or three hours I could easily come home with five hundred pictures or more. The shooting itself was largely instinctive: the angle, the framing, the positioning of the lights, the moment when a face relaxed or moved into the light, all of that came from my own eye and from whatever aesthetic sense I had built over the years.  Then the real work began. I would spend hours, sometimes days, going through those five hundred frames, comparing near-identical images side by side, discarding the obvious failures, returning to the ones I had rejected too quickly, until I was left with a handful, sometimes only three or four, that I felt represented what I had seen and what I had wanted to say.

That process was difficult for reasons that had little to do with photographic technique. The technical failures were easy to spot: a missed focus, a blink, a blurred hand. The hard part came after, when I had a dozen or two perfectly good pictures of the same person in similar poses and I had to decide which one would have the most impact on a viewer.  Two frames taken a fraction of a second apart can tell completely different stories, and choosing between them meant deciding which story a stranger looking at the photo would feel. Many times I discovered, during that slow process, that the picture I had thought was the best at the moment I took it was not the one I finally kept, and that my idea of what the session had been about had changed along the way.

A handful of lines

Is there a kind of selection that happens before anything is made at all? Whenever I go to the Guggenheim in New York, I walk to the Thannhauser Collection, and I never leave without standing for a while in front of Picasso’s Woman with Yellow Hair, painted in 1931. It is a portrait of Marie-Thérèse Walter asleep, her head resting on her arms, and what has always struck me is how little there is in it and how much emotion it can produce in a viewer.  A handful of lines, perhaps half a dozen that really matter: the long curve that runs from her forehead down to her nose in a single stroke, the round shapes of her arms, the yellow mass of her hair. From those few elements a sleeping woman appears, with her abandon, her calm, and the tenderness she inspires.

That painting has always made me think of poetry. A poem tries to hold an idea, a feeling, a piece of reality, with the smallest possible number of words, and the painting does the same with lines. In both cases the power comes from what has been left out. I can imagine Picasso, before the brush touched the canvas, or in the sketches that preceded it, weighing in his mind an infinite number of possible lines and discarding nearly all of them, until only the few that carried the emotion remained. He left a famous record of this kind of work in The Bull, a series of eleven lithographs made in the winter of 1945 and 1946, in which a heavy, fully modeled animal is stripped down, plate after plate, until only a few lines remain, and the bull seems more alive at the end than at the beginning. Joan Miró traced a similar path through his own face. In 1919 he painted a meticulous, almost photographic self-portrait, which Picasso owned; in 1937 and 1938 he drew himself again in a dense, visionary portrait; and in 1960 he returned to that image and painted over a version of it with a few thick black strokes, as if, after forty years, those were the only lines of his face that still mattered.

This is the same selection I was describing with my photographs, in a more concentrated form. As a photographer I generated first and chose later, and the two activities were separate in time: the shutter in the afternoon, the editing in the following days. For Picasso, as for a poet, generation and selection happen almost at once, in the mind and in the hand, and what we see on the canvas is the residue of thousands of choices we will never see. Those few lines are all that remains of an enormous amount of rejection, and that rejection is where the art lives.

The value of what is thrown away

Is there a way to measure what Picasso and Miró left out? Some years ago I read  a book by the Danish science writer Tor Nørretranders called The User Illusion, and one of its ideas has been in my mind ever since. Nørretranders opens with a story about Victor Hugo, who, after the publication of Les Misérables, is said to have sent his publisher a telegram containing only a question mark, and to have received in reply a telegram containing only an exclamation mark. Two symbols carried a whole conversation, because of everything the two men knew and left unsaid. Nørretranders called that discarded information exformation, and he argued that much of the value of a message lies in what was thrown away in order to produce it.

As I wrote in an earlier post, Hitchcock, Shannon, and AI, Claude Shannon’s measure of information tells us how much a message carries, but nothing about whether it is worth sending. Nørretranders, drawing on the work of the physicists Seth Lloyd and Heinz Pagels, pointed to a different kind of quantity, thermodynamic depth, which looks at how much information was used and discarded along the way to producing something, and he suggested that a measure of this kind might begin to capture what we mean by the depth of a poem, a painting, or a piece of writing. The more information a message has shed without losing its meaning, the more valuable it tends to be.

I do not think anyone has turned this into a practical meter for art, and I am not sure anyone should. But, as an intuition, it describes exactly what I see in front of Woman with Yellow Hair, and what I felt with my five hundred frames. The half dozen lines and the three or four photographs are valuable because of the enormous number of possibilities that were considered and rejected to reach them.

In that same post I borrowed from Robert Pirsig’s Zen and the Art of Motorcycle Maintenance the idea that a thing made by someone who cared, who was fully present for each small decision, is different from an identical thing made by someone who was not. There is another idea in that book that speaks directly to selection. Pirsig argued that Quality is something we recognize before we can define it: we know which frame is the right one, which line belongs and which does not, long before we can explain why. That moment of recognition is what selection is made of, and it is the part of the work I would be least willing to hand to a machine.

Choosing as authorship

Is selection really part of the creative act, or is it just housekeeping? Photographers have known the answer for a long time. The contact sheets of the great reportage photographers, with their red grease-pencil marks circling one frame out of thirty-six, are some of the most revealing documents in the history of the medium, because they show the moment when a sequence of exposures became a photograph. Their decision was the authorship.

The strangest illustration of this I know is the case of Garry Winogrand, the American street photographer, who died in 1984 leaving behind close to 300,000 images that he had never looked at with an editor’s eye. Curators later went through that archive and printed some of what they found. Ever since, critics have asked whether those pictures can truly be called Winogrand’s work: he pressed the shutter, but someone else did the choosing. Many people feel, rightly in my opinion, that half of the work is missing.

Puccini’s own story offers the same question in a different key. On the evening of April 25, 1926, at the first performance of Turandot at La Scala, Arturo Toscanini stopped the orchestra after the death of Liù, the point where Puccini’s own music ended, laid down his baton, and told the audience that here the opera ended, because here the Maestro had died. The finale usually performed today was completed by Franco Alfano from Puccini’s sketches, and in 2002 a new one by Luciano Berio was premiered. Both are serious works by serious musicians, and both carry, inevitably, the choices of someone other than Puccini. A finale written by a machine trained on all of Puccini’s music would be technically possible today, and it might even sound convincing, but it would raise the Winogrand question in its purest form: whose selections would we be listening to?

What the machine could have done for me

So where could AI have helped me with my five hundred frames? I think there are two distinct levels, and the distinction matters.

The first level is technical culling. Today’s vision models can easily recognize a picture that is out of focus, a subject with closed eyes, a frame that is a near-duplicate of the one before it, a crooked horizon, a face lost in shadow. That was perhaps a third of my editing time, and it was the least interesting third. A system that had done it for me, presenting the survivors grouped by similarity, would have given me back hours of my life without touching anything I cared about. I would not have missed that work, any more than a painter misses washing his brushes.

The second level is aesthetic ranking, and here things get more delicate. It is entirely possible today to train a model on the pictures I chose in the past, the few hundred keepers out of tens of thousands of frames, and ask it to predict which of tonight’s five hundred I am likely to prefer. Having spent a career building systems that learn from examples, I am sure such a model would do a respectable job, probably better than I would like to admit. It could put my finalists at the top of the pile and save me another day.

And yet I have doubts here, and the doubts come from my own experience. A model trained on my past choices is a model of my past taste. Part of what happened during those long editing sessions was that my taste moved. I would discover something in a frame I had dismissed, and that discovery would change how I shot the next time. A system that always shows me what I liked before risks becoming a very polite mirror, and a mirror is a poor teacher. I would want such a tool, but I would want it to show me its top choices together with a few deliberate surprises, the frames it found unlike anything I had kept, so that the machine helped me look more widely instead of narrowing my eye to what it had already learned.

Science fiction imagined a machine judging an artist’s work long before anyone could build one. In a famous scene of Stanley Kubrick’s 2001: A Space Odyssey, the astronaut Dave Bowman shows HAL 9000 his sketches of the hibernating crew, and HAL tells him, “That’s a very nice rendering, Dave. I think you’ve improved a great deal.” It is a clear vision of a computer making a judgment of quality, and of progress in an artist’s work over time. It is also a reminder of what to watch for: HAL’s compliment is the opening move of a conversation with a purpose of its own, and a machine that praises our taste is not necessarily helping us develop it.

Writing with a machine

And what about writing, the art closest to my own most recent passion? This is where the relationship between generation and selection becomes most intimate, and also, I think, most deceptive.

I have a particular perspective on this, because for a large part of my career I worked with statistical language models, the ancestors of today’s large language models. In speech recognition their job was to estimate which word was most likely to come next, so that the recognizer could choose between “recognize speech” and “wreck a nice beach.” The models of today are immensely more powerful, but at their core they still do something similar: they produce, one word after another, a continuation that is plausible given everything they have read. That is why they write so fluently, and it is also why their prose, left to itself, drifts toward the expected.

Poetry, and good writing in general, moves in the opposite direction. The poet looks for the word that is far from the most likely one and turns out to be the only right one, the word that surprises and then seems inevitable. If Picasso’s painting is a few lines chosen among infinite possibilities, a good sentence is a few words chosen from a vocabulary of hundreds of thousands, and a good paragraph is often what survives after most of what was first written has been deleted. Pascal apologized at the end of a long letter, saying that he had made it longer only because he did not have the time to make it shorter, and every writer knows what he meant.

I use language models in my own writing, and I have learned that they are extraordinary generators. They can produce a draft in seconds, propose ten ways to open a section, retrieve the fact I half remember, and notice that I have used the same word three times on one page. What they cannot do for me is decide what I actually think, or which of their many fluent sentences sounds like me and says what I mean. The work does not disappear; it moves. Less of my time goes into producing words and more into choosing, cutting, rewriting, and rejecting, and the rejecting is harder than it looks, because machine prose is so smooth that a weak sentence can slip through unnoticed. Only the writer can tell whether a draft that sounds right is also a draft that says something.

So the lesson of the photographs and of Picasso’s lines applies here too. The machine can multiply the drafts, and the writer remains the one who chooses the few words that carry the emotion. When writers forget this and accept the first plausible version, readers notice, even when they cannot say why: the text is fluent and weightless, like a photograph that is perfectly exposed and says nothing. Many people now have a less polite name for it, AI slop, and the word became common enough that Merriam-Webster chose “slop” as its word of the year for 2025.

What we might lose

Is there a price for all this help? I believe there is, and it is worth naming plainly.

The first cost is the loss of productive friction. Some of the slowness of artistic work is waste, and some of it is where the artist is formed. The days I spent with my five hundred frames were tedious, but they were also the days in which I learned to see. If a machine removes the tedium, we should be careful that it does not also remove the apprenticeship, especially for young artists who have not yet built the eye that the machine is supposed to serve.

The second cost is convergence. Models learn from what has already been made and liked, and they tend, by their nature, toward the statistically likely. An artist who leans too heavily on them for generation and for selection risks producing work that looks like the average of everything, polished and forgettable. The remedy is the one Puccini practiced with his librettists, from whom he demanded endless rewrites and scenes rebuilt from scratch, only to throw most of them away: use what the tools offer generously, and then trust your own judgment ruthlessly.

The third cost concerns authorship and rights. Puccini was also a lucid professional, attentive to his contracts with Casa Ricordi and to the rights on his works, and I imagine him at the front line of today’s arguments over who has the right to train a model on an artist’s work, what happens when a synthetic voice sings an aria that was never written, and how to distinguish the work from its mass-produced imitation. These questions do not have easy answers, but they will shape whether artists experience AI as a collaborator or as a competitor built from their own labor.

Back to the silence at La Scala

So what would Puccini have done with AI? I told Luigi, and I repeat it here, that I believe he would have used it the way he used his cars, his motorboats, and the gramophone, with enthusiasm, speed, and a touch of recklessness, while insisting that it remain at the service of the theater and of the emotion of the audience. Puccini said of himself, more or less, that God had touched him with his little finger and told him to write for the theater, and only for the theater. The measure of his work was whether the people in the hall would cry, and that measure came from a life lived, from loves, griefs, fears, and passions that no dataset can contain.

When I think of my own photographs, I reach a similar conclusion. I would happily have handed over the blurred frames and the closed eyes, the duplicates and the crooked horizons, and I would have welcomed an assistant that showed me what I might have missed. The last choice, the one picture out of five hundred that would make a stranger stop and feel something, I would have kept for myself, because that choice was the reason I was holding the camera in the first place.

This is why I believe the distinction between generation and selection matters so much for the future of art. The machine multiplies possibilities, and by doing so it makes the ability to choose more precious. The more abundant the possibilities become, the more the value of art will reside in the person who chooses and takes responsibility for the choice: the photographer with five hundred frames, the painter with half a dozen lines, the writer with a few right words. It is a demanding future, because it asks artists to cultivate their judgment more deliberately than ever, but it is also a hopeful one. The silence Toscanini left in the middle of Turandot reminds us where the hand of the Maestro ended. The tools we are building now can fill many silences, and the work ahead, for artists and for those of us who build the machines, is to make sure that the final word still belongs to someone.