Samuel R. Delany, SF Hall-of-Famer and professor of English and Creative Writing, lays out his theory of how science fiction differs from literary fiction (or, as he calls it, "literature" or "mundane fiction" with "mundane" referring to its literal meaning, "of the world") in an essay called "Disch II". This essay, ostensibly about the work of Thomas M. Disch, can be found in the newly reprinted 1984 essay collection Starboard Wine: More Notes on the Language of Science Fiction, which was a follow up to the 1979 collection The Jewel-Hinged Jaw: Notes on the Language of Science Fiction.
In the essay, he describes in detail how our contemporary concept of literature is fairly recent (dating to the late nineteenth century or so) as is the modern form of science fiction, practiced in the way he describes (which emerged in pulp magazines in the 1920's and '30's). He argues that rather than one kind of fiction artificially separated by genre borders, they are in fact two different kinds of writing, with different functions, priorities and strengths, and that in fact while literature can do things that science fiction cannot, equally science fiction can do things literature cannot, not only by the way it constructs and makes the reader consider the worlds that it creates, but by the way it uses language to very different ends than literature. He likes to bring up the example of the sentence "Her world exploded." which would be a metaphorical sentence in a work of literature but a literal one in a work of science fiction.
His arguments go against my own ideas from some time ago that genre boundaries should, by rights, vanish, and relate to the discussion about genre that I once had with Matthew Cheney, a Delany scholar who first turned me on to this book and who wrote its new introduction.
I'm going to quote from the essay at length below, beginning with a passage about how Kafka's Metamorphosis is read as literature, because I think he's making a fascinating case and one worth thinking about. I'm eliding a lot of his examples and history, and if you're interested in them, in fact if your interested in the subject at all, you should really buy and read all of Starboard Wine.
"... Because the world of mundane fiction is fixed, at least in comparison with the multiple worlds of science fiction, when we read some distortion in the representation of the world in a piece of mundane fiction we are lead to the questions, Why did the character (the fictive subject) perceive it this way? or Why did the writer (the auctorial subject) present it this way? These are the kinds of questions we ask when we read the fantasies of, say, Kafka. When Gregor Samsa, in The Metamorphosis, awakes to find himself become a beetle, our attention is immediately drawn to various psychological readings. Is Samsa mad? Or is the writer saying something obscure about psychology—his own, or possibly even ours? Where our attention is not drawn in such a fantasy is to the nuts-and-bolts workings of the world of the story itself. It doesn't go there because in the world—in Kafka's very real world of pre-World War I Prague, in my world of New York in the '80s, or in your world wherever and whenever it is—people don't turn into bugs. You know it. I know it. Kafka knew it. This priority of the subject as it guides our reading, our interpretations, the questions we ask of texts, the way we choose to make a text make sense, is what characterizes the traditional literary categories of writing; it is what joins both mundane fiction and the fantastic together as literature.
"The question then becomes, "What kind of sociological situation resonates with turning into a bug?"
"Science fiction, which by most conservative critical assessments is considered paraliterature (a genre of writing outside literature), works differently. It immediately focuses our attention on the workings of the world—or, as the phenomenologists would say, on the object. If we read in an SF story about a person who wakes up transformed into a bug, we are certainly concerned with how the person will react; but the underlying question that guides even that concern is this: What in the world of the story caused it to occur? Perhaps the character is not really human, but rather some neotenous life-form that has undergone a necessary change. Perhaps secret biological engineering has taken place during the night. Certainly if the character learned that one or another were the case, she might react very differently to the different situations, and in both cases probably differently from Samsa. Still, How would the world of the story have to be different from our world in order for this to occur? is the question around which the play of differences in the SF text is organized.
"At this point it should be clear that what separates science fiction from mundane fiction or the fantastic is basically that different expectations, different interpretive conventions, different underlying questions are brought into play the moment we identify a text (either from something within it or from what we have heard about it) as belonging to one mode or the other. A person might turn into a bug in either a fantasy tale or an SF tale; what distinguishes the two is the different ways we would read them, the different questions we would expect to be answered. And even if the text never gets around to answering those particular questions directly, they change the way we regard other things in the text because our attention is focused where it is and not somewhere else.
"What literature can criticize, and criticize extensively, is the subject—even as the subject affects one's living body (a fancy way of saying human behavior). Literature today is very limited, however, in what it can say of the object—of nature, culture, of institutions (in the broadest sense of the word).
"Literature can condone or condemn any behavior, judge it moral or immoral; but to speak to human behavior is still to speak to the subject. As far as criticizing culture and the institutions that compose it, however, all literature can say, of the provinces, for example, is, Get out of them. If you are smart enough to read this, they will stifle you. This is addressed to the subject. All literature can say of a particular government is, It oppresses you unto death. To accept it is to abnegate all your humanity. This is again addressed to the subject. All literature can say of a particular social class is, Life lived within its confines will foster you vulgarity, elegance, nobility or hypocrisy. This is still addressed to the subject. ... What literature cannot do is critique an institution directly, suggesting, say, how it might be restructured to excite the subject in a different way—at least it is highly limited in the suggestions it can make.
"Science fiction, because of the object priorities in the way we read it, in the questions we ask of it, in the modes by which we must interpret it simply for it to make sense, is able to critique directly both particular institutions and the larger cultural object in general—culture plus the infrastructural object culture proper is always a response to. When science fiction makes direct proposals it becomes propaganda just as much as literature does. What science fiction can do, however, is portray a different, an imagined, a nonexistent institution that works much better than, or often much worse than, or in the most interesting cases just very differently from, an existing one. The object priority in the reading conventions—which must begin with a consideration of some real institutions simply to understand how the science-fictional one works at all—generates the criticism directly in the understanding (cognition) process itself.
"Now that science fiction, a paraliterary category clearly outside of literature's priority (or tyranny) of the subject, has gained a certain respectability on its own, this ever so generous invitation to come inside the literary corral ("Let's remove the label," they cry) seems a last ditch effort to make sure science fiction will have to take a back seat as "second-class literature." To read science fiction as literature means to me to cease reading science fiction's presentation of alternate world-workings as complex commentary on the workings of our own world-that-is-the-case and to read them instead as yet another manifestation of the subject—perhaps another projection of the auctorial subject, as we read contemporary fantasy."
One wonders what Delany thinks about the recent trend of literary fiction appropriating science fiction's tropes. Is the trend toward a more plot-driven, genre-convention-employing literary fiction mean that we're harkening back to time before the late 19th century when "literature" had an object-priority? Are we entering into a new phase where the subject and object priority become mingled? Or is this just a temporary dalliance, a fad that will soon be cast off while literary fiction returns to the suburban homes, infidelities and quiet desperation that became its hallmark in the 20th century?