How and Why Plot Structure Works
You've heard this before:
There is a protagonist who wants something, badly. There are obstacles in the protagonist's way, usually an antagonist working against him/her. After being obstructed again and again, and finally, when all seems lost, the protagonist risks it all to win and succeeds (or perhaps fails).
Most of the books on writing you'll find in your average bookstore talk about the above, rudimentary plot structure. Often, especially in screenwriting books, the structure is divided up into three acts, with the rhythm going something like this: in the first act an inciting incident forces a protagonist to make a decision to pursue the object of desire, culminating in a minor first act climax in which the protagonist meets up against the full force of antagonism for the first time and manages some small success. Then, in a long second act, there are progressive complications as the protagonist tries to overcome successive obstacles, culminating in a second act climax, which often results in all seeming lost. Then, in the third act, the protagonist keeps fighting and risks it all, leading to a story climax in which he/she (usually) triumphs. Finally there is a denouement in which loose threads are wrapped up and things return to equilibrium. In some cases there are more acts, for instance the five act structure typical of Shakespeare, or fewer acts, a one or two act structure in a short story or half-hour television show, but three acts has become the base norm, especially in film.
You've seen/read/experienced this hundreds of times. And because of that, it's easy to dismiss it. It's easy to say, this structure is what leads to the formulaic shallowness typical of Hollywood, and I want no part of it. It's easy to say I'm going to make stories that bear no resemblance to the three act structure and they will blow your freaking mind. And while it's definitely possible to buck structural norms and create something wonderful, there are examples that could be cited (Roberto Bolaño springs to mind), the problem both with this attitude per se and with the way this information is presented in most writing books is that it ignores WHY this structure has become so standard, that is why it works. And even if you want to do something unconventional, it's important to understand why the conventions exist and how they work, to avoid falling into the traps that they are specifically designed to circumnavigate.
The first element of the convention is the most important. A character wants something. The reason why this is important is not merely because wanting things makes a character do stuff, but because wanting things is the human condition. Our lives are made up of successions of desires and attempts to satisfy those desires, and, as Buddhists tell us, it is these desires that cause our suffering. A human's life is a journey to try and eliminate suffering, hunger, pain, loneliness, etc, and to answer fundamental questions about what makes life worth living. Stories are depictions of an aspect of this journey, and become metaphors for our own experiences, wishes, and struggles. As a story teller this is the first thing you must understand.
The inciting incident the source of the desire that the story is about. It is the beginning. It upends the character's world, as new, powerful desires generally do; a new love, a new job, a new purpose in life, etc. In Hamlet, for example, the ghost of Prince Hamlet's father appears to tell him that he was killed by his Uncle Claudius and must be revenged. This changes everything for Hamlet. Justice and revenge are Hamlet's new desires, and they are desires we easily recognize, at times, in ourselves.
But there is an obstacle. Hamlet isn't sure if the ghost is a demon in disguise lying to him. He can't just march in and run Claudius through, or he might be killing an innocent man. He must come up with a plan to get him to reveal his guilt. And the story gets rolling.
Why must there always be obstacles to desire in the context of story? If Hamlet simply killed Claudius in the first act and that was the end of the play, we would feel that there wasn't much of a story. Why? Because if the protagonist is willing to put in the effort to overcome the obstacle shows that he/she values the object of desire. Further, overcoming the obstacle usually requires the protagonist to put him/herself at risk, and so the more he/she risks the more he/she must value the object of desire. If the object of desire is obtained too easily it's perceived by the audience as not having any value, not being worth the trouble, and thus not being "much of a story," since a story is about a desire. Hamlet desires justice and revenge so much that he's willing to risk his relationship with his girlfriend, his status in the court of Denmark, his sanity and finally his life, and in the end he gains his aim but at the cost of not only his own life but those his mother, his girlfriend, his girlfriend's family and even the liberty of his own country.
Of course, if the complications at the beginning of the story are bigger than those at the end, then the audience doesn't care about them as much. Hamlet confronting his mother would be less exciting if it happened after he'd already confronted and killed Claudius. (That is, unless the story was really about Hamlet's relationship with his mother, but that would be a different story altogether.) This is why the complications, the obstacles, should mount as the story progresses, to give the audience a sense of the growing risk and effort and thus a growing sense of the value the protagonist has for the object of desire.
Each act in a story allows for a rise and fall of action. Each act builds in a progression to a conclusion which then changes the situation in some fundamental way and leads into the next act, until the final climax where no further actions are necessary or possible vis a vis the plot. In screenwriting act climaxes are commonly referred to as reversals, that is they somehow reverse the situation of the protagonist. Usually they reversals are structured in an cycle of positive to negative value, that is the end of the first act is positive for the protagonist, the second act negative, and the third positive, though obviously this is not always or necessarily the case, it just creates a satisfying rhythm.
Why divide the story into acts at all? Why not simply have one progression of rising complication, one climax and the end? Of course you can, and this is certainly the case for one act plays and most short stories. Further, there is nothing about the human experience that says you can't have a long, slow progression and single climax. Certainly, in our lives that sort of thing does happen, say a long romance that culminates in marriage, or a long time spent working on a project that culminates in the project's completion and success. But even in human experience, the longer the situation the more likely there will be ups and downs, say a fight in the romance that leads to a break up and then a reunion, or a moment in the project when all seems lost and it will never get done, or whatever. Moreover, from a narrative perspective, the longer you spend building progression without a climax, the more chance for tedium to set in in your audience, especially if your progressions become repetitive. "Reversals", fundamental changes in the course of the story that force the protagonist to take on much greater risk, shock us and keep us interested. Three acts have become the standard simply because three climaxes and reversals are a satisfying rhythm for the audience. Hamlet has five acts and lots of twists and turns, so three acts isn't the only possibility, but one should be aware that a lot of contemporary audiences, trained on three acts, have trouble with Shakespeare and his plays are frequently shortened by contemporary directors. Again, that doesn't mean you shouldn't create something in five acts, only that you should make sure you understand what you're doing and why. For instance, long novels frequently have far more than three acts, but often manage this by being further subdivided in sections or "books" that are like mini-novels in themselves.
Plots are built the way that they are because they are a marriage of human needs--the acting out of desires that become metaphors for our own lives--and narrative needs--the need to stave off boredom and tedium by building from lesser complications to greater ones and by dividing the story into easily digestible acts. These are conventions, not rules or formulae, and they can be broken, played with, subverted, etc. but it is important to know why they exist and what their function is before you do so.