An extramarital affair threatens the fortunes of two families.

In Los Angeles, writer Roger Altar, who has won public acclaim but critical scorn for his pulp novels, hires architect Larry Coe to design a house based on the photos of a prize-winning house that Larry designed eight years earlier. While shopping at the local market, Larry meets his new neighbor, Margaret Gault, and is rendered speechless by her beauty. When Margaret returns home, her mother, Mrs. Wagner, whom Margaret has never forgiven for committing adultery, states that she knows that Margaret is not in love with her husband Ken and predicts that she also will be unfaithful one day. The next day, Larry deliberately walks his son David to the school bus stop so that he can meet Margaret, whom he calls Maggie, and convince her to accompany him to the Bel-Air site where he is to build Roger's house. Larry, excited by the prospect of building an innovative structure, is pleased that Maggie shares his enthusiasm and has even sought out the eight-year-old magazine article containing the photo of his last major accomplishment. When Larry sends the completed plans to Roger, Roger is fearful that he will be laughed at for building an "odd ball" house and balks until Larry accuses him of pandering to his critics and convinces him to take a chance. One night, while Larry and his wife Eve are dining with Larry's former employer, Stanley Baxter, Stanley asks Larry to design a new wing for a factory in Hawaii. Larry, who longs for his earlier acclaim and fears that life is passing him by, is reluctant to work on such a prosaic project, but Eve, concerned with financial stability, pressures him into accepting the commission. At the bus stop the next morning, Larry invites Maggie to join him at the Bel-Air job site, but she refuses. When Larry admits that he wants to see her again, however, Maggie, feeling neglected and unloved by her husband, whose ardor has dwindled over the years, agrees to meet Larry the following evening. On the night of her assignation, Maggie tells Ken that she is going out with friends, and when she mentions his lack of sexual interest, he silences her. At a Malibu hotel, Maggie confides to Larry that her father was the only other person who ever called her Maggie. After a tenuous start, the lovers finally consummate their affair. At home, Larry sees his wife, washing the dishes and wearing her hair in curlers, and longs for the sexy Maggie. Using the house project as an excuse for spending time away from home, Larry continues to see Maggie. One day, when Maggie goes to meet Larry at a restaurant, she is tailed by a man. Excusing herself from the table to call home, Maggie is accosted by the man in the lobby. Hearing the commotion, Larry comes to Maggie's defense and slugs the man, who then runs off. Although Maggie initially claims that she does not know her assailant, she finally admits that he forced himself on her the summer before. When she recalls the incident, in which she left her door open and took sleeping pills, Larry becomes enraged and accuses her of wanting that to happen. Stung, Maggie drives off. That night, the tormented Larry calls Maggie's house, but when Ken answers, hangs up the phone. When Maggie fails to appear at the bus stop the next morning, Larry goes to Maggie's house, where he is met by Mrs. Wagner. After Larry leaves, Maggie's perceptive mother comments "so now it has happened to you." Concerned with Larry's distance and hoping to draw him back into her life, Eve decides to throw a neighborhood party and calls Maggie to invite her and Ken, but Maggie hedges about accepting the invitation. At the party, Stanley tells Larry that his company was so impressed by his work on the factory that they want him to design an entire city in Hawaii, a project that would take five years to complete and require a move to Hawaii. Demurring, Larry asks Stanley not to mention the offer to Eve until he has a chance to discuss it with her. Larry is shocked by the arrival of Maggie and Ken, and later, alone with Maggie, asks her why she came. Just as Larry writes "I love you" on a sheet of paper, Felix Anders, the Coes's snide neighbor, enter the room and Larry crumples the paper. After the Gaults leave, Felix insinuates that he knows Larry is having an affair and warns him that Eve is also suspicious. When Larry denies the accusation, Felix hands him the crumpled love note. After Eve goes to Palm Springs for the weekend with her parents, Larry unexpectedly meets Maggie at an amusement park, where they have both taken their children. Warning that Felix knows about them, Larry declares that either she wants "more or less," to which Maggie responds she wants their relationship to stay the same. Larry then suggests that they stop seeing each other. Soon after, Stanley phones the house, and when Eve answers, tells her about the Hawaiian job offer. When Larry comes home, Eve, upset, asks why he never mentioned the offer and the two argue. As Larry storms out of the house, Eve tells him not to come back. Larry goes to visit Roger, who, although he has received rave reviews for his new book, is still unhappy. When the womanizing Roger envies Larry because "he is married, has a family and knows where he's going," Larry replies that he is a phony and confides that he has been unfaithful. Larry concludes by saying that although he loves his mistress, he does not want to hurt his wife. He then calls Maggie and arranges to meet her the next day at Roger's completed house. Meanwhile, Eve, who is about to step into the shower, is paid an unexpected visit by Felix, who begins to flirt with her menacingly. When he tugs at her bathrobe, she becomes hysterical and throws him out. As Felix leaves, Larry pulls up in his car and upon finding Eve in tears, deduces what has happened. After Larry slugs Felix, Felix smugly observes that Larry is no better than he. Finally admitting to Larry that she knows he is having an affair, Eve asks him to leave and he takes refuge in his office, located in a wing of the house. Later, Eve enters the office sobbing, and after asking what she did wrong, avows that she cannot live without him. The next day, when Larry and Maggie meet at Roger's vacant house, Larry tells her that he is going to Hawaii to build a city. As they acknowledge their love for each other, they are interrupted by the contractor, who assumes that Maggie is Larry's wife. Saying goodbye, Maggie drives down the driveway and out of Larry's life.