Writing what is popular versus what would happen

it is difficult to write for readers sometimes. Everyone wants a happy ending and not just any happy ending but the one they expect. Enter chaotic writer, Not all of my stories will end with a cherry milkshake. What do you think? anyone have thoughts on writing for the reader vs the story?