What kind of book

This is a question I get all the time. What kind of book are you writing? I find this question almost impossible to answer, because really, the answer is, a good one, I hope. After that, what does it matter what other descriptors? And what does the interlocutor mean to do with the information anyhow? What will it really tell anyone about Fallsy Downsies?
The other day, as I made my way into the office, the woman who works the front desk during the day asked me the question. I was late for a meeting, as usual, and had only had one coffee, and so I looked at her with confusion. And she said, you know, sci fi, romance…? This is the other part of the question that stymies me. The genre question. I think this is the only way we have to talk about stories in a broader way, it seems. I mean writers, and constant readers of a certain kind have specific ways to talk about books and kinds of stories. But what I think of as casual readers, and those who maybe haven’t read a book since high school, because it’s just not how they choose to consume narrative, seem to only be able to relate to books and stories by assigning them a genre.
The only answer I have is “literary fiction.” Highly unsatisfactory response. I was once accused, in my bookseller days, of being pretentious. Because I referred to a Jack Whyte book as “not our kind of book”–by which I meant the shop up the street had more a speciality in that area, and would likely have stock, while we had sold through the three copies we’d ordered. The customer didn’t like the implication, and plainly told me so. I didn’t need her to tell me I was pretentious–I was a writer/bookseller in my mid-twenties OF COURSE I was pretentious. It was basically my full time job to be so. Anyhow, older and only a little wiser, answering Literary Fiction to the genre question makes me feel genuinely pretentious.
So, here’s the thing. Why is genre the only way the casual/non-reading public at large can relate to a story?
How else to describe something that is not a mystery, nor a romance, nor sci fi, nor a western, but may have elements of all of those, except maybe the sci fi part?
What’s the point of describing something that isn’t even done? Whose purposes does it serve, anyhow?
Does any of it matter to anyone but writers?
What do you think?