Having previously talked about the book buying profession, here's one reason why brick-and-mortal bookstores won't stock a comprehensive and quality bookshelf: there's a limit to the familiarity and awareness of every book buyer. Or put it in another way, each book buyer only has enough time to familiarize himself or herself with a few genres. But a bookstore contains multiple genres so if there is only one book buyer, he or she won't always make the best choice for each and every genre shelf.
For example, I'm a science-fiction/fantasy fan--that genre is my so-called forte. Yet as a SF&F fan, I know my genre has various sub-genres that cater to different markets, so I already have my hands full catering to "my shelf". But what if I'm assigned to the philosophy section, a genre I know next to nothing about? Or the general fiction section? Or crime/suspense? I might have an inkling on what the children's book market and horror market is like but it's nowhere as comprehensive as SF&F. And unless you've spent your entire life doing nothing but reading all sorts of books (throw in speed-reading training in there), I think it's virtually impossible to cover all topics and genres.
The answer might seem simple: hire more book buyers, each of whom are specialized in a particular genre. The first problem there is looking for these so-called specialists. I mean I'm advertising my services for SF&F yet bookstores don't necessarily know that I exist, nor do they trust my judgment. Because there's a point where book buyer genre specialists (BBGS)--a term I'm coining now--have to ask, am I getting this book because I want it to be stocked, or am I getting this because it'll sell? If the latter isn't true, the bookstore shouldn't be hiring you. The second problem, of course, is the budget. I don't think bookstores have the budget to hire a lot of BBGS, especially if they're starting small. Yet the paradox there is most independent bookstores (which usually has one book buyer, namely the owner/manager) stock various genres of books. I have yet to see a successful independent bookstore that caters to one specific market, and then branch out to other genres as they expand. That, of course, is different from knowing your niche. Aeon Books, because it's situated opposite of two colleges, has had a boom in the philosophy genre. That doesn't mean they dumped all their other genres and start selling solely philosophy books. They know their niche and they're acquiring better selections of philosophy books, but that doesn't mean their other genres are withering either.
So how do individual book buyers determine what books their bookstore stocks? Well, if you're a veteran in the business, your records should show which books are selling and which aren't. Then sometimes, you acquire books that certain publishers are pushing, or the latest fad/popular book (i.e. The Da Vinci Code, Harry Potter). For me, what's a more interesting question is how do starting bookstores determine their book selection.
That's where the power of lists comes from. As much as readers benefit from best-seller lists, book buyers benefit from them more. Lists such as the New York Times Best Seller List may be imperfect, but they're the closest thing a book buyer has to a starting guide. Another list that is important might be a list of "classic books" in specific genres (i.e. Sir Arthur Conan Doyle for mystery, Ray Bradbury for fantasy, etc.).
Or a book buyer can trust his or her tastes. Which is fine when it's territory they're familiar with but doubt starts to creep in when it's a genre they haven't read a single book about yet has an entire shelf to fill up.
Then there'll be the nagging idea of am I getting this book because I like it or am I getting this book because I know it'll sell? This I think becomes a bigger problem when you're the owner--because you do want your inventory to sell. Or worse, if you're the owner and you confuse the two. If you're just a hired gun on the other hand, you might think to hell with the profits, as long as the bookstore stocks "good" books that the public should read.
That's also not taking into consideration your fickle market which you have no control of. You might acquire the books that are guaranteed to sell but if the public doesn't buy them anyway then you can't help but wonder whether you should restructure your book buying techniques or hire a new book buyer altogether (even if the cause for lack of sales isn't their fault).
So right now, there's no feasible solution that I can see, at least for physical stores. Online shops like Amazon.com are like Do-It-Yourself bookstores in the sense that consumers determine their own "shelf" because there is no actual shelf so to speak: we only browse the books we want to buy or are considering buying. Amazon tries to sway our choices by giving recommended reads or bargains if you buy this book along with that book, but at the end of the day, it's a proactive choice on our part what books we browse and what books we don't. But then again, online shops have less chances for serendipity than actual bookstores.