The concept of world building usually comes into play in science-fiction and fantasy because well, most world building in realist stories are based on the real world (that's not to say realist writers don't have to do some world building... they do, they just don't have to describe what an automobile is when they mention it in their stories for example). Of course having said that, I think as a reader, I don't want to see world building in novels. I'm there for the stories after all.
Now let me clarify that statement. When I say I don't want to see world building in stories, I don't mean there's no world building at all. What that simply means is that the world building happens in the background, not shoved down our throats. For example, Lord of the Rings has world building but it takes place as part of the story. The same goes for the Belgariad and the Mallorean. The Silmarillion and the Rivan Codex, on the other hand, are books about the world (and in the case of the latter, more or less the series's bible). They don't make interesting reading in themselves, unless you're a fan of their respective series. I don't think anyone would enjoy reading Rivan Codex for example without reading the Belgariad first (of course like all things, there will be exceptions). As a writer however, I'm curious about these kinds of stuff, but at the end of the day, world building isn't something I'd give to a typical reader (even a genre reader). The most world-building I've read in an actual novel is probably the first chapter in The Stars My Destination. The rest of the book quickly shifts to the actual narrative after that, however, so I'm okay with it.
Now RPG books, on the other hand, are different. There are basically two reasons why people buy RPG books: they either buy it for the game system, or they buy it for the world building. I mean I used to have the Forgotten Realms Campaign Setting at home and currently the Eberron Campaign Setting and a good chunk of the books are about histories and geographical descriptions and cultures of fictitious worlds. When I bought those books, I knew this was what I was buying. And while I derive pleasure from reading these fictional facts, it's only because I know they're a means to an end and that I'll subvert them for my own narrative or campaign. The closest thing I have to a narrative in those books are the histories but everything else, I'm thinking "how do I use this in my game?" and it's because of those kinds of thoughts that I find them interesting and useful.
At the end of the day, it's about finding the right tool for the right job. World building in fiction? Not so much. World building in gaming? Bring it on!