Yahoo may soon have to admit that the very service that made it the Web's Number One site is inherently flawed. As Web growth explodes, critics charge, up to a third of the sites seeking a listing in the popular directory don't get in.
Even Yahoo has long, if quietly, admitted that some sites may take months, even years, to get listed at all. And the impact can be dramatic, as a listing in Yahoo can mean feast or famine when it comes to attracting browsers to a Web site.
Just as it reaches new levels of popularity and business success - the Web's foremost directory service may soon be unable to call itself anywhere near up-to-date.
"They're getting out of the [directory] business, and they should be honest about that," said Louis Rosenfeld, a man who has had personal experience with the difficulty of getting a site listed in Yahoo.
In fact, unexpected reinforcement of his charge can be found in Yahoo's most recent published annual report of 1996, which, in outlining the threats to its business, reads "the Company has from time to time experienced significant delays in the processing of submissions, and further delays could have a material adverse effect on the Company's goodwill among Web users and content providers, and on the Company's business."
"Why is this not a problem anymore?" asks Danny Sullivan, editor of Search Engine Watch, which keeps an eye on the doings of Web directories and search engines.
Submitter Rosenfeld chronicled ongoing problems with his Yahoo listings. While he did get his site listed years ago, his efforts to make corrections when his listings became outdated or incorrect were exercises in frustration.
"I've had terrible problems with the same URLs going back for three years," he said. After waging a "campaign to reach a human" at Yahoo, he found his contact very friendly and responsive - but the fixes they made went only halfway. Today, half his listings remain incorrect, and after his initial go-round, he sees their complete correction as all but hopeless.
Something of a Yahoo competitor, Rosenfeld in fact runs his own service aimed at specialized, rather than generalized, cataloging of topical sites on the Web - an approach he says is the only kind that stands a chance in the onslaught of Web-based information.
Competitor or not, Rosenfeld is not alone. Usenet discussions and Internet mailing lists are rife with the same basic complaint: I can't get listed in Yahoo.
Another example is Joan Sarah Touzet, a designer of freeware fonts. In 1993, she set up a site offering her typefaces for download. It wasn't until just last week that her site showed up in Yahoo's new listings, coming as something of a blast from the past to Touzet.
She cannot now recall for sure when - or even if - she submitted the site to Yahoo. "It's very possible that I gave them a request, but it would have been two or three years ago," she said.
The uncertainty surrounding Touzet's listing tells much, in fact, about the revolving nature of Yahoo's listing process. Yahoo's Director of Surfing (aka editor in chief) Srinija Srinivasan readily admits the process is not based on one single order of business.
Explaining possible scenarios surrounding Touzet's surprise listing, said Srinivasan, "we don't ever lose submissions - it could mean that we actually found it finally, but more likely that that particular site was about something that a surfer here had on their list to find good content about."
Anyhow, she said, while users may perceive Yahoo as a comprehensive directory, it's a mistaken perception. "There's no question that people (heap) Yahoo together with other search engines as trying to serve the same purpose - when that's not true," Srinivasan said.
Instead, Yahoo is a "media company," she said, "aggregating information that we think is of use."
To Yahoo's team of an estimated 50 surfers (the company won't disclose exact numbers), far from attempting to process the tens of thousands of weekly submissions, "aggregating information" means doing whatever it can, within reason, to keep abreast of "what's happening," in Srinivasan's words, on the Web.
Critics say if that's what Yahoo is, it's time to be more explicit about it.
"The bulk of the sites that are submitted are not getting in," said Sullivan of Search Engine Watch - as few as a quarter of submitted URLs, in his estimation. The only rough numbers that Srinivasan would discuss suggest that anything from a third to half of weekly listings are added, an action she distinguishes from "processing."
Yet Sullivan said, "people tend to look at [Yahoo] as 'if there's a site that's out on the Web, it should be listed in Yahoo.'" While Yahoo used to suggest allowing two to three weeks for new listings, Sullivan said "they've shifted their attitude to 'we're listing what we get.'"
They've either got to be explicit about this shift, Sullivan said, or bolster the core directory service with more surfers (and thus more spending) to better keep up with demand.
But it's not a given, Srinivasan said, that more is better. Asked if she would like surfers on her team, she said "most of the time, you would like to have a little more."
But growing too fast, she said, can be at odds with the editorial coherence of the surfing team. "To the extent that we do this well at all is that we are all here in this collaborative environment and we help each other." The consistent approach to categorization and listing priorities would be threatened by a sudden doubling of staff.
Yet the upshot, Srinivasan recognizes, is the prospect of never getting ahead of the monumental task facing them daily. "We can't possibly scale in head-count to the growth of the Web," she said. Neither does she think that goal is desirable. "I don't think users want us to be a manual attempt at what the search engines do."
Some agree with Yahoo's user-centric, site-excluding Web view. "Will they continue to double it in size in terms of their own employees? That would be a challenge - and I'm not sure their users would want it," said Brewster Kahle, inventor of the Wide Area Information Server for electronic publishing and founder of Web-guide startup Alexa, which provides contextual site information as browsers surf.
Frustrated submitter Louis Rosenfeld has a background in library science and agrees that Yahoo's task - much like the physical world's Library of Congress - can no longer keep up with the breadth of information.
Specialization is therefore the better approach, observers say, and in fact, in its evolution to become an online service - or "media company" - this effectively defines Yahoo's new directions.
But that doesn't mean that Yahoo shouldn't make its evolution clearer in regard to user expectation.
"They've got to make a statement at some point about what Yahoo the directory is going to be," Rosenfeld said. "It's just a disservice to the entire Internet if they don't."
"We make it explicit that we don't guarantee [listings]," Srinivasan counters. Perhaps it's not abundantly obvious on Yahoo's front page, she said, but if one way or another the user finds what they wanted, Yahoo's mission is fulfilled.
Still, this focus on the user has changing implications for the Web site of the "little guy." While no placement or prioritization scheme at Yahoo puts one kind of site ahead of another, Srinivasan acknowledges that the sheer force of numbers on the Web makes it increasingly difficult for a site to get in.
The effect on small sites is illustrated by what happened in the case of Touzet's sudden listing on February 5th. Since that date, she said hits have increased 10- to 20-fold by her hit calculations.
"There has been an incredible increase in hits," Touzet said.
