Two weeks after Apple Computer released a so-called "metasearch" engine designed to piggyback on top of other Web search engines, a new product will now let just about anyone build their own.
On Thursday, Thunderstone released Webinator 2.5, software that lets any Unix or Windows NT user create a customized metasearch engine that will query and aggregate results from the big five search engines and major online databases.
"It's not just for Internet search engine content -- you can do a metasearch for anything," said Thunderstone general manager Bart Richards. "We want people to be creative, we want them to have fun."
The fun begins with Thunderstone's default metasearch front-end, which is set to query and compile results from Excite, Infoseek, Inktomi, Lycos, and Yahoo.
The idea of free and public metasearch engines is not new -- MetaCrawler has been feeding off other search engines for years. But Webinator is the first to allow users to create their own parasitic search tool. If the idea takes off, the big Internet directories might sit up and take notice.
Inktomi spokesman Kevin Brown said that his company has been dealing with this issue for several years. Inktomi technology is used in many search engines, including Wired Digital's HotBot. Wired Digital is the parent company of Wired News.
"Our long-run take on this is that for anyone who is going to piggyback on a service that costs us or our customers money, there's no significant free ride forever," Brown said. "We can support many, many small free riders -- but if something becomes a category killer, you then have to deal with it that way."
Richards said he shipped his product with links to the major search engines as "an example" of what the technology can do. He said he has no plans to license the output of those sites.
"It'd be fun to stir the pot a little bit to raise this as an issue, but the issue is very clear -- you're not supposed to republish without authorization," Richards said.
Erik Selberg, one of the authors of MetaCrawler, said that when his site first hit the radar of the big search engines, they decided to negotiate licensing arrangements rather than block it.
Selberg said that the big engines were hesitant to block MetaCrawler altogether, fearing that he might turn around and release his program's source code so that users could create their own similar engine.
"One MetaCrawler means one entity to negotiate with, which is nice," Selberg said. "But a custom MetaCrawler on every desktop? Bad news."
Selberg said that there's precious little that search engine companies can do to block client-side software like Webinator.
"It's easy to spoof something like Netscape so they can't tell you're a metasearch engine versus a browser," he said. "And it's unclear whether there's anything illegal about retrieving Web pages from an engine: People who filch off Lycos are likely [to be] under the same liability as Lycos is for filching off [an individual's] Web pages."
Richards said that the Webinator is not intended for use as a public metasearch engine, since that would almost certainly require a license from the search engines involved.
Instead, the company is aiming Webinator at private users and intranets.
"It would be amazingly useful if turned inwards, so somebody could search an intranet with a wide variety of search engines, like the personnel database, an internal patents [or] white-papers database, etc.," Richards said. "There's great value to aggregating metasearches, for competitive intelligence or whatever it is you happen to do."
For example, Richards said you could aggregate the results of several stock sites to get all the unique information each has for a particular stock symbol.
"With this software that we have, it is not rocket science."
