Search Engines: Gateways to the Digital Economy Madanmohan Rao (madanr@planetasia.com) reports from the Search Engine Summit in New York As entry points to the largest explosion of information the human race has ever seen, Web directories and search engines are increasingly being seen as key gateways to the Internet economy. And new technological innovations, business models, and stupendous investments are being poured into this industry. Forty-six percent of Internet users find new Web sites via search engines, and word of mouth (20 percent) and random searching (20 percent) are the next most popular methods, according to a February report by research firm IMT Strategies. "The search engine business is becoming increasingly complex and driven by high stakes. Your search engine registration strategy can have a dramatic effect on your site's visibility and will also raise important design issues when you are developing or updating your site," says Danny Sullivan, editor of popular site SearchEngineWatch.com, which hosted the recent Search Engine Strategies 2000 forum in New York. What began about five years ago as a clean distinction between human-edited directories and spider-driven search engines is becoming increasingly blurred, as directories draw on search engine technology to supplement their listings and as search engines try to augment their crawling capabilities with human-edited channels. Adding to the complexity of this mix is a growing network of alliances driven by third-party providers of private-label search technology, multilingual search and directory services, fee-driven pay-per-performance business models, popularity ratings mechanisms to tap user behavior patterns, and even user-driven directory compilation via distributed creation of so-called open directories. As a result, many publishers, marketers, researchers, and Web solutions companies are loudly complaining that they are unable to keep up with the growing complexity of searching on the Net or that they are unsure of how best to register a site and its contents at the various search portals. Confused? Fear not. Numerous sites like WordSpot.com, MarketPosition.com, SearchEngineWatch.com, and SpiderHunter.com are leaping into the fray here, offering tips, news, alerts, surveys, and research on the listings and search business. In fact, harnessing search engines has become such a key part of corporate marketing that entirely new categories of companies are springing up. They're called search engine optimizers (SEOs). And Web marketing agencies are adding a new job role: the search engine specialist (SES). A good example of a search engine optimizer is Massachusetts-based iProspect.com, whose offerings include HTML optimization, keyword analysis, individualized directory positioning, and click-through monitoring for clients like 3M, eCandy, Deutsche Bank, and Sharp Electronics. Other companies in this submission and optimization space include Submit-It, Register-It, WebPosition, PositionPro, RoIDirect, and TopDog. One of the more successful search engine specialists is Shari Thurow of Illinois-based GrantasticDesigns.com, who claims to have a 100 percent success rate in getting her numerous client sites ranked in the top 10 positions for their keywords in leading search engines. "I have been on every search engine almost every single day for the past five years-even on holidays," Thurow boasts. In addition to basic listing of company URL and description, some search engines offer additional options like logo display (for example, Ah-ha.com), advertorials (Infoseek), and natural-language query (AskJeeves.com). One of the innovative business models among search engines is pay for placement, pioneered by GoTo.com, whereby companies pay to have their sites appear at the top of listings under various keywords. Companies are billed only for click throughs, and they bid for priority placement through dynamic online auctions. GoTo has attracted more than 21,000 advertisers with this service and was recently ranked among the 20 most popular sites on the Web. This model has spawned numerous other pay-for-position search sites such as FindWhat.com, RocketLinks, SimpleSearch, HitsGalore, and Kanoodle. Some search engines monitor user behavior to find out which relevant sites users visit, how much time they spend on each site, and how successful the search may have been. Companies operating in this space include DirectHit.com, which was recently acquired by AskJeeves.com. DirectHit's popularity engine technology has been used by sites like ZDNet, ICQ, and InfoSpace.com. "Our approach is to humanize the Net by allowing users to type queries in natural language and also let our search engine reflect the popularity of the sites listed. Together with DirectHit, we deliver a higher quality experience for users as well as a more robust service for companies looking to retain customers," says Edward Boudrot, product manager at AskJeeves.com. Another innovative approach allows user participation not just in submission of entries but also in actual construction of directories: this approach is used by Open Directory, Go, PlanetClick, and Snap. "Instead of fighting the explosive growth of the Internet, Open Directory provides the means for the Internet to organize itself via its own communities of users," says Chris Tolles, marketing director at the Open Directory Project (http://www.dmoz.org). "We rope in an army of human volunteers who have edited and classified up to 1.5 million sites in 200,000 categories. We have 23,000 editors in over 200 countries, working on content in 50 languages. They vary in age from a 13-year-old editor of video game content all the way up to senior citizens who are nuclear physicists," says Tolles. In keeping with the open-source tradition driving the growth of operating systems like Linux, license for Open Directory content is also available for free use with attribution and is currently accessible via sites like Netscape, AT&T WorldNet, and Hotbot. "We want to create the largest collectively operated human-edited directory on the Net," says Tolles. Typically, directories focus on submissions from users and do not address content in metatags or individual pages. Search engines actually crawl through most content on the World Wide Web. Yahoo! is the most popular directory, followed by others like Netscape's Open Directory (http://www.dmoz.org), Looksmart (stand-alone and on the Microsoft Network), and Snap. The major search engines are Altavista, Google, Inktomi, Excite, Go, Lycos, Northern Light, and FAST. While many search engines have expanded into portal space by adding services like e-mail, others like Google continue to offer only search facilities. "We prefer to offer uncluttered, high-quality search services only; that is why we continue to draw over 12 million queries a day," says Aydin Senkut, product manager at Google, which was founded by two Stanford University students in 1998. The search engines typically create a periodically updated index of Web text that is then searched. But search engines differ widely in how frequently they update this index, in what search algorithms they use, in how they accept submissions, and in what kind of metadata they search (for example, Altavista and Inktomi search metakeyword and description tags, but Google and Lycos do not). Some search engines have been so caught up with updating their technology that they have fallen behind temporarily in keeping up with the growth of the Net. "Excite has not been able to include newly registered sites in its index since last November. Lycos has not added new sites since last December," complains Louis Villano, who runs a Web promotion company in New Jersey called NetActions. "I wish they would at least tell me whether I should keep resubmitting registration requests for my clients or just wait to hear from them." Faced with growing frustration from users who want their sites listed in a guaranteed amount of time, directories like Yahoo! and Looksmart have begun to charge users $199 for express submission, which guarantees that an editor will look at the submission within 48 hours. Competition between companies like Yahoo! and Looksmart in the international market is quite fierce, with each operating in over a dozen countries and languages. Looksmart has its own site as well as alliances with MSN, Excite, and Altavista. Looksmart's partnering agreements ensure that its services are the ones used in one form or another by 70 percent of all users in the U.S., says Kate Wingerson, editor in chief at Looksmart. Another area of expansion for search engines lies in the world of serious business and academic researchers, where it is important to judge not just the popularity but also the credibility of online content. NorthernLight.com combines searches on the Web with access to news and articles from 6,200 business, sports, and health care publications, which can be organized into folders. It also offers alert services when new articles or Web sites in those categories are available. IntelliSeek.com, founded in 1997 in Cincinnati by Indian entrepreneur Mahendra Vora, also targets the search space of hundreds of online business news sources-from Associated Press to ZDNet. Other offerings include comparison shopping. Many directories are emerging in the B2B and yellow pages arenas and are targeting specific demographic groups, such as women and children, or lifestyle areas, such as sports. A new arrival in the clean, family-oriented directory space is the site Ah-ha.com, which now has 600,000 users who prefer to stay away from pornographic and other adult content on the Web. The site Active.com provides listings of sports organizations and events. Another growing market for search engines consists in services that target content not on the entire Web but within organizational Intranets or within a specific World Wide Web site only. Companies operating in this space include SearchButton.com, which powers in-site search services for more than 6,000 sites-such as Quote.com, Shell Oil, and Web Radio-and smaller content sites-like Mom.com, which targets mothers. The exploding world of Internet-cell phone gateways is also being targeted by some search engine companies like FAST. Founded in Norway in 1997, FAST has launched search services for WAP (wireless-access-protocol) sites such as www.gelon.net and wap.fast.no. Keeping up with the search engine world is important not just for marketers and researchers but also for site developers, because the way a search engine views a site is closely related to site design and content structure. "Tables, splash pages, frames, and dynamically delivered content add to complexity of the site and are not always understood by a search engine," Sullivan says. "Site designers should keep in mind that sites should be designed with two audiences in mind: the end user and the search engines. Text-only features are good for users dialing up from slow lines and also for search engines," says Detlev Johnson, search engine specialist at Web marketing company MMG/Outrider. Search engine specialists should be brought in during the design and development of the Web site templates, and not after site creation, says Johnson. Link popularity is a growing feature affecting site ranking at such search engines as Google, thus providing growing impetus for the convergence between publishing and linking of content on the Web. Hence it is important for sites to actively promote links to themselves from other sites via techniques like link exchanges, referrals, affiliates, Web rings, and inclusion in yellow pages sites as well as in vertical industry hubs. Owners of larger, more complex sites should also consider separately registering some of their individual pages and even breaking up a site into smaller subsites with subdomain names (such as http://software.ibm.com). Challenges arise for search engine and directory companies in weeding out dead links, keeping up with sites that deal with rapidly breaking news, reporting traffic on cached pages, ensuring online privacy, dealing with pornographic or hate content, and thwarting spammers. Companies operating in the multilingual search space include U.S.-based Yahoo! and Altavista and the France-based Voilà network. Voila.com, a network of 10 portals in 9 languages that accounts for 150 million page views a month, is one of several sites offered by France Télécom, which also runs other sites focusing on music, books, and children. Challenges arise in devising search engines for double-byte languages, such as most Asian languages. The Japanese search engine market is growing fast and furious, with over two dozen players; it will be interesting to see how the market pans out for Indian-language search engines. The writer can be reached at madanr@microland.co.in About the Author: Madanmohan Rao