Directory Agent Profiles

Directory agents are AI workflows that automate listing aggregation, enrichment, deduplication, and classification — turning what was once a 200-hour manual research project into a recurring scheduled run. Common agent types cover web crawling, LLM-based summarization, embedding-based dedup, and outbound email enrichment.

The configuration tool helps spec out which agents a directory build needs given listing volume, source diversity, and update frequency. Output is an agent topology and rough cost estimate, not the running infrastructure itself.

The typical agent stack has four to six discrete components. A crawler ingests source data — RSS, sitemaps, scraped pages, public APIs. A normalizer cleans and structures the raw data into a consistent schema. An enricher fills in missing fields, often via paid APIs (Apollo, Clearbit, Crunchbase) or LLM-based extraction from associated content. A deduplicator compares incoming entries against existing records using embedding similarity (cosine > 0.92 is a common threshold) on canonical fields like name, normalized URL, and address. A classifier assigns each entry to taxonomy categories — embeddings plus a label vocabulary, or LLM-based classification with a prompt. Finally, a publisher writes accepted entries to the directory database. Some setups add a moderator agent that flags low-confidence classifications for human review.

A worked example. A regional services directory with 8,000 listings refreshed monthly. Sources: 5 industry feeds plus targeted web searches. Pipeline: crawler ingests roughly 1,500 candidates per month from feeds and 500 from search. Normalizer parses HTML and structured data into the standard schema. Enricher uses an LLM (gpt-4o-mini class) to write a 50-word description and assign 3 to 5 attributes per listing — about $0.005 per listing in token cost. Deduplicator embeds each candidate against the existing 8,000 and rejects matches above 0.92 cosine similarity — about 30 percent of candidates are dedups in steady state. Classifier assigns one of 24 categories using embedding-based nearest-neighbor classification, about 95 percent accurate. Cost per refresh cycle: $10 to $20 in LLM tokens, $50 to $100 in proxy/scraping infrastructure, plus the embedding service. Total monthly: $200 to $400 for the agent layer alone, plus the underlying compute.

Limitations and where production agent pipelines tend to break. Source format changes are the single biggest maintenance burden — websites redesign, feeds add or remove fields, APIs deprecate without notice. About 60 percent of agent runtime over the long term is spent on schema drift, not on the actual classification or extraction logic. Building monitoring that flags ingestion volume drops or shape changes is more valuable than getting initial accuracy 5 percentage points higher. Embedding-based dedup has tail-case problems: legitimately similar entities (a restaurant chain with multiple addresses) can match too aggressively; outright duplicates with different formatting can slip through. Manual review of dedup decisions in the first few weeks of a new pipeline catches most of these issues. LLM enrichment hallucinations are the second-biggest quality problem — models invent plausible-sounding but wrong data when source content is sparse. Constraining the LLM to only use facts present in the source text, and validating critical fields against structured signals (address against a postal database, business status against an active-company API), keeps quality high enough for production use.