out of curiosity, is there a self-hostable search engine that only crawls given domains? this could be useful for only indexing forums, for example, and this seems relatively reasonable for a small server i think
i guess Google CSE could work for that out of proprietary solutions