Custom Search Engine

[Shawn Gossman]

Have any of you ever programmed your own custom search engine?

How about a way for a "robot" to index content for it?

I feel like this would be a pretty cool thing to create.

 

[AWS]

Yes I did. I created a spider/robot to crawl and index forums. I still have the scripts I wrote for it. I would have to update it for the newer versions of PHP. It was coded when PHP 5 was first released. Not to mention the tools I used for screenshots are gone now. Would have to replace them. Maybe I'll dust it off and see if it can handle today's complex forums.

There is a post on vbulletin.com about it. I wanted to see if it affected their sites negatively. I always sent it out to go 1 level deep to prevent problems.

 

[Shawn Gossman]

Yes I did. I created a spider/robot to crawl and index forums. I still have the scripts I wrote for it. I would have to update it for the newer versions of PHP. It was coded when PHP 5 was first released. Not to mention the tools I used for screenshots are gone now. Would have to replace them. Maybe I'll dust it off and see if it can handle today's complex forums.

There is a post on vbulletin.com about it. I wanted to see if it affected their sites negatively. I always sent it out to go 1 level deep to prevent problems.

That's awesome! You could make a forum search engine!

I think a forum DMOZ-like site would be cool. Multiple editors and all. But you might have to keep links as "No Follow" to avoid Google penalization.

 

[AWS]

I actually created the spider because at the time I was an editor there. I wanted an easy way to populate a category. Later I built a frontend for it. The original grabbed the links and description and formatted to an xml file that I could import.