Gsd offers to get your web www ready and we use skill
regarding your keywords and descriptions to make your
web "web bot friendly"! You may ask why use this and
what is a web bot or web crawler...
A Web crawler is a computer program that browses the
World Wide Web in a methodical, automated manner or in
an orderly fashion. Other terms for Web crawlers are
ants, automatic indexers, bots, Web spiders, Web
robots, or—especially in the FOAF community—Web
This process is called Web crawling or spidering. Many
sites, in particular search engines, use spidering as a
means of providing up-to-date data. Web crawlers are
mainly used to create a copy of all the visited pages
for later processing by a search engine that will index
the downloaded pages to provide fast searches.
Crawlers can also be used for automating maintenance
tasks on a Web site, such as checking links or
validating HTML code. Also, crawlers can be used to
gather specific types of information from Web pages.
A Web crawler is one type of bot, or software agent. In
general, it starts with a list of URLs to visit, called
the seeds. As the crawler visits these URLs, it
identifies all the hyperlinks in the page and adds them
to the list of URLs to visit, called the crawl frontier.
URLs from the frontier are recursively visited according
to a set of policies.
The large volume implies that the crawler can only
download limited number of the Web pages within a given
time, so it needs to prioritize its downloads. The high
rate of change implies that the pages might have already
been updated or even deleted.
The number of possible crawlable URLs being generated by
server-side software has also made it difficult for web
crawlers to avoid retrieving duplicate content. Endless
combinations of HTTP GET (URL-based) parameters exist,
of which only a small selection will actually return
unique content. For example, a simple online photo
gallery may offer three options to users, as specified
through HTTP GET parameters in the URL.
If there exist four ways to sort images, three choices
of thumbnail size, two file formats, and an option to
disable user-provided content, then the same set of
content can be accessed with 48 different URLs, all of
which may be linked on the site. This mathematical
combination creates a problem for crawlers, as they must
sort through endless combinations of relatively minor
scripted changes in order to retrieve unique content.
This is where GSD Potch will help our skill and page
setup makes it easy for crawlers to mine the relevant
keywords and info...