crawling github

[8493834]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

API

Body

Hi my name is Joshua an i don't know if this fits into this category but anyway, i'v searched google how to get permission to crawl (index) github to my own search engine (SPS search) and i cant find a answer

[aimanfazal]

Answer

GitHub does not require a special "crawling permission" to index public repositories.

For an indexer/search engine, the recommended approach is to use GitHub's REST or GraphQL API instead of scraping GitHub pages.

API limits

• Unauthenticated: 60 requests/hour
• Authenticated: typically 5,000 requests/hour
• Use a GitHub App or personal access token when authentication is needed.
• Respect GitHub's rate limits and use caching/conditional requests where possible.

What permissions do I need?

It depends on what you're indexing:

• Public repositories: generally no special repository permission is required.
• Private repositories: your token/App must have access to those repositories.
• Organization data: additional organization/repository permissions may be required.

So, there is no separate permission that simply enables "GitHub crawling." Use the GitHub API and request only the permissions your application actually needs.

If this answer was helpful, please mark it as answered.

[8493834]

yes but how do i do that?

[8493834]

here (take note that it says blocked by robots.txt): github.com: 100 pages (skipped: 5 fetch errors, 6 redirected out of scope or duplicate, 1 blocked by robots.txt)

[8493834]

and i mean to index the docs and stuff like that

[8493834]

if you search up github on the SPS search website you will know what i mean

[beenish-majeed]

The blocked by robots.txt message basically means your crawler is respecting GitHub's rules. You shouldn't try to bypass those blocks. GitHub's current robots.txt tells crawlers that want to crawl GitHub to contact GitHub Support. ([github.com](https://github.com/robots.txt))

For what you're trying to build, though, there are probably better ways than crawling the whole site:

• GitHub Docs: The documentation is available in the github/docs repository, so you can clone that repo and index the Markdown files instead of crawling docs.github.com. Just check the repository's license before redistributing the content.

• Repository READMEs/files: Use the GitHub REST API. For example:

curl -H "Authorization: Bearer YOUR_TOKEN" \
  https://api.github.com/repos/OWNER/REPO/readme

The README endpoint also works without authentication for public repositories.

• Finding repositories: You can use GitHub's Search API to find repositories and then fetch their contents through the API. Keep in mind that the Search API has its own rate limits, which are more restrictive than the general REST API limits.

• Rate limits: Authenticated REST API requests generally have a 5,000 requests/hour primary limit. It's also a good idea to cache responses and use conditional requests with ETags (If-None-Match) so you don't repeatedly download unchanged data.

So for SPS Search, I'd use the API and public source repositories wherever possible. For pages that you specifically need to crawl from github.com and that are blocked by robots.txt, contact GitHub Support rather than trying to work around the restriction.

[PythonAlpha]

If your goal is to build a search engine that indexes GitHub content, I wouldn’t try to bypass the "robots.txt" restrictions.

For public repositories, you can use GitHub’s REST or GraphQL API to discover repositories and retrieve things like READMEs and repository files. For GitHub Docs specifically, you can also work from the public "github/docs" repository instead of crawling the documentation website directly.

For example, you can search for repositories through the GitHub API, then use the repository contents endpoints to retrieve the files you want to index. Caching the results will also help you avoid unnecessary API requests and rate-limit problems.

If SPS Search needs pages that GitHub specifically blocks from crawling, the safest route is to contact GitHub Support rather than trying to work around "robots.txt".