Free Log File Analyzer — See Which URLs Googlebot Crawls and What Status It Gets
Paste lines from an Apache or Nginx access log, or open a log file, and this log file analyzer shows how often Googlebot and other crawlers visit, which URLs they hit most, and how many requests return 404, 5xx or redirects. It runs in your browser: your log never leaves your device.
Paste log lines to start.
What the Log File Analyzer Reports
- Crawler share: requests from the crawler you pick, compared with all requests in the log
- Status codes: how many of that crawler's requests returned 200, 3xx, 404 and 5xx
- Top URLs: the pages it requests most, so you can see where crawling is spent
- Problem URLs: URLs returning 404, 5xx and redirects to the crawler, most frequent first
- Parameter URLs: how many crawler hits go to URLs with a query string, which can point to wasted crawling
- Crawl by day: daily request counts for the crawler, so spikes and gaps stand out
What a Server Log Tells You That Search Console Does Not
Google Search Console reports on crawling and indexing from Google's side, and its Crawl Stats report summarises requests. A server log is your own record of every request that reached your server, with the exact URL, time, status code and user agent. That makes it the place to see crawlers other than Google, to see every URL a crawler requested rather than a sample, and to spot errors that happen before a page ever reaches a report. A log is also the source for questions such as "does Googlebot ever reach this section?" and "how often does it get a 404?".
Important Limits
- User agents can be faked. Anyone can send "Googlebot" in the user agent. Google documents how to verify that a request really came from Googlebot, using reverse DNS or its published IP ranges. This tool matches the user-agent text only, so treat results as indicative.
- It reads combined log format. Apache and Nginx default logs use it. Custom formats may not parse, and the tool tells you how many lines it could not read.
- Logs may be partial. A CDN or host may keep its own logs. Use logs that cover all requests for the period.
- It does not render pages or follow redirects. It counts what the log says.
How to Get Your Logs
- Shared hosting: look for "Raw access logs" or "Logs" in your hosting panel and download a recent file.
- VPS or your own server: Apache logs are usually under /var/log/apache2 and Nginx under /var/log/nginx. Copy a day or a week of lines.
- CDN or platform: ask for access logs for the period, in a format with the request line, status and user agent.
Pair this with the robots.txt tester to see whether a crawled URL is allowed, the sitemap validator to check that your sitemap lists only good URLs, and the redirect mapper if the report shows many redirects.
Log File Analyzer FAQs
What is log file analysis in SEO?
It is reading your server's access log to see which URLs search engine crawlers request, how often, and what status your server returns. It shows crawling as it actually happened.
Which log format does the tool read?
The combined log format used by Apache and Nginx by default: client IP, date, request line, status, bytes, referrer and user agent. Lines in other formats are counted as unreadable.
Does it verify that a request really came from Googlebot?
No. It matches the user-agent text. User agents can be spoofed, so verify important findings using the method Google documents, such as reverse DNS lookup.
Is my log file uploaded anywhere?
No. The file or text is processed in your browser with JavaScript. Nothing is sent to any server, including ours.
How much log data should I use?
Enough to show a pattern: a few days to a few weeks for a small site. Very large logs may be slow in a browser, so use a sample.
What should I do about 404s the crawler hits?
Decide for each one whether the URL should exist. If it moved, add a permanent redirect. If it never existed or was removed on purpose, a 404 or 410 is fine, but fix any internal or external links that point to it.
Want a technical SEO review by a person?
Contomatix does audits, on-page work and link building for businesses and agencies.