-
Recent Posts
- Could AI Make Humans Extinct Within the Next Decade?
- Is Weebly Getting Out of the Website-Builder Business?
- Are You “Leaking SEO” by Linking to Other Websites?
- Wix Harmony Is Now in Google Gemini. So Why Are Users Still Complaining?
- When AI Sounds Confident but Gets It Wrong: The Mount Shasta Rescue
- When AI Agents Start Working Together, Who Is Really in Control?
- WPForms and Rank Math: Did These WordPress Plugins Create a Backdoor With Admin Access?
- AI Can Text Your Friends for You. Wait, Who Am I Talking To?
- Bill Gates Is Betting on AI to Help Healthcare – While Warning About What AI Could Do to Jobs
- Wix Says AI Website Owners Should Care About Portability. There’s Just One Problem.
- WordPress 7.1 Broke WP Rocket: The Fatal Error and the Plugin Problem
- WordPress vs. UltimateWB for an Artist Website: Which Should You Choose?
- Thinking About Switching From WordPress? What Should You Consider Before You Move?
- The Airtable Acquisition Raises a Bigger Question: Do You Really Own the Software Your Business Runs On?
- How to Use Google Search Console’s New Generative AI Data to Improve Your SEO
- How to Update the Same Website Content on Multiple Pages at Once
- Discovered vs. Crawled Not Indexed in Google Search Console: What’s the Difference and How to Fix Each
- The Small Business Guide to Auditing Google Ads and Website Costs
Categories
Search Results for: crawl
Discovered vs. Crawled Not Indexed in Google Search Console: What’s the Difference and How to Fix Each
When you log into Google Search Console (GSC) to check your site’s coverage, it can be very frustrating to find your URLs shoved into the "Not Indexed" bucket. But pay close attention to the specific reason GSC provides.
Two of the most common - and easily confused - statuses are "Discovered - currently not indexed" and "Crawled - currently not indexed."
While they might sound like minor variations of the same problem, they point to two completely different stages in
... Continue reading
Posted in Search Engine Optimization (SEO)
Tagged backlinks, crawl budget, crawled - currently not indexed, crawler, currently not indexed, discovered - currently not indexed, discovered vs crawled, domain authority, Google Search Console, Googlebot, gsc, internal linking, internal links, seo, seo missteps, seo tips, sitemap, xml sitemap
Leave a comment
How to Fix Crawl Budget Waste and “Crawled – Currently Not Indexed” in Google Search Console
If you've opened the Google Search Console (GSC) Page Indexing report recently and discovered thousands of URLs stuck in the "Crawled – currently not indexed" status, it's easy to assume something is wrong with your content.
Is the content too thin? Is Google penalizing your site? Did an algorithm update hurt your rankings?
In many cases, the answer is much less dramatic: your
... Continue reading
Posted in Search Engine Optimization (SEO)
Tagged crawl budget, crawl trap, crawled - currently not indexed, crawled not indexed, google algorithm, Google Search Console, google search engine ranking, Googlebot, gsc, robots.txt, rss feed, search engine crawl, search engine optimization, search parameter, seo, thin content, thin pages, wasted crawl budget, WordPress
Leave a comment
Why Google Deindexed 99% of My E-Commerce Site Overnight – And How to Fix “Crawled – Currently Not Indexed”
You log into Google Search Console (GSC) and see a vertical cliff. A WooCommerce (why didn't you use UltimateWB?!) store that held 5,000+ stable product URLs for months is suddenly down to just a handful of indexed pages. Your core catalog is completely gone. At least your homepage is left standing, you say to yourself.
You run the standard technical triage checklist:
- Robots.txt? Clean.
- Noindex tags? None. Everything is set to index, follow.
- Canonical tags? Correct and self-referencing.
Posted in E-commerce, Search Engine Optimization (SEO)
Tagged 301 redirect, algorithm changes, algorithmic filtering, bloat, bottlenecks, canonical tag, canonical tags, core updates, crawl priorities, crawl stats, crawled, crawled not indexed, database bottlenecks, deindexed, duplicate content, fast website, google algorithm penalties, Google Search Console, Googlebot, gsc, hpos, html, http errors, indexed, indexing, javascript, malware, noindex, noindex tag, pagespeed insights, performance bottlenecks, quality content, rendered html, robots.txt, security flags, seo, server log, site-wide quality signals, sitemap, soft 404, submit sitemap, technical penalty, technical seo, test live url, thin content, time to first byte, ttfb, unique value, url inspection tool, visibility, woocommerce, WordPress
Leave a comment
Webpage direct visits: bots, crawlers, or real visitors?
"Are webpage visits coming from a direct visit (vs a search) from bots or crawlers vs real visitors, or perhaps AI scrapers?"
Yes - a significant portion of direct webpage visits can come from bots, crawlers, and AI scraping content, not just real human visitors. But how much depends on the site, traffic level, and what kind of page you’re looking at.
Below is a clear breakdown.
1. What “Direct” Traffic Really Means
... Continue reading
Posted in AI, Ask David!, Website Traffic
Tagged ai, AI crawling, ai scrapers, analytics accuracy, bot traffic, bots, bots vs humans, content scraping, crawlers, direct traffic, fake direct traffic, fake traffic, google analytics, matomo, raw server logs, real visitors, robots, seo, SEO analytics, server logs, traffic analysis, web crawlers, website analytics, website monitoring
Leave a comment
Does Faster Website Speed Increase Googlebot Crawl Frequency?
Boosting your website’s speed can significantly enhance how often and how efficiently Google crawls your site. This ties directly into what Google refers to as your “crawl budget.”
What Is Crawl Budget?
Crawl budget is the number of pages Googlebot is willing and able to crawl on your site within a certain period. It’s determined by two key factors:
- Crawl Rate Limit: The maximum number of requests per second Googlebot will make without overwhelming your server.
- Crawl Demand:
Posted in Ask David!, Search Engine Optimization (SEO)
Tagged caching, clean code, crawl budget, crawl frequency, crawl rate limit, fast hosting, fast-loading times, faster indexing, fresh content, Googlebot, indexing, ranking factor, reliability, responsive, seo, server reliability, site optimization, site speed, ssd, technical seo, user experience, website performance, website speed
Leave a comment
What techniques do you use to improve website crawlability and indexability?
Good question! A well-designed website is only half the battle. For your website to truly thrive online, it needs to be seen by search engines and potential visitors. This is where crawlability and indexability come into play...
Conquer the Crawl: Techniques to Boost Website Crawlability and Indexability
Simply put, crawlability refers to how easily search engine bots (also known as crawlers) can navigate and access your website's content. Indexability determines whether those crawled pages are deemed worthy of inclusion in
... Continue reading
Posted in Ask David!, Search Engine Optimization (SEO)
Tagged analytics, backlinks, content creation, content is king, crawl, crawlers, engagement, fast loading times, Google Search Console, indexing, interactive, mobile friendly, organic traffic, responsive, responsive app, search engine crawl, search engine optimization, search engine optimization tips, search engine results pages, seo, serps, sitemap, sitemap generator, website speed, xml sitemap
Leave a comment
How to deal with bad bots and crawlers that waste your server resources and harm your website?
It is true that there are bad bots and crawlers that can potentially harm your website by consuming server resources, engaging in scraping, or attempting to exploit vulnerabilities. These malicious bots can negatively impact your website's performance, security, and user experience.
To address this issue, it's possible to implement measures to block or restrict the activities of bad bots and crawlers without affecting the good bots and crawlers that are used by legitimate search engines.
Here are some methods for
... Continue readingShould you block bots and crawlers from your website?
“Ask David!” question: "Is it a bad idea for SEO and search engine indexing and ranking to block bots and crawlers from accessing your website?"
Short Answer: Yes.
Blocking search engine bots and crawlers from accessing your website is generally a bad idea if your goal is to achieve good SEO rankings and search engine indexing. Here's why:
- Invisibility to Search Engines: When you block bots and crawlers, your website becomes invisible to search engines like Google. They won't be
Posted in Ask David!, Search Engine Optimization (SEO), Website Traffic
Tagged backlinks, bots, cloaking, crawlers, fast website, fast-loading times, high-quality content, invisibility, keyword ranking, meta robots, mobile optimization, organic traffic, responsive app, search engine crawl, search engine optimization, search engine ranking, seo, sitemap generator, target keywords, webpage indexing, website speed, xml sitemap
Leave a comment
How to make sure Google crawls all your webpages for indexing?
There are a few things you can do to make sure Google crawls all of your web pages for indexing:
- Create a sitemap and submit it to Google Search Console. A sitemap is a file that lists all of the pages on your website. Submitting your sitemap to Google Search Console helps Google to find and index all of your pages. This is easy to do with the UltimateWB built-in Sitemap Generator tool.
- Make sure your website is well-structured and
Posted in Ask David!, Search Engine Optimization (SEO), Website Traffic
Tagged backlinks, build backlinks, canonical tag, duplicate content, google, Google Search Console, indexing, meta data, meta description, robots.text, search engine crawl, sitemap, sitemap generator, social media, title
Leave a comment
Are You “Leaking SEO” by Linking to Other Websites?
You’ve probably heard this SEO advice before:
Don’t link to other websites. You’re just giving away your SEO.
Sometimes you’ll hear this described as “SEO leakage” - the idea that when you put an external link on your website, you’re giving some of your hard-earned SEO value to another website.
And if that other website happens to be a competitor, that can sound even worse.
So should website owners be afraid to link out?
No. Not when the link is
... Continue reading