Basics for SEO on JavaScript Crawl

In the last few years, One of the more significant changes in technical SEO is the topic "JavaScript crawling", Many SEOs have a lot of confusion when managing requests from developers about JavaScript, it’s typically a platform-specific.

My, intention is to provide a basic framework for anyone working in digital marketing or product management, regardless of your current experience with JavaScript.

JavaScript can significantly improve user experience, SEO's to not only encourage developers using it, but also ensure to minimize negative business impacts as a result of search engine behaviour.

Basics Of HTML Crawling

A basic understanding of how a traditional crawling works are required before understanding JavaScript crawling.

In simple, the crawl process looks like this:

1. Bot makes a GET request to the server for a page/file.
2. Bot downloads the raw HTML file.
3. HTML is parsed by Search Engines, and content, meta info and all associated content are extracted.
4. Content is stored (indexed), evaluated, and ranked in many ways.

The main “problem” with JavaScript is that the content that appears to users, what we see on the screen, can't be found via this method. In empirical terms, when you view the source, you do not see what appears to the user.

JavaScript crawling is just the process of getting the code the user is seeing rather than what they download and parse with the raw HTML. A browser is used by Search engines to crawl instead of relying only on a download of the HTML document.

To Understand What JavaScript Is Doing

Generally, what happens when a page is requested by the browser, depends on JavaScript-rendered content. Bot/JS crawler replicates this.

1. Initial Request – Bot & Browser makes a GET request for the HTML and all its assets.
2. DOM rendering – DOM stands for Document Object Model. The DOM will be rendered by bots. This DOM can be interpreted by the bots.
3. Loading DOM – Bot triggers events, one of them being DOMContentLoaded. In simple terms, this event means the initial HTML document has been loaded & parsed and now it’s ready for JavaScript to start doing work against the page.
4. JavaScript effect – JS can make changes to the page. In layman terms, think of something like modifying the content in HTML source. It’s just like opening a page in Notes and changing the title. JS can do many things, like effecting the page to get the desired effect.
5. Load Event – Browser fires a load event when the resources have finished loading. This is a key event, which says, that the page is “done.”
6. Post-Load Events & User Events – The pages can continue to change through user-driven events such as onClick or when new content is published.