Web scraping from JavaScript enabled website

Web scraping from JavaScript-enabled websites can be a bit more challenging than scraping static HTML websites. This is because the content of the website is generated dynamically by JavaScript, so the HTML source code you download may not contain the data you're interested in.

One way to scrape JavaScript-enabled websites is to use a tool such as Selenium, which allows you to automate a web browser and interact with the website as if you were a user. This allows you to execute the JavaScript code on the website and access the dynamically generated content. Here's an example of how you can use Selenium and Beautiful Soup to scrape a JavaScript-enabled website:

from bs4 import BeautifulSou
from selenium import webdriver


# Start a web driver (e.g. Chrome)
driver = webdriver.Chrome()


# Navigate to the website
driver.get('https://www.example.com')


# Wait for the JavaScript to load
driver.implicitly_wait(10)


# Get the HTML source code
html = driver.page_source


# Parse the HTML with Beautiful Soup
soup = BeautifulSoup(html, 'html.parser')


# Extract the data you're interested in
data = soup.find_all('tag_name')


# Close the web driver
driver.quit()        

Another way is to use a headless browser like Pyppeteer which allows you to interact with the website as if you are a user and it can also execute JavaScript and interact with the website as a user.

from pyppeteer import launc


async def main():
    browser = await launch()
    page = await browser.newPage()
    await page.goto('https://example.com')
    await page.waitForSelector('tag_name')
    data = await page.evaluate('''() => {
    return document.querySelectorAll('tag_name')
    }''')
    await browser.close()
    return data


results = main()        

It's important to note that scraping JavaScript-enabled websites can be more complex and may require more resources, such as processing power and memory. Additionally, some websites may have security measures in place specifically to prevent scraping with tools like Selenium or Pyppeteer, so it's important to be respectful of the terms of service and not scrape too aggressively.