Beautifulsoup的Web抓取问题

debugcn 发表于 Dev

恩迪

当我打开要从中抓取信息的网址时，HTML代码将显示所有内容。但是，当我在网络上抓取其HTML代码时，它仅显示了一部分，甚至没有匹配。现在，当网站在我的浏览器中打开时，它确实具有加载屏幕，但是我不确定那是问题所在。也许他们阻止了人们抓取它？我得到的HTML：

<!DOCTYPE html>

<html lang="en">
<head>
<meta charset="utf-8"/>
<title></title>
<base href="/app"/>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="favicon.ico" rel="icon" type="image/x-icon"/>
<link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"/>
<link href="styles.css" rel="stylesheet"/></head>
<body class="cl">
<app-root>
<div class="loader-wrapper">
<div class="loader"></div>
</div>
</app-root>
<script src="runtime.js" type="text/javascript"></script><script src="polyfills.js" type="text/javascript"></script><script src="scripts.js" type="text/javascript"></script><script src="main.js" type="text/javascript"></script></body>
<script src="https://www.google.com/recaptcha/api.js"></script>
<noscript>
<meta content="0; URL=assets/javascript-warning.html" http-equiv="refresh"/>
</noscript>
</html>

我使用的代码：

from twill.commands import *
import time
import requests
from bs4 import BeautifulSoup
go('url')
time.sleep(4)
showforms()

try:
    fv("1", "username", "username")
    fv("1", "password", "*********")
    submit('0')
except:
    pass
time.sleep(2.5)

url = "url_after_login"
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')
print(soup)
#name_box = soup.find('h1', attrs={'class': 'trend-and-value'})

亚历山德拉·杜德基纳（Alexandra Dudkina）

看来，网页内容是由javascript动态生成的。您可以结合硒/靓汤解析此类网页。硒的优点是它可以在浏览器中重现用户的行为-单击按钮或链接，在输入字段中输入文本等。

这是一个简短的示例：

from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup

# define 30 seconds delay 
DELAY = 30

# define URI
url = '<<WEBSITE_URL>>'

# define options for selenium driver
chrome_options = webdriver.ChromeOptions()
# this one make browser "invisible"
# comment it out to see all actions performed be selenium
chrome_options.add_argument('--headless')

# create selenium web driver
driver = webdriver.Chrome("<PATH_TO_CHROME_DRIVER>", options=chrome_options)

# open web page
driver.get(url)

# wait for h1 element to load for 30 seconds
h1_element = WebDriverWait(driver, DELAY).until(EC.presence_of_element_located((By.CSS_SELECTOR, 'h1.trend-and-value')))

# parse web page content using bs4
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')

print(soup)

替代解决方案可能是分析javascript呈现的网页。通常，此类网页以JSON格式从后端端点检索数据，也可以由您的抓取工具调用。

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。