Scrapy-有关刮论坛的一些问题

debugcn 发表于 Dev

Zheng

我想剪贴一个电影论坛，其结构类似于

Page 1
 Thread 1 in Page 1
 Thread 2 in Page 1
 ...
Page 2
 Thread 1 in Page 2
 Thread 2 in Page 2
 ...

页面和线程具有非常不同的html，因此我编写了xpath表达式来提取页面和线程所需的信息。

在parese()我的Spider方法中，我使用了文档中的示例来浏览每个页面：

page_links = ['page_1', 'page_2', ...]

for page_link in page_links:
  if page_link is not None:
     page_link = response.urljoin(page_link)
     yield scrapy.Request(page_link, callback=self.parse)

这样我就可以获得每个页面中每个线程的URL。

我想我下一步应该做的就是获取response每个主题，并运行一个函数来解析这些响应。但是由于我是OOP的新手，所以我对应该做的事情很困惑。

我有一个thread_links存储线程URL的列表，并且我正在尝试执行以下操作：

thread_links = ['thread_1', 'thread_2', ...]

for thread_link in thread_links:
    yield scrapy.Request(thead_link)

但是如何将这些响应传递给像这样的函数parse_thread(self, response)呢？

更新：这是我的代码：

# -*- coding: utf-8 -*-
import scrapy


class ShtSpider(scrapy.Spider):
    name = 'sht'
    allowed_domains = ['AAABBBCCC.com']
    start_urls = [
        'https://AAABBBCCC/forum-103-1.html',
                  ]
    thread_links = []

    def parse(self, response):
        temp = response.selector.xpath("//div[@class='pg']/a[@class='last']/@href").get()
        total_num_pages = int(temp.split('.')[0].split('-')[-1])

        for page_i in range(total_num_pages):
            page_link = temp.split('.')[0].rsplit('-', 1)[0] + '-' + str(page_i) + '.html'

            if page_link is not None:
                page_link = response.urljoin(page_link)
                print(page_link)
                yield scrapy.Request(page_link, callback=self.parse)

            self.thread_links.extend(response.selector.
                                     xpath("//tbody[contains(@id,'normalthread')]//td[@class='icn']//a/@href").getall())
            for thread_link in self.thread_links:
                thread_link = response.urljoin(thread_link)
                print(thread_link)
                yield scrapy.Request(url=thread_link, callback=self.parse_thread)

    def parse_thread(self, response):
        def extract_thread_data(xpath_expression):
            return response.selector.xpath(xpath_expression).getall()

        yield {
            'movie_number_and_title': extract_thread_data("//span[@id='thread_subject']/text()"),
            'movie_pics_links': extract_thread_data("//td[@class='t_f']//img/@file"),
            'magnet_link': extract_thread_data("//div[@class='blockcode']/div//li/text()"),
            'torrent_link': extract_thread_data("//p[@class='attnm']/a/@href"),
            'torrent_name': extract_thread_data("//p[@class='attnm']/a/text()"),
        }

我正在print()检查page_link和thread_link，它们似乎运行良好，所有页面和线程的URL正确显示，但是该程序仅在爬行一页后就停止了。这是来自consolo的信息：

2020-07-18 10:54:30 [scrapy.extensions.logstats] INFO: Crawled 1 pages (at 1 pages/min), scraped 0 items (at 0 items/min)
2020-07-18 10:54:30 [scrapy.core.engine] INFO: Closing spider (finished)
2020-07-18 10:54:30 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 690,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 17304,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/302': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 7, 18, 2, 54, 30, 777513),
 'log_count/DEBUG': 3,
 'log_count/INFO': 10,
 'memusage/max': 53985280,
 'memusage/startup': 48422912,
 'offsite/domains': 1,
 'offsite/filtered': 919087,
 'request_depth_max': 1,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2020, 7, 18, 2, 52, 54, 509604)}
2020-07-18 10:54:30 [scrapy.core.engine] INFO: Spider closed (finished)

更新：这是文档中的示例

def parse_page1(self, response):
    return scrapy.Request("http://www.example.com/some_page.html",
                          callback=self.parse_page)

def parse_page2(self, response):
    # this would log http://www.example.com/some_page.html
    self.logger.info("Visited %s", response.url)

如果我理解正确，它将打印它访问过的网址 http://www.example.com/some_page.html

这是我的蜘蛛，我刚刚创建了一个名为SMZDM的项目，并使用 scrapy genspider smzdm https://www.smzdm.com

# -*- coding: utf-8 -*-
import scrapy


class SmzdmSpider(scrapy.Spider):
    name = 'smzdm'
    allowed_domains = ['https://www.smzdm.com']
    start_urls = ['https://www.smzdm.com/fenlei/diannaozhengji/']

    def parse(self, response):
        return scrapy.Request("https://www.smzdm.com/fenlei/diannaozhengji/",
                              callback=self.parse_page)

    def parse_page(self, response):
        self.logger.info("Visited %s", response.url)
        print(f'Crawled {response.url}')

我已经https://www.smzdm.com/fenlei/diannaozhengji/在parse方法中进行了硬编码，只是想让它正常工作。

但是当使用运行时scrapy crawl smzdm，终端中没有任何显示。看来该parse_page方法从未执行过。

(Crawler) zheng@Macbook_Pro spiders % scrapy crawl smzdm
2020-07-29 17:10:03 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: SMZDM)
2020-07-29 17:10:03 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.6 (default, Jan  8 2020, 13:42:34) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Darwin-19.6.0-x86_64-i386-64bit
2020-07-29 17:10:03 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'SMZDM', 'NEWSPIDER_MODULE': 'SMZDM.spiders', 'SPIDER_MODULES': ['SMZDM.spiders']}
2020-07-29 17:10:03 [scrapy.extensions.telnet] INFO: Telnet Password: e3b9631aa810732d
2020-07-29 17:10:03 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2020-07-29 17:10:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-07-29 17:10:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-07-29 17:10:03 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-07-29 17:10:03 [scrapy.core.engine] INFO: Spider opened
2020-07-29 17:10:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-07-29 17:10:03 [py.warnings] WARNING: /Applications/anaconda3/envs/Crawler/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py:61: URLWarning: allowed_domains accepts only domains, not URLs. Ignoring URL entry https://www.smzdm.com in allowed_domains.
  warnings.warn(message, URLWarning)

2020-07-29 17:10:03 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-07-29 17:10:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.smzdm.com/fenlei/diannaozhengji/> (referer: None)
2020-07-29 17:10:03 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'www.smzdm.com': <GET https://www.smzdm.com/fenlei/diannaozhengji/>
2020-07-29 17:10:03 [scrapy.core.engine] INFO: Closing spider (finished)
2020-07-29 17:10:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 321,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 40270,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 7, 29, 9, 10, 3, 836061),
 'log_count/DEBUG': 2,
 'log_count/INFO': 9,
 'log_count/WARNING': 1,
 'memusage/max': 48422912,
 'memusage/startup': 48422912,
 'offsite/domains': 1,
 'offsite/filtered': 1,
 'request_depth_max': 1,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2020, 7, 29, 9, 10, 3, 381441)}
2020-07-29 17:10:03 [scrapy.core.engine] INFO: Spider closed (finished)

亚伦

完整的代码示例将帮助我指导您找到实现所需目标的最佳方法。但是听起来你走对了。

认为您只需要对parse_thread函数进行回调

编码示例

thread_links = ['thread1','thread2']

def thread_links(self,response)
    for thread_link in self.thread_links:
        yield scrapy.Request(url=thread_link,callback=self.parse_thread)

def parse_thread(self,response):
    print(response.text)

说明

在这里，我们从thread_links列表中获取链接，请注意，您必须要做self.thread_links，那是因为您正在定义thread_links列表而不是该函数。这就是所谓的类变量，需要通过self.VARIABLE在函数内部进行访问。

然后，我们将回调添加到parse_thread，再次在这里注意我们如何使用self.parse_thread。Scrapy发出请求并将响应传递给parse_thread函数。在这里，我刚刚打印了该回复。

更新的代码

由于您提供了一些代码，因此如果您检查了页面和线程链接的输出情况，我认为您可能会出错。

def parse_thread(self, response):
    def extract_thread_data(xpath_expression):
        return response.selector.xpath(xpath_expression).getall()

更改为

def parse_thread(self,response):
    yield response.xpath(xpath_expression).getall()

我不确定，因为我无法测试代码，但是嵌套函数可能会引起一些棘手的问题。

提示/澄清和编码建议

无需调用写response.selector，response.xpath就可以了。
当您从thread_links函数向parse_thread进行回调时，响应是您在thread_links中发出的刮擦请求的结果。无需将其包装在函数中。
确保您了解收益率和收益率的区别。一般来说，如果有大量数据，yield语句比return语句要有效得多。看到这里广泛的写作

更新2

您需要将这些包含在settings.py中

settings.py

USER_AGENT = 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36'


ROBOTSTXT_OBEY = False

说明

该网站是非常容易抓取的，因此您不想在该网站的robots.txt中抓取该网站。为了解决这个问题，我们设置ROBOTSTXT_OBEY = False.

除此之外，当scrapy发送HTTP请求时，您尚未定义用户代理，它可以是任何用户代理。我举了一个对我有用的例子。如果没有用户代理，它就不会检测到不是浏览器发出此类请求，并且scrapy不会抓取该URL。

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。

编辑于2021-04-2

我来说两句

0条评论

登录后参与评论

来自分类Dev

Related 相关文章

文章