我正在使用scrapy 0.24从网站上抓取数据。但是,我无法从我的回调方法发出任何请求parse_summary
。
class ExampleSpider(scrapy.Spider):
name = "tfrrs"
allowed_domains = ["example.org"]
start_urls = (
'http://www.example.org/results_search.html?page=0&sport=track&title=1&go=1',
)
def __init__(self, *args, **kwargs):
super(TfrrsSpider, self).__init__(*args, **kwargs)
self.start_urls = ['http://www.example.org/results_search.html?page=0&sport=track'&title=1&go=1',]
pass
# works without issue
def parse(self, response):
races = response.xpath("//table[@width='100%']").xpath(".//a[starts-with(@href, 'http://www.tfrrs.org/results/')]/@href").extract()
callback = self.parse_trackfieldsummary
for race in races:
yield scrapy.Request(race, callback=self.parse_summary)
pass
# works without issue
def parse_summary(self, response):
baseurl = 'http://www.example.org/results/'
results = response.xpath("//div[@class='data']").xpath('.//a[@style="margin-left: 20px;"]/@href').extract()
for result in results:
print(baseurl+result) # shows that url is correct everytime
yield scrapy.Request(baseurl+result, callback=self.parse_compiled)
# is never fired or shown in terminal
def parse_compiled(self, response):
print('test')
results = response.xpath("//table[@style='width: 935px;']")
print(results)
当我故意使请求parse_summary
失败(由于域错误等)时,我可以在提示符下看到该错误,但是当我使用正确的url时,就好像我什至没有调用它。我还测试所请求的网址parse_summary
的parse
方法,在那里他们正常工作。是什么导致它们没有在parse_summary
方法中被触发但在方法中被成功触发parse method
?提前谢谢你的帮助。
在对我进行一些更改之后Spider
,我仍然得到相同的结果。但是,如果我使用一个全新的项目,它将起作用。所以我想这与我的项目设置有关。
这是我的项目设置(项目raceretrieval
名称在哪里):
BOT_NAME = 'raceretrieval'
DOWNLOAD_DELAY= 1
CONCURRENT_REQUESTS = 100
SPIDER_MODULES = ['raceretrieval.spiders']
NEWSPIDER_MODULE = 'raceretrieval.spiders'
ITEM_PIPELINES = {
'raceretrieval.pipelines.RaceValidationPipeline':1,
'raceretrieval.pipelines.RaceDistanceValidationPipeline':2,
# 'raceretrieval.pipelines.RaceUploadPipeline':9999
}
如果我注释掉两 DOWNLOAD_DELAY= 1
和CONCURRENT_REQUESTS = 100
,蜘蛛按预期工作。为什么会这样呢?我不明白他们会如何影响这一点。
我更正了一些拼写错误,并正确设置了允许的域,并且parse_summary似乎工作正常。提取Urls,并在终端中正确显示parse_compile结果。
输出的行如下:
2014-12-29 12:19:05+0100 [example] DEBUG: Crawled (200) <GET
http://www.tfrrs.org/results/36288_f.html> (referer:
http://www.tfrrs.org/results/36288.html) <200
http://www.tfrrs.org/results/36288_f.html>
[<Selector xpath="//table[@style='width: 935px;']" data=u'<table width="0" border="0" cellspacing='>, <Selector xpath="//table[@style='width: 935px;']" data=u'<table width="0" border="0" cellspacing='> .....
这是更正的代码:
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["tfrrs.org"]
start_urls = (
'http://www.tfrrs.org/results_search.html?page=0&sport=track&title=1&go=1',
)
def __init__(self, *args, **kwargs):
super(ExampleSpider, self).__init__(*args, **kwargs)
self.start_urls = ['http://www.tfrrs.org/results_search.html?page=0&sport=track&title=1&go=1',]
# works without issue
def parse(self, response):
races = response.xpath("//table[@width='100%']").xpath(".//a[starts-with(@href, 'http://www.tfrrs.org/results/')]/@href").extract()
#callback = self.parse_trackfieldsummary
for race in races:
yield scrapy.Request(race, callback=self.parse_summary)
pass
# works without issue
def parse_summary(self, response):
baseurl = 'http://www.tfrrs.org/results/'
results = response.xpath("//div[@class='data']").xpath('.//a[@style="margin-left: 20px;"]/@href').extract()
for result in results:
#print(baseurl+result) # shows that url is correct everytime
yield scrapy.Request(baseurl+result, callback=self.parse_compiled)
# is never fired or shown in terminal
def parse_compiled(self, response):
print(response)
results = response.xpath("//table[@style='width: 935px;']")
print(results)
本文收集自互联网,转载请注明来源。
如有侵权,请联系[email protected] 删除。
我来说两句