Scrapy。在解析结果中创建复杂结构（dict中的dict）

while1pass 发表于 Dev

while1pass

我有一些描述对象属性的Item对象

import scrapy


class FullName(scrapy.Item):
    first = scrapy.Field()
    second = scrapy.Field()
    middle = scrapy.Field()

class Physical(scrapy.Item):
    growth = scrapy.Field()
    weight = scrapy.Field()
    hair = scrapy.Field()

我有这个项目，它属于主题。作为字段，我想插入对象的Item属性

class Human(scrapy.Item):
    sex = scrapy.Field()
    age = scrapy.Field()
    physical = <...Physical Item>
    full_name = <...FullName Item>

因此，当您将数据导出到具有指定嵌套的结构时

{
age: 23,
sex: male,
full_name: {first: test, second: test, middle: test}
physical: {growth: 90, height: 190, hair: blonde},
...
}

嵌套可以达到任何深度。

猫我用Scrapy做吗？蜘蛛的结构是什么？在有关扩展项и加载程序的草率文档中，我找不到。

还是我选择了错误的工具而需要手动进行？

UPD。关于蜘蛛。

蜘蛛的结构是什么？如您所知，“物理”字段需要与通过当前URL的蜘蛛PhysicalSpider关联。怎么了请修理我。

class PhysicalSpider(scrapy.Spider):
    name = "physical"

    def parse(self, response):
         item = PhysicalItem()
         item['weight'] = response.xpath('path').extract()
         yield item

class HumanSpider(scrapy.Spider):
    name = "human"
    start_urls = [
        "url1",
        "url2",
     ]

    def parse(self, response):
         item = HumanItem()
         item['sex'] = response.xpath('path').extract()
         item['age'] = response.xpath('path')[1].extract()
         item['physical'] = PhysicalSpider(???)
         yield item

霸王龙

class Human(scrapy.Item):
    sex = scrapy.Field()
    physical = scrapy.Field()
    full_name = scrapy.Field()

class Physical(scrapy.Item):
    height = scrapy.Field() 

p = Physical()
p['height'] = 180
h = Human()
h['physical'] = p
h['sex'] = 'yes'
return h

结果：

{'physical': {'height': 180}, 'sex': 'yes'}

根据您的蜘蛛示例：

class HumanSpider(scrapy.Spider):
    name = "human"
    start_urls = [
        "url1",
     ]

    def parse(self, response):
         item = HumanItem()
         item['sex'] = response.xpath('path').extract()
         item['age'] = response.xpath('path')[1].extract()
         physical_item = Physicalitem()
         physical_item['height'] = response.xpath('path').extract()
         item['physical'] = physical_item
         yield item

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。