Scrapy：在scrapy 中使用SQLAlchemy 有“NameError: name 'connection' is not defined”

debugcn 发表于 Dev

他们住

在scrapy中使用SQLAlchemy有NameError，报错信息如下：

Traceback (most recent call last):
  File "e:\weibo_spider\venv\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "e:\weibo_spider\venv\lib\site-packages\scrapy\crawler.py", line 79, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
NameError: name 'connection' is not defined

这是我的Scrapy Pipeline课：

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

from .models import MyModel # my sqlalchemy model


class WeiboSpiderPipeline(object):

    def open_spider(self, spider):
        # using pymysql as the conncetor
        engine = create_engine('mysql+pymysql://root@localhost/wbspider_data')
        Session = sessionmaker(bind=engine)
        self.conn = engine.connect()
        self.session = Session(bind=connection)

    def close_spider(self, spider):
        self.conn.close()

    def process_item(self, item, spider):
        return item

我在命令行中测试模型，它可以工作，但NameError在我运行scrapy crawl myspidername命令后发生。

帮助！

奥马尔·阿斯加尔

使用这种方法。

from sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=engine)
session = Session()

# it will send the connection back to the pool of connections in the engine, but it will not close the connection instead it will make the connection idle
session.close()
# to close the engine having pool of connections will close all the idle connections in the pool
engine.dispose()()


# for directing close the connection on session.close, use this code
# Disabling pooling using NullPool:

from sqlalchemy.pool import NullPool
engine = create_engine(
      'postgresql+psycopg2://scott:tiger@localhost/test',
      poolclass=NullPool)
session.close()

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。