玩蛇网提供最新Python编程技术信息以及Python资源下载!
您现在的位置: 玩蛇网首页 > Python问题解答 > 正文内容

Celery 队列做定向爬虫遇到自动退出问题

我们目前在基于 Celery 队列做定向爬虫。目标网页很简单,但存储的时候要将目标页上的各种信息转换、分解为结构化数据。在这期间需要查询很多次数据库。
采用 “发出任务 -> 队列 -> 在 Celery 中执行” 的方式。目前遇到以下问题

  1. 使用 supervisor 管理 Celery 的运行。但发现每隔一段时间,启动的多个 Celery 进程就好一起退出(同时)。

    supervisor.log 如下(部分)

    2015-05-09 03:02:56,462 INFO stopped: celerya (exit status 0)
    2015-05-09 03:02:57,457 INFO stopped: celeryb (exit status 0)
    2015-05-09 03:04:38,275 INFO spawned: 'celerya' with pid 3547
    2015-05-09 03:04:48,529 INFO success: celerya entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
    2015-05-09 03:07:55,995 INFO spawned: 'celeryb' with pid 3926
    2015-05-09 03:08:06,337 INFO success: celeryb entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
    2015-05-09 03:09:01,861 INFO stopped: celerya (exit status 0)
    2015-05-09 03:11:29,792 INFO stopped: celeryb (exit status 0)
    2015-05-09 03:12:02,037 INFO spawned: 'celerya' with pid 4706
    2015-05-09 03:12:03,044 INFO spawned: 'celeryb' with pid 4710
    

    退出时 celery 的异常信息如下:

    Traceback (most recent call last):
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
        self.blueprint.start(self)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
        step.start(parent)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
        return self.obj.start()
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
        blueprint.start(self)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
        step.start(parent)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
        c.loop(*c.loop_args())
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/loops.py", line 97, in synloop
        connection.drain_events(timeout=2.0)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
        return self.transport.drain_events(self.connection, **kwargs)
      File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 840, in drain_events
        message, queue = item
    TypeError: 'NoneType' object is not iterable
    
  2. 在运行 4 小时之后,celery 进程虽然活着,但却停止工作,无 log,手动重启之后恢复工作。但 80% 的任务出现数据库异常,提示 IntegrityError: (IntegrityError) (1062, u"Duplicate entry '201173000000006801' for key 'PRIMARY'") 错误。手动查找 改主键却不存在。

  3. 在运行 2 小时左右之后,开始出现 陆续出现主键重复错误,手动查找,该数据确实存在。但实际上应该不存在。因为该条数据就是在本次任务中创建的。

附: celery 运行参数:/usr/local/opt/pyenv/versions/calvino/bin/celery -A celeryd worker -P eventlet -c 30 -n spider02

不是提示了么,TypeError: 'NoneType' object is not iterable

看报错猜测是File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/transport/virtual/init.py", line 840, in drain_events
message, queue = item

这里的item是None,至于为什么是None,该怎么处理,得根据你的应用来调整了

玩蛇网文章,转载请注明出处和文章网址:http://www.iplaypy.com/wenda/wd18747.html [复制]



小额赞助,鼓励作者写更好的教程↓↓↓

玩蛇网Python互助QQ群,欢迎加入-->: 106381465 玩蛇网Python新手群
修订日期:2017年05月12日 - 20时03分59秒 发布自玩蛇网

我要分享到:

必知PYTHON教程 Must Know PYTHON Tutorials

必知PYTHON模块 Must Know PYTHON Modules