Celery 队列做定向爬虫遇到自动退出问题

我们目前在基于 Celery 队列做定向爬虫。目标网页很简单，但存储的时候要将目标页上的各种信息转换、分解为结构化数据。在这期间需要查询很多次数据库。
采用 “发出任务 -> 队列 -> 在 Celery 中执行” 的方式。目前遇到以下问题

使用 supervisor 管理 Celery 的运行。但发现每隔一段时间，启动的多个 Celery 进程就好一起退出（同时）。

supervisor.log 如下（部分）

2015-05-09 03:02:56,462 INFO stopped: celerya (exit status 0)
2015-05-09 03:02:57,457 INFO stopped: celeryb (exit status 0)
2015-05-09 03:04:38,275 INFO spawned: 'celerya' with pid 3547
2015-05-09 03:04:48,529 INFO success: celerya entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-05-09 03:07:55,995 INFO spawned: 'celeryb' with pid 3926
2015-05-09 03:08:06,337 INFO success: celeryb entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2015-05-09 03:09:01,861 INFO stopped: celerya (exit status 0)
2015-05-09 03:11:29,792 INFO stopped: celeryb (exit status 0)
2015-05-09 03:12:02,037 INFO spawned: 'celerya' with pid 4706
2015-05-09 03:12:03,044 INFO spawned: 'celeryb' with pid 4710

退出时 celery 的异常信息如下：

Traceback (most recent call last):
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
    c.loop(*c.loop_args())
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/loops.py", line 97, in synloop
    connection.drain_events(timeout=2.0)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 840, in drain_events
    message, queue = item
TypeError: 'NoneType' object is not iterable

在运行 4 小时之后，celery 进程虽然活着，但却停止工作，无 log，手动重启之后恢复工作。但 80% 的任务出现数据库异常，提示 IntegrityError: (IntegrityError) (1062, u"Duplicate entry '201173000000006801' for key 'PRIMARY'") 错误。手动查找改主键却不存在。
在运行 2 小时左右之后，开始出现陆续出现主键重复错误，手动查找，该数据确实存在。但实际上应该不存在。因为该条数据就是在本次任务中创建的。

附： celery 运行参数：/usr/local/opt/pyenv/versions/calvino/bin/celery -A celeryd worker -P eventlet -c 30 -n spider02

不是提示了么，TypeError: 'NoneType' object is not iterable
看报错猜测是File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/transport/virtual/init.py", line 840, in drain_events
message, queue = item
这里的item是None，至于为什么是None，该怎么处理，得根据你的应用来调整了

玩蛇网文章，转载请注明出处和文章网址：https://www.iplaypy.com/wenda/wd18747.html