zhang_crawl_bio/worker

prefect

prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"
prefect start

celery

To use this system you would:

1 Start Celery worker with: celery -A worker.celery.app worker --loglevel=info --concurrency=1 2 Send tasks using:

脚本方式

from worker.celery.models import KeywordTaskModel
from worker.celery.tasks import search_keyword_task
task_data = KeywordTaskModel(keyword="Acampe carinata essential oil").model_dump()
search_keyword_task.delay(task_data)

命令行方式

G:\code\upwork\zhang_crawl_bio\download\Redis-x64-5.0.14.1\redis-server.exe
celery -A worker.celery.app flower --persistent=True --db=".\output\flower_db"
# 为不同PC启动worker时指定配置
$env:PC_NAME="w1"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w2"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w3"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w4"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w5"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w6"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w7"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h