part-time-job/zhang_crawl_bio @ v1.0.0-proxy

mrh 2962b94f63 新增使用说明，新增 client 参数		9 luni în urmă
..
celery	2962b94f63 新增使用说明，新增 client 参数	9 luni în urmă
conf	ad5526ea13 备份。 celery 无法很好支持 playwright 上下文	10 luni în urmă
crawl_pages	50dcc6cd81 修改代理，使用说明，新增文件便于打包发布	9 luni în urmă
html_convert	1835feda6a 导出数据，后续测试，临时保存	9 luni în urmă
search_engine	1835feda6a 导出数据，后续测试，临时保存	9 luni în urmă
ARCHITECTURE.md	2b701092ff 删除 API 代码，准备改为 celery 的方式	10 luni în urmă
readme.md	61c7a90974 有些 resutl items 存在 cloudflare 人机验证，尝试跳过这些页面转换	10 luni în urmă

To use this system you would:

1 Start Celery worker with: celery -A worker.celery.app worker --loglevel=info --concurrency=1 2 Send tasks using:

脚本方式

from worker.celery.models import KeywordTaskModel
from worker.celery.tasks import search_keyword_task
task_data = KeywordTaskModel(keyword="Acampe carinata essential oil").model_dump()
search_keyword_task.delay(task_data)

命令行方式

G:\code\upwork\zhang_crawl_bio\download\Redis-x64-5.0.14.1\redis-server.exe
celery -A worker.celery.app flower --persistent=True --db=".\output\flower_db"
# 为不同PC启动worker时指定配置
$env:PC_NAME="w1"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w2"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w3"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w4"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w5"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w6"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h
$env:PC_NAME="w7"; celery -A worker.celery.app worker --hostname=$env:PC_NAME@%h

readme.md