haha/sbnews
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

迁移目录

Browse Source
This commit is contained in:
hajimi
2026-06-12 22:27:18 +08:00
parent 7381097582
commit cd44cd6e47
92 changed files with 1839 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/public/dongqiudi-crawler/.dockerignore
-10
View File
@@ -1,10 +0,0 @@
venv/
__pycache__/
**/__pycache__/
*.pyc
.pytest_cache/
logs/*
data/locks/*
test_output.txt
tmp_*.py
tmp_*.sql
server/public/dongqiudi-crawler/.gitignore
-21
View File
@@ -1,21 +0,0 @@
__pycache__/
*.py[cod]
*.so
*.egg-info/
dist/
build/
.eggs/
*.egg
.venv/
venv/
env/
.env
logs/*.log
data/*.json
data/*.csv
.idea/
.vscode/
*.swp
*.swo
*~
tmp_*
server/public/dongqiudi-crawler/Dockerfile
-33
View File
@@ -1,33 +0,0 @@
FROM python:3.11-slim-bookworm
ENV TZ=Asia/Shanghai \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
DQD_CONFIG_PATH=/app/config/settings.yaml \
DQD_TASKS_FILE=/app/config/crawler_tasks.yaml \
DQD_LOCK_DIR=/app/data/locks
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bash \
ca-certificates \
cron \
curl \
gcc \
g++ \
tzdata \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt /app/requirements.txt
RUN python -m pip install --upgrade pip setuptools wheel \
&& python -m pip install -r /app/requirements.txt \
&& python -m playwright install --with-deps chromium
COPY . /app
RUN mkdir -p /app/logs /app/data/locks /etc/sport-era-crawler \
&& chmod +x /app/docker/entrypoint.sh
ENTRYPOINT ["/app/docker/entrypoint.sh"]
server/public/dongqiudi-crawler/README.md
-120
View File
@@ -1,120 +0,0 @@
# 懂球帝数据爬虫
专门用于采集懂球帝(dongqiudi.com)足球数据的高级爬虫框架，内置多层反爬绕过机制。
## 功能特性
- **混合爬取引擎**: API 请求 → TLS 指纹伪装 → Playwright 浏览器自动化，三级降级
- **TLS 指纹伪装**: 基于 `curl_cffi` 模拟 Chrome/Edge 真实 JA3 指纹，绕过 TLS 检测
- **浏览器指纹池**: 50+ 指纹轮换，含 UA、Canvas、WebGL、屏幕分辨率等完整指纹
- **Playwright 隐身模式**: 移除 webdriver 痕迹、模拟人类鼠标/滚动行为
- **代理管理**: 代理池健康检测、自动轮换、按成功率排序
- **智能调度**: 高斯分布随机延迟、频率控制、自动退避重试
- **数据持久化**: MySQL 异步存储，UPSERT 避免重复
## 数据采集范围
| 数据类型 | API 端点 | 说明 |
|---------|---------|------|
| 积分榜 | `/soccer/biz/data/standing` | 联赛积分排名 |
| 赛程 | `/soccer/biz/data/schedule` | 比赛日程与比分 |
| 比赛详情 | `/soccer/biz/data/match_detail` | 事件、阵容、统计 |
| 比赛菜单 | `/api/v2/config/match_menu` | 联赛/赛事分类 |
| 实时比赛 | `/soccer/biz/data/live` | 进行中的比赛 |
| 新闻资讯 | `/api/v2/article/list` | 体育新闻列表 |
## 支持联赛
中超、中甲、亚冠、英超、西甲、德甲、意甲、法甲（可在配置文件中扩展）
## 安装
```bash
# 安装 Python 依赖
pip install -r requirements.txt
# 安装 Playwright 浏览器（用于 Playwright 通道）
playwright install chromium
```
## 使用方法
```bash
# 测试连通性
python main.py test
# 初始化数据库表
python main.py init-db
# 采集所有联赛积分榜
python main.py standings
# 采集所有联赛赛程
python main.py schedule
# 采集积分榜 + 赛程
python main.py all
# 采集比赛类型菜单
python main.py menu
# 采集单个联赛 (通过 season_id)
python main.py single 26322
# 启动定时调度
python main.py cron
```
## 项目结构
```
dongqiudi-crawler/
├── config/
│ └── settings.yaml # 配置文件
├── src/
│ ├── core/ # 核心模块
│ │ ├── config.py # 配置加载
│ │ ├── logger.py # 日志
│ │ └── exceptions.py # 异常定义
│ ├── anti_detect/ # 反检测模块
│ │ ├── fingerprint.py # 浏览器指纹生成与管理
│ │ ├── tls_client.py # TLS 指纹伪装 (curl_cffi)
│ │ ├── proxy_manager.py # 代理池管理
│ │ └── stealth_browser.py # Playwright 隐身浏览器
│ ├── engine/ # 爬虫引擎
│ │ ├── api_client.py # aiohttp API 客户端
│ │ └── hybrid_engine.py # 混合引擎 (API+TLS+Playwright)
│ ├── parser/ # 数据解析
│ │ └── data_parser.py # 懂球帝数据解析器
│ ├── storage/ # 数据存储
│ │ └── database.py # MySQL 异步存储
│ └── scheduler/ # 调度器
│ ├── task_runner.py # 任务运行器
│ └── cron_scheduler.py # 定时调度
├── main.py # 主入口
├── requirements.txt # Python 依赖
└── README.md
```
## 反爬策略说明
### 1. TLS 指纹伪装
懂球帝可能通过 JA3/JA4 指纹识别爬虫。`curl_cffi` 能完整模拟 Chrome 的 TLS 握手过程。
### 2. 浏览器指纹轮换
每次请求使用不同的 User-Agent、Sec-Ch-Ua 等头部，避免被指纹追踪。
### 3. 请求频率控制
高斯分布随机延迟（2-8秒），避免固定间隔被识别为机器人。
### 4. Playwright 降级
当 API 直接请求被拦截时，自动降级到 Playwright 浏览器渲染，绕过 JS 挑战。
## 配置说明
编辑 `config/settings.yaml` 可自定义：
- 爬取模式: `api_only` / `playwright_only` / `hybrid` / `smart`
- 延迟策略、重试次数、并发数
- 代理设置
- 目标联赛列表
- 数据库和 Redis 连接信息
server/public/dongqiudi-crawler/_test_fetch.py
-35
View File
@@ -1,35 +0,0 @@
#!/usr/bin/env python3
"""调试：查看页面结构"""
from scrapling.fetchers import Fetcher
url = "https://www.tennessean.com/story/news/local/2026/04/25/tennessee-lottery-results-saturday-april-25-2"
page = Fetcher.get(url, follow_redirects=True, timeout=15, stealthy_headers=True, impersonate="chrome")
print("status:", page.status)
print()
selectors_to_test = [
"article", "#article_content", ".article-content", ".art_content",
".article_body", "#artibody", "#mp-editor",
".content-article", ".article-text", "#article-container",
".article-content-wrap", ".article", "#content",
".post-content", ".main-content", ".TRS_Editor", ".text",
"[id*='article']", "[class*='article']", "[class*='art_']",
"[class*='content']", "[id*='content']",
"[class*='detail']", "[class*='news']", "[class*='body']",
".rich-text", ".news-content", ".detail-content",
"#js_content", ".content_area", ".main-text",
]
for sel in selectors_to_test:
el = page.css(sel)
if el:
text = el.css("::text").getall()
joined = " ".join(t.strip() for t in text if t.strip())
print(f"[FOUND] {sel} -> {len(joined)}字: {joined[:200]}")
else:
print(f"[MISS] {sel}")
# 输出 HTML 前8000字符
print("\n\n--- RAW HTML 前8000字 ---")
html_text = str(page.html) if hasattr(page, 'html') else page.text
print(html_text[:8000])
server/public/dongqiudi-crawler/ai_comment_dispatch.py
-1011
View File
File diff suppressed because it is too large Load Diff
server/public/dongqiudi-crawler/article_fetcher.py
-463
View File
@@ -1,463 +0,0 @@
"""
文章正文抓取定时任务 - 独立运行
从 la_article 中查找指定分类下没有正文的文章，逐条抓取正文并回写
仅处理分类: 加密资讯(9), NBA(17), CBA(18), 彩票资讯(21)
每次最多抓取 BATCH_SIZE 条，每条之间随机间隔 DELAY 秒，避免被限制
在 main.py 中通过 article_content_fetch 命令调用
"""
import html as html_mod
import random
import re
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.parse import urlparse
import pymysql
import requests
from bs4 import BeautifulSoup
from article_publish_helper import has_publishable_article_content
from scrapling.fetchers import Fetcher
# ─── 配置 ───
TARGET_CIDS = [9, 17, 18, 21] # 需要抓正文的分类
BATCH_SIZE = 3 # 每个分类每次抓取条数
MAX_WORKERS = 4 # 并行抓取线程数
MAX_RETRY = 2 # 最大重试次数，达到后永久跳过
REQUESTS_FALLBACK_HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/137.0.0.0 Safari/537.36"
),
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"Referer": "https://www.google.com/",
}
def decode_google_news_url(url: str) -> str:
"""解码 Google News 跳转链接为真实文章 URL"""
if "news.google.com" not in url:
return url
try:
from googlenewsdecoder import new_decoderv1
decoded = new_decoderv1(url, interval=2)
if decoded and decoded.get("status") and decoded.get("decoded_url"):
return decoded["decoded_url"]
except Exception as e:
print(f" [WARN] Google News URL 解码失败: {e}")
return url
def _clean_media_url(url: str, base_url: str = "") -> str:
"""保留原始 URL，仅过滤 data: URI 并补全相对路径"""
if not url or url.startswith("data:"):
return ""
# 协议相对 URL（//domain/path）直接补 https:
if url.startswith("//"):
return "https:" + url
# 站内相对路径（/path）补全域名
if url.startswith("/") and base_url:
parsed = urlparse(base_url)
return f"{parsed.scheme}://{parsed.netloc}{url}"
return url
def _extract_html_content(el, base_url: str = "") -> str:
"""从容器元素中提取正文 HTML
规则：
- 文字前的图片/视频不保留
- 文字后只保留第一张图片和第一个视频
- 文本去重
"""
items = [] # (type, value) type: text / img / video
seen_texts = set()
blocks = el.css("p, h2, h3, h4, blockquote, li, pre, img, video")
for block in blocks:
tag_name = getattr(block, 'tag', '') or ''
if tag_name == 'img':
src = block.attrib.get("src", "") or block.attrib.get("data-src", "")
cleaned = _clean_media_url(src, base_url)
if cleaned:
items.append(("img", cleaned))
continue
if tag_name == 'video':
src = block.attrib.get("src", "") or block.attrib.get("data-src", "")
poster = block.attrib.get("poster", "")
cleaned = _clean_media_url(src, base_url)
if cleaned:
items.append(("video", cleaned))
else:
cleaned = _clean_media_url(poster, base_url)
if cleaned:
items.append(("img", cleaned))
continue
# 文本段落
raw_parts = block.css("::text").getall()
text = "".join(t for t in raw_parts).strip()
if text and len(text) > 10 and text not in seen_texts:
seen_texts.add(text)
items.append(("text", html_mod.unescape(text)))
# 找到第一段文字的位置
first_text_idx = -1
for i, (kind, _) in enumerate(items):
if kind == "text":
first_text_idx = i
break
if first_text_idx < 0:
return ""
# 丢弃文字前的所有媒体
items = items[first_text_idx:]
# 找到最后一段文字的位置
last_text_idx = 0
for i, (kind, _) in enumerate(items):
if kind == "text":
last_text_idx = i
# 文字后的媒体只保留第一张图和第一个视频
result = items[:last_text_idx + 1]
got_img = False
got_video = False
for item in items[last_text_idx + 1:]:
if item[0] == "img" and not got_img:
result.append(item)
got_img = True
elif item[0] == "video" and not got_video:
result.append(item)
got_video = True
# 组装 HTML
parts = []
for kind, value in result:
if kind == "text":
parts.append(f"<p>{value}</p>")
elif kind == "img":
parts.append(f'<img src="{value}" />')
elif kind == "video":
parts.append(f'<video src="{value}" controls></video>')
return "".join(parts)
def _extract_html_content_bs4(el, base_url: str = "") -> str:
"""从 BeautifulSoup 容器中提取正文 HTML，规则与 scrapling 版本保持一致"""
items = []
seen_texts = set()
for block in el.select("p, h2, h3, h4, blockquote, li, pre, img, video"):
tag_name = (block.name or "").lower()
if tag_name == "img":
src = block.get("src", "") or block.get("data-src", "")
cleaned = _clean_media_url(src, base_url)
if cleaned:
items.append(("img", cleaned))
continue
if tag_name == "video":
src = block.get("src", "") or block.get("data-src", "")
poster = block.get("poster", "")
cleaned = _clean_media_url(src, base_url)
if cleaned:
items.append(("video", cleaned))
else:
cleaned = _clean_media_url(poster, base_url)
if cleaned:
items.append(("img", cleaned))
continue
text = block.get_text("", strip=True)
if text and len(text) > 10 and text not in seen_texts:
seen_texts.add(text)
items.append(("text", html_mod.unescape(text)))
first_text_idx = -1
for i, (kind, _) in enumerate(items):
if kind == "text":
first_text_idx = i
break
if first_text_idx < 0:
return ""
items = items[first_text_idx:]
last_text_idx = 0
for i, (kind, _) in enumerate(items):
if kind == "text":
last_text_idx = i
result = items[:last_text_idx + 1]
got_img = False
got_video = False
for item in items[last_text_idx + 1:]:
if item[0] == "img" and not got_img:
result.append(item)
got_img = True
elif item[0] == "video" and not got_video:
result.append(item)
got_video = True
parts = []
for kind, value in result:
if kind == "text":
parts.append(f"<p>{value}</p>")
elif kind == "img":
parts.append(f'<img src="{value}" />')
elif kind == "video":
parts.append(f'<video src="{value}" controls></video>')
return "".join(parts)
def _fetch_article_content_via_requests(url: str, timeout: int = 15) -> str:
"""requests 回退抓取，解决部分站点在 scrapling 下稳定返回 403 的问题"""
try:
resp = requests.get(url, timeout=timeout, headers=REQUESTS_FALLBACK_HEADERS)
except Exception as e:
print(f" [WARN] requests 回退抓取失败: {e}")
return ""
if resp.status_code != 200:
print(f" [WARN] requests 回退 HTTP {resp.status_code}")
return f"__HTTP_{resp.status_code}__"
soup = BeautifulSoup(resp.text, "html.parser")
result = _try_extract_bs4(soup, url)
if result:
print(" (requests 回退命中)")
return result
result = _fallback_p_tags_bs4(soup)
if result:
print(" (requests 回退p标签提取)")
return result
print(" [WARN] requests 回退提取未命中")
return ""
def fetch_article_content(url: str, timeout: int = 15) -> str:
"""抓取单篇文章正文，返回 HTML 格式（含 <p> 和 <img>）"""
# 先解码 Google News 链接
real_url = decode_google_news_url(url)
if real_url != url:
print(f" 真实URL: {real_url[:100]}")
try:
page = Fetcher.get(
real_url,
follow_redirects=True,
timeout=timeout,
stealthy_headers=True,
impersonate="chrome",
)
if page.status != 200:
print(f" [WARN] HTTP {page.status}")
if page.status == 403:
fallback = _fetch_article_content_via_requests(real_url, timeout=timeout)
return fallback or "__HTTP_403__"
return f"__HTTP_{page.status}__"
result = _try_extract(page, real_url)
if result:
return result
# 回退：全页面 <p> 标签
result = _fallback_p_tags(page, real_url)
if result:
print(f" (回退p标签提取)")
return result
print(f" [WARN] 所有提取方式均未命中")
return ""
except Exception as e:
print(f" [WARN] 抓取失败 {real_url[:80]}: {e}")
fallback = _fetch_article_content_via_requests(real_url, timeout=timeout)
return fallback or ""
# 内容选择器列表
_CONTENT_SELECTORS = [
"article", # 标准 article 标签
"#article_content", # 新浪移动端
".article-content", # 通用
".art_content", # 新浪 sports.sina.cn
".article_body", # 新浪财经
"#artibody", # 新浪PC
"#mp-editor", # 搜狐号正文
"[data-role='original-title'] ~ div", # 搜狐文章体
".content-article", # 搜狐
".article-text", # 搜狐移动端
"#article-container", # 搜狐 m.sohu.com
".article-content-wrap", # 搜狐PC
".article", # 通用
"#content", # 通用
".post-content", # 通用博客
".main-content", # 通用
".TRS_Editor", # 网易/政府站
".text", # 腾讯新闻
".story-body", # USA Today / Gannett 系
".gnt_ar_b", # Gannett (tennessean 等)
"[class*='detail']", # 详情页通用
"[class*='news-body']", # 新闻正文
"[class*='rich-text']", # 富文本
"[class*='article']", # 模糊匹配含article的class
"[class*='content']", # 模糊匹配含content的class
]
def _try_extract(page, base_url: str) -> str:
"""用选择器列表尝试提取正文"""
for selector in _CONTENT_SELECTORS:
el = page.css(selector)
if not el:
continue
result = _extract_html_content(el, base_url=base_url)
text_len = len(re.sub(r"<[^>]+>", "", result))
if text_len > 50:
return result
return ""
def _try_extract_bs4(soup: BeautifulSoup, base_url: str) -> str:
"""BeautifulSoup 版本的正文提取"""
for selector in _CONTENT_SELECTORS:
try:
elements = soup.select(selector)
except Exception:
continue
for el in elements:
result = _extract_html_content_bs4(el, base_url=base_url)
text_len = len(re.sub(r"<[^>]+>", "", result))
if text_len > 50:
return result
return ""
def _fallback_p_tags(page, base_url: str) -> str:
"""回退：提取全页面所有 <p> 标签"""
blocks = page.css("p")
if not blocks:
return ""
seen = set()
parts = []
for block in blocks:
raw = block.css("::text").getall()
text = "".join(t for t in raw).strip()
if text and len(text) > 15 and text not in seen:
seen.add(text)
parts.append(f"<p>{html_mod.unescape(text)}</p>")
content = "".join(parts)
if len(re.sub(r"<[^>]+>", "", content)) > 50:
return content
return ""
def _fallback_p_tags_bs4(soup: BeautifulSoup) -> str:
"""BeautifulSoup 版本的全页 <p> 标签回退提取"""
blocks = soup.select("p")
if not blocks:
return ""
seen = set()
parts = []
for block in blocks:
text = block.get_text("", strip=True)
if text and len(text) > 15 and text not in seen:
seen.add(text)
parts.append(f"<p>{html_mod.unescape(text)}</p>")
content = "".join(parts)
if len(re.sub(r"<[^>]+>", "", content)) > 50:
return content
return ""
def run(db_config: dict):
"""主入口：查询无正文文章 → 逐条抓取 → 回写数据库"""
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
# 每个分类各取 BATCH_SIZE 条，避免某个分类独占名额
rows = []
with conn.cursor() as cur:
for cid in TARGET_CIDS:
cur.execute(
"""SELECT id, title, source_url, content, is_show FROM la_article
WHERE cid = %s
AND ((content IS NULL OR content = '' OR CHAR_LENGTH(content) < 50) OR is_show = 0)
AND content_retry < %s
AND source_url IS NOT NULL AND source_url != ''
ORDER BY content_retry ASC, id DESC
LIMIT %s""",
(cid, MAX_RETRY, BATCH_SIZE * 12)
)
candidates = cur.fetchall()
selected = []
for row in candidates:
if has_publishable_article_content(row.get("content", "")):
continue
selected.append(row)
if len(selected) >= BATCH_SIZE:
break
rows.extend(selected)
if not rows:
print("没有需要抓取正文的文章")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "没有需要抓取正文的文章"}
print(f"待抓取正文: {len(rows)} 条")
# 并行抓取内容
def _fetch_one(row):
article_id = row["id"]
title = row["title"][:40]
url = row["source_url"]
print(f" [{article_id}] {title}...")
content = fetch_article_content(url)
return article_id, title, content
success = 0
fail = 0
skip = 0
futures = {}
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
for row in rows:
fut = executor.submit(_fetch_one, row)
futures[fut] = row["id"]
for fut in as_completed(futures):
article_id, title, content = fut.result()
if content and not content.startswith("__") and len(content) > 50:
with conn.cursor() as cur:
cur.execute(
"UPDATE la_article SET content = %s, is_show = 1, content_retry = 0, update_time = %s WHERE id = %s",
(content, int(time.time()), article_id)
)
conn.commit()
success += 1
print(f" ✓ [{article_id}] 成功 ({len(content)}字)")
else:
# 失败，content_retry +1
with conn.cursor() as cur:
cur.execute(
"UPDATE la_article SET content_retry = content_retry + 1 WHERE id = %s",
(article_id,)
)
conn.commit()
fail += 1
reason = content if content and content.startswith("__") else "解析失败"
print(f" ✗ [{article_id}] {reason}（retry+1）")
summary = f"抓取完成: 成功 {success}, 失败 {fail}, 共 {len(rows)} 条"
print(f"\n{summary}")
return {"success": True, "candidate_count": len(rows), "saved_count": success, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/article_publish_helper.py
-27
View File
@@ -1,27 +0,0 @@
import re
_BLOCK_TAG_PATTERN = re.compile(r"<(p|div|article|section|img|video|h1|h2|h3|h4|blockquote|li)\b", re.I)
_HTML_TAG_PATTERN = re.compile(r"<[^>]+>")
def has_publishable_article_content(content: str) -> bool:
"""判断资讯正文是否达到可发布标准。"""
raw = (content or "").strip()
if not raw:
return False
plain = _HTML_TAG_PATTERN.sub("", raw).strip()
if len(plain) < 50:
return False
# 抓取器和富文本编辑器通常会输出结构化 HTML；纯摘要文本即使较长也先保守视为未完成正文。
if _BLOCK_TAG_PATTERN.search(raw):
return True
return len(plain) >= 200
def resolve_crawled_article_is_show(content: str) -> int:
"""根据正文内容决定采集文章的默认状态：1=发布，0=稿子。"""
return 1 if has_publishable_article_content(content) else 0
server/public/dongqiudi-crawler/articledemo.js
-12591
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/articledetaildemo.js
-5118
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/cba_news.py
-285
View File
@@ -1,285 +0,0 @@
"""
CBA新闻采集模块 - 从 Google News RSS 获取CBA新闻并写入 la_article 表
在 main.py 中通过 cba_news 命令调用
数据源（RSS）:
1. Google News 中文简体 - CBA 篮球（主）
2. Google News 中文简体 - CBA 联赛（补充）
重复判断: 以 source_url 为唯一键，重复则更新标题/摘要
"""
import json
import re
import time
import xml.etree.ElementTree as ET
from datetime import datetime
from email.utils import parsedate_to_datetime
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
# ─── 配置 ───
ARTICLE_CID = 18 # la_article_cate 中"CBA"的 id
RSS_FEEDS = [
{
"name": "CBA篮球(中文)",
"url": "https://news.google.com/rss/search?q=CBA+%E7%AF%AE%E7%90%83&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
{
"name": "CBA联赛(中文)",
"url": "https://news.google.com/rss/search?q=CBA+%E8%81%94%E8%B5%9B&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
]
NS = {
"content": "http://purl.org/rss/1.0/modules/content/",
"media": "http://search.yahoo.com/mrss/",
"dc": "http://purl.org/dc/elements/1.1/",
}
def strip_html(text: str) -> str:
import html as html_mod
text = re.sub(r"<[^>]+>", "", text or "")
text = html_mod.unescape(text)
text = html_mod.unescape(text)
text = text.replace("\xa0", " ")
return text.strip()
def parse_rfc2822(date_str: str) -> tuple[int, str]:
"""解析 RSS pubDate (RFC 2822)，返回 (unix时间戳, 'Y-m-d H:i:s'格式字符串)"""
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = parsedate_to_datetime(date_str)
return int(dt.timestamp()), dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
# ─── RSS 采集 ───
def fetch_rss(session: requests.Session, feed: dict) -> list[dict]:
"""从单个 RSS 源拉取新闻，返回标准化列表"""
articles = []
url = feed["url"]
name = feed["name"]
try:
resp = session.get(url, timeout=30)
except Exception as e:
print(f" [WARN] {name} 请求失败: {e}")
return articles
if resp.status_code != 200:
print(f" [WARN] {name} HTTP {resp.status_code}")
return articles
try:
root = ET.fromstring(resp.content)
except ET.ParseError as e:
print(f" [WARN] {name} XML 解析失败: {e}")
return articles
items = root.findall(".//item")
for item in items:
title = (item.findtext("title", "") or "").strip()
link = (item.findtext("link", "") or "").strip()
if not title or not link:
continue
pub_date = item.findtext("pubDate", "")
desc_raw = item.findtext("description", "") or ""
desc = strip_html(desc_raw)[:255]
# Google News 标题格式: "标题 - 来源名"
author = feed.get("author_default", name)
if " - " in title:
parts = title.rsplit(" - ", 1)
if len(parts) == 2:
author = parts[1].strip()
title = parts[0].strip()
# 尝试提取图片
image = ""
media_el = item.find("media:content", NS)
if media_el is not None:
image = media_el.get("url", "")
if not image:
media_thumb = item.find("media:thumbnail", NS)
if media_thumb is not None:
image = media_thumb.get("url", "")
if not image:
img_match = re.search(r'<img[^>]+src=["\']([^"\']+)["\']', desc_raw)
if img_match:
image = img_match.group(1)
pub_ts, pub_fmt = parse_rfc2822(pub_date)
articles.append({
"title": title[:255],
"desc": desc,
"content": strip_html(desc_raw),
"source_url": link,
"image": image,
"author": author[:64],
"published_ts": pub_ts,
"published_at": pub_fmt,
"category": "CBA",
"ext": {
"source": "google_news",
"feed_name": name,
"feed_url": url,
},
})
return articles
# ─── 数据库同步 ───
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
"""同步到 la_article 表，返回 (总数, 新增数, 更新数)"""
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
if not urls:
return 0, 0, 0
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
# 按标题去重：同分类下相同标题的文章不再插入
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
if not source_url or not article.get("title"):
continue
create_ts = article.get("published_ts") or now
ext_json = json.dumps(article.get("ext", {}), ensure_ascii=False) if article.get("ext") else None
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title = %s, `desc` = %s, image = %s, author = %s,
category = %s, ext = %s, is_show = %s, update_time = %s
WHERE id = %s""",
(
article["title"][:255],
(article.get("desc") or "")[:255],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("category") or "")[:128],
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
# 标题去重：相同标题跳过
if article["title"][:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
article["title"][:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("published_at") or "")[:64],
(article.get("category") or "")[:128],
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
create_ts,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
existing_titles.add(article["title"][:255])
new_count += 1
return len(articles), new_count, update_count
# ─── 主入口 ───
def run(db_config: dict):
"""主入口，db_config 从 dongqiudi-crawler 配置中传入"""
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/xml, text/xml, application/rss+xml, */*",
})
all_articles = []
for feed in RSS_FEEDS:
print(f"=== {feed['name']} 采集 ===")
articles = fetch_rss(session, feed)
print(f" {feed['name']} 返回 {len(articles)} 条")
all_articles.extend(articles)
# 按 source_url 去重
seen = set()
deduped = []
for a in all_articles:
url = a.get("source_url", "")
if url and url not in seen:
seen.add(url)
deduped.append(a)
all_articles = deduped
print(f"去重后共 {len(all_articles)} 条")
if not all_articles:
print("所有数据源均无数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "所有数据源均无数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, all_articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/config/crawler_tasks.yaml
-156
View File
@@ -1,156 +0,0 @@
tasks:
- key: video
name: 视频列表采集
action: video
cron: "54 * * * *"
active: true
- key: lottery_draw
name: 彩种开奖数据采集
action: lottery_draw
cron: "*/2 * * * *"
active: true
- key: match_finish
name: 超时比赛收尾
action: match_finish
cron: "*/1 * * * *"
active: true
- key: error_report
name: 爬虫错误报告邮件
action: error_report
cron: "*/1 * * * *"
active: true
- key: csl_match
name: 中超赛程页面采集
action: csl_match
cron: "0 */1 * * *"
active: true
- key: nba_match
name: NBA赛程页面采集
action: nba_match
cron: "*/1 * * * *"
active: true
- key: cba_match
name: CBA赛程页面采集
action: cba_match
cron: "0 */1 * * *"
active: true
- key: epl_match
name: 英超赛程页面采集
action: epl_match
cron: "0 */1 * * *"
active: true
- key: bundesliga_match
name: 德甲赛程页面采集
action: bundesliga_match
cron: "0 */1 * * *"
active: true
- key: laliga_match
name: 西甲赛程页面采集
action: laliga_match
cron: "0 */1 * * *"
active: true
- key: seriea_match
name: 意甲赛程页面采集
action: seriea_match
cron: "0 */1 * * *"
active: true
- key: ligue1_match
name: 法甲赛程页面采集
action: ligue1_match
cron: "0 */1 * * *"
active: true
- key: ucl_match
name: 欧冠赛程页面采集
action: ucl_match
cron: "0 */1 * * *"
active: true
- key: uel_match
name: 欧联赛程页面采集
action: uel_match
cron: "0 */1 * * *"
active: true
- key: tennis_match
name: 网球赛程页面采集
action: tennis_match
cron: "0 */1 * * *"
active: true
- key: esports_match
name: 电竞赛程页面采集
action: esports_match
cron: "0 */1 * * *"
active: true
- key: sports_match
name: 体坛赛程页面采集
action: sports_match
cron: "0 */1 * * *"
active: true
- key: truth_social
name: Truth Social帖子采集+同步
action: truth_social
cron: "0 */12 * * *"
active: true
- key: league_news
name: 中超资讯采集
action: league_news
cron: "*/30 * * * *"
active: true
- key: article_content
name: 文章内容补全
action: article_content
cron: "*/1 * * * *"
active: true
- key: live_detail
name: 实时比赛详情更新
action: live_detail
cron: "*/1 * * * *"
active: true
- key: crypto_news
name: 加密货币新闻采集
action: crypto_news
cron: "0 */1 * * *"
active: true
- key: article_content_fetch
name: 资讯正文抓取
action: article_content_fetch
cron: "*/1 * * * *"
active: true
- key: dqd_worldcup
name: 懂球帝世界杯采集
action: dqd_worldcup
cron: "*/10 * * * *"
active: true
- key: fifa_worldcup_news
name: FIFA世界杯资讯
action: fifa_worldcup_news
cron: "*/10 * * * *"
active: true
- key: lottery_community
name: 49宝典新旧澳六合彩社区采集
action: lottery_community
cron: "*/10 * * * *"
active: true
- key: taiwan_lottery_news
name: 台湾彩券资讯采集
action: taiwan_lottery_news
cron: "15 */2 * * *"
active: true
- key: hkjc_lottery_news
name: 香港赛马会资讯采集
action: hkjc_lottery_news
cron: "45 */2 * * *"
active: true
- key: cba_news
name: CBA资讯采集
action: cba_news
cron: "20 * * * *"
active: true
- key: nba_news
name: NBA资讯采集
action: nba_news
cron: "10 * * * *"
active: true
- key: ai_comment_dispatch
name: Docker AI评论投放
action: ai_comment_dispatch
cron: "*/10 * * * *"
active: true
server/public/dongqiudi-crawler/config/settings.yaml
-183
View File
@@ -1,183 +0,0 @@
app:
name: "dongqiudi-crawler"
version: "1.0.0"
env: "development"
# 数据库配置
database:
host: "127.0.0.1"
port: 3300
database: "sbnews"
username: "sbnews"
password: "72m931X6K5eeWMpW"
charset: "utf8mb4"
prefix: "la_"
pool_size: 10
# Redis 配置
redis:
host: "127.0.0.1"
port: 6377
password: "hajiminanbeilvdou"
db: 4
# 爬虫引擎配置
crawler:
mode: "hybrid" # api_only / playwright_only / hybrid / smart
max_concurrent: 3
request_timeout: 30
max_retries: 3
retry_backoff: 2.0
# 延迟策略 (高斯分布)
delay:
base: 3.0
min: 2.0
max: 8.0
jitter: true # 随机抖动
# 会话管理
session:
rotate_interval: 300 # 秒，每5分钟轮换会话
max_requests_per_session: 50
# 反检测配置
anti_detect:
# 指纹池
fingerprint:
pool_size: 50
min_success_rate: 0.6
rotation_interval: 300 # 秒
cooldown: 60 # 使用后冷却时间
# TLS 指纹伪装 (curl_cffi)
tls:
enabled: true
impersonate: "chrome124" # 模拟 Chrome 124 的 TLS 指纹
# Playwright 反检测
playwright:
browser: "chromium"
headless: true
stealth: true
viewport:
width: 1920
height: 1080
anti_features:
remove_webdriver: true
fake_plugins: true
fake_languages: true
fake_chrome_runtime: true
disable_webrtc: true
random_viewport: true
human_mouse: true
random_scroll: true
# 代理配置
proxy:
enabled: false
rotate: true
providers: [] # 代理源列表
test_url: "https://httpbin.org/ip"
max_failures: 3
# 懂球帝 API 端点
dongqiudi:
base_url: "https://sport-data.dongqiudi.com"
web_url: "https://www.dongqiudi.com"
api:
standings: "/soccer/biz/data/standing"
schedule: "/soccer/biz/data/schedule"
match_detail: "/soccer/biz/data/match_detail"
match_menu: "/api/v2/config/match_menu"
team_info: "/soccer/biz/data/team_info"
player_stats: "/soccer/biz/data/player_stats"
live_matches: "/soccer/biz/data/live"
news: "/api/v2/article/list"
# 默认请求参数
default_params:
app: "dqd"
platform: "www"
version: "0"
language: "zh-cn"
# 采集目标联赛
targets:
- league: "中超"
league_code: "CSL"
season_id: 26322
priority: 1
active: true
- league: "中甲"
league_code: "CL1"
season_id: 26323
priority: 2
active: true
- league: "亚冠"
league_code: "ACL"
season_id: 26324
priority: 3
active: true
- league: "英超"
league_code: "EPL"
season_id: 24646
priority: 10
active: true
- league: "西甲"
league_code: "LAL"
season_id: 24651
priority: 11
active: true
- league: "德甲"
league_code: "BUN"
season_id: 24648
priority: 12
active: true
- league: "意甲"
league_code: "SA"
season_id: 24596
priority: 13
active: true
- league: "法甲"
league_code: "L1"
season_id: 24652
priority: 14
active: true
# 六合彩开奖数据源
lottery:
sources:
- name: "澳门六合彩"
code: "mc6"
category_id: 2
url: "https://4.194.133.188:8158/chajian/xam/result.txt"
- name: "香港六合彩"
code: "hk6"
category_id: 1
url: "https://4.194.133.188:8158/chajian/xg/result.txt"
# 调度器配置
scheduler:
standings_cron: "0 2 * * *" # 每天凌晨2点
schedule_cron: "0 */6 * * *" # 每6小时
live_cron: "*/5 * * * *" # 每5分钟（比赛日）
news_cron: "*/30 * * * *" # 每30分钟
content_cron: "*/10 * * * *" # 每10分钟补全文章详情
video_cron: "*/30 * * * *" # 每30分钟采集视频列表
# 日志配置
logging:
level: "INFO"
file: "logs/crawler.log"
max_size_mb: 50
backup_count: 5
format: "{time:YYYY-MM-DD HH:mm:ss} | {level:<8} | {name}:{function}:{line} - {message}"
# 告警配置
alert:
dispatch_enabled: false
wecom_webhook: ""
cooldown_seconds: 1800
request_timeout: 10
server/public/dongqiudi-crawler/crypto_news.py
-287
View File
@@ -1,287 +0,0 @@
"""
加密货币新闻采集模块 - 从多个 RSS 源获取新闻并写入 la_article 表
在 main.py 中通过 crypto_news 命令调用
数据源（RSS）:
1. Google News 中文简体 - 加密货币（主）
2. Google News 中文简体 - 比特币/以太坊/区块链（补充）
重复判断: 以 source_url 为唯一键，重复则更新标题/摘要
"""
import json
import re
import time
import xml.etree.ElementTree as ET
from datetime import datetime
from email.utils import parsedate_to_datetime
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
# ─── 配置 ───
ARTICLE_CID = 9 # la_article_cate 中"加密资讯"的 id
RSS_FEEDS = [
{
"name": "加密货币(中文)",
"url": "https://news.google.com/rss/search?q=%E5%8A%A0%E5%AF%86%E8%B4%A7%E5%B8%81&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
{
"name": "比特币/以太坊/区块链(中文)",
"url": "https://news.google.com/rss/search?q=%E6%AF%94%E7%89%B9%E5%B8%81+OR+%E4%BB%A5%E5%A4%AA%E5%9D%8A+OR+%E5%8C%BA%E5%9D%97%E9%93%BE&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
]
NS = {
"content": "http://purl.org/rss/1.0/modules/content/",
"media": "http://search.yahoo.com/mrss/",
"dc": "http://purl.org/dc/elements/1.1/",
}
def strip_html(text: str) -> str:
import html as html_mod
text = re.sub(r"<[^>]+>", "", text or "")
text = html_mod.unescape(text)
text = html_mod.unescape(text)
text = text.replace("\xa0", " ")
return text.strip()
def parse_rfc2822(date_str: str) -> tuple[int, str]:
"""解析 RSS pubDate (RFC 2822)，返回 (unix时间戳, 'Y-m-d H:i:s'格式字符串)"""
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = parsedate_to_datetime(date_str)
return int(dt.timestamp()), dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
# ─── RSS 采集 ───
def fetch_rss(session: requests.Session, feed: dict) -> list[dict]:
"""从单个 RSS 源拉取新闻，返回标准化列表"""
articles = []
url = feed["url"]
name = feed["name"]
try:
resp = session.get(url, timeout=30)
except Exception as e:
print(f" [WARN] {name} 请求失败: {e}")
return articles
if resp.status_code != 200:
print(f" [WARN] {name} HTTP {resp.status_code}")
return articles
try:
root = ET.fromstring(resp.content)
except ET.ParseError as e:
print(f" [WARN] {name} XML 解析失败: {e}")
return articles
items = root.findall(".//item")
for item in items:
title = (item.findtext("title", "") or "").strip()
link = (item.findtext("link", "") or "").strip()
if not title or not link:
continue
pub_date = item.findtext("pubDate", "")
desc_raw = item.findtext("description", "") or ""
desc = strip_html(desc_raw)[:255]
author = item.findtext("dc:creator", "", NS) or feed.get("author_default", name)
categories = [c.text for c in item.findall("category") if c.text]
# Google News 标题格式: "标题 - 来源名"
if " - " in title:
parts = title.rsplit(" - ", 1)
if len(parts) == 2:
if not author or author == name:
author = parts[1].strip()
title = parts[0].strip()
# 尝试提取图片
image = ""
media_el = item.find("media:content", NS)
if media_el is not None:
image = media_el.get("url", "")
if not image:
media_thumb = item.find("media:thumbnail", NS)
if media_thumb is not None:
image = media_thumb.get("url", "")
if not image:
img_match = re.search(r'<img[^>]+src=["\']([^"\']+)["\']', desc_raw)
if img_match:
image = img_match.group(1)
pub_ts, pub_fmt = parse_rfc2822(pub_date)
articles.append({
"title": title,
"desc": desc,
"content": strip_html(desc_raw),
"source_url": link,
"image": image,
"author": author.strip()[:64],
"published_ts": pub_ts,
"published_at": pub_fmt,
"category": ",".join(categories[:5]) if categories else "crypto",
"ext": {
"source": name.lower(),
"categories": categories,
"feed_url": url,
},
})
return articles
# ─── 数据库同步 ───
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
"""同步到 la_article 表，返回 (总数, 新增数, 更新数)"""
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
if not urls:
return 0, 0, 0
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
# 按标题去重：同分类下相同标题的文章不再插入
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
if not source_url or not article.get("title"):
continue
create_ts = article.get("published_ts") or now
ext_json = json.dumps(article.get("ext", {}), ensure_ascii=False) if article.get("ext") else None
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title = %s, `desc` = %s, image = %s, author = %s,
category = %s, ext = %s, is_show = %s, update_time = %s
WHERE id = %s""",
(
article["title"][:255],
(article.get("desc") or "")[:255],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("category") or "")[:128],
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
# 标题去重：相同标题跳过
if article["title"][:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
article["title"][:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("published_at") or "")[:64],
(article.get("category") or "")[:128],
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
create_ts,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
existing_titles.add(article["title"][:255])
new_count += 1
return len(articles), new_count, update_count
# ─── 主入口 ───
def run(db_config: dict):
"""主入口，db_config 从 dongqiudi-crawler 配置中传入"""
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/xml, text/xml, application/rss+xml, */*",
})
all_articles = []
for feed in RSS_FEEDS:
print(f"=== {feed['name']} 采集 ===")
articles = fetch_rss(session, feed)
print(f" {feed['name']} 返回 {len(articles)} 条")
all_articles.extend(articles)
# 按 source_url 去重
seen = set()
deduped = []
for a in all_articles:
url = a.get("source_url", "")
if url and url not in seen:
seen.add(url)
deduped.append(a)
all_articles = deduped
print(f"去重后共 {len(all_articles)} 条")
if not all_articles:
print("所有数据源均无数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "所有数据源均无数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, all_articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/data/49宝典-首頁.html
-5679
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/49宝典-首頁_files/_@astro-renderers.DFfZu430.css
-1
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/49宝典-首頁_files/c.js.下载
-1
View File
@@ -1 +0,0 @@
!function(){var _="http:"==function(){for(var _=document.getElementsByTagName("script"),t=0,e=_.length;t<e;t++){var n,i=_[t];if(i.src&&(n=/^(https?:)\/\/[\w\.\-]+\.cnzz\.com\//i.exec(i.src)))return n[1]}return window.location.protocol}()?"http:":"https:",t=encodeURIComponent,e="1281444702",n="",i="",o="z6.cnzz.com",c="1",r="text",a="z",s="&#31449;&#38271;&#32479;&#35745;",p=window["_CNZZDbridge_"+e].bobject,h=_+"//online.cnzz.com/o.js",f=[];if(f.push("id="+e),f.push("h="+o),f.push("on="+t(i)),f.push("s="+t(n)),h+="?"+f.join("&"),c)if(""!==i)p.createScriptIcon(h,"utf-8");else{var w,z;if(z="z"==a?"https://www.cnzz.com/stat/website.php?web_id="+e:"https://quanjing.cnzz.com","pic"===r)w="<a href='"+z+"' target=_blank title='"+s+"'><img border=0 hspace=0 vspace=0 src='"+(_+"//icon.cnzz.com/img/"+n+".gif")+"'></a>";else w="<a href='"+z+"' target=_blank title='"+s+"'>"+s+"</a>";p.createIcon([w])}}();
server/public/dongqiudi-crawler/data/49宝典-首頁_files/discover.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.7 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/home-active.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.0 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/index.CM6bLKGF.css
-1
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/49宝典-首頁_files/index.Dj3az3q3.css
-1
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/49宝典-首頁_files/index.aHasUa7_.css
-1
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/49宝典-首頁_files/market.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 3.4 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/my.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.0 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/no_content.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 25 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/treasure.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 KiB

server/public/dongqiudi-crawler/data/49宝典-首頁_files/z.js.下载
-1
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/data.html
-27
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/体坛页面.html
View File
server/public/dongqiudi-crawler/data/懂球帝足球获取.html
-12117
View File
File diff suppressed because it is too large Load Diff
server/public/dongqiudi-crawler/data/比赛详情示例页面.html
-1446
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/data/资讯详情.html
-616
View File
File diff suppressed because one or more lines are too long
server/public/dongqiudi-crawler/docker/entrypoint.sh
-38
View File
@@ -1,38 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
export TZ="${TZ:-Asia/Shanghai}"
TASKS_FILE="${DQD_TASKS_FILE:-/app/config/crawler_tasks.yaml}"
PYTHON_BIN="${PYTHON_BIN:-/usr/local/bin/python}"
mkdir -p /app/logs /app/data/locks /etc/sport-era-crawler
if [ "$#" -gt 0 ]; then
exec "$@"
fi
python - <<'PY'
import os
import shlex
from pathlib import Path
env_file = Path("/etc/sport-era-crawler/env.sh")
with env_file.open("w", encoding="utf-8") as fp:
for key, value in sorted(os.environ.items()):
if key.startswith("DQD_") or key in {"TZ", "PYTHONPATH", "PYTHONUNBUFFERED"}:
fp.write(f"export {key}={shlex.quote(value)}\n")
env_file.chmod(0o600)
PY
cd /app
"${PYTHON_BIN}" scripts/docker_task_runner.py render-cron \
--tasks "${TASKS_FILE}" \
--output /etc/cron.d/sport-era-crawler \
--python-bin "${PYTHON_BIN}" \
--app-dir /app
touch /var/log/cron.log
echo "[crawler] cron file rendered from ${TASKS_FILE}"
cat /etc/cron.d/sport-era-crawler
exec cron -f
server/public/dongqiudi-crawler/dqd_worldcup.py
-492
View File
@@ -1,492 +0,0 @@
"""
懂球帝世界杯专属采集
- 固定采集 2026 世界杯 season_id=26123 / competition_id=61
- 赛程写入 la_match
- 分组积分榜写入 la_worldcup_standing
- 射手榜/助攻榜写入 la_worldcup_person_ranking
"""
import json
import re
import time
from datetime import datetime
from urllib.parse import urlparse, parse_qsl, urlencode, urlunparse
import pymysql
import requests
SEASON_ID = 26123
COMPETITION_ID = 61
LEAGUE_NAME = "世界杯"
SPORT_TYPE = 1
LEAGUE_TAB_ID = 26123
LEAGUE_SORT = 285
CRAWLER_USER_AGENT = "Mozilla/5.0 (compatible; SportEra/1.0)"
BASE_PARAMS = {
"app": "dqd",
"version": "830",
"platform": "miniprogram",
"language": "zh-cn",
"app_type": "",
"from": "msite_com",
}
STANDING_URL = "https://sport-data.dongqiudi.com/soccer/biz/data/standing"
SCHEDULE_URL = "https://sport-data.dongqiudi.com/soccer/biz/data/schedule"
PERSON_RANKING_URL = "https://sport-data.dongqiudi.com/soccer/biz/data/person_ranking"
ROUND_ORDER = [
"小组赛 第1轮",
"小组赛 第2轮",
"小组赛 第3轮",
"1/16决赛",
"1/8决赛",
"1/4决赛",
"半决赛",
"三四名决赛",
"决赛",
]
BRACKET_ROUNDS = ROUND_ORDER[3:]
def normalize_prefix(prefix: str) -> str:
prefix = prefix or "la_"
return prefix if re.match(r"^[A-Za-z0-9_]+$", prefix) else "la_"
def console_print(message: str) -> None:
try:
print(message)
except UnicodeEncodeError:
print(message.encode("utf-8", "backslashreplace").decode("utf-8"))
def get_headers() -> dict:
return {
"User-Agent": CRAWLER_USER_AGENT,
"Accept": "application/json, text/plain, */*",
"Referer": "https://www.dongqiudi.com/",
}
def infer_stage(round_name: str) -> str:
if round_name.startswith("小组赛"):
return "小组赛"
return "淘汰赛"
def match_status_to_int(status: str) -> int:
status = (status or "").lower()
status_map = {
"fixture": 0,
"notstarted": 0,
"played": 2,
"finished": 2,
"live": 1,
"playing": 1,
"postponed": 3,
"cancelled": 3,
}
return status_map.get(status, 0)
def round_weight(round_name: str) -> int:
if round_name in ROUND_ORDER:
return ROUND_ORDER.index(round_name)
match = re.search(r"小组赛\s*第(\d+)轮", round_name)
if match:
return int(match.group(1)) - 1
return 999
def build_url(base_url: str, params: dict) -> str:
parsed = urlparse(base_url)
query = dict(parse_qsl(parsed.query, keep_blank_values=True))
query.update({k: str(v) for k, v in params.items() if v is not None})
return urlunparse(parsed._replace(query=urlencode(query)))
def fetch_json(session: requests.Session, url: str, params: dict | None = None) -> dict:
resp = session.get(url, params=params, timeout=30)
resp.raise_for_status()
return resp.json()
def fetch_standings(session: requests.Session) -> dict:
return fetch_json(session, STANDING_URL, {"season_id": SEASON_ID, **BASE_PARAMS})
def fetch_schedule_index(session: requests.Session) -> dict:
return fetch_json(session, SCHEDULE_URL, {"season_id": SEASON_ID, **BASE_PARAMS})
def fetch_round_schedule(session: requests.Session, round_url: str) -> dict:
return fetch_json(session, build_url(round_url, BASE_PARAMS))
def fetch_person_ranking(session: requests.Session, rank_type: str) -> dict:
params = {"season_id": SEASON_ID, "type": rank_type, "app_type": "", **BASE_PARAMS}
return fetch_json(session, PERSON_RANKING_URL, params)
def parse_standings(payload: dict) -> tuple[str, list[dict]]:
rounds = payload.get("content", {}).get("rounds", []) or []
if not rounds:
return "", []
rows = []
for round_item in rounds:
content = round_item.get("content", {}) or {}
stage_name = content.get("name", "") or "小组赛"
for group in content.get("data", []) or []:
group_name = group.get("name", "") or ""
for row_order, team in enumerate(group.get("data", []) or [], start=1):
rows.append({
"season_id": SEASON_ID,
"stage_name": stage_name,
"group_name": group_name,
"team_id": int(team.get("team_id") or 0),
"team_name": str(team.get("team_name") or ""),
"team_logo": str(team.get("team_logo") or ""),
"rank": int(team.get("rank") or 0),
"points": int(team.get("points") or 0),
"played": int(team.get("matches_total") or 0),
"won": int(team.get("matches_won") or 0),
"drawn": int(team.get("matches_draw") or 0),
"lost": int(team.get("matches_lost") or 0),
"goals_for": int(team.get("goals_pro") or 0),
"goals_against": int(team.get("goals_against") or 0),
"goal_diff": int(team.get("goals_pro") or 0) - int(team.get("goals_against") or 0),
"row_order": row_order,
})
return rows[0]["stage_name"] if rows else "", rows
def parse_schedule_matches(schedule_index: dict, session: requests.Session) -> list[dict]:
rounds = schedule_index.get("content", {}).get("rounds", []) or []
matches = []
seen_match_ids = set()
for round_item in rounds:
round_name = round_item.get("name", "") or ""
round_url = round_item.get("url", "")
if not round_name or not round_url:
continue
round_payload = fetch_round_schedule(session, round_url)
round_matches = round_payload.get("content", {}).get("matches", []) or []
for item in round_matches:
match_id = int(item.get("match_id") or 0)
if not match_id or match_id in seen_match_ids:
continue
seen_match_ids.add(match_id)
start_play = str(item.get("start_play") or "")
match_time = 0
if start_play:
try:
match_time = int(datetime.strptime(start_play, "%Y-%m-%d %H:%M:%S").timestamp())
except Exception:
match_time = 0
matches.append({
"match_id": match_id,
"competition_id": int(item.get("competition_id") or COMPETITION_ID),
"home_team_id": str(item.get("team_A_id") or ""),
"away_team_id": str(item.get("team_B_id") or ""),
"league_name": LEAGUE_NAME,
"round_name": round_name,
"stage": infer_stage(round_name),
"home_team": str(item.get("team_A_name") or ""),
"home_icon": str(item.get("team_A_logo") or ""),
"home_score": int(item.get("fs_A") or 0),
"away_team": str(item.get("team_B_name") or ""),
"away_icon": str(item.get("team_B_logo") or ""),
"away_score": int(item.get("fs_B") or 0),
"sport_type": SPORT_TYPE,
"status": match_status_to_int(str(item.get("status") or "")),
"match_time": match_time,
"current_minute": str(item.get("minute") or ""),
"half_score": "",
"sort": round_weight(round_name),
})
matches.sort(key=lambda item: (round_weight(item["round_name"]), item["match_time"], item["match_id"]))
return matches
def parse_person_rankings(payload: dict, rank_type: str) -> tuple[list[str], list[dict]]:
content = payload.get("content", {}) or {}
header = content.get("header", []) or []
rows = []
for item in content.get("data", []) or []:
rows.append({
"season_id": SEASON_ID,
"rank_type": rank_type,
"person_id": int(item.get("person_id") or 0),
"person_name": str(item.get("person_name") or ""),
"person_logo": str(item.get("person_logo") or ""),
"team_id": int(item.get("team_id") or 0),
"team_name": str(item.get("team_name") or ""),
"team_logo": str(item.get("team_logo") or ""),
"rank": int(item.get("rank") or 0),
"count": int(item.get("count") or 0),
"extra_json": json.dumps(item, ensure_ascii=False),
})
return header, rows
def ensure_tables(conn, prefix: str) -> None:
standing_table = f"{prefix}worldcup_standing"
ranking_table = f"{prefix}worldcup_person_ranking"
with conn.cursor() as cur:
cur.execute(
f"""
CREATE TABLE IF NOT EXISTS `{standing_table}` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`season_id` INT NOT NULL DEFAULT 0,
`stage_name` VARCHAR(50) NOT NULL DEFAULT '',
`group_name` VARCHAR(50) NOT NULL DEFAULT '',
`team_id` BIGINT NOT NULL DEFAULT 0,
`team_name` VARCHAR(100) NOT NULL DEFAULT '',
`team_logo` VARCHAR(500) NOT NULL DEFAULT '',
`rank` INT NOT NULL DEFAULT 0,
`points` INT NOT NULL DEFAULT 0,
`played` INT NOT NULL DEFAULT 0,
`won` INT NOT NULL DEFAULT 0,
`drawn` INT NOT NULL DEFAULT 0,
`lost` INT NOT NULL DEFAULT 0,
`goals_for` INT NOT NULL DEFAULT 0,
`goals_against` INT NOT NULL DEFAULT 0,
`goal_diff` INT NOT NULL DEFAULT 0,
`row_order` INT NOT NULL DEFAULT 0,
`create_time` INT UNSIGNED NOT NULL DEFAULT 0,
`update_time` INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_worldcup_group_team` (`season_id`, `stage_name`, `group_name`, `team_id`),
KEY `idx_worldcup_group` (`season_id`, `stage_name`, `group_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
cur.execute(
f"""
CREATE TABLE IF NOT EXISTS `{ranking_table}` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`season_id` INT NOT NULL DEFAULT 0,
`rank_type` VARCHAR(20) NOT NULL DEFAULT '',
`person_id` BIGINT NOT NULL DEFAULT 0,
`person_name` VARCHAR(100) NOT NULL DEFAULT '',
`person_logo` VARCHAR(500) NOT NULL DEFAULT '',
`team_id` BIGINT NOT NULL DEFAULT 0,
`team_name` VARCHAR(100) NOT NULL DEFAULT '',
`team_logo` VARCHAR(500) NOT NULL DEFAULT '',
`rank` INT NOT NULL DEFAULT 0,
`count` INT NOT NULL DEFAULT 0,
`extra_json` JSON NULL,
`create_time` INT UNSIGNED NOT NULL DEFAULT 0,
`update_time` INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_worldcup_rank_person` (`season_id`, `rank_type`, `person_id`),
KEY `idx_worldcup_rank_type` (`season_id`, `rank_type`, `rank`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
"""
)
def ensure_worldcup_league(conn, prefix: str) -> None:
league_table = f"{prefix}league"
now = int(time.time())
with conn.cursor() as cur:
cur.execute(
f"SELECT id FROM `{league_table}` WHERE label=%s OR league_id=%s OR tab_id=%s LIMIT 1",
(LEAGUE_NAME, COMPETITION_ID, LEAGUE_TAB_ID),
)
row = cur.fetchone()
if row:
cur.execute(
f"""
UPDATE `{league_table}`
SET label=%s, `type`='league', sport_type=%s, sort=%s, is_show=1, league_id=%s, update_time=%s
WHERE id=%s
""",
(LEAGUE_NAME, SPORT_TYPE, LEAGUE_SORT, COMPETITION_ID, now, row["id"]),
)
return
cur.execute(
f"""
INSERT INTO `{league_table}`
(tab_id, league_id, label, `type`, sport_type, api, icon, sort, is_show, extra_data, sessionid, create_time, update_time)
VALUES (%s, %s, %s, 'league', %s, '', '', %s, 1, NULL, NULL, %s, %s)
""",
(LEAGUE_TAB_ID, COMPETITION_ID, LEAGUE_NAME, SPORT_TYPE, LEAGUE_SORT, now, now),
)
def sync_matches(conn, prefix: str, matches: list[dict]) -> tuple[int, int]:
table = f"{prefix}match"
now = int(time.time())
inserted = 0
updated = 0
with conn.cursor() as cur:
for item in matches:
cur.execute(
f"""
INSERT INTO `{table}`
(match_id, competition_id, home_team_id, away_team_id,
league_name, round_name, stage, league_icon,
home_team, home_icon, home_score,
away_team, away_icon, away_score,
sport_type, status, match_time,
current_minute, half_score,
is_hot, is_show, sort, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,'',%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,0,1,%s,%s,%s)
ON DUPLICATE KEY UPDATE
competition_id=VALUES(competition_id),
league_name=VALUES(league_name),
round_name=VALUES(round_name),
stage=VALUES(stage),
home_team=VALUES(home_team),
home_icon=VALUES(home_icon),
home_score=VALUES(home_score),
away_team=VALUES(away_team),
away_icon=VALUES(away_icon),
away_score=VALUES(away_score),
sport_type=VALUES(sport_type),
status=VALUES(status),
match_time=VALUES(match_time),
current_minute=VALUES(current_minute),
half_score=VALUES(half_score),
sort=VALUES(sort),
update_time=VALUES(update_time)
""",
(
item["match_id"],
item["competition_id"],
item["home_team_id"],
item["away_team_id"],
item["league_name"],
item["round_name"][:50],
item["stage"][:50],
item["home_team"][:100],
item["home_icon"][:255],
item["home_score"],
item["away_team"][:100],
item["away_icon"][:255],
item["away_score"],
item["sport_type"],
item["status"],
item["match_time"],
item["current_minute"][:20],
item["half_score"][:50],
item["sort"],
now,
now,
),
)
if cur.rowcount == 1:
inserted += 1
elif cur.rowcount == 2:
updated += 1
return inserted, updated
def sync_standings(conn, prefix: str, rows: list[dict]) -> int:
table = f"{prefix}worldcup_standing"
now = int(time.time())
count = 0
with conn.cursor() as cur:
for item in rows:
cur.execute(
f"""
INSERT INTO `{table}`
(season_id, stage_name, group_name, team_id, team_name, team_logo, `rank`,
points, played, won, drawn, lost, goals_for, goals_against, goal_diff,
row_order, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
ON DUPLICATE KEY UPDATE
team_name=VALUES(team_name),
team_logo=VALUES(team_logo),
`rank`=VALUES(`rank`),
points=VALUES(points),
played=VALUES(played),
won=VALUES(won),
drawn=VALUES(drawn),
lost=VALUES(lost),
goals_for=VALUES(goals_for),
goals_against=VALUES(goals_against),
goal_diff=VALUES(goal_diff),
row_order=VALUES(row_order),
update_time=VALUES(update_time)
""",
(
item["season_id"], item["stage_name"][:50], item["group_name"][:50], item["team_id"],
item["team_name"][:100], item["team_logo"][:500], item["rank"], item["points"],
item["played"], item["won"], item["drawn"], item["lost"], item["goals_for"],
item["goals_against"], item["goal_diff"], item["row_order"], now, now,
),
)
count += 1
return count
def sync_rankings(conn, prefix: str, rows: list[dict], rank_type: str) -> int:
table = f"{prefix}worldcup_person_ranking"
now = int(time.time())
count = 0
with conn.cursor() as cur:
cur.execute(
f"DELETE FROM `{table}` WHERE season_id=%s AND rank_type=%s",
(SEASON_ID, rank_type),
)
for item in rows:
cur.execute(
f"""
INSERT INTO `{table}`
(season_id, rank_type, person_id, person_name, person_logo, team_id, team_name, team_logo,
`rank`, `count`, extra_json, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
""",
(
item["season_id"], item["rank_type"], item["person_id"], item["person_name"][:100],
item["person_logo"][:500], item["team_id"], item["team_name"][:100], item["team_logo"][:500],
item["rank"], item["count"], item["extra_json"], now, now,
),
)
count += 1
return count
def run(db_config: dict):
db_config = dict(db_config)
prefix = normalize_prefix(db_config.pop("prefix", "la_"))
session = requests.Session()
session.headers.update(get_headers())
standings_payload = fetch_standings(session)
schedule_index = fetch_schedule_index(session)
goals_payload = fetch_person_ranking(session, "goals")
assists_payload = fetch_person_ranking(session, "assists")
stage_name, standings = parse_standings(standings_payload)
matches = parse_schedule_matches(schedule_index, session)
goal_header, goal_rows = parse_person_rankings(goals_payload, "goals")
assist_header, assist_rows = parse_person_rankings(assists_payload, "assists")
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
ensure_tables(conn, prefix)
ensure_worldcup_league(conn, prefix)
inserted, updated = sync_matches(conn, prefix, matches)
standings_count = sync_standings(conn, prefix, standings)
goals_count = sync_rankings(conn, prefix, goal_rows, "goals")
assists_count = sync_rankings(conn, prefix, assist_rows, "assists")
conn.commit()
finally:
conn.close()
console_print(f"世界杯阶段: {stage_name or '小组赛'}")
console_print(f"赛程轮次: {len(schedule_index.get('content', {}).get('rounds', []) or [])}, 比赛 {len(matches)} 场")
console_print(f"la_match: 新增 {inserted}, 更新 {updated}")
console_print(f"积分榜: {standings_count} 条")
console_print(f"射手榜: {goals_count} 条, 表头: {' / '.join(goal_header or [])}")
console_print(f"助攻榜: {assists_count} 条, 表头: {' / '.join(assist_header or [])}")
return {
"success": True,
"candidate_count": len(matches) + len(standings) + len(goal_rows) + len(assist_rows),
"saved_count": inserted + updated + standings_count + goals_count + assists_count,
"summary_text": f"世界杯赛程 {len(matches)} 场，la_match 新增 {inserted} 更新 {updated}，积分榜 {standings_count}，射手榜 {goals_count}，助攻榜 {assists_count}",
}
server/public/dongqiudi-crawler/fifa_worldcup_news.py
-406
View File
@@ -1,406 +0,0 @@
"""
FIFA 世界杯资讯采集模块 - 从 FIFA CXM API 获取世界杯相关新闻并写入 la_article 表
在 main.py 中通过 fifa_worldcup_news 命令调用
"""
import html
import json
import re
import time
import zlib
from datetime import datetime, timezone
from urllib.parse import urljoin
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
BASE_URL = "https://cxm-api.fifa.com/fifaplusweb/api"
LOCALE = "en"
TOURNAMENT_PATH = "/en/tournaments/mens/worldcup/canadamexicousa2026"
TOURNAMENT_PAGE_URL = f"{BASE_URL}/pages{TOURNAMENT_PATH}"
NEWS_SECTION_URL = f"{BASE_URL}/sections/news/1aQDyhkYnKhkAW347zYi4Y?locale={LOCALE}"
ARTICLE_SECTION_URL = f"{BASE_URL}/sections/article/{{entry_id}}?locale={LOCALE}"
DEFAULT_CATEGORY_NAME = "世界杯"
DEFAULT_CATEGORY_SORT = 999
TITLE_MAX_LEN = 255
DESC_MAX_LEN = 255
ABSTRACT_MAX_LEN = 500
IMAGE_MAX_LEN = 128
AUTHOR_MAX_LEN = 255
PUBLISHED_AT_MAX_LEN = 30
CATEGORY_MAX_LEN = 50
SOURCE_URL_MAX_LEN = 500
def normalize_prefix(prefix: str) -> str:
prefix = prefix or "la_"
return prefix if re.match(r"^[A-Za-z0-9_]+$", prefix) else "la_"
def truncate(value: str, limit: int) -> str:
return (value or "")[:limit]
def console_print(message: str) -> None:
try:
print(message)
except UnicodeEncodeError:
print(message.encode("utf-8", "backslashreplace").decode("utf-8"))
def strip_html(text: str) -> str:
text = re.sub(r"<[^>]+>", "", text or "")
text = html.unescape(text)
text = text.replace("\xa0", " ")
return text.strip()
def parse_iso8601(date_str: str) -> tuple[int, str]:
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = datetime.fromisoformat(date_str.replace("Z", "+00:00"))
return int(dt.timestamp()), dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
def stable_article_id(entry_id: str) -> int:
"""将 FIFA entryId 映射成稳定的负整数，避开现有正整数 article_id。"""
if not entry_id:
return 0
return -int(zlib.crc32(f"fifa_worldcup:{entry_id}".encode("utf-8")) & 0x7FFFFFFF)
def get_headers() -> dict:
return {
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/json, text/plain, */*",
"Origin": "https://www.fifa.com",
"Referer": "https://www.fifa.com" + TOURNAMENT_PATH,
}
def node_text(node) -> str:
if node is None:
return ""
if isinstance(node, str):
return node
node_type = node.get("nodeType", "")
content = node.get("content", []) or []
if node_type == "text":
return html.escape(str(node.get("value", "")))
if node_type == "hyperlink":
href = node.get("data", {}).get("uri", "")
inner = "".join(node_text(child) for child in content)
if href:
return f'<a href="{html.escape(href, quote=True)}">{inner}</a>'
return inner
if node_type in ("paragraph", "heading-1", "heading-2", "heading-3", "heading-4", "heading-5", "heading-6"):
inner = "".join(node_text(child) for child in content).strip()
if not inner:
return ""
if node_type.startswith("heading-"):
level = node_type.split("-")[-1]
return f"<h{level}>{inner}</h{level}>"
return f"<p>{inner}</p>"
if node_type == "unordered-list":
items = [node_text(child) for child in content if node_text(child)]
if not items:
return ""
return "<ul>" + "".join(items) + "</ul>"
if node_type == "list-item":
inner = "".join(node_text(child) for child in content).strip()
return f"<li>{inner}</li>" if inner else ""
if node_type == "embedded-entry-block":
target = node.get("data", {}).get("target", {}) or {}
images = target.get("bynderImage") or []
if isinstance(images, list) and images:
image = images[0] or {}
src = image.get("src") or image.get("original") or image.get("transformBaseUrl") or ""
alt = image.get("title") or image.get("description") or ""
if src:
return f'<p><img src="{html.escape(src, quote=True)}" alt="{html.escape(str(alt), quote=True)}" /></p>'
url = target.get("url") or ""
if url:
text = target.get("title") or url
return f'<p><a href="{html.escape(url, quote=True)}">{html.escape(str(text))}</a></p>'
return ""
if node_type == "document":
parts = [node_text(child) for child in content]
return "\n".join(part for part in parts if part)
parts = [node_text(child) for child in content]
return "".join(parts)
def html_to_text(value: str) -> str:
return strip_html(value)
def fetch_tournament_page(session: requests.Session) -> dict:
resp = session.get(TOURNAMENT_PAGE_URL, timeout=30)
resp.raise_for_status()
return resp.json()
def fetch_news_section(session: requests.Session) -> dict:
resp = session.get(NEWS_SECTION_URL, timeout=30)
resp.raise_for_status()
return resp.json()
def fetch_article_detail(session: requests.Session, entry_id: str) -> dict:
url = ARTICLE_SECTION_URL.format(entry_id=entry_id)
resp = session.get(url, timeout=30)
resp.raise_for_status()
return resp.json()
def extract_article_detail(entry_json: dict) -> dict:
if not isinstance(entry_json, dict):
return {}
props = entry_json.get("properties") or {}
richtext = props.get("richtext") or entry_json.get("richtext") or {}
hero_image = (
props.get("heroImage")
or props.get("previewImage")
or entry_json.get("heroImage")
or entry_json.get("previewImage")
or {}
)
article_title = props.get("articleTitle") or entry_json.get("articleTitle") or entry_json.get("title") or ""
article_published = props.get("articlePublishedDate") or entry_json.get("articlePublishedDate") or ""
preview_text = props.get("articlePreviewText") or entry_json.get("articlePreviewText") or ""
content_html = node_text(richtext) if richtext else ""
image = ""
if isinstance(hero_image, dict):
image = hero_image.get("src") or hero_image.get("url") or hero_image.get("transformBaseUrl") or ""
return {
"title": article_title,
"desc": html_to_text(preview_text)[:255],
"content": content_html or (html.escape(html_to_text(preview_text)) if preview_text else ""),
"image": image,
"published_at": article_published,
}
def fetch_articles(session: requests.Session, section: dict) -> list[dict]:
items = []
for raw in section.get("items", []) or []:
entry_id = (raw.get("entryId") or "").strip()
slug = (raw.get("slug") or "").strip()
title = (raw.get("title") or "").strip()
if not entry_id or not slug:
continue
page_url = urljoin("https://www.fifa.com", raw.get("articlePageUrl") or f"{TOURNAMENT_PATH}/articles/{slug}")
preview_text = html_to_text(raw.get("previewText") or "")[:255]
published_at = raw.get("publishedDate") or ""
image = ""
img = raw.get("image") or {}
if isinstance(img, dict):
image = img.get("src") or img.get("url") or img.get("transformBaseUrl") or ""
try:
detail_json = fetch_article_detail(session, entry_id)
detail = extract_article_detail(detail_json)
except Exception:
detail = {}
if detail.get("title"):
title = detail["title"]
article_image = detail.get("image") or image
publish_ts, publish_fmt = parse_iso8601(detail.get("published_at") or published_at)
content = detail.get("content") or html.escape(preview_text)
items.append({
"entry_id": entry_id,
"title": truncate(title, TITLE_MAX_LEN),
"desc": truncate(detail.get("desc") or preview_text, DESC_MAX_LEN),
"content": content,
"source_url": page_url,
"image": article_image,
"author": "FIFA",
"published_ts": publish_ts,
"published_at": publish_fmt,
"category": "World Cup",
"ext": {
"source": "fifa_cxm",
"entry_id": entry_id,
"slug": slug,
"article_page_url": page_url,
"tournament_path": TOURNAMENT_PATH,
},
})
return items
def ensure_worldcup_category(conn, prefix: str) -> int:
cate_table = f"{prefix}article_cate"
with conn.cursor() as cur:
cur.execute(
f"SELECT id FROM `{cate_table}` WHERE delete_time IS NULL AND name = %s LIMIT 1",
(DEFAULT_CATEGORY_NAME,)
)
row = cur.fetchone()
now = int(time.time())
if row:
cid = int(row["id"])
cur.execute(
f"UPDATE `{cate_table}` SET is_show = 1, sort = %s, update_time = %s WHERE id = %s",
(DEFAULT_CATEGORY_SORT, now, cid)
)
return cid
cur.execute(
f"""INSERT INTO `{cate_table}` (pid, name, sort, is_show, create_time, update_time)
VALUES (0, %s, %s, 1, %s, %s)""",
(DEFAULT_CATEGORY_NAME, DEFAULT_CATEGORY_SORT, now, now)
)
return cur.lastrowid
def sync_to_db(conn, articles: list[dict], cid: int, prefix: str) -> tuple[int, int, int]:
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
if not urls:
return 0, 0, 0
article_table = f"{prefix}article"
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM `{article_table}` WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [cid]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM `{article_table}` WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [cid]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
title = (article.get("title") or "").strip()
if not source_url or not title:
continue
create_ts = article.get("published_ts") or now
article_id = stable_article_id(article.get("entry_id", ""))
ext_json = json.dumps(article.get("ext", {}), ensure_ascii=False) if article.get("ext") else None
image = truncate(article.get("image", ""), IMAGE_MAX_LEN)
author = truncate(article.get("author") or "", AUTHOR_MAX_LEN)
category = truncate(article.get("category") or "", CATEGORY_MAX_LEN)
published_at = truncate(article.get("published_at") or "", PUBLISHED_AT_MAX_LEN)
desc = truncate(article.get("desc") or "", DESC_MAX_LEN)
abstract = truncate(article.get("desc") or "", ABSTRACT_MAX_LEN)
source_url = truncate(source_url, SOURCE_URL_MAX_LEN)
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
f"""UPDATE `{article_table}` SET
title = %s, `desc` = %s, image = %s, author = %s,
category = %s, ext = %s, is_show = %s, update_time = %s
WHERE id = %s""",
(
truncate(title, TITLE_MAX_LEN),
desc,
image,
author,
category,
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
if title[:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
f"""INSERT INTO `{article_table}`
(article_id, cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
article_id,
cid,
truncate(title, TITLE_MAX_LEN),
desc,
abstract,
image,
author,
published_at,
category,
source_url,
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
create_ts,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
existing_titles.add(title[:255])
new_count += 1
return len(articles), new_count, update_count
def run(db_config: dict):
db_config = dict(db_config)
prefix = normalize_prefix(db_config.pop("prefix", "la_"))
session = requests.Session()
session.headers.update(get_headers())
try:
page = fetch_tournament_page(session)
news_section = fetch_news_section(session)
articles = fetch_articles(session, news_section)
console_print(f"世界杯页面: {page.get('meta', {}).get('title', '')}")
console_print(f"新闻区块: {news_section.get('entryId', '')}, 原始条数: {len(articles)}")
except Exception as e:
console_print(f"[WARN] FIFA 页面/新闻区块拉取失败: {e}")
return {"success": False, "candidate_count": 0, "saved_count": 0, "error_message": str(e), "summary_text": f"FIFA 页面/新闻区块拉取失败: {e}"}
if not articles:
console_print("世界杯资讯无数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "世界杯资讯无数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
cid = ensure_worldcup_category(conn, prefix)
total, new_count, update_count = sync_to_db(conn, articles, cid, prefix)
conn.commit()
console_print(f"分类ID: {cid}")
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
console_print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/hkjc_lottery_news.py
-245
View File
@@ -1,245 +0,0 @@
"""
香港赛马会新闻采集模块
- 列表页: https://racingnews.hkjc.com/chinese/?ny=2026&nm=6
- 详情页: 列表内详情链接
写入分类:
- la_article_cate.id = 21 (彩票资讯)
"""
import html
import json
import re
import time
from datetime import datetime
from urllib.parse import urljoin
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
ARTICLE_CID = 21
LIST_URL = "https://racingnews.hkjc.com/chinese/?ny=2026&nm=6"
SOURCE_NAME = "香港赛马会"
MAX_ITEMS = 20
def clean_text(text: str) -> str:
return re.sub(r"\s+", " ", html.unescape(text or "").replace("\u3000", " ")).strip()
def parse_hk_date(date_str: str) -> tuple[int, str]:
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
for fmt in ("%d/%m/%Y %H:%M", "%d/%m/%Y"):
try:
dt = datetime.strptime(date_str.strip(), fmt)
ts = int(dt.timestamp())
return ts, dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
continue
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
def fetch_list(session: requests.Session) -> list[dict]:
resp = session.get(LIST_URL, timeout=30)
resp.raise_for_status()
html_text = resp.text
matches = re.findall(
r'<div class="news-listing-container.*?<div class="news-listing-date">(.*?)</div>.*?<div class="news-listing-content-title">\s*<a href="([^"]+)">(.*?)</a>',
html_text,
re.I | re.S
)
items = []
for date_text, href, title in matches[:MAX_ITEMS]:
items.append({
"date_text": clean_text(re.sub(r"<[^>]+>", " ", date_text)),
"source_url": urljoin(LIST_URL, href),
"title": clean_text(re.sub(r"<[^>]+>", " ", title)),
})
return items
def parse_detail(session: requests.Session, url: str) -> dict:
resp = session.get(url, timeout=30)
resp.raise_for_status()
html_text = resp.text
heading_matches = re.findall(r"<h2[^>]*>(.*?)</h2>", html_text, re.I | re.S)
title = ""
for raw in heading_matches:
text = clean_text(re.sub(r"<[^>]+>", " ", raw))
if text and text != "其他新聞":
title = text
break
paragraphs = []
for raw in re.findall(r"<p[^>]*>([\s\S]*?)</p>", html_text, re.I):
text = clean_text(re.sub(r"<[^>]+>", " ", raw))
if len(text) > 8:
paragraphs.append(text)
images = []
for src in re.findall(r'<img[^>]+src="([^"]+res\.hkjc\.com[^"]+)"', html_text, re.I):
full = urljoin(url, src)
if full not in images:
images.append(full)
published_text = ""
if paragraphs and re.match(r"^\d{2}/\d{2}/\d{4}", paragraphs[0]):
published_text = paragraphs.pop(0)
content_parts = [f"<p>{text}</p>" for text in paragraphs]
for image in images[:3]:
content_parts.append(f'<img src="{image}" />')
desc = paragraphs[0][:255] if paragraphs else title[:255]
return {
"title": title[:255],
"desc": desc,
"content": "".join(content_parts),
"image": images[0] if images else "",
"published_text": published_text[:64],
"source_url": url,
}
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
now = int(time.time())
new_count = 0
update_count = 0
for article in articles:
title = article.get("title", "")
source_url = article.get("source_url", "")
if not title or not source_url:
continue
ext_json = json.dumps({
"source": "hkjc_racingnews",
"list_url": LIST_URL,
}, ensure_ascii=False)
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title=%s, `desc`=%s, abstract=%s, image=%s, author=%s,
published_at=%s, category=%s, content=%s, ext=%s, is_show=%s, update_time=%s
WHERE id=%s""",
(
title[:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
(article.get("image") or "")[:128],
SOURCE_NAME,
(article.get("published_at") or "")[:64],
"lottery,hk",
article.get("content") or "",
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
if title[:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
title[:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
(article.get("image") or "")[:128],
SOURCE_NAME,
(article.get("published_at") or "")[:64],
"lottery,hk",
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
article.get("published_ts") or now,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
new_count += 1
existing_titles.add(title[:255])
return len(articles), new_count, update_count
def run(db_config: dict):
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "text/html,*/*",
})
list_items = fetch_list(session)
print(f"HKJC 资讯列表返回 {len(list_items)} 条")
articles = []
for item in list_items:
detail = parse_detail(session, item["source_url"])
published_ts, published_at = parse_hk_date(detail.get("published_text") or item.get("date_text", ""))
articles.append({
"title": detail.get("title") or item["title"],
"desc": detail.get("desc", ""),
"content": detail.get("content", ""),
"image": detail.get("image", ""),
"published_ts": published_ts,
"published_at": published_at,
"source_url": item["source_url"],
})
if not articles:
print("无可写入数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "无可写入数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/insert_crontab_cba_news.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT 'CBA资讯采集', 1, 0, '从 Google News RSS 拉取 CBA 新闻并写入 CBA 分类', 'crawler', 'cba_news', 1, '20 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'cba_news' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_dqd_worldcup.sql
-73
View File
@@ -1,73 +0,0 @@
CREATE TABLE IF NOT EXISTS `la_worldcup_standing` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`season_id` INT NOT NULL DEFAULT 0,
`stage_name` VARCHAR(50) NOT NULL DEFAULT '',
`group_name` VARCHAR(50) NOT NULL DEFAULT '',
`team_id` BIGINT NOT NULL DEFAULT 0,
`team_name` VARCHAR(100) NOT NULL DEFAULT '',
`team_logo` VARCHAR(500) NOT NULL DEFAULT '',
`rank` INT NOT NULL DEFAULT 0,
`points` INT NOT NULL DEFAULT 0,
`played` INT NOT NULL DEFAULT 0,
`won` INT NOT NULL DEFAULT 0,
`drawn` INT NOT NULL DEFAULT 0,
`lost` INT NOT NULL DEFAULT 0,
`goals_for` INT NOT NULL DEFAULT 0,
`goals_against` INT NOT NULL DEFAULT 0,
`goal_diff` INT NOT NULL DEFAULT 0,
`row_order` INT NOT NULL DEFAULT 0,
`create_time` INT UNSIGNED NOT NULL DEFAULT 0,
`update_time` INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_worldcup_group_team` (`season_id`, `stage_name`, `group_name`, `team_id`),
KEY `idx_worldcup_group` (`season_id`, `stage_name`, `group_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
CREATE TABLE IF NOT EXISTS `la_worldcup_person_ranking` (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`season_id` INT NOT NULL DEFAULT 0,
`rank_type` VARCHAR(20) NOT NULL DEFAULT '',
`person_id` BIGINT NOT NULL DEFAULT 0,
`person_name` VARCHAR(100) NOT NULL DEFAULT '',
`person_logo` VARCHAR(500) NOT NULL DEFAULT '',
`team_id` BIGINT NOT NULL DEFAULT 0,
`team_name` VARCHAR(100) NOT NULL DEFAULT '',
`team_logo` VARCHAR(500) NOT NULL DEFAULT '',
`rank` INT NOT NULL DEFAULT 0,
`count` INT NOT NULL DEFAULT 0,
`extra_json` JSON NULL,
`create_time` INT UNSIGNED NOT NULL DEFAULT 0,
`update_time` INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
UNIQUE KEY `uk_worldcup_rank_person` (`season_id`, `rank_type`, `person_id`),
KEY `idx_worldcup_rank_type` (`season_id`, `rank_type`, `rank`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
UPDATE `la_league`
SET `label` = '世界杯',
`type` = 'league',
`sport_type` = 1,
`api` = '',
`icon` = '',
`sort` = 285,
`is_show` = 1,
`league_id` = 61,
`update_time` = UNIX_TIMESTAMP(),
`delete_time` = NULL
WHERE (`label` = '世界杯' OR `league_id` = 61 OR `tab_id` = 26123);
INSERT INTO `la_league`
(`tab_id`, `league_id`, `label`, `type`, `sport_type`, `api`, `icon`, `sort`, `is_show`, `extra_data`, `sessionid`, `create_time`, `update_time`)
SELECT 26123, 61, '世界杯', 'league', 1, '', '', 285, 1, NULL, NULL, UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1
FROM `la_league`
WHERE (`label` = '世界杯' OR `league_id` = 61 OR `tab_id` = 26123)
AND `delete_time` IS NULL
);
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT '懂球帝世界杯采集', 1, 0, '拉取世界杯赛程、积分榜、射手榜和助攻榜', 'crawler', 'dqd_worldcup', 1, '*/10 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'dqd_worldcup' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_fifa_worldcup_news.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT 'FIFA世界杯资讯', 1, 0, '拉取FIFA世界杯资讯并写入世界杯分类', 'crawler', 'fifa_worldcup_news', 1, '*/10 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'fifa_worldcup_news' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_hkjc_lottery_news.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT '香港赛马会资讯采集', 1, 0, '拉取香港赛马会新闻并写入彩票资讯分类', 'crawler', 'hkjc_lottery_news', 1, '45 */2 * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'hkjc_lottery_news' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_lottery_community.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT '49宝典新旧澳六合彩社区采集', 1, 0, '同步49宝典新旧澳社区帖子', 'crawler', 'lottery_community', 1, '*/10 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'lottery_community' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_nba_news.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT 'NBA资讯采集', 1, 0, '从 Google News RSS 拉取 NBA 新闻并写入 NBA 分类', 'crawler', 'nba_news', 1, '10 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'nba_news' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/insert_crontab_taiwan_lottery_news.sql
-5
View File
@@ -1,5 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT '台湾彩券资讯采集', 1, 0, '拉取台湾彩券官网新闻并写入彩票资讯分类', 'crawler', 'taiwan_lottery_news', 1, '15 */2 * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'taiwan_lottery_news' AND `delete_time` IS NULL
);
server/public/dongqiudi-crawler/lottery_community.py
-431
View File
@@ -1,431 +0,0 @@
import hashlib
import html
import json
import re
import time
from datetime import datetime
from urllib.parse import urljoin
import pymysql
import requests
BASE_URL = "https://cc33.49bd28.com/"
CDN_FALLBACK = "https://tk2cdn-hw.thc517.com"
IMAGE_CDN_HOST = "https://tk2cdn.ai4funs.com"
SOURCE = "49baodian"
USER_ACCOUNT = "lottery_publisher"
USER_NICKNAME = "彩票资料员"
USER_SN_START = 49000001
LOCK_WAIT_RETRY_CODES = {1205, 1213}
LOTTERY_VARIANTS = {
"a6": {
"name": "旧澳六合",
"aggregate_tag": "彩票",
"game_type": 2032,
"issue_info_url": "https://dokv.buyacard.cc/kv/gr/a6/issue/currentInfo",
"recommend_url": "https://ocs.ai4funs.com/pwtkprd/tk/pw02tk02/issue/a6/recommend",
"serial_list_url": "https://ocs.ai4funs.com/pwtkprd/tk/pw02tk02/a6/serialList",
},
"xa6": {
"name": "新澳六合",
"aggregate_tag": "彩票",
"game_type": 5,
"issue_info_url": "https://dokv.buyacard.cc/kv/gr/xa6/issue/currentInfo",
"recommend_url": "https://ocs.ai4funs.com/pwtkprd/tk/pw02tk02/issue/xa6/recommend",
"serial_list_url": "https://ocs.ai4funs.com/pwtkprd/tk/pw02tk02/xa6/serialList",
},
}
def table(prefix: str, name: str) -> str:
return f"`{prefix}{name}`"
def astro_decode(value):
if isinstance(value, list) and len(value) == 2 and isinstance(value[0], int):
value_type, payload = value
if value_type == 0:
if isinstance(payload, dict):
return {key: astro_decode(val) for key, val in payload.items()}
return payload
if value_type == 1:
return [astro_decode(item) for item in payload]
if value_type in (2, 3, 6, 7):
return payload
if value_type in (4, 5):
return [astro_decode(item) for item in payload]
if value_type in (8, 9, 10):
return payload
return payload
if isinstance(value, dict):
return {key: astro_decode(val) for key, val in value.items()}
if isinstance(value, list):
return [astro_decode(item) for item in value]
return value
def fetch_home_html(session: requests.Session) -> str:
resp = session.get(BASE_URL, timeout=30)
resp.raise_for_status()
return resp.content.decode("utf-8", errors="replace")
def extract_cdn_host(page_html: str) -> str:
match = re.search(r"window\.CDN_HOST\s*=\s*['\"]([^'\"]+)['\"]", page_html)
return match.group(1).rstrip("/") if match else CDN_FALLBACK
def extract_home_data(page_html: str) -> dict:
for match in re.finditer(r"<astro-island\b[^>]*\bprops=([\"'])(.*?)\1", page_html, re.S):
props_raw = html.unescape(match.group(2))
try:
props = astro_decode(json.loads(props_raw))
except json.JSONDecodeError:
continue
data = props.get("data") if isinstance(props, dict) else None
if isinstance(data, dict) and (data.get("lotteryResult") or data.get("galleryResult")):
return data
return {}
def request_json(session: requests.Session, url: str) -> dict:
resp = session.get(url, timeout=30)
resp.raise_for_status()
return resp.json()
WUXING_MAP = {"j": "金", "m": "木", "s": "水", "h": "火", "t": "土"}
ODD_EVEN_MAP = {"o": "单", "e": "双"}
SIZE_MAP = {"b": "大", "s": "小"}
def color_text(color: str) -> str:
return {"red": "红波", "blue": "蓝波", "green": "绿波", "R": "红波", "G": "绿波", "B": "蓝波"}.get(color, color or "")
def format_time(value) -> str:
if not value:
return ""
if isinstance(value, str) and re.match(r"^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}$", value):
return value
try:
ts = int(value)
except (TypeError, ValueError):
return ""
if ts > 100000000000:
ts = ts // 1000
return datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
def image_url(path: str, cdn_host: str) -> str:
if not path:
return ""
if path.startswith("http://") or path.startswith("https://"):
return path
normalized = path.lstrip("/")
if normalized.startswith("prod/") or normalized.startswith("prodmedia/"):
return urljoin(IMAGE_CDN_HOST + "/", normalized)
return urljoin(cdn_host.rstrip("/") + "/", normalized)
def int_value(value) -> int:
try:
return int(value)
except (TypeError, ValueError):
return 0
def is_retryable_db_error(exc: Exception) -> bool:
return isinstance(exc, pymysql.err.OperationalError) and len(exc.args) > 0 and exc.args[0] in LOCK_WAIT_RETRY_CODES
def build_draw_post(detail: dict, lottery_key: str) -> list[dict]:
if not isinstance(detail, dict):
return []
issue = detail.get("currentCompleteIssue") or detail.get("currentIssue") or detail.get("issue")
if not issue:
return []
numbers = detail.get("processedOpenCode") or detail.get("openCode") or detail.get("currentResult") or []
year = detail.get("currentYear") or detail.get("year", "")
lines = ["【49宝典开奖】", f"{year}年第{issue}期"]
for index, number in enumerate(numbers):
label = "特码" if index == len(numbers) - 1 else f"第{index + 1}球"
five_el = number.get("ws") or number.get("fiveElements") or ""
five_el = WUXING_MAP.get(five_el, five_el)
sz = number.get("size", "")
sz = SIZE_MAP.get(sz, sz)
oe = number.get("parity") or number.get("oddEven") or ""
oe = ODD_EVEN_MAP.get(oe, oe)
attrs = [
str(number.get("value") or number.get("num") or ""),
str(number.get("pet") or number.get("shengxiao") or ""),
str(five_el),
color_text(str(number.get("color", ""))),
str(oe),
str(number.get("combinedParity", "")),
str(sz),
str(number.get("tailSize", "")),
]
lines.append(f"{label}：" + " ".join([item for item in attrs if item]))
total_parity = detail.get("totalParity")
total_size = detail.get("totalSize")
if total_parity or total_size:
lines.append(f"总和：{total_parity or ''} {total_size or ''}".strip())
next_issue = detail.get("nextCompleteIssue") or detail.get("nextIssue")
if next_issue:
lines.append(f"下期：{next_issue}")
next_time = format_time(detail.get("nextOpenTime") or detail.get("nextTime"))
if next_time:
lines.append(f"下期开奖时间：{next_time}")
return [{
"origin_id": f"{SOURCE}:draw:{lottery_key}:{issue}",
"content": "\n".join(lines),
"images": [],
"like_count": 0,
"ext": {"type": "draw", "source": SOURCE, "lottery_key": lottery_key, "raw": detail},
"create_time": int(time.time()),
}]
def build_gallery_posts(items: list[dict], lottery_key: str, cdn_host: str) -> list[dict]:
if not isinstance(items, list):
return []
posts = []
for item in items:
if not isinstance(item, dict):
continue
target_id = str(item.get("targetId") or item.get("newspaperCode") or "")
if not target_id:
raw_key = json.dumps(item, ensure_ascii=False, sort_keys=True)
target_id = hashlib.md5(raw_key.encode("utf-8")).hexdigest()
images = []
for image in item.get("img") or []:
if isinstance(image, dict):
url = image_url(str(image.get("url") or ""), cdn_host)
if url:
images.append(url)
author = item.get("author") if isinstance(item.get("author"), dict) else {}
lines = [f"【{item.get('title') or '49宝典资料'}】"]
if item.get("year") or item.get("issue"):
lines.append(f"期号：{item.get('year', '')}年第{item.get('issue', '')}期")
if item.get("serialName"):
lines.append(f"系列：{item.get('serialName')}")
posts.append({
"origin_id": f"{SOURCE}:gallery:{lottery_key}:{target_id}",
"legacy_origin_ids": [f"{SOURCE}:gallery:{target_id}"] if lottery_key == "a6" else [],
"content": "\n".join(lines),
"images": images,
"like_count": int_value(item.get("totalLikeCount")),
"ext": {"type": "gallery", "source": SOURCE, "lottery_key": lottery_key, "raw": item},
"create_time": int(time.time()),
})
return posts
def build_posts(issue_detail: dict, gallery_items: list[dict], lottery_key: str, cdn_host: str) -> list[dict]:
posts = []
posts.extend(build_draw_post(issue_detail, lottery_key))
posts.extend(build_gallery_posts(gallery_items, lottery_key, cdn_host))
return posts
def ensure_user(conn, prefix: str) -> int:
with conn.cursor() as cur:
cur.execute(f"SELECT id FROM {table(prefix, 'user')} WHERE account=%s LIMIT 1", (USER_ACCOUNT,))
row = cur.fetchone()
if row:
return int(row["id"])
cur.execute(f"SELECT MAX(sn) AS max_sn FROM {table(prefix, 'user')}")
max_sn = int((cur.fetchone() or {}).get("max_sn") or 0)
sn = max(USER_SN_START, max_sn + 1)
now = int(time.time())
cur.execute(
f"""INSERT INTO {table(prefix, 'user')}
(sn, avatar, real_name, nickname, account, password, mobile, sex, channel, is_disable,
login_ip, login_time, is_new_user, is_vip, vip_level, vip_expire_time, ai_free_count,
invite_code, inviter_id, user_money, user_points, total_recharge_amount, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
(sn, "", "", USER_NICKNAME, USER_ACCOUNT, "", "", 0, 0, 0,
"", 0, 0, 0, 0, 0, 3, "", 0, 0, 0, 0, now, now)
)
return int(cur.lastrowid)
def ensure_tag(conn, prefix: str, tag_name: str) -> int:
with conn.cursor() as cur:
cur.execute(f"SELECT id, status FROM {table(prefix, 'community_tag')} WHERE name=%s LIMIT 1", (tag_name,))
row = cur.fetchone()
now = int(time.time())
if row:
tag_id = int(row["id"])
if int(row.get("status") or 0) != 1:
cur.execute(f"UPDATE {table(prefix, 'community_tag')} SET status=1 WHERE id=%s", (tag_id,))
return tag_id
cur.execute(
f"""INSERT INTO {table(prefix, 'community_tag')}
(name, icon, sort, post_count, is_hot, status, create_time)
VALUES (%s,%s,%s,%s,%s,%s,%s)""",
(tag_name, "", 10, 0, 1, 1, now)
)
return int(cur.lastrowid)
def save_post_images(conn, prefix: str, post_id: int, images: list[str], create_time: int):
with conn.cursor() as cur:
cur.execute(f"DELETE FROM {table(prefix, 'community_post_image')} WHERE post_id=%s", (post_id,))
for index, url in enumerate(images):
cur.execute(
f"INSERT INTO {table(prefix, 'community_post_image')} (post_id, image_url, sort, create_time) VALUES (%s,%s,%s,%s)",
(post_id, url, index, create_time)
)
def ensure_post_tag(conn, prefix: str, post_id: int, tag_id: int):
with conn.cursor() as cur:
cur.execute(
f"INSERT IGNORE INTO {table(prefix, 'community_post_tag')} (post_id, tag_id) VALUES (%s,%s)",
(post_id, tag_id)
)
def update_tag_count(conn, prefix: str, tag_id: int):
with conn.cursor() as cur:
cur.execute(
f"""UPDATE {table(prefix, 'community_tag')} SET post_count=(
SELECT COUNT(*) FROM {table(prefix, 'community_post_tag')} pt
INNER JOIN {table(prefix, 'community_post')} p ON p.id=pt.post_id
WHERE pt.tag_id=%s AND p.status=1 AND p.delete_time IS NULL
) WHERE id=%s""",
(tag_id, tag_id)
)
def sync_to_db(conn, prefix: str, posts: list[dict], user_id: int, tag_ids: list[int]) -> tuple[int, int]:
inserted = 0
updated = 0
now = int(time.time())
for post in posts:
images = post.get("images") or []
images_json = json.dumps(images, ensure_ascii=False)
ext_json = json.dumps(post.get("ext") or {}, ensure_ascii=False)
create_time = int(post.get("create_time") or now)
legacy_origin_ids = [item for item in (post.get("legacy_origin_ids") or []) if item]
post_id = 0
for attempt in range(3):
try:
with conn.cursor() as cur:
cur.execute(
f"SELECT id FROM {table(prefix, 'community_post')} WHERE origin_id=%s LIMIT 1",
(post["origin_id"],)
)
existing = cur.fetchone()
if not existing and legacy_origin_ids:
placeholders = ",".join(["%s"] * len(legacy_origin_ids))
cur.execute(
f"SELECT id FROM {table(prefix, 'community_post')} WHERE origin_id IN ({placeholders}) ORDER BY id DESC LIMIT 1",
tuple(legacy_origin_ids)
)
existing = cur.fetchone()
if existing:
post_id = int(existing["id"])
cur.execute(
f"""UPDATE {table(prefix, 'community_post')}
SET origin_id=%s, user_id=%s, content=%s, images=%s, ext=%s, like_count=%s, status=1, update_time=%s
WHERE id=%s""",
(post["origin_id"], user_id, post["content"], images_json, ext_json, int_value(post.get("like_count")), now, post_id)
)
updated += 1
else:
cur.execute(
f"""INSERT INTO {table(prefix, 'community_post')}
(origin_id, user_id, content, images, post_type, is_paid, price_points, free_content_len,
like_count, ext, status, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
(post["origin_id"], user_id, post["content"], images_json, 0, 0, 0, 100,
int_value(post.get("like_count")), ext_json, 1, create_time, now)
)
post_id = int(cur.lastrowid)
inserted += 1
break
except Exception as exc:
if not is_retryable_db_error(exc) or attempt == 2:
raise
conn.rollback()
time.sleep(0.5 * (attempt + 1))
for tag_id in tag_ids:
ensure_post_tag(conn, prefix, post_id, tag_id)
save_post_images(conn, prefix, post_id, images, create_time)
for tag_id in tag_ids:
update_tag_count(conn, prefix, tag_id)
return inserted, updated
def load_variant_payload(session: requests.Session, lottery_key: str) -> tuple[dict, list[dict]]:
variant = LOTTERY_VARIANTS[lottery_key]
issue_detail = request_json(session, variant["issue_info_url"])
gallery_payload = request_json(session, variant["recommend_url"])
gallery_items = gallery_payload.get("issueList") or []
return issue_detail, gallery_items
def run(db_config: dict):
config = dict(db_config)
prefix = config.pop("prefix", "la_")
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Referer": BASE_URL,
})
page_html = fetch_home_html(session)
cdn_host = extract_cdn_host(page_html)
data = extract_home_data(page_html)
if not data:
raise RuntimeError("未解析到49宝典首页props数据")
conn = pymysql.connect(**config, cursorclass=pymysql.cursors.DictCursor)
try:
user_id = ensure_user(conn, prefix)
aggregate_tag_id = ensure_tag(conn, prefix, "彩票")
total_inserted = 0
total_updated = 0
total_candidates = 0
for lottery_key, variant in LOTTERY_VARIANTS.items():
issue_detail, gallery_items = load_variant_payload(session, lottery_key)
posts = build_posts(issue_detail, gallery_items, lottery_key, cdn_host)
print(f"{variant['name']} 解析到 {len(posts)} 条49宝典数据")
total_candidates += len(posts)
if not posts:
continue
tag_id = ensure_tag(conn, prefix, variant["name"])
inserted, updated = sync_to_db(conn, prefix, posts, user_id, [aggregate_tag_id, tag_id])
conn.commit()
total_inserted += inserted
total_updated += updated
summary = f"发布用户ID {user_id}, 聚合分类ID {aggregate_tag_id}, 新增 {total_inserted} 条, 更新 {total_updated} 条"
print(summary)
return {"success": True, "candidate_count": total_candidates, "saved_count": total_inserted + total_updated, "summary_text": summary}
except Exception:
conn.rollback()
raise
finally:
conn.close()
server/public/dongqiudi-crawler/lottery_news.py
-284
View File
@@ -1,284 +0,0 @@
"""
彩票资讯采集模块 - 从 Google News RSS 获取彩票新闻并写入 la_article 表
在 main.py 中通过 lottery_news 命令调用
数据源（RSS）:
1. Google News 中文简体 - 彩票开奖（主）
重复判断: 以 source_url 为唯一键，重复则更新标题/摘要
"""
import json
import re
import time
import xml.etree.ElementTree as ET
from datetime import datetime
from email.utils import parsedate_to_datetime
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
# ─── 配置 ───
ARTICLE_CID = 21 # la_article_cate 中"彩票资讯"的 id
RSS_FEEDS = [
{
"name": "彩票开奖(中文)",
"url": "https://news.google.com/rss/search?q=%E5%BD%A9%E7%A5%A8+%E5%BC%80%E5%A5%96&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
]
NS = {
"content": "http://purl.org/rss/1.0/modules/content/",
"media": "http://search.yahoo.com/mrss/",
"dc": "http://purl.org/dc/elements/1.1/",
}
def strip_html(text: str) -> str:
import html as html_mod
text = re.sub(r"<[^>]+>", "", text or "")
text = html_mod.unescape(text)
text = html_mod.unescape(text)
text = text.replace("\xa0", " ")
return text.strip()
def parse_rfc2822(date_str: str) -> tuple[int, str]:
"""解析 RSS pubDate (RFC 2822)，返回 (unix时间戳, 'Y-m-d H:i:s'格式字符串)"""
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = parsedate_to_datetime(date_str)
return int(dt.timestamp()), dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
# ─── RSS 采集 ───
def fetch_rss(session: requests.Session, feed: dict) -> list[dict]:
"""从单个 RSS 源拉取新闻，返回标准化列表"""
articles = []
url = feed["url"]
name = feed["name"]
try:
resp = session.get(url, timeout=30)
except Exception as e:
print(f" [WARN] {name} 请求失败: {e}")
return articles
if resp.status_code != 200:
print(f" [WARN] {name} HTTP {resp.status_code}")
return articles
try:
root = ET.fromstring(resp.content)
except ET.ParseError as e:
print(f" [WARN] {name} XML 解析失败: {e}")
return articles
items = root.findall(".//item")
for item in items:
title = (item.findtext("title", "") or "").strip()
link = (item.findtext("link", "") or "").strip()
if not title or not link:
continue
pub_date = item.findtext("pubDate", "")
desc_raw = item.findtext("description", "") or ""
desc = strip_html(desc_raw)[:255]
# Google News 标题格式: "标题 - 来源名"，提取来源
author = feed.get("author_default", name)
if " - " in title:
parts = title.rsplit(" - ", 1)
if len(parts) == 2:
author = parts[1].strip()
title = parts[0].strip()
# 尝试提取图片
image = ""
media_el = item.find("media:content", NS)
if media_el is not None:
image = media_el.get("url", "")
if not image:
media_thumb = item.find("media:thumbnail", NS)
if media_thumb is not None:
image = media_thumb.get("url", "")
if not image:
img_match = re.search(r'<img[^>]+src=["\']([^"\']+)["\']', desc_raw)
if img_match:
image = img_match.group(1)
pub_ts, pub_fmt = parse_rfc2822(pub_date)
articles.append({
"title": title[:255],
"desc": desc,
"content": strip_html(desc_raw),
"source_url": link,
"image": image,
"author": author[:64],
"published_ts": pub_ts,
"published_at": pub_fmt,
"category": "lottery",
"ext": {
"source": "google_news",
"feed_name": name,
"feed_url": url,
},
})
return articles
# ─── 数据库同步 ───
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
"""同步到 la_article 表，返回 (总数, 新增数, 更新数)"""
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
if not urls:
return 0, 0, 0
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
# 按标题去重：同分类下相同标题的文章不再插入
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
if not source_url or not article.get("title"):
continue
create_ts = article.get("published_ts") or now
ext_json = json.dumps(article.get("ext", {}), ensure_ascii=False) if article.get("ext") else None
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title = %s, `desc` = %s, image = %s, author = %s,
category = %s, ext = %s, is_show = %s, update_time = %s
WHERE id = %s""",
(
article["title"][:255],
(article.get("desc") or "")[:255],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("category") or "")[:128],
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
# 标题去重：相同标题跳过
if article["title"][:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
article["title"][:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("published_at") or "")[:64],
(article.get("category") or "")[:128],
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
create_ts,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
existing_titles.add(article["title"][:255])
new_count += 1
return len(articles), new_count, update_count
# ─── 去重：两个 feed 可能有重叠 ───
def dedupe_articles(articles: list[dict]) -> list[dict]:
"""按 source_url 去重，保留先出现的"""
seen = set()
result = []
for a in articles:
url = a.get("source_url", "")
if url and url not in seen:
seen.add(url)
result.append(a)
return result
# ─── 主入口 ───
def run(db_config: dict):
"""主入口，db_config 从 dongqiudi-crawler 配置中传入"""
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/xml, text/xml, application/rss+xml, */*",
})
all_articles = []
for feed in RSS_FEEDS:
print(f"=== {feed['name']} 采集 ===")
articles = fetch_rss(session, feed)
print(f" {feed['name']} 返回 {len(articles)} 条")
all_articles.extend(articles)
all_articles = dedupe_articles(all_articles)
print(f"去重后共 {len(all_articles)} 条")
if not all_articles:
print("所有数据源均无数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "所有数据源均无数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, all_articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/main.py
-963
View File
@@ -1,963 +0,0 @@
#!/usr/bin/env python3
"""
懂球帝数据爬虫 - 主入口
用法:
python main.py standings # 采集所有联赛积分榜
python main.py schedule # 采集所有联赛赛程
python main.py menu # 采集比赛类型菜单
python main.py lottery # 采集香港/澳门六合彩开奖号码
python main.py match_data # 从la_league同步赛事比赛数据
python main.py csl_match # 采集中超赛程页面数据
python main.py nba_match # 采集NBA赛程页面数据
python main.py cba_match # 采集CBA赛程页面数据
python main.py epl_match # 采集英超赛程页面数据
python main.py bundesliga_match # 采集德甲赛程页面数据
python main.py laliga_match # 采集西甲赛程页面数据
python main.py seriea_match # 采集意甲赛程页面数据
python main.py ligue1_match # 采集法甲赛程页面数据
python main.py ucl_match # 采集欧冠赛程页面数据
python main.py uel_match # 采集欧联赛程页面数据
python main.py tennis_match # 采集网球赛程页面数据
python main.py esports_match # 采集电竞赛程页面数据
python main.py sports_match # 采集体坛赛程页面数据
python main.py truth_social # 采集Truth Social帖子并入库
python main.py crypto_news # 采集加密货币新闻资讯
python main.py lottery_news # 采集彩票资讯
python main.py taiwan_lottery_news # 采集台湾彩券资讯
python main.py hkjc_lottery_news # 采集香港赛马会资讯
python main.py fifa_worldcup_news # 采集 FIFA 世界杯资讯
python main.py dqd_worldcup # 采集懂球帝世界杯赛程/积分榜/球员榜
python main.py nba_news # 采集NBA新闻资讯
python main.py cba_news # 采集CBA新闻资讯
python main.py ai_comment_dispatch # Docker AI 评论调度
python main.py article_content_fetch # 抓取资讯文章正文内容
python main.py all # 采集积分榜 + 赛程
python main.py single <season_id> # 采集单个联赛积分榜
python main.py cron # 启动定时调度
python main.py init-db # 初始化数据库表
python main.py test # 测试连通性
"""
import asyncio
import sys
import time
import io
import os
from pathlib import Path
from contextlib import redirect_stdout
from src.core.config import load_config
from src.core.error_collector import ErrorCollector
from src.core.logger import setup_logger
TASK_ALERT_POLICY = {
"error_report": "never",
"article_content": "saved_if_candidates",
"article_content_fetch": "saved_if_candidates",
"live_detail": "saved_if_candidates",
"match_finish": "saved_if_candidates",
"lottery_draw": "saved_if_candidates",
"lottery_draw_force": "saved_if_candidates",
"ai_comment_dispatch": "saved_if_candidates",
}
DEFAULT_ACTION_TIMEOUT = 600
ACTION_TIMEOUTS = {
"csl_match": 180,
"nba_match": 180,
"cba_match": 180,
"epl_match": 180,
"bundesliga_match": 180,
"laliga_match": 180,
"seriea_match": 180,
"ligue1_match": 180,
"ucl_match": 180,
"uel_match": 180,
"tennis_match": 180,
"esports_match": 180,
"sports_match": 180,
"live_detail": 180,
"match_finish": 180,
"lottery_draw": 180,
"lottery_draw_force": 180,
"article_content": 600,
"article_content_fetch": 900,
"league_news": 900,
"fifa_worldcup_news": 600,
"dqd_worldcup": 600,
"ai_comment_dispatch": 1800,
}
def get_action_timeout(action: str) -> int:
return ACTION_TIMEOUTS.get(str(action or "").strip(), DEFAULT_ACTION_TIMEOUT)
class CrawlerActionLock:
"""Cross-process non-blocking lock for one crawler action."""
def __init__(self, action: str, lock_dir: Path | None = None):
safe_action = "".join(ch if ch.isalnum() or ch in ("-", "_") else "_" for ch in str(action or "all"))
self.lock_dir = Path(lock_dir or os.environ.get("DQD_LOCK_DIR") or (Path(__file__).resolve().parent / "data" / "locks"))
self.path = self.lock_dir / f"{safe_action}.lock"
self._fp = None
self._locked = False
def acquire(self) -> bool:
self.lock_dir.mkdir(parents=True, exist_ok=True)
self._fp = open(self.path, "a+b")
try:
self._fp.seek(0)
if os.name == "nt":
import msvcrt
msvcrt.locking(self._fp.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(self._fp.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
self._locked = True
self._fp.seek(0)
self._fp.truncate()
self._fp.write(f"pid={os.getpid()} started_at={int(time.time())}\n".encode("utf-8"))
self._fp.flush()
return True
except (BlockingIOError, OSError):
self.release()
return False
def release(self):
if not self._fp:
return
try:
if self._locked and os.name == "nt":
import msvcrt
self._fp.seek(0)
msvcrt.locking(self._fp.fileno(), msvcrt.LK_UNLCK, 1)
elif self._locked:
import fcntl
fcntl.flock(self._fp.fileno(), fcntl.LOCK_UN)
finally:
self._fp.close()
self._fp = None
self._locked = False
def _build_summary_text(action: str, raw_result: dict) -> str:
if not isinstance(raw_result, dict):
return f"{action} 执行完成"
for key in ("summary_text", "summary", "msg", "error"):
value = str(raw_result.get(key) or "").strip()
if value:
return value
if "count" in raw_result:
return f"{action} 执行完成: {raw_result.get('count', 0)}"
return f"{action} 执行完成"
def _normalize_task_result(action: str, raw_result, execution_success: bool) -> dict:
if isinstance(raw_result, list):
raw_result = {"results": raw_result}
raw_result = raw_result if isinstance(raw_result, dict) else {}
results = raw_result.get("results")
if isinstance(results, list):
candidate_count = sum(int(item.get("candidate_count") or item.get("count") or 0) for item in results if isinstance(item, dict))
saved_count = sum(int(item.get("saved_count") or item.get("count") or 0) for item in results if isinstance(item, dict))
raw_success = all(bool(item.get("success", False)) for item in results) if results else bool(execution_success)
else:
candidate_count = int(raw_result.get("candidate_count") or 0)
saved_count = int(raw_result.get("saved_count") or 0)
if not candidate_count and "pending" in raw_result:
candidate_count = int(raw_result.get("pending") or 0)
if not candidate_count and "total" in raw_result:
candidate_count = int(raw_result.get("total") or 0)
if not candidate_count and "fail" in raw_result:
candidate_count = int(raw_result.get("count") or 0) + int(raw_result.get("fail") or 0)
if not candidate_count and "count" in raw_result:
candidate_count = int(raw_result.get("count") or 0)
if not saved_count and "count" in raw_result:
saved_count = int(raw_result.get("count") or 0)
raw_success = bool(raw_result.get("success", execution_success))
run_stats = ErrorCollector.get().get_run_stats()
return {
"action": action,
"task_name": raw_result.get("task") or action,
"command": "crawler",
"params": action,
"policy": TASK_ALERT_POLICY.get(action, "saved_required"),
"execution_success": bool(execution_success),
"success": raw_success,
"candidate_count": candidate_count,
"saved_count": saved_count,
"http_error_count": int(run_stats.get("http_error_count") or 0),
"severe_http_statuses": list(run_stats.get("severe_http_statuses") or []),
"error_message": str(raw_result.get("error") or ""),
"summary_text": _build_summary_text(action, raw_result),
}
async def cmd_standings():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
results = await runner.crawl_all_leagues("standings")
_print_results(results, "积分榜")
return results
async def cmd_schedule():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
results = await runner.crawl_all_leagues("schedule")
_print_results(results, "赛程")
return results
async def cmd_menu():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_match_menu()
if result["success"]:
print(f"✅ 比赛菜单采集成功: {result['count']} 条")
else:
print(f"❌ 比赛菜单采集失败: {result.get('error')}")
return result
async def cmd_news():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_news()
if result["success"]:
print(f"✅ 新闻采集成功: {result['count']} 条, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ 新闻采集失败: {result.get('error')}")
return result
async def cmd_league_news():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_league_news()
if result["success"]:
print(result.get("summary", f"✅ 联赛资讯采集成功: {result['count']} 条"))
else:
print(f"❌ 联赛资讯采集失败: {result.get('error')}")
return result
async def cmd_article_content():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_article_content()
if result["success"]:
print(result.get("summary", f"✅ 文章内容补全: {result['count']} 篇"))
else:
print(f"❌ 文章内容补全失败: {result.get('error')}")
return result
async def cmd_video():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_videos()
if result["success"]:
print(f"✅ 视频采集成功: {result['count']} 条, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ 视频采集失败: {result.get('error')}")
return result
async def cmd_content():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_article_details()
if result["success"]:
msg = result.get("msg", "")
if msg:
print(f"✅ {msg}")
else:
print(f"✅ 文章详情补全: {result['count']}/{result.get('total', 0)} 篇, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ 文章详情补全失败: {result.get('error')}")
return result
async def cmd_live():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_live()
if result["success"]:
print(f"✅ 实时比赛更新: {result['count']}/{result.get('pending', 0)} 场, {result.get('elapsed', 0):.1f}s")
if result.get("msg"):
print(f" {result['msg']}")
else:
print(f"❌ 实时比赛采集失败: {result.get('error')}")
return result
async def cmd_lottery():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_lottery()
if result["success"]:
msg = result.get("msg", "")
if msg:
print(f"✅ {msg}")
else:
print(f"✅ 六合彩开奖采集成功: {result['count']} 条, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ 六合彩开奖采集失败: {result.get('error')}")
return result
async def _cmd_page_match(crawl_func_name: str, label: str):
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await getattr(runner, crawl_func_name)()
if result["success"]:
print(f"✅ {label}赛程采集成功: 新增 {result.get('inserted', 0)}, 更新 {result.get('updated', 0)}, 未变 {result.get('unchanged', 0)}, 共 {result['count']} 场, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ {label}赛程采集失败: {result.get('error')}")
return result
async def cmd_csl_match():
return await _cmd_page_match("crawl_csl_matches", "中超")
async def cmd_nba_match():
return await _cmd_page_match("crawl_nba_matches", "NBA")
async def cmd_cba_match():
return await _cmd_page_match("crawl_cba_matches", "CBA")
async def cmd_epl_match():
return await _cmd_page_match("crawl_epl_matches", "英超")
async def cmd_bundesliga_match():
return await _cmd_page_match("crawl_bundesliga_matches", "德甲")
async def cmd_laliga_match():
return await _cmd_page_match("crawl_laliga_matches", "西甲")
async def cmd_seriea_match():
return await _cmd_page_match("crawl_seriea_matches", "意甲")
async def cmd_ligue1_match():
return await _cmd_page_match("crawl_ligue1_matches", "法甲")
async def cmd_ucl_match():
return await _cmd_page_match("crawl_ucl_matches", "欧冠")
async def cmd_uel_match():
return await _cmd_page_match("crawl_uel_matches", "欧联")
async def cmd_tennis_match():
return await _cmd_page_match("crawl_tennis_matches", "网球")
async def cmd_esports_match():
return await _cmd_page_match("crawl_esports_matches", "电竞")
async def cmd_sports_match():
return await _cmd_page_match("crawl_sports_matches", "体坛")
async def cmd_truth_social():
import asyncio
from truth_social import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_crypto_news():
import asyncio
from crypto_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_lottery_news():
import asyncio
from lottery_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_taiwan_lottery_news():
import asyncio
from taiwan_lottery_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_hkjc_lottery_news():
import asyncio
from hkjc_lottery_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_fifa_worldcup_news():
import asyncio
from fifa_worldcup_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
"prefix": cfg.prefix,
}
return await asyncio.to_thread(run, db_config)
async def cmd_dqd_worldcup():
import asyncio
from dqd_worldcup import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
"prefix": cfg.prefix,
}
return await asyncio.to_thread(run, db_config)
async def cmd_lottery_community():
import asyncio
from lottery_community import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
"prefix": cfg.prefix,
}
return await asyncio.to_thread(run, db_config)
async def cmd_article_content_fetch():
import asyncio
from article_fetcher import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_nba_news():
import asyncio
from nba_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_cba_news():
import asyncio
from cba_news import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
}
return await asyncio.to_thread(run, db_config)
async def cmd_ai_comment_dispatch():
import asyncio
from ai_comment_dispatch import run
from src.core.config import load_config
cfg = load_config().database
db_config = {
"host": cfg.host, "port": cfg.port, "user": cfg.username,
"password": cfg.password, "database": cfg.database, "charset": cfg.charset,
"prefix": cfg.prefix,
}
return await asyncio.to_thread(run, db_config)
async def cmd_match_data():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_match_data()
if result["success"]:
print(f"✅ 赛事数据同步成功: {result['count']} 场比赛, {result.get('rounds', 0)} 轮次, {result.get('elapsed', 0):.1f}s")
if result.get("msg"):
print(f" {result['msg']}")
else:
print(f"❌ 赛事数据同步失败: {result.get('error')}")
return result
async def cmd_match_finish():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_match_finish()
if result["success"]:
print(f"✅ 超时比赛收尾完成: {result['count']} 场, 详情成功 {result.get('detail_ok', 0)} 场, 跳过 {result.get('skipped', 0)} 场, {result.get('elapsed', 0):.1f}s")
if result.get("msg"):
print(f" {result['msg']}")
else:
print(f"❌ 超时比赛收尾失败: {result.get('error')}")
return result
async def cmd_live_detail():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_live_detail()
if result["success"]:
print(f"✅ 实时比赛详情更新: {result['count']}/{result.get('pending', 0)} 场, "
f"结束 {result.get('finished', 0)} 场, 失败 {result.get('failed', 0)} 场, "
f"文字直播 {result.get('live_text', 0)} 条, {result.get('elapsed', 0):.1f}s")
if result.get("msg"):
print(f" {result['msg']}")
else:
print(f"❌ 实时比赛详情更新失败: {result.get('error')}")
return result
async def cmd_lottery_draw():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_lottery_draw()
if result["success"]:
print(f"✅ 彩种开奖数据采集成功: {result['count']} 条, {result.get('elapsed', 0):.1f}s")
if result.get("msg"):
print(f" {result['msg']}")
else:
print(f"❌ 彩种开奖数据采集失败: {result.get('error')}")
return result
async def cmd_lottery_draw_force():
"""强制全量刷新所有彩种最新开奖数据（跳过时间检查，用于断采后补数据）"""
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_lottery_draw(force=True)
if result["success"]:
print(f"✅ 彩种开奖数据强制刷新成功: {result['count']} 条, {result.get('elapsed', 0):.1f}s")
else:
print(f"❌ 彩种开奖数据强制刷新失败: {result.get('error')}")
result["task"] = "lottery_draw_force"
return result
async def cmd_error_report():
"""企业微信发送未恢复的定时任务告警"""
from src.core.alert_manager import CrontabAlertManager
manager = CrontabAlertManager()
result = await manager.dispatch_open_alerts(limit=50)
print(result.get("summary_text", "告警分发完成"))
return result
async def cmd_all():
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
print("=" * 60)
print("采集积分榜")
print("=" * 60)
r1 = await runner.crawl_all_leagues("standings")
_print_results(r1, "积分榜")
print()
print("=" * 60)
print("采集赛程")
print("=" * 60)
r2 = await runner.crawl_all_leagues("schedule")
_print_results(r2, "赛程")
return {
"results": [
{"task": "standings_batch", "success": all(r.get("success", False) for r in r1), "count": sum(int(r.get("count", 0) or 0) for r in r1)},
{"task": "schedule_batch", "success": all(r.get("success", False) for r in r2), "count": sum(int(r.get("count", 0) or 0) for r in r2)},
]
}
async def cmd_single(season_id: int):
from src.scheduler.task_runner import TaskRunner
async with TaskRunner() as runner:
result = await runner.crawl_standings(season_id, f"season_{season_id}")
if result["success"]:
print(f"✅ 采集成功: {result['count']} 条, {result['elapsed']:.1f}s")
else:
print(f"❌ 采集失败: {result.get('error')}")
return result
async def cmd_cron():
from src.scheduler.cron_scheduler import CronScheduler
scheduler = CronScheduler()
await scheduler.start()
print("定时调度器已启动，按 Ctrl+C 停止")
try:
while True:
await asyncio.sleep(1)
except (KeyboardInterrupt, asyncio.CancelledError):
await scheduler.stop()
print("调度器已停止")
async def cmd_init_db():
from src.storage.database import Database
async with Database() as db:
await db.ensure_tables()
print("✅ 数据库表初始化完成")
return {"success": True, "candidate_count": 1, "saved_count": 1, "summary_text": "数据库表初始化完成"}
async def cmd_test():
"""测试各通道连通性"""
from src.engine.hybrid_engine import HybridEngine
print("=" * 60)
print("懂球帝爬虫连通性测试")
print("=" * 60)
async with HybridEngine() as engine:
# 测试 API 通道
print("\n1. 测试 API 通道 (积分榜 - 中超)...")
try:
start = time.time()
data = await engine.get_standings(26322)
elapsed = time.time() - start
rounds = data.get("content", {}).get("rounds", [])
if rounds:
teams = rounds[0].get("content", {}).get("data", [])
print(f" ✅ 成功! {len(teams)} 支球队, {elapsed:.2f}s")
else:
print(f" ⚠️ 返回空数据, {elapsed:.2f}s")
except Exception as e:
print(f" ❌ 失败: {e}")
# 测试比赛菜单
print("\n2. 测试比赛菜单...")
try:
start = time.time()
data = await engine.get_match_menu()
elapsed = time.time() - start
if data.get("errCode") == 0:
items = data.get("data", {}).get("list", [])
print(f" ✅ 成功! {len(items)} 个比赛类型, {elapsed:.2f}s")
else:
print(f" ⚠️ 返回错误: {data}, {elapsed:.2f}s")
except Exception as e:
print(f" ❌ 失败: {e}")
# 测试数据库
print("\n3. 测试数据库连接...")
try:
from src.storage.database import Database
async with Database() as db:
result = await db.fetchone("SELECT 1 AS ok")
if result and result.get("ok") == 1:
print(" ✅ 数据库连接正常")
else:
print(" ❌ 数据库连接异常")
except Exception as e:
print(f" ❌ 数据库连接失败: {e}")
print("\n" + "=" * 60)
print("引擎统计:")
stats = engine.stats
for ch, s in stats.get("channels", {}).items():
print(f" {ch}: 成功={s['success']}, 失败={s['fail']}")
print("=" * 60)
return {"success": True, "candidate_count": 3, "saved_count": 3, "summary_text": "连通性测试完成"}
def _parse_extra_args(argv):
"""解析 --log-id=N --crontab-id=N 参数"""
extra = {}
for arg in argv:
if arg.startswith('--log-id='):
extra['log_id'] = int(arg.split('=', 1)[1])
elif arg.startswith('--crontab-id='):
extra['crontab_id'] = int(arg.split('=', 1)[1])
return extra
class _CrontabLogWriter:
"""实时写入执行日志到数据库"""
def __init__(self, log_id: int, crontab_id: int):
self.log_id = log_id
self.crontab_id = crontab_id
self._conn = None
self._cfg = None
async def connect(self):
import aiomysql
self._cfg = load_config().database
self._conn = await aiomysql.connect(
host=self._cfg.host, port=self._cfg.port, user=self._cfg.username,
password=self._cfg.password, db=self._cfg.database, charset=self._cfg.charset,
)
async def flush_output(self, output: str, elapsed: float):
"""实时刷新 output 到日志表（status 保持 0=执行中）"""
async with self._conn.cursor() as cur:
await cur.execute(
f"UPDATE `{self._cfg.prefix}dev_crontab_log` SET output=%s, elapsed=%s WHERE id=%s",
(output[:5000], elapsed, self.log_id)
)
await self._conn.commit()
async def finish(self, success: bool, output: str, elapsed: float):
"""最终更新状态"""
async with self._conn.cursor() as cur:
status = 1 if success else 2
await cur.execute(
f"UPDATE `{self._cfg.prefix}dev_crontab_log` SET status=%s, output=%s, elapsed=%s WHERE id=%s",
(status, output[:5000], elapsed, self.log_id)
)
if success:
await cur.execute(
f"UPDATE `{self._cfg.prefix}dev_crontab` SET last_time=%s, `time`=%s, error='' WHERE id=%s",
(int(time.time()), elapsed, self.crontab_id)
)
else:
await cur.execute(
f"UPDATE `{self._cfg.prefix}dev_crontab` SET error=%s WHERE id=%s",
(output[:500], self.crontab_id)
)
await self._conn.commit()
async def close(self):
if self._conn:
self._conn.close()
def _print_results(results, task_name):
success = sum(1 for r in results if r.get("success"))
total = len(results)
print(f"\n{task_name}采集完成: {success}/{total} 成功")
for r in results:
s = "✅" if r.get("success") else "❌"
league = r.get("league", "?")
count = r.get("count", 0)
elapsed = r.get("elapsed", 0)
err = r.get("error", "")
if r.get("success"):
print(f" {s} {league}: {count} 条, {elapsed:.1f}s")
else:
print(f" {s} {league}: {err}")
def main():
load_config()
setup_logger()
if len(sys.argv) < 2:
print(__doc__)
sys.exit(0)
cmd = sys.argv[1].lower()
extra = _parse_extra_args(sys.argv[2:])
cmd_map = {
"standings": cmd_standings,
"schedule": cmd_schedule,
"menu": cmd_menu,
"news": cmd_news,
"league_news": cmd_league_news,
"article_content": cmd_article_content,
"video": cmd_video,
"content": cmd_content,
"live": cmd_live,
"lottery": cmd_lottery,
"match_data": cmd_match_data,
"csl_match": cmd_csl_match,
"nba_match": cmd_nba_match,
"cba_match": cmd_cba_match,
"epl_match": cmd_epl_match,
"bundesliga_match": cmd_bundesliga_match,
"laliga_match": cmd_laliga_match,
"seriea_match": cmd_seriea_match,
"ligue1_match": cmd_ligue1_match,
"ucl_match": cmd_ucl_match,
"uel_match": cmd_uel_match,
"tennis_match": cmd_tennis_match,
"esports_match": cmd_esports_match,
"sports_match": cmd_sports_match,
"truth_social": cmd_truth_social,
"crypto_news": cmd_crypto_news,
"lottery_news": cmd_lottery_news,
"taiwan_lottery_news": cmd_taiwan_lottery_news,
"hkjc_lottery_news": cmd_hkjc_lottery_news,
"fifa_worldcup_news": cmd_fifa_worldcup_news,
"dqd_worldcup": cmd_dqd_worldcup,
"lottery_community": cmd_lottery_community,
"nba_news": cmd_nba_news,
"cba_news": cmd_cba_news,
"ai_comment_dispatch": cmd_ai_comment_dispatch,
"article_content_fetch": cmd_article_content_fetch,
"match_finish": cmd_match_finish,
"live_detail": cmd_live_detail,
"lottery_draw": cmd_lottery_draw,
"lottery_draw_force": cmd_lottery_draw_force,
"error_report": cmd_error_report,
"all": cmd_all,
"cron": cmd_cron,
"init-db": cmd_init_db,
"test": cmd_test,
}
if cmd == "single":
plain_args = [a for a in sys.argv[2:] if not a.startswith('--')]
if not plain_args:
print("用法: python main.py single <season_id>")
sys.exit(1)
coro = cmd_single(int(plain_args[0]))
elif cmd in cmd_map:
coro = cmd_map[cmd]()
else:
print(f"未知命令: {cmd}")
print(__doc__)
sys.exit(1)
if 'log_id' in extra and 'crontab_id' in extra:
asyncio.run(_run_with_log(coro, extra['log_id'], extra['crontab_id'], cmd))
else:
result = asyncio.run(coro)
normalized = _normalize_task_result(cmd, result, True)
if not normalized.get("success", True):
sys.exit(1)
async def _run_with_log(coro, log_id: int, crontab_id: int, action: str):
"""带实时日志刷新的异步执行"""
from loguru import logger as _logger
writer = _CrontabLogWriter(log_id, crontab_id)
await asyncio.wait_for(writer.connect(), timeout=15)
action_lock = CrawlerActionLock(action)
if not action_lock.acquire():
if hasattr(coro, "close"):
coro.close()
summary = f"{action} 已有任务执行中，跳过本次"
await writer.finish(True, summary, 0)
await writer.close()
return
ErrorCollector.get().reset_run_stats()
log_buf = io.StringIO()
stdout_buf = io.StringIO()
start_time = time.time()
execution_success = True
task_done = False
task_result = None
sink_id = _logger.add(log_buf, format="{time:HH:mm:ss} | {level:<8} | {message}", level="INFO")
def get_output():
parts = []
s = stdout_buf.getvalue()
if s:
parts.append(s.strip())
l = log_buf.getvalue()
if l:
parts.append(l.strip())
return "\n".join(parts) if parts else ""
async def flush_loop():
last_hash = 0
while not task_done:
await asyncio.sleep(3)
current = get_output()
h = len(current)
if h != last_hash:
last_hash = h
elapsed = round(time.time() - start_time, 2)
try:
await writer.flush_output(current, elapsed)
except Exception:
pass
flush_task = asyncio.create_task(flush_loop())
try:
with redirect_stdout(stdout_buf):
task_result = await asyncio.wait_for(coro, timeout=get_action_timeout(action))
except asyncio.TimeoutError:
execution_success = False
task_result = {"success": False, "error": "timeout", "summary_text": f"{action} 执行超时"}
stdout_buf.write(f"\n错误: {action} 执行超过 {get_action_timeout(action)} 秒，已终止")
except Exception as e:
execution_success = False
task_result = {"success": False, "error": str(e), "summary_text": f"执行异常: {e}"}
stdout_buf.write(f"\n错误: {e}")
finally:
task_done = True
_logger.remove(sink_id)
await asyncio.sleep(0)
flush_task.cancel()
try:
await flush_task
except asyncio.CancelledError:
pass
elapsed = round(time.time() - start_time, 2)
output = get_output()
normalized = _normalize_task_result(action, task_result, execution_success)
try:
await writer.finish(execution_success, output or normalized["summary_text"] or '执行完成', elapsed)
try:
from src.core.alert_manager import CrontabAlertManager
await CrontabAlertManager().record_crawler_result(action, crontab_id, log_id, normalized, output)
except Exception as alert_error:
print(f"告警记录失败: {alert_error}")
finally:
action_lock.release()
await writer.close()
await ErrorCollector.get().close()
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/nba_news.py
-279
View File
@@ -1,279 +0,0 @@
"""
NBA新闻采集模块 - 从 Google News RSS 获取NBA新闻并写入 la_article 表
在 main.py 中通过 nba_news 命令调用
数据源（RSS）:
1. Google News 中文简体 - NBA（主）
重复判断: 以 source_url 为唯一键，重复则更新标题/摘要
"""
import json
import re
import time
import xml.etree.ElementTree as ET
from datetime import datetime
from email.utils import parsedate_to_datetime
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
# ─── 配置 ───
ARTICLE_CID = 17 # la_article_cate 中"NBA"的 id
RSS_FEEDS = [
{
"name": "NBA(中文)",
"url": "https://news.google.com/rss/search?q=NBA&hl=zh-CN&gl=CN&ceid=CN:zh-Hans",
"author_default": "Google News",
},
]
NS = {
"content": "http://purl.org/rss/1.0/modules/content/",
"media": "http://search.yahoo.com/mrss/",
"dc": "http://purl.org/dc/elements/1.1/",
}
def strip_html(text: str) -> str:
import html as html_mod
text = re.sub(r"<[^>]+>", "", text or "")
text = html_mod.unescape(text)
text = html_mod.unescape(text)
text = text.replace("\xa0", " ")
return text.strip()
def parse_rfc2822(date_str: str) -> tuple[int, str]:
"""解析 RSS pubDate (RFC 2822)，返回 (unix时间戳, 'Y-m-d H:i:s'格式字符串)"""
if not date_str:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = parsedate_to_datetime(date_str)
return int(dt.timestamp()), dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
# ─── RSS 采集 ───
def fetch_rss(session: requests.Session, feed: dict) -> list[dict]:
"""从单个 RSS 源拉取新闻，返回标准化列表"""
articles = []
url = feed["url"]
name = feed["name"]
try:
resp = session.get(url, timeout=30)
except Exception as e:
print(f" [WARN] {name} 请求失败: {e}")
return articles
if resp.status_code != 200:
print(f" [WARN] {name} HTTP {resp.status_code}")
return articles
try:
root = ET.fromstring(resp.content)
except ET.ParseError as e:
print(f" [WARN] {name} XML 解析失败: {e}")
return articles
items = root.findall(".//item")
for item in items:
title = (item.findtext("title", "") or "").strip()
link = (item.findtext("link", "") or "").strip()
if not title or not link:
continue
pub_date = item.findtext("pubDate", "")
desc_raw = item.findtext("description", "") or ""
desc = strip_html(desc_raw)[:255]
# Google News 标题格式: "标题 - 来源名"
author = feed.get("author_default", name)
if " - " in title:
parts = title.rsplit(" - ", 1)
if len(parts) == 2:
author = parts[1].strip()
title = parts[0].strip()
# 尝试提取图片
image = ""
media_el = item.find("media:content", NS)
if media_el is not None:
image = media_el.get("url", "")
if not image:
media_thumb = item.find("media:thumbnail", NS)
if media_thumb is not None:
image = media_thumb.get("url", "")
if not image:
img_match = re.search(r'<img[^>]+src=["\']([^"\']+)["\']', desc_raw)
if img_match:
image = img_match.group(1)
pub_ts, pub_fmt = parse_rfc2822(pub_date)
articles.append({
"title": title[:255],
"desc": desc,
"content": strip_html(desc_raw),
"source_url": link,
"image": image,
"author": author[:64],
"published_ts": pub_ts,
"published_at": pub_fmt,
"category": "NBA",
"ext": {
"source": "google_news",
"feed_name": name,
"feed_url": url,
},
})
return articles
# ─── 数据库同步 ───
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
"""同步到 la_article 表，返回 (总数, 新增数, 更新数)"""
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
if not urls:
return 0, 0, 0
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
# 按标题去重：同分类下相同标题的文章不再插入
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
if not source_url or not article.get("title"):
continue
create_ts = article.get("published_ts") or now
ext_json = json.dumps(article.get("ext", {}), ensure_ascii=False) if article.get("ext") else None
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title = %s, `desc` = %s, image = %s, author = %s,
category = %s, ext = %s, is_show = %s, update_time = %s
WHERE id = %s""",
(
article["title"][:255],
(article.get("desc") or "")[:255],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("category") or "")[:128],
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
# 标题去重：相同标题跳过
if article["title"][:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
article["title"][:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", ""),
(article.get("author") or "")[:64],
(article.get("published_at") or "")[:64],
(article.get("category") or "")[:128],
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
create_ts,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
existing_titles.add(article["title"][:255])
new_count += 1
return len(articles), new_count, update_count
# ─── 主入口 ───
def run(db_config: dict):
"""主入口，db_config 从 dongqiudi-crawler 配置中传入"""
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/xml, text/xml, application/rss+xml, */*",
})
all_articles = []
for feed in RSS_FEEDS:
print(f"=== {feed['name']} 采集 ===")
articles = fetch_rss(session, feed)
print(f" {feed['name']} 返回 {len(articles)} 条")
all_articles.extend(articles)
# 按 source_url 去重
seen = set()
deduped = []
for a in all_articles:
url = a.get("source_url", "")
if url and url not in seen:
seen.add(url)
deduped.append(a)
all_articles = deduped
print(f"去重后共 {len(all_articles)} 条")
if not all_articles:
print("所有数据源均无数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "所有数据源均无数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, all_articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/requirements.txt
-42
View File
@@ -1,42 +0,0 @@
# 懂球帝爬虫项目依赖
# HTTP 请求
aiohttp>=3.9.0
requests>=2.31.0
httpx>=0.27.0
curl_cffi>=0.7.0 # TLS 指纹模拟 (绕过 JA3 检测)
# 浏览器自动化
playwright>=1.42.0
playwright-stealth>=1.0.6 # Playwright 反检测
# 反爬绕过
fake-useragent>=1.5.0
cloudscraper>=1.2.71 # 绕过 Cloudflare
undetected-chromedriver>=3.5.5
scrapling>=0.4.9 # 动态网页正文抓取
browserforge>=1.2.4 # scrapling 静态抓取指纹依赖
# 代理管理
PySocks>=1.7.1
# 数据处理
pydantic>=2.6.0
pydantic-settings>=2.2.0
PyYAML>=6.0.1
# 数据库
pymysql>=1.1.0
aiomysql>=0.2.0
redis>=5.0.0
aioredis>=2.0.0
# 调度
APScheduler>=3.10.4
# 工具
loguru>=0.7.2 # 高级日志
tenacity>=8.2.0 # 重试机制
python-dotenv>=1.0.0
orjson>=3.9.0 # 高性能 JSON
beautifulsoup4>=4.12.0 # requests 回退抓取时解析 HTML
server/public/dongqiudi-crawler/scripts/crawl_article_detail.py
-135
View File
@@ -1,135 +0,0 @@
"""
懂球帝文章正文采集脚本
- 从数据库读取已入库但无正文的文章
- 请求文章详情页HTML，提取 div.con 正文内容
- 将正文HTML写入 la_article.content
用法:
python scripts/crawl_article_detail.py # 采集最多50篇
python scripts/crawl_article_detail.py --limit 200 # 采集最多200篇
"""
import asyncio
import random
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from bs4 import BeautifulSoup
from curl_cffi import requests as curl_requests
from loguru import logger
from src.core.config import get_config
from src.storage.database import Database
DETAIL_URL = "https://www.dongqiudi.com/articles/{article_id}"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Referer": "https://www.dongqiudi.com/",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
def fetch_article_html(article_id: int) -> str:
url = DETAIL_URL.format(article_id=article_id)
resp = curl_requests.get(
url, headers=HEADERS,
impersonate="chrome", timeout=20,
)
resp.raise_for_status()
return resp.text
def extract_content(html: str) -> str:
soup = BeautifulSoup(html, "html.parser")
con_div = soup.select_one("div.con")
if not con_div:
return ""
inner = con_div.find("div", style=lambda v: v and "display:none" in v.replace(" ", ""))
target = inner if inner else con_div
for img in target.find_all("img"):
src = img.get("data-src") or img.get("orig-src") or img.get("src", "")
if src:
img["src"] = src
for attr in ["data-src", "orig-src", "data-width", "data-height"]:
if img.has_attr(attr):
del img[attr]
return str(target.decode_contents()).strip()
async def run(limit: int = 50):
cfg = get_config()
db = Database()
try:
articles = await db.get_articles_without_content(limit=limit)
logger.info(f"待采集正文: {len(articles)} 篇")
if not articles:
logger.info("所有文章已有正文，无需采集")
return
success_count = 0
fail_count = 0
for i, art in enumerate(articles, 1):
article_id = art["article_id"]
title = art["title"][:30]
logger.info(f"[{i}/{len(articles)}] article_id={article_id} {title}...")
try:
html = fetch_article_html(article_id)
content = extract_content(html)
if not content:
logger.warning(f" 正文为空，跳过")
fail_count += 1
continue
await db.update_article_content(article_id, content)
success_count += 1
logger.info(f" 正文长度: {len(content)} 字符 ✓")
except Exception as e:
logger.error(f" 采集失败: {e}")
fail_count += 1
delay = random.uniform(1.5, 3.0)
time.sleep(delay)
logger.info(f"✅ 正文采集完成: 成功 {success_count}, 失败 {fail_count}")
await db.log_crawl(
crawl_type="article_detail",
target="dongqiudi_article_html",
success=True,
elapsed=0,
record_count=success_count,
)
except Exception as e:
logger.error(f"采集失败: {e}", exc_info=True)
raise
finally:
await db.close()
def main():
import argparse
parser = argparse.ArgumentParser(description="懂球帝文章正文采集")
parser.add_argument("--limit", type=int, default=50, help="最多采集篇数 (默认50)")
args = parser.parse_args()
logger.info("=" * 60)
logger.info("懂球帝文章正文采集")
logger.info(f"最多采集 {args.limit} 篇")
logger.info("=" * 60)
asyncio.run(run(args.limit))
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/scripts/crawl_articles.py
-180
View File
@@ -1,180 +0,0 @@
"""
懂球帝文章资讯采集脚本
- 通过 /api/app/tabs/iphone/{tab_id}.json 接口获取文章列表
- 支持多Tab采集 + 分页翻页
- 按 article_id 去重，已有数据跳过
用法:
python scripts/crawl_articles.py # 采集所有Tab第1页
python scripts/crawl_articles.py --pages 5 # 每个Tab采集5页
python scripts/crawl_articles.py --tabs 1,3,5 # 只采集指定Tab
"""
import asyncio
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from curl_cffi import requests as curl_requests
from loguru import logger
from src.core.config import get_config
from src.storage.database import Database
API_URL = "https://www.dongqiudi.com/api/app/tabs/iphone/{tab_id}.json"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Referer": "https://www.dongqiudi.com/",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
TAB_CONFIG = {
1: "头条",
3: "英超",
4: "意甲",
5: "西甲",
6: "德甲",
}
def fetch_tab_page(tab_id: int, page_url: str = "") -> dict:
if not page_url:
page_url = API_URL.format(tab_id=tab_id)
resp = curl_requests.get(
page_url, headers=HEADERS,
impersonate="chrome", timeout=20,
)
resp.raise_for_status()
return resp.json()
def collect_tab(tab_id: int, max_pages: int = 1) -> list:
all_articles = []
page_url = ""
for page_idx in range(1, max_pages + 1):
label = TAB_CONFIG.get(tab_id, f"Tab{tab_id}")
logger.info(f"[{label}] 第{page_idx}页 请求...")
try:
data = fetch_tab_page(tab_id, page_url)
except Exception as e:
logger.error(f"[{label}] 第{page_idx}页 请求失败: {e}")
break
articles = data.get("articles", [])
logger.info(f"[{label}] 第{page_idx}页 返回 {len(articles)} 条文章")
if not articles:
break
all_articles.extend(articles)
next_url = data.get("next", "")
if not next_url:
logger.info(f"[{label}] 无下一页，停止")
break
page_url = next_url
time.sleep(2)
return all_articles
async def run(tab_ids: list, max_pages: int = 1):
cfg = get_config()
db = Database()
try:
existing_ids = await db.get_existing_article_ids()
cate_map = await db.get_article_cate_map()
logger.info(f"数据库已有 {len(existing_ids)} 条文章, {len(cate_map)} 个分类")
total_new = 0
total_skip = 0
for tab_id in tab_ids:
label = TAB_CONFIG.get(tab_id, f"Tab{tab_id}")
cid = cate_map.get(label, 0)
if not cid:
logger.warning(f"[{label}] 未找到对应分类，使用 cid=0")
all_articles = collect_tab(tab_id, max_pages)
unique = {}
for a in all_articles:
aid = int(a.get("id", 0))
if aid and aid not in unique:
unique[aid] = a
new_items = [a for aid, a in unique.items() if aid not in existing_ids]
skip_count = len(unique) - len(new_items)
logger.info(f"[{label}] 去重: {len(unique)} 条唯一, 新增: {len(new_items)}, 跳过: {skip_count}")
if new_items:
count = await db.upsert_articles(new_items, cid=cid)
total_new += count
existing_ids.update(int(a["id"]) for a in new_items)
total_skip += skip_count
time.sleep(1)
logger.info(f"✅ 采集完成: 新增 {total_new} 条, 跳过 {total_skip} 条")
await db.log_crawl(
crawl_type="articles",
target="dongqiudi_tabs_api",
success=True,
elapsed=0,
record_count=total_new,
)
except Exception as e:
logger.error(f"采集失败: {e}", exc_info=True)
try:
await db.log_crawl(
crawl_type="articles",
target="dongqiudi_tabs_api",
success=False,
elapsed=0,
error=str(e),
)
except Exception:
pass
raise
finally:
await db.close()
def main():
import argparse
parser = argparse.ArgumentParser(
description="懂球帝文章资讯采集",
epilog="示例:\n"
" python crawl_articles.py # 所有Tab各1页\n"
" python crawl_articles.py --pages 5 # 每Tab采5页\n"
" python crawl_articles.py --tabs 1,3,5 # 指定Tab\n",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("--pages", type=int, default=1, help="每个Tab采集页数 (默认1)")
parser.add_argument("--tabs", type=str, default="", help="Tab ID列表,逗号分隔 (默认全部)")
args = parser.parse_args()
if args.tabs:
tab_ids = [int(t.strip()) for t in args.tabs.split(",") if t.strip()]
else:
tab_ids = list(TAB_CONFIG.keys())
tab_names = [TAB_CONFIG.get(t, f"Tab{t}") for t in tab_ids]
logger.info("=" * 60)
logger.info("懂球帝文章资讯采集")
logger.info(f"Tabs: {', '.join(tab_names)}, 每Tab {args.pages} 页")
logger.info("=" * 60)
asyncio.run(run(tab_ids, args.pages))
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/scripts/crawl_live_detail.py
-238
View File
@@ -1,238 +0,0 @@
"""
懂球帝比赛实时详情采集脚本
- 每5秒轮询 la_match 表，找出已到开赛时间但未结束的比赛
- 异步并发采集每场比赛的 liveDetail 页面
- 解析 __NUXT__ 数据，更新 la_match 比分/状态，写入 la_match_data
- 每个异步任务间隔5-10秒随机
- 比赛结束后自动停止该场采集
"""
import asyncio
import random
import signal
import sys
import time
from datetime import datetime
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from curl_cffi import requests as curl_requests
from loguru import logger
from src.core.config import get_config
from src.storage.database import Database
from src.parser.nuxt_parser import parse_nuxt_match_sample
LIVE_DETAIL_URL = "https://www.dongqiudi.com/liveDetail/{match_id}"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Referer": "https://www.dongqiudi.com/",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
STATUS_MAP = {"Fixture": 0, "Playing": 1, "Played": 2}
# 正在采集中的 match_id 集合，防止重复调度
_active_tasks: dict = {} # match_id -> asyncio.Task
# 全局停止信号
_stop_event = asyncio.Event()
def fetch_live_detail(match_id: int) -> str:
"""同步请求 liveDetail 页面（在线程池中执行）"""
url = LIVE_DETAIL_URL.format(match_id=match_id)
resp = curl_requests.get(url, headers=HEADERS, impersonate="chrome", timeout=20)
resp.raise_for_status()
return resp.text
async def crawl_single_match(match_id: int, match_info: dict, db: Database):
"""采集单场比赛的实时数据，循环直到比赛结束"""
home = match_info.get("home_team", "")
away = match_info.get("away_team", "")
league = match_info.get("league_name", "")
logger.info(f"[开始采集] match_id={match_id} {league} {home} vs {away}")
consecutive_errors = 0
max_errors = 5
while not _stop_event.is_set():
try:
loop = asyncio.get_event_loop()
html = await loop.run_in_executor(None, fetch_live_detail, match_id)
match_sample = parse_nuxt_match_sample(html)
if not match_sample:
logger.warning(f"[{match_id}] 解析 __NUXT__ 失败，跳过本轮")
consecutive_errors += 1
if consecutive_errors >= max_errors:
logger.error(f"[{match_id}] 连续 {max_errors} 次失败，停止采集")
break
await asyncio.sleep(random.uniform(5, 10))
continue
consecutive_errors = 0
# 提取关键数据
status_str = str(match_sample.get("status", ""))
status_int = STATUS_MAP.get(status_str, 0)
fs_a = match_sample.get("fs_A", "")
fs_b = match_sample.get("fs_B", "")
home_score = int(fs_a) if fs_a and str(fs_a).isdigit() else 0
away_score = int(fs_b) if fs_b and str(fs_b).isdigit() else 0
hts_a = str(match_sample.get("hts_A", ""))
hts_b = str(match_sample.get("hts_B", ""))
half_score = f"{hts_a}-{hts_b}" if hts_a and hts_b and hts_a != "" and hts_b != "" else ""
minute = str(match_sample.get("minute", ""))
minute_period = str(match_sample.get("minute_period", ""))
current_minute = minute_period if minute_period else minute
logger.info(
f"[{match_id}] {league} {home} {home_score}-{away_score} {away} | "
f"status={status_str} minute={current_minute}"
)
# 更新 la_match
await db.update_match_live(
match_id=match_id,
status=status_int,
home_score=home_score,
away_score=away_score,
half_score=half_score,
current_minute=current_minute,
)
# 写入 la_match_data
await db.upsert_match_data(match_sample)
# 比赛结束
if status_str == "Played":
logger.info(f"[{match_id}] 比赛已结束: {home} {home_score}-{away_score} {away}")
break
except asyncio.CancelledError:
logger.info(f"[{match_id}] 任务被取消")
break
except Exception as e:
consecutive_errors += 1
logger.error(f"[{match_id}] 采集异常: {e}")
if consecutive_errors >= max_errors:
logger.error(f"[{match_id}] 连续 {max_errors} 次异常，停止采集")
break
# 随机间隔 5-10 秒
delay = random.uniform(5, 10)
try:
await asyncio.wait_for(_stop_event.wait(), timeout=delay)
break # stop_event 被设置，退出
except asyncio.TimeoutError:
pass # 正常超时，继续下一轮
logger.info(f"[结束采集] match_id={match_id} {league} {home} vs {away}")
async def poll_loop(db: Database, interval: int = 5):
"""主轮询循环：每 interval 秒检查一次需要采集的比赛"""
logger.info(f"启动轮询循环，间隔 {interval} 秒")
while not _stop_event.is_set():
try:
now_ts = int(time.time())
matches = await db.get_live_matches(now_ts)
# 清理已完成的任务
done_ids = [mid for mid, task in _active_tasks.items() if task.done()]
for mid in done_ids:
del _active_tasks[mid]
if matches:
new_count = 0
for m in matches:
mid = int(m["match_id"])
if mid in _active_tasks:
continue # 已在采集中
task = asyncio.create_task(crawl_single_match(mid, m, db))
_active_tasks[mid] = task
new_count += 1
# 每启动一个任务间隔 1-2 秒，避免瞬间并发太多
if new_count > 0:
await asyncio.sleep(random.uniform(1, 2))
if new_count > 0:
logger.info(f"新启动 {new_count} 个采集任务，当前活跃: {len(_active_tasks)}")
else:
active_count = len(_active_tasks)
if active_count > 0:
logger.debug(f"无新比赛，当前活跃任务: {active_count}")
except Exception as e:
logger.error(f"轮询异常: {e}")
# 等待 interval 秒或停止信号
try:
await asyncio.wait_for(_stop_event.wait(), timeout=interval)
break
except asyncio.TimeoutError:
pass
# 停止所有活跃任务
logger.info(f"停止所有活跃任务 ({len(_active_tasks)} 个)...")
for mid, task in _active_tasks.items():
task.cancel()
if _active_tasks:
await asyncio.gather(*_active_tasks.values(), return_exceptions=True)
_active_tasks.clear()
async def run(interval: int = 5):
db = Database()
try:
await db.connect()
logger.info("数据库连接成功")
await poll_loop(db, interval)
finally:
await db.close()
logger.info("数据库连接已关闭")
def main():
import argparse
parser = argparse.ArgumentParser(description="懂球帝比赛实时详情采集")
parser.add_argument("--interval", type=int, default=5, help="轮询间隔秒数 (默认5)")
args = parser.parse_args()
logger.info("=" * 60)
logger.info("懂球帝比赛实时详情采集 (liveDetail)")
logger.info(f"轮询间隔: {args.interval} 秒")
logger.info("按 Ctrl+C 停止")
logger.info("=" * 60)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
def handle_signal():
logger.info("收到停止信号...")
_stop_event.set()
try:
loop.add_signal_handler(signal.SIGINT, handle_signal)
loop.add_signal_handler(signal.SIGTERM, handle_signal)
except NotImplementedError:
pass # Windows 不支持 add_signal_handler
try:
loop.run_until_complete(run(interval=args.interval))
except KeyboardInterrupt:
logger.info("收到 Ctrl+C，正在停止...")
_stop_event.set()
loop.run_until_complete(asyncio.sleep(1))
finally:
loop.close()
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/scripts/crawl_matches.py
-187
View File
@@ -1,187 +0,0 @@
"""
懂球帝比赛数据采集脚本
- 通过 /api/data/tab/new/important 接口获取比赛记录
- 按周循环翻页 (start + 7天)
- 按天匹配，已有数据跳过
- 每天执行一次
"""
import asyncio
import sys
import time
from datetime import datetime, timedelta
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from curl_cffi import requests as curl_requests
from loguru import logger
from src.core.config import get_config
from src.storage.database import Database
API_URL = "https://www.dongqiudi.com/api/data/tab/new/important"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Referer": "https://www.dongqiudi.com/",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
def fetch_page(start_str: str) -> dict:
params = {
"start": f"{start_str}next",
"init": "1",
"platform": "www",
}
resp = curl_requests.get(
API_URL, params=params, headers=HEADERS,
impersonate="chrome", timeout=20,
)
resp.raise_for_status()
return resp.json()
def collect_range(start_date: datetime, end_date: datetime) -> list:
all_items = []
current = start_date
page = 0
while current < end_date:
page += 1
start_str = current.strftime("%Y-%m-%d") + " 00:00:00"
logger.info(f"[第{page}页] 请求: start={start_str}")
try:
data = fetch_page(start_str)
except Exception as e:
logger.error(f"请求失败: {e}")
current += timedelta(days=7)
time.sleep(2)
continue
items = data.get("list", [])
logger.info(f" 返回 {len(items)} 条记录")
for item in items:
all_items.append(item)
current += timedelta(days=7)
time.sleep(3)
return all_items
async def run(start_date: datetime, end_date: datetime):
cfg = get_config()
db = Database()
try:
existing_ids = await db.get_existing_match_ids()
logger.info(f"数据库已有 {len(existing_ids)} 条比赛记录(有match_id)")
days = (end_date - start_date).days
logger.info(f"采集范围: {start_date.strftime('%Y-%m-%d')} ~ {end_date.strftime('%Y-%m-%d')} ({days}天)")
all_items = collect_range(start_date, end_date)
unique_items = {}
for item in all_items:
mid = int(item["match_id"])
if mid not in unique_items:
unique_items[mid] = item
logger.info(f"去重后共 {len(unique_items)} 条唯一比赛")
new_items = [item for mid, item in unique_items.items() if mid not in existing_ids]
logger.info(f"新增比赛: {new_items.__len__()} 条 (跳过已有 {len(unique_items) - len(new_items)} 条)")
if not new_items:
logger.info("没有新比赛需要入库")
return
fixture_items = [i for i in new_items if i.get("status") == "Fixture"]
played_items = [i for i in new_items if i.get("status") == "Played"]
logger.info(f" 未开始: {len(fixture_items)} 条, 已结束: {len(played_items)} 条")
count = await db.upsert_la_match(new_items)
logger.info(f"✅ 入库完成: {count} 条比赛记录")
await db.log_crawl(
crawl_type="match_important",
target=API_URL,
success=True,
elapsed=0,
record_count=count,
)
except Exception as e:
logger.error(f"采集失败: {e}", exc_info=True)
try:
await db.log_crawl(
crawl_type="match_important",
target=API_URL,
success=False,
elapsed=0,
error=str(e),
)
except Exception:
pass
raise
finally:
await db.close()
def parse_date(s: str) -> datetime:
for fmt in ("%Y-%m-%d", "%Y/%m/%d", "%Y%m%d"):
try:
return datetime.strptime(s, fmt)
except ValueError:
continue
raise ValueError(f"无法解析日期: {s}，支持格式: YYYY-MM-DD / YYYY/MM/DD / YYYYMMDD")
def main():
import argparse
parser = argparse.ArgumentParser(
description="懂球帝比赛数据采集",
epilog="示例:\n"
" python crawl_matches.py --weeks 4 # 从今天起采集4周\n"
" python crawl_matches.py --start 2026-01-01 --end 2026-02-01 # 指定时间段\n"
" python crawl_matches.py --start 2026-01-01 # 从指定日期到今天\n",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("--weeks", type=int, default=None, help="从今天起采集未来N周 (默认4)")
parser.add_argument("--start", type=str, default=None, help="起始日期 (YYYY-MM-DD)")
parser.add_argument("--end", type=str, default=None, help="结束日期 (YYYY-MM-DD)，不指定则到今天")
args = parser.parse_args()
today = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)
if args.start:
start_date = parse_date(args.start)
if args.end:
end_date = parse_date(args.end)
else:
end_date = today + timedelta(days=1)
else:
weeks = args.weeks or 4
start_date = today
end_date = today + timedelta(weeks=weeks)
if start_date >= end_date:
logger.error(f"起始日期 {start_date.strftime('%Y-%m-%d')} >= 结束日期 {end_date.strftime('%Y-%m-%d')}")
sys.exit(1)
logger.info("=" * 60)
logger.info("懂球帝比赛数据采集 (important)")
logger.info(f"{start_date.strftime('%Y-%m-%d')} ~ {end_date.strftime('%Y-%m-%d')}")
logger.info("=" * 60)
asyncio.run(run(start_date, end_date))
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/scripts/docker_task_runner.py
-439
View File
@@ -1,439 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import os
import shlex
import socket
import subprocess
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Iterable, Optional
import pymysql
import yaml
APP_DIR = Path(__file__).resolve().parents[1]
if str(APP_DIR) not in sys.path:
sys.path.insert(0, str(APP_DIR))
from main import get_action_timeout # noqa: E402
from src.core.config import load_config # noqa: E402
STATUS_RUNNING = 0
STATUS_SUCCESS = 1
STATUS_FAILED = 2
STATUS_SKIPPED = 3
OUTPUT_LIMIT = 5000
@dataclass
class CrawlerTask:
task_key: str
task_name: str
action: str
cron_expression: str
active: bool = True
timeout: Optional[int] = None
@classmethod
def from_dict(cls, row: dict) -> "CrawlerTask":
action = str(row.get("action") or row.get("params") or "").strip()
return cls(
task_key=str(row.get("key") or row.get("task_key") or action).strip(),
task_name=str(row.get("name") or row.get("task_name") or action).strip(),
action=action,
cron_expression=str(row.get("cron") or row.get("cron_expression") or "* * * * *").strip(),
active=bool(row.get("active", True)),
timeout=int(row["timeout"]) if row.get("timeout") not in (None, "") else None,
)
def resolved_timeout(self) -> int:
return int(self.timeout or get_action_timeout(self.action))
@dataclass
class CommandResult:
exit_code: int
output: str
elapsed: float
timed_out: bool = False
@dataclass
class TaskRunResult:
log_id: int
status: int
output: str
elapsed: float
error_message: str = ""
class TaskLock:
def __init__(self, action: str, lock_dir: Path):
safe_action = "".join(ch if ch.isalnum() or ch in ("-", "_") else "_" for ch in str(action or "all"))
self.lock_dir = Path(lock_dir)
self.path = self.lock_dir / f"{safe_action}.lock"
self._fp = None
self._locked = False
def acquire(self) -> bool:
self.lock_dir.mkdir(parents=True, exist_ok=True)
self._fp = open(self.path, "a+b")
try:
self._fp.seek(0)
if os.name == "nt":
import msvcrt
msvcrt.locking(self._fp.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(self._fp.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
self._locked = True
self._fp.seek(0)
self._fp.truncate()
self._fp.write(f"pid={os.getpid()} started_at={int(time.time())}\n".encode("utf-8"))
self._fp.flush()
return True
except (BlockingIOError, OSError):
self.release()
return False
def release(self) -> None:
if not self._fp:
return
try:
if self._locked:
if os.name == "nt":
import msvcrt
self._fp.seek(0)
msvcrt.locking(self._fp.fileno(), msvcrt.LK_UNLCK, 1)
else:
import fcntl
fcntl.flock(self._fp.fileno(), fcntl.LOCK_UN)
finally:
self._locked = False
self._fp.close()
self._fp = None
class TaskLogStore:
def __init__(self):
cfg = load_config().database
self.prefix = cfg.prefix
self.conn = pymysql.connect(
host=cfg.host,
port=cfg.port,
user=cfg.username,
password=cfg.password,
database=cfg.database,
charset=cfg.charset,
autocommit=True,
)
self.ensure_table()
@property
def table(self) -> str:
return f"`{self.prefix}crawler_task_log`"
def close(self) -> None:
self.conn.close()
def ensure_table(self) -> None:
sql = f"""
CREATE TABLE IF NOT EXISTS {self.table} (
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
`task_key` VARCHAR(100) NOT NULL DEFAULT '' COMMENT '任务标识',
`task_name` VARCHAR(100) NOT NULL DEFAULT '' COMMENT '任务名称',
`action` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '爬虫动作',
`cron_expression` VARCHAR(64) NOT NULL DEFAULT '' COMMENT 'cron规则',
`source` VARCHAR(32) NOT NULL DEFAULT 'docker' COMMENT '来源',
`container_id` VARCHAR(128) NOT NULL DEFAULT '' COMMENT '容器ID',
`status` TINYINT UNSIGNED NOT NULL DEFAULT 0 COMMENT '0执行中 1成功 2失败 3跳过',
`trigger_type` TINYINT UNSIGNED NOT NULL DEFAULT 1 COMMENT '1自动 2手动',
`output` TEXT NULL COMMENT '执行输出',
`error_message` TEXT NULL COMMENT '错误信息',
`elapsed` DECIMAL(10,2) NOT NULL DEFAULT 0.00 COMMENT '耗时秒',
`started_at` INT UNSIGNED NOT NULL DEFAULT 0 COMMENT '开始时间',
`finished_at` INT UNSIGNED NOT NULL DEFAULT 0 COMMENT '结束时间',
`create_time` INT UNSIGNED NOT NULL DEFAULT 0,
`update_time` INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
KEY `idx_action_status` (`action`, `status`),
KEY `idx_create_time` (`create_time`),
KEY `idx_task_key` (`task_key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Docker爬虫任务执行日志'
"""
with self.conn.cursor() as cur:
cur.execute(sql)
def create_running(self, task: CrawlerTask) -> int:
now = int(time.time())
with self.conn.cursor() as cur:
cur.execute(
f"""
INSERT INTO {self.table}
(`task_key`, `task_name`, `action`, `cron_expression`, `source`, `container_id`,
`status`, `trigger_type`, `output`, `error_message`, `elapsed`,
`started_at`, `finished_at`, `create_time`, `update_time`)
VALUES (%s, %s, %s, %s, 'docker', %s, %s, 1, '', '', 0, %s, 0, %s, %s)
""",
(
task.task_key,
task.task_name,
task.action,
task.cron_expression,
socket.gethostname(),
STATUS_RUNNING,
now,
now,
now,
),
)
return int(cur.lastrowid)
def update_progress(self, log_id: int, output: str, elapsed: float) -> None:
with self.conn.cursor() as cur:
cur.execute(
f"UPDATE {self.table} SET output=%s, elapsed=%s, update_time=%s WHERE id=%s",
(output[-OUTPUT_LIMIT:], round(elapsed, 2), int(time.time()), log_id),
)
def finish(self, log_id: int, status: int, output: str, elapsed: float, error_message: str = "") -> None:
now = int(time.time())
with self.conn.cursor() as cur:
cur.execute(
f"""
UPDATE {self.table}
SET status=%s, output=%s, error_message=%s, elapsed=%s, finished_at=%s, update_time=%s
WHERE id=%s
""",
(status, output[-OUTPUT_LIMIT:], error_message[:2000], round(elapsed, 2), now, now, log_id),
)
def load_tasks(path: Path) -> list[CrawlerTask]:
raw = yaml.safe_load(Path(path).read_text(encoding="utf-8")) or {}
rows = raw if isinstance(raw, list) else raw.get("tasks", [])
return [CrawlerTask.from_dict(row) for row in rows]
def render_crontab(tasks: Iterable[CrawlerTask], python_bin: str = "/usr/local/bin/python", app_dir: str = "/app") -> str:
lines = [
"SHELL=/bin/bash",
"BASH_ENV=/etc/sport-era-crawler/env.sh",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"",
]
for task in tasks:
if not task.active:
continue
args = [
python_bin,
"scripts/docker_task_runner.py",
"run",
task.action,
"--task-key",
task.task_key,
"--task-name",
task.task_name,
"--cron-expression",
task.cron_expression,
]
if task.timeout:
args.extend(["--timeout", str(task.timeout)])
cmd = " ".join(shlex.quote(str(arg)) for arg in args)
lines.append(f"{task.cron_expression} root cd {shlex.quote(app_dir)} && {cmd}")
lines.append("")
return "\n".join(lines)
def run_main_action(action: str, timeout: int, output_callback: Callable[[str], None]) -> CommandResult:
start = time.monotonic()
env = os.environ.copy()
env["PYTHONUNBUFFERED"] = "1"
proc = subprocess.Popen(
[sys.executable, "main.py", action],
cwd=str(APP_DIR),
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
encoding="utf-8",
errors="replace",
env=env,
)
output_parts: list[str] = []
timed_out = False
if os.name == "nt":
try:
stdout, _ = proc.communicate(timeout=timeout)
stdout = stdout or ""
output_parts.append(stdout)
if stdout:
output_callback(stdout)
except subprocess.TimeoutExpired:
timed_out = True
proc.kill()
stdout, _ = proc.communicate()
stdout = stdout or ""
output_parts.append(stdout)
if stdout:
output_callback(stdout)
else:
import selectors
assert proc.stdout is not None
selector = selectors.DefaultSelector()
selector.register(proc.stdout, selectors.EVENT_READ)
try:
while proc.poll() is None:
remaining = timeout - (time.monotonic() - start)
if remaining <= 0:
timed_out = True
proc.kill()
break
for key, _ in selector.select(timeout=min(1.0, remaining)):
line = key.fileobj.readline()
if line:
output_parts.append(line)
output_callback(line)
rest = proc.stdout.read()
if rest:
output_parts.append(rest)
output_callback(rest)
finally:
selector.close()
elapsed = round(time.monotonic() - start, 2)
return CommandResult(
exit_code=proc.returncode if proc.returncode is not None else -1,
output="".join(output_parts),
elapsed=elapsed,
timed_out=timed_out,
)
def run_task(
task: CrawlerTask,
store=None,
lock_dir: Optional[Path] = None,
command_runner: Callable[[str, int, Callable[[str], None]], CommandResult] = run_main_action,
flush_interval: float = 5.0,
) -> TaskRunResult:
if store is None:
store = TaskLogStore()
if hasattr(store, "ensure_table"):
store.ensure_table()
log_id = int(store.create_running(task))
start = time.monotonic()
lock = TaskLock(task.action, Path(lock_dir or os.environ.get("DQD_LOCK_DIR") or APP_DIR / "data" / "locks"))
output_parts: list[str] = []
last_flush = 0.0
def append_output(text: str) -> None:
nonlocal last_flush
output_parts.append(text)
now = time.monotonic()
elapsed = round(now - start, 2)
if now - last_flush >= flush_interval:
store.update_progress(log_id, "".join(output_parts), elapsed)
last_flush = now
if not lock.acquire():
output = f"{task.action} 已有任务执行中，跳过本次"
store.finish(log_id, STATUS_SKIPPED, output, 0, "")
return TaskRunResult(log_id, STATUS_SKIPPED, output, 0, "")
try:
result = command_runner(task.action, task.resolved_timeout(), append_output)
output = result.output or "".join(output_parts) or "执行完成"
if result.timed_out:
status = STATUS_FAILED
error = f"{task.action} 执行超时，超过 {task.resolved_timeout()} 秒"
elif result.exit_code == 0:
status = STATUS_SUCCESS
error = ""
else:
status = STATUS_FAILED
error = f"{task.action} 执行失败，退出码 {result.exit_code}"
store.finish(log_id, status, output, result.elapsed, error)
return TaskRunResult(log_id, status, output, result.elapsed, error)
except Exception as exc:
elapsed = round(time.monotonic() - start, 2)
output = "".join(output_parts) or str(exc)
store.finish(log_id, STATUS_FAILED, output, elapsed, str(exc))
return TaskRunResult(log_id, STATUS_FAILED, output, elapsed, str(exc))
finally:
lock.release()
def build_task_from_args(args) -> CrawlerTask:
if args.tasks:
for task in load_tasks(Path(args.tasks)):
if task.action == args.action:
if args.timeout:
task.timeout = args.timeout
return task
return CrawlerTask(
task_key=args.task_key or args.action,
task_name=args.task_name or args.action,
action=args.action,
cron_expression=args.cron_expression or "",
active=True,
timeout=args.timeout,
)
def parse_args(argv: Optional[list[str]] = None):
parser = argparse.ArgumentParser(description="Sport Era Docker crawler task runner")
sub = parser.add_subparsers(dest="command", required=True)
run = sub.add_parser("run")
run.add_argument("action")
run.add_argument("--task-key", default="")
run.add_argument("--task-name", default="")
run.add_argument("--cron-expression", default="")
run.add_argument("--timeout", type=int, default=None)
run.add_argument("--tasks", default="")
render = sub.add_parser("render-cron")
render.add_argument("--tasks", required=True)
render.add_argument("--output", required=True)
render.add_argument("--python-bin", default="/usr/local/bin/python")
render.add_argument("--app-dir", default="/app")
return parser.parse_args(argv)
def main(argv: Optional[list[str]] = None) -> int:
args = parse_args(argv)
if args.command == "render-cron":
tasks = load_tasks(Path(args.tasks))
content = render_crontab(tasks, python_bin=args.python_bin, app_dir=args.app_dir)
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(content, encoding="utf-8")
output.chmod(0o644)
return 0
if args.command == "run":
store = TaskLogStore()
try:
result = run_task(build_task_from_args(args), store=store)
return 0 if result.status in (STATUS_SUCCESS, STATUS_SKIPPED) else 1
finally:
store.close()
return 1
if __name__ == "__main__":
raise SystemExit(main())
server/public/dongqiudi-crawler/scripts/fetch_match_detail.py
-159
View File
@@ -1,159 +0,0 @@
"""
获取单场比赛详情
用法: python scripts/fetch_match_detail.py <match_id> [--save]
--save 将数据写入 la_match_data 并更新 la_match
"""
import asyncio
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from curl_cffi import requests as curl_requests
from loguru import logger
from src.parser.nuxt_parser import parse_nuxt_match_sample
from src.storage.database import Database
LIVE_DETAIL_URL = "https://www.dongqiudi.com/liveDetail/{match_id}"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Referer": "https://www.dongqiudi.com/",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
STATUS_MAP = {"Fixture": 0, "Playing": 1, "Played": 2}
def fetch_html(match_id: int) -> str:
url = LIVE_DETAIL_URL.format(match_id=match_id)
resp = curl_requests.get(url, headers=HEADERS, impersonate="chrome", timeout=20)
resp.raise_for_status()
return resp.text
def display(ms: dict):
"""格式化输出比赛详情"""
status = ms.get("status", "")
fs_a = ms.get("fs_A", "")
fs_b = ms.get("fs_B", "")
hts_a = ms.get("hts_A", "")
hts_b = ms.get("hts_B", "")
print()
print("=" * 60)
print(f" 比赛ID: {ms.get('match_id')}")
print(f" 赛事: {ms.get('competition_name', '')} {ms.get('match_title', '')}")
print(f" 轮次: {ms.get('gameweek', '') or ms.get('round_name', '') or '-'}")
print("-" * 60)
print(f" 主队: {ms.get('team_A_name', '')} (ID: {ms.get('team_A_id', '')})")
print(f" 客队: {ms.get('team_B_name', '')} (ID: {ms.get('team_B_id', '')})")
print("-" * 60)
if status == "Played":
print(f" 比分: {fs_a} - {fs_b} (全场)")
if hts_a and hts_b:
print(f" 半场: {hts_a} - {hts_b}")
print(f" 状态: 已结束")
elif status == "Playing":
minute = ms.get("minute", "")
period = ms.get("minute_period", "")
print(f" 比分: {fs_a or 0} - {fs_b or 0} (进行中)")
print(f" 进行: {period} {minute}")
else:
print(f" 状态: 未开始")
print(f" 开赛时间: {ms.get('start_play', '')}")
print(f" 类型: {ms.get('cmp_type', '')}")
# 赔率
home_odds = ms.get("home", "")
draw_odds = ms.get("draw", "")
away_odds = ms.get("away", "")
if home_odds or draw_odds or away_odds:
print(f" 赔率: 主{home_odds} 平{draw_odds} 客{away_odds}")
# 直播源
tv = ms.get("livingTv", "")
if tv:
print(f" 直播: {tv}")
print("=" * 60)
# 额外信息
ht_info = ms.get("ht_info", "")
score_info = ms.get("score_info", "")
if ht_info:
print(f" 半场信息: {ht_info}")
if score_info:
print(f" 比分信息: {score_info}")
# Logo URLs
print(f"\n 赛事Logo: {ms.get('competition_bk_logo', '')}")
print(f" 主队Logo: {ms.get('team_A_logo', '')}")
print(f" 客队Logo: {ms.get('team_B_logo', '')}")
print()
async def save_to_db(ms: dict):
"""将数据写入数据库"""
db = Database()
try:
match_id = int(ms.get("match_id", 0))
status_str = str(ms.get("status", ""))
status_int = STATUS_MAP.get(status_str, 0)
fs_a = ms.get("fs_A", "")
fs_b = ms.get("fs_B", "")
home_score = int(fs_a) if fs_a and str(fs_a).isdigit() else 0
away_score = int(fs_b) if fs_b and str(fs_b).isdigit() else 0
hts_a = str(ms.get("hts_A", ""))
hts_b = str(ms.get("hts_B", ""))
half_score = f"{hts_a}-{hts_b}" if hts_a and hts_b else ""
minute_period = str(ms.get("minute_period", ""))
await db.update_match_live(
match_id=match_id,
status=status_int,
home_score=home_score,
away_score=away_score,
half_score=half_score,
current_minute=minute_period,
)
await db.upsert_match_data(ms)
logger.info(f"已保存到数据库: match_id={match_id}")
finally:
await db.close()
def main():
import argparse
parser = argparse.ArgumentParser(description="获取单场比赛详情")
parser.add_argument("match_id", type=int, help="懂球帝 match_id")
parser.add_argument("--save", action="store_true", help="写入数据库")
parser.add_argument("--json", action="store_true", help="输出原始 JSON")
args = parser.parse_args()
logger.info(f"获取比赛详情: match_id={args.match_id}")
html = fetch_html(args.match_id)
ms = parse_nuxt_match_sample(html)
if not ms:
logger.error("解析失败，无法提取比赛数据")
sys.exit(1)
if args.json:
print(json.dumps(ms, ensure_ascii=False, indent=2, default=str))
else:
display(ms)
if args.save:
asyncio.run(save_to_db(ms))
if __name__ == "__main__":
main()
server/public/dongqiudi-crawler/scripts/kb_worker.py
-708
View File
@@ -1,708 +0,0 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import asyncio
import hashlib
import html
import json
import math
import os
import re
import socket
import sys
import time
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional
import aiohttp
import aiomysql
import redis.asyncio as aioredis
import redis.exceptions as redis_exceptions
from loguru import logger
APP_DIR = Path(__file__).resolve().parents[1]
if str(APP_DIR) not in sys.path:
sys.path.insert(0, str(APP_DIR))
from src.core.config import get_config, load_config # noqa: E402
STREAM_KEY = "ai:kb:sync:stream"
DEAD_STREAM_KEY = "ai:kb:sync:dead"
GROUP_NAME = "ai_kb_workers"
COLLECTION_NAME = "global"
DEFAULT_CHUNK_SIZE = 600
DEFAULT_CHUNK_OVERLAP = 80
DEFAULT_EMBEDDING_BASE_URL = "https://dashscope.aliyuncs.com/compatible-mode"
DEFAULT_EMBEDDING_MODEL = "text-embedding-v4"
DEFAULT_EMBEDDING_DIM = 1024
def now_ts() -> int:
return int(time.time())
def normalize_plain_text(text: Any) -> str:
value = html.unescape(str(text or ""))
value = re.sub(r"\s+", " ", value.strip(), flags=re.UNICODE)
return value.strip()
def normalize_html_text(text: Any) -> str:
value = str(text or "")
value = re.sub(r"(?i)<br\s*/?>|</p>|</div>|</li>", "\n", value)
value = re.sub(r"<[^>]+>", "", value)
return normalize_plain_text(value)
def truncate_text(text: str, length: int) -> str:
text = text.strip()
if len(text) <= length:
return text
return text[:length] + "..."
def split_text(text: str, chunk_size: int = DEFAULT_CHUNK_SIZE, overlap: int = DEFAULT_CHUNK_OVERLAP) -> List[str]:
text = text.strip()
if not text:
return []
if len(text) <= chunk_size:
return [text]
chunks: List[str] = []
start = 0
step = max(1, chunk_size - overlap)
while start < len(text):
chunks.append(text[start : start + chunk_size].strip())
if start + chunk_size >= len(text):
break
start += step
return [chunk for chunk in chunks if chunk]
def vector_norm(vector: Iterable[float]) -> float:
total = 0.0
for value in vector:
total += float(value) * float(value)
return round(math.sqrt(total), 8) if total > 0 else 0.0
def json_loads_maybe(value: Any, default: Any) -> Any:
if value is None:
return default
if isinstance(value, (dict, list)):
return value
try:
return json.loads(str(value))
except Exception:
return default
def json_text(value: Any) -> str:
return json.dumps(value, ensure_ascii=False, separators=(",", ":"), default=str)
class KbDatabase:
def __init__(self):
cfg = get_config().database
self.cfg = cfg
self.prefix = cfg.prefix
self.pool: Optional[aiomysql.Pool] = None
async def connect(self) -> None:
if self.pool:
return
self.pool = await aiomysql.create_pool(
host=self.cfg.host,
port=self.cfg.port,
user=self.cfg.username,
password=self.cfg.password,
db=self.cfg.database,
charset=self.cfg.charset,
autocommit=False,
minsize=1,
maxsize=max(2, min(10, self.cfg.pool_size)),
)
logger.info("KB worker MySQL pool ready: {}:{}/{}", self.cfg.host, self.cfg.port, self.cfg.database)
async def close(self) -> None:
if self.pool:
self.pool.close()
await self.pool.wait_closed()
self.pool = None
def table(self, name: str) -> str:
return f"`{self.prefix}{name}`"
async def fetchone(self, sql: str, args: tuple = ()) -> Optional[Dict[str, Any]]:
await self.connect()
assert self.pool is not None
async with self.pool.acquire() as conn:
async with conn.cursor(aiomysql.DictCursor) as cur:
await cur.execute(sql, args)
return await cur.fetchone()
async def fetchall(self, sql: str, args: tuple = ()) -> List[Dict[str, Any]]:
await self.connect()
assert self.pool is not None
async with self.pool.acquire() as conn:
async with conn.cursor(aiomysql.DictCursor) as cur:
await cur.execute(sql, args)
return list(await cur.fetchall())
async def build_payload(self, domain: str, subtype: str, source_id: int) -> Optional[Dict[str, Any]]:
key = f"{domain}:{subtype}"
if key == "article:article":
return await self.build_article_payload(source_id)
if key == "post:post":
return await self.build_post_payload(source_id)
if key == "lottery:draw_result":
return await self.build_lottery_draw_result_payload(source_id)
if key == "lottery:ai_history":
return await self.build_lottery_ai_history_payload(source_id)
if key == "match:event":
return await self.build_match_event_payload(source_id)
return None
async def build_article_payload(self, source_id: int) -> Optional[Dict[str, Any]]:
row = await self.fetchone(
f"SELECT * FROM {self.table('article')} WHERE id=%s AND is_show=1 AND delete_time IS NULL LIMIT 1",
(source_id,),
)
if not row:
return None
cate = await self.fetchone(f"SELECT name FROM {self.table('article_cate')} WHERE id=%s LIMIT 1", (row.get("cid") or 0,))
cate_name = str((cate or {}).get("name") or "")
ext = json_loads_maybe(row.get("ext"), {})
translated = str(ext.get("translated_content") or "").strip() if isinstance(ext, dict) else ""
content = normalize_html_text(translated or row.get("content") or "")
summary = normalize_plain_text(row.get("abstract") or row.get("desc") or "")
keywords = list(dict.fromkeys(filter(None, [
str(row.get("title") or ""),
cate_name,
str(row.get("author") or ""),
str(row.get("category") or ""),
])))
return {
"title": str(row.get("title") or ""),
"summary": summary,
"canonical_text": "\n\n".join(filter(None, [str(row.get("title") or ""), summary, content])).strip(),
"keywords": ",".join(keywords),
"metadata_json": {
"cid": int(row.get("cid") or 0),
"cate_name": cate_name,
"author": str(row.get("author") or ""),
"category": str(row.get("category") or ""),
"article_id": int(row.get("article_id") or 0),
"published_at": str(row.get("published_at") or ""),
"is_video": int(row.get("is_video") or 0),
},
"source_created_at": int(row.get("create_time") or 0),
"source_updated_at": int(row.get("update_time") or row.get("create_time") or 0),
}
async def build_post_payload(self, source_id: int) -> Optional[Dict[str, Any]]:
row = await self.fetchone(
f"SELECT * FROM {self.table('community_post')} WHERE id=%s AND status=1 AND delete_time IS NULL LIMIT 1",
(source_id,),
)
if not row:
return None
tags = await self.fetchall(
f"""
SELECT t.name
FROM {self.table('community_post_tag')} pt
JOIN {self.table('community_tag')} t ON t.id = pt.tag_id
WHERE pt.post_id=%s
""",
(source_id,),
)
tag_names = [str(item.get("name") or "") for item in tags if item.get("name")]
ext = json_loads_maybe(row.get("ext"), {})
segments = [
str(row.get("content") or ""),
str(ext.get("translated_content") or "") if isinstance(ext, dict) else "",
str(ext.get("lottery_analysis_content") or "") if isinstance(ext, dict) else "",
" ".join(tag_names),
]
canonical = normalize_plain_text("\n\n".join(filter(None, segments)))
return {
"title": truncate_text(canonical, 40) or f"社区帖子#{source_id}",
"summary": truncate_text(canonical, 200),
"canonical_text": canonical,
"keywords": ",".join(dict.fromkeys(tag_names)),
"metadata_json": {
"post_type": int(row.get("post_type") or 0),
"match_id": int(row.get("match_id") or 0),
"is_paid": int(row.get("is_paid") or 0),
"user_id": int(row.get("user_id") or 0),
"tags": tag_names,
},
"source_created_at": int(row.get("create_time") or 0),
"source_updated_at": int(row.get("update_time") or row.get("create_time") or 0),
}
async def build_lottery_draw_result_payload(self, source_id: int) -> Optional[Dict[str, Any]]:
row = await self.fetchone(f"SELECT * FROM {self.table('lottery_draw_result')} WHERE id=%s LIMIT 1", (source_id,))
if not row:
return None
game = await self.fetchone(
f"SELECT name, template FROM {self.table('lottery_game')} WHERE code=%s LIMIT 1",
(row.get("code") or "",),
)
game = game or {}
numbers = [item for item in str(row.get("draw_code") or "").split(",") if item]
title = f"{game.get('name') or row.get('code')} 第{row.get('issue')}期开奖"
summary = f"{title} 于 {row.get('draw_time') or ''} 开奖，开奖号码为 {'、'.join(numbers)}。"
trend = json_loads_maybe(row.get("trend"), {})
canonical = "\n".join(filter(None, [
summary,
f"下一期期号：{row.get('next_issue')}" if row.get("next_issue") else "",
f"下期开奖时间：{row.get('next_time')}" if row.get("next_time") else "",
"走势数据：" + normalize_plain_text(json_text(trend)) if trend not in ({}, None) else "",
])).strip()
return {
"title": title,
"summary": summary,
"canonical_text": canonical,
"keywords": ",".join(filter(None, [
str(row.get("code") or ""),
str(game.get("name") or ""),
str(game.get("template") or ""),
str(row.get("issue") or ""),
])),
"metadata_json": {
"code": str(row.get("code") or ""),
"issue": str(row.get("issue") or ""),
"template": str(game.get("template") or ""),
"game_name": str(game.get("name") or ""),
"draw_time": str(row.get("draw_time") or ""),
},
"source_created_at": int(row.get("create_time") or 0),
"source_updated_at": int(row.get("update_time") or row.get("create_time") or 0),
}
async def build_lottery_ai_history_payload(self, source_id: int) -> Optional[Dict[str, Any]]:
row = await self.fetchone(f"SELECT * FROM {self.table('lottery_ai_analysis')} WHERE id=%s LIMIT 1", (source_id,))
if not row:
return None
result = row.get("result") if row.get("result") is not None else ""
result_text = result if isinstance(result, str) else json_text(result)
canonical = normalize_plain_text(result_text)
title = f"{row.get('code') or '彩票'} 第{row.get('issue') or '-'}期 {row.get('module') or 'module'} 历史AI分析"
return {
"title": title,
"summary": truncate_text(canonical, 220),
"canonical_text": canonical,
"keywords": ",".join(filter(None, [str(row.get("code") or ""), str(row.get("issue") or ""), str(row.get("module") or "")])),
"metadata_json": {
"code": str(row.get("code") or ""),
"issue": str(row.get("issue") or ""),
"module": str(row.get("module") or ""),
},
"source_created_at": int(row.get("create_time") or 0),
"source_updated_at": int(row.get("update_time") or row.get("create_time") or 0),
}
async def build_match_event_payload(self, source_id: int) -> Optional[Dict[str, Any]]:
row = await self.fetchone(f"SELECT * FROM {self.table('match')} WHERE id=%s AND is_show=1 LIMIT 1", (source_id,))
if not row:
return None
home = str(row.get("home_team") or "")
away = str(row.get("away_team") or "")
league = str(row.get("league_name") or "")
match_time_value = int(row.get("match_time") or 0)
match_time_text = time.strftime("%Y-%m-%d %H:%M", time.localtime(match_time_value)) if match_time_value > 0 else ""
history = await self.fetchall(
f"""
SELECT * FROM {self.table('match_history')}
WHERE (home_team=%s AND away_team=%s) OR (home_team=%s AND away_team=%s)
ORDER BY match_time DESC LIMIT 5
""",
(home, away, away, home),
)
home_recent = await self.fetchall(
f"SELECT * FROM {self.table('match_recent')} WHERE team_name=%s ORDER BY match_time DESC LIMIT 5",
(home,),
)
away_recent = await self.fetchall(
f"SELECT * FROM {self.table('match_recent')} WHERE team_name=%s ORDER BY match_time DESC LIMIT 5",
(away,),
)
title = f"{home} vs {away}".strip()
summary = (
f"{league} {title}，{home} 对阵 {away}，比赛时间 {match_time_text}，"
f"当前比分 {row.get('home_score') or 0}-{row.get('away_score') or 0}。"
)
canonical = "\n".join([
summary,
f"赔率：主胜{row.get('home_odds') or '-'}，平局{row.get('draw_odds') or '-'}，客胜{row.get('away_odds') or '-'}。",
"历史交锋：" + normalize_plain_text(json_text(history)),
f"{home}近期战绩：" + normalize_plain_text(json_text(home_recent)),
f"{away}近期战绩：" + normalize_plain_text(json_text(away_recent)),
])
return {
"title": title or f"赛事#{source_id}",
"summary": normalize_plain_text(summary),
"canonical_text": canonical.strip(),
"keywords": ",".join(filter(None, [league, home, away])),
"metadata_json": {
"league": league,
"teams": [team for team in [home, away] if team],
"match_time": match_time_text,
"status": int(row.get("status") or 0),
},
"source_created_at": int(row.get("create_time") or row.get("match_time") or 0),
"source_updated_at": int(row.get("update_time") or row.get("match_time") or row.get("create_time") or 0),
}
async def upsert_document(self, domain: str, subtype: str, source_id: int, embedder: "EmbeddingClient") -> Dict[str, Any]:
payload = await self.build_payload(domain, subtype, source_id)
if not payload:
await self.mark_document_inactive(domain, subtype, source_id)
return {"success": False, "error": "源内容不存在或不可用"}
content_hash = hashlib.md5(json_text([
payload["title"],
payload["summary"],
payload["canonical_text"],
payload["keywords"],
payload["metadata_json"],
]).encode("utf-8")).hexdigest()
existing = await self.fetchone(
f"""
SELECT * FROM {self.table('ai_kb_document')}
WHERE domain=%s AND subtype=%s AND source_id=%s LIMIT 1
""",
(domain, subtype, source_id),
)
if existing and str(existing.get("content_hash") or "") == content_hash and int(existing.get("is_active") or 0) == 1:
await self.update_document_metadata(int(existing["id"]), payload)
chunk_count = await self.fetchone(
f"SELECT COUNT(*) AS total FROM {self.table('ai_kb_chunk')} WHERE document_id=%s",
(int(existing["id"]),),
)
return {"success": True, "document_id": int(existing["id"]), "chunk_count": int((chunk_count or {}).get("total") or 0), "skipped": True}
chunks = split_text(str(payload["canonical_text"] or ""))
if not chunks:
chunks = [payload["summary"] or payload["title"] or "empty"]
embedding_result = await embedder.embed(chunks)
embeddings = embedding_result.get("embeddings") or []
if not embedding_result.get("success") or not embeddings:
return {"success": False, "error": embedding_result.get("error") or "embedding失败"}
await self.connect()
assert self.pool is not None
async with self.pool.acquire() as conn:
async with conn.cursor(aiomysql.DictCursor) as cur:
await conn.begin()
try:
metadata_json = json_text(payload["metadata_json"])
if existing:
document_id = int(existing["id"])
await cur.execute(
f"""
UPDATE {self.table('ai_kb_document')}
SET title=%s, summary=%s, canonical_text=%s, keywords=%s, metadata_json=%s,
source_created_at=%s, source_updated_at=%s, content_hash=%s, is_active=1, update_time=%s
WHERE id=%s
""",
(
payload["title"],
payload["summary"],
payload["canonical_text"],
payload["keywords"],
metadata_json,
int(payload["source_created_at"]),
int(payload["source_updated_at"]),
content_hash,
now_ts(),
document_id,
),
)
else:
await cur.execute(
f"""
INSERT INTO {self.table('ai_kb_document')}
(domain, subtype, source_id, title, summary, canonical_text, keywords, metadata_json,
source_created_at, source_updated_at, content_hash, is_active, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,1,%s,%s)
""",
(
domain,
subtype,
source_id,
payload["title"],
payload["summary"],
payload["canonical_text"],
payload["keywords"],
metadata_json,
int(payload["source_created_at"]),
int(payload["source_updated_at"]),
content_hash,
now_ts(),
now_ts(),
),
)
document_id = int(cur.lastrowid)
await cur.execute(f"DELETE FROM {self.table('ai_kb_chunk')} WHERE document_id=%s", (document_id,))
dimension = int(embedding_result.get("dimension") or len(embeddings[0] or []))
keyword_text = " ".join(filter(None, [
str(payload.get("title") or ""),
str(payload.get("summary") or ""),
str(payload.get("keywords") or ""),
json_text(payload.get("metadata_json") or {}),
]))
for index, chunk_text in enumerate(chunks):
embedding = [float(value) for value in embeddings[index]]
await cur.execute(
f"""
INSERT INTO {self.table('ai_kb_chunk')}
(document_id, collection_name, chunk_index, chunk_text, embedding_json,
embedding_model, embedding_dim, vector_norm, keyword_text, create_time, update_time)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
""",
(
document_id,
COLLECTION_NAME,
index,
chunk_text,
json_text(embedding),
embedding_result.get("model") or embedder.model,
dimension,
vector_norm(embedding),
keyword_text,
now_ts(),
now_ts(),
),
)
await conn.commit()
except Exception:
await conn.rollback()
raise
return {"success": True, "document_id": document_id, "chunk_count": len(chunks)}
async def update_document_metadata(self, document_id: int, payload: Dict[str, Any]) -> None:
await self.connect()
assert self.pool is not None
async with self.pool.acquire() as conn:
async with conn.cursor() as cur:
await cur.execute(
f"""
UPDATE {self.table('ai_kb_document')}
SET title=%s, summary=%s, keywords=%s, metadata_json=%s,
source_created_at=%s, source_updated_at=%s, update_time=%s
WHERE id=%s
""",
(
payload["title"],
payload["summary"],
payload["keywords"],
json_text(payload["metadata_json"]),
int(payload["source_created_at"]),
int(payload["source_updated_at"]),
now_ts(),
document_id,
),
)
await conn.commit()
async def mark_document_inactive(self, domain: str, subtype: str, source_id: int) -> None:
await self.connect()
assert self.pool is not None
async with self.pool.acquire() as conn:
async with conn.cursor() as cur:
await cur.execute(
f"""
UPDATE {self.table('ai_kb_document')}
SET is_active=0, update_time=%s
WHERE domain=%s AND subtype=%s AND source_id=%s
""",
(now_ts(), domain, subtype, source_id),
)
await conn.commit()
class EmbeddingClient:
def __init__(self):
self.api_key = os.environ.get("EMBEDDING_API_KEY") or os.environ.get("DASHSCOPE_API_KEY") or ""
self.base_url = (os.environ.get("EMBEDDING_BASE_URL") or DEFAULT_EMBEDDING_BASE_URL).rstrip("/")
self.model = os.environ.get("EMBEDDING_MODEL") or DEFAULT_EMBEDDING_MODEL
self.dimension = int(os.environ.get("EMBEDDING_DIM") or DEFAULT_EMBEDDING_DIM)
self.batch_size = max(1, int(os.environ.get("EMBEDDING_BATCH_SIZE") or 10))
async def embed(self, chunks: List[str]) -> Dict[str, Any]:
if not self.api_key or not self.base_url or not self.model:
return {"success": False, "error": "Embedding API配置未完成", "embeddings": []}
start = time.monotonic()
embeddings: List[List[float]] = []
tokens = 0
try:
async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=60)) as session:
for offset in range(0, len(chunks), self.batch_size):
batch = chunks[offset:offset + self.batch_size]
payload: Dict[str, Any] = {"model": self.model, "input": batch}
if self.dimension > 0:
payload["dimensions"] = self.dimension
async with session.post(
self.base_url + "/v1/embeddings",
json=payload,
headers={"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"},
ssl=False,
) as resp:
body = await resp.text()
try:
data = json.loads(body)
except Exception:
data = {}
if resp.status != 200 or not isinstance(data.get("data"), list):
message = ((data.get("error") or {}).get("message") if isinstance(data, dict) else "") or f"HTTP {resp.status}"
return {"success": False, "error": message, "embeddings": [], "cost_ms": int((time.monotonic() - start) * 1000)}
for row in data.get("data") or []:
embedding = row.get("embedding") if isinstance(row, dict) else None
if isinstance(embedding, list) and embedding:
embeddings.append([float(value) for value in embedding])
tokens += int(((data.get("usage") or {}).get("total_tokens") or 0) if isinstance(data, dict) else 0)
return {
"success": len(embeddings) == len(chunks),
"error": "" if len(embeddings) == len(chunks) else "Embedding结果数量不匹配",
"embeddings": embeddings,
"model": self.model,
"dimension": len(embeddings[0]) if embeddings else 0,
"tokens": tokens,
"cost_ms": int((time.monotonic() - start) * 1000),
}
except Exception as exc:
return {"success": False, "error": f"Embedding请求失败: {exc}", "embeddings": []}
class KbWorker:
def __init__(self, batch: int, block_ms: int, max_retries: int, worker_id: str):
self.batch = max(1, batch)
self.block_ms = max(100, block_ms)
self.max_retries = max(0, max_retries)
self.worker_id = worker_id
redis_cfg = get_config().redis
self.redis = aioredis.Redis(
host=redis_cfg.host,
port=redis_cfg.port,
password=redis_cfg.password or None,
db=redis_cfg.db,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=max(10, int(self.block_ms / 1000) + 5),
health_check_interval=30,
retry_on_timeout=True,
)
self.db = KbDatabase()
self.embedder = EmbeddingClient()
async def close(self) -> None:
await self.redis.close()
await self.db.close()
async def ensure_group(self) -> None:
try:
await self.redis.xgroup_create(STREAM_KEY, GROUP_NAME, id="0", mkstream=True)
logger.info("created redis stream group {} on {}", GROUP_NAME, STREAM_KEY)
except Exception as exc:
if "BUSYGROUP" not in str(exc):
raise
async def run_forever(self) -> None:
await self.ensure_group()
logger.info("KB worker started id={} batch={} block_ms={}", self.worker_id, self.batch, self.block_ms)
while True:
try:
await self.consume_once()
except (redis_exceptions.TimeoutError, redis_exceptions.ConnectionError, asyncio.TimeoutError) as exc:
logger.warning("redis read interrupted, continue polling: {}", exc)
await asyncio.sleep(1)
async def consume_once(self) -> int:
rows = await self.redis.xreadgroup(
GROUP_NAME,
self.worker_id,
streams={STREAM_KEY: ">"},
count=self.batch,
block=self.block_ms,
)
processed = 0
for _, messages in rows or []:
for message_id, fields in messages:
await self.handle_message(message_id, fields)
processed += 1
return processed
async def handle_message(self, message_id: str, fields: Dict[str, Any]) -> None:
domain = str(fields.get("domain") or "")
subtype = str(fields.get("subtype") or "")
source_id = int(fields.get("source_id") or 0)
attempts = int(fields.get("attempts") or 0)
started = time.monotonic()
try:
if not domain or not subtype or source_id <= 0:
raise ValueError("消息缺少 domain/subtype/source_id")
result = await self.db.upsert_document(domain, subtype, source_id, self.embedder)
if not result.get("success"):
raise RuntimeError(str(result.get("error") or "upsert failed"))
await self.redis.xack(STREAM_KEY, GROUP_NAME, message_id)
logger.info(
"kb upsert ok message={} source={}/{}/{} result={} elapsed={:.2f}s",
message_id,
domain,
subtype,
source_id,
result,
time.monotonic() - started,
)
except Exception as exc:
await self.handle_failure(message_id, fields, attempts, str(exc))
async def handle_failure(self, message_id: str, fields: Dict[str, Any], attempts: int, error: str) -> None:
next_attempt = attempts + 1
payload = dict(fields)
payload["attempts"] = str(next_attempt)
payload["last_error"] = error[:500]
payload["failed_at"] = str(now_ts())
if next_attempt > self.max_retries:
await self.redis.xadd(DEAD_STREAM_KEY, payload)
await self.redis.xack(STREAM_KEY, GROUP_NAME, message_id)
logger.error("kb message={} moved to dead stream after {} attempts: {}", message_id, next_attempt, error)
return
await self.redis.xadd(STREAM_KEY, payload)
await self.redis.xack(STREAM_KEY, GROUP_NAME, message_id)
logger.warning("kb message={} retry {}/{}: {}", message_id, next_attempt, self.max_retries, error)
def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Sport Era AI KB Redis Stream worker")
parser.add_argument("--batch", type=int, default=50)
parser.add_argument("--block-ms", type=int, default=5000)
parser.add_argument("--max-retries", type=int, default=3)
parser.add_argument("--worker-id", default="")
return parser.parse_args(argv)
async def async_main(argv: Optional[List[str]] = None) -> int:
args = parse_args(argv)
load_config()
worker_id = args.worker_id or f"{socket.gethostname()}-{os.getpid()}"
worker = KbWorker(args.batch, args.block_ms, args.max_retries, worker_id)
try:
await worker.run_forever()
finally:
await worker.close()
return 0
def main(argv: Optional[List[str]] = None) -> int:
return asyncio.run(async_main(argv))
if __name__ == "__main__":
raise SystemExit(main())
server/public/dongqiudi-crawler/src/__init__.py
View File
server/public/dongqiudi-crawler/src/anti_detect/__init__.py
View File
server/public/dongqiudi-crawler/src/anti_detect/fingerprint.py
-237
View File
@@ -1,237 +0,0 @@
"""
浏览器指纹生成与管理
- 生成逼真的浏览器指纹（UA、屏幕、Canvas、WebGL、TLS）
- 指纹池轮换、冷却、黑名单机制
"""
import asyncio
import hashlib
import random
import time
from datetime import datetime, timedelta
from typing import Dict, Any, List, Optional, Set
from loguru import logger
from fake_useragent import UserAgent
from src.core.config import get_config
# ── 常量 ──────────────────────────────────────────────
CHROME_VERSIONS = [
"120.0.6099.109", "121.0.6167.85", "122.0.6261.57",
"123.0.6312.86", "124.0.6367.91", "125.0.6422.60",
]
SCREEN_RESOLUTIONS = [
{"width": 1920, "height": 1080, "ratio": 1.0},
{"width": 1366, "height": 768, "ratio": 1.0},
{"width": 1536, "height": 864, "ratio": 1.25},
{"width": 2560, "height": 1440, "ratio": 1.5},
{"width": 1440, "height": 900, "ratio": 1.0},
{"width": 1680, "height": 1050, "ratio": 1.0},
{"width": 1280, "height": 800, "ratio": 1.0},
{"width": 3840, "height": 2160, "ratio": 2.0},
]
WEBGL_RENDERERS = [
"ANGLE (Intel, Intel(R) UHD Graphics 630 Direct3D11 vs_5_0 ps_5_0, D3D11)",
"ANGLE (Intel, Intel(R) UHD Graphics Direct3D11 vs_5_0 ps_5_0, D3D11)",
"ANGLE (AMD, AMD Radeon(TM) Graphics Direct3D11 vs_5_0 ps_5_0, D3D11)",
"ANGLE (NVIDIA, NVIDIA GeForce GTX 1060 6GB Direct3D11 vs_5_0 ps_5_0, D3D11)",
"ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Direct3D11 vs_5_0 ps_5_0, D3D11)",
"ANGLE (Intel, Intel(R) Iris(R) Xe Graphics Direct3D11 vs_5_0 ps_5_0, D3D11)",
]
FONTS_POOL = [
"Arial", "Arial Black", "Calibri", "Cambria", "Consolas",
"Courier New", "Georgia", "Impact", "Segoe UI", "Tahoma",
"Times New Roman", "Trebuchet MS", "Verdana",
"Microsoft YaHei", "SimHei", "SimSun", "NSimSun", "FangSong", "KaiTi",
]
SEC_CH_UA_TEMPLATES = {
"chrome": '"Chromium";v="{major}", "Google Chrome";v="{major}", "Not-A.Brand";v="99"',
"edge": '"Chromium";v="{major}", "Microsoft Edge";v="{major}", "Not-A.Brand";v="99"',
}
class FingerprintGenerator:
"""指纹生成器"""
def __init__(self):
try:
self._ua = UserAgent(browsers=["chrome", "edge"], os=["windows", "macos"])
except Exception:
self._ua = None
def generate(self) -> Dict[str, Any]:
browser = random.choice(["chrome", "edge"])
version = random.choice(CHROME_VERSIONS)
major = version.split(".")[0]
ua_string = self._build_ua(browser, version)
screen = random.choice(SCREEN_RESOLUTIONS)
fp_id = f"fp_{int(time.time())}_{random.randint(1000, 9999)}"
return {
"id": fp_id,
"browser": browser,
"version": version,
"user_agent": ua_string,
"screen": screen,
"sec_ch_ua": SEC_CH_UA_TEMPLATES.get(browser, SEC_CH_UA_TEMPLATES["chrome"]).format(major=major),
"http_headers": self._build_headers(ua_string, browser, major),
"canvas_hash": hashlib.md5(f"canvas_{random.randint(100000, 999999)}".encode()).hexdigest()[:16],
"webgl": {
"vendor": "Google Inc. (Intel)",
"renderer": random.choice(WEBGL_RENDERERS),
},
"fonts": sorted(random.sample(FONTS_POOL, k=random.randint(10, len(FONTS_POOL)))),
"timezone": "Asia/Shanghai",
"language": "zh-CN",
"created_at": datetime.now().isoformat(),
"success_rate": 1.0,
"usage_count": 0,
"is_active": True,
}
def _build_ua(self, browser: str, version: str) -> str:
os_strings = [
"Windows NT 10.0; Win64; x64",
"Windows NT 10.0; Win64; x64",
"Macintosh; Intel Mac OS X 10_15_7",
]
os_str = random.choice(os_strings)
major = version.split(".")[0]
if browser == "edge":
return (
f"Mozilla/5.0 ({os_str}) AppleWebKit/537.36 "
f"(KHTML, like Gecko) Chrome/{version} Safari/537.36 Edg/{version}"
)
return (
f"Mozilla/5.0 ({os_str}) AppleWebKit/537.36 "
f"(KHTML, like Gecko) Chrome/{version} Safari/537.36"
)
def _build_headers(self, ua: str, browser: str, major: str) -> Dict[str, str]:
headers = {
"User-Agent": ua,
"Accept": "application/json, text/plain, */*",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Referer": "https://www.dongqiudi.com/",
"Origin": "https://www.dongqiudi.com",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-site",
"Sec-Ch-Ua": SEC_CH_UA_TEMPLATES.get(browser, SEC_CH_UA_TEMPLATES["chrome"]).format(major=major),
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": '"Windows"',
}
return headers
class FingerprintPool:
"""指纹池管理器"""
def __init__(self):
cfg = get_config().anti_detect.fingerprint
self._pool_size = cfg.pool_size
self._min_success_rate = cfg.min_success_rate
self._cooldown = cfg.cooldown
self._generator = FingerprintGenerator()
self._pool: Dict[str, Dict[str, Any]] = {}
self._blacklist: Set[str] = set()
self._lock = asyncio.Lock()
async def initialize(self, size: Optional[int] = None):
size = size or self._pool_size
async with self._lock:
for _ in range(size):
fp = self._generator.generate()
self._pool[fp["id"]] = fp
logger.info(f"指纹池初始化完成，共 {len(self._pool)} 个指纹")
async def acquire(self) -> Dict[str, Any]:
async with self._lock:
now = datetime.now()
candidates = []
for fp_id, fp in self._pool.items():
if fp_id in self._blacklist or not fp.get("is_active", True):
continue
if fp.get("success_rate", 1.0) < self._min_success_rate:
fp["is_active"] = False
continue
last_used = fp.get("last_used")
if last_used:
last_dt = datetime.fromisoformat(last_used) if isinstance(last_used, str) else last_used
if (now - last_dt).total_seconds() < self._cooldown:
continue
candidates.append(fp)
if not candidates:
logger.warning("指纹池耗尽，生成新指纹")
fp = self._generator.generate()
self._pool[fp["id"]] = fp
candidates = [fp]
candidates.sort(key=lambda x: (x.get("usage_count", 0), -x.get("success_rate", 1.0)))
chosen = candidates[0]
chosen["last_used"] = now.isoformat()
chosen["usage_count"] = chosen.get("usage_count", 0) + 1
return chosen.copy()
async def mark_success(self, fp_id: str):
async with self._lock:
fp = self._pool.get(fp_id)
if not fp:
return
total = fp.get("usage_count", 1)
fails = fp.get("fail_count", 0)
fp["success_rate"] = 1.0 - (fails / total) if total > 0 else 1.0
async def mark_failure(self, fp_id: str, error_type: str = ""):
async with self._lock:
fp = self._pool.get(fp_id)
if not fp:
return
fp["fail_count"] = fp.get("fail_count", 0) + 1
total = fp.get("usage_count", 1)
fails = fp["fail_count"]
fp["success_rate"] = 1.0 - (fails / total) if total > 0 else 0.0
if error_type in ("ip_blocked", "captcha", "fingerprint_detected"):
self._blacklist.add(fp_id)
fp["is_active"] = False
logger.warning(f"指纹 {fp_id} 被加入黑名单: {error_type}")
async def cleanup(self):
async with self._lock:
to_remove = [
fp_id for fp_id, fp in self._pool.items()
if not fp.get("is_active") and fp.get("success_rate", 1.0) < self._min_success_rate * 0.5
]
for fp_id in to_remove:
self._pool.pop(fp_id, None)
self._blacklist.discard(fp_id)
deficit = self._pool_size - len(self._pool)
if deficit > 0:
for _ in range(deficit):
fp = self._generator.generate()
self._pool[fp["id"]] = fp
logger.info(f"指纹池补充 {deficit} 个新指纹")
@property
def stats(self) -> Dict[str, Any]:
active = sum(1 for fp in self._pool.values() if fp.get("is_active", True))
rates = [fp.get("success_rate", 1.0) for fp in self._pool.values()]
return {
"total": len(self._pool),
"active": active,
"blacklisted": len(self._blacklist),
"avg_success_rate": sum(rates) / len(rates) if rates else 1.0,
}
server/public/dongqiudi-crawler/src/anti_detect/proxy_manager.py
-162
View File
@@ -1,162 +0,0 @@
"""
代理管理器
- 代理池维护、健康检测、自动轮换
- 支持 HTTP / SOCKS5 代理
"""
import asyncio
import random
import time
from typing import Dict, Any, List, Optional
from loguru import logger
from src.core.config import get_config
from src.core.exceptions import ProxyError
class Proxy:
__slots__ = (
"url", "protocol", "host", "port", "username", "password",
"success_count", "fail_count", "last_used", "last_check",
"latency", "is_active",
)
def __init__(self, url: str, protocol: str = "http"):
self.url = url
self.protocol = protocol
parts = url.replace("http://", "").replace("https://", "").replace("socks5://", "")
auth_host = parts.split("@")
if len(auth_host) == 2:
auth = auth_host[0].split(":")
self.username = auth[0]
self.password = auth[1] if len(auth) > 1 else ""
host_port = auth_host[1].split(":")
else:
self.username = ""
self.password = ""
host_port = auth_host[0].split(":")
self.host = host_port[0]
self.port = int(host_port[1]) if len(host_port) > 1 else 80
self.success_count = 0
self.fail_count = 0
self.last_used = 0.0
self.last_check = 0.0
self.latency = 0.0
self.is_active = True
@property
def success_rate(self) -> float:
total = self.success_count + self.fail_count
return self.success_count / total if total > 0 else 1.0
def mark_success(self, latency: float):
self.success_count += 1
self.latency = latency
self.last_used = time.time()
def mark_failure(self):
self.fail_count += 1
self.last_used = time.time()
cfg = get_config().anti_detect.proxy
if self.fail_count >= cfg.max_failures:
self.is_active = False
logger.warning(f"代理 {self.host}:{self.port} 因连续失败被停用")
class ProxyManager:
"""代理池管理器"""
def __init__(self):
self._proxies: List[Proxy] = []
self._lock = asyncio.Lock()
self._enabled = get_config().anti_detect.proxy.enabled
@property
def enabled(self) -> bool:
return self._enabled and len(self._proxies) > 0
async def add_proxy(self, url: str, protocol: str = "http"):
async with self._lock:
proxy = Proxy(url, protocol)
self._proxies.append(proxy)
logger.info(f"添加代理: {proxy.host}:{proxy.port}")
async def add_proxies(self, urls: List[str], protocol: str = "http"):
for url in urls:
await self.add_proxy(url, protocol)
async def get_proxy(self) -> Optional[str]:
if not self._enabled:
return None
async with self._lock:
active = [p for p in self._proxies if p.is_active]
if not active:
logger.warning("无可用代理")
return None
active.sort(key=lambda p: (p.success_rate, -p.latency), reverse=True)
top_n = max(1, len(active) // 3)
chosen = random.choice(active[:top_n])
return chosen.url
async def mark_success(self, proxy_url: str, latency: float):
async with self._lock:
for p in self._proxies:
if p.url == proxy_url:
p.mark_success(latency)
break
async def mark_failure(self, proxy_url: str):
async with self._lock:
for p in self._proxies:
if p.url == proxy_url:
p.mark_failure()
break
async def health_check(self):
"""批量检测代理健康状态"""
import aiohttp
test_url = get_config().anti_detect.proxy.test_url
tasks = []
async def _check(proxy: Proxy):
try:
start = time.time()
async with aiohttp.ClientSession() as session:
async with session.get(
test_url,
proxy=proxy.url,
timeout=aiohttp.ClientTimeout(total=10),
) as resp:
if resp.status == 200:
proxy.latency = time.time() - start
proxy.is_active = True
proxy.last_check = time.time()
else:
proxy.is_active = False
except Exception:
proxy.is_active = False
async with self._lock:
for p in self._proxies:
tasks.append(_check(p))
await asyncio.gather(*tasks, return_exceptions=True)
active_count = sum(1 for p in self._proxies if p.is_active)
logger.info(f"代理健康检查完成: {active_count}/{len(self._proxies)} 可用")
@property
def stats(self) -> Dict[str, Any]:
active = sum(1 for p in self._proxies if p.is_active)
return {
"total": len(self._proxies),
"active": active,
"avg_latency": (
sum(p.latency for p in self._proxies if p.is_active) / active
if active else 0
),
}
server/public/dongqiudi-crawler/src/anti_detect/stealth_browser.py
-285
View File
@@ -1,285 +0,0 @@
"""
Playwright 隐身浏览器
- 深度反检测：移除 webdriver 痕迹、模拟人类行为
- 自动处理验证码页面、JS 挑战
"""
import asyncio
import random
import time
from typing import Dict, Any, Optional, List
from loguru import logger
from src.core.config import get_config
from src.core.exceptions import CaptchaError, BlockedError
class StealthBrowser:
"""隐身浏览器 - Playwright + 深度反检测"""
def __init__(self):
cfg = get_config().anti_detect.playwright
self._browser_type = cfg.browser
self._headless = cfg.headless
self._use_stealth = cfg.stealth
self._viewport = cfg.viewport
self._anti = cfg.anti_features
self._playwright = None
self._browser = None
self._context = None
self._running = False
async def start(self):
if self._running:
return
from playwright.async_api import async_playwright
self._playwright = await async_playwright().start()
launch_args = [
"--disable-blink-features=AutomationControlled",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-infobars",
"--disable-background-timer-throttling",
"--disable-backgrounding-occluded-windows",
"--disable-renderer-backgrounding",
"--disable-features=IsolateOrigins,site-per-process",
]
if self._anti.disable_webrtc:
launch_args.append("--disable-webrtc")
browser_launcher = getattr(self._playwright, self._browser_type)
self._browser = await browser_launcher.launch(
headless=self._headless,
args=launch_args,
)
context_opts = {
"viewport": {"width": self._viewport.width, "height": self._viewport.height},
"locale": "zh-CN",
"timezone_id": "Asia/Shanghai",
}
if self._anti.random_viewport:
w_offset = random.randint(-50, 50)
h_offset = random.randint(-30, 30)
context_opts["viewport"] = {
"width": self._viewport.width + w_offset,
"height": self._viewport.height + h_offset,
}
self._context = await self._browser.new_context(**context_opts)
self._running = True
logger.info(f"隐身浏览器已启动 ({self._browser_type}, headless={self._headless})")
async def stop(self):
if self._context:
await self._context.close()
self._context = None
if self._browser:
await self._browser.close()
self._browser = None
if self._playwright:
await self._playwright.stop()
self._playwright = None
self._running = False
logger.info("隐身浏览器已关闭")
async def new_page(self):
if not self._running:
await self.start()
page = await self._context.new_page()
if self._use_stealth:
await self._apply_stealth(page)
return page
async def _apply_stealth(self, page):
"""应用全套反检测措施"""
try:
from playwright_stealth import stealth_async
await stealth_async(page)
logger.debug("playwright-stealth 反检测已应用")
except ImportError:
logger.debug("playwright-stealth 未安装，使用内置反检测")
await self._inject_anti_detection(page)
async def _inject_anti_detection(self, page):
"""注入反检测 JS"""
if self._anti.remove_webdriver:
await page.add_init_script("""
Object.defineProperty(navigator, 'webdriver', {
get: () => undefined,
});
// 删除 __webdriver_evaluate 等属性
delete navigator.__webdriver_evaluate;
delete navigator.__webdriver_unwrap;
""")
if self._anti.fake_plugins:
await page.add_init_script("""
Object.defineProperty(navigator, 'plugins', {
get: () => {
const plugins = [
{name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer'},
{name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai'},
{name: 'Native Client', filename: 'internal-nacl-plugin'},
];
plugins.length = 3;
return plugins;
},
});
Object.defineProperty(navigator, 'mimeTypes', {
get: () => {
const mimes = [
{type: 'application/pdf', suffixes: 'pdf'},
{type: 'application/x-nacl', suffixes: ''},
];
mimes.length = 2;
return mimes;
},
});
""")
if self._anti.fake_languages:
await page.add_init_script("""
Object.defineProperty(navigator, 'languages', {
get: () => ['zh-CN', 'zh', 'en-US', 'en'],
});
Object.defineProperty(navigator, 'language', {
get: () => 'zh-CN',
});
""")
if self._anti.fake_chrome_runtime:
await page.add_init_script("""
window.chrome = {
app: { isInstalled: false, InstallState: {DISABLED: 'disabled', INSTALLED: 'installed', NOT_INSTALLED: 'not_installed'}, RunningState: {CANNOT_RUN: 'cannot_run', READY_TO_RUN: 'ready_to_run', RUNNING: 'running'} },
runtime: { OnInstalledReason: {CHROME_UPDATE: 'chrome_update', INSTALL: 'install', SHARED_MODULE_UPDATE: 'shared_module_update', UPDATE: 'update'}, OnRestartRequiredReason: {APP_UPDATE: 'app_update', OS_UPDATE: 'os_update', PERIODIC: 'periodic'}, PlatformArch: {ARM: 'arm', MIPS: 'mips', MIPS64: 'mips64', X86_32: 'x86-32', X86_64: 'x86-64'}, PlatformNaclArch: {ARM: 'arm', MIPS: 'mips', MIPS64: 'mips64', X86_32: 'x86-32', X86_64: 'x86-64'}, PlatformOs: {ANDROID: 'android', CROS: 'cros', LINUX: 'linux', MAC: 'mac', OPENBSD: 'openbsd', WIN: 'win'}, RequestUpdateCheckStatus: {NO_UPDATE: 'no_update', THROTTLED: 'throttled', UPDATE_AVAILABLE: 'update_available'} },
};
""")
# 隐藏自动化 permissions
await page.add_init_script("""
const originalQuery = window.navigator.permissions.query;
window.navigator.permissions.query = (parameters) => (
parameters.name === 'notifications' ?
Promise.resolve({ state: Notification.permission }) :
originalQuery(parameters)
);
""")
# 伪造 connection.rtt
await page.add_init_script("""
if (navigator.connection) {
Object.defineProperty(navigator.connection, 'rtt', { get: () => 50 });
}
""")
async def human_scroll(self, page, times: int = 3):
"""模拟人类滚动行为"""
for _ in range(times):
scroll_y = random.randint(100, 400)
await page.mouse.wheel(0, scroll_y)
await asyncio.sleep(random.uniform(0.3, 1.0))
async def human_mouse_move(self, page, x: int, y: int):
"""模拟人类鼠标移动（贝塞尔曲线）"""
steps = random.randint(10, 25)
current_x, current_y = 0, 0
for i in range(steps):
t = (i + 1) / steps
next_x = int(current_x + (x - current_x) * t + random.randint(-5, 5))
next_y = int(current_y + (y - current_y) * t + random.randint(-3, 3))
await page.mouse.move(next_x, next_y)
await asyncio.sleep(random.uniform(0.01, 0.05))
async def fetch_page(
self,
url: str,
wait_selector: Optional[str] = None,
wait_time: float = 0,
extract_json: bool = False,
) -> Dict[str, Any]:
"""使用隐身浏览器获取页面"""
page = None
try:
page = await self.new_page()
start = time.time()
await page.goto(url, wait_until="networkidle", timeout=30000)
nav_time = time.time() - start
if wait_selector:
await page.wait_for_selector(wait_selector, timeout=10000)
if self._anti.human_mouse:
await self.human_mouse_move(page, random.randint(200, 800), random.randint(200, 600))
if self._anti.random_scroll:
await self.human_scroll(page, times=random.randint(1, 3))
if wait_time > 0:
await asyncio.sleep(wait_time)
content = await page.content()
title = await page.title()
total_time = time.time() - start
# 检测反爬页面
if self._is_blocked_page(content, title):
raise BlockedError(f"检测到反爬页面: {title}")
result = {
"success": True,
"url": page.url,
"title": title,
"content": content,
"content_size": len(content),
"nav_time": nav_time,
"total_time": total_time,
}
if extract_json:
json_data = await self._extract_json(page)
if json_data:
result["json_data"] = json_data
return result
except (BlockedError, CaptchaError):
raise
except Exception as e:
logger.error(f"页面获取失败 {url}: {e}")
return {"success": False, "url": url, "error": str(e)}
finally:
if page:
await page.close()
def _is_blocked_page(self, content: str, title: str) -> bool:
blocked_keywords = ["验证码", "Captcha", "Access Denied", "403 Forbidden", "请稍后", "访问频繁"]
text = (content[:3000] + title).lower()
return any(kw.lower() in text for kw in blocked_keywords)
async def _extract_json(self, page) -> Optional[Any]:
"""从页面中提取 JSON 数据（用于 API 页面）"""
try:
text = await page.evaluate("() => document.body.innerText")
import json
return json.loads(text)
except Exception:
return None
async def __aenter__(self):
await self.start()
return self
async def __aexit__(self, *args):
await self.stop()
server/public/dongqiudi-crawler/src/anti_detect/tls_client.py
-148
View File
@@ -1,148 +0,0 @@
"""
TLS 指纹伪装客户端
使用 curl_cffi 模拟真实浏览器的 TLS 握手指纹 (JA3/JA4)
这是绕过懂球帝 TLS 指纹检测的核心模块
"""
import asyncio
import random
import time
from typing import Dict, Any, Optional
from urllib.parse import urlencode
from loguru import logger
from src.core.config import get_config
from src.core.exceptions import NetworkError, RateLimitError, BlockedError, CaptchaError
# curl_cffi 支持的浏览器指纹列表
IMPERSONATE_TARGETS = [
"chrome120", "chrome123", "chrome124",
"edge101", "edge99",
"safari15_5", "safari17_0",
]
class TLSClient:
"""
基于 curl_cffi 的 TLS 指纹伪装 HTTP 客户端
能够模拟真实浏览器的 JA3 指纹，绕过 TLS 指纹检测
"""
def __init__(self, impersonate: Optional[str] = None):
cfg = get_config()
self._impersonate = impersonate or cfg.anti_detect.tls.impersonate
self._timeout = cfg.crawler.request_timeout
self._session = None
self._request_count = 0
self._max_per_session = cfg.crawler.session.max_requests_per_session
self._created_at = None
async def _ensure_session(self):
if self._session is None or self._should_rotate():
await self._rotate_session()
def _should_rotate(self) -> bool:
if self._request_count >= self._max_per_session:
return True
cfg = get_config().crawler.session
if self._created_at and (time.time() - self._created_at) > cfg.rotate_interval:
return True
return False
async def _rotate_session(self):
await self.close()
try:
from curl_cffi.requests import AsyncSession
target = random.choice(IMPERSONATE_TARGETS) if self._impersonate == "random" else self._impersonate
self._session = AsyncSession(impersonate=target, timeout=self._timeout)
self._request_count = 0
self._created_at = time.time()
logger.debug(f"TLS 会话已创建，伪装: {target}")
except ImportError:
logger.warning("curl_cffi 未安装，回退到 aiohttp（无 TLS 指纹伪装）")
self._session = None
raise
async def get(
self,
url: str,
params: Optional[Dict[str, Any]] = None,
headers: Optional[Dict[str, str]] = None,
proxy: Optional[str] = None,
) -> Dict[str, Any]:
await self._ensure_session()
self._request_count += 1
full_url = f"{url}?{urlencode(params)}" if params else url
logger.info(f"[tls] GET {full_url}")
start = time.time()
try:
resp = await self._session.get(
url,
params=params,
headers=headers,
proxy=proxy,
allow_redirects=True,
)
elapsed = time.time() - start
if resp.status_code == 200:
try:
data = resp.json()
except Exception:
data = resp.text
resp_summary = str(data)[:300] if data else '(empty)'
logger.info(f"[tls] 200 OK {elapsed:.2f}s {full_url} resp={resp_summary}")
return {
"success": True,
"status": resp.status_code,
"data": data,
"elapsed": elapsed,
"headers": dict(resp.headers),
}
elif resp.status_code == 429:
logger.warning(f"[tls] 429 RateLimit {elapsed:.2f}s {full_url}")
retry_after = int(resp.headers.get("Retry-After", 60))
raise RateLimitError(retry_after=retry_after)
elif resp.status_code == 403:
logger.warning(f"[tls] 403 Forbidden {elapsed:.2f}s {full_url}")
raise BlockedError(f"403 Forbidden: {url}")
elif resp.status_code == 503:
text = resp.text
logger.warning(f"[tls] 503 {elapsed:.2f}s {full_url}")
if "captcha" in text.lower() or "验证码" in text:
raise CaptchaError("触发验证码")
raise NetworkError(f"503 Service Unavailable: {url}")
else:
logger.warning(f"[tls] HTTP {resp.status_code} {elapsed:.2f}s {full_url}")
raise NetworkError(f"HTTP {resp.status_code}: {url}")
except (RateLimitError, BlockedError, CaptchaError):
raise
except Exception as e:
elapsed = time.time() - start
if "curl_cffi" in str(type(e).__module__):
raise NetworkError(f"TLS 请求失败: {e}") from e
raise
async def close(self):
if self._session:
try:
await self._session.close()
except Exception:
pass
self._session = None
self._created_at = None
async def __aenter__(self):
await self._ensure_session()
return self
async def __aexit__(self, *args):
await self.close()
server/public/dongqiudi-crawler/src/core/__init__.py
View File
server/public/dongqiudi-crawler/src/core/alert_manager.py
-183
View File
@@ -1,183 +0,0 @@
import json
from datetime import datetime
from typing import Any, Dict, Optional
import aiohttp
from loguru import logger
from src.core.config import get_config
from src.storage.database import Database
class CrontabAlertManager:
def __init__(self, db: Optional[Database] = None):
self._cfg = get_config().alert
self._db = db or Database()
@staticmethod
def build_dedupe_key(crontab_id: int, alert_type: str) -> str:
return f"crontab:{int(crontab_id)}:{alert_type}"
@staticmethod
def _pick_http_status(severe_http_statuses: list[int]) -> int:
statuses = [int(item) for item in severe_http_statuses if item is not None]
if not statuses:
return 0
return sorted(statuses)[0]
async def record_crawler_result(
self,
action: str,
crontab_id: int,
crontab_log_id: int,
task_result: Dict[str, Any],
output: str,
):
await self._db.ensure_alert_tables()
if action == "error_report":
return
policy = task_result.get("policy", "saved_required")
candidate_count = int(task_result.get("candidate_count") or 0)
saved_count = int(task_result.get("saved_count") or 0)
http_error_count = int(task_result.get("http_error_count") or 0)
severe_http_statuses = list(task_result.get("severe_http_statuses") or [])
execution_success = bool(task_result.get("execution_success", True))
success = bool(task_result.get("success"))
error_message = str(task_result.get("error_message") or "")
summary_text = str(task_result.get("summary_text") or "")
alert_type = ""
http_status = 0
if http_error_count > 0:
alert_type = "http_error"
http_status = self._pick_http_status(severe_http_statuses)
elif not execution_success or (not success and error_message):
alert_type = "task_exception"
elif policy == "saved_required" and saved_count <= 0:
alert_type = "no_data"
elif policy == "saved_if_candidates" and candidate_count > 0 and saved_count <= 0:
alert_type = "no_data"
if not alert_type:
await self._db.resolve_crontab_alerts(crontab_id)
return
detail = {
"summary_text": summary_text[:1000],
"candidate_count": candidate_count,
"saved_count": saved_count,
"http_error_count": http_error_count,
"severe_http_statuses": severe_http_statuses,
"error_message": error_message[:1000],
"output_excerpt": output[:2000],
}
await self._db.upsert_crontab_alert(
crontab_id=crontab_id,
crontab_log_id=crontab_log_id,
task_name=str(task_result.get("task_name") or action),
command=str(task_result.get("command") or "crawler"),
params=str(task_result.get("params") or action),
alert_type=alert_type,
http_status=http_status,
detail=detail,
dedupe_key=self.build_dedupe_key(crontab_id, alert_type),
)
async def dispatch_open_alerts(self, limit: int = 50) -> Dict[str, Any]:
await self._db.ensure_alert_tables()
alerts = await self._db.get_dispatchable_crontab_alerts(self._cfg.cooldown_seconds, limit=limit)
if not alerts:
return {
"success": True,
"count": 0,
"candidate_count": 0,
"saved_count": 0,
"summary_text": "无待发送告警",
}
if not self._cfg.dispatch_enabled or not self._cfg.wecom_webhook:
logger.warning(f"[alert] 企业微信告警未启用或 webhook 未配置，当前待发送 {len(alerts)} 条")
return {
"success": True,
"count": 0,
"candidate_count": len(alerts),
"saved_count": 0,
"summary_text": f"企业微信告警未启用或 webhook 未配置，待发送 {len(alerts)} 条",
}
sent = 0
failed = 0
for alert in alerts:
ok = await self._send_wecom_alert(alert)
if ok:
await self._db.mark_crontab_alert_notified(int(alert["id"]))
sent += 1
else:
failed += 1
return {
"success": failed == 0,
"count": sent,
"candidate_count": len(alerts),
"saved_count": sent,
"failed": failed,
"summary_text": f"企业微信告警发送完成：成功 {sent} 条，失败 {failed} 条，待发送 {len(alerts)} 条",
}
async def _send_wecom_alert(self, alert: Dict[str, Any]) -> bool:
message = self._build_markdown(alert)
payload = {
"msgtype": "markdown",
"markdown": {"content": message},
}
timeout = aiohttp.ClientTimeout(total=max(3, int(self._cfg.request_timeout)))
try:
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.post(self._cfg.wecom_webhook, json=payload) as resp:
body = await resp.text()
if resp.status != 200:
logger.warning(f"[alert] 企业微信发送失败 HTTP {resp.status}: {body[:300]}")
return False
data = json.loads(body)
if int(data.get("errcode", -1)) != 0:
logger.warning(f"[alert] 企业微信返回失败: {body[:300]}")
return False
return True
except Exception as e:
logger.warning(f"[alert] 企业微信发送异常: {e}")
return False
@staticmethod
def _build_markdown(alert: Dict[str, Any]) -> str:
detail_raw = alert.get("detail") or ""
try:
detail = json.loads(detail_raw) if isinstance(detail_raw, str) else (detail_raw or {})
except Exception:
detail = {"summary_text": str(detail_raw)}
def fmt_ts(value: Any) -> str:
try:
ts = int(value or 0)
if ts <= 0:
return "-"
return datetime.fromtimestamp(ts).strftime("%Y-%m-%d %H:%M:%S")
except Exception:
return "-"
severe = detail.get("severe_http_statuses") or []
http_text = "/".join(str(item) for item in severe) if severe else (str(alert.get("http_status") or "-"))
summary = str(detail.get("summary_text") or detail.get("error_message") or "无摘要").replace("\n", " ")
return "\n".join([
"### Sport Era 定时任务告警",
f"> 任务：`{alert.get('task_name', '')}` / #{alert.get('crontab_id', 0)}",
f"> 命令：`{alert.get('command', '')} {alert.get('params', '')}`",
f"> 规则：`{alert.get('expression') or '-'}`",
f"> 类型：`{alert.get('alert_type', '')}`",
f"> HTTP：`{http_text}`",
f"> 保存：`{detail.get('saved_count', 0)}` / 候选：`{detail.get('candidate_count', 0)}` / 请求错误：`{detail.get('http_error_count', 0)}`",
f"> 首次发生：`{fmt_ts(alert.get('first_seen_at'))}`",
f"> 最近发生：`{fmt_ts(alert.get('last_seen_at'))}`",
f"> 摘要：{summary[:1000]}",
])
server/public/dongqiudi-crawler/src/core/config.py
-266
View File
@@ -1,266 +0,0 @@
"""
配置加载模块
"""
import os
from pathlib import Path
from typing import Dict, List, Optional, Any
import yaml
from pydantic import BaseModel, Field
class DatabaseConfig(BaseModel):
host: str = "localhost"
port: int = 3306
database: str = ""
username: str = ""
password: str = ""
charset: str = "utf8mb4"
prefix: str = "la_"
pool_size: int = 10
class RedisConfig(BaseModel):
host: str = "localhost"
port: int = 6379
password: str = ""
db: int = 0
class DelayConfig(BaseModel):
base: float = 3.0
min: float = 2.0
max: float = 8.0
jitter: bool = True
class SessionConfig(BaseModel):
rotate_interval: int = 300
max_requests_per_session: int = 50
class CrawlerConfig(BaseModel):
mode: str = "hybrid"
max_concurrent: int = 3
request_timeout: int = 30
max_retries: int = 3
retry_backoff: float = 2.0
delay: DelayConfig = Field(default_factory=DelayConfig)
session: SessionConfig = Field(default_factory=SessionConfig)
class FingerprintConfig(BaseModel):
pool_size: int = 50
min_success_rate: float = 0.6
rotation_interval: int = 300
cooldown: int = 60
class TLSConfig(BaseModel):
enabled: bool = True
impersonate: str = "chrome124"
class ViewportConfig(BaseModel):
width: int = 1920
height: int = 1080
class AntiFeaturesConfig(BaseModel):
remove_webdriver: bool = True
fake_plugins: bool = True
fake_languages: bool = True
fake_chrome_runtime: bool = True
disable_webrtc: bool = True
random_viewport: bool = True
human_mouse: bool = True
random_scroll: bool = True
class PlaywrightAntiConfig(BaseModel):
browser: str = "chromium"
headless: bool = True
stealth: bool = True
viewport: ViewportConfig = Field(default_factory=ViewportConfig)
anti_features: AntiFeaturesConfig = Field(default_factory=AntiFeaturesConfig)
class ProxyConfig(BaseModel):
enabled: bool = False
rotate: bool = True
providers: List[str] = Field(default_factory=list)
test_url: str = "https://httpbin.org/ip"
max_failures: int = 3
class AntiDetectConfig(BaseModel):
fingerprint: FingerprintConfig = Field(default_factory=FingerprintConfig)
tls: TLSConfig = Field(default_factory=TLSConfig)
playwright: PlaywrightAntiConfig = Field(default_factory=PlaywrightAntiConfig)
proxy: ProxyConfig = Field(default_factory=ProxyConfig)
class APIEndpoints(BaseModel):
standings: str = "/soccer/biz/data/standing"
schedule: str = "/soccer/biz/data/schedule"
match_detail: str = "/soccer/biz/data/match_detail"
match_menu: str = "/api/v2/config/match_menu"
team_info: str = "/soccer/biz/data/team_info"
player_stats: str = "/soccer/biz/data/player_stats"
live_matches: str = "/soccer/biz/data/live"
match_situation: str = "/mobile/match/situation"
match_lineup: str = "/mobile/match/lineup"
news: str = "/api/v2/article/list"
class DefaultParams(BaseModel):
app: str = "dqd"
platform: str = "www"
version: str = "0"
language: str = "zh-cn"
class TargetLeague(BaseModel):
league: str
league_code: str = ""
season_id: int
priority: int = 100
active: bool = True
class DongqiudiConfig(BaseModel):
base_url: str = "https://sport-data.dongqiudi.com"
web_url: str = "https://www.dongqiudi.com"
api: APIEndpoints = Field(default_factory=APIEndpoints)
default_params: DefaultParams = Field(default_factory=DefaultParams)
targets: List[TargetLeague] = Field(default_factory=list)
class LotterySource(BaseModel):
name: str
code: str
category_id: int
url: str
class LotteryConfig(BaseModel):
sources: List[LotterySource] = Field(default_factory=list)
class SchedulerConfig(BaseModel):
standings_cron: str = "0 2 * * *"
schedule_cron: str = "0 */6 * * *"
live_cron: str = "*/5 * * * *"
news_cron: str = "*/30 * * * *"
content_cron: str = "*/10 * * * *"
video_cron: str = "*/30 * * * *"
csl_match_cron: str = "0 */1 * * *"
nba_match_cron: str = "0 */1 * * *"
cba_match_cron: str = "0 */1 * * *"
epl_match_cron: str = "0 */1 * * *"
bundesliga_match_cron: str = "0 */1 * * *"
laliga_match_cron: str = "0 */1 * * *"
seriea_match_cron: str = "0 */1 * * *"
ligue1_match_cron: str = "0 */1 * * *"
ucl_match_cron: str = "0 */1 * * *"
uel_match_cron: str = "0 */1 * * *"
tennis_match_cron: str = "0 */1 * * *"
esports_match_cron: str = "0 */1 * * *"
sports_match_cron: str = "0 */1 * * *"
league_news_cron: str = "*/30 * * * *"
article_content_cron: str = "*/1 * * * *"
class LoggingConfig(BaseModel):
level: str = "INFO"
file: str = "logs/crawler.log"
max_size_mb: int = 50
backup_count: int = 5
format: str = "{time:YYYY-MM-DD HH:mm:ss} | {level:<8} | {name}:{function}:{line} - {message}"
class AlertConfig(BaseModel):
dispatch_enabled: bool = False
wecom_webhook: str = ""
cooldown_seconds: int = 1800
request_timeout: int = 10
class AppConfig(BaseModel):
database: DatabaseConfig = Field(default_factory=DatabaseConfig)
redis: RedisConfig = Field(default_factory=RedisConfig)
crawler: CrawlerConfig = Field(default_factory=CrawlerConfig)
anti_detect: AntiDetectConfig = Field(default_factory=AntiDetectConfig)
dongqiudi: DongqiudiConfig = Field(default_factory=DongqiudiConfig)
lottery: LotteryConfig = Field(default_factory=LotteryConfig)
scheduler: SchedulerConfig = Field(default_factory=SchedulerConfig)
logging: LoggingConfig = Field(default_factory=LoggingConfig)
alert: AlertConfig = Field(default_factory=AlertConfig)
_config: Optional[AppConfig] = None
def load_config(config_path: Optional[str] = None) -> AppConfig:
global _config
if config_path is None:
config_path = os.environ.get("DQD_CONFIG_PATH")
if config_path is None:
candidates = [
Path(__file__).parent.parent.parent / "config" / "settings.yaml",
Path("config/settings.yaml"),
]
for p in candidates:
if p.exists():
config_path = str(p)
break
if config_path and Path(config_path).exists():
with open(config_path, "r", encoding="utf-8") as f:
raw = yaml.safe_load(f) or {}
raw = _apply_env_overrides(raw)
_config = AppConfig(**raw)
else:
_config = AppConfig(**_apply_env_overrides({}))
return _config
def _apply_env_overrides(raw: Dict[str, Any]) -> Dict[str, Any]:
mapping = {
"DQD_DB_HOST": ("database", "host", str),
"DQD_DB_PORT": ("database", "port", int),
"DQD_DB_NAME": ("database", "database", str),
"DQD_DB_USER": ("database", "username", str),
"DQD_DB_PASSWORD": ("database", "password", str),
"DQD_DB_PREFIX": ("database", "prefix", str),
"DQD_DB_CHARSET": ("database", "charset", str),
"DQD_DB_POOL_SIZE": ("database", "pool_size", int),
"DQD_REDIS_HOST": ("redis", "host", str),
"DQD_REDIS_PORT": ("redis", "port", int),
"DQD_REDIS_PASSWORD": ("redis", "password", str),
"DQD_REDIS_DB": ("redis", "db", int),
"DQD_LOG_FILE": ("logging", "file", str),
"DQD_LOG_LEVEL": ("logging", "level", str),
}
for env_name, (section, key, caster) in mapping.items():
if env_name not in os.environ:
continue
value = os.environ.get(env_name, "")
try:
cast_value = caster(value)
except (TypeError, ValueError):
continue
raw.setdefault(section, {})
raw[section][key] = cast_value
return raw
def get_config() -> AppConfig:
global _config
if _config is None:
_config = load_config()
return _config
server/public/dongqiudi-crawler/src/core/error_collector.py
-93
View File
@@ -1,93 +0,0 @@
"""
全局错误收集器
- 收集所有HTTP请求错误，异步写入数据库
- 不阻塞主流程，写入失败只打日志
"""
import asyncio
from typing import Optional
from loguru import logger
class ErrorCollector:
_instance: Optional["ErrorCollector"] = None
_task_name: str = ""
def __init__(self):
self._db = None
self.reset_run_stats()
@classmethod
def get(cls) -> "ErrorCollector":
if cls._instance is None:
cls._instance = cls()
return cls._instance
def set_task_name(self, name: str):
self._task_name = name
def reset_run_stats(self):
self._http_error_count = 0
self._severe_http_statuses = set()
self._recent_errors = []
def get_run_stats(self) -> dict:
return {
"http_error_count": self._http_error_count,
"severe_http_statuses": sorted(self._severe_http_statuses),
"recent_errors": list(self._recent_errors),
}
def _record_run_error(self, request_url: str, http_status: int, error_type: str, error_message: str):
should_count = http_status == 404 or http_status == 0 or http_status >= 500
if should_count:
self._http_error_count += 1
self._severe_http_statuses.add(int(http_status))
if len(self._recent_errors) < 10:
self._recent_errors.append({
"request_url": request_url[:300],
"http_status": int(http_status),
"error_type": error_type[:100],
"error_message": error_message[:500],
})
async def _get_db(self):
if self._db is None:
from src.storage.database import Database
self._db = Database()
await self._db.connect()
return self._db
async def report(
self,
request_url: str,
request_method: str = "GET",
request_params: str = "",
http_status: int = 0,
error_type: str = "",
error_message: str = "",
response_body: str = "",
channel: str = "",
):
try:
self._record_run_error(request_url, http_status, error_type, error_message)
db = await self._get_db()
await db.insert_error_log(
task_name=self._task_name,
request_url=request_url,
request_method=request_method,
request_params=request_params,
http_status=http_status,
error_type=error_type,
error_message=error_message,
response_body=response_body,
channel=channel,
)
except Exception as e:
logger.debug(f"[error_collector] 写入错误日志失败: {e}")
async def close(self):
if self._db:
await self._db.close()
self._db = None
server/public/dongqiudi-crawler/src/core/exceptions.py
-45
View File
@@ -1,45 +0,0 @@
"""
异常定义
"""
class CrawlerError(Exception):
pass
class RateLimitError(CrawlerError):
def __init__(self, message="频率限制", retry_after: int = 60):
self.retry_after = retry_after
super().__init__(message)
class BlockedError(CrawlerError):
pass
class CaptchaError(CrawlerError):
pass
class NetworkError(CrawlerError):
pass
class ParsingError(CrawlerError):
pass
class FingerprintError(CrawlerError):
pass
class ProxyError(CrawlerError):
pass
class AntiDetectError(CrawlerError):
pass
class SessionExpiredError(CrawlerError):
pass
server/public/dongqiudi-crawler/src/core/logger.py
-36
View File
@@ -1,36 +0,0 @@
"""
日志模块 - 基于 loguru
"""
import sys
from pathlib import Path
from loguru import logger
from src.core.config import get_config
def setup_logger():
cfg = get_config().logging
logger.remove()
logger.add(
sys.stderr,
level=cfg.level,
format=cfg.format,
colorize=True,
)
log_path = Path(cfg.file)
log_path.parent.mkdir(parents=True, exist_ok=True)
logger.add(
str(log_path),
level=cfg.level,
format=cfg.format,
rotation=f"{cfg.max_size_mb} MB",
retention=cfg.backup_count,
encoding="utf-8",
enqueue=True,
)
return logger
server/public/dongqiudi-crawler/src/engine/__init__.py
View File
server/public/dongqiudi-crawler/src/engine/api_client.py
-275
View File
@@ -1,275 +0,0 @@
"""
懂球帝 API 客户端
- 支持 aiohttp / curl_cffi 双通道
- 自动指纹轮换、频率控制、重试
"""
import asyncio
import random
import time
from typing import Dict, Any, Optional
from urllib.parse import urlencode
import aiohttp
from loguru import logger
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from src.core.config import get_config
from src.core.exceptions import (
NetworkError, RateLimitError, BlockedError, CaptchaError, ParsingError,
)
from src.anti_detect.fingerprint import FingerprintPool
class APIClient:
"""懂球帝 API 客户端 (aiohttp 通道)"""
def __init__(self, fingerprint_pool: Optional[FingerprintPool] = None):
self._cfg = get_config()
self._fp_pool = fingerprint_pool
self._session: Optional[aiohttp.ClientSession] = None
self._request_count = 0
self._created_at: Optional[float] = None
self._stats = {
"total": 0, "success": 0, "fail": 0,
"rate_limit": 0, "blocked": 0, "total_time": 0.0,
}
async def initialize(self):
if self._session is None or self._should_rotate():
await self._new_session()
async def _new_session(self):
await self.close()
timeout = aiohttp.ClientTimeout(total=self._cfg.crawler.request_timeout)
self._session = aiohttp.ClientSession(timeout=timeout)
self._request_count = 0
self._created_at = time.time()
def _should_rotate(self) -> bool:
scfg = self._cfg.crawler.session
if self._request_count >= scfg.max_requests_per_session:
return True
if self._created_at and (time.time() - self._created_at) > scfg.rotate_interval:
return True
return False
async def close(self):
if self._session:
await self._session.close()
self._session = None
async def _get_headers(self) -> tuple:
"""获取请求头，返回 (headers, fp_id)"""
if self._fp_pool:
fp = await self._fp_pool.acquire()
return fp.get("http_headers", {}), fp.get("id")
return {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
"Referer": "https://www.dongqiudi.com/",
"Origin": "https://www.dongqiudi.com",
}, None
async def request(
self,
url: str,
params: Optional[Dict[str, Any]] = None,
proxy: Optional[str] = None,
) -> Dict[str, Any]:
await self.initialize()
headers, fp_id = await self._get_headers()
self._request_count += 1
self._stats["total"] += 1
start = time.time()
full_url = f"{url}?{urlencode(params)}" if params else url
logger.info(f"[api] GET {full_url}")
try:
async with self._session.get(
url, params=params, headers=headers, proxy=proxy,
allow_redirects=True, ssl=False,
) as resp:
elapsed = time.time() - start
self._stats["total_time"] += elapsed
return await self._handle_response(resp, url, fp_id, elapsed, params)
except (RateLimitError, BlockedError, CaptchaError):
self._stats["fail"] += 1
raise
except aiohttp.ClientError as e:
self._stats["fail"] += 1
if fp_id and self._fp_pool:
await self._fp_pool.mark_failure(fp_id, "network")
raise NetworkError(f"网络错误: {e}") from e
except Exception as e:
self._stats["fail"] += 1
raise NetworkError(f"请求异常: {e}") from e
async def _handle_response(
self, resp: aiohttp.ClientResponse, url: str, fp_id: Optional[str], elapsed: float,
params: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
status = resp.status
full_url = f"{url}?{urlencode(params)}" if params else url
if status == 200:
try:
data = await resp.json(content_type=None)
except Exception:
text = await resp.text()
raise ParsingError(f"JSON 解析失败，原始响应: {text[:500]}")
resp_summary = str(data)[:300] if data else '(empty)'
logger.info(f"[api] 200 OK {elapsed:.2f}s {full_url} resp={resp_summary}")
self._stats["success"] += 1
if fp_id and self._fp_pool:
await self._fp_pool.mark_success(fp_id)
return data
elif status == 429:
logger.warning(f"[api] 429 RateLimit {elapsed:.2f}s {full_url}")
self._stats["rate_limit"] += 1
retry_after = int(resp.headers.get("Retry-After", 60))
if fp_id and self._fp_pool:
await self._fp_pool.mark_failure(fp_id, "rate_limit")
raise RateLimitError(retry_after=retry_after)
elif status == 403:
logger.warning(f"[api] 403 Forbidden {elapsed:.2f}s {full_url}")
self._stats["blocked"] += 1
if fp_id and self._fp_pool:
await self._fp_pool.mark_failure(fp_id, "ip_blocked")
raise BlockedError(f"403: {url}")
elif status == 503:
text = await resp.text()
logger.warning(f"[api] 503 {elapsed:.2f}s {full_url}")
if "captcha" in text.lower() or "验证码" in text:
if fp_id and self._fp_pool:
await self._fp_pool.mark_failure(fp_id, "captcha")
raise CaptchaError("验证码")
raise NetworkError(f"503: {url}")
else:
logger.warning(f"[api] HTTP {status} {elapsed:.2f}s {full_url}")
raise NetworkError(f"HTTP {status}: {url}")
# ── 懂球帝专用 API 方法 ──
def _default_params(self, extra: Optional[Dict] = None) -> Dict[str, Any]:
dp = self._cfg.dongqiudi.default_params
params = {"app": dp.app, "platform": dp.platform, "version": dp.version, "language": dp.language}
if extra:
params.update(extra)
return params
async def get_standings(self, season_id: int) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.standings
params = self._default_params({"season_id": season_id})
return await self.request(url, params)
async def get_schedule(self, season_id: int, round_number: Optional[int] = None) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.schedule
extra = {"season_id": season_id}
if round_number:
extra["round"] = round_number
return await self.request(url, self._default_params(extra))
async def get_match_detail(self, match_id: int) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.match_detail
return await self.request(url, self._default_params({"match_id": match_id}))
async def get_match_menu(self) -> Dict[str, Any]:
url = self._cfg.dongqiudi.web_url + self._cfg.dongqiudi.api.match_menu
params = {"mark": "gif", "platform": "www", "version": "713"}
return await self.request(url, params)
async def get_live_matches(self) -> Dict[str, Any]:
"""从各联赛 tab API 汇总当前比赛数据"""
all_matches = []
league_ids = set()
for t in self._cfg.dongqiudi.targets:
if not t.active:
continue
lid = self._get_league_id(t.league)
if lid and lid not in league_ids:
league_ids.add(lid)
try:
url = f"https://api.dongqiudi.com/data/tab/league/new/{lid}"
raw = await self.request(url, None)
items = raw.get("list", []) if isinstance(raw, dict) else []
all_matches.extend(items)
except Exception as e:
logger.warning(f"获取联赛 {t.league} 实时数据失败: {e}")
return {"data": {"matches": all_matches}}
_LEAGUE_ID_MAP = {
"中超": 43, "中甲": 129, "亚冠": 226, "英超": 4, "西甲": 3,
"德甲": 5, "意甲": 9, "法甲": 12, "欧冠": 6, "欧联": 14,
}
def _get_league_id(self, league_name: str) -> Optional[int]:
return self._LEAGUE_ID_MAP.get(league_name)
async def get_news(self, tab_id: int = 1, size: int = 30) -> Dict[str, Any]:
"""通过 tabs API 获取资讯列表"""
url = f"{self._cfg.dongqiudi.web_url}/api/app/tabs/web/{tab_id}.json"
params = {"size": size} if size != 20 else None
return await self.request(url, params)
async def get_video_list(self, tab_id: int = 233) -> Dict[str, Any]:
"""抓取视频列表页 HTML，返回 {'html': str, 'tab_id': int}"""
await self.initialize()
url = f"{self._cfg.dongqiudi.web_url}/videoList/{tab_id}"
headers, fp_id = await self._get_headers()
headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
self._request_count += 1
self._stats["total"] += 1
try:
async with self._session.get(url, headers=headers, ssl=False) as resp:
if resp.status == 200:
html = await resp.text()
self._stats["success"] += 1
return {"html": html, "tab_id": tab_id}
raise NetworkError(f"HTTP {resp.status}: {url}")
except aiohttp.ClientError as e:
self._stats["fail"] += 1
raise NetworkError(f"网络错误: {e}") from e
async def get_article_detail(self, article_id: int) -> Dict[str, Any]:
"""抓取文章详情页 HTML，返回 {'html': str}"""
await self.initialize()
url = f"{self._cfg.dongqiudi.web_url}/articles/{article_id}.html"
headers, fp_id = await self._get_headers()
headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
self._request_count += 1
self._stats["total"] += 1
try:
async with self._session.get(url, headers=headers, ssl=False) as resp:
if resp.status == 200:
html = await resp.text()
self._stats["success"] += 1
return {"html": html, "article_id": article_id}
raise NetworkError(f"HTTP {resp.status}: {url}")
except aiohttp.ClientError as e:
self._stats["fail"] += 1
raise NetworkError(f"网络错误: {e}") from e
@property
def stats(self) -> Dict[str, Any]:
total = self._stats["total"]
return {
**self._stats,
"success_rate": self._stats["success"] / total if total else 1.0,
"avg_time": self._stats["total_time"] / total if total else 0.0,
}
async def __aenter__(self):
await self.initialize()
return self
async def __aexit__(self, *args):
await self.close()
server/public/dongqiudi-crawler/src/engine/hybrid_engine.py
-497
View File
@@ -1,497 +0,0 @@
"""
混合爬虫引擎
- API 优先 → TLS 伪装降级 → Playwright 降级
- 智能选择最优通道、自动重试、频率控制
"""
import asyncio
import json
import random
import re
import time
from enum import Enum
from typing import Dict, Any, Optional, List
import aiohttp
from loguru import logger
from src.core.config import get_config
from src.core.error_collector import ErrorCollector
from src.core.exceptions import (
CrawlerError, RateLimitError, BlockedError, CaptchaError, NetworkError,
)
from src.anti_detect.fingerprint import FingerprintPool
from src.anti_detect.proxy_manager import ProxyManager
from src.engine.api_client import APIClient
class Channel(Enum):
API = "api"
TLS = "tls"
PLAYWRIGHT = "playwright"
class HybridEngine:
"""混合爬虫引擎"""
def __init__(self):
self._cfg = get_config()
self._mode = self._cfg.crawler.mode
# 组件
self._fp_pool = FingerprintPool()
self._proxy_mgr = ProxyManager()
self._api_client: Optional[APIClient] = None
self._tls_client = None
self._browser = None
# 通道统计（用于智能模式选择）
self._channel_stats: Dict[str, Dict[str, int]] = {
ch.value: {"success": 0, "fail": 0} for ch in Channel
}
self._initialized = False
async def initialize(self):
if self._initialized:
return
await self._fp_pool.initialize()
self._api_client = APIClient(self._fp_pool)
# TLS 客户端（可选）
if self._cfg.anti_detect.tls.enabled:
try:
from src.anti_detect.tls_client import TLSClient
self._tls_client = TLSClient()
logger.info("TLS 伪装客户端已就绪")
except Exception as e:
logger.warning(f"TLS 客户端初始化失败（curl_cffi 可能未安装）: {e}")
self._initialized = True
logger.info(f"混合引擎初始化完成，模式: {self._mode}")
async def close(self):
if self._api_client:
await self._api_client.close()
if self._tls_client:
await self._tls_client.close()
if self._browser:
await self._browser.stop()
self._initialized = False
# ── 通用请求方法 ──
async def request(
self,
url: str,
params: Optional[Dict[str, Any]] = None,
max_retries: Optional[int] = None,
) -> Dict[str, Any]:
"""
统一请求入口，根据模式自动选择通道并降级
"""
await self.initialize()
retries = max_retries or self._cfg.crawler.max_retries
channels = self._get_channel_order()
last_error = None
ec = ErrorCollector.get()
full_url = url
if params:
from urllib.parse import urlencode
full_url = f"{url}?{urlencode(params)}"
params_str = json.dumps(params, ensure_ascii=False) if params else ""
for attempt in range(retries):
for channel in channels:
try:
await self._smart_delay()
proxy = await self._proxy_mgr.get_proxy() if self._proxy_mgr.enabled else None
result = await self._dispatch(channel, url, params, proxy)
self._channel_stats[channel.value]["success"] += 1
return result
except RateLimitError as e:
self._channel_stats[channel.value]["fail"] += 1
wait = e.retry_after * (1 + random.uniform(0, 0.3))
logger.warning(f"[{channel.value}] 频率限制，等待 {wait:.0f}s")
await ec.report(request_url=full_url, request_params=params_str, http_status=429, error_type="RateLimitError", error_message=str(e), channel=channel.value)
await asyncio.sleep(wait)
last_error = e
except BlockedError as e:
self._channel_stats[channel.value]["fail"] += 1
logger.warning(f"[{channel.value}] IP 被封，尝试下一通道")
await ec.report(request_url=full_url, request_params=params_str, http_status=403, error_type="BlockedError", error_message=str(e), channel=channel.value)
last_error = e
continue
except CaptchaError as e:
self._channel_stats[channel.value]["fail"] += 1
logger.warning(f"[{channel.value}] 触发验证码，降级到 Playwright")
await ec.report(request_url=full_url, request_params=params_str, http_status=503, error_type="CaptchaError", error_message=str(e), channel=channel.value)
if channel != Channel.PLAYWRIGHT:
continue
last_error = e
except (NetworkError, CrawlerError) as e:
self._channel_stats[channel.value]["fail"] += 1
logger.warning(f"[{channel.value}] 请求失败: {e}")
http_code = 0
err_msg = str(e)
if "404" in err_msg:
http_code = 404
elif "500" in err_msg:
http_code = 500
await ec.report(request_url=full_url, request_params=params_str, http_status=http_code, error_type=type(e).__name__, error_message=err_msg, channel=channel.value)
last_error = e
continue
# 本轮所有通道都失败，退避后重试
backoff = self._cfg.crawler.retry_backoff ** attempt * (1 + random.uniform(0, 0.5))
logger.info(f"第 {attempt + 1} 轮全部失败，退避 {backoff:.1f}s 后重试")
await asyncio.sleep(backoff)
raise last_error or CrawlerError(f"所有通道和重试均失败: {url}")
async def _dispatch(
self, channel: Channel, url: str, params: Optional[Dict], proxy: Optional[str],
) -> Dict[str, Any]:
"""分发到具体通道"""
if channel == Channel.API:
return await self._api_client.request(url, params, proxy)
elif channel == Channel.TLS:
if not self._tls_client:
raise NetworkError("TLS 客户端不可用")
fp = await self._fp_pool.acquire()
headers = fp.get("http_headers", {})
result = await self._tls_client.get(url, params=params, headers=headers, proxy=proxy)
if result.get("success"):
return result["data"]
raise NetworkError(result.get("error", "TLS 请求失败"))
elif channel == Channel.PLAYWRIGHT:
if not self._browser:
from src.anti_detect.stealth_browser import StealthBrowser
self._browser = StealthBrowser()
await self._browser.start()
full_url = url
if params:
from urllib.parse import urlencode
full_url = f"{url}?{urlencode(params)}"
logger.info(f"[playwright] GET {full_url}")
result = await self._browser.fetch_page(full_url, extract_json=True)
if result.get("success") and result.get("json_data"):
resp_summary = str(result["json_data"])[:300]
logger.info(f"[playwright] 200 OK {full_url} resp={resp_summary}")
return result["json_data"]
elif result.get("success"):
logger.info(f"[playwright] OK (no json) {full_url}")
return result
logger.warning(f"[playwright] FAIL {full_url} error={result.get('error','')}")
raise NetworkError(result.get("error", "Playwright 请求失败"))
raise CrawlerError(f"未知通道: {channel}")
def _get_channel_order(self) -> List[Channel]:
"""根据模式和历史成功率决定通道优先级"""
if self._mode == "api_only":
return [Channel.API]
elif self._mode == "playwright_only":
return [Channel.PLAYWRIGHT]
elif self._mode == "hybrid":
order = [Channel.API]
if self._tls_client:
order.append(Channel.TLS)
order.append(Channel.PLAYWRIGHT)
return order
elif self._mode == "smart":
return self._smart_channel_order()
return [Channel.API, Channel.PLAYWRIGHT]
def _smart_channel_order(self) -> List[Channel]:
"""智能排序：按成功率排序"""
def _rate(ch: Channel) -> float:
s = self._channel_stats[ch.value]
total = s["success"] + s["fail"]
return s["success"] / total if total > 5 else 0.5
candidates = [Channel.API]
if self._tls_client:
candidates.append(Channel.TLS)
candidates.append(Channel.PLAYWRIGHT)
candidates.sort(key=_rate, reverse=True)
return candidates
async def _smart_delay(self):
"""高斯分布随机延迟"""
dcfg = self._cfg.crawler.delay
delay = random.gauss(dcfg.base, (dcfg.max - dcfg.min) / 4)
delay = max(dcfg.min, min(dcfg.max, delay))
if dcfg.jitter:
delay += random.uniform(-0.5, 0.5)
delay = max(0.5, delay)
await asyncio.sleep(delay)
# ── 懂球帝专用方法（通过混合引擎转发） ──
async def get_standings(self, season_id: int) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.standings
dp = self._cfg.dongqiudi.default_params
params = {"season_id": season_id, "app": dp.app, "platform": dp.platform, "version": dp.version, "language": dp.language}
return await self.request(url, params)
async def get_schedule(self, season_id: int, round_number: Optional[int] = None) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.schedule
dp = self._cfg.dongqiudi.default_params
params = {"season_id": season_id, "app": dp.app, "platform": dp.platform, "version": dp.version, "language": dp.language}
if round_number:
params["round"] = round_number
return await self.request(url, params)
async def get_match_detail(self, match_id: int) -> Dict[str, Any]:
url = self._cfg.dongqiudi.base_url + self._cfg.dongqiudi.api.match_detail
dp = self._cfg.dongqiudi.default_params
params = {"match_id": match_id, "app": dp.app, "platform": dp.platform, "version": dp.version, "language": dp.language}
return await self.request(url, params)
async def get_match_situation(self, match_id: int) -> Dict[str, Any]:
"""获取比赛实况数据（situation API）"""
url = f"https://api.dongqiudi.com{self._cfg.dongqiudi.api.match_situation}/{match_id}"
return await self.request(url)
async def get_match_lineup(self, match_id: int) -> Dict[str, Any]:
"""获取比赛阵容数据（lineup API）"""
url = f"https://api.dongqiudi.com{self._cfg.dongqiudi.api.match_lineup}/{match_id}"
return await self.request(url)
async def fetch_match_detail_from_page(self, match_id: int) -> Dict[str, Any]:
"""从懂球帝移动端页面抓取比赛详情(SSR数据)"""
page_url = f"http://m.dongqiudi.com/matchDetail/{match_id}/lotteryOddsNew?cmp_type=soccer"
logger.info(f"[page] GET {page_url}")
start = time.time()
ec = ErrorCollector.get()
timeout = aiohttp.ClientTimeout(total=self._cfg.crawler.request_timeout)
headers = {
"User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) "
"AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9",
}
try:
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(page_url, headers=headers, ssl=False) as resp:
elapsed = time.time() - start
if resp.status != 200:
logger.warning(f"[page] HTTP {resp.status} {elapsed:.2f}s {page_url}")
await ec.report(request_url=page_url, http_status=resp.status, error_type=f"HTTP {resp.status}", error_message=f"页面请求失败 HTTP {resp.status}", channel="page")
raise NetworkError(f"页面请求失败 HTTP {resp.status}: {page_url}")
html = await resp.text()
logger.info(f"[page] 200 OK {elapsed:.2f}s {page_url} html_len={len(html)}")
except NetworkError:
raise
except Exception as e:
await ec.report(request_url=page_url, http_status=0, error_type=type(e).__name__, error_message=str(e), channel="page")
raise NetworkError(f"页面请求异常: {page_url} {e}") from e
m = re.search(r'window\.__INITIAL_STATE__\s*=\s*(\{.*?\})\s*;', html, re.DOTALL)
if not m:
await ec.report(request_url=page_url, http_status=200, error_type="ParseError", error_message="页面中未找到 __INITIAL_STATE__", response_body=html[:2000], channel="page")
raise NetworkError(f"页面中未找到 __INITIAL_STATE__: {page_url}")
state = json.loads(m.group(1))
header_data = (state.get("matchContentStore", {})
.get("matchDetailHeaderData", {}))
if not header_data:
await ec.report(request_url=page_url, http_status=200, error_type="ParseError", error_message="matchDetailHeaderData 为空", channel="page")
raise NetworkError(f"matchDetailHeaderData 为空: {page_url}")
match_obj = header_data.get("match", {})
info_obj = header_data.get("info", {})
logger.info(
f"[page] 解析完成: match_id={match_id} "
f"status={match_obj.get('status','')} "
f"team_A.fs={match_obj.get('team_A',{}).get('fs','')} "
f"team_B.fs={match_obj.get('team_B',{}).get('fs','')} "
f"events_count={len(info_obj.get('events') or [])}"
)
return header_data
async def fetch_match_list_from_page(self, tab_id: int = 5) -> List[Dict[str, Any]]:
"""从懂球帝移动端赛程页面抓取 matchListStore.matchList（SSR数据）"""
page_url = f"http://m.dongqiudi.com/match/{tab_id}"
logger.info(f"[page] GET {page_url}")
start = time.time()
ec = ErrorCollector.get()
timeout = aiohttp.ClientTimeout(total=self._cfg.crawler.request_timeout)
headers = {
"User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) "
"AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.9",
}
try:
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(page_url, headers=headers, ssl=False) as resp:
elapsed = time.time() - start
if resp.status != 200:
logger.warning(f"[page] HTTP {resp.status} {elapsed:.2f}s {page_url}")
await ec.report(request_url=page_url, http_status=resp.status, error_type=f"HTTP {resp.status}", error_message=f"页面请求失败 HTTP {resp.status}", channel="page")
raise NetworkError(f"页面请求失败 HTTP {resp.status}: {page_url}")
html = await resp.text()
logger.info(f"[page] 200 OK {elapsed:.2f}s {page_url} html_len={len(html)}")
except NetworkError:
raise
except Exception as e:
await ec.report(request_url=page_url, http_status=0, error_type=type(e).__name__, error_message=str(e), channel="page")
raise NetworkError(f"页面请求异常: {page_url} {e}") from e
m = re.search(r'window\.__INITIAL_STATE__\s*=\s*(\{.*?\})\s*;', html, re.DOTALL)
if not m:
await ec.report(request_url=page_url, http_status=200, error_type="ParseError", error_message="页面中未找到 __INITIAL_STATE__", response_body=html[:2000], channel="page")
raise NetworkError(f"页面中未找到 __INITIAL_STATE__: {page_url}")
state = json.loads(m.group(1))
match_list = state.get("matchListStore", {}).get("matchList", [])
logger.info(f"[page] 解析完成: tab_id={tab_id} matchList={len(match_list)} 条")
return match_list
async def get_schedule_round(self, url: str) -> Dict[str, Any]:
"""请求具体轮次的赛程URL（含round_id/gameweek参数）"""
if 'language=' not in url:
url += '&language=zh-cn'
return await self.request(url)
async def get_league_tab_matches(self, api_url: str) -> Dict[str, Any]:
"""通过联赛 tab API 获取比赛列表"""
return await self.request(api_url)
async def get_live_text(self, match_id: int, num: int = 200) -> Dict[str, Any]:
"""获取比赛文字直播"""
url = f"{self._cfg.dongqiudi.web_url}/api/v4/imserver/baidu/Chatroom/rooms"
params = {"match_id": match_id, "num": num}
return await self.request(url, params)
async def get_live_text_page(self, page_url: str) -> Dict[str, Any]:
"""文字直播翻页请求"""
return await self.request(page_url)
async def get_match_menu(self) -> Dict[str, Any]:
url = self._cfg.dongqiudi.web_url + self._cfg.dongqiudi.api.match_menu
params = {"mark": "gif", "platform": "www", "version": "713"}
return await self.request(url, params)
async def get_live_matches(self) -> Dict[str, Any]:
"""从各联赛 tab API 汇总当前比赛数据"""
all_matches = []
league_ids = set()
for t in self._cfg.dongqiudi.targets:
if not t.active:
continue
lid = self._LEAGUE_ID_MAP.get(t.league)
if lid and lid not in league_ids:
league_ids.add(lid)
try:
url = f"https://api.dongqiudi.com/data/tab/league/new/{lid}"
raw = await self.request(url, None)
items = raw.get("list", []) if isinstance(raw, dict) else []
all_matches.extend(items)
except Exception as e:
logger.warning(f"获取联赛 {t.league} 实时数据失败: {e}")
return {"data": {"matches": all_matches}}
_LEAGUE_ID_MAP = {
"中超": 43, "中甲": 129, "亚冠": 226, "英超": 4, "西甲": 3,
"德甲": 5, "意甲": 9, "法甲": 12, "欧冠": 6, "欧联": 14,
}
async def get_news(self, tab_id: int = 1, size: int = 30) -> Dict[str, Any]:
url = f"{self._cfg.dongqiudi.web_url}/api/app/tabs/web/{tab_id}.json"
params = {"size": size} if size != 20 else None
return await self.request(url, params)
async def get_league_news(self, tab_id: int, page: int = 1, after: int = 0) -> Dict[str, Any]:
"""从懂球帝 iPhone tabs API 获取联赛资讯（支持分页）"""
url = f"https://api.dongqiudi.com/app/tabs/iphone/{tab_id}.json"
params = {"mark": "gif", "version": "576", "from": f"tab_{tab_id}"}
if after:
params["after"] = after
params["page"] = page
return await self.request(url, params)
async def get_video_list(self, tab_id: int = 233) -> Dict[str, Any]:
await self.initialize()
return await self._api_client.get_video_list(tab_id)
async def get_article_detail(self, article_id: int) -> Dict[str, Any]:
await self.initialize()
return await self._api_client.get_article_detail(article_id)
async def get_article_html(self, article_id: int) -> str:
"""请求 m.dongqiudi.com 文章页面，返回 HTML 字符串"""
import aiohttp
url = f"https://m.dongqiudi.com/article/{article_id}.html"
headers = {
"User-Agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1",
}
timeout = aiohttp.ClientTimeout(total=15)
async with aiohttp.ClientSession(timeout=timeout, headers=headers) as session:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
raise NetworkError(f"HTTP {resp.status}: {url}")
async def get_lottery_draw_data(self, url: str) -> dict:
"""请求彩种开奖数据API（返回JSON）"""
await self.initialize()
import aiohttp
timeout = aiohttp.ClientTimeout(total=15)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.json()
raise NetworkError(f"HTTP {resp.status}: {url}")
async def get_lottery_result(self, url: str) -> str:
"""请求六合彩开奖结果（返回原始文本，自签名证书跳过验证）"""
await self.initialize()
import aiohttp, ssl
ssl_ctx = ssl.create_default_context()
ssl_ctx.check_hostname = False
ssl_ctx.verify_mode = ssl.CERT_NONE
ts = int(time.time() * 1000)
full_url = f"{url}?_={ts}"
timeout = aiohttp.ClientTimeout(total=15)
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get(full_url, ssl=ssl_ctx) as resp:
if resp.status == 200:
raw = await resp.read()
# 尝试 utf-8 解码，失败则用 gbk
try:
return raw.decode('utf-8')
except UnicodeDecodeError:
return raw.decode('gbk', errors='replace')
raise NetworkError(f"HTTP {resp.status}: {full_url}")
@property
def stats(self) -> Dict[str, Any]:
return {
"channels": self._channel_stats,
"fingerprint": self._fp_pool.stats,
"proxy": self._proxy_mgr.stats,
"api_client": self._api_client.stats if self._api_client else {},
}
async def __aenter__(self):
await self.initialize()
return self
async def __aexit__(self, *args):
await self.close()
server/public/dongqiudi-crawler/src/parser/__init__.py
View File
server/public/dongqiudi-crawler/src/parser/data_parser.py
-560
View File
@@ -1,560 +0,0 @@
"""
懂球帝数据解析器
- 积分榜、赛程、比赛详情、新闻等数据解析
- 数据校验和清洗
"""
import re
from datetime import datetime
from typing import Dict, Any, List, Optional, Tuple
from loguru import logger
from src.core.exceptions import ParsingError
class DongqiudiParser:
"""懂球帝数据解析器"""
# ── 积分榜 ──
def parse_standings(self, response: Dict[str, Any], season_id: int) -> List[Dict[str, Any]]:
"""解析积分榜 API 响应"""
if not isinstance(response, dict):
raise ParsingError(f"积分榜响应类型错误: {type(response)}")
content = response.get("content", {})
rounds = content.get("rounds", [])
if not rounds:
logger.warning(f"积分榜无 rounds 数据 (season_id={season_id})")
return []
standings = []
for round_data in rounds:
rc = round_data.get("content", {})
round_info = rc.get("info", {})
round_number = round_info.get("round", 0)
teams = rc.get("data", [])
for team in teams:
parsed = self._parse_team_standing(team, season_id, round_number)
if parsed:
standings.append(parsed)
logger.info(f"积分榜解析完成: season_id={season_id}, {len(standings)} 条记录")
return standings
def _parse_team_standing(self, team: Dict, season_id: int, round_number: int) -> Optional[Dict[str, Any]]:
team_id = team.get("team_id")
if not team_id:
return None
deduction_pts, deduction_reason = self._parse_deduction(team.get("instruction", {}))
recent = self._parse_recent_form(team.get("recent_record", ""))
gf = int(team.get("goals_pro", 0))
ga = int(team.get("goals_against", 0))
return {
"season_id": season_id,
"round_number": round_number,
"team_id": team_id,
"team_name": team.get("team_name", ""),
"team_logo": team.get("team_logo", ""),
"rank": int(team.get("rank", 0)),
"points": int(team.get("points", 0)),
"played": int(team.get("matches_total", 0)),
"won": int(team.get("matches_won", 0)),
"drawn": int(team.get("matches_draw", 0)),
"lost": int(team.get("matches_lost", 0)),
"goals_for": gf,
"goals_against": ga,
"goal_diff": gf - ga,
"recent_form": recent,
"deduction_points": deduction_pts,
"deduction_reason": deduction_reason,
}
# ── 赛程 ──
def parse_schedule(self, response: Dict[str, Any], season_id: int) -> List[Dict[str, Any]]:
"""解析赛程 API 响应"""
if not isinstance(response, dict):
raise ParsingError(f"赛程响应类型错误: {type(response)}")
content = response.get("content", {})
matches_raw = content.get("matches", [])
if not matches_raw:
logger.warning(f"赛程无 matches 数据 (season_id={season_id})")
return []
matches = []
for m in matches_raw:
parsed = self._parse_match(m, season_id)
if parsed:
matches.append(parsed)
logger.info(f"赛程解析完成: season_id={season_id}, {len(matches)} 场比赛")
return matches
def _parse_match(self, m: Dict, season_id: int) -> Optional[Dict[str, Any]]:
match_id = m.get("match_id")
if not match_id:
return None
start_play = self._parse_datetime(m.get("start_play"))
status = self._normalize_status(m.get("status", ""))
return {
"match_id": match_id,
"season_id": season_id,
"round_number": m.get("round_id", 0),
"round_name": m.get("round_name", ""),
"match_date": start_play,
"status": status,
"home_team_id": m.get("team_A_id"),
"home_team_name": m.get("team_A_name", ""),
"home_team_logo": m.get("team_A_logo", ""),
"away_team_id": m.get("team_B_id"),
"away_team_name": m.get("team_B_name", ""),
"away_team_logo": m.get("team_B_logo", ""),
"home_score": m.get("score_A") if status != "fixture" else None,
"away_score": m.get("score_B") if status != "fixture" else None,
"halftime_score": m.get("half_score", ""),
"venue": m.get("stadium", ""),
"competition_id": m.get("competition_id"),
"competition_name": m.get("competition_name", ""),
}
# ── 比赛详情 ──
def parse_match_detail(self, response: Dict[str, Any]) -> Dict[str, Any]:
"""解析比赛详情"""
if not isinstance(response, dict):
raise ParsingError(f"比赛详情响应类型错误: {type(response)}")
data = response.get("data", response)
return {
"match_id": data.get("match_id"),
"status": self._normalize_status(data.get("status", "")),
"home_team": data.get("team_A", {}),
"away_team": data.get("team_B", {}),
"home_score": data.get("score_A"),
"away_score": data.get("score_B"),
"events": data.get("events", []),
"statistics": data.get("statistics", {}),
"lineups": data.get("lineups", {}),
}
# ── 比赛菜单 / 联赛列表 ──
def parse_match_menu(self, response: Dict[str, Any]) -> List[Dict[str, Any]]:
"""解析比赛类型菜单"""
if response.get("errCode") != 0:
return []
items = response.get("data", {}).get("list", [])
result = []
for item in items:
result.append({
"id": item.get("id"),
"label": item.get("label", ""),
"type": item.get("type", ""),
"sort": item.get("sort", 0),
"api": item.get("api", ""),
})
return result
# ── 新闻 ──
def parse_news(self, response: Dict[str, Any], tab_name: str = "") -> List[Dict[str, Any]]:
"""解析 /api/app/tabs/web/{id}.json 返回的文章列表"""
articles = response.get("articles", [])
if not articles:
return []
import time
now_ts = int(time.time())
result = []
for art in articles:
aid = art.get("id")
if not aid:
continue
title = art.get("title", "")
if not title:
continue
author_info = art.get("author")
if isinstance(author_info, dict):
author = author_info.get("name", "")
else:
author = str(art.get("author_name", "") or "")
result.append({
"id": aid,
"title": title,
"description": art.get("description", "") or art.get("b_description", ""),
"thumb": art.get("thumb", ""),
"author_name": author,
"published_at": art.get("published_at", ""),
"share": art.get("share", ""),
"comments_total": art.get("comments_total", 0),
"category": tab_name or "news",
"is_video": art.get("is_video", False),
"sort_timestamp": now_ts,
})
logger.info(f"从tabs API解析到 {len(result)} 篇文章 (tab={tab_name})")
return result
# ── 视频列表 ──
def parse_video_list(self, response: Dict[str, Any], tab_name: str = "") -> List[Dict[str, Any]]:
"""从视频列表页 HTML 的 __NUXT__ 中提取视频条目列表"""
import json as _json
html = response.get("html", "")
if not html:
return []
nuxt_m = re.search(r'window\.__NUXT__\s*=\s*(.+?);\s*</script>', html, re.DOTALL)
if not nuxt_m:
return []
nuxt_raw = nuxt_m.group(1)
var_map = self._parse_iife_var_map(nuxt_raw)
ids = re.findall(r'\bid:(\d{5,})', nuxt_raw)
shares = re.findall(r'\bshare:"([^"]*)"', nuxt_raw)
thumbs = re.findall(r'\bthumb:"([^"]*)"', nuxt_raw)
video_srcs = re.findall(r'\bvideo_src:"([^"]*)"', nuxt_raw)
video_times = re.findall(r'\bvideo_time:"([^"]*)"', nuxt_raw)
title_refs = re.findall(r'\btitle:([a-zA-Z]\w*)', nuxt_raw)
desc_refs = re.findall(r'\bdescription:([a-zA-Z]\w*)', nuxt_raw)
# published_at 可能是字符串值或变量引用，统一按顺序提取
pub_matches = re.findall(r'\bpublished_at:(?:"([^"]*)"|([a-zA-Z]\w*))', nuxt_raw)
count = min(len(ids), len(shares), len(thumbs))
if count == 0:
return []
def resolve(refs: list, idx: int) -> str:
if idx >= len(refs):
return ""
val = var_map.get(refs[idx])
return str(val) if val and val is not None else ""
def resolve_pub(idx: int) -> str:
if idx >= len(pub_matches):
return ""
str_val, ref_val = pub_matches[idx]
if str_val:
return str_val
if ref_val:
val = var_map.get(ref_val)
return str(val) if val and val is not None else ""
return ""
now_ts = int(datetime.now().timestamp())
videos = []
for i in range(count):
try:
article_id = int(ids[i])
share_url = _json.loads(f'"{shares[i]}"')
thumb_url = _json.loads(f'"{thumbs[i]}"')
video_src = _json.loads(f'"{video_srcs[i]}"') if i < len(video_srcs) else ""
video_time = video_times[i] if i < len(video_times) else ""
except Exception:
continue
videos.append({
"id": article_id,
"title": resolve(title_refs, i),
"description": resolve(desc_refs, i),
"thumb": thumb_url,
"share": share_url,
"is_video": True,
"video_src": video_src,
"video_time": video_time,
"category": tab_name,
"published_at": resolve_pub(i),
"sort_timestamp": now_ts,
})
return videos
# ── 文章详情 ──
def parse_article_detail(self, response: Dict[str, Any]) -> Optional[Dict[str, str]]:
"""从文章详情页 HTML 的 __NUXT__ 中提取正文及元信息"""
import json as _json
html = response.get("html", "")
if not html:
return None
nuxt_m = re.search(r'window\.__NUXT__\s*=\s*(.+?);\s*</script>', html, re.DOTALL)
if not nuxt_m:
return None
nuxt_raw = nuxt_m.group(1)
body_m = re.search(r'\bbody:"((?:[^"\\]|\\.)*)"', nuxt_raw)
if not body_m:
return None
try:
content = _json.loads(f'"{body_m.group(1)}"')
except Exception:
return None
if not content or len(content) < 10:
return None
result = {"content": content}
for field in ("title", "description", "published_at"):
m = re.search(rf'\b{field}:"((?:[^"\\]|\\.)*)"', nuxt_raw)
if m:
try:
result[field] = _json.loads(f'"{m.group(1)}"')
except Exception:
pass
author_m = re.search(r'\bauthor_name:"((?:[^"\\]|\\.)*)"', nuxt_raw)
if author_m:
try:
result["author"] = _json.loads(f'"{author_m.group(1)}"')
except Exception:
pass
return result
def parse_article_html(self, html: str) -> Optional[Dict[str, str]]:
"""从 m.dongqiudi.com 文章 HTML 中提取 <article> 标签内的富文本内容"""
if not html:
return None
article_m = re.search(r'<article[^>]*>(.*?)</article>', html, re.DOTALL)
if not article_m:
return None
article_html = article_m.group(1)
title = ""
h1_m = re.search(r'<h1[^>]*>(.*?)</h1>', article_html, re.DOTALL)
if h1_m:
title = re.sub(r'<[^>]+>', '', h1_m.group(1)).strip()
author = ""
writer_m = re.search(r'<span[^>]*class="writer"[^>]*>(.*?)</span>', article_html, re.DOTALL)
if writer_m:
author = re.sub(r'<[^>]+>', '', writer_m.group(1)).strip()
con_m = re.search(r'<div[^>]*class="con"[^>]*>(.*)', article_html, re.DOTALL)
if con_m:
content = con_m.group(1).strip()
content = re.sub(r'</div>\s*$', '', content, count=1).strip()
else:
content = ""
content = re.sub(r'data-src="([^"]*)"', r'src="\1"', content)
# 将 GIF 缩略图替换为原始 GIF（data-gif-src 属性中存储了可播放的原图）
content = re.sub(
r'<img([^>]*)\bsrc="[^"]*"([^>]*)\bdata-gif-src="([^"]*)"',
r'<img\1src="\3"\2data-gif-src="\3"',
content
)
if not content or len(content) < 20:
parts = []
if title:
parts.append(f'<h1>{title}</h1>')
time_text = ""
time_m = re.search(r'<time[^>]*>(.*?)</time>', article_html, re.DOTALL)
if time_m:
time_text = re.sub(r'<[^>]+>', '', time_m.group(1)).strip()
if author or time_text:
meta = []
if author:
meta.append(author)
if time_text:
meta.append(time_text)
parts.append(f'<p>{" · ".join(meta)}</p>')
desc_m = re.search(r'<meta[^>]*name="description"[^>]*content="([^"]*)"', html)
if desc_m:
desc_text = desc_m.group(1).strip()
desc_text = re.sub(r'&lt;[^&]*&gt;', '', desc_text)
if desc_text:
parts.append(f'<p>{desc_text}</p>')
if not parts:
return None
content = "\n".join(parts)
result = {"content": content}
if title:
result["title"] = title
if author:
result["author"] = author
video_urls = re.findall(r'<video[^>]*\bsrc="([^"]+)"', article_html)
if not video_urls:
video_urls = re.findall(r'<source[^>]*\bsrc="([^"]+)"', article_html)
if video_urls:
result["video_url"] = video_urls[0]
return result
# ── IIFE __NUXT__ 解析 ──
@staticmethod
def _parse_iife_var_map(nuxt_raw: str) -> Dict[str, Any]:
"""解析 (function(a,b,...){...}(val_a,val_b,...)) 压缩格式，返回变量名->值映射"""
import json as _json
param_m = re.match(r'^\(function\(([^)]*)\)', nuxt_raw)
if not param_m:
return {}
param_names = [p.strip() for p in param_m.group(1).split(',')]
last_pos = nuxt_raw.rfind('})(')
if last_pos >= 0:
args_raw = nuxt_raw[last_pos + 3:-1]
else:
last_pos = nuxt_raw.rfind('}(')
if last_pos >= 0:
args_raw = nuxt_raw[last_pos + 2:-2]
else:
return {}
args: list = []
i = 0
n = len(args_raw)
while i < n:
c = args_raw[i]
if c in ' \n\r\t,':
i += 1
continue
if c == '"':
j = i + 1
while j < n:
if args_raw[j] == '\\':
j += 2
elif args_raw[j] == '"':
break
else:
j += 1
try:
args.append(_json.loads(args_raw[i:j + 1]))
except Exception:
args.append(args_raw[i + 1:j])
i = j + 1
elif c in ('{', '['):
depth, j, close_c = 0, i, '}' if c == '{' else ']'
in_str = False
while j < n:
ch = args_raw[j]
if in_str:
if ch == '\\':
j += 1
elif ch == '"':
in_str = False
elif ch == '"':
in_str = True
elif ch == c:
depth += 1
elif ch == close_c:
depth -= 1
if depth == 0:
break
j += 1
args.append(args_raw[i:j + 1])
i = j + 1
else:
j = i
while j < n and args_raw[j] not in ',)\n':
j += 1
token = args_raw[i:j].strip()
if token == 'true':
args.append(True)
elif token == 'false':
args.append(False)
elif token in ('null', 'void 0'):
args.append(None)
else:
try:
args.append(int(token))
except ValueError:
try:
args.append(float(token))
except ValueError:
args.append(token)
i = j
return {name: args[idx] for idx, name in enumerate(param_names) if idx < len(args)}
# ── 工具方法 ──
def _parse_deduction(self, instruction: Dict) -> Tuple[int, str]:
if not instruction:
return 0, ""
desc = instruction.get("description", "")
if not desc:
return 0, ""
for pattern in [r"扣(\d+)分", r"扣除(\d+)分", r"罚(\d+)分"]:
match = re.search(pattern, desc)
if match:
return int(match.group(1)), desc
return 0, desc
def _parse_recent_form(self, record: str) -> str:
if not record:
return ""
mapping = {"胜": "W", "赢": "W", "平": "D", "负": "L", "输": "L"}
return "".join(mapping.get(c, "?") for c in record)[-5:]
def _normalize_status(self, status: str) -> str:
if not status:
return "fixture"
s = status.lower()
if "finished" in s or "完场" in s:
return "finished"
elif "live" in s or "进行" in s:
return "live"
elif "postponed" in s or "延期" in s:
return "postponed"
elif "cancelled" in s or "取消" in s:
return "cancelled"
return "fixture"
def _parse_datetime(self, dt_str: Optional[str]) -> Optional[datetime]:
if not dt_str:
return None
for fmt in [
"%Y-%m-%d %H:%M:%S", "%Y/%m/%d %H:%M:%S",
"%Y-%m-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S.%f",
"%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M", "%Y-%m-%d",
]:
try:
return datetime.strptime(dt_str, fmt)
except ValueError:
continue
return None
def validate_standing(self, data: Dict) -> List[str]:
errors = []
for field in ["season_id", "team_id", "rank", "points"]:
if field not in data:
errors.append(f"缺少字段: {field}")
if all(k in data for k in ("played", "won", "drawn", "lost")):
if data["played"] != data["won"] + data["drawn"] + data["lost"]:
errors.append(f"场次不匹配: {data['played']} != {data['won']}+{data['drawn']}+{data['lost']}")
return errors
def validate_match(self, data: Dict) -> List[str]:
errors = []
for field in ["match_id", "season_id", "home_team_id", "away_team_id"]:
if field not in data:
errors.append(f"缺少字段: {field}")
if data.get("status") == "finished":
if data.get("home_score") is None or data.get("away_score") is None:
errors.append("已完场比赛缺少比分")
return errors
server/public/dongqiudi-crawler/src/parser/nuxt_parser.py
-179
View File
@@ -1,179 +0,0 @@
"""
从懂球帝 liveDetail 页面的 window.__NUXT__ IIFE 中解析比赛详情数据
"""
import re
from typing import Optional, Dict, Any
def parse_js_args(args_str: str) -> list:
"""解析 JS IIFE 的实参列表，返回 Python 值列表"""
results = []
i = 0
n = len(args_str)
while i < n:
c = args_str[i]
if c in (' ', '\n', '\r', '\t'):
i += 1
continue
if c == ',':
i += 1
continue
if c == '"':
j = i + 1
while j < n:
if args_str[j] == '\\':
j += 2
continue
if args_str[j] == '"':
break
j += 1
raw = args_str[i:j+1]
inner = raw[1:-1]
if '\\u' in inner:
val = inner.encode('utf-8').decode('unicode_escape')
else:
val = inner.replace('\\n', '\n').replace('\\t', '\t').replace('\\"', '"').replace('\\\\', '\\')
results.append(val)
i = j + 1
continue
if args_str[i:i+4] == 'true':
results.append(True)
i += 4
continue
if args_str[i:i+5] == 'false':
results.append(False)
i += 5
continue
if args_str[i:i+4] == 'null':
results.append(None)
i += 4
continue
if args_str[i:i+6] == 'void 0':
results.append(None)
i += 6
continue
if c in '0123456789.-':
j = i + 1
while j < n and args_str[j] in '0123456789.eE+-':
j += 1
num_str = args_str[i:j]
try:
val = int(num_str)
except ValueError:
val = float(num_str)
results.append(val)
i = j
continue
if c == '{':
depth = 1
j = i + 1
while j < n and depth > 0:
if args_str[j] == '{':
depth += 1
elif args_str[j] == '}':
depth -= 1
elif args_str[j] == '"':
j += 1
while j < n and args_str[j] != '"':
if args_str[j] == '\\':
j += 1
j += 1
j += 1
results.append(args_str[i:j])
i = j
continue
if c == '[':
depth = 1
j = i + 1
while j < n and depth > 0:
if args_str[j] == '[':
depth += 1
elif args_str[j] == ']':
depth -= 1
elif args_str[j] == '"':
j += 1
while j < n and args_str[j] != '"':
if args_str[j] == '\\':
j += 1
j += 1
j += 1
results.append(args_str[i:j])
i = j
continue
j = i
while j < n and args_str[j] not in ',)]}':
j += 1
token = args_str[i:j].strip()
results.append(token)
i = j
return results
def _decode_str(val_expr: str) -> str:
"""解码 JS 字符串字面量（去掉引号）"""
inner = val_expr[1:-1]
if '\\u' in inner:
try:
return inner.encode('utf-8').decode('unicode_escape')
except Exception:
return inner
return inner.replace('\\n', '\n').replace('\\t', '\t').replace('\\"', '"').replace('\\\\', '\\')
def _resolve_value(val_expr: str, var_map: dict):
"""将 JS 赋值右侧表达式解析为 Python 值"""
if val_expr in var_map:
return var_map[val_expr]
if val_expr.startswith('"') and val_expr.endswith('"'):
return _decode_str(val_expr)
if val_expr == 'true':
return True
if val_expr == 'false':
return False
if val_expr == 'null':
return None
try:
return int(val_expr)
except ValueError:
pass
try:
return float(val_expr)
except ValueError:
pass
return val_expr
def parse_nuxt_match_sample(html: str) -> Optional[Dict[str, Any]]:
"""从 liveDetail HTML 中解析 matchSample 对象"""
pattern = r'window\.__NUXT__\s*=\s*\(function\(([^)]+)\)\s*\{(.+)\}\((.+)\)\);'
m = re.search(pattern, html, re.DOTALL)
if not m:
return None
params = [p.strip() for p in m.group(1).split(',')]
body = m.group(2)
args_str = m.group(3)
args = parse_js_args(args_str)
var_map = {}
for i, p in enumerate(params):
if i < len(args):
var_map[p] = args[i]
ms_var_match = re.search(r'(\w+)\.match_id\s*=\s*(\w+)', body)
if not ms_var_match:
return None
ms_var = ms_var_match.group(1)
assign_pattern = re.compile(rf'\b{re.escape(ms_var)}\.(\w+)\s*=\s*(.+?)\s*;', re.MULTILINE)
match_sample = {}
for am in assign_pattern.finditer(body):
field = am.group(1)
val_expr = am.group(2).strip()
match_sample[field] = _resolve_value(val_expr, var_map)
return match_sample if match_sample else None
server/public/dongqiudi-crawler/src/scheduler/__init__.py
View File
server/public/dongqiudi-crawler/src/scheduler/cron_scheduler.py
-376
View File
@@ -1,376 +0,0 @@
"""
定时调度器
- 基于 APScheduler 的定时任务管理
- 支持 Cron 表达式
"""
import asyncio
from typing import Optional
from loguru import logger
from src.core.config import get_config
from src.scheduler.task_runner import TaskRunner
class CronScheduler:
"""定时调度器"""
def __init__(self):
self._cfg = get_config().scheduler
self._runner: Optional[TaskRunner] = None
self._scheduler = None
async def start(self):
from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger
self._runner = TaskRunner()
await self._runner.initialize()
self._scheduler = AsyncIOScheduler()
# 积分榜采集
self._scheduler.add_job(
self._job_standings,
CronTrigger.from_crontab(self._cfg.standings_cron),
id="standings",
name="积分榜采集",
)
# 赛程采集
self._scheduler.add_job(
self._job_schedule,
CronTrigger.from_crontab(self._cfg.schedule_cron),
id="schedule",
name="赛程采集",
)
# 新闻资讯采集
self._scheduler.add_job(
self._job_news,
CronTrigger.from_crontab(self._cfg.news_cron),
id="news",
name="新闻资讯采集",
)
# 实时比赛采集
self._scheduler.add_job(
self._job_live,
CronTrigger.from_crontab(self._cfg.live_cron),
id="live",
name="实时比赛采集",
)
# 文章详情补全
self._scheduler.add_job(
self._job_content,
CronTrigger.from_crontab(self._cfg.content_cron),
id="content",
name="文章详情补全",
)
# 视频列表采集
self._scheduler.add_job(
self._job_video,
CronTrigger.from_crontab(self._cfg.video_cron),
id="video",
name="视频列表采集",
)
# 中超赛程页面采集
self._scheduler.add_job(
self._job_csl_match,
CronTrigger.from_crontab(self._cfg.csl_match_cron),
id="csl_match",
name="中超赛程页面采集",
)
# NBA赛程页面采集
self._scheduler.add_job(
self._job_nba_match,
CronTrigger.from_crontab(self._cfg.nba_match_cron),
id="nba_match",
name="NBA赛程页面采集",
)
# CBA赛程页面采集
self._scheduler.add_job(
self._job_cba_match,
CronTrigger.from_crontab(self._cfg.cba_match_cron),
id="cba_match",
name="CBA赛程页面采集",
)
# 英超赛程页面采集
self._scheduler.add_job(
self._job_epl_match,
CronTrigger.from_crontab(self._cfg.epl_match_cron),
id="epl_match",
name="英超赛程页面采集",
)
# 德甲赛程页面采集
self._scheduler.add_job(
self._job_bundesliga_match,
CronTrigger.from_crontab(self._cfg.bundesliga_match_cron),
id="bundesliga_match",
name="德甲赛程页面采集",
)
# 西甲赛程页面采集
self._scheduler.add_job(
self._job_laliga_match,
CronTrigger.from_crontab(self._cfg.laliga_match_cron),
id="laliga_match",
name="西甲赛程页面采集",
)
# 意甲赛程页面采集
self._scheduler.add_job(
self._job_seriea_match,
CronTrigger.from_crontab(self._cfg.seriea_match_cron),
id="seriea_match",
name="意甲赛程页面采集",
)
# 法甲赛程页面采集
self._scheduler.add_job(
self._job_ligue1_match,
CronTrigger.from_crontab(self._cfg.ligue1_match_cron),
id="ligue1_match",
name="法甲赛程页面采集",
)
# 欧冠赛程页面采集
self._scheduler.add_job(
self._job_ucl_match,
CronTrigger.from_crontab(self._cfg.ucl_match_cron),
id="ucl_match",
name="欧冠赛程页面采集",
)
# 欧联赛程页面采集
self._scheduler.add_job(
self._job_uel_match,
CronTrigger.from_crontab(self._cfg.uel_match_cron),
id="uel_match",
name="欧联赛程页面采集",
)
# 网球赛程页面采集
self._scheduler.add_job(
self._job_tennis_match,
CronTrigger.from_crontab(self._cfg.tennis_match_cron),
id="tennis_match",
name="网球赛程页面采集",
)
# 电竞赛程页面采集
self._scheduler.add_job(
self._job_esports_match,
CronTrigger.from_crontab(self._cfg.esports_match_cron),
id="esports_match",
name="电竞赛程页面采集",
)
# 体坛赛程页面采集
self._scheduler.add_job(
self._job_sports_match,
CronTrigger.from_crontab(self._cfg.sports_match_cron),
id="sports_match",
name="体坛赛程页面采集",
)
# 联赛资讯采集
self._scheduler.add_job(
self._job_league_news,
CronTrigger.from_crontab(self._cfg.league_news_cron),
id="league_news",
name="联赛资讯采集",
)
# 文章内容补全
self._scheduler.add_job(
self._job_article_content,
CronTrigger.from_crontab(self._cfg.article_content_cron),
id="article_content",
name="文章内容补全",
)
self._scheduler.start()
logger.info("定时调度器已启动")
logger.info(f" 积分榜: {self._cfg.standings_cron}")
logger.info(f" 赛程: {self._cfg.schedule_cron}")
logger.info(f" 新闻: {self._cfg.news_cron}")
logger.info(f" 联赛资讯: {self._cfg.league_news_cron}")
logger.info(f" 文章补全: {self._cfg.article_content_cron}")
logger.info(f" 实时: {self._cfg.live_cron}")
logger.info(f" 详情: {self._cfg.content_cron}")
logger.info(f" 视频: {self._cfg.video_cron}")
logger.info(f" 中超: {self._cfg.csl_match_cron}")
logger.info(f" NBA: {self._cfg.nba_match_cron}")
logger.info(f" CBA: {self._cfg.cba_match_cron}")
logger.info(f" 英超: {self._cfg.epl_match_cron}")
logger.info(f" 德甲: {self._cfg.bundesliga_match_cron}")
logger.info(f" 西甲: {self._cfg.laliga_match_cron}")
logger.info(f" 意甲: {self._cfg.seriea_match_cron}")
logger.info(f" 法甲: {self._cfg.ligue1_match_cron}")
logger.info(f" 欧冠: {self._cfg.ucl_match_cron}")
logger.info(f" 欧联: {self._cfg.uel_match_cron}")
logger.info(f" 网球: {self._cfg.tennis_match_cron}")
logger.info(f" 电竞: {self._cfg.esports_match_cron}")
logger.info(f" 体坛: {self._cfg.sports_match_cron}")
async def stop(self):
if self._scheduler:
self._scheduler.shutdown(wait=False)
if self._runner:
await self._runner.close()
logger.info("定时调度器已停止")
async def _job_standings(self):
logger.info("[CRON] 触发积分榜采集")
try:
await self._runner.crawl_all_leagues("standings")
except Exception as e:
logger.error(f"[CRON] 积分榜采集失败: {e}")
async def _job_schedule(self):
logger.info("[CRON] 触发赛程采集")
try:
await self._runner.crawl_all_leagues("schedule")
except Exception as e:
logger.error(f"[CRON] 赛程采集失败: {e}")
async def _job_news(self):
logger.info("[CRON] 触发新闻资讯采集")
try:
await self._runner.crawl_news()
except Exception as e:
logger.error(f"[CRON] 新闻采集失败: {e}")
async def _job_live(self):
logger.info("[CRON] 触发实时比赛采集")
try:
await self._runner.crawl_live()
except Exception as e:
logger.error(f"[CRON] 实时比赛采集失败: {e}")
async def _job_content(self):
logger.info("[CRON] 触发文章详情补全")
try:
await self._runner.crawl_article_details(batch_size=3)
except Exception as e:
logger.error(f"[CRON] 文章详情补全失败: {e}")
async def _job_video(self):
logger.info("[CRON] 触发视频列表采集")
try:
await self._runner.crawl_videos()
except Exception as e:
logger.error(f"[CRON] 视频列表采集失败: {e}")
async def _job_csl_match(self):
logger.info("[CRON] 触发中超赛程页面采集")
try:
await self._runner.crawl_csl_matches()
except Exception as e:
logger.error(f"[CRON] 中超赛程采集失败: {e}")
async def _job_nba_match(self):
logger.info("[CRON] 触发NBA赛程页面采集")
try:
await self._runner.crawl_nba_matches()
except Exception as e:
logger.error(f"[CRON] NBA赛程采集失败: {e}")
async def _job_cba_match(self):
logger.info("[CRON] 触发CBA赛程页面采集")
try:
await self._runner.crawl_cba_matches()
except Exception as e:
logger.error(f"[CRON] CBA赛程采集失败: {e}")
async def _job_epl_match(self):
logger.info("[CRON] 触发英超赛程页面采集")
try:
await self._runner.crawl_epl_matches()
except Exception as e:
logger.error(f"[CRON] 英超赛程采集失败: {e}")
async def _job_bundesliga_match(self):
logger.info("[CRON] 触发德甲赛程页面采集")
try:
await self._runner.crawl_bundesliga_matches()
except Exception as e:
logger.error(f"[CRON] 德甲赛程采集失败: {e}")
async def _job_laliga_match(self):
logger.info("[CRON] 触发西甲赛程页面采集")
try:
await self._runner.crawl_laliga_matches()
except Exception as e:
logger.error(f"[CRON] 西甲赛程采集失败: {e}")
async def _job_seriea_match(self):
logger.info("[CRON] 触发意甲赛程页面采集")
try:
await self._runner.crawl_seriea_matches()
except Exception as e:
logger.error(f"[CRON] 意甲赛程采集失败: {e}")
async def _job_ligue1_match(self):
logger.info("[CRON] 触发法甲赛程页面采集")
try:
await self._runner.crawl_ligue1_matches()
except Exception as e:
logger.error(f"[CRON] 法甲赛程采集失败: {e}")
async def _job_ucl_match(self):
logger.info("[CRON] 触发欧冠赛程页面采集")
try:
await self._runner.crawl_ucl_matches()
except Exception as e:
logger.error(f"[CRON] 欧冠赛程采集失败: {e}")
async def _job_uel_match(self):
logger.info("[CRON] 触发欧联赛程页面采集")
try:
await self._runner.crawl_uel_matches()
except Exception as e:
logger.error(f"[CRON] 欧联赛程采集失败: {e}")
async def _job_tennis_match(self):
logger.info("[CRON] 触发网球赛程页面采集")
try:
await self._runner.crawl_tennis_matches()
except Exception as e:
logger.error(f"[CRON] 网球赛程采集失败: {e}")
async def _job_esports_match(self):
logger.info("[CRON] 触发电竞赛程页面采集")
try:
await self._runner.crawl_esports_matches()
except Exception as e:
logger.error(f"[CRON] 电竞赛程采集失败: {e}")
async def _job_sports_match(self):
logger.info("[CRON] 触发体坛赛程页面采集")
try:
await self._runner.crawl_sports_matches()
except Exception as e:
logger.error(f"[CRON] 体坛赛程采集失败: {e}")
async def _job_league_news(self):
logger.info("[CRON] 触发联赛资讯采集")
try:
await self._runner.crawl_league_news()
except Exception as e:
logger.error(f"[CRON] 联赛资讯采集失败: {e}")
async def _job_article_content(self):
logger.info("[CRON] 触发文章内容补全")
try:
await self._runner.crawl_article_content()
except Exception as e:
logger.error(f"[CRON] 文章内容补全失败: {e}")
server/public/dongqiudi-crawler/src/scheduler/task_runner.py
-1537
View File
File diff suppressed because it is too large Load Diff
server/public/dongqiudi-crawler/src/storage/__init__.py
View File
server/public/dongqiudi-crawler/src/storage/database.py
-1436
View File
File diff suppressed because it is too large Load Diff
server/public/dongqiudi-crawler/taiwan_lottery_news.py
-267
View File
@@ -1,267 +0,0 @@
"""
台湾彩券资讯采集模块
- 列表接口: https://api.taiwanlottery.com/TLCAPIWeB/News/List
- 详情页面: https://www.taiwanlottery.com/news/news/{newsId}
写入分类:
- la_article_cate.id = 21 (彩票资讯)
"""
import html
import json
import re
import time
from datetime import datetime
from urllib.parse import urljoin
import pymysql
import requests
from article_publish_helper import has_publishable_article_content, resolve_crawled_article_is_show
ARTICLE_CID = 21
LIST_API = "https://api.taiwanlottery.com/TLCAPIWeB/News/List?keyword&from=2025-06&to=2026-06&type=2&webtag=1&pageSize=20&pageNo={page}"
DETAIL_URL = "https://www.taiwanlottery.com/news/news/{news_id}"
DETAIL_API = "https://api.taiwanlottery.com/TLCAPIWeB/News/Detail/{news_id}"
SOURCE_NAME = "台湾彩券"
MAX_PAGES = 3
def clean_text(text: str) -> str:
return re.sub(r"\s+", " ", html.unescape(text or "").replace("\u3000", " ")).strip()
def parse_announce_date(value: str) -> tuple[int, str]:
if not value:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
try:
dt = datetime.fromisoformat(value.replace("Z", "+00:00"))
ts = int(dt.timestamp())
return ts, dt.strftime("%Y-%m-%d %H:%M:%S")
except Exception:
now = int(time.time())
return now, datetime.fromtimestamp(now).strftime("%Y-%m-%d %H:%M:%S")
def fetch_list(session: requests.Session, page_no: int) -> list[dict]:
resp = session.get(LIST_API.format(page=page_no), timeout=30)
resp.raise_for_status()
data = resp.json()
if data.get("rtCode") != 0:
raise RuntimeError(f"台湾彩券列表接口异常: {data.get('rtMsg')}")
return data.get("content", {}).get("newsListRes", []) or []
def strip_wrapper_tags(html_text: str) -> str:
content = html_text or ""
content = re.sub(r"^\ufeff", "", content)
content = re.sub(r"</?(?:html|body)[^>]*>", "", content, flags=re.I)
content = re.sub(r"<head[\s\S]*?</head>", "", content, flags=re.I)
content = re.sub(r"<style[\s\S]*?</style>", "", content, flags=re.I)
content = re.sub(r"<meta[^>]*>", "", content, flags=re.I)
return content.strip()
def parse_detail(session: requests.Session, news_id: str) -> dict:
api_url = DETAIL_API.format(news_id=news_id)
resp = session.get(api_url, timeout=30)
resp.raise_for_status()
data = resp.json()
if data.get("rtCode") != 0:
raise RuntimeError(f"台湾彩券详情接口异常: {data.get('rtMsg')}")
detail = data.get("content") or {}
url = DETAIL_URL.format(news_id=news_id)
title = clean_text(detail.get("newsTitle", ""))
published_ts, published_at = parse_announce_date(detail.get("announceDate", ""))
raw_html = strip_wrapper_tags(detail.get("content", ""))
inner_doc = raw_html
image_urls = []
for src in re.findall(r'<img[^>]+src="([^"]+)"', inner_doc, re.I):
full = urljoin(url, src)
if full not in image_urls:
image_urls.append(full)
paragraphs = []
for raw in re.findall(r"<p[^>]*>([\s\S]*?)</p>", inner_doc, re.I):
text = clean_text(re.sub(r"<[^>]+>", " ", raw))
if len(text) > 8:
paragraphs.append(text)
published_text = ""
if paragraphs and re.match(r"^\d{2}/\d{2}/\d{4}", paragraphs[0]):
published_text = paragraphs.pop(0)
content_parts = [f"<p>{text}</p>" for text in paragraphs]
for image in image_urls[:3]:
content_parts.append(f'<img src="{image}" />')
desc = paragraphs[0][:255] if paragraphs else title[:255]
return {
"title": title[:255],
"desc": desc,
"content": "".join(content_parts),
"image": image_urls[0] if image_urls else "",
"published_text": published_text[:64],
"published_ts": published_ts,
"published_at": published_at,
"source_url": url,
}
def dedupe_by_source(items: list[dict]) -> list[dict]:
seen = set()
result = []
for item in items:
key = item.get("source_url", "")
if key and key not in seen:
seen.add(key)
result.append(item)
return result
def sync_to_db(conn, articles: list[dict]) -> tuple[int, int, int]:
if not articles:
return 0, 0, 0
urls = [a["source_url"] for a in articles if a.get("source_url")]
with conn.cursor() as cur:
placeholders = ",".join(["%s"] * len(urls))
cur.execute(
f"SELECT id, source_url, content, is_show FROM la_article WHERE source_url IN ({placeholders}) AND cid = %s",
urls + [ARTICLE_CID]
)
existing = {row["source_url"]: row for row in cur.fetchall()}
titles = [a["title"][:255] for a in articles if a.get("title")]
existing_titles = set()
if titles:
with conn.cursor() as cur:
t_placeholders = ",".join(["%s"] * len(titles))
cur.execute(
f"SELECT title FROM la_article WHERE title IN ({t_placeholders}) AND cid = %s",
titles + [ARTICLE_CID]
)
existing_titles = {row["title"] for row in cur.fetchall()}
new_count = 0
update_count = 0
now = int(time.time())
for article in articles:
source_url = article.get("source_url", "")
title = article.get("title", "")
if not source_url or not title:
continue
ext_json = json.dumps({
"source": "taiwan_lottery",
"news_id": article.get("news_id", ""),
}, ensure_ascii=False)
if source_url in existing:
existing_row = existing[source_url]
next_is_show = existing_row["is_show"] if has_publishable_article_content(existing_row.get("content", "")) else 0
with conn.cursor() as cur:
cur.execute(
"""UPDATE la_article SET
title=%s, `desc`=%s, abstract=%s, image=%s, author=%s,
published_at=%s, category=%s, content=%s, ext=%s, is_show=%s, update_time=%s
WHERE id=%s""",
(
title[:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", "")[:128],
SOURCE_NAME,
article.get("published_at", "")[:64],
"lottery,taiwan",
article.get("content") or "",
ext_json,
next_is_show,
now,
existing_row["id"],
)
)
update_count += 1
else:
if title[:255] in existing_titles:
continue
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_article
(cid, title, `desc`, abstract, image, author, published_at,
category, source_url, content, ext, is_show, sort,
click_virtual, click_actual, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
ARTICLE_CID,
title[:255],
(article.get("desc") or "")[:255],
(article.get("desc") or "")[:500],
article.get("image", "")[:128],
SOURCE_NAME,
article.get("published_at", "")[:64],
"lottery,taiwan",
source_url[:500],
article.get("content") or "",
ext_json,
resolve_crawled_article_is_show(article.get("content") or ""),
0,
0,
0,
article.get("published_ts") or now,
now,
)
)
existing[source_url] = {"id": cur.lastrowid, "content": article.get("content") or "", "is_show": resolve_crawled_article_is_show(article.get("content") or "")}
new_count += 1
existing_titles.add(title[:255])
return len(articles), new_count, update_count
def run(db_config: dict):
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; SportEra/1.0)",
"Accept": "application/json,text/html,*/*",
})
all_articles = []
for page_no in range(1, MAX_PAGES + 1):
print(f"=== 台湾彩券资讯 第{page_no}页 ===")
list_items = fetch_list(session, page_no)
print(f" 列表返回 {len(list_items)} 条")
if not list_items:
break
for item in list_items:
news_id = item.get("newsId", "")
if not news_id:
continue
detail = parse_detail(session, news_id)
all_articles.append({
"news_id": news_id,
"title": detail.get("title") or clean_text(item.get("newsTitle", "")),
"desc": detail.get("desc", ""),
"content": detail.get("content", ""),
"image": detail.get("image", ""),
"published_ts": detail.get("published_ts"),
"published_at": detail.get("published_at"),
"source_url": detail.get("source_url"),
})
all_articles = dedupe_by_source(all_articles)
print(f"去重后共 {len(all_articles)} 条")
if not all_articles:
print("无可写入数据，跳过")
return {"success": True, "candidate_count": 0, "saved_count": 0, "summary_text": "无可写入数据，跳过"}
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total, new_count, update_count = sync_to_db(conn, all_articles)
conn.commit()
summary = f"共 {total} 条, 新增 {new_count}, 更新 {update_count}"
print(summary)
return {"success": True, "candidate_count": total, "saved_count": new_count + update_count, "summary_text": summary}
finally:
conn.close()
server/public/dongqiudi-crawler/test_output.txt
BIN
View File
Binary file not shown.
server/public/dongqiudi-crawler/tests/__init__.py
-1
View File
@@ -1 +0,0 @@
server/public/dongqiudi-crawler/tests/test_ai_comment_dispatch.py
-214
View File
@@ -1,214 +0,0 @@
import unittest
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from ai_comment_dispatch import (
AiCommentDispatcher,
AiCommentSettings,
CommentCandidate,
DispatchTask,
VirtualUser,
extract_post_text,
merge_candidates,
select_balanced_candidates,
)
class FakeRepo:
def __init__(self, article_candidates=None, post_candidates=None, daily_success=0, users=None):
self.article_candidates = article_candidates or []
self.post_candidates = post_candidates or []
self.daily_success = daily_success
self.users = users or []
self.created_tasks = []
self.saved_comments = []
self.failed_tasks = []
self.ensure_schema_called = False
self.ensure_users_target = None
def ensure_schema(self):
self.ensure_schema_called = True
def ensure_virtual_users(self, target_count):
self.ensure_users_target = target_count
return list(self.users)
def count_daily_success(self, day_start, day_end):
return self.daily_success
def load_article_candidates(self, settings):
return list(self.article_candidates)
def load_post_candidates(self, settings):
return list(self.post_candidates)
def create_task(self, candidate, virtual_user, persona_key):
task = DispatchTask(
task_id=len(self.created_tasks) + 1,
target_type=candidate.target_type,
target_id=candidate.target_id,
virtual_user_id=virtual_user.user_id,
persona_key=persona_key,
selection_sources=tuple(candidate.selection_sources),
)
self.created_tasks.append(task)
return task
def mark_task_success(self, task, comment_content):
self.saved_comments.append((task, comment_content))
def mark_task_failed(self, task, message):
self.failed_tasks.append((task, message))
class FakeClient:
def __init__(self, outputs=None):
self.outputs = outputs or {}
self.calls = []
def generate_comment(self, candidate, virtual_user, persona_key, settings):
self.calls.append((candidate, virtual_user, persona_key))
return self.outputs.get(
(candidate.target_type, candidate.target_id),
{"success": True, "content": f"{candidate.target_type}-{candidate.target_id}-comment"},
)
class AiCommentDispatchTest(unittest.TestCase):
def test_merge_candidates_merges_selection_sources_by_target(self):
merged = merge_candidates(
[
CommentCandidate("article", 10, "title-a", "body-a", ("cate:1",), created_at=100, rank_value=8),
CommentCandidate("article", 10, "title-a", "body-a", ("top",), created_at=100, rank_value=8),
CommentCandidate("article", 11, "title-b", "body-b", ("hot",), created_at=90, rank_value=5),
]
)
self.assertEqual(len(merged), 2)
self.assertEqual(merged[0].target_id, 10)
self.assertEqual(set(merged[0].selection_sources), {"cate:1", "top"})
def test_select_balanced_candidates_prefers_25_25_then_backfills(self):
articles = [
CommentCandidate("article", index, f"title-{index}", "body", (f"cate:{index}",), created_at=200 - index, rank_value=index)
for index in range(1, 9)
]
posts = [
CommentCandidate("post", index, "", f"post-{index}", (f"tag:{index}",), created_at=100 - index, rank_value=index)
for index in range(1, 3)
]
selected = select_balanced_candidates(articles, posts, total_limit=6)
self.assertEqual(len(selected), 6)
self.assertEqual(sum(1 for item in selected if item.target_type == "post"), 2)
self.assertEqual(sum(1 for item in selected if item.target_type == "article"), 4)
self.assertEqual([item.target_id for item in selected[:3]], [1, 2, 3])
def test_extract_post_text_prefers_image_only_analysis_content(self):
content = extract_post_text(
"",
{
"lottery_analysis_source": "image_only",
"lottery_analysis_content": "杀码思路和冷热分析",
},
)
self.assertEqual(content, "杀码思路和冷热分析")
def test_dispatcher_stops_when_daily_cap_reached(self):
repo = FakeRepo(daily_success=500, users=[VirtualUser(1, "ai_comment_0001", "AI评论员01", "data_analyst")])
client = FakeClient()
dispatcher = AiCommentDispatcher(repo, client)
settings = AiCommentSettings(per_run=10, daily_cap=500, user_pool_size=100)
summary = dispatcher.run(settings)
self.assertTrue(repo.ensure_schema_called)
self.assertEqual(repo.ensure_users_target, 100)
self.assertEqual(summary["status"], "skipped")
self.assertEqual(summary["reason"], "daily_cap_reached")
self.assertEqual(len(repo.created_tasks), 0)
self.assertEqual(len(client.calls), 0)
def test_dispatcher_creates_tasks_and_marks_success_or_failure(self):
article_candidates = [
CommentCandidate("article", 101, "article-101", "body", ("cate:10",), created_at=10, rank_value=3),
CommentCandidate("article", 102, "article-102", "body", ("hot",), created_at=9, rank_value=2),
]
post_candidates = [
CommentCandidate("post", 201, "", "post-201", ("tag:14",), created_at=8, rank_value=5),
CommentCandidate("post", 202, "", "post-202", ("tag:15",), created_at=7, rank_value=4),
]
users = [
VirtualUser(1, "ai_comment_0001", "AI评论员01", "data_analyst"),
VirtualUser(2, "ai_comment_0002", "AI评论员02", "old_fan"),
VirtualUser(3, "ai_comment_0003", "AI评论员03", "coach_view"),
VirtualUser(4, "ai_comment_0004", "AI评论员04", "night_shift_editor"),
]
repo = FakeRepo(article_candidates=article_candidates, post_candidates=post_candidates, users=users)
client = FakeClient(
outputs={
("article", 101): {"success": True, "content": "文章评论A"},
("article", 102): {"success": False, "error": "上游超时"},
("post", 201): {"success": True, "content": "帖子评论B"},
("post", 202): {"success": True, "content": "帖子评论C"},
}
)
dispatcher = AiCommentDispatcher(repo, client)
settings = AiCommentSettings(per_run=4, daily_cap=500, user_pool_size=4, concurrency=2)
summary = dispatcher.run(settings)
self.assertEqual(summary["status"], "success")
self.assertEqual(summary["selected_count"], 4)
self.assertEqual(summary["success_count"], 4)
self.assertEqual(summary["failed_count"], 0)
self.assertEqual(len(repo.created_tasks), 4)
self.assertEqual(len(repo.saved_comments), 4)
self.assertEqual(len(repo.failed_tasks), 0)
self.assertIn("后续", repo.saved_comments[1][1])
self.assertEqual(len({task.virtual_user_id for task in repo.created_tasks}), 4)
def test_settings_reads_new_config_values(self):
settings = AiCommentSettings.from_config_map(
{
"auto_comment_enabled": "1",
"auto_comment_user_pool_size": "100",
"auto_comment_per_run": "50",
"auto_comment_daily_cap": "500",
"auto_comment_concurrency": "4",
"auto_comment_request_timeout": "180",
"auto_comment_article_prompt": "article prompt",
"auto_comment_post_prompt": "post prompt",
"openai_api_key": "sk-demo",
"openai_base_url": "https://example.com",
"openai_model": "gpt-5.5",
"openai_wire_api": "responses",
"openai_provider_label": "OpenAI",
"openai_reasoning_effort": "high",
"openai_disable_response_storage": "1",
"max_tokens": "300",
"temperature": "0.65",
}
)
self.assertTrue(settings.enabled)
self.assertEqual(settings.user_pool_size, 100)
self.assertEqual(settings.per_run, 50)
self.assertEqual(settings.daily_cap, 500)
self.assertEqual(settings.concurrency, 4)
self.assertEqual(settings.request_timeout, 180)
self.assertEqual(settings.article_prompt, "article prompt")
self.assertEqual(settings.post_prompt, "post prompt")
self.assertEqual(settings.openai_api_key, "sk-demo")
self.assertEqual(settings.max_tokens, 300)
self.assertAlmostEqual(settings.temperature, 0.65)
if __name__ == "__main__":
unittest.main()
server/public/dongqiudi-crawler/tests/test_docker_task_runner.py
-203
View File
@@ -1,203 +0,0 @@
import os
import sys
import tempfile
import unittest
import importlib
from pathlib import Path
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
class FakeStore:
def __init__(self):
self.created = []
self.progress = []
self.finished = []
self.next_id = 100
def create_running(self, task):
self.created.append(task)
self.next_id += 1
return self.next_id
def update_progress(self, log_id, output, elapsed):
self.progress.append((log_id, output, elapsed))
def finish(self, log_id, status, output, elapsed, error_message=""):
self.finished.append((log_id, status, output, elapsed, error_message))
class DockerTaskRunnerTest(unittest.TestCase):
def test_config_uses_dqd_environment_overrides(self):
sys.modules.pop("src.core.config", None)
src_core = importlib.import_module("src.core")
if hasattr(src_core, "config"):
delattr(src_core, "config")
config_module = importlib.import_module("src.core.config")
with tempfile.TemporaryDirectory() as tmp:
settings = Path(tmp) / "settings.yaml"
settings.write_text(
"\n".join(
[
"database:",
' host: "yaml-db"',
" port: 3306",
' database: "yaml_name"',
' username: "yaml_user"',
' password: "yaml_password"',
' prefix: "la_"',
"redis:",
' host: "yaml-redis"',
" port: 6379",
" db: 0",
]
),
encoding="utf-8",
)
env = {
"DQD_DB_HOST": "docker-db",
"DQD_DB_PORT": "3307",
"DQD_DB_NAME": "docker_name",
"DQD_DB_USER": "docker_user",
"DQD_DB_PASSWORD": "docker_password",
"DQD_DB_PREFIX": "sx_",
"DQD_REDIS_HOST": "docker-redis",
"DQD_REDIS_PORT": "6380",
"DQD_REDIS_DB": "5",
}
with patch.dict(os.environ, env, clear=False):
config_module._config = None
cfg = config_module.load_config(str(settings))
self.assertEqual(cfg.database.host, "docker-db")
self.assertEqual(cfg.database.port, 3307)
self.assertEqual(cfg.database.database, "docker_name")
self.assertEqual(cfg.database.username, "docker_user")
self.assertEqual(cfg.database.password, "docker_password")
self.assertEqual(cfg.database.prefix, "sx_")
self.assertEqual(cfg.redis.host, "docker-redis")
self.assertEqual(cfg.redis.port, 6380)
self.assertEqual(cfg.redis.db, 5)
def test_render_crontab_skips_inactive_tasks_and_quotes_arguments(self):
from scripts.docker_task_runner import CrawlerTask, render_crontab
tasks = [
CrawlerTask("nba", "NBA赛程", "nba_match", "*/1 * * * *", True),
CrawlerTask("old", "旧任务", "standings", "0 2 * * *", False),
]
rendered = render_crontab(tasks, python_bin="/usr/local/bin/python", app_dir="/app")
self.assertIn("SHELL=/bin/bash", rendered)
self.assertIn("BASH_ENV=/etc/sport-era-crawler/env.sh", rendered)
self.assertIn("*/1 * * * * root cd /app && /usr/local/bin/python scripts/docker_task_runner.py run nba_match --task-key nba --task-name", rendered)
self.assertIn("--cron-expression '*/1 * * * *'", rendered)
self.assertNotIn("standings", rendered)
def test_load_tasks_accepts_top_level_list_yaml(self):
from scripts.docker_task_runner import load_tasks
with tempfile.TemporaryDirectory() as tmp:
tasks_file = Path(tmp) / "crawler_tasks.yaml"
tasks_file.write_text(
"\n".join(
[
"- key: live",
" name: 实时详情",
" action: live_detail",
" cron: '*/1 * * * *'",
]
),
encoding="utf-8",
)
tasks = load_tasks(tasks_file)
self.assertEqual(len(tasks), 1)
self.assertEqual(tasks[0].task_key, "live")
self.assertEqual(tasks[0].action, "live_detail")
def test_run_task_marks_success_and_flushes_output(self):
from scripts.docker_task_runner import CommandResult, CrawlerTask, run_task
store = FakeStore()
task = CrawlerTask("nba", "NBA赛程", "nba_match", "*/1 * * * *", True, timeout=3)
def fake_runner(action, timeout, output_callback):
output_callback("hello\n")
return CommandResult(exit_code=0, output="hello\nok\n", elapsed=0.2, timed_out=False)
with tempfile.TemporaryDirectory() as tmp:
result = run_task(task, store=store, lock_dir=Path(tmp), command_runner=fake_runner)
self.assertEqual(result.status, 1)
self.assertEqual(store.created[0].action, "nba_match")
self.assertEqual(store.progress[0][1], "hello\n")
self.assertEqual(store.finished[0][1], 1)
self.assertIn("ok", store.finished[0][2])
def test_run_task_marks_failure(self):
from scripts.docker_task_runner import CommandResult, CrawlerTask, run_task
store = FakeStore()
task = CrawlerTask("news", "资讯", "league_news", "*/30 * * * *", True, timeout=3)
def fake_runner(action, timeout, output_callback):
return CommandResult(exit_code=7, output="boom\n", elapsed=0.1, timed_out=False)
with tempfile.TemporaryDirectory() as tmp:
result = run_task(task, store=store, lock_dir=Path(tmp), command_runner=fake_runner)
self.assertEqual(result.status, 2)
self.assertEqual(store.finished[0][1], 2)
self.assertIn("退出码 7", store.finished[0][4])
def test_run_task_marks_timeout_as_failure(self):
from scripts.docker_task_runner import CommandResult, CrawlerTask, run_task
store = FakeStore()
task = CrawlerTask("live", "实时", "live_detail", "*/1 * * * *", True, timeout=1)
def fake_runner(action, timeout, output_callback):
return CommandResult(exit_code=-1, output="slow\n", elapsed=1.0, timed_out=True)
with tempfile.TemporaryDirectory() as tmp:
result = run_task(task, store=store, lock_dir=Path(tmp), command_runner=fake_runner)
self.assertEqual(result.status, 2)
self.assertIn("执行超时", store.finished[0][4])
def test_run_task_skips_duplicate_action(self):
from scripts.docker_task_runner import CrawlerTask, TaskLock, run_task
store = FakeStore()
task = CrawlerTask("live", "实时", "live_detail", "*/1 * * * *", True, timeout=1)
with tempfile.TemporaryDirectory() as tmp:
lock_dir = Path(tmp)
lock = TaskLock("live_detail", lock_dir)
self.assertTrue(lock.acquire())
try:
result = run_task(
task,
store=store,
lock_dir=lock_dir,
command_runner=lambda action, timeout, output_callback: self.fail("duplicate should not run"),
)
finally:
lock.release()
self.assertEqual(result.status, 3)
self.assertEqual(store.finished[0][1], 3)
self.assertIn("已有任务执行中", store.finished[0][2])
if __name__ == "__main__":
unittest.main()
server/public/dongqiudi-crawler/tests/test_kb_worker.py
-111
View File
@@ -1,111 +0,0 @@
import unittest
from pathlib import Path
import sys
import types
APP_DIR = Path(__file__).resolve().parents[1]
if str(APP_DIR) not in sys.path:
sys.path.insert(0, str(APP_DIR))
fake_aiomysql = types.ModuleType("aiomysql")
fake_aiomysql.DictCursor = object
fake_aiomysql.create_pool = None
sys.modules.setdefault("aiomysql", fake_aiomysql)
fake_aiohttp = types.ModuleType("aiohttp")
fake_aiohttp.ClientSession = object
fake_aiohttp.ClientTimeout = object
sys.modules.setdefault("aiohttp", fake_aiohttp)
fake_redis = types.ModuleType("redis")
fake_redis_asyncio = types.ModuleType("redis.asyncio")
fake_redis_asyncio.Redis = object
fake_redis.asyncio = fake_redis_asyncio
sys.modules.setdefault("redis", fake_redis)
sys.modules.setdefault("redis.asyncio", fake_redis_asyncio)
fake_loguru = types.ModuleType("loguru")
fake_loguru.logger = types.SimpleNamespace(info=lambda *args, **kwargs: None, warning=lambda *args, **kwargs: None, error=lambda *args, **kwargs: None)
sys.modules.setdefault("loguru", fake_loguru)
fake_src = types.ModuleType("src")
fake_core = types.ModuleType("src.core")
fake_config = types.ModuleType("src.core.config")
fake_config.get_config = lambda: None
fake_config.load_config = lambda: None
fake_core.config = fake_config
fake_src.core = fake_core
sys.modules.setdefault("src", fake_src)
sys.modules.setdefault("src.core", fake_core)
sys.modules.setdefault("src.core.config", fake_config)
from scripts.kb_worker import DEAD_STREAM_KEY, GROUP_NAME, STREAM_KEY, KbWorker # noqa: E402
class FakeRedis:
def __init__(self):
self.acked = []
self.added = []
async def xack(self, stream, group, message_id):
self.acked.append((stream, group, message_id))
async def xadd(self, stream, payload):
self.added.append((stream, payload))
return "2-0"
class FakeDb:
def __init__(self, result=None, exc=None):
self.result = result if result is not None else {"success": True, "document_id": 1, "chunk_count": 1}
self.exc = exc
self.calls = []
async def upsert_document(self, domain, subtype, source_id, embedder):
self.calls.append((domain, subtype, source_id))
if self.exc:
raise self.exc
return self.result
class FakeEmbedder:
pass
class KbWorkerQueueTest(unittest.IsolatedAsyncioTestCase):
def make_worker(self, db):
worker = object.__new__(KbWorker)
worker.redis = FakeRedis()
worker.db = db
worker.embedder = FakeEmbedder()
worker.max_retries = 1
return worker
async def test_successful_message_is_acked(self):
worker = self.make_worker(FakeDb())
await worker.handle_message("1-0", {"domain": "article", "subtype": "article", "source_id": "123"})
self.assertEqual(worker.db.calls, [("article", "article", 123)])
self.assertEqual(worker.redis.acked, [(STREAM_KEY, GROUP_NAME, "1-0")])
self.assertEqual(worker.redis.added, [])
async def test_message_goes_to_dead_stream_after_retry_limit(self):
worker = self.make_worker(FakeDb(exc=RuntimeError("boom")))
await worker.handle_message(
"1-0",
{"domain": "article", "subtype": "article", "source_id": "123", "attempts": "1"},
)
self.assertEqual(worker.redis.acked, [(STREAM_KEY, GROUP_NAME, "1-0")])
self.assertEqual(len(worker.redis.added), 1)
stream, payload = worker.redis.added[0]
self.assertEqual(stream, DEAD_STREAM_KEY)
self.assertEqual(payload["attempts"], "2")
self.assertIn("boom", payload["last_error"])
if __name__ == "__main__":
unittest.main()
server/public/dongqiudi-crawler/tests/test_runtime_guard.py
-46
View File
@@ -1,46 +0,0 @@
import importlib.util
import pathlib
import sys
import tempfile
import types
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
sys.modules.setdefault("src.core.alert_manager", types.ModuleType("src.core.alert_manager"))
sys.modules["src.core.alert_manager"].CrontabAlertManager = object
sys.modules.setdefault("src.core.config", types.ModuleType("src.core.config"))
sys.modules["src.core.config"].load_config = lambda: None
sys.modules.setdefault("src.core.error_collector", types.ModuleType("src.core.error_collector"))
sys.modules["src.core.error_collector"].ErrorCollector = object
sys.modules.setdefault("src.core.logger", types.ModuleType("src.core.logger"))
sys.modules["src.core.logger"].setup_logger = lambda: None
SPEC = importlib.util.spec_from_file_location("crawler_main", ROOT / "main.py")
crawler_main = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(crawler_main)
class RuntimeGuardTest(unittest.TestCase):
def test_action_lock_blocks_duplicate_running_action(self):
with tempfile.TemporaryDirectory() as tmp:
first = crawler_main.CrawlerActionLock("nba_match", pathlib.Path(tmp))
second = crawler_main.CrawlerActionLock("nba_match", pathlib.Path(tmp))
self.assertTrue(first.acquire())
try:
self.assertFalse(second.acquire())
finally:
first.release()
self.assertTrue(second.acquire())
second.release()
def test_action_timeout_uses_specific_value_or_default(self):
self.assertEqual(crawler_main.get_action_timeout("nba_match"), 180)
self.assertEqual(crawler_main.get_action_timeout("live_detail"), 180)
self.assertEqual(crawler_main.get_action_timeout("article_content_fetch"), 900)
self.assertEqual(crawler_main.get_action_timeout("unknown_task"), 600)
if __name__ == "__main__":
unittest.main()
server/public/dongqiudi-crawler/truth_social.py
-158
View File
@@ -1,158 +0,0 @@
"""
Truth Social 采集模块 - 从 RapidAPI 获取帖子并入库
在 main.py 中通过 truth_social 命令调用
"""
import json
import re
import time
from datetime import datetime
import pymysql
import requests
# ─── 配置 ───
RAPIDAPI_KEY = "482d762c92mshfd65e1ec2ce81e4p103837jsna4df0033beb4"
RAPIDAPI_HOST = "truth-social-api.p.rapidapi.com"
RAPIDAPI_URL = f"https://{RAPIDAPI_HOST}"
SOURCE = "truthsocial"
TARGETS = [
{"username": "realDonaldTrump", "user_id": 8, "tag_id": 13},
]
def strip_html(html: str) -> str:
return re.sub(r"<[^>]+>", "", html or "").strip()
def parse_timestamp(ts_str: str) -> int:
if not ts_str:
return int(time.time())
ts_str = ts_str.replace("Z", "+00:00")
try:
return int(datetime.fromisoformat(ts_str).timestamp())
except Exception:
return int(time.time())
def fetch_feed(session: requests.Session, username: str, max_pages: int = 3) -> list[dict]:
all_posts = []
continue_id = None
for page in range(max_pages):
url = f"{RAPIDAPI_URL}/users/{username}/feed"
params = {}
if continue_id:
params["continue_from_id"] = continue_id
try:
resp = session.get(url, params=params, timeout=30)
except Exception as e:
print(f" [WARN] 请求失败: {e}")
break
if resp.status_code != 200:
print(f" [WARN] API {resp.status_code}: {resp.text[:200]}")
break
data = resp.json()
posts = data if isinstance(data, list) else data.get("data", data.get("statuses", []))
if not posts:
break
for p in posts:
if "content" in p and "content_plain" not in p:
p["content_plain"] = strip_html(p["content"])
all_posts.extend(posts)
continue_id = str(posts[-1].get("id", ""))
if page < max_pages - 1:
time.sleep(2)
return all_posts
def sync_to_db(conn, posts: list[dict], user_id: int, tag_id: int) -> tuple[int, int]:
with conn.cursor() as cur:
cur.execute(
"SELECT origin_id FROM la_community_post WHERE origin_id LIKE %s",
(f"{SOURCE}:%",)
)
existing_ids = {row["origin_id"] for row in cur.fetchall()}
new_count = 0
for post in posts:
post_id = str(post.get("id", ""))
origin_id = f"{SOURCE}:{post_id}"
if not post_id or origin_id in existing_ids:
continue
content_plain = post.get("content_plain") or strip_html(post.get("content", ""))
created_at = parse_timestamp(post.get("created_at", ""))
images = []
for media in post.get("media_attachments", []):
url = media.get("url", "")
if url and media.get("type") in ("image", "gifv"):
images.append(url)
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO la_community_post
(origin_id, user_id, content, images, post_type, ext, status, create_time, update_time)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)""",
(
origin_id, user_id, content_plain,
json.dumps(images) if images else "[]",
0, json.dumps(post, ensure_ascii=False), 1,
created_at, created_at,
)
)
new_post_id = cur.lastrowid
cur.execute(
"INSERT INTO la_community_post_tag (post_id, tag_id) VALUES (%s, %s)",
(new_post_id, tag_id)
)
for idx, img_url in enumerate(images):
cur.execute(
"INSERT INTO la_community_post_image (post_id, image_url, sort, create_time) VALUES (%s, %s, %s, %s)",
(new_post_id, img_url, idx, created_at)
)
existing_ids.add(origin_id)
new_count += 1
return len(existing_ids), new_count
def run(db_config: dict):
"""主入口，db_config 从 dongqiudi-crawler 配置中传入"""
session = requests.Session()
session.headers.update({
"Content-Type": "application/json",
"x-rapidapi-host": RAPIDAPI_HOST,
"x-rapidapi-key": RAPIDAPI_KEY,
})
conn = pymysql.connect(**db_config, cursorclass=pymysql.cursors.DictCursor)
try:
total_candidates = 0
total_saved = 0
for target in TARGETS:
username = target["username"]
print(f"=== @{username} ===")
posts = fetch_feed(session, username)
print(f" API返回 {len(posts)} 条")
total_candidates += len(posts)
if not posts:
continue
existing, new_count = sync_to_db(conn, posts, target["user_id"], target["tag_id"])
conn.commit()
print(f" 已入库 {existing} 条, 新增 {new_count} 条")
total_saved += new_count
return {"success": True, "candidate_count": total_candidates, "saved_count": total_saved, "summary_text": f"Truth Social 共拉取 {total_candidates} 条，新增 {total_saved} 条"}
finally:
conn.close()
server/public/dongqiudi-crawler/update_crontab_error_report.sql
-12
View File
@@ -1,12 +0,0 @@
INSERT INTO `la_dev_crontab` (`name`, `type`, `system`, `remark`, `command`, `params`, `status`, `expression`, `create_time`, `update_time`)
SELECT '爬虫错误报告邮件', 1, 0, '每分钟分发未恢复的定时任务企业微信告警', 'crawler', 'error_report', 1, '*/1 * * * *', UNIX_TIMESTAMP(), UNIX_TIMESTAMP()
WHERE NOT EXISTS (
SELECT 1 FROM `la_dev_crontab` WHERE `command` = 'crawler' AND `params` = 'error_report' AND `delete_time` IS NULL
);
UPDATE `la_dev_crontab`
SET `name` = '爬虫错误报告邮件',
`remark` = '每分钟分发未恢复的定时任务企业微信告警',
`expression` = '*/1 * * * *',
`update_time` = UNIX_TIMESTAMP()
WHERE `command` = 'crawler' AND `params` = 'error_report' AND `delete_time` IS NULL;