ScrapeGraphAI-experiments/README.md

ScrapeGraphAI

ScrapeGraphAI是一个用于网络爬虫和数据抓取的AI工具。

• https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/docs/chinese.md
• https://github.com/ScrapeGraphAI/ScrapegraphLib-Examples

Reference

https://www.aivi.fyi/aiagents/introduce-ScrapeGraphAI+LangChain+LangGraph#可用管道

Dependencies

pip install scrapegraphai
playwright install
pip install --upgrade duckduckgo-search
pip install scrapegraphai'[other-language-models]'
pip install scrapegraphai'[more-semantic-options]'
pip install scrapegraphai'[more-browser-options]'

ollama pull mistral-nemo
ollama list

Tips

• Comment
  • 小参数模型的api，比调用gpt-4o的省钱很多
  • Playwright +plugins 能解决一部分captcha。如果再加上llm，基本就不是什么问题了
  • 这个repo就是传统爬虫套了一个ai的壳子，数据解析部分用ai来做代替以前的hard code, 反爬只能通过ip proxy (家宅ip供应商最好) + playwright or chrome driver&selenium attach到 chrome进程来解决