一个神奇的蜘蛛?项目，适用于数据采集任务，源码结构简洁明了。
index页面示例：
---
项目地址https://github.com/lixi5338619/magical_spider
使用指南1、配置settings.py，启动flask服务
2、参考demo文件夹中的代码进行测试，主要通过runflow.py运行。
代码语言：javascript代码运行次数：0运行复制```javascript import requestshost = 'http://127.0.0.1:5000'def magical_start(project_name,base_url = 'https://www.php.cn/link/b6f05a7baab2fe0eea07e59bd5b0b317'): # 1、创建浏览器并选择session_id result = requests.post(f'{host}/create',data={'name':project_name,'url':base_url}).json() session_id,process_url = result['session_id'],result['process_url'] return session_id,process_urldef magical_request(session_id,process_url,request_url): # 2、请求浏览器_xhr data = {'session_id':session_id,'process_url':process_url, 'request_url':request_url,'request_type':'get'} result = requests.post(f'{host}/xhr',data=data).json() return result['result']def magical_close(session_id,process_url,process_name): # 4、关闭浏览器 close_data = {'session_id':session_id,'process_url':process_url,'process_name':process_name} requests.post(f'{host}/close',data=close_data).json()
<code>
3、测试代码
<p>GET请求</p><p>代码语言：javascript代码运行次数：0<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="<a href="https://www.php.cn/link/c9041cfd2a40932691855abd98fd219a">http://www.w3.org/2000/svg"><path</a> d="M6.66666 10.9999L10.6667 7.99992L6.66666 4.99992V10.9999ZM7.99999 1.33325C4.31999 1.33325 1.33333 4.31992 1.33333 7.99992C1.33333 11.6799 4.31999 14.6666 7.99999 14.6666C11.68 14.6666 14.6667 11.6799 14.6667 7.99992C14.6667 4.31992 11.68 1.33325 7.99999 1.33325ZM7.99999 13.3333C5.05999 13.3333 2.66666 10.9399 2.66666 7.99992C2.66666 5.05992 5.05999 2.66659 7.99999 2.66659C10.94 2.66659 13.3333 5.05992 13.3333 7.99992C13.3333 10.9399 10.94 13.3333 7.99999 13.3333Z" fill="currentcolor"></path></svg>运行<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="<a href="https://www.php.cn/link/c9041cfd2a40932691855abd98fd219a">http://www.w3.org/2000/svg"><path</a> clip-rule="evenodd" d="M4.5 15.5V3.5H14.5V15.5H4.5ZM12.5 5.5H6.5V13.5H12.5V5.5ZM9.5 2.5H3.5V12.5H1.5V0.5H11.5V2.5H9.5Z" fill="currentcolor" fill-rule="evenodd"></path></svg>复制<code>javascript from demo.runflow import magical_start,magical_request,magical_closeproject_name = 'cnipa'base_url = 'https://www.cnipa.gov.cn'session_id,process_url = magical_start(project_name,base_url)print(len(magical_request(session_id, process_url,'https://www.cnipa.gov.cn/col/col57/index.html')))magical_close(session_id,process_url,project_name)</code></pre><p>POST请求</p><p>代码语言：javascript代码运行次数：0<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path d="M6.66666 10.9999L10.6667 7.99992L6.66666 4.99992V10.9999ZM7.99999 1.33325C4.31999 1.33325 1.33333 4.31992 1.33333 7.99992C1.33333 11.6799 4.31999 14.6666 7.99999 14.6666C11.68 14.6666 14.6667 11.6799 14.6667 7.99992C14.6667 4.31992 11.68 1.33325 7.99999 1.33325ZM7.99999 13.3333C5.05999 13.3333 2.66666 10.9399 2.66666 7.99992C2.66666 5.05992 5.05999 2.66659 7.99999 2.66659C10.94 2.66659 13.3333 5.05992 13.3333 7.99992C13.3333 10.9399 10.94 13.3333 7.99999 13.3333Z" fill="currentcolor"></path></svg>运行<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path clip-rule="evenodd" d="M4.5 15.5V3.5H14.5V15.5H4.5ZM12.5 5.5H6.5V13.5H12.5V5.5ZM9.5 2.5H3.5V12.5H1.5V0.5H11.5V2.5H9.5Z" fill="currentcolor" fill-rule="evenodd"></path></svg>复制</code>javascript
from demo.runflow import magical_start,magical_request,magical_closeimport jsonproject_name = 'chinadrugtrials'base_url = '<a href="https://www.php.cn/link/1751a9a215df522dcf207ba51b327643"><a href="https://www.php.cn/link/1751a9a215df522dcf207ba51b327643">http://www.chinadrugtrials.org.cn'session_id,process_url</a></a> = magical_start(project_name,base_url)data = {"id": "","ckm_index": "","sort": "desc","sort2": "","rule": "CTR","secondLevel": "0","currentpage": "2","keywords": "","reg_no": "","indication": "","case_no": "","drugs_name": "","drugs_type": "","appliers": "","communities": "","researchers": "","agencies": "","state": ""}formdata = json.dumps(data)print(magical_request(session_id=session_id, process_url=process_url, request_url='<a href="https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268"><a href="https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268">https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268</a></a>', request_type='post',formdata=formdata ))magical_close(session_id,process_url,project_name)</p><pre class="brush:php;toolbar:false;"><code></p><p>4、index页面可以查看和管理当前运行中的任务，同时可以查看系统的内存和磁盘使用情况。</p><p>5、demo文件夹包含任务流程汇总runflow.py，以及抖音和药监局的案例，提供了单任务和多任务的示例。</p><hr /><p>linux部署1.安装chrome（选择安装位置） yum install <a href="https://www.php.cn/link/6b17d006a2ed6f12f07c7ea60b8002b5">https://www.php.cn/link/6b17d006a2ed6f12f07c7ea60b8002b5</a></p><div class="aritcle_card flexRow">
<div class="artcardd flexRow">
<a class="aritcle_card_img" href="/ai/1715" title="Chromox"><img
src="https://img.php.cn/upload/ai_manual/000/000/000/175680315684832.png" alt="Chromox" onerror="this.onerror='';this.src='/static/lhimages/moren/morentu.png'" ></a>
<div class="aritcle_card_info flexColumn">
<a href="/ai/1715" title="Chromox">Chromox</a>
<p>Chromox是一款领先的AI在线生成平台，专为喜欢AI生成技术的爱好者制作的多种图像、视频生成方式的内容型工具平台。</p>
</div>
<a href="/ai/1715" title="Chromox" class="aritcle_card_btn flexRow flexcenter"><b></b><span>下载</span> </a>
</div>
</div><p>2.检查chrome版本 google-chrome --version</p><p>3.安装对应版本的chromedriver_linux64 例如，我的chrome版本是104.0.5112.79 wget <a href="https://www.php.cn/link/5bb5f8d030f5e6e4b6d137c6990f0772">https://www.php.cn/link/5bb5f8d030f5e6e4b6d137c6990f0772</a></p><p>4.解压chromedriver unzip chromedriver_linux64</p><p>5.授权chromedriver chmod 777 chromedriver</p><p>6.修改项目代码settings.py中的chromedriver路径</p><p>7.安装python依赖并启动flask项目</p><p>Python依赖：flask、sqlite3、selenium、websockets、opencv-python、numpyflask启动方式：python3 server.py8.开启服务器端口访问权限</p><p>9.运行项目测试</code> 
