Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrapy crawl v2ex 出错了 #1

Open
xinmans opened this issue Jul 6, 2023 · 5 comments
Open

scrapy crawl v2ex 出错了 #1

xinmans opened this issue Jul 6, 2023 · 5 comments

Comments

@xinmans
Copy link

xinmans commented Jul 6, 2023

包都pip install了
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/home/xinmans/mydev/v2ex_scrapy/v2ex_scrapy/spiders/V2exSpider.py", line 7, in
from v2ex_scrapy.DB import DB
File "/home/xinmans/mydev/v2ex_scrapy/v2ex_scrapy/DB.py", line 7, in
from v2ex_scrapy.items import (
File "/home/xinmans/mydev/v2ex_scrapy/v2ex_scrapy/items.py", line 13, in
class Base(DeclarativeBase):
File "/home/xinmans/mydev/v2ex_scrapy/v2ex_scrapy/items.py", line 15, in Base
list[str]: JSON,
TypeError: 'type' object is not subscriptabl

@oldshensheep
Copy link
Owner

需要python版本python>=3.10

@xinmans
Copy link
Author

xinmans commented Jul 6, 2023

可以做个docker

Dockerfile

FROM python:3.10
WORKDIR /app

COPY requirements.txt /app/requirements.txt

RUN pip install -r requirements.txt

CMD cd /app && scrapy crawl v2ex

docker-compose.yml

version: "3.7"
services:
v2ex_scrpay:
container_name: v2ex_scrpay
build: .
volumes:
- /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
- ./:/app/
restart: always

运行sudo docker-compose up -d即可

@oldshensheep
Copy link
Owner

这个应用太简单了,加个Docker没必要

@xinmans
Copy link
Author

xinmans commented Jul 7, 2023

这个应用太简单了,加个Docker没必要

是的
但是我基础环境是python3.8版本的,换版本影响其他应用了

@oldshensheep
Copy link
Owner

Python有很多第三方的环境管理工具,可以用不同的Python版本。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants