Ubuntu 16.04 系统(解释器为 python3.12)在Pycharm虚拟环境中安装 pyspider 爬虫工具
<p>一:安装步骤<br>步骤1. 系统Terminal命令行执行如下命令安装依赖的组件 PhantomJS<br> </p><div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
$ sudo tar -xvf phantomjs-2.1.1-linux-x86_64.tar.bz2 -C /usr/local/
$ sudo ln -s /usr/local/phantomjs-2.1.1-linux-x86_64/bin/phantomjs/usr/local/bin/phantomjs
$ phantomjs --version
</pre>
</div>
<p> </p>
<p>步骤2. 系统Terminal命令行执行如下命令安装其他依赖包</p>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">$ sudo apt update
$ sudo apt install libcurl4-openssl-dev backports
</pre>
</div>
<p> </p>
<br><br>步骤3. pyspider 依赖的 tornado 库在 Python 3.12 环境下需要 backports.ssl_match_hostname 模块，而 pyspider 尚未完全适配这些改动。需通过pycharm的Terminal中执行如下命令解决<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ pip install backports.ssl_match_hostname
</pre>
</div>
<p> </p>
<p>步骤4（可选）. pyspider 在 Python 3.12 环境中运行时存在兼容性问题。附件中是已修复好兼容问题的终版压缩包，可以直接解压使用。然后进行步骤6（前提是先通过pip install强制安装了异常版本 pip install --force-reinstall pyspider），如果想自己一步步修改源码，可跳过此步，参考"步骤5.兼容问题修复"<br>点击下载压缩包：</p>
<p><br>步骤5. 兼容问题修复。pyspider 在 Python 3.12 环境中运行时存在兼容性问题。这是由于 Python 3.12 对部分旧模块进行了移除或者调整，而 pyspider 尚未完全适配这些改动，需要做如下修改<br> 5.1 兼容问题1：Python3 中的 async 已经变成了关键字。需将源码中 async 替换成其他变量，</p>
<p>如： 将下面位置的 async 改为 mark_async<br> </p>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> pyspider/run.py 的231行、245行(两个)、365行
pyspider/webui/app.py 的95行
pyspider/fetcher/tornado_fetcher.py 的81行、89行(两个)、95行、117行
</pre>
</div>
<p> </p>
5.2 兼容问题2：部分旧模块进行了移除或者调整，而 pyspider 尚未完全适配这些改动。需通过如下步骤手动修改pyspider代码来解决兼容性问题<br> a). 修复 UserDict 和 Mapping 导入问题：<br> 把 pyspider/libs/counter.py 文件里的python代码第14行： <br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from UserDict import DictMixin
except ImportError:
from collections import Mapping as DictMixin
</pre>
</div>
<br> 改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from collections import UserDict as DictMixin
except ImportError:
from collections.abc import Mapping as DictMixin
</pre>
</div>
<br> 把 pyspider/scheduler/task_queue.py 文件里的python代码第12行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from UserDict import DictMixin
except ImportError:
from collections import Mapping as DictMixin
</pre>
</div>
<br> 改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from collections import UserDict as DictMixin
except ImportError:
from collections.abc import Mapping as DictMixin
</pre>
</div>
<br> b). 修复 imp 模块缺失问题：<br> 把 pyspider/processor/project_module.py 文件中的python代码第11行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">import imp
</pre>
</div>
<p> 改成： </p>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> import importlib.util
</pre>
</div>
<br> c). 修复 MutableMapping 导入问题：<br> tornado 库引用MutableMapping出现错误，可修改 tornado/httputil.py 文件中的python代码第106行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">class HTTPHeaders(collections.MutableMapping):
</pre>
</div>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class HTTPHeaders(collections.abc.MutableMapping):
</pre>
</div>
5.3 兼容问题3：pyspider 在 Python 3.12 环境下运行时存在兼容性问题，fractions模块已被移除，而 pyspider 尚未完全适配这些改动。修改如下将 fractions 替换成 math<br> a). pyspider/libs/base_handler.py的python代码第12行空白行新增<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">import math
</pre>
</div>
<br> b).修改 pyspider/libs/base_handler.py的python代码将其中的第115行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">min_tick = fractions.gcd(min_tick, each.tick)
</pre>
</div>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">min_tick = math.gcd(min_tick, each.tick)
</pre>
</div>
<p> </p>
5.4 兼容问题4：pyspider 使用的 Flask 版本不兼容。pyspider 是基于 Flask 旧版本开发的，而新版本（如 Flask 2.3+）移除了 before_first_request 装饰器，需进行如下修改<br> 修改 pyspider/webui/debug.py的python代码将其中的第 64 行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> @app.before_first_request
def enable_projects_import():
sys.meta_path.append(ProjectFinder(app.config['projectdb']))
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> @app.before_request
def enable_projects_import():
if not hasattr(app, '_got_first_request'):
app._got_first_request = True
sys.meta_path.append(ProjectFinder(app.config['projectdb']))
</pre>
</div>
<p> </p>
5.5 兼容问题5：pyspider 的 WebDAV 模块在 Python 3.12 环境下存在兼容性问题。<br> a) 在 Python 3.12 中，抽象基类（ABC）的检查变得更加严格，ScriptProvider 类没有实现其基类要求的所有抽象方法。修改如下<br> 修改 pyspider/webui/webdav.py的python代码第165行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class ScriptProvider(DAVProvider):
def __init__(self, app):
super(ScriptProvider, self).__init__()
self.app = app
def __repr__(self):
return "pyspiderScriptProvider"
def getResourceInst(self, path, environ):
path = os.path.normpath(path).replace('\\', '/')
if path in ('/', '.', ''):
path = '/'
return RootCollection(path, environ, self.app)
else:
return ScriptResource(path, environ, self.app)
</pre>
</div>
<p> </p>
改为：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class ScriptProvider(DAVProvider):
def __init__(self, app):
super(ScriptProvider, self).__init__()
self.app = app
def __repr__(self):
return "pyspiderScriptProvider"
def getResourceInst(self, path, environ):
path = os.path.normpath(path).replace('\\', '/')
if path in ('/', '.', ''):
path = '/'
return RootCollection(path, environ, self.app)
else:
return ScriptResource(path, environ, self.app)
# 添加缺失的抽象方法实现
def get_resource_inst(self, path, environ):
return ScriptResource(path, self, environ)
</pre>
</div>
<p> </p>
b) WsgiDAV 库的配置格式发生了改变，domaincontroller 选项已被弃用，需要使用 http_authenticator.domain_controller 替代。修改如下<br> 修改 pyspider/webui/webdav.py的python代码第207行<br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
'domaincontroller': NeedAuthController(app),
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)
</pre>
</div>
<p> </p>
改成：<br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
# 更新认证配置
"http_authenticator": {
"domain_controller": NeedAuthController,# 移动到 http_authenticator 下
"accept_basic": True,
"accept_digest": False,
"default_to_digest": False,
},
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)
</pre>
</div>
<br> c) WsgiDAV 的认证控制器接口发生变化，存在兼容性问题，修改如下<br> 修改 pyspider/webui/webdav.py的python代码第186行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class NeedAuthController(object):
def __init__(self, app):
self.app = app
def getDomainRealm(self, inputRelativeURL, environ):
return 'need auth'
def requireAuthentication(self, realmname, environ):
return self.app.config.get('need_auth', False)
def isRealmUser(self, realmname, username, environ):
return username == self.app.config.get('webui_username')
def getRealmUserPassword(self, realmname, username, environ):
return self.app.config.get('webui_password')
def authDomainUser(self, realmname, username, password, environ):
return username == self.app.config.get('webui_username') \
and password == self.app.config.get('webui_password')
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class NeedAuthController(object):
def __init__(self, app, config=None):
self.app = app
# 处理额外的config参数，使其兼容WsgiDAV的初始化方式
if config is not None:
self.config = config
else:
# 如果config未提供，尝试从app中获取
self.config = app.config.get("http_authenticator", {})
def getDomainRealm(self, inputRelativeURL, environ):
return 'need auth'
def requireAuthentication(self, realmname, environ):
return self.app.config.get('need_auth', False)
def isRealmUser(self, realmname, username, environ):
return username == self.app.config.get('webui_username')
def getRealmUserPassword(self, realmname, username, environ):
return self.app.config.get('webui_password')
def authDomainUser(self, realmname, username, password, environ):
return username == self.app.config.get('webui_username') \
and password == self.app.config.get('webui_password')
# 添加WsgiDAV期望的接口方法，转发到原有方法
def get_domain_realm(self, input_path, environ):
return self.getDomainRealm(input_path, environ)
def basic_auth_user(self, realm, user_name, password, environ):
return self.authDomainUser(realm, user_name, password, environ)
def supports_http_digest_auth(self):
return False# 我们不支持摘要认证
def is_share_anonymous(self, share_path):
"""检查指定的共享路径是否允许匿名访问"""
# 如果不需要认证，则所有共享都允许匿名访问
return not self.app.config.get('need_auth', False)
</pre>
</div>
<p> </p>
5.6 兼容问题6：Werkzeug 库版本与 pyspider 不兼容。从 Python 3.12 开始，Werkzeug v2.3.0 及以上版本已经移除了DispatcherMiddleware，将其移至独立的werkzeug.middleware.dispatcher模块中。<br> 而 pyspider 仍在使用旧的导入方式。修改如下<br> 修改 pyspider/webui/app.py的python代码第64行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">from werkzeug.wsgi import DispatcherMiddleware
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> from werkzeug.middleware.dispatcher import DispatcherMiddleware
</pre>
</div>
<br>步骤6. pycharm的Terminal中执行如下命令安装pyspider<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ pip install pyspider
</pre>
</div>
<p> </p>
步骤7. Pycharm的Terminal命令行执行如下命令，验证pyspider是否安装成功，打印所示当前安装的pyspider最新版本是 0.3.10<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ pyspider --version
pyspider, version 0.3.10
</pre>
</div>
<br><br>步骤8. pycharm的Terminal命令行执行 pyspider 启动 pyspider 网页端控制台，如下打印结果表示成功启动 pyspider，并且启用了5000端口。浏览器可以访问 <span style="color: rgba(255, 0, 0, 1)"><strong>http://localhost:5000/</strong></span> 进入PySpider网页控制台爬数据了。<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ pyspider<br> phantomjs fetcher running on port 25555<br> result_worker starting...<br> processor starting...<br> fetcher starting...<br> scheduler starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br> in 5m: new:0,success:0,retry:0,failed:0<br> webui running on 0.0.0.0:5000
</pre>
</div>
<br><br>步骤9. 如果运行 pyspider 命令，出现错误Error: Could not create web server listening on port 25555，原因是25555端口被占用，需要释放端口重新执行步骤8<br>解决方案： 使用 lsof -i 25555查看端口被哪个PID占用，然后用 kill -9 <PID> 释放端口 <br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ lsof -i :25555
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
phantomjs 16615 wanghu 7uIPv4 278700 0t0TCP *:25555 (LISTEN)
$ kill -9 16615
</pre>
</div>
<p> </p>
二：pyspider踩坑过程与解决方案：<br>1.安装 pyspider 时出现了如下的 ConfigurationError: Could not run curl-config 错误，这是因为系统缺少 curl-devel 或 libcurl 开发库，而 pycurl 依赖这些库来编译。 <br>解决方案：使用如下命令安装 libcurl4-openssl-dev 包（Ubuntu/Debian 系统），它包含了编译 pycurl 所需的 curl-config 工具和头文件<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">sudo apt install libcurl4-openssl-dev
</pre>
</div>
<br><br>错误详情：<br>Getting requirements to build wheel ... error<br> error: subprocess-exited-with-error<br> <br> × Getting requirements to build wheel did not run successfully.<br> │ exit code: 1<br> ╰─> <br> Traceback (most recent call last):<br> File "<string>", line 230, in configure_unix<br> File "/usr/local/lib/python3.12/subprocess.py", line 1026, in __init__<br> self._execute_child(args, executable, preexec_fn, close_fds,<br> File "/usr/local/lib/python3.12/subprocess.py", line 1950, in _execute_child<br> raise child_exception_type(errno_num, err_msg, err_filename)<br> FileNotFoundError: No such file or directory: 'curl-config'<br> <br> During handling of the above exception, another exception occurred:<br> <br> Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module><br> main()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main<br> json_out["return_val"] = hook(**hook_input["kwargs"])<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel<br> return hook(config_settings)<br> ^^^^^^^^^^^^^^^^^^^^^<br> File "/tmp/pip-build-env-6by6cyoh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel<br> return self._get_build_requires(config_settings, requirements=[])<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/tmp/pip-build-env-6by6cyoh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires<br> self.run_setup()<br> File "/tmp/pip-build-env-6by6cyoh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 512, in run_setup<br> super().run_setup(setup_script=setup_script)<br> File "/tmp/pip-build-env-6by6cyoh/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup<br> exec(code, locals())<br> File "<string>", line 1016, in <module><br> File "<string>", line 676, in get_extension<br> File "<string>", line 93, in __init__<br> File "<string>", line 235, in configure_unix<br> ConfigurationError: Could not run curl-config: No such file or directory: 'curl-config'<br> <br> <br> note: This error originates from a subprocess, and is likely not a problem with pip.<br>error: subprocess-exited-with-error<br><br>× Getting requirements to build wheel did not run successfully.<br>│ exit code: 1<br>╰─> See above for output.<br><br>2. 安装完pyspider后在Terminal中输入 pyspider命令运行后出现如下错误，是因为Python3 中的 async 已经变成了关键字。<br>解决方案：将 async 替换成其他变量，如 将下面位置的 async 改为 mark_async<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> pyspider/run.py 的231行、245行(两个)、365行
pyspider/webui/app.py 的95行
pyspider/fetcher/tornado_fetcher.py 的81行、89行(两个)、95行、117行
</pre>
</div>
<br>错误详情：<br>$ pyspider<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 5, in <module><br> from pyspider.run import main<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 231<br> async=True, get_object=False, no_input=False):<br> ^^^^^<br>SyntaxError: invalid syntax<br><br>3. 再次尝试运行 pyspider 命令，出现如下错误，是因为pyspider 在 Python 3.12 环境中运行时存在兼容性问题。这是由于 Python 3.12 对部分旧模块进行了移除或者调整，而 pyspider 尚未完全适配这些改动。<br>解决方案：通过如下步骤手动修改pyspider代码来解决兼容性问题<br>a). 修复 UserDict 和 Mapping 导入问题：<br>把 pyspider/libs/counter.py 文件里的python代码第14行： <br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from UserDict import DictMixin
except ImportError:
from collections import Mapping as DictMixin
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from collections import UserDict as DictMixin
except ImportError:
from collections.abc import Mapping as DictMixin
</pre>
</div>
<br>把 pyspider/scheduler/task_queue.py 文件里的python代码第12行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from UserDict import DictMixin
except ImportError:
from collections import Mapping as DictMixin
</pre>
</div>
<br>改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> try:
from collections import UserDict as DictMixin
except ImportError:
from collections.abc import Mapping as DictMixin
</pre>
</div>
<p> </p>
b). 修复 imp 模块缺失问题：<br>把 pyspider/processor/project_module.py 文件中的python代码第11行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">import imp
</pre>
</div>
<br>改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">import importlib.util
</pre>
</div>
<br>c). 修复 MutableMapping 导入问题：<br>tornado 库引用MutableMapping出现错误，可修改 tornado/httputil.py 文件中的python代码第106行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">class HTTPHeaders(collections.MutableMapping):
</pre>
</div>
<br>改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">class HTTPHeaders(collections.abc.MutableMapping):
</pre>
</div>
错误详情：<br><br>$ pyspider<br> phantomjs not found, continue running without it.<br> result_worker starting...<br>Process Process-5:<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/counter.py", line 14, in <module><br> from UserDict import DictMixin<br>ModuleNotFoundError: No module named 'UserDict'<br><br>During handling of the above exception, another exception occurred:<br><br>Traceback (most recent call last):<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap<br> self.run()<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run<br> self._target(*self._args, **self._kwargs)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 192, in scheduler<br> Scheduler = load_cls(None, None, scheduler_cls)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/scheduler/__init__.py", line 1, in <module><br> from .scheduler import Scheduler, OneScheduler, ThreadBaseScheduler # NOQA<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/scheduler/scheduler.py", line 19, in <module><br> from pyspider.libs import counter, utils<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/counter.py", line 16, in <module><br> from collections import Mapping as DictMixin<br>ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/lib/python3.12/collections/__init__.py)<br>Process Process-4:<br>Traceback (most recent call last):<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap<br> self.run()<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run<br> self._target(*self._args, **self._kwargs)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 236, in fetcher<br> Fetcher = load_cls(None, None, fetcher_cls)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/__init__.py", line 1, in <module><br> from .tornado_fetcher import Fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/tornado_fetcher.py", line 21, in <module><br> import tornado.httputil<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/httputil.py", line 106, in <module><br> class HTTPHeaders(collections.MutableMapping):<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br>AttributeError: module 'collections' has no attribute 'MutableMapping'<br>Process Process-3:<br>Traceback (most recent call last):<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap<br> self.run()<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run<br> self._target(*self._args, **self._kwargs)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 273, in processor<br> Processor = load_cls(None, None, processor_cls)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/processor/__init__.py", line 1, in <module><br> from .processor import ProcessorResult, Processor<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/processor/processor.py", line 20, in <module><br> from .project_module import ProjectManager, ProjectFinder<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/processor/project_module.py", line 11, in <module><br> import imp<br>ModuleNotFoundError: No module named 'imp'<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 333, in webui<br> app = load_cls(None, None, webui_instance)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/__init__.py", line 8, in <module><br> from . import app, index, debug, task, result, login<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 17, in <module><br> from pyspider.fetcher import tornado_fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/__init__.py", line 1, in <module><br> from .tornado_fetcher import Fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/tornado_fetcher.py", line 21, in <module><br> import tornado.httputil<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/httputil.py", line 106, in <module><br> class HTTPHeaders(collections.MutableMapping):<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br>AttributeError: module 'collections' has no attribute 'MutableMapping'<br><br><br><br>4. 再次尝试运行 pyspider 命令，出现如下import backports.ssl_match_hostname ModuleNotFoundError: No module named 'backports'错误，是因为pyspider 依赖的 tornado 库在 Python 3.12 环境下需要 backports.ssl_match_hostname 模块，而 pyspider 尚未完全适配这些改动。<br>解决方案：<br>backports.ssl_match_hostname 模块缺失问题，可通过pycharm的Terminal中执行如下命令解决<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">$ pip install backports.ssl_match_hostname
</pre>
</div>
<br>补充： Error: Could not create web server listening on port 25555 属于端口占用问题，将其他问题都解决了再处理此问题<br><br>错误详情：<br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> processor starting...<br>Process Process-4:<br>Traceback (most recent call last):<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap<br> self.run()<br> File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run<br> self._target(*self._args, **self._kwargs)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 236, in fetcher<br> Fetcher = load_cls(None, None, fetcher_cls)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/__init__.py", line 1, in <module><br> from .tornado_fetcher import Fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/tornado_fetcher.py", line 31, in <module><br> from tornado.simple_httpclient import SimpleAsyncHTTPClient<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/simple_httpclient.py", line 8, in <module><br> from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/http1connection.py", line 30, in <module><br> from tornado import iostream<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/iostream.py", line 40, in <module><br> from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/netutil.py", line 56, in <module><br> import backports.ssl_match_hostname<br>ModuleNotFoundError: No module named 'backports'<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 333, in webui<br> app = load_cls(None, None, webui_instance)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/__init__.py", line 8, in <module><br> from . import app, index, debug, task, result, login<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 17, in <module><br> from pyspider.fetcher import tornado_fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/__init__.py", line 1, in <module><br> from .tornado_fetcher import Fetcher<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/fetcher/tornado_fetcher.py", line 31, in <module><br> from tornado.simple_httpclient import SimpleAsyncHTTPClient<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/simple_httpclient.py", line 8, in <module><br> from tornado.http1connection import HTTP1Connection, HTTP1ConnectionParameters<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/http1connection.py", line 30, in <module><br> from tornado import iostream<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/iostream.py", line 40, in <module><br> from tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateError, _client_ssl_defaults, _server_ssl_defaults<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/tornado/netutil.py", line 56, in <module><br> import backports.ssl_match_hostname<br>ModuleNotFoundError: No module named 'backports'<br><br>5. 再次尝试运行 pyspider 命令，出现如下错误信息AttributeError: module 'fractions' has no attribute 'gcd'，是因为pyspider 在 Python 3.12 环境下运行时存在兼容性问题，fractions模块已被移除，而 pyspider 尚未完全适配这些改动。<br><br>解决方案：<br>a). pyspider/libs/base_handler.py的python代码第12行空白行新增<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> import math
</pre>
</div>
<p> </p>
b).修改 pyspider/libs/base_handler.py的python代码将其中的第115行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">min_tick = fractions.gcd(min_tick, each.tick)
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">min_tick = math.gcd(min_tick, each.tick)
</pre>
</div>
<p> </p>
错误详情：<br><br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 333, in webui<br> app = load_cls(None, None, webui_instance)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/__init__.py", line 8, in <module><br> from . import app, index, debug, task, result, login<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/debug.py", line 22, in <module><br> from pyspider.libs import utils, sample_handler, dataurl<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/sample_handler.py", line 9, in <module><br> class Handler(BaseHandler):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/base_handler.py", line 115, in __new__<br> min_tick = fractions.gcd(min_tick, each.tick)<br> ^^^^^^^^^^^^^<br>AttributeError: module 'fractions' has no attribute 'gcd'<br><br><br>6. 再次尝试运行 pyspider 命令，出现如下Flask 兼容性问题错误信息（AttributeError: 'QuitableFlask' object has no attribute 'before_first_request'），这个错误是因为 pyspider 使用的 Flask 版本不兼容。pyspider 是基于 Flask 旧版本开发的，而新版本（如 Flask 2.3+）移除了 before_first_request 装饰器<br><br>解决方案：<br>修改 pyspider/webui/debug.py的python代码将其中的第 64 行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> @app.before_first_request
def enable_projects_import():
sys.meta_path.append(ProjectFinder(app.config['projectdb']))
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> @app.before_request
def enable_projects_import():
if not hasattr(app, '_got_first_request'):
app._got_first_request = True
sys.meta_path.append(ProjectFinder(app.config['projectdb']))
</pre>
</div>
<p> </p>
<br><br>错误详情：<br><br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br> fetcher starting...<br>Error: Could not create web server listening on port 25555<br> scheduler starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br>Error: Could not create web server listening on port 25555<br> in 5m: new:0,success:0,retry:0,failed:0<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 333, in webui<br> app = load_cls(None, None, webui_instance)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 48, in load_cls<br> return utils.load_object(value)<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/libs/utils.py", line 369, in load_object<br> module = __import__(module_name, globals(), locals(), )<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/__init__.py", line 8, in <module><br> from . import app, index, debug, task, result, login<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/debug.py", line 64, in <module><br> @app.before_first_request<br> ^^^^^^^^^^^^^^^^^^^^^^^^<br>AttributeError: 'QuitableFlask' object has no attribute 'before_first_request'. Did you mean: '_got_first_request'?<br>Error: Could not create web server listening on port 25555<br><br><br>7. 再次尝试运行 pyspider 命令，出现错误 TypeError: Can't instantiate abstract class ScriptProvider without an implementation for abstract method 'get_resource_inst' ，原因是pyspider 的 WebDAV 模块在 Python 3.12 环境下存在兼容性问题。在 Python 3.12 中，抽象基类（ABC）的检查变得更加严格，ScriptProvider 类没有实现其基类要求的所有抽象方法。<br><br>解决方案：<br>修改pyspider/webui/webdav.py的python代码第165行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class ScriptProvider(DAVProvider):
def __init__(self, app):
super(ScriptProvider, self).__init__()
self.app = app
def __repr__(self):
return "pyspiderScriptProvider"
def getResourceInst(self, path, environ):
path = os.path.normpath(path).replace('\\', '/')
if path in ('/', '.', ''):
path = '/'
return RootCollection(path, environ, self.app)
else:
return ScriptResource(path, environ, self.app)
</pre>
</div>
<p> </p>
改为：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class ScriptProvider(DAVProvider):
def __init__(self, app):
super(ScriptProvider, self).__init__()
self.app = app
def __repr__(self):
return "pyspiderScriptProvider"
def getResourceInst(self, path, environ):
path = os.path.normpath(path).replace('\\', '/')
if path in ('/', '.', ''):
path = '/'
return RootCollection(path, environ, self.app)
else:
return ScriptResource(path, environ, self.app)
# 添加缺失的抽象方法实现
def get_resource_inst(self, path, environ):
return ScriptResource(path, self, environ)
</pre>
</div>
<p> </p>
<br>错误详情：<br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br>Error: Could not create web server listening on port 25555<br> scheduler starting...<br> fetcher starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br> in 5m: new:0,success:0,retry:0,failed:0<br>Error: Could not create web server listening on port 25555<br> webui exiting...<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br>Error: Could not create web server listening on port 25555<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 384, in webui<br> app.run(host=host, port=port)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 59, in run<br> from .webdav import dav_app<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/webdav.py", line 207, in <module><br> '/': ScriptProvider(app)<br> ^^^^^^^^^^^^^^^^^^^<br>TypeError: Can't instantiate abstract class ScriptProvider without an implementation for abstract method 'get_resource_inst'<br><br>8. 再次尝试运行 pyspider 命令，出现错误 ValueError: Invalid configuration: - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.，原因是 WsgiDAV 库的配置格式发生了改变，domaincontroller 选项已被弃用，需要使用 http_authenticator.domain_controller 替代。<br><br>解决方案：<br>将pyspider/webui/webdav.py的python代码第207行<br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
'domaincontroller': NeedAuthController(app),
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)
</pre>
</div>
<p> </p>
改成：<br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> config = DEFAULT_CONFIG.copy()
config.update({
'mount_path': '/dav',
'provider_mapping': {
'/': ScriptProvider(app)
},
# 更新认证配置
"http_authenticator": {
"domain_controller": NeedAuthController,# 移动到 http_authenticator 下
"accept_basic": True,
"accept_digest": False,
"default_to_digest": False,
},
'verbose': 1 if app.debug else 0,
'dir_browser': {'davmount': False,
'enable': True,
'msmount': False,
'response_trailer': ''},
})
dav_app = WsgiDAVApp(config)
</pre>
</div>
<p> </p>
错误详情：<br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br> scheduler starting...<br>Error: Could not create web server listening on port 25555<br> fetcher starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br> in 5m: new:0,success:0,retry:0,failed:0<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> webui exiting...<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 384, in webui<br> app.run(host=host, port=port)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 59, in run<br> from .webdav import dav_app<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/webdav.py", line 220, in <module><br> dav_app = WsgiDAVApp(config)<br> ^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/wsgidav/wsgidav_app.py", line 155, in __init__<br> _check_config(config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/wsgidav/wsgidav_app.py", line 129, in _check_config<br> raise ValueError("Invalid configuration:\n - " + "\n - ".join(errors))<br>ValueError: Invalid configuration:<br> - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.<br><br><br>9. 再次尝试运行 pyspider 命令，出现错误”TypeError: NeedAuthController.__init__() takes 2 positional arguments but 3 were given“，原因是WsgiDAV 的认证控制器接口发生变化，存在兼容性问题<br><br>解决方案：<br>pyspider/webui/webdav.py的python代码第186行：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class NeedAuthController(object):
def __init__(self, app):
self.app = app
def getDomainRealm(self, inputRelativeURL, environ):
return 'need auth'
def requireAuthentication(self, realmname, environ):
return self.app.config.get('need_auth', False)
def isRealmUser(self, realmname, username, environ):
return username == self.app.config.get('webui_username')
def getRealmUserPassword(self, realmname, username, environ):
return self.app.config.get('webui_password')
def authDomainUser(self, realmname, username, password, environ):
return username == self.app.config.get('webui_username') \
and password == self.app.config.get('webui_password')
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> class NeedAuthController(object):
def __init__(self, app, config=None):
self.app = app
# 处理额外的config参数，使其兼容WsgiDAV的初始化方式
if config is not None:
self.config = config
else:
# 如果config未提供，尝试从app中获取
self.config = app.config.get("http_authenticator", {})
def getDomainRealm(self, inputRelativeURL, environ):
return 'need auth'
def requireAuthentication(self, realmname, environ):
return self.app.config.get('need_auth', False)
def isRealmUser(self, realmname, username, environ):
return username == self.app.config.get('webui_username')
def getRealmUserPassword(self, realmname, username, environ):
return self.app.config.get('webui_password')
def authDomainUser(self, realmname, username, password, environ):
return username == self.app.config.get('webui_username') \
and password == self.app.config.get('webui_password')
# 添加WsgiDAV期望的接口方法，转发到原有方法
def get_domain_realm(self, input_path, environ):
return self.getDomainRealm(input_path, environ)
def basic_auth_user(self, realm, user_name, password, environ):
return self.authDomainUser(realm, user_name, password, environ)
def supports_http_digest_auth(self):
return False# 我们不支持摘要认证
def is_share_anonymous(self, share_path):
"""检查指定的共享路径是否允许匿名访问"""
# 如果不需要认证，则所有共享都允许匿名访问
return not self.app.config.get('need_auth', False)
</pre>
</div>
<p> </p>
错误详情：<br><br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br> fetcher starting...<br>Error: Could not create web server listening on port 25555<br> scheduler starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br> in 5m: new:0,success:0,retry:0,failed:0<br>Error: Could not create web server listening on port 25555<br> webui exiting...<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 384, in webui<br> app.run(host=host, port=port)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 59, in run<br> from .webdav import dav_app<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/webdav.py", line 226, in <module><br> dav_app = WsgiDAVApp(config)<br> ^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/wsgidav/wsgidav_app.py", line 257, in __init__<br> app = mw(self, self.application, config)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/wsgidav/http_authenticator.py", line 140, in __init__<br> dc = make_domain_controller(wsgidav_app, config)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/wsgidav/http_authenticator.py", line 111, in make_domain_controller<br> dc = dc(wsgidav_app, config)<br> ^^^^^^^^^^^^^^^^^^^^^^^<br>TypeError: NeedAuthController.__init__() takes 2 positional arguments but 3 were given<br>Error: Could not create web server listening on port 25555<br><br><br>10. 再次尝试运行 pyspider 命令，出现错误ImportError: cannot import name 'DispatcherMiddleware' from 'werkzeug.wsgi'，原因是 Werkzeug 库版本与 pyspider 不兼容。从 Python 3.12 开始，Werkzeug v2.3.0 及以上版本已经移除了DispatcherMiddleware，将其移至独立的werkzeug.middleware.dispatcher模块中。而 pyspider 仍在使用旧的导入方式。<br><br>解决方案：<br>修改 pyspider/webui/app.py的python代码第64行<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> from werkzeug.wsgi import DispatcherMiddleware
</pre>
</div>
<p> </p>
改成：<br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;">from werkzeug.middleware.dispatcher import DispatcherMiddleware
</pre>
</div>
<p> </p>
错误详情：<br><br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br>Error: Could not create web server listening on port 25555<br> scheduler starting...<br> fetcher starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br> in 5m: new:0,success:0,retry:0,failed:0<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> webui exiting...<br>Traceback (most recent call last):<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/bin/pyspider", line 8, in <module><br> sys.exit(main())<br> ^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 754, in main<br> cli()<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__<br> return self.main(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1363, in main<br> rv = self.invoke(ctx)<br> ^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1808, in invoke<br> rv = super().invoke(ctx)<br> ^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke<br> return ctx.invoke(self.callback, **ctx.params)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 165, in cli<br> ctx.invoke(all)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 497, in all<br> ctx.invoke(webui, **webui_config)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/core.py", line 794, in invoke<br> return callback(*args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func<br> return f(get_current_context(), *args, **kwargs)<br> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/run.py", line 384, in webui<br> app.run(host=host, port=port)<br> File "/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/pyspider/webui/app.py", line 64, in run<br> from werkzeug.wsgi import DispatcherMiddleware<br>ImportError: cannot import name 'DispatcherMiddleware' from 'werkzeug.wsgi' (/home/wanghu/PycharmProjects/PythonProject/getHarmonyVideoDatas/.venv/lib/python3.12/site-packages/werkzeug/wsgi.py)<br>Error: Could not create web server listening on port 25555<br><br>11. 再次尝试运行 pyspider 命令，出现错误Error: Could not create web server listening on port 25555，原因是25555端口被占用，需要释放端口重新执行 pyspider 命令<br>解决方案： 使用 lsof -i 25555查看端口被哪个PID占用，然后用 kill -9 <PID> 释放端口后再重新执行 pyspider 命令。看到 如下打印结果表示成功启动 pyspider，并且启用了5000端口。浏览器可以访问 http://localhost:5000/ 进入PySpider网页控制台爬数据了。<br><br>
<div class="cnblogs_Highlighter">
<pre class="brush:csharp;gutter:false;"> $ lsof -i :25555
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
phantomjs 16615 wanghu 7uIPv4 278700 0t0TCP *:25555 (LISTEN)
(.venv) wanghu@td-1:~/PycharmProjects/PythonProject/getHarmonyVideoDatas$ kill -9 16615
(.venv) wanghu@td-1:~/PycharmProjects/PythonProject/getHarmonyVideoDatas$ pyspider
phantomjs fetcher running on port 25555
result_worker starting...
processor starting...
fetcher starting...
scheduler starting...
scheduler.xmlrpc listening on 127.0.0.1:23333
in 5m: new:0,success:0,retry:0,failed:0
webui running on 0.0.0.0:5000
</pre>
</div>
<p> </p>
<br><br><br>错误详情：<br><br>$ pyspider<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br>Error: Could not create web server listening on port 25555<br> result_worker starting...<br>Error: Could not create web server listening on port 25555<br> processor starting...<br> fetcher starting...<br> scheduler starting...<br> scheduler.xmlrpc listening on 127.0.0.1:23333<br>Error: Could not create web server listening on port 25555<br> in 5m: new:0,success:0,retry:0,failed:0<br>Error: Could not create web server listening on port 25555<br> webui running on 0.0.0.0:5000<br><br><br><br>
来源：https://www.cnblogs.com/bravestarrhu/p/18878872
頁:
[1]