[python]requests VS httpx VS aiohttp
<h2 id="前言">前言</h2><p>前段时间想着把一个python服务的接口逐渐改成异步的，其中用到requests的地方就要改成httpx或者aiohttp，有点好奇异步请求相较于同步请求有哪些提升，遂做了点小实验。</p>
<p>首先有个服务A提供接口，这个接口会停顿1秒，模拟数据库操作。服务B去请求服务A的这个接口，并把响应返回给客户端C。服务B提供4个接口，这4个接口分别用requests、httpx同步、httpx异步和aiohttp去请求服务A。</p>
<p>客户端使用wrk做请求测试。</p>
<h2 id="实现服务a">实现服务A</h2>
<p>服务A使用Go编写，用标准库即可完成</p>
<pre><code class="language-go">package main
import (
"net/http"
"time"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("GET /a", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(1 * time.Second)
w.Write([]byte("ok"))
})
if err := http.ListenAndServe("127.0.0.1:8000", mux); err != nil {
panic(err)
}
}
</code></pre>
<p>先用wrk直接请求试试，以此作为基准</p>
<pre><code class="language-bash">$ wrk -t8 -c1000 -d30s http://127.0.0.1:8000/a
Running 30s test @ http://127.0.0.1:8000/a
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.00s 7.62ms 1.17s 97.34%
Req/Sec 194.17 266.07 1.24k 86.82%
29491 requests in 30.10s, 3.32MB read
Requests/sec: 979.85
Transfer/sec: 112.91KB
</code></pre>
<h2 id="服务b-fastapi">服务B: FastAPI</h2>
<p>先用FastAPI做服务B试试</p>
<pre><code class="language-python">from fastapi import FastAPI
import httpx
import aiohttp
import uvicorn
import requests
app = FastAPI()
url = "http://127.0.0.1:8000/a"
@app.get("/sync1")
def sync1():
try:
resp = requests.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print(f"sync request failed, {e}")
else:
return resp.text
@app.get("/sync2")
def sync2():
try:
with httpx.Client() as client:
resp = client.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print(f"sync2 request failed, {e}")
else:
return resp.text
@app.get("/async1")
async def async1():
try:
async with httpx.AsyncClient(timeout=2) as client:
resp = await client.get(url)
resp.raise_for_status()
except Exception as e:
print(f"async1 request failed, {e}")
else:
return resp.text
@app.get("/async2")
async def async2():
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, timeout=2) as resp:
resp.raise_for_status()
content = await resp.text()
except Exception as e:
print(f"async2 request failed, {e}")
else:
return content
if __name__ == "__main__":
uvicorn.run("server1:app", host="127.0.0.1", port=8001, workers=4, access_log=False)
</code></pre>
<p>wrk请求结果。httpx不仅同步请求性能不如requests，没想到连异步请求性能也不如requests。而aiohttp以五倍多第二名的性能冠绝群雄。</p>
<table>
<thead>
<tr>
<th>API</th>
<th>Total request</th>
<th>QPS</th>
<th>timeout</th>
<th>comment</th>
</tr>
</thead>
<tbody>
<tr>
<td>/sync1</td>
<td>4640</td>
<td>154.15</td>
<td>4480</td>
<td>requests 同步请求</td>
</tr>
<tr>
<td>/sync2</td>
<td>3631</td>
<td>120.87</td>
<td>3570</td>
<td>httpx 同步请求</td>
</tr>
<tr>
<td>/async1</td>
<td>4313</td>
<td>143.40</td>
<td>4254</td>
<td>httpx 异步请求</td>
</tr>
<tr>
<td>/async2</td>
<td>25379</td>
<td>843.35</td>
<td>0</td>
<td>aiohttp 异步请求</td>
</tr>
</tbody>
</table>
<p>异步比同步性能还差，着实有点费解，遂找大模型问了下，大模型回复说httpx默认配置参数不高，可以额外指定参数，还需要避免反复创建http client。似乎有点道理，但是同步性能不如开箱即用的requests，异步性能不如开箱即用的aiohttp，我为什么还要折腾httpx呢？</p>
<h2 id="服务b-flask">服务B: Flask</h2>
<p>Flask 2.0 也支持异步接口，但是之前测试性能并不是很好，拉出来一并测试瞧瞧实力。</p>
<p>Flask 版本：3.1.1。因为gunicorn运行异步接口会报错，所以用的flask内置webserver。</p>
<pre><code class="language-python">from flask import Flask
import requests
import httpx
import logging
import aiohttp
app = Flask(__name__)
url = "http://127.0.0.1:8000/a"
@app.get("/sync1")
def sync1():
try:
resp = requests.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print("request failed")
return "request failed"
else:
return resp.text
@app.get("/sync2")
def sync2():
try:
with httpx.Client() as client:
resp = client.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print("request failed")
return "request failed"
else:
return resp.text
@app.get("/async1")
async def async1():
try:
async with httpx.AsyncClient(timeout=2) as client:
resp = await client.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print("request failed")
return "request failed"
else:
return resp.text
@app.get("/async2")
async def async2():
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, timeout=2) as response:
response.raise_for_status()
resp = await response.text()
except Exception as e:
print("request failed")
return "request failed"
else:
return resp
if __name__ == "__main__":
werkzeug_logger = logging.getLogger("werkzeug")
werkzeug_logger.disabled = True
app.run(host="127.0.0.1", port=8001)
</code></pre>
<p>测试结果。看来flask还是跟requests更搭，异步还不如同步。</p>
<table>
<thead>
<tr>
<th>API</th>
<th>Total request</th>
<th>QPS</th>
<th>timeout</th>
<th>comment</th>
</tr>
</thead>
<tbody>
<tr>
<td>/sync1</td>
<td>13279</td>
<td>441.27</td>
<td>248</td>
<td>requests 同步请求</td>
</tr>
<tr>
<td>/sync2</td>
<td>2324</td>
<td>77.26</td>
<td>2323</td>
<td>httpx 同步请求</td>
</tr>
<tr>
<td>/async1</td>
<td>2330</td>
<td>77.46</td>
<td>2330</td>
<td>httpx 异步请求</td>
</tr>
<tr>
<td>/async2</td>
<td>8277</td>
<td>275.03</td>
<td>6887</td>
<td>aiohttp 异步请求</td>
</tr>
</tbody>
</table>
<h2 id="服务b-sanic">服务B: Sanic</h2>
<p>再用Sanic测试一遍</p>
<pre><code class="language-python">from sanic import Sanic
from sanic.response import text
import requests
import httpx
import aiohttp
app = Sanic(__name__)
url = "http://127.0.0.1:8000/a"
@app.get("/sync1")
def sync1(request):
try:
resp = requests.get(url, timeout=2)
resp.raise_for_status()
except Exception as e:
print(f"sync1 request failed, {e}")
else:
return text(resp.text)
@app.get("/sync2")
def sync2(request):
try:
with httpx.Client() as client:
response = client.get(url, timeout=2)
response.raise_for_status()
except Exception as e:
print(f"sync2 request failed, {e}")
else:
return text(response.text)
@app.get("/async1")
async def async1(request):
try:
async with httpx.AsyncClient() as client:
response = await client.get(url, timeout=2)
response.raise_for_status()
except Exception as e:
print(f"async1 request failed, {e}")
else:
return text(response.text)
@app.get("/async2")
async def async2(request):
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, timeout=2) as response:
response.raise_for_status()
content = await response.text()
except Exception as e:
print(f"async2 request failed, {e}")
else:
return text(content)
if __name__ == "__main__":
app.run(host="127.0.0.1", port=8001, debug=False, access_log=False, workers=4)
</code></pre>
<p>可能是我对Sanic了解不多，单就这个测试结果来看，Sanic根本不适合编写同步API。而且使用httpx异步请求的时候有大量报错，wrk结果显示 <code>Non-2xx or 3xx responses: 1244</code></p>
<table>
<thead>
<tr>
<th>API</th>
<th>Total request</th>
<th>QPS</th>
<th>timeout</th>
<th>comment</th>
</tr>
</thead>
<tbody>
<tr>
<td>/sync1</td>
<td>37</td>
<td>1.23</td>
<td>35</td>
<td>requests 同步请求</td>
</tr>
<tr>
<td>/sync2</td>
<td>16</td>
<td>0.53</td>
<td>16</td>
<td>httpx 同步请求</td>
</tr>
<tr>
<td>/async1</td>
<td>5481</td>
<td>182.09</td>
<td>5339</td>
<td>httpx 异步请求</td>
</tr>
<tr>
<td>/async2</td>
<td>28116</td>
<td>934.67</td>
<td>0</td>
<td>aiohttp 异步请求</td>
</tr>
</tbody>
</table>
<h2 id="服务b-go">服务B: Go</h2>
<p>最后再用Go实现下请求</p>
<pre><code class="language-go">func GetA(w http.ResponseWriter, r *http.Request) {
resp, err := http.Get("http://127.0.0.1:8000/a")
if err != nil {
log.Println(err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Println(err)
}
w.Write(body)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("GET /b", GetA)
if err := http.ListenAndServe("127.0.0.1:8001", mux); err != nil {
panic(err)
}
}
</code></pre>
<p>测试结果，和直接请求服务A差别不大。</p>
<pre><code class="language-bash">$ wrk -t8 -c1000 -d30s http://127.0.0.1:8001/b
Running 30s test @ http://127.0.0.1:8001/b
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.02s 85.69ms 1.66s 96.55%
Req/Sec 210.49 175.05 740.00 63.65%
29000 requests in 30.08s, 3.26MB read
Requests/sec: 963.97
Transfer/sec: 111.08KB
</code></pre>
<h2 id="小结">小结</h2>
<p>用python编写同步请求还是老老实实用requests，异步接口应该用aiohttp，httpx的性能只能说能用。</p>
</div>
<div id="MySignature" role="contentinfo">
<p>本文来自博客园，作者：花酒锄作田，转载请注明原文链接：https://www.cnblogs.com/XY-Heruo/p/18924647</p><br><br>
来源：https://www.cnblogs.com/XY-Heruo/p/18924647
頁:
[1]