刘宠君 發表於 2021-3-3 23:03:00

面试题-python 如何读取一个大于 10G 的txt文件？

<h1 id="前言">前言</h1>
<p>用python 读取一个大于10G 的文件，自己电脑只有8G内存，一运行就报内存溢出：MemoryError<br>
python 如何用open函数读取大文件呢？</p>
<h1 id="读取大文件">读取大文件</h1>
<p>首先可以自己先制作一个大于10G的txt文件</p>
<pre><code>a = '''
2021-02-02 21:33:31,678 - Not Found: /http:/123.125.114.144/
2021-02-02 21:33:31,679 - "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2021-02-02 22:14:04,121 - code 400, message Bad request version ('HTTP')
2021-02-02 22:14:04,122 - "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2021-02-02 22:16:21,052 - "GET /api/login HTTP/1.1" 301 0
2021-02-02 22:16:21,123 - "GET /api/login/ HTTP/1.1" 200 3876
2021-02-02 22:16:21,192 - "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2021-02-02 22:16:21,196 - "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2021-02-02 22:16:21,229 - "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2021-02-02 22:16:21,307 - "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2021-02-02 22:16:23,525 - "POST /api/login/ HTTP/1.1" 302 0
2021-02-02 22:16:23,618 - "GET /api/index/ HTTP/1.1" 200 18447
2021-02-02 22:16:23,709 - "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2021-02-02 22:16:23,712 - "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2021-02-02 22:16:23,712 - "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2021-02-02 22:16:23,714 - "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2021-02-02 22:16:26,509 - "GET /api/report_list/1/ HTTP/1.1" 200 14649
2021-02-02 22:16:51,496 - "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-02 22:16:51,721 - "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-02 22:16:59,707 - "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-03 22:16:59,909 - "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-03 22:17:01,306 - "GET /api/edit_case/1/ HTTP/1.1" 200 36504
2021-02-03 22:17:06,265 - "GET /api/add_project/ HTTP/1.1" 200 17737
2021-02-03 22:17:07,825 - "GET /api/project_list/1/ HTTP/1.1" 200 29789
2021-02-03 22:17:13,116 - "GET /api/add_config/ HTTP/1.1" 200 24816
2021-02-03 22:17:19,671 - "GET /api/config_list/1/ HTTP/1.1" 200 19532
'''
while True:
    with open("xxx.log", "a", encoding="utf-8") as fp:
         fp.write(a)
</code></pre>
<p>循环写入到 xxx.log 文件，运行 3-5 分钟，pycharm 打开查看文件大小大于 10G</p>
<p><img src="https://img2020.cnblogs.com/blog/1070438/202103/1070438-20210303230141204-660680391.png" alt="" loading="lazy"></p>
<p>于是我用open函数 直接读取</p>
<pre><code># 作者-上海悠悠 QQ交流群:717225969
# blog地址 https://www.cnblogs.com/yoyoketang/


f = open("xxx.log", 'r')
print(f.read())
f.close()
</code></pre>
<p>抛出内存溢出异常:MemoryError</p>
<pre><code>Traceback (most recent call last):
File "D:/2021kecheng06/demo/txt.py", line 35, in &lt;module&gt;
    print(f.read())
MemoryError
</code></pre>
<p>运行的时候可以看下自己电脑的内存已经占了100%， cpu高达91% ，不挂掉才怪了！</p>
<p><img src="https://img2020.cnblogs.com/blog/1070438/202103/1070438-20210303223255379-484532918.png" alt="" loading="lazy"></p>
<p>这种错误的原因在于，read()方法执行操作是一次性的都读入内存中，显然文件大于内存就会报错。</p>
<h1 id="read-的几种方法">read() 的几种方法</h1>
<p>1.read() 方法可以带参数 n， n 是每次读取的大小长度，也就是可以每次读一部分，这样就不会导致内存溢出</p>
<pre><code>f = open("xxx.log", 'r')
print(f.read(2048))
f.close()
</code></pre>
<p>运行结果</p>
<pre><code>2019-10-24 21:33:31,678 - Not Found: /http:/123.125.114.144/
2019-10-24 21:33:31,679 - "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2019-10-24 22:14:04,121 - code 400, message Bad request version ('HTTP')
2019-10-24 22:14:04,122 - "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2019-10-24 22:16:21,052 - "GET /api/login HTTP/1.1" 301 0
2019-10-24 22:16:21,123 - "GET /api/login/ HTTP/1.1" 200 3876
2019-10-24 22:16:21,192 - "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2019-10-24 22:16:21,196 - "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2019-10-24 22:16:21,229 - "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 - "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 - "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 - "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 - "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 - "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 - "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 - "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2019-10-24 22:16:26,509 [I
</code></pre>
<p>这样就只读取了2048个字符，全部读取的话，循环读就行</p>
<pre><code># 作者-上海悠悠 QQ交流群:717225969
# blog地址 https://www.cnblogs.com/yoyoketang/


f = open("xxx.log", 'r')
while True:
    block = f.read(2048)
    print(block)
    if not block:
      break
f.close()
</code></pre>
<p>2.readline()：每次读取一行，这个方法也不会报错</p>
<pre><code>f = open("xxx.log", 'r')

while True:
    line = f.readline()
    print(line, end="")
    if not line:
      break
f.close()
</code></pre>
<p>3.readlines()：读取全部的行，生成一个list，通过list来对文件进行处理，显然这种方式依然会造成：MemoyError</p>
<h1 id="真正-pythonic-的方法">真正 Pythonic 的方法</h1>
<p>真正 Pythonci 的方法，使用 with 结构打开文件，fp 是一个可迭代对象，可以用 for 遍历读取每行的文件内容</p>
<pre><code>with open("xxx.log", 'r') as fp:
    for line in fp:
      print(line, end="")
</code></pre>
<h1 id="yield-生成器读取大文件">yield 生成器读取大文件</h1>
<p>前面一篇讲yield 生成器的时候提到读取大文件，函数返回一个可迭代对象，用next()方法读取文件内容</p>
<pre><code># 作者-上海悠悠 QQ交流群:717225969
# blog地址 https://www.cnblogs.com/yoyoketang/


def read_file(fpath):
    BLOCK_SIZE = 1024
    with open(fpath, 'rb') as f:
      while True:
            block = f.read(BLOCK_SIZE)
            if block:
                yield block
            else:
                return
if __name__ == '__main__':
    a = read_file("xxx.log")
    print(a)            # generator objec
    print(next(a))      # bytes类型
    print(next(a).decode("utf-8"))   # str
</code></pre>
<p>运行结果</p>
<pre><code>&lt;generator object read_file at 0x00000226B3005258&gt;
b'\r\n2019-10-24 21:33:31,678 - Not Found: /http:/123.125.114.144/\r\n2019-10-24 21:33:31,679 - "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678\r\n2019-10-24 22:14:04,121 - code 400, message Bad request version (\'HTTP\')\r\n2019-10-24 22:14:04,122 - "GET ../../mnt/custom/ProductDefinition HTTP" 400 -\r\n2019-10-24 22:16:21,052 - "GET /api/login HTTP/1.1" 301 0\r\n2019-10-24 22:16:21,123 - "GET /api/login/ HTTP/1.1" 200 3876\r\n2019-10-24 22:16:21,192 - "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801\r\n2019-10-24 22:16:21,196 - "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638\r\n2019-10-24 22:16:21,229 '
- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 - "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 - "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 - "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 - "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 - "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 - "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 - "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2019-10-24 22:16:26,509 [basehtt
</code></pre><br><br>
来源：https://www.cnblogs.com/yoyoketang/p/14477562.html
頁: [1]
查看完整版本: 面试题-python 如何读取一个大于 10G 的txt文件？