DataGrip 连接 HiveServer2 报错的问题
<p>datagrip 连接 hiveserver2 报错</p>
<p>
<span><strong>一、heap memory</strong></span></p>
<blockquote>
<p>
exception in thread "main" java.lang.outofmemoryerror: java heap space</p>
</blockquote>
<p>
因为 datagrip 会开好几个 session，我这里是 4 个 session</p>
<p>
<img title="DataGrip 连接 HiveServer2 报错的问题" alt="DataGrip 连接 HiveServer2 报错的问题" src="https://zhuji.jb51.net/uploads/img/202305/41e1e3fe8e2b0e1519c864706a4ba198.jpg"></p>
<p>
因此有时候如果执行的语句过大，会造成堆内存来不起回收。导致堆内存溢出，反应在 datagrip 就是执行语句卡住没反应，等很久然后报错。这里有个超时时间，等很久就是超过这个超时时间后报错。</p>
<p>
我们可以适当调大 heap 内存:</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_541204">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain"># 在 hive/bin 目录下的 hive-config.sh，最后一行就是设置 heap 的大小。</code>
</div>
<div class="line number2 index1 alt1">
<code class="java spaces"> </code><code class="java value">69</code> <code class="java plain"># </code><code class="java keyword">default</code> <code class="java plain">to use 256mb</code>
</div>
<div class="line number3 index2 alt2">
<code class="java spaces"> </code><code class="java value">70</code> <code class="java plain">export hadoop_heapsize=${hadoop_heapsize:-</code><code class="java value">2048</code><code class="java plain">}</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong><span>二、too many open files</span></strong></p>
<p>
<img title="DataGrip 连接 HiveServer2 报错的问题" alt="DataGrip 连接 HiveServer2 报错的问题" src="https://zhuji.jb51.net/uploads/img/202305/9f2a4800093b5142bb6ebc81ffe74767.jpg"></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_524879">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">lzq</code><code class="java color1">@hadoop102</code> <code class="java plain">~ ulimit -a</code>
</div>
<div class="line number2 index1 alt1">
<code class="java plain">-t: cpu time (seconds) unlimited</code>
</div>
<div class="line number3 index2 alt2">
<code class="java plain">-f: file size (blocks) unlimited</code>
</div>
<div class="line number4 index3 alt1">
<code class="java plain">-d: data seg size (kbytes) unlimited</code>
</div>
<div class="line number5 index4 alt2">
<code class="java plain">-s: stack size (kbytes) </code><code class="java value">8192</code>
</div>
<div class="line number6 index5 alt1">
<code class="java plain">-c: core file size (blocks) </code><code class="java value">0</code>
</div>
<div class="line number7 index6 alt2">
<code class="java plain">-m: resident set size (kbytes) unlimited</code>
</div>
<div class="line number8 index7 alt1">
<code class="java plain">-u: processes </code><code class="java value">10000</code>
</div>
<div class="line number9 index8 alt2">
<code class="java plain">-n: file descriptors </code><code class="java value">4096</code>
</div>
<div class="line number10 index9 alt1">
<code class="java plain">-l: locked-in-memory size (kbytes) </code><code class="java value">64</code>
</div>
<div class="line number11 index10 alt2">
<code class="java plain">-v: address space (kbytes) unlimited</code>
</div>
<div class="line number12 index11 alt1">
<code class="java plain">-x: file locks unlimited</code>
</div>
<div class="line number13 index12 alt2">
<code class="java plain">-i: pending signals </code><code class="java value">31830</code>
</div>
<div class="line number14 index13 alt1">
<code class="java plain">-q: bytes in posix msg queues </code><code class="java value">819200</code>
</div>
<div class="line number15 index14 alt2">
<code class="java plain">-e: max nice </code><code class="java value">0</code>
</div>
<div class="line number16 index15 alt1">
<code class="java plain">-r: max rt priority </code><code class="java value">0</code>
</div>
<div class="line number17 index16 alt2">
<code class="java plain">-n </code><code class="java value">15</code><code class="java plain">: unlimited</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong><span>修改最大文件连接数量</span></strong></p>
<ul>
<li>
linux 任何外部设备(socket、文件)都可以看做一个文件，而一个文件就可以通过一个文件描述符来读写。</li>
<li>
<code>-n: file descriptors 4096</code> 描述的就是一进程可以打开文件最大的数量</li>
<li>
默认是1024，我们可以通过命令 <code>ulimit -n 4096</code> 把最大打开文件调整到 4096，如果重启会还原成默认值</li>
<li>
永久设置方法</li>
</ul>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_523522">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">vim /etc/security/limits.conf </code>
</div>
<div class="line number2 index1 alt1">
<code class="java plain">在最后加入</code>
</div>
<div class="line number3 index2 alt2">
<code class="java plain">* soft nofile </code><code class="java value">65535</code>
</div>
<div class="line number4 index3 alt1">
<code class="java plain">* hard nofile </code><code class="java value">65535</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
<strong>安装 lsof</strong></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_223497">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">sudo yum install lsof -y</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
查看当前系统打开的文件数量</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_937239">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">lsof | wc -l </code>
</div>
<div class="line number2 index1 alt1">
<code class="java plain">watch </code><code class="java string">"lsof | wc -l"</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_296927">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">lzq</code><code class="java color1">@hadoop102</code> <code class="java plain">~ jps</code>
</div>
<div class="line number2 index1 alt1">
<code class="java value">25696</code> <code class="java plain">jps</code>
</div>
<div class="line number3 index2 alt2">
<code class="java value">1522</code> <code class="java plain">namenode</code>
</div>
<div class="line number4 index3 alt1">
<code class="java value">22627</code> <code class="java plain">runjar</code>
</div>
<div class="line number5 index4 alt2">
<code class="java value">1716</code> <code class="java plain">datanode</code>
</div>
<div class="line number6 index5 alt1">
<code class="java value">3140</code> <code class="java plain">kafka</code>
</div>
<div class="line number7 index6 alt2">
<code class="java value">2309</code> <code class="java plain">nodemanager</code>
</div>
<div class="line number8 index7 alt1">
<code class="java value">2647</code> <code class="java plain">quorumpeermain</code>
</div>
<div class="line number9 index8 alt2">
<code class="java value">22889</code> <code class="java plain">runjar</code>
</div>
<div class="line number10 index9 alt1">
<code class="java value">23322</code> <code class="java plain">runjar</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
查看某一进程的打开文件数量</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_956862">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain"># runjar 就是 hiveserver2 的进程</code>
</div>
<div class="line number2 index1 alt1">
<code class="java plain">lsof -p pid | wc -l </code>
</div>
<div class="line number3 index2 alt2">
<code class="java plain">lsof -p </code><code class="java value">22627</code> <code class="java plain">| wc -l </code>
</div>
<div class="line number4 index3 alt1">
<code class="java plain">lsof -p </code><code class="java value">22889</code> <code class="java plain">| wc -l</code>
</div>
<div class="line number5 index4 alt2">
<code class="java plain">lsof -p </code><code class="java value">23322</code> <code class="java plain">| wc -l</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
</div>
<p>
到此这篇关于datagrip 连接 hiveserver2 报错的文章就介绍到这了,更多相关datagrip 连接 hiveserver2 报错内容请搜索以前的文章或继续浏览下面的相关文章希望大家以后多多支持！</p>
<p>
原文链接：https://blog.csdn.net/u010022158/article/details/107114019</p>
頁:
[1]