kubernetes支持local volume
<p>目录</p><ul>
<li>local volume</li>
<li>创建一个storage class</li>
<li>静态创建PV</li>
<li>使用local volume PV</li>
<li>动态创建PV</li>
</ul>
<h3 id="local-volume">local volume</h3>
<p>kubernetes从1.10版本开始支持local volume（本地卷），workload（不仅是statefulsets类型）可以充分利用本地快速SSD，从而获取比remote volume（如cephfs、RBD）更好的性能。</p>
<p>在local volume出现之前，statefulsets也可以利用本地SSD，方法是配置hostPath，并通过nodeSelector或者nodeAffinity绑定到具体node上。但hostPath的问题是，管理员需要手动管理集群各个node的目录，不太方便。</p>
<p>下面两种类型应用适合使用local volume。</p>
<ul>
<li>数据缓存，应用可以就近访问数据，快速处理。</li>
<li>分布式存储系统，如分布式数据库Cassandra ，分布式文件系统ceph/gluster</li>
</ul>
<p>下面会先以手动方式创建PV、PVC、Pod的方式，介绍如何使用local volume，然后再介绍external storage提供的半自动方式，最后介绍社区的一些发展。</p>
<h3 id="创建一个storage-class">创建一个storage class</h3>
<p>首先需要有一个名为<code class="highlighter-rouge">local-volume</code>的sc。</p>
<div class="language-yaml highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">kind: StorageClass
apiVersion: storage.k8s.io</span>/<span style="color: rgba(0, 0, 0, 1)">v1
metadata:
name: local</span>-<span style="color: rgba(0, 0, 0, 1)">volume
provisioner: kubernetes.io</span>/no-<span style="color: rgba(0, 0, 0, 1)">provisioner
volumeBindingMode: WaitForFirstConsumer</span></pre>
</div>
<p> </p>
</div>
</div>
<p>sc的provisioner是 <code class="highlighter-rouge">kubernetes.io/no-provisioner</code>。</p>
<p><code class="highlighter-rouge">WaitForFirstConsumer</code>表示PV不要立即绑定PVC，而是直到有Pod需要用PVC的时候才绑定。调度器会在调度时综合考虑选择合适的local PV，这样就不会导致跟Pod资源设置，selectors，affinity and anti-affinity策略等产生冲突。很明显：如果PVC先跟local PV绑定了，由于local PV是跟node绑定的，这样selectors，affinity等等就基本没用了，所以更好的做法是先根据调度策略选择node，然后再绑定local PV。</p>
<h3 id="静态创建pv">静态创建PV</h3>
<p>通过kubectl命令，静态创建一个5GiB的PV；该PV使用node ubuntu-1的 <code class="highlighter-rouge">/data/local/vol1</code> 目录；该PV的sc为local-volume。</p>
<div class="language-yaml highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">apiVersion: v1
kind: PersistentVolume
metadata:
name: example</span>-local-<span style="color: rgba(0, 0, 0, 1)">pv
spec:
capacity:
storage: 5Gi
accessModes:
</span>-<span style="color: rgba(0, 0, 0, 1)"> ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local</span>-<span style="color: rgba(0, 0, 0, 1)">volume
local:
path: </span>/data/local/<span style="color: rgba(0, 0, 0, 1)">vol1
nodeAffinity:
required:
nodeSelectorTerms:
</span>-<span style="color: rgba(0, 0, 0, 1)"> matchExpressions:
</span>- key: kubernetes.io/<span style="color: rgba(0, 0, 255, 1)">hostname</span><span style="color: rgba(0, 0, 0, 1)">
operator: In
values:
</span>- ubuntu-<span style="color: rgba(128, 0, 128, 1)">1</span></pre>
</div>
<p> </p>
</div>
</div>
<p>Retain（保留）是指，PV跟PVC释放后，管理员需要手工清理，重新设置该卷。</p>
<p>需要指定PV对应的sc；目录<code class="highlighter-rouge">/data/local/vol1</code>也需要创建。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre>kubectl get pv example-local-<span style="color: rgba(0, 0, 0, 1)">pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
example</span>-local-pv 5Gi RWO Retain Available local-volume 8d</pre>
</div>
<p> </p>
</div>
</div>
<h3 id="使用local-volume-pv">使用local volume PV</h3>
<p>接下来创建一个关联 sc:local-volume的PVC，然后将该PVC挂到nginx容器里。</p>
<div class="language-yaml highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)">apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myclaim
spec:
accessModes:
</span>-<span style="color: rgba(0, 0, 0, 1)"> ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: local</span>-<span style="color: rgba(0, 0, 0, 1)">volume
</span>---<span style="color: rgba(0, 0, 0, 1)">
kind: Pod
apiVersion: v1
metadata:
name: mypod
spec:
containers:
</span>-<span style="color: rgba(0, 0, 0, 1)"> name: myfrontend
image: nginx
volumeMounts:
</span>- mountPath: <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">/usr/share/nginx/html</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">
name: mypd
volumes:
</span>-<span style="color: rgba(0, 0, 0, 1)"> name: mypd
persistentVolumeClaim:
claimName: myclaim</span></pre>
</div>
<p> </p>
</div>
</div>
<p>进入到容器里，会看到挂载的目录，大小其实就是上面创建的PV所在磁盘的size。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre>/dev/sdb 503G235M478G <span style="color: rgba(128, 0, 128, 1)">1</span>% /usr/share/nginx/html</pre>
</div>
<p> </p>
</div>
</div>
<p>在宿主机的<code class="highlighter-rouge">/data/local/vol1</code>目录下创建一个<code class="highlighter-rouge">index.html</code>文件：</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 255, 1)">echo</span> <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">hello world</span><span style="color: rgba(128, 0, 0, 1)">"</span> > /data/local/vol1/index.html</pre>
</div>
<p> </p>
</div>
</div>
<p>然后再去curl容器的IP地址，就可以得到刚写入的字符串了。</p>
<p>删除Pod/PVC，之后PV状态改为Released，该PV不会再被绑定PVC了。</p>
<h3 id="动态创建pv">动态创建PV</h3>
<p>手工管理local PV显然是很费劲的，社区提供了external storage可以动态的创建PV（实际仍然不够自动化）。</p>
<p>local volume provisioner的官方编排在<code class="highlighter-rouge">local-volume/provisioner/deployment/kubernetes/example/default_example_provisioner_generated.yaml</code>目录里，不过官方文档一会fast-disk，一会local-storage，有点混乱。我这里统一都用<code class="highlighter-rouge">local-volume</code>。</p>
<div class="language-yaml highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: v1
kind: ConfigMap
metadata:
name: local</span>-provisioner-<span style="color: rgba(0, 0, 0, 1)">config
namespace: default
data:
storageClassMap: </span>|<span style="color: rgba(0, 0, 0, 1)">
local</span>-<span style="color: rgba(0, 0, 0, 1)">volume:
hostDir: </span>/data/<span style="color: rgba(0, 0, 0, 1)">local
mountDir:</span>/data/<span style="color: rgba(0, 0, 0, 1)">local
blockCleanerCommand:
</span>- <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">/scripts/shred.sh</span><span style="color: rgba(128, 0, 0, 1)">"</span>
- <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">2</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">
volumeMode: Filesystem
fsType: ext4
</span>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: extensions</span>/<span style="color: rgba(0, 0, 0, 1)">v1beta1
kind: DaemonSet
metadata:
name: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">provisioner
namespace: default
labels:
app: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">provisioner
spec:
selector:
matchLabels:
app: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">provisioner
template:
metadata:
labels:
app: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">provisioner
spec:
serviceAccountName: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">admin
containers:
</span>- image: <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">silenceshell/local-volume-provisioner:v2.1.0</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">
imagePullPolicy: </span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">Always</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">
name: provisioner
securityContext:
privileged: </span><span style="color: rgba(0, 0, 255, 1)">true</span>
<span style="color: rgba(0, 0, 255, 1)">env</span><span style="color: rgba(0, 0, 0, 1)">:
</span>-<span style="color: rgba(0, 0, 0, 1)"> name: MY_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
</span>- mountPath: /etc/provisioner/<span style="color: rgba(0, 0, 0, 1)">config
name: provisioner</span>-<span style="color: rgba(0, 0, 0, 1)">config
readOnly: </span><span style="color: rgba(0, 0, 255, 1)">true</span>
- mountPath:/data/<span style="color: rgba(0, 0, 0, 1)">local
name: local
mountPropagation: </span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">HostToContainer</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">
volumes:
</span>- name: provisioner-<span style="color: rgba(0, 0, 0, 1)">config
configMap:
name: local</span>-provisioner-<span style="color: rgba(0, 0, 0, 1)">config
</span>-<span style="color: rgba(0, 0, 0, 1)"> name: local
hostPath:
path: </span>/data/<span style="color: rgba(0, 0, 0, 1)">local
</span>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: v1
kind: ServiceAccount
metadata:
name: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">admin
namespace: default
</span>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: rbac.authorization.k8s.io</span>/<span style="color: rgba(0, 0, 0, 1)">v1
kind: ClusterRoleBinding
metadata:
name: local</span>-volume-provisioner-pv-<span style="color: rgba(0, 0, 0, 1)">binding
namespace: default
subjects:
</span>-<span style="color: rgba(0, 0, 0, 1)"> kind: ServiceAccount
name: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">admin
namespace: default
roleRef:
kind: ClusterRole
name: system:persistent</span>-volume-<span style="color: rgba(0, 0, 0, 1)">provisioner
apiGroup: rbac.authorization.k8s.io
</span>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: rbac.authorization.k8s.io</span>/<span style="color: rgba(0, 0, 0, 1)">v1
kind: ClusterRole
metadata:
name: local</span>-volume-provisioner-node-<span style="color: rgba(0, 0, 0, 1)">clusterrole
namespace: default
rules:
</span>- apiGroups: [<span style="color: rgba(128, 0, 0, 1)">""</span><span style="color: rgba(0, 0, 0, 1)">]
resources: [</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">nodes</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">]
verbs: [</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)">get</span><span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(0, 0, 0, 1)">]
</span>---<span style="color: rgba(0, 0, 0, 1)">
apiVersion: rbac.authorization.k8s.io</span>/<span style="color: rgba(0, 0, 0, 1)">v1
kind: ClusterRoleBinding
metadata:
name: local</span>-volume-provisioner-node-<span style="color: rgba(0, 0, 0, 1)">binding
namespace: default
subjects:
</span>-<span style="color: rgba(0, 0, 0, 1)"> kind: ServiceAccount
name: local</span>-volume-<span style="color: rgba(0, 0, 0, 1)">admin
namespace: default
roleRef:
kind: ClusterRole
name: local</span>-volume-provisioner-node-<span style="color: rgba(0, 0, 0, 1)">clusterrole
apiGroup: rbac.authorization.k8s.io</span></pre>
</div>
<p> </p>
</div>
</div>
<p>kubectl创建后，由于是daemonset类型，每个节点上都会启动一个provisioner。该provisioner会监视 “discovery directory”，即上面配置的<code class="highlighter-rouge">/data/local</code>。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre>$ kubectl get pods -o wide|<span style="color: rgba(0, 0, 255, 1)">grep</span> local-<span style="color: rgba(0, 0, 0, 1)">volume
local</span>-volume-provisioner-rrsjp <span style="color: rgba(128, 0, 128, 1)">1</span>/<span style="color: rgba(128, 0, 128, 1)">1</span> Running <span style="color: rgba(128, 0, 128, 1)">0</span> 5m <span style="color: rgba(128, 0, 128, 1)">10.244</span>.<span style="color: rgba(128, 0, 128, 1)">1.141</span> ubuntu-<span style="color: rgba(128, 0, 128, 1)">2</span> <none><span style="color: rgba(0, 0, 0, 1)">
local</span>-volume-provisioner-v87b7 <span style="color: rgba(128, 0, 128, 1)">1</span>/<span style="color: rgba(128, 0, 128, 1)">1</span> Running <span style="color: rgba(128, 0, 128, 1)">0</span> 5m <span style="color: rgba(128, 0, 128, 1)">10.244</span>.<span style="color: rgba(128, 0, 128, 1)">2.69</span> ubuntu-<span style="color: rgba(128, 0, 128, 1)">3</span> <none><span style="color: rgba(0, 0, 0, 1)">
local</span>-volume-provisioner-x65k9 <span style="color: rgba(128, 0, 128, 1)">1</span>/<span style="color: rgba(128, 0, 128, 1)">1</span> Running <span style="color: rgba(128, 0, 128, 1)">0</span> 5m <span style="color: rgba(128, 0, 128, 1)">10.244</span>.<span style="color: rgba(128, 0, 128, 1)">0.174</span> ubuntu-<span style="color: rgba(128, 0, 128, 1)">1</span> <none></pre>
</div>
<p> </p>
</div>
</div>
<p>前面<code class="highlighter-rouge">mypod/myclaim</code>已经删除了，我们重新创建一个，此时pvc myclaim是Pending状态，provisoner并没有自动供给存储。为什么呢？</p>
<p>原来<code class="highlighter-rouge">external-storage</code>的逻辑是这样的：其Provisioner本身其并不提供local volume，但它在各个节点上的provisioner会去动态的“发现”挂载点（discovery directory），当某node的provisioner在<code class="highlighter-rouge">/data/local/</code>目录下发现有挂载点时，会创建PV，该PV的<code class="highlighter-rouge">local.path</code>就是挂载点，并设置nodeAffinity为该node。</p>
<p>那么如何获得挂载点呢？</p>
<p>直接去创建目录是行不通的，因为provsioner希望PV是隔离的，例如capacity，io等。试着在ubuntu-2上的<code class="highlighter-rouge">/data/local/</code>下创建一个<code class="highlighter-rouge">xxx</code>目录，会得到这样的告警。</p>
<div class="highlighter-rouge">
<div class="highlight">
<pre class="highlight"><code>discovery.go:201] Path "/data/local/xxx" is not an actual mountpoint
</code></pre>
</div>
</div>
<p>目录不是挂载点，不能用。</p>
<p>该目录必须是真材实料的mount才行。一个办法是加硬盘、格式化、mount，比较麻烦，实际可以通过本地文件格式化(loopfs)后挂载来“欺骗”provisioner，让它以为是一个mount的盘，从而自动创建PV，并与PVC绑定。</p>
<p>如下。</p>
<p>将下面的代码保存为文件 <code class="highlighter-rouge">loopmount</code>，加执行权限并拷贝到<code class="highlighter-rouge">/bin</code>目录下，就可以使用该命令来创建挂载点了。</p>
<div class="language-bash highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre>#!/bin/<span style="color: rgba(0, 0, 0, 1)">bash
# Usage: </span><span style="color: rgba(0, 0, 255, 1)">sudo</span> loopmount <span style="color: rgba(0, 0, 255, 1)">file</span> size <span style="color: rgba(0, 0, 255, 1)">mount</span>-<span style="color: rgba(0, 0, 0, 1)">point
</span><span style="color: rgba(0, 0, 255, 1)">touch</span> $<span style="color: rgba(128, 0, 128, 1)">1</span><span style="color: rgba(0, 0, 0, 1)">
truncate </span>-s $<span style="color: rgba(128, 0, 128, 1)">2</span> $<span style="color: rgba(128, 0, 128, 1)">1</span><span style="color: rgba(0, 0, 0, 1)">
mke2fs </span>-t ext4 -F $<span style="color: rgba(128, 0, 128, 1)">1</span> <span style="color: rgba(128, 0, 128, 1)">1</span>> /dev/<span style="color: rgba(0, 0, 255, 1)">null</span> <span style="color: rgba(128, 0, 128, 1)">2</span>> /dev/<span style="color: rgba(0, 0, 255, 1)">null</span>
<span style="color: rgba(0, 0, 255, 1)">if</span> [[ ! -d $<span style="color: rgba(128, 0, 128, 1)">3</span> ]]; <span style="color: rgba(0, 0, 255, 1)">then</span>
<span style="color: rgba(0, 0, 255, 1)">echo</span> $<span style="color: rgba(128, 0, 128, 1)">3</span> <span style="color: rgba(128, 0, 0, 1)">"</span><span style="color: rgba(128, 0, 0, 1)"> not exist, creating...</span><span style="color: rgba(128, 0, 0, 1)">"</span>
<span style="color: rgba(0, 0, 255, 1)">mkdir</span> $<span style="color: rgba(128, 0, 128, 1)">3</span>
<span style="color: rgba(0, 0, 255, 1)">fi</span>
<span style="color: rgba(0, 0, 255, 1)">mount</span> $<span style="color: rgba(128, 0, 128, 1)">1</span> $<span style="color: rgba(128, 0, 128, 1)">3</span>
<span style="color: rgba(0, 0, 255, 1)">df</span> -h |<span style="color: rgba(0, 0, 255, 1)">grep</span> $<span style="color: rgba(128, 0, 128, 1)">3</span></pre>
</div>
<p> </p>
</div>
</div>
<p>使用脚本创建一个6G的文件，并挂载到<code class="highlighter-rouge">/data/local</code>下。之所以要6G，是因为前面PVC需要的是5GB，而格式化后剩余空间会小一点，所以设置文件更大一些，后面才好绑定PVC。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre># loopmount xxx 6G /data/local/<span style="color: rgba(0, 0, 0, 1)">xxx
</span>/data/local/<span style="color: rgba(0, 0, 0, 1)">xxxnot exist, creating...
</span>/dev/loop0 <span style="color: rgba(128, 0, 128, 1)">5</span>.9G 24M<span style="color: rgba(128, 0, 128, 1)">5</span>.6G <span style="color: rgba(128, 0, 128, 1)">1</span>% /data/local/x1</pre>
</div>
<p> </p>
</div>
</div>
<p>查看PV，可见Provisioner自动创建了PV，而kubernetes会将该PV供给给前面的PVC myclam，mypod也run起来了。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 0, 1)"># kubectl get pv
NAME CAPACITYACCESS MODES RECLAIM POLICY STATUSCLAIM STORAGECLASS REASON AGE
local</span>-pv-600377f7 5983Mi RWO Delete Bound default/myclaimlocal-volume 1s</pre>
</div>
<p> </p>
</div>
</div>
<p>可见，目前版本的local volume还无法做到像cephfs/RBD一样的全自动化，仍然需要管理员干涉，显然这不是一个好的实现。</p>
<p>社区有人提交了基于LVM做local volume动态供给的Proposal，不过进展很缓慢。作者是huawei的员工，应该huawei已经实现了。</p>
<blockquote>
<p>除了基于LVM，也可以基于 <code class="highlighter-rouge">ext4 project quota</code> 来实现LV的动态供给。</p>
</blockquote>
<p>除了使用磁盘，还可以考虑使用内存文件系统，从而获取更高的io性能，只是容量就没那么理想了。一些特殊的应用可以考虑。</p>
<div class="highlighter-rouge">
<div class="highlight">
<div class="cnblogs_code">
<pre><span style="color: rgba(0, 0, 255, 1)">mount</span> -t tmpfs -o size=1G,nr_inodes=10k,mode=<span style="color: rgba(128, 0, 128, 1)">700</span> tmpfs /data/local/tmpfs</pre>
</div>
<p> </p>
</div>
</div>
<p>总的来说，local volume本地卷目前不支持动态供给，还无法真正推广使用，但可以用来解决一些特定问题。</p>
<p>Ref:</p>
<ul>
<li>Local Persistent Volumes for Kubernetes Goes Beta</li>
<li>Create a loopfs / loopback device with ext4</li>
</ul>
<h3> </h3>
<p>参考文档：</p>
<p>https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/</p>
<p>https://ieevee.com/tech/2019/01/17/local-volume.html</p><br><br>
来源：https://www.cnblogs.com/davygeek/p/10944208.html
頁:
[1]