8.5 Ingress实现基于域名的多虚拟主机、URL转发、及多域名https实现等案例
<h2 id="1什么是ingress">1.什么是Ingress</h2><p>Ingress 公开了从k8s集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。<br>
可以将 Ingress 配置为服务提供外部可访问的 URL、负载均衡流量、终止 SSL/TLS，以及提供基于名称的虚拟主机等能力<br>
<img src="https://img2020.cnblogs.com/blog/2510032/202111/2510032-20211111104651922-1892901042.png"></p>
<p><strong>ingress具体的工作原理如下:</strong><br>
step1：ingress contronler通过与k8s的api进行交互，动态的去感知k8s集群中ingress服务规则的变化，然后读取它，并按照定义的ingress规则，转发到k8s集群中对应的service。<br>
step2：而这个ingress规则写明了哪个域名对应k8s集群中的哪个service，然后再根据ingress-controller中的nginx配置模板，生成一段对应的nginx配置。<br>
step3：然后再把该配置动态的写到ingress-controller的pod里，该ingress-controller的pod里面运行着一个nginx服务，控制器会把生成的nginx配置写入到nginx的配置文件中，然后reload一下，使其配置生效,以此来达到域名分配置及动态更新的效果。</p>
<p><strong>ingress可以解决的问题</strong><br>
1）动态配置服务<br>
如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作。<br>
2）减少不必要的端口暴露<br>
配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式。<br>
原文链接：https://blog.csdn.net/weixin_44729138/article/details/105978555</p>
<h2 id="2ingress控制器部署">2.ingress控制器部署</h2>
<p>为了让 Ingress 资源工作，集群必须有一个正在运行的 Ingress 控制器<br>
与作为 kube-controller-manager 可执行文件的一部分运行的其他类型的控制器不同， Ingress 控制器不是随集群自动启动的。 基于此页面，你可选择最适合你的集群的 ingress 控制器实现。<br>
Kubernetes 作为一个项目，目前支持和维护 AWS， GCE 和 nginx Ingress 控制器。</p>
<p>但是基于nginx服务的ingress controller根据不同的开发公司，又分为k8s社区的ingres-nginx和nginx公司的nginx-ingress。<br>
在此根据github上的活跃度和关注人数，我们选择的是k8s社区的ingres-nginx进行部署测试</p>
<h3 id="21-准备ingress控制器部署使用的docker镜像v100的版本">2.1 准备ingress控制器部署使用的docker镜像(v1.0.0的版本)</h3>
<pre><code>docker pull willdockerhub/ingress-nginx-controller:v1.0.0
docker tag 192.168.1.110/base/ingress-nginx-controller:v1.0.0
docker push 192.168.1.110/base/ingress-nginx-controller:v1.0.0
docker pull jettech/kube-webhook-certgen:v1.0.0
docker tag 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
docker push 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
</code></pre>
<h3 id="22-修改ingress-controller-deployyaml">2.2 修改ingress-controller-deploy.yaml</h3>
<p><strong>1.官方文件中创建的controller service 的type是LoadBalancer，改成NodePort，并添加相应的端口<br>
2.将文件中的镜像替换为本地harbor中的镜像</strong></p>
<pre><code># Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
#type: LoadBalancer
type: NodePort
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 40080
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 40443
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
</code></pre>
<details>
<summary>点击查看完整的deploy.yaml文件</summary>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ''
resources:
- nodes
verbs:
- get
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ''
resources:
- namespaces
verbs:
- get
- apiGroups:
- ''
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- ingress-controller-leader
verbs:
- get
- update
- apiGroups:
- ''
resources:
- configmaps
verbs:
- create
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
type: ClusterIP
ports:
- name: https-webhook
port: 443
targetPort: webhook
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
#type: LoadBalancer
type: NodePort
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
nodePort: 40080
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
nodePort: 40443
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
containers:
- name: controller
image: 192.168.1.110/base/ingress-nginx-controller:v1.0.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
runAsUser: 101
allowPrivilegeEscalation: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
volumeMounts:
- name: webhook-cert
mountPath: /usr/local/certificates/
readOnly: true
resources:
requests:
cpu: 100m
memory: 90Mi
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/controller-ingressclass.yaml
# We don't support namespaced ingressClass yet
# So a ClusterRole and a ClusterRoleBinding is required
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: nginx
namespace: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
name: ingress-nginx-admission
webhooks:
- name: validate.nginx.ingress.kubernetes.io
matchPolicy: Equivalent
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
service:
namespace: ingress-nginx
name: ingress-nginx-controller-admission
path: /networking/v1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx-admission
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- get
- create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ingress-nginx-admission
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-create
namespace: ingress-nginx
annotations:
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-create
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: create
image: 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
imagePullPolicy: IfNotPresent
args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: ingress-nginx-admission-patch
namespace: ingress-nginx
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
template:
metadata:
name: ingress-nginx-admission-patch
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: admission-webhook
spec:
containers:
- name: patch
image: 192.168.1.110/base/ingress-nginx_kube-webhook-certgen:v1.0.0
imagePullPolicy: IfNotPresent
args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 2000
</code></pre>
</details>
<h3 id="23-创建ingress-controller">2.3 创建ingress-controller</h3>
<pre><code>kubectl apply -fingress-controller-deploy-v1.0.0.yml
</code></pre>
<h3 id="24-查看相关资源是否创建成功">2.4 查看相关资源是否创建成功</h3>
<pre><code>root@k8-deploy:~/k8s-yaml/ingress# kubectl get svc,pod -n ingress-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/ingress-nginx-controller NodePort 10.0.2.50 <none> 80:40080/TCP,443:40443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission ClusterIP 10.0.135.64 <none> 443/TCP 21h app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/ingress-nginx-admission-create-mr6qt 0/1 Completed 0 21h 10.100.112.44 192.168.2.19 <none> <none>
pod/ingress-nginx-admission-patch-sxrl6 0/1 Completed 1 21h 10.100.112.45 192.168.2.19 <none> <none>
pod/ingress-nginx-controller-556976fb4d-dljbp 1/1 Running 0 21h 10.100.224.73 192.168.2.18 <none> <none>
# 查看ingress-nginx-controller 版本
# kubectl exec -it ingress-nginx-controller-556976fb4d-dljbp -n ingress-nginx -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.0.0
Build: 041eb167c7bfccb1d1653f194924b0c5fd885e10
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.20.1
-------------------------------------------------------------------------------
# 查看node节点上是否监听ingress-controller的40080,40443 NodePort端口
root@k8-deploy:~/k8s-yaml/ingress# for i in 17 18 19;do echo node:192.168.1.$i;ssh 192.168.2.$i -C "netstat -ntlp |grep -E '40080|40443'";echo;done
node:192.168.1.17
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 3452053/kube-proxy
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 3452053/kube-proxy
node:192.168.1.18
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 1364498/kube-proxy
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 1364498/kube-proxy
node:192.168.1.19
tcp 0 0 0.0.0.0:40443 0.0.0.0:* LISTEN 251137/kube-proxy
tcp 0 0 0.0.0.0:40080 0.0.0.0:* LISTEN 251137/kube-proxy
</code></pre>
<h3 id="25-配置haproxy边缘节点-代理ingress">2.5 配置haproxy（边缘节点） 代理ingress</h3>
<pre><code># cat /etc/haproxy/haproxy.cfg
...
frontend ingress
bind 192.168.2.110:80
mode tcp
log global
default_backend ingress-servers
frontend ingress-https
bind 192.168.2.110:443
mode tcp
log global
default_backend ingress-servers-https
backend ingress-servers
balance source
server websrv1 192.168.2.17:40080 check maxconn 2000
server websrv2 192.168.2.18:40080 check maxconn 2000
server websrv3 192.168.2.19:40080 check maxconn 2000
backend ingress-servers-https
balance source
server websrv1 192.168.2.17:40443 check maxconn 2000
server websrv2 192.168.2.18:40443 check maxconn 2000
server websrv3 192.168.2.19:40443 check maxconn 2000
</code></pre>
<h2 id="3-基于域名的多虚拟主机的测试">3 基于域名的多虚拟主机的测试</h2>
<p>下图显示了客户端是如果通过 Ingress Controller 连接到其中一个 Pod 的流程，客户端首先对 域名 执行 DNS 解析，得到 Ingress Controller 所在节点的 IP，然后客户端向 Ingress Controller 发送 HTTP 请求，然后根据 Ingress 对象里面的描述匹配域名，找到对应的 Service 对象，并获取关联的 Endpoints 列表，将客户端的请求转发给其中一个 Pod。<br>
<img src="https://img2020.cnblogs.com/blog/2510032/202111/2510032-20211112155239461-1547969547.png"></p>
<h3 id="31-准备2个域名和镜像">3.1 准备2个域名和镜像</h3>
<p>为方便测试，域名放到hosts文件中</p>
<pre><code>192.168.2.110 app1.test.com app2.test.com
</code></pre>
<p>测试的预期效果为访问不通的域名，会返回不同的内容。</p>
<p>2个镜像制作如下:<br>
基础镜像使用之前已经做好的nginx镜像(应代码发布目录改为了/usr/share/nginx/html/)</p>
<pre><code>nginx-app1# ll
-rw-r--r-- 1 root root 83 11月 11 13:54 Dockerfile
-rw-r--r-- 1 root root 23 11月 11 13:54 index.html
# cat Dockerfile
FROM 192.168.1.110/web/nginx-set-config:v1
COPY index.html /usr/share/nginx/html/
# cat index.html
nginx test html : app1
# docker build -t 192.168.1.110/web/nginx-set-config-app1:v1 .
# docker push 192.168.1.110/web/nginx-set-config-app1:v1
## ==========================
nginx-app2# ll
-rw-r--r-- 1 root root 83 11月 11 13:56 Dockerfile
-rw-r--r-- 1 root root 23 11月 11 13:56 index.html
# cat Dockerfile
FROM 192.168.1.110/web/nginx-set-config:v1
COPY index.html /usr/share/nginx/html/
# cat index.html
nginx test html : app2
# docker build -t 192.168.1.110/web/nginx-set-config-app2:v1 .
# docker push 192.168.1.110/web/nginx-set-config-app2:v1
</code></pre>
<h3 id="32-编写2个service的yaml文件">3.2 编写2个service的yaml文件</h3>
<pre><code>root@k8-deploy:~/k8s-yaml/ingress# cat nginx-app1-svc.yml
apiVersion: v1
kind: Service
metadata:
name: nginx-app1-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 40003
protocol: TCP
selector:
app: nginx-app1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-app1-deployment
namespace: yan-test
labels:
app: nginx-app1
spec:
replicas: 1
selector:
matchLabels:
app: nginx-app1
template:
metadata:
labels:
app: nginx-app1
spec:
containers:
- name: nginx-app1-ct
image: 192.168.1.110/web/nginx-set-config-app1:v1
ports:
- containerPort: 80
## =========================
root@k8-deploy:~/k8s-yaml/ingress# cat nginx-app2-svc.yml
apiVersion: v1
kind: Service
metadata:
name: nginx-app2-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 40004
protocol: TCP
selector:
app: nginx-app2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-app2-deployment
namespace: yan-test
labels:
app: nginx-app2
spec:
replicas: 1
selector:
matchLabels:
app: nginx-app2
template:
metadata:
labels:
app: nginx-app2
spec:
containers:
- name: nginx-app2-ct
image: 192.168.1.110/web/nginx-set-config-app2:v1
ports:
- containerPort: 80
</code></pre>
<h3 id="33-创建service">3.3 创建service</h3>
<pre><code># kubectl apply -f nginx-app1-svc.yml
# kubectl apply -f nginx-app2-svc.yml
# kubectl get svc -n yan-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-app1-svc NodePort 10.0.7.177 <none> 80:40003/TCP 23h
nginx-app2-svc NodePort 10.0.63.21 <none> 80:40004/TCP 23h
</code></pre>
<h3 id="34-编写2个域名对应的2个service的ingress配置文件">3.4 编写2个域名对应的2个service的ingress配置文件</h3>
<pre><code># cat ingress-muti-host-nginx.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-web
namespace: yan-test
annotations:
kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客户端上传文件，最大大小，默认为20m
#nginx.ingress.kubernetes.io/rewrite-target: / ##URL重写
#nginx.ingress.kubernetes.io/app-root: /index.html
spec:
rules:
- host: app1.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app1-svc
port:
number: 80
- host: app2.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app2-svc
port:
number: 80
</code></pre>
<h3 id="35-创建ingress规则">3.5 创建ingress规则</h3>
<pre><code># kubectl apply -f ingress-muti-host-nginx.yml
# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-web <none> app1.test.com,app2.test.com 10.0.2.50 80 21h
</code></pre>
<h3 id="36-使用域名访问测试">3.6 使用域名访问测试</h3>
<pre><code>root@k8-deploy:~# curl app1.test.com
nginx test html : app1
root@k8-deploy:~# curl app2.test.com
nginx test html : app2
</code></pre>
<h2 id="4-单域名多url转发">4 单域名多URL转发</h2>
<p>预期效果为访问tomcat.test.com 返回tomato默认页面内容，访问tomcat.test.com/app1/index.html返回app1项目的内容</p>
<h3 id="41-构建tomcat镜像">4.1 构建tomcat镜像</h3>
<pre><code>tomcat-app1# ll
总用量 20
drwxr-xr-x 3 root root 4096 11月 12 13:42 ./
drwxr-xr-x 6 root root 4096 11月 11 13:56 ../
drwxr-xr-x 2 root root 4096 11月 10 16:24 app1/
-rw-r--r-- 1 root root164 11月 10 16:24 app1.tgz
-rw-r--r-- 1 root root 96 11月 10 16:23 Dockerfile
root@k8-deploy:~/k8s-yaml/ingress/images/tomcat-app1# cat Dockerfile
FROM 192.168.1.110/web/alpine-jdk-8u192-tomcat-8.5.70:v20211020-1014
ADD app1.tgz /opt/webapps
root@k8-deploy:~/k8s-yaml/ingress/images/tomcat-app1# cat app1/index.html
tomcat app1 html
# docker build -t 192.168.1.110/web/tomcat-app1:v1 .
# docker push 192.168.1.110/web/tomcat-app1:v1
</code></pre>
<h3 id="42-编写svc-ymal并创建tomcat-svc">4.2 编写svc ymal并创建tomcat svc</h3>
<pre><code># cat tomcat-app1-svc.yml
apiVersion: v1
kind: Service
metadata:
name: tomcat-app1-svc
namespace: yan-test
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 40001
protocol: TCP
selector:
app: tomcat-app1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-app1-deployment
namespace: yan-test
labels:
app: tomcat-app1
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-app1
template:
metadata:
labels:
app: tomcat-app1
spec:
containers:
- name: tomcat-app1-ct
image: 192.168.1.110/web/tomcat-app1:v1
ports:
- containerPort: 8080
</code></pre>
<h3 id="43-创建tomcat-svc并查看">4.3 创建tomcat svc并查看</h3>
<pre><code># kubectl apply -f tomcat-app1-svc.yml
# kubectl get svc,pod -n yan-test -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/tomcat-app1-svc NodePort 10.0.144.64 <none> 80:40001/TCP 24h app=tomcat-app1
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/tomcat-app1-deployment-754855ff99-jfqwz 1/1 Running 0 24h 10.100.224.72 192.168.2.18 <none> <none>
</code></pre>
<h3 id="44-编写ingress-yaml">4.4 编写ingress yaml</h3>
<pre><code># cat ingress-muti-path-tomcat.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tomcat-ingress
namespace: yan-test
annotations:
kubernetes.io/ingress.class: "nginx" ##指定Ingress Controller的类型
nginx.ingress.kubernetes.io/use-regex: "true" ##指定后面rules定义的path可以使用正则表达式
nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" ##连接超时时间,默认为5s
nginx.ingress.kubernetes.io/proxy-send-timeout: "600" ##后端服务器回转数据超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-read-timeout: "600" ##后端服务器响应超时时间,默认为60s
nginx.ingress.kubernetes.io/proxy-body-size: "10m" ##客户端上传文件，最大大小，默认为20m
#nginx.ingress.kubernetes.io/rewrite-target: / ##URL重写
#nginx.ingress.kubernetes.io/app-root: /index.html
spec:
rules:
- host: tomcat.test.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tomcat-app1-svc
port:
number: 80
- path: /app1
pathType: Prefix
backend:
service:
name: tomcat-app1-svc
port:
number: 80
</code></pre>
<h3 id="45-创建tomcat-ingress规则">4.5 创建tomcat ingress规则</h3>
<pre><code># kubectl apply -f ingress-muti-path-tomcat.yml
# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
tomcat-ingress <none> tomcat.test.com 10.0.2.50 80 26m
</code></pre>
<h3 id="46-通过域名和不同的url进行验证">4.6 通过域名和不同的url进行验证</h3>
<pre><code># curl tomcat.test.com
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Apache Tomcat/8.5.70</title>
<link href="favicon.ico" rel="icon" type="image/x-icon" />
<link href="tomcat.css" rel="stylesheet" type="text/css" />
</head>
...
# curl tomcat.test.com/app1/index.html
tomcat app1 html
</code></pre>
<h2 id="5-ingress-https配置测试">5 ingress https配置测试</h2>
<h3 id="51-准备ssl证书">5.1 准备ssl证书</h3>
<p>申请证书的域名需要完成ICP备案，使用阿里云免费的ssl了证书</p>
<pre><code>root@k8-deploy:~/k8s-yaml/ingress/cert# ll
-rw-r--r-- 1 root root 1679 11月 16 15:02 6611016_ingress.t.faxuan.net.key
-rw-r--r-- 1 root root 4153 11月 16 15:02 6611016_ingress.t.faxuan.net_nginx.zip
-rw-r--r-- 1 root root 3813 11月 16 15:02 6611016_ingress.t.faxuan.net.pem
</code></pre>
<h3 id="52-将ssl证书创建为secret引入k8s">5.2 将ssl证书创建为secret引入k8s</h3>
<pre><code>root@k8-deploy:~/k8s-yaml/ingress/cert# kubectl create secret tls ingress.t.faxuan.net-secret \
> --key=./6611016_ingress.t.faxuan.net.key \
> --cert=./6611016_ingress.t.faxuan.net.pem \
> -n yan-test
secret/ingress.t.faxuan.net-secret created
root@k8-deploy:~/k8s-yaml/ingress/cert# kubectl get secrets -n yan-test
NAME TYPE DATA AGE
default-token-8fd2j kubernetes.io/service-account-token 3 68m
ingress.t.faxuan.net-secret kubernetes.io/tls 2 5s
</code></pre>
<h3 id="53-编写-ingress-htts规则的yaml">5.3 编写 ingress htts规则的yaml</h3>
<p>后端的server 使用之前已经创建的nginx-app1-svc</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
namespace: yan-test
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
spec:
ingressClassName: nginx
tls:
- hosts:
- ingress.t.faxuan.net
secretName: ingress.t.faxuan.net-secret
rules:
- host: ingress.t.faxuan.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-app1-svc
port:
number: 80
</code></pre>
<h3 id="54-使用yaml文件创建ingress规则">5.4 使用yaml文件创建ingress规则</h3>
<pre><code>kubectl apply -f ingress-https-single.yml
root@k8-deploy:~/k8s-yaml/ingress# kubectl get ingress -n yan-test
NAME CLASS HOSTS ADDRESS PORTS AGE
myapp nginx ingress.t.faxuan.net 10.0.233.49 80, 443 17m
</code></pre>
<p>遇到的错误：</p>
<pre><code>Error from server (InternalError): error when creating "ingress-https-single.yml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": the server rejected our request for an unknown reason
</code></pre>
<p>2种解决方法：<br>
1.使用ingress的v1beta1版本格式的yaml文件<br>
2.删除 ValidatingWebhookConfiguration资源中ingress-nginx-admission<br>
kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission<br>
参考：https://knowledge.broadcom.com/external/article/227955/installation-reported-during-nginx-setup.html</p>
<h3 id="55-haproxy-代理配置">5.5 haproxy 代理配置</h3>
<pre><code>listen ingress-443
bind *:443
mode tcp
server k8s1 192.168.2.17:40443 check inter 3s fall 3 rise 5
server k8s2 192.168.2.18:40443 check inter 3s fall 3 rise 5
server k8s3 192.168.2.19:40443 check inter 3s fall 3 rise 5
# 重启haproxy
systemctl restart haproxy
</code></pre>
<h3 id="56-验证ingress-https">5.6 验证ingress https</h3>
<p>域名解析，比如：ingress.t.faxuan.net解析到haproxy所在的服务器IP<br>
<img src="https://img2020.cnblogs.com/blog/2510032/202111/2510032-20211116172613976-1527597460.png"></p><br><br>
来源：https://www.cnblogs.com/yanql/p/15538815.html
頁:
[1]