wso2~分布式集群部署总结
<p>集群模式需要有关注这块，如果是all-in-one模式，你部署多个节点，这不叫集群模式，它会有多种状态信息不一致，会有多种问题，如果希望将网关tw,平台acp(admin,consumer,publisher)，密钥管理km，限流管理tm分开部署多份，需要使用Distributed Deployment 模式(https://apim.docs.wso2.com/en/latest/install-and-setup/setup/kubernetes-deployment/kubernetes/am-pattern-3-acp-tm-gw/)。</p><ul>
<li>https://www.yenlo.com/blogs/custom-rate-limiting-with-wso2-apim/</li>
<li>https://apim.docs.wso2.com/en/4.1.0/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/</li>
<li>https://apim.docs.wso2.com/en/latest/manage-apis/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/</li>
<li>https://github.com/wso2/api-manager/issues/3961</li>
<li>https://github.com/wso2/api-manager/issues/359</li>
</ul>
<h1 id="helm-distribute分布式部署">helm-distribute分布式部署</h1>
<h3 id="1-部署顺序">1. 部署顺序</h3>
<ol>
<li>control-plane</li>
<li>gateway</li>
<li>traffic-manager</li>
<li>key-manager[未在私有仓库找到景象，目前它集成到control-plane里]</li>
</ol>
<h3 id="2-各子服务dockerfile添加jdbc">2. 各子服务Dockerfile添加jdbc</h3>
<pre><code>FROM wso2/wso2am-tm:4.5.0
ARG USER=wso2carbon
ARG USER_HOME=/home/${USER}
ARG WSO2_SERVER_NAME=wso2am-tm
ARG WSO2_SERVER_VERSION=4.5.0
ARG WSO2_SERVER=${WSO2_SERVER_NAME}-${WSO2_SERVER_VERSION}
ARG WSO2_SERVER_HOME=${USER_HOME}/${WSO2_SERVER}
# Copy JDBC MySQL driver
ADD --chown=wso2carbon:wso2 https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar ${WSO2_SERVER_HOME}/repository/components/lib
</code></pre>
<h3 id="3-gateway中valuesyaml中配置acp的服务名和实例名">3. gateway中values.yaml中配置acp的服务名和实例名</h3>
<pre><code>wso2:
apim:
configurations
throttling:
# -- Traffic manager loadbalancer service url for throttling.
# If the traffic manager is separated from control-plane, all urls should be pointed to traffic-manager ones.
serviceUrl: "wso2am-acp-service" # 对应control-plane的主服务名
eventhub:
# -- Event hub (control plane) enabled.
# This should be enabled if the traffic manager is separated from the control-plane.
enabled: false
# -- Event hub (control plane) loadbalancer service url
serviceUrl: "wso2am-acp-service"
</code></pre>
<h3 id="4-各服务的deploymentyaml文件添加本地时区变量">4. 各服务的deployment.yaml文件添加本地时区变量</h3>
<ul>
<li>control-plane\templates\control-plane\instance-1\wso2am-cp-deployment.yaml</li>
<li>control-plane\templates\control-plane\instance-2\wso2am-cp-deployment.yaml</li>
<li>gateway\templates\gateway\wso2am-gateway-deployment.yaml</li>
<li>traffic-manager\templates\traffic-manager\instance-1\wso2am-tm-deployment.yaml</li>
<li>traffic-manager\templates\traffic-manager\instance-2\wso2am-tm-deployment.yaml</li>
</ul>
<pre><code>spec:
template:
spec:
containers:
- name: wso2am-traffic-manager
image: {{ .Values.wso2.deployment.image.registry }}/{{ .Values.wso2.deployment.image.repository }}@{{ .Values.wso2.deployment.image.digest }}
imagePullPolicy: {{ .Values.wso2.deployment.imagePullPolicy }}
env:
- name: PROFILE_NAME
value: traffic-manager
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: JVM_MEM_OPTS
value: "-Xms{{ .Values.wso2.deployment.resources.jvm.memory.xms }} -Xmx{{ .Values.wso2.deployment.resources.jvm.memory.xmx }}"
- name: TZ
value: Asia/Shanghai
</code></pre>
<h3 id="5-多节点集群">5. 多节点集群</h3>
<ol>
<li>cp(control-plane) 支持多副本，多个deployment</li>
<li>gateway 无状态，直接使用deployment多副本功能</li>
<li>traffic-manager 支持多副本，多个deployment</li>
</ol>
<p><img src="https://images.cnblogs.com/cnblogs_com/lori/2430318/o_250721090013_wso2_distributes.png" alt="" loading="lazy"></p>
<h1 id="redis-throttle相关需再调研">redis throttle相关【需再调研】</h1>
<h3 id="相关动态">相关动态</h3>
<p>Update 02/07/2025</p>
<ul>
<li>
<p>Until now, Redis was only tested in an all-in-one deployment. Since the actual requirement is to support a multi-Traffic Manager (TM) node deployment, set up a distributed APIM deployment—with support from @kavindasr —consisting of 2 ACP nodes, 2 GW nodes, 2 TM nodes, and a Redis Cluster hosted in AWS.</p>
</li>
<li>
<p>To support the Redis Cluster, I refactored the implementation to use JedisCluster instead of JedisPooled, and verified the functionality in this distributed setup.</p>
</li>
<li>
<p>Initially, a configuration mismatch caused the Redis count to be incremented multiple times per API request—once by each Traffic Manager. After identifying the issue, adjusted the configuration to ensure correct and consistent counter updates.</p>
</li>
</ul>
<h1 id="分布式部署问题点说明">分布式部署问题点说明</h1>
<ul>
<li>两个acp配置问题，acp需要配置自己和对方的发布消息的端口9611和9711，他们需要通讯，配置对方的5672的mq端口</li>
<li>一个gateway没有正常配置两个acp，gateway需要配置所有acp的5672，9611，9711端口，和一个公共的acp服务的9433端口</li>
<li>两个tm配置问题，tm需要配置对方tm的5672端口，和一个公共的acp服务的9433端口</li>
</ul>
<h2 id="apim-acp-两个实例配置时1配置22配置1">apim-acp 两个实例，配置时1配置2，2配置1</h2>
<p>wso2am-acp-1配置</p>
<pre><code>
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://localhost:5672"]
event_duplicate_url = ["tcp://wso2am-acp-2-service:5672"]
[]
urls = ["tcp://wso2am-acp-1-service:9611"]
auth_urls = ["ssl://wso2am-acp-1-service:9711"]
[]
urls = ["tcp://wso2am-acp-2-service:9611"]
auth_urls = ["ssl://wso2am-acp-2-service:9711"]
</code></pre>
<p>wso2am-acp-2配置</p>
<pre><code>
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://localhost:5672"]
event_duplicate_url = ["tcp://wso2am-acp-1-service:5672"]
[]
urls = ["tcp://wso2am-acp-1-service:9611"]
auth_urls = ["ssl://wso2am-acp-1-service:9711"]
[]
urls = ["tcp://wso2am-acp-2-service:9611"]
auth_urls = ["ssl://wso2am-acp-2-service:9711"]
</code></pre>
<h2 id="apim-gateway-两个副本配置acp1和acp2">apim-gateway 两个副本，配置acp1和acp2</h2>
<pre><code># Traffic Manager configurations
username= "$ref{super_admin.username}"
password= "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
throttle_decision_endpoints = ["tcp://wso2am-acp-1-service:5672","tcp://wso2am-acp-2-service:5672"]
enable_unlimited_tier = true
enable_header_based_throttling = true
enable_jwt_claim_based_throttling = true
enable_query_param_based_throttling = true
[]
traffic_manager_urls = ["tcp://wso2am-acp-1-service:9611","tcp://wso2am-acp-2-service:9611"]
traffic_manager_auth_urls = ["ssl://wso2am-acp-1-service:9711","ssl://wso2am-acp-2-service:9711"]
type = "loadbalance"
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-1-service:5672","tcp://wso2am-acp-2-service:5672"]
</code></pre>
<h2 id="apim-tm-两个实例配置时1配置22配置1">apim-tm 两个实例，配置时1配置2，2配置1</h2>
<p>wso2am-tm-1配置</p>
<pre><code>
service_url = "https://wso2am-acp-service:9443/services/"
# Traffic Manager configurations
event_duplicate_url = ["tcp://wso2am-tm-2-service:5672"]
service_url = "https://wso2am-tm-service:9443/services/"
throttle_decision_endpoints = ["tcp://localhost:5672"]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-2-service:5672"]
</code></pre>
<p>wso2am-tm-2配置</p>
<pre><code>
service_url = "https://wso2am-acp-service:9443/services/"
# Traffic Manager configurations
event_duplicate_url = ["tcp://wso2am-tm-1-service:5672"]
service_url = "https://wso2am-tm-service:9443/services/"
throttle_decision_endpoints = ["tcp://localhost:5672"]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-1-service:5672"]
</code></pre>
</div>
<div id="MySignature" role="contentinfo">
<p></p>
<div class="navgood">
<p>作者：仓储大叔，张占岭，<br>
荣誉：微软MVP<br>QQ：853066980</p>
<p><strong>支付宝扫一扫，为大叔打赏!</strong>
<br><img src="https://images.cnblogs.com/cnblogs_com/lori/237884/o_IMG_7144.JPG"></p>
</div><br><br>
来源：https://www.cnblogs.com/lori/p/18998591
頁:
[1]