在 K8S 中部署 jenkins
github上有一个1.5k star的项目:https://github.com/jenkinsci/kubernetes-plugin
上面提供了jenkins在kubernetes中容器化的部署方式,以及yaml文件,现在直接 git clone 下来。
[root@master ~]# cd /work/
[root@master work]# git clone https://github.com/jenkinsci/kubernetes-plugin.git
克隆完之后进入到这个目录。
[root@master work]# cd kubernetes-plugin/src/main/kubernetes/
[root@master kubernetes]# ls
jenkins.yml service-account.yml
直接克隆下来的文件,可能在某些k8s版本上会报错,请根据报错提示修改;
下面的yaml在k8s v1.16.3版本上做过了测试。
可以看到有两个文件,service-account.yml 文件是创建 RBAC 授权相关的东西,这个不要动,主要看一下 jenkins.yml,需要修改一下,修改后的文件如下:
# jenkins
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
selector:
matchLabels:
name: jenkins
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
command:
- /sbin/tini
- /usr/local/bin/jenkins.sh
- --prefix=/jenkins
image: jenkins/jenkins:lts-alpine
imagePullPolicy: Always
ports:
- containerPort: 8080
- containerPort: 50000
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 0.5
memory: 500Mi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -D--prefix=/jenkins
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /jenkins/login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /jenkins/login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12
securityContext:
fsGroup: 1000
volumes:
- hostPath:
path: /data/jenkins
name: jenkins-home
---
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
selector:
name: jenkins
ports:
-
name: http
port: 80
targetPort: 8080
protocol: TCP
-
name: agent
port: 50000
protocol: TCP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jenkins
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/tls-acme: "true"
# "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
# For nginx-ingress controller < 0.9.0.beta-18
ingress.kubernetes.io/ssl-redirect: "true"
# "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
ingress.kubernetes.io/proxy-body-size: 50m
ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
rules:
- http:
paths:
- path: /jenkins
backend:
serviceName: jenkins
servicePort: 80
host: example.cn
tls:
- hosts:
- example.cn
secretName: litemall-admin-secret
注意到,上面yaml中主要修改ingress中的host为自己的域名,我的域名是在阿里备案过的域名(如果有国外的机器,不用备案也可以),而且申请了阿里的免费https证书,将证书配置到ingress-nginx中(通过创建一个secret,secret中包含了证书的信息);
还有需要修改ingress中的path为带前缀的:/jenkins
,因为对于入口网关:ingress-nginx
,有很多路径为/
的服务创建ingress
去暴露,为了区分,所以通过带前缀的方式。这样就需要在Statefulset的podTemplate中配置jenkins的参数--prefix=/jenkins
,这样Pod启动执行command:/sbin/tini -- /usr/local/bin/jenkins.sh --prefix=/jenkins
后,jenkins.sh
脚本中就会调用java -jar jenkins.war --prefix=/jenkins
将参数传入。
默认path
加了/jenkins
前缀后,同时对应的修改健康检查中的path
为/jenkins/login
;
另外,jenkins的数据目录需要做持久化,也就是/var/jenkins_home
,jenkins
所有的数据都是存在这个目录下面的。
如果没有公网域名,那就没必要创建Ingress,直接通过Service的NodePort方式去暴露,最后通过{k8s集群任一主机IP}:{nodePort端口}/jenkins
方式去访问。
最后顺便贴出Service Account
的配置,这里对github上的原有yaml没做修改:
# In GKE need to get RBAC permissions first with
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin [--user=<user-name>|--group=<group-name>]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
最后通过域名去访问:
作者简洁
作者:小碗汤,一位热爱、认真写作的小伙,目前维护原创公众号:『我的小碗汤』,专注于写go语言、docker、kubernetes、java等开发、运维知识等提升硬实力的文章,期待你的关注。转载说明:务必注明来源(注明:来源于公众号:我的小碗汤,作者:小碗汤)