本文主要介绍如何在阿里云容器服务Kubernetes上快速安装部署Jenkins X Platform并结合demo实践演示GitOps的操作流程。
注意：
本文中使用的jx工具、cloud-environments等做过改造用以适配阿里云Kubernetes容器服务，并未在自建Kubernetes集群中做过验证。
先决条件：
首先，需要在阿里云容器服务控制台创建一个Kubernetes集群，本次实践使用的环境信息如下：
master1 192.168.0.119
master2 192.168.0.120
master3 192.168.0.121
worker1 192.168.0.122
jx-node 192.168.0.123
一、 部署Jenkins X Platform并创建Staging Production Env
1. kubectl 、helm、git和jx的安装以及初始化
$ scp 192.168.0.119:/usr/bin/kubectl /usr/bin/kubectl$ scp 192.168.0.119:/usr/local/bin/helm /usr/local/bin/helm$ mkdir -p $HOME/.kube将容器服务控制台上显示的kubeconfig文件内容复制到计算机 $HOME/.kube/config
初始化Helm Client：
$ helm init --client-only --stable-repo-url https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/$ helm repo update安装git
$ yum install git -y安装jx
$ wget http://acs-jenkins-x.oss-cn-beijing.aliyuncs.com/v1.3.699/jx && mv jx /usr/local/bin/ && chmod +x /usr/local/bin/jx验证jx：(注意不要选择升级，jx做了定制化修改，目前版本1.3.699)
$ jx versionError loading team settings. the server could not find the requested resource (get environments.jenkins.io)Error loading team settings. the server could not find the requested resource (get environments.jenkins.io)Failed to get git version: failed to run ‘git version‘ command in directory ‘‘, output: ‘‘: exec: "git": executable file not found in $PATHNAME ??????????????VERSIONjx ????????????????1.3.699Kubernetes cluster v1.11.5kubectl ???????????v1.11.5helm client ???????v2.11.0+g2e55dbehelm server ???????v2.11.0+g2e55dbeOperating System ??"CentOS Linux release 7.4.1708 (Core) "A new jx version is available: 1.3.736? Would you like to upgrade to the new jx version? No2. 安装jenkins-x-platform
$ mkdir -p ~/.jx$ cd ~/.jx$ jx install --provider=kubernetes ?????????????--domain aliyunk8s-bj.com ?????????????--ingress-deployment=nginx-ingress-controller ?????????????--ingress-service=nginx-ingress-lb ??????????????--cloud-environment-repo=https://github.com/AliyunContainerService/cloud-environments.git ?????????????--no-tiller=true ?????????????--no-default-environments=true交互及日志输出如下：
Namespace jx created Using helmBinary helm with feature flag: noneContext "kubernetes-admin-ccf3ebc497d9e4083aab8c89bb1b217bc" modified.Storing the kubernetes provider kubernetes in the TeamSettingsEnabling helm template mode in the TeamSettings? Please enter the name you wish to use with git: ?jenkins-x-bot? Please enter the email address you wish to use with git: ?haoshuwei24@gmail.comGit configured for user: jenkins-x-bot and email haoshuwei24@gmail.comTrying to create ClusterRoleBinding kubernetes-admin-cluster-admin-binding for role: cluster-admin for user kubernetes-admin clusterrolebindings.rbac.authorization.k8s.io "kubernetes-admin-cluster-admin-binding" not foundCreated ClusterRoleBinding kubernetes-admin-cluster-admin-bindingUsing helm2Skipping tillerhelm installed and configuredexisting ingress controller found, no need to install a new oneWaiting for external loadbalancer to be created and update the nginx-ingress-controller service in kube-system namespaceExternal loadbalancer createdWaiting to find the external host name of the ingress controller Service in namespace kube-system with name nginx-ingress-lbIf you are installing Jenkins X on premise you may want to use the ‘--on-premise‘ flag or specify the ‘--external-ip‘ flags. See: https://jenkins-x.io/getting-started/install-on-cluster/#installing-jenkins-x-on-premiseYou can now configure your wildcard DNS aliyunk8s-bj.com to point to 39.97.21.40nginx ingress controller installed and configuredLets set up a Git user name and API token to be able to perform CI/CDCreating a local Git user for GitHub server? GitHub user name: jenkins-x-botTo be able to create a repository on GitHub we need an API TokenPlease click this URL https://github.com/settings/tokens/new?scopes=repo,read:user,read:org,user:email,write:repo_hook,delete_repoThen COPY the token and enter in into the form below:? API Token: ****************************************Select the CI/CD pipelines Git server and user? Do you wish to use GitHub as the pipelines Git server: Yes? Do you wish to use jenkins-x-bot as the pipelines Git user for GitHub server: YesSetting the pipelines Git server https://github.com and user name jenkins-x-bot.Saving the Git authentication configurationCurrent configuration dir: /root/.jxoptions.Flags.CloudEnvRepository: https://github.com/AliyunContainerService/cloud-environments.gitoptions.Flags.LocalCloudEnvironment: falseCloning the Jenkins X cloud environments repo to /root/.jx/cloud-environmentsEnumerating objects: 1317, done.Counting objects: 100% (1317/1317), done.Compressing objects: 100% (618/618), done.Total 1317 (delta 673), reused 1317 (delta 673), pack-reused 0No default password set, generating a random oneGenerated helm values /root/.jx/extraValues.yamlCreating Secret jx-install-config in namespace jxInstalling Jenkins X platform helm chart from: /root/.jx/cloud-environments/env-kubernetes? Select Jenkins installation type: Static Master Jenkins? Pick workload build pack: ?Kubernetes Workloads: Automated CI+CD with GitOps PromotionSetting the team build pack to kubernetes-workloads repo: https://github.com/AliyunContainerService/jenkins-x-kubernetes.git ref: masterInstalling jx into namespace jxAdding values file /root/.jx/cloud-environments/env-kubernetes/myvalues.yamlAdding values file /root/.jx/adminSecrets.yamlAdding values file /root/.jx/extraValues.yamlAdding values file /root/.jx/cloud-environments/env-kubernetes/secrets.yamlFetched chart jenkins-x/jenkins-x-platform to dir /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platformGenerating Chart Template ‘template --name jenkins-x --namespace jx /tmp/helm-template-workdir-860490114/jenkins-x/chartFiles/jenkins-x-platform --output-dir /tmp/helm-template-workdir-860490114/jenkins-x/output --debug --values /root/.jx/cloud-environments/env-kubernetes/myvalues.yaml --values /root/.jx/adminSecrets.yaml --values /root/.jx/extraValues.yaml --values /root/.jx/cloud-environments/env-kubernetes/secrets.yaml‘Applying generated chart jenkins-x/jenkins-x-platform YAML via kubectl in dir: /tmp/helm-template-workdir-860490114/jenkins-x/outputdeployment.extensions/jenkins-x-chartmuseum createdpersistentvolumeclaim/jenkins-x-chartmuseum createdsecret/jenkins-x-chartmuseum createdservice/jenkins-x-chartmuseum createdrole.rbac.authorization.k8s.io/cleanup createdrolebinding.rbac.authorization.k8s.io/cleanup createdserviceaccount/cleanup createdclusterrole.rbac.authorization.k8s.io/controllercommitstatus-jx createdclusterrolebinding.rbac.authorization.k8s.io/controllercommitstatus-jx createddeployment.apps/jenkins-x-controllercommitstatus createdrole.rbac.authorization.k8s.io/controllercommitstatus createdrolebinding.rbac.authorization.k8s.io/controllercommitstatus createdserviceaccount/jenkins-x-controllercommitstatus createdclusterrole.rbac.authorization.k8s.io/controllerrole-jx createdclusterrolebinding.rbac.authorization.k8s.io/controllerrole-jx createddeployment.apps/jenkins-x-controllerrole createdrole.rbac.authorization.k8s.io/controllerrole createdrolebinding.rbac.authorization.k8s.io/controllerrole createdserviceaccount/jenkins-x-controllerrole createdclusterrole.rbac.authorization.k8s.io/controllerteam-jx createdclusterrolebinding.rbac.authorization.k8s.io/controllerteam-jx createddeployment.apps/jenkins-x-controllerteam createdrole.rbac.authorization.k8s.io/controllerteam createdrolebinding.rbac.authorization.k8s.io/controllerteam createdserviceaccount/jenkins-x-controllerteam createdclusterrole.rbac.authorization.k8s.io/controllerworkflow-jx createdclusterrolebinding.rbac.authorization.k8s.io/controllerworkflow-jx createddeployment.apps/jenkins-x-controllerworkflow createdrole.rbac.authorization.k8s.io/controllerworkflow createdrolebinding.rbac.authorization.k8s.io/controllerworkflow createdserviceaccount/jenkins-x-controllerworkflow createdconfigmap/exposecontroller createdrole.rbac.authorization.k8s.io/expose createdrolebinding.rbac.authorization.k8s.io/expose createdserviceaccount/expose createdclusterrole.rbac.authorization.k8s.io/gcactivities-jx createdclusterrolebinding.rbac.authorization.k8s.io/gcactivities-jx createdcronjob.batch/jenkins-x-gcactivities createdrole.rbac.authorization.k8s.io/gcactivities createdrolebinding.rbac.authorization.k8s.io/gcactivities createdserviceaccount/jenkins-x-gcactivities createdcronjob.batch/jenkins-x-gcpods createdrole.rbac.authorization.k8s.io/gcpods createdrolebinding.rbac.authorization.k8s.io/gcpods createdserviceaccount/jenkins-x-gcpods createdclusterrole.rbac.authorization.k8s.io/gcpreviews-jx createdclusterrolebinding.rbac.authorization.k8s.io/gcpreviews-jx createdcronjob.batch/jenkins-x-gcpreviews createdrole.rbac.authorization.k8s.io/gcpreviews createdrolebinding.rbac.authorization.k8s.io/gcpreviews createdserviceaccount/jenkins-x-gcpreviews createddeployment.extensions/jenkins-x-heapster createdservice/heapster createdconfigmap/jenkins createdpersistentvolumeclaim/jenkins createdservice/jenkins-agent createddeployment.extensions/jenkins createdservice/jenkins createdconfigmap/jenkins-x-git-kinds createdclusterrolebinding.rbac.authorization.k8s.io/jenkins-jx-role-binding createdsecret/jenkins createdserviceaccount/jenkins createdconfigmap/jenkins-tests createdconfigmap/nexus createddeployment.extensions/jenkins-x-nexus createdpersistentvolumeclaim/jenkins-x-nexus createdsecret/nexus createdservice/nexus createdrole.rbac.authorization.k8s.io/committer createdclusterrolebinding.rbac.authorization.k8s.io/jenkins-x-team-controller createdconfigmap/jenkins-x-team-controller createdsecret/jenkins-docker-cfg createdconfigmap/jenkins-x-devpod-config createdconfigmap/jenkins-x-docker-registry createdconfigmap/jenkins-x-extensions createdconfigmap/jenkins-x-pod-templates createdsecret/jx-basic-auth createdrole.rbac.authorization.k8s.io/jx-view createdsecret/jenkins-maven-settings createdsecret/jenkins-npm-token createdrole.rbac.authorization.k8s.io/owner createdsecret/jenkins-release-gpg createdsecret/jenkins-ssh-config createdrole.rbac.authorization.k8s.io/viewer createdApplying Helm hook post-upgrade YAML via kubectl in file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yamljob.batch/expose createdWaiting for helm post-upgrade hook Job expose to complete before removing it如果是首次部署jenkins x platfrom的话，此处需等待几分钟进行镜像的拉取和容器的启动：
$ kubectl -n jx get poNAME ???????????????????????????????????????????????READY ????STATUS ?????????????RESTARTS ??AGEexpose-2bhmv ???????????????????????????????????????0/1 ??????ContainerCreating ??0 ?????????1mjenkins-568884c766-sxc7d ???????????????????????????0/1 ??????Init:0/1 ???????????0 ?????????1mjenkins-x-chartmuseum-6cf566bfb-jm7cv ??????????????0/1 ??????ContainerCreating ??0 ?????????1mjenkins-x-controllercommitstatus-6f57d857d8-kzllt ??1/1 ??????Running ????????????0 ?????????1mjenkins-x-controllerrole-57d864c96f-4lrrk ??????????1/1 ??????Running ????????????0 ?????????1mjenkins-x-controllerteam-c48fc44f-tsrkh ????????????0/1 ??????ContainerCreating ??0 ?????????1mjenkins-x-controllerworkflow-c758649d6-2kpfj ???????0/1 ??????ContainerCreating ??0 ?????????1mjenkins-x-heapster-7fbdb867d9-wt6sl ????????????????0/2 ??????ContainerCreating ??0 ?????????1mjenkins-x-nexus-5d5455cfd7-4xsck ???????????????????0/1 ??????ContainerCreating ??0 ?????????1m接下来的日志交互及输出如下：
^@^@^@Deleting helm hook sources from file: /tmp/helm-template-workdir-860490114/jenkins-x/helmHooks/jenkins-x-platform/charts/expose/templates/job.yamljob.batch "expose" deletedRemoving Kubernetes resources from older releases using selector: jenkins.io/chart-release=jenkins-x,jenkins.io/version!=0.0.3193waiting for install to be ready, if this is the first time then it will take a while to download images^@Jenkins X deployments ready in namespace jx ???******************************************************** ????????NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp ???********************************************************Getting Jenkins API TokenUsing url http://jenkins.jx.aliyunk8s-bj.com/me/configureUnable to automatically find API token with chromedp using URL http://jenkins.jx.aliyunk8s-bj.com/me/configureError: creating the chrome client: fork/exec /usr/bin/google-chrome: no such file or directoryPlease go to http://jenkins.jx.aliyunk8s-bj.com/me/configure and click Show API Token to get your API TokenThen COPY the token and enter in into the form below:? API Token:此处如果没有DNS service解析域名jenkins.jx.aliyunk8s-bj.com的话， 需要先在jx-node上手动绑定hosts：
$ kubectl -n jx get ingNAME ?????????HOSTS ????????????????????????????ADDRESS ??????PORTS ????AGEchartmuseum ??chartmuseum.jx.aliyunk8s-bj.com ??xx.xx.xx.xx ??80 ???????5mjenkins ??????jenkins.jx.aliyunk8s-bj.com ??????xx.xx.xx.xx ??80 ???????5mnexus ????????nexus.jx.aliyunk8s-bj.com ????????xx.xx.xx.xx ??80 ???????5m$ echo "xx.xx.xx.xx jenkins.jx.aliyunk8s-bj.com" >> /etc/hosts根据日志提示生成jenkins api token：
完成安装：
Created user admin API Token for Jenkins server jenkins.jx.aliyunk8s-bj.com at http://jenkins.jx.aliyunk8s-bj.comUpdating Jenkins with new external URL details http://jenkins.jx.aliyunk8s-bj.comJenkins X installation completed successfully ???******************************************************** ????????NOTE: Your admin password is: qub6n#mKkh0oN!S59nPp ???********************************************************Your Kubernetes context is now set to the namespace: jxTo switch back to your original namespace use: jx namespace defaultFor help on switching contexts see: https://jenkins-x.io/developing/kube-context/To import existing projects into Jenkins: ??????jx importTo create a new Spring Boot microservice: ??????jx create spring -d web -d actuatorTo create a new microservice from a quickstart: jx create quickstart3. 创建staging env
$ jx create env -n staging -l Staging --namespace staging --fork-git-repo=‘https://github.com/haoshuwei/default-environment-charts.git‘ --domain=aliyunk8s-bj.com --promotion=Auto --prefix=‘bj‘....Creating GitHub webhook for haoshuwei/environment-bj-staging for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/由于我们默认使用Ingress暴露服务， 所以需要修改environment-bj-staging的webhook：
$ kubectl -n jx get svc |grep LoadBalancerkubectl -n jx get svc |grep LoadBalancerjenkins ????????????????LoadBalancer ??172.19.11.179 ??xx.xx.xx.xx ??8080:30456/TCP ??13hjenkins会自动创建environment-bj-staging job并进行扫描和构建：
4. 创建production env:
$ jx create env -n production -l Production --namespace production --fork-git-repo=‘https://github.com/haoshuwei/default-environment-charts.git‘ --domain=aliyunk8s-bj.com --promotion=Manual --prefix=‘bj‘...Creating GitHub webhook for haoshuwei/environment-bj-production for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/修改environment-hz-production的webhook。
jenkins会自动创建environment-bj-production job并进行扫描和构建：
5. 配置访问阿里云容器镜像服务的docker registry secret：
$ kubectl -n jx delete secrets jenkins-docker-cfgsecret "jenkins-docker-cfg" deleted$ docker login -u xxx -p xxx registry.cn-beijing.aliyuncs.com$ kubectl create secret generic jenkins-docker-cfg -n jx --from-file=/root/.docker/config.jsonsecret/jenkins-docker-cfg created二、 运用GitOps管理应用发布
1. 安装openjdk
$ yum install -y java-1.8.0-openjdk2. 创建一个Spring示例应用
$ jx create spring -d web -d actuator? Language: java? Group: com.example? Artifact: jenkins-x-demoCreated Spring Boot project at /root/.jx/jenkins-x-demoNo username defined for the current Git server!? Do you wish to use jenkins-x-bot as the Git user name: YesThe directory /root/.jx/jenkins-x-demo is not yet using git? Would you like to initialise git now? Yes? Commit message: ?Initial importGit repository createdselected pack: /root/.jx/draft/packs/github.com/AliyunContainerService/jenkins-x-kubernetes/packs/maven? Which organisation do you want to use? jenkins-x-botreplacing placeholders in directory /root/.jx/jenkins-x-demoapp name: jenkins-x-demo, git server: github.com, org: jenkins-x-bot, Docker registry org: jenkins-x-botskipping directory "/root/.jx/jenkins-x-demo/.git"Using Git provider GitHub at https://github.comAbout to create repository jenkins-x-demo on server https://github.com with user jenkins-x-bot? Enter the new repository name: ?jenkins-x-demoCreating repository jenkins-x-bot/jenkins-x-demoPushed Git repository to https://github.com/haoshuwei/jenkins-x-demoCreated Jenkins Project: http://jenkins.jx.aliyunk8s-bj.com/job/haoshuwei/job/jenkins-x-demo/Watch pipeline activity via: ???jx get activity -f jenkins-x-demo -wBrowse the pipeline log via: ???jx get build logs haoshuwei/jenkins-x-demo/masterOpen the Jenkins console via ???jx consoleYou can list the pipelines via: jx get pipelinesWhen the pipeline is complete: ?jx get applicationsFor more help on available commands see: https://jenkins-x.io/developing/browsing/Note that your first pipeline may take a few minutes to start while the necessary images get downloaded!Creating GitHub webhook for haoshuwei/jenkins-x-demo for url http://jenkins.jx.aliyunk8s-bj.com/github-webhook/修改 jenkins-x-demo的webhook。
jenkins-x-demo项目成功创建后，jenkins job jenkins-x-demo会自动扫描并构建jenkins-x-demo-0.0.1.tgz发布到chartmuseum并自动promote到staging环境：
查看environment-bj-staging项目的PR：
PR的Check通过后会自动merge：
主要修改内容为：
然后environment-bj-staging项目自动构建并部署jenkins-x-demo应用到Staging Env。
3. 访问Staging环境中的jenkins-x-demo应用服务
$ jx get appsAPPLICATION ??????PRODUCTION PODS URL ???????????????????????????????????????????STAGING PODS URLjx-jenkins-x-demo 0.0.1 ?????1/1 ?http://jenkins-x-demo.staging.aliyunk8s-bj.com访问http://jenkins-x-demo.staging.aliyunk8s-bj.com：
4. 开发jenkins-x-demo应用并发布新版本到Staging环境
步骤3中， 我们访问了一个并没有准备好进入生产环境的jenkins-x-demo应用， 下面我们添加一个新的feature：
$ cd jenkins-x-demo$ git checkout -b feature/add-index新建src/main/resources/static/index.html并写入如下内容：
<html> ?<head> ???<title> Jenkins X Spring Demo </title> ?</head> ?<body bgcolor=white> ???<table border="0" cellpadding="10"> ?????<tr> ???????<td> ?????????<img height="300" width="300" src="https://github.com/jenkins-x/jenkins-x-website/raw/e5aae999166a67d6220aa469eed1f23e0996c1f7/images/logo.png"> ???????</td> ???????<td> ?????????<h1>Jenkins X Spring Demo</h1> ???????</td> ?????</tr> ???</table> ?</body></html>提交和推送后我们在GitHub上创建一个feature/add-index到master分支的PR：
jenkins会自动对PR进行构建和部署，这个过程中jenkins-x-demo的部署并不是部署到Staging环境，而是会部署为Preview环境提供预览：
预览jenkins-x-demo应用http://jenkins-x-demo.jx-haoshuwei-jenkins-x-demo-pr-1.aliyunk8s-bj.com/：
5. 发布新版本jenkins-x-demo应用到Staging环境
确认提交的PR没问题后，即可merge到master分支：
jenkins会重新构建jenkins-x-demo v0.0.2并发布到Staging环境:
$ jx get appsAPPLICATION ??????PRODUCTION PODS URL ???????????????????????????????????????????STAGING PODS URLjx-jenkins-x-demo 0.0.2 ?????1/1 ?http://jenkins-x-demo.staging.aliyunk8s-bj.com6. jenkins-x-demo应用推送到Production环境
一旦应用通过测试验证并准备好进入生产后， 我们就可以手动发布应用到Production环境：
$ jx get appsAPPLICATION ??????PRODUCTION PODS URL ??????????????????????????????????????????????STAGING PODS URLjx-jenkins-x-demo 0.0.2 ?????1/1 ?http://jenkins-x-demo.production.aliyunk8s-bj.com 0.0.2 ??1/1 ?http://jenkins-x-demo.staging.aliyunk8s-bj.com访问Production环境中的jenkins-x-demo：
三、 Production环境中应用的回滚
回滚到版本v0.x.x
$ jx promote --version 0.x.x --env production --timeout 20m四、 删除或重装Jenkins X Platform
$ kubectl config current-context$ jx uninstall$ cd ~/.jx $ rm -rf *五、 结束语
目前Jenkins X Platform还在不断完善和改进中， 比如支持多集群等问题https://github.com/jenkins-x/jx/issues/479， 我们也会持续优化Jenkins X 在阿里云Kubernetes容器服务上的最佳实践并更新博客。
原文链接
更多技术干货 请关注阿里云云栖社区微信号 ：yunqiinsight
15分钟在阿里云Kubernetes服务上快速建立Jenkins X Platform并运用GitOps管理应用发布
原文地址：https://www.cnblogs.com/zhaowei121/p/10334280.html