Kubernetes部署（十二）：helm部署harbor企业级镜像仓库

发布时间：2023-09-06 02:31责任编辑：傅花花关键词：暂无标签

harbor简介

harbor官方github：https://github.com/goharbor
Harbor是一个用于存储和分发Docker镜像的企业级Registry服务器。Harbor通过添加用户通常需要的功能（如安全性，身份和管理）来扩展开源Docker Distribution。使registry更接近构建和运行环境可以提高图像传输效率。Harbor支持在registry之间复制映像，还提供高级安全功能，如用户管理，访问控制和活动审计。

特征

云原生注册表：Harbour 支持容器镜像和Helm chart，可用作本地云环境（如容器运行和业务流程平台）的注册表。
基于角色的访问控制：用户和存储库通过“项目”进行组织，用户可以对项目下的镜像拥有不同的权限。
基于策略的映像复制：可以基于具有多个过滤器（存储库，标记和标签）的策略在多个注册表实例之间复制（同步）映像。如果遇到任何错误，Harbor将自动重试进行复制。非常适合负载平衡，高可用性，多数据中心，混合和多云场景。
漏洞扫描：Harbor定期扫描镜像并警告用户漏洞。
LDAP / AD支持：Harbor与现有企业LDAP / AD集成以进行用户身份验证和管理，并支持将LDAP组导入Harbor并为其分配适当的项目角色。
镜像删除和垃圾收集：可以删除图像，并可以回收它们的空间。
公证：可以确保镜像的真实性。
图形用户门户：用户可以轻松浏览，搜索存储库和管理项目。
审计：跟踪存储库的所有操作。
RESTful API：适用于大多数管理操作的RESTful API，易于与外部系统集成。
易于部署：提供在线和离线安装程序。
先决条件
Kubernetes集群 1.10+
helm 2.8.0+

Harbor部署

1. 添加域名解析。

将h.cnlinux.club和n.cnlinux.club的A记录解析到我的负载均衡IP 10.31.90.200，用于创建ingress。

2. 下载harbor的chart包

[root@node-01 harbor]# wget https://github.com/goharbor/harbor-helm/archive/1.0.0.tar.gz -O harbor-helm-v1.0.0.tar.gz

3. 修改配置文件

提取harbor-helm-v1.0.0.tar.gz文件中的values.yaml文件，并放到和harbor-helm-v1.0.0.tar.gz同一级的目录中。
修改values.yaml，我的配置修改了如下几个字段：
需要说明的是如果k8s集群中存在storageclass就可以直接用storageclass，在几个persistence.persistentVolumeClaim.XXX.storageClass中指定storageclass名就可以了，会自动多个pvc，但是我这里为了防止创建多个pvc增加管理难度，我在部署前创建了一个pvc，harbor下所有的服务都使用这一个pvc，具体每个字段的作用请查看官方文档https://github.com/goharbor/harbor-helm。
- expose.ingress.hosts.core
- xpose.ingress.hosts.notary
- externalURL
- persistence.persistentVolumeClaim.registry.existingClaim
- persistence.persistentVolumeClaim.registry.subPath
- persistence.persistentVolumeClaim.chartmuseum.existingClaim
- persistence.persistentVolumeClaim.chartmuseum.subPath
- persistence.persistentVolumeClaim.jobservice.existingClaim
- persistence.persistentVolumeClaim.jobservice.subPath
- persistence.persistentVolumeClaim.database.existingClaim
- persistence.persistentVolumeClaim.database.subPath
- persistence.persistentVolumeClaim.redis.existingClaim
- persistence.persistentVolumeClaim.redis.subPath

expose: ?type: ingress ?tls: ???enabled: true ???secretName: "" ???notarySecretName: "" ???commonName: "" ?ingress: ???hosts: ?????core: h.cnlinux.club ?????notary: n.cnlinux.club ???annotations: ?????ingress.kubernetes.io/ssl-redirect: "true" ?????nginx.ingress.kubernetes.io/ssl-redirect: "true" ?????ingress.kubernetes.io/proxy-body-size: "0" ?????nginx.ingress.kubernetes.io/proxy-body-size: "0" ?clusterIP: ???name: harbor ???ports: ?????httpPort: 80 ?????httpsPort: 443 ?????notaryPort: 4443 ?nodePort: ???name: harbor ???ports: ?????http: ???????port: 80 ???????nodePort: 30002 ?????https: ????????port: 443 ???????nodePort: 30003 ?????notary: ????????port: 4443 ???????nodePort: 30004externalURL: https://h.cnlinux.clubpersistence: ?enabled: true ?resourcePolicy: "keep" ?persistentVolumeClaim: ???registry: ?????existingClaim: "pvc-harbor" ?????storageClass: "" ?????subPath: "registry" ?????accessMode: ReadWriteOnce ?????size: 5Gi ???chartmuseum: ?????existingClaim: "pvc-harbor" ?????storageClass: "" ?????subPath: "chartmuseum" ?????accessMode: ReadWriteOnce ?????size: 5Gi ???jobservice: ?????existingClaim: "pvc-harbor" ?????storageClass: "" ?????subPath: "jobservice" ?????accessMode: ReadWriteOnce ?????size: 1Gi ???database: ?????existingClaim: "pvc-harbor" ?????storageClass: "" ?????subPath: "database" ?????accessMode: ReadWriteOnce ?????size: 1Gi ???redis: ?????existingClaim: "pvc-harbor" ?????storageClass: "" ?????subPath: "redis" ?????accessMode: ReadWriteOnce ?????size: 1Gi ?imageChartStorage: ???type: filesystem ???filesystem: ?????rootdirectory: /storageimagePullPolicy: IfNotPresentlogLevel: debugharborAdminPassword: "Harbor12345"secretKey: "not-a-secure-key"nginx: ?image: ???repository: goharbor/nginx-photon ???tag: v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}portal: ?image: ???repository: goharbor/harbor-portal ???tag: v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}core: ?image: ???repository: goharbor/harbor-core ???tag: v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}adminserver: ?image: ???repository: goharbor/harbor-adminserver ???tag: v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}jobservice: ?image: ???repository: goharbor/harbor-jobservice ???tag: v1.7.0 ?replicas: 1 ?maxJobWorkers: 10 ?jobLogger: file ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}registry: ?registry: ???image: ?????repository: goharbor/registry-photon ?????tag: v2.6.2-v1.7.0 ?controller: ???image: ?????repository: goharbor/harbor-registryctl ?????tag: v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}chartmuseum: ?enabled: true ?image: ???repository: goharbor/chartmuseum-photon ???tag: v0.7.1-v1.7.0 ?replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}clair: ?enabled: true ?image: ???repository: goharbor/clair-photon ???tag: v2.0.7-v1.7.0 ?replicas: 1 ?httpProxy: ?httpsProxy: ?updatersInterval: 12 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}notary: ?enabled: true ?server: ???image: ?????repository: goharbor/notary-server-photon ?????tag: v0.6.1-v1.7.0 ???replicas: 1 ?signer: ???image: ?????repository: goharbor/notary-signer-photon ?????tag: v0.6.1-v1.7.0 ???replicas: 1 ?nodeSelector: {} ?tolerations: [] ?affinity: {} ?podAnnotations: {}database: ?type: internal ?internal: ???image: ?????repository: goharbor/harbor-db ?????tag: v1.7.0 ???password: "changeit" ???nodeSelector: {} ???tolerations: [] ???affinity: {} ?podAnnotations: {}redis: ?type: internal ?internal: ???image: ?????repository: goharbor/redis-photon ?????tag: v1.7.0 ???nodeSelector: {} ???tolerations: [] ???affinity: {} ?podAnnotations: {}

4. 创建存储卷

因为harbor需要使用到mysql，为防止mysql在调度过程中造成数据丢失，我们需要将mysql的数据存储在gluster的存储卷里。

[root@node-01 harbor]# vim pvc-harbor.yaml apiVersion: v1kind: PersistentVolumeClaimmetadata: ?name: pvc-harborspec: ?storageClassName: gluster-heketi ?accessModes: ???- ReadWriteMany ?resources: ???requests: ?????storage: 50Gi

[root@node-01 harbor]# kubectl apply -f pvc-harbor.yaml

5. 安装harbor

[root@node-01 harbor]# helm install ?--name harbor harbor-helm-v1.0.0.tar.gz -f values.yaml

如果安装不成功可以用helm del --purge harbor删除重新安装。

6. 测试

在一段时间后可以看到harbor所有相关的pod都已经运行起来了，我们就可以访问了，默认用户密码是admin/Harbor12345,可以通过修改values.yaml来更改默认的用户名和密码。

[root@node-01 ~]# kubectl get podNAME ??????????????????????????????????????????READY ????STATUS ???RESTARTS ??AGEharbor-harbor-adminserver-7fffc7bf4d-vj845 ????1/1 ??????Running ??1 ?????????15dharbor-harbor-chartmuseum-bdf64f899-brnww ?????1/1 ??????Running ??0 ?????????15dharbor-harbor-clair-8457c45dd8-9rgq8 ??????????1/1 ??????Running ??1 ?????????15dharbor-harbor-core-7fc454c6d8-b6kvs ???????????1/1 ??????Running ??1 ?????????15dharbor-harbor-database-0 ??????????????????????1/1 ??????Running ??0 ?????????15dharbor-harbor-jobservice-7895949d6b-zbwkf ?????1/1 ??????Running ??1 ?????????15dharbor-harbor-notary-server-57dd94bf56-txdkl ??1/1 ??????Running ??0 ?????????15dharbor-harbor-notary-signer-5d64c5bf8d-kppts ??1/1 ??????Running ??0 ?????????15dharbor-harbor-portal-648c56499f-g28rz ?????????1/1 ??????Running ??0 ?????????15dharbor-harbor-redis-0 ?????????????????????????1/1 ??????Running ??0 ?????????15dharbor-harbor-registry-5cd9c49489-r92ph ???????2/2 ??????Running ??0 ?????????15d

接下来我们创建test的私有项目用来测试。
因为我们创建的harbor仓库是https的所以在docker pull或者push镜像之前，需要先把证书加到docker对应的配置目录里，不然docker是无法登录harbor的。
进入test项目，点解“注册证书”下载harbor的CA证书。
在每个node节点创建目录（以后可能会在master上传镜像，所以此次我的master节点也都一起创建了）

for n in `seq -w 01 06`;do ssh node-$n "mkdir -p /etc/docker/certs.d/h.cnlinux.club";done#将下载下来的harbor CA证书拷贝到每个node节点的etc/docker/certs.d/h.cnlinux.club目录下for n in `seq -w 01 06`;do scp ca.crt node-$n:/etc/docker/certs.d/h.cnlinux.club/;done

在node节点上功登录harbor,登录成功后的信息保存在当前用户家目录下的.docker/config.json里。

[root@node-06 ~]# docker login h.cnlinux.clubUsername: adminPassword: Login Succeeded[root@node-06 ~]# cat .docker/config.json { ???????"auths": { ???????????????"h.cnlinux.club": { ???????????????????????"auth": "YWRtaW46SGFyYm9yMTIzNDU=" ???????????????} ???????}}

在官方docker仓库pull一个nginx镜像，任何打上tag，push到harbor仓库，如下就可以看到harbor的test项目下已经有nginx的镜像了

[root@node-06 ~]# docker pull nginx:latest[root@node-06 ~]# docker tag nginx:latest h.cnlinux.club/test/nginx:latest[root@node-06 ~]# docker push h.cnlinux.club/test/nginx:latest

问题：如果我的k8s集群很多的node节点是不是每个node节点都要上去登录才能pull harbor仓库的镜像？这样是不是就非常麻烦了？

其实在k8s里有一种secret的类型是kubernetes.io/dockerconfigjson就是用来解决这种问题的。
首先将docker的登录信息转换成base64格式

[root@node-06 ~]# cat .docker/config.json |base64ewoJImF1dGhzIjogewoJCSJoLmNubGludXguY2x1YiI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZTR0Z5WW05eU1USXpORFU9IgoJCX0KCX0sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2VyLUNsaWVudC8xOC4wNi4xLWNlIChsaW51eCkiCgl9Cn0=

创建secret

apiVersion: v1kind: Secretmetadata:name: harbor-registry-secretnamespace: defaultdata:.dockerconfigjson: ewoJImF1dGhzIjogewoJCSJoLmNubGludXguY2x1YiI6IHsKCQkJImF1dGgiOiAiWVdSdGFXNDZTR0Z5WW05eU1USXpORFU9IgoJCX0KCX0sCgkiSHR0cEhlYWRlcnMiOiB7CgkJIlVzZXItQWdlbnQiOiAiRG9ja2VyLUNsaWVudC8xOC4wNi4xLWNlIChsaW51eCkiCgl9Cn0=type: kubernetes.io/dockerconfigjson

[root@node-01 ~]# kubectl create -f harbor-registry-secret.yaml secret/harbor-registry-secret created

创建nginx demo并使用harbor上的nginx镜像。并将nginx.cnlinux.club解析到负载均衡10.31.90.200。

apiVersion: apps/v1kind: Deploymentmetadata: ?name: deploy-nginx ?labels: ???app: nginxspec: ?replicas: 3 ?selector: ???matchLabels: ?????app: nginx ?template: ????metadata: ?????labels: ???????app: nginx ???spec: ?????containers: ?????- name: nginx ???????image: h.cnlinux.club/test/nginx:latest ???????ports: ???????- containerPort: 80 ?????imagePullSecrets: ???????- name: harbor-registry-secret---apiVersion: v1kind: Servicemetadata: ?name: nginxspec: ?selector: ???app: nginx ?ports: ?- name: nginx ???protocol: TCP ???port: 80 ???targetPort: 80 ?type: ClusterIP ---apiVersion: extensions/v1beta1kind: Ingressmetadata: ?name: nginx ?annotations: ???# nginx.ingress.kubernetes.io/rewrite-target: / ???kubernetes.io/ingress.class: nginxspec: ?rules: ???- host: nginx.cnlinux.club ?????http: ???????paths: ?????????- path: ????????????backend: ?????????????serviceName: nginx ?????????????servicePort: 80

可以看到3过node节点上的nginx已经运行，证明harbor上的镜像已经成功pull下来。

[root@node-01 ~]# kubectl get pod -o wide|grep nginx deploy-nginx-647f9649f5-88mkt ?????????????????1/1 ????Running ???????????0 ?????????2m41s ??10.34.0.5 ?????node-06 ??<none> ??????????<none>deploy-nginx-647f9649f5-9z842 ?????????????????1/1 ????Running ???????????0 ?????????2m41s ??10.40.0.5 ?????node-04 ??<none> ??????????<none>deploy-nginx-647f9649f5-w44ck ?????????????????1/1 ????Running ???????????0 ?????????2m41s ??10.46.0.6 ?????node-05 ??<none> ??????????<none>

最后我们访问http://nginx.cnlinux.club,至此所有的都已完成。

后续会陆续更新所有的k8s相关文档，如果你觉得我写的不错，希望大家多多关注点赞，如有问题可以在下面给我留言，非常感谢！

Kubernetes部署（十二）：helm部署harbor企业级镜像仓库

原文地址：http://blog.51cto.com/billy98/2345517

Kubernetes部署（十二）：helm部署harbor企业级镜像仓库

harbor简介

特征

先决条件

Harbor部署

1. 添加域名解析。

2. 下载harbor的chart包

3. 修改配置文件

4. 创建存储卷

5. 安装harbor

6. 测试

知识推荐