Private registry for #kubernetes cluster

Last post I told you how to create a simple registry for your Docker. Now, we need to make it work with our kubernetes cluster and that will require some extra steps: Authentication and Manage some additional parameters in your ReplicationController config.

I am using an update Centos 7 with docker-engine installed.
Most of the info I’ve got it from nikvdp’s post.

Prepare your system

First of all, you have to get your tools. Install apache-utils and docker composer as follow:

yum update -y && yum install httpd httpd-tools -y
curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose


==== Begin Update May 25th, 2017 ==========================
Selinux must be disabled to make it work
use “setenforce 0”
==== End Update May 25th, 2017 ===========================

Create your containers/nginx files

Run the following command to create folder and files

mkdir ~/docker-registry
cd ~/docker-registry
cat << 'EOF' >> docker-compose.yml
nginx:
  image: "nginx:1.9"
  ports:
    - 443:443
  links:
    - registry:registry
  volumes:
    - ./nginx/:/etc/nginx/conf.d:ro
registry:
  image: registry:2
  ports:
    - 127.0.0.1:5000:5000
  environment:
    REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
  volumes:
    - ./data:/data
EOF
mkdir ~/docker-registry/data
mkdir ~/docker-registry/nginx
cd ~/docker-registry/nginx
cat << 'EOF' >> registry.conf
upstream docker-registry {
  server registry:5000;
}

server {
  listen 443;
  server_name registry.nuage.lab;

  # SSL
   ssl on;
   ssl_certificate /etc/nginx/conf.d/domain.crt;
   ssl_certificate_key /etc/nginx/conf.d/domain.key;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location /v2/ {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
      return 404;
    }

    # To add basic authentication to v2 use auth_basic setting plus add_header
     auth_basic "registry.localhost";
     auth_basic_user_file /etc/nginx/conf.d/registry.password;
     add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

    proxy_pass                          http://docker-registry;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}
EOF
cd ~/docker-registry

Create your credentials:

cd ~/docker-registry/nginx
htpasswd -c registry.password pinrojas
# I used password 'pinrojas'

Create your self-signed certificates:

openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout ~/docker-registry/nginx/domain.key \
-x509 -days 365 -out ~/docker-registry/nginx/domain.crt

Test your new registry:

# We'll run it in background
docker-compose up &
curl -k https://pinrojas:pinrojas@registry.nuage.lab/v2/

You will get a “{}” as response.
Check out my test:

[root@docker01 docker-registry]# docker-compose up &
[1] 23866
[root@docker01 docker-registry]# Starting dockerregistry_registry_1
Recreating dockerregistry_nginx_1
Attaching to dockerregistry_registry_1, dockerregistry_nginx_1
registry_1  | time="2016-09-13T21:59:40Z" level=info msg="Starting upload purge in 55m0s" go.version=go1.6.3 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 
registry_1  | time="2016-09-13T21:59:40Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.6.3 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 
registry_1  | time="2016-09-13T21:59:40Z" level=info msg="redis not configured" go.version=go1.6.3 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 
registry_1  | time="2016-09-13T21:59:40Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.6.3 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 
registry_1  | time="2016-09-13T21:59:40Z" level=info msg="listening on [::]:5000" go.version=go1.6.3 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 

[root@docker01 docker-registry]# curl -k https://pinrojas:pinrojas@registry.nuage.lab/v2/
{}[root@docker01 docker-registry]# registry_1  | time="2016-09-13T21:59:53Z" level=info msg="response completed" go.version=go1.6.3 http.request.host=registry.nuage.lab http.request.id=c3dcb0b8-d002-4865-b5dc-d2ab78fd0a87 http.request.method=GET http.request.remoteaddr=10.10.10.16 http.request.uri="/v2/" http.request.useragent="curl/7.29.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4.390192ms http.response.status=200 http.response.written=2 instance.id=89538e7a-60e1-4f60-92b0-0b28a3059e41 version=v2.5.1 
registry_1  | 172.17.0.3 - - [13/Sep/2016:21:59:53 +0000] "GET /v2/ HTTP/1.0" 200 2 "" "curl/7.29.0"
nginx_1     | 10.10.10.16 - pinrojas [13/Sep/2016:21:59:53 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "curl/7.29.0" "-"

Config your kubernetes cluster

Copy your domain.crt into /etc/docker/certs.d/registry.nuage.lab/ca.crt file in all kubernetes nodes in your cluster.

==== Begin Update Sept 22, 2016 ====================
The next step: adding a “–insecure-registry” options to your docker service is not require if you copied the registry’s certificate to all kubernetes nodes. You can skip to set docker login credentials.
===== End Update Sept 22, 2016 =====================

Then change the docker config file adding “–insecure-registry registry.nuage.lab” as follow:

[root@k8snode01 ~]# cat /etc/sysconfig/docker | grep registry.nuage.lab
OPTIONS='--selinux-enabled --log-driver=journald --insecure-registry registry.nuage.lab'

Restart docker: service docker restart
We have to set credential into every node in your cluster as follow:

[root@k8snode01 ~]# docker login registry.nuage.lab
Username: pinrojas
Password: ******
Email: pinrojas@gmail.com
WARNING: login credentials saved in /root/.docker/config.json
Login Succeeded
[root@k8snode01 ~]# cat ~/.docker/config.json 
{
	"auths": {
		"registry.nuage.lab": {
			"auth": "cGlucm9qYXM6cGlucm9qYXM=",
			"email": "pinrojas@gmail.com"
		}
	}
}

You can copy config.json file into ~/.docker/config.json in all nodes to save time.
Finally create your secret image registry

kubectl create secret docker-registry myregistrykey --docker-server=registry.nuage.lab --docker-username=pinrojas --docker-password=pinrojas --docker-email=pinrojas@gmail.com

Don’t forget to use this registry when you create your pods.

Launch my ReplicationCluster using my new private registry

I will use the following yaml file to create a replication controller with 4 replicas of hello-world
The images has been created using the Dockefile and server.js defined in my previous post.

IMPORTANT: I am using my k8s setup with Nuage. Then I have some extra metadata that you may not need.

cat << 'EOF' >> hello-world.yaml
apiVersion: v1
kind: ReplicationController
metadata:
   name: hellow-world-test
spec:
   replicas: 4
   selector:
      app: hello-node
   template:
      metadata:
         name: hellow-world-test
         labels:
            app: hello-node
            nuage.io/subnet: hello01
            nuage.io/zone: hello01

      spec:
         containers:
         - name: hello-world-test
           image: registry.nuage.lab/hello-node:v1
           ports:
           - containerPort: 8080
         imagePullSecrets:
         - name: myregistrykey
EOF

This is how was going:

[root@k8scluster ~]# kubectl create -f hello-world.yaml 
replicationcontroller "hellow-world-test" created
[root@k8scluster ~]# kubectl get all
NAME                        DESIRED      CURRENT       AGE
hellow-world-test           4            4             1m
NAME                        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes                  10.254.0.1   <none>        443/TCP    14d
NAME                        READY        STATUS        RESTARTS   AGE
hellow-world-test-26nl5     1/1          Running       0          1m
hellow-world-test-4kvp2     1/1          Running       0          1m
hellow-world-test-bglyb     1/1          Running       0          1m
hellow-world-test-ifp59     1/1          Running       0          1m
my-nginx-3800858182-r0om7   1/1          Running       2          13d
my-nginx-3800858182-xtffj   1/1          Running       2          13d
[root@k8scluster ~]# kubectl get rc
NAME                DESIRED   CURRENT   AGE
hellow-world-test   4         4         6m
[root@k8scluster ~]# kubectl expose rc hellow-world-test --type="LoadBalancer"
service "hellow-world-test" exposed
[root@k8scluster ~]# kubectl get all
NAME                        DESIRED         CURRENT       AGE
hellow-world-test           4               4             7m
NAME                        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hellow-world-test           10.254.254.91                 8080/TCP   7s
kubernetes                  10.254.0.1      <none>        443/TCP    14d
NAME                        READY           STATUS        RESTARTS   AGE
hellow-world-test-26nl5     1/1             Running       0          7m
hellow-world-test-4kvp2     1/1             Running       0          7m
hellow-world-test-bglyb     1/1             Running       0          7m
hellow-world-test-ifp59     1/1             Running       0          7m
my-nginx-3800858182-r0om7   1/1             Running       2          13d
my-nginx-3800858182-xtffj   1/1             Running       2          13d
[root@k8scluster ~]# curl http://10.254.254.91:8080
Hello World version ONE! Host/Pod: hellow-world-test-bglyb
[root@k8scluster ~]# curl http://10.254.254.91:8080
Hello World version ONE! Host/Pod: hellow-world-test-4kvp2

This is what I’ve got from my Nuage VSD
kubernetes-private-registry-dcoker-nuage-01

See ya!

9 thoughts on “Private registry for #kubernetes cluster

  1. Great article Mauricio,
    Which version of kubernetes are you running ?
    I’m personally running 1.3 and using Deployments and ReplicaSets instead of ReplicationControllers,
    Using RC is a hard request for Nuage+kubernetes ?

  2. Great article Mauricio,
    Which version of kubernetes are you running ?
    I’m personally running 1.3 and using Deployments and ReplicaSets instead of ReplicationControllers,
    Using RC is a hard request for Nuage+kubernetes ?

    1. Hi yazpik. I am installing everything over again. Just give a couple of hours to let you know what I will finally use. I haven’t issues with ReplicationControllers and Nuage. Why do you prefer Deployments and ReplicaSets over RCs?

      1. AFAIK, Basically Deployment is a higher level or new generation of ReplicationControllers since 1.2, you can manage Replica Sets as well as Services you can rollback between versions and/or replicas to meet the desired state very easily.
        I would say rolling-update of the replication controller is cooler than deployments because you see how kubernetes create/destroy pods, but rolling-update is running as a client tool “client-server”, now the approach is changing using deployments running all the instructions on the cluster.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s