Docker deployment of high-availability etcd cluster practice

Introduction to Etcd

etcd is a highly consistent distributed key-value store, which provides a reliable way to store data that needs to be accessed by distributed systems or machine clusters. It handles leader election gracefully during network partitioning and can tolerate machine failures, even in the leader node

environment

• docker

• docker-compose

• docker swarm (optional)

• python3

• pipenv

• pip3

Configuration

nginx

Need to create an upstream part and declare our etcd server name

upstream etcd_servers {
        least_conn;
        server etcd-00:2379 max_fails=3 fail_timeout=5s;
        server etcd-01:2379 max_fails=3 fail_timeout=5s;
        server etcd-02:2379 max_fails=3 fail_timeout=5s;
}

Note: Since we are running on docker, the server name is resolved by internal dns. name is the name of the service declared in docker-compose.yml (etcd-00, etcd-01, etcd-02)

We need to declare the server part, use the listening port (etcd default listening port) and proxy_pass rules to route traffic to our etcd service

server {
    listen 2379;
    proxy_pass etcd_servers;
}

Final configuration

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    log_format basic'$time_iso8601 $remote_addr'
                        '$protocol $status $bytes_sent $bytes_received '
                        '$session_time $upstream_addr'
                        '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log /dev/stdout basic;

    upstream etcd_servers {
        least_conn;
        server etcd-00:2379 max_fails=3 fail_timeout=5s;
        server etcd-01:2379 max_fails=3 fail_timeout=5s;
        server etcd-02:2379 max_fails=3 fail_timeout=5s;
    }
    server {
        listen 2379;
        proxy_pass etcd_servers;
    }
}

Etcd

To configure etcd in cluster mode, we need to specify the following settings on each container

command:
      -etcd
      ---name=etcd-02
      ---data-dir=data.etcd
      ---advertise-client-urls=http://etcd-02:2379
      ---listen-client-urls=http://0.0.0.0:2379
      ---initial-advertise-peer-urls=http://etcd-02:2380
      ---listen-peer-urls=http://0.0.0.0:2380
      ---initial-cluster=etcd-00=http://etcd-00:2380,etcd-01=http://etcd-01:2380,etcd-02=http://etcd-02:2380
      ---initial-cluster-state=new
      ---initial-cluster-token=etcd-cluster-1

• --name: The human readable name of this member.
• --data-dir: The path of the data directory.
• --advertise-client-urls: A list of client URLs for this member, used to advertise to the rest of the cluster. These URLs can include domain names.
• --listen-client-urls: List of URLs to monitor client traffic. This flag tells etcd to accept incoming requests from the client with the specified scheme://IP:port combination.
• --initial-advertise-peer-urls: A list of peer URLs for this member, used to advertise to the rest of the cluster. These addresses are used to communicate etcd data around the cluster. At least one must be routable to all cluster members. These URLs can include domain names.
• --listen-peer-urls: List of URLs to listen to peer-to-peer traffic. This flag tells etcd to accept incoming requests from peers on the specified scheme://IP:port combination.
• --initial-cluster: the initial cluster configuration for booting.
• --initial-cluster-state: Initial cluster state ("new" or "existing"). Set all members that existed during the initial static or DNS boot to new. If this option is set to existing, etcd will try to join the existing cluster. If the wrong value is set, etcd will try to start but fail safely.
• --initial-cluster-token: the initial cluster token of etcd cluster during boot.

docker compose deploy etcd cluster

1. Clone this repository

git clone https://github.com/elephdev/docker-etcd-cluster.git

2. Start

cd docker-etcd-cluster
docker-compose up -d

3. Check the environment status and wait for the container to be ready

docker-compose ps


     Name Command State Ports
-------------------------------------------------- -------------------------------------------------- ------
etcd_etcd-00_1 etcd --name=etcd-00 --data ... Up 2379/tcp, 2380/tcp
etcd_etcd-01_1 etcd --name=etcd-01 --data ... Up 2379/tcp, 2380/tcp
etcd_etcd-02_1 etcd --name=etcd-02 --data ... Up 2379/tcp, 2380/tcp
etcd_nginx_1 /docker-entrypoint.sh ngin ... Up 0.0.0.0:2379->2379/tcp,:::2379->2379/tcp, 80/tcp

test environment

Need to install pipenv environment

pipenv shell
pip install -r requirements.txt
python test/etcd-test.py
hey key1
hey key2

View the log of the nginx service, view the traffic redirected to the etcd host

docker-compose logs -f nginx

Attaching to etcd_nginx_1
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx_1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx_1 | 2021-10-08T09:41:23+00:00 172.28.0.1 TCP 200 422 665 0.052 172.28.0.3:2379 "665" "422" "0.000"
nginx_1 | 2021-10-08T09:41:24+00:00 172.28.0.1 TCP 200 422 665 0.046 172.28.0.2:2379 "665" "422" "0.000"
nginx_1 | 2021-10-08T09:50:56+00:00 172.28.0.1 TCP 200 422 665 0.029 172.28.0.4:2379 "665" "422" "0.000"

Docker swarm stack

docker stack deploy -c etcd-stack.yml etcd

Check deployment status

docker stack ps etcd

mx6fvfwye547 etcd_etcd-00.1 quay.io/coreos/etcd:v3.5.0 node-2 Running Running 3hours ago
wybd7n4oitae etcd_etcd-01.1 quay.io/coreos/etcd:v3.5.0 node-4 Running Running 3 hours ago
rmlycc3uvc8t etcd_etcd-02.1 quay.io/coreos/etcd:v3.5.0 node-2 Running Running 3 hours ago
rexh1smoalpo etcd_nginx.1 nginx:alpine node-2 Running Running 21 hours ago

docker service ls

ID NAME MODE REPLICAS IMAGE PORTS
1u709kzgmo2b etcd_etcd-00 replicated 1/1 quay.io/coreos/etcd:v3.5.0
m7ze76xi58ww etcd_etcd-01 replicated 1/1 quay.io/coreos/etcd:v3.5.0
1535r562g3az etcd_etcd-02 replicated 1/1 quay.io/coreos/etcd:v3.5.0
v8n8qlo3dm30 etcd_nginx replicated 1/1 nginx:alpine *:2379->2379/tcp

If you want to test swarm settings, please open test/etcd-test.py and change 127.0.0.1 to an ip (or LB ip) of the server in the docker swarm cluster and run the test