ClickHouse 이중화(HA) 구성 가이드

아키텍처

테스트 환경: onprem - docker

Docker Compose 설정

docker-compose.yml

yaml
clickhouse:
  <<: *clickhouse-defaults
  container_name: signoz-clickhouse
  hostname: clickhouse
  user: ${USER_UID}:${USER_UID}
  ports:
    - "9000:9000"
    - "8123:8123"
    - "9181:9181"
  volumes:
    - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
    - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
    - ./custom-function.xml:/etc/clickhouse-server/custom-function.xml
    - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
    - ${DATA}/clickhouse/:/var/lib/clickhouse/
  sysctls:
    - net.ipv6.conf.all.disable_ipv6=1

clickhouse-2:
  <<: *clickhouse-defaults
  container_name: signoz-clickhouse-2
  hostname: clickhouse-2
  user: ${USER_UID}:${USER_UID}
  ports:
    - "9001:9000"
    - "8124:8123"
    - "9182:9181"
  volumes:
    - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
    - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
    - ./custom-function.xml:/etc/clickhouse-server/custom-function.xml
    - ./clickhouse-cluster-node2.xml:/etc/clickhouse-server/config.d/cluster.xml# 다른 설정 파일
    - ./data/clickhouse-2/:/var/lib/clickhouse/
  sysctls:
    - net.ipv6.conf.all.disable_ipv6=1

---

클러스터 설정 파일

clickhouse-cluster.xml (Node 1용)

xml
<?xml version="1.0"?>
<clickhouse>
    <zookeeper>
        <node index="1">
            <host>zookeeper-1</host><!-- 실제 ZooKeeper 컨테이너 이름 -->
            <port>2181</port>
        </node>
    </zookeeper>

    <remote_servers>
        <cluster>
            <shard>
                <internal_replication>true</internal_replication><!-- 복제 활성화 -->
                <replica>
                    <host>clickhouse</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-2</host>
                    <port>9000</port>
                </replica>
            </shard><!-- 같은 샤드에 두 복제본 -->
        </cluster>
    </remote_servers>

<!-- 첫 번째 노드용 매크로 -->
    <macros>
        <cluster>cluster</cluster>
        <shard>01</shard>
        <replica>replica_01</replica><!-- 고유한 복제본 ID -->
    </macros>
</clickhouse>

clickhouse-cluster-node2.xml (Node 2용)

xml
<?xml version="1.0"?>
<clickhouse>
    <zookeeper>
        <node index="1">
            <host>zookeeper-1</host>
            <port>2181</port>
        </node>
    </zookeeper>

    <remote_servers>
        <cluster>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>clickhouse</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-2</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster>
    </remote_servers>

<!-- 두 번째 노드용 매크로 -->
    <macros>
        <cluster>cluster</cluster>
        <shard>01</shard>
        <replica>replica_02</replica><!- 다른 복제본 ID -->
    </macros>
</clickhouse>

---

테스트 및 검증

1. 기본 연결 테스트

# Node 1 연결 확인
curl "http://localhost:8123/?query=SELECT%20'Node1'%20as%20node"
Node1

# Node 2 연결 확인
curl "http://localhost:8124/?query=SELECT%20'Node2'%20as%20node"
Node2

2. 클러스터 상태 확인

curl "http://localhost:8123/?query=SELECT%20*%20FROM%20system.clusters%20FORMAT%20Pretty"

성공적인 결과 예시:

┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ cluster ┃ shard_num ┃ shard_weight ┃ internal_replication ┃ replica_num ┃ host_name    ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ cluster │         1 │            1 │                    1 │           1 │ clickhouse   │
├─────────┼───────────┼──────────────┼──────────────────────┼─────────────┼──────────────┤
│ cluster │         1 │            1 │                    1 │           2 │ clickhouse-2 │
└─────────┴───────────┴──────────────┴──────────────────────┴─────────────┴──────────────┘

🔑 성공 지표:

• shard_num: 1, 1 (같은 샤드)
• replica_num: 1, 2 (두 개의 복제본)
• internal_replication: 1 (복제 활성화)

3. 복제 테이블 테스트

3-1. 복제 테이블 생성

# 복제 테이블 생성 (ON CLUSTER 사용)
curl -X POST "http://localhost:8123/" -d "
CREATE TABLE test_replication ON CLUSTER cluster
(
    id UInt32,
    message String,
    created DateTime,
    node_info String DEFAULT hostName()
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test_replication', '{replica}')
ORDER BY id"

성공 시 응답:

clickhouse-2    9000    0        1    0
clickhouse    9000    0        0    0

실패 시 응답 예시:

Code: 253. DB::Exception: There was an error on [clickhouse-2:9000]: Code: 253. DB::Exception: Replica /clickhouse/tables/01/test_replication/replicas/replica_02 already exists. (REPLICA_ALREADY_EXISTS)

3-2. 테이블 생성 확인

bash
# 양쪽 노드에서 테이블 확인
curl "http://localhost:8123/?query=SELECT%20name,%20engine%20FROM%20system.tables%20WHERE%20name='test_replication'%20FORMAT%20Pretty"
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ name             ┃ engine              ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ test_replication │ ReplicatedMergeTree │
└──────────────────┴─────────────────────┘

curl "http://localhost:8124/?query=SELECT%20name,%20engine%20FROM%20system.tables%20WHERE%20name='test_replication'%20FORMAT%20Pretty"
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ name             ┃ engine              ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ test_replication │ ReplicatedMergeTree │
└──────────────────┴─────────────────────┘

3-3. 데이터 삽입 테스트

bash
# Node 1에 데이터 삽입
curl -X POST "http://localhost:8123/" -d "
INSERT INTO test_replication (id, message, created) VALUES
(1, 'Hello from Node 1', now()),
(2, 'Test replication', now()),
(3, 'ClickHouse HA', now())"

3-4. 복제 확인

# Node 1에서 데이터 조회
curl "http://localhost:8123/?query=SELECT%20*%20FROM%20test_replication%20ORDER%20BY%20id%20FORMAT%20Pretty"
┏━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ id ┃ message           ┃             created ┃ node_info  ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│  1 │ Hello from Node 1 │ 2025-06-13 05:58:32 │ clickhouse │
├────┼───────────────────┼─────────────────────┼────────────┤
│  2 │ Test replication  │ 2025-06-13 05:58:32 │ clickhouse │
├────┼───────────────────┼─────────────────────┼────────────┤
│  3 │ ClickHouse HA     │ 2025-06-13 05:58:32 │ clickhouse │
└────┴───────────────────┴─────────────────────┴────────────┘

# Node 2에서 데이터 조회 (자동 복제 확인)
curl "http://localhost:8124/?query=SELECT%20*%20FROM%20test_replication%20ORDER%20BY%20id%20FORMAT%20Pretty"
┏━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ id ┃ message           ┃             created ┃ node_info  ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│  1 │ Hello from Node 1 │ 2025-06-13 05:58:32 │ clickhouse │
├────┼───────────────────┼─────────────────────┼────────────┤
│  2 │ Test replication  │ 2025-06-13 05:58:32 │ clickhouse │
├────┼───────────────────┼─────────────────────┼────────────┤
│  3 │ ClickHouse HA     │ 2025-06-13 05:58:32 │ clickhouse │
└────┴───────────────────┴─────────────────────┴────────────┘

3-5. 복제 상태 확인

# 복제 상태 상세 확인
curl "http://localhost:8123/?query=SELECT%20*%20FROM%20system.replicas%20WHERE%20table='test_replication'%20FORMAT%20Pretty"

 

4. 장애 복구 테스트

# Node 1 중지
docker stop signoz-clickhouse

# Node 2로 쿼리 (이중화 확인)
curl "http://localhost:8124/?query=SELECT%20count()%20FROM%20system.databases"

# Node 1 재시작
docker start signoz-clickhouse

추가 모니터링 명령어

curl "[http://localhost:8123/?query=SELECT * FROM system.clusters FORMAT Pretty](http://localhost:8123/?query=SELECT%20*%20FROM%20system.clusters%20FORMAT%20Pretty)"
curl "[http://localhost:8123/?query=SELECT * FROM system.replicas FORMAT Pretty](http://localhost:8123/?query=SELECT%20*%20FROM%20system.replicas%20FORMAT%20Pretty)"
curl "[http://localhost:8123/?query=SELECT * FROM system.replication_queue FORMAT Pretty](http://localhost:8123/?query=SELECT%20*%20FROM%20system.replication_queue%20FORMAT%20Pretty)"