OKD-baremetal

사용된 서버 사양

okd 4.10

1. DNS 설정 (bastion server)

1) bind install

yum install -y bind bind-utils

2) zone 등록

cat <<EOF >> /etc/named.rfc1912.zones

zone "test.domain.co.kr" IN {
        type master;                                                            
        file "test.domain.co.kr.zone";
        allow-update { none; };
};

zone "100.168.192.in-addr.arpa" IN {
        type master;
        file "/var/named/reverse.test.domain.co.kr.zone";
        allow-update { none; } ;
};
EOF

3) 정방향 DNS

cat <<EOF >> /var/named/test.domain.co.kr.zone
$TTL 900D
@   IN SOA  @ ns.test.domain.co.kr. (
                    20230908   ; serial
                    1D  ; refresh
                    1H  ; retry 
                    999W  ; expire
                    3H )    ; minimum

                    IN NS   ns.test.domain.co.kr
                    IN A    192.168.100.200

; Bastion or Jumphost
bastion.test.domain.co.kr.IN A 192.168.100.200
registry.test.domain.co.kr.IN A 192.168.100.201
ns.test.domain.co.kr.      IN A    192.168.100.200
test.domain.co.kr     IN A    192.168.100.200
; Ancillary services
lb.test.domain.co.kr. IN A 192.168.100.200

;ocp cluster
bootstrap.test.domain.co.kr.IN  A  192.168.100.202

master1.test.domain.co.kr.IN  A   192.168.100.203
master2.test.domain.co.kr.IN  A   192.168.100.204
master3.test.domain.co.kr.IN  A   192.168.100.205

worker1.test.domain.co.kr.IN  A   192.168.100.206
worker2.test.domain.co.kr.IN  A   192.168.100.207

infra1.test.domain.co.kr.IN  A   192.168.100.208
infra2.test.domain.co.kr.IN  A   192.168.100.209

;ocp internal cluster ip
etcd-0  IN A    192.168.100.203
etcd-1  IN A    192.168.100.204
etcd-2  IN A    192.168.100.205

api-int         IN A 192.168.100.200
api             IN A 192.168.100.200
*.apps          IN A 192.168.100.200
apps            IN A 192.168.100.200

_etcd-server-ssl._tcp.test.domain.co.kr. IN SRV 0 10 2380 etcd-0.test.domain.co.kr.
_etcd-server-ssl._tcp.test.domain.co.kr. IN SRV 0 10 2380 etcd-1.test.domain.co.kr.
_etcd-server-ssl._tcp.test.domain.co.kr. IN SRV 0 10 2380 etcd-2.test.domain.co.kr.
EOF

4) 역방향 DNS

cat <<EOF >> /var/named/reverse.test.domain.co.kr.zone
$TTL 900D
@   IN SOA  @ ns.test.domain.co.kr. (
            4020154001  ; serial
            3H          ; refresh
            1H          ; retry
            999W          ; expiry
            1H )        ; minimum

@           IN NS       ns.test.domain.co.kr.
            IN A        192.168.100.200

200.100.168.192.in-addr.arpa. IN  PTR  test.domain.co.kr.

200.100.168.192.in-addr.arpa. IN  PTR  bastion.test.domain.co.kr.
200.100.168.192.in-addr.arpa. IN  PTR  lb.test.domain.co.kr.

202.100.168.192.in-addr.arpa. IN  PTR  bootstrap.test.domain.co.kr.
203.100.168.192.in-addr.arpa. IN  PTR  master1.test.domain.co.kr. 
204.100.168.192.in-addr.arpa. IN  PTR  master2.test.domain.co.kr. 
205.100.168.192.in-addr.arpa. IN  PTR  master3.test.domain.co.kr. 

206.100.168.192.in-addr.arpa. IN  PTR  worker1.test.domain.co.kr. 
207.100.168.192.in-addr.arpa. IN  PTR  worker2.test.domain.co.kr. 
208.100.168.192.in-addr.arpa. IN  PTR  infra1.test.domain.co.kr. 
209.100.168.192.in-addr.arpa. IN  PTR  infra2.test.domain.co.kr. 

EOF

정방향 역방향 파일 생성 후, 해당 파일 링크로 접속해서 vi 편집기로 열면, $TTL 이 누락되어 작성되는 경우가 있으니 꼭 한 번 열어서 확인할 것.

5) 생성한 zone 파일의 권한을 변경

chown root.named /var/named/test.domain.co.kr.zone
chown root.named /var/named/reverse.test.domain.co.kr.zone

6) zone 파일이 문법적으로 맞는지 확인

named-checkzone test.domain.co.kr /var/named/test.domain.co.kr.zone
named-checkzone 107.1.0.10.in-addr.arpa /var/named/reverse.test.domain.co.kr.zone

#확인 결과 
OK

7) /etc/named.conf 파일 수정

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { none; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { any; };

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";

        /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
        include "/etc/crypto-policies/back-ends/bind.config";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

8) /etc/resolv.conf 수정

9) named 데몬 재시작

systemctl restart named

10) nslookup으로 확인

vi /etc/hosts

systemctl restart dnsmasq

dig fedoraCoreOS2

dig -x 10.0.1.103

부트 시간

vi /etc/default/grub

설정 적용후 재부팅

grub2-mkconfig -o /boot/grub2/grub.cfg

reboot

2. LoadBalancer - HAploxy (bastion server)

Master와 bootstrap은 6443, 22623 port 등록
•Worker와 infra는 80, 443 port 등록

1) 필요한 util 다운로드

yum -y install haproxy

2) /etc/haproxy/haproxy.cfg 파일 수정

vi /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   https://www.haproxy.org/download/1.8/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
#    ssl-default-bind-ciphers PROFILE=SYSTEM
#    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 4000

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
frontend openshift-api-server
        bind *:6443
        default_backend openshift-api-server
        mode tcp
        option tcplog

backend openshift-api-server
        balance source
        mode tcp
        server bootstrap 192.168.100.202:6443 check
        server master1.test.domain.co.kr. 192.168.100.203:6443 check
        server master2.test.domain.co.kr. 192.168.100.204:6443 check
        server master3.test.domain.co.kr. 192.168.100.205:6443 check

frontend machine-config-server
        bind *:22623
        default_backend machine-config-server
        mode tcp
        option tcplog

backend machine-config-server
        balance source
        mode tcp
        server bootstrap 192.168.100.202:22623 check
        server master1.test.domain.co.kr. 192.168.100.203:22623 check
        server master2.test.domain.co.kr. 192.168.100.204:22623 check
        server master3.test.domain.co.kr. 192.168.100.205:22623 check

frontend ingress-http
        bind *:80
        default_backend ingress-http
        mode tcp
        option tcplog

backend ingress-http
        balance source
        mode tcp
        server infra1.test.domain.co.kr. 192.168.100.208:80 check
        server infra2.test.domain.co.kr. 192.168.100.209:80 check
        server worker1.test.domain.co.kr. 192.168.100.206:80 check
        server worker2.test.domain.co.kr. 192.168.100.207:80 check

frontend ingress-https
        bind *:443
        default_backend ingress-https
        mode tcp
        option tcplog

backend ingress-https
        balance source
        mode tcp
        server infra1.test.domain.co.kr. 192.168.100.208:443 check
        server infra2.test.domain.co.kr. 192.168.100.209:443 check
        server worker1.test.domain.co.kr. 192.168.100.206:443 check
        server worker2.test.domain.co.kr. 192.168.100.207:443 check

config파일 확인

haproxy -f /etc/haproxy/haproxy.cfg -c

haproxy 서비스 시작

systemctl enable haproxy.service --now
systemctl status haproxy.service

3. Install (bastion server)

1) okd 파일을 설치하고, /usr/local/bin 폴더에 압축을 풀어준다.

wget https://github.com/okd-project/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz
wget https://github.com/okd-project/okd/releases/download/4.10.0-0.okd-2022-07-09-073606/openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz

tar -xvf openshift-client-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz -C /usr/local/bin/
tar -xvf openshift-install-linux-4.10.0-0.okd-2022-07-09-073606.tar.gz -C /usr/local/bin/

2) ssh 키 생성

ssh-keygen -t rsa -b 4096

3) Ignition file을 만들기 위한 install-config.yaml 작성

cd /var/www/html/
mkdir okd4
vi install-config.yaml

pullSecret: ‘’ > redhat에서
sshKey: '’ > ssh-keygen 명령 생성하고 나면

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is: ...

이렇게 뜨는데, /root/.ssh/id_rsa.pub 경로에 가서 공개키를 복사해서 sshKey: 항목에 붙여준다.

apiVersion: v1
baseDomain: test.domain.co.kr
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3     
metadata:
  name: tests
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
  - 192.168.100.0/24
platform:
  none: {}
fips: false
pullSecret: '{"auths":{"cloud.opeshift.com":{"auth":"b3Blbn...}}}'
sshKey: 'ssh-ed25519 AAAA...'

4) openshift를 설치한 폴더에서 manifests 명령어 수행

—dir={ install-config.yaml 이 존재하는 폴더 경로 }
manifests 명령 수행되고 나면, 작성했던 install-config.yaml은 삭제되므로 미리 백업해 둘 것

./openshift-install create manifests --dir=/root/okd4

생성된 manifest 확인

#cd openshift
-rw-r-----. 1 root root  181 Sep 11 07:27 99_kubeadmin-password-secret.yaml
-rw-r-----. 1 root root 2466 Sep 11 07:27 99_openshift-cluster-api_master-user-data-secret.yaml
-rw-r-----. 1 root root 2466 Sep 11 07:27 99_openshift-cluster-api_worker-user-data-secret.yaml
-rw-r-----. 1 root root 1169 Sep 11 07:27 99_openshift-machineconfig_99-master-ssh.yaml
-rw-r-----. 1 root root 1169 Sep 11 07:27 99_openshift-machineconfig_99-worker-ssh.yaml
-rw-r-----. 1 root root  237 Sep 11 07:27 openshift-install-manifests.yaml

#cd manifest
-rw-r-----. 1 root root 1544 Sep 11 07:27 cluster-config.yaml
-rw-r-----. 1 root root  155 Sep 11 07:27 cluster-dns-02-config.yml
-rw-r-----. 1 root root  505 Sep 11 07:27 cluster-infrastructure-02-config.yml
-rw-r-----. 1 root root  160 Sep 11 07:27 cluster-ingress-02-config.yml
-rw-r-----. 1 root root 9607 Sep 11 07:27 cluster-network-01-crd.yml
-rw-r-----. 1 root root  276 Sep 11 07:27 cluster-network-02-config.yml
-rw-r-----. 1 root root  142 Sep 11 07:27 cluster-proxy-01-config.yaml
-rw-r-----. 1 root root  170 Sep 11 07:27 cluster-scheduler-02-config.yml
-rw-r-----. 1 root root  262 Sep 11 07:27 cvo-overrides.yaml
-rw-r-----. 1 root root  118 Sep 11 07:27 kube-cloud-config.yaml
-rw-r-----. 1 root root 1304 Sep 11 07:27 kube-system-configmap-root-ca.yaml
-rw-r-----. 1 root root 4054 Sep 11 07:27 machine-config-server-tls-secret.yaml
-rw-r-----. 1 root root 3841 Sep 11 07:27 openshift-config-secret-pull-secret.yaml

5) ignition 파일 생성

./openshift-install create ignition-configs --dir=/root/okd4

생성된 .ign 파일 확인

├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign

drwxr-x---. 2 root root     50 Sep 11 07:31 auth
-rwxr-xr-x. 1 root root 278442 Sep 11 07:31 bootstrap.ign
-rw-r--r--. 1 root root   3924 Sep 11 07:26 install-config.yaml.bak
-rwxr-xr-x. 1 root root   1728 Sep 11 07:31 master.ign
-rw-r-----. 1 root root     98 Sep 11 07:31 metadata.json
-rwxr-xr-x. 1 root root   1728 Sep 11 07:31 worker.ign

6) 생성된 ignition 파일을 파일 서버 디렉토리로 복사하고 다운로드 가능하도록 퍼미션 조정

# cp -arp *.ign /var/www/html/

# chmod 644 /var/www/html/*

7) curl 확인

curl http://192.168.100.200:8080/okd4
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

4. bootstrap 설치

(bastion server에서)

1) ign 검증하는 hash값 파일로 전환하기

추후 bootstrap.ign 검증을 위해 해시값이 필요한데, 해시값의 길이가 길어 타이핑이 어려우므로 bootstrap.hash 파일에 해시값 넣습니다.

# cd  /var/www/html/

# sha512sum bootstrap.ign |awk {'print $1'} > bootstrap.hash

# sha512sum master.ign |awk {'print $1'} > master.hash

# sha512sum worker.ign |awk {'print $1'} > worker.hash

./openshift-install coreos print-stream-json | grep '\.iso[^.]' | grep x86_64
                                "location": "https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/x86_64/fedora-coreos-35.20220327.3.0-live.x86_64.iso",

(bootstrap server에서)

$ hash=`curl http://192.168.100.200:8080/bootstrap.hash`
$ sudo coreos-installer install --copy-network --ignition-url http://115.68.142.99:8080/bootstrap.ign /dev/sda --ignition-hash sha512-${hash}

hash=`curl http://10.0.1.107:8080/master.hash`
sudo coreos-installer install /dev/vdc --ignition-url http://10.0.1.107:8080/master.ign --ignition-hash sha512-${hash} --copy-network  --insecure-ignition

hash=`curl http://10.0.1.107:8080/worker.hash`
sudo coreos-installer install /dev/vdc --ignition-url http://10.0.1.107:8080/worker.ign --ignition-hash sha512-${hash} --copy-network  --insecure-ignition

[root@bootstrap ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0   99M  0 part /boot/efi
├─vda2 252:2    0 1000M  0 part /boot
├─vda3 252:3    0    4M  0 part 
├─vda4 252:4    0    1M  0 part 
└─vda5 252:5    0 98.9G  0 part /
vdb    252:16   0   10G  0 disk /mnt
vdc    252:32   0   10G  0 disk 
[root@bootstrap ~]# ./coreos-installer_amd64 install --copy-network --ignition-url http://192.168.100.200:8080/bootstrap.ign /dev/vdc --ignition-hash sha512-${hash}
Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature
> Read disk 618.5 MiB/618.5 MiB (100%)   
gpg: Signature made Tue 05 Sep 2023 10:42:45 AM KST
gpg:                using RSA key 6A51BBABBA3D5467B6171221809A8D7CEB10B464
gpg: Good signature from "Fedora (38) <fedora-38-primary@fedoraproject.org>" [ultimate]

Writing Ignition config
Copying networking configuration from /etc/NetworkManager/system-connections/

Note: detected other devices with a filesystem labeled `boot`:
  - /dev/vda2
The installed OS may not work correctly if there are multiple boot filesystems.
Before rebooting, investigate whether these filesystems are needed and consider
wiping them with `wipefs -a`.

Install complete.

드라이브에 ISO 파일 마운트

sudo dd if= of=/dev/sr0 bs=4M status=progress

#sudo dd if=<ISO_FILE_PATH> of=/dev/sr0 bs=4M status=progress

sudo dd if=fedora-coreos-38.20230819.3.0-live.x86_64.iso of=/dev/vdc bs=4M status=progress

해당 작업(Master 구성)까지 ignition file 생성 기준으로 24시간 안에 해야함

./coreos-installer_amd64 iso kargs modify fedora-coreos-38.20230819.3.0-live.x86_64.iso --append ip=10.0.1.75::10.0.1.1:255.255.255.0:bootstrap.test.domain.co.kr:eth0:none --append nameserver=10.0.1.107 --append coreos.inst.install_dev=vdc --append coreos.inst.ignition_url=http://10.0.1.107:8080/bootstrap.ignls

[root@bootstrap ~]# ll
total 789412
-rwxrwxrwx. 1 root root  13525985 Sep 12 09:48 coreos-installer_amd64
-rw-r--r--. 1 root root 794820608 Sep  5 10:38 fedora-coreos-38.20230819.3.0-live.x86_64.iso
-rw-------. 1 root root      6330 May 18 11:55 original-ks.cfg
[root@bootstrap ~]# ./coreos-installer_amd64 iso kargs show
error: The following required arguments were not provided:
    <ISO>

USAGE:
    coreos-installer_amd64 iso kargs show [OPTIONS] <ISO>

For more information try --help
[root@bootstrap ~]# ./coreos-installer_amd64 iso kargs show fedora-coreos-38.20230819.3.0-live.x86_64.iso 
mitigations=auto,nosmt coreos.liveiso=fedora-coreos-38.20230819.3.0 ignition.firstboot ignition.platform.id=metal
[root@bootstrap ~]# nmctl d
-bash: nmctl: command not found
[root@bootstrap ~]# nmcli d
DEVICE  TYPE      STATE      CONNECTION  
eth0    ethernet  connected  System eth0 
lo      loopback  unmanaged  --          
[root@bootstrap ~]# l;;
-bash: syntax error near unexpected token `;;'
[root@bootstrap ~]# ;;
-bash: syntax error near unexpected token `;;'
[root@bootstrap ~]# ll
total 789412
-rwxrwxrwx. 1 root root  13525985 Sep 12 09:48 coreos-installer_amd64
-rw-r--r--. 1 root root 794820608 Sep  5 10:38 fedora-coreos-38.20230819.3.0-live.x86_64.iso
-rw-------. 1 root root      6330 May 18 11:55 original-ks.cfg
[root@bootstrap ~]# ./coreos-installer_amd64 iso kargs modify fedora-coreos-38.20230819.3.0-live.x86_64.iso --append ip=10.0.1.75::10.0.1.1:255.255.255.0:bootstrap.test.domain.co.kr:System eth0:none --append nameserver=10.0.1.107 --append coreos.inst.install_dev=vdc --append coreos.inst.ignition_url=http://10.0.1.107:8080/bootstrap.ign
error: Found argument 'eth0:none' which wasn't expected, or isn't valid in this context

USAGE:
    coreos-installer_amd64 iso kargs modify [OPTIONS] <ISO>

For more information try --help
[root@bootstrap ~]# ./coreos-installer_amd64 iso kargs modify fedora-coreos-38.20230819.3.0-live.x86_64.iso --append ip=10.0.1.75::10.0.1.1:255.255.255.0:bootstrap.test.domain.co.kr:eth0:none --append nameserver=10.0.1.107 --append coreos.inst.install_dev=vdc --append coreos.inst.ignition_url=http://10.0.1.107:8080/bootstrap.ign
[root@bootstrap ~]# ./coreos-installer_amd64 iso kargs show fedora-coreos-38.20230819.3.0-live.x86_64.iso 
mitigations=auto,nosmt coreos.liveiso=fedora-coreos-38.20230819.3.0 ignition.firstboot ignition.platform.id=metal ip=10.0.1.75::10.0.1.1:255.255.255.0:bootstrap.test.domain.co.kr:eth0:none nameserver=10.0.1.107 coreos.inst.install_dev=vdc coreos.inst.ignition_url=http://10.0.1.107:8080/bootstrap.ign
[root@bootstrap ~]# ls
coreos-installer_amd64  fedora-coreos-38.20230819.3.0-live.x86_64.iso  original-ks.cfg
[root@bootstrap ~]# hash=`curl http://192.168.100.200:8080/bootstrap.hash`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  12900      0 --:--:-- --:--:-- --:--:-- 21500
[root@bootstrap ~]# ./coreos-installer_amd64 install /dev/vdc --ignition-hash sha512-${hash}
Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature
> Read disk 306.6 MiB/618.5 MiB (49%)   

Error: decoding and writing image

Caused by:
    0: request or response body error: error reading a body from connection: Connection reset by peer (os error 104)
    1: error reading a body from connection: Connection reset by peer (os error 104)
    2: Connection reset by peer (os error 104)

Resetting partition table
Error: install failed
[root@bootstrap ~]#./coreos-installer_amd64 install /dev/vdc --ignition-hash sha512-${hash}

참고

방화벽에서 80, 53번 포트 오픈 및 확인

[root]# firewall-cmd --permanent --zone=public --add-service=dns
[root]]# firewall-cmd --permanent --add-port=80/tcp
[root]# firewall-cmd --reload
[root]# firewall-cmd --list-all

참고 블로그 및 페이지

공식 문서 https://docs.okd.io/4.10/installing/installing_bare_metal/installing-bare-metal.html

운영체제 iso https://fedoraproject.org/coreos/download/?stream=stable

마루님 https://maru1000.tistory.com/73

호롤리님 https://gruuuuu.github.io/ocp/ocp4.7-restricted/

서버나라 개발왕자 https://sysdocu.tistory.com/1765

ㅏㅏsudo systemctl reset-failed
- ddd
- [core@fedoraCoreOS2 ~]$ sudo systemctl list-units --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● afterburn-sshkeys@core.service loaded failed failed Afterburn (SSH Keys) LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. [core@fedoraCoreOS2 ~]$ unit -bash: unit: command not found [core@fedoraCoreOS2 ~]$ [core@fedoraCoreOS2 ~]$ [core@fedoraCoreOS2 ~]$ [core@fedoraCoreOS2 ~]$ afterburn Error: error: the following required arguments were not provided: <--cmdline|--provider <name>> Usage: afterburn multi <--cmdline|--provider <name>> For more information, try '--help'. [core@fedoraCoreOS2 ~]$ afterburn --helt Error: error: unexpected argument '--helt' found tip: a similar argument exists: '--help' Usage: afterburn multi --help <--cmdline|--provider <name>> <--provider <name>|--cmdline|--attributes <path>|--check-in|--hostname <path>|--network-units <path>|--ssh-keys <username>|--legacy-cli> For more information, try '--help'. [core@fedoraCoreOS2 ~]$ afterburn --help Perform multiple tasks in a single call Usage: afterburn multi [OPTIONS] <--cmdline|--provider <name>> Options: --provider <name> The name of the cloud provider --cmdline Read the cloud provider from the kernel cmdline --attributes <path> The file into which the metadata attributes are written --check-in Check-in this instance boot with the cloud provider --hostname <path> The file into which the hostname should be written --network-units <path> The directory into which network units are written --ssh-keys <username> Update SSH keys for the given user -h, --help Print help -V, --version Print version [core@fedoraCoreOS2 ~]$ afterburn --version Afterburn-multi 5.4.2 [core@fedoraCoreOS2 ~]$ systemctl status core.service Unit core.service could not be found. [core@fedoraCoreOS2 ~]$ systemctl status afterbun Unit afterbun.service could not be found. [core@fedoraCoreOS2 ~]$ systemctl status afterburn.service ○ afterburn.service - Afterburn (Metadata) Loaded: loaded (/usr/lib/systemd/system/afterburn.service; disabled; preset: disabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: inactive (dead) Docs: https://coreos.github.io/afterburn/usage/attributes/ [core@fedoraCoreOS2 ~]$ systemctl start afterburn.service ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ==== Authentication is required to start 'afterburn.service'. Authenticating as: CoreOS Admin (core) Password: Failed to start afterburn.service: Connection timed out See system logs and 'systemctl status afterburn.service' for details. [core@fedoraCoreOS2 ~]$ sudo systemctl start afterburn.service [core@fedoraCoreOS2 ~]$ sudo systemctl list-units --status=failed systemctl: unrecognized option '--status=failed' [core@fedoraCoreOS2 ~]$ sudo systemctl list-units --state=failed UNIT LOAD ACTIVE SUB DESCRIPTION ● afterburn-sshkeys@core.service loaded failed failed Afterburn (SSH Keys) LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. [core@fedoraCoreOS2 ~]$ systemctl reset-failed ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ==== Authentication is required to manage system services or other units. Authenticating as: CoreOS Admin (core) Password: [core@fedoraCoreOS2 ~]$ sudo systemctl reset-failed [core@fedoraCoreOS2 ~]$ sudo systemctl list-units --state=failed UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed. [core@fedoraCoreOS2 ~]$ sudo su [root@fedoraCoreOS2 core]#
sudo systemctl list-units --state=failed
SSH Connecting to 192.168.100.212 SSH Host key fingerprint: SSH ecdsa-sha2-nistp256 orzUR98jstCq3/26hFfhBIJ13wfqbsuy6ZD6dwzygKM= SSH Loading private key: file://C:\Users\DIR-N-0153\Desktop\openshift\test.pem Fedora CoreOS 38.20230819.3.0 Tracker: https://github.com/coreos/fedora-coreos-tracker Discuss: https://discussion.fedoraproject.org/tag/coreos Last failed login: Wed Sep 13 08:27:58 UTC 2023 on tty1 There was 1 failed login attempt since the last successful login. [core@localhost ~]$ hostname -f localhost [core@localhost ~]$ ping 10.0.1.107 PING 10.0.1.107 (10.0.1.107) 56(84) bytes of data. 64 bytes from 10.0.1.107: icmp_seq=1 ttl=64 time=2.27 ms 64 bytes from 10.0.1.107: icmp_seq=2 ttl=64 time=0.988 ms 64 bytes from 10.0.1.107: icmp_seq=3 ttl=64 time=0.238 ms ^C --- 10.0.1.107 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.238/1.163/2.265/0.836 ms [core@localhost ~]$ nmcli d DEVICE TYPE STATE CONNECTION ens3 ethernet connected Wired connection 1 lo loopback connected (externally) lo [core@localhost ~]$ nmcli connection add type ethernet autoconnect yes con-name ens160 ifname ens160 Error: Failed to add 'ens160' connection: Insufficient privileges [core@localhost ~]$ nmcli connection add type ethernet autoconnect yes con-name ens3 ifname ens3 Error: Failed to add 'ens3' connection: Insufficient privileges [core@localhost ~]$ sudo su [root@localhost core]# nmcli connection add type ethernet autoconnect yes con-name ens3 ifname ens3 Connection 'ens3' (58cca060-0f38-4b4c-af14-a8f60a5b001b) successfully added. [root@localhost core]# nmcli connection modify ens3 ipv4.addresses 10.0.1.103/24 ipv4.method manual [root@localhost core]# nmcli connection modify ens3 ipv4.dns 10.0.1.107 [root@localhost core]# nmcli connection modify ens3 ipv4.gateway 10.0.1.1 [root@localhost core]# nmcli connection modify ens3 ipv4.dns-search test.domain.co.kr [root@localhost core]# nmcli connection up ens3 Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3) [root@localhost core]# nmcli d DEVICE TYPE STATE CONNECTION ens3 ethernet connected ens3 lo loopback connected (externally) lo [root@localhost core]# sudo fdisk -l Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 83C3D115-FCFA-4420-8F14-C27039EF02BD Device Start End Sectors Size Type /dev/vda1 2048 4095 2048 1M BIOS boot /dev/vda2 4096 264191 260096 127M EFI System /dev/vda3 264192 1050623 786432 384M Linux filesystem /dev/vda4 1050624 209715166 208664543 99.5G Linux filesystem Disk /dev/vdb: 10 GiB, 10737418240 bytes, 20971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [root@localhost core]# coreos-installer coreos-installer 0.17.0 Installer for Fedora CoreOS and RHEL CoreOS USAGE: coreos-installer coreos-installer <SUBCOMMAND> OPTIONS: -h, --help Print help information -V, --version Print version information SUBCOMMANDS: install Install Fedora CoreOS or RHEL CoreOS download Download a CoreOS image list-stream List available images in a Fedora CoreOS stream iso Commands to manage a CoreOS live ISO image pxe Commands to manage a CoreOS live PXE image [root@localhost core]# cd /etc/NetworkManager/system-connections/ [root@localhost system-connections]# ll total 4 -rw-------. 1 root root 263 Sep 13 08:31 ens3.nmconnection [root@localhost system-connections]# cat ens3.nmconnection [connection] id=ens3 uuid=58cca060-0f38-4b4c-af14-a8f60a5b001b type=ethernet interface-name=ens3 [ethernet] [ipv4] address1=10.0.1.103/24,10.0.1.1 dns=10.0.1.107; dns-search=test.domain.co.kr; method=manual [ipv6] addr-gen-mode=default method=auto [proxy] [root@localhost system-connections]# hostnamectl set-hostname fedoraCoreOS2.test.domain.co.kr [root@localhost system-connections]# hostname -f fedoraCoreOS2.test.domain.co.kr [root@localhost system-connections]# hash=`curl http://10.0.1.107:8080/bootstrap.hash` % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 19724 0 --:--:-- --:--:-- --:--:-- 21500 [root@localhost system-connections]# sudo coreos-installer install /dev/vda --ignition-hash sha512-${hash} --copy-network --insecure-ignition Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature Partitions in use on /dev/vda: /dev/vda3 mounted on /boot /dev/vda4 mounted on /sysroot Error: checking for exclusive access to /dev/vda Caused by: found busy partitions [root@localhost system-connections]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 252:0 0 100G 0 disk ├─vda1 252:1 0 1M 0 part ├─vda2 252:2 0 127M 0 part ├─vda3 252:3 0 384M 0 part /boot └─vda4 252:4 0 99.5G 0 part /var /sysroot/ostree/deploy/fedora-coreos/var /usr /etc / /sysroot vdb 252:16 0 10G 0 disk [root@localhost system-connections]# sudo coreos-installer install /dev/vda^C-ignition-hash sha512-${hash} --copy-network --insecure-ignition [root@localhost system-connections]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 252:0 0 100G 0 disk ├─vda1 252:1 0 1M 0 part ├─vda2 252:2 0 127M 0 part ├─vda3 252:3 0 384M 0 part /boot └─vda4 252:4 0 99.5G 0 part /var /sysroot/ostree/deploy/fedora-coreos/var /usr /etc / /sysroot vdb 252:16 0 10G 0 disk vdc 252:32 0 10G 0 disk - = = [root@localhost system-connections]# sudo coreos-installer install /dev/vdc --ignition-hash sha512-${hash} --copy-network --insecure-ignition Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature > Read disk 618.5 MiB/618.5 MiB (100%) gpg: Signature made Tue Sep 5 01:42:45 2023 UTC gpg: using RSA key 6A51BBABBA3D5467B6171221809A8D7CEB10B464 gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 4 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 4u gpg: Good signature from "Fedora (38) <fedora-38-primary@fedoraproject.org>" [ultimate] Copying networking configuration from /etc/NetworkManager/system-connections/ Copying /etc/NetworkManager/system-connections/ens3.nmconnection to installed system Note: detected other devices with a filesystem labeled `boot`: - /dev/vda3 The installed OS may not work correctly if there are multiple boot filesystems. Before rebooting, investigate whether these filesystems are needed and consider wiping them with `wipefs -a`. Install complete. ================== reboot ================ Fedora CoreOS 38.20230819.3.0 Tracker: https://github.com/coreos/fedora-coreos-tracker Discuss: https://discussion.fedoraproject.org/tag/coreos Last login: Wed Sep 13 08:52:49 2023 from 192.168.100.53 [systemd] Failed Units: 1 afterburn-sshkeys@core.service [core@fedoraCoreOS2 ~]$ ll total 0 [core@fedoraCoreOS2 ~]$

### master 노드 작업

203~207

```jsx
wipefs --all --force /dev/vda4
```

```jsx
[systemd]
Failed Units: 1
  afterburn-sshkeys@core.service
[core@worker ~]$ sudo fdisk -l
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4B7D0C2E-15A3-4F0A-B231-5E43B4B8E809

Device       Start       End   Sectors  Size Type
/dev/vda1     2048      4095      2048    1M BIOS boot
/dev/vda2     4096    264191    260096  127M EFI System
/dev/vda3   264192   1050623    786432  384M Linux filesystem
/dev/vda4  1050624 209715166 208664543 99.5G Linux filesystem

Disk /dev/vdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
GPT PMBR size mismatch (4968447 != 20971519) will be corrected by write.
The backup GPT table is not on the end of the device.

Disk /dev/vdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001

Device       Start     End Sectors  Size Type
/dev/vdc1     2048    4095    2048    1M BIOS boot
/dev/vdc2     4096  264191  260096  127M EFI System
/dev/vdc3   264192 1050623  786432  384M Linux filesystem
/dev/vdc4  1050624 4968414 3917791  1.9G Linux filesystem
[core@worker ~]$ sudo systemctl reset-failed
[core@worker ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part /boot
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
├─vdc1 252:33   0    1M  0 part 
├─vdc2 252:34   0  127M  0 part 
├─vdc3 252:35   0  384M  0 part 
└─vdc4 252:36   0  1.9G  0 part 
[core@worker ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs            12G     0   12G   0% /dev/shm
tmpfs           4.7G  636K  4.7G   1% /run
/dev/vda4       100G  2.2G   98G   3% /sysroot
tmpfs            12G     0   12G   0% /tmp
/dev/vda3       350M  106M  222M  33% /boot
tmpfs           2.4G     0  2.4G   0% /run/user/1000
[core@worker ~]$ sudo fdisk
fdisk: bad usage
Try 'fdisk --help' for more information.
[core@worker ~]$ sudo fdisk -l
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4B7D0C2E-15A3-4F0A-B231-5E43B4B8E809

Device       Start       End   Sectors  Size Type
/dev/vda1     2048      4095      2048    1M BIOS boot
/dev/vda2     4096    264191    260096  127M EFI System
/dev/vda3   264192   1050623    786432  384M Linux filesystem
/dev/vda4  1050624 209715166 208664543 99.5G Linux filesystem

Disk /dev/vdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
GPT PMBR size mismatch (4968447 != 20971519) will be corrected by write.
The backup GPT table is not on the end of the device.

Disk /dev/vdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 00000000-0000-4000-A000-000000000001

Device       Start     End Sectors  Size Type
/dev/vdc1     2048    4095    2048    1M BIOS boot
/dev/vdc2     4096  264191  260096  127M EFI System
/dev/vdc3   264192 1050623  786432  384M Linux filesystem
/dev/vdc4  1050624 4968414 3917791  1.9G Linux filesystem
[core@worker ~]$ 
[core@worker ~]$ df -h /dev/vd*
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
/dev/vda3       350M  106M  222M  33% /boot
/dev/vda4       100G  2.2G   98G   3% /
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
devtmpfs        4.0M     0  4.0M   0% /dev
[core@worker ~]$ df -h /dev/vdc
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
[core@worker ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part /boot
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
├─vdc1 252:33   0    1M  0 part 
├─vdc2 252:34   0  127M  0 part 
├─vdc3 252:35   0  384M  0 part 
└─vdc4 252:36   0  1.9G  0 part 
[core@worker ~]$ wipefs /dev/vdc
wipefs: error: /dev/vdc: probing initialization failed: Permission denied
[core@worker ~]$ sudo su
[root@worker core]# wipefs /dev/vdc
DEVICE OFFSET      TYPE UUID LABEL
vdc    0x200       gpt       
vdc    0x27ffffe00 gpt       
vdc    0x1fe       PMBR      
[root@worker core]# fdisk --help

Usage:
 fdisk [options] <disk>         change partition table
 fdisk [options] -l [<disk>...] list partition table(s)

Display or manipulate a disk partition table.

Options:
 -b, --sector-size <size>      physical and logical sector size
 -B, --protect-boot            don't erase bootbits when creating a new label
 -c, --compatibility[=<mode>]  mode is 'dos' or 'nondos' (default)
 -L, --color[=<when>]          colorize output (auto, always or never)
                                 colors are enabled by default
 -l, --list                    display partitions and exit
 -x, --list-details            like --list but with more details
 -n, --noauto-pt               don't create default partition table on empty devices
 -o, --output <list>           output columns
 -t, --type <type>             recognize specified partition table type only
 -u, --units[=<unit>]          display units: 'cylinders' or 'sectors' (default)
 -s, --getsz                   display device size in 512-byte sectors [DEPRECATED]
     --bytes                   print SIZE in bytes rather than in human readable format
     --lock[=<mode>]           use exclusive device lock (yes, no or nonblock)
 -w, --wipe <mode>             wipe signatures (auto, always or never)
 -W, --wipe-partitions <mode>  wipe signatures from new partitions (auto, always or never)

 -C, --cylinders <number>      specify the number of cylinders
 -H, --heads <number>          specify the number of heads
 -S, --sectors <number>        specify the number of sectors per track

 -h, --help                    display this help
 -V, --version                 display version

Available output columns:
 gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
 dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S
      Start-C/H/S
 bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
 sgi: Device Start End Sectors Cylinders Size Type Id Attrs
 sun: Device Start End Sectors Cylinders Size Type Id Flags

For more details see fdisk(8).
[root@worker core]# fdisk -wipe  /dev/vdc
fdisk: unsupported wipe mode
[root@worker core]# fdisk --wipe  /dev/vdc
fdisk: unsupported wipe mode
[root@worker core]# wipefs --all --force /dev/vda
/dev/vda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/vda: 8 bytes were erased at offset 0x18fffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/vda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
[root@worker core]# wipefs --all --force /dev/vdc
/dev/vdc: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/vdc: 8 bytes were erased at offset 0x27ffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/vdc: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
[root@worker core]# sudo coreos-installer install /dev/vdc --ignition-url http://10.0.1.107:8080/master.ign --ignition-hash sha512-${hash} --copy-network 
error: Invalid value "sha512-" for '--ignition-hash <digest>': wrong digest length (0)

For more information try --help
[root@worker core]# hash=`curl http://10.0.1.107:8080/master.hash`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   129  100   129    0     0  23809      0 --:--:-- --:--:-- --:--:-- 25800
[root@worker core]# sudo coreos-installer install /dev/vda --ignition-url http://10.0.1.107:8080/master.ign --ignition-hash sha512-${hash} --copy-network 
Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature
Partitions in use on /dev/vda:
    /dev/vda3 mounted on /boot
    /dev/vda4 mounted on /sysroot
Error: checking for exclusive access to /dev/vda

Caused by:
    found busy partitions
[root@worker core]# blkid
/dev/vdb: LABEL="ephemeral0" UUID="3acfd544-7fb4-45fc-bb9b-b558460f6d26" BLOCK_SIZE="4096" TYPE="ext4"
/dev/vda4: LABEL="root" UUID="f808356c-4d6a-48ed-9c87-1ba7b1eff4ed" BLOCK_SIZE="512" TYPE="xfs"
/dev/vda2: SEC_TYPE="msdos" LABEL_FATBOOT="EFI-SYSTEM" LABEL="EFI-SYSTEM" UUID="FCFE-5903" BLOCK_SIZE="512" TYPE="vfat"
/dev/vda3: LABEL="boot" UUID="03ffe148-feef-438d-8717-cfb343094e39" BLOCK_SIZE="1024" TYPE="ext4"
[root@worker core]# blkid
/dev/vdb: LABEL="ephemeral0" UUID="3acfd544-7fb4-45fc-bb9b-b558460f6d26" BLOCK_SIZE="4096" TYPE="ext4"
/dev/vda4: LABEL="root" UUID="f808356c-4d6a-48ed-9c87-1ba7b1eff4ed" BLOCK_SIZE="512" TYPE="xfs"
/dev/vda2: SEC_TYPE="msdos" LABEL_FATBOOT="EFI-SYSTEM" LABEL="EFI-SYSTEM" UUID="FCFE-5903" BLOCK_SIZE="512" TYPE="vfat"
/dev/vda3: LABEL="boot" UUID="03ffe148-feef-438d-8717-cfb343094e39" BLOCK_SIZE="1024" TYPE="ext4"
[root@worker core]# fdisk -l
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vdb: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/vdc: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@worker core]# sudo coreos-installer install /dev/vda^C-ignition-url http://10.0.1.107:8080/master.ign --ignition-hash sha512-${hash} --copy-network 
[root@worker core]# fdisk /dev/vda

Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

This disk is currently in use - repartitioning is probably a bad idea.
It's recommended to umount all file systems, and swapoff all swap
partitions on this disk.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x3ac3bb98.

Command (m for help): d
No partition is defined yet!

Command (m for help): w

The partition table has been altered.
Syncing disks.

[root@worker core]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part /boot
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
[root@worker core]# wipefs --all --force /dev/vda3
/dev/vda3: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
[root@worker core]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part 
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
[root@worker core]# wipefs --all --force /dev/vda4
/dev/vda4: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42
[root@worker core]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part 
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
[root@worker core]# wipefs --all --force /dev/vda4
[root@worker core]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    252:0    0  100G  0 disk 
├─vda1 252:1    0    1M  0 part 
├─vda2 252:2    0  127M  0 part 
├─vda3 252:3    0  384M  0 part 
└─vda4 252:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot
vdb    252:16   0   10G  0 disk 
vdc    252:32   0   10G  0 disk 
[root@worker core]# sudo coreos-installer install /dev/vdc --ignition-url http://10.0.1.107:8080/master.ign --ignition-hash sha512-${hash} --copy-network 
Downloading Fedora CoreOS stable x86_64 metal image (raw.xz) and signature
> Read disk 618.5 MiB/618.5 MiB (100%)    
gpg: Signature made Tue Sep  5 01:42:45 2023 UTC
gpg:                using RSA key 6A51BBABBA3D5467B6171221809A8D7CEB10B464
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   4  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 4u
gpg: Good signature from "Fedora (38) <fedora-38-primary@fedoraproject.org>" [ultimate]
Writing Ignition config
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/ens3.nmconnection to installed system
Install complete.
```

 `wipefs --all --force /dev/vdc`

```jsx
vi /etc/sysctl.conf
```

```jsx
coreos.inst.install_dev=vdb
coreos.inst.image_url=http://10.0.1.107:8080/fedora-coreos-38.20230819.3.0-metal.x86_64.raw.xz
coreos.inst.ignition_url=http://10.0.1.107:8080/master.ign
ip=10.0.1.144::10.0.1.1:255.255.255.0:master2.test.domain.co.kr:ens3:none
nameserver=10.0.1.107

```

```
[rocky@worker ~]$ openshift-install coreos print-stream-json | grep -Eo '"https.*(kernel-|in
itramfs.|rootfs.)\w+(\.img)?"'
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/aarch64/
fedora-coreos-35.20220327.3.0-live-kernel-aarch64"
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/aarch64/
fedora-coreos-35.20220327.3.0-live-initramfs.aarch64.img"
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/aarch64/
fedora-coreos-35.20220327.3.0-live-rootfs.aarch64.img"
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/x86_64/f
edora-coreos-35.20220327.3.0-live-kernel-x86_64"
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/x86_64/f
edora-coreos-35.20220327.3.0-live-initramfs.x86_64.img"
"https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/35.20220327.3.0/x86_64/f
edora-coreos-35.20220327.3.0-live-rootfs.x86_64.img"

```

kernel, initramfs, rootfs 다운로드 

[222.xxx](http://222.xxx) → 라우터 

### 1006

```jsx
domain@maascontroller:~/okd/install-directory3$ oc login --username=kubeadmin --password=yiQn9-jgxuH-bYbyi-Sj39w
Login successful.

You have access to 68 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "openshift-console".
domain@maascontroller:~/okd/install-directory3$ oc get pod
NAME                        READY   STATUS    RESTARTS   AGE
console-57d86f9796-cx48b    1/1     Running   0          42m
console-57d86f9796-sqjch    1/1     Running   0          42m
downloads-b57469dd7-5fxdt   1/1     Running   0          49m
downloads-b57469dd7-m7g7b   1/1     Running   0          49m
domain@maascontroller:~/okd/install-directory3$ oc -n openshift-console get route
NAME        HOST/PORT                                        PATH   SERVICES    PORT    TERMINATION          WILDCARD
console     console-openshift-console.apps.test.test            console     https   reencrypt/Redirect   None
downloads   downloads-openshift-console.apps.test.test          downloads   http    edge/Redirect        None
domain@maascontroller:~/okd/install-directory3$ oc -n openshift-console get service
NAME        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
console     ClusterIP   172.30.163.114   <none>        443/TCP   52m
downloads   ClusterIP   172.30.189.133   <none>        80/TCP    52m
domain@maascontroller:~/okd/install-directory3$ oc -n openshift-authentication get route
NAME              HOST/PORT                            PATH   SERVICES          PORT   TERMINATION            WILDCARD
oauth-openshift   oauth-openshift.apps.test.test          oauth-openshift   6443   passthrough/Redirect   None
domain@maascontroller:~/okd/install-directory3$ oc -n openshift-authentication get service
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
oauth-openshift   ClusterIP   172.30.35.132   <none>        443/TCP   68m
domain@maascontroller:~/okd/install-directory3$ oc -n openshift-authentication get route oauth-openshift -o json | jq .status
{
{
  "ingress": [
    {
      "conditions": [
        {
          "lastTransitionTime": "2023-10-06T07:04:04Z",
          "status": "True",
          "type": "Admitted"
        }
      ],
      "host": "oauth-openshift.apps.test.test",
      "routerCanonicalHostname": "router-default.apps.test.test",
      "routerName": "default",
      "wildcardPolicy": "None"
    }
  ]
}
```

저작자표시

'Container & Orchestration > OKD' 카테고리의 다른 글

OKD - on openstack (0)	2025.02.11
OKD Dashboard 이용해서 Helm Release 쉽게 진행하기 (0)	2025.02.07
okd 명령어 (0)	2025.01.30

abstract.jiiiiin

OKD - baremetal

OKD-baremetal

사용된 서버 사양

1. DNS 설정 (bastion server)

1) bind install

2) zone 등록

3) 정방향 DNS

4) 역방향 DNS

5) 생성한 zone 파일의 권한을 변경

6) zone 파일이 문법적으로 맞는지 확인

7) /etc/named.conf 파일 수정

8) /etc/resolv.conf 수정

9) named 데몬 재시작

10) nslookup으로 확인

2. LoadBalancer - HAploxy (bastion server)

1) 필요한 util 다운로드

2) /etc/haproxy/haproxy.cfg 파일 수정

config파일 확인

haproxy 서비스 시작

3. Install (bastion server)

1) okd 파일을 설치하고, /usr/local/bin 폴더에 압축을 풀어준다.

2) ssh 키 생성

3) Ignition file을 만들기 위한 install-config.yaml 작성

4) openshift를 설치한 폴더에서 manifests 명령어 수행

5) ignition 파일 생성

6) 생성된 ignition 파일을 파일 서버 디렉토리로 복사하고 다운로드 가능하도록 퍼미션 조정

7) curl 확인

4. bootstrap 설치

1) ign 검증하는 hash값 파일로 전환하기

드라이브에 ISO 파일 마운트

참고

참고 블로그 및 페이지

'Container & Orchestration > OKD' 카테고리의 다른 글

티스토리툴바

OKD - baremetal

OKD-baremetal

사용된 서버 사양

1. DNS 설정 (bastion server)

1) bind install

2) zone 등록

3) 정방향 DNS

4) 역방향 DNS

5) 생성한 zone 파일의 권한을 변경

6) zone 파일이 문법적으로 맞는지 확인

7) /etc/named.conf 파일 수정

8) /etc/resolv.conf 수정

9) named 데몬 재시작

10) nslookup으로 확인

2. LoadBalancer - HAploxy (bastion server)

1) 필요한 util 다운로드

2) /etc/haproxy/haproxy.cfg 파일 수정

config파일 확인

haproxy 서비스 시작

3. Install (bastion server)

1) okd 파일을 설치하고, /usr/local/bin 폴더에 압축을 풀어준다.

2) ssh 키 생성

3) Ignition file을 만들기 위한 install-config.yaml 작성

4) openshift를 설치한 폴더에서 manifests 명령어 수행

5) ignition 파일 생성

6) 생성된 ignition 파일을 파일 서버 디렉토리로 복사하고 다운로드 가능하도록 퍼미션 조정

7) curl 확인

4. bootstrap 설치

1) ign 검증하는 hash값 파일로 전환하기

드라이브에 ISO 파일 마운트

참고

참고 블로그 및 페이지

'Container & Orchestration > OKD' 카테고리의 다른 글

관련글

티스토리툴바