https://yum.enterprisedb.com/tools/redhat/rhel-7-x86_64/repoview/edb-efm32.html URL : https://yum.enterprisedb.com UserName : Nijamutheen mail : nijam@2ndquadrant.in Password : HcMMBdpGnFEyi8J3 URL :https://enterprisedb.com UserName : Nijamutheen Password : Demo@12345 curl -v https://www.oracle.com for third party URL access This tutorial describes quickly configuring a Replication and Failover Manager cluster in EnterpriseDB SOFTWARES & IP Description: EnterpriseDB : 9.6 // https://www.enterprisedb.com/software-downloads-postgres EnterpriseDB Failover Manager(EFM) : 3.1 // https://www.enterprisedb.com/download-edb-failover-manager Master : 10.11.128.81 PORT :5444 Slave : 10.11.128.82 PORT :5444 EDB Failover Manager Witness Node + EDB BART + PEM : 10.11.133.193 PORT :5430 OS : Red Hat Enterprise Linux Server release 7.4 (Maipo) Simple 3 steps TO install EDB 9.6 cd /home/465632 chown .root edb-as96-server-9.6.10.17-1-linux-x64.run chmod 775 edb-as96-server-9.6.10.17-1-linux-x64.run ./edb-as96-server-9.6.10.17-1-linux-x64.run CONTENTS: Topic-I . NETOWORK & FIREWALL Topic-II. STREAMING REPLICATION Topic-III. FAILOVER MANAGER Topic-Iv. EDB emf SWITCHOVER: Topic-V . EDB emf AUTOMATIC FAILOVERS Basic Facts of EDB EFM: 10.11.128.81 Currernt PostgreSQL primary instance 10.11.128.82 Current PostgreSQL hot standby instance 10.11.133.193 EDB Failover Manager Witness Node + EDB BART //The purpose of the witness is to do an automatic failover when the primary is down. //Virtual-IP Virtual IP that is used for client connections to the current master database or new master I . NETOWORK & FIREWALL First we have to complete the network configuration between those server then we have to clear the firewall issues. Our IP'S are 10.11.128.81, 10.11.128.82, 10.11.133.193. At : 10.11.128.81 vi /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE= ONBOOT=yes NETMASK=255.255.255.240 IPADDR=10.11.128.81 GATEWAY=10.11.128.94 TYPE=Ethernet At : 10.11.128.82 vi /etc/sysconfig/network-scripts/ifcfg-eth0 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) DEVICE= ONBOOT=yes NETMASK=255.255.255.240 IPADDR=10.11.128.82 GATEWAY=10.11.128.94 TYPE=Ethernet At : 10.11.133.193 vi /etc/sysconfig/network-scripts/ifcfg-eth0 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) DEVICE=ens32 ONBOOT=yes NETMASK=255.255.255.224 IPADDR=10.11.133.193 GATEWAY=10.11.133.222 TYPE=Ethernet SELINUX will be helpful to reach different domain PING optiom Here 128 & 133 two different domain is there. you need to disable firewall as well. vi /etc/sysconfig/selinux SELINUX=disabled --Restart the network services # systemctl restart network (Or) # service network restart STOPPING OR DISABLING FIREWALL: If the firewall is active / running, enter this command to stop it: # systemctl stop firewalld.service To completely disable the firewalld service, so it does not reload when you restart the host machine: # systemctl disable firewalld.service AT 10.11.133.193 but 81 and 82 is not have HOST entries. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.11.133.193 NCG-Hospital-Prod-Web 10.11.11.201 hydesm001 Topic-II. STREAMING REPLICATION : Following procedure is used to EDB PostgreSQL Streaming Replication. Identify the master and slave database. Get the database down time to set up streaming replication. Follow the steps mentioned below. Master Server setup: 1) Note down master and slave server IP address. 2) Create replication user in master server CREATE USER replica REPLICATION LOGIN ENCRYPTED PASSWORD 'TcsReplica@123'; 3) Make the following changes in postgressql.conf file ------------------------------------------------------------------------------ # CONNECTIONS AND AUTHENTICATION ------------------------------------------------------------------------------ listen_addresses = 'localhost,10.11.128.81' ------------------------------------------------------------------------------ # WRITE AHEAD LOG ------------------------------------------------------------------------------ wal_level = hot_standby synchronous_commit = local ------------------------------------------ # - Archiving ------------------------------------------ archive_mode = on archive_command= 'cp %p /opt/edb/as9.6/data/archive/%f' archive_timeout = 2 (seconds) ------------------------------------------------------------------------------------------- # REPLICATION ------------------------------------------------------------------------------------------- max_wal_senders = 2 wal_keep_segments = 10 synchronous_standby_names = 'pgslave001' 4) Edit pg_hba.conf file to add the following line in replication host details. vi /opt/edb/as9.6/data/pg_hba.conf host Replication replica 10.11.128.82/32 md5 5) Restart master server. /opt/edb/as9.6/bin/pg_ctl -D /opt/edb/as9.6/data restart Slave Server setup: 1) Stop the services in slave servers. /opt/edb/as9.6/bin/pg_ctl -D /opt/edb/as9.6/data stop 2) Remove existing data directory. mv /opt/edb/as9.6/data/* /home/postgres/backup/ 3) Take rsync backup and create data directory in Master server by using the following command. /opt/edb/as9.6/bin/psql -c "select pg_start_backup('initial_backup');" rsync -cva --inplace /opt/edb/as9.6/data/* enterprisedb@10.11.128.82:/opt/edb/as9.6/data/ /opt/edb/as9.6/bin/psql -c "select pg_stop_backup();" 4) After completing previous step, edit postgresql.conf file in the slave server and make the following changes. ------------------------------------------------------------------------------ # CONNECTIONS AND AUTHENTICATION ------------------------------------------------------------------------------ listen_addresses = 'localhost,10.11.128.82' ------------------------------------------------------------------------------ # REPLICATION ------------------------------------------------------------------------------ hot_standby = on 5) Create the recovery.conf file in slave server in data directory. vi /opt/edb/as9.6/data/recovery.conf standby_mode = 'on' primary_conninfo='host=10.11.128.81 port=5444 user=replica password=TcsReplica@123 application_name=pgslave001' restore_command = 'cp /opt/edb/as9.6/data/archive/%f %p' trigger_file = '/tmp/postgresql.trigger.5444' 6) Create access permissions to recovery.conf file Chown –R enterprisedb. /opt/edb/as9.6/data/recovery.conf 7) Start the slave server. /opt/edb/as9.6/bin/pg_ctl -D /opt/edb/as9.6/data start Verification: Make some changes (DDL / DML) in master database and verify in the slave database for the same change. If the changes appears in the slave database, streaming replication setup is successful. Using pg_stat_replication views check master server whether is recovery mode or not,master will not be recovery mode slave only will be recovery mode. select pg_is_in_recovery(); using linux command check the wal sender/receiver process whether started or not: $ ps -ef|grep postgres Topic-III. FAILOVER MANAGER Configuration: we are using 3 servers Red Hat Enterprise Linux Server release 7.4 (Maipo) FEATURES OF EFM: - Automatic failover - Switchover to a standby (s) close to master in terms of transaction xlog - Set priority of standby (s) - Next master if you have multiple standby - Customize notifications/alerts - Transfer VIP to the new master with failover (provided, the master has been configured with VIP) A database server is running, and streaming replication is set up between the master and standby nodes. You have installed the Failover Manager RPM on each node means master,standby and witness node. The example that follows creates a cluster named efm. You should start the configuration process on a master or standby node. Then, you can copy the configuration files to other nodes to save time. Step 1. EFM must be installed on both 3 nodes.To install EDB EFM, I used the rpm provided by EnterpriseDB. Note that you will need an Edb account.We will show the installation on only one node, but is the same for both nodes. rpm -Uvh http://yum.enterprisedb.com/edbrepos/edb-repo-latest.noarch.rpm or yum install -y http://yum.enterprisedb.com/edbrepos/edb-repo-latest.noarch.rpm ALTERNATIVE METHOD: DIRECTLY INSTALLING USING DOWNLOADED RPM : If your server not allowing internet connectivity or if you feel complicated repository configuration then directly you can download and upload (WINSCP will help) the emf rpm using following account i will give 2ndquadrant enterprisedb account username and password for educational pupose enjoy ! . URL : https://yum.enterprisedb.com/tools Username : Nijamutheen Password : HcMMBdpGnFEyi8J3 chown .root edb-efm32-3.2.0-1.rhel7.x86_64.rpm chmod 775 edb-efm32-3.2.0-1.rhel7.x86_64.rpm rpm -ivh edb-efm32-3.2.0-1.rhel7.x86_64.rpm Get it table https://www.enterprisedb.com/docs/en/3.1/edbfm/EDB_Failover_Manager_Guide.1.10.html EMF Utilities bash-4.2$ cd /usr/edb/efm-3.2/bin bash-4.2$ ls -l total 44 -rwxr-xr-x 1 root root 388 Jul 23 12:13 efm -rwxr-xr-x 1 root root 1380 Jul 23 12:13 efm_address -rwxr-xr-x 1 root root 13813 Jul 23 12:13 efm_db_functions -rwxr-xr-x 1 root root 5339 Jul 23 12:13 efm_root_functions -rwxr-xr-x 1 root root 1483 Jul 23 12:13 runefm.sh -rwxr-xr-x 1 root root 2500 Jul 23 12:13 runJavaApplication.sh drwxr-x--- 2 root efm 4096 Oct 23 13:25 secure bash-4.2$ cd secure/ bash-4.2$ ls -l total 8 -rwsr-x--- 1 root efm 7192 Jul 23 12:13 manage-vip Note : please don't use for unwanted things, 2ndquadrant.in Giving this educational purpose only. Step 2. After we have to enable the corresponding edb repository (You will need a login and password) skip this step if you install the emf alternate method. # vi /etc/yum.repos.d/edb.repo [edbas96] name=EnterpriseDB Advanced Server 9.6 $releasever - $basearch baseurl=https://Nijamutheen:HcMMBdpGnFEyi8J3@yum.enterprisedb.com/9.6/redhat/rhel-$releasever-$basearch enabled=1 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/ENTERPRISEDB-GPG-KEY Step 3.efm requires also openjdk. So we also have installed openjdk 1.8 on all nodes: Install the Openjdk as well in all nodes note the point your server should be on internet connection becouse openjdk will ask some dependencies packages as well. [root@2ndquadrant.in]# yum install java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 [root@2ndquadrant.in]# java -version openjdk version "1.8.0_181" OpenJDK Runtime Environment (build 1.8.0_181-b13) OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode) [root@2ndquadrant.in]# rpm -q --whatprovides java java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.i686 java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 Note: this will install java 1 Package (+22 Dependent packages) but server need internet connection. Refer this link for more Step 4: To manage efm we create a dedicated user in the primary cluster postgres=# create user efm with login password 'root' superuser; Note:emf user will be created during installation. Step 5: We have to add following entries to pg_hba.conf of all databases clusters to allow connection for user efm. vi /opt/edb/as9.6/data/pg_hba.conf ##for efm host edb efm 10.11.128.81/32 md5 host edb efm 10.11.128.82/32 md5 host edb efm 10.11.133.193/32 md5 Step 6: Create Working Configuration Files The configuration of efm consists of editing 2 main configuration files: efm.nodes and efm.properties. In my case these files are located in /etc/edb/efm-3.1. There are already two sample files that we can copy and then edit. cd /etc/edb/efm-3.2 cp efm.properties.in efm.properties cp efm.nodes.in efm.nodes chown efm:efm efm.properties chown efm:efm efm.nodes Note: By default, Failover Manager expects the cluster properties file to be named efm.properties. If you name the properties file something other than efm.properties, you must modify the service script or unit file to instruct Failover Manager to use a different name. The property files are owned by root. The Failover Manager service script expects to find the files in the /etc/edb/efm-3.2 directory. If you move the property file to another location, you must create a symbolic link that specifies the new location. Step 7: Create the encrypted password in all nodes (needed for the properties file) for emf user "db.password.encrypted": [root@2ndquadrant.in]# /usr/edb/efm-3.2/bin/efm encrypt efm This utility will generate an encrypted password for you to place in your EFM cluster property file: /etc/edb/efm-3.2/efm.properties Please enter the password and hit enter: Please enter the password again to confirm: Could not encode password: Illegal key size or default parameters If full-strength encryption has not been installed in the current version of Java, search for " java full strength encryption" for more information. ### Solution JDK 9 and later ship with, and use by default, the unlimited policy files. The unlimited policy files for earlier releases available here are required only for JDK 8, 7, and 6 updates earlier than 8u161, 7u171, and 6u16. On those versions and later the policy files are included, but not enabled by default. STEPS TO FIX THIS ISSUES: 1. For this download “Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 8” for JDK 1.8. from below link https://www.oracle.com/technetwork/java/javase/downloads/jce-all-download-5170447.html 2. change appropriate file permission and unzip the the folder. cd /home/465632 chown .root jce_policy.zip chmod 775 jce_policy.zip unzip jce_policy.zip 3. after unzip you will get two policy files which is given belo [root@2ndquadrant.in]# cd UnlimitedJCEPolicyJDK8/ [root@2ndquadrant.in]# ls -l total 16 -rw-rw-r-- 1 root root 3035 Dec 21 2013 local_policy.jar -rw-r--r-- 1 root root 7323 Dec 21 2013 README.txt -rw-rw-r-- 1 root root 3023 Dec 21 2013 US_export_policy.jar 4.find the security location file first issue the "java -version" you will see the version of java the locate the security folder location using use "locate jdk" then update paste the both policy files into the security folder if files is already exists do overwrite. our server security file location is "/dpp/JDK/installation/jdk1.8.0_101/jre/lib/security" then copied mention files in security location. [root@2ndquadrant.in]# sudo alternatives --config javac [root@2ndquadrant.in]# cp /home/465632/UnlimitedJCEPolicyJDK8/* /dpp/JDK/installation/jdk1.8.0_101/jre/lib/security after copied policy files , format should be like this [root@2ndquadrant.in]# pwd /dpp/JDK/installation/jdk1.8.0_101/jre/lib/security [root@2ndquadrant.in]# ls -l total 176 -rw-r--r-- 1 tomcat tomcat 4054 Jun 22 2016 blacklist -rw-r--r-- 1 tomcat tomcat 1273 Jun 22 2016 blacklisted.certs -rw-r--r-- 1 tomcat tomcat 112860 Jun 22 2016 cacerts -rw-r--r-- 1 tomcat tomcat 2466 Jun 22 2016 java.policy -rw-r--r-- 1 tomcat tomcat 26224 Jun 22 2016 java.security -rw-r--r-- 1 tomcat tomcat 98 Jun 22 2016 javaws.policy -rw-r--r-- 1 tomcat tomcat 3035 Oct 25 17:05 local_policy.jar -rw-r--r-- 1 root root 7323 Oct 25 17:05 README.txt -rw-r--r-- 1 tomcat tomcat 0 Jun 22 2016 trusted.libraries -rw-r--r-- 1 tomcat tomcat 3023 Oct 25 17:05 US_export_policy.jar 5. then again issue the encrypt " ./efm encrypt efm " you will get encrypted password [root@2ndquadrant.in]# /usr/edb/efm-3.2/bin/efm encrypt efm This utility will generate an encrypted password for you to place in your EFM cluster property file: /etc/edb/efm-3.2/efm.properties Please enter the password and hit enter: Please enter the password again to confirm: The encrypted password is: f2cbe18f2c761c951093e45cb5b03917 Please paste this into your efm.properties file db.password.encrypted=f2cbe18f2c761c951093e45cb5b03917 ERROR 2: [root@2ndquadrant.in]# /usr/edb/efm-3.2/bin/efm encrypt efm 64 bit java not found. ### SOLUTION Only the OpenJDK JRE is installed by default. Google’s Android Studio, however, requires the full Java JDK, and the recommended version is Oracle’s Java (JDK or SE 6 and above). So the first thing you have to do is to install the full Oracle’s Java JDK. Point your web browser to the Oracle SE download page here(https://www.oracle.com/technetwork/java/javase/downloads/index.html) and download the latest RPM version of the 64-bit JDK (not Server JRE or the JRE) [root@2ndquadrant.in]# java -version openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-b12) OpenJDK Server VM (build 25.191-b12, mixed mode) [root@2ndquadrant.in]# rpm -qa | grep java java-1.8.0-openjdk-headless-1.8.0.191.b12-0.el7_5.i686 SYMCnbjava-7.6.0.1-1.x86_64 [root@2ndquadrant.in]# yum install jdk-8u191-linux-x64.rpm [root@2ndquadrant.in]# yum install jdk-8u191-linux-x64-demos.rpm The first two commands above will install the JDK, demos and samples to the ‘/usr/java’ directory. [root@2ndquadrant.in]# yum remove java-1.8.0-openjdk-headless-1.8.0.191.b12-0.el7_5.i686 [root@2ndquadrant.in]# java -version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 1.8.0_191-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode) Step 8: Update the efm.properties File The properties file (cluster_name.properties) contains parameters that specify connection properties and behaviors for your Failover Manager cluster. Modifications to property settings are applied when Failover Manager starts. If you modify a property value you must restart Failover Manager to apply the changes. for more info : https://www.enterprisedb.com/docs/en/3.1/edbfm/EDB_Failover_Manager_Guide.1.13.html#pID0E0N50HA AT MASTER EFM.PROPERTIES & EFM.NODES FILE: $ cat /etc/edb/efm-3.1/efm.properties | grep -v ^# db.user=efm db.password.encrypted=ff7f041651e5e864013c1102d26a5e08 # you will get from step 7, if you change the cluster name(emf) , you must encrypt the password again with the new name. db.port=5444 db.database=edb db.service.owner=enterprisedb db.service.name= # not mandatory to give "systemctl restart " we given bin path emf will take pg_ctl automatically. db.bin=/opt/edb/as9.6/bin/ db.recovery.conf.dir=/opt/edb/as9.6/data/ jdbc.sslmode=disable user.email=nijam@2ndquadrant.in,info@2ndquadrant.in script.notification= bind.address=10.11.128.81:5430 # ip address and port that jgroups and not the same as the "admin.port" admin.port=9999 is.witness=false local.period=10 local.timeout=60 local.timeout.final=10 remote.timeout=10 node.timeout=50 stop.isolated.master=false pingServerIp=10.11.133.193 # Testing network connectivity to your name server if GOOGLE DNS 8.8.8.8, we will not preferred master, standby,witness IP. pingServerCommand=/bin/ping -q -c3 -w5 auto.allow.hosts=false db.reuse.connection.count=0 auto.failover=true auto.reconfigure=true promotable=true minimum.standbys=0 recovery.check.period=2 auto.resume.period=0 script.fence= script.post.promotion= script.resumed= script.db.failure= script.master.isolated= script.remote.pre.promotion= script.remote.post.promotion= script.custom.monitor= custom.monitor.interval= custom.monitor.timeout= custom.monitor.safe.mode= sudo.command=sudo sudo.user.command=sudo -u %u log.dir=/var/log/efm-3.1 jgroups.loglevel=INFO efm.loglevel=INFO jvm.options=-Xmx32m cat /etc/edb/efm-3.1/efm.nodes # List of node address:port combinations separated by whitespace. # The list should include at least the membership coordinator's address. 10.11.133.193:5430 10.11.128.82:5430 AT SLAVE EMF.PROPERTIES & EMF.NODES FILE: $ cat /etc/edb/efm-3.1/efm.properties | grep -v ^# db.user=efm db.password.encrypted=ff7f041651e5e864013c1102d26a5e08 # you will get from step 6, if you change the cluster name(emf) , you must encrypt the password again with the new name. db.port=5444 db.database=edb db.service.owner=enterprisedb db.service.name= # not mandatory to give "systemctl restart " we given bin path emf will take pg_ctl automatically. db.bin=/opt/edb/as9.6/bin/ db.recovery.conf.dir=/opt/edb/as9.6/data/ jdbc.sslmode=disable user.email=nijam@2ndquadrant.in,info@2ndquadrant.in script.notification= bind.address=10.11.128.82:5430 # ip address and port that jgroups and not the same as the "admin.port" admin.port=9999 is.witness=false local.period=10 local.timeout=60 local.timeout.final=10 remote.timeout=10 node.timeout=50 stop.isolated.master=false pingServerIp=10.11.133.193 # Testing network connectivity to your name server if GOOGLE DNS 8.8.8.8, we will not preferred master, standby,witness IP. pingServerCommand=/bin/ping -q -c3 -w5 auto.allow.hosts=true db.reuse.connection.count=0 auto.failover=true auto.reconfigure=true promotable=true minimum.standbys=0 recovery.check.period=2 auto.resume.period=0 script.fence= script.post.promotion= script.resumed= script.db.failure= script.master.isolated= script.remote.pre.promotion= script.remote.post.promotion= script.custom.monitor= custom.monitor.interval= custom.monitor.timeout= custom.monitor.safe.mode= sudo.command=sudo sudo.user.command=sudo -u %u log.dir=/var/log/efm-3.1 jgroups.loglevel=INFO efm.loglevel=INFO jvm.options=-Xmx32m cat /etc/edb/efm-3.1/efm.nodes # List of node address:port combinations separated by whitespace. # The list should include at least the membership coordinator's address. 10.11.133.193:5430 10.11.128.81:5430 AT WITNESS SERVER EFM.PROPERTIES & EFM.NODES FILE: $ cat /etc/edb/efm-3.1/efm.properties | grep -v ^# db.user=efm db.password.encrypted=ff7f041651e5e864013c1102d26a5e08 # you will get from step 6, if you change the cluster name(emf) , you must encrypt the password again with the new name. db.port=5444 db.database=edb db.service.owner=enterprisedb db.service.name= # not mandatory to give "systemctl restart " we given bin path emf will take pg_ctl automatically. db.bin=/opt/edb/as9.6/bin/ db.recovery.conf.dir=/opt/edb/as9.6/data/ jdbc.sslmode=disable user.email=nijam@2ndquadrant.in,info@2ndquadrant.in script.notification= bind.address=10.11.128.82:5430 # ip address and port that jgroups and not the same as the "admin.port" admin.port=9999 is.witness=true local.period=10 local.timeout=60 local.timeout.final=10 remote.timeout=10 node.timeout=50 stop.isolated.master=false pingServerIp=10.11.133.193 # Testing network connectivity to your name server if GOOGLE DNS 8.8.8.8, we will not preferred master, standby,witness IP. pingServerCommand=/bin/ping -q -c3 -w5 auto.allow.hosts=false db.reuse.connection.count=0 auto.failover=true auto.reconfigure=true promotable=true minimum.standbys=0 recovery.check.period=2 auto.resume.period=0 script.fence= script.post.promotion= script.resumed= script.db.failure= script.master.isolated= script.remote.pre.promotion= script.remote.post.promotion= script.custom.monitor= custom.monitor.interval= custom.monitor.timeout= custom.monitor.safe.mode= sudo.command=sudo sudo.user.command=sudo -u %u log.dir=/var/log/efm-3.1 jgroups.loglevel=INFO efm.loglevel=INFO jvm.options=-Xmx32m cat /etc/edb/efm-3.1/efm.nodes # List of node address:port combinations separated by whitespace. # The list should include at least the membership coordinator's address. 10.11.128.81:5430 10.11.128.82:5430 Step 9: Start the EFM Cluster Now let’s start efm on both nodes. If there is any error check logs on /var/log/efm-3.1/startup-efm.log I started on following order: MASTER, SALVE AND WITNESS SERVER. Services can be configured to be started automatically when the server starts. Start the Failover Manager on any node of the cluster. The agent is named efm-3.1; use your platform-specific service command to control the service. For example, on a CentOS or RHEL 7.x host, use the command: systemctl start efm-3.1 On a CentOS or RHEL 6.x host, use the command: service efm-3.1 start oracle LINUX: service efm-3.1 start Step 9: Once started we can verified from any node the status of our cluster $ /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 10.11.133.193 UP N/A Master 10.11.128.81 UP UP Standby 10.11.128.82 UP UP Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 10.11.133.193 Standby priority host list: 10.11.128.82 Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Master 10.11.128.81 0/430001A8 Standby 10.11.128.82 0/430001A8 Standby database(s) in sync with master. It is safe to promote. Iv. EDB emf SWITCHOVER: If the cluster status output shows that the master and standby(s) are in sync, you can invoke the following command to perform a switchover: $ /usr/edb/efm-3.1/bin/efm promote efm -switchover Promote/switchover command accepted by local agent. Proceeding with promotion and will reconfigure original master. Run the 'cluster-status' command for information about the new cluster state. And if we run the cluster-status command during the switchover $ /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 10.11.133.193 UP N/A Idle 10.11.128.81 UP UNKNOWN Standby 10.11.128.82 UP UP Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 10.11.133.193 Standby priority host list: 10.11.128.82 Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Standby 10.11.128.82 0/44000098 No master database was found. Idle Node Status (idle nodes ignored in XLog location comparisons): Address XLog Loc Info -------------------------------------------------------------- 10.11.128.81 UNKNOWN Connection to 10.11.128.81:5444 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. And we can see the promoting status $ /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 10.11.133.193 UP N/A Idle 10.11.128.81 UP UNKNOWN Promoting 10.11.128.82 UP UP Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 192.168.56.30 Standby priority host list: (List is empty.) Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Master 10.11.128.82 0/44000170 No standby databases were found. Idle Node Status (idle nodes ignored in XLog location comparisons): Address XLog Loc Info -------------------------------------------------------------- 10.11.128.81 UNKNOWN Connection to 10.11.128.81:5444 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. And after a few time we can see that the new master is on 10.11.128.82 # /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 10.11.133.193 UP N/A Standby 10.11.128.81 UP UP Master 10.11.128.82 UP UP Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 192.168.56.30 Standby priority host list: 10.11.128.81 Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Master 10.11.128.82 0/440001A8 Standby 10.11.128.81 0/440001A8 Standby database(s) in sync with master. It is safe to promote. To switch back, invoke the "promote" command once again. Iv. EDB emf AUTOMATIC FAILOVERS. Let’s simulate a crash of our primary database by killing the corresponding postgres process.you can use following anyone to killing # kill -9 postgres process id # pkill postgres If we check the cluster status from the witness server, we can see that the master DB STATUS is in an UNKNOWN. $ /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 10.11.133.193 UP N/A Standby 10.11.128.81 UP UP Idle 10.11.128.82 UP UNKNOWN Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 192.168.56.30 Standby priority host list: 10.11.128.81 Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Standby 10.11.128.81 0/440001A8 No master database was found. Idle Node Status (idle nodes ignored in XLog location comparisons): Address XLog Loc Info -------------------------------------------------------------- 10.11.128.82 UNKNOWN Connection to 10.11.128.82:5444 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. In the alert log in our standby server 10.11.128.81, we can see that that the database is converted to a primary one a few time after. we can confirm by querying the cluster status $ /usr/edb/efm-3.1/bin/efm cluster-status efm Cluster Status: efm VIP: Agent Type Address Agent DB Info -------------------------------------------------------------- Witness 192.168.56.30 UP N/A Master 10.11.128.81 UP UP Allowed node host list: 10.11.128.81 10.11.128.82 10.11.133.193 Membership coordinator: 192.168.56.30 Standby priority host list: (List is empty.) Promote Status: DB Type Address XLog Loc Info -------------------------------------------------------------- Master 10.11.128.81 0/440002B8 No standby databases were found. Note: On the old primary 10.11.128.82 we can see the contents of the file recovery.conf automatically created by EDB Failover manager To rebuild our standby database we have to edit the recovery.conf file. A virtual IP can be also configured. The official documentation can help regarding this. VIRTUAL IP: When using a virtual IP (VIP) address with Failover Manager, it is important to test the VIP functionality manually before starting failover manager. This will catch any network-related issues before they cause a problem during an actual failover. The following steps test the actions that failover manager will take. The example uses the following property values: virtualIp=172.24.38.239 virtualIp.interface=eth0 virtualIp.prefix=255.255.255.0=24 pingServerCommand=/bin/ping -q -c3 -w5 When instructed to ping the VIP from a node, use the command defined by the pingServerCommand property. 1. Ping the VIP from all nodes to confirm that the address is not already in use: # /bin/ping -q -c3 -w5 172.24.38.239 PING 172.24.38.239 (172.24.38.239) 56(84) bytes of data. --- 172.24.38.239 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3000ms You should see 100% packet loss. 2. Run the efm_address add4 command on the Master node to assign the VIP and then confirm with ip address: # efm_address add4 eth0 172.24.38.239/24 # ip address eth0 Link encap:Ethernet HWaddr 36:AA:A4:F4:1C:40 inet addr:172.24.38.239 Bcast:172.24.38.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:247 3. Ping the VIP from the other nodes to verify that they can reach the VIP: # /bin/ping -q -c3 -w5 172.24.38.239 PING 172.24.38.239 (172.24.38.239) 56(84) bytes of data. --- 172.24.38.239 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.023/0.025/0.029/0.006 ms You should see no packet loss. 4. Use the efm_address del command to release the address on the master node and confirm the node has been released with ip address: # efm_address del eth0 172.24.38.239/24 # ip address eth0 Link encap:Ethernet HWaddr 22:00:0A:89:02:8E inet addr:10.137.2.142 Bcast:10.137.2.191 ... The output from this step should not show an eth0 interface 5. Repeat step 3, this time verifying that the Standby and Witness do not see the VIP in use: # /bin/ping -q -c3 -w5 172.24.38.239 PING 172.24.38.239 (172.24.38.239) 56(84) bytes of data. --- 172.24.38.239 ping statistics --- 4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3000ms You should see 100% packet loss. Repeat this step on all nodes. 6. Repeat step 2 on all Standby nodes to assign the VIP to every node. You can ping the VIP from any node to verify that it is in use. # efm_address add4 eth0 172.24.38.239/24 # ip address eth0 Link encap:Ethernet HWaddr 36:AA:A4:F4:1C:40 inet addr:172.24.38.239 Bcast:172.24.38.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:247 After the test steps above, release the VIP from any non-Master node before attempting to start Failover Manager. More Documents: https://www.enterprisedb.com/docs/en/3.1/edbfm/EDB_Failover_Manager_Guide.1.08.html# https://www.enterprisedb.com/docs/en/3.1/edbfm/EDB_Failover_Manager_Guide.1.13.html#pID0E0N50HA https://blog.dbi-services.com/maintenance-scenarios-with-edb-failover-manager-2-primary-node/ https://www.enterprisedb.com/docs/en/3.1/edbfm/EDB_Failover_Manager_Guide.1.25.html# https://blog.dbi-services.com/edb-failover-manager-3-0-and-postgresql-10-1/ https://blog.dbi-services.com/whats-new-in-edb-efm-3-1/ https://vibhorkumar.blog/tag/edb/ https://clouddba.co/postgresql-dba-part-6-postgresql-high-availability-setup-using-efm/