Combined repository of OCF agents from the RHCS and Linux-HA projects
License: GNU General Public License v2.0
This repository contains resource agents (RAs) compliant with the Open Cluster Framework (OCF) specification.
These resource agents are used by two cluster resource management implementations:
More information about OCF resource agents is available in the OCF Resource Agent Developer's guide (link below).
On a PostgreSQL async streamin replication cluster with 2 nodes. When I stop the slave, then start it after waiting some minutes and some writes on the master, postgresql starts correctly on slave, restoring archive files and finish with this:
LOG: restored log file "000000200000000000000087" from archive
LOG: restored log file "000000200000000000000088" from archive
cp: cannot stat /var/lib/pgsql/tmp/archive/000000200000000000000089': No such file or directory cp: cannot stat/var/lib/pgsql/tmp/archive/00000021.history': No such file or directory
LOG: streaming replication successfully connected to primary
so in fact streaming replication is ok (and I tested writes on master, thay are written on slave).
but crm_mon -A does not reflect this:
Master/Slave Set: ms_pgsql [p_pgsql]
Masters: [ A ]
Slaves: [ B ]
Clone Set: cl_pingd [p_pingd]
Started: [ B A ]
Node Attributes:
Work to include MHA support in the MysQL module was at least being considered. Is this continuing? I wrote about it on MySQLFanBoy dot com.
the pgsql file is redhat specific. In Ubuntu server the paths are different (e.g. the path to pg_ctl is completely different in the resource agent it is specified as /usr/bin/pg_ctl whereas in ubuntu server it is /usr/lib/postgresql/9.3/bin/pg_ctl). The RA is not usable under ubuntu server as is.
Inspired by https://bugs.launchpad.net/ubuntu/+source/resource-agents/+bug/893023 & my recent experience:
The Apache RA calls httpd directly, not, say, through apache2ctl. One of the consequences is that certain setup and sanity-checking usually done by apache2ctl goes undone - including the creation of /var/run/apache2 if it doesn't exist already. This can result in Apache complaining something like "errorNo such file or directory: Cannot create SSLMutex with file `/var/run/apache2/ssl_mutex'"
This script has too many errors to run with any amount of confidence. I will submit a patched version. It in no way provides any confidence on a Ethernet interface being up and usable in its current form
Steps to reproduce:
Step 1: Kill mysql processes of Master.
Step 2: Wait until Heartbeat/Pacemaker switched Master/Slave.
Step 3: Repeat step 1 and step 2 two times.
Step 4: Observe Master/Slave status.
Expected result: Heartbeat/Pacemaker switches Master/Slave successfully.
Actually result: Heartbeat/Pacemaker stops switching Master/Slave.
After killed Master in 2nd time, I check the new Master 's log (ha-log) , the message "MySQL monitor succeeded (master)" didn't show up in log. Then i kill mysql processes of new Master (3rd time), the result is heartbeat/pacemaker stops switching Master/Slave.
And this is my pacemaker config:
node $id="fabe2f8e-9ba2-4f85-a644-fa16fe492830" ares
attributes apollo-log-file-p_mysql="mysql-bin.000067" apollo-log-pos-p_mysql="107"
node $id="fd5a954a-aadc-450e-9dda-ca2c18e980c2" apollo
primitive MailTo ocf:heartbeat:MailTo
params email="[email protected]"
primitive p_mysql ocf:heartbeat:mysql
params config="/etc/mysql/my.cnf" pid="/var/run/mysqld/mysqld.pid" socket="/var/run/mysqld/mysqld.sock" binary="/usr/bin/mysqld_safe" replication_user="root" replication_passwd="nec" test_user="root" test_passwd="nec" max_slave_lag="10" evict_outdated_slaves="false"
op monitor interval="1s" role="Master" timeout="120s"
op monitor interval="3s" timeout="120s"
op start interval="0" role="Stopped" timeout="120s" on-fail="restart"
op stop interval="0" timeout="120s"
meta is-managed="true"
primitive virtualIP ocf:heartbeat:IPaddr
params ip="192.168.103.223" cidr_netmask="255.255.255.0"
op monitor interval="1s"
meta is-managed="true"
ms ms_MySQL p_mysql
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true"
colocation mysql_co_ip inf: virtualIP ms_MySQL:Master
order my_MySQL_promote_before_vip inf: ms_MySQL:promote virtualIP:start
property $id="cib-bootstrap-options"
dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c"
cluster-infrastructure="Heartbeat"
stonith-enabled="false"
default-action-timeout="30"
cluster-recheck-interval="30s"
no-quorum-policy="ignore"
property $id="mysql_replication"
p_mysql_REPL_INFO="ares|mysql-bin.000034|107"
rsc_defaults $id="rsc-options"
resource-stickiness="1"
migration-threshold="1"
failure-timeout="15s
I was a bit surprised that despite description of IPAddr[2] just and IP address never was enough for configuration and I had to supply both netmask and NIC name to get things working properly. But a lot of tutorials said that's how it should be done, so I accepted it until got a configuration where NICs had different names on two machines in a cluster and migration of IP didn't work.
I never could get such a configuration to work in Debian squeeze, but on Debian lenny it worked. After digging into the code and configuration it came out that on lenny 'cat /proc/net/route' gives routing table like:
Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT eth0 006BD35F 00000000 0001 0 0 0 E0FFFFFF 0 0 0 eth1 0000000A 00000000 0001 0 0 0 00FFFFFF 0 0 0 eth2 0001000A 00000000 0001 0 0 0 00FFFFFF 0 0 0 eth0 00000000 1E6BD35F 0003 0 0 0 00000000 0 0 0
When on squeeze the order of the information is different:
Iface Destination Gateway Flags RefCnt Use Metric Mask MTU Window IRTT bond0 00000000 9E77D35F 0003 0 0 0 00000000 0 0 0 eth1 0000000A 00000000 0001 0 0 0 00FFFFFF 0 0 0 * FA00000A 00000000 0205 0 0 0 FFFFFFFF 0 0 0 bond0 8077D35F 00000000 0001 0 0 0 E0FFFFFF 0 0 0
Basically, with the newer kernels the default route 0/0 comes as a first line and matches any supplied IP address/mask combination, giving interface of the default route as a match. It may be suitable result in some cases, but just incidently. Also, the code tries to take route metric into account and rejects anything that is below already found one, although 'man route' says:
Metric The 'distance' to the target (usually counted in hops). It is not used by recent kernels, but may be needed by routing daemons.
And in most cases it's value 0 for all the entries. But current code just picks the first matching line with such a metric rejecting any following better routes with the same metric (0).
So, my proposal is to make this check more relaxed in the first place and consider equal metrics suitable as well. Still in this case default route possibly would be the best choice, so I add additional constrain - pick most specific matching route(with narrower netmask).
Here is the small patch:
--- findif.c.orig 2010-04-14 13:57:20.000000000 +0200
+++ findif.c 2012-01-27 22:06:14.000000000 +0100
@@ -171,11 +171,13 @@
long metric = LONG_MAX;
long best_metric = LONG_MAX;
int rc = 0;
-
+
char buf[2048];
char interface[MAXSTR];
FILE *routefd = NULL;
+ *best_netmask = 0;
+
if ((routefd = fopen(PROCROUTE, "r")) == NULL) {
snprintf(errmsg, errmsglen
, "Cannot open %s for reading"
@@ -199,17 +201,18 @@
, PROCROUTE, buf);
rc = -1; goto out;
}
- if ( (in->s_addr&mask) == (in_addr_t)(dest&mask)
- && metric < best_metric) {
- best_metric = metric;
- *best_netmask = mask;
- strncpy(best_if, interface, best_iflen);
+ if ((in->s_addr&mask) == (in_addr_t)(dest&mask)) {
+ if(metric <= best_metric && mask >= *best_netmask) {
+ best_metric = metric;
+ *best_netmask = mask;
+ strncpy(best_if, interface, best_iflen);
+ }
}
}
if (best_metric == LONG_MAX) {
snprintf(errmsg, errmsglen, "No route to %s\n", address);
- rc = 1;
+ rc = 1;
}
out:Testing both on lenny and squeeze gave consistent right results :)
Still, this and current code doesn't take into account, for example case when 'reject' route present in the routing table:
* FA00000A 00000000 0205 0 0 0 FFFFFFFF 0 0 0
If desired IP happen to fall into the rejected range an '*' interface returned as a result. Well, it can easily be verified in the code, as well as on a higher level.
Regards,
Timur.
Hi!
LSB specification recently introduced /run as storage for run-time information with the idea to have it available as soon as possible in the boot process(as /var could be separate FS mounted later). At least Debian implements /run as tmpfs, which means that you can't rely on the presence of any sub-directories there.
There are at least several RAs which expect to find their PID files and other run-time information in the subdirectories of the [/var]/run. Obviously they fail on new Debian(and Ubuntu, I guess).
Those have to be fixed by checking the existence of the PID directory and creation of it if necessary.
Affected RAs at least named(#350), zabbixserver, mysql, mysql-proxy and lxc.
apache2, seems, has a cure for it, although I haven't checked(Could have permissions problems).
With regards,
Timur Bakeyev.
Small issue:
On a cluster configured with rep_mode=sync, I stopped all pgsql resources, then edited the cluster configuration to use rep_mode=async. Then pgsql will not start because postgresql.conf still has the include entry to the non existent rep_mode.conf file.
May be, before starting postgresql, the Ra should check postgresql.conf and remove the RA rep_mode include if cluster rep_mode configuration is not "sync"?
In Filesystem RA and parameter force_clones the $OCF_RESKEY_force_clones_default
is never set in the meta data. Copy/paste error or is there some reason for it? I suppose it should be "false"
thanks
During switch overing in logs:
Aug 8 11:53:10 server2 lrmd: [2462]: info: RA output: (P_NFS:stop:stderr) umount: /var/lib/nfs: device is busy
Aug 8 11:53:10 server2 lrmd: [2462]: info: RA output: (P_NFS:stop:stderr) umount: /var/lib/nfs: device is busy
Aug 8 11:53:10 server2 nfsserver[23115]: INFO: NFS server stopped
I'm use drbr with nfs server. NFS config part:
primitive P_NFS ocf:heartbeat:nfsserver
params nfs_ip="10.4.24.77" nfs_init_script="/etc/init.d/nfs" nfs_shared_infodir="/mnt/data/varlibnfs" nfs_notify_cmd="/sbin/rpc.statd"
op monitor interval="5s"
primitive P_NFSLOCK lsb:nfslock
primitive P_NFS_IP ocf:heartbeat:IPaddr2
params ip="10.4.24.77" nic="eth0"
op monitor interval="5s"
group G_NFS P_DRBD_FS P_NFS_IP P_NFSLOCK P_NFS
IN case we perform lazy umount (-l option) all seem work fine:
unbind_tree ()
{
if mount | grep -q "rpc_pipefs on /var/lib/nfs/rpc_pipefs"; then
umount /var/lib/nfs/rpc_pipefs
fi
if is_bound $fp /var/lib/nfs; then
umount -l /var/lib/nfs
fi
}
What do I do wrong? =)
In the old days (using vm.sh and rgmanager), a domain was never defined on a kvm server. In stead "virsh create" was used to start a domain in a non-persistent way and when a live migration happened, the domain then just moved to the other side without issues or traces on the original kvm server. Now I have 2 kvm servers that both have all virtual machines defined, which is imho dangerous if somebody just did a "virsh start" on the wrong kvm server ... Sanlock or virtlockd protects against that of course, but still ...
It seems the "define" is used here for 2 things:
To get the domain name, one could simply do:
egrep ".$" /path/to/XMLFILE |sed -e 's/.(.*)</name>$/\1/'
And to get the domain state (since running is the one that interests us), the main logic of the function VirtualDomain_Status() would be:
virsh list|grep running|grep DOMAINNAME
As a result, the functions VirtualDomain_Define() and VirtualDomain_Cleanup_Statefile() could be removed, and also the use of a statefile per domain (containing just the domain name) would no longer be needed.
One could argue that we could use a undefine when stopping a vm, and a --undefinefromsource when migrating, but in the case the kvm server crashes it would still leave the domain defined.
If there's interest in a patch, I can create one ...
Franky
I'm running a simple symmetric cluster, resources are configured like this :
primitive ip-ldap-lo ocf:heartbeat:IPaddr2 \
params ip="10.0.1.7" cidr_netmask="32" nic="lo" \
op monitor interval="60"
primitive slapd ocf:heartbeat:slapd params....
group ip-slapd ip-ldap-lo slapd
clone c-ip-slapd ip-slapd meta interleave="true"
This is "backend cluster" for IPVS/DR
In that case IPAddr2 is assuming the ip as a cluster ip. I don't need it ! :)
Looking at the https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/IPaddr2 i can see:
what do you reckon ?
Is this a correct use-case ?
If so, how would change the behavior ?
how about adding a && [ "$NIC" != "lo" ] line 467 ?
Environment 1:(32bit Fedora12)
uname -a
Linux localhost.localdomain 2.6.32.26-175.fc12.i686.PAE #1 SMP Wed Dec 1 21:45:50 UTC 2010 i686 i686 i386 GNU/Linux
cat /etc/issue
Fedora release 12 (Constantine)
resource-agents: resource-agents-3.0.17-1.fc12.i686
pacemaker: pacemaker-1.0.7-1.fc12.i686
crm configure primitive test_IPv6 ocf:heartbeat:IPv6addr params ipv6addr="fe80::0250:56ff:fea2:ff3" cidr_netmask="64" nic="eth0" op monitor timeout="15s" interval="15s"
we can find fe80::0250:56ff:fea2:ff3 binded to eth0,but there's error message in /var/log/messages:
ocalhost IPv6addr: [13246]: ERROR: bind() failed: Invalid argument
This really can bind ipv6 address on adaptor ,but there's error message!!!!!!
Environment 1:(64bit Centos5.5)
uname -a
Linux localhost.localdomain 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/issue
CentOS release 5.5 (Final)
resource-agents : resource-agents-1.0.4-1.1.el5
pacemaker:pacemaker-1.0.11-1.2.el5
crm configure primitive test_IPv6 ocf:heartbeat:IPv6addr params ipv6addr="fe80::0250:56ff:fea2:ff3" cidr_netmask="64" nic="eth0" op monitor timeout="15s" interval="15s",
we can find we can find fe80::0250:56ff:fea2:ff3 could not bind to eth0,
error log in /var/log/messages:
Nov 1 03:06:37 localhost IPv6addr: [5372]: ERROR: bind() failed: Invalid argument
Nov 1 03:06:37 localhost kernel: IPv6addr[5372]: segfault at 0000003ffffffffd rip 0000003dae27273e rsp 00007fffe1cb8b00 error 4
Nov 1 03:06:37 localhost lrmd: [5330]: WARN: Managed testbni_1:start process 5372 killed by signal 11 [SIGSEGV - Segmentation violation].
Nov 1 03:06:37 localhost lrmd: [5330]: ERROR: Managed testbni_1:start process 5372 dumped core
I debug the code find that the Invalid argument is not a big deal,it seemed u_int8_t payload in send_ua(struct in6_addr src_ip, char* if_name) not initiallized on 64-bit os,and then when free(payload) ,kernel error happens!!!
I check the source code of these 2 IPv6addr.c is the same ,if i compile on the first env(fedora12) ,then copied to centos(64-bit),and create resource ,it can bind ipv6 address to adaptor ,the same with working on env1,also have error messages.
I'm looking at the 'anything' RA as an example and basis to my own one.
And I can see an issue there with the 'stop' method.
At some stage it removes the pid file, and then calls the 'anything_status' function that returns success automatically if the pid file is not found:
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/anything#L134
As of linux 3.2.0 at least, an nfs mounted directory is reported in /proc/mounts with a trailing "/":
192.168.3.56:/srv/nfs/fweb/ /srv/fweb nfs4 [...]
thus a resource configured with:
params name="fweb_fs" mountpoint="/srv/fweb" host="192.168.3.56" export="/srv/nfs/fweb" fstype="nfs" [...]
ends with a condition like:
-n 192.168.3.56:/srv/nfs/fweb/ -a 192.168.3.56:/srv/nfs/fweb/ = 192.168.3.56:/srv/nfs/fweb
which does not returns the expected result:
<err> filesystem:fweb_fs: 192.168.3.56:/srv/nfs/fweb is not mounted on /srv/fweb
here is a quick patch:
--- tmp/fs-lib.sh 2012-04-26 12:38:39.000000000 +0200
+++ fs-lib.sh 2012-04-26 12:57:44.000000000 +0200
@@ -264,6 +264,10 @@
real_device "$1"
dev="$REAL_DEVICE"
+ # On recent Linux versions, nfs mountpoint have a trailing / in
+ # /proc/mounts
+ dev="${dev%/}"
+
if [ -z "$dev" ]; then
ocf_log err "$OCF_RESOURCE_INSTANCE: is_mounted: Could not match $1 with a real device"
return $OCF_ERR_ARGS
@@ -291,6 +295,11 @@
real_device "$tmp_dev"
tmp_dev="$REAL_DEVICE"
+ # On recent Linux versions, nfs mountpoint have a trailing / in
+ # /proc/mounts
+ tmp_dev="${tmp_dev%/}"
+
+
# XXX fork/clone warning XXX
# Mountpoint from /proc/mounts containing spaces will
# have spaces represented in octal. printf takes care
Hello
I'm having a problem getting conntrackd ms to work with a colocation constraint. I need to have Master conntrackd only on the node that has the "lip" IPaddr2 primitive running on it.
Here are my specs:
Ubuntu: 12.04
corosync: 1.4.2
crm: 1.1.6
conntrackd: 1.0.0
Here is my configuration (notice I weighted the fw02 higher than fw01 to test my failover):
node fw01 \
attributes firewall="100"
node fw02 \
attributes firewall="150"
primitive lip ocf:heartbeat:IPaddr2 \
meta target-role="Started" \
params ip="192.168.100.2" cidr_netmask="24" nic="eth0"
primitive updated-conntrackd ocf:updates:conntrackd
ms conntrackd updated-conntrackd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Master"
location conntrackd-run conntrackd \
rule $id="conntrackd-run-rule-0" -inf: not_defined firewall or firewall number:lte 0 \
rule $id="conntrackd-run-rule-1" firewall: defined firewall
location lip-loc lip \
rule $id="lip-loc-rule-0" -inf: not_defined firewall or firewall number:lte 0 \
rule $id="lip-loc-rule-1" firewall: defined firewall
colocation conntrackd-loc inf: conntrackd:Master lip:Started
property $id="cib-bootstrap-options" \
dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
symmetric-cluster="false" \
no-quorum-policy="ignore" \
stonith-enabled="false" \
last-lrm-refresh="1339040513"So based on my configuration above, I would expect conntrackd to be Master on fw02, but it's not:
root@fw01:~# crm status
============
Last updated: Wed Jun 6 20:46:55 2012
Last change: Wed Jun 6 20:41:53 2012 via crmd on fw01
Stack: openais
Current DC: fw01 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
3 Resources configured.
============
Online: [ fw01 fw02 ]
Master/Slave Set: conntrackd [updated-conntrackd]
Masters: [ fw01 ]
Slaves: [ fw02 ]
lip (ocf::heartbeat:IPaddr2): Started fw02
Interesting thing is... if I add "lip" to a group with another primitive and run the same logic, failover works just fine:
root@fw01:~# crm configure show
node fw01 \
attributes firewall="100"
node fw02 \
attributes firewall="150"
primitive lip ocf:heartbeat:IPaddr2 \
params ip="192.168.100.2" cidr_netmask="24" nic="eth0"
primitive lip2 ocf:heartbeat:IPaddr2 \
params ip="192.168.100.101" cidr_netmask="24" nic="eth0"
primitive updated-conntrackd ocf:updates:conntrackd
group gateway lip lip2 \
meta target-role="Started"
ms conntrackd updated-conntrackd \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Master"
location conntrackd-run conntrackd \
rule $id="conntrackd-run-rule-0" -inf: not_defined firewall or firewall number:lte 0 \
rule $id="conntrackd-run-rule-1" firewall: defined firewall
location gateway-loc gateway \
rule $id="lip-loc-rule-0" -inf: not_defined firewall or firewall number:lte 0 \
rule $id="lip-loc-rule-1" firewall: defined firewall
colocation conntrackd-loc inf: conntrackd:Master gateway:Started
property $id="cib-bootstrap-options" \
dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
symmetric-cluster="false" \
no-quorum-policy="ignore" \
stonith-enabled="false" \
last-lrm-refresh="1339041080"
root@fw01:~# crm status
============
Last updated: Wed Jun 6 20:52:17 2012
Last change: Wed Jun 6 20:52:03 2012 via cibadmin on fw01
Stack: openais
Current DC: fw01 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
4 Resources configured.
============
Online: [ fw01 fw02 ]
Master/Slave Set: conntrackd [updated-conntrackd]
Masters: [ fw02 ]
Slaves: [ fw01 ]
Resource Group: gateway
lip (ocf::heartbeat:IPaddr2): Started fw02
lip2 (ocf::heartbeat:IPaddr2): Started fw02It would be nice to be able to preserve a single source IPv6 address on a host after additional addresses (usually a /64) are added to the interface. It appears most people (including me, outside the HA framework) are accomplishing this by passing preferred_lft 0 to ip addr. However that doesn't appear to be an option right now with the ipv6addr ocf agent.
I'm not much of a C programmer or I'd submit a patch. Figured I'd add an issue, though, to get some thoughts on this idea. Perhaps there's a better way to accomplish this, but I haven't found it? I tried to research using gar.conf, but it appears to be focused more on destination address selection than source addressing.
Asterisk resource does not failover to the other node validate function fail during the stop command.
Because the stop will return negative answer (5)
Either use: [ "$1" != "stop" ] && asterisk_validate
Instead of asterisk_validate
or change the asterisk_validate function.
I created a VirtualDomain RA configuration, but failed to put the libvirt xml for the guest vm in place.
When the RA was started, it failed to start with rc=5 (not installed) as expected, and tried to migrate the guest away to another node.
However, in doing so, it called 'stop' against the RA, which also failed with the same rc=5 error. This apparent failure to stop resulted in a stonith of the node.
Can VirtualDomain be fixed to avoid this?
Abbreviated Logs:
node1 (vm host):
Sep 24 15:07:18 node1 VirtualDomain(vm4)[22276]: ERROR: Configuration file
/etc/libvirtcfg/vm4.xml does not exist or is not readable.
Sep 24 15:07:18 node1 crmd[5634]: info: process_lrm_event: LRM operation vm4_start_0 (call=54, rc=5, cib-update=68, confirmed=true) not installed
Sep 24 15:07:18 node1 lrmd: [5631]: info: rsc:vm4:55: stop
Sep 24 15:07:18 node1 VirtualDomain(vm4)[22310]: ERROR: Configuration file /etc/libvirtcfg/vm4.xml does not exist or is not readable.
Sep 24 15:07:18 node1 crmd[5634]: info: process_lrm_event: LRM operation vm4_stop_0 (call=55, rc=5, cib-update=69, confirmed=true) not installed
STONITH
node2 (DC calling stonith):
Sep 24 15:07:18 node2 pengine[5057]: notice: unpack_rsc_op: Preventing vm4 from re-starting on node1: operation start failed 'not installed' (rc=5)
Sep 24 15:07:18 node2 pengine[5057]: warning: unpack_rsc_op: Processing failed op vm4_last_failure_0 on node1: not installed (5)
Sep 24 15:07:18 node2 pengine[5057]: warning: common_apply_stickiness: Forcing vm4 away from node1 after 1000000 failures (max=5)
Sep 24 15:07:18 node2 pengine[5057]: notice: LogActions: Stop vm4#011(node1)
Sep 24 15:07:18 node2 stonith-ng[5054]: info: initiate_remote_stonith_op: Initiating remote operation reboot for node1: 6d82f19a-75d6-41d5-815d-36176b57d940
The function get_link_status in ethmonitor does not return the correct link status in the following scenarios:
1). when the wire is unplugged from network interface
eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
2). when lower layer is down for a VLAN
vlan2@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN
I tested link-local address ,it can bind on adaptor ,but site-local/global(2001::1) address can not ,please verify if this is an issue.
Hi,
I have patched the version to not use tmpfile on disk. It saves I/O and usage spaces. Here the code:
http://pastebin.com/zactvNXL
And also, log typo error with demote, promote.
Best regards,
I was trying to use named RA from the GIT repository to get redundant DNS server(s) in my environment. Cluster is running on Debian 6.0 Squeeze with the stock corosync/pacemaker.
Configuration looks like:
primitive named ocf:heartbeat:named \
params named_user="bind" named_config="/etc/bind/named.conf" monitor_request="ns.domain.nl" monitor_response="10.0.0.201" \
op monitor interval="20s" timeout="30s" \
op start interval="0" timeout="60s" \
op stop interval="0" timeout="60s"
clone run_named named \
meta clone-max="2" clone-node-max="1" target-role="Started"
colocation use_named inf: ip_int run_named
order activate_named inf: ip_int:start run_named:start
Log file showed that named is starting normally, but immediately gets killed. A bit of debugging of the RA revealed that although named is starting OK the following code that supposed to find it in the list of running processes fails to do so on whatever reason and named considered dead, effectively disabling RA:
pid=`ps -e -o pid,command,args | grep "$pattern" | grep -v grep | awk '{print $1}'`
Running the same sequence(with expanded $pattern) from the command line perfectly worked and returned the PID of the named process.
I've tried to simplify the expression and replaced it with:
pid=`pgrep -f "$pattern"`
Which is a functional equivalent of the previous command.
Amusingly enough it worked fine and RA for the last two months didn't cause any problems.
Grepping RAs shows that at least 10 of them use "pgrep", so I'd consider it's presence in the system a hard dependency anyhow.
--- /root/GIT/resource-agents/heartbeat/named 2012-03-13 18:35:01.000000000 +0100
+++ named 2012-08-03 06:10:01.000000000 +0200
@@ -247,7 +247,7 @@
pattern="$pattern.*-c $OCF_RESKEY_named_config"
fi
- pid=`ps -e -o pid,command,args | grep "$pattern" | grep -v grep | awk '{print $1}'`
+ pid=`pgrep -f "$pattern"`
echo $pid
}
In findif.sh OCF_ERR_CONFIGURED is often returned, for instance if the NIC is not found. How do we know that it is the configuration issue and not just that the NIC doesn't exist for another reason? Didn't test this or anythin so I might be completely off here.
This breaks the backward compatibility with the heartbeat stack.
#285 Low: build: Move binaries in /usr/lib/heartbeat to /usr/libexec/heartbeat to avoid multi-lib inconsistencies
The heatbeat init script expects the heartbeat binary resides in $HA_BIN/heartbeat (/usr/lib64/heartbeat/heatbeat) but it now tries to look /usr/libexec/heartbeat/heartbeat and fails.
When a virtual machine managed by the cluster is rebooted from inside the vm with a shutdown -r now command the cluster occasionally detects it as failed (if your monitor operation fires while it's in the off state). VMs are normally only in this off state on reboot for <1second so this may be a very minor tweak to the monitor to sample more than once before deciding that a previously running VM has actually failed.
The commits 981c45d and f00dcaf fundamentally change how the VirtualDomain RA script manage domains with libvirt. While I agree that this changed functionality is a good improvement to the script, the changes are likely to break many existing configurations.
In all the examples I can see, including on http://www.linux-ha.org/wiki/VirtualDomain_(resource_agent), the domain XML file is kept under /etc/libvirt/qemu/{domain}.xml (e.g. params config="/etc/libvirt/qemu/test.xml" hypervisor="qemu:///system"). With this configuration, the XML file vanishes when the resource agent attempts to start the VM; it gets undefined first, then create attempts to start it from the no-longer-existing file.
Recently IPaddr got back handy hooks for local start and stop scripts. I find them handy and think that in certain situations such hooks can be useful in IPaddr2 too. Also that would make semantics of both script more coherent.
OCF_RESKEY_local_start_script
OCF_RESKEY_local_stop_script
My initial reason for getting such functionality is a result of using Cluster IP option in the IPaddr2. Enabling it brings ip_conntrack as a dependency which without additional iptables rules tries to track all the traffic routed through the host. On busy sites that very quickly overflows conntrack table and leads to the problems with the new and existing connections.
iptables allows to exclude certain traffic from tracking by using NOTRACK target in the raw table.
Having simple script that mangles iptables on start and stop of the RA makes the problem disappear.
I can imagine other situations when it is necessary to perform some additional actions on RA start and stop.
So, would it be possible to add local start/stop script hooks to IPaddr2 as well?
With regards,
Timur Bakeyev.
This param has been pluralised to directories in 9.3
[ $check_config_rc -eq 0 ] && : ${OCF_RESKEY_socketdir=`get_pgsql_param unix_socket_directory`}
needs either version detection or some logic to try get_pgsql_param unix_socket_directoriesThe patch below removes superfluous headings, renders option names in bold, and rephrases the sentence dealing with IPv6 literal addresses.
--- ldirectord/ldirectord.in.orig 2013-02-07 13:17:42.000000000 +0100
+++ ldirectord/ldirectord.in 2013-12-17 13:32:04.450002165 +0100
@@ -424,7 +424,6 @@
the IP-address and port of the real server is overridden, otherwise the
IP-address and port of the real server is used.
-=head2
For TCP and UDP (non fwmark) virtual services, unless the forwarding method
is B<masq> and the IP address of a real server is non-local (not present on
a interface on the host running ldirectord) then the port of the real
@@ -433,10 +432,9 @@
method is B<masq>. This is due to the way that the underlying LVS code in
the kernel functions.
-=head2
More than one of these entries may be inside a virtual section. The
-checktimeout, negotiatetimeout, checkcount, fallback, emailalert,
-emailalertfreq and quiescent options listed above may also appear inside a
+B<checktimeout>, B<negotiatetimeout>, B<checkcount>, B<fallback>, B<emailalert>,
+B<emailalertfreq> and B<quiescent> options listed above may also appear inside a
virtual section, in which case the global setting is overridden.
B<checktype =
@@ -703,8 +701,8 @@
=head1 IPv6
Directives for IPv6 are virtual6, real6, fallback6.
-IPv6 addresses specified for virtual6, real6, fallback6 and a file
-of maintenance directory should be enclosed by
+Literal IPv6 addresses specified for these directives and used as file names
+in the maintenance directory should be enclosed in
brackets ([2001:db8::abcd]:80).
Following checktype and service are supported.
If LVS_SUPPORT is enabled,
conflicting ip addresses need to be removed from the local loopback adapter.
Currently the Interface with the conflicting IP cannot be found, since the subnet mask passed to the function find_interface() is the SNM to be assiged to the interface that should be brought up.
/usr/lib/ocf/resource.d/heartbeat/IPaddr2
682- if ocf_is_true ${OCF_RESKEY_lvs_support}; then
683: for i in `find_interface $OCF_RESKEY_ip $NETMASK`; do
684- case $i in
685- lo*)
686- remove_conflicting_loopback $OCF_RESKEY_ip 32 255.255.255.255 lo
687- ;;
688- esac
689- done
690- fiIf we want to remove an IP address with SNM 32 with (pls see line 686) , i suggest to search for the corresponding interface which has IP address with SNM 32 or alternatively introduce a new parameter for this loopback_netmask.
683c683
< for i in `find_interface $OCF_RESKEY_ip $NETMASK`; do
---
> for i in `find_interface $OCF_RESKEY_ip 32`; doThe pgslq RA should really use rsync to synchronise the archive - as an option.
I get that the author doesn't want to mess up with rsync, but there really should be an option to do it.
Otherwise restart is required where we could get away without one. The issue is fixed in 9.3, but for 9.2 - it would be nice to have the rsync option please.
I'd like to understand why there are calls in
resource-agents / rgmanager / src / resources / lvm_by_vg.sh
that use vgreduce --removemissing
(Currently at lines 232)
if ! vgreduce --removemissing --force $OCF_RESKEY_vg_name; then
ocf_log err "Failed to make $OCF_RESKEY_vg_name consistent"
return $OCF_ERR_GENERIC
fi
This has caused problems for us with a major client in this scenario:
We were able to recover the situation using vgcfgrestore and archived metadata.
However this is very unexpected behaviour! I would expect the Resource start up to FAIL with an ERROR notice if PV's are missing so that we could fix the problem (simply rescan the scsi_hosts) and not to hose the volume group metadata to the point that it looked initially that we had either major on disc corruption or actual data loss.
So right now this looks like a bug or design fault - and I cannot see a really good reason for the code in question. Please help me understand why its there and consider an alternative approach to making sure the VG can be imported without unexpected side effects.
Many thanks!
The OCF Resource Agent Developer Guide at http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html states that "An OCF compliant resource agent can be implemented in any programming language. The API is not language specific. However, most resource agents are implemented as shell scripts, which is why this guide primarily uses example code written in shell language."
The current implementation of the ocf-tester invokes the tested ra script with constructs like (eg line 214):
bash /path/to/resourceagent validate-all | cd /usr/share/resource-agents && $METADATA_LINT)
Presumably this should be something more like:
(/path/to/resourceagent validate-all) | cd /usr/share/resource-agents && $METADATA_LINT)
To remove the assumption that the resource agent is implemented in sh or bash.
On line 335 of the nginx RA, it loads $pid from the config file. Immediately after, it proceeds to check $PidFile and then set it to the default PID file location. This means it would work if you only had one instance of nginx running (& the pid file was in the default path), but it would not work with multiple instances of nginx (or non-default PID locations).
Recent Debian introduced support of the LSB /run directory, ewhich is mounted as tmpfs FS. Which implies that on startup it's entirely empty. RA named expects to find named pid file in the {/var}/run/named/named.pid and so fails, as /named/ sub-directory doesn't exist.
Here is a small patch to fix the situation.
--- named.orig 2013-11-02 05:28:37.728810791 +0100
+++ named 2013-11-02 23:03:57.000000000 +0100
@@ -202,6 +202,8 @@
# Validate most critical parameters
named_validate_all() {
+ local pid_dir
+
check_binary $OCF_RESKEY_named
check_binary $OCF_RESKEY_rndc
check_binary $OCF_RESKEY_host
@@ -216,6 +218,11 @@
fi
fi
+ pid_dir=`dirname "${OCF_RESKEY_named_pidfile}"`
+ if [ -n "$pid_dir" -a ! -d "$pid_dir" ]; then
+ ocf_log info "PID file directory $pid_dir doesn't exist"
+ fi
+
getent passwd $OCF_RESKEY_named_user >/dev/null 2>&1
if [ ! $? -eq 0 ]; then
ocf_log err "User $OCF_RESKEY_named_user doesn't exist";
@@ -334,7 +341,7 @@
named_start() {
local root_dir_opt
- local pid
+ local pid_dir pid group
root_dir_opt=""
named_status && return $OCF_SUCCESS
@@ -360,6 +367,15 @@
return $OCF_ERR_GENERIC
fi
+ group=`id -gn ${OCF_RESKEY_named_user}`
+ # dirs under [/var]/run can go away on reboots.
+ pid_dir=`dirname "${OCF_RESKEY_named_pidfile}"`
+ if [ ! -d "$pid_dir" ]; then
+ ocf_log info "Creating PID dir: $pid_dir"
+ mkdir -p "$pid_dir"
+ chmod 775 "$pid_dir"
+ chown root:"$group" "$pid_dir" >/dev/null 2>&1 || true
+ fi
pid=`named_getpid`
With regards,
Timur Bakeyev
In iscsi script:
2)Thesession state verification is not enough just to list. In case of failure, it is still possible to list. The right thing
would be to check the status of session. Agree RFC3720, and open_iscsi, the session transitions are:
FREE -> ACTIVE -> LOGGED_IN -> FAILED -> FREE
There are various configurable timeouts between state transitions "LOGGED_IN <-> FAILED" and "FAILED <-> FREE"
I think the disk should be considered as lost if it reaches the state "FREE"
3)For both suggestions, here is a patch for last version:
--- iscsi 2012-06-21 12:21:43.229289181 -0300
+++ iscsi.new 2012-06-21 12:34:07.457286218 -0300
@@ -257,7 +257,22 @@
$iscsiadm -m node -p $1 -T $2 -u
}
open_iscsi_status() {
- $iscsiadm -m session 2>/dev/null | grep -qs "$2$"
+ #Agree RFC3720, and open_iscsi, the session transitions are:
+ # FREE -> ACTIVE -> LOGGED_IN -> FAILED -> FREE
+ # \-->Here there is a connection failure
+ #There are various configurable timeouts between state transitions
+ # "LOGGED_IN <-> FAILED" and "FAILED <-> FREE"
+ #We consider the disk lost, if it reach the state "FREE"
+ local session_id=$($iscsiadm -m session 2>/dev/null| \
+ grep -s "$2$"|sed -rn -e 's/.*\[([0-9]+)\].*/\1/p')
+ [ -z "$session_id" ] && return 1
+ local session_state=$($iscsiadm -m session -r $session_id -P 1| \
+ sed -rn -e 's/.*iSCSI +Session +State *: *([A-Z_]+)/\1/p')
+ case $session_state in
+ ACTIVE|LOGGED_IN) return 0;; #All ok
+ FAILED) return 0;; #still there's a chance to recover
+ FREE|*) return 1;;
+ esac
}
#
@@ -378,33 +393,35 @@
discovery_type=${OCF_RESKEY_discovery_type}
udev=${OCF_RESKEY_udev}
-$discovery # discover and setup the real portal string (address)
-case $? in
-0) ;;
-1) [ "$1" = stop ] && exit $OCF_SUCCESS
- [ "$1" = monitor ] && exit $OCF_NOT_RUNNING
- [ "$1" = status ] && exit $LSB_STATUS_STOPPED
- exit $OCF_ERR_GENERIC
-;;
-2) [ "$1" = stop ] && {
- iscsi_monitor || exit $OCF_SUCCESS
- }
- ocf_is_probe && {
- iscsi_monitor; exit
- }
- exit $OCF_ERR_GENERIC
-;;
-3) ocf_is_probe && exit $OCF_NOT_RUNNING
- if ! is_iscsid_running; then
- [ $setup_rc -eq 1 ] &&
- ocf_log warning "iscsid.startup probably not correctly set in /etc/iscsi/iscsid.conf"
- [ "$1" = stop ] && exit $OCF_SUCCESS
- exit $OCF_ERR_INSTALLED
- fi
- exit $OCF_ERR_GENERIC
-;;
-esac
-
+if [ -n "$1" -a "$1" != monitor -a "$1" != status ];then
+ $discovery # discover and setup the real portal string (address)
+ case $? in
+ 0) ;;
+ 1) [ "$1" = stop ] && exit $OCF_SUCCESS
+ [ "$1" = monitor ] && exit $OCF_NOT_RUNNING
+ [ "$1" = status ] && exit $LSB_STATUS_STOPPED
+ exit $OCF_ERR_GENERIC
+ ;;
+ 2) [ "$1" = stop ] && {
+ iscsi_monitor || exit $OCF_SUCCESS
+ }
+ ocf_is_probe && {
+ iscsi_monitor; exit
+ }
+ exit $OCF_ERR_GENERIC
+ ;;
+ 3) ocf_is_probe && exit $OCF_NOT_RUNNING
+ if ! is_iscsid_running; then
+ [ $setup_rc -eq 1 ] &&
+ ocf_log warning "iscsid.startup probably not correctly set in /etc/iscsi/iscsid.conf"
+ [ "$1" = stop ] && exit $OCF_SUCCESS
+ exit $OCF_ERR_INSTALLED
+ fi
+ exit $OCF_ERR_GENERIC
+ ;;
+ esac
+fi
+
# which method was invoked?
case "$1" in
start) iscsi_start
Changing the OCF_RESKEY_clientspec to a character class makes exportfs_monitor correctly report the status
For example:
web* will not pass the grep and monitor will fail
web1.full.domain.com will pass the grep and pass monitor
Changing clientspec web* to web[*] will pass the grep and monitor will work.
186c148,149
NEW=`echo ${OCF_RESKEY_clientspec} | sed 's/*/[*]/'` exportfs | grep -zqs "${OCF_RESKEY_directory}[[:space:]]*${NEW}"
In the apache resource agent, the code uses the exit code from grep directly - as though it were a proper OCF exit code. It is not.
This causes a "not running" condition to become "generic error" - which is not right. Pacemaker treats the two very differently.
The second problem is the non-conforming implementation of OCF_CHECK_LEVEL. The OCF specification states that if you receive an OCF_CHECK_LEVEL that you do not support, you are to treat it as the next lower level that you do support - and you're required to support OCF_CHECK_LEVEL of 0. In effect, this means that any positive OCF_CHECK_LEVEL is legal for any resource agent - unlike this implementation of the apache agent, which allows only one value.
The following code looks wrong to me, because even on infiniband if it's an IPv6 address, we should use send_ua, right? @kskmori: do you agree?
806 if is_infiniband; then
807 run_send_ib_arp
808 elif [ $FAMILY = "inet" ];then
809 if [ -x $SENDARP ]; then
810 run_send_arp
811 fi
812 else
813 if [ -x $SENDUA ]; then
814 run_send_ua
815 fi
816 fi
IPADDR/IPADDR2 somteimes fails to update arp cache on all nodes
Look at the screenshot https://dl.dropbox.com/u/14312824/ocf_ipaddr.png
P_MYSQL_MASTER_IP -> 10.8.12.200
P_MYSQL_SLAVE_IP -> 10.8.12.202
Both resources are supposed to be on the same machine, therefore MAC addresses should be the same after migration, but some time it fails to update an arp cache for quite a long time (~5 minutes or so).
Please take a look.
Thanks
Referring to the #52 I got the idea that shell script for finding interface by IP address could be better solution than parsing routing table in C.
For (most) Linux distributions 'ip' command is a part of the standard installation and:
# ip route get 10.0.0.220 10.0.0.220 dev eth1 src 10.0.0.106
gives better and more portable results.
For FreeBSD(and Solaris I guess):
$ route -n get 192.168.0.30
route to: 192.168.0.30
destination: 192.168.0.0
mask: 255.255.255.192
interface: xl0
flags:
recvpipe sendpipe ssthresh rtt,msec rttvar hopcount mtu expire
0 0 0 0 0 0 1500 -2151423
Also gives meaningful results(and currently used in 'findif').
Even parsing of '/proc/net/route' is easier in shell, so I see no good reason for quite complex and not flexible C binary for these purpose(except the speed, possibly).
If this idea gets support I can try to write such a replacement.
With regards,
Timur.
I've tried adding 2600:3c00::34:c007/116 with the IPv6addr resource agent, but I get an unknown error when trying to start it.
The error line reads:
ipv6test_start_0 (node=node-62, call=6, rc=1, status=complete): unknown error
I've emailed the Linux-HA mailing list and was suggested to report the bug. I would love any insight on this and really want to get this working.
Link to the mailing list thread: http://old.nabble.com/Heartbeat-IPv6addr-OCF-to35211501.html
In start method of nfsserver run sm-notify command.
If 2 IPs are set to nfs_ip, sm-notify is run 2 times as below.
for example:
nfs_ip=192.168.137.22,192.168.30.15
->RA do below commands
sm-notify -f -v 192.168.137.22 -P /var/lib/nfs/statd/sm.ha
sm-notify -f -v 192.168.30.15 -P /var/lib/nfs/statd/sm.ha
When the server is mounted and locked any files from 2 clients, the client1 using addrss 192.168.137.22, and client2 using addrss 192.168.30.15.
Then 2 lock status files (192.168.137.10 and 192.168.30.10) are created in /var/lib/nfs/statd/sm.ha .
for example:
client1's IP -> 192.168.30.10/24
client2's IP -> 192.168.137.10/24
->both clients lock any file..
# ls -l /var/lib/nfs/statd/sm*
Total 8
-rw------- 1 rpcuser rpcuser 89 Jun 18 17:42 2013 192.168.137.10
-rw------- 1 rpcuser rpcuser 88 Jun 18 17:43 2013 192.168.30.10
The server do a failover at this time, the sm-notify command does not exit and do retry forevery. So nfsserver start op timed out.
Because sm-notify can not send notify from 192.168.137.22 to 192.168.30.10.
I don't know method for solve this problem.
Do anyone have a good idea?
(It should the limit to only one IP can be specified to nfs_ip?)
Regards,
Agree with iSCSI RFC and open-iscsi implementation, you must check the "session state", and not the "connection state" . I suggest the following:
--- resource-agents/heartbeat/iscsi.orig 2012-11-06 08:55:14.850216774 -0300
+++ resource-agents /heartbeat/iscsi 2012-11-06 09:27:07.522217785 -0300
@@ -284,7 +284,7 @@
}
open_iscsi_status() {
local target="$1"
- local session_id conn_state outp
+ local session_id sess_state outp
local msg_logged
local recov
@@ -295,22 +295,22 @@
while :; do
outp=`$iscsiadm -m session -r $session_id -P 1` ||
return 2
- conn_state=`echo "$outp" | sed -n '/Connection State/s/.*: //p'`
+ sess_state=`echo "$outp" | sed -n '/iSCSI Session State/s/.*: //p'`
# some drivers don't return connection state, in that case
# we'll assume that we're still connected
- case "$conn_state" in
- "LOGGED IN")
+ case "$sess_state" in
+ "LOGGED_IN")
[ -n "$msg_logged" ] &&
- ocf_log info "connection state $conn_state. Session restored."
+ ocf_log info "session state $sess_state. Session restored."
return 0;;
"Unknown"|"") # this is also probably OK
[ -n "$msg_logged" ] &&
- ocf_log info "connection state $conn_state. Session restored."
+ ocf_log info "session state $sess_state. Session restored."
return 0;;
*) # failed
if [ "$__OCF_ACTION" != stop ] && ! ocf_is_probe && ocf_is_true $recov; then
if [ -z "$msg_logged" ]; then
- ocf_log warning "connection state $conn_state, waiting for recovery..."
+ ocf_log warning "connection state $sess_state, waiting for recovery..."
msg_logged=1
fi
sleep 1
If you uses $OCF_RESKEY_try_recovery=true , you will have a infinite loop waiting
for recovery. Ok it is the try_recovery intention. But..
Agree iSCSI rfc, the transition state is LOGGED_IN->FAILED->FREE->LOGGED_IN->.... ,being FREE, the session timeout state. "It is an event defined to occur when the last connection
state timeout expires...." (replacement_timeout in open-iscsi).
I think that the FREE state can be an end to infinite loop.
The label/alias on the loopback interface (lo:vip) is not restored if the Virtual IP is migrated. This is a small patch to the commit of January 14th 2013.
443,453c443
< case $NIC in
< *:*)
< IFLABEL=$NIC
< NIC=`echo $NIC | sed 's/:.*//'`
< ;;
< *)
< if [ -n "$IFLABEL" ]; then
< IFLABEL=${NIC}:${IFLABEL}
< fi
< ;;
< esac
---
> NIC=`echo $NIC | sed 's/:.*//'`
545,546c535,536
< cmd="$cmd label $label"
< msg="${msg} (with label $label)"
---
> cmd="$cmd label $iface:$label"
> msg="${msg} (with label $iface:$label)"
617c607
< add_interface $ifinfo
---
> add_interface $ifinfo $IFLABELLdirectord is using LWP for it's negotiate checks for the HTTP/HTTPS sites. Since LWP 6.0 by default it verifies the correspondence of the SSL certificate and the server hostname. In 99.9% of the cases this is the VIP hostname and RIP are identified by their internal hostnames or, most common - by their IP addresses.
That breaks hostname verification and hence - marks HTTPS backends as invalid and kicks them off the pool. This problem did hit me in the production when we've upgraded from Debian squeeze to Debian wheezy, which brought newer version of LWP.
http://search.cpan.org/~gaas/LWP-Protocol-https-6.04/lib/LWP/Protocol/https.pm
Luckily, the fix to the problem is easy:
--- ldirectord.orig 2013-12-03 11:59:11.114983525 +0100
+++ ldirectord 2013-12-03 11:59:34.703026282 +0100
@@ -2834,7 +2834,7 @@
&ld_debug(2, "check_http: url=\"$$r{url}\" "
. "virtualhost=\"$virtualhost\"");
- my $ua = new LWP::UserAgent();
+ my $ua = new LWP::UserAgent(ssl_opts => { verify_hostname => 0 });
my $h = undef;
if ($$v{service} eq "http_proxy") {
I haven't verified that with older version of LWP, but I believe it should just ignore unknown parameters to the constructor.
When passing an IP and NIC findif generates an invalid netmask:
[root@node-14]# OCF_RESKEY_ip=10.108.37.99 OCF_RESKEY_nic=eth0 /usr/lib64/heartbeat/findif
eth0 netmask 255.255.255.127 broadcast 10.108.37.227
This is a problem because the broadcast address it generates is incorrect.
However, it works when the NIC isn't specified:
[root@node-14]# OCF_RESKEY_ip=10.108.37.99 /usr/lib64/heartbeat/findif
eth0 netmask 255.255.255.0 broadcast 10.108.37.255
The issue seems to be that when a NIC is specified the default value of best_netmask is used in all calculations which is INT_MAX (https://github.com/ClusterLabs/resource-agents/blob/master/tools/findif.c#L571). best_netmask ends up as 0x7fffffff when it seems like it should be 0xffffffff. Changing the default to UINT_MAX makes findif behave correctly in this case.
IPaddr2 may failed to assign an IPv6 address with 'dadfailed' status on LVS configuration, which has the same IPv6 address on lo too.
I consider that this is a blocker for the release.
The steps to reproduce is below. Step 4a. or 4b. is a failed scenario. Step 4c. is a succeeded scenario. (thanks to @nozawatm for testing it)
Diagnosis:
It fails when send_ua is called if the VIP is still 'tentative' status (accomplishing the assignment in the kernel).
The cause seems that send_ua is sending ping for waiting the tentative flag is disappeared (i.e. assingment has finished) but the ping packet makes IPv6 DAD (duplicate address detection) protocol fail in the case of LVS configuration.
(L317-L322 in IPaddr2.c)
17d9f6a#L0R317
One strange thing is that this problem was not observed on RHEL5.7 (send_ua suceeds even if it was tentative status). We are unsure that if the kernel behavior does matter or not, but anyway we have to fix it somehow.
Solution:
I think we should take another way for waiting to finish to assign the IPv6 address. Probably we are going to remove the ping loop in send_ua and alternatively use ip command to check tentative flag in the RA.
I and @nozawatm are working on this issue right now.
I would appreceate if you have any comments.
steps to reproduce
# more /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)
# uname -a
Linux devnode1 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
# git log -1 --oneline
66f939e Merge pull request #176 from t-matsuo/delete-unnecessary-line
# export OCF_ROOT=/usr/lib/ocf
# export OCF_RESKEY_ip=2001:db8:100::101
# export OCF_RESKEY_nic=eth0
# export OCF_RESKEY_cidr_netmask=64
# export OCF_RESKEY_lvs_ipv6_addrlabel=true
# ip -o -6 addr add $OCF_RESKEY_ip/128 dev lo
# ip -o -6 addr show dev lo
1: lo inet6 2001:db8:100::101/128 scope global \ valid_lft forever preferred_lft forever
1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever
# ip -o -6 addr show dev eth0
2: eth0 inet6 2001:db8:100:0:5054:ff:fe11:0/64 scope global dynamic \ valid_lft 86128sec preferred_lft 14128sec
2: eth0 inet6 fe80::5054:ff:fe11:0/64 scope link \ valid_lft forever preferred_lft forever
# /usr/lib/ocf/resource.d/heartbeat/IPaddr2 start ; echo $?
INFO: Adding IPv6 address label prefix 2001:db8:100::101 label 99
INFO: Adding inet6 address 2001:db8:100::101/64 to device eth0
INFO: Bringing device eth0 up
INFO: /usr/lib64/heartbeat/send_ua -i 200 -c 5 2001:db8:100::101 64 eth0
0
# ip -o -6 addr show dev eth0
2: eth0 inet6 2001:db8:100::101/64 scope global tentative dadfailed \ valid_lft forever preferred_lft forever
(...)
# ip addr add 2001:db8:100::101/64 dev eth0 ; ip -o -6 addr show dev eth0; /usr/lib64/heartbeat/send_ua 2001:db8:100::101 64 eth0
2: eth0 inet6 2001:db8:100::101/64 scope global tentative \ valid_lft forever preferred_lft forever
(...)
# ip -o -6 addr show dev eth0
2: eth0 inet6 2001:db8:100::101/64 scope global tentative dadfailed \ valid_lft forever preferred_lft forever
(...)
# ip addr add 2001:db8:100::101/64 dev eth0 ; sleep 3; ip -o -6 addr show dev eth0; /usr/lib64/heartbeat/send_ua 2001:db8:100::101 64 eth0
2: eth0 inet6 2001:db8:100::101/64 scope global \ valid_lft forever preferred_lft forever
# ip -o -6 addr show dev eth0
2: eth0 inet6 2001:db8:100::101/64 scope global \ valid_lft forever preferred_lft forever
(...)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.