Exadata & OVM: Unable to get map of the virtual network interfaces

This week I had a "pleasure" of reimaging Exadata server to use OVM.
During this process we hit interesting problem – after the reimaging process there was no bridge at DOM0, corresponding to the client network – because of this, the OEDA sofware returned an error like this:

[ERROR    ] More than two single interfaces are found in DOM0 that match criteria. Single interfaces; IP; Netmask, Gateway: ; 10.20.10.70; 255.255.0.0; 10.20.0.1

The error was produced by this script: /opt/exadata_ovm/exadata.img.domu_maker

A short lookup through the script revealed the nature of the error – apparently we had to add appropriate the bridge to both DOM0s. So we did:

[root@exa2dbadm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
HOTPLUG=no
BRIDGE=vmeth1
NM_CONTROLLED=no
ETHTOOL_OPTS="-K eth1 lro off"
IPV6INIT=no
[root@exa2dbadm01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-vmeth1
#### DO NOT REMOVE THESE LINES ####
#### %GENERATED BY CELL% ####
DEVICE=vmeth1
TYPE=Bridge
BOOTPROTO=none
ONBOOT=yes
DELAY=0
ETHTOOL_OPTS="-K eth1 lro off"
IPADDR=10.20.10.60
NETMASK=255.255.0.0
GATEWAY=10.20.0.1
NETWORK=10.20.0.0
BROADCAST=10.20.255.255
[root@exa2dbadm01 ~]# ip a show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmeth1 state UP qlen 1000
    link/ether 00:21:28:a2:aa:ad brd ff:ff:ff:ff:ff:ff
    inet6 fe80::221:28ff:fea2:aaad/64 scope link
       valid_lft forever preferred_lft forever
[root@exa2dbadm01 ~]# ip a show vmeth1
13: vmeth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:21:28:a2:aa:ad brd ff:ff:ff:ff:ff:ff
    inet 10.20.10.60/16 brd 10.20.255.255 scope global vmeth1

After this operation we got success – other error 😉

[ERROR    ] Unable to get map of the virtual network interfaces

Fortunately the script /opt/exadata_ovm/exadata.img.domu_maker is well commented – and here is what we found:

# Get existing bridges from the cell config of "this" dom0 from /opt/oracle.cellos/cell.conf
# Check for mismatch between existing bridges in cell.conf and what "brctl show" has

Following that hint we have updated the XML file "/opt/oracle.cellos/cell.conf" and that was it – problem solved 🙂

The missing entry it the file looked like this:

<Interfaces>
    <Bridge>vmeth1</Bridge>
    <Gateway>10.20.0.1</Gateway>
    <Hostname>exa2dbadm01.arrowecs.hub</Hostname>
    <IP_address>10.20.10.60</IP_address>
    <IP_enabled>yes</IP_enabled>
    <IP_ssh_listen>enabled</IP_ssh_listen>
    <Inet_protocol>IPv4</Inet_protocol>
    <Name>eth1</Name>
    <Net_type>Management</Net_type>
    <Netmask>255.255.0.0</Netmask>
    <State>1</State>
    <Status>UP</Status>
    <Vlan_id>0</Vlan_id>
  </Interfaces>

Well, that’s it. Hope it saves someone a few hours of searching 😉
Of course you have to remember to edit appropriate files on all DOM0 machines in your Exadata configuration and to be very careful while editing the config files.

Of course the better solution would be to double check your OEDA configuration files before reimaging… But it’s better to alter some configs, than start everything from scratch.