Skip to content

4.11.2 HA VRs with Second Public NIC get different routes between Startup and Failover #3353

Closed
Closed
4.11.2 HA VRs with Second Public NIC get different routes between Startup and Failover#3353
Milestone
4.11.3.0

Description

@richardlawley
richardlawley
opened on May 24, 2019
ISSUE TYPE
  • Bug Report
COMPONENT NAME
VR
CLOUDSTACK VERSION
4.11.2
CONFIGURATION

Advanced Networking, HA VRs

OS / ENVIRONMENT

XenServer 6.5

SUMMARY

When a network has public IPs from multiple ranges and a VM has a Static NAT rule from the second IP range (different range from the Source NAT IP), the outbound IP changes when a VR fails over. This is because default routes for each interface are not restored when a failover occurs, but are added correctly when the VR is built.

STEPS TO REPRODUCE
  1. Create zone with 2 or more public IP ranges
  2. Create an advanced network, allocate IPs until you get IPs from both networks (or use the database to assign a specific IP)
  3. Create a VM and assign a Static NAT IP to the IP from the second range
  4. Reboot both VRs at the same time to get a "clean" situation
  5. Log into Master VR
  6. Run ip route table list all > /root/route.1
  7. Run systemctl restart keepalived to force a failover without rebuilding VR
  8. Log into the new VR and fail back using systemctl restart keepalived
  9. Run ip route table list all > /root/route.2
  10. Compare routing tables with diff /root/route.1 /root/route.2
EXPECTED RESULTS
Routing table should be identical between first build and failover
ACTUAL RESULTS

Default routes for each interface were missing, resulting in the outgoing traffic from the VM with static NAT originating from the Source NAT IP

# diff /root/route.1 /root/route.2
2d1
< default via X.X.X.X dev eth2 table Table_eth2 proto static 
6d4
< default via X.X.X.X dev eth3 table Table_eth3 proto static

We are able to fix this manually by re-adding the route:

ip route add default via X.X.X.X table Table_eth3 proto static

NB - the eth2 route doesn't seem to be necessary, but adding the eth3 route fixes the originating IPs.

Activity

  1. ustcweizhou commented on May 24, 2019

    @ustcweizhou
    ustcweizhou
    on May 24, 2019
    Contributor

    This is one of the issues that happen on network/vpc with public IPs in mutiple subnets.
    We have seen many issues in this scenario from 4.7. Some have already been fixed. Some issue still exist

  2. richardlawley commented on May 24, 2019

    @richardlawley
    richardlawley
    on May 24, 2019
    ContributorAuthor

    These routes also get restored when a new static NAT association is made.

  3. yadvr
    added this to the 4.11.3.0 milestone on May 27, 2019

  4. richardlawley commented on May 28, 2019

    @richardlawley
    richardlawley
    on May 28, 2019
    ContributorAuthor

    I've found some additional cases where routes aren't added properly, leading to two additional problems with static NAT VMs with multiple public subnets:

    1. VMs with static NAT IPs on range corresponding to eth3 cannot ping other IPs from another network on the same range
    2. VMs cannot always ping each other using their external IP address. This one I know has been fixed before, so this is either a regression or a missed test case.

    I traced the cause of these to be here. The intention of the code is that it runs when the first IP address is added, however in practice this means it only runs when there is one IP assigned to the interface - if you start the VR with multiple IPs present then the route is added correctly.

    The route table of a "broken" router looks like this:

    # ip route show table Table_eth3
default via X.X.X.X dev eth3 proto static 
throw X.X.X.X/Y proto static

    The fixed version is:

    # ip route show table Table_eth3
default via X.X.X.X dev eth3 proto static 
throw X.X.X.X/Y proto static 
throw 10.1.1.0/24 proto static 
throw 169.254.0.0/16 proto static

    My current working code is by changing the affected block to:

                    if self.get_type() in ["public"]:
                    route.set_route("table %s throw %s proto static" % (tableName, self.config.address().dbag['eth0'][0]['network']))
                    route.set_route("table %s throw %s proto static" % (tableName, self.config.address().dbag['eth1'][0]['network']))

    Once we're done rolling this out, I'm happy to put together a PR, but I'm very aware that the VR code contains quite a few different scenarios, and I'm not yet familiar with how to write tests for this functionality.

  5. yadvr
    added a commit that references this issue on May 31, 2019

    systemvm: fix VR issues with Multiple Public Subnets (#3361)

    2f268fb
  6. yadvr
    closed this as completedin 2f268fbon May 31, 2019
  7. shwstppr
    added a commit that references this issue on May 31, 2019

    systemvm: fix VR issues with Multiple Public Subnets (apache#3361)

    c5708ab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions