aws.RouteTable routes are not notified of changes by aws.Route

[rdanno]

What happened?

Route resources are linked dependencies of RouteTables - but 'inline' Routes don't properly notify the RouteTable when they make a route entry change - and since routes=[] is a property of RouteTable - when you refresh state then the RouteTable sees drift.

I've added ignoreChanges to RouteTable::routes but the refresh still picks up the changes

Expected Behavior

Route should notify RouteTable when it makes changes to its routes so the refresh operation does not need to be run

Steps to reproduce

deploy a routetable with no routes, then use Route to add a route entry then run a refresh

Output of pulumi about

`CLI
Version 3.70.0
Go Version go1.20.5
Go Compiler gc

Plugins
NAME VERSION
aws 5.41.0
python unknown

Host
OS darwin
Version 11.7
Arch x86_64`

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[danielrbradley]

Hi @rdanno thanks for writing this up.

If I've understood you correctly ... you're using both inline routes (within the RouteTable resource) and standalone route resources. Then when you do a refresh you see discrepencies because the standalone routes are shown as being removed on the RouteTable. Is that correct.

I believe this is a similar issue to #1790

I think the initial resolution here to to avoid creating routes via the RouteTable property and only use the standalone Route resources. The root cause is likely a limitation of the underlying upstream implementation of this provider.

[danielrbradley]

As noted on the terraform resource, this is a known issue in the upstream provider:

Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

[rdanno]

Creating the RouteTable with no routes. Adding routes with Route later.

[danielrbradley]

Ok, that sounds like it's a different issue to the one noted then. We'll need to investigate the upstream implementation to identify if we can address this there.

[t0yv0]

I have a repro here that I hope demonstrates the issue, unfortunately it is still present in recent Pulumi.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const exampleAwsVpc = new aws.ec2.Vpc("test", {cidrBlock: "10.1.0.0/16"});

const exampleRouteTable = new aws.ec2.RouteTable("example", {
    vpcId: exampleAwsVpc.id,
    // routes: [],
    tags: {
        Name: "example",
    },
});

const egress = new aws.ec2.EgressOnlyInternetGateway("egress", {vpcId: exampleAwsVpc.id});

const exampleRoute = new aws.ec2.Route("route1", {
    routeTableId: exampleRouteTable.id,
    destinationIpv6CidrBlock: "::/0",
    egressOnlyGatewayId: egress.id,
});


export const exampleRouteId = exampleRoute.id;

After the first pulumi up, RouteTable receives inputs nil but outputs as routes: [].

However refreshing this stack shows a warning that RouteTable wants to register the route:

Refresh shows:

Previewing refresh (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2561/dev/previews/d1c94b52-d216-412f-84c4-774e5f61bc72

     Type                                  Name          Plan       Info
     pulumi:pulumi:Stack                   aws-2561-dev
     ├─ aws:ec2:EgressOnlyInternetGateway  egress
     ├─ aws:ec2:Vpc                        test
 ~   ├─ aws:ec2:RouteTable                 example       update     [diff: ~routes]
     └─ aws:ec2:Route                      route1

Resources:
    ~ 1 to update
    4 unchanged

Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
 details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::aws-2561::pulumi:pulumi:Stack::aws-2561-dev]
    ~ aws:ec2/routeTable:RouteTable: (update)
        [id=rtb-01ff6ea4cd740d0c1]
        [urn=urn:pulumi:dev::aws-2561::aws:ec2/routeTable:RouteTable::example]
        [provider=urn:pulumi:dev::aws-2561::pulumi:providers:aws::default_6_32_0::c96d3f21-e412-40ac-bfd8-c9677165d4fd]
        --outputs:--
      ~ routes         : [
          + [0]: {
                  + carrierGatewayId       : ""
                  + cidrBlock              : ""
                  + coreNetworkArn         : ""
                  + destinationPrefixListId: ""
                  + egressOnlyGatewayId    : "eigw-063e404ff733110ab"
                  + gatewayId              : ""
                  + ipv6CidrBlock          : "::/0"
                  + localGatewayId         : ""
                  + natGatewayId           : ""
                  + networkInterfaceId     : ""
                  + transitGatewayId       : ""
                  + vpcEndpointId          : ""
                  + vpcPeeringConnectionId : ""
                }
        ]

Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
  [Use arrows to move, type to filter]
  yes
> no
  details

If the refresh is accepted (our pulumi up --refresh is run), then the system gets into a steady state with clean pulumi preview and refresh, because now the route is duplicated into the outputs of RouteTable as well as into its own resource.

CLI          
Version      3.111.1
Go Version   go1.22.1
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.32.0
awsx    2.9.0
docker  4.5.3
docker  3.6.1
nodejs  unknown

Host     
OS       darwin
Version  14.4.1
Arch     x86_64

This project is written in nodejs: executable='/Users/t0yv0/bin/node' version='v18.18.2'

Current Stack: anton-pulumi-corp/aws-2561/dev

TYPE                                                         URN
pulumi:pulumi:Stack                                          urn:pulumi:dev::aws-2561::pulumi:pulumi:Stack::aws-2561-dev
pulumi:providers:aws                                         urn:pulumi:dev::aws-2561::pulumi:providers:aws::default_6_32_0
aws:ec2/vpc:Vpc                                              urn:pulumi:dev::aws-2561::aws:ec2/vpc:Vpc::test
aws:ec2/egressOnlyInternetGateway:EgressOnlyInternetGateway  urn:pulumi:dev::aws-2561::aws:ec2/egressOnlyInternetGateway:EgressOnlyInternetGateway::egress
aws:ec2/routeTable:RouteTable                                urn:pulumi:dev::aws-2561::aws:ec2/routeTable:RouteTable::example
aws:ec2/route:Route                                          urn:pulumi:dev::aws-2561::aws:ec2/route:Route::route1


Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/awsx    2.9.0
@pulumi/pulumi  3.113.3
@types/node     18.19.31
typescript      5.4.5
@pulumi/aws     6.32.0

Pulumi locates its logs in /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/com.apple.shortcuts.mac-helper// by default

[t0yv0]

This seems to be very similar to #2246 just for a different pair of coupled resources that are not recommended to be used together, there seems to be a pattern going on here. The issue likely reproduces in TF but is less of a problem there because TF does not warn on refresh of this sort and it refreshes by default when apply is called, so after two terraform apply calls the system gets into a steady state.

[t0yv0]

Similarly to #2246 ignoreChanges is not able to affect refresh, but there is a new feature being developed for an ignoreRefreshChanges flag that is the recommended workaround for 2246 that should work here as well: pulumi/pulumi#16015

[t0yv0]

Adding pulumi/pulumi-policy-aws#110 to make the NOTE a visible warning for users of AWSGuard.