NFR: Arrangements with large numbers (500+) of accounts

[perlboy]

Description

During the MI an operational concern was raised, particularly due to the implementation of Energy C&I accounts in the energy majors with respect to the real (or at least Rules mandated) possibility of situations where possibly thousands of accounts would be included in a single arrangement. I can confirm that there are legitimate account situations which are eligible under the Rules but for which there are >1500 accounts.

There are various challenges with this:

1. Even if the Primary Data Holder can deliver 1000 energy accounts at once a call to the Secondary Data Holder could result in potentially tens of thousands of NMIs requested to Get Service Points. This request is very cheap for the Data Recipient (it is literally one call without a filter) yet enormously expensive to the Secondary Data Holder
2. Even if the Primary Data Holder can deliver 1000 energy accounts at once a call to the Secondary Data Holder could result in potentially tens of thousands of NMIs requested to Get Usage by Service Point. This request is cheap for the Data Recipient (it is literally one call without a filter) yet enormously expensive to both the Primary Data Holder, who must piece together interspersed FRMP responsibility periods, and the Secondary Data Holder who must conjur up, sort and deliver usage data
3. There is a threshold at which point it will approach theoretical impossibility to deliver 1000 records (the page-size limit) within the NFR of 1 second, this gets much worse with progressive plan changes (i.e. long lived but regularly renegotiated plan changes across the fleet of accounts)
4. CX design in these circumstances is completely different, there's been some alteration (with a MAY added) but the Standards here are quite ambiguous

Resolving this without Rules changes is challenging but there are very real operational issues occurring right now for active Data Holders as a result. Based on my exposure to the next Tranche it is apparent this problem actually gets worse as they seem to be skewed to have a proportionally larger count of C&I customers with large account counts.

Note: I've created this pretty quickly for consideration in this MI and therefore have not had a chance to pass this through Biza.io's Data Standards Committee.

Note 2: I put this under NFR but, having written it, it seems broader than that, sorry in advance.

Area Affected

• Any endpoint involving either directly or indirectly Energy Accounts. At first glance this essentially all energy endpoints except those relating to customers.

Change Proposed

Without Rules changes this is a hard problem, some [embryonic] thoughts in order of priority/capability/ease. I've used 100 as an arbitrary number here, up for discussion, as I think the lowest constraint is probably the SDH side and this is a guess.

Limit maximum per arrangement

Provide immediate Standards relief by introducing a clause similar to "A Data Holder within the Energy sector MAY limit the maximum number of Energy Accounts in a single arrangement to 100 accounts".

Our read of this limit is that:

1. Retail Consumer impact would be very very (possibly zero) low
2. C&I Consumers beyond these thresholds would either
   • not be users of the CDR or;
   • would be able to establish multiple arrangements, 100 accounts at a time, while tedious this would still achieve the Rules objective and could be revisited

Limit maximum result set per endpoint

1. Introduce maximum page-size for List Energy Accounts, Get Service Points, GET DER for Service Points, Get Bulk Balances for Energy, of 100.
2. Introduce enhanced error of urn:au-cds:error:cds-energy:Resource/TooManyAccounts
3. Introduce (2) error into Get Bulk DER, Get Bulk Usage
4. Introduce a maximum number (100?) of servicePointIds for all "for Specific Service Points" endpoints

Note 1: It is possible there may need to be some revision of NFRs for requests related to arrangements with very large accounts

Note 2: I came up with the endpoint list on the fly, definitely possible there might be some changes here

Introduce specific calls for lots of accounts

This one is a lot more work and I'm not sure it would work so great but the idea would be to introduce an asynchronous version of each endpoint specifically for bulk things of very large counts. I suspect it may be better to concentrate on the first two groups of suggestions rather than shoehorn/rush towards this state.

[perlboy]

Adding an addition here that this use case is now live with incidents being raised regarding large numbers of accounts. In the absence of any relief we will now be proceeding with enabling this use case via other means and will consequently start passing requests to secondary data holders of 50+ NMIs at a time.