Ready for real-world

[davidhealey]

Hello,

Is this app ready for real world usage?

[richpeck]

Hi David, unfortunately not. There is an app which I use myself called 100pcvatfreebridge that works extremely well: https://www.comsci.co.uk/100PcVatFreeBridge.html

[davidhealey]

Thanks, is it free software though?

[richpeck]

Yes

[davidhealey]

Yes

Oh excellent! Do you have a link for the source code? I can't see it on their site.

[richpeck]

They don't publish the source code. I've not seen an open source HMRC VAT integration so decided to do my own. Unfortunately, I got pulled into other projects and have yet to finish it.

[davidhealey]

They don't publish the source code.

That's a shame, when I said free software I meant freedom respecting, not freeware/gratis. I've seen one other project that looks like it might do the job, it integrates via GnuCash and each user has to put in an API key from HMRC which is a pain. I'm not sure if it's being actively updated though.

[richpeck]

The correct term is 'open source'. Plenty of free software that's proprietary! In terms of what you're looking for - I can explain how one of these systems works if you'd like, its very simple. Perhaps that would be of assistance? I do wish to finish this but, as mentioned, got caught up with other things.

[davidhealey]

The correct term is 'open source'.

Ah that term came about long after "free software" https://www.gnu.org/philosophy/free-sw.html https://www.fsf.org/about/what-is-free-software

Perhaps that would be of assistance?

Yes this would be very helpful, thank you!

[richpeck]

Apologies, didn't realize there was a distinction :)

I'll get something to eat then will reply with an overview afterwards!

[richpeck]

I'm back!

To explain: -

HMRC have been pushing toward their "MTD" (Making Tax Digital) paradigm for a while, and have (I believe) only made the transition fully with VAT. There was talk of doing it for other services, but there was pushback because some people don't have access to the Internet etc.

VAT-MTD is the service HMRC have implemented to make this transition work for VAT.

Anyone who is VAT registered in the UK have undoubtedly heard of it as many a brown envelop ended up doormats explaining the matter over the past few years -- from April 2019, the only way to submit VAT returns has been via the MTD platform.

Ultimately, it's a REST API endpoint with accompanying authentication, allowing application developers to interface with HMRC's service to provide access to clients. The problem with this is that it renders spreadsheet / manual submission void, as you have to format a request specifically in order to make it work. I'll explain this in a second.

You can read up on the API here: https://developer.service.hmrc.gov.uk/api-documentation/docs/api/service/vat-api/1.0

You can read up on VAT-MTD's technical scope here: https://developer.service.hmrc.gov.uk/guides/vat-mtd-end-to-end-service-guide/#overview

The importance of it being an API is that it opens up the platform to developers. This has lead to the proliferation of "done for you" systems such as Xero, Sage etc.

As I don't like the idea of a software application accessing my bank accounts, I prefer to keep my records in a spreadsheet and submit the information manually. When the MTD changes came in, I, therefore, looked around for an applicable solution. Turns out the type of software that acts as a "bridge" between HMRC's new API stack and third party software (such as a spreadsheet) is called bridging software.

I originally discovered "VitalTax", which was a simple Excel plugin that interfaced with the VAT-MTD endpoint. Unfortunately, whilst it started out gratis, it's now a fully paid solution.

From here, I turned to HMRC to see if I could find an alternative. Turns out they have a comprehensive directory of applications which you can use - I searched through the "free" ones to come up with the 100pcVatFreeBridge package I referred to before.

After using the 100pc software for some time, I decided that, having had experience with this sort of thing, to attempt to replicate its functionality. The functionality is not hard to replicate - I got a little carried away with some of the things I was doing. I'll go through the technical details in another comment!

[richpeck]

API Connectivity

I don't know what your experience is with API's.

Having had a cursory look at your profile, it seems my initial assumption that you wanted free VAT software was incorrect - seems you're quite experienced (this is why I was quite terse with my responses before -- a lot of people don't understand things) 🙂

If you're not familiar with API's, the basis is that a networked computer (server) is running an application which accepts inbound HTTP requests to an endpoint. These requests can be populated with a payload of data which the hosted application will process and provide a response to. The typical setup of these API services is in what's known as the "REST" format, which I'm not fully familiar with the definition for in honesty.

The bottom line with REST is that you're able to send specifically-formatted HTTP requests to web based endpoints, with the appropriate authentication headers, in order to input data into web based services. In the case of HMRC, it means we are able to submit VAT returns across the Internet.

Despite the scary sounding name and a lot of documentation, the VAT-MTD API is very simple: -

• The test (sandbox) endpoint is: https://test-api.service.hmrc.gov.uk
• The production (live) endpoint is : https://api.service.hmrc.gov.uk

As per HMRC's website, the following API "actions" are accessible through the above URL's: -

• retrieve VAT obligations
• submit and view a VAT return
• retrieve a VAT liability
• retrieve a VAT payment

You can, essentially, use the above to send pre-formatted packets of data and receive different responses.

I'll detail how to do this in the next comment.

[richpeck]

Technical Basis

The basis of all software is to take someone's inputs and use them to manipulate data in a way conducive to beneficial responses.

This means that each time you create software, you're essentially providing the user with a series of "forms" through which they can input different things, and have the software "compute" a response based on data that it has pre-loaded.

In the case of the VAT MTD thing - all you're really doing is providing a means for a user to load a source of data (spreadsheet) and giving them the means to allocate that data to the different "boxes" apparent on the typical VAT submission form.

As per the API discussion above, this means that you're aiming to get to a point where you're able to send a payload of data to the VAT-MTD "submission" endpoint, which fills in the various VAT boxes for the user.

The payload should look like this: -

{
  "periodKey": "A001",
  "vatDueSales": 105.50,
  "vatDueAcquisitions": -100.45,
  "totalVatDue": 5.05,
  "vatReclaimedCurrPeriod": 105.15,
  "netVatDue": 100.10,
  "totalValueSalesExVAT": 300,
  "totalValuePurchasesExVAT": 300,
  "totalValueGoodsSuppliedExVAT": 3000,
  "totalAcquisitionsExVAT": 3000,
  "finalised": true
}

My code was aimed at following the 100PCVatBridge pattern of having a simple interface to upload a spreadsheet and allocate its data to a specific VAT return, providing a means to submit it to HMRC.

I'd have to look through my code again to provide any specific details on how I was looking to do this - however, I will explain several key points that are extremely pertinent: -

1. oAuth

The way in which the API authenticates users is with the Open Auth protocol. This is how web service providers such as Facebook etc give you the means to use their data/services without handing your password over to third parties.

HMRC provide an oAuth service, whereby your application is able to accept user requests to authenticate and then have the means to interface with their account in HMRC. You can read up on the specifics of this here.

In order to connect to a user's account, they must authorize your solution to interface with HMRC on their behalf. This is tricky because, when I was doing this code, there was no pre-baked solution to handle it. I managed to create my own, which I can obviously discuss if you want.

2. Ruby

My solution (this repository) is based on my own blend of Ruby. This means that it's a web based solution and not a native one.

I use Sinatra for the web framework and ActiveRecord to store/manage user data. I have a number of other gems for oAuth compatibility etc.

I mention this because, although the HMRC API is standardized, there isn't a huge amount of coverage from third party libraries yet. This means that a lot of the solutions will typically end up homebrew, which could be a problem if you prefer working with different stacks.

3. Data Integrity

The most important concern for me was the integrity of data being submitted to HMRC.

Obviously, you don't want to make an app that is going to cause problems, so I wanted to be sure that what I was doing was going to work 100% each time. Fortunately, the VAT side of things is not hugely problematic in this regard, as there are a set number of boxes which are clearly defined in scope.

The key point, I suppose, is that whilst it may seem interesting or tempting to create API integrations with HMRC, one of the reasons this sort of product costs money is because the companies behind them have, presumably, put the time and money into validating their ability to get the calculations correct etc.

--

Not much more I can think of adding.

If you'd like me to make some specific explanations about user flow, I can do that no problem.

[richpeck]

By the way, happy to do a call or you can email me directly [email protected] if you'd like to talk about things more specifically.

[davidhealey]

Thanks that really helps understand it, I'm actually getting tempted now to have a go at writing my own app to do this. I'm not familiar with Ruby but I think poking around in your code could still give me a kickstart. I'd make an app in Electron as I'm most familiar with that way of working and it almost certainly has packages for handling oAuth.

[richpeck]

Electron is just a frontend package manager for JS if I'm not mistaken?

You may be able to use React to do it -- you basically need the following: -

1. User authentication/management (not a problem if you're shipping a native app)
2. oAuth connectivity to allow your app to interface with a user's HMRC account
3. The means to upload a spreadsheet and assign cells / named ranges to the VAT return boxes
4. Submit the VAT return to HMRC with the API endpoints listed above

[davidhealey]

Electron is just a frontend package manager for JS if I'm not mistaken?

I think you're thinking of nodeJS, which Electron uses. Electron is a framework for building desktop apps using web tech. I've built a few apps with it in the past.

I haven't used React but I will have a look into it and see how it compares to Electron. I'd rather use something I'm already familiar with just to get it done quickly but if React is better in the long run I might go with it.

[richpeck]

RE Electron. Yes - that's what I meant. It's used by a LOT of great apps. I always thought it took existing JS code and put it in a browser type format (Electron just runs on Chromium when it's compiled from what I've read).

If you're familiar with Electron, use that!