Implement Audit Logs

[mind-ar]

Feature Request

Description

Audit Logs allows the administrators to have a detailed log related to authentication (login, logouts, resource access, etc)
Many companies, such as banks, require products to provide audit trails/logs.
Also, an audit log will be useful when the management portal is developed, as pointed at #595 (comment)

First version can be easily developed, using the current logger, but on different file:

audit:
  level: info
  keep_stdout: false
  local:
    format: text
    file_path: ""

A second version could support db backend

audit:
  level: info
  keep_stdout: false
  mysql: {}
  postgres: {}
  ...: {}

Future version (too future) could allow to push audit logs to common logging systems, e.g. Common Log File System (CLFS) or Common Event Format (CEF)

Things to define:

• Fields to be logged
• What must be logged for each level

What do you think?

Use Case

[seandlg]

Currently, we set the log level to debug to achieve this. However, this logs access against all endpoints, also non-sensitive ones. Being able to audit logs against /api-endpoints would be awesome indeed!

[james-d-elliott]

Can you explain what you mean by /api endpoints?

Also if we added audit logging it would only be to a log facility like file or gelf etc, not SQL. It'd likely take this form:

log:
  audit:
    enabled: false
    directory: "/audit"
    ## Categories to log. When using the directory logger each of these has its own file.
    ## Most of these can probably be relatively easily implemented, with openid_connect being the least likely to be implemented initially.
    categories:
    - authentication  # logs all user authentication attempts 
    - authorization  # logs all authorization requests to /api/verify
    - openid_connect  # logs all openid connect related things
    - request  # logs requests, method, path, remote IP, response status code, maybe response headers.

[seandlg]

Oh, sure! I meant that only certain endpoints would have to be audited, e.g. if the rules section were to be (simplified example):

  rules:
    - domain: "web.app.com"
      subject:
        - "group:admins"
      policy: one_factor
      audit: false # SPA, cannot do anything by itself

    # This is required for a preflight OPTIONS request, that sets the CORS policy
    - domain: "api.app.com"
      policy: bypass
      methods:
        - OPTIONS
      audit: false # not of interest

    - domain: "api.app.com"
      subject:
        - "group:admins"
      policy: one_factor
      audit: true # audit any requests that match here

I guess my thinking here is that I could extend the security-sensitive, existing access control logic to my auditing needs. Does that make sense?

[mind-ar]

log:
  audit:
    enabled: false
    directory: "/audit"
    ## Categories to log. When using the directory logger each of these has its own file.
    ## Most of these can probably be relatively easily implemented, with openid_connect being the least likely to be implemented initially.
    categories:
    - authentication  # logs all user authentication attempts 
    - authorization  # logs all authorization requests to /api/verify
    - openid_connect  # logs all openid connect related things
    - request  # logs requests, method, path, remote IP, response status code, maybe response headers.
  
... 

  rules:
    - domain: "web.app.com"
      subject:
        - "group:admins"
      policy: one_factor
      audit: false # SPA, cannot do anything by itself

    - domain: "api.app.com"
      policy: bypass
      methods:
        - OPTIONS
      audit: false # not of interest

    - domain: "api.app.com"
      subject:
        - "group:admins"
      policy: one_factor
      audit: true # audit any requests that match here

[james-d-elliott]

I'm not convinced making it per policy is a good idea. Getting too granular is going to make it easier to configure it in a way which misses important logs. I feel like this is probably a good realm for logging engines which are purpose built for this kind of filtering. But I can get the feedback of the other team members.

Audit logs are traditionally meant to be used in postmortem when a security breach has occurred or to detect security breaches in advance of the postmortem. While it's on the user, if they use the below information they will be in for a nasty surprise at a point where it's too late to fix it if the API is ever breached by a non-admin account:

   - domain: "api.app.com"
     subject:
       - "group:admins"
     policy: one_factor
     networks: 192.168.1.0/24
     audit: true # audit any requests that match here
   - domain: "api.app.com"
     subject:
       - "group:admins"
     policy: two_factor
     audit: true # audit any requests that match here
   - domain: "api.app.com"
     policy: one_factor
     audit: false

[seandlg]

I see your point and I must admit that your reasoning makes much sense! If the audit engine allows for powerful filtering, this already addresses my above use-case.

Just brainstorming here: While auditing in case of a postmortem is certainly the most crucial, there's also a use-case of sanity auditing, e.g. detecting scripting against security sensitive endpoints. One advantage of authelia, as opposed to e.g. traefik is that it already knows which endpoints are sensitive, plus it knows the identity of users. Of course, this information is certainly extractable otherwise.

[james-d-elliott]

If the audit engine allows for powerful filtering, this already addresses my above use-case.

Not entirely sure what you meant by this. If we were to implement audit logging it would utilize structured logging. Fields which could theoretically be filtered on with something like graylog or other log storage soltions, or various simple programmatic methods. As long as we ensure the initial audit logs are comprehensive enough and wont theoretically need to change in the future then each file will have a static format, or could be directed to a specific collector in something like graylog.

My thoughts are that if we did audit logging it'd be part of the log revamp (looking at zap or zerolog), and if we did allow filtering on authelia's side it'd be category based (i.e. what kind of log message it is, not the subject of the log), and if we added subject based logging it would probably come later and we'd make it completely decoupled from the policy engine so we could theoretically offload this work to a subroutine rather than doing it as a synchronous action.