m365-endpoint-api-digester

An Microsoft 365 endpoint API utility

For information see: https://docs.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide

The Office 365 IP Address and URL web service helps you better identify and differentiate Office 365 network traffic, making it easier for you to evaluate, configure, and stay up to date with changes

An extension of offically provided start scripts at: https://docs.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-ip-web-service?view=o365-worldwide#example-python-script

---

What does this do?

Consumes the M365 endpoint API, which supplies a list of IPv4/6 addresses and DNS addresses which are used by M365 related products (Teams, OneDrive, Sharepoint & O365 Applications). With this list is is possible to create rules for network infrastructure devices such as firewalls, IDS, proxies etc, to permit, monitor, throttle or control access.

This python command line application can currently digest the M365 API, and product output files in the following formats:

• Generic CSV - For further processing
• PuppetSquid - For use in a puppet controlled environment, likely as part of your CI/CD workflow. See m365digester/Outputs/PuppetSquid.py
• Squid3 via a template - For use directly in your Squid configuration. See m365digester/Outputs/SquidConfig.py and examples/squidconfig.template

Motivation

This script began as part of a requirement to produce a Squid3 based proxy running on Puppet manage infrastructure, with the sole purpose of proxying MS Teams and MS OneDrive connections to M365 from networks that were not permitted to be on a routable network, nor were they permitted to have generic proxied internet access in the interests of security. The first iteration of this script produced YAML only, and had little configurability. Squid3 uses Splay Trees, which does not necessarily work well when trying to translate the rules provided by the M365 Endpoint API (see here), so a primitive 'collapser' is included, to reduce the rule sets produced to their minimum, and thus keep Squid happy.

---

Configuration

Uses the standard 'logging' module from Python ~2+ for log levels:

• CRITICAL
• ERROR
• WARNING
• INFO
• DEBUG

Options

Parameters can be set in several ways

• Environment variables
• Command line
• As part of a python Dict() (when used as a module)

Parameters

Short	Long	ENVVAR	Type	Default	Information
-v	--version	N/A	Semantic version		Return version information
	--log-level-file	LOG_LEVEL_FILE	Log Level	DEBUG	Set the log level for file output
	--log-level-console	LOG_LEVEL_CONSOLE	Log Level	INFO	Set the log level for console output
-l	--log-file-output	LOG_FILE_PATH	File path and name	None	Log file target
-k	--keep-sqlitedb	SQLITEDB_KEEP	Switch (Bool)	False	If set, any SQLite databases used on disk will not be deleted at the termination of this application
-j	--sqlitedb-file-path	SQLITEDB_FILE_PATH	File path and name	./{APP_NAME}.db	If set, all SQLite operations will be performed on this file on disk, not in memory
-W	--disable-wildcards	WILDCARDS_DISABLED	Bool	False	Prevent the replacement of wildcards eg: '*.domain.com' with single prefix dots '.'
-w	--wildcard-pattern	WILDCARD_PATTERN	String (regex)	'^(*).'	Regex to use for the detection and replacement of wildcards
-C	--collapse-acls-disable	ACL_COLLAPSE_DISABLED	Switch (Bool)	True	If disabled, ACLs will not be reduced to a smaller set based on inner/outer subdomain tree positioning
-z	--categories-include	CATEGORIES_INCLUDE	Domain List (space seperator)	Allow Default	List of categories from API to process
-q	--disable-domains	DOMAINS_DISABLED	Switch (Bool)	Disable processing of domain names from API	False
-n	--disable-ipv4	IPV4_DISABLED	Switch (Bool)	Disable processing of IPv4 addresses from API	False
-m	--disable-ipv6	IPV6_DISABLED	Switch (Bool)	Disable processing of IPv6 addresses from API	False
-i	--client-request-id	M365_REQUEST_ID	String (GUID)	Automatically generated from host NIC MAC	Request ID to use with M365 API
-s	--service-instance	M365_SERVICE_INSTANCE	String (Choice)	Worldwide	Specify M365 service instance API type
-e	--extra-known-domains	EXTRA_KNOWN_DOMAINS	Domain list (space separated)	Not specified	Use for your tenancy domain names or other extras including overrides, do not use quotations, wildcards permitted, ie: '-e mycompany-files.sharepoint.net *.live.com autodiscover.mycompany.mail.onmicrosoft.com'
-E	--extra-known-ips	EXTRA_KNOWN_IPS	IP address list (space separated)	Not specified	Use for other extras IP addresses including overrides, do not use quotations, wildcards not permitted, ie: '-E 192.168.1.0/24'
-x	--exclude-addresses	EXCLUDE_ADDRESSES	Domain/Address list (space separated)	Not specified	Use to exclude entries from consideration when processing or generating files, ie: '-x autodiscover.*.onmicrosoft.com'
-u	--output-path	OUTPUT_PATH	File path without name	'./'	Path on disk to place output file. Mutually exclusive with -o
-p	--output-prefix	OUTPUT_PREFIX	File name only without extension	'{APP_NAME}'	Filename without extension for output file
-o	--output-file	OUTPUT_FILE	File name and path	Unset	Full path and filename for output file. Mutually exclusive with -u and -p
-t	--output-type	OUTPUT_TYPE	String (Choice)	yaml	Output file type, from: [ GENERALCSV PUPPETSQUID SQUIDCONFIG ]
	--output-template	OUTPUT_TEMPLATE	File name and path	Unset	Input template file for output file types supporting it (ie: SQUIDCONFIG)
	--linesep	LINESEP	String (Choice)	Python os.linesep	Specify line separator (new line), CRLF on Windows, LF on nix* ")

---

Use as a Docker container

NOTE: This container is not yet published, but the included Dockerfile has been tested locally and does work.

docker run -v /host/output/target:/output:rw dougbarry/m365digester:latest -l /output/m365digester.log -k -j /output/m365digester.db -z Allow Default Optimize -m -e testcompany-files.sharepoint.com testcompany-cloud.microsoft.com *.live.com -t puppetsquid -o /output/puppet-squid-snippet.yml

Use via docker-compose

services:
  m365digester:
    image: dougbarry/m365digester:latest
    environment:
      - "M365_REQUEST_ID=b10c5ed1-bad1-445f-b386-b919946339a7"
      - "OUTPUT_PATH=/output"
    volumes:
      - /host/path/output:/output:rw

---

Use as command line application:

./m365digester-cli -l ./m365-digester.log -k -j ./m365-digester.db -z Allow Default Optimize -m -e testcompany-files.sharepoint.com testcompany-cloud.microsoft.com *.live.com -o puppet-squid-snippet.yml

Use as module:

Default config in M365Digester/Defaults.py
Examples in examples/

Minimum viable use as module:

#!/usr/bin/env python3
import pprint
from m365digester.M365Digester import M365Digester

app = M365Digester()
app.main()
pprint.pprint(app.rule_list)

Module usage with argument parsing

See M365Digester/M365DigesterCli.py

Custom M365 domains in module

from m365digester.M365Digester import M365Digester

...
config = dict()
config.setdefault('extra_known_domains', [
    'mytenancy.sharepoint.com',
    'mytenancy-files.sharepoint.com',
    'mytenancy-my.sharepoint.com',
    'mytenancy-myfiles.sharepoint.com',
    # really just for *.officeapps.live.com but squids splay trees don't always like it
    '.live.com'
])
...
app = M365Digester(config, my_logger)
...

Contributing

If you have any issues or suggestions, please submit an issue on GitHub. All contributions considered and welcomed

License

MIT License