The latest version of this script works with Pandas dataframes and supports Azure Firewall logs sent to the same storage account (see parameter mode).
You might have set your Azure Vnet, with some NSGs. You start rolling apps, to the point where you have many VMs, and many NSGs. Somebody makes an application upgrade, or installs a new application, but traffic is not flowing through. Which NSG is dropping traffic? Which TCP ports should be opened?
One possibility is using Traffic Analytics. Traffic Analytics is a two-step process:
- NSG logs are stored in a storage account
- NSG logs from the storage account are processed (consolidated and enriched with additional information) and made queriable
One of the nicest features of Traffic Analytics is being able to query logs with the KQL (Kusto Query Language). For example, you can use this query to find out the dropped flows in the last 3 hours for IP address 1.2.3.4:
AzureNetworkAnalytics_CL
| where TimeGenerated >= ago(1h)
| where SubType_s == "FlowLog"
| where DeniedInFlows_d > 0 or DeniedOutFlows_d > 0
| where SrcIP_s == "1.2.3.4"
| project NSGName=split(NSGList_s, "/")[2],NSGRules_s,DeniedInFlows_d,DeniedOutFlows_d,SrcIP_s,DestIP_s,DestPort_d,L7Protocol_s
However, you will notice that there is a time lag, and you will not find the very latest logs in Log Analytics. The original NSG Flow logs are stored in storage account, in JSON format, so you could get those logs using the Azure Storage SDK.
You can use the Python script in this repository (assuming you have the Python SDK for storage installed). You can use different flags, like the --help option to get usage information. If you do not have handy a Python console you can use Azure Cloud Shell and clone this repository:
git clone https://github.com/erjosito/get_nsg_logs
If you don't have the Azure Storage Blob and pandas Python modules, you need to install them:
pip3 install azure-storage-blob pandas --user
Now you can have a look at the different options:
$ cd ./get_nsg_logs/
$ python3 ./get_nsg_logs.py --help
usage: get_nsg_logs.py [-h] [--account-name ACCOUNT_NAME] [--display-lb]
[--display-allowed]
[--display-direction DISPLAY_DIRECTION]
[--display-hours DISPLAY_HOURS] [--version VERSION]
[--only-non-zero] [--flow-state FLOW_STATE_FILTER]
[--ip IP_FILTER] [--port PORT_FILTER]
[--nsg-name NSG_NAME_FILTER] [--aggregate] [--verbose]
Get the latest flow logs in a storage account
optional arguments:
-h, --help show this help message and exit
--account-name ACCOUNT_NAME
you need to supply an storage account name. You can get a list of your storage accounts with this command: az storage account list -o table
--display-lb display or hide flows generated by the Azure LB (default: False)
--display-allowed display as well flows allowed by NSGs (default: False)
--display-direction DISPLAY_DIRECTION
display flows only in a specific direction. Can be in, out, or both (default in)
--display-hours DISPLAY_HOURS
How many hours to look back (default: 1)
--display-minutes DISPLAY_MINUTES
How many minutes to look back (default: 0/unlimited)
--only-non-zero display only v2 flows with non-zero packet/byte counters (default: False)
--flow-state FLOW_STATE_FILTER
filter the output to a specific v2 flow type (B/C/E)
--ip IP_FILTER filter the output to a specific IP address
--port PORT_FILTER filter the output to a specific TCP/UDP port
--protocol PROTOCOL_FILTER
filter the output to a specific protocol (T/U/I)
--nsg-name NSG_NAME_FILTER
filter the output to a specific NSG
--mode MODE can be nsg,fw,both (default: nsg)
--aggregate prints byte/packet count aggregates (default: False)
--no-output does not print out any output, useful with --verbose flag (default: False)
--verbose run in verbose mode (default: False)
There is something you need to do before being able to access Azure Blob Storage: finding out the Azure Storage Account key. The script will read it from the environtment variable STORAGE_ACCOUNT_KEY, that you can set with this command:
storage_account_name=yourstorageaccount123
export STORAGE_ACCOUNT_KEY=$(az storage account keys list -n $storage_account_name --query '[0].value' -o tsv)For example, in order to show dropped and allowed traffic of ingress NSG logs stored in a storage account (if following the previous instructions, stored in the variable $storage_account_name), excluding Azure LB probe traffic for the last 2 hours:
$ python3 ./get_nsg_logs.py --account-name $storage_account_name --display-hours 2 --display-direction in --display-allowed
timestamp type resource rule state src_ip dst_ip src_port dst_port protocol direction action
1276 2022-05-06 07:08:02.967603300+00:00 nsg FLOWLOGS DefaultRule_AllowVnetOutBound E 192.168.1.4 192.168.1.6 33286 1433 T O A
1262 2022-05-06 07:08:02.967603300+00:00 nsg FLOWLOGS DefaultRule_AllowInternetOutBound B 192.168.1.4 13.69.67.61 36552 443 T O A
1261 2022-05-06 07:08:02.967603300+00:00 nsg FLOWLOGS DefaultRule_AllowInternetOutBound E 192.168.1.4 13.69.67.61 36516 443 T O A
1277 2022-05-06 07:08:02.967603300+00:00 nsg FLOWLOGS DefaultRule_AllowVnetOutBound B 192.168.1.4 192.168.1.7 55280 1433 T O A
1275 2022-05-06 07:08:02.967603300+00:00 nsg FLOWLOGS DefaultRule_AllowVnetOutBound E 192.168.1.4 192.168.1.5 58948 80 T O A
...
Another example is showing logs captured to troubleshoot communication between specific machines on a specific port (on the last 3 minutes to reduce output):
❯ python3 ./get_nsg_logs.py --account-name $storage_account_name --mode=nsg --display-hours 1 --display-allowed --display-minutes 3 --ip 192.168.1.4 --ip2 192.168.1.5 --aggregate
timestamp type resource rule state packets_src_to_dst bytes_src_to_dst packets_dst_to_src bytes_dst_to_src src_ip dst_ip src_port dst_port protocol direction action
579 2022-05-06 07:18:34.404934100+00:00 nsg FLOWLOGS DefaultRule_AllowVnetInBound B 192.168.1.4 192.168.1.5 60142 80 T I A
580 2022-05-06 07:18:34.404934100+00:00 nsg FLOWLOGS DefaultRule_AllowVnetInBound E 1 74 1 54 192.168.1.4 192.168.1.5 60142 80 T I A
bytes_src_to_dst 74
packets_src_to_dst 1
bytes_dst_to_src 54
packets_dst_to_src 1
dtype: object
Happy troubleshooting!