How to connect AlienVault OTX to Azure Sentinel

I am excited to announce an updated AlienVault OTX playbook for Azure Sentinel. Inspired by Matt Eagan’s Sentinel Ingestion article. This is a Logic App (playbook) to import threat indicators from AlienVault into Azure Sentinel using the Graph Security API.

My goal was to expand on Matt’s example to create an easy-to-use template. I was introduced to a great example by Rich Lilly that includes a Defender ATP (Defender Endpoint Protection) feed. These are good examples of a daily OTX feed.

Side Note: When sending IOCs to MDE using the Graph Security API, there is a limit of 15,000 custom indicators per tenant and only Files, IP addresses, URLs, domains, and Certificates are allowed.

My next goal was to provide a historic IOC collection option. I wanted to ingest months or years of IOC information. My secondary goal was to look for additional ways to further enrich the data stored in Sentinel.

I ran into two obstacles:

1. There is a 1000 workflow limit per logic app execution. When using a single For-Loop I was unable to pull in large numbers of IOCs.
2. Working with a large dataset it would be useful to have a date ingested value. The TimeGenerated value in the threat intelligence table gets updated periodically for records older than 14 days. This is to support the 14 day lookback limit on analytic rules.

The updated playbook overcomes this limitation by breaking the request into pages (1000 indicators each). I pulled in 5 years of IOC data (roughly 200,000 records) in testing. This runs for about 10 minutes for every 10,000 records. This is intended to be a one time lookback followed by a daily maintenance update. Despite being a rather complex logic app, each record counts as only 2 action executions (200k records costs around $10).

To improve usability and data enrichment, I added more setup variables and made some minor adjustments. I added a lookup URL to the additionalInformation column that links back the AlienVault lookup for each IOC. I also use the “FileCreatedDateTime” column to log the time ingested. This date0time value was not previously being used.

Instructions:

1. Get an API key from AlienVault: https://otx.alienvault.com/
2. Create an App Registration in Azure AD.
3. Import the Logic App (disabled by default)
4. Set the run variables (Tennant ID, Client ID, App Secret, and OTX API Key).
5. Enable and run.
6. Activate the appropriate TI Map rules to enable alerting.

Note: Consider running a one-time historic lookback (described below).

Historic Data Lookback (RUN ONCE):

1. Set the lookback days to a desired value (example 365)
2. Enable and run the Logic App (estimate 10 minutes processing time for every 10k records)
3. Make sure to reset the Lookback days to the default 1 day when complete

Note: During testing the provider returned some incorrectly formatted records. This was only observed in large collections. The app does not have error checking. Incorrectly formatted records will fail if encountered but the overall app will complete. This will cause the log to show the parent app as failed.

App Registration Troubleshooting:

I found in testing and working with customers that the App Registration setup used to authenticate the API is a common cause for issues.

• Make sure the correct API permissions are assigned to your app registration. Some of the examples do not demonstrate the correct permissions. All you need is “ThreatIndicators.ReadWrite.OwnedBy”.
• Make sure to Grant Admin Consent on the API Permission page. You must be a Global Administrator, an Application Administrator, or a Cloud Application Administrator to activate (look for the Green check mark).
• Your App Registration can be assigned to roles at the workspace or RG. You may need to assign additional credentials. If your granted App Registration fails, add your application identity to Log Analytics Reader or Contributor on the workspace.