FTD URL Filtering

  • Post author:
  • Post category:Cisco / FTD

With the URL Filtering license, the Cisco Secure Firewall FTD devices can filter based on category and/or reputation of the URL, a URL database is frequently updated from the Cisco Cloud so should always be up to date with the correct information. Without the URL Filtering license (just the Base license) an administrator can manually define URL(s); this does create a management overhead and not practical when permitting access to the internet as a whole.

This post covers URL Filtering on the Cisco Secure Firewall FTD.

Overview

Categories and Reputation

The Cisco cloud contains the analysis of millions of URLs, which are then classified into approximately 100 different classes. The FMC downloads the URL database directly from the Cisco cloud, this database maintains the Web Reputation Index (WRI).

The WRI is calculated dynamically based on points from sources such as:-

  • Age and history of the site
  • Reputation and location of hosting IP address
  • Subject and context of the content.

Web Reputation Level and description:

Reputation Level Name Description
1 Untrusted Site pose a high risk; known to have exposed malicious software to clients.
2 Questionable Sites are suspicious, threat level is higher than average.
3 Neutral Generally benign sites, but they could expose clients to risks.
4 Favourable Benign sites may expose clients to risk, but exposure rare.
5 Trusted Well known trustworthy sites have very strong security features.

 

URL Database

The FMC downloads the URL database from the cloud and receives incremental/smaller updates from the cloud periodically (if updates configured). The FTD downloads the URL database from the FMC and loads the URL dataset into its memory for faster lookup.

Two types of URL datasets are published for local install on the FTD, 20 million URLs and 1 million URLs. FTD devices with memory less than or equal to 3.4Gb RAM 1 million URLs are download, on devices with more than 3.4Gb RAM the entire 20 million URL dataset are downloaded.

URL Lookup

NOTE – When using URL Filtering (and also includes application detection, rate limiting and Intelligent Application Bypass) a few packets must pass in order for the identification to be completed.

The following steps are taken to identify the URL.

  1. The FTD inspects the client traffic, and a session is established between a client and a server.
  2. The FTD uses the snort engine to identify the requested URL and resolves the URL into its category and reputation by looking at the local URL dataset.
    • With HTTPS traffic SSL decryption is not required, the URL is retrieved from the Client Hello packet which contains the Server Name Indication
    • When the web server is using TLS 1.3 to secure communications, the TLS 1.3 protocol encrypts the server certificate for additional security, the certificate is required to match application and URL filtering. The FTD can extract the server certificate without decrypting the packet, using the setting Early application detection and URL categorisation option for TLS Server Identity Discovery.
    • Traffic identification would typically occur within 3 to 5 packets, or after the server certificate exchange in TLS handshake.
  3. If the FTD cannot resolve the URL from its local cache the query is sent to the FMC which performs a lookup on its own database, which is typically a larger URL dataset.
  4. If the FMC fails to resolve a URL it can send the query to the Cisco Cloud.
    • If cloud lookup is disabled, the FMC places an unknown URL into the Uncategorised

Uncategorised URLs

The FTD does not let uncategorised traffic pass until URL lookup is complete or the lookup process times out. The URL remains uncategorised until a category is determined via a cloud lookup. The initial flows are allowed, subsequent connections continued to lookup the URL via the cloud, this can lead to poor performance.

To avoid this, you can let the FTD immediately pass traffic whenever a URL appears un-cached and the URL category cannot be determined locally by disabling the Retry URL Cache Miss Lookup option, bypassing the cloud lookups.

Best Practices

The following are some of Cisco’s recommended best practices when using URL Filtering.

  • Enable the URL Filtering monitor in the health policy.
  • Enable the Cisco Cloud Services for URL Filtering, this ensures the most up to date classifications of URLs.
  • Enable cached URLs to improve user’s web browsing.
  • Ensure the FMC updates the URL database periodically (automatic or manually), the FMC communicates with the Cisco Cloud Services every 30 minutes to receive updates for the URL Filtering database.

Configuring URL Filtering

Ensure the FTD is licensed for URL Filtering

  • Navigate to Devices > Device Management > DEVICENAME
  • Click Device
  • Under the License select URL Filtering
  • Click Save to exit.

Configuring Updates

Confirm URL Filtering automatic updates are enabled, this ensures the URL Filtering database will be up to date.

  • Navigate to System > Integration > Cloud Services
  • Ensure Enable Automatic Updates is enabled to update URL database.
  • Ensure Query Cisco Cloud for unknown URLs (optional)

Configuring Access Control Policy

In this testing scenario we shall create two URL filtering rules, one rule to block the Social Networking category which includes sites such as twitter, Instagram etc and then permit any URL with a favourable or trusted reputation.

  • Navigate to Policies > Access Control > Access Control Policy
  • Edit the Policy assigned to the FTD.
  • Click Add Rule
    • Set and appropriate Name
    • Specify the Action as Block
    • Click URLs and from the Category section select category to block, i.e., Social Networking, click Add to Rule
    • From the Reputations column select Any, click Add to Rule
  • Click Logging, and at select Log at Start of Connection.
  • Click Save

Create another rule to allow any URL that has a favourable reputation, this rule is created below the Block rule above.

  • Click Add Rule
    • Set and appropriate Name, i.e., Allow favourable URL.
    • Specify the Action as Allow
    • Click URLs and from the Category section select category to block, i.e., Any (Except Uncategorized), click Add to Rule
    • From the Reputations column select Favourable, click Add to Rule

  • Click Logging, and at select Log at End of Connection.
  • Click Save

 

Advanced Options

  • From with the Access Control Policy, click the Advanced
  • Edit the TLS Server Identity Discovery and enable Early application detection and URL categorisation – this feature requires FTD running version 6.7 or newer.

Verification

Testing social networking website

For testing we shall access twitter.com, this should be blocked as it is a Social Networking site.

  • From the CLI of the FTD run the command system support firewall-engine-debug filter on the source IP address of the client and generate traffic to twitter.com

From the output below we can determine

  • The rule matching is pending until the reputation and category of the url is determined.
  • The URL is identified as twitter.com.
  • The URL lookup determines the reputation is 90 (trusted) and the category is 2069 (Social Networking) for twitter.com.
  • The final rule is determined as matching Block social media, and the connection is dropped.
> system support firewall-engine-debug

Please specify an IP protocol: tcp
Please specify a client IP address: 192.168.11.100
Please specify a client port:
Please specify a server IP address:
Please specify a server port:
Monitoring firewall engine debug messages.

192.192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 New firewall session
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 app event with app id no change, url no change, tls host no change, bits 0x1
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Starting with minimum 2, 'Block social media', and SrcZone first with zones 2 -> 1, geo 0 -> 0, vlan 0, src sgt: 0, src sgt type: unknown, dst sgt: 0, dst sgt type: unknown, svc 0, payload 0, client 0, misc 0, user 9999997
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 pending rule order 2, 'Block social media', AppID for URL
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 rule order 2, 'Block social media', action Block continue eval of pending deny
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 pending rule order 3, 'Any favourable URL', AppID for URL
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Deferring trust until a rule is matched
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 pending rule id 268437504 nothing has changed
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 wait for decryption event
192.168.11.100 51855 -- 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 InsightUrlListEventHandler: No active URL entries
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 app event with app id changed, url no change, tls host changed, bits 0x118
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 not decrypting event
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 app event with app id changed, url no change, tls host no change, bits 0x4
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Starting with minimum 2, 'Block social media', and SrcZone first with zones 2 -> 1, geo 0(xff 0) -> 0, vlan 0, src sgt: 0, src sgt type: unknown, dst sgt: 0, dst sgt type: unknown, svc 1122, payload 882, client 1296, misc 0, user 9999997, url twitter.com, host twitter.com, no xff
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 no match rule order 2, 'Block social media', url 0 (twitter.com) custom url
192.168.11.100 51855 -- 104.244.42.193 443 6 AS=0 ID=1 GR=1-1: returned from url lookup, url_info is 90 2069 0 0 0 0 0 0 0 0
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 URL lookup for twitter.com found rep 90, cat 2069, 0, 0, 0, 0, 0, 0
192.168.11.100 51855 -- 104.244.42.193 443 6 AS=0 ID=1 GR=1-1: DataMessaging_GetURLData: Returning URL_DBTYPE for twitter.com
192.168.11.100 51855 -- 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 InsightUrlDataEventHandler: status 0x00010000, INSIGHT_FOUND (0x00010000) | SHMDB (1), twitter.com, reputation 90, category_list [ 2069 0 0 0 0 0 0 ]
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 rule order 2, 'Block social media', URL Lookup Success: twitter.com waited: 0ms
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 match rule order 2, 'Block social media', action Block
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 MidRecovery data sent for rule id: 268437504, rule_action:4, rev id:1359738725, rule_match flag:0x1
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Matched a rule, stop deferring trust
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Generating an SOF event
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 deny action
192.168.11.100 51855 -> 104.244.42.193 443 6 AS=0 ID=1 GR=1-1 Deleting Firewall session

From the FMC events we can confirm that access to twitter was blocked.

The brightcloud online URL lookup can be used to double check the reputation and category of a website.

From the screenshot below we can determine that twitter.com has a reputation of 92 and is classified in the category of Social Networking.

Testing a favourable reputation website

Testing using any favourable website, from the output below we can confirm:

  • The rule matching is pending until the reputation and category of the url is determined.
  • The URL is identified as bbc.co.uk.
  • The URL lookup determines the reputation is 70 (trusted) and the category is 2020 (News and Media) for bbc.co.uk.
  • The final rule is determined as matching Any Favourable URL, and the connection is allowed.
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 New firewall session
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 app event with app id no change, url no change, tls host no change, bits 0x1
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 Starting with minimum 2, 'Block social media', and SrcZone first with zones 2 -> 1, geo 0 -> 0, vlan 0, src sgt: 0, src sgt type: unknown, dst sgt: 0, dst sgt type: unknown, svc 0, payload 0, client 0, misc 0, user 9999997
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 pending rule order 2, 'Block social media', AppID for URL
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 rule order 2, 'Block social media', action Block continue eval of pending deny
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 pending rule order 3, 'Any favourable URL', AppID for URL
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 Deferring trust until a rule is matched
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 pending rule id 268437504 nothing has changed
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 wait for decryption event
192.168.11.100 52306 -- 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 InsightUrlListEventHandler: No active URL entries
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 app event with app id changed, url no change, tls host changed, bits 0x118
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 not decrypting event
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 app event with app id changed, url no change, tls host no change, bits 0x4
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 Starting with minimum 2, 'Block social media', and SrcZone first with zones 2 -> 1, geo 0(xff 0) -> 0, vlan 0, src sgt: 0, src sgt type: unknown, dst sgt: 0, dst sgt type: unknown, svc 1122, payload 1376, client 1296, misc 0, user 9999997, url bbc.co.uk, host bbc.co.uk, no xff
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 no match rule order 2, 'Block social media', url 0 (bbc.co.uk) custom url
192.168.11.100 52306 -- 151.101.0.81 443 6 AS=0 ID=1 GR=1-1: returned from url lookup, url_info is 70 2020 0 0 0 0 0 0 0 0
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 URL lookup for bbc.co.uk found rep 70, cat 2020, 0, 0, 0, 0, 0, 0
192.168.11.100 52306 -- 151.101.0.81 443 6 AS=0 ID=1 GR=1-1: DataMessaging_GetURLData: Returning URL_DBTYPE for bbc.co.uk
192.168.11.100 52306 -- 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 InsightUrlDataEventHandler: status 0x00010000, INSIGHT_FOUND (0x00010000) | SHMDB (1), bbc.co.uk, reputation 70, category_list [ 2020 0 0 0 0 0 0 ]
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 rule order 2, 'Block social media', URL Lookup Success: bbc.co.uk waited: 0ms
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 no match rule order 2, 'Block social media', host 0 (bbc.co.uk) c=2020 r=70
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 no match rule order 3, 'Any favourable URL', url 0 (bbc.co.uk) custom url
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 match rule order 3, 'Any favourable URL', action Allow
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 MidRecovery data sent for rule id: 268437506, rule_action:2, rev id:2400384869, rule_match flag:0x1
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 Matched a rule, stop deferring trust
192.168.11.100 52306 -> 151.101.0.81 443 6 AS=0 ID=1 GR=1-1 allow action

Summary

With the rules created any Social Networking site and websites with a reputation of less than 60 will be disallowed, all other URLs with a reputation of more than 61 will be permitted.

Tags: , ,