Content Discovery – PortSwigger
This feature can be used to discover content and functionality that is not related to visible content that you can browse or search.
To access this feature, select an HTTP request anywhere in Burp, or any part of the target sitemap, and choose “Discover Content” under “Engagement Tools” from the context menu.
Burp uses a variety of techniques to discover content, including name searching, web crawling, and extrapolation from naming conventions seen in the app. The discovered content is displayed in a special sitemap that is specific to the discovery session and can also be optionally added to the master suite sitemap.
This tab shows you the current state of the discovery session.
The toggle button indicates whether the session is running and allows you to pause and restart the session.
The following information is displayed about the progress of the discovery session:
Number of requests made
Number of bytes transferred in server responses
Number of network errors
Number of Queued Discovery Jobs
Number of spider requests queued
Number of responses awaiting analysis
The individual discovery tasks that are queued are displayed in a table. The discovery engine works recursively and when a new directory or file is discovered, other tasks follow, depending on the configuration. For example, when a new directory is discovered, Burp can add tasks to find subdirectories and files in that directory; or, when a new file is discovered, Burp can add a task to find the same base file name with different file extensions. Newly added tasks are prioritized based on their likelihood of quickly discovering new content.
These options allow you to define the starting directory for the content discovery session and whether files or directories should be targeted. The following options are available:
- Startup Directory – This is where Burp will start looking for content. Only the elements of this path and its subdirectories will be requested during the session.
- Discover – This option determines whether the session will search for files or directories or both. If you are searching for directories, you can choose whether and how deep to recursion in the discovered subdirectories.
These options allow you to configure the sources that Burp should use to generate the filenames to test. The following options are available
Built-in short file list
Integrated short directory list
Long built-in file list
Long built-in directory listing
Custom File List
Custom directory listing
Names discovered in use at the target site. If this option is selected, Burp will keep a list of all directories and filename roots that have been discovered on the target site, and will also check them in each new directory tested.
Derivatives based on the elements discovered. If this option is selected, Burp will attempt to guess the names of the items based on those that have already been discovered. For example, if the directory
AnnualReport2018is discovered, Burp will also check
These settings control how the discovery session adds file extensions to the file stems under test. The file stems themselves are derived based on the file name options. As each file stalk is tested, Burp looks for a variety of different extensions, based on these settings. The following options are available:
- Test these extensions – This option allows you to configure a list of extensions that Burp will always check. You can refine the default list based on technologies known to be used on the target application.
- Test all extensions observed on the target site – If this option is selected, then Burp will automatically check for file extensions that have been observed in use on the target site. This option is useful when you are not sure what extensions or technologies are being used. You can also configure a list of extensions that you don’t want to check even if they are in use (like image files).
- Test these variant extensions on the discovered files – This option allows you to configure a list of extensions that Burp will additionally check using the roots of the file names discovered. This option is useful for checking backup copies of existing files.
- Test the stems of files without extension – If this option is selected, Burp will check every file stalk with no extension added.
These settings control the engine used to make HTTP requests when discovering content and interacting with the suite sitemap. The following options are available:
- Case sensitivity – This setting controls whether Burp will handle filenames with sensitivity. If “Auto detect” is selected, then Burp will first process file names with case sensitivity, and upon discovery of the first new item, will test the server’s handling of case variations. Depending on this processing, Burp may revert to case-insensitive handling of file names.
- Add the discovered content to the sitemap of the suite – If this option is selected, the new elements identified in the current discovery session will be automatically added to the site map of the main suite.
- Copy the contents of the sitemap of the suite – If this option is selected, the discovery session will copy any existing relevant content from the master suite sitemap into the discovery sitemap, to provide a more solid starting point for discovering new content.
- Spider of the discovered content – If this option is selected, the discovery session will perform conventional web crawling and process responses to discovery requests looking for links to additional new content.
- Number of discovery threads – This option controls the number of simultaneous requests that the discovery engine is able to make.
- Number of spider threads – This option controls the number of simultaneous requests that the exploration function is able to make, if it is activated.
The discovery session uses its own sitemap, showing all of the content that has been discovered within the defined scope. If you’ve configured Burp to do so, newly discovered items will also be added to Burp’s main site map.