Our Crawler
Digital Dignity is a Swiss non-profit that helps people affected by image-based sexual abuse find non-consensual intimate images of themselves online. To make that possible, our crawler visits public web pages and downloads the images and videos on them, so that our members can search them for their own face and request removal.
How to recognise our traffic
Every request of our crawler and of its image and video downloads comes from one IP address:
- IPv4 address:
62.238.91.149 - Reverse DNS:
crawler.digitaldignity.io
Page requests use common browser User-Agent strings. Image and video downloads identify as DigitalDignity-MediaWorker/1.0 or DigitalDignity-VideoWorker/1.0. The IP address is the reliable way to identify our traffic.
What it fetches
Only publicly reachable pages and the images and videos they show. It does not log in, does not pass paywalls, and does not solve or bypass CAPTCHAs or other challenges.
- A page the site refuses (for example with HTTP 403, 451, 503 or a challenge page) is not fetched again in the same run.
- An image or video download refused with HTTP 401 or 403 is retried once right away, without the Referer and Cookie headers, if it carried them. Images and video files are then given up. A streamed video (HLS or DASH) can be tried again later, at most three times in all.
- After HTTP 429 on a page, the crawler slows down for that site and fetches the page again later. A download answered with HTTP 429 is tried again later, at most five times for an image and three times for a video.
How fast
- At most one page request per second per host, shared by all our crawler processes. Many sites are crawled more slowly.
- Separately, at most one image or video download per second per host. A single video download may use up to four parallel range requests.
Contact us
If our crawler causes a problem on your site, or you received an abuse report about 62.238.91.149, write to abuse@digitaldignity.io. We answer within 24 hours.
It helps us to know the affected host, the time of the requests with its time zone, and what you would like us to do, for example to slow down or to stop fetching your site.