Dark Web Investigation Search Engine

An India-focused investigative research tool for authorized researchers to discover, index, and safely examine relevant content from selected Tor hidden services.

GitHub repository

Problem statement

Investigating illicit activities, threat actor communications, and leaked credential caches on the dark web poses significant technical and operational risks: malware distribution, hostile scripts, and non-deterministic hidden service availability. Investigators need a controlled, safe environment for targeted discovery.

Motivation and intended users

To build an investigative research platform using Django, SQLite, and Tor proxy routing that enforces source boundaries, verifies link availability concurrently, and renders retrieved page artifacts in an isolated, script-restricted preview environment.

Technologies and stack

PythonDjangoSQLiteTor (SOCKS5)BeautifulSoup4Content Security Policy

Dark Web Investigation Search Engine: System Architecture & Safety Controls

The Investigative Challenge

Researchers and cybercrime analysts monitoring leaked indicators, compromised infrastructure, and threat actor publications on Tor hidden services face acute operational hazards:

Client-Side Exploits: Malicious JavaScript or browser zero-days embedded in onion pages.
Deanonymization Hazards: External resource calls (web beacons, remote stylesheets, tracking pixels) that bypass SOCKS proxies.
Volatile Lifespans: Onion endpoints frequently rotate, go offline, or switch hosting nodes.

text
[Authorized Investigator]
          │ (Submits Query / Selects Source Registry)
          ▼
   [Django Search Controller]
          │
          ├──> [Source Registry Whitelist (SQLite)]
          │
          ▼
   [Tor Proxy Worker Pool (SOCKS5 127.0.0.1:9050)]
          │
          ▼ (Tor Circuit Routing)
   [.onion Hidden Services]
          │
          ▼ (Raw Response Ingestion)
   [Sanitization & Parsing Engine (BeautifulSoup)]
          ├─ Remove <script>, <object>, <iframe>, <embed>
          ├─ Strip external inline style handlers (onload, onerror)
          ├─ Normalize metadata, headers, timestamps
          │
          ▼
   [Artifact Persistence (SQLite)]
          │
          ▼
   [Sandboxed Render View] (iframe sandbox="allow-same-origin" CSP: default-src 'none')

Safety Controls & Content Security Policy (CSP)

The platform prioritizes researcher safety through defense-in-depth:

Network Boundary: All external HTTP requests pass through the local Tor daemon via SOCKS5 proxying. Direct socket connections outside the proxy are blocked.
HTML Sanitization: Responses are parsed into a DOM tree where all active script tags, frames, and event attributes are aggressively stripped.

CSP Sandbox: Previews are served with HTTP headers enforcing:

http
Content-Security-Policy: default-src 'none'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; sandbox allow-same-origin;
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN

Verified Implementation Scope

Backend: Django 4.x application structured with modular apps: sources, crawler, search, and audit.
Concurrency: Parallel link checking using Python's concurrent.futures.ThreadPoolExecutor configured with conservative worker caps to respect Tor connection pool limits.
Disclaimer: This project does not provide anonymity guarantees against state-level traffic correlation and is maintained strictly for defensive threat analysis.

Key features and decisions

›Controlled Source Registry: Restricts crawling strictly to explicitly permitted .onion hidden service addresses.
›SOCKS5 Tor Routing: Decouples requests through the Tor client to prevent origin IP leakage during reconnaissance.
›Concurrent Link Availability Checker: Multithreaded validation worker verifying node reachability.
›Isolated Preview Engine: Strips active content, JavaScript, external resources, and renders via sandboxed iframes with CSP `default-src 'none'`.
›Administrative Investigation Log: Audits search queries, fetched artifacts, and timestamps for legal/evidentiary traceability.

Challenges and limitations

!This tool is an investigative research utility, not an automated web crawler indexing the entire dark web.
!Tor network latency and onion routing circuits result in variable response times (3s–30s per request).
!Safety mechanisms strictly prohibit executing client-side scripts, which limits rendering of JavaScript-dependent SPAs.
!Access and deployment are restricted to authorized research and educational scenarios.