Soldier 1ce45cfe97 Add URL scraping with ethical web crawling ... Replace sleep with actual URL fetching. Worker scrapes HTML title from URLs, respects robots.txt, and includes proper User-Agent headers. Scraped titles stored in SQLite and sent via webhook callback.		2025-11-16 08:18:31 +00:00
..
db.go	Add URL scraping with ethical web crawling	2025-11-16 08:18:31 +00:00