3 Commits

Author	SHA1	Message	Date
Soldier	1ce45cfe97	Add URL scraping with ethical web crawling ... Replace sleep with actual URL fetching. Worker scrapes HTML title from URLs, respects robots.txt, and includes proper User-Agent headers. Scraped titles stored in SQLite and sent via webhook callback.	2025-11-16 08:18:31 +00:00
Soldier	018d699e31	Add webhook callback support ... Add webhook_url column to jobs table. POST /jobs endpoint accepts JSON payload with optional webhook_url. After job completion, worker POSTs to webhook with status and duration.	2025-11-16 08:01:53 +00:00
Soldier	40d194beb1	Add SQLite persistence and worker ... Add jobs table with ID, status, and created_at fields. POST /jobs endpoint creates pending jobs in SQLite. Worker polls every 5s for pending jobs, processes them with 2s delay, and marks as done.	2025-11-16 07:50:59 +00:00