About HarZaviyaBot
HarZaviyaBot is the crawler behind HarZaviya. It fetches the public RSS/Atom feeds of Pakistani news outlets to build one combined headline feed that links back to each outlet.
It identifies itself with this User-Agent:
HarZaviyaBot/0.1 (+https://harzaviya.com/about-bot)How it behaves
- Public feeds only. It reads RSS/Atom feeds, and — to fill in a missing thumbnail or summary — the Open Graph metadata of an article's own public page. It never stores or republishes the full text of an article.
- It identifies itself with the User-Agent above, which links here.
- Conditional requests. It sends
ETag/If-Modified-Sinceso it only downloads a feed when something has changed. - Gentle pace. It checks each feed roughly once every 20 minutes and throttles any per-article requests.
- Honors robots.txt.
Contact & opt-out
If you run an outlet and would like HarZaviyaBot to stop, want a different feed used, or have any
concern, email [email protected]
— or simply disallow the HarZaviyaBot User-Agent in your robots.txt and it will comply.