Block Bad Bots and Spiders using .htaccess

Below is a useful code block for blocking a lot of the known bad bots and site rippers currently out there.

You might also check out the following .htaccess rules to Harden your website’s Security even further.

Simply add the code to your /public_html/.htaccess file:

# Bad bot
SetEnvIfNoCase User-Agent "^abot" bad_bot
SetEnvIfNoCase User-Agent "^aipbot" bad_bot
SetEnvIfNoCase User-Agent "^asterias" bad_bot
SetEnvIfNoCase User-Agent "^EI" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl" bad_bot
SetEnvIfNoCase User-Agent "^LWP" bad_bot
SetEnvIfNoCase User-Agent "^lwp" bad_bot
SetEnvIfNoCase User-Agent "^MSIECrawler" bad_bot
SetEnvIfNoCase User-Agent "^nameprotect" bad_bot
SetEnvIfNoCase User-Agent "^PlantyNet_WebRobot" bad_bot
SetEnvIfNoCase User-Agent "^UCmore" bad_bot
SetEnvIfNoCase User-Agent "Alligator" bad_bot
SetEnvIfNoCase User-Agent "AllSubmitter" bad_bot
SetEnvIfNoCase User-Agent "Anonymous" bad_bot
SetEnvIfNoCase User-Agent "Asterias" bad_bot
SetEnvIfNoCase User-Agent "autoemailspider" bad_bot
SetEnvIfNoCase User-Agent "Badass" bad_bot
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "BecomeBot" bad_bot
SetEnvIfNoCase User-Agent "Bitacle" bad_bot
SetEnvIfNoCase User-Agent "bladder\ fusion" bad_bot
SetEnvIfNoCase User-Agent "Blogshares\ Spiders" bad_bot
SetEnvIfNoCase User-Agent "Board\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Board\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Convera" bad_bot
SetEnvIfNoCase User-Agent "ConveraMultiMediaCrawler" bad_bot
SetEnvIfNoCase User-Agent "c-spider" bad_bot
SetEnvIfNoCase User-Agent "DA" bad_bot
SetEnvIfNoCase User-Agent "DnloadMage" bad_bot
SetEnvIfNoCase User-Agent "Download\ Demon" bad_bot
SetEnvIfNoCase User-Agent "Download\ Express" bad_bot
SetEnvIfNoCase User-Agent "Download\ Wonder" bad_bot
SetEnvIfNoCase User-Agent "dragonfly" bad_bot
SetEnvIfNoCase User-Agent "DreamPassport" bad_bot
SetEnvIfNoCase User-Agent "DSurf" bad_bot
SetEnvIfNoCase User-Agent "DTS Agent" bad_bot
SetEnvIfNoCase User-Agent "EBrowse" bad_bot
SetEnvIfNoCase User-Agent "eCatch" bad_bot
SetEnvIfNoCase User-Agent "edgeio" bad_bot
SetEnvIfNoCase User-Agent "Email\ Extractor" bad_bot
SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "EmeraldShield" bad_bot
SetEnvIfNoCase User-Agent "ESurf" bad_bot
SetEnvIfNoCase User-Agent "Exabot" bad_bot
SetEnvIfNoCase User-Agent "ExtractorPro" bad_bot
SetEnvIfNoCase User-Agent "FileHeap!\ file downloader" bad_bot
SetEnvIfNoCase User-Agent "FileHound" bad_bot
SetEnvIfNoCase User-Agent "Forex" bad_bot
SetEnvIfNoCase User-Agent "Franklin\ Locator" bad_bot
SetEnvIfNoCase User-Agent "FreshDownload" bad_bot
SetEnvIfNoCase User-Agent "FrontPage" bad_bot
SetEnvIfNoCase User-Agent "FSurf" bad_bot
SetEnvIfNoCase User-Agent "Gaisbot" bad_bot
SetEnvIfNoCase User-Agent "Gamespy_Arcade" bad_bot
SetEnvIfNoCase User-Agent "genieBot" bad_bot
SetEnvIfNoCase User-Agent "GetBot" bad_bot
SetEnvIfNoCase User-Agent "GetRight" bad_bot
SetEnvIfNoCase User-Agent "Gigabot" bad_bot
SetEnvIfNoCase User-Agent "Go!Zilla" bad_bot
SetEnvIfNoCase User-Agent "Go-Ahead-Got-It" bad_bot
SetEnvIfNoCase User-Agent "GOFORITBOT" bad_bot
SetEnvIfNoCase User-Agent "heritrix" bad_bot
SetEnvIfNoCase User-Agent "HLoader" bad_bot
SetEnvIfNoCase User-Agent "HooWWWer" bad_bot
SetEnvIfNoCase User-Agent "HTTrack" bad_bot
SetEnvIfNoCase User-Agent "iCCrawler" bad_bot
SetEnvIfNoCase User-Agent "ichiro" bad_bot
SetEnvIfNoCase User-Agent "iGetter" bad_bot
SetEnvIfNoCase User-Agent "imds_monitor" bad_bot
SetEnvIfNoCase User-Agent "Industry\ Program" bad_bot
SetEnvIfNoCase User-Agent "Indy\ Library" bad_bot
SetEnvIfNoCase User-Agent "InetURL" bad_bot
SetEnvIfNoCase User-Agent "InstallShield\ DigitalWizard" bad_bot
SetEnvIfNoCase User-Agent "IRLbot" bad_bot
SetEnvIfNoCase User-Agent "IUPUI\ Research\ Bot" bad_bot
SetEnvIfNoCase User-Agent "Java" bad_bot
SetEnvIfNoCase User-Agent "jeteye" bad_bot
SetEnvIfNoCase User-Agent "jeteyebot" bad_bot
SetEnvIfNoCase User-Agent "JoBo" bad_bot
SetEnvIfNoCase User-Agent "JOC\ Web\ Spider" bad_bot
SetEnvIfNoCase User-Agent "Kapere" bad_bot
SetEnvIfNoCase User-Agent "Larbin" bad_bot
SetEnvIfNoCase User-Agent "LeechGet" bad_bot
SetEnvIfNoCase User-Agent "LightningDownload" bad_bot
SetEnvIfNoCase User-Agent "Linkie" bad_bot
SetEnvIfNoCase User-Agent "Mac\ Finder" bad_bot
SetEnvIfNoCase User-Agent "Mail\ Sweeper" bad_bot
SetEnvIfNoCase User-Agent "Mass\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "MetaProducts\ Download\ Express" bad_bot
SetEnvIfNoCase User-Agent "Microsoft\ Data\ Access" bad_bot
SetEnvIfNoCase User-Agent "Microsoft\ URL\ Control" bad_bot
SetEnvIfNoCase User-Agent "Missauga\ Locate" bad_bot
SetEnvIfNoCase User-Agent "Missauga\ Locator" bad_bot
SetEnvIfNoCase User-Agent "Missigua Locator" bad_bot
SetEnvIfNoCase User-Agent "Missouri\ College\ Browse" bad_bot
SetEnvIfNoCase User-Agent "Mister\ PiX" bad_bot
SetEnvIfNoCase User-Agent "MovableType" bad_bot
SetEnvIfNoCase User-Agent "Mozi!" bad_bot
SetEnvIfNoCase User-Agent "Mozilla/3.0 (compatible)" bad_bot
SetEnvIfNoCase User-Agent "Mozilla/5.0 (compatible; MSIE 5.0)" bad_bot
SetEnvIfNoCase User-Agent "MSIE_6.0" bad_bot
SetEnvIfNoCase User-Agent "MSIECrawler" badbot
SetEnvIfNoCase User-Agent "MVAClient" bad_bot
SetEnvIfNoCase User-Agent "MyFamilyBot" bad_bot
SetEnvIfNoCase User-Agent "MyGetRight" bad_bot
SetEnvIfNoCase User-Agent "NASA\ Search" bad_bot
SetEnvIfNoCase User-Agent "Naver" bad_bot
SetEnvIfNoCase User-Agent "NaverBot" bad_bot
SetEnvIfNoCase User-Agent "NetAnts" bad_bot
SetEnvIfNoCase User-Agent "NetResearchServer" bad_bot
SetEnvIfNoCase User-Agent "NEWT\ ActiveX" bad_bot
SetEnvIfNoCase User-Agent "Nextopia" bad_bot
SetEnvIfNoCase User-Agent "NICErsPRO" bad_bot
SetEnvIfNoCase User-Agent "NimbleCrawler" bad_bot
SetEnvIfNoCase User-Agent "Nitro\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "Nutch" bad_bot
SetEnvIfNoCase User-Agent "Offline\ Explorer" bad_bot
SetEnvIfNoCase User-Agent "OmniExplorer" bad_bot
SetEnvIfNoCase User-Agent "OutfoxBot" bad_bot
SetEnvIfNoCase User-Agent "P3P" bad_bot
SetEnvIfNoCase User-Agent "PagmIEDownload" bad_bot
SetEnvIfNoCase User-Agent "pavuk" bad_bot
SetEnvIfNoCase User-Agent "PHP\ version" bad_bot
SetEnvIfNoCase User-Agent "playstarmusic" bad_bot
SetEnvIfNoCase User-Agent "Program\ Shareware" bad_bot
SetEnvIfNoCase User-Agent "Progressive Download" bad_bot
SetEnvIfNoCase User-Agent "psycheclone" bad_bot
SetEnvIfNoCase User-Agent "puf" bad_bot
SetEnvIfNoCase User-Agent "PussyCat" bad_bot
SetEnvIfNoCase User-Agent "PuxaRapido" bad_bot
SetEnvIfNoCase User-Agent "Python-urllib" bad_bot
SetEnvIfNoCase User-Agent "RealDownload" bad_bot
SetEnvIfNoCase User-Agent "RedKernel" bad_bot
SetEnvIfNoCase User-Agent "relevantnoise" bad_bot
SetEnvIfNoCase User-Agent "RepoMonkey\ Bait\ &\ Tackle" bad_bot
SetEnvIfNoCase User-Agent "RTG30" bad_bot
SetEnvIfNoCase User-Agent "SBIder" bad_bot
SetEnvIfNoCase User-Agent "script" bad_bot
SetEnvIfNoCase User-Agent "Seekbot" bad_bot
SetEnvIfNoCase User-Agent "SiteSnagger" bad_bot
SetEnvIfNoCase User-Agent "SmartDownload" bad_bot
SetEnvIfNoCase User-Agent "sna-" bad_bot
SetEnvIfNoCase User-Agent "Snap\ bot" bad_bot
SetEnvIfNoCase User-Agent "SpeedDownload" bad_bot
SetEnvIfNoCase User-Agent "Sphere" bad_bot
SetEnvIfNoCase User-Agent "sproose" bad_bot
SetEnvIfNoCase User-Agent "SQ\ Webscanner" bad_bot
SetEnvIfNoCase User-Agent "Stamina" bad_bot
SetEnvIfNoCase User-Agent "Star\ Downloader" bad_bot
SetEnvIfNoCase User-Agent "Teleport" bad_bot
SetEnvIfNoCase User-Agent "TurnitinBot" bad_bot
SetEnvIfNoCase User-Agent "UdmSearch" bad_bot
SetEnvIfNoCase User-Agent "URLGetFile" bad_bot
SetEnvIfNoCase User-Agent "User-Agent" bad_bot
SetEnvIfNoCase User-Agent "UtilMind\ HTTPGet" bad_bot
SetEnvIfNoCase User-Agent "WebAuto" bad_bot
SetEnvIfNoCase User-Agent "WebCapture" bad_bot
SetEnvIfNoCase User-Agent "webcollage" bad_bot
SetEnvIfNoCase User-Agent "WebCopier" bad_bot
SetEnvIfNoCase User-Agent "WebFilter" bad_bot
SetEnvIfNoCase User-Agent "WebReaper" bad_bot
SetEnvIfNoCase User-Agent "Website\ eXtractor" bad_bot
SetEnvIfNoCase User-Agent "WebStripper" bad_bot
SetEnvIfNoCase User-Agent "WebZIP" bad_bot
SetEnvIfNoCase User-Agent "Wells\ Search" bad_bot
SetEnvIfNoCase User-Agent "WEP\ Search\ 00" bad_bot
SetEnvIfNoCase User-Agent "Wget" bad_bot
SetEnvIfNoCase User-Agent "Wildsoft\ Surfer" bad_bot
SetEnvIfNoCase User-Agent "WinHttpRequest" bad_bot
SetEnvIfNoCase User-Agent "WWWOFFLE" bad_bot
SetEnvIfNoCase User-Agent "Xaldon\ WebSpider" bad_bot
SetEnvIfNoCase User-Agent "Y!TunnelPro" bad_bot
SetEnvIfNoCase User-Agent "YahooYSMcm" bad_bot
SetEnvIfNoCase User-Agent "Zade" bad_bot
SetEnvIfNoCase User-Agent "ZBot" bad_bot
SetEnvIfNoCase User-Agent "zerxbot" bad_bot

# Apache < 2.4
<IfModule !mod_authz_core.c>
 <Limit GET POST>
  Order allow,deny
  Allow from all
  Deny from env=bad_bot
 </Limit>
</IfModule>
# Apache >= 2.4
<IfModule mod_authz_core.c>
 <Limit GET POST>
  <RequireAll>
   Require all granted
   Require not env bad_bot
  </RequireAll>
 </Limit>
</IfModule>

Was this answer helpful?

Also Read

Secure Joomla with .htaccess

Advanced .htaccess rules that will overall secure and increase the speed of your Joomla...

What does File Permissions "chmod" mean?

To chmod a file means to set certain permissions. By default, all files uploaded to the server...

.htaccess URL Masking Examples

If you would like to access a specific URL for your domain but do not display it in the browser's...

How to Remove Multiple Slashes from URL

You can use this .htaccess rule for removing multiple trailing slashes anywhere in URL:...

.htaccess rules to Harden your website’s Security

Below you see a list of the important rules that you can add to your .htaccess file to harden...