We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I tested this module to scrap https://www.doordash.com/ which implement the cloudfare bot. Here is the output. b'<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><link href="/cdn-cgi/styles/challenges.css" rel="stylesheet"><meta http-equiv="refresh" content="375"></head><body class="no-js"><div class="main-wrapper" role="main"><div class="main-content"><noscript><div id="challenge-error-title"><div class="h2"><span class="icon-wrapper"><div class="heading-icon warning-icon"></div></span><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: \'2\',cZone: "www.doordash.com",cType: \'managed\',cNounce: \'39291\',cRay: \'822a8eb44b584aa8\',cHash: \'602f515f9358e2a\',cUPMDTk: "\\/?__cf_chl_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",cFPWv: \'g\',cTTimeMs: \'1000\',cMTimeMs: \'375000\',cTplV: 5,cTplB: \'cf\',cK: "visitor-time",fa: "\\/?__cf_chl_f_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",md: "z94vGId17dyb7Qd4Pdg.yRRwFM3QgdMOnhliB3UVCE4-1699412339-0-AaTCUhoD_0HKzcTKXdVb5pyCYowLHIPqifSkfKGwdhw-n7lz2PeEZ3bXCjsCG-k69kS0mk81vAvm10qnqBd1AeJmcG_ieZIT3JEUfpybxy9HfL51pPbVNfP6JEA3mqStLsPNeQudDiTi4VqLtfhdZuy1n7M8seXhXOampBeCdFb9rRw2_dOhDPI94gyW1tz43HDVpHlBbnSmiIb83CKyXtj4csJJCwA5255J0iJ7z_wwEbALIePHyQySJzI4ALIrnaogKqiKnUGrNk3Vk8PQ1Fem47zS0dEfXGrOq-xpecw5n--3Ux_usuYWLnWqRQAo0irhCmp82_jQfPSMATqF37T5Q8d2NHcOvocyWHgUdaXFTgUzDtdG2syIn_L8xj_bsixzQ1__QjZB1UM3fTHSJr-paHjVcv6_vMg5astW54lq4fwi-H5oi0ayXJ5HRELLU7LfShV9r-FbDONoC9KTCyd7rdswFPV9N-vsr8FcD-TIvmE2yC9-CkbGHwG0NtipfbcXotkga__2AUe4VomsJ7bedr79QFt2koxU7y8rW9jPqneOkYbwUU0fOQ3mK7BUUvEvlUrAp9VUR7Fvl4te2LA1LbuP_c-Di6A98DBr0kN36KVP9ugt8JdGuIlHRKrL8pMMk5epOcCxO9CSguigSLPgqTjuyjQuf5Stk_tY1TOCBc0SiAy_6vMk63KS-jPUyKGdS-SNSdOt_lR6vpRGc034f42O7V8mrlWYFTb280UVQnl4ZlAbIh2YEm1h2dNcIIuIdO5EG71z1-woFwTOaKcKtIMIyVhCPNwEcLMvBPIbh4yQ2NIeFDh85GCq7XW0piPYVdUTo1991vnvpBEsThmdPX2VDJ21OTIptWDKITX1rjO_GNy0zS8dxnUPCu24kgYapIzoKVF27bNVWVH-RGkP9Lcvk0YCyWEUCU8HN6Zys_Jt5z4OyBoUXkOqN-QFmzBZ6mgFcIvJh-aUt1GozL0Zk8M0QzNYphNnL4KU0zGC2xlbEqAoPTxI8oIe9pN67NNcfdozh4u9sm1k6OH5j6C75PkWEFUiCE7Bs2EgimQKrcyz6vubFNbk7k4S_tYKtaIc-8Q2Oi3hTnFnjGApZdKN-B5xM3VUfkwcohWIaM-DZah5Kxm6ZfSHS607IdL5aCLX6r5Gg7gSTJivv0bpe0kpqVSEEfW1Q8SdkqWF_lg1NfKloT8cv-i0DB0pUbmvprFTOKY6sa7Xmi-bAu2L8XtQA8vC71LLlyK4CCM_HcyDG86-qpgWOhOCYJKGxUJ9IA4snTM7Fhp9Jr_gK7fkh3w-uky88DQ-lKXWsN8kJ0s8V9F0iHe2-mEZN0rG4-AtdBBofvKiKLMUodH7TExsYbN8tL0ZZoagzkZ4vqXXeNsP3G8BHUvDfApfMjGLBz9Jy3gHf8z-sp7S5byLmaf7KEg6zalyy4_vZ0QXEXxOPba8Bx1Qeh1LJN_yly2hMuoDVa6dREtSqJBwpMM5DmD6lfzPseOZdlhzL6d9m7_EkOCOxRTlORd92lLY1QP-ODJRixHtW4XTwQ0oL56k7kJWxVhb-RZLbOT0qgJNHPdK38B_LAtuXgTBt2IMaMbUMrjiTJBekxIyGUULAFDU33HJ73f8wRDL14Ts6sOv8NMHd78tyAaldEJHXuPCIIbC5GJC1WTli1osmbE1DmrHsMDoEuuRJLKPC5SSbQmHMG7tkFkCs514gMZd84PJ8cSGLX9-kt9bPBlJZB9tLhsWBg5ongw-2h7-AtQGUe0RVVP_mxCV1f07B9JfCegiJ33fspvoIEmExYQzKK1vMjAhfazIHXfCd4KeoKD2ZArd9jurDIv7Xg1hXV2tm8YmV552rQbWssKylnDLJnIAwfjSm1WYUbu1k4Aabk0ULycxv-qn3L89lb_gZHXMw3aMDKEvyXtpndKBpEREIHi4lvrUrYtiCYxkPxo6g8EC_28a4sR00peuzRhQRCGOPa80zcAJkmoEjguJanofUaVq8aa75sfGKOyDpRBvb7wVOcL8gPLv9nbqgny-_8Rk0sP3lDADkJpx5DzjgW3v7uWWgVHYbXT98l2yn9wn-vNyotm_lpBvuxMBC2vHNGDhKDEs98qf2YBYPAOMQ_rfF1GyjPc6U7Lfai0OwtxeywMEsygzZGVwKf2yR64YnC0d5YqQIRsgKbXKx7WkgV0qTiLUBuqwNNP0sgsQnsfyvkAuH-qc-zjyQXBmubduTH-ZZV44EZqEss_rC69q6RTi3NW8fCPZmePTLj8ocHUyKhi_2TrhxqCMTA6eXRe1J9K2Ffxmk48M7iEhFAjZziDf9rLUkgb9rY5h6_x5ItQ18AJ7XAu6ZRxGdP6RP44UtLBIWE8Shn9Myda_qe038ScdINF013iKKpW84v43noiK9dpRpz3GqhxT5bnqtIyyJe7zdM80aWsGzRPK3iKSM1EBOtHgurvguHe4tQ5Wau04otO8xYUuSno4yPndOuk3Ryp1HgRJ2769OyNPJXagv7veglfYO5EY2b5wrZuU8tsnMfy0lJj8pmnw8MgivEpS5lwrTtajO1T4KEwOU6WxJGN5rArxyLFZrbxdlzsfykuzOBs9ns7CBnvEiozxgr8lpPvMZDycnlQ4PkPqk7c1vGFK5Qa39BbeJDnDyAYcxd_p4xjfOONBKHzHSktZXHMXnppXzxfhAh9Nwf_bX2PSztIeBTgW54vDzMtiQRTFpBEIzEKywXeyHJ6P-D7RrITwZpRmkbEp1ZBq5_nuoA",cRq: {ru: \'aHR0cHM6Ly93d3cuZG9vcmRhc2guY29tLw==\',ra: \'TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTFfNikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzU5LjAuMzA3MS44NiBTYWZhcmkvNTM3LjM2\',rm: \'R0VU\',d: \'U4UZeO5kXKz6wvnM7ZvNiRldadESuA5j9BBSa1hRk9JmnHgtgrF29dLZmIlY+EBo89k+hJElCuGatsXbW7YKvO9ZJYTFnAstaeNaPXb984R0DGiZY1oYGPVbp7qpdVGbAWKswyqDZgc45oWZ2vsIZO4W+XjhOUPf9GylIqsi/2yWr11g/eNMDaCZijgFPozXWvmHhDDEQTBtS1tkch6xpSWTs2/jEw4Hz4ZgOQDYi65BpEHl93NsXB3SojAYgEdyGf4vpvvaCd2L5xcR+CISfdkhJMmoVUuOOPdkLlAVlv2BDgR+60pQQ0KYUON4QZ50SpmRYsiCwypJH8shg0PiVCp0fviyPJV4tWobCkmEN/0QJLBY6sPENed7clrxICXwsNSRR2HVjC19R/uDD6lzOXywsipDBVed8iKagVZHv44hwdTMDd+bnBnpk/dNCdSfxqdJPNsP7KZjuA63qqT0saru/ChFdOE1MjN9YpbyjCexlgHCgPGyLBv/ayWiHiY+it5Xk39ijlMxzprILFNiT/pR9nXvuvW5hnMMXbeHl0k=\',t: \'MTY5OTQxMjMzOS44OTEwMDA=\',cT: Math.floor(Date.now() / 1000),m: \'L3ZkQAdEKOHUaX9iur1Sr9sbx7oS0LL6gGNjkUNombQ=\',i1: \'T5qgz3Bud5TzogubxjP0SQ==\',i2: \'UKTuwcBi0QeRKvntURnRsw==\',zh: \'RiyS0sWHUt5On9P85BVgc8YSgvSVHcs0aHcZp4WMEhU=\',uh: \'75ubduWSuOViar7Qqb1rEZiYXMGez0Iqfl4zN15H62A=\',hh: \'1UmR1vv1MybJBf3wFVWgLp34ob+busJ/ICsY5lG4l+0=\',}};var cpo = document.createElement(\'script\');cpo.src = \'/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=822a8eb44b584aa8\';window._cf_chl_opt.cOgUHash = location.hash === \'\' && location.href.indexOf(\'#\') !== -1 ? \'#\' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === \'\' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf(\'?\') !== -1 ? \'?\' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\\/?__cf_chl_rt_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName(\'head\')[0].appendChild(cpo);}());</script><script defer src="https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317" integrity="sha512-euoFGowhlaLqXsPWQ48qSkBSCFs3DPRyiwVu3FjR96cMPx+Fr+gpWRhIafcHwqwCqWS42RZhIudOvEI+Ckf6MA==" data-cf-beacon=\'{"rayId":"822a8eb44b584aa8","b":1,"version":"2023.10.0","token":"f335665aff0f4b738e4fdf1c07bfea53"}\' crossorigin="anonymous"></script>\n</body></html>'
b'<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><link href="/cdn-cgi/styles/challenges.css" rel="stylesheet"><meta http-equiv="refresh" content="375"></head><body class="no-js"><div class="main-wrapper" role="main"><div class="main-content"><noscript><div id="challenge-error-title"><div class="h2"><span class="icon-wrapper"><div class="heading-icon warning-icon"></div></span><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: \'2\',cZone: "www.doordash.com",cType: \'managed\',cNounce: \'39291\',cRay: \'822a8eb44b584aa8\',cHash: \'602f515f9358e2a\',cUPMDTk: "\\/?__cf_chl_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",cFPWv: \'g\',cTTimeMs: \'1000\',cMTimeMs: \'375000\',cTplV: 5,cTplB: \'cf\',cK: "visitor-time",fa: "\\/?__cf_chl_f_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",md: "z94vGId17dyb7Qd4Pdg.yRRwFM3QgdMOnhliB3UVCE4-1699412339-0-AaTCUhoD_0HKzcTKXdVb5pyCYowLHIPqifSkfKGwdhw-n7lz2PeEZ3bXCjsCG-k69kS0mk81vAvm10qnqBd1AeJmcG_ieZIT3JEUfpybxy9HfL51pPbVNfP6JEA3mqStLsPNeQudDiTi4VqLtfhdZuy1n7M8seXhXOampBeCdFb9rRw2_dOhDPI94gyW1tz43HDVpHlBbnSmiIb83CKyXtj4csJJCwA5255J0iJ7z_wwEbALIePHyQySJzI4ALIrnaogKqiKnUGrNk3Vk8PQ1Fem47zS0dEfXGrOq-xpecw5n--3Ux_usuYWLnWqRQAo0irhCmp82_jQfPSMATqF37T5Q8d2NHcOvocyWHgUdaXFTgUzDtdG2syIn_L8xj_bsixzQ1__QjZB1UM3fTHSJr-paHjVcv6_vMg5astW54lq4fwi-H5oi0ayXJ5HRELLU7LfShV9r-FbDONoC9KTCyd7rdswFPV9N-vsr8FcD-TIvmE2yC9-CkbGHwG0NtipfbcXotkga__2AUe4VomsJ7bedr79QFt2koxU7y8rW9jPqneOkYbwUU0fOQ3mK7BUUvEvlUrAp9VUR7Fvl4te2LA1LbuP_c-Di6A98DBr0kN36KVP9ugt8JdGuIlHRKrL8pMMk5epOcCxO9CSguigSLPgqTjuyjQuf5Stk_tY1TOCBc0SiAy_6vMk63KS-jPUyKGdS-SNSdOt_lR6vpRGc034f42O7V8mrlWYFTb280UVQnl4ZlAbIh2YEm1h2dNcIIuIdO5EG71z1-woFwTOaKcKtIMIyVhCPNwEcLMvBPIbh4yQ2NIeFDh85GCq7XW0piPYVdUTo1991vnvpBEsThmdPX2VDJ21OTIptWDKITX1rjO_GNy0zS8dxnUPCu24kgYapIzoKVF27bNVWVH-RGkP9Lcvk0YCyWEUCU8HN6Zys_Jt5z4OyBoUXkOqN-QFmzBZ6mgFcIvJh-aUt1GozL0Zk8M0QzNYphNnL4KU0zGC2xlbEqAoPTxI8oIe9pN67NNcfdozh4u9sm1k6OH5j6C75PkWEFUiCE7Bs2EgimQKrcyz6vubFNbk7k4S_tYKtaIc-8Q2Oi3hTnFnjGApZdKN-B5xM3VUfkwcohWIaM-DZah5Kxm6ZfSHS607IdL5aCLX6r5Gg7gSTJivv0bpe0kpqVSEEfW1Q8SdkqWF_lg1NfKloT8cv-i0DB0pUbmvprFTOKY6sa7Xmi-bAu2L8XtQA8vC71LLlyK4CCM_HcyDG86-qpgWOhOCYJKGxUJ9IA4snTM7Fhp9Jr_gK7fkh3w-uky88DQ-lKXWsN8kJ0s8V9F0iHe2-mEZN0rG4-AtdBBofvKiKLMUodH7TExsYbN8tL0ZZoagzkZ4vqXXeNsP3G8BHUvDfApfMjGLBz9Jy3gHf8z-sp7S5byLmaf7KEg6zalyy4_vZ0QXEXxOPba8Bx1Qeh1LJN_yly2hMuoDVa6dREtSqJBwpMM5DmD6lfzPseOZdlhzL6d9m7_EkOCOxRTlORd92lLY1QP-ODJRixHtW4XTwQ0oL56k7kJWxVhb-RZLbOT0qgJNHPdK38B_LAtuXgTBt2IMaMbUMrjiTJBekxIyGUULAFDU33HJ73f8wRDL14Ts6sOv8NMHd78tyAaldEJHXuPCIIbC5GJC1WTli1osmbE1DmrHsMDoEuuRJLKPC5SSbQmHMG7tkFkCs514gMZd84PJ8cSGLX9-kt9bPBlJZB9tLhsWBg5ongw-2h7-AtQGUe0RVVP_mxCV1f07B9JfCegiJ33fspvoIEmExYQzKK1vMjAhfazIHXfCd4KeoKD2ZArd9jurDIv7Xg1hXV2tm8YmV552rQbWssKylnDLJnIAwfjSm1WYUbu1k4Aabk0ULycxv-qn3L89lb_gZHXMw3aMDKEvyXtpndKBpEREIHi4lvrUrYtiCYxkPxo6g8EC_28a4sR00peuzRhQRCGOPa80zcAJkmoEjguJanofUaVq8aa75sfGKOyDpRBvb7wVOcL8gPLv9nbqgny-_8Rk0sP3lDADkJpx5DzjgW3v7uWWgVHYbXT98l2yn9wn-vNyotm_lpBvuxMBC2vHNGDhKDEs98qf2YBYPAOMQ_rfF1GyjPc6U7Lfai0OwtxeywMEsygzZGVwKf2yR64YnC0d5YqQIRsgKbXKx7WkgV0qTiLUBuqwNNP0sgsQnsfyvkAuH-qc-zjyQXBmubduTH-ZZV44EZqEss_rC69q6RTi3NW8fCPZmePTLj8ocHUyKhi_2TrhxqCMTA6eXRe1J9K2Ffxmk48M7iEhFAjZziDf9rLUkgb9rY5h6_x5ItQ18AJ7XAu6ZRxGdP6RP44UtLBIWE8Shn9Myda_qe038ScdINF013iKKpW84v43noiK9dpRpz3GqhxT5bnqtIyyJe7zdM80aWsGzRPK3iKSM1EBOtHgurvguHe4tQ5Wau04otO8xYUuSno4yPndOuk3Ryp1HgRJ2769OyNPJXagv7veglfYO5EY2b5wrZuU8tsnMfy0lJj8pmnw8MgivEpS5lwrTtajO1T4KEwOU6WxJGN5rArxyLFZrbxdlzsfykuzOBs9ns7CBnvEiozxgr8lpPvMZDycnlQ4PkPqk7c1vGFK5Qa39BbeJDnDyAYcxd_p4xjfOONBKHzHSktZXHMXnppXzxfhAh9Nwf_bX2PSztIeBTgW54vDzMtiQRTFpBEIzEKywXeyHJ6P-D7RrITwZpRmkbEp1ZBq5_nuoA",cRq: {ru: \'aHR0cHM6Ly93d3cuZG9vcmRhc2guY29tLw==\',ra: \'TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTFfNikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzU5LjAuMzA3MS44NiBTYWZhcmkvNTM3LjM2\',rm: \'R0VU\',d: \'U4UZeO5kXKz6wvnM7ZvNiRldadESuA5j9BBSa1hRk9JmnHgtgrF29dLZmIlY+EBo89k+hJElCuGatsXbW7YKvO9ZJYTFnAstaeNaPXb984R0DGiZY1oYGPVbp7qpdVGbAWKswyqDZgc45oWZ2vsIZO4W+XjhOUPf9GylIqsi/2yWr11g/eNMDaCZijgFPozXWvmHhDDEQTBtS1tkch6xpSWTs2/jEw4Hz4ZgOQDYi65BpEHl93NsXB3SojAYgEdyGf4vpvvaCd2L5xcR+CISfdkhJMmoVUuOOPdkLlAVlv2BDgR+60pQQ0KYUON4QZ50SpmRYsiCwypJH8shg0PiVCp0fviyPJV4tWobCkmEN/0QJLBY6sPENed7clrxICXwsNSRR2HVjC19R/uDD6lzOXywsipDBVed8iKagVZHv44hwdTMDd+bnBnpk/dNCdSfxqdJPNsP7KZjuA63qqT0saru/ChFdOE1MjN9YpbyjCexlgHCgPGyLBv/ayWiHiY+it5Xk39ijlMxzprILFNiT/pR9nXvuvW5hnMMXbeHl0k=\',t: \'MTY5OTQxMjMzOS44OTEwMDA=\',cT: Math.floor(Date.now() / 1000),m: \'L3ZkQAdEKOHUaX9iur1Sr9sbx7oS0LL6gGNjkUNombQ=\',i1: \'T5qgz3Bud5TzogubxjP0SQ==\',i2: \'UKTuwcBi0QeRKvntURnRsw==\',zh: \'RiyS0sWHUt5On9P85BVgc8YSgvSVHcs0aHcZp4WMEhU=\',uh: \'75ubduWSuOViar7Qqb1rEZiYXMGez0Iqfl4zN15H62A=\',hh: \'1UmR1vv1MybJBf3wFVWgLp34ob+busJ/ICsY5lG4l+0=\',}};var cpo = document.createElement(\'script\');cpo.src = \'/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=822a8eb44b584aa8\';window._cf_chl_opt.cOgUHash = location.hash === \'\' && location.href.indexOf(\'#\') !== -1 ? \'#\' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === \'\' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf(\'?\') !== -1 ? \'?\' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\\/?__cf_chl_rt_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName(\'head\')[0].appendChild(cpo);}());</script><script defer src="https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317" integrity="sha512-euoFGowhlaLqXsPWQ48qSkBSCFs3DPRyiwVu3FjR96cMPx+Fr+gpWRhIafcHwqwCqWS42RZhIudOvEI+Ckf6MA==" data-cf-beacon=\'{"rayId":"822a8eb44b584aa8","b":1,"version":"2023.10.0","token":"f335665aff0f4b738e4fdf1c07bfea53"}\' crossorigin="anonymous"></script>\n</body></html>'
This is normal bot script which can get through normal scraping. Could you let me know example website list so that I can have a test? Thank you.
The text was updated successfully, but these errors were encountered:
No branches or pull requests
I tested this module to scrap https://www.doordash.com/ which implement the cloudfare bot.
Here is the output.
b'<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><link href="/cdn-cgi/styles/challenges.css" rel="stylesheet"><meta http-equiv="refresh" content="375"></head><body class="no-js"><div class="main-wrapper" role="main"><div class="main-content"><noscript><div id="challenge-error-title"><div class="h2"><span class="icon-wrapper"><div class="heading-icon warning-icon"></div></span><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: \'2\',cZone: "www.doordash.com",cType: \'managed\',cNounce: \'39291\',cRay: \'822a8eb44b584aa8\',cHash: \'602f515f9358e2a\',cUPMDTk: "\\/?__cf_chl_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",cFPWv: \'g\',cTTimeMs: \'1000\',cMTimeMs: \'375000\',cTplV: 5,cTplB: \'cf\',cK: "visitor-time",fa: "\\/?__cf_chl_f_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs",md: "z94vGId17dyb7Qd4Pdg.yRRwFM3QgdMOnhliB3UVCE4-1699412339-0-AaTCUhoD_0HKzcTKXdVb5pyCYowLHIPqifSkfKGwdhw-n7lz2PeEZ3bXCjsCG-k69kS0mk81vAvm10qnqBd1AeJmcG_ieZIT3JEUfpybxy9HfL51pPbVNfP6JEA3mqStLsPNeQudDiTi4VqLtfhdZuy1n7M8seXhXOampBeCdFb9rRw2_dOhDPI94gyW1tz43HDVpHlBbnSmiIb83CKyXtj4csJJCwA5255J0iJ7z_wwEbALIePHyQySJzI4ALIrnaogKqiKnUGrNk3Vk8PQ1Fem47zS0dEfXGrOq-xpecw5n--3Ux_usuYWLnWqRQAo0irhCmp82_jQfPSMATqF37T5Q8d2NHcOvocyWHgUdaXFTgUzDtdG2syIn_L8xj_bsixzQ1__QjZB1UM3fTHSJr-paHjVcv6_vMg5astW54lq4fwi-H5oi0ayXJ5HRELLU7LfShV9r-FbDONoC9KTCyd7rdswFPV9N-vsr8FcD-TIvmE2yC9-CkbGHwG0NtipfbcXotkga__2AUe4VomsJ7bedr79QFt2koxU7y8rW9jPqneOkYbwUU0fOQ3mK7BUUvEvlUrAp9VUR7Fvl4te2LA1LbuP_c-Di6A98DBr0kN36KVP9ugt8JdGuIlHRKrL8pMMk5epOcCxO9CSguigSLPgqTjuyjQuf5Stk_tY1TOCBc0SiAy_6vMk63KS-jPUyKGdS-SNSdOt_lR6vpRGc034f42O7V8mrlWYFTb280UVQnl4ZlAbIh2YEm1h2dNcIIuIdO5EG71z1-woFwTOaKcKtIMIyVhCPNwEcLMvBPIbh4yQ2NIeFDh85GCq7XW0piPYVdUTo1991vnvpBEsThmdPX2VDJ21OTIptWDKITX1rjO_GNy0zS8dxnUPCu24kgYapIzoKVF27bNVWVH-RGkP9Lcvk0YCyWEUCU8HN6Zys_Jt5z4OyBoUXkOqN-QFmzBZ6mgFcIvJh-aUt1GozL0Zk8M0QzNYphNnL4KU0zGC2xlbEqAoPTxI8oIe9pN67NNcfdozh4u9sm1k6OH5j6C75PkWEFUiCE7Bs2EgimQKrcyz6vubFNbk7k4S_tYKtaIc-8Q2Oi3hTnFnjGApZdKN-B5xM3VUfkwcohWIaM-DZah5Kxm6ZfSHS607IdL5aCLX6r5Gg7gSTJivv0bpe0kpqVSEEfW1Q8SdkqWF_lg1NfKloT8cv-i0DB0pUbmvprFTOKY6sa7Xmi-bAu2L8XtQA8vC71LLlyK4CCM_HcyDG86-qpgWOhOCYJKGxUJ9IA4snTM7Fhp9Jr_gK7fkh3w-uky88DQ-lKXWsN8kJ0s8V9F0iHe2-mEZN0rG4-AtdBBofvKiKLMUodH7TExsYbN8tL0ZZoagzkZ4vqXXeNsP3G8BHUvDfApfMjGLBz9Jy3gHf8z-sp7S5byLmaf7KEg6zalyy4_vZ0QXEXxOPba8Bx1Qeh1LJN_yly2hMuoDVa6dREtSqJBwpMM5DmD6lfzPseOZdlhzL6d9m7_EkOCOxRTlORd92lLY1QP-ODJRixHtW4XTwQ0oL56k7kJWxVhb-RZLbOT0qgJNHPdK38B_LAtuXgTBt2IMaMbUMrjiTJBekxIyGUULAFDU33HJ73f8wRDL14Ts6sOv8NMHd78tyAaldEJHXuPCIIbC5GJC1WTli1osmbE1DmrHsMDoEuuRJLKPC5SSbQmHMG7tkFkCs514gMZd84PJ8cSGLX9-kt9bPBlJZB9tLhsWBg5ongw-2h7-AtQGUe0RVVP_mxCV1f07B9JfCegiJ33fspvoIEmExYQzKK1vMjAhfazIHXfCd4KeoKD2ZArd9jurDIv7Xg1hXV2tm8YmV552rQbWssKylnDLJnIAwfjSm1WYUbu1k4Aabk0ULycxv-qn3L89lb_gZHXMw3aMDKEvyXtpndKBpEREIHi4lvrUrYtiCYxkPxo6g8EC_28a4sR00peuzRhQRCGOPa80zcAJkmoEjguJanofUaVq8aa75sfGKOyDpRBvb7wVOcL8gPLv9nbqgny-_8Rk0sP3lDADkJpx5DzjgW3v7uWWgVHYbXT98l2yn9wn-vNyotm_lpBvuxMBC2vHNGDhKDEs98qf2YBYPAOMQ_rfF1GyjPc6U7Lfai0OwtxeywMEsygzZGVwKf2yR64YnC0d5YqQIRsgKbXKx7WkgV0qTiLUBuqwNNP0sgsQnsfyvkAuH-qc-zjyQXBmubduTH-ZZV44EZqEss_rC69q6RTi3NW8fCPZmePTLj8ocHUyKhi_2TrhxqCMTA6eXRe1J9K2Ffxmk48M7iEhFAjZziDf9rLUkgb9rY5h6_x5ItQ18AJ7XAu6ZRxGdP6RP44UtLBIWE8Shn9Myda_qe038ScdINF013iKKpW84v43noiK9dpRpz3GqhxT5bnqtIyyJe7zdM80aWsGzRPK3iKSM1EBOtHgurvguHe4tQ5Wau04otO8xYUuSno4yPndOuk3Ryp1HgRJ2769OyNPJXagv7veglfYO5EY2b5wrZuU8tsnMfy0lJj8pmnw8MgivEpS5lwrTtajO1T4KEwOU6WxJGN5rArxyLFZrbxdlzsfykuzOBs9ns7CBnvEiozxgr8lpPvMZDycnlQ4PkPqk7c1vGFK5Qa39BbeJDnDyAYcxd_p4xjfOONBKHzHSktZXHMXnppXzxfhAh9Nwf_bX2PSztIeBTgW54vDzMtiQRTFpBEIzEKywXeyHJ6P-D7RrITwZpRmkbEp1ZBq5_nuoA",cRq: {ru: \'aHR0cHM6Ly93d3cuZG9vcmRhc2guY29tLw==\',ra: \'TW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTFfNikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzU5LjAuMzA3MS44NiBTYWZhcmkvNTM3LjM2\',rm: \'R0VU\',d: \'U4UZeO5kXKz6wvnM7ZvNiRldadESuA5j9BBSa1hRk9JmnHgtgrF29dLZmIlY+EBo89k+hJElCuGatsXbW7YKvO9ZJYTFnAstaeNaPXb984R0DGiZY1oYGPVbp7qpdVGbAWKswyqDZgc45oWZ2vsIZO4W+XjhOUPf9GylIqsi/2yWr11g/eNMDaCZijgFPozXWvmHhDDEQTBtS1tkch6xpSWTs2/jEw4Hz4ZgOQDYi65BpEHl93NsXB3SojAYgEdyGf4vpvvaCd2L5xcR+CISfdkhJMmoVUuOOPdkLlAVlv2BDgR+60pQQ0KYUON4QZ50SpmRYsiCwypJH8shg0PiVCp0fviyPJV4tWobCkmEN/0QJLBY6sPENed7clrxICXwsNSRR2HVjC19R/uDD6lzOXywsipDBVed8iKagVZHv44hwdTMDd+bnBnpk/dNCdSfxqdJPNsP7KZjuA63qqT0saru/ChFdOE1MjN9YpbyjCexlgHCgPGyLBv/ayWiHiY+it5Xk39ijlMxzprILFNiT/pR9nXvuvW5hnMMXbeHl0k=\',t: \'MTY5OTQxMjMzOS44OTEwMDA=\',cT: Math.floor(Date.now() / 1000),m: \'L3ZkQAdEKOHUaX9iur1Sr9sbx7oS0LL6gGNjkUNombQ=\',i1: \'T5qgz3Bud5TzogubxjP0SQ==\',i2: \'UKTuwcBi0QeRKvntURnRsw==\',zh: \'RiyS0sWHUt5On9P85BVgc8YSgvSVHcs0aHcZp4WMEhU=\',uh: \'75ubduWSuOViar7Qqb1rEZiYXMGez0Iqfl4zN15H62A=\',hh: \'1UmR1vv1MybJBf3wFVWgLp34ob+busJ/ICsY5lG4l+0=\',}};var cpo = document.createElement(\'script\');cpo.src = \'/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=822a8eb44b584aa8\';window._cf_chl_opt.cOgUHash = location.hash === \'\' && location.href.indexOf(\'#\') !== -1 ? \'#\' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === \'\' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf(\'?\') !== -1 ? \'?\' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\\/?__cf_chl_rt_tk=sZ7nFSLwBdIDvz8tEP__CYEkNCuLa7N3qmF1DGukFAI-1699412339-0-gaNycGzNCvs" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName(\'head\')[0].appendChild(cpo);}());</script><script defer src="https://static.cloudflareinsights.com/beacon.min.js/v84a3a4012de94ce1a686ba8c167c359c1696973893317" integrity="sha512-euoFGowhlaLqXsPWQ48qSkBSCFs3DPRyiwVu3FjR96cMPx+Fr+gpWRhIafcHwqwCqWS42RZhIudOvEI+Ckf6MA==" data-cf-beacon=\'{"rayId":"822a8eb44b584aa8","b":1,"version":"2023.10.0","token":"f335665aff0f4b738e4fdf1c07bfea53"}\' crossorigin="anonymous"></script>\n</body></html>'This is normal bot script which can get through normal scraping.
Could you let me know example website list so that I can have a test?
Thank you.
The text was updated successfully, but these errors were encountered: