技术问答Scrapy鎬庝箞澶勭悊骞跺彂闄愬埗鍜岄€熺巼闄愬埗 - 扬州沐宇科技

Scrapy鎻愪緵浜嗕互涓嬫柟寮忔潵澶勭悊骞跺彂闄愬埗鍜岄€熺巼闄愬埗锛?/p>

1. 骞跺彂闄愬埗锛歋crapy涓彲浠ヤ娇鐢–ONCURRENT_REQUESTS鍜孋ONCURRENT_REQUESTS_PER_DOMAIN閰嶇疆椤规潵鎺у埗骞跺彂璇锋眰鏁伴噺銆傚彲浠ラ€氳繃璁剧疆杩欎袱涓厤缃」鐨勫€兼潵闄愬埗骞跺彂璇锋眰鐨勬暟閲忥紝浠ラ槻姝?span class="in-link" data-id="53">鏈嶅姟鍣?/span>杩囪浇鎴栬灏佺銆傚€煎緱娉ㄦ剰鐨勬槸锛屽苟鍙戦檺鍒跺彲鑳戒細褰卞搷鐖彇閫熷害锛屽洜姝ら渶瑕佹牴鎹叿浣撴儏鍐佃繘琛岃皟鏁淬€?/p>

2. 閫熺巼闄愬埗锛歋crapy鎻愪緵浜咲OWNLOAD_DELAY鍜孉UTOTHROTTLE閰嶇疆椤规潵鎺у埗鐖彇閫熺巼銆侱OWNLOAD_DELAY鎸囧畾姣忎釜璇锋眰涔嬮棿鐨勫欢杩熸椂闂达紝浠ラ伩鍏嶅鏈嶅姟鍣ㄩ€犳垚澶ぇ鐨勫帇鍔涖€傝€孉UTOTHROTTLE鍙互鏍规嵁鏈嶅姟鍣ㄧ殑鍝嶅簲鏃堕棿鍔ㄦ€佽皟鏁磋姹傞€熺巼锛岄伩鍏嶈繃搴﹂绻佸湴鐖彇缃戠珯銆?/p>

閫氳繃鍚堢悊鍦拌缃繖浜涢厤缃」锛屽彲浠ユ湁鏁堝湴鎺у埗骞跺彂璇锋眰鏁伴噺鍜岀埇鍙栭€熺巼锛屼粠鑰屾彁楂樼埇鍙栨晥鐜囧苟閬垮厤瀵?span class="in-link" data-id="273">缃戠珯鏈嶅姟鍣?/span>閫犳垚杩囧ぇ鍘嬪姏銆?/p>