Rolling cURL: PHP²¢·¢×î¼Ñʵ¼ù
ÔÎÄ£ºhttp://blogread.cn/it/article/5501
ÔÚʵ¼ÊÏîÄ¿»òÕß×Ô¼º±àдС¹¤¾ß(±ÈÈçÐÂΞۺÏ,ÉÌÆ·¼Û¸ñ¼à¿Ø,±È¼Û)µÄ¹ý³ÌÖÐ, ͨ³£ÐèÒª´ÓµÚ3·½ÍøÕ¾»òÕßAPI½Ó¿Ú»ñÈ¡Êý¾Ý, ÔÚÐèÒª´¦Àí1¸öURL¶ÓÁÐʱ, ΪÁËÌá¸ßÐÔÄÜ, ¿ÉÒÔ²ÉÓÃcURLÌṩµÄcurl_multi_*×庯ÊýʵÏÖ¼òµ¥µÄ²¢·¢.
±¾ÎĽ«Ì½ÌÖÁ½ÖÖ¾ßÌåµÄʵÏÖ·½·¨, ²¢¶Ô²»Í¬µÄ·½·¨×ö¼òµ¥µÄÐÔÄܶԱÈ.
1. ¾µäcURL²¢·¢»úÖÆ¼°Æä´æÔÚµÄÎÊÌâ
¾µäµÄcURLʵÏÖ»úÖÆÔÚÍøÉϺÜÈÝÒ×ÕÒµ½, ±ÈÈç²Î¿¼PHPÔÚÏßÊÖ²áµÄÈçÏÂʵÏÖ·½Ê½:
function classic_curl($urls, $delay) { $queue = curl_multi_init(); $map = array(); foreach ($urls as $url) { // create cURL resources $ch = curl_init(); // set URL and other appropriate options curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_TIMEOUT, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_NOSIGNAL, true); // add handle curl_multi_add_handle($queue, $ch); $map[$url] = $ch; } $active = null; // execute the handles do { $mrc = curl_multi_exec($queue, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); while ($active > 0 && $mrc == CURLM_OK) { if (curl_multi_select($queue, 0.5) != -1) { do { $mrc = curl_multi_exec($queue, $active); } while ($mrc == CURLM_CALL_MULTI_PERFORM); } } $responses = array(); foreach ($map as $url=>$ch) { $responses[$url] = callback(curl_multi_getcontent($ch), $delay); curl_multi_remove_handle($queue, $ch); curl_close($ch); } curl_multi_close($queue); return $responses; }
Ê×ÏȽ«ËùÓеÄURLѹÈë²¢·¢¶ÓÁÐ, È»ºóÖ´Ðв¢·¢¹ý³Ì, µÈ´ýËùÓÐÇëÇó½ÓÊÕÍêÖ®ºó½øÐÐÊý¾ÝµÄ½âÎöµÈºóÐø´¦Àí. ÔÚʵ¼ÊµÄ´¦Àí¹ý³ÌÖÐ, ÊÜÍøÂç´«ÊäµÄÓ°Ïì, ²¿·ÖURLµÄÄÚÈÝ»áÓÅÏÈÓÚÆäËûURL·µ»Ø, µ«ÊǾµäcURL²¢·¢±ØÐëµÈ´ý×îÂýµÄÄǸöURL·µ»ØÖ®ºó²Å¿ªÊ¼´¦Àí, µÈ´ýÒ²¾ÍÒâζ×ÅCPUµÄ¿ÕÏкÍÀË·Ñ. Èç¹ûURL¶ÓÁкܶÌ, ÕâÖÖ¿ÕÏкÍÀË·Ñ»¹´¦ÔڿɽÓÊܵķ¶Î§, µ«Èç¹û¶ÓÁкܳ¤, ÕâÖֵȴýºÍÀ˷ѽ«±äµÃ²»¿É½ÓÊÜ.
2. ¸Ä½øµÄRolling cURL²¢·¢·½Ê½
×Ðϸ·ÖÎö²»ÄÑ·¢ÏÖ¾µäcURL²¢·¢»¹´æÔÚÓÅ»¯µÄ¿Õ¼ä, ÓÅ»¯µÄ·½Ê½Ê±µ±Ä³¸öURLÇëÇóÍê±ÏÖ®ºó¾¡¿ÉÄÜ¿ìµÄÈ¥´¦ÀíËü, ±ß´¦Àí±ßµÈ´ýÆäËûµÄURL·µ»Ø, ¶ø²»ÊǵȴýÄǸö×îÂýµÄ½Ó¿Ú·µ»ØÖ®ºó²Å¿ªÊ¼´¦ÀíµÈ¹¤×÷, ´Ó¶ø±ÜÃâCPUµÄ¿ÕÏкÍÀË·Ñ. Ïл°²»¶à˵, ÏÂÃæÌùÉϾßÌåµÄʵÏÖ:
function rolling_curl($urls, $delay) { $queue = curl_multi_init(); $map = array(); foreach ($urls as $url) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_TIMEOUT, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, 0); curl_setopt($ch, CURLOPT_NOSIGNAL, true); curl_multi_add_handle($queue, $ch); $map[(string) $ch] = $url; } $responses = array(); do { while (($code = curl_multi_exec($queue, $active)) == CURLM_CALL_MULTI_PERFORM) ; if ($code != CURLM_OK) { break; } // a request was just completed -- find out which one while ($done = curl_multi_info_read($queue)) { // get the info and content returned on the request $info = curl_getinfo($done['handle']); $error = curl_error($done['handle']); $results = callback(curl_multi_getcontent($done['handle']), $delay); $responses[$map[(string) $done['handle']]] = compact('info', 'error', 'result'); // remove the curl handle that just completed curl_multi_remove_handle($queue, $done['handle']); curl_close($done['handle']); } // Block for data in / output; error handling is done by curl_multi_exec if ($active > 0) { curl_multi_select($queue, 0.5); } } while ($active); curl_multi_close($queue); return $responses; }3. Á½ÖÖ²¢·¢ÊµÏÖµÄÐÔÄܶԱÈ
¸Ä½øÇ°ºóµÄÐÔÄܶԱÈÊÔÑéÔÚLINUXÖ÷»úÉϽøÐÐ, ²âÊÔʱʹÓõIJ¢·¢¶ÓÁÐÈçÏÂ:
http://item.taobao.com/item.htm?id=14392877692
http://item.taobao.com/item.htm?id=16231676302
http://item.taobao.com/item.htm?id=17037160462
http://item.taobao.com/item.htm?id=5522416710
http://item.taobao.com/item.htm?id=16551116403
http://item.taobao.com/item.htm?id=14088310973
¼òҪ˵Ã÷ÏÂʵÑéÉè¼ÆµÄÔÔòºÍÐÔÄܲâÊÔ½á¹ûµÄ¸ñʽ: Ϊ±£Ö¤½á¹ûµÄ¿É¿¿, ÿ×éʵÑéÖØ¸´20´Î, ÔÚµ¥´ÎʵÑéÖÐ, ¸ø¶¨ÏàͬµÄ½Ó¿ÚURL¼¯ºÏ, ·Ö±ð²âÁ¿Classic(Ö¸¾µäµÄ²¢·¢»úÖÆ)ºÍRolling(Ö¸¸Ä½øºóµÄ²¢·¢»úÖÆ)Á½ÖÖ²¢·¢»úÖÆµÄºÄʱ(ÃëΪµ¥Î»), ºÄʱ¶ÌÕßʤ³ö(Winner), ²¢¼ÆËã½ÚÊ¡µÄʱ¼ä(Excellence, ÃëΪµ¥Î»)ÒÔ¼°ÐÔÄÜÌáÉý±ÈÀý(Excel. %). ΪÁ˾¡Á¿Ìù½üÕæÊµµÄÇëÇó¶øÓÖ±£³ÖʵÑéµÄ¼òµ¥, ÔÚ¶Ô·µ»Ø½á¹ûµÄ´¦ÀíÉÏÖ»ÊÇ×öÁ˼òµ¥µÄÕýÔò±í´ïʽƥÅä, ¶øÃ»ÓнøÐÐÆäËû¸´ÔӵIJÙ×÷. ÁíÍâ, ΪÁËÈ·¶¨½á¹û´¦Àí»Øµ÷¶ÔÐÔÄܶԱȲâÊÔ½á¹ûµÄÓ°Ïì, ¿ÉÒÔʹÓÃusleepÄ£ÄâÏÖʵÖбȽϸºÔðµÄÊý¾Ý´¦ÀíÂß¼(ÈçÌáÈ¡, ·Ö´Ê, дÈëÎļþ»òÊý¾Ý¿âµÈ).
ÐÔÄܲâÊÔÖÐÓõ½µÄ»Øµ÷º¯ÊýΪ:
function callback($data, $delay) { preg_match_all("/(.+)<\/h3>/iU", $data, $matches); usleep($delay); return compact('data', 'matches'); }Êý¾Ý´¦Àí»Øµ÷ÎÞÑÓ³Ùʱ: Rolling CurlÂÔÓÅ, µ«ÐÔÄÜÌáÉýЧ¹û²»Ã÷ÏÔ.
------------------------------------------------------------------------------------------------
Delay: 0 micro seconds, equals to 0 milli seconds
------------------------------------------------------------------------------------------------
Counter Classic Rolling Winner Excellence Excel. %
------------------------------------------------------------------------------------------------
1 0.1193 0.0390 Rolling 0.0803 67.31%
2 0.0556 0.0477 Rolling 0.0079 14.21%
3 0.0461 0.0588 Classic -0.0127 -21.6%
4 0.0464 0.0385 Rolling 0.0079 17.03%
5 0.0534 0.0448 Rolling 0.0086 16.1%
6 0.0540 0.0714 Classic -0.0174 -24.37%
7 0.0386 0.0416 Classic -0.0030 -7.21%
8 0.0357 0.0398 Classic -0.0041 -10.3%
9 0.0437 0.0442 Classic -0.0005 -1.13%
10 0.0319 0.0348 Classic -0.0029 -8.33%
11 0.0529 0.0430 Rolling 0.0099 18.71%
12 0.0503 0.0581 Classic -0.0078 -13.43%
13 0.0344 0.0225 Rolling 0.0119 34.59%
14 0.0397 0.0643 Classic -0.0246 -38.26%
15 0.0368 0.0489 Classic -0.0121 -24.74%
16 0.0502 0.0394 Rolling 0.0108 21.51%
17 0.0592 0.0383 Rolling 0.0209 35.3%
18 0.0302 0.0285 Rolling 0.0017 5.63%
19 0.0248 0.0553 Classic -0.0305 -55.15%
20 0.0137 0.0131 Rolling 0.0006 4.38%
------------------------------------------------------------------------------------------------
Average 0.0458 0.0436 Rolling 0.0022 4.8%
------------------------------------------------------------------------------------------------
Summary: Classic wins 10 times, while Rolling wins 10 times
Êý¾Ý´¦Àí»Øµ÷ÑÓ³Ù5ºÁÃë: Rolling CurlÍêʤ, ÐÔÄÜÌáÉý40%×óÓÒ.
------------------------------------------------------------------------------------------------
Delay: 5000 micro seconds, equals to 5 milli seconds
------------------------------------------------------------------------------------------------
Counter Classic Rolling Winner Excellence Excel. %
------------------------------------------------------------------------------------------------
1 0.0658 0.0352 Rolling 0.0306 46.5%
2 0.0728 0.0367 Rolling 0.0361 49.59%
3 0.0732 0.0387 Rolling 0.0345 47.13%
4 0.0783 0.0347 Rolling 0.0436 55.68%
5 0.0658 0.0286 Rolling 0.0372 56.53%
6 0.0687 0.0362 Rolling 0.0325 47.31%
7 0.0787 0.0337 Rolling 0.0450 57.18%
8 0.0676 0.0391 Rolling 0.0285 42.16%
9 0.0668 0.0351 Rolling 0.0317 47.46%
10 0.0603 0.0317 Rolling 0.0286 47.43%
11 0.0714 0.0350 Rolling 0.0364 50.98%
12 0.0627 0.0215 Rolling 0.0412 65.71%
13 0.0617 0.0401 Rolling 0.0216 35.01%
14 0.0721 0.0226 Rolling 0.0495 68.65%
15 0.0701 0.0428 Rolling 0.0273 38.94%
16 0.0674 0.0352 Rolling 0.0322 47.77%
17 0.0452 0.0425 Rolling 0.0027 5.97%
18 0.0596 0.0366 Rolling 0.0230 38.59%
19 0.0679 0.0480 Rolling 0.0199 29.31%
20 0.0657 0.0338 Rolling 0.0319 48.55%
------------------------------------------------------------------------------------------------
Average 0.0671 0.0354 Rolling 0.0317 47.24%
------------------------------------------------------------------------------------------------
Summary: Classic wins 0 times, while Rolling wins 20 times
ͨ¹ýÉÏÃæµÄÐÔÄܶԱÈ, ÔÚ´¦ÀíURL¶ÓÁв¢·¢µÄÓ¦Óó¡¾°ÖÐRolling cURLÓ¦¸ÃÊǸü¼ÓµÄÑ¡Ôñ, ²¢·¢Á¿·Ç³£´ó(1000+)ʱ, ¿ÉÒÔ¿ØÖƲ¢·¢¶ÓÁеÄ×î´ó³¤¶È, ±ÈÈç20, ÿµ±1¸öURL·µ»Ø²¢´¦ÀíÍê±ÏÖ®ºóÁ¢¼´¼ÓÈë1¸öÉÐδÇëÇóµÄURLµ½¶ÓÁÐÖÐ, ÕâÑùд³öÀ´µÄ´úÂë»á¸ü¼Ó½¡×³, ²»ÖÁÓÚ²¢·¢ÊýÌ«´ó¶ø¿¨ËÀ»ò±ÀÀ£.
ÍÆ¼öÐÅÏ¢
- ¡¾ÊÓÆµ²¥·Å¡¿JplayerÊÓÆµ²¥·ÅÆ÷µÄʹÓÃ
- memcacheÄÚ´æÔÀí
- Memcache¼¼Êõ·ÖÏí£º½éÉÜ¡¢Ê¹Óᢴ洢¡¢Ëã·¨¡¢ÓÅ»¯....
- php³£ÓÃÕýÔò±í´ïʽ
- phpÐÔÄܼà²âÄ£¿éXHProf
- ÈÃCI¿ò¼ÜÖ§³Öservice²ã
- ʹÓÃPHPÉú³É´øLOGOµÄ¸öÐÔ»¯¶þάÂëͼÏñ
- ¹ØÓÚCodeIgniterÄã¿ÉÄܲ»ÖªµÀµÄ5¸ö֪ʶµã
- Memcache ºÁÃë¼¶³¬Ê±¼°ÆäËû³£¼ûÎÊÌâ»ã×Ü
- [PHP±Ê¼Ç]PHPQueryÒ»¸ö´¦ÀíDOMµÄÀûÆ÷
ÈÈÃÅÐÅÏ¢
- nohup: redirecting stderr to stdou....
- ʹÓÃlog_formatΪNginx·þÎñÆ÷ÉèÖøüÏêϸµÄÈÕÖ¾¸ñʽ
- jquery easyUI--dataGrid-Json
- [Ô´´]·ÂGoogle Reader¡¢ÐÂÀË΢²©¡¢ÌÚѶ΢²©µ....
- ÀûÓÃKeepalived+mysql¹¹½¨¸ß¿ÉÓÃMySQLË«Ö÷×Ô¶....
- Nginx+keepalivedʵÏÖ¸ºÔؾùºâºÍË«»úÈȱ¸¸ß¿ÉÓÃ
- jqueryʵÏÖÒ³Ãæ¼ÓÔØ½ø¶ÈÌõ
- Rolling cURL: PHP²¢·¢×î¼Ñʵ¼ù
- codeigniter ·ÓÉÖÕ¼«ÓÅ»¯(url rewrite)
- linuxÏÂÉèÖÃsshÎÞÃÜÂëµÇ¼
×î½ü¸üÐÂ
- PHP»ñÈ¡Óû§µÄÕæÊµIP£¬²¢ÅжÏÊÇ·ñÄÚÍøIP
- PHP ´íÎóÈÕÖ¾ error_log
- ÀûÓÃbigpipe»úÖÆÊµÏÖÒ³ÃæÄ£¿éµÄÒì²½äÖȾ chunked¼¼Êõ
- php¿ØÖÆÎļþÏÂÔØËÙ¶È
- js + php ¶ÁÈ¡¡¢²¥·ÅÊÓÆµÁ÷ ¼æÈÝfirefox£¬c....
- ¡¾ÊÓÆµ²¥·Å¡¿JplayerÊÓÆµ²¥·ÅÆ÷µÄʹÓÃ
- UNICODE Óë UTF-8 µÄ¹ØÏµ
- memcacheÄÚ´æÔÀí
- Memcache¼¼Êõ·ÖÏí£º½éÉÜ¡¢Ê¹Óᢴ洢¡¢Ëã·¨¡¢ÓÅ»¯....
- phpʹÓÃmb_detect_encoding¼ì²â×Ö·û´®±àÂë
ÆÀÂÛ