웹 스크래핑을 위한 Python 도구

컴퓨터 과학에서 웹 스크래핑은 웹사이트에서 데이터를 추출하는 것을 의미합니다. 이 기술을 사용하여 웹의 비정형 데이터를 정형 데이터로 변환합니다.

Python3에서 가장 일반적인 웹 스크래핑 도구는 -

• Urllib2
• 요청
• 뷰티풀 수프
• Lxml
• 셀레늄
• 기계식 수프

Urllib2 − 이 도구는 Python과 함께 사전 설치되어 있습니다. 이 모듈은 URL을 추출하는 데 사용됩니다. urlopen() 함수를 사용하여 다른 프로토콜(FTP, HTTP 등)을 사용하여 URL을 가져옵니다.

예시 코드

urllib.request에서 가져오기 urlopenmy_html =urlopen("https://www.tutorialspoint.com/")print(my_html.read())

출력

b'\r\n \r\n
\r\n
\r\n
\r\n홈
\r\n
\r\n
\r\nQ/A\r\n
\r\n
\r\n도구
\ r\n
\r\n
\r\n코딩 그라운드 
 \ r\n
\r\n
\r\n시사
\r\n
\r\n
\r\nUPSC 노트
\r\n
\r\n
\r\n온라인 교사
\r\n
\r\n
\r\n….

요청 − 이 모듈은 사전 설치되어 있지 않으므로 명령 프롬프트에서 명령줄을 작성해야 합니다.요청 HTTP/1.1에 요청을 보냅니다.

pip 설치 요청

예시

요청 가져오기# get URLmy_req =requests.get('https://www.tutorialspoint.com/') print(my_req.encoding) print(my_req.status_code) print(my_req.elapsed) print(my_req.url) print(my_req.history)print(my_req.headers['콘텐츠 유형'])

출력

UTF-82000:00:00.205727https://www.tutorialspoint.com/[]text/html; charset=UTF-8

뷰티풀 수프 - 이것은 다른 파서에서 사용되는 파싱 라이브러리입니다. Python의 표준 라이브러리는 BeautifulSoup의 기본 파서를 제공합니다. HTML 페이지에서 데이터를 추출하는 데 사용되는 파서 트리를 구축합니다.

이 모듈을 설치하기 위해 명령 프롬프트에 명령줄을 작성합니다.

핍 설치 beautifulsoup4

예시

bs4 import BeautifulSoup# 가져오기 요청 가져오기 요청# get URLmy_req =requests.get("https://www.tutorialspoint.com/")my_data =my_req.textmy_soup =BeautifulSoup(my_data)for my_soup.find_all(' a'):print(my_link.get('href'))

출력

https://www.tutorialspoint.com/index.htmhttps://www.tutorialspoint.com/questions/index.phphttps://www.tutorialspoint.com/online_dev_tools.htmhttps://www.tutorialspoint.com /codingground.htmhttps://www.tutorialspoint.com/current_affairs/index.htmhttps://www.tutorialspoint.com/upsc_ias_exams.htmhttps://www.tutorialspoint.com/tutor_connect/index.phphttps://www.tutorialspoint .com/programming_examples/https://www.tutorialspoint.com/whiteboard.htmhttps://www.tutorialspoint.com/netmeeting.phphttps://www.tutorialspoint.com/articles/https://www.tutorialspoint.com /index.htmhttps://www.tutorialspoint.com/tutorialslibrary.htmhttps://www.tutorialspoint.com/videotutorials/index.htmhttps://store.tutorialspoint.comhttps://www.tutorialspoint.com/html_online_training/index .asphttps://www.tutorialspoint.com/css_online_training/index.asphttps://www.tutorialspoint.com/3d_animation_online_training/index.asphttps://www.tutorialspoint.com/swift_4_online_training/index.asphttps://www.tutorialspoint .com/blo ckchain_online_training/index.asphttps://www.tutorialspoint.com/reactjs_online_training/index.asphttps://www.tutorialspoint.com/tutorialslibrary.htmhttps://www.tutorialspoint.com/computer_fundamentals/index.htmhttps://www. tutorialspoint.com/compiler_design/index.htmhttps://www.tutorialspoint.com/operating_system/index.htmhttps://www.tutorialspoint.com/data_structures_algorithms/index.htmhttps://www.tutorialspoint.com/dbms/index. htmhttps://www.tutorialspoint.com/data_communication_computer_network/index.htmhttps://www.tutorialspoint.com/academic_tutorials.htmhttps://www.tutorialspoint.com/html/index.htmhttps://www.tutorialspoint.com/ CSS/index.htmhttps://www.tutorialspoint.com/javascript/index.htmhttps://www.tutorialspoint.com/php/index.htmhttps://www.tutorialspoint.com/angular4/index.htmhttps:// www.tutorialspoint.com/mysql/index.htmhttps://www.tutorialspoint.com/web_development_tutorials.htmhttps://www.tutorialspoint.com/cprogramming/index.htmhttps://www.tutorialspoint.com/cplusplus/ind ex.htmhttps://www.tutorialspoint.com/java8/index.htmhttps://www.tutorialspoint.com/python/index.htmhttps://www.tutorialspoint.com/scala/index.htmhttps://www. tutorialspoint.com/csharp/index.htmhttps://www.tutorialspoint.com/computer_programming_tutorials.htmhttps://www.tutorialspoint.com/java8/index.htmhttps://www.tutorialspoint.com/jdbc/index.htmhttps://www.tutorialspoint.com/servlets/index.htmhttps://www.tutorialspoint.com/spring/index.htmhttps://www.tutorialspoint.com/hibernate/index.htmhttps://www.tutorialspoint.com/ 스윙/index.htmhttps://www.tutorialspoint.com/java_technology_tutorials.htmhttps://www.tutorialspoint.com/android/index.htmhttps://www.tutorialspoint.com/swift/index.htmhttps://www. tutorialspoint.com/ios/index.htmhttps://www.tutorialspoint.com/kotlin/index.htmhttps://www.tutorialspoint.com/react_native/index.htmhttps://www.tutorialspoint.com/xamarin/index. htmhttps://www.tutorialspoint.com/mobile_development_tutorials.htmhttps://www.tutorialspoint.com/mongodb/index.htmhtt ps://www.tutorialspoint.com/plsql/index.htmhttps://www.tutorialspoint.com/sql/index.htmhttps://www.tutorialspoint.com/db2/index.htmhttps://www.tutorialspoint. com/mysql/index.htmhttps://www.tutorialspoint.com/memcached/index.htmhttps://www.tutorialspoint.com/database_tutorials.htmhttps://www.tutorialspoint.com/asp.net/index.htmhttps://www.tutorialspoint.com/entity_framework/index.htmhttps://www.tutorialspoint.com/vb.net/index.htmhttps://www.tutorialspoint.com/ms_project/index.htmhttps://www.tutorialspoint. com/excel/index.htmhttps://www.tutorialspoint.com/word/index.htmhttps://www.tutorialspoint.com/microsoft_technologies_tutorials.htmhttps://www.tutorialspoint.com/big_data_analytics/index.htmhttps:// www.tutorialspoint.com/hadoop/index.htmhttps://www.tutorialspoint.com/sas/index.htmhttps://www.tutorialspoint.com/qlikview/index.htmhttps://www.tutorialspoint.com/power_bi/ index.htmhttps://www.tutorialspoint.com/tableau/index.htmhttps://www.tutorialspoint.com/big_data_tutorials.htmhttps://ww w.tutorialspoint.com/tutorialslibrary.htmhttps://www.tutorialspoint.com/codingground.htmhttps://www.tutorialspoint.com/coding_platform_for_websites.htmhttps://www.tutorialspoint.com/developers_best_practices/index.htmhttps:// www.tutorialspoint.com/effective_resume_writing.htmhttps://www.tutorialspoint.com/computer_glossary.htmhttps://www.tutorialspoint.com/computer_whoiswho.htmhttps://www.tutorialspoint.com/questions_and_answers.htmhttps://www. tutorialspoint.com/multi_language_tutorials.htmhttps://itunes.apple.com/us/app/tutorials-point/id914891263?ls=1&mt=8https://play.google.com/store/apps/details?id=com. tutorialspoint.onlineviewerhttps://www.windowsphone.com/s?appid=91249671-7184-4ad6-8a5f-d11847946b09/about/index.htm/about/about_team.htm/about/about_careers.htm/about/about_privacy.htm/about/about_privacy. about/about_terms_of_use.htmhttps://www.tutorialspoint.com/articles/https://www.tutorialspoint.com/online_dev_tools.htmhttps://www.tutorialspoint.com/free_web_graphics.htmhttps://www.tutorialspoint.com /online_file_conversion.htmhttps://www.tutorialspoint.com/shared-tutorials.phphttps://www.tutorialspoint.com/netmeeting.phphttps://www.tutorialspoint.com/free_online_whiteboard.htmhttps://www.tutorialspoint.comhttps ://www.facebook.com/tutorialspointindiahttps://plus.google.com/u/0/+tutorialspointhttps://www.twitter.com/tutorialspointhttps://www.linkedin.com/company/tutorialspointhttps:// www.youtube.com/channel/UCVLbzhxVTiTLiVKeGV7WEBghttps://www.tutorialspoint.com/index.htm/about/about_privacy.htm#cookies/about/faq.htm/about/about_helping.htm/about/contact_us.htm

Lxml - 이것은 파싱 라이브러리, 고성능, 생산 품질의 HTML 및 XML 파싱 라이브러리입니다. 고품질의 최대 속도를 원한다면 이 라이브러리를 사용해야 합니다. 웹 사이트에서 데이터를 추출할 수 있는 많은 모듈이 있습니다.

설치를 위해 명령 프롬프트

pip 설치 lxml

예시

lxml에서 가져오기 etreemy_root_elem =etree.Element('html')etree.SubElement(my_root_elem, 'head')etree.SubElement(my_root_elem, 'title')etree.SubElement(my_root_elem, '본문')print(etree. tostring(my_root_elem, pretty_print =True).decode("utf-8"))

출력