-
Notifications
You must be signed in to change notification settings - Fork 343
/
README.Rmd
80 lines (60 loc) · 2.72 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
```
# rvest <a href="https://rvest.tidyverse.org"><img src="man/figures/logo.png" align="right" height="138" alt="rvest website" /></a>
<!-- badges: start -->
[](https://cran.r-project.org/package=rvest)
[](https://github.com/tidyverse/rvest/actions/workflows/R-CMD-check.yaml)
[](https://app.codecov.io/gh/tidyverse/rvest?branch=main)
<!-- badges: end -->
## Overview
rvest helps you scrape (or harvest) data from web pages.
It is designed to work with [magrittr](https://github.com/tidyverse/magrittr) to make it easy to express common web scraping tasks, inspired by libraries like [beautiful soup](https://www.crummy.com/software/BeautifulSoup/) and [RoboBrowser](http://robobrowser.readthedocs.io/en/latest/readme.html).
If you're scraping multiple pages, I highly recommend using rvest in concert with [polite](https://dmi3kno.github.io/polite/).
The polite package ensures that you're respecting the [robots.txt](https://en.wikipedia.org/wiki/Robots_exclusion_standard) and not hammering the site with too many requests.
## Installation
```{r, eval = FALSE}
# The easiest way to get rvest is to install the whole tidyverse:
install.packages("tidyverse")
# Alternatively, install just rvest:
install.packages("rvest")
```
## Usage
```{r, message = FALSE}
library(rvest)
# Start by reading a HTML page with read_html():
starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")
# Then find elements that match a css selector or XPath expression
# using html_elements(). In this example, each <section> corresponds
# to a different film
films <- starwars %>% html_elements("section")
films
# Then use html_element() to extract one element per film. Here
# we the title is given by the text inside <h2>
title <- films %>%
html_element("h2") %>%
html_text2()
title
# Or use html_attr() to get data out of attributes. html_attr() always
# returns a string so we convert it to an integer using a readr function
episode <- films %>%
html_element("h2") %>%
html_attr("data-id") %>%
readr::parse_integer()
episode
```
If the page contains tabular data you can convert it directly to a data frame with `html_table()`:
```{r}
html <- read_html("https://en.wikipedia.org/w/index.php?title=The_Lego_Movie&oldid=998422565")
html %>%
html_element(".tracklist") %>%
html_table()
```