mirror of
https://github.com/tnypxl/rollup.git
synced 2025-12-16 07:23:17 +00:00
docs: update configuration section in README.md to include scrape parameters and example usage
This commit is contained in:
23
README.md
23
README.md
@@ -63,19 +63,19 @@ rollup [command] [flags]
|
|||||||
|
|
||||||
Rollup can be configured using a YAML file. By default, it looks for `rollup.yml` in the current directory. You can specify a different configuration file using the `--config` flag.
|
Rollup can be configured using a YAML file. By default, it looks for `rollup.yml` in the current directory. You can specify a different configuration file using the `--config` flag.
|
||||||
|
|
||||||
Example `rollup.yml`:
|
**Scrape Configuration Parameters:**
|
||||||
|
|
||||||
|
- `requests_per_second`: *(float, optional)* The rate at which requests are made per second during web scraping. Default is `1.0`.
|
||||||
|
- `burst_limit`: *(integer, optional)* The maximum number of requests that can be made in a burst. Default is `5`.
|
||||||
|
|
||||||
|
These parameters help control the request rate to avoid overloading the target servers and to comply with their rate limits.
|
||||||
|
|
||||||
|
**Example `rollup.yml` with Scrape Configuration:**
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
file_types:
|
|
||||||
- go
|
|
||||||
- md
|
|
||||||
ignore:
|
|
||||||
- node_modules/**
|
|
||||||
- vendor/**
|
|
||||||
- .git/**
|
|
||||||
code_generated:
|
|
||||||
- **/generated/**
|
|
||||||
scrape:
|
scrape:
|
||||||
|
requests_per_second: 1.0
|
||||||
|
burst_limit: 5
|
||||||
sites:
|
sites:
|
||||||
- base_url: https://example.com
|
- base_url: https://example.com
|
||||||
css_locator: .content
|
css_locator: .content
|
||||||
@@ -94,9 +94,6 @@ scrape:
|
|||||||
css_locator: .special-content
|
css_locator: .special-content
|
||||||
exclude_selectors:
|
exclude_selectors:
|
||||||
- .special-ads
|
- .special-ads
|
||||||
output_type: single
|
|
||||||
requests_per_second: 1.0
|
|
||||||
burst_limit: 3
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Examples
|
## Examples
|
||||||
|
|||||||
Reference in New Issue
Block a user