tnypxl/rollup
1
0
Fork 0
mirror of https://github.com/tnypxl/rollup.git synced 2025-12-15 15:03:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: Update README.md to reflect recent changes in functionality

Browse Source
This commit is contained in:
Arik Jones (aider)
2024-09-22 18:31:06 -05:00
parent 7d8e25b1ad
commit ea410e4abb
1 changed files with 31 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
README.md
View File

@@ -4,16 +4,18 @@ Rollup aggregates the contents of text-based files and webpages into a markdown
## Features ## Features
- File type filtering - File type filtering for targeted content aggregation
- Ignore patterns for excluding files - Ignore patterns for excluding specific files or directories
- Support for code-generated file detection - Support for code-generated file detection and exclusion
- Advanced web scraping functionality - Advanced web scraping functionality with depth control
- Verbose logging option for detailed output - Verbose logging option for detailed operation insights
- Exclusionary CSS selectors for web scraping - Exclusionary CSS selectors for precise web content extraction
- Support for multiple URLs in web scraping - Support for multiple URLs in web scraping operations
- Configurable output format for web scraping (single file or separate files) - Configurable output format for web scraping (single file or separate files)
- Configuration file support (YAML) - Flexible configuration file support (YAML)
- Generation of default configuration file - Automatic generation of default configuration file
- Custom output file naming
- Concurrent processing for improved performance
## Installation ## Installation
@@ -43,6 +45,8 @@ rollup [command] [flags]
- `--types, -t`: Comma-separated list of file extensions to include (default: .go,.md,.txt) - `--types, -t`: Comma-separated list of file extensions to include (default: .go,.md,.txt)
- `--codegen, -g`: Comma-separated list of glob patterns for code-generated files - `--codegen, -g`: Comma-separated list of glob patterns for code-generated files
- `--ignore, -i`: Comma-separated list of glob patterns for files to ignore - `--ignore, -i`: Comma-separated list of glob patterns for files to ignore
- `--output, -o`: Name of the output file (default: rollup_output.md)
- `--concurrent, -c`: Number of concurrent goroutines for processing (default: 4)
### Flags for `web` command ### Flags for `web` command
@@ -51,11 +55,13 @@ rollup [command] [flags]
- `--depth, -d`: Depth of link traversal (default: 0, only scrape the given URLs) - `--depth, -d`: Depth of link traversal (default: 0, only scrape the given URLs)
- `--css`: CSS selector to extract specific content - `--css`: CSS selector to extract specific content
- `--exclude`: CSS selectors to exclude from the extracted content (comma-separated) - `--exclude`: CSS selectors to exclude from the extracted content (comma-separated)
- `--concurrent, -c`: Number of concurrent goroutines for processing (default: 4)
### Global flags ### Global flags
- `--config, -f`: Path to the configuration file (default: rollup.yml in the current directory) - `--config, -f`: Path to the configuration file (default: rollup.yml in the current directory)
- `--verbose, -v`: Enable verbose logging - `--verbose, -v`: Enable verbose logging
- `--timeout, -t`: Timeout for web requests in seconds (default: 30)
## Configuration ## Configuration
@@ -92,10 +98,10 @@ scrape:
rollup files rollup files
``` ```
2. Web scraping with multiple URLs: 2. Web scraping with multiple URLs and increased concurrency:
```bash ```bash
rollup web --urls=https://example.com,https://another-example.com rollup web --urls=https://example.com,https://another-example.com --concurrent=8
``` ```
3. Generate a default configuration file: 3. Generate a default configuration file:
@@ -104,15 +110,25 @@ scrape:
rollup generate rollup generate
``` ```
4. Use a custom configuration file: 4. Use a custom configuration file and specify output:
```bash ```bash
rollup files --config=my-config.yml rollup files --config=my-config.yml --output=project_summary.md
``` ```
5. Web scraping with separate output files: 5. Web scraping with separate output files and custom timeout:
```bash ```bash
rollup web --urls=https://example.com,https://another-example.com --output=separate rollup web --urls=https://example.com,https://another-example.com --output=separate --timeout=60
```
6. Rollup files with specific types and ignore patterns:
```bash
rollup files --types=.go,.md --ignore=vendor/**,*_test.go
```
7. Web scraping with depth and CSS selector:
```bash
rollup web --urls=https://example.com --depth=2 --css=.main-content
``` ```
## Contributing ## Contributing