Commit Graph

12 Commits

Author SHA1 Message Date
Claude
ff13012408 fix: address functionality gaps identified in code review
- Wire up --config/-f flag to actually load custom config files
  - Move config loading to PersistentPreRunE in root.go
  - Simplify main.go to just call cmd.Execute()
  - Move Playwright init to web command's PreRunE/PostRunE

- Remove unused functions from cmd/web.go (~90 lines of dead code)
  - Remove writeSingleFile, writeMultipleFiles, generateDefaultFilename
  - Remove scrapeURL, extractAndConvertContent, testExtractAndConvertContent
  - Remove unused mock function from web_test.go

- Add OutputType validation to Config.Validate()
  - Only allow "single", "separate", or empty string
  - Add test cases for valid and invalid output types
2025-11-27 16:05:42 +00:00
Claude
09608cf073 fix: resolve 5 bugs identified in code review
- Fix malformed YAML in config_test.go (incorrect indentation)
- Add validation for empty file_extensions in Config.Validate()
- Remove obsolete max_depth test case (field no longer exists)
- Remove unused global cfg variable in main.go
- Fix race condition in ScrapeSites by counting URLs before goroutines
- Remove unreachable JavaScript code in scroll script, add proper delay
- Standardize file extensions to not include leading dot
2025-11-27 15:56:37 +00:00
tnypxl
02e39baf38 flatten scrape config to 'sites:'
* flatten scrape config to 'sites:'. Update unit tests and readme.
* remove check for file_extensions configuration. 
* show progress indication after 5 seconds.
* add documentation to functions
* fix: remove MaxDepth and link extraction functionality
* fix: Remove MaxDepth references from cmd/web.go
2024-10-14 16:09:58 -05:00
Arik Jones (aider)
21d3e8ee68 fix: Handle missing configuration file for help command 2024-09-16 09:52:48 -05:00
Arik Jones (aider)
a6ebf0062a fix: Add --verbose flag to web subcommand 2024-09-14 20:41:23 -05:00
Arik Jones (aider)
6f4750c900 fix: Remove references to non-existent CSSLocator field in Config struct 2024-09-14 20:36:31 -05:00
Arik Jones (aider)
b1db362a94 fix: Initialize logger before calling InitPlaywright 2024-09-14 19:59:39 -05:00
Arik Jones (aider)
595c451ad9 feat: Pass scraper configuration to command execution 2024-09-14 15:15:39 -05:00
Arik Jones
a3b23a6d34 ... 2024-09-14 15:11:24 -05:00
Arik Jones (aider)
8932f503c6 feat: Pass configuration to command execution 2024-09-14 15:09:57 -05:00
Arik Jones
0163c4e504 Adds a configuration layer for use rollup.yml which may be preferred over CLI flags. 2024-09-05 23:41:39 -05:00
Arik Jones
ef89da6c39 Add main.go 2024-09-03 11:07:45 -05:00