tnypxl
02e39baf38
flatten scrape config to 'sites:'
...
* flatten scrape config to 'sites:'. Update unit tests and readme.
* remove check for file_extensions configuration.
* show progress indication after 5 seconds.
* add documentation to functions
* fix: remove MaxDepth and link extraction functionality
* fix: Remove MaxDepth references from cmd/web.go
2024-10-14 16:09:58 -05:00
Arik Jones
691832e282
fix: Update expectation
2024-09-22 18:18:03 -05:00
Arik Jones (aider)
574800c241
fix: Update runRollup function to accept config parameter
2024-09-22 17:06:18 -05:00
Arik Jones (aider)
274ef7ea79
test: enhance and expand test coverage for file operations
2024-09-22 16:56:52 -05:00
Arik Jones
a55e8df02a
refactor: improve error handling and variable naming in TestRunRollup
2024-09-22 16:56:51 -05:00
Arik Jones (aider)
364b185269
fix: resolve test failures in TestRunRollup, TestExtractContentWithCSS, and TestExtractLinks
2024-09-21 16:04:20 -05:00
Arik Jones (aider)
e5d4c514a7
fix: resolve build errors in test files
2024-09-21 15:59:39 -05:00
Arik Jones (aider)
6ff44f81bb
fix: resolve nil pointer dereference in ExtractContentWithCSS test
2024-09-21 15:59:08 -05:00
Arik Jones (aider)
2fd411ce65
test: add debugging info and fix reflect import
2024-09-21 15:57:05 -05:00
Arik Jones
73116e8d82
Fix logging and other issues from preventing scraping
2024-09-21 15:54:33 -05:00
