Web Fetch

1---2name: web-fetch3description: Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.4---5 6# Web Content Fetching7 8Fetch web content in this order:91. Prefer markdown-native endpoints (`content-type: text/markdown`)102. Use selector-based HTML extraction for known sites113. Use the bundled Bun fallback script when selectors fail12 13## Prerequisites14 15Verify required tools before extracting:16 17```bash18command -v curl >/dev/null || echo "curl is required"19command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"20command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"21```22 23Install Bun dependencies for the bundled script:24 25```bash26cd ~/.claude/skills/web-fetch && bun install27```28 29## Default Workflow30 31Use this as the default flow for any URL:32 33```bash34URL="<url>"35CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"36 37if echo "$CONTENT_TYPE" | grep -q "markdown"; then38  curl -sL "$URL"39else40  curl -sL "$URL" \41    | html2markdown \42        --include-selector "article,main,[role=main]" \43        --exclude-selector "nav,header,footer,script,style"44fi45```46 47## Known Site Selectors48 49| Site | Include Selector | Exclude Selector |50|------|------------------|------------------|51| platform.claude.com | `#content-container` | - |52| docs.anthropic.com | `#content-container` | - |53| developer.mozilla.org | `article` | - |54| github.com (docs) | `article` | `nav,.sidebar` |55| Generic | `article,main,[role=main]` | `nav,header,footer,script,style` |56 57Example:58 59```bash60curl -sL "<url>" \61  | html2markdown \62      --include-selector "#content-container" \63      --exclude-selector "nav,header,footer"64```65 66## Finding the Right Selector67 68When a site isn't in the patterns list:69 70```bash71# Check what content containers exist72curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -1073 74# Test a selector75curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -3076 77# Check line count78curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l79```80 81## Universal Fallback Script82 83When selectors produce poor output, run the bundled parser:84 85```bash86bun ~/.claude/skills/web-fetch/fetch.ts "<url>"87```88 89If already in the skill directory:90 91```bash92bun fetch.ts "<url>"93```94 95## Options Reference96 97```bash98--include-selector "CSS"  # Keep only matching elements99--exclude-selector "CSS"  # Remove matching elements100--domain "https://..."    # Convert relative links to absolute101```102 103## Troubleshooting104 105**Empty output with selectors**: The page might be markdown-native. Check headers first:106 107```bash108curl -sIL "<url>" | grep -i '^content-type:'109```110 111**Wrong content selected**: The site may have multiple article/main regions:112 113```bash114curl -s "<url>" | grep -o '<article[^>]*>'115```116 117**`html2markdown` not found**: Install it, then retry selector-based extraction.118 119**`bun` or script deps missing**: Run `cd ~/.claude/skills/web-fetch && bun install`.120 121**Missing code blocks**: Check if the site uses non-standard code formatting.122 123**Client-rendered content**: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.