API Documentation

Introduction

The Indo Data Works API turns any public web page into clean, structured data. Send a URL to /scrape and get back Markdown, HTML, or JSON. Send a query to /search and get back search results with the full content of each page already scraped — one call instead of two.

• Zero restrictions — no domain whitelists. If it's on the public web, we can fetch it.
• Anti-bot bypass — handles Cloudflare challenges, JavaScript rendering, and rate limiting on your behalf.
• AI-agent first — JSON/Markdown output designed to be piped straight into LLM context windows.
• Pay per request — $0.01 per scrape, $0.03 per search. No subscriptions.

All endpoints accept and return JSON with Content-Type: application/json. Requests over HTTPS only.

Getting Started

From zero to your first scrape in under a minute. You need nothing but a terminal or a few lines of code — no signup form, no credit card.

curl -X POST https://api.indodataworks.asia/api-keys \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com", "name": "my-first-key"}'

# => {"api_key": "d01a726c-85b9-...", "created_at": "2026-10-18T16:00:00Z", "user_id": "..."}

export IDW_API_KEY="paste_the_api_key_you_saved_above"

curl -X POST https://api.indodataworks.asia/scrape \
  -H "X-API-Key: $IDW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": "markdown"}'

curl "https://api.indodataworks.asia/usage?api_key=$IDW_API_KEY"

# => {"total_requests": 3, "requests_today": 3, "requests_this_hour": 1, "rate_limit": {...}}

Authentication

Authenticated endpoints (/scrape and /search) require a single header:

X-API-Key: YOUR_API_KEY

A missing or invalid key returns 401 Unauthorized:

{
  "detail": "Missing API key. Provide X-API-Key header."
}

Rate Limits

Every API key is limited to 100 requests per hour, tracked per key on a sliding window. The limit applies to /scrape and /search combined.

• Exceeding the limit returns 429 Too Many Requests with a JSON detail message.
• Check your current window at any time with GET /usage → requests_this_hour.
• On 429, back off and retry when the window resets — exponential backoff (e.g. 30s → 60s → 120s) works well.
• Need more than 100/hour? Contact us for an enterprise plan.

{
  "detail": "Rate limit exceeded: 100 requests/hour. Try again later."
}

Pricing

Pay-per-request, no subscription, no commitment. /api-keys, /usage, and /health are free.

Endpoints

Five endpoints, one base URL: https://api.indodataworks.asia. Each section below documents the request schema, response schema, errors, and runnable examples in cURL, Python, and JavaScript.

Scrape a webpage

Fetches the target URL, defeats bot protection and renders JavaScript when needed, then converts the page to your chosen format. The fastest way to get clean, LLM-ready text out of any public page.

Request body

Headers

Response body 200 OK

Examples

curl -X POST https://api.indodataworks.asia/scrape \
  -H "X-API-Key: $IDW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "markdown",
    "timeout": 30
  }'

import os
import requests

resp = requests.post(
    "https://api.indodataworks.asia/scrape",
    headers={"X-API-Key": os.environ["IDW_API_KEY"]},
    json={"url": "https://example.com", "format": "markdown", "timeout": 30},
    timeout=60,
)
resp.raise_for_status()
data = resp.json()

if data["success"]:
    print(data["metadata"]["title"])
    print(data["content"][:500])
else:
    print("Scrape failed:", data["error"])

const res = await fetch("https://api.indodataworks.asia/scrape", {
  method: "POST",
  headers: {
    "X-API-Key": process.env.IDW_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    format: "markdown",
    timeout: 30,
  }),
});

if (!res.ok) throw new Error(`API error: ${res.status}`);
const data = await res.json();
console.log(data.metadata.title);
console.log(data.content.slice(0, 500));

Example response

{
  "success": true,
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "metadata": {
    "title": "Example Domain",
    "url": "https://example.com",
    "status_code": 200,
    "content_length": 1120
  },
  "error": null,
  "strategy": "httpx",
  "scrape_time": 0.4748
}

Error codes

Search the web & scrape results

One call does both halves of a research loop: it runs a web search, then scrapes each top result so you get full page content — not just snippets. Ideal for AI agents answering questions that need fresh sources.

Request body

Response body 200 OK

Examples

curl -X POST https://api.indodataworks.asia/search \
  -H "X-API-Key: $IDW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "AI trends 2026",
    "max_results": 5,
    "scrape_content": true
  }'

import os
import requests

resp = requests.post(
    "https://api.indodataworks.asia/search",
    headers={"X-API-Key": os.environ["IDW_API_KEY"]},
    json={"query": "AI trends 2026", "max_results": 5, "scrape_content": True},
    timeout=120,
)
resp.raise_for_status()
data = resp.json()

for r in data["results"]:
    print(f"- {r['title']}  ({r['url']})")
    if r.get("content"):
        print(f"  {r['content'][:200]}...")

const res = await fetch("https://api.indodataworks.asia/search", {
  method: "POST",
  headers: {
    "X-API-Key": process.env.IDW_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    query: "AI trends 2026",
    max_results: 5,
    scrape_content: true,
  }),
});

if (!res.ok) throw new Error(`API error: ${res.status}`);
const { results } = await res.json();
for (const r of results) {
  console.log(`- ${r.title} (${r.url})`);
}

Example response

{
  "success": true,
  "results": [
    {
      "title": "The State of AI in 2026",
      "url": "https://example.com/ai-2026",
      "snippet": "Agents, inference costs, and open models...",
      "content": "# The State of AI in 2026\n\n..."
    }
  ],
  "error": null
}

Error codes

Generate an API key

Creates a new API key tied to your email. This is the entry point of the whole system — you call it once, store the key, and use it on every subsequent request.

Request body

Response body 200 OK

Examples

curl -X POST https://api.indodataworks.asia/api-keys \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com", "name": "my-first-key"}'

import requests

resp = requests.post(
    "https://api.indodataworks.asia/api-keys",
    json={"email": "you@example.com", "name": "my-first-key"},
    timeout=30,
)
resp.raise_for_status()
key = resp.json()["api_key"]
print("Save this key now:", key)

const res = await fetch("https://api.indodataworks.asia/api-keys", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ email: "you@example.com", name: "my-first-key" }),
});

if (!res.ok) throw new Error(`API error: ${res.status}`);
const { api_key } = await res.json();
console.log("Save this key now:", api_key);

Example response

{
  "api_key": "d01a726c-85b9-4736-89d6-ded094865658",
  "user_id": "usr_9f2c41",
  "created_at": "2026-10-18T16:00:00Z"
}

Error codes

Check usage statistics

Returns request counts for your API key — total, today, and inside the current rate-limit hour, plus a live view of your remaining quota. Use it in dashboards, cron jobs, or budget alerts.

Query parameters

Response body 200 OK

Examples

curl "https://api.indodataworks.asia/usage?api_key=$IDW_API_KEY"

import os
import requests

resp = requests.get(
    "https://api.indodataworks.asia/usage",
    params={"api_key": os.environ["IDW_API_KEY"]},
    timeout=15,
)
resp.raise_for_status()
usage = resp.json()

rl = usage["rate_limit"]
print(f"Total: {usage['total_requests']}, today: {usage['requests_today']}, "
      f"this hour: {usage['requests_this_hour']} ({rl['remaining']}/{rl['limit']} left)")

const url = new URL("https://api.indodataworks.asia/usage");
url.searchParams.set("api_key", process.env.IDW_API_KEY);

const res = await fetch(url);
if (!res.ok) throw new Error(`API error: ${res.status}`);

const usage = await res.json();
console.log(`This hour: ${usage.requests_this_hour} used, ` +
  `${usage.rate_limit.remaining}/${usage.rate_limit.limit} remaining`);

Example response

{
  "api_key": "d01a726c-85b9-4736-89d6-ded094865658",
  "total_requests": 142,
  "requests_today": 37,
  "requests_this_hour": 8,
  "rate_limit": {
    "limit": 100,
    "remaining": 92,
    "used": 8
  }
}

Error codes

Health check

Public liveness probe. No authentication, no billing. Use it for uptime monitors and to distinguish API problems from your own network problems.

Response body 200 OK

Examples

curl https://api.indodataworks.asia/health

import requests

resp = requests.get("https://api.indodataworks.asia/health", timeout=10)
print(resp.json()["status"])  # "healthy"

const res = await fetch("https://api.indodataworks.asia/health");
const { status } = await res.json();
console.log(status); // "healthy"

Example response

{
  "status": "healthy",
  "timestamp": "2026-10-18T16:00:00.000000",
  "scraper_initialized": true
}

Error Codes

All errors return JSON with a human-readable detail message. HTTP status is the contract — branch on it, log detail for debugging.

Best Practices

• Prefer format: "markdown" for LLM pipelines — it strips navigation/boilerplate and tokenizes efficiently.
• Set a client timeout above the server timeout. If you pass timeout: 30, wait at least 45s on the client so the server's structured error beats your socket abort.
• Retry idempotently on 429/5xx with exponential backoff + jitter; never retry 401/422 blindly.
• Cache results keyed by URL. Scraping the same page twice in one run costs double for zero new information.
• Use scrape_content: false on /search when the agent only needs links, then scrape the 1–2 pages it actually reads.
• Monitor with GET /usage before the window matters: alert at 80 requests/hour to avoid mid-run 429s.
• Store keys in environment variables or a secrets manager — never in source control.

Try It Live

Send a real request without leaving this page. Enter your API key and a URL — the request runs from your browser against the live API, and the JSON response appears below. Your key goes only to api.indodataworks.asia, never anywhere else, and is never stored.