REST API reference for datalake.do - A managed Apache Iceberg data catalog for large-scale analytics on object storage with ACID transactions and schema flexibility.

DataLake API

A managed Apache Iceberg data catalog for large-scale analytics on object storage with ACID transactions and schema flexibility.

Endpoint

POST https://api.do/datalake

Authentication

All API requests require authentication:

curl https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

Request

Headers

Authorization: Bearer YOUR_API_KEY
Content-Type: application/json
X-Request-ID: unique-request-id (optional)

Request Body

{
  "operation": "datalake",
  "parameters": {
    // Operation-specific parameters
  },
  "options": {
    "timeout": 30000,
    "retries": 3
  }
}

Response

Success Response

{
  "success": true,
  "data": {
    // Response data
  },
  "meta": {
    "requestId": "req_123",
    "timestamp": "2025-01-01T12:00:00Z",
    "duration": 145
  }
}

Error Response

{
  "success": false,
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid parameters",
    "details": {
      // Error details
    }
  },
  "meta": {
    "requestId": "req_123",
    "timestamp": "2025-01-01T12:00:00Z"
  }
}

Operations

create

Create a new Iceberg data catalog.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "create",
    "parameters": {}
  }'

delete

Permanently remove the data lake and all its tables.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "delete",
    "parameters": {}
  }'

createTable

Create a new Iceberg table in the catalog.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "createTable",
    "parameters": {}
  }'

dropTable

Remove an Iceberg table from the catalog.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "dropTable",
    "parameters": {}
  }'

query

Execute SQL queries using R2 SQL engine.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "query",
    "parameters": {}
  }'

insert

Insert data into an Iceberg table.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "insert",
    "parameters": {}
  }'

update

Update existing records in an Iceberg table.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "update",
    "parameters": {}
  }'

deleteRecords

Delete records from an Iceberg table.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "deleteRecords",
    "parameters": {}
  }'

snapshot

Create a new snapshot for time-travel queries.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "snapshot",
    "parameters": {}
  }'

timeTravel

Query data as it existed at a specific point in time.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "timeTravel",
    "parameters": {}
  }'

evolveSchema

Add, rename, or delete columns without rewriting data.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "evolveSchema",
    "parameters": {}
  }'

listTables

List all tables in the catalog.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "listTables",
    "parameters": {}
  }'

getSchema

Retrieve the schema for a specific table.

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "getSchema",
    "parameters": {}
  }'

Examples

cURL

curl -X POST https://api.do/datalake \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "operation": "datalake",
    "parameters": {}
  }'

JavaScript/TypeScript

const response = await fetch('https://api.do/datalake', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    operation: 'datalake',
    parameters: {},
  }),
})

const data = await response.json()

Python

import requests

response = requests.post(
    'https://api.do/datalake',
    headers={
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json',
    },
    json={
        'operation': 'datalake',
        'parameters': {}
    }
)

data = response.json()

Rate Limiting

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 1609459200

Status Codes

200 OK - Success
400 Bad Request - Invalid request
401 Unauthorized - Missing/invalid API key
403 Forbidden - Insufficient permissions
404 Not Found - Resource not found
429 Too Many Requests - Rate limit exceeded
500 Internal Server Error - Server error
503 Service Unavailable - Service temporarily unavailable

Error Codes

VALIDATION_ERROR - Invalid parameters
AUTHENTICATION_ERROR - Invalid API key
AUTHORIZATION_ERROR - Insufficient permissions
NOT_FOUND - Resource not found
RATE_LIMIT_EXCEEDED - Too many requests
TIMEOUT - Operation timeout
INTERNAL_ERROR - Server error

Webhooks

Subscribe to events:

curl -X POST https://api.do/webhooks \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhook",
    "events": ["datalake.created", "datalake.deleted", "datalake.Table.created", "datalake.Table.dropped", "datalake.queried", "datalake.Data.inserted", "datalake.Data.updated", "datalake.Data.deleted", "datalake.Snapshot.created", "datalake.Query.timeTraveled", "datalake.Schema.evolved"]
  }'

Best Practices

API Keys - Store securely, never commit to git
Error Handling - Handle all error codes gracefully
Retries - Implement exponential backoff
Rate Limiting - Respect rate limits
Idempotency - Use X-Idempotency-Key header
Logging - Log requests for debugging

DataLake API

On this page