MT2Data Datalake Gateway

Secure, index-based access to the MT2Data data lake. The gateway provides authenticated access to data files for authorized enterprise clients.

Base URL: https://datalake.mt2data.cloud

What the Datalake Gateway provides

• OAuth 2.0 Client Credentials — enterprise M2M authentication with 1-hour JWT tokens
• Symbol-based file access — files are identified by human-readable Symbols, not raw paths
• Subscription-scoped access — your account defines exactly which files you can retrieve
• Fast catalog browsing — browse all authorized Symbols and descriptions in ~1s

Quick links

• Authentication — OAuth 2.0 flow, credentials, token caching
• API Reference — all endpoints with request/response details
• Code Examples — Python, JavaScript, C# clients
• Performance Guide — endpoint speeds, recommended workflows
• Troubleshooting — common errors, security best practices

API Reference

Base URL: https://datalake.mt2data.cloud

All data endpoints require authentication. See Authentication for how to obtain an access token.

POST /oauth/token

Obtain an OAuth 2.0 access token using client credentials.

Request:

POST /oauth/token HTTP/1.1
Content-Type: application/json

{
  "grant_type": "client_credentials",
  "client_id": "your-client-id",
  "client_secret": "your-client-secret"
}

Also accepts application/x-www-form-urlencoded.

Response (200):

{
  "access_token": "eyJhbGciOiJIUzI1NiIs...",
  "token_type": "Bearer",
  "expires_in": 3600
}

Response (401):

{
  "error": "invalid_client",
  "error_description": "Invalid client credentials"
}

GET /catalog

Get metadata for all files you are authorized to access. Returns Symbol identifiers, human-readable descriptions, and R2 file paths.

The Description field is read from the index CSV's ShortDescriptionPt column (the short Portuguese label), falling back to ShortDescription where that cell is empty.

Response (200):

{
  "clientId": "your-account",
  "files": [
    {
      "Symbol": "IbgeGdpAgroInd",
      "Description": "PIB - Agropecuária Ind",
      "FileName": "BaseDeDadosMacro/ContasNacionais/IBGE/IbgeGdpAgroInd.csv"
    }
  ],
  "count": 1
}

Performance: ~1s — use this as your primary way to discover available files.

GET /list

List files that appear in your catalog and physically exist in R2, with size and upload timestamp. Uses batched HEAD verification.

Parameters:

Response (200):

{
  "files": [
    {
      "symbol": "IbgeGdpAgroInd",
      "description": "PIB - Agropecuária Ind",
      "size": 1048576,
      "uploaded": "2026-01-15T10:30:00Z"
    }
  ],
  "count": 1,
  "limit": 1000,
  "truncated": false
}

Performance: ~2s for a full catalog (~500 files).

GET /count

Returns the total number of accessible files that physically exist in R2.

Response (200):

{ "count": 506 }

Performance: ~2s.

GET /info/

Get metadata for a specific file by its Symbol identifier without downloading it.

GET /info/YOUR_SYMBOL HTTP/1.1
Authorization: Bearer <access_token>

Response (200):

{
  "symbol": "YOUR_SYMBOL",
  "size": 1048576,
  "uploaded": "2026-01-15T10:30:00Z",
  "etag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
  "httpMetadata": {},
  "customMetadata": {}
}

etag is also returned as an ETag header, alongside Last-Modified. Keep it and send it back as If-None-Match on the download — see below.

Performance: ~0.5s.

GET /download/

Download a file by its Symbol identifier. Returns binary content.

GET /download/YOUR_SYMBOL HTTP/1.1
Authorization: Bearer <access_token>

Response (200): File content as binary stream. Content-Type and Content-Disposition are forwarded from R2, alongside ETag, Last-Modified, Content-Length and Accept-Ranges: bytes.

Don't re-download what hasn't changed

Send the ETag you last received as If-None-Match. An unchanged file answers 304 Not Modified with no body, which matters for the larger objects — a 20 MB file costs one small round trip instead of 20 MB.

curl -H "Authorization: Bearer $TOKEN" \
     -H 'If-None-Match: "d41d8cd98f00b204e9800998ecf8427e"' \
     -o file.parquet -w '%{http_code}\n' \
     "https://datalake.mt2data.cloud/download/YOUR_SYMBOL"

HEAD works the same way if you only want the headers.

Resuming a large download

Range is supported, and a ranged request answers 206 Partial Content with a Content-Range header:

curl -H "Authorization: Bearer $TOKEN" -H 'Range: bytes=0-1048575' \
  "https://datalake.mt2data.cloud/download/YOUR_SYMBOL" -o first-mb.bin

Response (304): Empty, when If-None-Match matches the current file.

Response (403):

{
  "error": "Access denied",
  "message": "Symbol not found in your subscription"
}

Response (404):

{
  "error": "Not found",
  "message": "File does not exist"
}

Performance: ~0.5s + file transfer time.

GET /

Direct file download by file path (alternative to Symbol-based access).

curl -H "Authorization: Bearer $TOKEN" \
  "https://datalake.mt2data.cloud/path/to/your-file.parquet" -o your-file.parquet

Error Codes

Authentication

The MT2Data Datalake Gateway uses OAuth 2.0 Client Credentials flow for authentication. You receive a client_id and client_secret from MT2Data, exchange them for a short-lived access token, and include that token in all API requests.

Base URL: https://datalake.mt2data.cloud

Your Credentials

You will receive credentials from MT2Data:

Step 1 — Get an Access Token

curl -X POST https://datalake.mt2data.cloud/oauth/token \
  -H "Content-Type: application/json" \
  -d '{
    "grant_type": "client_credentials",
    "client_id": "your-client-id",
    "client_secret": "your-client-secret"
  }'

Response:

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "Bearer",
  "expires_in": 3600
}

Tokens are valid for 1 hour (3600 seconds).

Step 2 — Use the Token

Include the access token in the Authorization header on all API requests:

curl https://datalake.mt2data.cloud/catalog \
  -H "Authorization: Bearer <your-access-token>"

Token Expiry & Refresh

When a token expires you will receive a 401. Request a new token via Step 1.

Best practice: track expires_in in your application and refresh the token ~60 seconds before expiry to avoid interruption. See Code Examples for complete implementations with automatic token refresh.

Code Examples

Complete client implementations in Python, JavaScript/Node.js, and C# with automatic token refresh.

Python

import requests
from datetime import datetime, timedelta

class MT2DataClient:
    def __init__(self, client_id: str, client_secret: str):
        self.base_url = "https://datalake.mt2data.cloud"
        self.client_id = client_id
        self.client_secret = client_secret
        self.access_token = None
        self.token_expires_at = None

    def _get_token(self) -> str:
        """Get or refresh access token."""
        if self.access_token and self.token_expires_at > datetime.now():
            return self.access_token

        response = requests.post(
            f"{self.base_url}/oauth/token",
            json={
                "grant_type": "client_credentials",
                "client_id": self.client_id,
                "client_secret": self.client_secret
            }
        )
        response.raise_for_status()

        data = response.json()
        self.access_token = data["access_token"]
        # Refresh 60s before actual expiry
        self.token_expires_at = datetime.now() + timedelta(seconds=data["expires_in"] - 60)
        return self.access_token

    def _headers(self) -> dict:
        return {"Authorization": f"Bearer {self._get_token()}"}

    def get_catalog(self) -> dict:
        """Get all authorized Symbols and descriptions."""
        return requests.get(f"{self.base_url}/catalog", headers=self._headers()).json()

    def list_files(self, limit: int = 1000) -> dict:
        """List accessible files with existence verification (~2s)."""
        return requests.get(
            f"{self.base_url}/list",
            headers=self._headers(),
            params={"limit": limit}
        ).json()

    def count_files(self) -> int:
        """Count total accessible files."""
        return requests.get(f"{self.base_url}/count", headers=self._headers()).json()["count"]

    def get_file_info(self, symbol: str) -> dict:
        """Get metadata for a file by Symbol."""
        return requests.get(f"{self.base_url}/info/{symbol}", headers=self._headers()).json()

    def download_file(self, symbol: str, local_path: str) -> None:
        """Download a file by Symbol to disk."""
        with requests.get(
            f"{self.base_url}/download/{symbol}",
            headers=self._headers(),
            stream=True
        ) as r:
            r.raise_for_status()
            with open(local_path, "wb") as f:
                for chunk in r.iter_content(chunk_size=8192):
                    f.write(chunk)


# Usage
if __name__ == "__main__":
    client = MT2DataClient(
        client_id="your-client-id@company.com",
        client_secret="your-secret-key-here"
    )

    # Browse available files
    catalog = client.get_catalog()
    print(f"Available Symbols: {catalog['count']}")
    for f in catalog["files"][:5]:
        print(f"  {f['Symbol']}: {f['Description']}")

    # Download a file by Symbol
    client.download_file("YOUR_SYMBOL", "dataset.parquet")
    print("Downloaded dataset.parquet")

JavaScript / Node.js

const axios = require('axios');
const fs = require('fs');

class MT2DataClient {
  constructor(clientId, clientSecret) {
    this.baseUrl = 'https://datalake.mt2data.cloud';
    this.clientId = clientId;
    this.clientSecret = clientSecret;
    this.accessToken = null;
    this.tokenExpiresAt = null;
  }

  async _getToken() {
    if (this.accessToken && this.tokenExpiresAt > Date.now()) {
      return this.accessToken;
    }
    const response = await axios.post(`${this.baseUrl}/oauth/token`, {
      grant_type: 'client_credentials',
      client_id: this.clientId,
      client_secret: this.clientSecret
    });
    this.accessToken = response.data.access_token;
    // Refresh 60s before expiry
    this.tokenExpiresAt = Date.now() + (response.data.expires_in - 60) * 1000;
    return this.accessToken;
  }

  async _headers() {
    return { Authorization: `Bearer ${await this._getToken()}` };
  }

  async getCatalog() {
    return (await axios.get(`${this.baseUrl}/catalog`, { headers: await this._headers() })).data;
  }

  async listFiles(limit = 1000) {
    return (await axios.get(`${this.baseUrl}/list`, {
      headers: await this._headers(),
      params: { limit }
    })).data;
  }

  async getFileInfo(symbol) {
    return (await axios.get(`${this.baseUrl}/info/${symbol}`, { headers: await this._headers() })).data;
  }

  async downloadFile(symbol, localPath) {
    const response = await axios.get(`${this.baseUrl}/download/${symbol}`, {
      headers: await this._headers(),
      responseType: 'stream'
    });
    await new Promise((resolve, reject) => {
      response.data.pipe(fs.createWriteStream(localPath))
        .on('finish', resolve)
        .on('error', reject);
    });
  }
}

// Usage
async function main() {
  const client = new MT2DataClient(
    'your-client-id@company.com',
    'your-secret-key-here'
  );

  const catalog = await client.getCatalog();
  console.log(`Available Symbols: ${catalog.count}`);

  await client.downloadFile('YOUR_SYMBOL', 'dataset.parquet');
  console.log('Downloaded dataset.parquet');
}

main().catch(console.error);

C# / .NET

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;

public class MT2DataClient
{
    private readonly string _baseUrl = "https://datalake.mt2data.cloud";
    private readonly string _clientId;
    private readonly string _clientSecret;
    private readonly HttpClient _httpClient = new();
    private string _accessToken;
    private DateTime _tokenExpiresAt;

    public MT2DataClient(string clientId, string clientSecret)
    {
        _clientId = clientId;
        _clientSecret = clientSecret;
    }

    private async Task<string> GetTokenAsync()
    {
        if (_accessToken != null && _tokenExpiresAt > DateTime.UtcNow)
            return _accessToken;

        var content = new StringContent(
            JsonSerializer.Serialize(new {
                grant_type = "client_credentials",
                client_id = _clientId,
                client_secret = _clientSecret
            }),
            Encoding.UTF8, "application/json"
        );

        var response = await _httpClient.PostAsync($"{_baseUrl}/oauth/token", content);
        response.EnsureSuccessStatusCode();

        using var doc = JsonDocument.Parse(await response.Content.ReadAsStringAsync());
        _accessToken = doc.RootElement.GetProperty("access_token").GetString();
        var expiresIn = doc.RootElement.GetProperty("expires_in").GetInt32();
        _tokenExpiresAt = DateTime.UtcNow.AddSeconds(expiresIn - 60);
        return _accessToken;
    }

    public async Task<string> GetCatalogAsync()
    {
        var token = await GetTokenAsync();
        _httpClient.DefaultRequestHeaders.Authorization =
            new AuthenticationHeaderValue("Bearer", token);
        var response = await _httpClient.GetAsync($"{_baseUrl}/catalog");
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }

    public async Task DownloadFileAsync(string symbol, string localPath)
    {
        var token = await GetTokenAsync();
        _httpClient.DefaultRequestHeaders.Authorization =
            new AuthenticationHeaderValue("Bearer", token);
        var bytes = await _httpClient.GetByteArrayAsync($"{_baseUrl}/download/{symbol}");
        await System.IO.File.WriteAllBytesAsync(localPath, bytes);
    }
}

// Usage
var client = new MT2DataClient("your-client-id@company.com", "your-secret-key-here");
Console.WriteLine(await client.GetCatalogAsync());
await client.DownloadFileAsync("YOUR_SYMBOL", "dataset.parquet");

Performance Guide

Endpoint Speed Reference

Recommended Workflows

Downloading a known file

If you already know the Symbol:

# Direct download — fastest path
GET /download/IbgePnadAgricultureStk

Discovering and downloading files

# Step 1: Browse catalog (~1s) — find the Symbol you need
GET /catalog
# Returns: [{ "Symbol": "IbgePnadAgricultureStk", "Description": "Agricultura - estoque" }, ...]

# Step 2: Download by Symbol (~0.5s + transfer)
GET /download/IbgePnadAgricultureStk

Verifying which files exist

# Use /list when you need to confirm files exist in R2 with metadata
GET /list?limit=1000
# ~2s with batched HEAD calls — returns Symbol, size, upload date

Quick file count

GET /count
# ~2s, returns total number of accessible files

Token Caching

Tokens are valid for 1 hour. Do not request a new token on every API call — cache it and refresh when it is ~60 seconds from expiry.

# Good — reuse token until near-expiry
if not self.access_token or self.token_expires_at < datetime.now():
    self.access_token = fetch_new_token()

# Bad — fetches a new token on every request
token = fetch_new_token()
call_api(token)

Parallelism

/list and /count internally make batched parallel HEAD requests against R2 — they are already optimized. For downloading multiple files, parallelize client-side:

from concurrent.futures import ThreadPoolExecutor

symbols = ["SymbolA", "SymbolB", "SymbolC"]

with ThreadPoolExecutor(max_workers=5) as executor:
    futures = [executor.submit(client.download_file, s, f"{s}.parquet") for s in symbols]
    for f in futures:
        f.result()

Troubleshooting

Common Errors

401 — "Invalid client credentials"

Cause: Client ID or secret is incorrect, or the OAuth client does not exist.

Fix:

• Verify credentials are exactly correct (no trailing spaces or newlines)
• Contact MT2Data to confirm your account is active

401 — "Token expired"

Cause: Access token has expired (lifetime is 1 hour).

Fix: Request a new token via POST /oauth/token. Implement token caching with expiry tracking in your application — see the Performance Guide.

403 — "Access denied" / "Symbol not found in your subscription"

Cause: You are trying to access a file that is not in your index.

Fix:

• Use GET /catalog to see which Symbols you are authorized to access
• Contact MT2Data if you need access to additional files

403 — "Invalid API key"

Cause: API key does not exist or has the wrong prefix.

Fix: Verify the key exists and is prefixed with mt2_admin_ or mt2_client_.

404 — File not found

Cause: The file exists in your catalog but has not yet been synced to R2, or the path/Symbol is incorrect.

Fix:

• Use GET /catalog to verify the Symbol is in your subscription
• Use GET /list to confirm the file actually exists in R2
• Check that the Symbol spelling is exact (case-sensitive)

404 — "Catalog not found"

Cause: Your client does not have an index file configured in R2.

Fix: Contact MT2Data to ensure your index file has been uploaded.

400 — "unsupported_grant_type"

Cause: grant_type is missing or not set to client_credentials.

Fix:

• Set grant_type=client_credentials in the request body
• Ensure Content-Type is application/json or application/x-www-form-urlencoded

Changes not applying immediately

KV-stored credentials (new keys, updated index files) can take up to 60 seconds to propagate globally. Wait and retry.

Getting Help

If you encounter an issue not covered here, reach out through any of the following:

• Contact form: mt2data.cloud/#contact
• Email: support@mt2data.cloud

When contacting support, include:

• Your Client ID (never the secret)
• The exact error message and HTTP status code
• The endpoint you were calling
• Approximate timestamp of the request

Security Best Practices

1. Never share your client secret — treat it like a password
2. Use environment variables or a secrets manager — do not hardcode secrets in source code
3. Cache tokens — request a new token only when the current one is near expiry, not on every call
4. Use HTTPS only — all endpoints enforce HTTPS
5. Rotate compromised secrets immediately — contact MT2Data if you suspect a secret has been exposed