Project repository: https://github.com/lhoestq/duckdb-hf-extension

Installation

SET allow_extension_repositories = 'allowed';
CREATE OR REPLACE EXTENSION REPOSITORY hf_repo
  WITH PREFIX 'https://hf.co/buckets/duckdb-extensions/builds/resolve';
INSTALL hf FROM hf_repo;

Usage

Read/write to Hugging Face buckets, datasets, models and spaces. Optimized with Xet.

LOAD hf FROM hf_repo;

SELECT * FROM 'hf://datasets/fka/prompts.chat/prompts.csv';
SELECT * FROM hf_ls('hf://buckets/commoncrawl/commoncrawl/crawl-data');
COPY (SELECT * FROM range(10) t(i)) TO 'hf://datasets/me/demo/out.parquet';
COPY (SELECT * FROM read_parquet('hf://datasets/me/demo/*.parquet'))
  TO 'hf://datasets/me/demo/partitioned' (FORMAT PARQUET, PARTITION_BY (kind));

Authentication

Login with the hf CLI:

hf auth login

Or pass a token manually:

CREATE SECRET (TYPE HUGGINGFACE, TOKEN 'hf_...');