An embeddable SQL database like SQLite, but supports Postgres features - DuckDB / github / HN

The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research - HN - DuckDB Internals (CMU Advanced Databases / Spring 2023) - duck DB use a vectorized push-base model (implemented in C++) - support larger than memory execution - DuckDB – The SQLite for Analytics (Mark Raasveldt, CWI)

• HN is a lightweight and super fast library/CLI for working with Parquet. It’s SQLite for column formats
• The DuckDB Local UI / HN - the UI is not open source.
• Should You Ditch Spark for DuckDb or Polars? / HN
• How does DuckDB compares to SQLite (chatGPT)

DuckLake

• DuckLake - advanced data lake features without traditional lakehouse complexity by using Parquet files and your SQL database. It’s an open, standalone format from the DuckDB team
• Understanding DuckLake: A Table Format with a Modern Architecture - a new table format for storing data on blobstore
  • data are stored on a blob storage (parquet on S3)
  • metadata are stored in RDB (Postgres)

see also

• Duck-UI – Browser-Based SQL IDE for DuckDB
  • there is an embeded one in duckdb
• A sharded DuckDB on 63 nodes runs 1T row aggregation challenge in 5 sec