"We propose to re-think data management system parser design to create modern, extensible #parsers, which allow a dynamic configuration of the accepted syntax at run-time, for example to allow syntax extensions, new statements, or to add entirely new query languages."
#duckdb
New blog post on #tiledb and #snowflake integration and what is possible for #biomedical #research and what I wish I could do in #duckdb
https://brianrepko.github.io/blog/posts/2025-12-20-tiledb-snowflake/
Quick research project: using DuckDB for deterministic data profiling and drift, with LLMs only doing interpretation - not math. All local, no data in context.
If that sounds interesting to any data/ML folks (especially Ollama users), I’m open to feedback.
https://www.mostlylucid.net/blog/datasummarizer-how-it-works
https://github.com/scottgal/mostlylucidweb/blob/main/Mostlylucid.DataSummarizer/README.md
Worth continuing with / it's doing what some other tool does quicker and better? 🤓
#duckdb #ollama #dataengineering #analytics #privacybydesign #opensource
The reason I made a sample dataset was that I thought it was a bit sluggish querying the GeoPackage file from DuckDB. The query in the image took 2.56 s on the GeoPackage file. I now tried to save the entire dataset into a Parquet file (sorted on county and municipality) and compressed with ZSTD. The same query takes 0.0140s.
Also the Parquet file is 141 MiB compared to 1.18 GiB for the GeoPackage file. The Parquet file is smaller than the original zip file with the GeoPackage file.
New prototype (v0.1.0): DataSummarizer
A statistics-first data profiling CLI built on DuckDB.
Computes deterministic profiles (nulls, skew, leakage, drivers, outliers) locally, then optionally lets a local LLM narrate or generate SQL — never the other way round.
Early days, but already useful for first-pass analysis, data contracts, drift checks, and segment comparison.
LLMs reason. Databases compute.
https://www.mostlylucid.net/blog/datasummarizer-how-it-works
https://github.com/scottgal/mostlylucidweb/tree/main/Mostlylucid.DataSummarizer
For the past few days I've been trying to figure out to use #duckdb COLUMNS(...) to do a ANY reduce on multiple bool columns, instead of the default which is ALL.
I think there's a syntax proposal but not merged in yet? Anyways, in the meantime, I've discovered that `GREATEST(*COLUMNS(...))` works to this effect. Only took me a whole week to figure this out 😅
(Fellow #rstats nerds will recognize these ops as if_any and if_all!)
Big week for FOSS! 🎉 FreeBSD 15.0 dropped Dec 2 with major improvements to pkgbase and security. Meanwhile, NVIDIA open-sourced NeMo Data Designer under Apache 2.0 - awesome for AI development! DuckDB 1.4.3 LTS also landed Dec 9. The community keeps pushing forward! #FreeBSD15 #NeMo #DuckDB #FOSS #opensource
Learn how to conduct geospatial analysis to determine which UK city has the safest drivers. Thomas Reid's new article uses GeoPandas to process boundary data and #DuckDB to query 5 years of traffic accident statistics.
https://towardsdatascience.com/geospatial-exploratory-data-analysis-with-geopandas-and-duckdb/
Deep dive on pruning: Dewey Dunnington’s latest blog post dives deep into how pruning, the selective reading of relevant data, makes #GeoParquet blazing fast in both local and #cloudnative contexts. Featuring hands-on comparisons across #SedonaDB, #DuckDB, #GeoPandas,...
https://spatialists.ch/posts/2025/12/16-deep-dive-on-pruning/ #GIS #GISchat #geospatial #SwissGIS
Ever wonder how tools analyse gigabytes of data, while you’re fighting to squeeze a few KB of code into an Ollama context window?
The trick isn’t bigger models.
It’s not giving the data to the LLM at all.
LLM → SQL → DuckDB → CSV-on-disk
Schema + samples only. Zero data leakage. Sub-100ms queries on 500MB+ files.
Local. Private. Scales past RAM.
Full C# walkthrough:
https://www.mostlylucid.net/blog/analysing-large-csv-files-with-local-llms
So many to choose from: In his latest post, Mark Litwintschik compares a range of global administrative #boundary datasets, from #OpenStreetMap to #NaturalEarth, assessing geometric #accuracy, data #completeness, and information content. The analysis, powered by #DuckDB...
https://spatialists.ch/posts/2025/12/13-so-many-to-choose-from/ #GIS #GISchat #geospatial #SwissGIS
I have published #DuckDB #NodeJS bindings for #FreeBSD. https://github.com/duckdb/duckdb-node-neo/discussions/349
You know, in case I ever need it (needed an excuse to play with sourcehut's CI).
From the @DSLC :rstats:chives:
:rstats: :python: :julia: DuckDB in Action: Club meetings & An introduction to DuckDB https://youtu.be/Cdi1lPdMfG8 #RStats #PyData #JuliaLang #DuckDB
:rstats: pharmaverse Examples: Demographic Table https://youtu.be/Vgpej0_C3lw #RStats
:rstats: R for Data Science: https://youtu.be/HicrANUfnj0 #RStats
Visit https://dslc.video for hours of new #DataScience videos every week!
Recent @DSLC club meetings:
🦆 DuckDB in Action: Executing SQL Queries https://youtu.be/EHgXpnrJAB4 #RStats #PyData #JuliaLang #duckdb
From the @DSLC :rstats:chives:
:rstats: Advanced R: Functionals https://youtu.be/tYqFMtmhmiI #RStats
:rstats: :javascript: JS4R: Managing JavaScript https://youtu.be/NyBUNAqENhg #RStats #JavaScript
:rstats: SMLTAR: Word Embeddings Part 1 https://youtu.be/2ZxXAUivrVc #RStats
Visit https://dslc.video for hours of new #DataScience videos every week!
The Advent of SQL has started 🎄
This year it's hosted by "Database School" and requires signing up: https://databaseschool.com/series/advent-of-sql/videos/308
I'll be solving the challenges using #DuckDB and my solutions for the first two days are up on my website: https://francoismichonneau.net/2025/12/advent-of-sql/
I'm a tad bit nervous to be presenting an Introductory talk about #DuckDB at the RubySG Dec meetup today (9 Dec)! But I have a good 11million record synthetic dataset to show some of the features!
Register: https://luma.com/0em8ixuy
Using the #duckdb COLUMNS(<regex>) pattern with capture groups to compute and rename multiple columns all in one line of SELECT is *sooo slick*
Been on a DuckDB learning spree lately and really proud of myself for figuring this out by just reading the docs with no AI help (they all come up with more verbose solutions than this and look gross)
No more sf::st_as_text() or st_geomfromtext() to move geospatial data between {sf} and duckdb spatial 😃 https://cidree.github.io/duckspatial/ #gisChat #Rstats #DuckDB
Client Info
Server: https://mastodon.social
Version: 2025.07
Repository: https://github.com/cyevgeniy/lmst