Released scrapy-contrib-bigexporter 1.0.0 (https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexporters) - additional export formats for the webscraping framework Scrapy.
Migrated parquet export from fastparquet to pyarrow as fastparquet is deprecated (https://docs.dask.org/en/stable/changelog.html#fastparquet-engine-deprecated)
Migrated orc export from pyorc to pyarrow to reduce the number of dependencies
#scrapy #crawling #python #parquet #orc #pyarrow #webcrawling #scraping
If the purpose of a library is to "process and transport large data sets" but the code base contains an error message like "array cannot contain more than 2147483646 bytes" then there must be a big misunderstanding somewhere. #pyarrow
Easily obtain OSM and OMF data: #Python and CLI tools #QuackOSM and #OvertureMaestro offer easier access to data from #OpenStreetMap (#OSM) and the Overture Maps Foundation (#OMF) through #PyArrow, #GeoParquet, or #DuckDB. These tools can simplify large-scale geospatial data...
https://spatialists.ch/posts/2025/05/23-easily-obtain-osm-and-omf-data/ #GIS #GISchat #geospatial #SwissGIS
Easily obtain OSM and OMF data: #Python and CLI tools #QuackOSM and #OvertureMaestro offer easier access to data from #OpenStreetMap (#OSM) and the Overture Maps Foundation (#OMF) through #PyArrow, #GeoParquet, or #DuckDB. These tools can simplify large-scale geospatial data...
https://spatialists.ch/posts/2025/05-23-easily-obtain-osm-and-omf-data/ #GIS #GISchat #geospatial #SwissGIS
#TalkPythonToMe 503: The #PyArrow Revolution
Episode webpage: https://talkpython.fm/episodes/show/503/the-pyarrow-revolution
Media file: https://talkpython.fm/episodes/download/503/the-pyarrow-revolution.mp3
Currently taking a look at refreshing some of the #ApacheArrow and #PyArrow docs, so if you use Arrow in #rstats or Python and there's any areas you'd like to understand better, give me a shout, and we'll see what we can do!
#python #pyarrow #default_serialization_context
pyarrow.serialization and default_serialization_context are used for converting Python objects to Arrow binary format and vice versa.
pyarrow.parquet.core provides functionality for reading/writing Parquet files in Python. write_to_dataset is used to write a PyArrow table to a Parquet dataset. #python #pyarrow #write_to_dataset
🌗 Lance - 現代化的列式資料格式，適用於機器學習和大型機器學習模型，使用 Rust 實現。
➤ 只需兩行程式碼即可從 Parquet 轉換，實現 100 倍更快的隨機存取、向量索引和資料版本控制。與 Pandas、DuckDB、Polars、Pyarrow 兼容，更多整合正在到來。
https://github.com/lancedb/lance
Lance 是一種現代化的列式資料格式，針對機器學習工作流程和資料集進行了優化。它具有高性能的隨機存取、向量索引、資料版本控制等特點，並與 Pandas、DuckDB、Polars、Pyarrow 等生態系統整合。只需兩行程式碼即可從 Parquet 轉換，實現 100 倍更快的隨機存取、向量索引和資料版本控制。
+ 這是一個非常有用的工具，特別是對於需要處理大型資料集的機器學習工作流程。它的性能比 Parquet 更好，而且與多個生態系統整合，使得使用起來非常方便。
+ 使用 Rust 實現是一個
#Lance #資料格式 #機器學習 #Rust #Parquet #Pandas #DuckDB #Polars #Pyarrow
Estaba probando la nueva versión de #Pandas y la integración con #PyArrow.
Se nota que es muy rápido, pero parece que hay opciones pendientes de implementar. Por ejemplo "converter" al leer un CSV.
PyIceberg: Python Development Setup
This video will walk you through the steps required to set up the Python development environment for PyIceberg. We will set up a local instance of Spark, Rest catalog, and MinIO for querying an actual table. This makes it easy to do interactive development and test everything end to end.
#iceberg #python #pyiceberg #tabular #minio #spark #datalake #datalakehouse #pyarrow
https://youtu.be/D0HJuB0uSio
Fokko Driesprong has written a very interesting new blog on using the latest version of #PyIceberg with #PyArrow and DuckDB Labs to load data from an #Iceberg table into PyArrow or DuckDB with PyIceberg.
https://tabular.medium.com/pyiceberg-0-2-1-pyarrow-and-duckdb-79effbd1077f
With #PyIceberg 0.2.1 now available, we thought a video that illustrates using it with #PyArrow and DuckDB Labs would be in order. Thank you Fokko Driesprong for the content.
#iceberg #apacheiceberg #duckdb #voltrondata #datalake #datalakehouse
A hearty thank you to the PyIceberg community on the release of Apache PyIceberg release 0.2.0!
This release includes a few major features, such as
* Read support using PyArrow and DuckDB
* Support for AWS Glue
Please check the updated docs (https://py.iceberg.apache.org/) for the details.
This release can be downloaded from: https://pypi.org/project/pyiceberg/0.2.0/
And can be installed using: pip3 install pyiceberg==0.2.0
Hey #dataNerds 🤓, good news:
#DuckDB v0.6.0 brings reading #CSV data on par with #PyArrow & #Polars and loads 1.66 GB of #ChicagoCrimes data in 1.9s with 12 cores/24 threads when experimental parallel CSV reader & unordered insertion are enabled.
🧐 https://github.com/RandomFractals/chicago-crimes#with-duckdb
#dataTools 🔬 ...
Today I'm doing some #MachineLearning archeology, and once again dabbling in #PyArrow to read and convert parquet to #Pandas so that I can do some minimal data exploration and answer questions on how the models were trained. If you store data in parquet format, PyArrow is a great resource.
