Lmst

Yay! Just discovered via Github that Unidata received its latest funding tranche and their developers are back to work!

I am really looking forward to a time when scientific data analysis is less of a constant fuckaround and fight with technical bullshit. I'd *really* like

- #netCDF natively supporting complex numbers
- #Python #xarray and #pandas to natively support physical units (#pint is great on its own but the integrations leave a LOT to be desired)
- #Jupyter notebooks to suck less (crashes, glitches, widget plots not saved statically, an effing BUILTIN formatter, etc.)
- proper data pipeline systems
...

Satisfaction comes from wherever you can get it. An issue I filed on Github in 2021 and which users gradually piled onto to say, oh yeah, we need that? It's resolved!

New release of my software coming in the next couple days.

#netcdf #grib2

𝗖𝗙 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗖𝗼𝗻𝘃𝗲𝗻𝘁𝗶𝗼𝗻𝘀 support #OpenScience by automating processes via metadata in #NetCDF and #Zarr files. They establish a unified language for the #weather, #climate, #ocean, and #EO community. https://buff.ly/rGPuoku
#remotesensing #EarthObs

how to tell if #netcdf was compiled with s3 support ? (on linux, just installed from RPM)

even an example netcdf url that should work from a bucket would be helpful

GSPy - A New Toolbox And Data Standard For Geophysical Datasets
--
https://doi.org/10.3389/feart.2022.907614 <-- shared paper
--
https://doi.org/10.5066/P9XNQVGQ | https://code.usgs.gov/g3sc/gspy <-- shared code repository
--
[an older paper, but code is in active and ongoing development/evolution]
#GIS #spatial #mapping #geophysics #geophysical #NetCDF #datatypes #code #opensource #library #dataformats #standardisation #standardization #openstandard #portable #metadata #Python #package #GSPy #methods #workflows #xarray #CRS #opendata #architecture #toolbox

Conceptual diagram of GSPy workflow. Data from a variety of formats and types are read into GSPy, along with required metadata files. Through the GSPy software, data are converted into a standardized NetCDF file containing the dataset and metadata appropriate for archiving and sharing.

GS data convention. (A) Datasets are structured into three fundamental group types based on content and data geometry. The Survey group contains general metadata about the dataset. Unstructured datasets, such as from CSV or TXT files, form Tabular groups, whereas structured (gridded) datasets are categorized under the Raster group. Metadata is attached to all groups, with various required attributes (green text) that expands on the CF-1.8 convention. (B) Groups follow a strict hierarchy in the NetCDF file, with a single Survey group at the top to which all data groups are attached. Datasets are indexed within their respective group type. (C) Tabular and Raster data groups must contain clearly defined dimensions, such as index or x, y, z, as well as coordinate variables. Raster groups are distinct in that dimensions are also coordinates, whereas Tabular datasets are assigned spatial coordinates that align with the index dimension. Lastly, the coordinate variable “spatial_ref” is required for all data groups, which expands on the “coordinate_information” variable required in the Survey metadata.

photo - rigs preparing to do a seismic survey, Middle East

GSPy code base - Writing and plotting examples. Once all groups have been attached to a Survey, the “write_netcdf” and “write_ncml” methods will write the GS NetCDF and NcML files, respectively. GSPy also provides methods to generate scatter and pcolor plots for variables.

🎉 Generate spreadsheet templates for recording data and metadata for CF-NetCDF files! 🚀

The latest update adds a sheet for variable attributes—metadata describing each variable.

🔗 https://www.nordatanet.no/aen/template-generator/

Templates bring structure and enable automated CF-NetCDF creation with scripts. In the future, I plan to develop a converter tool ion top of this—upload a template, download a CF-NetCDF!

#NetCDF #CFConventions #FAIRData

Excited to share that I'm now a member of the CF Conventions Committee! 🎉

CF (Climate and Forecast) conventions standardise how scientific data and metadata are stored in NetCDF files, ensuring consistency and machine readability. Looking forward to contributing and collaborating with the community to shape its future! 🌍

#CFConventions #NetCDF #FAIRData #ScientificData #DataScience

🚀 Excited to share my new FREE video tutorial on creating CF-NetCDF files using R! 🌟

In this video, you'll learn to:
• Ensure compliance with CF and ACDD conventions
• Work with 1D, 2D, and 3D data
• Manage irregular grids and moving instruments

🔗 Watch it here: https://youtu.be/IZDygRjfMIg

📖 Plus, check out the accompanying Jupyter book for code and explanations: https://nordatanet.github.io/NetCDF_in_R_from_beginner_to_pro/04_creating_a_cfnetcdf_file.html

#CFConventions #NetCDF #RProgramming #RStats #DataScience #ClimateData #FAIRData

I’m pleased to share that my NetCDF in Python and R tutorials are now available on NorDataNet. 🎉

Access them here:

Python: https://www.nordatanet.no/NetCDF-in-Python-from-beginner-to-pro
R: https://www.nordatanet.no/NetCDF-in-R-from-beginner-to-pro

These tutorials cover everything from basic data extraction to creating your own NetCDF files, ensuring they meet CF and ACDD standards for FAIR data.

NorDataNet enhances access to environmental datasets, supporting long-term preservation and interoperability. 🌍📊

#NetCDF #FAIRData #Python #R #NorDataNet

Using Mesh Data In QGIS [including accessing NetCDF data]
--
https://courses.gisopencourseware.org/course/view.php?id=72 <-- shared tutorial
--
https://youtu.be/OBi4wuxLiAE?si=_5DCAcsy8qjhsA-k <-- tutorial as a video
--
“Mesh data represents an unstructured grid that can include temporal and other components. It consists of vertices, edges, and faces that form a spatial structure in 2D or 3D space. Each vertex can store multiple datasets, which can also have a temporal dimension, making mesh data highly versatile for various applications such as meteorology, hydrodynamics, and environmental modeling…
This course is designed to equip you with the knowledge and skills needed to effectively work with mesh data within the QGIS environment…"
#QGIS #GIS #mesh #datavisualisation #NetCDF #climate #animation #openaccess #onlinelearning #learning #spatial #mapping #tutorial

maps and charts - tutorial output - precipitation and temperature at IHE Delft on Feb 21st 2022

screen shoot - QGIS - making the plot from the mesh data

🌍 Heading to the 2024 CF Workshop in Norrköping, Sweden! 🇸🇪 CF conventions are vital for ensuring data adheres to the FAIR principles. While NetCDF files are great, CF standards make data truly interoperable across disciplines.

I'll be sharing ideas on introducing CF to new disciplines like biology and geology, helping them unlock the full potential of their data. Excited to learn how CF conventions evolve and meet the community! 🌟

#FAIRData #CFConventions #Interoperability #NetCDF #OpenScience

Having one of those 3:22 in the a.m. moments when I got "stuff" to work!

Also, ugh, dealing with #netCDF libraries reading hierarchical (grouped) #HDF datasets and having to deal with unnamed dimensions. Also, ugh, again.

🌍📊 Want to work with NetCDF files in Python? My tutorial series covers everything from opening and plotting NetCDF data to creating CF-compliant files for FAIR data publication.

Whether you're new to NetCDF or looking to enhance your skills, I've got you covered! 🚀 Check it out: https://lhmarsden.github.io/NetCDF_in_Python_from_beginner_to_pro

Topics include:
🔸Extracting data 📝
🔸Plotting 📈
🔸Creating CF-compliant files 🌐
🔸Granularity 🖥️
🔸CF & ACDD 🖥️

Suggestions? Let me know! #Python #DataScience #NetCDF #xarray #FAIRData #ClimateData

🚀 New Video Alert! 🚀

I’ve just released a tutorial on how to convert NetCDF data to CSV or XLSX using R. Whether you’re dealing with 1D, 2D, or 3D variables, this video will guide you through the process step-by-step.

🎥 Watch here: https://youtu.be/9-EDaRQ8Aps

#RStats #NetCDF #DataScience #FAIRdata #CSV #XLSX #Tutorial #YouTube

Day six of the sixth eruption near #Svartsengi and the town of #Grindavík This eruption is the largest one yet. Here is a 4-day series of #Sentinel5p satellite data from the #TROPOMI instrument showing the mean amount of SO₂ in a vertical column of atmosphere (Dobson units). With time this will turn into sulfate particles forming volcanic smog.

This level 3 data is made available by The German Aerospace Center as #NetCDF files and can easily be opend in #QGIS and visualized with MDAL library.

@roadskater
1) #netCDF is so much better than the individual formats thousands of scientists have made up over the decades. But I really wish people would use the metadata fields consistently (or at all).

2) Have you discovered IrfanView? It’s mostly just a viewer, but it handles every image type and many grid types that I’ve thrown at it. It is great for checking whether the netCDF shows what it’s supposed to.

The person who pinged me about this won't look too good dealing with the data product developers on his end if he keeps arguing for standards that, e.g., mean my software can read and plot the data. But forget moi, CF rules!

Otherwise. Ooof. My bad.

#netcdf #hdf

Meanwhile, other software packages apparently can plot this stuff? I guess they have multiple coders, and funding to pay them all?

Otherwise, crud. The data grid mapping issue is a pain, but I figured that out. But… the data are in a structure, which… the library doesn't handle well (cough, gag) and disaster ensues.

There could be a career in sci software dev here. If one had the funding.

#netcdf

Spent some time the last few days examining whether I need to blow up and rewrite code in order to plot a satellite data product. Seemed from perusing a very necessary library API that it wouldn't need much work. On re-read, maybe more work. Sigh, now realizing that the API itself would need refactoring. Which, 1) their developers might say no, or 2) worse, they might say yes but they have SO many other demands.

The user who pinged me about this will probably be less than surprised. F!

#netcdf

#NetCDF

Client Info