Lmst

#MachineLearning #ModelEvaluation #Calibration #HealthcareAI #AIethics #DecisionTheory

💻 #AllometricEquations for estimating above- and below-ground #Biomass of #PhragmitesAustralisMarshes.
Characteristics:
1️⃣ Divided into saltwater marshes and freshwater marshes.
2️⃣ Using plant height as the sole predictor.
3️⃣ It is a power-law allometric model.
#ModelEvaluation
https://doi.org/10.1093/jpe/rtae113

Scatter plots for of predicted and observed values of AGB based on logarithmic transformed allometric model with plant height (H) alone as predictor variable for reed marsh.

Verification of selected AGB estimate model with plant height alone as predictor variable by comparing it and a new model with literature data on larger scale added.

Grab your copy:

Gumroad -> https://valeman.gumroad.com/l/MasteringModernTimeSeriesForecasting

https://leanpub.com/mastering_modern_time_series_forecasting

#TimeSeries #Forecasting #ModelEvaluation #MetricsMatter #MachineLearning #AI #FVA #ChapterDrop
9/9

The lesson: just because a model is branded for uncertainty doesn’t mean it delivers. Stop the monkey data science—science is not crowd-following.

#DataScience #NGBoost #ProbabilisticModels #ModelEvaluation #MachineLearning #StopTheHype

---

@data @datadon 🧵

Accuracy! To counter regression dilution, a method is to add a constraint on the statistical modeling.
Regression Redress restrains bias by segregating the residual values.
My article: http://data.yt/kit/regression-redress.html

#bias #modeling #dataDev #AIDev #modelEvaluation #regression #modelling #dataLearning #linearRegression #probability #probabilities #statistics #stats #correctionRatio #ML #distributions #accuracy #RegressionRedress #Python #RStats

@data @datadon 🧵

How to assess a statistical model?
How to choose between variables?

Pearson's #correlation is irrelevant if you suspect that the relationship is not a straight line.

If monotonic relationship:
"#Spearman’s rho is particularly useful for small samples where weak correlations are expected, as it can detect subtle monotonic trends." It is "widespread across disciplines where the measurement precision is not guaranteed".
"#Kendall’s Tau-b is less affected [than Spearman’s rho] by outliers in the data, making it a robust option for datasets with extreme values."
Ref: https://statisticseasily.com/kendall-tau-b-vs-spearman/

#normality #normalDistribution #modeling #dataDev #AIDev #ML #modelEvaluation #regression #modelling #dataLearning #featureEngineering #linearRegression #modeling #probability #probabilities #statistics #stats #correctionRatio #ML #Pearson #bias #regressionRedress #distributions

Dive into the world of model evaluation in machine learning! Discover how to optimize logistic regression, compare ensemble techniques, and select the best model using key performance metrics. #MachineLearning #ModelEvaluation #DataScience

https://teguhteja.id/model-evaluation-mastering-performance-metrics-and-selection-in-machine-learning/

There's still time to submit a proposal to “Genomic forecasting of adaptation under environmental change” special feature in AmNat! Click here for all the details: https://www.amnat.org/announcements/genomic-forecasting-feature.html

#genomicForecasting #Genomics #theory #methods #modelEvaluation #empiricalStudies #criticalViewpoints

Genomic forecasting! See https://www.amnat.org/announcements/genomic-forecasting-feature.html

Learn how to interpret R², the 'coefficient of determination' in regression models! 📈📚 #DataScience #RegressionAnalysis #ModelEvaluation https://towardsdatascience.com/interpreting-r%C2%B2-a-narrative-guide-for-the-perplexed-086a9a69c1ec

https://eugeneyan.com//writing/aieng-reflections/

Very nice summary of the AI Engineer Summit 2023 by @eugeneyan

#llms #AI #mlops #chatgpt #machinelearning #modelevaluation

4/

Below, key points:

- "lack of #ModelEvaluation"

- statistical #uncertainty & "#robustness of event attribution results"

#References

[4] Seneviratne, et al., 2021. Chapter 11: weather and climate extreme events in a changing climate. In: Climate Change 2021: The Physical Science Basis - Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. IPCC, Geneva, Switzerland, pp. 1513–1766. https://purl.org/INRMM-MiD/z-ED8RQFV5

https://www.ipcc.ch/report/ar6/wg1/downloads/report/IPCC_AR6_WGI_Chapter11.pdf#page=28

Excerpt from [4], p. 1540 (p. 28 of the PDF). Cited literature: pp. 1706-1758

Section "11.2.3 Attribution of Extremes"

"Apart from the detection and attribution of trends in extremes, new approaches have been developed to answer the question of whether, and to what extent, external drivers have altered the probability and intensity of an individual extreme event (NASEM, 2016). In AR5, there was an emerging consensus that the role of external drivers of climate change in specific extreme weather events could be estimated and quantified in principle, but related assessments were still confined to particular case studies, often using a single model, and typically focusing on high-impact events with a clear attributable signal.

However, since AR5, the attribution of extreme weather events has emerged as a growing field of climate research with an increasing body of literature (see series of supplements to the annual State of the Climate report (Peterson et al., 2012, 2013a; Herring et al., 2014, 2015, 2016, 2018), including the number of approaches to examining extreme events (described in Easterling et al., 2016; Otto, 2017; Stott et al., 2016))."

Excerpt from [4], p. 1540 (p. 28 of the PDF).

Section "11.2.3 Attribution of Extremes"

"The outcome of event attribution is dependent on the definition of the event [...], as well as the framing [...] and uncertainties in observations and modelling. Observational uncertainties arise in estimating the magnitude of an event as well as its rarity [...]. Results of attribution studies can also be very sensitive to the choice of climate variables [...]. Attribution statements are also dependent on the spatial [...] and temporal [...] extent of event definitions, as events of different scales involve different processes [...] and large-scale averages generally yield higher attributable changes in magnitude or probability due to the smoothing out of noise. In general, confidence in attribution statements for large-scale heat and lengthy extreme precipitation events have higher confidence than shorter and more localized events, such as extreme storms, an aspect also relevant for determining the emergence of signals in extremes or the confidence in projections [...].

The reliability of the representation of the event in question in the climate models used in a study is essential [...]. Extreme events characterized by atmospheric dynamics that stretch the capabilities of current-generation models [...] limit the applicability of the probability-based approach of event attribution."

Excerpt from [4], p. 1541 (p. 29 of the PDF). Cited literature: pp. 1706-1758

Section "11.2.3 Attribution of Extremes"

"The lack of model evaluation, in particular in early event attribution studies, has led to criticism of the emerging field of attribution science as a whole (Trenberth et al., 2015) and of individual studies (Angélil et al., 2017). In this regard, the storyline approach (Shepherd, 2016) provides an alternative option that does not depend on the model’s ability to represent the circulation reliably. In addition, several ways of quantifying statistical uncertainty (Paciorek et al., 2018) and model evaluation (Lott and Stott, 2016; Philip et al., 2018b, 2020) have been employed to evaluate the robustness of event attribution results. For the unconditional probability-based approach, multi-model and multi-approach (e.g., combining observational analyses and model experiments) methods have been used to improve the robustness of event attribution (Hauser et al., 2017; Otto et al., 2018a; Philip et al., 2018b, 2019, 2020; van Oldenborgh et al., 2018; Kew et al., 2019)."

Excerpt from [4], p. 1553 (p. 41 of the PDF).

Section "11.3.4 Detection and Attribution, Event Attribution"

"Local forcing may mask or enhance the warming effect of greenhouse gases [...]
Irrigation and crop intensification [...] lead to a cooling in some regions [...]
Deforestation has contributed about one third of the total warming of hot extremes in some mid-latitude regions [...]. Despite [...] larger uncertainties at the regional scale, nearly all studies demonstrated that human influence has contributed to an increase in the frequency or intensity of hot extremes and to a decrease in the frequency or intensity of cold extremes.

In summary, long-term changes in various aspects of long- and short-duration extreme temperatures, including intensity, frequency, and duration have been detected in observations and attributed to human influence at global and continental scales. It is extremely likely that human influence is the main contributor to the observed increase in the intensity and frequency of hot extremes and the observed decrease in the intensity and frequency of cold extremes on the global scale. It is very likely that this applies on continental scales as well. Some specific recent hot extreme events would have been extremely unlikely to occur without human influence on the climate system. Changes in aerosol concentrations have affected trends in hot extremes in some regions, with [...] aerosols leading to attenuated warming, in particular from 1950 to 1980."

Interesting paper: "Direction Augmentation in the Evaluation of Armed Conflict Predictions" by @johannes (Johannes Bracher), Lotta Rüter, Fabian Krüger, @sebastianlerch and Melanie Schienle
#Prediction #Forecasting #modelevaluation #conflictprediction
https://arxiv.org/abs/2304.12108

New open-access article on "Decomposition of the mean absolute error (MAE) into systematic and unsystematic components" in #PLOSONE, if you're into that sort of thing.
#Statistics #ModelEvaluation #ModelError
https://dx.plos.org/10.1371/journal.pone.0279774

Model Evaluation, Model Selection, and Algorithm
Selection in Machine Learning

#MachineLearning #ModelEvaluation #CrossValidation
#HyperparameterOptimization

https://arxiv.org/pdf/1811.12808.pdf

Anyway, I keep meaning to write up a blog post on “falsehoods I have believed about measuring model performance” touching on #AppliedML issues related to #modelEvaluation, #metrics, #monitoring, #observability, and #experiments (#RCTs). The cool kids would call this #AIAlignment in their VC pitch decks, but even us #NormCore ML engineers have to wrestle with how to measure and optimize the real-world impact of our models.

I'm thinking about how to relate #ModelBuilding and #ModelEvaluation to #OpenScience. I'm not there yet, but anyone who wants to think along, feel free! Model evaluation, thinking about #validity and how #standardization and #generalisation interact, among others...

Figure on #standardisation vs #generalisation from https://doi.org/10.1111/j.1601-183X.2010.00628.x

#ModelEvaluation

Client Info