In the world of data science, raw data serves as the foundation for generating actionable insights. However, managing, processing, and transforming this data into a usable format requires specialized tools.
#ApacheSpark #ApacheHadoop #Kafka #Airflow #DBT #Presto #SQL #Python #PySpark #Snowflake #Redshift #BigQuery #Kubernetes #Docker #Terraform #Databricks
Podcast "Techtiefen": Big Data auf Kubernetes mit Stackable
Für diese Podcast-Episode hat sich Nico Kreiling Lars Francke eingeladen. Sie schauen auf frühere Hadoop-Zeiten und moderne Datenverarbeitung.
#ApacheHadoop #BigData #CloudComputing #Containerisierung #Kubernetes #news
I hope we open source some of the #ApacheHadoop work we are doing because Dark Lord knows this is harder than it should be.
"⚠️ New Cyber Threat Alert: Apache Hadoop & Flink Under Siege 🔒"
Researchers at Aqua Nautilus have identified a new cyberattack targeting Apache Hadoop and Flink applications. Attackers exploit misconfigurations in the YARN's (Yet Another Resource Negotiator) Resource Manager to deploy malware and cryptocurrency miners. This stealth approach is a significant challenge for traditional security defenses, highlighting the need for robust configurations and monitoring in Apache applications.
Source: Aqua Security Blog
Tags: #Cybersecurity #ApacheHadoop #ApacheFlink #Malware #Rootkits #SecurityDefense #InfoSec 🚨🖥️🔐
Authors: Nitzan Yaakov & Assaf Morag
For those a little familiar with Cascading, in #java, it was originally designed to run on #ApacheHadoop, and then #ApacheTez, but it also has a local planner.
This lets developers create non-clustered data applications, without the Hadoop/Tez etc dependencies or runtime.
I've been using the local planner in production for over 5 years now.
But Parquet requires Hadoop libraries, and this is ok, there is a shim between the libraries that allow Parquet and S3AFileSystem to be used locally.
A little more color on this announcement..
https://fosstodon.org/@cwensel/110549001614086663
First, #ApacheParquet removed #Cascading support, so I had to splice the original source into Cascading. But the ParquetScheme didn't honor type information fully. So there is a new TypedParquetScheme that has native support for JSON and Timestamps.
Second, Parquet requires the #ApacheHadoop FileSystem, which means we get the wonderful S3A implementation. But we also get a 331MB jar dependency with the aws bundle.
