There's a lot talk about "ZeroDisk" infrastructure backed by S3. The pitch is "move your data from locally attached NVMe storage to S3 and your applications will scale easier and be more performant!"

Maybe I'm getting too old for this shit, but I swear to dog this is the 4th such cycle in my career:

1. NFS
2. iSCSI / Fibrechannel
3. Hadoop / HDFS
4. ZeroDisk with S3

Am I the only one that's like: "wait, move TBs of data to S3 from NVMe to increase performance? Are you high?"

It doesn't work, so you scale up. Now you're back to local NVMe "cache disks" running instances as expensive as the locally attached NVMe instances when you add those costs to your S3 bill. The performance is worse because of course it is.

It always comes back to the two hard problems in computer science: naming things, cache invalidation, and off-by-one errors. 😂

#zerodisk #s3 #hadoop #cache #datalake #GetOffMyLawn

Modern Database Architecture: #Cloud Storage Evolution 🚀

🔹 #ZeroDisk architecture leverages #AWS #S3 as primary storage, eliminating need for traditional disk management

🔸 #Database systems gain instant scalability and elasticity through separation of compute and storage

🔹 New approach enables three deployment options:
- Direct #S3 writes
- #S3Express as cache layer
- #Raft cluster with S3 backend

🔸 Major platforms adopting this architecture:
#Snowflake, #Clickhouse, #Neon, #TiDB

🔹 Trade-offs balance between latency, cost, and durability metrics

🔸 Benefits include serverless operation, instant failover, and unlimited storage capacity

https://avi.im/blag/2024/zero-disk-architecture/

#DevOps