Ever wonder what a HyperLogLog data structure is? (Who hasn’t!?) In our latest video, learn how Dragonfly implements this memory-efficient counter to track millions of unique users with just 1.5KB of memory. https://youtu.be/EIJbC9lxzts #DragonflyDB #HyperLogLog
https://youtu.be/EIJbC9lxzts

@bkastl Hm, feel you!

Arbeite durchaus in dem Bereich und war bisher immer ein großer Freund des Ethikrates.

Evtl. sollte sie eine Befürworterin des #HyperLogLog werden.

https://media.ccc.de/v/38c3-privacy-preserving-health-data-processing-is-possible

#38c3 #Patientenakte

Completed the First Assignment of #645 @CMUDB , Hyperloglog was an interesting data structure to learn about.

#hyperloglog #presto

Google 的 HyperLogLog++

算是接續昨天寫的「Redis 對 HyperLogLog 省空間的實作」，在 Redis 的 HyperLogLog 實作有提到 Google 的論文「HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm」，裡面提出了 HyperLogLog++ (HLL++)。

論文中 Google 提出來的改進主要有三個，第一個是用了 64-bit hash function：

5.1 Using a 64 Bit Hash Fu

https://blog.gslin.org/archives/2024/03/21/11709/google-%e7%9a%84-hyperloglog/

#Computer #Murmuring #Programming #algorithm #data #google #hll #hyperloglog #structure

Redis 對 HyperLogLog 省空間的實作

HyperLogLog (HLL) 是用統計方式解決 Count-distinct problem 的資料結構以及演算法，不要求完全正確，而是大概的數量。

演算法其實沒有很難懂，在 2007 年的原始論文「HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm」裡面可以讀到演算法是長這樣：

可以

https://blog.gslin.org/archives/2024/03/20/11705/redis-%e5%b0%8d-hyperloglog-%e7%9c%81%e7%a9%ba%e9%96%93%e7%9a%84%e5%af%a6%e4%bd%9c/

#Computer #Murmuring #Software #algorithm #count #data #distinct #hyperloglog #problem #redis #structure

Here is the recording of my lightning talk at #37c3:

https://media.ccc.de/v/37c3-lightningtalks-58040-more-privacy-for-the-ehds

#privacy #ehds #hyperloglog @bkastl @dleisegang

#HyperLogLog is super clever.

It can count any number of unique values in constant space (i.e. without storing the values) within a specified margin of error.

And HLLs can be merged to count unique number of values in both sets! So you can quickly count something like "unique number of requests per day", and combine these into "per month", and "per year", without storing a year worth of history.

Fantastic explanation of #HyperLogLog algorithm: https://www.youtube.com/watch?v=lJYufx0bfpw. What's great about this video is that it uses very basic concepts, so that even non-programmers will understand it. On the other hand, CS terms like hash functions or sorted sets are mentioned in fine print, so the video doesn't sound childish

#TIL about the #HyperLogLog algorithm and I think it's a damn brilliant way to estimate the number of unique elements of a potentially gargantuan set of items and only running in O(n) time and O(1) space. The fact that variants of the algorithm can be done in parallel makes it even more awesome!

https://youtu.be/lJYufx0bfpw

#algorithms #ComputerScience #SoME3 #mathematics #maths #statistics

Approximate COUNT(DISTINCT ...) in
#postgresql and #yugabytedb https://dev.to/yugabyte/approximate-count-distinct-in-yugabytedb-and-postgresql-with-hyperloglog-1i13… #hyperloglog #skipscan #analyze

Something a little different today on the channel: HyperLogLog!

It's one of my favorite algorithms, used to estimate cardinality of a set. Typically used in environments with very large datasets (spread across many servers in a cluster) where a true, accurate distinct count would be very expensive.

HLL uses a simple observation about coin flipping probabilities, and extends that to cardinality estimation. Really clever algo, and provides a very fast and compact datastructure with reasonably small errors (<2% across billions of unique elements, typically in just a few kb of memory).

https://www.youtube.com/watch?v=lJYufx0bfpw

#programming #algorithm #hyperloglog #cardinality #datastructures