Big Data news and engineering summaries

Polars 2.0 brings default streaming collection for LazyFrame, initial out-of-core spill-to-disk support, major optimizer and engine improvements, first-class SQL, a new Arrow MapType-based Map dtype, and stricter dtype validation. Its SQL benchmarks on TPC-H/TPC-DS data reportedly outperform DuckDB and DataFusion, though the results are not official benchmark results. Streaming improves memory use but may change row order; spilling starts around 80% RAM with a 64GB disk budget. Strict schemas enable earlier errors and faster iteration.

F3 is a next-generation open-source columnar data file format designed for efficiency, extensibility, and interoperability. It addresses layout limitations in formats like Parquet by embedding Wasm decoders within files, ensuring future-proof compatibility and platform-agnostic performance. The project provides a research prototype and API for developers, enabling adaptable data encoding schemes for evolving analytics workloads.

Google DeepMind’s AlphaGenome Atlas precomputes AlphaGenome’s predictions for all 9 billion possible single-nucleotide variants across human DNA, creating a 1-petabyte searchable dataset. Its AlphaGenome Variant Impact (AVI) score combines coding and non-coding effects, helping researchers prioritize variants. Early applications supported a rare-disease diagnosis and revealed 22% more non-coding associations in UK Biobank data.

An extensive analysis of 2025 US tariffs reveals a significant economic impact on the domestic economy, characterizing the trade policy as a self-inflicted wound. By examining shipment-level data involving over 25 million transactions valued at 4 trillion dollars, researchers found that American importers and consumers bear approximately 96% of the tariff costs. In contrast, foreign exporters absorbed only 4% of the burden. Events involving specific tariff shocks on Brazil and India demonstrate that export prices remained stable while trade volumes collapsed, indicating that international suppliers chose to reduce shipments rather than lowering prices to offset the taxes. This resulted in a 200 billion dollar surge in US customs revenue, effectively acting as a massive tax paid by American citizens and businesses rather than foreign entities.

China's Dongfang Suanxin is tackling the AI memory wall by prioritizing high memory bandwidth over chip node miniaturization. Using 14nm DF1000 and DF2000 chips with 3D stacked DRAM architectures and 3.5D Infinity Chiplet layouts, their TY64 SuperNode systems aim to outperform NVIDIA's GB200 NVL72 in memory throughput to maximize AI workload efficiency.

A recent investigate report reveals that Palantir is developing a sophisticated surveillance tool for Immigration and Customs Enforcement (ICE) called ELITE (Enhanced Leads Identification & Targeting for Enforcement). This application utilizes advanced analytics and geospatial mapping to identify potential deportation targets, providing agents with detailed dossiers and 'address confidence scores' based on data from various government and commercial sources. Internal records indicate that the tool helps ICE supervisors coordinate field operations by analyzing population density and targeting specific individuals or groups. This technological infrastructure marks a significant expansion in the use of high-tech and big-data solutions for real-time immigration enforcement and mass surveillance activities.

This project highlights the discovery of the world's longest lines of sight using a custom-developed algorithm named CacheTVS. By exhaustively analyzing topographical data across the planet, researchers identified the longest possible direct view: a 530km span from the Hindu Kush to Pik Dankova. Other notable mentions include a 504km line from Antioquia to Pico Cristobal in Colombia and a 483km view from Mount Elbrus in Russia across the Black Sea to the Pontic Mountains in Turkey. The project provides an interactive map containing approximately 4.5 billion calculated lines of sight, showcasing how geography, atmospheric conditions, and advanced computational algorithms can uncover unique geographical records.

This article explores how standard command-line tools can significantly outperform complex distributed systems like Hadoop for certain data processing tasks. The author demonstrates that processing 1.75GB to 3.46GB of chess game data using Amazon Elastic Map Reduce took approximately 26 minutes, while a optimized shell pipeline achieved the same results in just 12 seconds on a single laptop. By leveraging tools such as cat, grep, awk, and xargs, the author constructs a stream-processing pipeline that utilizes multi-core parallelism and requires virtually no memory. The resulting solution is over 235 times faster than the Hadoop implementation, highlighting that 'Big Data' tools are often overkill for tasks measurable in gigabytes. This approach emphasizes the power of traditional Unix philosophies—parallel processing via pipes—over high-overhead distributed architectures for datasets that fit within local storage throughput limits.

Polars 2.0 release candidate introduces better defaults rather than major features. LazyFrame queries now use the streaming engine by default, improving memory usage and performance but not guaranteeing row order. The API is stricter, requiring explicit handling of type conversions, concatenation, and parsing, while clearer migration errors guide users through removed APIs and parameters.

This dataset provides a comprehensive, live-updated archive of Hacker News, encompassing every item submitted since 2006. Available in Parquet format, it allows researchers to easily query and analyze two decades of technical discussions, trends, and community dynamics using tools like DuckDB and the Hugging Face datasets library.

The GM-SEUS dataset, documenting solar installations across the U.S., has updated to version 2, now including over 3.4 million panels and a new rooftop array dataset. This analysis utilizes high-performance hardware, QGIS for visualization, and DuckDB with geospatial extensions for data processing and statistical evaluation of solar farm capacities and installation trends.

A recent report from 404 Media reveals the deep integration of Palantir technology within U.S. Immigration and Customs Enforcement (ICE) operations through a tool called ELITE. This application functions like a sophisticated map interface, allowing agents to identify high-density areas of potential deportation targets. The software provides detailed dossiers including photos, Alien Numbers, and specific addresses, accompanied by a 'confidence score' for locations. This system facilitates large-scale operations and raises significant ethical concerns regarding racial profiling, surveillance, and the role of tech workers in building tools for extrajudicial enforcement. The article highlights the contrast between the professional tech environment and the tangible human consequences of the software's deployment on the ground.

Netflix evolved its TimeSeries tiered storage from an Apache Kafka/Flink pipeline writing compacted Parquet to S3 into native Cassandra cold reads from existing S3 backups. Immutable Time Slices use ReadOnlyCompactionStrategy to align SSTables, while custom backup Memtables cache metadata locally and fetch data chunks on demand. An optional separate Cassandra cold cluster protects hot traffic. The system now stores over 15 PB, reduced disk usage from 80% to 7%, improved p90 latency by ~30%, and keeps p99 below 500 ms.

Apache Arrow celebrates its 10th anniversary, reflecting on its 2016 launch as a stable, language-agnostic standard for columnar data. Since its inception, the project has maintained exceptional backward compatibility, expanded to include native libraries for 12 languages, and fostered a massive ecosystem of subprojects like DataFusion, ADBC, and GeoArrow for high-performance data exchange.

DuckLake is an open lakehouse format and DuckDB extension that separates metadata from data: a catalog database stores metadata, while Parquet files store table data. It lets DuckDB attach DuckLake databases and use standard SQL to create, query, update, and evolve tables. Built-in capabilities include time travel and change data feeds, distinguishing it from a simple Parquet-based storage layer by providing integrated cataloging and versioned data operations.

The release of pandas 3.0.0 marks a significant milestone for the popular data manipulation library, introducing performance enhancements and API consistency. Key highlights include the adoption of a dedicated string data type by default, which improves type safety and memory efficiency by moving away from generic objects. Additionally, Copy-on-Write (CoW) is now the permanent default behavior, eliminating the notorious SettingWithCopyWarning and ensuring more predictable data modification patterns. The release also updates the default datetime resolution to microseconds to prevent out-of-bounds errors and introduces a new pd.col syntax for cleaner DataFrame assignments. Developers are encouraged to migrate cautiously by first updating to pandas 2.3 to identify deprecated features before moving to the 3.0 version.

Netflix rebuilt its Ads Event Publisher as a stateful Apache Flink streaming job to support spiky Live traffic, replacing synchronous Metadata Registry writes with a Kafka-based streaming join. The job normalizes and reroutes events, matches client telemetry with serving responses within 60 minutes, deduplicates revenue events, and uses Spark recovery for delayed arrivals. Deterministic rollout and audits achieved over 99.99% agreement, removed the ad-serving critical-path dependency, retired legacy components, and processed hundreds of thousands of events per second during load tests.

Redis 8.8 is now available, featuring a new 'array' data structure, a window counter rate limiter, and message NACKing for Streams. It delivers significant performance gains across various operations and adds subkey notifications for hashes, multiple time series aggregators, explicit JSON floating-point storage control, and a new COUNT aggregator for sorted sets.

AMD's MI355X GPU outperforms NVIDIA's B200 and B300 in performance-per-dollar when running the 2.8T parameter Kimi K3 model. High HBM capacity allows single-node deployment, overcoming NVIDIA's cross-node constraints. While AMD faces minor software hurdles in ROCm, targeted optimizations in speculative decoding and prefill kernels demonstrate that AMD is a highly competitive, cost-effective alternative for frontier LLMs.

DuckDB 2.0’s async I/O substantially accelerates unchanged queries over AWS S3 by overlapping downloads with Parquet decoding. On an M5 laptop, a query counting vote types in a 2.2 GB, 228-million-row Parquet file took 18.8 seconds in DuckDB 1.5.5 and 7.7 seconds in DuckDB 2.0 alpha. The file’s 2,268 row groups are prefetched by dedicated download threads while workers decode buffered data, keeping network and CPU active simultaneously. Results depend on network conditions.