This is exactly how I would describe my experience. When I talk to others about polars now I usually quickly mention its fast up front, but then mostly talk about the API, its composability, small surface area, etc. are really what make it great to work with. Having these same semantics backed by eager execution, query optimized lazy API, streaming engine, GPU engine, and now distributed auto-magical ephemeral boxes in the sky engine just make it that much better of a tool.

  df = (
    pl.scan_parquet('hf://datasets/minimaxir/mtg-embeddings/mtg_embeddings.parquet')
    .filter(
        pl.col("type").str.contains("Sorcery"),
        pl.col("manaCost").str.contains("B"),
    )
    .collect()

Polars is awesome to use, would highly recommend. Single node it is excellent at saturating CPUs, if you need to distribute the work put it in a Ray Actor with some POLARS_MAX_THREADS applied depending on how much it saturates a single node.

This was captured well in their company announcement blogpost [0]:

> A strict, consistent and composable API. Polars gives you the hangover up front and fails fast, making it very suitable for writing correct data pipelines.

[0] https://pola.rs/posts/company-announcement/

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

https://arxiv.org/abs/2402.00159

Config doesn't have to be very complicated, I've used this for a while.

  set -g mouse on                # allow mouse
  set -g history-limit 999999999 # unlimited history
  set -sg escape-time 0          # vim esc response faster

We ran into many unsolvable or 30m+ solvable envs with conda that mamba handled quickly.

The underlying solver can be used with conda directly as well, but I have not done that (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...)

Will definitely try it out.