Gopi Krishna Tummala

Tag: data-engineering

All the articles with the tag "data-engineering".

  • Advanced MLOps & Production
    30 MIN READ

    Beyond Parquet: How Lance and LanceDB Rebuild the Data Layer for Machine Learning

    Parquet was built for scans; ML needs random access, fat embedding columns, cheap column backfills, and versioned datasets. A practitioner's look at the Lance format and LanceDB: what the peer-reviewable numbers actually show, where the hype outruns the evidence, and how it changes the dataloader and RAG designs from my earlier posts.

  • Advanced MLOps & Production
    40 MIN READ

    Datasets & Dataloaders: The Art of Never Starving Your GPU

    GPU utilization is a lagging indicator — the real battle is in the data pipeline. A practitioner's deep dive into PyTorch DataLoader internals, zero-copy data pumps, WebDataset streaming, and the exact questions this gets you in ML system design interviews.