just POSTed a 15GB file of CSV data (35mil rows) into a server with curl. it took 20GB of RAM but it worked
3
10
Replying to @stdlib
20GB of RAM on which side?

Nov 5, 2019 · 7:12 PM UTC

1
Replying to @gluegadget
curl to clickhouse. curl sits at ~20GB, CH ~4GB. Tbf curl is doing ok here; it has to buffer the whole (15GB) POST body, because I decompress in memory and let it hold the whole thing. this is a conscious trade off because I have a lot of RAM, but less NVMe for scratch space