22.9 Partitioning

  • Splitting data into many files makes it easier to work with increasingly larger and larger files
  • May take trial and error to find what you think is best way to partition
  • Ideally, have a decent number of partitions (i.e., < 10,000 partitions) with each partitions neither too small (i.e., < 20MB) or large (i.e., > 2GB)