24 Databases on New Hardware

non-volatile memory

  • low latency read/write like DRAM with persistent writes and large capacities like SSD
  • first mentioned in 1971
  • found by HP labs by accident in 2008
  • 3 category phase change ram, resistive ram, magnetoresistive ram (I’m not much into electronic things.)
  • 3 ways of using NVM(block addressable, it’s like a faster SSD. but slower than DRAM)
    1. use hardware managed NVM as a larger cache for DRAM. no SSD/HDD.
    2. use NVM alongside DRAM as a storage for logs. no SSD/HDD.
    3. use old style buffer pool with disk file system. use NVM directly alongside them.
  • byte-addressable NVM(not emerged yet)
  • more tillable issues on storage/recovery methods, synchronization
  • DBMS engine architecture optimization(not applicable now)
    1. in-place update. WAL + snapshot. VoltDB.
    2. copy on write. create shadow copy, no WAL, LMDB
    3. log structure. RocksDB
  • storage optimization(no duplicate data)
  • recoverty optimization(may save the effort of processing the log)

    GPU acceleration

  • suitable for workload that do not branch & do not block, like scanning with predicates
  • about speed
    • transfer between cpu and DRAM (ddr4) ~40GB/s
    • transfer between cpu and GPU RAM (PCIe bus)~ 16GB/s
    • NVLink between two GPU ~ 25GB/s
    • possible NVLink between GPU & CPU ~ 25GB/s
  • 3 ways of using GPUs
    • put the entire database in GPU’s RAM & run queries in there
    • a certain portion of data onto GPU, run queries using both CPU & GPU
    • streaming data to GPU, then get result from GPU

      Hardware Transactional Memory

  • using instructions to indicate txns, let hardware do concurrent control