Alex Cai

<p>In the previous post, I talked briefly about the journey we've taken to find the best way to get SSTable data loaded onto the GPU for data analytics. We looked at the existing methods of pulling data from the driver and manually converting it into a cuDF, then at using the existing Cassandra driver, and finally at working on a custom parser implementation, <em>sstable-to-arrow</em>, which will be the focus of this post. We&rsquo;ll go over its capabilities, limitations, and how to get started.</p>
<h2>Implementation details</h2>
<p><em>sstable-to-arrow</em> is written in C++17. It uses the <a href="https://kaitai.io/">Kaitai Struct</a> library to declaratively specify the layout of the SSTable files using YAML, a data serialization language. The Kaitai Struct compiler then compiles these Kaitai declarations into C++ classes, which can be included in the source code to actually parse the SSTables into data in memory. Then, it takes the data and converts each column in the table into an Arrow Vector. <em>sstable-to-arrow</em> can then ship the Arrow data to any client, where the data can be turned into a cuDF and be available for GPU analysis.</p>
<h2>Limitations</h2>
<ul>
<li><em>sstable-to-arrow</em> can only read one SSTable at a time. To handle multiple SSTables, the user must configure a cuDF for each SSTable and use the GPU to merge them based on last write wins semantics. <em>sstable-to-arrow</em> exposes internal cassandra timestamps and tombstone markers so that merging can be done at the cuDF layer.</li>
<li>Some data, including the names of the partition key and clustering columns, can't actually be deduced from the SSTable files since they require the schema to be stored in the system tables.</li>
<li>Cassandra stores data in memtables and commitlogs before flushing to SSTables, so analytics performed via only <em>sstable-to-arrow</em> will potentially be stale / not real-time.</li>
<li>Currently, the parser only supports files written by Cassandra OSS 3.11.</li>
</ul>
<ul>
<li>The system is set up to scan entire SSTables (not read specific partitions). More work will be needed if we ever do predicate pushdown.</li>
</ul>
<ul>
<li>The following <a href="https://docs.datastax.com/en/cql-oss/3.x/cql/cql_reference/cql_data_types_c.html">cql types</a> are not supported: <em>counter</em>, <em>frozen</em>, and user-defined types.</li>
<li><em>varint</em>s can only store up to 8 bytes. Attempting to read a table with larger <em>varint</em>s will crash.</li>
<li>The parser can only read tables with up to 64 columns.</li>
<li>The parser loads each SSTable into memory, so it is currently unable to handle large SSTables that exceed the machine&rsquo;s memory capacity.</li>
<li><em>decimal</em>s are converted into an 8-byte floating point value because neither C++ nor Arrow has native support for arbitrary-precision integers or decimals like the Java <em>BigInteger</em> or <em>BigDecimal</em> classes. This means that operations on decimal columns will use floating point arithmetic, which may be inexact.</li>
</ul>
<ul>
<li><em>set</em>s are treated as lists because Arrow has no equivalent of a set.</li>
</ul>
<h2>Roadmap and future developments</h2>
<p>The ultimate goal for this project is to have some form of a <em>read_sstable</em> function included in the RAPIDS ecosystem, similar to how <em>cudf.DataFrame.from_csv</em> currently works. Performance is also going to be a continuous area of development, and I'm currently looking into ways in which reading the SSTables can be further parallelized to make the most use of the GPU. I&rsquo;m also working on solving or improving the limitations addressed above, especially broadening support for different CQL types and enabling the program to handle large datasets.</p>
<h2>How to run</h2>
<p>You can run <em>sstable-to-arrow</em> using <a href="https://hub.docker.com/r/datastaxlabs/sstable-to-arrow">Docker</a>.</p>
<div class="code">
<div class="code__heading">Bash<button class="code__copy">Copy</button></div>
<div class="code__body">
<pre><code class="language-ini">
# -s flag loads and sends a sample Internet Of Things (IOT) database
docker run --rm -itp 9143:9143 --name sstable-to-arrow datastaxlabs/sstable-to-arrow -s
      </code></pre>
</div>
</div>
<p>This will listen for a connection on port 9143. It expects the client to send a message first, and then it will send data in the following format:</p>
<p>1. The number of Arrow Tables being transferred as an 8-byte big-endian unsigned integer</p>
<p>2. For each table:</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp;1. Its size in bytes as an 8-byte big-endian unsigned integer</p>
<p>&nbsp;&nbsp;&nbsp;&nbsp;2. The contents of the table in Arrow IPC Stream Format</p>
<p>You can then fetch the data from your port using any client. To get started with the sample Python client, follow these steps if your system does <strong>not</strong> support CUDA:</p>
<div class="code">
<div class="code__heading">Bash<button class="code__copy">Copy</button></div>
<div class="code__body">
<pre><code class="language-ini">
# 1. Clone the repository:
git clone https://github.com/datastax/sstable-to-arrow.git
# 2. Navigate to the client directory:
cd sstable-to-arrow/client
# 3. Create a new Python virtual environment:
python3 -m venv ./sstable_to_arrow_venv
# 4. Activate the virtual environment:
source ./sstable_to_arrow_venv/bin/activate
# 5. Make sure you have the latest version of pip:
pip3 install --upgrade pip
# 6. Install the requirements:
pip3 install -r requirements.txt
# 7. Start the sstable-to-arrow server in another terminal (see above)
# 8. Run the sample client
python3 no_cuda.py
      </code></pre>
</div>
</div>
<p>If your system does support CUDA, we recommend creating a conda environment with the following commands. You will also need to pass the -x flag when starting the <em>sstable-to-arrow</em> server above to turn all non-cudf-supported types into hex strings.</p>
<div class="code">
<div class="code__heading">Bash<button class="code__copy">Copy</button></div>
<div class="code__body">
<pre><code class="language-ini">
# 1. Clone the repository:
git clone https://github.com/datastax/sstable-to-arrow.git
# 2. Navigate to the client directory:
cd sstable-to-arrow/client
# 3. Create a new conda environment:
conda create -y -n sstable-to-arrow
# 4. Activate the virtual environment:
conda activate sstable-to-arrow
# 5. Install the requirements:
conda install -y -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql cudf pyarrow
# 6. Start the sstable-to-arrow server with the -x flag in another terminal (see above)
# 7. Run the sample client
python3 with_cuda.py
      </code></pre>
</div>
</div>
<p>To experiment with other datasets, you will need raw SSTable files on your machine. You can download sample IOT data at <a href="https://drive.google.com/drive/folders/1y-f6rRH3OfC8AvVTNuhcmvjihnaMWN4p?usp=sharing">this Google Drive folder</a>. You can also generate IOT data using the <a href="https://github.com/datastax/sstable-to-arrow/blob/main/test/generate-data"><em>generate-data</em></a> script in the repository, or you can manually create a table using CQL and the Cassandra Docker image (see the <a href="https://cassandra.apache.org/_/quickstart.html">Cassandra quickstart</a> for more info). Make sure to use Docker volumes to share the SSTable files with the container:</p>
<div class="code">
<div class="code__heading">Bash<button class="code__copy">Copy</button></div>
<div class="code__body">
<pre><code class="language-ini">
docker run --rm -itp 9143:9143 -v /absolute/path/to/sstable/directory:/mnt/sstables --name sstable-to-arrow datastaxlabs/sstable-to-arrow /mnt/sstables
      </code></pre>
</div>
</div>
<p>You can also pass the -h flag to get information about other options.</p>
<p>If you would like to build the project from source, follow the steps in the <a href="https://github.com/datastax/sstable-to-arrow">GitHub repository</a>.</p>
<h2>SSTable to Parquet</h2>
<p><em>sstable-to-arrow</em> is also able to save the SSTable data as a <a href="https://parquet.apache.org/">Parquet</a> file, a common format for storing columnar data. Again, it does not yet support deduplication, so it will simply output the sstable and all metadata to the given Parquet file.</p>
<p>You can run this by passing the <em>-p</em> flag followed by the path where you would like to store the Parquet file:</p>
<div class="code">
<div class="code__heading">Bash<button class="code__copy">Copy</button></div>
<div class="code__body">
<pre><code class="language-ini">
# will save the parquet file into the same directory as the sstable files
docker run --rm -it -v /absolute/path/to/sstable/directory:/mnt/sstables --name sstable-to-arrow datastaxlabs/sstable-to-arrow -p /mnt/sstables/data.parquet /mnt/sstables
      </code></pre>
</div>
</div>
<h2>Conclusion</h2>
<p><em>sstable-to-arrow</em> is an early but promising approach for leveraging cassandra data for analytics using GPUs. The project is available on <a href="https://github.com/datastax/sstable-to-arrow">GitHub</a> and accessible via <a href="https://hub.docker.com/repository/docker/datastaxlabs/sstable-to-arrow">Docker Hub</a> as an alpha release.</p>
<p>We will be holding a free online workshop which will go deeper into this project with hands-on examples in mid August! You can sign up <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.eventbrite.com_e_workshop-2Danalyzing-2Dcassandra-2Ddata-2Dwith-2Dgpus-2Dtickets-2D164294668777&amp;d=DwMFaQ&amp;c=adz96Xi0w1RHqtPMowiL2g&amp;r=gH-IK_otrGtjAo1mJrnEP1bXaeKATcXEiyjRwmQ7a4k&amp;m=pfTxAVvEOpWcSAFk5TqfQmelZlZRv11GjxQfUHGbMkQ&amp;s=J__a98te0yqKWnULu2ZBgut_4dqip8BBr_CPDwcIK9o&amp;e=">here</a> if you&rsquo;re interested.&nbsp;</p>
<p>If you&rsquo;re interested in trying out <em>sstable-to-arrow,</em> take a look at the second blog post in this two part series and feel free to reach out to <a href="mailto:seb@datastax.com">seb@datastax.com</a> with any feedback or questions.</p>

In the previous post, I talked briefly about the journey we've taken to find the best way to get SSTable data loaded onto the GPU for data analytics. We looked at the existing methods of pulling data from the driver and manually converting it into a cuDF, then at using the existing Cassandra driver, and finally at working on a custom parser implementation, 

, which will be the focus of this post. We’ll go over its capabilities, limitations, and how to get started.

 library to declaratively specify the layout of the SSTable files using YAML, a data serialization language. The Kaitai Struct compiler then compiles these Kaitai declarations into C++ classes, which can be included in the source code to actually parse the SSTables into data in memory. Then, it takes the data and converts each column in the table into an Arrow Vector. 

 can then ship the Arrow data to any client, where the data can be turned into a cuDF and be available for GPU analysis.

 can only read one SSTable at a time. To handle multiple SSTables, the user must configure a cuDF for each SSTable and use the GPU to merge them based on last write wins semantics. 

 exposes internal cassandra timestamps and tombstone markers so that merging can be done at the cuDF layer.

Some data, including the names of the partition key and clustering columns, can't actually be deduced from the SSTable files since they require the schema to be stored in the system tables.

Cassandra stores data in memtables and commitlogs before flushing to SSTables, so analytics performed via only 

 will potentially be stale / not real-time.

Currently, the parser only supports files written by Cassandra OSS 3.11.

The system is set up to scan entire SSTables (not read specific partitions). More work will be needed if we ever do predicate pushdown.

s can only store up to 8 bytes. Attempting to read a table with larger 

The parser can only read tables with up to 64 columns.

The parser loads each SSTable into memory, so it is currently unable to handle large SSTables that exceed the machine’s memory capacity.

s are converted into an 8-byte floating point value because neither C++ nor Arrow has native support for arbitrary-precision integers or decimals like the Java 

 classes. This means that operations on decimal columns will use floating point arithmetic, which may be inexact.

s are treated as lists because Arrow has no equivalent of a set.

The ultimate goal for this project is to have some form of a 

 function included in the RAPIDS ecosystem, similar to how 

 currently works. Performance is also going to be a continuous area of development, and I'm currently looking into ways in which reading the SSTables can be further parallelized to make the most use of the GPU. I’m also working on solving or improving the limitations addressed above, especially broadening support for different CQL types and enabling the program to handle large datasets.

 # -s flag loads and sends a sample Internet Of Things (IOT) database docker run --rm -itp 9143:9143 --name sstable-to-arrow datastaxlabs/sstable-to-arrow -s 

This will listen for a connection on port 9143. It expects the client to send a message first, and then it will send data in the following format:

1. The number of Arrow Tables being transferred as an 8-byte big-endian unsigned integer

1. Its size in bytes as an 8-byte big-endian unsigned integer

2. The contents of the table in Arrow IPC Stream Format

You can then fetch the data from your port using any client. To get started with the sample Python client, follow these steps if your system does 

 # 1. Clone the repository: git clone https://github.com/datastax/sstable-to-arrow.git # 2. Navigate to the client directory: cd sstable-to-arrow/client # 3. Create a new Python virtual environment: python3 -m venv ./sstable_to_arrow_venv # 4. Activate the virtual environment: source ./sstable_to_arrow_venv/bin/activate # 5. Make sure you have the latest version of pip: pip3 install --upgrade pip # 6. Install the requirements: pip3 install -r requirements.txt # 7. Start the sstable-to-arrow server in another terminal (see above) # 8. Run the sample client python3 no_cuda.py 

If your system does support CUDA, we recommend creating a conda environment with the following commands. You will also need to pass the -x flag when starting the 

 server above to turn all non-cudf-supported types into hex strings.

 # 1. Clone the repository: git clone https://github.com/datastax/sstable-to-arrow.git # 2. Navigate to the client directory: cd sstable-to-arrow/client # 3. Create a new conda environment: conda create -y -n sstable-to-arrow # 4. Activate the virtual environment: conda activate sstable-to-arrow # 5. Install the requirements: conda install -y -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql cudf pyarrow # 6. Start the sstable-to-arrow server with the -x flag in another terminal (see above) # 7. Run the sample client python3 with_cuda.py 

To experiment with other datasets, you will need raw SSTable files on your machine. You can download sample IOT data at 

. You can also generate IOT data using the 

 script in the repository, or you can manually create a table using CQL and the Cassandra Docker image (see the 

 for more info). Make sure to use Docker volumes to share the SSTable files with the container:

 docker run --rm -itp 9143:9143 -v /absolute/path/to/sstable/directory:/mnt/sstables --name sstable-to-arrow datastaxlabs/sstable-to-arrow /mnt/sstables 

You can also pass the -h flag to get information about other options.

If you would like to build the project from source, follow the steps in the 

 is also able to save the SSTable data as a 

 file, a common format for storing columnar data. Again, it does not yet support deduplication, so it will simply output the sstable and all metadata to the given Parquet file.

 flag followed by the path where you would like to store the Parquet file:

 # will save the parquet file into the same directory as the sstable files docker run --rm -it -v /absolute/path/to/sstable/directory:/mnt/sstables --name sstable-to-arrow datastaxlabs/sstable-to-arrow -p /mnt/sstables/data.parquet /mnt/sstables 

 is an early but promising approach for leveraging cassandra data for analytics using GPUs. The project is available on 

We will be holding a free online workshop which will go deeper into this project with hands-on examples in mid August! You can sign up 

 take a look at the second blog post in this two part series and feel free to reach out to 

Analyzing Cassandra Data using GPUs, Part 2

Alex Cai

Share

Share

Implementation details

Limitations

Roadmap and future developments

How to run

SSTable to Parquet

Conclusion

More Technology

How Winweb Built its AI Assistant with DataStax Astra DB and LangChain

Vercel + Astra DB: Get Data into Your GenAI Apps Fast

Simplifying Agent Development with Astra DB Connector for Vertex AI Search

Making Astra DB easier for MongoDB developers

One-stop Data API for Production GenAI