Apache Spark

All integration guides. This guide reads SQL rows through Flight SQL and writes a separate output table, then verifies it through a new SELECT.

Install

Start the server and sample table. From the repository root, create a separate environment:

python3.12 -m venv .venv-spark
. .venv-spark/bin/activate
python -m pip install -r requirements/integrations/spark.txt
make bin/flight-sql-jdbc-driver-19.0.0.jar
export FLIGHT_JDBC_JAR="$PWD/bin/flight-sql-jdbc-driver-19.0.0.jar"

The profile pins the November 3 dependency versions. The examples default to grpc://localhost:8815 without client TLS/auth; use FLIGHT_URI for another reachable local development endpoint. Use the connection reference for secure deployment configuration.

Read, write and verify

Save as spark_example.py and run python spark_example.py.

import os
import uuid
import pyarrow as pa
import adbc_driver_flightsql.dbapi as sql
from pyspark.sql import SparkSession
uri = os.environ.get("FLIGHT_URI", "grpc://localhost:8815")
jar = os.path.abspath(os.environ["FLIGHT_JDBC_JAR"])
spark = (SparkSession.builder.master("local[2]").appName("flight-guide")
    .config("spark.jars", jar)
    .config("spark.driver.extraJavaOptions", "--add-opens=java.base/java.nio=ALL-UNNAMED")
    .config("spark.executor.extraJavaOptions", "--add-opens=java.base/java.nio=ALL-UNNAMED")
    .getOrCreate())
try:
    frame = spark.read.jdbc(
        "jdbc:arrow-flight-sql://" + uri.removeprefix("grpc://") + "/?useEncryption=false",
        "flight_tool_sample", properties={"driver": "org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver"})
    frame.show()
    transformed = frame.selectExpr("id", "amount * 2 AS amount").orderBy("id")
    # Small local bridge: collect rows before writing through ADBC.
    output = pa.Table.from_pylist([row.asDict() for row in transformed.collect()])
    # Write the tool's output through Flight SQL, then verify the stored values.
    name = "guide_" + uuid.uuid4().hex
    with sql.connect(uri, autocommit=True) as conn, conn.cursor() as cur:
        try:
            cur.adbc_ingest(name, output, mode="create")
            cur.execute(f'SELECT * FROM "{name}" ORDER BY id')
            received = cur.fetch_arrow_table()
            assert received.equals(output)
            print(received)
        finally:
            cur.execute(f'DROP TABLE IF EXISTS "{name}"')
            cur.fetch_arrow_table()
finally:
    spark.stop()

Install a full JDK 21 and set JAVA_HOME. The recorded route validated Spark JDBC reads, including predicate partitions. This write example collects a small result in the driver and uses ADBC; it does not establish Spark DataFrameWriter.jdbc support or a distributed Flight SQL sink. Do not collect unbounded production data.

Validation and cleanup

The live guide recipe run passed this example's read, write, fresh read-back and cleanup. The report records runtime versions and local execution limits. SQL uses autocommit without transactions. The example drops its own unique output table; remove the shared input with the fixture cleanup after finishing the guides. The historical run does not establish Enterprise, TLS/auth or distributed execution qualification.