Metaflow

All integration guides. This guide reads SQL rows through Flight SQL and writes a separate output table, then verifies it through a new SELECT.

Install

Start the server and sample table. From the repository root, create a separate environment:

python3.12 -m venv .venv-metaflow
. .venv-metaflow/bin/activate
python -m pip install -r requirements/integrations/orchestrators.txt

The profile pins the November 3 dependency versions. The examples default to grpc://localhost:8815 without client TLS/auth; use FLIGHT_URI for another reachable local development endpoint. Use the connection reference for secure deployment configuration.

Read, write and verify

Save as metaflow_example.py, set METAFLOW_DEFAULT_DATASTORE=local and METAFLOW_DEFAULT_METADATA=local, then run python metaflow_example.py run. If the local runtime cannot identify a user, set METAFLOW_USER to your username.

from metaflow import FlowSpec, step

class FlightFlow(FlowSpec):
    @step
    def start(self):
        import os
        import uuid
        import pyarrow as pa
        import adbc_driver_flightsql.dbapi as sql

        uri = os.environ.get("FLIGHT_URI", "grpc://localhost:8815")
        with sql.connect(uri, autocommit=True) as conn, conn.cursor() as cur:
            cur.execute("SELECT id, label, amount FROM flight_tool_sample ORDER BY id")
            source = cur.fetch_arrow_table()
        output = pa.table({"id": source["id"],
                           "amount": [value * 2 for value in source["amount"].to_pylist()]})

        # Write the tool's output through Flight SQL, then verify the stored values.
        name = "guide_" + uuid.uuid4().hex
        with sql.connect(uri, autocommit=True) as conn, conn.cursor() as cur:
            try:
                cur.adbc_ingest(name, output, mode="create")
                cur.execute(f'SELECT * FROM "{name}" ORDER BY id')
                received = cur.fetch_arrow_table()
                assert received.equals(output)
                print(received)
            finally:
                cur.execute(f'DROP TABLE IF EXISTS "{name}"')
                cur.fetch_arrow_table()
        self.count = output.num_rows
        self.next(self.end)

    @step
    def end(self):
        assert self.count == 3
        print("Flight read/write verified")

if __name__ == "__main__":
    FlightFlow()

Use the local datastore and metadata service for this example; cloud execution and retry guarantees were not tested. Local .metaflow artifacts remain on disk. The connection is opened in the step, not serialized across steps.

Validation and cleanup

The live guide recipe run passed this example's read, write, fresh read-back and cleanup. The report records runtime versions and local execution limits. SQL uses autocommit without transactions. The example drops its own unique output table; remove the shared input with the fixture cleanup after finishing the guides. The historical run does not establish Enterprise, TLS/auth or distributed execution qualification.