pyspark.sql.DataFrameWriter.option#

DataFrameWriter.option(key, value)[source]#

Adds an output option for the underlying data source.

New in version 1.5.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

keystr: The key for the option to set.
value: The value for the option to set.

Examples

>>> spark.range(1).write.option("key", "value")
<...readwriter.DataFrameWriter object ...>

Specify the option ‘nullValue’ with writing a CSV file.

>>> import tempfile
>>> with tempfile.TemporaryDirectory(prefix="option") as d:
...     # Write a DataFrame into a CSV file with 'nullValue' option set to 'Hyukjin Kwon'.
...     df = spark.createDataFrame([(100, None)], "age INT, name STRING")
...     df.write.option("nullValue", "Hyukjin Kwon").mode("overwrite").format("csv").save(d)
...
...     # Read the CSV file as a DataFrame.
...     spark.read.schema(df.schema).format('csv').load(d).show()
+---+------------+
|age|        name|
+---+------------+
|100|Hyukjin Kwon|
+---+------------+