what’s the best way to format a date, with the month in full, for example, I get the value 01-01-2022 and I need to return 01JAN2022, I tried the following code but it didn’t work:
.withColumn(‘dt_ori’,upper(date_format(col("dt_ori"),"ddMMMyyyy")))
>Solution :
This is working as expected , based on the various Date Time patterns available within Spark 3.x
‘MMM’: Short textual representation in the standard form. The month
pattern should be a part of a date pattern not just a stand-alone
month except locales where there is no difference between stand and
stand-alone forms like in English.
Spark Version
- Spark Version – 3.1.2
- Java Version – 1.8.0_292 (AdoptOpenJDK)
- Scala Version – 2.12.10
Date Format
s = StringIO("""
ID,date_str
101,01-01-2022
""")
df = pd.read_csv(s,delimiter=',')
sparkDF = sql.createDataFrame(df)\
.withColumn('date_parsed',F.to_date(F.col('date_str'), 'dd-MM-yyyy'))\
.withColumn('date_formated',F.upper(F.date_format(F.col('date_parsed'), 'ddMMMyyyy')))\
.drop('date_str')
sparkDF.show()
+---+-----------+-------------+
| ID|date_parsed|date_formated|
+---+-----------+-------------+
|101| 2022-01-01| 01Jan2022|
+---+-----------+-------------+