Skip to content

[SPARK-46158][PYTHON][PS] Support axis columns in DataFrame.xs#56284

Open
manhha2502 wants to merge 1 commit into
apache:masterfrom
manhha2502:SPARK-46158-xs-axis-columns
Open

[SPARK-46158][PYTHON][PS] Support axis columns in DataFrame.xs#56284
manhha2502 wants to merge 1 commit into
apache:masterfrom
manhha2502:SPARK-46158-xs-axis-columns

Conversation

@manhha2502
Copy link
Copy Markdown

What changes were proposed in this pull request?

This PR adds support for DataFrame.xs with axis=1 / axis="columns" in pandas API on Spark.

It supports selecting cross-sections from columns, including single-level columns, MultiIndex columns, and level-based selection by level number or level name.

Why are the changes needed?

DataFrame.xs currently supports index-axis selection, but column-axis selection raises NotImplementedError.

This improves compatibility with pandas and addresses SPARK-46158.

Does this PR introduce any user-facing change?

Yes.

Previously, DataFrame.xs(..., axis=1) raised NotImplementedError.

After this change, users can select column cross-sections, for example:

psdf.xs("metrics", axis=1)
psdf.xs("metrics", axis="columns")
psdf.xs("num_legs", axis=1, level="feature")

How was this patch tested?

Added positive and negative test cases in FrameIndexingMixin.test_xs covering axis=1, axis="columns", MultiIndex columns, level-based selection, missing column keys, and invalid column levels.

Also ran:

python -m py_compile python/pyspark/pandas/frame.py python/pyspark/pandas/tests/indexes/test_indexing.py
git diff --check

I attempted to run the targeted unittest locally, but the source checkout has not built Spark jars yet.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

@manhha2502 manhha2502 force-pushed the SPARK-46158-xs-axis-columns branch 2 times, most recently from a3eef60 to a01852f Compare June 3, 2026 02:11
@manhha2502 manhha2502 force-pushed the SPARK-46158-xs-axis-columns branch from a01852f to 7da11c0 Compare June 3, 2026 02:57
@HyukjinKwon
Copy link
Copy Markdown
Member

cc @devin-petersohn fyi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants