Skip to content

Commit

Permalink
[SPARK-47098][INFRA] Migrate from AppVeyor to GitHub Actions for Spar…
Browse files Browse the repository at this point in the history
…kR tests on Windows

### What changes were proposed in this pull request?

This PR proposes to migrate from AppVeyor to GitHub Actions for SparkR tests on Windows.

### Why are the changes needed?

Reduce the tools we use for better maintenance.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

- [x] Tested in my fork

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45175 from HyukjinKwon/SPARK-47098.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
HyukjinKwon authored and dongjoon-hyun committed Feb 20, 2024
1 parent 8e82887 commit 6a56494
Show file tree
Hide file tree
Showing 8 changed files with 83 additions and 422 deletions.
1 change: 0 additions & 1 deletion .github/labeler.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ INFRA:
- changed-files:
- any-glob-to-any-file: [
'.github/**/*',
'appveyor.yml',
'tools/**/*',
'dev/create-release/**/*',
'.asf.yaml',
Expand Down
81 changes: 81 additions & 0 deletions .github/workflows/build_sparkr_window.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
name: "Build / SparkR-only (master, 4.3.2, windows-2019)"

on:
schedule:
- cron: '0 17 * * *'

jobs:
build:
name: "Build module: sparkr"
runs-on: windows-2019
timeout-minutes: 300
steps:
- name: Download winutils Hadoop binary
uses: actions/checkout@v4
with:
repository: cdarlint/winutils
- name: Move Hadoop winutil into home directory
run: |
Move-Item -Path hadoop-3.3.5 -Destination ~\
- name: Checkout Spark repository
uses: actions/checkout@v4
- name: Cache Maven local repository
uses: actions/cache@v4
with:
path: ~/.m2/repository
key: build-sparkr-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
build-sparkr-windows-maven-
- name: Install Java 17
uses: actions/setup-java@v4
with:
distribution: zulu
java-version: 17
- name: Install R 4.3.2
uses: r-lib/actions/setup-r@v2
with:
r-version: 4.3.2
- name: Install R dependencies
run: |
Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival', 'arrow', 'xml2'), repos='https://cloud.r-project.org/')"
Rscript -e "pkg_list <- as.data.frame(installed.packages()[,c(1, 3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]"
shell: cmd
- name: Build Spark
run: |
rem 1. '-Djna.nosys=true' is required to avoid kernel32.dll load failure.
rem See SPARK-28759.
rem 2. Ideally we should check the tests related to Hive in SparkR as well (SPARK-31745).
rem 3. setup-java installs Maven 3.8.7 but does not allow changing its version, so overwrite
rem Maven version as a workaround.
mvn -DskipTests -Psparkr -Djna.nosys=true package -Dmaven.version=3.8.7
shell: cmd
- name: Run SparkR tests
run: |
set HADOOP_HOME=%USERPROFILE%\hadoop-3.3.5
set PATH=%HADOOP_HOME%\bin;%PATH%
.\bin\spark-submit2.cmd --driver-java-options "-Dlog4j.configuration=file:///%CD:\=/%/R/log4j2.properties" --conf spark.hadoop.fs.defaultFS="file:///" R\pkg\tests\run-all.R
shell: cmd
env:
NOT_CRAN: true
# See SPARK-27848. Currently installing some dependent packages causes
# "(converted from warning) unable to identify current timezone 'C':" for an unknown reason.
# This environment variable works around to test SparkR against a higher version.
R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ and Structured Streaming for stream processing.
<https://spark.apache.org/>

[![GitHub Actions Build](https://github.com/apache/spark/actions/workflows/build_main.yml/badge.svg)](https://github.com/apache/spark/actions/workflows/build_main.yml)
[![AppVeyor Build](https://img.shields.io/appveyor/ci/ApacheSoftwareFoundation/spark/master.svg?style=plastic&logo=appveyor)](https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark)
[![PySpark Coverage](https://codecov.io/gh/apache/spark/branch/master/graph/badge.svg)](https://codecov.io/gh/apache/spark)
[![PyPI Downloads](https://static.pepy.tech/personalized-badge/pyspark?period=month&units=international_system&left_color=black&right_color=orange&left_text=PyPI%20downloads)](https://pypi.org/project/pyspark/)

Expand Down
75 changes: 0 additions & 75 deletions appveyor.yml

This file was deleted.

186 changes: 0 additions & 186 deletions dev/appveyor-guide.md

This file was deleted.

Loading

0 comments on commit 6a56494

Please sign in to comment.