Gradle Plugin Integration Tests

This module contains integration tests for main libraries/tools/kotlin-gradle-plugin plugin and other Gradle subplugins (‘kapt’, ‘allopen’, etc...).

How to run

To run all tests for all Gradle plugins use check task.

More fine-grained test tasks exist covering different parts of Gradle plugins:

• kgpJvmTests - runs all tests for Kotlin Gradle Plugin/Jvm platform (parallel execution)
• kgpJsTests - runs all tests for Kotlin Gradle Plugin/Js platform (parallel execution)
• kgpAndroidTests - runs all tests for Kotlin Gradle Plugin/Android platform (parallel execution)
• kgpMppTests - run all tests for Kotlin Gradle Multiplatform plugin (parallel execution)
• kgpNativeTests - run all tests for Kotlin Gradle Plugin with K/N (parallel execution)
• kgpDaemonTests - runs all tests for Gradle and Kotlin daemons (sequential execution)
• kgpOtherTests - run all tests for support Gradle plugins, such as kapt, allopen, etc (parallel execution)
• kgpAllParallelTests - run all tests for all platforms except daemons tests (parallel execution)

Also, few deprecated tasks still exist until all tests will be migrated to the new setup:

• kgpSimpleTests - runs all migrated Kotlin Gradle Plugin tests (parallel execution)
• test - runs all tests with the oldest supported Gradle version (sequential execution)
• testAdvancedGradleVersion - runs all tests with the latest supported Gradle version (sequential execution)

The old tests that use the Gradle plugins DSL (PluginsDslIT) also require the Gradle plugin marker artifacts to be installed:

./gradlew :kotlin-gradle-plugin:plugin-marker:install :kotlin-noarg:plugin-marker:install :kotlin-allopen:plugin-marker:install
./gradlew :kotlin-gradle-plugin-integration-tests:test

If you want to run only one test class, you need to append --tests flag with value of test class, which you want to run

./gradlew :kotlin-gradle-plugin-integration-tests:kgpAllParallelTests --tests <class-name-with-package>

How to Run Using Kotlin Native from Master

Currently, Kotlin Native from master involves three configurations: kgpMppTests, kgpNativeTests, and kgpOtherTests. Depending on your development environment, there are a few different ways you can run this.

• On Local Environment In the case of Local Environment builds, you have two options, which you can add in your local.properties file:
  • kotlin.native.enabled=true - this property adds building Kotlin Native full bundle step before Integration Tests, then this bundle will be used in the Integration Tests.
  • kotlin.native.local.distribution.for.tests.enabled=false - include this line if you need to disable running Integration Tests with Kotlin/Native from master, even when kotlin.native.enabled is set to true.
• On TeamCity In the case of TeamCity builds, no extra setting is necessary. The mentioned configurations depend on the full bundle, which stores its artifacts in the -DkonanDataDirForIntegrationTests directory. Also, you can specify the konan directory for test by providing -DkonanDataDirForIntegrationTests, for example:

 ./gradlew :kotlin-gradle-plugin-integration-tests:kgpNativeTests -DkonanDataDirForIntegrationTests=/tmp/.konan

How to work with the tests

Few rules you should follow while writing tests:

• All tests should be written using JUnit 5 platform.
• Consider writing tests for specific supported platform in plugin rather than for specific supported Gradle feature. For example, if you want to add some tests for Gradle build cache, add them in the related test suites for Kotlin/Jvm, Kotlin/Js, etc...
• Don't create one big test suite (class). Consider splitting tests into smaller suites. All tests are running in parallel (except daemon tests) and having small tests suites should improve overall tests running time.
• In tests consider using more specific tasks over general one. For example, use assemble instead of build when test does not need to also compile tests and run them. This should reduce test execution time.
• By default, tests are running with LogLevel.INFO log level. Don't set LogLevel.DEBUG unless it is really required. Debug log level produces a lot of output, that slows down test execution.
• Add @DisplayName(...) with meaningful description both for test class and methods inside. This will allow developers easier to understand what test is about.
• Add to test related Kotlin tag. For example for Kotlin/Jvm tests - @JvmGradlePluginTests. Other available tags are located nearby @JvmGradlePluginTests - check yourself what suites best for the test. You could add tag onto test suite once, but then all tests in test suite should be for the related tag. Preferably add tag for each test.
• Consider using Gradle Plugin DSL while adding new/modifying existing test projects.

Tests run using Gradle TestKit and may reuse already active Gradle TestKit daemon. Shared TestKit caches are located in ./.testKitDir directory. It is cleared on CI after test run is finished, but not locally. You could clean it locally by running cleanTestKitCache task.

How to debug Kotlin daemon

1. Create Remote JVM debug configuration in IDEA.
   1. Modify debug port to be 5005.
   2. In Debugger mode floating menu select Listen to remote JVM.
   3. (Optional) You can check Auto restart to automatically restart configuration after each debug session.
2. Specify correct debug port in build call arguments kotlinDaemonDebugPort = 5005.
3. Run newly created configuration in Debug mode and after that run test in simple Run mode.

Adding new test suite

Select appropriate tag annotation and add it to the test class, so it will be assigned to the related test task. Extend test class from KGPBaseTest.

For each test method add @GradleTest annotation and gradleVersion: GradleVersion method parameter. All tests annotated with @GradleTest are parameterized tests, where provided parameter is Gradle version. By default, test will receive minimal and latest supported Gradle versions. It is possible to modify/add additional Gradle versions by adding @GradleTestVersions annotation either to the whole suite or to the specific test method. Prefer using TestVersions to define required versions instead of writing them directly as String.

Use test DSL defined here to write actual test case:

project("someProject", gradleVersion) {
    build("assemble") {
        assertTasksExecuted(":compileKotlin")
    }
}

All test projects are located in resources/testProject directory. You could use existing test projects or add a new one. Test setup, on running the test, will automatically add new settings.gradle file or missing pluginsManagement { ... } block into existing file, so you could just use plugins without version in build scripts:

plugins {
    id "org.jetbrains.kotlin.jvm"
}

A bunch of additional useful assertions available to use, such as file assertions, output assertions and task assertions. If you want to add a new assertion, add as a reviewer someone from Kotlin build tools team.

Additional test helpers

• Whenever you need to test combination of different JDKs and Gradle versions - you could use @GradleWithJdkTest instead of @GradleTest. Then test method will receive requires JDKs as a second parameter:

@JdkVersions(version = [JavaVersion.VERSION_11, JavaVersion.VERSION_17])
@GradleWithJdkTest
fun someTest(
    gradleVersion: GradleVersion, 
    providedJdk: JdkVersions.ProvidedJdk
) {
    project("simple", gradleVersion, buildJdk = providedJdk.location) {
        build("assemble")
    }
}

• Whenever Android Gradle plugin different versions should be checked in the tests - it is possible to use @GradleAndroidTest annotation instead of @GradleTest. Test will receive additionally to Gradle version AGP version and required JDK version:

@AndroidTestVersions(additionalVersions = [TestVersions.AGP.AGP_42])
@GradleAndroidTest
fun someTest(
    gradleVersion: GradleVersion,
    agpVersion: String,
    jdkVersion: JdkVersions.ProvidedJdk
) {
    project(
        "simpleAndroid",
        gradleVersion,
        buildOptions = defaultBuildOptions.copy(androidVersion = agpVersion),
        buildJdk = jdkVersion.location
    ) {
        build("assembleDebug")
    }
}

• If you want to copy current state of the test project and play with it separately - you could use makeSnapshotTo(destinationPath) function.

Common test fixes

Test infrastructure adds following common fixes to all test projects:

• applies ‘org.jetbrains.kotlin.test.fixes.android’ plugin. If you are using custom settings.gradle or settings.gradle.kts content in the test project, you need to add this plugin into pluginManagement:

pluginManagement {
    repositories {
        mavenLocal()
    }

    val test_fixes_version: String by settings
    plugins {
       id("org.jetbrains.kotlin.test.fixes.android") version test_fixes_version
    }
}

pluginManagement {
    repositories {
        mavenLocal()
    }
    
    plugins {
       id "org.jetbrains.kotlin.test.fixes.android" version $test_fixes_version
    }
}

Deprecated tests setup

When you create a new test, figure out which Gradle versions it is supposed to run on. Then, when you instantiate a test project, specify one of:

• project("someProjectName", GradleVersionRequired.None) or just project("someProjectName") – the test can run on the whole range of the supported Gradle versions;
• project("someProjectName", GradleVersionRequired.AtLeast("X.Y")) – the test is supposed to run on Gradle version X.Y and newer (e.g. it tests integration with a Gradle feature that was released in version X.Y);
• project("someProjectName", GradleVersionRequired.Exact("X.Y")) – the test is supposed to run only with Gradle version X.Y (e.g. it tests a workaround for that version or records some special behavior that is not reproducible with newer versions).

:warning: When your tests target multiple Gradle versions, make sure they pass when run with both tasks test and testAdvanceGradleVersion (see above). In the IDE, you can modify a test run configuration to use a Gradle task other than test.

You can check a Gradle version that the test runs with using Project.testGradleVersionAtLeast("X.Y") and Project.testGradleVersionBelow("X.Y").

Since Gradle output layouts differ from version to version, you can access classes and resources output directories using the functions that adapt to the Gradle version that is used for each test:

• CompiledProject.kotlinClassesDir() with optional arguments for subproject and source set, and its Java counterpart CompiledProject.javaClassesDir() (note that Gradle versions below 4.0 use the same directory for both)

• Project.resourcesDir() (with optional arguments for subproject and source set) for the resources directory;

• Project.classesDir(), which is a general way to get the output directory for a specific subproject, source set, and language.