| This document briefly outlines how the compilation with |
| [Kotlin/Native](https://kotlinlang.org/docs/native-overview.html) works in |
| terms of components and artifacts, including the cases with interoperability |
| with different languages. |
| |
| # Kotlin/Native compilation in Gradle |
| |
| This section describes how compilation with K/N from source to binary works in |
| Gradle for a single target. The approach is not really specific to Gradle, but |
| using Gradle in this description seems practical. |
| |
| By "Gradle" this document mostly means |
| "[Kotlin Gradle Plugin](../../libraries/tools/kotlin-gradle-plugin)". |
| |
| When compiling projects with multiple K/N targets, the build has to handle |
| common parts between these targets, which is not trivial. |
| This is out of scope for this document, and additionally involves things like |
| [commonizer](../../native/commonizer) and common metadata compilation. |
| |
| So, this document focuses on the single target case. Below is the chart: |
| |
| ```mermaid |
| --- |
| title: Compilation of a binary in project :B, which depends on project :A |
| --- |
| flowchart LR |
| subgraph Inputs |
| Asource[project :A source code] |
| Adeps[project :A klib dependencies] |
| Bsource[project :B source code] |
| Bdeps[project :B klib dependencies] |
| end |
| |
| Acompile{{:A:compileKotlin$Target}} |
| Bcompile{{:B:compileKotlin$Target}} |
| |
| Aklib[project :A klib] |
| Bklib[project :B klib] |
| |
| link{{:B:linkDebug...}} |
| |
| Asource -->|translated| Acompile |
| Adeps -.->|used| Acompile |
| |
| Acompile --> Aklib |
| |
| Aklib -.-> Bcompile |
| Bsource --> Bcompile |
| Bdeps -.-> Bcompile |
| |
| Bcompile --> Bklib |
| |
| Aklib --> link |
| Adeps --> link |
| Bklib --> link |
| Bdeps --> link |
| |
| link --> binary |
| ``` |
| |
| Any Kotlin sources are first compiled to a klib. |
| The `compileKotlin$Target` tasks do that. |
| |
| Klib is a very high-level representation of a Kotlin library. |
| It contains serialized [Kotlin metadata](../../core/metadata) and serialized |
| [Kotlin backend IR](../../compiler/ir/ir.tree) (resolved AST). |
| |
| Klibs are designed to have binary compatibility. They are publishable. In |
| particular, dependencies (e.g., those from Maven repositories) are also |
| represented as klibs. |
| |
| So, "compile" tasks don't perform an actual compilation to machine code or |
| LLVM IR. |
| Instead, "link" tasks do that. A "link" task takes a transitively closed set |
| of klibs as an input, and produces an actual native binary as an output. |
| |
| In addition to compiling klibs to machine code, a "link" task also compiles and |
| includes the K/N runtime, containing the implementation of e.g., garbage |
| collector. |
| |
| Both "compile" and "link" tasks delegate the whole compilation to the K/N |
| compiler. |
| In other words, any such task invokes the K/N compiler once. |
| They don't split the action into pieces delegating them to the compiler |
| one-by-one. |
| The entry point of the compiler can be found [here](../../native/cli-native). |
| |
| So the rest of this document will sometimes operate the terms "first/second |
| compilation stage", referring to the compilations from sources to klib and from |
| klibs to binary accordingly. |
| Performed by Gradle or the compiler. |
| |
| # Kinds of binaries |
| |
| The compiler can produce a few different kinds of binaries. |
| * **Executables** — native runnable executables, calling `fun main` upon |
| launch. |
| * **Test executables** — native runnable executables, running all the `@Test` |
| methods. They are used for the test tasks in Gradle. |
| * **Static and dynamic native libraries**. Those are designed to be consumed |
| from sources in different languages. For example, the compiler can produce a |
| library with C API or Objective-C API. In both cases the compiler takes care |
| of generating all the necessary bridging code. |
| |
| In the Objective-C case, the compiler emits a self-sufficient Objective-C |
| framework (i.e. the native library, the headers and additional metadata are |
| packed into a `.framework` bundle). |
| |
| The relevant compilation flags specifying what the compiler produces are: |
| `-produce`, `-generate-test-runner`, `-Xstatic-framework`. |
| The corresponding Gradle documentation is |
| [here](https://kotlinlang.org/docs/multiplatform-dsl-reference.html#binaries). |
| |
| # Closed-world compilation model |
| |
| As hinted in the compilation scheme overview, compilation of binaries follows |
| the closed-world model: a binary always comprises a transitively enclosed set |
| of klibs, including stdlib. |
| |
| Among other things, this means that two different binaries can't share any |
| code, e.g., in the form of a shared native library. |
| In other words, K/N binary is always an independent, self-contained closed |
| "world". |
| |
| # Compilation time optimization |
| |
| So, the second compilation stage (confusingly named "link" in Kotlin Gradle |
| Plugin) performs a global compilation, which might take a long time. |
| That's why the compiler has a few optimizations to that, reusing the machine |
| code produced by the previous invocations. |
| |
| It is worth mentioning that there are "debug" and "release" binaries. |
| The compilation time optimizations apply only to debug binaries. |
| That's because compiling release binaries involves massive global |
| optimizations for the generated code, which are incompatible with compiling |
| anything separately. |
| |
| So, in practice, producing the release binaries ("linkRelease" Gradle tasks) |
| takes significantly longer than the debug binaries, both because of heavier |
| compiler optimizations and lack of caching. |
| |
| ## "Compiler caches" |
| |
| As mentioned above, the compiler tries to reuse the machine code produced by |
| previous invocations when compiling debug binaries. This is called "compiler |
| caches" or "compilation caches". |
| |
| Under the hood, it works the following way. |
| |
| When the compiler compiles a set of klibs to a binary, it attempts to compile |
| project dependencies to separate chunks of native code, storing them under |
| `~/.konan`. So, those are fairly persistent, and thus can be reused across |
| compiler invocations. |
| |
| The entry point to this machinery can be found |
| [here](../../kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/CacheBuilder.kt). |
| |
| ### Cache format |
| |
| Such chunks of native code are represented as static libraries (`.a` files). |
| They are augmented with |
| [additional meta-information](../../kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/CacheStorage.kt) |
| (stored as separate files), for example, class field layouts. |
| |
| Without compiler caches, such information is typically deduced from the Kotlin |
| backend IR, but adding it to a cache allows the compiler skipping |
| deserialization of IR in "cached" klibs. |
| |
| ### Versioning the caches |
| |
| Within the caching implementation, the compiler additionally takes care of |
| properly accounting parameters affecting the compilation, e.g., compiler flags |
| or dependencies. |
| |
| For example, compiling with `-Xgc=cms` and `-Xgc=pmcs` produces different |
| machine code. |
| |
| Compiling klib2 against klib1 (its dependency) might produce different machine |
| code after changing the version of klib1 (even if those are binary compatible |
| with regard to Kotlin). |
| |
| In such cases, the compiler can distinguish incompatible compiler caches. |
| Basically, it stores such different cached versions of the same library into |
| different directories. |
| |
| This is all encapsulated in the compiler itself, as part of its operation |
| within the second compilation stage. |
| Implementing this in Gradle instead is possible, but would break encapsulation. |
| |
| ### Only project dependencies |
| |
| As briefly mentioned, the caching is applied only to dependencies, i.e., klibs |
| downloaded from remote (Maven) repositories, and not to klibs produced by |
| compiling project source code. |
| This is a deliberate design choice: |
| Dependencies do not change — a klib residing at some path on the local |
| filesystem will always stay the same for the same path. So, |
| * up-to-date checks are trivial |
| * no need for a machinery cleaning outdated compiler versions (caches) of |
| such modules. It would be useful even for project dependencies, but not as |
| necessary as with project modules (which change that often, so caching |
| without purging would lead to significant unlimited disk footprint). |
| |
| To handle project modules, there is a different (experimental) machinery — |
| "incremental compilation", covered in the next section. |
| |
| ## Incremental second compilation stage |
| |
| The compiler is |
| [able](https://kotlinlang.org/docs/whatsnew1920.html#incremental-compilation-of-klib-artifacts) |
| to reuse the machine code produced by the previous invocations even for the |
| project source code. |
| Only this time the compiler produces a chunk of machine code per source file, |
| not per klib. |
| |
| More specifically, the compiler compiles each Kotlin file to an individual |
| static library (`.a` file), and also stores the graph of dependencies between |
| files. On the next binary compilation, the compiler can check the set of |
| changed files, and recompile only these files and files depending on them. |
| |
| The up-to-date checks for files here are based not on source files as such, but |
| on "compiled files" files represented as Kotlin backend IR in klibs. |
| This decision serves at least two purposes: |
| * The second compilation stage operates klibs, not source files. So using the |
| klib-level representation is a matter of encapsulation. |
| * It allows handling subtle changes that don't affect the contents of file or |
| its dependencies, but affect the resolution and thus change the actual |
| compilation. |
| |
| The incremental compilation is tightly integrated with the compiler caches |
| under the hood: for example, a compilation can be incremental and use compiler |
| caches at the same time. |
| In fact, incremental compilation is implemented with the help of |
| "per-file caches" under the hood. |
| |
| The entry point to this machinery can be found |
| [here](../../kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/CacheBuilder.kt), |
| same as the compiler caches. |
| |
| # Interoperability |
| |
| Kotlin/Native provides a few ways to perform interaction between Kotlin and |
| other languages. |
| |
| ## Consuming C or Objective-C libraries in Kotlin, aka "cinterop" |
| |
| ### "cinterop klibs" |
| |
| Kotlin/Native |
| [allows consuming C](https://kotlinlang.org/docs/native-c-interop.html) and |
| [Objective-C libraries](https://kotlinlang.org/docs/native-objc-interop.html) |
| in Kotlin code. |
| To achieve that, there is a special tool — `cinterop`. |
| The entry point can be found [here](../../kotlin-native/Interop/StubGenerator). |
| It takes a C/Objective-C library headers (`.h`) as an input and produces |
| a special "cinterop klib" as an output. |
| |
| "Cinterop klib" differs from regular Kotlin klibs in some details. |
| For example, it contains Kotlin metadata, but not Kotlin backend IR. |
| Instead, it includes some other information for the compiler, enabling the |
| latter to actually generate machine code for native library use sites. |
| |
| The compiler can accept a cinterop klib as a dependency, just like any regular |
| Kotlin klib. |
| So, to import a C or Objective-C library to Kotlin, one needs to make a |
| cinterop klib and pass it to the compiler as a dependency. |
| |
| In Gradle, this is implemented in the following way: |
| * A `compilation` can define a `cinterops` description |
| * Under the hood, Gradle produces a cinterop klib based on such a description, |
| and passes it as a dependency to the corresponding compilation. |
| * Any other compilation depending on this compilation will also get this |
| cinterop klib as a dependency. That's why cinterop klibs are published along |
| with regular Kotlin klibs. |
| |
| See also https://kotlinlang.org/docs/multiplatform-dsl-reference.html#cinterops. |
| |
| ### Linking native libraries |
| |
| Typically, a C or Objective-C library consists of header files and a binary — |
| either static (`.a`) or dynamic (`.dylib`/`.so`/`.dll`) library. |
| |
| cinterop klibs cover the header part, which usually contains only the API. |
| The actual implementation of the library is located in the binary part, which |
| has to be linked to the binary the compiler produces. The following chart |
| illustrates the entire compilation flow: |
| |
| ```mermaid |
| flowchart LR |
| subgraph nativelib['foo' native library] |
| dylib[libfoo.dylib] |
| header[foo.h] |
| end |
| |
| sources[Kotlin sources] |
| klib |
| cinteropKlib[cinterop klib] |
| |
| compiler1{{compiler}} |
| subgraph compiler2[compiler encapsulates that] |
| codegen{{code generator}} |
| machineCode[.o] |
| linker{{native linker}} |
| end |
| |
| sources --> compiler1 |
| |
| header -->|cinterop| cinteropKlib |
| sources -.->|imports| cinteropKlib |
| cinteropKlib -.-> compiler1 |
| cinteropKlib --> codegen |
| |
| compiler1 --> klib |
| klib --> codegen |
| |
| codegen --> machineCode |
| machineCode --> linker |
| |
| dylib -.-> linker |
| |
| linker --> binary |
| |
| ``` |
| |
| ## Consuming Swift or C++ libraries in Kotlin |
| |
| The previous section describes how importing C and Objective-C to Kotlin works. |
| Importing Swift or C++ works only through Objective-C or C correspondingly. |
| |
| For example, when annotating Swift declarations with `@objc`, Swift compiler |
| can produce an Objective-C header file for such declarations. |
| This header file can then be (processed with `cinterop` and) imported to |
| Kotlin. |
| |
| Similar for C++ — one needs to wrap C++ declarations to C, and import C to |
| Kotlin. |
| |
| ## Consuming Kotlin libraries in Objective-C or Swift |
| |
| As mentioned above, the K/N compiler can produce an Objective-C framework, |
| which can then be imported to Objective-C or Swift code. |
| |
| The entry point to the implementation can be found |
| [here](../../kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/objcexport/ObjCExport.kt). |
| |
| For such a framework, it is possible to instruct the compiler which modules to |
| include to the framework API as a whole (declarations from the rest will be |
| added one-by-one, if used in other parts of the resulting API). |
| |
| ### Closed world strikes back |
| |
| The thing worth mentioning (repeating) here is |
| [the closed-world compilation model](#closed-world-compilation-model). |
| A compiled framework is a closed world. So, it always contains all the |
| dependencies. When working with multiple such frameworks, this might be |
| surprising. Consider the following case. |
| ```mermaid |
| flowchart RL |
| core["// project :core |
| class Data"] |
| feature1["// project :feature1 |
| fun produceData(): Data"] |
| feature2["// project :feature2 |
| fun consume(data: Data)"] |
| |
| feature1 -.->|depends| core |
| feature2 -.->|depends| core |
| |
| framework1["// framework Feature1 |
| @interface Feature1Data |
| + (Feature1Data*)produceData;"] |
| framework2["// framework Feature2 |
| @interface Feature2Data |
| + (void)consumeData:(Feature2Data*);"] |
| |
| framework1 -->|includes|feature1 |
| framework1 -->|includes|core |
| framework2 -->|includes|core |
| framework2 -->|includes|feature2 |
| |
| swift["// Swift code |
| consume(data: produceData()) |
| // ^ doesn't work, different types"] |
| |
| swift -.->|depends| framework1 |
| swift -.->|depends| framework2 |
| |
| ``` |
| In other words, both frameworks (`Feature1` and `Feature2`) contain project |
| `:core`, and its class `Data`. So Swift code ends up with two different |
| incompatible versions of the class: one from `Feature1`, one from `Feature2`. |
| |
| Since `consumeData` expects `Data` from `Feature2`, but `produceData` returns |
| `Data` from `Feature1`, `consume(data: produceData())` in Swift won't work. |
| |
| That's why the recommended approach is to use the "umbrella framework" |
| technique: |
| ```mermaid |
| flowchart RL |
| core[:core] |
| feature1[:feature1] |
| feature2[:feature2] |
| |
| framework[Framework] |
| |
| feature1 -.->|depends| core |
| framework -->|includes| feature1 |
| framework -->|includes| core |
| framework -->|includes| feature2 |
| feature2 -.->|depends| core |
| |
| swift[Swift code] |
| swift -.->|depends| framework |
| ``` |
| |
| Naturally, that poses a limitation for consuming K/N libraries in |
| Objective-C/Swift. In fact, it is never a single library, but a whole "world". |
| |
| See also |
| [here](https://www.jetbrains.com/help/kotlin-multiplatform-dev/multiplatform-project-configuration.html#several-shared-modules). |
| |
| ## Consuming Kotlin libraries in C/C++ |
| |
| Basically everything from the section |
| [above](#Consuming-Kotlin-libraries-in-Objective-C-or-Swift) applies to |
| [consuming Kotlin in C/C++](https://kotlinlang.org/docs/native-dynamic-libraries.html). |
| |
| The implementation can be found |
| [here](../../kotlin-native/backend.native/compiler/ir/backend.native/src/org/jetbrains/kotlin/backend/konan/cexport). |
| |
| ## Consuming Kotlin libraries in Swift directly, without Objective-C |
| |
| As mentioned above, the K/N compiler can produce an Objective-C framework, |
| which can be imported to Swift code. |
| |
| There is an ongoing development initiative for implementing this without |
| Objective-C in the middle. See [the documentation](../swift-export/README.md) |
| for more details. |