Add external reader support in pkl-gradle#1067
Closed
HT154 wants to merge 2 commits into
Closed
Conversation
HT154
commented
May 11, 2025
HT154
force-pushed
the
gradle-external-readers
branch
from
498201d to
fc0567e
Compare
bioball
reviewed
May 12, 2025
HT154
force-pushed
the
gradle-external-readers
branch
from
fc0567e to
a18074d
Compare
Member
|
@HT154 can we add some test coverage for this? I think this would involve moving Hint: the Groovy syntax for maps is externalModuleReaders = ["foo": new org.pkl.gradle.ExternalReader("bar")] |
HT154
force-pushed
the
gradle-external-readers
branch
5 times, most recently
from
e6b911e to
f6d3224
Compare
netvl
reviewed
Jun 24, 2025
Comment on lines
+148
to
+154
| @Nested | ||
| @Optional | ||
| public abstract MapProperty<String, ExternalReader> getExternalModuleReaders(); | ||
|
|
||
| @Nested | ||
| @Optional | ||
| public abstract MapProperty<String, ExternalReader> getExternalResourceReaders(); |
Contributor
There was a problem hiding this comment.
These should be marked as @Input properties, not @Nested. The @Nested annotation in Gradle is used to make Gradle process values of a map property as beans. All @Nested properties are treated as @Input, so it will work as is, but semantically values of this map are just data carriers without any annotations which do not require further processing, so I think that @Input would be more appropriate here.
Contributor
Author
There was a problem hiding this comment.
Thanks @netvl! Do you know of any documentation that covers these subtleties? I wasn't able to find anything when I searched.
Contributor
There was a problem hiding this comment.
This is all described in Gradle docs, but it may be hard to find, especially since Gradle docs constantly evolve. Most of it comes with experience with writing plugins and sifting through tons of github tickets in the Gradle repo :)
On this particular thing, I believe, the description is here: https://docs.gradle.org/current/userguide/incremental_build.html#sec:task_input_output_annotations
HT154
force-pushed
the
gradle-external-readers
branch
2 times, most recently
from
818061b to
5b2a4c8
Compare
netvl
reviewed
Aug 14, 2025
HT154
force-pushed
the
gradle-external-readers
branch
from
5b2a4c8 to
a1565c2
Compare
Contributor
Author
|
It looks like this hangs right now for reasons I'm not understanding. It's intermittent on *nix systems and seemingly consistent on Windows (but I don't have a system to test that locally). Will likely need a hand debugging this. |
HT154
force-pushed
the
gradle-external-readers
branch
from
a1565c2 to
cbff681
Compare
HT154
force-pushed
the
gradle-external-readers
branch
from
cbff681 to
30972cc
Compare
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 11, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 11, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 11, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 11, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 13, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
netvl
added a commit
to netvl/pkl
that referenced
this pull request
May 14, 2026
This PR adds support for configuring external module and resource readers in the Gradle plugin, for all tasks which may need it.
Currently, no external readers are passed to the CliBaseOptions instance which is constructed by every task. Therefore, the only way external readers can be used with Gradle is via PklProject definitions, which sometimes may be limiting. After this change, it will be possible to configure external readers for any tasks, including those which operate on individual modules and not the entire projects.
I'm using NamedDomainObjectContainer to store reader definitions. The name of each reader's definition is passed through into the CLI classes as the map key. Therefore, you can define an external reader like this:
```kotlin
val myPklTask by pkl.operationKind.creating {
val foo by externalModuleReaders.creating {
executable = "external-executable"
arguments = listOf("--bar", "baz")
}
}
```
and then use `foo` as the URI schema in the Pkl code:
```pkl
import "foo:a/b/c"
```
Properties of each external reader spec are defined as proper properties and are propagated as input dependencies to the task, so the caching and task dependency resolution behaves as expected.
I completely forgot that apple#1067 exists and reimplemented this from scratch. But my approach to the interface is slightly different and more in line with Gradle recommendations, IMO, and I really need this functionality so I intend to see this through. As in that PR, there are no tests which validate the actual invocation of the external command because creating one requires writing a command which does msgpack, which I'm unsure of how to do properly within this project.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test on [windows] please