/cc @mit-mit @lrhn @mkustermann @vsmenon

@grouma does this affect your teams?
@Hixie this mentions that it does not touch flutter.

@mraleph if we are targeting 3.2 this would most likely give users around 6 months to migrate.

I think it's a good idea that the Dart team provides the the low-level mutex and message-passing package, implemented using dart:ffi. Probably as a tools package.
That makes it possible for us to change the implementation in the future, if we, for example, stop using pthreads in the stand-alone VM.

I'd rather not have user-written Dart code depend on pthreads being available in the dart process.
In fact, we should probably say somewhere that we do not guarantee such things.

We should make sure the package is designed abstractly enough, so we don't bind ourselves to a particular mutex implementation. (And we should document that in the package, so future maintainers won't start providing more implementation specific primitives.)

LGTM.

@mraleph if we are targeting 3.2 this would most likely give users around 6 months to migrate.

I was somehow thinking that we only do stable release every 6 months, but it seems that target cadence is quarterly. I am happy to commit to removing it in 3.4 instead of 3.2 to give full year. Updated the issue text to reflect this timeline.

@grouma does this affect your teams?

Implicitly through sass but not directly. The sass impact is constrained to a single file.

This seems like a good path forward. Thanks, @mraleph.

To simplify downstream compatibility, it would be helpful to have at least one release where both waitFor and the new sync APIs (such as resolvePackageUriSync) are available. That is, if we introduce resolvePackageUriSync in 3.1, we shouldn't flag waitFor in that same release.

I'm also OK with not flagging it at all, but simply removing it after a couple of releases where we've had the migration options available.

This doesn't meet the bar of providing a viable alternative for existing use-cases. Even if this synchronous isolate package addresses Sass's needs—which is still totally hypothetical at this point—it definitely doesn't meet dcli's. You say "the usage of asynchronous dart:io methods can be replaced with synchronous counterparts", but there's no synchronous equivalent of many dart:io features, particularly running a process with interactive access to standard IO which dcli relies heavily on.

A viable alternative for this feature that addresses all its uses looks like fibers, for which there's a considerable amount of prior art of integrating it into evented systems like Dart. But if you're not ready to go there, please leave thing as they are as you said in #39390 (comment).

@nex3 Mitigation section provides a number of migration paths which together address all existing uses of waitFor. In my opinion primary migration path for dcli is to convert their code to use await, though other alternatives are also available.

there's no synchronous equivalent of many dart:io features, particularly running a process with interactive access to standard IO which dcli relies heavily on.

Unless I am misunderstanding something this is also addressed by dart:ffi and dart:isolate in the very same way you do it in SASS: simplest approach here is to spawn a worker isolate which uses asynchronous dart:io methods to run a process and handle IO and have this worker isolate communicate back to the main isolate using a synchronous channel built with dart:ffi.

A viable alternative for this feature that addresses all its uses looks like fibers

If we only had native to worry about we would probably already done fibers. Unfortunately JS throws a wrench in this: adding fibers means sacrificing portability.

My current thinking here is that the best we could do without sacrificing portability is to have fibers underneath async-await implementation (so that async-to-async calls become as cheap as normal calls), but that would not allow people to suspend execution across synchronous frames.

This is anyway tangential to the desire to remove waitFor and has different timelines: fibers (even as an implementation detail) might or might not come, but we already have all the tools for current users to migrate of waitFor.

I understand that you're eager to get rid of waitFor. It's definitely not the right abstraction for what it's trying to do—I agree with that completely. But please don't let that eagerness translate into putting users into an untenable situation. We all want to find the right alternative here.

"Mitigation" isn't the same thing as a "viable alternative", though. Have you discussed with @bsutton or other maintainers of tools using waitFor to see if they consider this viable? I don't want to put words in their mouth, but I'd be very surprised if they find the prospect of re-implementing and maintaining large dart:io APIs as isolate proxies to be a good solution.

waitFor() provides three important capabilities that are otherwise missing from Dart today:

1. Allowing async code to be passed as callbacks to synchronous APIs. This is essentially the Sass embedded compiler's use-case.

2. Providing synchronous access to dart:io APIs that are async-only, such as Process.start, HTTP, sockets, etc. This is dcli's primary use case.

3. Making it possible to share logic across synchronous and asynchronous APIs. Even when both APIs exist, there's no other way to write code that can operate either synchronously or asynchronously other than copying it and everything it transitively touches. This comes up occasionally for Google-internal scripting.

The proposed solution addresses 1 acceptably, but it doesn't address 2 without a huge amount of additional wrapper work on the part of users and it doesn't address 3 at all. All of these are exacerbated by the fact that "single-threaded" async code is much slower than the equivalent synchronous code even when there's no IO going on.

If we only had native to worry about we would probably already done fibers. Unfortunately JS throws a wrench in this: adding fibers means sacrificing portability.

My expectation would be that Fibers would be a VM-only feature, like waitFor() is today. Everything we're talking about—isolates, dart:io, dart:ffi—is already VM-only anyway, so it seems like it would make sense to give the VM special functionality for interacting with the event loop in more powerful ways.

we still have the problem that the linter doesn't cover all of
the 'you forgot to await x' scenarios which makes writing cli apps using an
async library dangerous.

Do you expect this to be provided by the Dart analyzer / linter or are open to accept external solutions as well?

I view this a fundamental requirement and believe it should be in the dart
linter and in the recommended set of lints in the lints package.

Sorry, but this reads as if your intentions are to get this rule in the linter and the core set, not to find any good option that unblocks the migration. If this is such a problem, why you're focused only on one specific solution?

Fair point. Now I better understand the reasons behind it, thank you.

@pq - do we have a pointer to the linter work in this area (unawaited async)?

@bsutton I'm curious to know -- is there a good list of sync APIs you'd need that might make it easier to migrate from waitFor?

That is, can you expand on @nex3's comment here?

Providing synchronous access to dart:io APIs that are async-only, such as Process.start, HTTP, sockets, etc. This is dcli's primary use case.

In my own package, I can drop waitFor altogether as soon as resolvePackageUriSync is exposed...

@bsutton, regarding lint support, would a combination of unawaited_futures and discarded_futures be sufficient? (Assuming false positives are addressed in the latter?)

I have spent time trying to understand implications for both https://github.com/sass/dart-sass-embedded and https://github.com/onepub-dev/dcli by eliminating waitFor myself.

dart-sass-embedded

Here is the branch that completely eliminates waitFor from dart-sass-embedded by switching it to an model where a new isolate is spawned for each incoming compilation request.

The main isolate works with stdin/stdout communication channel using asynchronous APIs and then dispatches compilations to worker isolates. When a worker isolate needs to call back to to host they send request back to the main isolate (using a normal SendPort) and suspends execution using native synchronisation primitives. Main isolate will then route response back to the worker isolate once it is received on the stdin.

I think this effectively repeats work that @nex3 and @ntkme have already done previously.

Unsurprisingly it works without any issues (all tests pass).

There is a bit too much serialization / deserialisation going on - which is probably wasteful (especially for large messages like compilation request / responses). This can be addressed by either changing protocol slightly or by changing how we deserialise these messages (protocol buffers allow you to peak inside them based on tag numbers).

My conclusion is that there are really no blockers for Sass here.

dcli

I have started going through dcli as well eliminating waitFor usage (here is snapshot of my changes). I have managed to mostly eliminate waitFor in the code with the exception of few large pieces and tests. The following patterns left unmigrated:

• Synchronous flushing of stdin and stderr.
• fetch operation which uses HttpClient
• hard lock in named_lock which uses binding to port as a locking primitive:
• process related API (RunnableProcess and Progress)

My current assessment is that for flush (first item) we could probably include builtin synchronous flushing API. The rest can be covered by a helper isolate + blocking synchronization primitives. I have included dcli_core/bin/sync_process.dart as an illustration of how one could run process iteractively and communicate with it synchronously. Though how exactly to refactor these APIs are better decided by the original author.

I have observed that the lion share of waitFor usage in dcli can be eliminated simply by using synchronous counterparts of dart:io asynchronous APIs. Using synchronous API also makes code considerably simpler in many places, when compared to using asynchronous counterparts and waitFor.

So my conclusion for dcli is that same as for Sass - there are no real blockers, though it would be nice to extend dart:io with synchronous flush.

Other uses

I have also used Github Code Search to search for other instances of waitFor and all that I have found is related to resolvePackageUri being async. (Plus a couple uses in code which has not been updated for 3+ years).

There is a bit too much serialization / deserialisation going on - which is probably wasteful (especially for large messages like compilation request / responses). This can be addressed by either changing protocol slightly or by changing how we deserialise these messages (protocol buffers allow you to peak inside them based on tag numbers).

Issue for addressing this inefficiency: sass/sass#3579

My conclusion is that there are really no blockers for Sass here.

That's fair, but I suggest to not rush setting timeline for removing waitFor. What we got so far is only a POC that works for linux, we still need to work on porting for Windows, abstract ffi logic as a package, infra and testing, etc. There are tons of todos if we decide to go down this route.

Deprecation can be declared early to suggest new user to not use a to-be removed feature, but obsoletion timeline should only be declared when the alternative solution is production ready. Declare an obsoletion when all the alternative solutions are still just POC is way too aggressive. If Dart team is committed to build the alternative that is needed for migration, I think it won't be too late to talk about obsoletion timeline after you have most developers successfully migrate off the deprecated feature.

What we got so far is only a POC that works for linux, we still need to work on porting for Windows, abstract ffi logic as a package, infra and testing, etc. There are tons of todos if we decide to go down this route.

POC from yesterday worked on any POSIX system (originally developed on Mac OS X, but will also work on Linux, iOS and Android). I spend ~1h today abstracting the code a bit and porting it to Windows - which was fairly trivial as well just as one would expect. All dart-sass-embedded tests pass there as well.

Deprecation can be declared early to suggest new user to not use a to-be removed feature, but obsoletion timeline should only be declared when the alternative solution is production ready.

The waitFor was deprecated for more than a year now and we have been talking about removing it for more than 3 years. That neither prevented people from using it, nor did it motivate Sass and DCli to migrate away from it.

My experiments demonstrate without any doubt that Dart already today provides all necessary tools for users to migrate off waitFor in a matter of days by rewriting their code. So I think it would be well within our policies to just remove waitFor in the next stable release. However, I acknowledge that these tools are fairly low-level and possibly too heavyweight for some use-cases (like Isolate.resolvePackageUri), so I am not going to propose such unpleasantly aggressive route and instead set obsoletion date to 1 year from now (3.4), which gives us all plenty of time to migrate.

We (the Sass and dcli teams) are trying to be patient and work with the Dart team to ensure that we can find a solution we're all happy with to move forward here. I think we've been quite clear about what the critical parts of the status quo are for us, and we've provided a number of possible solutions to move forward that would satisfy our needs. I don't feel like we're getting the same patience or care from you, @mraleph.

Unilaterally moving the goalposts from "when we have a viable alternative" to a drop-dead date is an aggressive, anti-collaborative tactic. It makes me, as a user, worry that my needs or even commitments made by the Dart team are not relevant at all to the decision-making that actually happens. It also makes me feel like the effort we've put into working with you in good faith to try and find a mutually satisfactory solution was wasted.

This isn't just an academic exercise. There are real people who rely on this software—sass-embedded alone gets 65k downloads a week, and it's part of Google's larger strategic plans with Angular. Creating it was only possible at all because waitFor() existed. Acting like we're naughty children for using it all this time, when the Dart team wasn't putting any effort into providing an alternative, is both unhelpful and insulting. Please: work with us like we're trying to work with you.

@nex3 - @mraleph is digging into the code to propose viable alternatives above. We'd all like reasonable off ramps for sass and dcli.

I don't think we need a general purpose alternative such as fibers to move forward. As you've noted, V8 chose to not go that way: https://sass-lang.com/blog/node-fibers-discontinued#the-death-of-fibers

@nex3 I am sorry you feel this way. I would like to understand which part of this you don't consider viable. When reading that code you have to assume that Mailbox has a slightly nicer API and lives together with underlying synchronisation primitives in a separate package maintained by the Dart team. My understanding, from discussions on #44804, is that this will be an acceptable compromise. If I missed something - please list these things.

I also encourage you to see timeline from our perspective:

• The consensus within the Dart team is that we want this experimental feature removed.
• We have agreed to postpone removal until we have some alternative, we have marked the feature as deprecated to encourage people to move away from it;
• We have waited;
• We have found an alternative;
• We have made PoCs which demonstrate that alternative indeed works (e.g. for Sass it passes all tests on Windows/Linux and Mac OS X);
• We have committed to package an alternative into a nice package which hides low-level details away from users which don't feel comfortable dealing with low-level code.
• We have committed to a generous 1 year removal timeline.

@bsutton One (simplest) solution for NamedLock problem would be to build your own waitFor out of Isolate.run and package:native_synchronization: you create an isolate which is responsible for managing ServerSocket instance and use blocking synchronization primitives in the main isolate to work with it.

Another solution (more involved one) could be to use FFI and implement named lock on top of platform specific named mutex APIs: CreateMutexW on Windows and sem_open (or PTHREAD_PROCESS_SHARED mutexes residing in shared memory) on POSIX OSes.

Hi, I'm using waitFor in a number of cloud build scripts. Could someone please point me in the direction of the best way to migrate this code?

    // testing cloud_functions_core
    final pccGet = waitFor<Process>(Process.start(
      'dart',
      ['test', '--chain-stack-traces'],
      mode: ProcessStartMode.inheritStdio,
      workingDirectory: '../cloud_functions_core',
      environment: {'DOCKER_BUILDKIT': '1'},
    ));
    if (waitFor<int>(pccGet.exitCode) != 0) {
      stderr.writeln('ERROR: Testing failed.');
      exit(1);
    }

Thanks

@jimmyff
Why are you not using Process.runSync?

@jimmyff Why are you not using Process.runSync?

Ah, now I feel silly -that's exactly when I needed. Sorry for noise!

The removal is complete.

@bsutton One (simplest) solution for NamedLock problem would be to build your own waitFor out of Isolate.run and package:native_synchronization: you create an isolate which is responsible for managing ServerSocket instance and use blocking synchronization primitives in the main isolate to work with it.

Another solution (more involved one) could be to use FFI and implement named lock on top of platform specific named mutex APIs: CreateMutexW on Windows and sem_open (or PTHREAD_PROCESS_SHARED mutexes residing in shared memory) on POSIX OSes.

@mraleph I'm trying to attempt exactly this!

Repository here:

https://github.com/open-runtime/native_named_locks/blob/aot_monorepo_compat_dart_ffi/lib%2Fwindows_named_lock.dart

I'd absolutely love some feedback as this is a first stab at an implementation and using some pretty new stuff that I haven't used before 😅 I come from the GRPC + Protobuf / Traditional Backend World - haven't done too much on Windows FFI but giving it a really solid effort & hope to share this out there and get it into DCLI at some point.

For unix it tries to use Flock and fctnl from the stdlibc package that Canonical just released on pub.dev.

It also tries to wrap things in an extra layer of protection with Mutex from native_synchronization.

Gentlefolk, I need some help as I've come to a grinding halt on converting dcli away from waitfor.

The problem is with implementing synchronous process execution.

@mraleph helped by providing the mailbox class which allows for sync data transfer between isolates.

The problem is with spawning an isolate to run the process.

Spawning an isolate is an async process but I can't use an await when spawning as that makes the call async.

I spawn the process - without awaiting it - and then immediately called the synchronous mailbox.take method.

The problem is that calling a sync method immediately after the spawn seems to stop the spawn from completing.
The result is the code essentially deadlocks.

pseudo code:

unawaited(Isolate.spawn(_startProcess, mailbox);

mailbox.take(); // code hangs here as spawn never completes

I'm hoping I've misses something because at the moment I don't see any way forward.

The good news is that the rest of the conversion is essentially complete except for some cleanup.

Hey @bsutton! Quick question here - is this what you were thinking about using named locks for?

I have been meaning to post an update but I have it implemented and passing on MacOS, Linux and Windows with unit tests that verify cross-process synchronization.

Apologies if this is unrelated.. can sync with you on the named locks back over on DCLI repo!

Regarding the above - do you need to pass the mailbox as a sendable?

I had to do something like that when I used it for the PR a little bit ago.

Link here

@tsavo-at-pieces no this isn't related to named locks. The issue isn't with passing the mailbox, I have that working.

But yes we should talk about named locks - I've been a little distracted over the last few days.

Okay solid - interesting situation here - are you able to share a git link to the problematic code? I'm wrapping up named locks tonight and can take a look at this as well (no promises but definitely happy to take a crack at it locally)!

I'm assuming the you're calling something like mailbox.put within the isolate before calling mailbox.take?

The code in question is here:

https://github.com/onepub-dev/dcli/blob/master/dcli/lib/src/process/process/process_in_isolate.dart

After spawning isolate the call to _connectSendPort makes the call to take.

I'm locally playing with some alternate patterns, but if you have any ideas I would be pleased to here them.

@bsutton okay solid - I believe named locks can actually help here with something like the following usage.

1. Create a NamedLock instance on the Main Thread and immediately lock it
2. Pass in the name of the NamedLock to the Isolate through the _startIsolate(settings, channel); call
3. Create a NamedLock instance within the Isolate Thread from the passed name
4. The Isolate then tries to acquire the NamedLock with a blocking NamedLock.acquire() call
5. Back on the main thread, right after the _startIsolate(settings, channel); call, we call proceed to call NamedLock.unlock() from the main thread allowing the isolate to successfully complete its NamedLock.acquire() call, lock the NamedLock within the Isolate and allow it to proceed and call mailbox.put()
6. Again on the main thread, we call NamedLock.acquire() ahead of our mailbox.take() can introduce a small sleep if needed but this main thread NamedLock.acquire() will block mailbox.take() until the isolate calls NamedLock.unlock()
7. Lastly on the Isolate side right after it calls mailbox.put() it proceeds to call NamedLock.unlock() or NamedLock.dispose() unblocking the main threads blocking NamedLock.acquire() preceding the call to mailbox.take()

Absolutely - looking forward to input from Dart Devs as well!

Last thing to note is that the NamedLocks implementation I've been working on has a native shared memory map/counter to count the number of threads/processes that have a NamedLocks instance with an unresolved NamedLocks.acquire() call. If the number of expected & unique threads calling NamedLock.acquire() is deterministic then perhaps we can leverage this to know?

I've raised this issue to create spawnSync as I think it is the only workable solution:

#55356

Awesome - good looks 🫡 Looking forward to hearing more from the Dart team!

We'll get there - it's always a bit frustrating in the moment but gonna be solid 🤝

appreciate you @bsutton!

Deleted my comment after further research since I can see you want to interact with the spawned process in a sync matter with stdin and stdout.