Investigate a Cylc UI within JupyterLab

[kinow]

Describe exactly what you would like to see in an upcoming release

From comments in this PR: #90

The pull request was closed, but there was a different approach, suggested by @bollwyvl, where we could actually have the Cylc UI within JupyterLab.

JupyterLab is extensible, and allows widgets/plugins to customize and manipulate the interface. As we already use JupyterHub for authentication and starting and managing the UI Server, perhaps that would be a good idea too.

Additional context

Pull requests welcome!

[kinow]

Did some experimentation with JupyterLab yesterday and posted to Element. Almost forgot to write here too, if it were not for @hjoliver reminding me (thanks! 👍 )

---

Went as far as I could with GScan in JupyterLab. It almost worked: https://github.com/kinow/cylc-gscan

issues:

• it's really hard to load Vue in React due to outdated libraries.
• The current code starts a new Vue app, which is more expensive (we don't want to start all over for each component/widget, sharing data would be problematic too)
• JupyterLab 3 is coming with build improvements. Developing with JupyterLab is quite annoying, having to re-run commands. There migth be a --watch command somewhere, but I haven't found it yet
• JupyterLab does not expose webpack and babel objects/configuration; with this we could modify the build of the application
• In the current code, I tried to load the template by compiling it then rendering, but it has complications with the runtime as we cannot modify webpack/babel. It would be really hard to get our Vue components to work in JupyterLab
• The simplest way might be to use React (i.e. rewrite components in React)
• Vuetify is for Vue, so our templates would need udpating for a Material Design with React
• Oh, there's no easy way to get our data store (Vuex) to work with Redux (I'm still not sure what options we have for data sharing between components & widgets in JupyterLab, there might be something useful/cool)
• I only did this work today because I had some spare time between tasks, and because this weekend I found a good repository with an example app (https://github.com/mkery/JupyterLab-CodeAnalysisDemo) via Twitter. This demo loads a sidebar, which is exactly what I had in mind for GScan (but it could be another kind of widget) - the interesting thing here is that you need to learn what classes you will extend, in order to have access to the sidepanel, or to have a listener to changes in the app, or to create widgets. The docs are not great, so the curve for learning is a little steep.
• Had a lot of TypeScript type errors, but it's similar to the way Mypy/PEP-484/PHP works, and some other parts of the TS type system are similar to Java. Still, it brings some challenges, like one library I wanted to use had no compatibility with Typescript (i.e. no @types/$package, nor shims, or anything to help the build chain to import the code
• I converted all the code (including GScan) to TypeScript, but some parts are hard-coded, like the user name (for the URL like /user/$name/subscriptions)

There are several benefits of moving to JupyterLab. I listed the issues I found, but they are not really blockers (if we have time), just things that would need to be reviewed/clarified if/when considering the move 👍️ (which would be really fun IMO 😬).

[bollwyvl]

Good to hear your updates! There will be a lot more information about JupyterLab 3 soon, but my take is: greenfields stuff should start in JupyterLab 3, as the end-user experience of pip install and it works finally makes it something that can be supported as-delivered, and won't be broken tomorrow by leftpad upgrading.

• it's really hard to load Vue in React due to outdated libraries.

What libraries in specific? You should generally get your own copy of your own stuff
if you specify it... React, lumino, etc. however, are, and will continue to be singletons.

React is not required for building new greenfields JupyterLab extensions... and the
contract that is required (lumino.Widget) is "here's a DOM node, have fun". Some of the existing extensions do use React at their interfaces (e.g. icons, status bar items), so you might be stuck there.

• The current code starts a new Vue app, which is more expensive (we don't want to start all over for each component/widget, sharing data would be problematic too)

I don't know what I don't know about vue, but if the relationship is 1-root-node-to-1-app, that might have to be the way

• JupyterLab 3 is coming with build improvements.

The "improvement" for users: there is no build. pip, conda or even bash will be able to
install a pre-built package. Unfortunately, lab 2 extensions won't work with lab 3 extension,
but that's what major revs are for, I guess.

For devs: the build occurs only in the scope of your package and core, and
emits a "federated module" (in webpack 5 terminology). It brings all its dependencies,
but deduplicates them at run time... sure, this is a little more expensive, and
the existing webpack 4 workflow is still available, but greenfeilds

Developing with JupyterLab is quite annoying, having to re-run commands. There migth be a --watch command somewhere, but I haven't found it yet

https://jupyterlab.readthedocs.io/en/stable/developer/extension_dev.html#extension-authoring

A fairly bog-standard lab 2 extension might have a workflow like this:

jlpm
jlpm build                          # this could be tsc + babel + webpack
jupyter labextension install .
jlpm watch                          # whatever you have to do to make your chain watch
jupyter lab --watch --debug         # in another terminal

While lab 3 will be:

jlpm
jlpm build
jupyter labextension build .
jupyter labextension develop --overwrite .
jupyter labextension watch .        # one terminal
jupyter lab --debug                 # another terminal, doesn't use node at all!

• JupyterLab does not expose webpack and babel objects/configuration; with this we could modify the build of the application

In JupyterLab 3, you'll have more flexibility, but the federation stuff has some pointy bits.

• In the current code, I tried to load the template by compiling it then rendering, but it has complications with the runtime as we cannot modify webpack/babel. It would be really hard to get our Vue components to work in JupyterLab

Again, don't know what I don't know, but if vue+webpack+babel can generate a "normal"
es6 module (e.g. into dist), it can be loaded, rather than forcing all the webpack
complexity on the end user's computer.

• The simplest way might be to use React (i.e. rewrite components in React)
• Vuetify is for Vue, so our templates would need udpating for a Material Design with React

🤷?

Oh, there's no easy way to get our data store (Vuex) to work with Redux (I'm still not sure what options we have for data sharing between components & widgets in JupyterLab, there might be something useful/cool)

JupyterLab doesn't use Redux, or any top-level data store. All of the existing
stuff in Lab core uses lumino signals and messages, which may give you the correct
resolution for what you need. Most of these signals will eminate from a IFooManager,
exported by other extensions, or from a Widget instance.

• I only did this work today because I had some spare time between tasks, and because this weekend I found a good repository with an example app (https://github.com/mkery/JupyterLab-CodeAnalysisDemo) via Twitter. This demo loads a sidebar, which is exactly what I had in mind for GScan (but it could be another kind of widget) - the interesting thing here is that you need to learn what classes you will extend, in order to have access to the sidepanel, or to have a listener to changes in the app, or to create widgets. The docs are not great, so the curve for learning is a little steep.

Aside from the above link, there's:

• https://github.com/jupyterlab/extension-examples
• https://github.com/jupyterlab/extension-cookiecutter-ts

Both of the above have WIP JupyterLab 3 variants.

So reading source of extensions and/or core extensions that do something like
what you want will help with finding the right APIs. Having it checked out,
and loaded in an editor that can do reference jumping is clutch.

• Had a lot of TypeScript type errors, but it's similar to the way Mypy/PEP-484/PHP works, and some other parts of the TS type system are similar to Java. Still, it brings some challenges, like one library I wanted to use had no compatibility with Typescript (i.e. no @types/$package, nor shims, or anything to help the build chain to import the code

TypeScript is not required for JupyterLab. Just get to a bog-standard lib/index.js or whatever.

JupyterLab uses TypeScript extensively, as it makes finding API regressions a lot easier, and
the editor support more reasonable.

If you come across a library that doesn't have typescript, you can create an empty
declaration file, e.g. foo.d.ts

declare module 'foo.d.ts' {}

And it will import as any... you can iteratively add more robustness: a little
goes a long way, as often you might just be calling one or two methods.

• I converted all the code (including GScan) to TypeScript,

A couple years in: TypeScript is going to reward the maintainers of a project. It might
be a turn-off for casual contributors that don't know TypeScript, and users don't care at all,
or about react, redux, vue, revue, vuetify, unless it helps you ship better software faster,
but they might care about material design vs... whatever JupyterLab is.

... but some parts are hard-coded, like the user name (for the URL like /user/$name/subscriptions)

The PageConfig object has baseUrl, which you can embed. Unless this is some vue thing that is magic and special...

• There are several benefits of moving to JupyterLab. I listed the issues I found, but they are not really blockers (if we have time), just things that would need to be reviewed/clarified if/when considering the move +1 (which would be really fun IMO grimacing).

If your pieces are small (or at least self-contained), and fit nicely into UI concepts already in JupyterLab (e.g. a sidebar, a menu, a main area) it can be pretty nice. Also, if you leave extension points (exposed on your top-level plugin), others can extend your stuff.

[bollwyvl]

Sorry for the wall of text above, but good luck! Let us know how we can do a better job of enabling the kinds of cool features you want to deliver to your users!

[hjoliver]

Thanks @bollwyvl - that will be super helpful to us 👍

[kinow]

Hi @bollwyvl !

Sorry for the wall of text above, but good luck! Let us know how we can do a better job of enabling the kinds of cool features you want to deliver to your users!

That was super helpful (again!).

it's really hard to load Vue in React due to outdated libraries.

What libraries in specific? You should generally get your own copy of your own stuff if you specify it... React, lumino, etc. however, are, and will continue to be singletons.

It could be that I misunderstood how JupyterLab works. I saw React in the transitive dependencies, and assumed JupyterLab was a React application? The example I was using was also using React, so I was trying to load a Vue component, without having to create a Vue application for the cylc-gscan project.

We have other components that would need to be converted into Lumino widgets too, so I thought the easiest way would be to leave the React app control everything, hold the data, and pass it down to Vue components.

Following this train of thought, I decided to try libraries that integrate React and Lumino, like Vuera - https://github.com/akxcv/vuera. But regardless of having quite a few stars, it didn't work following their docs for installation without babel.

React is not required for building new greenfields JupyterLab extensions... and the
contract that is required (lumino.Widget) is "here's a DOM node, have fun". Some of the existing extensions do use React at their interfaces (e.g. icons, status bar items), so you might be stuck there.

That should be easier then!

Q: is there a way to initialize a Vue application once during the initialization of JupyterLab JS/TS application? This would be easier for us to transition, if I had an initialization hook where I could load a Vue app.

function jupyterlabInit(domElement, options) {
    # create a Vue app, loading Vuetify for CSS components and Vuex for the central store
    new Vue({
       el: domElement,
       ...options
    })
}

After that the widgets for "Tree" and "Graph" etc for Cylc workflows would just need to be kept in sync with the Vue components (which we are doing already with code similar to this - https://github.com/tupilabs/vue-lumino).

JupyterLab does not expose webpack and babel objects/configuration; with this we could modify the build of the application

In JupyterLab 3, you'll have more flexibility, but the federation stuff has some pointy bits.

All good. Vue provides ways to customize webpack and babel, but it's not super straightforward. Devs are given a chance to include/delete/edit parts of the chain of execution in webpack/babel. Which doesn't always work, but is good enough for enabling features, loading new plugins, etc.

From what I saw in some open issues in jupyterlab looks like it will solve the issues I had once it's avaialble/fixed.

I only did this work today because I had some spare time between tasks, and because this weekend I found a good repository with an example app (https://github.com/mkery/JupyterLab-CodeAnalysisDemo) via Twitter. This demo loads a sidebar, which is exactly what I had in mind for GScan (but it could be another kind of widget) - the interesting thing here is that you need to learn what classes you will extend, in order to have access to the sidepanel, or to have a listener to changes in the app, or to create widgets. The docs are not great, so the curve for learning is a little steep.

Aside from the above link, there's:

https://github.com/jupyterlab/extension-examples
https://github.com/jupyterlab/extension-cookiecutter-ts

Both of the above have WIP JupyterLab 3 variants.

I had seen the cookiecutter (I think I used it to create cylc-gscan repo), but don't recall seeing https://github.com/jupyterlab/extension-examples! Lots of useful examples, thanks!

So reading source of extensions and/or core extensions that do something like what you want will help with finding the right APIs. Having it checked out, and loaded in an editor that can do reference jumping is clutch.

👍 that's exactly how I learned Jenkins extension points. By finding the plugins doing what I wanted, and reading the source code.

Had a lot of TypeScript type errors, but it's similar to the way Mypy/PEP-484/PHP works, and some other parts of the TS type system are similar to Java. Still, it brings some challenges, like one library I wanted to use had no compatibility with Typescript (i.e. no @types/$package, nor shims, or anything to help the build chain to import the code

TypeScript is not required for JupyterLab. Just get to a bog-standard lib/index.js or whatever.

Ah! I can do that? I thought it had a hard requirement in the extension being written in TS! It will be much easier to start with JS.

JupyterLab uses TypeScript extensively, as it makes finding API regressions a lot easier, and the editor support more reasonable.

👍 I spent most of my time as dev writing Java, and later jumped to PHP/Py/JS, so I'm definitely +1 for TS, type checking, etc, for less bugs.

If you come across a library that doesn't have typescript, you can create an empty declaration file, e.g. foo.d.ts

declare module 'foo.d.ts' {}

And it will import as any... you can iteratively add more robustness: a little goes a long way, as often you might just be calling one or two methods.

I kind of understood that part from past experiments. But I still get confused whenever I see these .d.ts files. Mainly because I need to understand more of how they work to feel confident I understand and can use them.

I added one for Vue after I had a build error, but I got that from a StackOverflow I believe - https://github.com/kinow/cylc-gscan/blob/3c14245623dd0552f48020fb852b603d7f882203/src/vue-shim.d.ts

Vuetify is for Vue, so our templates would need udpating for a Material Design with React

Sorry, this was me thinking that JupyterLab was a React app. Our user interface at the moment uses CSS/HTML components provided by Vuetify. In order to keep the look and feel, and perhaps re-use some of what we created, it would be easier to use some library that implemented the Google Material Design, but worked with React (as Vuetify is Vue only).

But looks like if we use JupyterLab 3 we can do whatever we want, even keep using Vue + Vuetify, or vanilla JS/TS with some other CSS library, or write our own scss/css layout.

A couple years in: TypeScript is going to reward the maintainers of a project. It might be a turn-off for casual contributors that don't know TypeScript, and users don't care at all, or about react, redux, vue, revue, vuetify, unless it helps you ship better software faster, but they might care about material design vs... whatever JupyterLab is.

That doesn't look too bad IMO, as I think maintaining the code will be much simpler 🙏

... but some parts are hard-coded, like the user name (for the URL like /user/$name/subscriptions)

The PageConfig object has baseUrl, which you can embed. Unless this is some vue thing that is magic and special...

Any way to get the currently logged-in user, if any? I see only the URL in PageConfig - https://github.com/jupyterlab/jupyterlab/blob/de778fae6c253d45dc63f3299b29b1f8ded06dcf/packages/coreutils/src/pageconfig.ts.

But we also need the user name for GraphQL queries.

There are several benefits of moving to JupyterLab. I listed the issues I found, but they are not really blockers (if we have time), just things that would need to be reviewed/clarified if/when considering the move +1 (which would be really fun IMO grimacing).

If your pieces are small (or at least self-contained), and fit nicely into UI concepts already in JupyterLab (e.g. a sidebar, a menu, a main area) it can be pretty nice. Also, if you leave extension points (exposed on your top-level plugin), others can extend your stuff.

How do you handle if you have multiple Lumino widgets that need the same data? Or if you have an extension like GScan in the side panel, would it be possible to share the data from GScan with widgets created?

Thanks a lot @bollwyvl ! I've got a lot to learn (though most of that is fun to learn! 🎉 🤓 )

[kinow]

Oh, almost forgot. @bollwyvl any good documentation on updating an extension project from JLab2 to JLab3, please? Just to know whether it will be easy to update, or if I should wait or use the existing JLab3 code for the extension.

[bollwyvl]

I believe the migration guides are in lab master, etc. but again, cranking the cookiecutter with "--checkout 3.0", getting up to the watch command and playing with stuff can be the most informative. It's always hard to make some of these calls. While the individual js/css might change very little, the boilerplate is very different... As would be the eventual end user documentation. But going from a section of "install nodejs and jump through these hoops" to a line of "pip install" is priceless.