From git repo to easy ML pipelines: How we improved user experience.

Posted on 26-okt-2020 9:47:55

It's about time for our first official development update since the launch of our platform! Thanks to the hard work of our developers we've got some big updates to our platform aimed at improving the user experience, with a focus on data scientists. Let's take a closer look at all of them.

A better graph view, easier to navigate and manipulate

Let’s start with what’s staring us in the face, we made a big overhaul to the design of our graph view!

What used to be one big cluster, is now more logically grouped in different mini clusters. This completely changes the way you can interact with your data environment, allowing you to group related components or microservices together thanks to Kubernetes Namespaces and making it easier to find what you need.

New optimized UI for Tengu

We also made the graph a lot more stable, allowing you to move resources with more precision. You might see the occasional jittery node, but that’s only because the graph is now constantly being updated in real-time, no more pressing F5!

To accompany the new way it updates, we added a small visual update, namely color coding the nodes to quickly see which ones are updated, updating, or experiencing an error.

Not only is the graph view a bit clearer, providing better oversight, and more stable. It’s also easier to configure. We made it easier to set up a lot of connections between resources and configs, and added actions to a lot of resources to easily do what you want to. 

For example when selecting a database, instead of searching for the IP address, you can now access it with the single click of a button.

Perform actions straight from Tengu

 

More resources, with a focus on data science and ML

When it comes to adding functionality, we focussed on the data science flow, and specifically on easy integration of Machine Learning. To facilitate this we focussed on a couple of things, let’s do a quick rundown.
  • Elyra: our custom Jupyter Lab has a new supported function, and we do mean it when we say brand new because the software has only been released a couple of months ago. Elyra is an ingenious solution to build pipelines from jupyter notebooks. Just drag and drop, and connect the dots to create your ML pipeline!

Easily build pipelines with Elyra

  • Kubeflow runtime: to run your pipeline with sufficient resources, we suggest using the integrated Kubeflow runtime, we made the integration into Tengu as easy as possible. This requires storage as well, which leads us to buckets.
  • S3 MinIO Buckets: to support both Elyra and Kubeflow runtime functions, we integrated a storage resource. These work perfectly with your Elyra pipelines executed in the Kubeflow runtime.

With these 3 new functionalities, you can set up an ML flow from start to finish. Combine your Jupyter Notebook scripts into a pipeline with Elyra, execute it in Kubeflow runtime and store your data in a S3 MinIO bucket.

Quickly run your pipeline thanks to Kubeflow and MinIO buckets

 

Need some code? Git over here

Last but not least…

Elyra isn’t the only new thing in our custom Jupyter Lab, you can now directly import GitHub repositories into your workspace!

Now you can work on notebooks, scripts and configs through GitHub and import those straight into your pipeline, all confirming with the git protocol ensuring easy collaboration and tons of existing resources you can use. 

Collaborate on scripts, and clone your repo into our workspace

Interested in how all these integrations make for a easy and seamless data science project flow? 

Stay tuned for our first use case tutorial coming soon!

Topics: Tengu, Data Science, Dev update

Jaco Van Laere

Written by Jaco Van Laere

Jaco is a marketeer and content creator at TENGU, and has a profound love for all things data, tech and media.