[DMP 2024] Generate a project.yaml file from a list of steps

ayodele · June 13, 2024, 11:19am

We are excited to participate in the 2024 Direct Mentoring Program by supporting two software engineers interested in contributing to Open Source Gov4Tech programs. This public channel is dedicated to discussing our sponsored feature: Generating project.yaml files from a list of steps.

Mentors, contributors, and community members are welcome to ask and answer questions related to this topic here.

Feel free to engage and share your insights!

github.com/OpenFn/kit

[DMP 2024] Generate a project.yaml file from a list of steps

opened 01:15PM - 04 Mar 24 UTC

closed 05:17PM - 29 Dec 24 UTC

christad92

DMP 2024

## Overview We are looking to integrate an AI agent that is able to generate …a workflow for a user, based on an instructions specifying the required steps and adaptors. This feature will enable new users of OpenFn (Lightning) to get started faster after registration by entering the steps they need. The agent will automatically build a workflow can run without error and ready for editing. ## Deliverables * A new service in the apollo rep called `gen_project` * The service should take a series of steps as an input, and return a workflow definition as an output * The workflow can optionally be defined as a project.yaml or workflow.json file * A demo script which allows the project to be imported to lightning. To be clear, this work does NOT include any integration on the Lightning side. Extensions to this work include: * Generating job expressions to implement the workflow (this likely leverages existing apollo functionality) ## Sample Inputs Below are examples of prompts and their corresponding workflow.yaml for reference and testing. Prompt 1 - Get Data from DHIS2 - Filter out children under 2 - Aggregate the data - Make a comment on Asana ``` Workflow-1: name: Simple workflow jobs: Get-data-from-DHIS2: name: Get data from DHIS2 adaptor: '@openfn/language-dhis2@latest' # credential: # globals: body: | Filter-out-children-under-2: name: Filter out children under 2 adaptor: '@openfn/language-common@latest' # credential: # globals: body: | Aggregate-data-based-on-gender: name: Aggregate data based on gender adaptor: '@openfn/language-common@latest' # credential: # globals: body: | make-a-comment-on-Asana: name: make a comment on Asana adaptor: '@openfn/language-asana@latest' # credential: # globals: body: | triggers: webhook: type: webhook enabled: true edges: webhook->Get-data-from-DHIS2: source_trigger: webhook target_job: Get-data-from-DHIS2 condition_type: always enabled: true Get-data-from-DHIS2->Filter-out-children-under-2: source_job: Get-data-from-DHIS2 target_job: Filter-out-children-under-2 condition_type: on_job_success enabled: true Filter-out-children-under-2->Aggregate-data-based-on-gender: source_job: Filter-out-children-under-2 target_job: Aggregate-data-based-on-gender condition_type: on_job_success enabled: true Aggregate-data-based-on-gender->make-a-comment-on-Asana: source_job: Aggregate-data-based-on-gender target_job: make-a-comment-on-Asana condition_type: on_job_success enabled: true ``` Prompt 2: - Fetch submissions from KoboCollect with language-kobotoolbox@latest - Push the data to the a postgresSQL database with language-postgresql@latest - Send text message to an admin using language-twilio@0.3.4 with status of sent message ``` workflow-1: name: another simple workflow jobs: Fetch-submissions-from-KoboCollect: name: Fetch submissions from KoboCollect adaptor: '@openfn/language-kobotoolbox@latest' # credential: # globals: body: | // Get started by adding operations from the API reference Push-Data-to-PostgreSQL: name: Push Data to PostgreSQL adaptor: '@openfn/language-postgresql@latest' # credential: # globals: body: | // Get started by adding operations from the API reference Send-a-text-message-to-admin: name: Send a text message to admin adaptor: '@openfn/language-twilio@0.3.4' # credential: # globals: body: | // Get started by adding operations from the API reference triggers: webhook: type: webhook enabled: true edges: webhook->Fetch-submissions-from-KoboCollect: source_trigger: webhook target_job: Fetch-submissions-from-KoboCollect condition_type: always enabled: true Fetch-submissions-from-KoboCollect->Push-Data-to-PostgreSQL: source_job: Fetch-submissions-from-KoboCollect target_job: Push-Data-to-PostgreSQL condition_type: on_job_success enabled: true Push-Data-to-PostgreSQL->Send-a-text-message-to-admin: source_job: Push-Data-to-PostgreSQL target_job: Send-a-text-message-to-admin condition_type: on_job_success enabled: true ``` ## Demo script To validate the work, we would like a demo script which lets us easily test various inputs and see them visualised in Lightning. The demo should be executable from a terminal. It could be a python or bash script. It should not be integrated with the the OpenFn CLI. The demo should do the following: * Read in a list of steps from a file, where each line of the file is one step * Call out to the apollo service (locally, by default, but can take a URL) to generate a project.yaml * Generate a Lightning project structure on the file system * Deploy the project to demo.openfn.org ## Suggested API We recommend the following API for the service JSON input: ``` { steps: ["instructions for step 1", "instructions for step"] format: "yaml" | "json" } ``` JSON output: ``` { files: { 'project.yaml': '...' } } ``` The resulting structure and the `files` key makes for easy integration with the apollo server. ## Background OpenFn is an open source platform for data integration and workflow automation which can be used via a [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) or a [web UI](http://github.com/openfn/lightning). Projects on OpenFn can be encoded in a yaml file containing workflows, steps, jobs, triggers and edges. ### Key terms Workflows contains one trigger and one or more steps connected by edges. Workflows are at the base of users activities and steps related to a objective are built and saved in a workflow. A step is a task or instruction that users write mostly in Javascript. The output of a step can be anything from sending a text message, transform data, make an API call, send data to an external systems or fetch data from an external system. These jobs are performed with the help of adaptors. Triggers are useful for nudging workflows to run based on an event or at a scheduled time. Edges connects two steps and determines the order and conditions for the steps in a workflow. See more details at https://docs.openfn.org/documentation/get-started/terminology ### Sample Project.yaml Below is an example of a yaml file for a project `openhie-project` which has one workflow `OpenHIE-Workflow` and 4 steps `[ "FHIR-standard-Data-with-change", "Send-to-OpenHIM-to-route-to-SHR", "Send-to-OpenHIM-to-route-to-SHR", "Notify-CHW-upload-successful"]`. The trigger is a Webhook event and there are also 4 edges with `source_job`, `target_job` and `condition` variables that guide how the workflow should be executed. ```yaml name: openhie-project description: Some sample # credentials: # globals: workflows: OpenHIE-Workflow: name: OpenHIE Workflow jobs: FHIR-standard-Data-with-change: name: FHIR-standard-Data-with-change adaptor: '@openfn/language-http@latest' enabled: true # credential: # globals: body: | fn(state => { console.log("hello world") return state }); Send-to-OpenHIM-to-route-to-SHR: name: Send-to-OpenHIM-to-route-to-SHR adaptor: '@openfn/language-http@latest' enabled: true # credential: # globals: body: | fn(state => state); Notify-CHW-upload-successful: name: Notify-CHW-upload-successful adaptor: '@openfn/language-http@latest' enabled: true # credential: # globals: body: | fn(state => state); Notify-CHW-upload-failed: name: Notify-CHW-upload-failed adaptor: '@openfn/language-http@latest' enabled: true # credential: # globals: body: | fn(state => state); triggers: webhook: type: webhook edges: webhook->FHIR-standard-Data-with-change: source_trigger: webhook target_job: FHIR-standard-Data-with-change condition: always FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: source_job: FHIR-standard-Data-with-change target_job: Send-to-OpenHIM-to-route-to-SHR condition: on_job_success Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: source_job: Send-to-OpenHIM-to-route-to-SHR target_job: Notify-CHW-upload-successful condition: on_job_success Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: source_job: Send-to-OpenHIM-to-route-to-SHR target_job: Notify-CHW-upload-failed condition: on_job_failure ``` ### Here's what a list of steps might look like (also pictured in the image below): - Fetch Referrals From Primero using the primero adaptor - Send a text message to case officer with telerivet adaptor - Add patient to DHIS2 with the dhis2 adaptor ### When this feature is integrated with Lighnting, here is how a user would use it to generate a workflow. Building this UI/X is NOT part of this project but the image below is helpful for contributors to understand the mission. ![Image](https://github.com/OpenFn/kit/assets/5230530/00b2b18d-dfb0-4446-a22f-29f90b6fad79) ## Documentation and relevant links - [Workflows](https://docs.openfn.org/documentation/build/workflows) - [CLI workflows](https://docs.openfn.org/documentation/cli-usage#run-a-workflow) - [Web UI](http://github.com/openfn/lightning) - [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli)

Pin4sf · June 16, 2024, 11:53am

I’m Shivansh, and I’ve been selected to work on this issue for DMP 2024. As a Data Science and ML enthusiast, I enjoy working on various AI-related projects. I am grateful to the OpenFn team and DMP for giving me this opportunity to work with them. I’m really excited to learn more about DPGs and industry-leading software practices through this mentorship program.

I’ve been developing a Python module that generates YAML files from workflow steps using NLP, rule-based parsing, and NER. You can check the initial AI pipeline I am following here:

My initial YAML generator implementation can be found in this Google Colab: RB+NER_Yaml.ipynb.

I hope my work aligns with the project goals. I would greatly appreciate your feedback and guidance on any potential improvements or clarifications regarding my responsibilities.

Pin4sf · August 9, 2024, 1:21pm

Hello everyone,

I have reached the midpoint of my service generation issue DMP 619 and have made significant progress in generating the project workflow (project.yaml), which should be lightning import ready.

I have created the gen_project service in the OpenFn Apollo repository and have been closely working with the project maintainers to resolve issues.

You can see the update in this video.

Feel free to let me know if there’s anything else you’d like to add or adjust!

ayodele · August 16, 2024, 9:21am

Thanks for the update @Pin4sf. It’s good to see that everything is coming together nicely.

keep it up

#DMP2024

Pin4sf · September 21, 2024, 2:30pm

I have completed my issue [DMP 2024] Generate a project.yaml file from a list of steps.

You can access the contribution dashboard here C4GT DMP 24

I would like to express my gratitude to all the maintainers who helped me throughout my mentorship program. I’ve learned so much about open source development and feel much more confident in my abilities. It was a privilege to work on this issue, and I hope to contribute more to the project in the future!

I am attaching the final demo of the project Video

ayodele · September 30, 2024, 12:18pm

Congrats Shivansh and thanks for your contributions.