Workflow engines 101

I’m Daniel, senior GO developer at Adiuvo, and today I want to explain what workflow engines are and provide you with some interesting cases from my experience with them.

Workflows: The Backbone of Business Processes

At the heart of any organization’s operations lies a myriad of processes, tasks, and activities. Workflows, in the realm of business, refer to the orchestrated series of steps designed to achieve a specific goal. Think of it as a roadmap guiding a task from initiation to completion. Workflows can range from straightforward linear processes to difficult, branching procedures, depending on the complexity of the task at hand. Workflows consist of the activities. Let’s have a look at what an activity is.

Within a workflow, individual units of work are known as activities. These can be as simple as sending an email or as complex as processing large datasets. Activities are the building blocks that, when combined in a specific sequence, make up a workflow. Each activity contributes to the overall goal, and their orchestrated execution ensures the seamless flow of the entire process.

Now, when we know a little bit more about workflow and activities, let’s dive in and check how it works behind the scenes.

How Workflow Engines Work: Behind the Scenes

Workflow engines operate as the silent conductors orchestrating the complex symphony of tasks and processes within an organization. Let’s dig into the inner workings of these engines to understand how they seamlessly coordinate activities, automate processes, and ensure the smooth flow of operations.

To make everything work, workflow engines need to handle a few things, which are:

• Workflow Definition
• Task Execution
• Dependency Management
• State Management
• Error Handling
• Integration Capabilities
• Rule Engine
• Monitoring and Reporting
• Adaptability and Flexibility

Let’s have a closer look at each of these.

1. Workflow Definition:

At the core, the process begins with defining the workflow. This involves mapping out the series of tasks, decisions, and interactions that constitute a particular business process. Users typically design workflows through graphical interfaces, creating a visual representation of the entire sequence.

2. Task Execution:

Once the workflow is defined, the workflow engine takes charge of executing the tasks or activities in the specified order. Each task represents a specific action, such as data processing, approvals, or system interactions. The engine ensures that tasks are executed in a coordinated manner, adhering to the defined workflow logic.

3. Dependency Management:

Tasks within a workflow often have dependencies. For instance, Task B may rely on the completion of Task A before it can start. Workflow engines manage these dependencies, ensuring that tasks are executed in the correct sequence. This helps in maintaining the integrity of the overall process.

4. State Management:

Workflows often involve processes that span extended periods. Workflow engines keep track of the state of each workflow and individual tasks. This means they remember where a particular process is in its execution, allowing for resumption from specific points in case of interruptions or failures.

5. Error Handling:

In the real world, errors are inevitable. Workflow engines include robust error-handling mechanisms to deal with unexpected issues. This involves logging errors, initiating retries, and, in some cases, triggering predefined error-handling workflows to address exceptional situations.

6. Integration Capabilities:

Workflow engines are designed to integrate seamlessly with various software applications, databases, and systems. This integration capability enables them to interact with external tools and services, creating a unified environment where disparate technologies can work together harmoniously.

7. Rule Engine:

Embedded within the workflow engine is a rule engine. This component allows organizations to define and enforce business rules within the workflow. These rules dictate the behavior of the workflow, ensuring that tasks are performed in compliance with regulations and organizational policies.

8. Monitoring and Reporting:

Workflow engines provide real-time visibility into the status and progress of ongoing processes. Users and administrators can monitor workflows through intuitive dashboards, track task completion, and access detailed logs. This visibility enhances control and facilitates informed decision-making.

9. Adaptability and Flexibility:

The best workflow engines are designed to be flexible. They allow organizations to adapt and modify processes easily. Whether it’s a change in business requirements or an optimization effort, workflow engines provide the agility needed to adjust workflows without significant disruptions.

In essence, workflow engines serve as the invisible architects that transform static business processes into dynamic, automated, and adaptive workflows. Their ability to manage complexity, ensure consistency, and enhance collaboration makes them indispensable tools for organizations navigating the ever-evolving landscape of modern business.

Knowing how things work, you need to understand what types of workflow engines are there and what are the differences.

What types are there and what are the differences?

1. Business-Oriented Workflow Engines:

Camunda BPM, Activiti, IBM BPM: These engines are geared towards business processes, providing high-level modeling and management tools for business analysts and managers. They typically support standards such as BPMN (Business Process Model and Notation).

2. Technical Workflow Engines:

Apache Airflow, Camunda: These engines focus on automating technical processes. They usually offer broader integration capabilities with the technical stack.

3. Hybrid Workflow Engines:

Microsoft Power Automate (Flow): This engine offers a hybrid approach by combining business processes and task automation across various domains including IT, marketing, etc.

4. Cloud-Based Workflow Engines:

Microsoft Power Automate, Zapier: These engines provide cloud-based solutions for process automation utilizing web services and APIs.

5. Microservices Orchestration Engines:

Temporal, Cadence: These engines are oriented towards orchestrating microservices in distributed applications and provide tools for managing execution and coordinating services.

6. Low-Code/No-Code Workflow Platforms:

Appian, OutSystems: Platforms that offer tools for creating business applications and workflows without requiring deep programming knowledge.

That was a theory part and I know that all of you want to try to create and run workflow on your own, so let’s do it. Let’s code!

Let’s code

Now let’s try to write our first workflow on our own.

For that I will use Temporal and Go.

Temporal

Temporal is an open-source platform developed by Temporal Technologies that provides a framework for building scalable and reliable distributed applications. Temporal is designed to simplify the development of complex, long-running, and mission-critical business processes by managing the state and execution of workflows. It focuses on solving the challenges associated with building resilient, fault-tolerant, and scalable distributed systems.

Key features of Temporal include:

1. Stateful Workflows: Temporal allows developers to write workflows as code in their preferred programming language. Workflows can maintain their state across multiple function invocations, making it easier to handle long-running processes.
2. Fault Tolerance: Temporal is designed to handle failures gracefully. It can automatically retry failed tasks, handle worker crashes, and recover from various types of failures without losing the state of the workflow.
3. Distributed Coordination: Temporal manages the coordination and execution of workflows in a distributed environment. It helps with scheduling tasks, managing dependencies, and orchestrating the flow of activities.
4. Scalability: Temporal is built to scale horizontally, allowing it to handle a large number of concurrent workflows and tasks. It can be deployed in a variety of environments, including on-premises and in the cloud.
5. Temporal Server: The Temporal server is the core component that manages the execution of workflows and the storage of their state. It supports various databases for state storage, such as Cassandra and MySQL.
6. Open Source: Temporal is open-source, which means that the source code is freely available for inspection, modification, and contribution by the community. This openness promotes transparency and collaboration.

Temporal is particularly well-suited for applications that involve complex business processes, require long-term execution, and demand high levels of reliability and fault tolerance. It can be used in a variety of industries, including finance, healthcare, e-commerce, and more.

Let’s code – Example

First, I have to run a Temporal server. For that I’ll recommend getting docker-compose from the Temporal team on github.

Then create an alias for temporal-admin tools, this is just for convenience.

alias tctl=”docker exec temporal-admin-tools tctl”

Then create a namespace for workflows

tctl –ns wf-article namespace register

My workflow will be very simple. It will consist of 2 activities (greetings, farewell). First will print into console a greeting message and second will print a farewell.

Use go get to install the Temporal Go SDK

go get go.temporal.io/sdk

Create a new GO project.

Then create an activity.go file where describe your activities logic

Let’s start the workflow.

Create start/main.go

Start the workflow

Check the UI

Here it is.

But wait a second, seems something is wrong here! No worker is set to pick up that task.

Let’s fix that.

Create worker/main.go

WF completed successfully

Congratulations, you have written your first workflow with GO and Temporal workflow engine. Remember this was a simple example. You can create powerful and complicated systems with the help of Temporal. If you are interested in learning more about Temporal I encourage you to find more examples on their official website temporal.io.

Conclusion

Workflow engines are very great orchestration tools for software development processes of any scale. Today we’ve delved into the anatomy of workflow engines from a theoretical point of view and then written our first workflow. Now you can start using them in your real world applications, making managing scenarios especially easy for complex business processes that may require long-term execution operations and that are highly reliable and fault tolerant.

Author: Daniel Gorshkov

You may also like:

GO Myths we believe: Capacity