Hi! Here is the second part of my “go myths” series. The first part can be read here.

Today I’m gonna to talk about strings. There are two popular myths about strings, and I want to share real state of affairs.

1. String is NOT the slice of bytes.

Yep, you can read everywhere that string is a slice of bytes, even on go.dev, but it’s just not true. Imagine, how will the string struct look like in this case? `type string []byte`? Maybe! But the source code is always the source of truth.

So, it’s definitely NOT a slice. It may look like it, but that’s as far as it goes.

2. Strings concatenations NOT always lead to allocations.

As you can hear, strings are immutable and that is why all concatenation operations lead to allocations. But it’s a very simplified thesis.

Let’s do some benchmarking:

Output:

Now we can be confident that no-allocation concatenations are possible. But how? Let’s ask our single source of truth in industry — the source code.

Here is the buffer. If the string is small enough to store the new value (inital + concatenated data) in 32 bytes, we will not get an allocation!

Output:

Now let’s force an allocation:

Output:

Gotcha! Here is how it’s decided to allocate or not.

It just compares the length of “l” (resulting string) with the size of buffer array. If “l” less than 32 bytes (for go 1.22) no allocation will happen, for another case calling “rawstring” leads to call “mallocgc” function.

Fun fact

Just change the concat syntax to “+=” and get an allocation!

Output:

Author: Herman Samolazov

You may also like:

GO Myths we believe: Capacity

Workflow engines 101

I’m Daniel, senior GO developer at Adiuvo, and today I want to explain what workflow engines are and provide you with some interesting cases from my experience with them.

Workflows: The Backbone of Business Processes

At the heart of any organization’s operations lies a myriad of processes, tasks, and activities. Workflows, in the realm of business, refer to the orchestrated series of steps designed to achieve a specific goal. Think of it as a roadmap guiding a task from initiation to completion. Workflows can range from straightforward linear processes to difficult, branching procedures, depending on the complexity of the task at hand. Workflows consist of the activities. Let’s have a look at what an activity is.

Within a workflow, individual units of work are known as activities. These can be as simple as sending an email or as complex as processing large datasets. Activities are the building blocks that, when combined in a specific sequence, make up a workflow. Each activity contributes to the overall goal, and their orchestrated execution ensures the seamless flow of the entire process.

Now, when we know a little bit more about workflow and activities, let’s dive in and check how it works behind the scenes.

How Workflow Engines Work: Behind the Scenes

Workflow engines operate as the silent conductors orchestrating the complex symphony of tasks and processes within an organization. Let’s dig into the inner workings of these engines to understand how they seamlessly coordinate activities, automate processes, and ensure the smooth flow of operations.

To make everything work, workflow engines need to handle a few things, which are:

• Workflow Definition
• Task Execution
• Dependency Management
• State Management
• Error Handling
• Integration Capabilities
• Rule Engine
• Monitoring and Reporting
• Adaptability and Flexibility

Let’s have a closer look at each of these.

1. Workflow Definition:

At the core, the process begins with defining the workflow. This involves mapping out the series of tasks, decisions, and interactions that constitute a particular business process. Users typically design workflows through graphical interfaces, creating a visual representation of the entire sequence.

2. Task Execution:

Once the workflow is defined, the workflow engine takes charge of executing the tasks or activities in the specified order. Each task represents a specific action, such as data processing, approvals, or system interactions. The engine ensures that tasks are executed in a coordinated manner, adhering to the defined workflow logic.

3. Dependency Management:

Tasks within a workflow often have dependencies. For instance, Task B may rely on the completion of Task A before it can start. Workflow engines manage these dependencies, ensuring that tasks are executed in the correct sequence. This helps in maintaining the integrity of the overall process.

4. State Management:

Workflows often involve processes that span extended periods. Workflow engines keep track of the state of each workflow and individual tasks. This means they remember where a particular process is in its execution, allowing for resumption from specific points in case of interruptions or failures.

5. Error Handling:

In the real world, errors are inevitable. Workflow engines include robust error-handling mechanisms to deal with unexpected issues. This involves logging errors, initiating retries, and, in some cases, triggering predefined error-handling workflows to address exceptional situations.

6. Integration Capabilities:

Workflow engines are designed to integrate seamlessly with various software applications, databases, and systems. This integration capability enables them to interact with external tools and services, creating a unified environment where disparate technologies can work together harmoniously.

7. Rule Engine:

Embedded within the workflow engine is a rule engine. This component allows organizations to define and enforce business rules within the workflow. These rules dictate the behavior of the workflow, ensuring that tasks are performed in compliance with regulations and organizational policies.

8. Monitoring and Reporting:

Workflow engines provide real-time visibility into the status and progress of ongoing processes. Users and administrators can monitor workflows through intuitive dashboards, track task completion, and access detailed logs. This visibility enhances control and facilitates informed decision-making.

9. Adaptability and Flexibility:

The best workflow engines are designed to be flexible. They allow organizations to adapt and modify processes easily. Whether it’s a change in business requirements or an optimization effort, workflow engines provide the agility needed to adjust workflows without significant disruptions.

In essence, workflow engines serve as the invisible architects that transform static business processes into dynamic, automated, and adaptive workflows. Their ability to manage complexity, ensure consistency, and enhance collaboration makes them indispensable tools for organizations navigating the ever-evolving landscape of modern business.

Knowing how things work, you need to understand what types of workflow engines are there and what are the differences.

What types are there and what are the differences?

1. Business-Oriented Workflow Engines:

Camunda BPM, Activiti, IBM BPM: These engines are geared towards business processes, providing high-level modeling and management tools for business analysts and managers. They typically support standards such as BPMN (Business Process Model and Notation).

2. Technical Workflow Engines:

Apache Airflow, Camunda: These engines focus on automating technical processes. They usually offer broader integration capabilities with the technical stack.

3. Hybrid Workflow Engines:

Microsoft Power Automate (Flow): This engine offers a hybrid approach by combining business processes and task automation across various domains including IT, marketing, etc.

4. Cloud-Based Workflow Engines:

Microsoft Power Automate, Zapier: These engines provide cloud-based solutions for process automation utilizing web services and APIs.

5. Microservices Orchestration Engines:

Temporal, Cadence: These engines are oriented towards orchestrating microservices in distributed applications and provide tools for managing execution and coordinating services.

6. Low-Code/No-Code Workflow Platforms:

Appian, OutSystems: Platforms that offer tools for creating business applications and workflows without requiring deep programming knowledge.

 

That was a theory part and I know that all of you want to try to create and run workflow on your own, so let’s do it. Let’s code!

Let’s code

Now let’s try to write our first workflow on our own.

For that I will use Temporal and Go.

Temporal

Temporal is an open-source platform developed by Temporal Technologies that provides a framework for building scalable and reliable distributed applications. Temporal is designed to simplify the development of complex, long-running, and mission-critical business processes by managing the state and execution of workflows. It focuses on solving the challenges associated with building resilient, fault-tolerant, and scalable distributed systems.

Key features of Temporal include:

1. Stateful Workflows: Temporal allows developers to write workflows as code in their preferred programming language. Workflows can maintain their state across multiple function invocations, making it easier to handle long-running processes.
2. Fault Tolerance: Temporal is designed to handle failures gracefully. It can automatically retry failed tasks, handle worker crashes, and recover from various types of failures without losing the state of the workflow.
3. Distributed Coordination: Temporal manages the coordination and execution of workflows in a distributed environment. It helps with scheduling tasks, managing dependencies, and orchestrating the flow of activities.
4. Scalability: Temporal is built to scale horizontally, allowing it to handle a large number of concurrent workflows and tasks. It can be deployed in a variety of environments, including on-premises and in the cloud.
5. Temporal Server: The Temporal server is the core component that manages the execution of workflows and the storage of their state. It supports various databases for state storage, such as Cassandra and MySQL.
6. Open Source: Temporal is open-source, which means that the source code is freely available for inspection, modification, and contribution by the community. This openness promotes transparency and collaboration.

Temporal is particularly well-suited for applications that involve complex business processes, require long-term execution, and demand high levels of reliability and fault tolerance. It can be used in a variety of industries, including finance, healthcare, e-commerce, and more.

Let’s code – Example

First, I have to run a Temporal server. For that I’ll recommend getting docker-compose from the Temporal team on github.

Then create an alias for temporal-admin tools, this is just for convenience.

alias tctl=”docker exec temporal-admin-tools tctl”

Then create a namespace for workflows

tctl –ns wf-article namespace register

My workflow will be very simple. It will consist of 2 activities (greetings, farewell). First will print into console a greeting message and second will print a farewell.

Use go get to install the Temporal Go SDK

go get go.temporal.io/sdk

Create a new GO project.

Then create an activity.go file where describe your activities logic

Let’s start the workflow.

Create start/main.go

Start the workflow

Check the UI

Here it is.

But wait a second, seems something is wrong here! No worker is set to pick up that task.

Let’s fix that.

Create worker/main.go

WF completed successfully

Congratulations, you have written your first workflow with GO and Temporal workflow engine. Remember this was a simple example. You can create powerful and complicated systems with the help of Temporal. If you are interested in learning more about Temporal I encourage you to find more examples on their official website temporal.io.

Conclusion

Workflow engines are very great orchestration tools for software development processes of any scale. Today we’ve delved into the anatomy of workflow engines from a theoretical point of view and then written our first workflow. Now you can start using them in your real world applications, making managing scenarios especially easy for complex business processes that may require long-term execution operations and that are highly reliable and fault tolerant.

Author: Daniel Gorshkov

You may also like:

GO Myths we believe: Capacity

My name is Herman, and I am here to bust some popular myths about the GO language. The one for today is, “Slice capacity doubles after appending.”

Having served as a hiring manager in over 500 interviews, I often joke, “Juniors claim that slices’ capacity just doubles, Middles say that it doubles, but after reaching a certain length behavior changes…, Seniors say that it doubles, but after reaching a certain length…, but after a specific go’s release formula was changed… All of these explanations are valid as answers, but in reality, everything is a lie!”

Let’s examine the code:

| Output: len=18, cap=36

As we can see, if we create a slice with a length and capacity of 17 and then append to it, we receive a slice with a capacity of 36. However, 17 * 2 equals 34! Moreover, 17 is smaller than 256, which is the threshold in the growslice function in version 1.21.

Let’s dive deeper! We’ll create a function that indicates how many times the capacity doubles and how many times it doesn’t.

| Output: 44

As we can see, the slice capacity doubles only 44 times out of 1000! What’s even more intriguing is that the mythical ‘threshold’ doesn’t always function as expected!

| Output: len=337, cap=672

Wow, 336 * 2 is indeed 672! However, it exceeds the threshold (256) and the result “should be” lower. Alright, let me present some more interesting examples, and then we can dig into the underlying mechanics to understand why it appears so confusing based on our current understanding.

Moreover, why do we always default to using int64? Let’s experiment with something different.

| Output: 28

Ah, now we see it’s related to the data type! Let’s further explore this insight.

| Output: 1

Which iteration is it that gives us this result, you ask? It’s the second one, with both capacity and length equal to 1.

| Output:

| slice: len=1, cap=1

| slice: len=2, cap=2

| slice: len=3, cap=3

| …

It just adds one to the cap every time.

Now we have enough evidence to understand that the capacity doesn’t always double, but only in specific cases. So, how does it work?

Let’s take a closer look at the ‘growslice’ function.

The function calculates the new capacity for the slice. In the initial section of the function, we observe the ‘default’ behavior of capacity growth: if the capacity is lower than the threshold, it will be set to double the current capacity. Otherwise, there is a formula for a smoother growth:

However, the most intriguing segment lies below, within the switch statement based on the size of the slice’s elements. Here, the `capmem` variable shows the amount of memory required to accommodate the new capacity of elements in the slice. This is where the complexity arises. We cannot simply allocate an arbitrary amount of memory, as ‘malloc’ doesn’t operate in that manner. Instead, ‘malloc’ allocates memory based on specific size classes. You can find these size classes here. By the way this code is generated, you can read a source here. The Go team decided to align the memory allocated for the new capacity with these size classes to optimize memory usage. If we request ‘malloc’ to allocate memory for 34 elements of 8 bytes each, it will allocate 288 bytes (36 * 8), because the calculated result depends on the specific size-class. The underlying logic is located here.

As we can see depending on the amount of memory needed to be allocated, amount maps it to one of size classes and then size class maps to the actual amount of memory. In this particular case allocated memory is enough to store 2 additional elements. So why should we lose these elements? Let’s simply add them to our slice. These calculations aim to align the capacity with the amount of memory allocated by ‘malloc’.

As you can observe, the formulas and computation methods vary for different types: those that are equal to the pointer’s size (dependent on the architecture), those with a size of 1 byte, those with a size that is a power of 2, and everything else. However, the concept remains the same across all these cases: calculate the minimum amount of memory required and align it with the results of ‘malloc’s work as closely as possible.

But what about empty structs? The size of an empty struct is 0, which leads to a special case for it.

For slices containing empty structs, slices serve as counters for these empty structs. There’s no need to allocate any memory for ‘data’ in such slices. Consequently, the concept of capacity doesn’t apply; it’s always equal to the current length of the slice.

Lastly, I should mention how `append` functions when we add multiple elements:

Keep in mind that the rules we discovered above are working for this case too. Demonstration.

| Output: 8

Conclusion:

So, now we can finally answer correctly on “how does slice cap grow”. Slice just tries to double the capacity, but reality is tougher.

Now we can show that calculation is based on a lot of factors, not only reaching threshold or not:

• It depends on element type(their size)
• It depends on architecture (bitness i.e. pointer size)
• It depends on current len and cap of slice
• It depends on size of slice(`roundupsize` part)
• It depends on current implementation of ‘malloc’
• It depends on number of elements we append

 

Author: Herman Samolazov