51Degrees API Documentation  4.5

Documentation

• Pipeline API
• Concepts
• Types of data
• Flow data

Flow data

Introduction🔗

Flow data holds the inputs, outputs, metadata, and errors for one Pipeline processing request.

See the Specification for more technical details.

Data structure🔗

Flow data has several sub-containers that are used to segment the data that it contains:

• Evidence
• Element data
• Errors

Evidence🔗

Before the flow data is passed into the Pipeline, input data is supplied. We refer to this data as 'evidence'. The evidence can be set manually or automatically by using a web integration package (where available) for your web framework of choice.

Visit the evidence page for more details.

Element data🔗

The responses from each flow element are stored in a key/value pair structure within flow data. In each case, the key is the string key of the flow element and the value is an element data instance. The element data structure is visible to each flow element so one element can use the result from another element in its processing.

An example where this is required is the 51Degrees Cloud Engines. First, an element makes an HTTP request to the cloud and stores the JSON response in the flow data. Later, another element takes that JSON response and parses it to populate a strongly typed object with values for the specific aspect it is concerned with.

Visit the element data page for more details.

Errors🔗

The errors collection stores the details of any errors that occur during processing. The language's default exception handling mechanism will be used to catch and record any exceptions that occur when a flow element is processing. However, the processing of later flow elements will continue as normal.

By default, once all flow elements have been processed, an exception will be thrown with details of any errors that have occurred.

The pipeline builder has an option to modify this behavior so that exceptions are totally suppressed. In this situation, the caller is responsible for handling any exceptions by checking the errors collection after processing.

Lifecycle🔗

Creation🔗

Flow data is only ever created by a Pipeline, when the CreateFlowData method is called. This allows the Pipeline to create the flow data internal data structures using implementations that are most appropriate for the configuration of the flow elements in the Pipeline.

For example, thread-safe but slower data collections only need to be used if the Pipeline is configured to execute elements in parallel.

Disposal and cleanup🔗

Dispose of flow data as soon as it is no longer needed. Flow data can hold native resources, such as the results of an on-premise engine, and under high load the garbage collector may not free them fast enough, which can cause an out-of-memory error.

In C# use a using block and in Java a try-with-resources block, as shown below. In Python use a with statement, and in PHP call close(). Node.js has no disposal method, so leave flow data to the garbage collector.

C# Java

Select a language.

In .NET, use a using block to dispose of flow data:

// Create the FlowData instance.

using (var data = pipeline.CreateFlowData())

{

...

}

In Java, use a try-with-resources block:

// A try-with-resource block MUST be used for

// the FlowData instance. This ensures that native resources created

// by the device detection engine are freed.

try (FlowData data = pipeline.createFlowData()) {

...

}

Thread-safety🔗

C# Java PHP Node.js

Select a language.

In .NET, by default, the non-thread-safe Dictionary class is used for both element data and evidence.

In both cases, this can be overridden to use another IDictionary implementation such as the thread-safe ConcurrentDictionary.

The errors collection uses the List class. This is not thread-safe. As performance is less of an issue with this collection, a simple lock is used to synchronize items being added to the list.

In Java, by default, the non-thread-safe TreeMap class is used for element data and the thread-safe ConcurrentSkipListMap class for evidence.

In both cases, this can be overridden to use another Map implementation such as the thread-safe ConcurrentHashMap. However, it should be noted that not all Map implementations support case insensitivity.

PHP runs in a single thread. Consequently, elements cannot run in parallel and concurrency issues are not a concern.

Node runs in a single thread with asynchronous execution, except where worker threads are used. It is strongly recommended to not share an element data object with a worker thread.

Have a data or code question? Get in touch and explore subscription options.