Liquid Pipe

Beijun Desai, Intern – 11/2021

Project Name: Liquid-Pipe

            This project is one of the first big projects I worked on at Sophron Networks LLC. The goal of this project is to create a pipeline that consists of the following:

• Receive messages as JSON arrays

• Debatch the messages into individual JSON objects

• Validate each object against a well-defined set of validation rules

• Transform each object into a specific JSON schema

• Run another set of user-configured business rules, where the business rules are defined by a JSON or XML configuration file

• Insert the transformed JSON object into a database

• Insert all errors and warnings encountered during transformation and/or validation steps into a database.

The project was created using Azure Logic Apps, Azure Storage Queues, Liquid templates, and Python. The project took way longer than expected because of the steep learning curve associated with Azure and the Liquid templating language. The first steps were to create templates using the given validation and business rules. Liquid templates were chosen so that new business and validation rules could be created in the future easily. After that it was developing the pipeline using logic apps and the queues. It was interesting and new to work with that. Creating the templates was hard because of the learning curve and the way the language worked with the different filters. It was fun to get used to working with JSON in that kind of environment and learning how Liquid worked. After much trial and error using the datasets, messages were being sent and validated, but new problems kept popping up dealing with the incorrect number of messages in the queue and duplicate messages. I tried switching to different logic apps and counting the number of messages within the Azure functions themselves, but it still didn’t work. I intended to keep this project separated into multiple steps as follows:

• Retrieve message file from a storage container

• Debatch messages from the file and send them to a queue

• Retrieve messages from the queue in a loop until all messages in said queue had been validated

• If the message was validated and was correct, it was then sent to a function to transform it. If not, it was deleted from the queue and put on another queue that contained all failing messages

• Transformed message in its own queue processed for business rules.

• If the message passed, it was sent to a final new queue. If not, it was deleted from said queue and then put in the same fail queue as the validation errors.

This process took way too long for it to be of any use in the grand scheme. The next step was to try to fix the errors that arose while dealing with Azure. There were a lot of 404 errors which I couldn’t solve, and there were still errors with the long running times and incorrect message counts. I then decided to remove some of the retrieve and send message steps by combining a lot of the steps written earlier into a single function, so only one function was called for all the validation/transformation/business rules operations instead of multiple. The sending of the messages to the original queue was also scrapped because it was unnecessary for the project requirements. Now the time required to run the pipeline locally went to 11 minutes for 10,000 messages to 6 – 7 minutes. Everything is being ran asynchronously which also helps. The project was also tested locally using Postman and Visual Studio Code. The only problem is figuring out how to run the pipeline on Azure because I keep getting mystery errors and the logs aren’t helpful until I learn how to read them which is another task. I keep trying new things, and I hope they will work eventually!