mirror_farmos

Introduction

The objective of the mongo_mirror is to centralize, transform and serve data from a huge number of discrete and private farmOS instances into a unified, fast, efficient, privacy respectful and versatile database. A mirror allows data operations that would be overly costly, time consuming and privacy harming if done instancywise. This library provides a very simple interface that will allow a mirror to be built from scratch by only providing the adequate security keys and some variables describing the expected behaviour. As the farmOS data structure is fiarly complex to be directly consumed by some applications, this library is also able to consolidate it and de reference it into structures that are as flat as possible without loosing data content or precission. This also makes data analysis easier. The mongo_mirror can optionally supervise the application of as many data pipelines as needed to be applied over this consolidated and simplified dataset, in order to deliver the exact data that each applicatino needs. For example, a first dataset might only have anonimized GPS positions to show in a map the farms

How this library works

Transformations Applied to Entities

• Entities are mirrored inside a tree structure before dereferencing.

• Each entity gets a field called related. It contains the entities mentioned in relations. Each one of this entities contains its own related field, the entities look the same when searched directly or as relationships of other related entities.

• Entities inside related are tagged with a relationship_class attribute. It informs under which relations attribute it was originally stored and allows to distinguis what kind of information should be obtained from it.

• I traversed the FarmOs UI and identified the source (either an attribute on the main entity or inside a relationship) for each bit of data that is shown there for each type of entity. These are copied into the main entity with appropiate data names as first level object properties, obtained either from attributes on the original main entity or inside any of its related sub entities.

Modules

Nightly Build

Meant to grab information from FarmOS instances and centralize it into a MongoDB unique database.

Main Functions

• copyFarmToMongo: Provided a farm and a list of types, all entitities for each type are retrieved, tagged with the adequate farmDomain attribute and stored inside raw_entities collection.
• copyManyFarmsToMongo: Helps iterate through the whole list of farms.

Dereference and Simplify

Meant to reformat the data into a consolidated, as flat as possible without loosing details planting centric database. It assembles whole objects based on FarmOS references attributes so that each object contains directly all of its related information. It employs a set of parsers called Instructions to grab only the data attributes that are relevant from each entity and arrange them in the most possible meaningful way without making wrong structure assumptions.

• processOneFarm retrieves all entities from raw_entities for a given farm, dereferences them, simplifies them and copies them into simplified_entities.

Coffeeshop Aggregations

Not currently a library.

An aggregation grabs plantings from simplified_entities, attaches events (as they are reverse dependencies) and consolidates planting rows into coffeeshop_full. This is further simplified and stored as coffeeshop_simplified, which is the exact payload the coffeeshop frontend expects.

Declarative Configuration

Where to find the Summaries

Main Branch

• Latest version of QA artifacts