Accountable Event Overview

The Accounting service provides an aggregation functionality through which various services can centrally log accounting information as it is gradually accumulated by their usage from the respective authorized clients. Different accumulation policies and granularity of information is supported depending on the respective services. The information consists of globally defined key point indicators and the aggregation and reports generated based on these becomes available for further usage, as appropriate.

Accounting Model

The model for an accounted event can contain the following information:

• timestamp
  • DateTime (ISO 8601)
  • Mandatory
  • Date of the entry
• starttime
  • DateTime (ISO 8601)
  • Optional
  • Date of the operation starts
• endtime
  • DateTime (ISO 8601)
  • Optional
  • Date of the operation ends
• serviceid
  • Text
  • Mandatory
  • Global unique identifier for the service that the log entry refers to. It may be generated by the probe rather than be harvested from the logs.
• level
  • ENUM
  • Mandatory
  • Value should be "accounting"
• resource
  • Text
  • Mandatory
  • The resource handled by the call.
• action
  • Text
  • Mandatory
  • The action identifier in internal semantics.
• userid
  • Text
  • Mandatory
  • An identifier for the user on whos purpose the work is performed
• userdelegate
  • Text
  • Optional
  • An identifier for the user who is the delegate to perform the work
• value
  • Double
  • Optional
  • The measure value if omitted 1
• measure
  • Text
  • Optional
  • The measure currently available messures time(ms), information(mb), throughput(mb/sec), unit, others to be considered, optional if omitted unit
• type
  • ENUM
  • Optional
  • The measure type available types + - 0, optional if omitted +
• comment
  • Text
  • Optional
  • A comment of accounting

Ingestion Pipeline

The accounting service offers HTTP API endpoints through which event ingestion can take place. For the purpose of the DataGEMS integration we have choosen a less intrucive approach for the integrating components that is also offered by the Logging Service.

We make use of the logging capabilities and the infrastructure and processes put in place for the Logging Service. Integrating components can produce log entries that are explicitly marked to be interpreted as Accounting Entries. These entires are not picked up by the logging service aggregation mechanism but from a pipeline that explicitly picks them up, transforms and ingests them to the Accounting Service Elastic Search index.

Through the deployment model utilized, an agent will read the logs produced by each pod and will scan for entries marked as Accounting Events. A transformation step will apply any needed mutations to the log entry retrieved to match the accounting model expected by the accounting service. The trasformation can be customized based on specific deployment labels. A set of supported schemas is provided and each deployed service can declare the accounting event schema it conforms to. Based on this schema the transformation is applies. After the accounting entry model is transformed, enriched with environment metadata and generated, it is pused to the accounting service datastore where it becomes available for aggregations.

The supported accounting entry formats are the following:

TODO: Provide examples*