Summary
The Field Enricher (Feature Service) Processor can be used to enrich (or join) event records with attribute data from a published feature service’s feature layer or nonspatial table. The attributes are appended as new fields in the processed event records.
Examples
- The Field Enricher (Feature Service) Processor can be used to give nonspatial data a spatial component. For example, attribute data from a sensor can be enriched with a geometry (location) coming from a related feature layer. This allows the ability to visualize the sensor on a map as it receives real-time attribute updates. The sensor can then be symbolized based on the changing attributes.
- Using the Field Enricher (Feature Service) Processor, real-time vehicle data from an AVL feed can be enriched with daily driver assignment information stored in a related feature layer. As drivers are assigned vehicles for the day, the real-time location and status of the vehicle can be enriched with the corresponding driver’s information from the feature layer. This is useful for monitoring incidents in real-time.
Usage notes
- The Field Enricher (Feature Service) Processor requires a registered ArcGIS Server connection be specified; including the service folder, the feature service name, and the target layer in the feature service. The data types for each appended field will be carried over from the enrichment source—they are not specified as part of the processor configuration.
- GeoEvent data enrichment relies on what database professionals refer to as a table join. You can specify a field name from the feature service's table and the name of the GeoEvent field on which a join can be performed. While the actual field name from the feature service's table must be provided, the GeoEvent field on which the join will be performed can be specified using either the name of the GeoEvent field or a tag applied to a field in the GeoEvent Definition associated with the event being processed.
- A comma-separated list of the fields to be included in the enrichment can be constructed by selecting fields or entering them manually. Optionally, specify the tags GeoEvent Server should apply to each new field it creates as a second list of comma-separated values.
- The processor works by caching polled feature records. This improves performance and reduces the total number of requests made to the feature service each time an event record is processed. As an event record is received, the processor polls the feature service for the corresponding feature record. Once polled, the feature record is then cached and reused based on the duration of time configured by the Cache Refresh Time Interval (minutes) parameter. The total number of feature records that can be cached is controlled by the Maximum Number of Feature Records parameter.
- GeoEvent data enrichment with the creation of new fields alters the event record's schema, which requires GeoEvent Server to create a new GeoEvent Definition. The new GeoEvent Definition will be managed by GeoEvent Server and deleted if changes are made to the processor or the GeoEvent Service in which the processor is used.
- ArcGIS Enterprise, ArcGIS Online, and ArcGIS Server (stand-alone) map services or feature services can be used with the Field Enricher (Feature Service) Processor.
- When identifying an existing field to use in the GeoEvent Join Field parameter, it is not required you specify a GeoEvent Definition first. Choosing a GeoEvent Definition from the Definition menu only narrows the list of available fields to choose from in the Field menu.
Parameters
| Parameter | Description |
|---|---|
| Name | A descriptive name for the processor used for reference in GeoEvent Manager. |
| Processor | The name of the selected processor. |
| ArcGIS Server Connection | The ArcGIS Server, ArcGIS Enterprise, or ArcGIS Online connection registered with GeoEvent Server as a data store. Registered server connections cache information about feature services, feature layers, and layer properties. Provide the registered data store connection containing the feature service used for event data enrichment. Note:The processor provides the ability to register a new ArcGIS Server, ArcGIS Enterprise, or ArcGIS Online data store connection by clicking Register ArcGIS Server. Alternatively, a new data store connection can be registered in GeoEvent Manager by navigating to Site > Data Stores. |
| Folder | The ArcGIS Server services folder or ArcGIS Enterprise portal/ArcGIS Online content item folder where the map or feature service for event record enrichment is located. |
| Service | The name of the feature service from which feature records will be polled. |
| Layer | The name of the feature layer from which feature records will be polled. |
| Feature Layer Join Field | The name of the field from the feature layer used to perform an attribute join with the processed event records. Joins are made on the corresponding data value in this field and the field specified in the GeoEvent Join Field parameter. |
| Target Fields | The target field(s) used for the enriched data obtained from the feature layer. The default is New Fields.
|
| Enrichment Fields | Specifies the fields from the feature layer to use to enrich (join to) processed event records. Use a comma-separated list, with no spaces, to choose more than one field for event record enrichment (for example, DriverName,Driver_ID,Route). Note:Use Select Fields as an alternative method for choosing fields from the feature layer. |
| GeoEvent Join Field | The name of the field from the event record used to perform an attribute join with data from the feature layer. Joins are made on the corresponding data value in this field and the field selected in the Feature Layer Join Field parameter. Note:Use the Definition menu to identify the GeoEvent Definition of the inbound event record. Choosing a GeoEvent Definition will limit the number of fields to choose from. Use the Field menu to identify the name of an existing field to be used for the join. |
| Field Tags (Conditional) | The tag, or a comma-separated list of tags, to apply to the new fields appended to the processed event record. The order of the tags must match the order of the specified enrichment fields (for example, GEOMETRY,TRACK_ID,TIME_START). Note:The tag(s) must already exist in GeoEvent Manager, the processor will not create new tags. Property is shown when Target Fields is set to New Fields and is hidden when set to Existing Fields. |
| New GeoEvent Definition Name (Conditional) | The name assigned to the new GeoEvent Definition. The new GeoEvent Definition will combine the schema of the inbound event record with the new fields used for storing the enrichment values from the feature layer. Property is shown when Target Fields is set to New Fields and is hidden when set to Existing Fields. |
| Cache Refresh Time Interval (minutes) | Specifies the amount of time, in minutes, individual feature records will be stored in memory (cached) after being used to enrich an event record. Cached feature records are used to enrich additional corresponding event records. Once the cache refresh time interval for a feature record is exceeded, the processor will clear the feature record from its cache to make room for additional feature records. A new request will be made to the feature service for the feature record upon receiving another event record requiring a join. The default is 1. Note:Cached feature records are stored in memory. |
| Maximum Number of Feature Records | Specifies the maximum number of feature records to be stored in memory (cached). When the number of feature records exceeds the value specified in this parameter, the oldest feature records in the cache are cleared to make room for the new feature records. This process occurs regardless of the value specified in the Cache Refresh Time Interval (minutes) parameter. The default is 1000. |
Considerations and limitations
- Consider finding a balance when adjusting the cache settings for this processor. An increase to the time interval feature records are cached will result in a reduction in the number of requests made to the feature service. This will improve the total throughput performance of the processor since feature records will be joined from memory instead of from repeated network requests. The trade-off with this approach is that the feature records used for joins will be inherently older. It is recommended the Cache Refresh Time Interval (minutes) parameter be increased when the feature records are not updating often. For example, if the feature records are updated every 72 hours, consider adjusting the cache refresh time to be nearly the same rather than every minute (which is the default). This will reduce the number of service requests made for feature data that is otherwise not changing and can be stored in memory.
- Increasing the Maximum Number of Feature Records parameter will result in more feature records stored in memory (cached). Consider making this change when the Cache Refresh Time Interval (minutes) has also been increased. Feature records that are cached for longer may accumulate in greater number depending on the rate of inbound event data. The trade off with increasing the Maximum Number of Feature Records is that more system memory is used.
- Reducing the Maximum Number of Feature Records is often not recommended. Doing so can have an adverse impact on the performance of the processor. For example, setting the Maximum Number of Feature Records to 0 means that no feature records will be cached. If the number of unique event records coming through is 2000 per second, the processor will effectively have to issue 2000 requests to the feature service per second, every second, to obtain the feature records for enrichment. This can not only reduce the overall performance of GeoEvent Server, but also negatively impact the external server that must handle the high number of requests.
- When the Maximum Number of Feature Records size is exceeded, GeoEvent Server will remove the oldest cached feature record in order to accommodate the new feature record. For example, if the Maximum Number of Feature Records is set to 100, and 101 unique event records are processed, the first cached feature record will be purged to make room for the 101st feature record. This happens regardless of whether the Cache Refresh Time Interval (minutes) is exceeded.
- When this processor is used for the first time in a GeoEvent Service with a high-volume, high-velocity data feed, it is not uncommon for a backlog of inbound event records to accumulate. The reason being, GeoEvent Server must issue a series of requests to the feature service to build its cache of feature records to join on for the first time. Once the feature records begin to accumulate in the cache, the processor can then join subsequent event records from memory; thereby improving processing performance.
- Network latency can have a significant impact on the performance of this processor. For example, if the time it takes to retrieve a feature record increases from 100 milliseconds to 200 milliseconds because of fluctuations in network latency, the number of event records that can be processed in that same unit of time will have been reduced by half. The reason being, it takes twice as long to retrieve the feature record for a valid join to happen.