Feature Manipulation Engine (FME) Geospatial Review

The Resource Applications and Technology Implementation (RATI) unit at the Geospatial Technology and Applications Center (GTAC) completed an assessment of the geospatial capabilities and US Forest Service (USFS) use cases related to the Feature Manipulation Engine (FME) software.

OUTLINE OF CONTENT:

Navigational links are included at the top of this Story Map for quicker routing to specific sections.

  1. Introductory Information
  2. Geospatial Capabilities
  3. Use Cases
  4. Data System Interactions
  5. Application Interactions
  6. Issues and Recommendations
  7. Administrative Information
  8. Versioning and the Road Ahead

Introductory Information

To browse through all the components of the FME platform, click the buttons to the left and right of the following interactive graphic to navigate through the pages.

Geospatial Capabilities

GIS was a starting point for the development of FME; thus, it's foundational ties to, and power in, the geospatial realm. Not only does FME support many geospatial (spatial and tabular) formats, it has many out-of-the-box transformers that provide a wide variety of geoprocessing capabilities. Further, FME functionality can be integrated into Esri's ArcGIS Pro or ArcMap via the Data Interoperability extension.

These topics are covered in the following set of slides.

  • Data Management and Compatibility
  • Data QA/QC
  • Data Monitoring
  • Cartographic Uses
  • Metadata
  • ArcGIS Data Interoperability Extension
  • Ties to Web GIS Applications
  • Working with 'Big Data'
  • Limitations
  • Performance and User Experience

Use Cases

This section reviews many existing FS-specific use cases, as well as discusses things to consider related to transitioning workflows within FME.

Transitioning Workflows: Development to Maintenance

There are 2 levels of transition to consider when managing an FME workflow: (1) transitioning an existing workflow to an FME Workbench workspace (either creating new or using an existing entity like an ArcGIS ModelBuilder model); and (2) transitioning a workspace in FME Workbench to FME Server to be run manually and/or automated.

Regarding the second point, there is a need for an FME-advanced user to intially develop a workspace (FMW file) for a process in FME Workbench. Afterwards, that workspace can be handed over to a basic user to be run via FME Server with little expertise. Parameters can be set up to allow for a user interface tool, similar to ArcGIS ModelBuilder. Automation may be enabled, allowing the process to be very hands-off within FME Server. However, maintenance of the foundational FME workspace typically requires attention of an advanced user.

To fully take advantage of FME as a geospatial tool, more expertise is needed in the FS to sustain the development and maintenance required for the agency's business needs.

Data System Interactions

NRM: Must consider both the NRM Legacy (IWebDB) and NRM ProdB (NextGen) environments. NRM Legacy uses SDO as the data storage type and utilizes SDE formats for transactional components. NRM ProdB uses Oracle Spatial data formats; SDO only. All of these data types are supported by FME. Currently, FME is used for some loading of NRM data.

Some NRM datasets utilize custom COM objects (i.e. FACTS, FSVeg, Wildlife, AQS (Aquatic Surveys), Watershed Quality Survey and WITT); these are not supported by ArcGIS Pro. For more information on how to access these NRM Feature Classes through FME, refer to  these instructions (FME Desktopsection) .

Distribution of jobs run in FME Server; ETL_PROD is mainly EDW jobs

Distribution of jobs run in FME Server; ETL_PROD is mainly EDW jobs

VDC: Geodatabase structure in the VDC is SDE data format, using ST storage type. FME supports SDE data. Regarding applications within the VDC, Safe supports current ArcMap and ArcGIS Pro versions pertinent to time of FME release.

EDW: Corporate datasets can be easily accessed from all FME components when developing workspaces. FME Server powers numerous ETLs for uploading and refreshing EDW datasets. Approximately 1600 jobs run each night. There are also some validation and error reporting automations completed by FME Server.

A portion of a job list run nightly by FME Server

A portion of a job list run nightly by FME Server

EDAPT: FME Server is currently being used to push text files from the FS enterprise system into the USDA EPAPT environment. This is an automated process, and once text files are migrated, a Python script incorporates them in EDAPT. There are plans to place an FME Server instance directly within EDAPT, which would allow information to be ingested into the USDA EDAPT environment from the FS using a more streamlined ETL process.

Application Interactions

For a list of integration opportunities available with FME, visit the  FME Integrations Gallery .

Graphic representing data integration - a computer monitor with floating bubbles of data types like Excel, Oracle, PostgreSQL, Esri, etc. in front.

In general, FME behaves the same with a local install of FME or with an instance in Citrix or the VDI environments. Occasionally, FME has a dependency on local installations of certain applications, such as Oracle or Esri products. In the case of a dependency, a local install of the interacting software MUST exist in the environment (i.e. local, Citrix, or VDI) the FME application is being run.

Within the  Matrix View  of the 'FME Integrations Gallery', any data format that requires local installations of interacting software has a note stating 'Requires Additional Software' in the list. An example is pictured below in the excerpt of the FME Integrations Gallery -- note the Esri ArcGIS Layer and Esri ArcGIS Map Document items.

A snippet of the Matrix View of the FME Integrations Gallery specifically highlighting how to tell which data types require additional software.

An excerpt of the Matrix View of the FME Integrations Gallery

A user must consider whether the version of FME being used is 32-bit (i.e. FME 2020 and earlier) or 64-bit (i.e. FME 2022) and how that relates to the interacting application. If using FME 2020, and need to interact with Esri and Oracle, a 32-bit version of ArcMap and the 32-bit Oracle client must be installed. Further, since FME 2022 is a 64-bit application, it will only function alongside ArcGIS Pro, Esri's 64-bit Desktop GIS program.

Esri/ArcGIS

FME programs can interact with a number of Esri products including ArcMap, ArcGIS Pro, Portal, AGOL and other web applications like Field Maps and Survey123. As just mentioned, the version of FME in use must be considered when interacting with ArcMap and/or ArcGIS Pro.

Common data formats used in the FS, supported by FME, include ArcGIS layer files (.lyr/.lyrx), ArcGIS Map Documents, Esri Shapefiles (.shp), AGOL/Portal/ArcGIS Server Feature Services, ArcInfo Coverage and E00 files, Esri ASCII Grid files, Esri SDE Geodatabase files, Esri File Geodatabase (.gdb), Esri Personal Geodatabase (.mdb), XML Workspace Documents, Esri Mobile Geodatabases, etc.

For more information about interacting with FME and ArcGIS Pro in the VDI environment, please refer to the  ArcGIS Pro SharePoint site –Applications on the VDI Desktop/FME Desktop portion .

ERDAS IMAGINE

FME can interact with ERDAS native data formats such as ER Mapper ERS, RAW, and IMG files.

Additionally FME supports many other commonly used raster-based file types in the FS, such as TIFFs, GeoTIFFs, Cloud Optimized GeoTIFF (COG), and Meta Raster Format (MRF).

For more information,  click here .

Tableau

FME is most commonly used as a precursor tool for Tableau by either assisting with data setup (i.e. create, analyze, simplify) or creating a TDE (Tableau Data Extract) file. FME Server is used to bring data into Tableau automatically. Also, FME 2022 can interact with Tableau Hyper files (not supported in prior versions).

For more information,  click here .

Power BI

FME does not support PBIX files (native Power BI project files). There is interest, raised in the  FME Community , for a writer specific to PBIX files. Users are still are able to work with FME alongside Power BI by employing readers/writers associated with Excel, CSV, SQL, etc. to update source data. Once the source data is altered, triggers can be set to refresh Power BI data instances.

Oracle

FME supports reading and writing of Oracle data. Common conversions executed in FME: MS Excel to Oracle; Oracle to MS Excel; MS Access to Oracle; Oracle to Esri SHP.

For more information,  click here .

PostgreSQL

Like Oracle, PostgreSQL data can be read and written with FME. Common conversions executed by FME: PostgreSQL to PostGIS; CSV to PostGreSQL; PostGreSQL to Esri SHP; MS Access to PostGreSQL.

For more information,  click here .

Microsoft Applications

FME supports many Microsoft products including SQL Server, Access, Excel, and Azure services. Additionally, it supports several on-premises and cloud solutions such as Amazon Web Services and Dropbox, and basically any web service with an API.

For more information,  click here .

Adobe Products

FME can read and write geospatial PDF documents using the PDF2D reader/writer. There are additional transformers within FME that can extract certain types of data from a PDF, such as the TesseractCaller transormer that can read specific PDF text (consider pulling label information). Other elements can be extracted from geoPDFs such as maps, rasters/images, vector data, text, tables, and metadata.

The following is useful documentation related to FME and Adobe interactions:

For more information,  click here .

Issues and Recommendations

A table detailing issues and recommendations Issue, Summary of Issue; Users Effected; Recommendation for Next Step(s) Need for more FME-specific expertise and technical support resources within the FS; Although there are several geospatial users throughout the agency that are familiar with FME and/or see potential for it for their business needs, few have the advanced-level resources necessary to independently, and fully, support their unit/region. Further, the FS currently has no dedicated support role for FME users.; All existing or potential FS Geospatial FME Users; (i) Identify pathways for FME users of all levels to get help and technical support.(ii) Create and provide FS-specific training opportunities for the FME platform applications. No centralized entity for information gathering/sharing (CIO updates, users providing feedback, etc.) and collaborative support related to FME; There is frequently either confusion or lack of knowledge amongst FS FME users about FS-specific best practices, guidance, support options, usage parameters, etc., related to FME-related business cases.; All FS Geospatial staff using or seeking to use FME; (i) Consider fostering a group, like the ArcGIS Pro Working Group or the Tableau Community of Practice, to allow for more cross-pollination and collaboration amongst the FS FME user group. (ii) Create and deploy a SharePoint site, or some comparable hub-like location, to host information specific to FME guidance and updates within the agency. Future enterprise system updates have potential to break business served by FME tools; Changes to enterprise infrastructure, such as the Oracle to PostGreSQL migration, have had, and will continue to have rippling effects on existing FME workspaces. There has been limited communication to FS users on how to proactively address these effects.; FS Geospatial Users currently developing and maintaining FME workspaces and workflows; (i) Ensure all information pertinent to known upcoming changes is shared widely with FS FME user community. (ii) Incorporate an FME component into plans for changes in enterprise applications. Out of sync update/versioning schedule; Safe Software updates FME on a 1-year cycle, while the FS updates FME applications every 2 years. This out-of-sync updating schedule leads to a general disconnect, and creates problems with support for bugs in older versions, access to the most-up-to-date functionality, etc.; All FS Geospatial staff using and seeking to use FME; (i) Update FME Desktop and Server software on an annual basis.

Administrative Information

The following slides supply information about software acquisition, licensing, user spin-up and training options, and 508 compliance considerations.

Versioning and the Road Ahead

FME is a mature software that updates roughly once or twice a year with minor releases. The FS generally adopts a new version every two years.

Notable updates in the coming-soon FME 2022 release:

  • Compare 2 workspaces, side-by-side, to assess changes made
  • Merge workspaces
  • Schema Scanner now available (helps manage active metadata and detect schema shift)
  • Enhanced EsriReprojector transformer; now supports vertical and time-dependent coordinate systems
  • Pertinent new formats supported: (1) OGC Catalogue Service for the Web, and (2) SpatiaLite [GDAL and FDO] now centralized into the 'SpatiaLite' format
  • Webhook that allows for processing web form submissions in real-time

A forthcoming FS-specific concern is the transition from Oracle to PostGreSQL and how that will effect existing FME workspaces. The CIO has reported that Esri is working with PostGreSQL to help ensure the transition will be fairly seamless. There is an additional concern about potential syntax issues depending on what 'flavor' of SQL is used in the new PostGreSQL environment. The CIO also reported that there is a tool that will be stood up with the new PostGreSQL build to address FME scripts/workspaces with SQL code and provide clean-up opportunities.

For more specifics on the most recent road map, please visit this  Safe Sotware Products webpage .

Geospatial Technology and Applications Center (GTAC), Resource Applications and Technology Implementation (RATI) unit

 Tim Love  -- Senior Geospatial Coordinator (Author and Point of Contact)

Dalinda Damm -- Senior Geospatial Analyst (Author)

Distribution of jobs run in FME Server; ETL_PROD is mainly EDW jobs

A portion of a job list run nightly by FME Server

An excerpt of the Matrix View of the FME Integrations Gallery