Feature Manipulation Engine (FME) Geospatial Review

The Resource Applications and Technology Implementation (RATI) unit at the Geospatial Technology and Applications Center (GTAC) completed an assessment of the geospatial capabilities and US Forest Service (USFS) use cases related to the Feature Manipulation Engine (FME) software.

OUTLINE OF CONTENT:

Navigational links are included at the top of this Story Map for quicker routing to specific sections.

1. Introductory Information
2. Geospatial Capabilities
3. Use Cases
4. Data System Interactions
5. Application Interactions
6. Issues and Recommendations
7. Administrative Information
8. Versioning and the Road Ahead

To browse through all the components of the FME platform, click the buttons to the left and right of the following interactive graphic to navigate through the pages.

GIS was a starting point for the development of FME; thus, it's foundational ties to, and power in, the geospatial realm. Not only does FME support many geospatial (spatial and tabular) formats, it has many out-of-the-box transformers that provide a wide variety of geoprocessing capabilities. Further, FME functionality can be integrated into Esri's ArcGIS Pro or ArcMap via the Data Interoperability extension.

These topics are covered in the following set of slides.

• Data Management and Compatibility
• Data QA/QC
• Data Monitoring
• Cartographic Uses
• Metadata
• ArcGIS Data Interoperability Extension
• Ties to Web GIS Applications
• Working with 'Big Data'
• Limitations
• Performance and User Experience

This section reviews many existing FS-specific use cases, as well as discusses things to consider related to transitioning workflows within FME.

Transitioning Workflows: Development to Maintenance

There are 2 levels of transition to consider when managing an FME workflow: (1) transitioning an existing workflow to an FME Workbench workspace (either creating new or using an existing entity like an ArcGIS ModelBuilder model); and (2) transitioning a workspace in FME Workbench to FME Server to be run manually and/or automated.

Regarding the second point, there is a need for an FME-advanced user to intially develop a workspace (FMW file) for a process in FME Workbench. Afterwards, that workspace can be handed over to a basic user to be run via FME Server with little expertise. Parameters can be set up to allow for a user interface tool, similar to ArcGIS ModelBuilder. Automation may be enabled, allowing the process to be very hands-off within FME Server. However, maintenance of the foundational FME workspace typically requires attention of an advanced user.

To fully take advantage of FME as a geospatial tool, more expertise is needed in the FS to sustain the development and maintenance required for the agency's business needs.

NRM: Must consider both the NRM Legacy (IWebDB) and NRM ProdB (NextGen) environments. NRM Legacy uses SDO as the data storage type and utilizes SDE formats for transactional components. NRM ProdB uses Oracle Spatial data formats; SDO only. All of these data types are supported by FME. Currently, FME is used for some loading of NRM data.

Some NRM datasets utilize custom COM objects (i.e. FACTS, FSVeg, Wildlife, AQS (Aquatic Surveys), Watershed Quality Survey and WITT); these are not supported by ArcGIS Pro. For more information on how to access these NRM Feature Classes through FME, refer to  these instructions (FME Desktopsection) .

VDC: Geodatabase structure in the VDC is SDE data format, using ST storage type. FME supports SDE data. Regarding applications within the VDC, Safe supports current ArcMap and ArcGIS Pro versions pertinent to time of FME release.

EDW: Corporate datasets can be easily accessed from all FME components when developing workspaces. FME Server powers numerous ETLs for uploading and refreshing EDW datasets. Approximately 1600 jobs run each night. There are also some validation and error reporting automations completed by FME Server.

EDAPT: FME Server is currently being used to push text files from the FS enterprise system into the USDA EPAPT environment. This is an automated process, and once text files are migrated, a Python script incorporates them in EDAPT. There are plans to place an FME Server instance directly within EDAPT, which would allow information to be ingested into the USDA EDAPT environment from the FS using a more streamlined ETL process.

For a list of integration opportunities available with FME, visit the  FME Integrations Gallery .

In general, FME behaves the same with a local install of FME or with an instance in Citrix or the VDI environments. Occasionally, FME has a dependency on local installations of certain applications, such as Oracle or Esri products. In the case of a dependency, a local install of the interacting software MUST exist in the environment (i.e. local, Citrix, or VDI) the FME application is being run.

Within the  Matrix View  of the 'FME Integrations Gallery', any data format that requires local installations of interacting software has a note stating 'Requires Additional Software' in the list. An example is pictured below in the excerpt of the FME Integrations Gallery -- note the Esri ArcGIS Layer and Esri ArcGIS Map Document items.

A user must consider whether the version of FME being used is 32-bit (i.e. FME 2020 and earlier) or 64-bit (i.e. FME 2022) and how that relates to the interacting application. If using FME 2020, and need to interact with Esri and Oracle, a 32-bit version of ArcMap and the 32-bit Oracle client must be installed. Further, since FME 2022 is a 64-bit application, it will only function alongside ArcGIS Pro, Esri's 64-bit Desktop GIS program.

Esri/ArcGIS

FME programs can interact with a number of Esri products including ArcMap, ArcGIS Pro, Portal, AGOL and other web applications like Field Maps and Survey123. As just mentioned, the version of FME in use must be considered when interacting with ArcMap and/or ArcGIS Pro.

Common data formats used in the FS, supported by FME, include ArcGIS layer files (.lyr/.lyrx), ArcGIS Map Documents, Esri Shapefiles (.shp), AGOL/Portal/ArcGIS Server Feature Services, ArcInfo Coverage and E00 files, Esri ASCII Grid files, Esri SDE Geodatabase files, Esri File Geodatabase (.gdb), Esri Personal Geodatabase (.mdb), XML Workspace Documents, Esri Mobile Geodatabases, etc.

For more information about interacting with FME and ArcGIS Pro in the VDI environment, please refer to the  ArcGIS Pro SharePoint site –Applications on the VDI Desktop/FME Desktop portion .

ERDAS IMAGINE

FME can interact with ERDAS native data formats such as ER Mapper ERS, RAW, and IMG files.

Additionally FME supports many other commonly used raster-based file types in the FS, such as TIFFs, GeoTIFFs, Cloud Optimized GeoTIFF (COG), and Meta Raster Format (MRF).

For more information,  click here .

Tableau

FME is most commonly used as a precursor tool for Tableau by either assisting with data setup (i.e. create, analyze, simplify) or creating a TDE (Tableau Data Extract) file. FME Server is used to bring data into Tableau automatically. Also, FME 2022 can interact with Tableau Hyper files (not supported in prior versions).

For more information,  click here .

 FME Community Article: How to Write Spatial Data into Tableau with FME 

Power BI

FME does not support PBIX files (native Power BI project files). There is interest, raised in the  FME Community , for a writer specific to PBIX files. Users are still are able to work with FME alongside Power BI by employing readers/writers associated with Excel, CSV, SQL, etc. to update source data. Once the source data is altered, triggers can be set to refresh Power BI data instances.

Oracle

FME supports reading and writing of Oracle data. Common conversions executed in FME: MS Excel to Oracle; Oracle to MS Excel; MS Access to Oracle; Oracle to Esri SHP.

For more information,  click here .

PostgreSQL

Like Oracle, PostgreSQL data can be read and written with FME. Common conversions executed by FME: PostgreSQL to PostGIS; CSV to PostGreSQL; PostGreSQL to Esri SHP; MS Access to PostGreSQL.

For more information,  click here .

Microsoft Applications

FME supports many Microsoft products including SQL Server, Access, Excel, and Azure services. Additionally, it supports several on-premises and cloud solutions such as Amazon Web Services and Dropbox, and basically any web service with an API.

For more information,  click here .

Adobe Products

FME can read and write geospatial PDF documents using the PDF2D reader/writer. There are additional transformers within FME that can extract certain types of data from a PDF, such as the TesseractCaller transormer that can read specific PDF text (consider pulling label information). Other elements can be extracted from geoPDFs such as maps, rasters/images, vector data, text, tables, and metadata.

The following is useful documentation related to FME and Adobe interactions:

•  Adobe Geospatial PDF Writer (Help Documentation) 
•  Adobe Geospatial PDF Reader/Writer (Help Documentation) 
•  Extracting Geospatial Data from PDFs (Blog) 

For more information,  click here .

The following slides supply information about software acquisition, licensing, user spin-up and training options, and 508 compliance considerations.

FME is a mature software that updates roughly once or twice a year with minor releases. The FS generally adopts a new version every two years.

Notable updates in the coming-soon FME 2022 release:

• Compare 2 workspaces, side-by-side, to assess changes made
• Merge workspaces
• Schema Scanner now available (helps manage active metadata and detect schema shift)
• Enhanced EsriReprojector transformer; now supports vertical and time-dependent coordinate systems
• Pertinent new formats supported: (1) OGC Catalogue Service for the Web, and (2) SpatiaLite [GDAL and FDO] now centralized into the 'SpatiaLite' format
• Webhook that allows for processing web form submissions in real-time

A forthcoming FS-specific concern is the transition from Oracle to PostGreSQL and how that will effect existing FME workspaces. The CIO has reported that Esri is working with PostGreSQL to help ensure the transition will be fairly seamless. There is an additional concern about potential syntax issues depending on what 'flavor' of SQL is used in the new PostGreSQL environment. The CIO also reported that there is a tool that will be stood up with the new PostGreSQL build to address FME scripts/workspaces with SQL code and provide clean-up opportunities.

For more specifics on the most recent road map, please visit this  Safe Sotware Products webpage .