Processors for Apache NiFi

Code & Downloads

• GitHub Repository »

Apache NiFi is a data flow platform which helps automate the movement of data between disparate systems. Flexible and secure from inception, NiFi started life as an internal project for the NSA before becoming a part of the Apache community.

Built on Java, NiFi was a great candidate for integration with the MarkLogic Java Client API and DMSDK. Its focus on security and provenance align with our core principles. NiFi's collection of processors, easy to use GUI, and production monitoring capabilities make it even more attractive as a strategy for data orchestration.

MarkLogic is proud to announce its officially supported NiFi processors. As a member of the NiFi community, MarkLogic is leveraging the proven capabilities of NiFi as part of a data flow orchestration ecosystem. You can now ingest data from multiple source systems into MarkLogic using Apache NiFi which can also act as spokes for your data hub.

Currently, we support two processors built for MarkLogic – PutMarkLogic processor, for ingesting data into MarkLogic and the QueryMarkLogic processor, used for querying documents in MarkLogic.

Getting started with NiFi and MarkLogic could not be easier:

• Download and unpack NiFi
  Download the NiFi binaries from http://nifi.apache.org/download.html. Make sure you're on the latest release of NiFi (1.7). Unpack (i.e. unzip) the tar or zip files in a directory of your choice (for example: /abc).
• Download the latest MarkLogic processors from https://github.com/marklogic/nifi/releases
• Place the MarkLogic-specific processor files in the correct directory
  Copy the two .nar files provided by MarkLogic in the zip folder into the lib folder (nifi-1.7.0/lib) of the unpacked NiFi distribution.
• Start NiFi
  Go to the Quickstart guide by Apache NiFi and follow the commands in the Decompress and Launch section. Note that you don't need to follow the decompress instructions. Also, make sure that you are present in the directory of your nifi installation. If not, change your directory using a command which might look like "cd /abc/nifi-1.7.0". Now, you are ready to follow the launch instructions provided in the Quickstart guide's decompress and launch section for your particular environment.
• Run NiFi using your browser
  You can point a web browser at http://localhost:8080/nifi/ to run NiFi. Make sure you're on the latest MarkLogic version (9.0+).

More instructions for getting started are available in GitHub. We also have a video tutorial on MarkLogic and NiFi. We have submitted our processors to the NiFi Community for inclusion in an upcoming release.