motion-html-pipeline

This gem is a port of the html-pipeline gem to RubyMotion, for use on iOS and macOS. Currently synced with html-pipeline release v.2.12.3

GitHub HTML processing filters and utilities. This module includes a small framework for defining DOM based content filters and applying them to user provided content. Read an introduction about this project in this blog post.

• Installation
• Usage
  • Examples
• Filters
  • Disabled Filters
• Documentation
• Extending
  • 3rd Party Extensions
• Instrumenting
• Contributing
  • Contributors

Installation

Add this line to your application's Gemfile:

gem 'motion-html-pipeline'

and to your Rakefile

require 'motion-html-pipeline'

And then execute:

$ bundle

Usage

This library provides a handful of chain-able HTML filters to transform user content into markup. A filter takes an HTML string or a MotionHTMLPipeline::DocumentFragment, optionally manipulates it, and then outputs the result.

For example, to transform an image URL into an image tag:

filter = MotionHTMLPipeline::Pipeline::ImageFilter.new("http://example.com/test.jpg")
filter.call

would output

<img src="http://example.com/test.jpg" alt=""/>

Filters can be combined into a pipeline which causes each filter to hand its output to the next filter's input. So if you wanted to have content be filtered through the ImageFilter and then wrap it in an <a> tag with a max-width inline style:

pipeline = MotionHTMLPipeline::Pipeline.new([
  MotionHTMLPipeline::Pipeline::ImageFilter,
  MotionHTMLPipeline::Pipeline::ImageMaxWidthFilter
])
result = pipeline.call("http://example.com/test.jpg")
result[:output].to_s

Prints:

<a href="http://example.com/test.jpg" target="_blank"><img src="http://example.com/test.jpg" alt="" style="max-width:100%;"></a>

Some filters take an optional context and/or result hash. These are used to pass around arguments and metadata between filters in a pipeline. For example, with the AbsoluteSourceFilter you can pass in :image_base_url in the context hash:

filter = MotionHTMLPipeline::Pipeline::AbsoluteSourceFilter.new('<img src="/test.jpg">', image_base_url: 'http://example.com')
filter.call

Examples

We define different pipelines for different parts of our app. Here are a few paraphrased snippets to get you started.

Note: these are examples from the original gem since they illustrate how the pipelines can be used. Many of the filters are not currently usable yet, as mentioned in the Filters section below.

# The context hash is how you pass options between different filters.
# See individual filter source for explanation of options.
context = {
  :asset_root => "http://your-domain.com/where/your/images/live/icons",
  :base_url   => "http://your-domain.com"
}

# Pipeline providing sanitization and image hijacking but no mention
# related features.
SimplePipeline = Pipeline.new [
  SanitizationFilter,
  TableOfContentsFilter, # add 'name' anchors to all headers and generate toc list
  CamoFilter,
  ImageMaxWidthFilter,
  SyntaxHighlightFilter,
  EmojiFilter,
  AutolinkFilter
], context

# Pipeline used for user provided content on the web
MarkdownPipeline = Pipeline.new [
  MarkdownFilter,
  SanitizationFilter,
  CamoFilter,
  ImageMaxWidthFilter,
  HttpsFilter,
  MentionFilter,
  EmojiFilter,
  SyntaxHighlightFilter
], context.merge(:gfm => true) # enable github formatted markdown


# Define a pipeline based on another pipeline's filters
NonGFMMarkdownPipeline = Pipeline.new(MarkdownPipeline.filters,
  context.merge(:gfm => false))

# Pipelines aren't limited to the web. You can use them for email
# processing also.
HtmlEmailPipeline = Pipeline.new [
  PlainTextInputFilter,
  ImageMaxWidthFilter
], {}

# Just emoji.
EmojiPipeline = Pipeline.new [
  PlainTextInputFilter,
  EmojiFilter
], context

Filters

• AbsoluteSourceFilter - replace relative image urls with fully qualified versions
• HttpsFilter - HTML Filter for replacing http github urls with https versions.
• ImageMaxWidthFilter - link to full size image for large images

Disabled Filters

Several of the standard filters, such as AutolinkFilter and EmojiFilter, are initially disabled, as they rely on other Ruby gems that don't have RubyMotion equivalents. Please feel free to submit a pull request that enables any of them.

• MentionFilter - replace @user mentions with links
• TeamMentionFilter - replace @org/team mentions with links
• AutolinkFilter - auto_linking urls in HTML
• CamoFilter - replace http image urls with camo-fied https versions
• EmailReplyFilter - util filter for working with emails
• EmojiFilter - everyone loves emoji!
• MarkdownFilter - convert markdown to html
• PlainTextInputFilter - html escape text and wrap the result in a div
• SanitizationFilter - whitelist sanitize user markup
• SyntaxHighlightFilter - code syntax highlighter
• TableOfContentsFilter - anchor headings with name attributes and generate Table of Contents html unordered list linking headings

Documentation

Full reference documentation for the original html-pipeline gem can be found here.

Extending

To write a custom filter, you need a class with a call method that inherits from MotionHTMLPipeline::Pipeline::Filter.

For example this filter adds a base url to images that are root relative:

class RootRelativeFilter < MotionHTMLPipeline::Pipeline::Filter

  def call
    doc.css("img").each do |img|
      next if img['src'].nil?

      src = img['src'].strip

      if src.start_with? '/'
        base_url   = NSURL.URLWithString(context[:base_url])
        img["src"] = NSURL.URLWithString(src, relativeToURL: base_url).absoluteString
      end
    end

    doc
  end

end

Now this filter can be used in a pipeline:

Pipeline.new [ RootRelativeFilter ], { base_url: 'http://somehost.com' }

We use HTMLKit for document parsing in MotionHTMLPipeline::DocumentFragment.

3rd Party Extensions

Many people have built their own filters for html-pipeline. Although these have not been converted to run with RubyMotion, most of them should be easy convert.

Here are some extensions people have built:

• html-pipeline-asciidoc_filter
• jekyll-html-pipeline
• nanoc-html-pipeline
• html-pipeline-bitly
• html-pipeline-cite
• tilt-html-pipeline
• html-pipeline-wiki-link' - WikiMedia-style wiki links
• task_list - GitHub flavor Markdown Task List
• html-pipeline-nico_link - An HTML::Pipeline filter for niconico description links
• html-pipeline-gitlab - This gem implements various filters for html-pipeline used by GitLab
• html-pipeline-youtube - An HTML::Pipeline filter for YouTube links
• html-pipeline-flickr - An HTML::Pipeline filter for Flickr links
• html-pipeline-vimeo - An HTML::Pipeline filter for Vimeo links
• html-pipeline-hashtag - An HTML::Pipeline filter for hashtags
• html-pipeline-linkify_github - An HTML::Pipeline filter to autolink GitHub urls
• html-pipeline-redcarpet_filter - Render Markdown source text into Markdown HTML using Redcarpet
• html-pipeline-typogruby_filter - Add Typogruby text filters to your HTML::Pipeline
• korgi - HTML::Pipeline filters for links to Rails resources

Instrumenting

Although instrumenting was ported, it has not been used real-world, and may not work properly at this time.

Filters and Pipelines can be set up to be instrumented when called. The pipeline must be setup with an ActiveSupport::Notifications compatible service object and a name. New pipeline objects will default to the MotionHTMLPipeline::Pipeline.default_instrumentation_service object.

# the AS::Notifications-compatible service object
service = ActiveSupport::Notifications

# instrument a specific pipeline
pipeline = MotionHTMLPipeline::Pipeline.new [MarkdownFilter], context
pipeline.setup_instrumentation "MarkdownPipeline", service

# or set default instrumentation service for all new pipelines
MotionHTMLPipeline::Pipeline.default_instrumentation_service = service
pipeline = HTML::Pipeline.new [MarkdownFilter], context
pipeline.setup_instrumentation "MarkdownPipeline"

Filters are instrumented when they are run through the pipeline. A call_filter.html_pipeline event is published once the filter finishes. The payload should include the filter name. Each filter will trigger its own instrumentation call.

service.subscribe "call_filter.html_pipeline" do |event, start, ending, transaction_id, payload|
  payload[:pipeline] #=> "MarkdownPipeline", set with `setup_instrumentation`
  payload[:filter] #=> "MarkdownFilter"
  payload[:context] #=> context Hash
  payload[:result] #=> instance of result class
  payload[:result][:output] #=> output HTML String or Nokogiri::DocumentFragment
end

The full pipeline is also instrumented:

service.subscribe "call_pipeline.html_pipeline" do |event, start, ending, transaction_id, payload|
  payload[:pipeline] #=> "MarkdownPipeline", set with `setup_instrumentation`
  payload[:filters] #=> ["MarkdownFilter"]
  payload[:doc] #=> HTML String or Nokogiri::DocumentFragment
  payload[:context] #=> context Hash
  payload[:result] #=> instance of result class
  payload[:result][:output] #=> output HTML String or Nokogiri::DocumentFragment
end

FAQ

I have left this FAQ item here for when we get the PlainTextInputFilter working.

1. Why doesn't my pipeline work when there's no root element in the document?

To make a pipeline work on a plain text document, put the PlainTextInputFilter at the beginning of your pipeline. This will wrap the content in a div so the filters have a root element to work with. If you're passing in an HTML fragment, but it doesn't have a root element, you can wrap the content in a div yourself. For example:

EmojiPipeline = Pipeline.new [
  PlainTextInputFilter,  # <- Wraps input in a div and escapes html tags
  EmojiFilter
], context

plain_text = "Gutentag! :wave:"
EmojiPipeline.call(plain_text)

html_fragment = "This is outside of an html element, but <strong>this isn't. :+1:</strong>"
EmojiPipeline.call("<div>#{html_fragment}</div>") # <- Wrap your own html fragments to avoid escaping

Contributing

Contributors

Thanks to all of these contributors, who have made the original gem possible.