A User Guide For The Craft CMS Feed Me Plugin

Feed Me Plugin

Overview of the Plugin's Capabilities

The Feed Me plugin is a must-have for any Craft CMS website that needs to import and manage feeds. It provides a suite of advanced yet easy-to-use tools for importing all kinds of feeds, scheduling imports, filtering data, and mapping feed elements to Craft entries.

At its core, Feed Me allows you to import RSS, Atom, JSON, and XML feeds into Craft with just a few clicks. Simply install the plugin, add your feed URL, configure any options, and let Feed Me work its magic. It can import the full contents of feeds ranging from blog posts to products to podcast episodes.

Once imported, you get full control over the feed data. Feed filters give you precision over what gets imported, letting you narrow down to specific feed elements. The field mapper allows you to map feed data to Craft entry fields and categories for a seamless transfer. Imports can be scheduled to run automatically at any interval, ensuring your site always has the latest content.

For developers, Feed Me offers a robust API and the ability to create advanced workflows around imports. It saves huge amounts of time compared to parsing feeds in Twig or building custom integrations. The plugin handles all the heavy lifting so developers can focus on other projects.

Overall, if you need to get external content into Craft in a structured way, Feed Me is the go-to solution. It takes the pain out of imports and unlocks the full power of feeds for your website.

Use Cases and Benefits of Using Feed Me

Feed Me shines for a range of websites and use cases where importing external content provides value. For example, blogs can automatically pull in posts from contributors' Medium or Tumblr feeds. Ecommerce sites can sync product data from suppliers via XML or JSON feeds. News outlets and aggregators can collate articles through RSS.

In all these cases, Feed Me offers significant benefits beyond what you could achieve with Craft's native feed support:

• Automated content flow: Imports can be fully automated on any schedule. This keeps content fresh and reduces manual overhead for developers/editors.

• Customised data mapping: The field mapper gives granular control over how feed elements map to Craft entries and fields. This ensures seamless integration with existing content structures.

• Robust import options: Features like filters, rate limiting, and import logs allow fine-tuned control over imports. This provides flexibility to handle diverse feeds.

• Time and cost savings: For developers, Feed Me reduces the need for custom integrations and feed parsing. For publishers, it automates content aggregation tasks. This efficiency boost saves time and money.

• Reliable performance: The plugin is optimised to handle large feeds without performance issues. Developers can trigger imports via CLI or APIs for additional flexibility.

In summary, Feed Me supercharges your content pipeline by enabling automated, customisable feed imports unmatched by out-of-the-box Craft feeds. The time and effort savings quickly add up for both developers and content teams.

Comparison to Native Craft Feeds

Compared to Craft's native feed capabilities, Feed Me offers advanced functionality in a user-friendly package.

Craft CMS provides basic support for RSS and Atom feeds via its native Feed Me class. However, it lacks key features needed for robust feed integration:

• No import scheduling

• Limited feed formats (only RSS/Atom)

• No filtering or fine-grained mapping

• Difficult to customise and extend

• Manual CLI parsing is required for large feeds

This is where Feed Me plugs the gaps. It brings enterprise-grade feed management to Craft with easy-to-use controls and a flexible API for developers.

Feed Me works seamlessly on top of native Craft feeds. For example, it uses Craft's FeedEntryModel under the hood for imported elements. But it adds several layers of functionality to supercharge imports in ways not possible using Craft alone.

For sites needing powerful and customizable feed integration, Feed Me is a must-have plugin.

While Craft provides basic feed support, Feed Me offers the advanced features required for production deployments. It saves developers from building custom solutions through superior out-of-the-box capabilities.

Installing the Feed Me Plugin

System and Craft CMS Requirements

Before installing Feed Me, ensure your system meets the minimum Craft CMS requirements for compatibility:

• PHP 7.0 or later

• Craft CMS 3.1.0 or later

• At least 128MB of RAM allocated to PHP

• Ability to make outbound HTTP requests

Feed Me requires PHP 7.0 as it utilises modern PHP features and syntax. It is optimised for Craft 3 and will not work on earlier versions. Make sure your server has adequate RAM, as Feed Me can use additional memory when importing large feeds.

The plugin needs to make outbound requests to access and import feeds. If your server has a firewall blocking external requests, you may need to whitelist the Feed Me user agent string.

Provided these requirements are met, Feed Me can be installed on shared, cloud, and local Craft CMS environments. There are no additional prerequisites beyond a standard Craft 3 system. Just ensure it is up-to-date and running PHP 7 or higher.

Installation Methods

There are two ways to install Feed Me - through the official Craft CMS Plugin Store, or by manually uploading the ZIP file:

Plugin Store (recommended):

1. In your Craft control panel, navigate to Settings > Plugins and select "Install" under Feed Me.

2. Click through the installation prompts to complete the process.

The Plugin Store method ensures you'll get notified of and receive all plugin updates automatically.

Manual ZIP upload:

1. Download the latest Feed Me ZIP file from https://plugins.craftcms.com.

2. In your Craft control panel, go to Settings > Plugins and click the "Upload plugin" button.

3. Upload the Feed Me ZIP file.

With manual uploads, you'll need to repeat the process whenever new versions of Feed Me are released to get updates.

Verifying Installation Success

Once installation is complete, you can verify Feed Me is properly installed and ready to use:

• A "Feed Me" item will now appear in your Craft control panel main navigation. This is where all feed importing tasks are handled.

• Navigate to Settings > Plugins to check Feed Me appears in the list of installed plugins.

• Go to Utilities > System Reports. Feed Me should show in the list of installed plugins, with no errors.

• Check your Craft log files for any Feed Me installation errors.

With the plugin now installed, you can proceed to creating your first feed import. Feed Me does not make any database or config changes during installation, so it is ready for use immediately after a successful install.

If the plugin is not showing up or you see errors, try re-uploading the ZIP file or restarting your Craft instance. This typically resolves any installation issues. You may also need to double check your server meets the PHP and RAM requirements as detailed above.

But in most cases, Feed Me will install seamlessly in just a few clicks. Once installed, you'll have a robust, highly customisable feed importer at your fingertips to supercharge your content workflow.

Configuring Feed Me Settings

General Settings and Configuration

The general settings section in Feed Me offers several ways to control overall plugin behaviour and configure things to your needs. One of the first things to do is enter your purchased pro license key, which unlocks additional features like multi-site support. Enabling debug logging is highly recommended during initial setup and testing, as it provides very detailed log files that help diagnose any issues that crop up during imports.

Another key setting is the schema cache duration - this determines how long Feed Me will cache the schema for each feed source. The default 3600 seconds (1 hour) works well for most use cases, providing a good performance boost while ensuring schema changes are relatively quickly reflected. If using queued importing, you can also set a queue timeout which determines how long jobs can remain queued before being re-run.

Beyond that, settings like date format, title generation, and image caching allow you to customize Feed Me's behaviour for your specific importing needs. The plugin makes it easy to override default settings on a feed-by-feed or even import-by-import basis. Overall the defaults provide a robust starting point suitable for most use cases, but Feed Me offers endless flexibility to tweak settings for your unique requirements.

Managing Feed Sources

Adding feed sources is where the importing magic happens. Under the Feeds menu item, simply click "New Feed" and select your desired feed type - RSS, Atom, JSON or XML. Give your feed a name and enter the source URL. Each feed can have its own configuration settings, so you can enable/disable, set limits, timeouts, date filters and more on a feed level.

When adding feeds, be sure to test with smaller limits and conservative timeouts at first. You can progressively raise these as you confirm everything is importing smoothly. Also check that "Enabled" is selected or else the feed won't run at all.

The key settings are limit, timeout, date filtering, and HTTPS/SSL verification. Limit sets the maximum number of feed items to import per run. Timeout determines how long the Feed Me can take to import before timing out. Date filtering allows you to only import items within a certain date range. Disabling SSL verification can help if you run into cert issues, but ideally, you'll want to leave this on for security.

You can add as many feed sources as needed - the key is to start small and slowly build up sources while troubleshooting any issues. Feed Me will automatically detect feed changes like new and removed items, so you can keep multiple sources in sync easily.

Advanced Feed Settings

Beyond the basics, Feed Me offers a wealth of advanced settings accessible under each feed's "Advanced" tab. One of the most useful is the ability to add custom HTTP headers - this becomes essential for feeds requiring API keys or OAuth for access. You can also set up scheduled importing using crontab expressions - for example, daily syncs of product catalogs or hourly checks for new blog posts.

For data mapping, the "Fallback Fields" setting enables you to populate Craft fields with default values if the feed field is empty. This prevents missing fields on import. The before/after save actions let you trigger custom PHP callbacks during stages of the import process.

Other handy advanced settings include rate limiting, randomization, and custom Guzzle configuration. Rate limiting is great for decreasing server load by throttling imports.

Randomizing the import order can help distribute server load as well. For complete HTTP flexibility, you can pass custom Guzzle handler settings to customize the request handling.

Between the main and advanced settings, Feed Me offers endless options to customize importing exactly for your unique needs. Whether you need only the basics or extremely advanced workflows, the plugin has you covered.

Importing Feeds into Craft CMS

One-Time vs Continuous Importing

Feed Me gives you the choice between one-time and continuous feed importing.

One-time imports grab the feed once and import the current set of items. No previously imported items will be re-imported on subsequent one-time runs. This is good for importing a feed just once.

Continuous imports sync the feed on an ongoing basis. It will import new items while also updating existing ones if they change upstream. This keeps your imported feed content continuously up to date.

In general, continuous importing is the best option for most use cases. One-time can be handy for one-off imports from dynamic sources that change drastically. But for blogs, news, products etc you typically want continuous import to capture updates.

Newly created feeds default to continuous importing. You can toggle between the modes at any time on the feed settings page. Just be aware this will impact how future imports handle existing data.

Immediate vs Scheduled Importing

By default, Feed Me immediately runs a feed import whenever you hit the "Import Feed Now" button. This instantly pulls down the latest feed content.

However, you can also schedule recurring imports using cron expressions. For example, import a product catalog feed every 6 hours, or check a news feed every 15 minutes.

Scheduled importing is configured on the Advanced tab of each feed source. Use standard cron syntax like:

*/20 * * * *

To import every 20 minutes. You can schedule feeds to import as often as needed.

Scheduled imports run in the background per your settings. You can monitor their status and history in the Feed Queue. Scheduled and immediate imports also populate the feed's individual import log for inspection.

For feeds that don't change often, scheduled importing is preferable to reduce resource usage. But for rapidly updating sources, use immediate importing combined with frequent cron schedules.

Managing the Feed Queue

The Feed Queue provides an overview of all your import jobs, both immediate and scheduled. It displays the status of imports that are:

• Pending - Scheduled but not yet run

• Running - Currently importing

• Successful - Completed with no errors

• Failed - Completed but encountered errors

You can drill into each job to inspect details like:

• Feed source

• Import type (one-time or continuous)

• Start time and duration

• Number of new/updated/deleted items

• Full log output

For scheduled imports, regularly checking the queue is a good way to monitor their health. Any failed jobs will be prominently displayed. You can also easily re-run failed imports with a single click.

The queue provides both a high-level overview and granular detail for all importing. It's an invaluable tool for observing how your feeds perform over time. Are they consistently successful? Do they occasionally fail? How long do imports take?

Leverage the queue logs for metrics, troubleshooting, and insight into how Feed Me imports your content over time. For complex importing workflows, it's an essential tool.

Filtering and Transforming Feed Data

Global vs Per Feed Filtering

When it comes to filtering imported feeds, Feed Me provides two powerful options - global filters that apply across all sources, and granular per feed filters unique to each source. Global filters come in handy for sitewide rules, like removing certain categories of content or blocking items from a staging source. Since they apply universally, global filters are great for high-level filtering needs shared by all feeds.

But for more precise control, per feed conditional filters will be the more common choice. These allow crafting specific filters tailored to each feed source. For example, you may want to only import book reviews from one source, while filtering out retweets and duplicates from another.

And for a third feed, you may want to only bring in content from certain geo regions. Per feed filters enable this by honing in on the exact types of items you want from each unique source.

In most cases, a combined approach works best - global filters to establish base rules for all feeds, then additional tight filters per source for targeted importing needs. Together they provide maximum flexibility to import only what you need.

Searching and Filtering with XPath

When working with XML-based feeds like RSS and Atom, XPath queries are incredibly powerful for precision filtering. XPath allows selecting matching elements based on criteria like text content, attribute values, node relationships, and more. For example, you can filter items where the title contains certain text, the category matches a value, the author name starts with a substring, or the view count exceeds a number. The possibilities are endless for surgically targeting just the content you want to import.

Some examples of XPath filters:

//item[contains(@title, 'craftcms')]

Imports only items where the title contains "craftcms".

//entry[author = 'Mark Smith']

Selects entries whose author value equals "Mark Smith".

//post[viewCount > 1000]

Imports posts with over 1000 views.

The important thing is that this filtering happens before any field mapping occurs. So XPath gives you control over the structure of the XML itself, which items are included or excluded for import. To modify data within an item, Field Mapping is used. The combination of XPath filtering then field mapping allows full control over both the shape of the feed and the data mapping.

Removing Elements via Filters

Along with targeting elements to import, XPath filters can also fully remove items from the feed prior to import. This is accomplished using "not" filters and exclusionary criteria.

For example:

//item[not(contains(@category, 'jobs'))]

Would import all items except those with a category of "jobs".

Other techniques like filtering by date or rejecting incomplete items can also help strip down the feed to only the content you need to be imported into Craft. This prevents bloating your Craft database with unnecessary data.

Exclusionary filters are especially helpful for avoiding duplicates on continuous imports, by filtering out previously imported items based on a unique ID or other field.

Overall intelligent feed filtering and removal will keep your imports optimized. Feed Me provides all the tools needed to deeply customize this cleansing process.

Mapping and Structuring Imported Data

Mapping Feed Fields to Craft Entries

One of the most important steps when importing feeds into Craft involves properly mapping the feed data over to corresponding Craft entry fields. Feed Me makes this easy through its drag-and-drop Field Mapper interface. Simply connect the feed field names over to your desired Craft field handles like title and body.

When mapping, be sure to map fields of similar semantics together - for example map the feed's title field to your Craft title field. Don't incorrectly map unrelated fields together. Also, use the actual Craft field handles like postDate rather than their display labels. For any unneeded fields, you can just leave them unmapped so they are ignored on import.

Feed Me will check this field mapping for each imported item and populate the Craft entry accordingly. So taking care to define the mappings correctly up front is crucial to ensure your imported content ends up in the right places.

Over time, make sure to consult the feed provider's documentation to check for any field changes. Adjust your mappings accordingly if needed to avoid mismatches. Also, leverage fallback fields to provide default values for any blank fields. Overall, clean and accurate feed field mapping will keep your imported data structured perfectly for your templates.

Using Categories and Category Rules

In addition to basic field mapping, Feed Me also provides tools to automatically categorize content on import. Under the Category Mapper tab, you can map feed categories directly over to corresponding Craft categories. For example, map the feed's "News" category over to a Craft category called "News".

This alone handles simple one-to-one category mapping. But you can also create category rules for more advanced logic, like "if feed category equals X, assign Craft category Y". Finally, a default category can be set to handle any unmapped feed categories.

For example:

Feed Category "News" => Craft Category "News"

Feed Category "Sports" => Craft Category "Sports"

Default => Craft Category "Uncategorized"

Used together, field mapping and category mapping ensure imported content is automatically structured just the way you need it for managing, editing, and displaying entries.

Advanced Mapping with JSON and XML

For more complex JSON-based feeds, Feed Me supports dot notation mapping for nested attributes like:

item.blog.postDate => postDate

With XML feeds, namespace prefixes can be used to prevent ambiguity between overlapping field names:

content:encoded => body

media:content => image

This unlocks the ability to map deeply nested or namespaced data from even sophisticated feeds into Craft's entry fields.

Overall, Feed Me delivers all the tools needed to map external feed data into Craft entries with whatever structure fits your content model. Even very complex import sources can be tamed for seamless usage in your templates.

Feed Import Optimization and Performance

Optimizing Memory Usage for Large Feeds

When importing feeds with very large numbers of items, memory usage can quickly spike leading to exhausting your server's available RAM. This results in PHP memory errors and failed incomplete imports. A few techniques can help optimize memory when working with massive feeds:

First, ensure you're running PHP 7+ in 64 bit mode with ample memory allocated - the more RAM the better. At the PHP level, bump up limits in php.ini for memory_limit and max_execution_time. Inside Craft, also increase the memoryLimit and maxExecutionTime config settings. Together this will prevent PHP from running out of memory or reaching timeout limits.

Strategically filtering the feed to remove unneeded elements will also help reduce the number of items being processed, keeping memory requirements lower. Importing in smaller, targeted batches using Feed Me's limit setting can also distribute the load over time rather than one huge single import.

There are also some Feed Me specific optimizations that can assist. Enabling the "Disable GC" option will restrict PHP's garbage collection during imports to avoid memory spikes. For the largest feeds, queued importing that fully processes in the background can be very helpful as well.

With adequate resources, selective importing, and Feed Me's built-in optimizations, even feeds with millions of items can be processed efficiently. It's all about finding the optimal batch size and import workflow for your server's memory capabilities when working at scale.

Limiting API Requests Through Field Mapping

One important factor in Feed Me performance relates to Craft's API request requirements. Each entry being saved generates a request to the API, which can pile up with thousands of items. To help:

First, be strategic with your field mappings - only bring in the data you actually need for your templates and content model. Avoid bulky mappings for unimportant fields as this creates unnecessary API overhead. At the template level, fields you won't use should be set to propagate=false in your field layouts.

For categories, map just one or two important, high-level groupings rather than dozens of granular categories that will bloat requests. You can also leverage Feed Me's setting to optimize category saves - this combines them into a single request.

Finally, using a control panel plugin to selectively disable search indexing during import helps take out CRaft's indexing requests from the process for a big boost.

By importing leanly and mapping only essential data, you can significantly reduce API requests allowing much faster feed processing.

Using Throttling to Avoid Timeouts

For very large feeds, enable Feed Me's throttling setting to carefully control the import speed and avoid server timeouts or overload. Start with a smaller throttle value like 50ms to insert a slight delay between saving each item.

Test progressively higher values while assessing import duration and monitoring for any timeout errors. The optimal throttle amount will balance performance and timeout avoidance without being too conservative.

Monitor your server load and resources as you adjust the throttling to dial in the best value. With throttling, you can achieve successful imports of even millions of items that would normally time out.

When combined with optimization techniques like batching, filtering, and queuing, Feed Me provides every tool needed to handle feeds at a massive scale.

Troubleshooting Guide

Fixing Common XML Parsing Errors

Getting an XML feed to parse properly can be frustrating when you encounter errors during import. There are a few troubleshooting techniques that can help get things working again:

First, validate the XML feed using an online validator tool to check for malformed markup that may be triggering issues. Resolve any structural problems identified. Also inspect the feed for non-standard XML entities and unusual character encoding that Feed Me may not be handling properly. Adjust the parsing settings or filter out problem characters.

For API responses, carefully check the headers for clues - are authentication credentials missing causing authorization issues? Enable Feed Me's debugging logs to capture the raw response being received, which often reveals the problem.

If the default XML parser continues to fail, try switching to an alternate option like SimpleXML in case that handles the feed's structure better. As a last resort, pre-processing the XML in your own script into completely valid markup can resolve parsing problems before handing to Feed Me.

With patience and a focus on common XML formatting and encoding pitfalls, the specific issue preventing parsing can usually be uncovered and addressed.

Debugging Feeds Failing to Update

It can be puzzling when a feed that was previously imported without issue suddenly stops updating in Feed Me. Some techniques to debug:

Start by reviewing the feed URL - did it change or is the site currently down? Check for any feed markup changes as well, like new XML namespaces or element names that need updated field mappings. See if the provider introduced new API requirements like headers for authentication, throttling, or caching.

Use a feed validator to check for XML syntax problems that may be choking up Feed Me. Enable Feed Me's debug logging to inspect what's received from the feed - are items parsed successfully but failing on import for some reason?

Examining other signs like timeout errors in the logs can provide direction too. With systematic and analytical debugging, the root cause of most feed issues can be uncovered. Don't assume the problem lies with Feed Me as an issue in the feed provider or authentication often causes feeds to unexpectedly stop updating.

Identifying and Resolving Timeout Issues

Performance timeouts in Feed Me typically originate from a few common sources. The first place to check is server-level resource limits - increase PHP memory, execution time, and Craft's config settings to allow longer processing.

For very large feeds, try filtering items before import to reduce load. Smaller batches and throttling can help as well. If calling an external API, see if headers indicate throttling or request limits. Introduce delays between requests.

To pinpoint the exact timeout source, take note if it fails during fetching, parsing, or saving entries. Failures fetching point to network issues, parsing suggests encoding/memory problems, while saving indicates Craft API limits.

Matching the failure stage to potential solutions based on Feed Me's logs enables targeted troubleshooting. With the right debugging approach, nearly any timeout or performance issue can be optimized for smooth and speedy feed imports.